All I know is that most NV40 core based GPU can be unlocked to 6800GT level (although some pipelines may be laser cut or damaged).
This chart shows most of them (but no XT or NV41). According to it, the 6800GS AGP is NV40, so it's probably unlockable.
Edit: from Rivatuner.
|NV40 graphics processor's internal configuration is rather flexible and amount of active pixel and vertex units can be easily controlled by special GPU units configuration register. State of each pixel and vertex unit is mapped to the corresponding bit in the configuration register, logical 1 in this bit enables mapped unit, logical 0 - disables unit and prevents graphics processor from using it. This technique allows creating 16pp,6vp / 12pp,5vp / 8pp,4vp boards based on NV40 graphics processor core. So any NV40 graphics processor can be easily reprogrammed to use 1-4 pixel units (or 4-16 pixel pipelines) and 1-6 vertex units via masking the corresponding bits in the configuration register.
|First, you should look at the graphics processor's revision. If it is NV43 revision A4 or newer, then I've to dissappoint you - NVIDIA seem to progress with their hardware protections, so currently these chips are completely unmoddable. You won't be able to unlock hardware masked pipelines, as well as you will not be able to unlock professional capabilities on it. The same applies to all revisions of NV41, NV42, NV44 and NV47 graphics processors.
So 6800GS PCIe (nv42) is unmoddable. And anurevolution, I checked the Rivatuner FAQ, and it says that if the card is described as "NV41(A1,8pp,4vp)", then the mod wasn't successful. The custom settings are what Rivatuner is trying to force the GPU to. Here:
|Q: I have 6800 NU and right after installation RivaTuner displayed 12x1,5vp configuration on the main tab. In the NVStrap tab I saw that I have pixel unit 1 and vertex unit 3 disabled, so I ticked them and restarted PC. After reboot I still see 12pp,5vp on the main tab, however NVStrap tab shows 16pp,6vp configuration. What's going on?
A: NVStrap's status bar (displayed in the upper part of Custom graphics processor configuration panel) shows you default, current and target configurations. 16pp,6vp configuration you see is the target configuration, i.e. configuration you've asked the NVStrap to set after the next reboot. The value you should look at is the current configuration, showing which pixel and vertex units are actually currently enabled. Both context help and FAQ clearly state that the units can be hardware masked, so current and target configurations can be different due to your attempts to activate hardware masked units. So if you see differences in target and current configurations after reboot and see "disabled" state near the unit you tried to enable - it is protected by hardware mask. In this case you may either try to enable it using "Allow enabling hardware masked units" or forget about softmodding, if you don't want to risk with enabling a unit, which can be potentially damaged.
I guess this resolves the OP's issue...