Have you tried running the artifact test in ATI Tool? I once voltmodded my 7900GT. First, the vcore mod with conductive ink. Worked like a charm, gave core clock a massive boost.
Then, I did the vmem pencil mod. That screwed my card up. It still works, but I somehow did some damage to it. ATI Tool would detect artifacts at every clock speed, and overclocking it above stock clocks put artifacts in my games. So, that may have happened to you.