Which upgrade should I do?

the GPU.

though, why not go for a Phenom II chip? they run cooler, and are generally better than the Phenom I chips. and i've seen Phenom II X4 940's (DDR2 compatible) for $100 on here. epic value

im trying to stay on a low budget and my socket am2 will only support phenom x4 9650 or a phenom II x2 3.1 callisto black ediition

I used to have an 8600gt before I switched to my gtx260. HUGE difference in my games, and I'm pretty sure the 5770 is roughly equivalent to a gtx260. If you could oc your current proc, it wouldn't bottleneck the 5770 as bad.

then look to get a Phenom II X2 550. maybe you can unlock?

but still, GPU first

How about you save for a new system?

Your PSU is poor quality (Main thing I'm concerned about)
http://www.hardwaresecrets.com/article/557

Your motherboard is also not able to handle higher wattage CPU's because of the small VRM and does not appear to be a very good overclocker

If that's not possible, go for the GPU upgrade. The 5770 uses very little power, whereas the Phenom 9650 would add considerable power draw and heat.