There's not much you can do about it. There's not much point turning it off though. It's a normal thing. It clocks down when you're not running graphically intensive programs so that it consumes less power. For comparison, my Hawk clocks down to 51MHz when it's not in use.
Trust me on this one, just leave it. It'll be fine.