not much to it. If you use the AsExtreme software, you can apply the new OC changes on the fly. Make sure you're running CPUZ and RealTemp also.
Open up the app, click on "overclocking" tab. Now you can change the multiplier to 40 to reach 4Ghz. Click Save. Use Prime95 to stress your CPU and check CPUZ for the changes.
Prime95 should also be used to check stability. If your computer restarts, or BSOD, then you need to do more stabilizing work.
edit: If you have the 2600 non-k like myself, you're pretty limited on how far you can o/c. The max multiplier you can get to is around x39. I've tried to set it to x40 but it was a no go. From here, you have to use BCLK. I would increment by .5 every time to test it out (100.5->101->101.5). Smaller increments as you reach your stability ceiling. 4.0GHz is around BLCK=102.67 Mult=x39.