Did you try to OC the RAM and CPU seperatly first?
The CPU... set the RAM to a much lower speed like 133 (DDR266) to underclock it that way it wont effect the CPU's overclocking.
Increase the vcore (CPU voltage) but keep the full load temp below 50c.
Increase the HTT and check for stability... once you hit an instable OC back down a few MHz on the HTT.
The RAM... set the CPU's multiplyer 3x below default so the CPU wont effect the RAM's OCing. Or in your case reduce from 10x to 7x
Set the vdimm (RAM voltage) to the recommended max whick should be 2.9v
The RAM utilizes an HT
factor and the default is ... 5 or
an HTT of 201-250 use 4x
251-333 use 3x
Increasing the HTT and reducing the HT will require testing for instability ...
The memtest86 Quick test is at the botton of the below article "Memtest".
Once you know each stable MAX your OCing should come together much easier.