The layout of the ASRock Z77E shouldn't worry you if you go with an all in one cooler option. You may have to cut a tiny bit of plastic off of the backplate to clear the mSATA slot on the underside of the motherboard, but it isn't a problem. I did it lol. I agree with the others, drop the 7200RPM hard drive and pick up a SSD. You'll appreciate the speed boost over the mechanical hard drive. I also agree, at least get a 660ti or a hd7950. I would lean towards a HD7950, just because they are beasts at overclocking which makes the bang for the buck amazing. Just make sure to get a card with the voltages unlocked. I know my MSI R7950 is unlocked and overclocks past the point of a stock HD7970. If it takes another paycheck to save for the upgrades in the GPU and SSD, it's well worth the wait. A few weeks is worth the better hardware that will handle games further into the future. My layout in my Prodigy may give you an idea of how to build yours. It's basic, nothing insane like you see in the Prodigy case owners thread.
I do not have the H80...I'm using Thermaltake's H2O Water Pro cooler. I'm just showing to give you an idea of clearance.
Edit: Oh yeah, some of the pics are old. I'm currently using some of that awesome Samsung magic green RAM. Stock was 1600 and I overclocked it to 2133MHz and timings to 9-9-9-22 without touching the voltages. Stuff is pretty good lol. I'm going to see how well I can get the OCs with this board. It seems to handle very well.
As for overclocking guides for this motherboard, go to this thread:
The layout for the BIOS screen is a bit different, but you can make it through.
For PSUs, a 600 watt would be PLENTY for this system. I have a 600W and it covers everything great and has TONS of headroom for overclocking. Just make sure to get the most modular one you can get and that it has a good, single, 12V rail.Edited by mwayne5 - 10/16/12 at 7:05pm