As others have said, there's nothing wrong with putting a micro-ATX board in an ATX case.
As far as air coolers are concerned, the two measurements you need to be concerned about are the maximum height cooler that will fit in your case and the height of your memory. Your Nanoxia case allows for coolers up to 185mm wide and I don't know the height of your memory unless you give us specific model of memory you're talking about (HyperX Blue, HyperX Beast, etc.) Almost any air cooler will fit within your case dimensions, but you'll need to find out the height of your memory before we can make a good recommendation.
Water cooling is also an option. There are a number of AIO (All-in-One) water cooling kits you can use, and generally which ones you can use are dictated by the fan mounting points on the back and top of your case.
There are also high-end "custom loop" water cooling systems. These are expensive and not recommended for the first-time overclocker.
Let me try to summarize your cooling options:
Air cooling ($15 to $90):
Pros: Most cost-effective coolers (most cooling for the money). Most reliable. Quieter than similarly-performing AIO water coolers.
Cons: For small to medium coolers, none. However, large air coolers have a few disadvantages. Large air coolers may not fit in all cases and with all memory configurations. Large air coolers can put dangerous stress on the motherboard, especially while the computer is being moved. Large air coolers can also significantly impede airflow, increasing case temperatures.
All-in-One (AIO) water cooling ($50 to $180):
Pros: Many AIO systems fit where large air coolers do not. Case width and memory height are usually of no concern. The best AIO coolers outperform the best air coolers. AIO coolers allow for a cleaner look inside the case, better airflow, and easier system maintenance.
Cons: More complexity than air cooling makes AIO coolers less reliable than air coolers. Pump failure is a possibility, and leakage, which may destroy components, is rare but possible. System lifespan is limited to a few years because of evaporation. Most units are louder and less cost-effective than similarly-performing air counterparts. Larger units have large radiators that may not fit in certain cases.
"Custom loop" water cooling ($150 and up):
Pros: Superior cooling performance. Case width and memory height are usually of no concern. Allow for good airflow inside the case. Numerous customization options. Replaceable system parts. Well-designed systems are extremely quiet. May be expanded to cool video card(s) and motherboard chipset in addition to CPU. Customization and planning leads to pride of ownership for many owners.
Cons: Expensive. Value of spending large amounts of money on cooling is questionable to many. Typically require medium to large cases with appropriate mounting points for radiators. Complex selection process and setup. Complexity makes these less reliable than air coolers (but more reliable than most AIO coolers, because custom systems tend to use higher-quality parts). Incorrect installation may lead to leakage and equipment damage. Regular maintenance, especially fluid level checks, are required. Not recommended for beginners.
I think that does it. Let us know what your budget and goals are and we can make some recommendations. If you're unsure, I recommend starting with a medium-sized air cooler in the $25-$60 range.
Edited by synge - 5/16/14 at 8:00am