And the biggest problem? They are single socket only. That means you have more physical servers to manage and keep track of. In addition, you spend much more time passing data and instructions between nodes vs. between sockets. Message passing at the socket level is an order of magnitude faster.
Combine this with some 4P platforms and you are looking at 48 cores in one platform today and 64 cores in that same platform in a few more months.
If you had to process a scalable workload with lots of interdependencies across 64 cores would you rather do it with 1 box, all local or on 32 boxes, all over a fabric interconnect.
Maybe I am just a simple guy, but that just sounds like a bad idea. Especially if they are using Gb as their interconnect. Maybe it would be faster with infiniband, but the cost of 32 host adapters and switches would kill you.
For plain processing power you can't beat the 4P Opteron. Especially when you can configure 4x 6168 CPUs, 32GB, a drive and a 3 year warranty for only $7500. To do 24 nodes against today's platform you'd need those nodes to be $234 each. And your interconnect cost would have to be $0.
Trust me, I have done this math more times than anyone else here - the numbers just don't work. Unless an Atom core is ~3-4 the performance of an Opteron core. Anyone want to take a bet on that one?