I'm a pretty experienced ex-linux sys admin/engineer; I'm going to 2nd going with CentOS for your test bed environment, if working with clustering I would check out ipvsadm/LVS(Linux Virtual Server) http://www.linuxvirtualserver.org/
I've used both RHEL and CentOS, i you have the budget for RHEL liscense's AND the support then I'd say go RHEL, however without the premium support package, the support is nothing to write home about.
CentOS is the free community version of RHEL; it's the community standard in the web hosting industry atm.
Fedora is (*imho*) a test bed for Red Hat and not as stable as CentOS.
RHEL/CentOS should be pretty straight forward if you have worked with AIX and Solaris (except for the whole intel architecture ;p) .....the System V Init config is a PITA; prefer BSD style init system.
If you want a resume builder, big companies use RHEL, all smaller companies/startups/people that know what they are doing, will usually use CentOS or a mix of CentOS/RHEL.
Perhaps some more details into how the mathematics program operates and we could answer some of your questions better, at first I read that you wanted a cluster but then I'm hearing about granular changes in multi-threading to improve your program's performance; at this point, one has to ask--are you willing to code? What is your skill level at?
Hope we helped so far but; moar details plz!