Sadly a swap file will still be slow, even with RAIDed SSD's. Even if the sequential read/writes come close to being 50% of RAM you will never achieve the same performance with random reads and writes, which is why RAM is indespensible. But you are right, 2TB RAM is bloody expensive. The guys I know who work in DNA sequencing always complain about RAM prices, despite RAM being really cheap.
I know. But it would still be faster than runs crashing or having them run on one or two cores out of 32 because otherwise it tops out RAM.
It's mostly sequential read anyway - large files (1-5GB each) which are read as a block, processed and written out as much smaller results (normally to a different drive). Extractions from further processing of these results can give files 300GB or more... which again are read sequentially for the most part. It's a pretty specialist requirement, I admit, but it may work out superior to insane amounts of RAM. To be honest I'd have to try it to decide whether it would be a valid option or not. I can't/won't do it out of my own pocket, though, and I doubt my boss will drop that much on an "it might work, it might not", although it is possible.
RAM is only cheap when you want <256GB. It's "OK" price wise at the 256-512GB level, and higher than that you start bleeding money.
Yep server memory is suprisingly affordable. I am looking at buying 192 gigs DDR3 ECC ram for roughly $1 per GB. DDR4 isn't too much pricier than that for the same capacity. But when you start looking into high density sticks the prices just hurt to look at.