So after a little bit of messing around opening up my case I've determined that at idle, as suspected, the spinning hard drives are hands down the loudest thing in the case.
One that was on happened to turn off due to inactivity as I was listening to the case and the thing went from vibrating the whole case a bit and being clearly audible (with my ear centimeters from where the drives are, not really at about 1 metter, and I'm talking front open wide) to being completely dead silent.
The padding clearly deafens that sound significantly.

But...
Since I've been waiting on a 4tb WD blue for storage that should be at the very least a bit quieter, and the ventilation itself is dead silent, I'm gonna try out a Meshify. I'm betting I'll get it to be quieter on load than a silent focused case just on being able to turn the fans down (3 Phanteks PH 140 should handle the intake pretty nice and quietly).

We'll see how it turns out.