Well I run with the side panels off all the time. The only thing I find loud is usually the single HDD, and ok the GPU fan in games is louder. But overall, my system is pretty quiet compared to a lot of people with a full case.
Also less dust builds up for sure. And yup it's cooler for sure. I should put the panels back on to see how much cooler. I don't use any case fans, just a 2x140mm SilverArrow and a monster cooler on a Zotac rtx 2070 AmpExtremeCore