Whoops, this actually turned out to be a wall of text. Oh well
I mean, a couple of reviewers are kinda already doing it, but the majority still go off of mfg specs which is a poor indicator of real-world performance.
1. RPMs don't matter
With all of these unique fan blade designs, motors, bearings, frames, etc. one fan at 1100rpm may move as much air as another at 1500rpm if not more.
Instead, airflow at a fixed noise level ("noise-normalized" as popularized by Gamer's Nexus) will give a more accurate indicator of fan performance. Increasing those datapoints will give you a more comprehensive breadth of the fan's performance.
2. MFG specs don't matter (airflow)
They can be a poor indicator of how the fan performs in real-world usage. These fans are measured in free air, usually at 12v. We know that dust filters, solid front panels, that hexagon case mesh material and whathaveyou, and especially radiators will shape the performance of the fan. Depending on the fan type, those restrictions could take out a chunk of your airflow.
Instead of relying on those specs, having airflow measurements through various obstructions would paint a better picture of the fan's airflow performance. For instance, fan a may have 120cfm in free air, but it loses 30 going through a dust filter. Fan b may also do 120cfm in free air, but loses 45cfm through a dust filter. Or a radiator. Or a front panel. Those are the numbers we need.
On a side note, I'm also curious about how the airflow shape contributes to thermals. For instance, those silverstone AP fans with the neat looking grill on the backside shapes the exhaust airflow so that it travels straight. Your typical fan blows out air in a V-shape. You have the most pressure around the edge of the fan, and there is a deadzone with turbulence near the middle. If you're trying to push some air towards your graphics card or other component in the case, wouldn't a fan with a more focused airflow path be more desireable? Though, with all of the stuff going on in people's cases nowadays, that's probably a wash, especially if you can brute force it with more fans. Again, a side note.
3. dB doesn't matter
The noise a fan makes is much more than just a peak dB result. You'd think a fan that's advertised for 32dba would be quiet until you hear the ball bearings growling at 1000rpm, the turbulence of the intake fan next to a grill or radiator, or the harmonics and resonances the fan produces. I remember reading some amazon review of the a1225 about some guy returning the fans because of the sound signature. Could have been a shill review, but it's still a very important point, possibly more important than airflow metrics if you've a keen ear.
Instead, I think we should adopt something similar to speaker and headphone measurements. Some of you audiophiles on here know how some headphones have their peaks and dips in the frequency range, and that resonance could make or break the headphones without equalization. There is no equalizer for fans; what you hear is what you get.
So, being able to point out specific noises in a frequency chart, and how it relates to what you'd hear in person would be great information to have. For example:
Fan a: This 3dB peak at 500hz is the DBB growl, this peak at 2kHz is the motor noise resonance. In fan b, the fan at 500Hz is 6dB quieter due do it using FDB, and the rubber isolators have dampened the resonances from the frame which are pretty common at 4kHz. However, you can see here in the 700-1kHz range that the mesh screen in front of the intake fan is creating 4db more noise on fan b than fan a, so it's recommended to use this fan in a push configuration or with a bit of a gap in pull configuration to reduce the turbulence.
And then, you put a sound clip on top of that to really put things into perspective, and maybe some spectrometer results of the fans on a cooler or in a case vs. open air to see the tonality differences. Wouldn't that be nice to have?
4. Static pressure
doesn is too ambiguous
But lol yeah. Static pressure isn't really something that is important nowadays when you have an airflow:noise graph. If it can push a lot of air through a radiator, then that's all that matters because that's where your temps are going to be based off of (considering well ventilated case, optimized airflow path, blah blah). While static pressure has been used as an indicator of how well it can perform in restrictive environments, it doesn't do a good enough job of how much that mm/h2o metric actually affects performance. And in the grand scheme of things, the combination of the above points is all you really need to gauge the performance of the fan. That mm/h2o number would just be a reference point as to why a P12 is performing better than an S12A on a radiator. By itself, it doesn't mean much, but when benchmarked it really puts it into perspective, and that's what I feel we need in these fan reviews.
5. Temperatures don't matter
When I see youtube reviewers reviewing fans by putting them in a case or air cooler, I wonder how well they're actually controlling all of those variables. Do you know how much is going on inside a case, that ONE case to get those temperatures? It's too personalized, and honestly I think it's an unnecessary datapoint.
Again, I think an airflow:noise graph is all that's needed because the more heat you can dissipate from the heat exchanger, the better your temperatures will be. A drop in temperatures will be the effect, and the cause of that is the 40fpm increase in airflow at 30dba versus the other fan.
6. PWM and Voltage scaling matter
I don't see this being measured enough. Some fans don't scale linearly with PWM, and if you're trying to target a specific percentage range that gives you a good balance of noise and performance for your auto config, then having a linear rpm spread would be ideal instead of having the fan jump from 1300 to 1800rpm with just 15% PWM increase.
The same applies to voltage adjustment, or basically undervolting. Some fans exhibit clicking noises when undervolted or have high starting voltages that can make them unusable if you're trying to keep the noise down. Granted, anomalies like this aren't as common as they used to be, and I'm sure the engineers have tested these metrics, but I still think it's important to validate them regardless.
However, and this is a big however and likely the reason why this isn't tested more often, is the fact that motherboards nowadays aren't the same when adjusting fan speeds. Some fans can't even hit their advertised speeds because of motherboard fan control, or the PWM curve changes, or you have a limit on how low the starting voltage is. But I guess that's why you use purpose-built measuring equipment for it instead (read: expensive).
But yeah that's about it. You have VSG on thermalbench doing the nice airflow:noise charts as well as pwm/v scaling. VSG also does the fan reviews on TPU, but aren't as good as the ones on his website. Overclockers.com has some old fan roundups that have fans tested with a dust filter, air cooler and radiator, but lack noise normalization. SPCR has some nice measuring equipment, but the scope of their tests are small. A number of youtubers have some microphone shots of the fan's sound signature, though that's all you can really take from those reviews. None that I know of do smoke tests. Cooling technique has his machine and good sound clips as well as probably the best spectrometer readouts, though he only measures free air and the spectrometer results don't really define any quirks of the fan.
The point is, nobody has put all of them into one review, and you have people having to reach in the nether-regions of the internet trying to supplement what one review doesn't have, or just trying to find any quantitative data on a particular fan. And we all know that doesn't work due to different testing methodologies, QC variance, measuring equipment, and so on.
So, what do you guys think? Is it time for fan reviews to be held to a higher standard, to take the next step like frame-time measurements were for graphics cards? Are current reviews good enough for Joe Blow?