In the case of GTX 670's I do not fully agree. Reference cooler for GTX 670 makes a terrible noise due to the fan being mounted in a plastic housing. At first I owned a MSI GTX 670 with the reference cooler. The humming noise from the fan almost drove me crazy. Also, it had alot of coil whine. Eventually I sold it and got these two ASUS GTX 670 DC II's (Non TOP). No humming noise from the fans and no coil whine. Very happy owner.
However, in most cases a reference cooler will be fine, but not for the GTX 670's. Also, when buying a certain card it's worth checking on the internet if other people are complaining on coil whine. GTX 670 reference are also known for it's coil whine, not just that annoying humming from the fan. GTX 680 reference on the other hand, is fine as far as I know.