I think one thing people fail to realize is that there are so many millions of combinations in a PC (especially our custom machines) - of hardware and software that there is simply no 100% fool proof way that Nvidia- or any other company for that matter can produce drivers that will be 100% perfect in all situations.
I've never had any problems with Nvidia drivers, keep them up to date, the GTX280 was good, the GTX480 was better - and now a Titan... always been solid as a rock and stable.
Look at your computer a bit closer- how much bloatwarez junk do you have installed? Are you keeping other hardware like chipset drivers updated... Are you running 100 extra services in the background for things you uninstalled a year ago that just had a crappy uninstaller
Do you have 10 different programs open and THEN a game on top of it?
Another thing- Chrome and many other browsers now are utilizing the GPU A LOT more... having a bunch of Chrome tabs open DOES amount for some GPU overhead- even it it's not much, perhaps it's causing some conflicts....
I just think it's unfair that Nvidia get's such bad rep for their drivers when in reality there isn't a company out there that is making "flawless" software... having a PC ( IMO ) implies you're going to have to do some maintenance and bug fixing... no different than an oil change in you're car.