That doesn't really seem like a problem. Any number of things could be eating resources in the background or not that could cause performance fluctuation. FPS also naturally fluctuates depending one what you're looking at at any particular time.
One thing I would recommend doing differently though is doing a driver clean between the uninstall of the old driver and the install of the new one. I use DriverCleaner.NET, but Driver Sweeper works just the same and is free.
- Have the new driver you plan to install already downloaded and ready to go.
- Download and install
Driver Sweeper
- Type "msconfig" into the run/search box in the start menu and press Enter.
- Choose the Boot tab at the top and tick Safe boot under Boot options. Hit apply, and exit without restarting when prompted.
- Uninstall PhysX and Stereoscopic 3D drivers through the conventional windows uninstall pane. Uninstall the display driver. Restart when prompted.
- You should be in safe mode after the reboot. Run Driver Sweeper with both NVIDIA - Display and NVIDIA - PhysX ticked. Clean.
- You can probably reboot in normal mode from here, but I usually reboot into safe mode once more just in case Driver Sweeper cleans any files after the reboot that might be locked with a normal boot.
- Go into msconfig again, untick Safe boot, reboot.
- Install new drivers.