There will be no performance difference between 2GB and 4GB on the same GPU chipset. The question comes into play when you want to jack up the visual settings to the maximum, some 2GB cards can't hold the entire set of texture data and will bog down when swapping textures in and out of VRAM on demand. If you have more VRAM, the entire texture cache set can remain in memory and you'll have smoother transitions from place to place in the game.
They say that if you're running 2560x1440p or 2560x1600p, or multiple monitors on a single card, that you need more VRAM because the display frame buffers are larger for the larger displays and/or multiple displays, leaving less VRAM available for texture caching.
There was a general consensus that with the GTX 680 (the GTX 770 is basically a 680, but with higher clocks and faster VRAM) was usually at a point where the GPU itself was becoming the bottleneck when working with maximum visual details that required more than 2GB of VRAM. So, while you could buy 4GB GTX 680s - setting them up with maximum details "requiring more than 2GB", the card would slow down due to the load and not the VRAM texture swapping so the point was moot. LOL.
p.s. I had three GTX 680's with 4GB in Tri-SLI and I never once saw them hit more than 2.2 GB of VRAM, and that was with Crysis 3 under maxed out details at 2560x1600p. 2GB is "plenty" in my opinion for current generation games.
p.p.s. Rumor has it that XBOX One and PS4 console development are going to push multi-core and VRAM requirements over the next several years, so 4GB might be a good idea even if you can play most things with 2GB today. It's your money, mileage may vary!