When considering overall performance on a Linux system, unless you're just concerned with benchmark numbers, what you really mean is some subjective "user experience". So you can avoid the TLDR syndrome, let me state right here that while I'm running an Asrock Z77 Extreme with an i5-3550 with 8 Gigs of RAM, and at any given time have 3-6 Operating Systems on it, some (especially Windows) 64bit, my main OpSys is still 32bit and I notice very little if any difference in performance on many benchmarks, games or what have you.
My overall user experience is what keeps me from moving to 64bit on my main. It's less work that has little gain..not yet a good tradeoff. For my reasons why, just keep reading. If you just want one opinion from someone who has both you can stop here. There is barely a noticeable difference for most people not doing lots of video decoding and the like.
The extended version (underlying causes) -
It is difficult to get a truly objective answer on 32bit vs/ 64bit for several reasons. Obviously 64bit has greater potential and at some point the number of 64bit programs will so far outweigh 32bit that 32bit will become all but obsolete. So devs encourage us to move to 64bit now to possibly more quickly effect that changeover. Some reasons for this are further down this post. Additionally, since it is a Windows World still and Windows PAE bloze goats, one really has to go 64bit with Windows.
Linux PAE is substantially superior. One example is that Windows limits PAE to 4 GB for licensing purposes ONLY, while Linux PAE can address up to 64 GB RAM on a 32 Bit CPU/System. You may notice that even here on a Linux board people refer to "The 4GB Tipping Point" as if it applied in Linux when in fact it does not.
One can set how much memory is addressable by PAE in the Linux Kernel and apparently when one exceeds 32GB there are some penalties and considerations but up to 32GB works a treat. I've worked on 32GB Linux PAE machines that are solid and fast. 4GB is a walk in the park. So as far as hardware and it's support within the kernel goes, there is almost NO DIFFERENCE (less than 1%) in performance, with the exception of some benchmark programs that are weighted in favor of 64bit. Phoronix verifies this BTW.
Another reason many tout 64bit over 32bit is simply because that's what they have chosen (and as you can already see, often for mistaken reasons) and people tend to justify their choices to others to justify it to themselves. Hopefully I am somewhat less subjective because I use both and have not made a hard choice yet simply because it isn't time yet in my estimation. Here's why I think that.............
Still a very large number of programs are 32 bit and very few are exclusively 64 bit. So there is almost no penalty (programs that can't be run) with 32bit. Plus, if you use a 64bit system and have any programs that are 32bit, and almost everyone does, this requires MultLib.
When compiling programs to satisfy Multilibs, they are compiled by gcc at least twice and often multiple times. Additionally one must have two sets of libraries for those programs to make specific, non-conflicting calls. I hesitate to refer to these as "bloat" but anyone can see this is less simple than it presently has to be. To be clear, I'm not saying this takes a noticeable increase in human time to run such multilib programs, just that it makes ones' system more complicated, especially if one compiles anything from source.
To avoid the inevitable flames, let me state that nobody presently running 64bit has any compelling reason to reinstall as 32bit. Again, the difference is on the order of 1% for over 90% of common programs performance. My point is simply that, at least for now and a few more years, there is also no compelling reason for most people to choose 64bit over 32bit in Linux, unless you have more than 32GB RAM. There are only minor differences that most Linux users will never feel.
The bottom line is that you can feel free to experiment, unless you're running the likes of CAD, Video Decoding, etc. with almost no penalty and with some gain in simplicity.