I lowered my overclock on my GPU to be 100% stable regardless of what's happening, even though it could run 3DMark and Heaven no problems at what it was. Going to lower the clocks on the CPU as well to 4.2ghz and have a bit more voltage then needed and see if that fixes my issue. It's definitely not my internet that's the problem since (even though I am paying for 1mb/s upload) I get an average of 5-10mb/s upload speed and I'm streaming at 2,750. Running it at 3,500 shouldn't be an issue either, but it's managed to be. Literally only does this when I'm streaming.
So, even though I'm not using my CPU to stream with directly, do you really think that's the reason? Not that I don't have a reason not to trust you as you most likely know more than I do.