One thing we have to remember is how the Internet itself is cobbled together along with how DNS resolution actually works and how CDN (Content Delivery Network) technologies such as Akamai and Amazon cloud massage DNS to force users to a specific edge service hopefully based on their geographic location.
As we all know, the Internet itself is just a collection of loosely connected networks with a few backbone carriers forming the mass of the connectivity between them. General consumer ISP's like Comcast, Windstream, etc. will typically have multiple peer points with major backbone carriers such as Level3, Verizon (VzB), AT&T, Sprint (who owns the old MCI transports). Companies like Google are also getting in on the game with their Google fiber projects. And Amazon is building it's own private backbone in an attempt to maintain control of it's own content.
So, in the olden days, a user sitting in Maine trying to get to a Google server in Mountain View would have to bounce around inside his local ISP, reach his ISP's local peer then get forwarded to his ISP's regional data center where it would pick-up a connection to his ISP's ISP. From there he would ride to the regional NAP in New York, NY. and pick up one of the primary east-west backbones - most likely AT&T at this point. His traffic would route across the US to the AT&T's regional NAP in LA where it would then hop off onto Google's local carrier and finally hit the server he wanted.
Today, 70-80% of all traffic if served via a CDN - the majority of which is either Akamai with Amazon catching up quickly. What happens is that a CDN will have an agreement with regional ISP's (with some services they're pushing it even closer to the user) where they establish an Edge network at that carrier. They also have agreements with content providers such as Google, Netflix, etc. which they then push cached content for those content providers to these Edge networks that is used to feed traffic to the ISP's users.
By massaging the DNS hierarchy they CDN ensures that all users on ISP xxx will directed to Edge network xxx for content xxx. There's actually 3 or 4 ways these work and the details are complex, but if you're interested here's some
good slides that covers a few.
Using my example above, that same guy tries to go to Netflix - which has an agreement with one of the CDN's. The guys PC does a DNS call to his ISP's DNS servers who through DNS Black Magic returns the IP of the ISP's local Edge network and the content is served.
Now lets suppose that user changed his DNS to point to Level3's 4.2.2.4 server. His "perceived" download speeds appear slower. I say perceived and appears because DNS does nothing to your actual bandwidth - but it can do a ton for optimization downstream. What has happened is that now when his PC attempts to resolve for Netflix he is bypassing his local ISP and sending it to L3. This server may be sitting in Seattle for all we know. If so, then instead of the cached content at his local ISP's Edge network he's now getting cached content for Level3's edge network in Seattle. Also, we have to keep in mind that now his media stream is competing with the traffic of the millions of other users flowing between east and west coasts. And given that most carriers rely on CDN's to serve content locally/regionally most carriers will put classify streaming media lower than other traffic to force users to use local CDN copies.
Now if that same user up and moved to Seattle and kept his DNC pointed to 4.2.2.4 his problems would go away because his geographical proximity to the content he's forcing his PC to pull from has changed. This is why some people have different experiences with stuff like Google DNS and OpenDNS. It's all relative.
I hope this cleared some things up a bit and didn't just make things more confusing. But, I've seen this problem countless times in my consulting gig's and it's a very hard concept for some folks to grasp.