The way I see it, the crappy conditions the rest of the world suffers DOES affect us here in America.
1) Work is cheap and exploitable in developing nations like China, India, Malaysia, Thailand, Singapore, Bangladesh, Myanmar, etc.
2) American companies save large amounts of overhead by shipping manufacturing overseas.
3) Less work for American workers = more people on the dole, more unemployment which can lead to more secondary things like crime, poverty, neglect of children, slums, disease, which can then lead to tertiary outcomes like increased payout via medicare and medicaid, more shelters, more prisons, etc.
4)When these workers start receiving better pay, better living conditions, improved civil rights - then the cost of manufacturing will go up, and as the cost of oil will inevitably go up (and therefore shipping) in the future - then stateside manufacturing will become a more viable option.
5) Manufacturing will return to the states.
6) End outcome: You will pay more for things in the future, but the world will probably be better off for it, overall.
The above is all my idle speculation, but I hope someday when everyone is paid a fair wage for fair work, then things will get better for everyone, including us (as a nation).