1. Follow the signal. For you, Source->Sound Card->Speakers. The quality of your sound will depend on the lowest quality component of those three. If you're listening to poor 96kbps or 128kbps mp3s, then upgrading the other two components will have little affect on the sound. Oftentimes, improving the sound card and speakers can actually make poor quality music sound worse because they are capable of reproducing the poor quality more accurately.
2. If the source is good, then start with upgrading the speakers. Only get 5.1ch if you do a lot of 1st-person gaming or movie watching on your system; 2.0 or 2.1 would serve music much better. Personally I'm fond of either the Audioengine A5+ or A2 with subwoofer for 2.0/2.1. For their size, the A2 is pretty remarkable.
3. Whether you want a better sound card or an external DAC depends on your requirements. If you're going to use the digital out to go to speakers or a DAC then it's silly to upgrade the sound card as it will do almost nothing for you. The best possible sound quality would be achieved by running digital to an external DAC, but a few sound cards have good enough DACs for the majority of sub-$500 speakers. My personal choice would be either the Asus Xonar STX or Creative ZXR; probably the ZXR for actual EAX support instead of emulated (although I dislike Creative pricing).