New Posts  All Forums:Forum Nav:

UDP Packet Loss

post #1 of 18
Thread Starter 
Hey,

I've got a problem where by we recently changed our network from 100mbps to gigabit. Since then we have started to drop packets, seemingly as the traffic increases.
We're sending about 6200 packets per second at an average of 750kb per packet.
The network requires the lowest latency and packet loss.
We changed to gigabit in order to push the packets through at a faster rate, but I think the packets are just being held in the buffers instead (where as with 100mpbs they weren't/we had no packet loss). The buffer size has been changed to 4092 on the NIC's and that has lowered the packet loss.

1. Is there any program that goes down deep into the network card and can see if the buffers are filling.
2. Are there any there any other options to decrease the packet loss? I am about to try increasing the receive descriptors and see if that has an effect.

If anyone needs any more info, I am pleased to supply.

Thank you in advance smile.gif
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
post #2 of 18
Are you using cat 5 or cat6 cables? Also it could be in /proc/net/udp in *nix, the higher the number the larger the queue

UDP is bad if you want to get reliable communication as well, you should look at yusing TCP if you want to make sure packets arive smile.gif
Escobar
(9 items)
 
Supercomputer ^_^
(13 items)
 
 
CPUMotherboardGraphicsRAM
1055T M4A88T-D EVO USB3 ATI 6850 4 GB 
Optical DriveOSMonitorKeyboard
DVD RW Windows 8 Pro lp1900 + 2 X 15 inch dell Microsoft Comfort Curve 
PowerCase
600watt thermaltake antec 200 
  hide details  
Reply
Escobar
(9 items)
 
Supercomputer ^_^
(13 items)
 
 
CPUMotherboardGraphicsRAM
1055T M4A88T-D EVO USB3 ATI 6850 4 GB 
Optical DriveOSMonitorKeyboard
DVD RW Windows 8 Pro lp1900 + 2 X 15 inch dell Microsoft Comfort Curve 
PowerCase
600watt thermaltake antec 200 
  hide details  
Reply
post #3 of 18
Thread Starter 
Cat 6 cables and packets cannot be retransmitted, it can basically be classed as voice data.
What do you mean by:
Quote:
/proc/net/udp in *nix
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
post #4 of 18
Ah coolness, hmm, any switches / routers between you and the destination? Whats the load on the network?

in anything linux everythings a file, so the "file" /proc/net/udp contains the size of teh waiting UDP packets, if the number is more then 0 it means that you are sending too much data for the reciver to process
Escobar
(9 items)
 
Supercomputer ^_^
(13 items)
 
 
CPUMotherboardGraphicsRAM
1055T M4A88T-D EVO USB3 ATI 6850 4 GB 
Optical DriveOSMonitorKeyboard
DVD RW Windows 8 Pro lp1900 + 2 X 15 inch dell Microsoft Comfort Curve 
PowerCase
600watt thermaltake antec 200 
  hide details  
Reply
Escobar
(9 items)
 
Supercomputer ^_^
(13 items)
 
 
CPUMotherboardGraphicsRAM
1055T M4A88T-D EVO USB3 ATI 6850 4 GB 
Optical DriveOSMonitorKeyboard
DVD RW Windows 8 Pro lp1900 + 2 X 15 inch dell Microsoft Comfort Curve 
PowerCase
600watt thermaltake antec 200 
  hide details  
Reply
post #5 of 18
Thread Starter 
Ah right, using windows sorry :/

The network is - hosts(80ish) : layer 3 switch : Server

The server multicasts updates to the hosts and the network is used for nothing else, so there isn't much load. We just need the updates to get there as fast as possible.
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
post #6 of 18
There are a few things that need to be clarified here. First of all, in your original post, you mentioned your server is sending 6200 packets per second, and that packet size was 750 kb per packet. Don't you mean 750 bytes? 750 kilobytes or 750 kilobits, either one, is well in excess of the 1500 (standard) to 9000 (jumbo) byte MTU limits on the switch. That being said, I think there's some terminology discrepancy here that needs to be corrected.

Lets assume you're talking 750 bytes, because 750 kilobits * 6200 PPS is 4.65 Gbps, which far exceeds your Gigabit link rate.

Never, ever, EVER use UDP for broadcast unless you also write into your broadcast protocol an ACK process. UDP, by design, is a connectionless protocol, and per RFC can and will be dropped as first priority in every case. A properly written UDP based transaction looks like this:

Server UDP packet ---> Client
Client UDP ACK packet ---> Server
Server UDP ACK ACK packet ---> Client

In this scenario, you are using connectionless UDP to handle all the traffic, so you MUST implement two-way ACK to guarantee delivery of the packet. If the packet is lost, it must be retransmitted.

TCP is connection-based and is better suited to this type of communication, because the underlying protocol and network drivers will handle the serialization and ACK/ACK process for you. With UDP protocols, you have to write it all yourself 100% from scratch in your communication code.

This sounds like somebody wrote a homebrew application to trigger events or updates on a bunch of client PCs, without understanding what UDP is and how it's supposed to be used. I would have written a TCP connection-based daemon on the client PCs and send updates from the server over the socket.

Greg
Edited by hammong - 1/16/14 at 8:22am
post #7 of 18
Thread Starter 
I can't remember the numbers exactly, left my notes in work. It might have been 750kb with 620pps average. (I'll post the actual stats when I find the numbers)

I'm not using UDP for broadcast and I do expect some packet loss. I am trying to minimize packet loss by fixing aspects that I know are causing it. I am well aware of TCP and it is not an option to use it.

There were very few dropped packets before we switched up to gigabit and after increasing the buffers there are less, so the buffers are a source of packet loss. Just trying to get around that.
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
post #8 of 18
Wouldn't that be something to do with Switch Load then?
Assuming it's routing packets as it is in Layer 3 mode, and the Switches CPU can't handle it (Which is why there are buffers in the first place).

Like stated UDP packets aren't supposed to be reliable so it maybe be dropping it because of that.

Also some switches have Cut Through Forwarding which may help keep the buffers a bit lower.
ESXi Home Box
(6 items)
 
The Workstation.
(16 items)
 
 
CPURAMHard DriveOS
Dual L5630 72GB DDR3 RECC 120GB VERTEX 4 + 6TB RAID5 ESXi 6.0 U2 
Other
Dell PowedgeR710  
CPUGraphicsRAMHard Drive
Intel Core i5 2.4GHZ Intel Iris Pro 16GB DDR3 256GB PCI-e x2 
OSKeyboardMouse
OSX 10.11 + Win 10 Apple Wireless Keyboard Apple Magic Trackpad 
  hide details  
Reply
ESXi Home Box
(6 items)
 
The Workstation.
(16 items)
 
 
CPURAMHard DriveOS
Dual L5630 72GB DDR3 RECC 120GB VERTEX 4 + 6TB RAID5 ESXi 6.0 U2 
Other
Dell PowedgeR710  
CPUGraphicsRAMHard Drive
Intel Core i5 2.4GHZ Intel Iris Pro 16GB DDR3 256GB PCI-e x2 
OSKeyboardMouse
OSX 10.11 + Win 10 Apple Wireless Keyboard Apple Magic Trackpad 
  hide details  
Reply
post #9 of 18
You do have both ends of the connections set to Auto Negotiation and not manually set to Gigabit/Full Duplex, right? If you have Auto on one end of the link, and Manual on the other end of the link, the data stream will have a packets dropped frequently.

I'd consider anything between 0 and 0.5% packet loss on a gigabit point to point link to be within tolerance. Better quality switches in the middle can cut it back a bit, but if you're using some cheap $200 24-port gigabit switch, you can expect loss to be significantly higher.

Best of luck!

Greg
post #10 of 18
Thread Starter 
The load on the switch is fine, plus its just using the switch function as the server is on the same LAN.

Yes, both ends are set to auto negotiate, I thought about testing both sides on manual (even though it violates the standard) and seeing if it made any difference.

Its a 3950 stackable cisco switch, not cheap bit of kit.
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
Skynet HQ
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Athlon X2 Dual Core 2.70GHz Asus M3N-78 PRO ATI XFX 5770 5GB Corsair DDR2 
OSMonitorKeyboardPower
Windows 7 Ultimate 2x 23" LG Flatrons Logitech MX3100 850w PSU 
CaseMouse
X-Blade Gaming Case Logitech MX1100 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Networking & Security