Overclock.net › Forums › Components › Network Hardware › Help configuring Infiniband network properly
New Posts  All Forums:Forum Nav:

Help configuring Infiniband network properly

post #1 of 11
Thread Starter 
I have had my network set up for a while with no issues, it consisted of a router going to two different switches. One is a 10GbE switch where all of my own devices are connected, and the other is a Cisco 1GbE switch that all the roommates devices and the guest AP is connected to. This has worked out well but I wanted to move my own PCs and server over to Infiniband after Skyl3r got me more interested in it. So I just hooked up the Infiniband switch yesterday to the network.



So right now my network looks like this:









I just got the Voltaire/Mellanox 4036E infiniband switch yesterday, and have moved all my computers and server over to it. I was able to SSH into the switch via PuTTY and configure it. I have full network connectivity right now, I can be on my computer and talk to the router just fine and change settings around and all that. However, I cannot get internet on any of the computers that are connected via Infiniband. They have network access to the router, but no internet access. So that is problem #1.

The way the 4036E switch works is that is has thirty four 40gb ports, and two 10GbE SFP+ ports. There is a built in subnet manager that translates the Infiniband traffic to the Ethernet interfaces and vise versa to allow Infiniband clients to talk to the rest of the network. This means there is no need for a separate gateway to bridge the traffic between network types. I know the subnet manager is working, because otherwise I would not be able to talk to the router from my computer right now. I am just using the default minhop subnet manager.

The router IP is set to 192.168.1.1, I set the Infiniband switch to 192.168.1.2. It turns out this is just setting the management interface's Ethernet port to that address. When I plugged the 10GbE SFP+ port from the infiniband switch into my netgear switch, the SFP+ port seems to have assigned itself 192.168.1.3. I assume it must choose the next sequential number after what the management interface is set to, as I did not assign it .3 myself nor is that within the range set for the DHCP server on the router. Either way, thats what it got and the router sees that address and I can talk to the router just fine.

When I plug the cable from my Infiniband HSA (same as an Ethernet NIC for all intents and purposes) to the switch, I am given a proper DHCP address. In this case I was assigned 192.168.1.163. I can also manually assign myself an ipv4 address in the adapter just like any regular ethernet adapter in Windows.

So as far as I can tell, the network seems to be functioning ok. I can ping every device on the network from my computer that is currently on Infiniband, and yet I still cant get internet access on it. All computers that are plugged in over regular Ethernet can get to the internet, and if I unplug my computer from the Infiniband switch and plug it in over Ethernet then my computer can get on the internet. So I am not really sure where the problem is, but I am hoping someone on this forum is knowledgeable enough in networking that they can steer me in the right direction and help me get this configured so the Infiniband connected PCs can get on the internet.
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #2 of 11
@OP

Check the routing table of the computers connected to the Infiniband switch, specifically that the default route is set (should be the IP address of the router). Also double check that the Infiniband switch doesn't have its own DHCP server...
Mythica
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Western Digital Scorpio Black Samsung HD204UI Linux Mint 17 HP L1800 
KeyboardPowerCaseMouse
Logitech Basic Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Mythica
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Western Digital Scorpio Black Samsung HD204UI Linux Mint 17 HP L1800 
KeyboardPowerCaseMouse
Logitech Basic Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #3 of 11
Thread Starter 
The gateway seems to be set right, it comes out as 192.168.1.1 which is the router. The infiniband switch does not have a DHCP server as far as I can see, the only DHCP options are to allow the switch to get its management interface address over dhcp or to turn it off and set the IP manually which is what I did. Here are a couple screenshots of how things are saying they are configured. The left terminal in the first image is the infiniband switch. The others are showing the congis of the PC I am using right now to try and get internet access working.





Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #4 of 11
@OP

So, just to be clear: while connected to the Infiniband switch you can ping devices on both Netgear and Cisco switches? What is the address range of the devices connected to the Cisco switch (I'm looking for weird conflicts, if any)? As for the lack of Internet access, does this apply to all devices connected to the Infiniband switch or just your PC?
Mythica
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Western Digital Scorpio Black Samsung HD204UI Linux Mint 17 HP L1800 
KeyboardPowerCaseMouse
Logitech Basic Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Mythica
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Western Digital Scorpio Black Samsung HD204UI Linux Mint 17 HP L1800 
KeyboardPowerCaseMouse
Logitech Basic Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #5 of 11
Thread Starter 
Quote:
Originally Posted by parityboy View Post

@OP

So, just to be clear: while connected to the Infiniband switch you can ping devices on both Netgear and Cisco switches? What is the address range of the devices connected to the Cisco switch (I'm looking for weird conflicts, if any)? As for the lack of Internet access, does this apply to all devices connected to the Infiniband switch or just your PC?




I was actually slightly mistaken, I cannot PING other computers and real devices from the computers on the infiniband network. I can PING all routers, switches, and APs though.
However, I capitalized ping above because I simply cannot ping them (as in actual ping command in a cmd window), I CAN send and receive data. I can browse mapped network drives from PC#1 and PC#2 on the infiniband switch and the drives are located on PC#5 that is an Ethernet client on a completely different switch than the Infiniband clients are on (I just tested watching a few movies over the network earlier to verify). I also took a screenshot on PC#2 and sent it to PC#1 over Infiniband. I can also log in to all the switches and my router and make changes and all of that. So they talk to each other just fine, but for some reason the actual, specific "ping" command in a cmd window does not respond back from any of the computers on the network, only get a response from routers and switches. And of course all the Infiniband connected computers do not have internet access. Here is a screenshot of my PC#2 doing a ping to the router (.1), the 10GbE switch (.4) and the 1GbE switch (.5):






So all Infiniband computers have no internet, but can send/receive data to all other devices on the network just fine.
All computers that use Infiniband, if I disable the Infiniband adapters and enable the Ethernet adapters they get to the internet just fine. (Which is how I am typing this post)





EDIT: As for address ranges of devices on the switches, all devices on the Cisco switch are connected via DHCP. The DHCP Server has a range of 192.168.1.150-254. The devices right now are:

.180
.183
.223
.226
.247
.252

and connected to the AP that connects through the Cisco switch I only have 1 device right now:
.179



The Netgear switch is mostly not used right now, but it can have:

.48 (loft computer when the ethernet adapter is enabled. this adapter is disabled when I am trying to work on the infiniband stuff)
.49 (my computer when the ethernet adapter is enabled. this adapter is disabled when I am trying to work on the infiniband stuff)
.100
.103 (turned off most of the time)
.125 (roku device that is off most of the time and gets used on weekends mostly)
and an xbox connected via dhcp but it isnt on right now so IDK what ip it will get when I use it next


The Infiniband switch right now just has 2 of the 5 computers plugged in. The IPs are:

.148 (loft PC, PC#2)
.149 (my PC, PC#1)
Edited by EniGma1987 - 4/25/17 at 9:30am
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #6 of 11
@OP

Based on everything you've posted, I'm beginning to suspect the culprit is the subnet manager on the IB switch. The fact that the stations on the IB switch can communicate with the rest of the LAN but not outside of the LAN tells me there's an issue between the IB switch and the Netgear switch, and the only thing I can think of that would affect traffic at the IP level is the subnet manager.

The switches sound like managed switches. Is there a way you can monitor traffic between the IB switch and the Netgear switch?
Mythica
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Western Digital Scorpio Black Samsung HD204UI Linux Mint 17 HP L1800 
KeyboardPowerCaseMouse
Logitech Basic Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Mythica
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Western Digital Scorpio Black Samsung HD204UI Linux Mint 17 HP L1800 
KeyboardPowerCaseMouse
Logitech Basic Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #7 of 11
Thread Starter 
Quote:
Originally Posted by parityboy View Post

@OP

Based on everything you've posted, I'm beginning to suspect the culprit is the subnet manager on the IB switch. The fact that the stations on the IB switch can communicate with the rest of the LAN but not outside of the LAN tells me there's an issue between the IB switch and the Netgear switch, and the only thing I can think of that would affect traffic at the IP level is the subnet manager.

The switches sound like managed switches. Is there a way you can monitor traffic between the IB switch and the Netgear switch?


I have been leaning towards the subnet manager the switch uses as well, though I dont know of a way to know whether it is working as intended or not.

In what way do you mean monitor the traffic? What should I be looking for?


Im not sure if any of this info really matters, but this shows the port statistics for the Neatgeat switch. Port 12 (xg12) is the one that connects to the Infiniband switch:

Port 11 of that switch is the port that connects to the main router.




Here is port 12's detailed view:

Edited by EniGma1987 - 4/25/17 at 3:04pm
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #8 of 11
@OP

What I meant was, can you capture the traffic arriving on the port from the IB switch and inspect it? If so, you could see what happens (or not) when you send data intended for the Internet. Also, are there other subnet managers you can try?
Mythica
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Western Digital Scorpio Black Samsung HD204UI Linux Mint 17 HP L1800 
KeyboardPowerCaseMouse
Logitech Basic Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Mythica
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Western Digital Scorpio Black Samsung HD204UI Linux Mint 17 HP L1800 
KeyboardPowerCaseMouse
Logitech Basic Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #9 of 11
Thread Starter 
I dont really know much about capturing and analyzing network traffic so I dont think I can be much help in that area. I did install Wireshark on my computer and let it capture data while I tried going to a couple websites (google.com and speedtest.net). I attached the file if you think it might help:
WiresharkLog.zip 101k .zip file


There are 2 subnet managers in the switch, minhop and updn. Minhop is default and I have tried updn before and it didnt change anything. I just turned updn on now to give it a try again.
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #10 of 11
Thread Starter 
I downloaded this program from a company called SolarWind and did a network discovery of my network. I was hoping to use it to get a little more info about where the problem might be, but the network discovery seems to only detect my computer, the Infiniband switch, and the Neatgear switch:






This hints that the Infiniband switch is somehow not able to "see" past the switch directly above it in the chain. The odd thing though is that I can log in to the router and Cisco switch down the other side of the router just fine, but perhaps there is something either in the MNetgear switch or the router that is translating my LAN traffic well enough to function but isnt/wont translate the internet bound traffic? IDK. I am not really the most knowledgeable about networking once I get into the more advanced things but I just thought it was odd that the network discovery was unable to make it past the Netgear switch directly above the Infiniband switch.
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Network Hardware
Overclock.net › Forums › Components › Network Hardware › Help configuring Infiniband network properly