Overclock.net › Forums › Components › Network Hardware › Infiniband for Main Storage File Server
New Posts  All Forums:Forum Nav:

Infiniband for Main Storage File Server - Page 3

post #21 of 34
Thread Starter 
Quote:
Originally Posted by EniGma1987 View Post

The guy whos article I was reading (dont have it saved here at work) had used a 10gb infiniband network and estimated he should get roughly 700MB/s throughput. Why 700 IDK, my estimations should put it at 1GB or slightly over. Anyway, after quite a bit of testing he found that the best it was able to achieve was 135MB/s due to the way the Infiniband functioned on Win7 and in Ethernet mode with IPoIB. Something about how the network couldnt use RDMA and other instructions or transfer types or whatever that are supposed to be inherent to Infiniband. I hope Win10 has brought improvements to this area, it would be nice if Infiniband could reach its potential on our more "home type" networks.


I have SSDs on both ends of 3 of the computers in my 10GbE network right now. So when I get home today I will do a crystalDiskMark test on each computer and then a test over the network itself and see what it comes up with.

Okay, sounds good.

Also, I did do a read/write test when I first set it up and I do recall my read speeds being around 250-300MB/s
This is with two HDD's in RAID0, so that's probably accurate to local read speeds.
My write speeds were around 135MB/s, which was weird.
Edited by Skyl3r - 4/12/17 at 7:39am
post #22 of 34
A lot of people test with RAM disks for transfers to alleviate any storage bottlenecks when testing. thumb.gif

Depending how the drive is presented will dictate a lot of performance too. If you mounted a block device via iSCSI, that will perform to a much higher degree than a SMB/CIFS share.
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
post #23 of 34
Alright so I did my tests. I ran Crystal Disk Mark on both drives C drives, which are m.2 SSDs, then I ran it on a RAMDisk on each PC to get a baseline for performance.

I first ran a test called LAN Speed Test (Lite) with a 1000MB file test from my PC to the downstairs server and to the loft PC which is the 2nd fastest one here and has the other m.2 drive and the RAMDisk. It shows quite a drastic slowdown on the write speed for both the SSD and RAMDisk. I can only assume that is a Windows file system thing since even the RAMDisk shows that. Read speed of that file over the network ranges from 7.9gb/s on the SSD to 8.7gb/s on the RAMDisk.





Next I tried a file transfer over the network of a TV show. Unfortunately the free version of the RAMDisk software I used only allows up to a 1023MB disk size for the free version, and didnt want to spend $40 to do this test on the two computers with a larger file. Sadly the drive size and file size is too small to really see what Windows says the transfer speed is, so I had to settle for transferring an HD movie of 5GB file size from my m.2 drive to the loft's m.2 drive over the 10gigabit Ethernet network. Transfer speed was a constant 1.11GB/s speed, which is right about the max speed of 10GbE once you take overhead into account. I transfered this file both from my computer to the loft and from the loft to my computer, speed was the same 1.10-1.11GB/s in both transfers.





My Samsung 960 EVO m.2 SSD:



Loft's Intel m.2 SSD:








Lastly, I wanted to try running a Crystal Disk Mark on the RAMDisk itself when it was mounted as a mapped network drive. So this is the speed of the disk over the network:

you can see it also maxing out around that same 1.11GB speed. That is the max 10gb Ethernet speed. Interesting that Ethernet has such a large penalty when doing 4k random read and write operations over it. I am curious if Infiniband will have much better performance there because of the significantly lower latency.


And this is the same RAMDisk when the test is run locally:
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #24 of 34
Thread Starter 
Quote:
Originally Posted by EniGma1987 View Post

Lastly, I wanted to try running a Crystal Disk Mark on the RAMDisk itself when it was mounted as a mapped network drive. So this is the speed of the disk over the network: Warning: Spoiler! (Click to show)
And this is the same RAMDisk when the test is run locally: Warning: Spoiler! (Click to show)

Glad you posted, I almost forgot biggrin.gif


LOCAL - RAMDISK


WINDOWS SHARE - RAMDISK



Other Tests

Warning: Spoiler! (Click to show)




Couple notes:
  • The file copy is limited to the local read of my desktop, which is slow (thus why I solely work off the file server). I forgot about that when testing, but I included it anyways.
  • My RAMDISK was created with Radeon RAMDISK - I don't know if any driver overhead is associated with this




I realize now that these results are hard to compare. It would work a lot better if I tested with 10GbE and then Infiniband using all the same hardware. It's hard to pinpoint where bottlenecks might be. That said, LAN Speed Test and Crystal Diskmark reported somewhat similar results.



EDIT:

RAMDISK to RAMDISK over network:



If this is accurate, then it looks like there could be somewhere in the neighborhood of a 46-50% overhead. Which is significant; but indicates that 40gbps could still yield significant gains.

If you are looking to move forward with this, I'd recommend just getting two cards and a cable. Test computer to computer without the switch. If you see the kinds of gains you are hoping for, or at least significant gains, then you can make the judgment call on the upgrade.



For me personally, I spent $22 on the two network cards, then $5 + shipping for the cable. So my total came out to be ~$60 for this network increase. I am not complaining biggrin.gif


I almost want to get another network cable and test again to see if I get 10gbps. A cool thing with Infiniband is you can literally just run two cables, and set the cables to "link aggregation" mode and it doubles bandwidth.

Edited by Skyl3r - 4/12/17 at 8:28pm
post #25 of 34
Quote:
Originally Posted by Skyl3r View Post

[*] My RAMDISK was created with Radeon RAMDISK - I don't know if any driver overhead is associated with this

I tested many RAMDisks in the past. I found every company's software had very different file transfer speeds. Some had better 4k, some had much better sustained. Dataram had the best speeds on average across the range, which is the company I used to do the RAMDisk in my tests above. I dont know who makes Radeon's, it could be Dataram themselves. But not only does the driver have different overhead and optimizations, the ram speed and timings has a pretty big effect on the speed for obvious reasons.



Quote:
Originally Posted by Skyl3r View Post

If you are looking to move forward with this, I'd recommend just getting two cards and a cable. Test computer to computer without the switch. If you see the kinds of gains you are hoping for, or at least significant gains, then you can make the judgment call on the upgrade.


Smart thinking. I will buy a couple NICs and a cable and try it out.
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #26 of 34
Thread Starter 
Quote:
Originally Posted by EniGma1987 View Post

I tested many RAMDisks in the past. I found every company's software had very different file transfer speeds. Some had better 4k, some had much better sustained. Dataram had the best speeds on average across the range, which is the company I used to do the RAMDisk in my tests above. I dont know who makes Radeon's, it could be Dataram themselves. But not only does the driver have different overhead and optimizations, the ram speed and timings has a pretty big effect on the speed for obvious reasons.
Smart thinking. I will buy a couple NICs and a cable and try it out.

Yeah... The RAM in my server is not high end RAM by any stretch of the imagination. I'll test again with dataram and see how much, if at all, my results change.

If you do end up doing this, be sure to post results here, I'm really interested in how well 40gbps Infiniband works.
Edited by Skyl3r - 4/13/17 at 6:20am
post #27 of 34
Quote:
Originally Posted by Skyl3r View Post

In regards to your edit, all of the steps required to go from out of the box to working are as follows:

  1. Add mellanox cards to computers (I'm using Win10, FYI)
  2. From what I read, nothing extra should be required to use a switch. I have never verified though, so you'll have to test
  3. Connect cable(s)
  4. Boot into Windows. On my computer it immediately identified it and displayed a Mellanox IPoIB driver in Network Connections
  5. I installed the latest ConnectX (OFED) from here http://www.mellanox.com/page/software_overview_ib
  6. I HAD to update the firmware on the card before it worked. I had to use MFT for this http://www.mellanox.com/page/management_tools
    You might not have to do this, so test without doing it first. Save yourself the trouble. I will dig up how I did it if you run into problems
  7. After this you need to set them to ethernet mode. You go to Device Manager. You'll see the IPoIB drivers under Network Adapters, do NOT click this, Instead, expand System devices and you'll find the mellanox device in there. One of the tabs has two radio buttons. One says IB and one says ETH. Switch the ports over from IB to ETH.
  8. Then all you need to do is assign IP's and you're good to go!




So I got the infiniband 40gb cards and cables today. Installed them in two PCs to test this out and driver and firmware install was perfect and easy. The driver package actually installed one of the newest firmwares for me with just putting a checkbox in the install window.

I switched them both to ethernet mode from the default infiniband and they got IPs right away. One computer came up with 168.254.33.149, the other came up with 168.254.233.1 or something like that. I could not ping the computers from each other, so I assigned IPs of 192.168.1.50 and 192.168.1.51 to get them all set on the same subnet. Unfortunately, I still cant ping properly. Here is a screenshot of my ipconfig if it might help with some ideas as to why things aren't working. Note that right now I have an RJ45 Ethernet port turned on so that I can get to the internet and post here. I have tested pinging the other computer with that onboard motherboard NIC both disabled and enabled.

Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #28 of 34
Well, its working now. No idea why, but I was screwing around with IP addresses. I set my computer to be 192.168.1.1, and then I was able to ping the computer in my loft from my computer. I set my IP back to .50 and could still ping the loft now. I set the loft computer then to 192.168.1.1 and I was able to ping my computer. Set the loft back to .51 and they could still ping each other. Weird. But at least it works now. Stay tuned for benchmarks
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #29 of 34
You need a different subnet than your other NIC is on.
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
post #30 of 34
Thread Starter 
Quote:
Originally Posted by EniGma1987 View Post

Well, its working now. No idea why, but I was screwing around with IP addresses. I set my computer to be 192.168.1.1, and then I was able to ping the computer in my loft from my computer. I set my IP back to .50 and could still ping the loft now. I set the loft computer then to 192.168.1.1 and I was able to ping my computer. Set the loft back to .51 and they could still ping each other. Weird. But at least it works now. Stay tuned for benchmarks
Quote:
Originally Posted by beers View Post

You need a different subnet than your other NIC is on.

beers is right. You need to assign a different subnet than your ethernet NIC. Try something like:
100.100.100.X
That is what I use. So computer one would be 100.100.100.1 and computer two would be 100.100.100.2.

Short explanation for why would be that your computer does not know which NIC to use when both are on the same subnet. It decides to try to resolve the IP using network adapter priorities. Typically, your ethernet adapter would have highest priority, so if you try to connect to, ping or otherwise reach an IP address on the same network as the ethernet adapter, it will attempt to use the ethernet adapter. Think of if you have a WiFi connection and an Ethernet connection on the same network. Your computer will always try to use the ethernet connection.

The fix is simply to assign a different subnet to the Infiniband adapter as beers said.


Also great to hear you were able to get it setup so fast! I'm excited to see what kind of results you can get outta that!
Edited by Skyl3r - 4/17/17 at 8:45pm
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Network Hardware
Overclock.net › Forums › Components › Network Hardware › Infiniband for Main Storage File Server