Overclock.net › Forums › Components › Network Hardware › Infiniband for Main Storage File Server
New Posts  All Forums:Forum Nav:

Infiniband for Main Storage File Server - Page 2

post #11 of 34
Nice, a lot of games work pretty well over a SMB share. I've only had a couple that stuttered pretty badly even when using 10 Gbps NICs (Looking at you CS:GO).

Running all of the crap off of the server is super convenient, especially if you ever have to reinstall windows or anything. Just import the steam folder and done.
Quote:
Originally Posted by spinFX View Post

check out mikrotik's gear, they have some affordable routers with 4-8 1gbit ethernet ports and 2 SFP+ (10gbit/s) ports. I plan a similar upgrade to home network with a 10Gbit link between main desktop and main file/backup server and the rest of the network on 100mbit.
Very cheap way to get 10Gbit networking without dropping stupid amounts of cash (limited cable runs though - my file server will need to be quite close to the router and the main desktop)

I don't understand this suggestion. He already has an infiniband switch and some NICs which are running IP over Infiniband (IPoIB). For LAN transfers you'd be better off utilizing a switch as well.
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
post #12 of 34
i use 4 15000 rpm drives in RAID 0 over an infiniband connection from my server.

great speeds but you have to configure internet explorer (yes internet explorer) to allow execution of programs over your local intranet or else you'll get a security risk pop up every time.

http://woshub.com/how-to-disable-open-file-security-warnings-in-windows-7/

also 50 foot fiber infiniband cable FTW! lol.
Gaming Rig
(15 items)
 
Sons Rig
(11 items)
 
File Server
(13 items)
 
CPUMotherboardGraphicsGraphics
Core i7 4790K ASUS Z97-WS Gigabyte GeForce GTX 980 Gigabyte GeForce GTX 980 
RAMHard DriveCoolingOS
16 Gb's Mushkin DDR3 @ 1600 MHz Crucial M500 SSD 240gb Coolit ECO C240 Windows 7 Ultimate 
MonitorMonitorPowerCase
ASUS PB287Q 4K Monitor HTC Vive Coolermaster Silent Pro M 1000W Rosewill Blackhawk Ultra 
MouseMouse PadAudio
Razer Deathadder 2013 Steelseries Qck Mass Super Thick Cloth Mouse Pad Genius SW-G2.1 1250 4PC Gaming Speakers 
CPUMotherboardGraphicsGraphics
Core i7 3770k Asus P8Z68 Deluxe Galaxy Geforce GTX 780 Galaxy Geforce GTX 780 
RAMHard DriveCoolingOS
16 Gb's G.Skill DDR3 1866 Crucial M500 SSD 240Gb Coolit ECO C240 Windows 7 Ultimate 
MonitorPowerCase
Benq G2420HD Coolermaster Silent Pro 1000 Watt Deepcool Kendomen 
CPUMotherboardRAMHard Drive
Core i3 2120 Supermicro X9SCM-F 4 Gb Kingston 1600mhz DDR3 ECC 12 Segate 2tb drives (RAID 6) 
Hard DriveHard DriveOSPower
ADATA SP600 SSD 24 Toshiba DT01ACA200 drives (2 RAID 6's) Windows 7 Ultimate 64 bit. OCZ ZT750 Supermicro PWS-665-PQ 
CaseOtherOtherOther
Norco 4020 and Norco 4224 LSI 9260-4i Raid Card Intel RES2SV240 20 port Expander. HP SAS 24 por... Voltaire 410Ex Hca Infiniband HBA 
  hide details  
Reply
Gaming Rig
(15 items)
 
Sons Rig
(11 items)
 
File Server
(13 items)
 
CPUMotherboardGraphicsGraphics
Core i7 4790K ASUS Z97-WS Gigabyte GeForce GTX 980 Gigabyte GeForce GTX 980 
RAMHard DriveCoolingOS
16 Gb's Mushkin DDR3 @ 1600 MHz Crucial M500 SSD 240gb Coolit ECO C240 Windows 7 Ultimate 
MonitorMonitorPowerCase
ASUS PB287Q 4K Monitor HTC Vive Coolermaster Silent Pro M 1000W Rosewill Blackhawk Ultra 
MouseMouse PadAudio
Razer Deathadder 2013 Steelseries Qck Mass Super Thick Cloth Mouse Pad Genius SW-G2.1 1250 4PC Gaming Speakers 
CPUMotherboardGraphicsGraphics
Core i7 3770k Asus P8Z68 Deluxe Galaxy Geforce GTX 780 Galaxy Geforce GTX 780 
RAMHard DriveCoolingOS
16 Gb's G.Skill DDR3 1866 Crucial M500 SSD 240Gb Coolit ECO C240 Windows 7 Ultimate 
MonitorPowerCase
Benq G2420HD Coolermaster Silent Pro 1000 Watt Deepcool Kendomen 
CPUMotherboardRAMHard Drive
Core i3 2120 Supermicro X9SCM-F 4 Gb Kingston 1600mhz DDR3 ECC 12 Segate 2tb drives (RAID 6) 
Hard DriveHard DriveOSPower
ADATA SP600 SSD 24 Toshiba DT01ACA200 drives (2 RAID 6's) Windows 7 Ultimate 64 bit. OCZ ZT750 Supermicro PWS-665-PQ 
CaseOtherOtherOther
Norco 4020 and Norco 4224 LSI 9260-4i Raid Card Intel RES2SV240 20 port Expander. HP SAS 24 por... Voltaire 410Ex Hca Infiniband HBA 
  hide details  
Reply
post #13 of 34
Thread Starter 
Quote:
Originally Posted by Master__Shake View Post

i use 4 15000 rpm drives in RAID 0 over an infiniband connection from my server.

great speeds but you have to configure internet explorer (yes internet explorer) to allow execution of programs over your local intranet or else you'll get a security risk pop up every time.

http://woshub.com/how-to-disable-open-file-security-warnings-in-windows-7/

also 50 foot fiber infiniband cable FTW! lol.

Thanks for the link. I do get this popup all the time. Better fix it smile.gif
How much throughput do you normally see?
Did you have troubles setting it up?
Quote:
Originally Posted by beers View Post

Nice, a lot of games work pretty well over a SMB share. I've only had a couple that stuttered pretty badly even when using 10 Gbps NICs (Looking at you CS:GO).

Running all of the crap off of the server is super convenient, especially if you ever have to reinstall windows or anything. Just import the steam folder and done.

Exactly smile.gif
I actually have my Desktop, Documents, Downloads, etc. all mapped to the drive. I blow up my computer somewhat frequently, so it'll be super nice to just boot up, map drives and everything is back how it was.
post #14 of 34
Thread Starter 
Most annoying problem I've had yet. I wanted to try Fallout NV off my server.

Installed it and tried to launch and kept getting some ".ini not found" error. So, I browsed through my registry and changed all fallout references to folders to be FQN directories instead of the mapped drive letter.
IE:
"\\SERVER IP\..."
instead of
"X:\"

This fixed one error, but I kept getting an .ini error. I assumed it wasn't seeing my "My Games" folder because I had remapped my Documents to my server. So I changed it back to my User folder on my Windows Installation drive. No luck. I thought maybe Steam was doing something crazy, so I tried to launch the .exe directly.

Now I get a new error:
"Application load error 5:0000065434"

A quick google search tells me this means FalloutNV can't see Steam or vice versa. So, I tried closing down steam and launching it as admin. Voila, it works now.
post #15 of 34
Quote:
Originally Posted by Skyl3r View Post

Thanks for the link. I do get this popup all the time. Better fix it smile.gif
How much throughput do you normally see?
Did you have troubles setting it up?
Exactly smile.gif
I actually have my Desktop, Documents, Downloads, etc. all mapped to the drive. I blow up my computer somewhat frequently, so it'll be super nice to just boot up, map drives and everything is back how it was.

no troubles setting it up other than the annoying pop ups about security which i fixed.

it acts like a local drive and loads as fast as it did when it actually was local.

i have no idea what is happening with your fallout install though.
Gaming Rig
(15 items)
 
Sons Rig
(11 items)
 
File Server
(13 items)
 
CPUMotherboardGraphicsGraphics
Core i7 4790K ASUS Z97-WS Gigabyte GeForce GTX 980 Gigabyte GeForce GTX 980 
RAMHard DriveCoolingOS
16 Gb's Mushkin DDR3 @ 1600 MHz Crucial M500 SSD 240gb Coolit ECO C240 Windows 7 Ultimate 
MonitorMonitorPowerCase
ASUS PB287Q 4K Monitor HTC Vive Coolermaster Silent Pro M 1000W Rosewill Blackhawk Ultra 
MouseMouse PadAudio
Razer Deathadder 2013 Steelseries Qck Mass Super Thick Cloth Mouse Pad Genius SW-G2.1 1250 4PC Gaming Speakers 
CPUMotherboardGraphicsGraphics
Core i7 3770k Asus P8Z68 Deluxe Galaxy Geforce GTX 780 Galaxy Geforce GTX 780 
RAMHard DriveCoolingOS
16 Gb's G.Skill DDR3 1866 Crucial M500 SSD 240Gb Coolit ECO C240 Windows 7 Ultimate 
MonitorPowerCase
Benq G2420HD Coolermaster Silent Pro 1000 Watt Deepcool Kendomen 
CPUMotherboardRAMHard Drive
Core i3 2120 Supermicro X9SCM-F 4 Gb Kingston 1600mhz DDR3 ECC 12 Segate 2tb drives (RAID 6) 
Hard DriveHard DriveOSPower
ADATA SP600 SSD 24 Toshiba DT01ACA200 drives (2 RAID 6's) Windows 7 Ultimate 64 bit. OCZ ZT750 Supermicro PWS-665-PQ 
CaseOtherOtherOther
Norco 4020 and Norco 4224 LSI 9260-4i Raid Card Intel RES2SV240 20 port Expander. HP SAS 24 por... Voltaire 410Ex Hca Infiniband HBA 
  hide details  
Reply
Gaming Rig
(15 items)
 
Sons Rig
(11 items)
 
File Server
(13 items)
 
CPUMotherboardGraphicsGraphics
Core i7 4790K ASUS Z97-WS Gigabyte GeForce GTX 980 Gigabyte GeForce GTX 980 
RAMHard DriveCoolingOS
16 Gb's Mushkin DDR3 @ 1600 MHz Crucial M500 SSD 240gb Coolit ECO C240 Windows 7 Ultimate 
MonitorMonitorPowerCase
ASUS PB287Q 4K Monitor HTC Vive Coolermaster Silent Pro M 1000W Rosewill Blackhawk Ultra 
MouseMouse PadAudio
Razer Deathadder 2013 Steelseries Qck Mass Super Thick Cloth Mouse Pad Genius SW-G2.1 1250 4PC Gaming Speakers 
CPUMotherboardGraphicsGraphics
Core i7 3770k Asus P8Z68 Deluxe Galaxy Geforce GTX 780 Galaxy Geforce GTX 780 
RAMHard DriveCoolingOS
16 Gb's G.Skill DDR3 1866 Crucial M500 SSD 240Gb Coolit ECO C240 Windows 7 Ultimate 
MonitorPowerCase
Benq G2420HD Coolermaster Silent Pro 1000 Watt Deepcool Kendomen 
CPUMotherboardRAMHard Drive
Core i3 2120 Supermicro X9SCM-F 4 Gb Kingston 1600mhz DDR3 ECC 12 Segate 2tb drives (RAID 6) 
Hard DriveHard DriveOSPower
ADATA SP600 SSD 24 Toshiba DT01ACA200 drives (2 RAID 6's) Windows 7 Ultimate 64 bit. OCZ ZT750 Supermicro PWS-665-PQ 
CaseOtherOtherOther
Norco 4020 and Norco 4224 LSI 9260-4i Raid Card Intel RES2SV240 20 port Expander. HP SAS 24 por... Voltaire 410Ex Hca Infiniband HBA 
  hide details  
Reply
post #16 of 34
This thread made me actually go and look at some Infiniband stuff. Turns out I can connect my 5 main PCs in the house and get a switch, cables and all for a 40gb/s inifniband network and keep the cost under $1900 total. Now that's a great price thumb.gif The 10gb/s Ethernet network equipment I bought just last year cost substantially more than that.


So @Skyl3r , Were you able to get it all set up and working with just the connection utility the cards came with and the default Windoes drivers? I am seeing a brand new Mellanox 34-port switch, and Mellanox 40gb PCI-EE adapters for cheap on Ebay so if it was relatively easy I might just upgrade part of my network to infiniband right now.
Edited by EniGma1987 - 4/11/17 at 11:48am
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #17 of 34
Thread Starter 
Quote:
Originally Posted by EniGma1987 View Post

This thread made me actually go and look at some Infiniband stuff. Turns out I can connect my 5 main PCs in the house and get a switch, cables and all for a 40gb/s inifniband network and keep the cost under $1900 total. Now that's a great price thumb.gif The 10gb/s Ethernet network equipment I bought just last year cost substantially more than that.

If you did 20gb/s, I'd wager you could do it for less than half that cost.

My roommates and I spec'd out how much it'd cost us to connect our 3 computers to the server. It came out to be $108 to add there computers to the Infiniband network. That's a total of $170 for 3 computers at 10gb/s. That also leaves space to run an additional cable from each computer to double bandwidth. Quite frankly though, I don't have the hardware to max the 10gb/s bandwidth, so no need biggrin.gif


In regards to your edit, all of the steps required to go from out of the box to working are as follows:

  1. Add mellanox cards to computers (I'm using Win10, FYI)
  2. From what I read, nothing extra should be required to use a switch. I have never verified though, so you'll have to test
  3. Connect cable(s)
  4. Boot into Windows. On my computer it immediately identified it and displayed a Mellanox IPoIB driver in Network Connections
  5. I installed the latest ConnectX (OFED) from here http://www.mellanox.com/page/software_overview_ib
  6. I HAD to update the firmware on the card before it worked. I had to use MFT for this http://www.mellanox.com/page/management_tools
    You might not have to do this, so test without doing it first. Save yourself the trouble. I will dig up how I did it if you run into problems
  7. After this you need to set them to ethernet mode. You go to Device Manager. You'll see the IPoIB drivers under Network Adapters, do NOT click this, Instead, expand System devices and you'll find the mellanox device in there. One of the tabs has two radio buttons. One says IB and one says ETH. Switch the ports over from IB to ETH.
  8. Then all you need to do is assign IP's and you're good to go!


And yes, it works flawlessly for me. I don't have to run anything on boot or anything like that. It just comes up and works. I can do remote desktop, share files and folders, etc. Just like it was ethernet.

If you need details or have questions about anything I mentioned, let me know and I'll get you specifics when I get home.
After having done this once, I am sure I could set it up again in around 10 minutes biggrin.gif It's really not a terribly difficult process. The hardest part is the lack of / Difficult to find Windows documentation.
Edited by Skyl3r - 4/11/17 at 12:09pm
post #18 of 34
I looked into this quite a bit last night and almost pulled the trigger on a Mellanox 4036E (So the switch can get a 10gb connection to my router and a 10gb connection to my second switch). All in all it wouldnt have cost me very much to upgrade to the infiniband 40gb network. But right before I put all the stuff in my cart I saw someone who did this same thing and found that even though the link speed says 10gb (in his setup), the way the network works in Ethernet mode and with Windows does not allow the full speed of the network, it actually gets 4-5x lower speed than the reported link speed should get.

So if you have a solid state drive on each end of one of your infiniband links, could you do a couple file transfers and report back on what the real world transfer rates are over your 10gb infiniband network? I need to know if things have changed in since 4-5 years ago when the guy I saw did his tests. I am hoping Windows 10 has brought some new functionality so infiniband can get the higher speed it should be getting.
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #19 of 34
Thread Starter 
Quote:
Originally Posted by EniGma1987 View Post

I looked into this quite a bit last night and almost pulled the trigger on a Mellanox 4036E (So the switch can get a 10gb connection to my router and a 10gb connection to my second switch). All in all it wouldnt have cost me very much to upgrade to the infiniband 40gb network. But right before I put all the stuff in my cart I saw someone who did this same thing and found that even though the link speed says 10gb (in his setup), the way the network works in Ethernet mode and with Windows does not allow the full speed of the network, it actually gets 4-5x lower speed than the reported link speed should get.

So if you have a solid state drive on each end of one of your infiniband links, could you do a couple file transfers and report back on what the real world transfer rates are over your 10gb infiniband network? I need to know if things have changed in since 4-5 years ago when the guy I saw did his tests. I am hoping Windows 10 has brought some new functionality so infiniband can get the higher speed it should be getting.

I'm not sure what 4-5x slower than 10gb is tongue.gif What I was hearing was that there's a processor overhead that doesn't exist with Ethernet. That said, there's other benefits besides throughput, such as extremely low latency.

I'm onboard with doing this test. As I said before, I don't have the hardware to max out the network speed; but I do have an SSD I can test with. I'll do a Crystal Disk Mark test locally on the SSD, then over the network and see what results I get and how much/if they differ. I think this will be a good test.

You could do a similar test on your 10GbE to see if overhead is obvious on Infiniband.
post #20 of 34
Quote:
Originally Posted by Skyl3r View Post

I'm not sure what 4-5x slower than 10gb is tongue.gif What I was hearing was that there's a processor overhead that doesn't exist with Ethernet. That said, there's other benefits besides throughput, such as extremely low latency.

I'm onboard with doing this test. As I said before, I don't have the hardware to max out the network speed; but I do have an SSD I can test with. I'll do a Crystal Disk Mark test locally on the SSD, then over the network and see what results I get and how much/if they differ. I think this will be a good test.

You could do a similar test on your 10GbE to see if overhead is obvious on Infiniband.

The guy whos article I was reading (dont have it saved here at work) had used a 10gb infiniband network and estimated he should get roughly 700MB/s throughput. Why 700 IDK, my estimations should put it at 1GB or slightly over. Anyway, after quite a bit of testing he found that the best it was able to achieve was 135MB/s due to the way the Infiniband functioned on Win7 and in Ethernet mode with IPoIB. Something about how the network couldnt use RDMA and other instructions or transfer types or whatever that are supposed to be inherent to Infiniband. I hope Win10 has brought improvements to this area, it would be nice if Infiniband could reach its potential on our more "home type" networks.


I have SSDs on both ends of 3 of the computers in my 10GbE network right now. So when I get home today I will do a crystalDiskMark test on each computer and then a test over the network itself and see what it comes up with.
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Network Hardware
Overclock.net › Forums › Components › Network Hardware › Infiniband for Main Storage File Server