Overclock.net › Forums › Components › Network Hardware › Infiniband for Main Storage File Server
New Posts  All Forums:Forum Nav:

Infiniband for Main Storage File Server - Page 4

post #31 of 34
Quote:
Originally Posted by beers View Post

You need a different subnet than your other NIC is on.

Even when they are a completely separate network right now and not touching the main Ethernet? I only had the second NIC enabled so I could make those posts cause the Infiniband network doesn't have internet access right now till I get a switch and actually integrate the networks. And what should I do when I have my Infiniband switch that all my main computers connect to, and that switch is connected to the main networks 10G Ethernet switch? Here is an image of how my network will be laid out come next week:





I realize it is not the typical enterprise layout. Once I get a new router that has a 10G SFP+ port on it then I will connect the router straight to the Infiniband switch, and the Infiniband switch will connect to the 10G Ethernet switch. But that will come later this year.
Right now all "my" traffic is completely on the 10GbE and Infiniband side of the network. All guest traffic and traffic for roomates devices are on the 1G side of the network.
With how I showed the network in the picture above, wouldnt the Infiniband switch stay on the same 192.168.1.x subnet as the rest of the network to allow me to talk to other devices of mine and get out to the internet?













As for benchmarks for the Infiniband 40gb and 80gb LAG:



wooo its working! wheee.gifwheee.gif





lol I think I broke it. But both computers report this when LAG is set up.






Transfer speed of a 5GB movie from one PC to the other. This would be faster, but my m.2 drives max out around this, and the RAMDisks I have wont go higher than 1GB unless I buy a license. And 1GB file transfer is over so quick I cant get a screenshot. tongue.gif





Crystal Disk Mark test of a RAMDisk over Infiniband 40gb




Crystal Disk Mark test of a RAMDisk over the Infiniband 80gb network.






LAN Speed Test (Lite) reports this. For some reason the write is still pretty low, though it is far higher than what this tool reported over the 10GbE network. About 2.5x higher. Drive "Y" is the RAMDisk when mounted over the network
Edited by EniGma1987 - 4/17/17 at 9:34pm
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
Gaming
(17 items)
 
Gaming PC
(20 items)
 
 
CPUMotherboardGraphicsRAM
7700K AS Rock Z170 OC Formula Titan X Pascal 2050MHz 64GB DDR4-3200 14-14-14-34-1T 
Hard DriveHard DriveHard DriveCooling
950 EVO m.2 OS drive 850 EVO 1TB games drive Intel 730 series 500GB games drive Custom water cooling 
OSMonitorKeyboardPower
Win 10 Pro x64 AMH A399U E-Element mechanical, black switches, Vortex b... EVGA G3 1kw 
CaseMouseAudioAudio
Lian-Li PC-V1000L Redragon M901 LH Labs Pulse X Infinity DAC Custom built balanced tube amp with SS diamond ... 
Audio
MrSpeakers Alpha Prime 
  hide details  
Reply
post #32 of 34
Thread Starter 
Quote:
Originally Posted by EniGma1987 View Post

Even when they are a completely separate network right now and not touching the main Ethernet? I only had the second NIC enabled so I could make those posts cause the Infiniband network doesn't have internet access right now till I get a switch and actually integrate the networks. And what should I do when I have my Infiniband switch that all my main computers connect to, and that switch is connected to the main networks 10G Ethernet switch? Here is an image of how my network will be laid out come next week:

Due to what I described in my last post, it's a problem specifically because they are separate networks. If you disabled your ethernet adapter, I expect it would work fine with the IP's you said previously.
Once you have the network setup as shown in your topological picture, this shouldn't be a problem anymore. That looks like a fine setup for the network.

Warning: Spoiler! (Click to show)
Quote:
Originally Posted by EniGma1987 View Post

Lastly, I wanted to try running a Crystal Disk Mark on the RAMDisk itself when it was mounted as a mapped network drive. So this is the speed of the disk over the network:
Warning: Spoiler! (Click to show)
Quote:
Originally Posted by EniGma1987 View Post


Crystal Disk Mark test of a RAMDisk over Infiniband 40gb

Crystal Disk Mark test of a RAMDisk over the Infiniband 80gb network.

Looks like a pretty nice bump in performance there thumb.gif
I've seen a couple people test out infiniband as I did, but you'll be the first person I've heard of using it in a home network like this. Hope it goes well!
post #33 of 34
A cheaper option is probably to just get an HDD for storage connected via SATA.
But sure this file server option connected via 10gbps+ networks has been done before countless times.
The infiniband are a popular choice because you can get used adapters relatively cheap, ebay/amazon, but getting a switch or router with these connections and speeds is still going to be pricey I think if you want to connect multiple machines. Plus outside US the prices may not be that great.
On top of that your server would need a decent storage setup to be able to get high speeds if that is what you want, meaning HDDs are slow even in RAID and SSD server storage for home? A bit overkill.

Seems like a hefty investment if you want high speed networked storage for multiple devices. If you need just one then using the regular SATA/PCIe ports is better.

LTT has forum and videos about making one or two infiniband setups.

For testing network speed and disks there is IOMETER, free. More of a pro tool so don't expect user friendly one click solution.

Personally I don't understand this ramdisk stuff, why not just run a bandwidth and latency test with a program, why bother with ramdisk and crystal disk mark.
What are your permanent drive to permanent drive speeds? What are your real network speeds? I guess you are using ramdisk because your drives are not fast enough to test with programs such as CDM?
Try the IOMETER it can measure disks, networks, ...

What do you use to manage and share/access the file server? Is it just a file server or setup to be able to run virtual machines as well?

Works for you, enjoy the infiniband.
Edited by JackCY - 4/18/17 at 5:22am
post #34 of 34
Thread Starter 
Quote:
Originally Posted by JackCY View Post

A cheaper option is probably to just get an HDD for storage connected via SATA.
But sure this file server option connected via 10gbps+ networks has been done before countless times.
The infiniband are a popular choice because you can get used adapters relatively cheap, ebay/amazon, but getting a switch or router with these connections and speeds is still going to be pricey I think if you want to connect multiple machines. Plus outside US the prices may not be that great.
On top of that your server would need a decent storage setup to be able to get high speeds if that is what you want, meaning HDDs are slow even in RAID and SSD server storage for home? A bit overkill.

An HDD connected via SATA is probably cheaper than 5 computers connected via infiniband. I don't know if that's relevant though, because they both serve different purposes.

I don't know what sort of setup EniGma has, but he already has 10GbE and he's looking for more; so I'm assuming he's aware of the limitations of his hardware.
For me personally, I have the hardware to see maybe 3gbps. The problem is that I need to see that 3gbps to make use of my fileserver in the capacity I want. Infiniband was the best/cheapest option to reach my goal.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Network Hardware
Overclock.net › Forums › Components › Network Hardware › Infiniband for Main Storage File Server