Overclock.net › Forums › Specialty Builds › Servers › File- / Media- / Hyper-v- / Web- /etc- Server with Infiniband?
New Posts  All Forums:Forum Nav:

File- / Media- / Hyper-v- / Web- /etc- Server with Infiniband?

post #1 of 7
Thread Starter 
Hi!

Actually i'm planning on rebuilding my homeserver, I use it for the things in the title and, crucial: I hate wating. I hate wating for Filetransfers, I hate waiting for my VMs, I hate waiting for anything when i sit on my computer.
So, this is the server right now:

Case: Lian Li A77B (This has to be the worst case ever built!!)
Mainboard: ASUS P8C WS
CPU: Intel Xeon E3-1240v2
RAM: Corsair Vengeance LP 1600MHz CL6, 2x 4GB
SSD: Intel 330 60GB
NIC: HP NC365T 4-port Ethernet Server Adapter + 2x Onboard 1GB Ports Teamed
Raidcontroller: Adaptec 5805
Disks: 2TB WD Caviar Black (RAID 5)
OS: Windows Server 2012

Other:
Switch: Cisco SG200-26

Now the Problem:
Part of me really likes the full switch, it looks good and the performance is amazing! But, there are just to much cables through my appartment. Now that i have to buy a new case (Rackmount, 24 Hotswap Bay, 4 HE) i'd like to upgrade. But since 10GBe is so ******* expensive i have read something about Infiniband which is quite favourable (What means I could go 40GB Infiniband (for all my clients) for the Price of 10GBe for the server and my main-rig.

But I have really no idea how infiniband works.Can I teach my Server to just through filetransfers through infiniband and rest of the traffic through ethernet? Is it possible to work with infiniband and ethernet at the same time?
Is this a good idea at all?

... i'm agonising about this ... can someone make this a little bit clearer for me?

Greetings & Thanks!

beatfried
post #2 of 7
While IB can definitely be done on the cheap, I don't think it's what you really need. Cheap IB is 10Gbps stuff (not 40Gbps), and is ideal when you have 1-2 hypervisor hosts with a remote storage solution (for home or lab use). If you are asking about putting IB adapters in all your devices and getting an IB switch, expect to spend several hundred for a decent switch, $60 for each adapter, and ~$50 for each cable. Also, IP over IB doesn't work all that great either. With Windows Server 2012, you would use RDMA and you would really only want to use it for your hypervisor storage. To use RDMA over IB in Server 2012, you need at least Mellanox ConnectX-2 cards, which are over $125 used.

If your problem is slow storage on your VMs, the first thing I would do is ditch the RAID 5 and go with RAID 10. Your IOPS will be higher, and so will your read and write speeds. Depending on the number of VMs you run, you could also just buy a 500GB SSD and still save money over the IB stuff.

If you problem is with copying files from your PC to the server, my question is how often are you copying files that are over several gigs? Copying data from VM to VM in Hyper-V is 10Gbps, as that's what the vNICs run at.

Are you looking to connect everything on your network with IB? That just won't be cost effective, and I seriously doubt you will need that. When/what are you copying to where and from where that gigabit just isn't enough? Is it multiple devices copying to/from a file server at the same time? If so, then LACP is your answer.

Let's take a step back and find out what your problem is before assuming IB is the right solution for you. Don't get me wrong, if you want to go all IB go for it, but you will still need your gigabit switch for a TCP/IP network. IB is primarily used for storage-to-device connections (for the few people that even use it). IB is dying tech, which is why it can be had for cheap. Things like 10Gbps ethernet, FC, and FCoE are more common than IB.
post #3 of 7
Thread Starter 
Thank you very much for your reply!

So, first: actually I got no performance problem. Its crazy fast right now and i like that! But i my appartment is 100qm^2 big and only one room. At the moment there are locations which got 14(!!!) cables at the same time and thats more than a 10cm cable bundle which i simply can't hide biggrin.gif I paid for the cables alone what other people pay for their whole computer -.- My main goal ist to get rid of the cables without losing (better winning -> Faster = Better) performance.

IB isnt anywhere fixed I just researched for a cheap way to go 10GbE and came across IB. I searched a little and saw that the prices ar significantly under 10GbE.

I need this performance only between my workstation and my server. Theres a backup server coming soon, which also should be in there. So my calculation:

10GbE:
1x Netgear XS708E -> 922 CHF -> 922 CHF
2x Intel X540-T1 -> 409 CHF -> 818 CHF

IB 10Gb:
1x Switch ~ 500 CHF (i found many of them...) -> 500 CHF -> 500 CHF
2x Card ~ 50 - 100 CHF -> 200 CHF
or all together: http://www.ebay.ch/itm/10Gb-Infiniband-dual-ports-cards-with-switch-and-cables-/181160154274?pt=Altro_Networking&hash=item2a2dfc94a2#ht_500wt_1414
The only expensive thing are the cables which are about 200-300 CHF for a 15m cable.
Why do you specifically name the Mellanox ConnectX-2?

So now we have 1'700 CHF vs. ~1000 CHF (with a bit of searching and waiting..)

Now the pros of the Ethernet are that I really know what i'm doing and what it can do and its futureproof. But the price is heavy :/
The Infiniband is about half the price but I don't know what i'm doing and would have a lot to learn... (well, thats not really a bad thing... but the time i'm losing..) and its dead more or less...

For your other questions: When i'm not gaming i do nothing else than copy hundrets of gigabytes from and to my workstation. I'm editing Videos and Fotos alot and the new go pro black edition with 4k resolution didn't really make it better -.- i'll also get an upgrade in terms of storage so I will get a new raidcontroller + expander and 24 WD RE 4TB HDDs plus a little bit later a Second Server (Or Maybe a NAS Box) with 24 WD RED 4 TB Drives for Backup.

I hope my English is understandable biggrin.gif Ah and CHF -> $ is more or less 1:1 but everything is more expensive here to get...
post #4 of 7
So you're running 6Gbps now and you want to make a huge investment to get to 10Gbps which may or may not work the way you want it to? You'd be better off just going with 2 4 port NICs with LACP and call it a day. That will increase your bandwidth by 2Gbps at a minimal cost.
STARSCREAM
(17 items)
 
SKYWARP
(14 items)
 
 
CPUMotherboardGraphicsRAM
Core i7-3770k ASUS Sabertooth Z77 ASUS GeForce GTX550 Ti G.Skill Ripjaws Z 
Hard DriveHard DriveHard DriveOptical Drive
Samsung 830 Western Digital Caviar Black Western Digital Caviar Green LG Bluray/HD-DVD 
CoolingOSMonitorKeyboard
Corsair H100 w/SP120 Performance Windows 7 64-bit Ultimate Dell U2410 Filco Ninja Majestouch-2 Black TKL w/Cherry MX ... 
PowerCaseMouseMouse Pad
Corsair AX650 Antec P280 CoolerMaster Storm Spawn Thunder 8 
Audio
Dell Soundbar 
CPUMotherboardGraphicsRAM
Intel Core i5-2500k ASUS Maximus V Gene HIS IceQ X Turbo Radeon 6850 G.Skill Ripjaws X 1866MHz 2x4GB 
Hard DriveCoolingOSMonitor
Samsung 830 64GB Corsair H80 w/SP120 Performance Windows 7 Ultimate Dell 2007WFP 
KeyboardPowerCaseMouse
DasKeyboard Professional NZXT Hale90 750W Antec Mini P180 Logitech G400 
Mouse PadAudio
XTrac Ripper Logitech 2.1 
  hide details  
Reply
STARSCREAM
(17 items)
 
SKYWARP
(14 items)
 
 
CPUMotherboardGraphicsRAM
Core i7-3770k ASUS Sabertooth Z77 ASUS GeForce GTX550 Ti G.Skill Ripjaws Z 
Hard DriveHard DriveHard DriveOptical Drive
Samsung 830 Western Digital Caviar Black Western Digital Caviar Green LG Bluray/HD-DVD 
CoolingOSMonitorKeyboard
Corsair H100 w/SP120 Performance Windows 7 64-bit Ultimate Dell U2410 Filco Ninja Majestouch-2 Black TKL w/Cherry MX ... 
PowerCaseMouseMouse Pad
Corsair AX650 Antec P280 CoolerMaster Storm Spawn Thunder 8 
Audio
Dell Soundbar 
CPUMotherboardGraphicsRAM
Intel Core i5-2500k ASUS Maximus V Gene HIS IceQ X Turbo Radeon 6850 G.Skill Ripjaws X 1866MHz 2x4GB 
Hard DriveCoolingOSMonitor
Samsung 830 64GB Corsair H80 w/SP120 Performance Windows 7 Ultimate Dell 2007WFP 
KeyboardPowerCaseMouse
DasKeyboard Professional NZXT Hale90 750W Antec Mini P180 Logitech G400 
Mouse PadAudio
XTrac Ripper Logitech 2.1 
  hide details  
Reply
post #5 of 7
Thread Starter 
Did you even read what I wrote?
4 more Cables is the last I want....
post #6 of 7
Quote:
Originally Posted by beatfried View Post

Did you even read what I wrote?
4 more Cables is the last I want....

Just twist the 8 ethernet cables together..... or run them through conduit. You're certainly free to make such a purchase but it seems a little ridiculous to spend so much money on something that isn't going to solve your problem. You're still going to have ethernet cables and the infiniband cables are as thick as about 3 or 4 ethernet cables anyway.
STARSCREAM
(17 items)
 
SKYWARP
(14 items)
 
 
CPUMotherboardGraphicsRAM
Core i7-3770k ASUS Sabertooth Z77 ASUS GeForce GTX550 Ti G.Skill Ripjaws Z 
Hard DriveHard DriveHard DriveOptical Drive
Samsung 830 Western Digital Caviar Black Western Digital Caviar Green LG Bluray/HD-DVD 
CoolingOSMonitorKeyboard
Corsair H100 w/SP120 Performance Windows 7 64-bit Ultimate Dell U2410 Filco Ninja Majestouch-2 Black TKL w/Cherry MX ... 
PowerCaseMouseMouse Pad
Corsair AX650 Antec P280 CoolerMaster Storm Spawn Thunder 8 
Audio
Dell Soundbar 
CPUMotherboardGraphicsRAM
Intel Core i5-2500k ASUS Maximus V Gene HIS IceQ X Turbo Radeon 6850 G.Skill Ripjaws X 1866MHz 2x4GB 
Hard DriveCoolingOSMonitor
Samsung 830 64GB Corsair H80 w/SP120 Performance Windows 7 Ultimate Dell 2007WFP 
KeyboardPowerCaseMouse
DasKeyboard Professional NZXT Hale90 750W Antec Mini P180 Logitech G400 
Mouse PadAudio
XTrac Ripper Logitech 2.1 
  hide details  
Reply
STARSCREAM
(17 items)
 
SKYWARP
(14 items)
 
 
CPUMotherboardGraphicsRAM
Core i7-3770k ASUS Sabertooth Z77 ASUS GeForce GTX550 Ti G.Skill Ripjaws Z 
Hard DriveHard DriveHard DriveOptical Drive
Samsung 830 Western Digital Caviar Black Western Digital Caviar Green LG Bluray/HD-DVD 
CoolingOSMonitorKeyboard
Corsair H100 w/SP120 Performance Windows 7 64-bit Ultimate Dell U2410 Filco Ninja Majestouch-2 Black TKL w/Cherry MX ... 
PowerCaseMouseMouse Pad
Corsair AX650 Antec P280 CoolerMaster Storm Spawn Thunder 8 
Audio
Dell Soundbar 
CPUMotherboardGraphicsRAM
Intel Core i5-2500k ASUS Maximus V Gene HIS IceQ X Turbo Radeon 6850 G.Skill Ripjaws X 1866MHz 2x4GB 
Hard DriveCoolingOSMonitor
Samsung 830 64GB Corsair H80 w/SP120 Performance Windows 7 Ultimate Dell 2007WFP 
KeyboardPowerCaseMouse
DasKeyboard Professional NZXT Hale90 750W Antec Mini P180 Logitech G400 
Mouse PadAudio
XTrac Ripper Logitech 2.1 
  hide details  
Reply
post #7 of 7
I am in currently in the process of setting up a low budget IB windows network between two machines. Two 2x20Gbps cards and two 5 meter cables were about $225USD shipped to Canada. The ones I bought are rather old though and take a bit of work to get them to work in Win 8 / Server 2012 but it is possible. The cards definitely do not support RDMA so I'll be using IPoIB. Unfortunately the board I bought for my server has a bad DIMM so it will be a few more weeks till I get results.
Big Lian Li
(13 items)
 
Forever Alone
(8 items)
 
 
CPUMotherboardGraphicsRAM
i5-6400 @ 4.7GHz Asus Z170-A Asus GTX980 Strix 4x16GB G.Skill DDR4 @ 2800MHz 
Hard DriveHard DriveMonitorMonitor
256GB Samsung 950 Pro 480GB Mushkin Striker Asus ROG Swift PG278Q  QNIX QX2710  
KeyboardPowerCaseMouse
Topre Realforce 87U Silverstone ST1000-G Lian Li PC-A77F Mionix Naos 3200 
Other
Intel 10GbE EXPX9502AFXSR  
CPUMotherboardRAMHard Drive
Opteron 6128 Supermicro H8SGL-F Kingston Registered ECC Sandisk SSD 
Hard DriveHard DrivePowerCase
OCZ Revodrive 3 FusionIO ioXtreme Silverstone Strider 550W FT02BW 
  hide details  
Reply
Big Lian Li
(13 items)
 
Forever Alone
(8 items)
 
 
CPUMotherboardGraphicsRAM
i5-6400 @ 4.7GHz Asus Z170-A Asus GTX980 Strix 4x16GB G.Skill DDR4 @ 2800MHz 
Hard DriveHard DriveMonitorMonitor
256GB Samsung 950 Pro 480GB Mushkin Striker Asus ROG Swift PG278Q  QNIX QX2710  
KeyboardPowerCaseMouse
Topre Realforce 87U Silverstone ST1000-G Lian Li PC-A77F Mionix Naos 3200 
Other
Intel 10GbE EXPX9502AFXSR  
CPUMotherboardRAMHard Drive
Opteron 6128 Supermicro H8SGL-F Kingston Registered ECC Sandisk SSD 
Hard DriveHard DrivePowerCase
OCZ Revodrive 3 FusionIO ioXtreme Silverstone Strider 550W FT02BW 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Servers
Overclock.net › Forums › Specialty Builds › Servers › File- / Media- / Hyper-v- / Web- /etc- Server with Infiniband?