Overclock.net › Forums › Specialty Builds › Servers › Infiniband : first steps...
New Posts  All Forums:Forum Nav:

Infiniband : first steps...

post #1 of 8
Thread Starter 
Hello,

I finally bought the items needed to setup a cheap 10Gbs network :

MHEA28-XTC + HP IB 4X DDR (Ref : MHGA28-1TC) + SFF-8470 Cable

I flashed each one to the last fw version.

Initially, i had setup OpenSM to run as a service on each computer (2 * Win 7), as i had read that even if 2 OpenSM are running, as on is "detected", the other one becomes inactive... But, finally, I had to only keep one OpenSM starting and running, in order to have the network working.

Well, once done, i made some benchmarks after learning about the Infiniband tools (that were new to me).

I admit have been disappointed : some file transfers between smb folders, some iperf tests (playing on TCP window size), etc., gave me an average 3Gbs transfer speed :/ ...

Unless i had misunderstood, i thought being safe from the well-known TCP/IP overhead on 10GbE networks... but while testing, i thought about the fact that, even w/ Infiniband, perhaps problem will be persistent until keeping using TCP/IP ?

Anyway, i can't find any valuable reason about such a performance decrease...

(I hardly achieved 5Gbs with unrealistic TCP window size value of 1MB).

Even if i obviously noticed a load on CPU charge, it can't be the bottleneck... (E8400 & E5300).

I found some tools about RDMA protocol and was so glad when obtained such results : +900MBs and ~5µsec latency !!!

But, apart TCP/IP, i don't know any protocol from this new (for me) Infiniband world : RDAM, iSER, SRP, etc...

I can imagine that these protocols are dedicated to specific usages, but i think (hope) that thse protocols can possibly be setup for the basic usage : files transfers.

Obviously, i'm not requiring a quick drag-n-drop way as usual way. Just an achievable way to transfer files between 2 computers linked w/ my right new IB LAN !!!

Without using IPoIB in order to achieve right transfer speeds (unless someone know something that could explain the performance decrease).

Meanwhile, I read a alot about RDMA, SDP, WSD, SRP, iSER, etc...

I quite thought having found the solution : set up SRP/iSER in order to transfer files and optimize bw using iSCSI over RDMA. But, it seems impossible to setup a simple iSCSI target/client scheme between 2 Windows computers (to be accurate, iSCSI target doesn't exist for Windows (only initiator) : only for Linux w/ SCST program)).

So, still asking how some forums posts tell how much they are proud reaching 8Gbs (10Gbs cards) speeds only w/ IPoIB w/o any hassle....

Thank you.

Sincerely,

XZed
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
post #2 of 8
If you find something for iSCSI Target for windows tell me!

Great post though will be keeping an eye on this thread
Ragnarok
(15 items)
 
Yggdrassil
(8 items)
 
 
CPUCPUMotherboardGraphics
Opteron 6272 Opteron 6272 Asus KGPE-D16 Sapphire R9 290X 
GraphicsRAMHard DriveCooling
Sapphire R9 290X 8x Samsung 8GB DDR3 1333 ECC 3TBx2 in Raid 1 Noctua NH-U12DO 
CoolingOSMonitorPower
120mm Fractal Design 1200rpm Case Fan Silent Se... Windows 7 Pro x64 1x24" 2x22" Corsair Professional HX1050 
CaseAudioOther
Fractal Design XL R2 Titanium Grey Asus MIO 892 Purple and white Braided/sleeved cables 
  hide details  
Reply
Ragnarok
(15 items)
 
Yggdrassil
(8 items)
 
 
CPUCPUMotherboardGraphics
Opteron 6272 Opteron 6272 Asus KGPE-D16 Sapphire R9 290X 
GraphicsRAMHard DriveCooling
Sapphire R9 290X 8x Samsung 8GB DDR3 1333 ECC 3TBx2 in Raid 1 Noctua NH-U12DO 
CoolingOSMonitorPower
120mm Fractal Design 1200rpm Case Fan Silent Se... Windows 7 Pro x64 1x24" 2x22" Corsair Professional HX1050 
CaseAudioOther
Fractal Design XL R2 Titanium Grey Asus MIO 892 Purple and white Braided/sleeved cables 
  hide details  
Reply
post #3 of 8
Quote:
Originally Posted by Norse View Post
If you find something for iSCSI Target for windows tell me!

Great post though will be keeping an eye on this thread
Windows Storage Server 2008 includes iSCSI target.

Alternatively, StarWind iSCSI target also works.

I've used both...
ESXi Host 1
(15 items)
 
  
CPUMotherboardGraphicsRAM
(2x) Intel Xeon E5520 Dell OnBoard Matrox G200 24GB DDR3 12x2GB UDIMMS (18 slots total) 
Hard DriveHard DriveHard DriveHard Drive
PERC6-RAID50 Intel 730 480GB Intel 320 300GB Synology DS414 iSCSI SAN 
OSMonitorKeyboardPower
VMWare vSphere5 Enterprise Plus Dell iDRAC6 Remote Management [KVM-Over-IP] Dell iDRAC6 KVM Dell Hot-Swap Redundant 1100W 
CaseMouse
Dell PowerEdge T710 Stock Dell iDRAC6 KVM 
  hide details  
Reply
ESXi Host 1
(15 items)
 
  
CPUMotherboardGraphicsRAM
(2x) Intel Xeon E5520 Dell OnBoard Matrox G200 24GB DDR3 12x2GB UDIMMS (18 slots total) 
Hard DriveHard DriveHard DriveHard Drive
PERC6-RAID50 Intel 730 480GB Intel 320 300GB Synology DS414 iSCSI SAN 
OSMonitorKeyboardPower
VMWare vSphere5 Enterprise Plus Dell iDRAC6 Remote Management [KVM-Over-IP] Dell iDRAC6 KVM Dell Hot-Swap Redundant 1100W 
CaseMouse
Dell PowerEdge T710 Stock Dell iDRAC6 KVM 
  hide details  
Reply
post #4 of 8
Thread Starter 
Quote:
Originally Posted by Norse View Post
If you find something for iSCSI Target for windows tell me!

Great post though will be keeping an eye on this thread
Thank you very much ! So glad to see that i'm not alone in such a quest !

Quote:
Originally Posted by ComGuards
Windows Storage Server 2008 includes iSCSI target.

Alternatively, StarWind iSCSI target also works.

I've used both...
I can't believe not having found such easy solution ....

As soon as i can test it, i'll post here my results.

Anyway : thank you very much for your answer !
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
post #5 of 8
Thread Starter 
I read too quickly your answer...

I was talking about SRP/iSER targets for Windows :

SRP

iSER

Ayway, thank you for your answer that brought me a precious information .
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
post #6 of 8
I happen to know the iSCSI bit of server 2008 is separate: I had a subscription to MSDN and never downloaded the iSCSI-related ISO. Then the subscription expired and I decided I wanted to play with iSCSI. Well too late, can't do iSCSI.

Luckily FreeBSD, if you find the right tutorial/howto/walkthrough, works incredibly well for iSCSI.

I only comprehended a relatively small subset of your post but I did get the 3 Gbps part. Is there a possibility the hard drives/SATA controller (looked at those wikepedia links, guessing you're not using SATA...) could be the bottle neck? Again, lots of acronyms over my head, maybe I'm just missing something. Course I'd be happy with 3Gbps
Edited by subassy - 3/28/11 at 6:41pm
 
VM Server
(17 items)
 
 
CPUGraphicsRAMHard Drive
Intel Ivy Bridge Core i7-3630QM nVidia GeForce GTX 680M 16GB DDR3 1600MHz Dual Channel Memory (2 SODIMMS) Hard Drive: Serial-ATA II 3GB/s 
Hard DriveOSMonitorPower
Hard Drive: Serial-ATA II 3GB/s Windows 10 Pro x64 17.3" FHD 16:9 (1920x1080) Battery: Smart Li-ion Battery (8-Cell) 
Audio
Sound Blaster Compatible 3D Audio 
CPUMotherboardGraphicsRAM
Intel Core i7 860 Biostar T5 XE Radeon HD 5870 Corsair 16GB  
Hard DriveHard DriveOptical DriveOS
Western Digital hard drive wd1001fals-00e8b0 Maxtor 300GB I don't need no stinking optical drive Microsoft Windows 7 Ultimate x64 
MonitorMonitorKeyboardPower
HP ZR24w 24'' Samsung SyncMaster 24" logitech wireless k360 Seventeam ST-850ZAF 850W ATX 
CaseMouseAudioAudio
Thermaltake V9 Black Edition Logitech G500 Programmable Gaming Mouse FiiO E7 USB DAC and Portable Headphone Amplifier Sennheiser HD555 Professional Headphones 
  hide details  
Reply
 
VM Server
(17 items)
 
 
CPUGraphicsRAMHard Drive
Intel Ivy Bridge Core i7-3630QM nVidia GeForce GTX 680M 16GB DDR3 1600MHz Dual Channel Memory (2 SODIMMS) Hard Drive: Serial-ATA II 3GB/s 
Hard DriveOSMonitorPower
Hard Drive: Serial-ATA II 3GB/s Windows 10 Pro x64 17.3" FHD 16:9 (1920x1080) Battery: Smart Li-ion Battery (8-Cell) 
Audio
Sound Blaster Compatible 3D Audio 
CPUMotherboardGraphicsRAM
Intel Core i7 860 Biostar T5 XE Radeon HD 5870 Corsair 16GB  
Hard DriveHard DriveOptical DriveOS
Western Digital hard drive wd1001fals-00e8b0 Maxtor 300GB I don't need no stinking optical drive Microsoft Windows 7 Ultimate x64 
MonitorMonitorKeyboardPower
HP ZR24w 24'' Samsung SyncMaster 24" logitech wireless k360 Seventeam ST-850ZAF 850W ATX 
CaseMouseAudioAudio
Thermaltake V9 Black Edition Logitech G500 Programmable Gaming Mouse FiiO E7 USB DAC and Portable Headphone Amplifier Sennheiser HD555 Professional Headphones 
  hide details  
Reply
post #7 of 8
Thread Starter 
Hello,

No bottleneck as tests were made w/ forged packets from card to card.

Well, of course, 3Gbs is cool ...

Thank you for your extra informations about iSCSI experience .
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
post #8 of 8
Thread Starter 
Quote:
Originally Posted by ComGuards View Post
Windows Storage Server 2008 includes iSCSI target.
Can't believe that !!! :

Free Microsoft iSCSI Target
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
bubulle_b0x
(13 items)
 
  
CPUMotherboardRAMHard Drive
Intel Pentium Dual-Core E5300 Gigabyte GA-73PVM-S2H 2 Go DDR2-800 CL6 250Go ES.2 - PERC 6/i : 8*1To 7200.12 (RAID6) 
Optical DriveOSPowerCase
Optiarc AD-7240S Windows 7 Corsair CX400 Cooler Master Elite 310 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Servers
Overclock.net › Forums › Specialty Builds › Servers › Infiniband : first steps...