Overclock.net › Forums › Specialty Builds › Servers › 10GB/s Fiber NAS
New Posts  All Forums:Forum Nav:

10GB/s Fiber NAS

post #1 of 13
Thread Starter 
[ame="http://www.amazon.com/Fiber-Optic-Cable-Multi-Duplex/dp/B003WP1OAU/ref=sr_1_7?s=electronics&ie=UTF8&qid=1295601371&sr=1-7"]Amazon.com: 10Gb Fiber Optic Cable, LC/SC, Multi Mode, Duplex - 5 Meter (50/125 Type) - Aqua: Electronics[/ame]

what is required for me to build a NAS based on 10GB/s fiber? Do I just need an Emulex PCI-E card?
post #2 of 13
Well your server is going to need a PCI-e Fiber network card and you will need a 10GB fiber router to connect to your network, probably a Cisco Catalyst router or switch.
post #3 of 13
@OP:

So what's inside your NAS, SSDs in RAID?
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #4 of 13
What do you have that can saturate 10 gigabit? wink.gif

You'll likely be limited on standard gigabit unless you are using SSD's on workstations and a RAID array on your server. Even then, ~100-110 MB/sec isn't that much to wait on ...

The cost for your server's NIC, a 10 Gb switch and NICs + SSD for your workstations will likely cost more than this undertaking is worth.
Edited by beers - 1/21/11 at 2:51am
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
post #5 of 13
Thread Starter 
I'm using 256GB SSDs and 2TB RE4 drives in RAID. I think i have it sorted out. I'm installing openfiler and then I will have that partitioned to my ESXi cluster.

I'm not using SCSI because they are so dang expensive. I might as well get SSDs.
Quote:
Originally Posted by beers;12087301 
What do you have that can saturate 10 gigabit? wink.gif

I put the rest of my servers' virtual memory on the SAN that has SSDs and it gets 73% of the performance of DDR2 at less than a fifth of the cost. And since it's in RAID it doesn't matter when a SSD fails because i can just pop it out and replace it, then RMA the one I trashed and get another tongue.gif. Warranties FTW

I think I've got it from here, did a lot of research today. Thanks biggrin.gif +rep
Edited by pon - 1/21/11 at 12:50pm
post #6 of 13
Quote:
Originally Posted by pon;12091781 
I put the rest of my servers' virtual memory on the SAN that has SSDs and it gets 73% of the performance of DDR2 at less than a fifth of the cost. And since it's in RAID it doesn't matter when a SSD fails because i can just pop it out and replace it, then RMA the one I trashed and get another tongue.gif. Warranties FTW

Interesting use...sounds like the kind of thing that'd appear in a supercomputing lab. As SSDs get faster, and 10GigE gets cheaper, we may well see this sort of thing appear more in production environments.

Might sound loopy now, but in 10-15 years' time when 40GigE and 100GigE reach commodity...? Memory Area Network? tongue.gif Oh...one more thing...what do you have that's memory-intensive, just out of interest?
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #7 of 13
I'd be interested to know where you get the 73% of performance stat from - the latency alone will kill off performance much more than that, not to mention the bandwidth caps due to the PCIe buses.

Also, why do you want a fiber NAS? If you basically want more RAM for caching etc, why not put the SSDs directly in the hosts, as you would have otherwise done with the RAM? Or instead set up an infiniband DAS for each host - this would likely be much cheaper (you can get second hand 10G infiniband adapters pretty cheap (~$75 BIN from eBay). A full 10G fiber switch & NICs is going to be pretty spendy, but if you can do it as DAS instead you should save a fair bit of cash.
post #8 of 13
Quote:
Originally Posted by the_beast;12098827 
I'd be interested to know where you get the 73% of performance stat from - the latency alone will kill off performance much more than that, not to mention the bandwidth caps due to the PCIe buses.

That's depending on the boards' capabilities.

I've seen those kind of virtual memory interfaces previously, in a resort network our company managed.

It had 4 HP ProLiant G6 (2x quadcore Xeon Conroe-based with 4GB DDR2-667 registered & FB each) servers on RAID0+1 made of Seagate Cheetah SCSI drives, 74GB drives. The servers were wired together along 1km^2 of land with 8.4Gb/s optical interfaces with PCI Express LAN cards.

It performed incredibly well, don't know up to what level, but performance was more than 10x the performance of the virtual memory using only system drives.

All wired up to a kind of datacenter that contained databases of millions of customers, and thousands of registers of different parameters from all over the resort. The processing was done on the servers, then sent to the datacenter. The datacenter buffered all the data and spooled it to the RAID drives inside it.


So yes, it works. Not up to a physical RAM level, but sure it does work WAY better than regular virtual memory on a single system disk. And if you swap HDDs for SDDs, performance will be way better.
   
AGP bencher
(14 items)
 
CPUMotherboardGraphicsRAM
Ryzen R7 1700 Gigabyte GA-AX370-Gaming 5 Sapphire HD 6950 2GiB 2x8GB KFA2 HOF DDR4-3600 
Hard DriveHard DriveHard DriveHard Drive
Crucial MX100 256GB Seagate 600 Series 240GB Seagate 7200.14 2TB Samsung F3 1TB 
CoolingCoolingCoolingCooling
EKWB Supreme HF XSPC Rasa GPU EK XT360 EK 4.0 
OSMonitorMonitorKeyboard
W10 Pro LG IPS235 LG E2250V KUL ES-87 
PowerCaseMouseAudio
SF Leadex II 650W Lian Li PC-A05NB Logitech G9 Xonar DX 
AudioAudio
SMSL SA-S3+Technics CB-250 Sennheiser HD555 
CPUMotherboardRAMHard Drive
AMD A10-5700 Gigabyte F2A75M-HD2 G.SKILL Ares 2133 CL9 Hitachi 5K750 
Hard DriveCoolingOSMonitor
Momentus .7 200GB Noctua NH-L9a Server 2012 R2 Standard AUO B156HW01 
PowerCaseOther
PicoPSU-80-WI-25V AIO Aluminium Handmade TP-Link Archer Something Something Wi-Fi AC 
CPUCPUCPUMotherboard
Core2Duo E6400 Core2Quad Q6600 Pentium Dual Core E5200 AsRock 4COREDUAL-SATA2 R2.0 
GraphicsRAMHard DriveOptical Drive
A dumpload of ancient AGP cards Kingston Value DDR2-667 CL4 2T @CL3 1T Seagate 160GB 7200.10 LG IDE DVD-ROM 
CoolingCoolingOSMonitor
Ghettomade CPU waterblock 49cc 2stroke engine copper radiator WinXP SP2 32bit ProView 17" 
PowerCase
Tacens Radix V 550W Ghetto aluminium bench 
  hide details  
Reply
   
AGP bencher
(14 items)
 
CPUMotherboardGraphicsRAM
Ryzen R7 1700 Gigabyte GA-AX370-Gaming 5 Sapphire HD 6950 2GiB 2x8GB KFA2 HOF DDR4-3600 
Hard DriveHard DriveHard DriveHard Drive
Crucial MX100 256GB Seagate 600 Series 240GB Seagate 7200.14 2TB Samsung F3 1TB 
CoolingCoolingCoolingCooling
EKWB Supreme HF XSPC Rasa GPU EK XT360 EK 4.0 
OSMonitorMonitorKeyboard
W10 Pro LG IPS235 LG E2250V KUL ES-87 
PowerCaseMouseAudio
SF Leadex II 650W Lian Li PC-A05NB Logitech G9 Xonar DX 
AudioAudio
SMSL SA-S3+Technics CB-250 Sennheiser HD555 
CPUMotherboardRAMHard Drive
AMD A10-5700 Gigabyte F2A75M-HD2 G.SKILL Ares 2133 CL9 Hitachi 5K750 
Hard DriveCoolingOSMonitor
Momentus .7 200GB Noctua NH-L9a Server 2012 R2 Standard AUO B156HW01 
PowerCaseOther
PicoPSU-80-WI-25V AIO Aluminium Handmade TP-Link Archer Something Something Wi-Fi AC 
CPUCPUCPUMotherboard
Core2Duo E6400 Core2Quad Q6600 Pentium Dual Core E5200 AsRock 4COREDUAL-SATA2 R2.0 
GraphicsRAMHard DriveOptical Drive
A dumpload of ancient AGP cards Kingston Value DDR2-667 CL4 2T @CL3 1T Seagate 160GB 7200.10 LG IDE DVD-ROM 
CoolingCoolingOSMonitor
Ghettomade CPU waterblock 49cc 2stroke engine copper radiator WinXP SP2 32bit ProView 17" 
PowerCase
Tacens Radix V 550W Ghetto aluminium bench 
  hide details  
Reply
post #9 of 13
Those servers must have been wired together with FibreChannel, or something close to it. Even that would be cheaper than 10GigE. tongue.gif

Seriously though, as exotic as your proposal is, right now you'd get more and cheaper performance by attaching a decent 32GB SSD directly to the host server. I plan on doing something similar to my next workstation build - using an SSD for virtual memory.
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #10 of 13
Hrmm... I haven't decided what's next on my plate...
ESXi Host 1
(15 items)
 
  
CPUMotherboardGraphicsRAM
(2x) Intel Xeon E5520 Dell OnBoard Matrox G200 24GB DDR3 12x2GB UDIMMS (18 slots total) 
Hard DriveHard DriveHard DriveHard Drive
PERC6-RAID50 Intel 730 480GB Intel 320 300GB Synology DS414 iSCSI SAN 
OSMonitorKeyboardPower
VMWare vSphere5 Enterprise Plus Dell iDRAC6 Remote Management [KVM-Over-IP] Dell iDRAC6 KVM Dell Hot-Swap Redundant 1100W 
CaseMouse
Dell PowerEdge T710 Stock Dell iDRAC6 KVM 
  hide details  
Reply
ESXi Host 1
(15 items)
 
  
CPUMotherboardGraphicsRAM
(2x) Intel Xeon E5520 Dell OnBoard Matrox G200 24GB DDR3 12x2GB UDIMMS (18 slots total) 
Hard DriveHard DriveHard DriveHard Drive
PERC6-RAID50 Intel 730 480GB Intel 320 300GB Synology DS414 iSCSI SAN 
OSMonitorKeyboardPower
VMWare vSphere5 Enterprise Plus Dell iDRAC6 Remote Management [KVM-Over-IP] Dell iDRAC6 KVM Dell Hot-Swap Redundant 1100W 
CaseMouse
Dell PowerEdge T710 Stock Dell iDRAC6 KVM 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Servers
Overclock.net › Forums › Specialty Builds › Servers › 10GB/s Fiber NAS