Overclock.net › Forums › Specialty Builds › Servers › PROPOSED 144 TB V-Unraid box(s)
New Posts  All Forums:Forum Nav:

PROPOSED 144 TB V-Unraid box(s)

post #1 of 11
Thread Starter 
Hello all, I have been planning a new future proof file serving machine that I can build over time for my home media and backup solutions for my family of 6. The goal is to ditch all other forms of media except plex throughout my house, yet still have the versatility of shows/movies for my very diverse family. Which equals LOTS of needed space. What I came up with will be built over the next year or two, then populated with hard drives as needed. Here is the proposed build for my 144TB virtual unraid machine. It will be comprised of a 4u head unit, and 2 DIY JBOD’S. I am simply in the R&D phase right now, with the head unit halfway complete.

The host box
this will be ESXI 5 with the following specs, ESXI will be loaded to the USB that is on the face of the mobo, the unraid os’s will be on usb’s that are passed through to the client machines

Case - Norco RPC-470

Mobo/owned - P5B-E/4L

RAM/owned - 4 sticks of ecc udim 4g ram = 16 gig ram

CPU - Xeon e3-1240v2

Host SSD will simply be a 128 gig ssd directly attached to the mobo

PSU will be a single rail 750 power unit

Raid cards
PIKE 2008
passed through for 8 drive unraid client (drives will be populated in the norco rpc-470)

2 M1015 passed through to 2 identical jbods for 2 more unraid clients

The JBODS
these will both be identical with...

Case - Norco 4220

SAS Expander –
Intel RES2SV240
or
HP 468406-B21

I'm on the fence on whether to go with the intel or hp, simply because intel is 6g, but for simple file serving and hosting files for 2 plex servers im not sure if ill need much more than 3g from the hp expander, also with the hp, that will be 2 less 8088 to 8087 pci adapters I will have to buy

Any thoughts ideas and criticisms are more than welcome
    
CPUCPUMotherboardMotherboard
Intel Xeon Intel Xeon unknown unknown 
GraphicsRAMRAMRAM
Rage XL/XC Samsung  Samsung  Samsung  
RAM
Samsung  
  hide details  
Reply
    
CPUCPUMotherboardMotherboard
Intel Xeon Intel Xeon unknown unknown 
GraphicsRAMRAMRAM
Rage XL/XC Samsung  Samsung  Samsung  
RAM
Samsung  
  hide details  
Reply
post #2 of 11
Thread Starter 
one question would be... is the 1240v2 going to be enough to handle 48 wd red 3tb drives?
    
CPUCPUMotherboardMotherboard
Intel Xeon Intel Xeon unknown unknown 
GraphicsRAMRAMRAM
Rage XL/XC Samsung  Samsung  Samsung  
RAM
Samsung  
  hide details  
Reply
    
CPUCPUMotherboardMotherboard
Intel Xeon Intel Xeon unknown unknown 
GraphicsRAMRAMRAM
Rage XL/XC Samsung  Samsung  Samsung  
RAM
Samsung  
  hide details  
Reply
post #3 of 11
@OP

To clear up a couple points:

If the host machine is booted from a USB stick running ESXi, and the VMs running UnRAID are also on USB sticks, what is the SSD for?

As for the expander question, the good thing about the Intel is that it doesn't have to be powered from a PCIe slot, unlike the HP. Some "domestic" motherboards simply don't like the HP and won't boot properly with one installed. treadstone over on [H] had to modify a resistor on his board to get it to boot properly with an HP expander installed.

In your case however, I'll go ahead and assume you'll use one of those special boards that simply provides power. The Intel is still simpler to power though, since you can use a 4 pin Molex connector, and you can use motherboard stanchions and screws to secure the Intel to the side or floor of the 4220 chassis.

For power in the RPC-470, you don't need a 750W unit. A good quality 420-500W unit will be more than enough. For the JBODs, make sure that the M1015s support staggered spin-up, and get a minimum 500W good quality power supply. For further information, have a look here. smile.gif
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #4 of 11
Thread Starter 
Quote:
Originally Posted by parityboy View Post

@OP

To clear up a couple points:

If the host machine is booted from a USB stick running ESXi, and the VMs running UnRAID are also on USB sticks, what is the SSD for?

As for the expander question, the good thing about the Intel is that it doesn't have to be powered from a PCIe slot, unlike the HP. Some "domestic" motherboards simply don't like the HP and won't boot properly with one installed. treadstone over on [H] had to modify a resistor on his board to get it to boot properly with an HP expander installed.

In your case however, I'll go ahead and assume you'll use one of those special boards that simply provides power. The Intel is still simpler to power though, since you can use a 4 pin Molex connector, and you can use motherboard stanchions and screws to secure the Intel to the side or floor of the 4220 chassis.

For power in the RPC-470, you don't need a 750W unit. A good quality 420-500W unit will be more than enough. For the JBODs, make sure that the M1015s support staggered spin-up, and get a minimum 500W good quality power supply. For further information, have a look here. smile.gif

the SSD will be the drive pool that is going to host a linux based Plex/seedbox temporarily, until i can build a vm-box to host all of my process OS's. after that build the SSD will go away, and this unit will strictly for unraid vm's. or would it be possible or even better to run the unraid OS's off of an SSD? The only way i know how to use Unraid is with the loaded usb thumb drive

from what I read, the M1015 only supports staggered spin up in IR mode, you need it in IT mode when connecting it to an expander, which doesn't support staggered spin. Is there a card that you'd recommend that would play nice with either expander and do staggered spin?

I think your right though about the Intel expander, life would be far easier if I were to use a power board specifically this one it would make the JBOD's pretty damn nice:thumb:

On another note, Im having trouble finding if the PIKE 2008 card is supported (officially or unofficially) in Unraid, anyone have any info or experience with this?
    
CPUCPUMotherboardMotherboard
Intel Xeon Intel Xeon unknown unknown 
GraphicsRAMRAMRAM
Rage XL/XC Samsung  Samsung  Samsung  
RAM
Samsung  
  hide details  
Reply
    
CPUCPUMotherboardMotherboard
Intel Xeon Intel Xeon unknown unknown 
GraphicsRAMRAMRAM
Rage XL/XC Samsung  Samsung  Samsung  
RAM
Samsung  
  hide details  
Reply
post #5 of 11
With that many disks, there is bound to be a few failures even from the start. I suggest you set up your arrays and stress test them with random reads and writes for a couple days to a week before you start adding your media collection to them.

Also, keep in mind any type of hardware or software raid is not a backup solution. So keep your BDs and DVDs disks you are ripping (I assume that's what you plan on doing) as a form of backup.
VH001
(17 items)
 
VH002
(15 items)
 
VH003
(16 items)
 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 SuperMicro X9DR7-LN4F Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS 8 x Toshiba 5TB 
Hard DriveCoolingCoolingOS
Intel 750 400GB NVME SSD 2 x Dynatron R22 6 x SuperMicro 5K RPM Fans VMWare vSphere 6 
MonitorPowerCaseOther
IPMI SuperMicro 650W PSU SuperMicro CSE-833T-650B Black 3U Rackmount Ser... Intel X520-T2 Dual Port 10GbE Server Adapter 
Other
HP P420 1GB Smart Array Controller 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 Intel S2600CP2J Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS Intel 750 400GB NVME SSD 
CoolingOSMonitorPower
2 x Intel AUPSRCBTP VMWare vSphere 6 IPMI Intel 750W PSU 
CaseOtherOther
Intel P4216XXMHGR 4U Server Chassis Intel X520-T2 Dual Port 10GbE Server Adapter HP P420 1GB Smart Array Controller 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 SuperMicro X9DR3-F Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS Intel 750 400GB NVME SSD 
CoolingCoolingOSMonitor
2 x Dynatron R22 3 x SuperMicro 5K RPM Fans VMWare vSphere 6 IPMI 
PowerCaseOtherOther
SuperMicro 563W PSU SuperMicro CSE-825TQ-563LPB 2U Rackmount Server... Intel X520-T2 Dual Port 10GbE Server Adapter HP P420 1GB Smart Array Controller 
  hide details  
Reply
VH001
(17 items)
 
VH002
(15 items)
 
VH003
(16 items)
 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 SuperMicro X9DR7-LN4F Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS 8 x Toshiba 5TB 
Hard DriveCoolingCoolingOS
Intel 750 400GB NVME SSD 2 x Dynatron R22 6 x SuperMicro 5K RPM Fans VMWare vSphere 6 
MonitorPowerCaseOther
IPMI SuperMicro 650W PSU SuperMicro CSE-833T-650B Black 3U Rackmount Ser... Intel X520-T2 Dual Port 10GbE Server Adapter 
Other
HP P420 1GB Smart Array Controller 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 Intel S2600CP2J Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS Intel 750 400GB NVME SSD 
CoolingOSMonitorPower
2 x Intel AUPSRCBTP VMWare vSphere 6 IPMI Intel 750W PSU 
CaseOtherOther
Intel P4216XXMHGR 4U Server Chassis Intel X520-T2 Dual Port 10GbE Server Adapter HP P420 1GB Smart Array Controller 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 SuperMicro X9DR3-F Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS Intel 750 400GB NVME SSD 
CoolingCoolingOSMonitor
2 x Dynatron R22 3 x SuperMicro 5K RPM Fans VMWare vSphere 6 IPMI 
PowerCaseOtherOther
SuperMicro 563W PSU SuperMicro CSE-825TQ-563LPB 2U Rackmount Server... Intel X520-T2 Dual Port 10GbE Server Adapter HP P420 1GB Smart Array Controller 
  hide details  
Reply
post #6 of 11
Thread Starter 
@Jibesh

whereas I will be building the JBOD's and VM box, I wont be populating these with hard drives in bulk, (Wish I had that kind of money though)biggrin.gif I'm starting with the bare min of 2 3TB drives and a cache disk (not decided on SSD or 7200 SATA yet) and build from there. In the past I've simply just done a 48hr burn to weed out errors, and was planning on continuing that practice, whereas more time is better... well I'm impatient!

As per the backups, I have a 3tb datastore that I keep on another machine for family photo's, OS backup, and important document parity. Also a set of 1TB externals that take turns living off site for the truly important stuff. Other than that, it's all media - and I think we all agree that original DVD/Bluray/CD copies are the best backup, as you stated in your post.
    
CPUCPUMotherboardMotherboard
Intel Xeon Intel Xeon unknown unknown 
GraphicsRAMRAMRAM
Rage XL/XC Samsung  Samsung  Samsung  
RAM
Samsung  
  hide details  
Reply
    
CPUCPUMotherboardMotherboard
Intel Xeon Intel Xeon unknown unknown 
GraphicsRAMRAMRAM
Rage XL/XC Samsung  Samsung  Samsung  
RAM
Samsung  
  hide details  
Reply
post #7 of 11
So I assume you know that you will need two UnRAID license and two UnRAID installations? The latest version of UnRAID Server Pro supports up to 24 HDDs, and you will want to use at least 1 for parity giving you a maximum of 23 drives per UnRAID install. If you use 4TB drives, that gives you an approximate 92TB of storage. Seeing how you are using 2 20-bay JBODs, I suspect you will be using 19 storage drives in each so you would have to use 4TB drives to achieve your desired total amount of storage. I would also consider adding in a SSD or a velociraptor as a cache drive to speed up writes.

I say go with the Intel. Some rare few HP SAS Expanders only neogotiate SATA I (1.5Gbps), which is a bug with no known fix. Also, when using UnRAID, your performance will be no faster than 1 drive. With a 7200 RPM SATA III drive, a SATA III port on the motherboard/RAID Controller/HBA/etc will give you NO FASTER speeds than a SATA II port. There is no spindle drives that can move data faster than 3Gbps (which is the speed of a SATA II port).

While I'm not a fan of UnRAID, you can definitely do what you are talking about doing, but you will have two UnRAID server installs. You could always use something like DFS-N to make all your shares look like they come from one server; that works well for me. Make sure you test each and every drive before adding it to UnRAID, and also have spares available.

Good luck!
post #8 of 11
@OP

Controller
If you can get a cheap MegaRAID SAS 9240-4i or 9240-8i you'll have the staggered spin up. Combine that with SFF8087-SFF8088 backplates and the Intel expander, and you'll be set.

PIKE card
To be brutally honest, I've never heard of that card, and I'm willing to bet it will cause you grief. Check the UnRAID HCL, and then go buy an LSI card anyway. tongue.gif Seriously, they'll give you the least amount of hassle. In fact since you won't need an expander in the RPC-470, I can recommend a Dell SAS 6/iR or any card based on the same chip (LSI SAS1068 I think). I have one in my sig rig driving a 7 drive software RAID 6, and it works fine. Just make sure you disable it from booting and you're good to go. You can also flash it to be a vanilla LSI 3081 HBA.

That Supermicro power board looks kinda sweet. I bookmarked that page. biggrin.gif
Edited by parityboy - 5/25/13 at 2:33pm
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #9 of 11
That is the exact same motherboard that I use in my storage box, paired with a LSI MegaRAID 9261-8i.

In your case, why any RAID controllers? Why not something like a Dell PERC 6e, which can be had cheap used on eBay. Use those to pass through to your unRAID VMs to connect to your JBODs.
post #10 of 11
@tycoonbob

The PERC 6 controllers don't do JBOD mode, so UnRAID won't see the individual disks. They are a pain in that regard. Also, they are RAID controllers. tongue.gif Do you know if the LSI SAS1068 chip supports SAS expanders?
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Servers
Overclock.net › Forums › Specialty Builds › Servers › PROPOSED 144 TB V-Unraid box(s)