Overclock.net › Forums › Specialty Builds › Servers › Desktop/File Server hybrid: Ubuntu+mdadm?
New Posts  All Forums:Forum Nav:

Desktop/File Server hybrid: Ubuntu+mdadm?

post #1 of 14
Thread Starter 
So for the past two years or so I've been very pleased with my Linux box, which runs Ubuntu Desktop 10.04 with Samba sharing configured for my 8x1.5 TB (~10TB usable) RAID 5 on a PERC 5i. In addition to file-sharing to my HTPC, gaming box and Android handheld, I tend to use this box as my "daily driver" for surfing the web, finding & downloading torrents, and a variety of basic desktop tasks like editing video & audio files, working on office documents, etc.

My only problem is that it's starting to approach "full" and I have a handful of HD-DVDs & Blu-Rays that I would archive if there was room for them. I also feel like I've learned a few things from my experience building & using my first "server", and I'm hoping that, with some help, I can avoid the pitfalls I ran into last time.

First and foremost, I've learned how useful it is to have an always on PC which is capable of handling simple desktop tasks, often saving me the need to power up my Windows box(es). For this reason, I'm disinclined to go with unRAID in my new build. However, the second thing I've learned is how inflexible RAID-5 is when it comes to practical matters like expanding through the purchase of new drives (either of larger capacity, or just of more quantity). This suggests that I want some sort of software RAID functionality that will allow me to do what unRAID does...

It seems that running mdadm on Ubuntu may be the best solution to my problems, and I was wondering if anyone could offer confirmation, insight and/or alternatives?

I was also hoping some folks might be able to offer some advice on hardware, as I haven't kept up with the latest motherboard, CPU & graphics chipsets. As of right now the only piece of the new system I've ordered is the case.

Most critically I'm wondering if anyone has specific recommendations for hard drive controller cards? I know this is one of the components about which unRAID is most picky, and I was worried mdadm might also be particular. I'm planning to start with 5x or 6x 3TB drives for starters, and will max out at something like 15 drives before I'd have to migrate to a real server case (which will likely be when I'm next starting over from scratch). I'm also planning on a discrete OS drive, which may end up being an SSD, and will probably run directly off of the motherboard.

I figure the need to support specific PCI-e, PCI or PCI-X hard drive cards will play a big factor in my selection of other components, but suggestions on cost effective, upgradeable and moderately powerful (but energy efficient!) motherboard/CPU/GPU Linux-friendly combos would also be welcome. Oh, and I already have some spare DDR3 sitting around, but I think pretty much all mobos are using that now anyways.
    
CPUMotherboardGraphicsRAM
Phenom II X4 830 GA-990FXA-UD5 (temp) Sapphire Radeon HD3450 G. Skill DDR3-1333 2x4GB Ripjaws 
Hard DriveHard DriveOptical DriveCooling
OCZ Vertex WD Scoprio Blue Samsung 22x DVD±RW CoolerMaster Hyper 212+ 
OSPowerCase
Ubuntu TBD (I loathe 11.10) PC Power & Cooling Silencer 760W Lian-Li PC-V2120B 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Phenom II X4 830 GA-990FXA-UD5 (temp) Sapphire Radeon HD3450 G. Skill DDR3-1333 2x4GB Ripjaws 
Hard DriveHard DriveOptical DriveCooling
OCZ Vertex WD Scoprio Blue Samsung 22x DVD±RW CoolerMaster Hyper 212+ 
OSPowerCase
Ubuntu TBD (I loathe 11.10) PC Power & Cooling Silencer 760W Lian-Li PC-V2120B 
  hide details  
Reply
post #2 of 14
You've run out of space on 10TB? Wow. So you're looking to build a new Server(ish?) PC, eh?

Anyway, mdadm is great. I use it on my server (which is only RAID 1, but still). That said, you can use any controller you like - if Linux sees it, mdadm will see it. mdadm uses your CPU as the RAID controller through software so having a dedicated RAID card isn't necessarily necessary (lol). If your motherboard has ~6 SATA ports, you could even connect all your drives to your motherboard and save yourself from having to use a dedicated controller card, since you're going to be using mdadm anyway. Or, you could perhaps re-use your PERC 5i. Up to you.

Anyway, if you're building yourself a new server PC, I'd say that the best route to go is to get yourself a previous-generation i3 or i5 (non-Sandy Bridge) simply because you can overclock them (if needed/wanted, but for a server, it's not really necessary), and that the CPUs + the motherboards are cheaper than their Sandy Bridge counterparts. Plus, you can get a motherboard that has the amount of SATA ports that you want. As far as RAM goes, DDR3 is super cheap these days, you could go ~8GB which would be plenty and be pretty cheap (~$40-50). GPU-wise, you can disregard it as it's not necessary. You can just use the integrated controller.
Orion IV
(12 items)
 
Supernaut II
(8 items)
 
 
CPUMotherboardGraphicsRAM
Core i7 3930k Gigabyte GA-X79-UD3 2x EVGA GTX 770 SC in SLI 16GB (4x4GB) G.SKILL Sniper 1600 
Hard DriveCoolingOSMonitor
ADATA 128GB SSD + 2x WD Blue 500GB AAKX (RAID 0) Thermaltake Water2.0 Extreme Dual Rad | 10x Yat... Windows 7 Ultimate x64 ASUS PB278Q, ASUS VG248QE, 2x Dell 20" 16:10 
KeyboardPowerCaseMouse
Two. Browns & Blues LEPA G900 (900W) CoolerMaster HAF 932 Advanced Logitech G700 
CPUMotherboardGraphicsRAM
AMD A6 6400k Black Edition Gigabyte F2A85XM-D3H Integrated 8GB Kingston HyperX 
Hard DriveOSPowerCase
4x 2TB WD Green (RAID 5), 80GB OS Windows 7 Ultimate 400W Generic Thermaltake V3 Black 
CPUMotherboardGraphicsRAM
Core i5 750 ASUS P55 XFX 5750 8GB Kingston 
Hard DriveOSPowerCase
160GB WD Blue Windows 7 Ultimate 600w FEMA Diablotek Short-Midtower 
  hide details  
Reply
Orion IV
(12 items)
 
Supernaut II
(8 items)
 
 
CPUMotherboardGraphicsRAM
Core i7 3930k Gigabyte GA-X79-UD3 2x EVGA GTX 770 SC in SLI 16GB (4x4GB) G.SKILL Sniper 1600 
Hard DriveCoolingOSMonitor
ADATA 128GB SSD + 2x WD Blue 500GB AAKX (RAID 0) Thermaltake Water2.0 Extreme Dual Rad | 10x Yat... Windows 7 Ultimate x64 ASUS PB278Q, ASUS VG248QE, 2x Dell 20" 16:10 
KeyboardPowerCaseMouse
Two. Browns & Blues LEPA G900 (900W) CoolerMaster HAF 932 Advanced Logitech G700 
CPUMotherboardGraphicsRAM
AMD A6 6400k Black Edition Gigabyte F2A85XM-D3H Integrated 8GB Kingston HyperX 
Hard DriveOSPowerCase
4x 2TB WD Green (RAID 5), 80GB OS Windows 7 Ultimate 400W Generic Thermaltake V3 Black 
CPUMotherboardGraphicsRAM
Core i5 750 ASUS P55 XFX 5750 8GB Kingston 
Hard DriveOSPowerCase
160GB WD Blue Windows 7 Ultimate 600w FEMA Diablotek Short-Midtower 
  hide details  
Reply
post #3 of 14
@OP

Look at my sig. rig for a guide. I use Ubuntu 10.04 LTS as a desktop, but when I build my "proper" workstation I'll be turning the sig. rig into a full-time server running Ubuntu Server 11.xx.

Storage-wise I use 6 + hot spare Samsung F4EG HD204UI drives attached to a Dell SAS 6/iR, arranged as RAID 5 using Linux software RAID + LVM. The SAS 6/iR supports 8 SATA drives via SFF-8484 cables, and uses the LSI SAS1068E chip, which is well supported by the Linux kernel. The card can be had for $50 on eBay if you're patient.

My case will support 10 3.5" total, plus a 2.5" drive for the OS, so this will ultimately be the arrangement for this box, as well as moving to RAID 6 once I hit 10 drives.

I share files mainly via Samba, although I've been experimenting with NFS also. I use combination of Disk Utility and Webmin to administer this box.
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #4 of 14
Thread Starter 
Is it true that mdadm will allow me to add drives at will, without having to wait on an array rebuild, and will support a variety of drive sizes at the same time?

10 TB is surprisingly easy to burn through with ~400 titles of DVDs, Blu-Rays and HD-DVDs, when you add in ~300+ hours of 720p TV shows...

I've also just stumbled upon ZFS and am now curious about it at as well...
Edited by EvilGenius007 - 10/14/11 at 11:38am
    
CPUMotherboardGraphicsRAM
Phenom II X4 830 GA-990FXA-UD5 (temp) Sapphire Radeon HD3450 G. Skill DDR3-1333 2x4GB Ripjaws 
Hard DriveHard DriveOptical DriveCooling
OCZ Vertex WD Scoprio Blue Samsung 22x DVD±RW CoolerMaster Hyper 212+ 
OSPowerCase
Ubuntu TBD (I loathe 11.10) PC Power & Cooling Silencer 760W Lian-Li PC-V2120B 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Phenom II X4 830 GA-990FXA-UD5 (temp) Sapphire Radeon HD3450 G. Skill DDR3-1333 2x4GB Ripjaws 
Hard DriveHard DriveOptical DriveCooling
OCZ Vertex WD Scoprio Blue Samsung 22x DVD±RW CoolerMaster Hyper 212+ 
OSPowerCase
Ubuntu TBD (I loathe 11.10) PC Power & Cooling Silencer 760W Lian-Li PC-V2120B 
  hide details  
Reply
post #5 of 14
there are so many, you could continue to use your current box and build an openfiler server specifically for storage. LVM is cool, but I find it easier to use on other linux distros like cent os. You can't really isntall unbuntu onto lvm where as you can with other linux distro. LVM would allow you to create a directory say "movies" and then in the creation of the logical volume, you could make it mirrored. LVM takes care of making sure you have your data redundant. You could always make your volumes mirrored just to be consistent, but LVM will give you the ability to add another drive and expand those volumes as they fill up. By using a volume for each thing, say photos, videos, and music as examples, you could easily track your usage.

It's on thing the system admins I work with taught me that you just don't give out all the data. So if you estimate that you will need 1.5 TB just for music, then you can allocate 1.5 and if you are approaching the limit you can just add more storage to it. LVM takes care of where the data is stored.

I remember a system admin showing me the way they had it. A software developer called them and asked for another 10 GB for extra database storage. He literally checked to make sure that he had enough un allocated space, which was like 150 GB, and then did a lvexpand and voila, there was 10 more GB on that drive.

The only other thing you could look at in the future is an external drive array.

http://www.newegg.com/Product/Produc...82E16816132016

There are so many out there. But it would allow you to continue using your current setup and just expand.
post #6 of 14
Quote:
Is it true that mdadm will allow me to add drives at will, without having to wait on an array rebuild, and will support a variety of drive sizes at the same time?
Not as far as I am aware. With Linux software RAID you can of course add drives as you grow the array - Online Capacity Expansion - and you can add more than one drive at a time, but you will have to wait for the array to rebuild as it re-weaves the data to include the new drive(s) into the array. For my system, with 2TB drives it can take 30 hours to complete an OCE.

As far as I am aware, Linux software RAID will not support using multiple different drive sizes at the same time; for each array the device size used for each array member will be the size of the smallest drive in the array. There are ways of migrating the array to bigger drive sizes, ways that are both inventive and convoluted, but they do work.

Using LVM is useful, because it allows finer control over storage allocation than simply sharing an entire array. Typically, you would create your RAID array (to gain the redundancy), then assign it to be an LVM Physical Volume (PV). Out of the PV you would then create one or more Logical Volumes (LVs). These are block devices and need to be formatted like any disk.

They can be expanded and shrunk on demand, but bear in mind that they would need to be formatted with a filesystem which supports shrinking; not many do. I personally have my 10TiB array managed with LVM through Webmin.


@herkalurk

I'm not sure that LVM supports mirroring; I know it supports striping - I'm not sure how useful that is either. It certainly isn't a patch on RAID 0...
Edited by parityboy - 10/17/11 at 5:05pm
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #7 of 14
Thread Starter 
Quote:
Originally Posted by parityboy View Post
Nor as far as I am aware. With Linux software RAID you can of course add drives as you grow the array - Online Capacity Expansion - and you can add more than one drive at a time, but you will have to wait for the array to rebuild as it re-weaves the data to include the new drive(s) into the array. For my system, with 2TB drives it can take 30 hours to complete an OCE.
Ugh, that's not really ideal at all. Looks like I get to do more research on alternatives...
    
CPUMotherboardGraphicsRAM
Phenom II X4 830 GA-990FXA-UD5 (temp) Sapphire Radeon HD3450 G. Skill DDR3-1333 2x4GB Ripjaws 
Hard DriveHard DriveOptical DriveCooling
OCZ Vertex WD Scoprio Blue Samsung 22x DVD±RW CoolerMaster Hyper 212+ 
OSPowerCase
Ubuntu TBD (I loathe 11.10) PC Power & Cooling Silencer 760W Lian-Li PC-V2120B 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Phenom II X4 830 GA-990FXA-UD5 (temp) Sapphire Radeon HD3450 G. Skill DDR3-1333 2x4GB Ripjaws 
Hard DriveHard DriveOptical DriveCooling
OCZ Vertex WD Scoprio Blue Samsung 22x DVD±RW CoolerMaster Hyper 212+ 
OSPowerCase
Ubuntu TBD (I loathe 11.10) PC Power & Cooling Silencer 760W Lian-Li PC-V2120B 
  hide details  
Reply
post #8 of 14
You're going to run into that rebuild time with anything to be honest (anything that uses parity, raid, zfs, etc.). The nice thing about it is your CPU is much, much faster than the built-in one on the RAID controller so it rebuilds much faster than on a hardware card. Also you might as well max out your ram as it uses it for caching.
Mitty
(16 items)
 
  
CPUMotherboardGraphicsGraphics
Intel i7-3930k Asrock Fatal1ty X79 Champion EVGA GTX 480 EVGA GTX 480 
GraphicsRAMHard DriveOptical Drive
EVGA GTX 480 Mushkin Redline WD Velociraptor Asus DVD±RW 
CoolingOSMonitorKeyboard
EK 2011 waterblock + 2x140 rad + respump Windows 7 Pro x64 Dell U2711 IBM Model M 
PowerCaseMouseAudio
Seasonic X-1250 Fractal ARC MIDI Logitech G9X Asus Xonar DX 
  hide details  
Reply
Mitty
(16 items)
 
  
CPUMotherboardGraphicsGraphics
Intel i7-3930k Asrock Fatal1ty X79 Champion EVGA GTX 480 EVGA GTX 480 
GraphicsRAMHard DriveOptical Drive
EVGA GTX 480 Mushkin Redline WD Velociraptor Asus DVD±RW 
CoolingOSMonitorKeyboard
EK 2011 waterblock + 2x140 rad + respump Windows 7 Pro x64 Dell U2711 IBM Model M 
PowerCaseMouseAudio
Seasonic X-1250 Fractal ARC MIDI Logitech G9X Asus Xonar DX 
  hide details  
Reply
post #9 of 14
@D-EJ915

Agreed, rebuild time is unavoidable with any kind of RAID topology, although RAID 1 is the quickest, because there's no parity to calculate, and no striping either.

One note on ZFS though: as far as I am aware, you cannot expand a vdev in ZFS once it is created. You can expand a storage pool by adding another vdev, but the data already in the pool will not automatically stripe across the new vdev; only new data added to the pool will do that.


@OP

An alternative to RAID 5/6 is FlexRAID. It works at the file level rather than the disk level, and uses no striping. You can add multiple redundancy levels to safeguard your data, and the chance of losing the entire array to disk failure is near zero.

However, you are limited to the speed of a single drive; of little consequence across GigE, but less desirable when operating locally. However, it does carry the advantage of allowing the other drives to spin down when not being used, unlike traditional RAID mechanisms.
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #10 of 14
I my main server have 4x 1TB's in it and I have Ubuntu installed on top of LVM on top of mdraid. I am looking to move it to pure LVM once I get an SSD for a main boot disk. The problem is no linux installer I have seen lets you setup LVM stripe or mirroring, so I have to setup raid 10 for that. LVM has full support for mirroring and striping, and from what I've seen it is faster than mdraid on mirrioring, but a little slower on stripes.

Another note, if you have or can afford 2GB+ of ram, I would setup a KVM virtual machine. Have the main OS be barebones, and have the file server setup as a VM. That way if you are messing around with it locks up/dies/looses network, you can VNC into the VM as if you were at the local console. It is pretty easy todo and Ubuntu has a great howto
Black Lightning
(24 items)
 
  
CPUMotherboardGraphicsGraphics
4770K Gigabyte Z87 OC Force Sapphire 290x  Sapphire 290x  
RAMHard DriveHard DriveHard Drive
Crucial Ballistix Tactical LP Toshiba Q Series Pro Samsung 840 Pro WD Black 
Hard DriveCoolingCoolingCooling
Seagate XSPC Raystorm EK-FB GA Z87X-OC Force EK-FC R9-290X - Acetal+Nickel  
CoolingCoolingCoolingCooling
EK-FC R9-290X - Acetal+Nickel  XSPC EX420 XSPC D5 Dual Bay Reservoir/Pump Combo External Chiller 
OSMonitorKeyboardPower
Windows 10 Dell U2210H x 5 On Custom Mount Logitech Orion Spark G910 Silverstone ST1000-G Evolution 
CaseMouseMouse PadAudio
Rosewill Blackhawk-ULTRA Logitech G600 Razer Destructor Sony 5.1 Surround Sound with Powered Subwoofer 
  hide details  
Reply
Black Lightning
(24 items)
 
  
CPUMotherboardGraphicsGraphics
4770K Gigabyte Z87 OC Force Sapphire 290x  Sapphire 290x  
RAMHard DriveHard DriveHard Drive
Crucial Ballistix Tactical LP Toshiba Q Series Pro Samsung 840 Pro WD Black 
Hard DriveCoolingCoolingCooling
Seagate XSPC Raystorm EK-FB GA Z87X-OC Force EK-FC R9-290X - Acetal+Nickel  
CoolingCoolingCoolingCooling
EK-FC R9-290X - Acetal+Nickel  XSPC EX420 XSPC D5 Dual Bay Reservoir/Pump Combo External Chiller 
OSMonitorKeyboardPower
Windows 10 Dell U2210H x 5 On Custom Mount Logitech Orion Spark G910 Silverstone ST1000-G Evolution 
CaseMouseMouse PadAudio
Rosewill Blackhawk-ULTRA Logitech G600 Razer Destructor Sony 5.1 Surround Sound with Powered Subwoofer 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Servers
Overclock.net › Forums › Specialty Builds › Servers › Desktop/File Server hybrid: Ubuntu+mdadm?