Overclock.net › Forums › Specialty Builds › Servers › Heavy Load Server Build
New Posts  All Forums:Forum Nav:

Heavy Load Server Build

post #1 of 28
Thread Starter 
Hi all,

I work in VFX, daytimes are at a major film effects company and evenings I generally freelance on commercial contracts...basically just to pay for me to be able to have awesome computers around the house all the time! smile.gif

I've added a few rendering nodes to my team recently and they all pull from my main workstation's HDD. So far it's been going okay like that, but not great. Noticing problems when I've got all the machines rendering frames at the same time.

Probably time for me to put together a server I figure. Now alot of folks around here are making servers simply for media, hosting games and storing random files. My work however can get as heavy as several GB for a single second of rendered animation, so my needs are quite different. Lots of storage required (would say 6TB or so at least) and lots of bandwidth + processing power.

My future plans include adding much more hardware; possibly building my render farm up to 10 "blades". That means it could be a situation where 10 machines are constantly writing LARGE images and reading textures from the server's drives. I assume this requires quite a powerful CPU in the server.

I'm leaning towards something like this (though it's not a terribly pressing need, would likely make more sense to wait on Sandy-E), but have some questions along the way:

- Xeon E3 1230 @ 3.2GHz, would this be a solid choice or should 1155 be avoided for server use?
- RAID5, 6x WD Caviar Black 2TB, or should I do 5x and setup an SSD CacheCade? (I have no idea how, but I'll learn)
- Dual LAN ports...does this make it possible to run two separate switches and double your bandwidth?
- Not sure what type of case to use. Maybe a 2U or 3U makes the most sense.
- Does Windows Home Server have the ability to support what I'm looking to do? If not, would a Linux server communicate and network nicely enough with Windows machines to allow me to render and pull files in peace?

Now another thing that I was maybe thinking about, is what if I were to build myself a Quad CPU Interlagos machine (once they're out) with 60 cores and a RAID5 setup (essentially a completely awesome server) and simply work off that while the other machines I own pull and write files to it?

Does it make more financial sense to someone like myself to do that? At least with an option like that one, I get an awesome server and an awesome workstation all at the same time. It's just a matter of whether or not I'll run into any issues if my "server" is being taxed by heavy workloads all the time (ie: rendering things while the render farm does as well).

I'm sure there's a lot of important stuff I haven't asked yet. If you can think of anything, just let me know!

Thanks in advance.
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
post #2 of 28
Is this just for storage? As I wrote in the thread next to yours, take a look at synology products. In your case, I would get a raid 5, or raid 6 capable box (nearly all of the 3+ disk devices) and they have 2 NICS that team together to give double bandwidth. Offloads all of the file work to the NAS instead of wasting storage calculations on your workstation. If you are in a windows environment, it syncs into active directory too. food for thought.

look @ http://www.newegg.com/Product/Product.aspx?Item=N82E16822108072
Forced Induction
(14 items)
 
  
CPUMotherboardGraphicsRAM
2600K Gigabyte Z68X-UD3H-B3 GIGABYTE 970 GTX 16GB DDR3 1600 Corsair Vengeance 9-9-9-24 
Hard DriveOptical DriveOSMonitor
Samsung 840 Plextor Windows 8.1 Asus PB287q 4K 
KeyboardPowerCaseMouse
Corsair Mech Corsair 850 Antec 300 MS3 
Mouse PadAudio
Big one Xonar DX 
  hide details  
Reply
Forced Induction
(14 items)
 
  
CPUMotherboardGraphicsRAM
2600K Gigabyte Z68X-UD3H-B3 GIGABYTE 970 GTX 16GB DDR3 1600 Corsair Vengeance 9-9-9-24 
Hard DriveOptical DriveOSMonitor
Samsung 840 Plextor Windows 8.1 Asus PB287q 4K 
KeyboardPowerCaseMouse
Corsair Mech Corsair 850 Antec 300 MS3 
Mouse PadAudio
Big one Xonar DX 
  hide details  
Reply
post #3 of 28
personally, dual quad core CPU, 16gigs of ram and use both gbps nic, the network would then max out before the hard drives do

Now you say you have alot of space so i'd suggest like you did, 6x2TB but run in raid 5 or 6 (10 or 8TB)

The rest of stuff you said about, after reading it twice you seem to mostly need hard drive speed and network speed. possible have two servers for differant projects so you dont end up restricting each other and due to your space requirements you'd then not risk running out of space whilst doing a project (SAN/NAS might be helpful for dumping any completed files)
Ragnarok
(15 items)
 
Yggdrassil
(8 items)
 
 
CPUCPUMotherboardGraphics
Opteron 6272 Opteron 6272 Asus KGPE-D16 Sapphire R9 290X 
GraphicsRAMHard DriveCooling
Sapphire R9 290X 8x Samsung 8GB DDR3 1333 ECC 3TBx2 in Raid 1 Noctua NH-U12DO 
CoolingOSMonitorPower
120mm Fractal Design 1200rpm Case Fan Silent Se... Windows 7 Pro x64 1x24" 2x22" Corsair Professional HX1050 
CaseAudioOther
Fractal Design XL R2 Titanium Grey Asus MIO 892 Purple and white Braided/sleeved cables 
  hide details  
Reply
Ragnarok
(15 items)
 
Yggdrassil
(8 items)
 
 
CPUCPUMotherboardGraphics
Opteron 6272 Opteron 6272 Asus KGPE-D16 Sapphire R9 290X 
GraphicsRAMHard DriveCooling
Sapphire R9 290X 8x Samsung 8GB DDR3 1333 ECC 3TBx2 in Raid 1 Noctua NH-U12DO 
CoolingOSMonitorPower
120mm Fractal Design 1200rpm Case Fan Silent Se... Windows 7 Pro x64 1x24" 2x22" Corsair Professional HX1050 
CaseAudioOther
Fractal Design XL R2 Titanium Grey Asus MIO 892 Purple and white Braided/sleeved cables 
  hide details  
Reply
post #4 of 28
@Norse, the only catch with your proposal is that system would then be a hybrid-purpose box. Rendering and massive file transferring? I don't think that mixes well... I reckon he would be best served with a dedicated storage server/device and leave all the CPU's to focus on spitting out frames. my 2 cents.
Forced Induction
(14 items)
 
  
CPUMotherboardGraphicsRAM
2600K Gigabyte Z68X-UD3H-B3 GIGABYTE 970 GTX 16GB DDR3 1600 Corsair Vengeance 9-9-9-24 
Hard DriveOptical DriveOSMonitor
Samsung 840 Plextor Windows 8.1 Asus PB287q 4K 
KeyboardPowerCaseMouse
Corsair Mech Corsair 850 Antec 300 MS3 
Mouse PadAudio
Big one Xonar DX 
  hide details  
Reply
Forced Induction
(14 items)
 
  
CPUMotherboardGraphicsRAM
2600K Gigabyte Z68X-UD3H-B3 GIGABYTE 970 GTX 16GB DDR3 1600 Corsair Vengeance 9-9-9-24 
Hard DriveOptical DriveOSMonitor
Samsung 840 Plextor Windows 8.1 Asus PB287q 4K 
KeyboardPowerCaseMouse
Corsair Mech Corsair 850 Antec 300 MS3 
Mouse PadAudio
Big one Xonar DX 
  hide details  
Reply
post #5 of 28
Quote:
Originally Posted by dbxuau;14554809 
@Norse, the only catch with your proposal is that system would then be a hybrid-purpose box. Rendering and massive file transferring? I don't think that mixes well... I reckon he would be best served with a dedicated storage server/device and leave all the CPU's to focus on spitting out frames. my 2 cents.

yea the thing is, if the workstations are transfering files, they are doing the CPU workload not the server unless the software is running on the server
Ragnarok
(15 items)
 
Yggdrassil
(8 items)
 
 
CPUCPUMotherboardGraphics
Opteron 6272 Opteron 6272 Asus KGPE-D16 Sapphire R9 290X 
GraphicsRAMHard DriveCooling
Sapphire R9 290X 8x Samsung 8GB DDR3 1333 ECC 3TBx2 in Raid 1 Noctua NH-U12DO 
CoolingOSMonitorPower
120mm Fractal Design 1200rpm Case Fan Silent Se... Windows 7 Pro x64 1x24" 2x22" Corsair Professional HX1050 
CaseAudioOther
Fractal Design XL R2 Titanium Grey Asus MIO 892 Purple and white Braided/sleeved cables 
  hide details  
Reply
Ragnarok
(15 items)
 
Yggdrassil
(8 items)
 
 
CPUCPUMotherboardGraphics
Opteron 6272 Opteron 6272 Asus KGPE-D16 Sapphire R9 290X 
GraphicsRAMHard DriveCooling
Sapphire R9 290X 8x Samsung 8GB DDR3 1333 ECC 3TBx2 in Raid 1 Noctua NH-U12DO 
CoolingOSMonitorPower
120mm Fractal Design 1200rpm Case Fan Silent Se... Windows 7 Pro x64 1x24" 2x22" Corsair Professional HX1050 
CaseAudioOther
Fractal Design XL R2 Titanium Grey Asus MIO 892 Purple and white Braided/sleeved cables 
  hide details  
Reply
post #6 of 28
What are you willing to invest in this system?

You could start with a beefy home made SAN for the storage; then some day move to a blade enclosure and a couple of blades.

With a good storage back end you can use existing systems for the processing work load.

The storage server would be just a single socket solution with a low end dual/quad core CPU. Anandtech has some good reads on creating a DIY SAN if that is the route you wish to pursue.

There are a number of good reads on using large numbers of consumer grade, mid range systems for the rendering.
    
CPUMotherboardGraphicsRAM
C2D T7100 1.8 ghz (undervolted) ummm... Dell Intel X3100 2 x 1gb 667mhz 
Hard DriveOptical DriveOSMonitor
Fujitsu 7200 RPM 120gb CD-RW/DVD dual boot Vista business 1440x900 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
C2D T7100 1.8 ghz (undervolted) ummm... Dell Intel X3100 2 x 1gb 667mhz 
Hard DriveOptical DriveOSMonitor
Fujitsu 7200 RPM 120gb CD-RW/DVD dual boot Vista business 1440x900 
  hide details  
Reply
post #7 of 28
On the network side, you would need managed switches to capitalize on port channeling, but if you made a low overhead linux system just for files storage you would be set. Storage your on the right track with raid5 or 6, perhaps you could do raid 5 with a hot spare already spinning. I work at a university, and our main remote drive server uses 2 1gig ports grouped together. The students don't push nearly as much data at a time as you're looking so you may want to buy a 4 port PCI E x4 nic and group all ports.
post #8 of 28
@OP:

It sounds like what you need is fast, centralised and networked storage. Personally, I would suggest a dedicated storage box that did storage and nothing else.

Since this is a server build, I'd look at getting something like a Supermicro X9SCM for the motherboard.

The Xeon CPU you listed should be fine, along with some ECC RAM. For storage, I'd say a "plain Jane" 8-port HBA (like a Dell SAS 6/iR, probably $50-60 on eBay) combined with the software RAID 6 capabilities of OpenFiler or the RAIDZ2 capabilities of FreeNAS would be perfect. Personally, with 8 drives I'd go straight to RAID 6.

If you think hardware RAID is the way to go, then hit eBay and bag yourself a Dell PERC 6/i for $120. Both OpenFiler and FreeNAS will support the PERC 6/i, AFAIK.
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #9 of 28
If the rendering isn't being done on the to-be-built server itself, you don't need anything more than a recent desktop-class dual core. If the server will be doing any rendering at all, then you'd be correct in going to server-class Xeon chips.

As others have said, RAID5/6 is the way to go. With RAID 5 I'd say throw together a small Linux box to act as a storage server with 2-4 NICs (can be a single 4-port NIC) teamed for higher bandwidth. Buy an extra drive to act as a hot-spare for the RAID 5 array. Be sure to buy enterprise-class drives like the WD RE4s or Samsung F3Rs. Better firmware, better hardware and better warranty, for not much more money. The hot-spare will also allow you to have some breathing room to replace the drive (or more accurately, load another hot-spare) in the event of a failure. For this, use hot-swap bays obviously.

If you decide to go with RAID 6, I suggest hardware RAID. There's a bit of debate on this subject but suffice to say this is my opinion and my recommendation.

Either way, the standard 7200 RPM drives should be enough for your network bandwidth needs. If you think you'll saturate more than 2gbps of bandwidth, you could check out 10K drives but that's oddles more cash. With RAID 5 more drives = more speed, so 8 drives is pretty damn snappy.

Again, if no rendering is being done on this server RAM wouldn't really be an issue. I would think 4GB would be enough, but I don't have experience in render farm setups. 8GB of ECC RAM is still relatively cheap however, and gives you some overhead.


As far as networking equipment goes, as said before you'd need managed switches to handle the NIC teaming on the switch end of things. My recommendation is to go with Juniper or Cisco equipment. Stay far, far, far, FAR away from Dell, HP, etc... brand smart/managed switches. You'll be in for nothing but headaches. If you go the Cisco route, at very minimum talk to a Cisco rep if you don't know Cisco hardware intimately. Yes, they'll try and up sell you, but they're usually pretty decent at being straight forward with what product will do exactly what.
    
CPUMotherboardGraphicsRAM
Core i7 970 @ 4.0 GHz 1.22 Vcore Asus Rampage II Gene GTX 260 216SP G.SKILL PI 3x2gb DDR3 1600 @ 7-8-7-24 
Hard DriveOSMonitorPower
2x 500gb Seagates RAID 0, 1x 500gb non-RAID Windows 7 Professional x64 ASUS 24'' VH242H / Spectre 24'' WS Corsair 750TX 
Case
Corsair 300R 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Core i7 970 @ 4.0 GHz 1.22 Vcore Asus Rampage II Gene GTX 260 216SP G.SKILL PI 3x2gb DDR3 1600 @ 7-8-7-24 
Hard DriveOSMonitorPower
2x 500gb Seagates RAID 0, 1x 500gb non-RAID Windows 7 Professional x64 ASUS 24'' VH242H / Spectre 24'' WS Corsair 750TX 
Case
Corsair 300R 
  hide details  
Reply
post #10 of 28
@TurboTurtle

Just to pick up on a couple of the points you've made:

1. If you're recommending Linux MD RAID (i.e software RAID) for a RAID setup, why not include ECC RAM, considering this is work machine?

2. Samsung F3Rs: are they actually for sale? I've never seen them in the retailers. tongue.gif

3. I'm glad you make the point about Dell switches. I was actually considering these for my business, along with some Dell servers and workstations; are their switches really that bad?
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Servers
Overclock.net › Forums › Specialty Builds › Servers › Heavy Load Server Build