Overclock.net › Forums › Specialty Builds › Servers › Heavy Load Server Build
New Posts  All Forums:Forum Nav:

Heavy Load Server Build - Page 3

post #21 of 28
I agree on the RAM, ZFS will swallow whatever you throw at it. tongue.gif A pretty cheap price to pay for the level of data integrity you get, and as long as the box isn't doing anything else, performance should be good.
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #22 of 28
Thread Starter 
Quote:
Originally Posted by darknight670;14625479 
Sure CPU is quite important but RAM is in my opinion a lot more important with ZFS. But a Quad Core should be enough to reach 1Go/s with ZFS.

Honestly, CPUs are so cheap these days if you're just looking for a single socket machine. For $300 I can throw a 3.4GHz quad core Sandy Bridge at this thing and not have to worry about having enough CPU power for a few years I bet.

I've been getting into Mari lately as well which is an extremely capable tool set (it's also extremely capable of generating 20+ GB texture directories) so bandwidth might start to get even more important down the road. Mari allows you to paint 3D geometry with full floating point HDR textures and the memory overhead starts to get crazy.

So with all this having been said and let's say my end goal is a setup along these lines:
1 completely badass workstation; dual overclocked SB-E 8 core Xeons + Quad SLI GTX780 (those cards will probably be out in the next year or so).
10 or so render boxes (likely also all dual overclocked 8 core Xeons).

Do I really NEED a server? Can I buy an amazing 10Gbps NIC for the workstation that will pump out data very quickly to a 1Gbps switch to feed the render nodes? Actually...would the switch also need to be 10Gbps if all I care about is a 1Gb connection to each render node?

I'm not even adverse to an SSD RAID setup in the workstation or buying one of those enterprise level SSD cards.

I put a 2TB HDD into each render node considering how cheap they are these days, so backups and file STORING isn't an issue at all as I basically have a collective 20TB of disk space on the nodes alone. File SPEED on the other hand could be optimized a ton on the main workstation.

Is a server a waste of money for me when all is said and done? Is it better spent on the main machine that I work on? For the $2-$3K to build the server, I could have an entry level enterprise SSD in the workstation instead and then get the benefit of working directly off that.

I've recently gone from an i7 975EE to an i7 980 in my main PC and I think even just adding those extra 2 cores has already made a decent difference for I/O speeds to my render nodes. They no longer choke for lengthy (upwards of 20 secs or so previously) periods of time upon completion of renders while my main PC (also @ 100% CPU load) tries to save the file.

Another important question actually is in terms of Windows licenses. I've been outfitting the render nodes with simple cuts of Win 7 Home Premium...but I believe that the main drawback of that OS is you can't have a network with more than 10 other PCs.

If my main workstation is running Windows Pro or Ultimate or whatever, can I connect with more than 10 other computers even if they can't connect with each other? Would that work out at all or do all machines need Win Pro?

Does my main machine possibly need to be running a variant of Windows Server and if so...can I get away with running WHS or would I need one of those $700+ copies of Windows?

Also how scalable would this all be? What if I want to go to 20 render nodes some day?

Thanks again guys, the server forum here is amazing. I know I'm asking a ton of questions, but I really want to learn about all this stuff.
Edited by kweechy - 8/19/11 at 10:33pm
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
post #23 of 28
It seems you already have a workstation that plays the role of distribution "server" and that your problem is bandwidth...

With 10 rendering nodes you are far beyond the power-user setup.

The only way to improve the responsiveness of your setup would be :
1/ SSD in the workstation
2/ Connections higher than 1Gbit ( 10+ Gbit )

But these things ( especially the 10Gbit connections ) costs a LOT and are quite uncommon for non professional users

I think you should go on hardforum ( networking ) and ask here.
Store-IT
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 955 BE Gigabyte 880GA-UD3H v2.2 Integrated 8 x 2 Gio DDR3 
Hard DriveOSPowerCase
6 x 2 To Samsung F4, 2 x 1 Tb F3 Openindiana 151b 430W Seasonic Fractal R3 
  hide details  
Reply
Store-IT
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 955 BE Gigabyte 880GA-UD3H v2.2 Integrated 8 x 2 Gio DDR3 
Hard DriveOSPowerCase
6 x 2 To Samsung F4, 2 x 1 Tb F3 Openindiana 151b 430W Seasonic Fractal R3 
  hide details  
Reply
post #24 of 28
Quote:
Originally Posted by parityboy;14578344 
@TurboTurtle

Just to pick up on a couple of the points you've made:

2. Samsung F3Rs: are they actually for sale? I've never seen them in the retailers. tongue.gif

Fry's has them in stock all the time. They are only really worth paying the extra dough for the considerably longer warranty.
"The SR-2 Grid"
(21 items)
 
  
CPUMotherboardGraphicsGraphics
2x Xeon e5620 @ 3.4 Ghz Evga SR-2 ATI 5870 ATI 5870 
GraphicsRAMHard DriveHard Drive
Nvidia GTS 240 12 GB DDR3 Corsair Dominator OCZ Vertex III ... 2x Samsung Spinpoint F3 1 TB 
Hard DriveHard DriveHard DriveOptical Drive
300 GB WD Velociraptor WD Caviar Blue 640 GB WD Caviar Green 500 GB LG Blu Ray Combo Drive 
CoolingOSMonitorKeyboard
Corsair H50 x2 Windows 7 Professional / Snow Leopard/ubuntu/Win 7 LG 55 inch tv, I-Inc 28 inch, Acer 20 Inch, 11.... Corsair K90, Logitech g13 
PowerCaseMouseMouse Pad
Antec CP-1000 Coolermaster Cosmos II Razer DeathAdder Drumpad 
Audio
Sennheiser HD 650 
  hide details  
Reply
"The SR-2 Grid"
(21 items)
 
  
CPUMotherboardGraphicsGraphics
2x Xeon e5620 @ 3.4 Ghz Evga SR-2 ATI 5870 ATI 5870 
GraphicsRAMHard DriveHard Drive
Nvidia GTS 240 12 GB DDR3 Corsair Dominator OCZ Vertex III ... 2x Samsung Spinpoint F3 1 TB 
Hard DriveHard DriveHard DriveOptical Drive
300 GB WD Velociraptor WD Caviar Blue 640 GB WD Caviar Green 500 GB LG Blu Ray Combo Drive 
CoolingOSMonitorKeyboard
Corsair H50 x2 Windows 7 Professional / Snow Leopard/ubuntu/Win 7 LG 55 inch tv, I-Inc 28 inch, Acer 20 Inch, 11.... Corsair K90, Logitech g13 
PowerCaseMouseMouse Pad
Antec CP-1000 Coolermaster Cosmos II Razer DeathAdder Drumpad 
Audio
Sennheiser HD 650 
  hide details  
Reply
post #25 of 28
I second the raid 0+1. You lose quite a bit of write performance with raid 5 and 6.
 
VM Server
(15 items)
 
 
CPUMotherboardGraphicsRAM
2600k ASRock Z68 Extreme 4 Sapphire 7970 8GB Vengeance 1600 CL8 
Hard DriveOptical DriveOSMonitor
Crucial M4 256GB HP dvd1160i Windows 7 x64 Ultimate Samsung BX2231  
MonitorMonitorMonitorKeyboard
Samsung BX2231  Samsung BX2231  Panasonic 42" Plasma  Ducky Shine Green LED with Red Switches 
PowerCaseMouse
Corsair 750w Fractal Midi  G500 
CPUMotherboardRAMHard Drive
Intel i7 3770s Asrock Z77 Extreme-m 32GB G.SKILL Ripjaws X Series Samsung F3 1TB 
Hard DriveHard DriveHard DriveHard Drive
Western Digital 2TB Green Seagate Barracuda 1Tb  Western Digital 500GB Western Digital 250GB 
CoolingOSPowerCase
Dark Knight II ESXi 5.1 Antec Earth Watts 380w RAIDMAX Platinum 
OtherOtherOther
iStarUSA Four 3.5in drive in Three 5.25in Hot S... IBM m1015 - IT Flash IBM m1015 - IT Flash 
  hide details  
Reply
 
VM Server
(15 items)
 
 
CPUMotherboardGraphicsRAM
2600k ASRock Z68 Extreme 4 Sapphire 7970 8GB Vengeance 1600 CL8 
Hard DriveOptical DriveOSMonitor
Crucial M4 256GB HP dvd1160i Windows 7 x64 Ultimate Samsung BX2231  
MonitorMonitorMonitorKeyboard
Samsung BX2231  Samsung BX2231  Panasonic 42" Plasma  Ducky Shine Green LED with Red Switches 
PowerCaseMouse
Corsair 750w Fractal Midi  G500 
CPUMotherboardRAMHard Drive
Intel i7 3770s Asrock Z77 Extreme-m 32GB G.SKILL Ripjaws X Series Samsung F3 1TB 
Hard DriveHard DriveHard DriveHard Drive
Western Digital 2TB Green Seagate Barracuda 1Tb  Western Digital 500GB Western Digital 250GB 
CoolingOSPowerCase
Dark Knight II ESXi 5.1 Antec Earth Watts 380w RAIDMAX Platinum 
OtherOtherOther
iStarUSA Four 3.5in drive in Three 5.25in Hot S... IBM m1015 - IT Flash IBM m1015 - IT Flash 
  hide details  
Reply
post #26 of 28
Thread Starter 
Thought I'd post a follow up to this post. I got myself up and running with RAMdisk and mapped network drives to it on all the slaves.

Gigabit LAN is really more than fast enough for my needs, I'm shipping off maybe 200-500MB worth of data for each work unit and receiving 20MB per frame as they complete.

I tend to average work units that take 20-30 minutes and return frames every 5-10, so the traffic isn't so bad on those jobs.

The ones that REALLY killed me were when I fire off composites through my render management software. Those can render a frame every 2-3 seconds and even though it's only 3-4MB per image...the HDD is NOT happy when it's constantly getting a stream of 6-8 images per second.

RAMdisk on the other hand, is the best thing ever more or less. All writes come in instantly.

I can fire off 1 work unit per CPU thread for example, so with a 5 frame work unit, each machine is rendering 40 frames...I'll get back an entire 160 frame Nuke comp is around 3 seconds now.

Anyway maybe that can help anyone in a similar position.

I'm sure an SSD would work alright too, I tried with mine, but it's a 1st gen and barely faster than a current platter drive anyway.
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
post #27 of 28
RAMdisk eh? I hope you have a good UPS in place. Does it allow you to (in the event of a power failure), trigger a script to do a clean shutdown of your software and then your server?
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
Ryzen
(12 items)
 
  
CPUMotherboardGraphicsRAM
Ryzen 7 1700 Gigabyte GA-AB350M Gaming 3 Palit GT-430 Corsair Vengeance LPX CMK16GX4M2B3000C15 
Hard DriveCoolingOSMonitor
Samsung 850 EVO AMD Wraith Spire Linux Mint 18.x Dell UltraSharp U2414H 
KeyboardPowerCaseMouse
Apple Basic Keyboard Thermaltake ToughPower 850W Lian-Li PC-A04B Logitech Trackman Wheel 
  hide details  
Reply
post #28 of 28
Thread Starter 
Quote:
Originally Posted by parityboy;15420036 
RAMdisk eh? I hope you have a good UPS in place. Does it allow you to (in the event of a power failure), trigger a script to do a clean shutdown of your software and then your server?

It CAN save off a disk image, but it takes a while. An SSD is a much more ideal solution in reality, but the RAMdisk so far is truly fantastic as the only bottle neck whatsoever anymore for saving renders to my workstation is my LAN connection. For the data sizes I'm working with, that basically means zero bottleneck anymore...I think the largest files that get sent out are these light calculation maps that are around 400MB, so at the start of a job when ALL nodes pick it up for the first time, my network usage is at 100% for around 6 secs.

I copy the files off the RAMdisk every hour or two while working away. Worst case scenario is that I lose a night of renders I suppose, but I don't sleep much so that's only 3-4 hours anyway.
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Servers
Overclock.net › Forums › Specialty Builds › Servers › Heavy Load Server Build