Overclock.net › Forums › Specialty Builds › Servers › Some Home Server Questions (I'm a noob)
New Posts  All Forums:Forum Nav:

Some Home Server Questions (I'm a noob)

post #1 of 9
Thread Starter 
I do visual effects work from home as a freelancer, but I'm more on the art side of things than the technical and I don't really know a TON about this stuff. I managed to get myself running with a setup like this so far though:

Main PC:
i7 975x, 12GB RAM, GTX480

Render Boxes (several):
i7 2600k, 16GB RAM, barebones $20 vid cards (just to be able to VNC into them).

The way it works is that I'll hit render on my main machine and Maya writes out temp rendering files that get picked up by the render boxes. They then need to read the irradiance map that gets generated (can be several dozen megs) as well as any file textures from my main PC as well.

This can get a bit bogged down on heavy scenes because while these network machines are calling for all this data to be sent to them, my main computer is already beginning to render the scene (all cores @ 100% use). I'm likely bandwidth limited by the speed my main PC's drive can spool out the information as well.

My idea was to simply have some sort of server box to facilitate all of this and take the burden off my PC and speed things up in general. I'd likely RAID 0 2+ drives (it wouldn't be my primary storage, failure isn't a huge concern as much as speed) and see if there was some way I could host the distributed rendering on it somehow instead of my computer.

What sort of hardware would I need for something like this?

- 4 core Xeon e3
- 16GB RAM
- Let's say 4 x 2TB HDD in RAID 0

Would I need a RAID controller or how does that sort of stuff work? I have literally zero experience with servers. Workstations I know somewhat alright, but not the server side of things.

Is a standard LAN connection good enough to send multiple 100mb files to multiple machines at the same time? That would be the largest amount of data I can possibly recall ever using.

Thanks in advance. You guys really know your ****, I appreciate any help.

Thanks in advance!
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
post #2 of 9
First off, apologies because I don't understand the intricacies of distributed rendering (only briefly worked with backburner at university a while back).

I suppose the first question is wouldn't a raid setup on your main PC eliminate the IO bottleneck you speak of?

Also, if it's single large files you're moving around, ignore people if they tell you to get an SSD - a 4x1tb samsung F3 raid0 SS array would push 600mb/s sustained r/w speeds, far surpassing any sata ssd. You wouldn't really need a dedcated controller for this - that's only for large or complex arrays such as raid5, or to get the absolute maximum performance from the array.

As for the network bandwidth question, I don't know whether you're using 100mbps/1000mbps full/half duplex, but 100mbps would be on the slow side for shoving big 100mb files around if you ask me
space heater
(29 items)
 
S-HVH-01
(11 items)
 
PfSense
(5 items)
 
CPUMotherboardGraphicsRAM
Intel 2500k 4.7GHz  Asus P8P67 Pro(B3) 2 x Club3D 280X 3GB 16GB g.skill 1600MHz C9 
Hard DriveHard DriveHard DriveHard Drive
Samsung 840 Pro 256GB 4 x 2TB Barracudas in RAID10 2 x WD Blue 1TB RAID0 2 x 3TB Seagate Barracuda 
Hard DriveHard DriveOptical DriveCooling
8 x 146GB HP 10k SAS - various RAID levels RAID Controller LSI 9260-8i 512MB LiteOn IHAS124 Noctua NH-D15 
OSMonitorMonitorMonitor
8.1 Enterprise Acer GD245HQ (120hz) LG W2420P LG IPS231P 
MonitorMonitorKeyboardPower
HP ZR24w LG 32LH3000 (w/IPS) Razer Blackwidow Antec TruePower New 750W 
CaseMouseAudioAudio
Corsair 750D Modified original G5 Asus Xonar PCIe Sennheiser HD595 
AudioAudioAudioAudio
Tannoy Mercury 5 NAD C165BEE Bowers & Wilkins 683 Cambridge Audio DACMagic+ 
CPUCPUMotherboardRAM
Intel® Xeon® Processor E5-2630 v3 (20M Cache, 2... Intel® Xeon® Processor E5-2630 v3 (20M Cache, 2... Lenovo RD650 128GB DDR4 ECC REG 
Hard DriveHard DriveHard DriveHard Drive
720ix 12Gbps SAS RAID Controller with 1GB FBWC RAID 0 - 2 x Intel S3500 120GB SSD (deduped VHD... Intel 530 120GB SSD RAID 10 - 8 x Seagate 146GB 10k SAS  
OSPowerCase
Windows Hyper-V Server 2012 R2 2 x LiteION 750W 80PLUS Platinum RD650 24x2.5" chassis 
CPURAMHard DriveOS
Intel® Xeon® Processor X5650  8GB ECC 2 x 146GB SCSI RAID1 PfSense 2.2b 
Other
Intel 4 x 1GbE 
  hide details  
Reply
space heater
(29 items)
 
S-HVH-01
(11 items)
 
PfSense
(5 items)
 
CPUMotherboardGraphicsRAM
Intel 2500k 4.7GHz  Asus P8P67 Pro(B3) 2 x Club3D 280X 3GB 16GB g.skill 1600MHz C9 
Hard DriveHard DriveHard DriveHard Drive
Samsung 840 Pro 256GB 4 x 2TB Barracudas in RAID10 2 x WD Blue 1TB RAID0 2 x 3TB Seagate Barracuda 
Hard DriveHard DriveOptical DriveCooling
8 x 146GB HP 10k SAS - various RAID levels RAID Controller LSI 9260-8i 512MB LiteOn IHAS124 Noctua NH-D15 
OSMonitorMonitorMonitor
8.1 Enterprise Acer GD245HQ (120hz) LG W2420P LG IPS231P 
MonitorMonitorKeyboardPower
HP ZR24w LG 32LH3000 (w/IPS) Razer Blackwidow Antec TruePower New 750W 
CaseMouseAudioAudio
Corsair 750D Modified original G5 Asus Xonar PCIe Sennheiser HD595 
AudioAudioAudioAudio
Tannoy Mercury 5 NAD C165BEE Bowers & Wilkins 683 Cambridge Audio DACMagic+ 
CPUCPUMotherboardRAM
Intel® Xeon® Processor E5-2630 v3 (20M Cache, 2... Intel® Xeon® Processor E5-2630 v3 (20M Cache, 2... Lenovo RD650 128GB DDR4 ECC REG 
Hard DriveHard DriveHard DriveHard Drive
720ix 12Gbps SAS RAID Controller with 1GB FBWC RAID 0 - 2 x Intel S3500 120GB SSD (deduped VHD... Intel 530 120GB SSD RAID 10 - 8 x Seagate 146GB 10k SAS  
OSPowerCase
Windows Hyper-V Server 2012 R2 2 x LiteION 750W 80PLUS Platinum RD650 24x2.5" chassis 
CPURAMHard DriveOS
Intel® Xeon® Processor X5650  8GB ECC 2 x 146GB SCSI RAID1 PfSense 2.2b 
Other
Intel 4 x 1GbE 
  hide details  
Reply
post #3 of 9
Thread Starter 
Distributed rendering is pretty simple as far as I can really tell. Maya writes out and sends a VRay scene that the blades are able to render in a standalone copy of VRay. Once the data for global illumination is calculated, that gets written out and sent over to the blades as well. All textures used in the scene need to be loaded up into RAM by each blade (hence the 16GB RAM on all machines).

All of this stuff comes from my main PC and happens while all 8 threads are under 100% load unfortunately which I'm sure slows down the process a lot.

And yeah this all happens from a single, 7200rpm 1TB HDD which can't be helping either.

So what you're thinking is that I'd be better off not worrying about the server idea at all and simply setting up a RAID 0 array on my main PC to speed up my data transfers? Each render box has a 1TB HDD inside so I could make backups and not worry too much about corruption if a RAID drive dies.

How do I learn about what my network bandwidth actually is and how could I improve it once I do know?

What sort of stuff do I need to get myself started with let's say a 4 HDD RAID 0 array in my main PC? Or can you point me towards some resources that will teach me about this stuff? I hate asking for spoon-fed info from people.
Edited by kweechy - 5/25/11 at 3:58pm
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
post #4 of 9
Quote:
Originally Posted by kweechy View Post
All of this stuff comes from my main PC and happens while all 8 threads are under 100% load unfortunately which I'm sure slows down the process a lot.
Is it possible for the rendering on the main pc to take place after sending the files to the render boxes?

Quote:
Originally Posted by kweechy View Post
How do I learn about what my network bandwidth actually is and how could I improve it once I do know?
Seeing as you have an i7 processor, the motherboard will have a gigaethernet (1000mbps) NIC on it.

Quote:
Originally Posted by kweechy View Post
What sort of stuff do I need to get myself started with let's say a 4 HDD RAID 0 array in my main PC? Or can you point me towards some resources that will teach me about this stuff? I hate asking for spoon-fed info from people.
Again, seeing as how you have an i7 system, it should be safe to say it comes with intergrated (fake) raid built in. You would need to check your motherboard manual (see manufacturer support site) to make sure that it does support RAID0 and for instructions on how to set it up within the bios and controller software (pretty easy to do).
VH001
(17 items)
 
VH002
(15 items)
 
VH003
(16 items)
 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 SuperMicro X9DR7-LN4F Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS 8 x Toshiba 5TB 
Hard DriveCoolingCoolingOS
Intel 750 400GB NVME SSD 2 x Dynatron R22 6 x SuperMicro 5K RPM Fans VMWare vSphere 6 
MonitorPowerCaseOther
IPMI SuperMicro 650W PSU SuperMicro CSE-833T-650B Black 3U Rackmount Ser... Intel X520-T2 Dual Port 10GbE Server Adapter 
Other
HP P420 1GB Smart Array Controller 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 Intel S2600CP2J Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS Intel 750 400GB NVME SSD 
CoolingOSMonitorPower
2 x Intel AUPSRCBTP VMWare vSphere 6 IPMI Intel 750W PSU 
CaseOtherOther
Intel P4216XXMHGR 4U Server Chassis Intel X520-T2 Dual Port 10GbE Server Adapter HP P420 1GB Smart Array Controller 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 SuperMicro X9DR3-F Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS Intel 750 400GB NVME SSD 
CoolingCoolingOSMonitor
2 x Dynatron R22 3 x SuperMicro 5K RPM Fans VMWare vSphere 6 IPMI 
PowerCaseOtherOther
SuperMicro 563W PSU SuperMicro CSE-825TQ-563LPB 2U Rackmount Server... Intel X520-T2 Dual Port 10GbE Server Adapter HP P420 1GB Smart Array Controller 
  hide details  
Reply
VH001
(17 items)
 
VH002
(15 items)
 
VH003
(16 items)
 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 SuperMicro X9DR7-LN4F Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS 8 x Toshiba 5TB 
Hard DriveCoolingCoolingOS
Intel 750 400GB NVME SSD 2 x Dynatron R22 6 x SuperMicro 5K RPM Fans VMWare vSphere 6 
MonitorPowerCaseOther
IPMI SuperMicro 650W PSU SuperMicro CSE-833T-650B Black 3U Rackmount Ser... Intel X520-T2 Dual Port 10GbE Server Adapter 
Other
HP P420 1GB Smart Array Controller 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 Intel S2600CP2J Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS Intel 750 400GB NVME SSD 
CoolingOSMonitorPower
2 x Intel AUPSRCBTP VMWare vSphere 6 IPMI Intel 750W PSU 
CaseOtherOther
Intel P4216XXMHGR 4U Server Chassis Intel X520-T2 Dual Port 10GbE Server Adapter HP P420 1GB Smart Array Controller 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 Intel Xeon E5-2670 SuperMicro X9DR3-F Onboard Graphics 
RAMHard DriveHard DriveHard Drive
128GB (16 x 8GB) 1333MHz ECC Registered DDR3 4 x Micron M500 480GB SSD 4 x Hitachi 450GB 10K SAS Intel 750 400GB NVME SSD 
CoolingCoolingOSMonitor
2 x Dynatron R22 3 x SuperMicro 5K RPM Fans VMWare vSphere 6 IPMI 
PowerCaseOtherOther
SuperMicro 563W PSU SuperMicro CSE-825TQ-563LPB 2U Rackmount Server... Intel X520-T2 Dual Port 10GbE Server Adapter HP P420 1GB Smart Array Controller 
  hide details  
Reply
post #5 of 9
As above, there isn't really much advantage in building a server for this.

How big are the files that the multiple machines need to access, and are they accessed sequentially or is it random requests?

If they're small files with random I/O, then a small SSD would be best.

If they're large and sequential, a couple of fast drives in RAID10 (or RAID0 if a failure can be tolerated - this will really depend on how long a single operation takes - you'd probably be ok in RAID0 if a render operation takes a few hours, but you'd want RAID10 if the rendering takes days to complete). Something like 4 or 6 320GB F4s would be a decent idea (although with the cost of those drives an SSD might still be a better buy, depending on file size).

If the shared files are very small then a RAM disk might even be an option - setting aside a GB or so of your main system memory probably won't affect your rendering performance, but may speed up your nodes considerably.

You should double check all your network settings and make sure everything is running gigabit and full duplex (note that you shouldn't set full duplex unless you have a managed switch - you should use 'auto' or you will actually force half duplex mode). Check jumbo frames also, and if you enable jumbo make sure every device on the network supports the frame size you choose or your performance will be terrible. Depending on how many nodes you have you should also get a decent switch - if your network load is high then a cheapy switch will bog down.

Finally - where are the output files stored? If you can move them to another drive (even a single drive) you will eliminate much more of the bottleneck.

Hope this helps - or at least prompts you to think of a few more options...
post #6 of 9
Thread Starter 
Very helpful yes, thanks a lot for that information.

So since I'm likely bottlenecked by my home network...what if I were to grab a networking card for my main computer that had two LAN ports, pipe in one from my router for internet etc and then pipe out one into a gigabit switch that then feeds all the rendering machines?

What is the best way to see what kind of network speeds you're achieving on your home network?
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
post #7 of 9
What network infrastructure do you have currently? If everything is gigabit (even if it's crap kit) then your bottleneck is still most likely to be in you disk subsystem.

And a dual port network card likely isn't going to help either, but will be pricey. Your general LAN use for internet etc will barely use your connection (unless you have VERY fast internet and are torrenting like a mentalist), so shifting that load off the NIC won't help. It could be worthwhile putting dedicated NICs in your main rig to feed each rendering node separately - but only if your disks can keep up. So that's the problem you need to address first.
post #8 of 9
Thread Starter 
I actually neglected to mention that due to the pitfalls of WiFi networks in downtown areas, I have been forced into a powerline ethernet situation. It's decent, but obviously not as good as a straight up LAN connection, so that was my thinking in terms of splitting off the internet router's LAN from my connection to the render slaves.

So if I understand correctly, my ideal setup is simply a RAID array in my workstation and a proper LAN connection to my render slaves (not powerline).

I guess my other question would be in terms of setting up the RAID in the first place...is it fine to simply do it on my motherboard's native SATA ports or should I be buying a PCI card RAID controller to really get it running properly? Or are those more for arrays with like 8+ drives?

What kind of tests can I run to see what speeds I'm getting at various stages of the transfers?
- Speed from my main PC's drive to the LAN port
- Speed from main PC's LAN port to a render slave's LAN port
- Speed from render slave's LAN port to its own drive
Edited by kweechy - 5/30/11 at 8:13am
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
2P Workstation
(13 items)
 
  
CPUCPUMotherboardGraphics
Xeon E5 2699 v3 Xeon E5 2699 v3 ASUS Z10PE-D16 EVGA TITAN X 
RAMHard DriveHard DriveCooling
128GB Crucial DDR4 @ 2133MHz Intel 750 Series 1.2TB 4 x 4TB Constellation RAID5 2 x XSPC Raystorms 
CoolingCoolingMonitorMonitor
RS480 MCP655 Dell U3011 Dell U3011 
Case
Silverstone TJ07 
  hide details  
Reply
post #9 of 9
If you are running RAID 0, the on board controller should be just fine. However, if you are going to keep any important data on the RAID, you will need to back it up. Even if you run RAID 10, you must have a backup solution(And don't forget to test it from time to time).

For IO speed (Drive speed), you can just use HDTune
And for LAN speed, you can yse LAN Speed Test http://download.cnet.com/LAN-Speed-T...-10908738.html

If all you are pushing is 100MB files, then the render boxes IO speed shouldn't be an issue. Most modern 7200rpm HDD can do at least 100MB/s

Having a Gigabit LAN will help tremendously, Gigabit LANs can push ~125 MB/s
 
VM Server
(15 items)
 
 
CPUMotherboardGraphicsRAM
2600k ASRock Z68 Extreme 4 Sapphire 7970 8GB Vengeance 1600 CL8 
Hard DriveOptical DriveOSMonitor
Crucial M4 256GB HP dvd1160i Windows 7 x64 Ultimate Samsung BX2231  
MonitorMonitorMonitorKeyboard
Samsung BX2231  Samsung BX2231  Panasonic 42" Plasma  Ducky Shine Green LED with Red Switches 
PowerCaseMouse
Corsair 750w Fractal Midi  G500 
CPUMotherboardRAMHard Drive
Intel i7 3770s Asrock Z77 Extreme-m 32GB G.SKILL Ripjaws X Series Samsung F3 1TB 
Hard DriveHard DriveHard DriveHard Drive
Western Digital 2TB Green Seagate Barracuda 1Tb  Western Digital 500GB Western Digital 250GB 
CoolingOSPowerCase
Dark Knight II ESXi 5.1 Antec Earth Watts 380w RAIDMAX Platinum 
OtherOtherOther
iStarUSA Four 3.5in drive in Three 5.25in Hot S... IBM m1015 - IT Flash IBM m1015 - IT Flash 
  hide details  
Reply
 
VM Server
(15 items)
 
 
CPUMotherboardGraphicsRAM
2600k ASRock Z68 Extreme 4 Sapphire 7970 8GB Vengeance 1600 CL8 
Hard DriveOptical DriveOSMonitor
Crucial M4 256GB HP dvd1160i Windows 7 x64 Ultimate Samsung BX2231  
MonitorMonitorMonitorKeyboard
Samsung BX2231  Samsung BX2231  Panasonic 42" Plasma  Ducky Shine Green LED with Red Switches 
PowerCaseMouse
Corsair 750w Fractal Midi  G500 
CPUMotherboardRAMHard Drive
Intel i7 3770s Asrock Z77 Extreme-m 32GB G.SKILL Ripjaws X Series Samsung F3 1TB 
Hard DriveHard DriveHard DriveHard Drive
Western Digital 2TB Green Seagate Barracuda 1Tb  Western Digital 500GB Western Digital 250GB 
CoolingOSPowerCase
Dark Knight II ESXi 5.1 Antec Earth Watts 380w RAIDMAX Platinum 
OtherOtherOther
iStarUSA Four 3.5in drive in Three 5.25in Hot S... IBM m1015 - IT Flash IBM m1015 - IT Flash 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Servers
Overclock.net › Forums › Specialty Builds › Servers › Some Home Server Questions (I'm a noob)