Overclock.net › Forums › Software, Programming and Coding › Networking & Security › Rendering Farm(Cluster)
New Posts  All Forums:Forum Nav:

Rendering Farm(Cluster)

post #1 of 12
Thread Starter 
I am currently running HandBrake/RipBot264 to process Blu-Ray movies into compressed .MKV files on my i7 920 D0 @ 4.1Ghz, its taking between 2-3 hours per rip on a single computer, im wondering if anybody could give PRECISE details on how to spread a video encode to multiple computers over Gigabit Ethernet (or faster if needed) in some kind of network media/cluster setup... step-by-step gunner.gif
post #2 of 12
Hmm.. Cluster rendering?

Generally what happens is that one would work on a project on there PC then send the work to a super computer. And when i say super computer i mean 100+ processors and a terabyte of ram.
Full Of Stuff
(15 items)
 
The nEEEtBook
(13 items)
 
CPUMotherboardGraphicsRAM
AMD Phenom II 1100t X6 MSI 990XA-GD55 GTX 465 SLI Mushkin Ridge Back 
Hard DriveHard DriveOptical DriveCooling
OCZ Agility 3 Seagate 7200.12 Sony 18X DVD-ROM 48X CD-ROM SATA DVD-ROM Drive Scythe MUGEN-2 SCMG-2000 
OSMonitorKeyboardPower
Windows 7 Ultimate Planar SA2311W 3D Monitor Microsoft Keyboard PC Performance & Cooling Silencer 750W 
CaseMouseAudio
Cooler Master HAF 912 [Window Mod] OCZ BEHEMOTH Laser Mouse Realtek 5.1 Onboard 
CPUMotherboardGraphicsRAM
Intel Atom N270 AsusTek 1005HA Mobile Intel 945 Express 1GB DDR2 
Hard DriveOSMonitorKeyboard
OCZ Vertex 2 (Upgrade) Windows 7 Starter 10.1" Wide screen LED Lit Display Full Keyboard 
PowerCaseMouse
40W 19V Asus Adapter 1005HA Case Trackpad 
  hide details  
Reply
Full Of Stuff
(15 items)
 
The nEEEtBook
(13 items)
 
CPUMotherboardGraphicsRAM
AMD Phenom II 1100t X6 MSI 990XA-GD55 GTX 465 SLI Mushkin Ridge Back 
Hard DriveHard DriveOptical DriveCooling
OCZ Agility 3 Seagate 7200.12 Sony 18X DVD-ROM 48X CD-ROM SATA DVD-ROM Drive Scythe MUGEN-2 SCMG-2000 
OSMonitorKeyboardPower
Windows 7 Ultimate Planar SA2311W 3D Monitor Microsoft Keyboard PC Performance & Cooling Silencer 750W 
CaseMouseAudio
Cooler Master HAF 912 [Window Mod] OCZ BEHEMOTH Laser Mouse Realtek 5.1 Onboard 
CPUMotherboardGraphicsRAM
Intel Atom N270 AsusTek 1005HA Mobile Intel 945 Express 1GB DDR2 
Hard DriveOSMonitorKeyboard
OCZ Vertex 2 (Upgrade) Windows 7 Starter 10.1" Wide screen LED Lit Display Full Keyboard 
PowerCaseMouse
40W 19V Asus Adapter 1005HA Case Trackpad 
  hide details  
Reply
post #3 of 12
I've thought about building an MPI cluster as well but haven't really found a use to spend the cash on extra hardware since I only use handbrake infrequently.

Looking up some sort of MPI/Beowulf cluster would likely point you in the right direction.
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
Waiting on X399
(13 items)
 
  
CPUMotherboardGraphicsRAM
AMD Phenom II B57 @ X4 3.9 Gigabyte 790FXTA-UD5 Sapphire Radeon 290 8 GB G.Skill 2133 
Hard DriveCoolingOSKeyboard
250 GB 840 EVO Noctua NH-D14 Windows 10 Logitech K350 
PowerCaseMouseMouse Pad
Seasonic x750 Corsair 600T Logitech G100s Razer Goliathus Speed 
Audio
Plantronics Gamecom 788 
  hide details  
Reply
post #4 of 12
Quote:
Originally Posted by OCTDBADBRO;11883550 
I am currently running HandBrake/RipBot264 to process Blu-Ray movies into compressed .MKV files on my i7 920 D0 @ 4.1Ghz, its taking between 2-3 hours per rip on a single computer, im wondering if anybody could give PRECISE details on how to spread a video encode to multiple computers over Gigabit Ethernet (or faster if needed) in some kind of network media/cluster setup... step-by-step gunner.gif


How many are you doing a day? Can you do multiple ones simutaneously?

You will be better off increases overall throughput instead of trying to reduce the runtime of each individual compression. AMD hexcore builds would accomplish this without adding so much complexity.
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
post #5 of 12
Thread Starter 
well i have 1 blu-ray drive, 2 servers and 3 computers right now but im getting 8 extra computers all 6-8+ cores for WCG/F@H

I'm doing about 20 rips per week right now thumb.gif
post #6 of 12
Quote:
Originally Posted by OCTDBADBRO;11889165 
well i have 1 blu-ray drive, 2 servers and 3 computers right now but im getting 8 extra computers all 6-8+ cores for WCG/F@H

I'm doing about 20 rips per week right now thumb.gif

Does HandBrake/RipBot264 support clusters?


Just get 1-2 more Blu-Ray drives and rip them to a file server. RDP into your other computer to run HandBrake/RipBot264.
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
post #7 of 12
Thread Starter 
Okay so lets say i want to open 1 billion copies of solitaire... except for been a waste of time, is there anybody that can give exact step by step instructions on how to combine say 8 computers together to split the load between them.
I have tried lots of Cluster based Linux distro's and only had success with 1 and then i dont know MPI which that version of cluster required instead of something like OpenMosix which automatically load balances baaasmiley.gif
post #8 of 12
Quote:
Originally Posted by OCTDBADBRO;11908375 
Okay so lets say i want to open 1 billion copies of solitaire... except for been a waste of time, is there anybody that can give exact step by step instructions on how to combine say 8 computers together to split the load between them.
I have tried lots of Cluster based Linux distro's and only had success with 1 and then i dont know MPI which that version of cluster required instead of something like OpenMosix which automatically load balances baaasmiley.gif

...but 1 billion instances of solitaire is not suited for clusters. The usage patterns does not make sense.

You don't build clusters and then try to get programs to run on them... because you won't unless you have the source code or are VERY good at hardware and software. Even so, performance will probably suck.

You build programs that are suited for clusters.
Edited by DuckieHo - 1/6/11 at 9:07am
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
post #9 of 12
perhaps windows HPC would work? does it have to be handbrake? M$ has a new transcoding alpha for their streaming services and its made to work with HPC

This dude says he has a handbreak cluster

http://falconfour.wordpress.com/2009/05/19/yes-the-handbrake-svn-is-stable-enough/


I am going to build and test a HPC cluster in early july
post #10 of 12
I've looked into building a beowulf cluster and from my information gathering, the only (cheap) way to do it is linux-based, meaning a lot of windows-based programs aren't going to apply.

Also, clusters are for parallel-proccess optimization. DVD encoding/decoding is sorta-kinda along those lines, but you don't need a cluster. Just building another rig and having two dedicated instead of one would be my advice. Clustering is not a simple process and can be relatively high-maintenance, depending on what you're doing.

EDIT: Actually, re-reading the OP, if you're just doing a rip, you're probably limited by your optical drive, not your system's processing power. Just a thought.
Edited by Aeschylus - 5/2/11 at 11:21pm
Rosemari
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 @ 4.0 GHz|1.26250v Asus P6T deluxe V2 XFX AMD Radeon 6950 2GB RF 12 GB G.Skill DDR3 1600 
Hard DriveOSMonitorPower
WD 1 TB Caviar Black 7200 RPM Win 7 64-bit Asus 24" 1080p Corsair TX 750W 
Case
Thermaltake Xaser VI 
  hide details  
Reply
Rosemari
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 @ 4.0 GHz|1.26250v Asus P6T deluxe V2 XFX AMD Radeon 6950 2GB RF 12 GB G.Skill DDR3 1600 
Hard DriveOSMonitorPower
WD 1 TB Caviar Black 7200 RPM Win 7 64-bit Asus 24" 1080p Corsair TX 750W 
Case
Thermaltake Xaser VI 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Networking & Security
Overclock.net › Forums › Software, Programming and Coding › Networking & Security › Rendering Farm(Cluster)