Overclock.net › Forums › Cooling › Water Cooling › [Gallery / Build Log] ALDEBARAN - quad-socket server on supermicro H8QG6-i
New Posts  All Forums:Forum Nav:

[Gallery / Build Log] ALDEBARAN - quad-socket server on supermicro H8QG6-i

post #1 of 15
Thread Starter 
Hello, here is a story about building 4-socket water-cooled server that I'll use for my 3D renders.
I've made a little video qith some additional photos of the machine https://vimeo.com/83923217



So the goal was to take all power of 4x AMD Opteron 6380 without having some overheat or cooling noise problems, taking into consideration that this machine has to work whole days and nights at 100% CPU load.

I was looking for a case capable to bear supermicro H8QG6-i motherboard, because Supermicro's chassis that they sell for those boards are ugly and it was difficult to plan to use it in watercolled system. So I've stopped my choice on a Coolermaster's Cosmos II case. It is the biggest one I found with a convinient price of about 300 euros. In addition it's very handy for transportation.

All water-cooling components qre from Aquacomputer and EK.
2x 360mm radiators + 1x 240 radiator from Aquacomputer
880 ml reservoir and Fan controller Aquaero XT from Aquacomputer
4x waterblocks cuplex kryos Delrin für Socket G34 - from aquacomputer
All fitting parts and pump EK-DDC 3.25 X-TOP CSQ - Plexi from EK
Fun grill from MPCTECH.

However I should rebuild the case before I could put in the motherboard.

Firstly i should remove all this stuff from the case



Than i made some mounts in the bottom part to be able to fix a bottom 3 fan radiator.



And of course, as the case can't bear my MBoard directly I installed a plexi plate with all fixes on it to be able to fix the motherboard.



So one radiator at the bottom one on the top and one on the front side of the box.




I need to remake all ugly cables manually.



Finally I planned to replace factory's side panel of the box with one made of plexi with 3 holes for bottom fans.




Screwing up all components together and it works!


After 24 hours full CPUs load render test I had 63 C max temperature on the CPU1, and 61-62 C on CPUs 2-4. So i'm putty glad with this result.
post #2 of 15
wow that is totally awesome, looks phenomenal. what sort of stuff are you rendering?
No 1 Rig
(12 items)
 
  
CPUMotherboardGraphicsRAM
i5 2380p Asus P8z68 Delixe/Gen 3 EVGA GTX 470 2 x 4Gb Corsair Vengeance Blue 
Hard DriveHard DriveCoolingOS
WD black OCZ Solid 3 Laing DDC in XSPC Dual Bay Res Windows 8.1 Pro 64Bit 
MonitorPowerCaseMouse
24" AOC Corsair Builders 650W Modular Coolermaster 690II Cyborg R.A.T. 5 
  hide details  
Reply
No 1 Rig
(12 items)
 
  
CPUMotherboardGraphicsRAM
i5 2380p Asus P8z68 Delixe/Gen 3 EVGA GTX 470 2 x 4Gb Corsair Vengeance Blue 
Hard DriveHard DriveCoolingOS
WD black OCZ Solid 3 Laing DDC in XSPC Dual Bay Res Windows 8.1 Pro 64Bit 
MonitorPowerCaseMouse
24" AOC Corsair Builders 650W Modular Coolermaster 690II Cyborg R.A.T. 5 
  hide details  
Reply
post #3 of 15
Thread Starter 
Generally video FX.
Images from Autodesk Maya, with mental ray and maxwell render. It can be also After effects jobs.
You can find some videos on my vimeo channel https://vimeo.com/user5762334/videos
Edited by hanumant - 1/13/14 at 5:41am
post #4 of 15
Thread Starter 
runing on win 2008 server SP2
post #5 of 15
Hi Hanumant,

we've also tried to build one of these. smile.gif But for some reason this thing is incredibly slow.
We ran a few tests and the CPU is doing quite OK I think. But the Large memory write speed is the worst I've ever seen.

Somehow the temperature of the CPU's doesn't rise above 43C°. Even if this thing renders all the night at 100% CPU usage.

Can you please take look at our test results?
www.viscircle.com/server/

We tried to change interleaving settings for banks, nodes and channels in the bios. But only the node interleaving ( so in effect NUMA) had an effect on the memory latency (from 177 to 68 which is still a bad value)

Do you have any idea what else we could tweak?

Thank you very much.

VisCircle
post #6 of 15
Thread Starter 
Hello, sorry for my late response. I Was very busy last weeks.
Your information surprised me. Really I can confirm that there are a memory latency 123-160 on my tests.
I tried to find any solution in Supermicro Thech support, but it doesn't respond at all.
I've contacted my distributor in order to have a competent and adequate tech. support from supermicro. I wait now.

I'll inform you as I have any news.

You say that you've made down the latency to 68, which setting you've done in BIOS ?

Thank you!
post #7 of 15
We ran a few tests and the CPU is doing quite OK I think. But the Large memory write speed is the worst I've ever seen.zgjULx
post #8 of 15
Thread Starter 
Did you used all 16 dimms during your tests?
post #9 of 15
We set the numa support to on and the node interleaving to disabled i think.

I hope that passmark uses al of the 16 dimms. :/ How can i test it?

I ran some other benchmarks and evrywhere the same large ram problem.
post #10 of 15
Thread Starter 
Numa disable does not help at all, in my case frown.gif
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Water Cooling
Overclock.net › Forums › Cooling › Water Cooling › [Gallery / Build Log] ALDEBARAN - quad-socket server on supermicro H8QG6-i