Overclock.net › Forums › Case Mods & Cases › Builds & Case Mods › Build Logs › Super Pallet Server - Quieting supermicro servers MK II - Watercooling
New Posts  All Forums:Forum Nav:

Super Pallet Server - Quieting supermicro servers MK II - Watercooling

post #1 of 34
Thread Starter 
Hi,

A bit of background:

2 years ago I was able to purchase 4 x E7-2870 xeon chips from ebay for a fantastic price (~£400 total), these retailed at around £2k each at the time (or they did then). The problems I had was sourcing motherboards/servers to use them. It seemed only supermicro offered a reasonably priced solution. These particular chips (E7-2***) are allowed in pairs per motherboard which was ideal because I wanted a live database server and an analysis server at home. I managed to buy 2 servers: the "SuperServer 8046B-6RF" hosting the ""X8QB6-F" motherboards.

On first run I was in for a shock! The noise levels were astronomical, from memory the servers had 6 x 9 inch fans that were capable of 8000RPM, it sounded like a jet engine, there was no way I could work with these in the same room. My initial idea was to look and find cpu heatsink/fans and a bespoke server case to host. The problem I found was the chips use the LGA-1567 socket, which is very obscure. I couldn't find any cpu heatsink-fans that would fit this enclosure. These chips are obviously designed for hosting centres where noise/heat is not an issue. I wanted these servers at home because of data sensitivity and didn't want the additional costs and problems with remote hosting.

The other problem is the dimensions of the SuperMicro motherboards, these are none standard and will not fit most cases (400mm+ by 400mm+). I did some research at the time, and copied some ideas from the bitcoin mining community in building my own case hosting the 2 motherboards and a large powerful fan:





(Btw, for live use, I used a much larger, more powerful fan wink.gif )
Okay, you can stop laughing now smile.gif ... This was still quite noisy but nowhere near as bad, I hosted this in my garage (quiet enough not to upset my neighbours). This worked fine for almost 2 years until one of my server's 2 ssd's failed at the same time, maybe a power surge or perhaps overheating? Anyway this made me look at other solutions (again)! Finally I came across the perfect solution, I found a link to a guy in Russia who had built his own "fitting" for the LGA1567 for a Koolance gpu-210 block: http://www.ebay.co.uk/itm/LGA1567-CPU-Mounting-Plate-for-Koolance-GPU-210-Cold-Plate-SNK-P0045P-Supermicro-/161254223847, here is a picture:


Though the gpu-210 (http://koolance.com/gpu-210-video-card-vga-chipset-water-block) is for a gpu, this is only because of its generic usage (multiple fittings etc). The surface area on this block's heat plate is exactly the same as the supermicro provided cpu heatsinks, so it is a perfect fit:

(on the left is the water block, on the right is the stock supermicro heatsink)

So I am now partly through my water build and my newbie status will probably show smile.gif. I have read everything I can on watercooling so hoping to get it right first time. Expensive otherwise :-S

I wanted to log here in case there are other supermicro/server sufferers out there, and for my own future perusal for documenting correct fittings etc. Also thought it would give some of you a laugh ;-)
post #2 of 34
Nice idea

Sent from my SM-N910P using Tapatalk
 
New Main Gaming
(16 items)
 
250 Delta
(16 items)
 
CPUMotherboardGraphicsRAM
6850k MSI X99A Tomahawk NVIDIA GTX 1080 Ti G Skill Ripjaws V 
Hard DriveHard DriveHard DriveHard Drive
Intel 530 SSD Samsung 850 Pro Samsung 850 Pro Crucial  
Hard DriveHard DriveCoolingCooling
WD Blue WD Blue EK Supremacy EVO X99 Nickel 480 Rad 
CoolingCoolingOSMonitor
480 Rad EK GTX Titan XP Nickel Plexi Win 10 Pro LG 29UM58-P 
MonitorMonitorMonitorKeyboard
LG 29UM58-P LG 29UM58-P HTC VIVE Corsair K70 
PowerCaseMouse
EVGA 850 BQ Corsair 900D Logitech G700s 
CPUMotherboardGraphicsRAM
i7 6700k MSI Z170 Tomahawk AC GTX 980 G. SKill Ripjaws Series 4 
Hard DriveHard DriveHard DriveHard Drive
Intel 320 120Gb SSD WD Green 750Gb WD Green 1Tb Samsung SSD 850 Evo 500GB SSD (Steam) 
CoolingOSMonitorMonitor
Custom Loop Windows 10 Vizio E24i-B1 Vizio E24i-B1 
KeyboardPowerCaseMouse
Saitek Eclipse II XFX 650 Black Edition Corsair 900D Logitech G700s 
CPUMotherboardGraphicsRAM
i5 4690k Asus Z97i-Plus Onboard HyperX Fury Black 
Hard DriveHard DriveHard DriveHard Drive
Transcend m.2 Western Digital Blue 320Gb Western Digital Blue 500Gb Seagate 500gb 
CoolingCoolingCoolingCooling
XSPC CPU Waterblock Swiftech MCR220 AlphaCool DC-LT AlphaCool DC-LT Top 
CoolingOSPowerCase
AlphaCool DC-LT Res Windows 10 Pro x64 Corsair CX600M Corsair 250D 
  hide details  
Reply
 
New Main Gaming
(16 items)
 
250 Delta
(16 items)
 
CPUMotherboardGraphicsRAM
6850k MSI X99A Tomahawk NVIDIA GTX 1080 Ti G Skill Ripjaws V 
Hard DriveHard DriveHard DriveHard Drive
Intel 530 SSD Samsung 850 Pro Samsung 850 Pro Crucial  
Hard DriveHard DriveCoolingCooling
WD Blue WD Blue EK Supremacy EVO X99 Nickel 480 Rad 
CoolingCoolingOSMonitor
480 Rad EK GTX Titan XP Nickel Plexi Win 10 Pro LG 29UM58-P 
MonitorMonitorMonitorKeyboard
LG 29UM58-P LG 29UM58-P HTC VIVE Corsair K70 
PowerCaseMouse
EVGA 850 BQ Corsair 900D Logitech G700s 
CPUMotherboardGraphicsRAM
i7 6700k MSI Z170 Tomahawk AC GTX 980 G. SKill Ripjaws Series 4 
Hard DriveHard DriveHard DriveHard Drive
Intel 320 120Gb SSD WD Green 750Gb WD Green 1Tb Samsung SSD 850 Evo 500GB SSD (Steam) 
CoolingOSMonitorMonitor
Custom Loop Windows 10 Vizio E24i-B1 Vizio E24i-B1 
KeyboardPowerCaseMouse
Saitek Eclipse II XFX 650 Black Edition Corsair 900D Logitech G700s 
CPUMotherboardGraphicsRAM
i5 4690k Asus Z97i-Plus Onboard HyperX Fury Black 
Hard DriveHard DriveHard DriveHard Drive
Transcend m.2 Western Digital Blue 320Gb Western Digital Blue 500Gb Seagate 500gb 
CoolingCoolingCoolingCooling
XSPC CPU Waterblock Swiftech MCR220 AlphaCool DC-LT AlphaCool DC-LT Top 
CoolingOSPowerCase
AlphaCool DC-LT Res Windows 10 Pro x64 Corsair CX600M Corsair 250D 
  hide details  
Reply
post #3 of 34
Thread Starter 
Thanks Millerboy3 smile.gif

Design:

I hope to eventually cool both motherboards in the same loop but because I need one of my servers always running, I have made this a 2 stage setup:

Initial setup:
Reservoir/Pump > CPU1 > CPU2 > v large Radiator > (same) Reservoir/pump

Later, extended to:
Reservoir/Pump > CPU1 > CPU2 > *** > CPU3 > CPU4 > *** > v large Radiator > (same) Reservoir/pump
*** = (additional pump maybe required in one of these locations due to extended size? If so, I'll test to find best location.)

I will build my own acrylic casing for each motherboard with fans to keep memory/components cool.

This setup is primarily for reducing heat and noise and NOT aesthetics, so expect a pretty ugly system smile.gif In fact, I have decided to build this all on top of a "customised" pallet (as I can place the PSU underneath and feed wires through top), hence the title "Super Pallet Server" smile.gif
post #4 of 34
Thread Starter 
Initial Cleaning

I decided to go for a whopping radiator, the MO-RA3 420. Because of its sheer size and weight, the usual initial radiator cleaning tips can't be used (I.e., using very hot water and lots of shaking wheee.gif) So I decided to try out an idea I'd seen on youtube here which is basically using a fish tank pump and drawing out all the crap into a filter/bucket. In fact, in hindsight I don't see why you couldn't just do this using your normal watercooling pump... I suppose, it just makes it easier when maintaining your loop if you don't have to dismantle everything (including pump)?

So I bought a 1500 L/H pump:
Racksoy Professional Aquarium 1.8M 1500L/H 20W Internal Submersible Water Pump For Freshwater Saltwater from amazon.
and a
10 inch Water Filter Housing 1/4 inch Ports
and
10 inch PP Sediment Water Filter Cartridge (1 Micron) to fit inside it (from a company in Ireland called TheWaterFilterMen.





I decided to use a 1500 L/H pump because I didn't want to go far from the pressure I would expect from my watercooling setup (according to specs the D5 Vario pump runs at 1200 L/H). When observing this pressure it was like a tap half way on. In hindsight maybe I should have gone for a much higher pressure pump here?
post #5 of 34
Thread Starter 
(It's an expensive radiator, to get this wrong frown.gif...) Suppose better to have to clean out loop later than replace a bust radiator!?
post #6 of 34
Thread Starter 
Got it set up:


and ran over night:


It seems there was a small amount of gunk, not sure if this was worth going to the lengths I went to, to clean this out? But it is done now anyway!
post #7 of 34
Thread Starter 
Adding Noctua Fans and side panel to the MO-RA3 420 Radiator

I bought 18 x 140mm Noctua NF-A14PWM fans (pretty ugly but always get good write-ups wrt noise/performance ratios). 9 will be push fans and 9 will be rear exhaust pull fans. I have also bought the custom side fittings and feet...

Pain in the backside, took hours to screw them all in and so little space for cables between fans:



You may have to zoom in on this picture (below), but it shows the MO-RA3 side panel that goes over the fans, I have highlighted where they expect you to pass through 9 fan cables ohno-smiley.gif :


Annoyingly, because the Noctua fans have pretty thick cable sleeves it is impossible to fit them through this hole (I am not even sure they would still fit with sleeves removed?). Cutting the holes bigger with a hacksaw seemed the easiest/quickest way:
post #8 of 34
Thread Starter 
In order to control all these fans I bought a custom built device from a guy in the UK. This device takes in one PWM input and outputs the same signal to upto 11 PWM devices(fans). Fortunately, the Noctua fans come with several other cables, one of which is a y-splitter cable. So I was able to reduce the number of fan cables from 18 to 9 using these y cables.

PWM Splitter

This is the pwm splitter device in action (where I originally found it):
https://www.youtube.com/watch?v=Kd5iifufgg8

I bought from ebay, here is the item (PWM fan splitter). The guy selling these is the designer and builder of these and he will build them on request, just message him. I bought an additional 2 of these to control my server fans too. A brilliant piece of equipment smile.gif
post #9 of 34
Subscribed and best of luck !
Shadownet
(12 items)
 
 
CPUMotherboardGraphicsRAM
5930k Asus rampage V Extreme  Inno3D GTX 970 Hercluez x4 airboss ultra  Gskill ripjaws DDR4 3200mhz 
Hard DriveHard DriveHard DriveCooling
WD black  WD Black  Seagate ES  Custom loop  
OSKeyboardPowerCase
Windows 10  Corsair k70  Super flower 1000 W plat  Corsair 900D 
  hide details  
Reply
Shadownet
(12 items)
 
 
CPUMotherboardGraphicsRAM
5930k Asus rampage V Extreme  Inno3D GTX 970 Hercluez x4 airboss ultra  Gskill ripjaws DDR4 3200mhz 
Hard DriveHard DriveHard DriveCooling
WD black  WD Black  Seagate ES  Custom loop  
OSKeyboardPowerCase
Windows 10  Corsair k70  Super flower 1000 W plat  Corsair 900D 
  hide details  
Reply
post #10 of 34
Thread Starter 
Thanks kaistledine smile.gif


Problems with the MO RA 3
The problem with all these fans is the wiring gets pretty messy especially the way I've gone about it wrt to y-splitters and extension cables. If I were interested in aesthetics, I'd have probably removed all the cable sleeves and used longer cables, then housed them neatly into one thick sleeve. In fact, I'd probably buy 18 extra long pwm cables and re-wire them with regards to length and splits (I did something similar for a signal cable which I'll explain later).

The pwm splitter device I mentioned above also needs a molex cable to deal with powering all the fans, so again, this would need considering wrt aesthetics!
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Build Logs
Overclock.net › Forums › Case Mods & Cases › Builds & Case Mods › Build Logs › Super Pallet Server - Quieting supermicro servers MK II - Watercooling