APOLLO (2CPU LGA1366 Server | InWin PP689 | 24 Disks Capacity) - by alpenwasser - Page 2 - Overclock.net - An Overclocking Community

Forum Jump: 

APOLLO (2CPU LGA1366 Server | InWin PP689 | 24 Disks Capacity) - by alpenwasser

Thread Tools
post #11 of 30 (permalink) Old 04-01-2014, 04:11 PM - Thread Starter
New to Overclock.net
alpenwasser's Avatar
Join Date: Jul 2013
Location: Switzerland
Posts: 557
Rep: 41 (Unique: 36)
Mounting the PSU, Testing LSI Controller

Finally, UPDATE TIME! smile.gif

Yeah, it's taking a lot longer to finish this than I'd initially hoped (doesn't
it always with these sort of projects...). But I've been working on it in the last
few weeks and now finally have something to share. smile.gif

PSU Fitting Issue

The PSU slides into this case through an opening from behind, and since the case
isn't really made for normal ATX-sized PSUs (but server PSUs instead, it's a
rather tight fit. To be more specific: The PSU in its stock config does not fit,
the screws for the fan grill and the fan grill itself bump up against the
case. An easy fix though, just needed to remove the fan grill on the PSU.

(click image for full res)

And voilà:

(click image for full res)

Furthermore, since normal server PSUs usually blow air through along their
longitudinal axis, there is no ventilation hole on the case for a fan on the top
of the PSU, which most of today's PSUs have. Not to worry, I still had an old
Aquacomputer rad grill laying around. A bit of dremeling should be able to fix
this problem. Marking for cutting:

(click image for full res)

And with the grill mounted:
(click image for full res)

Bracket Collision

Another issue with mounting a PSU in the case that wasn't intended to be mounted
in this case: The bracket for the PSU does not quite line up correctly with the
power inlet.
(click image for full res)

The power plug can still be connected, but the PSU sits crooked in the case and
it's a huge pain to mount like this.

(click image for full res)

Again, a little bit of cutting was required:

(click image for full res)

To give you an idea of how the PSU fits into the case:
(click image for full res)

(click image for full res)

Needed to hook up some HDDs to test the LSI controller. Looks very ghetto, worked
like a charm. smile.gif

(click image for full res)

(click image for full res)

Next Up...

Manufacturing the drive cages, the so-called pièce de resistance... biggrin.gif

So long
alpenwasser is offline  
Sponsored Links
post #12 of 30 (permalink) Old 04-01-2014, 04:25 PM
Starting to become a BOFH
legoman786's Avatar
Join Date: Mar 2006
Location: Tucson, AZ
Posts: 10,463
Rep: 364 (Unique: 304)
Ok... I'm interested.

Originally Posted by Syrillian go_quote.gif
The persistance of a human may never win against the persistance of time, but the fruits of the labors are wonderous to behold, even in our brevity.
⚜ R.I.P Syrillian ⚜

legoman786 is offline  
post #13 of 30 (permalink) Old 04-01-2014, 04:45 PM - Thread Starter
New to Overclock.net
alpenwasser's Avatar
Join Date: Jul 2013
Location: Switzerland
Posts: 557
Rep: 41 (Unique: 36)
Originally Posted by legoman786 View Post

Ok... I'm interested.

Thanks! smile.gif

It's not going to be super pretty (not worth the time and money, especially since I don't
really have either at the moment rolleyes.gif ), but it will hold 24 3.5" disks in the final config,
which I'm hoping should be enough for the foreseeable future and provide some sort
of eye-candy at least for storage fetishists. biggrin.gif
alpenwasser is offline  
Sponsored Links
post #14 of 30 (permalink) Old 04-02-2014, 08:11 AM - Thread Starter
New to Overclock.net
alpenwasser's Avatar
Join Date: Jul 2013
Location: Switzerland
Posts: 557
Rep: 41 (Unique: 36)
The Disk Racks

A.k.a. the main part of this undertaking. smile.gif

As mentioned elsewhere, one of the two main problem of our current server is
that it only has seven HDD slots, and they're already all filled up. The only
way to get more storage is to install larger disks, which isn't really all that

One of the main points of this build was to have more disk slots. The PP689 only
offers four in its stock form, which you can upgrade to a maximum of thirteen
drives. You would need to buy another four-disk enclosure (which btw. I could
not find anywhere to buy), and a five-disk enclosure for the 5.25" bays. Since
13 drives aren't really that many, and since these enclosures aren't exactly
cheap, I decided to go another route.

It took me a while to figure out how to do it, but in the end this is what I
came up with. I had very generous help from one of my neighbours, who has a mill
and a lathe at his disposal, as well as plenty of time (he's a pensioneer biggrin.gif ).

So off we went:

The Mill

(click image for full res)

First Steps

(click image for full res)

(click image for full res)

The Mill can also serve as a drill press. The drill chuck he looks ridiculously
huge when you put a small drill bit into it (he said they didn't have the
smaller model in stock when he needed to buy his, so he went with the large
one). biggrin.gif

(click image for full res)

Stumbled upon this when going through my pics. My dog's girlfriend, basically
(she's a labrador and belongs to another one of our neighbours). I was
dogsitting here for an evening a few weeks back. She can be a bit hyperactive at
times, but is a very lovely dog. smile.gif

(click image for full res)

Drilling and Milling

Lots of holes needed to be drilled for the pop rivets that were going to hold it
all together.

(click image for full res)

(click image for full res)

Milling out the slots for the screwheads:

(click image for full res)

Phase I Complete

The side panels of the disk racks completed. Testing with some broken old HDDs
I had laying around to make sure it all fits as it should. It does. smile.gif

(click image for full res)

(click image for full res)

(click image for full res)

Rail Detail

This is how the construct looks on the side where you slide in the disks. You
can see the pop rives I used to assemble it, the slots which are pictured being
milled above for the screwheads and the screws on the disks. You can also see
the recesses into which the screws mounted on the HDDs lock. The system works
very well.

(click image for full res)

(click image for full res)


Obviously, 24 HDDs are going to put out some heat, so some ventilation is
required. I'm using six Papst fans for that. The fans will be bolted onto the
panels with some L profiles. Unfortunately, 120 mm fans have 105 mm hole
distance, and HDDs are ~100 mm wide, so it's not possible to mount the fan on
both sides, only two screws can be used. It's not really a problem though, two
screws tightened down nicely give sufficient stability.

(click image for full res)

(click image for full res)

(click image for full res)

(click image for full res)

Mounting Brackets

The panels are mounted to the bottom and top of the case with screws. To have
some leeway in adjusting things, there are slots instead of round holes in some

(click image for full res)


(click image for full res)


(click image for full res)

Fan Mounting

Since the fan screws need to be tightened rather heavily, the screws exert quite
a bit of pressure on the fan frames. To prevent the fan frames from being
crushed and/or breaking, we made some brass bushings that take the brunt of the

(click image for full res)

And Mounted

And finally the disk racks are mounted inside the case. smile.gif

(click image for full res)

(click image for full res)

Disk Mounting

The Disks just slide into the slots and lock into place in the recesses you can
see above. Since I can't tighten the screws, I'm using Loctite to prevent them
from falling out due to vibration. I tried to get some screws similar to those
Lian Li use for their HDD mounting, but the only ones I could find were so
expensive that they'd have cost me more than 100 USD. So yeah, nope...

(click image for full res)

There's still lots to do, but that was by far the most work intensive part of
this build, took us quite a while to get it done. And no, it won't be painted or
anything, the server will stand in a closed room in our appartment anyway. I'd
have loved to make it all pretty and nice, but at the moment I just don't have
the time.

So long, and until next time. smile.gif
alpenwasser is offline  
post #15 of 30 (permalink) Old 04-08-2014, 12:59 PM - Thread Starter
New to Overclock.net
alpenwasser's Avatar
Join Date: Jul 2013
Location: Switzerland
Posts: 557
Rep: 41 (Unique: 36)
Chipset Cooling, Adventures in Instability

As some may be aware, I originally had some issues when trying to
get this machine to run stable. While stress testing with mprime,
it repeatedly and reproduceably crashed after less than an hour,
sometimes even already after a few minutes. Each time after
crashing, it took me several tries and about 10 to 20 minutes to
get the board to POST again.

After some troubleshooting and running a few diagnostics, it
turned out that the 5520 chipset was running really hot. It's
temperature threshold as indicated by the system is 95 degrees
Celsius, and when I was last able to check on it before a crash,
it had already passed 85 deg C, so I suspected that it was bumping
up against the threshold, upon which the board did an emergency
shutoff and mandated a cooldown period until it would run again.

As an emergency fix, I took the 80 mm San Ace fan that came with
the case and mounted it to the chipset heatsink with some waxed
cotton cord, and voilà somewhere slightly above 70 deg C maximum.

Unfortunately I forgot to take pictures of that rather ghetto
setup before dismantling it again and replacing it with something
more solid, but I have managed to blow up some sections from
another picture that should at least give you an idea of how it

Some Improvisation

Apologies for the horrid picture quality, as said this is a blowup
from a picture of which this section is only a small part.
(click image for full res)

A More Permanent Solution

The chipset heatsink is just your run of the mill alu heatsink held
on by a spring clamp with some hooks.

(click image for full res)

And the naked chipset after cleaning off the TIM. That stuff was a
***** to get off, it had dried up rather significantly.

(click image for full res)

Since the 80 mm fan is quite a bit larger than the chipset
heatsink itself, I needed to either replace the heatsink or modify
it in order to be able to mount the fan to it. I took a
rather crude, but very effective approach: I took an L piece of
aluminium, drilled two holes across the heatsink, cut some M4
threads on those two holes (which worked despite the holes only
going through the fins and not being continuous), then bolted the
L piece to the heatsink with two M4 screws. Works like a charm. smile.gif

Don't mind the unclean alu bits from the drilling and cutting on
the heatsink between the fins, it wasn't really possible to
properly clean that off and make the holes as clean as one usually

(click image for full res)

And from the other side...

(click image for full res)

The fan itself is held down by three screws, two in the L piece...

(click image for full res)

... and one in the corner of the heatsink itself. The bent fins
are from drilling and cutting the thread, they got a bit
structurally weak at their edges due to that. Doesn't impair
functionality, so not such a big deal since it won't be visible

(click image for full res)

And the whole package:

(click image for full res)

The heatsink unit mounted on the M/B. You need to unmount the fan
to do that. You can again see the bent fins here.

(click image for full res)

And mounted, with the fan:

(click image for full res)

That's it for today, thanks for stopping by. smile.gif

alpenwasser is offline  
post #16 of 30 (permalink) Old 04-09-2014, 01:23 PM - Thread Starter
New to Overclock.net
alpenwasser's Avatar
Join Date: Jul 2013
Location: Switzerland
Posts: 557
Rep: 41 (Unique: 36)
Disk Ventilation

Although disks have become quite frugal when it comes to
power consumption these days (at least some of them) and HDD
cooling is not really a huge issue for most people, packing
24 disks as closely together as in this build will cause
heat issues without ventilation. There is no need for 3k rpm
Delta fans though, a whiff of cool air breezing over the
disks should do the job nicely.

For this purpose, as you may have seen in some previous
pics, I have chosen 6 120 mm Papst fans, specifically the
4412 GLL model, and am running them at 7 V. The fans draw
air in through a vent area, and it then gets passed through
the M/B compartment and out the back.

Each fan is fixed to a rail riveted to one of the disk rack
panels with two screws.

You've seen this before, but for completeness' sake I'm
adding the pics of the bushings used to prevent the fan
frames from being crushed to this update as well:

(click image for full res)

I exchanged the copper screws for some silver ones, and in
the process added some dampening foam between the mouning
rails and the fan frame.

(click image for full res)

The whole fan panel assembly:

(click image for full res)

While doing some test runs, I noticed that a rather large
amount of air was being expelled through the front of the
case instead of going into the M/B compartment and out the
back (I wasn't really surprised by this seeing as how open
the front was). Obviously, this was not optimal. So I took a
1.5 mm panel of alu and bolted it to the front.

Because the existing front has a few folds in it, I needed
to do some cutting on the case first.

(click image for full res)

(click image for full res)

(click image for full res)

(click image for full res)

After having done that, I turned my attention to the side
panel, making an opening for the ventilation. I thought of
several ways of doing this, but all of them were a bit more
complicated than I'd have liked them to be. Cutting such a
big hole with a dremel isn't really practical, so I
considered doing it with our jigsaw, but after doing a few
test cuts I didn't really like the result as I couldn't get
a straight enough cut. And the cut needed to be clean,
because there's no space to fit a U channel over the edge,
and I don't really like the idea of covering it up on the

Anyway, the guy just used a nice big angle grinder for the
cut, and since he's a metal worker by trade, it turned out
almost perfectly straight (not 100%, but it's still cut by
hand, after all wink.gif). After that, I painted the bare edge
with some model paint to not have the blank metal staring at

I thought about painting the mesh, but at the moment I don't
really have the time, plus I kind of like the look of this
bare piece of alu, so I've left it as-is.

(click image for full res)

The mesh doesn't cover the whole fan area (nor is it very
open with those rather narrow slots), but there is no need
for high-power ventilation here, so this is not a big deal.

(click image for full res)

It's fixed to the inside of the panel with some double-sided
adhesive tape.

(click image for full res)

And in its final config:

(click image for full res)

Drive temperatures hover between 28 deg C and 35 deg C at
the moment, ambient is about 23 deg C. smile.gif

Until next time,
alpenwasser is offline  
post #17 of 30 (permalink) Old 04-11-2014, 10:24 AM - Thread Starter
New to Overclock.net
alpenwasser's Avatar
Join Date: Jul 2013
Location: Switzerland
Posts: 557
Rep: 41 (Unique: 36)
Triple Fan Unit

As hinted at earlier, the airflow in this build will go from
the front compartment through the middle wall into the M/B
compartment and out the back.

This is pretty much how the stock configuration works,
except in that the air gets in through the front panel, not
through the side panel.

Unfortunately I forgot to take pics of the stock config, but
luckily tweaktown.com did a review on this case and took them
for me. biggrin.gif

Source article where I got the image from can be found here.

In the stock config, the 92 mm fans are mounted inside
some plastic fan cages that allow quick and toolless fan
replacement in case of failure.

(click image for full res)

And without the fan cages:

(click image for full res)

Originally I just screwed the fans to two aluminium L
profile bars.

(click image for full res)

(click image for full res)

(click image for full res)

It was fixed to the middle wall with double-sided adhesive
tape. It's very strong stuff, so the fan unit falling off
was not a concern. Additionally, the tape has some thickness
to it, which should provide some dampening between the fan
unit and the middle wall.

(click image for full res)

Unfortunately, due to some bumps on the middle wall getting in
the way, the tape on the rear angle didn't make proper contact
with the wall. It held, but not very well.

Additinoally, I noticed that there were rather strong
vibtrations on the middle wall. It turned out that the tape
did indeed offer some decoupling, but it also did not
offer any additional strength to the middle wall (i.e. no
additional stiffness), which meant the wall could easily

(click image for full res)

So, I took the unit out, and while I was at it, I also cut
out some recesses for the fans which I didn't bother doing
before. I also put some dampening foam between the fans and
the alu angles.

(click image for full res)

Aaand of course I mounted the fans the wrong way
round. Sigh.

(click image for full res)

Disassemble again, reassemble.

(click image for full res)

Also: Foam between the alu angles and the wall itself:
(click image for full res)

This time I bolted it to the wall with some screws. Much
more solid now, no more vibrations. smile.gif

(click image for full res)

How it looks from the other side:

(click image for full res)

alpenwasser is offline  
post #18 of 30 (permalink) Old 04-12-2014, 12:19 PM - Thread Starter
New to Overclock.net
alpenwasser's Avatar
Join Date: Jul 2013
Location: Switzerland
Posts: 557
Rep: 41 (Unique: 36)
Storage Topology & Cabling

Storage Topology

In case you can't read the text, the full res version should
be more easily readable.

(click image for full res)

The idea behind the storage topology is based on the concept
Any one of the three LSI controllers can fail and I still
have all my data available.

You'll see below that I haven't yet gotten around to
installing the Velociraptor.

I use coloured zip ties to mark the cables that go to the
different controllers.

BLUE = controller 0
YELLOW = controller 1
GREEN = controller 2


There isn't really any space to hide the cables, so this was
rather tricky and required three attempts until I was
satisfied with the result. In the end I hid the extra cable
behind the triple fan unit, good thing they're 38 mm fans,
which makes the space behind them just about large enough to
fit the extra cable bits.

The power cables for the disks are two cables that came with
the PSU and onto which I just put a lot more connectors
while taking off the stock connectors because those were
neither placed in the correct locations nor facing in the
right direction.

Looks harmless, right? Yeah...

(click image for full res)

And the disks:
(click image for full res)

OK then, first try:
(click image for full res)

I soon realized that this wasn't going to work. The problem
was that I had the disks arranged in the same way as the
will be set up in the storage pool layout, so the disks
which go into the same storage pool were also mounted below
each other. Sounds nice in theory, but if you want to
have disk from each pool distributed among the different
controllers, you'll get quite the cable mess.

(click image for full res)

(click image for full res)

Second Try

Next try, this time I arranged the disks to that the cables
to the controllers could be better laid out. Since I wanted
to set up all the cables for all the disk slots, even ones
that will stay empty for now, I had to shuffle the disks
around when laying out the cables.

(click image for full res)

(click image for full res)

(click image for full res)

Better. But I still wasn't quite happy, mainly because...

(click image for full res)

(click image for full res)

(click image for full res)

... of this:

(click image for full res)

Third Try

This time I made sure the cables stayed tidy on both ends
while hiding the mess (which cannot be avoided since all
cables are the same length but lead to different end points,
obviously) behind the triple fan unit.

(click image for full res)

The loop of extra cable length for the top cable loom:

(click image for full res)

And the cable loom for controller 0, from the disk side...

(click image for full res)

and the M/B side. Much better IMHO. smile.gif

(click image for full res)

The bottom controller had a bit more extra cable length to hide, so
that part is a bit messier.

(click image for full res)

And the middle one:
(click image for full res)

Tada! While not perfect (I'd need longer cables for that to
make cleaner runs, but I'm not buying more cables just for
the sake of that for a build that has a closed side panel),
with this iteration of my cabling I'm now rather happy:

(click image for full res)

(click image for full res)

(click image for full res)

And the other side. Much better than before methinks. smile.gif

(click image for full res)

(click image for full res)

The SATA cable for the system SSD:

(click image for full res)

And the controller LEDs when there's some activity:

(click image for full res)

Now if you'll excuse me, there's a dinner waiting to be
cooked. smile.gif

alpenwasser is offline  
post #19 of 30 (permalink) Old 04-13-2014, 06:32 AM
New to Overclock.net
Aximous's Avatar
Join Date: Dec 2009
Location: Hungary
Posts: 490
Rep: 23 (Unique: 21)
Very nice build, I love the HDD rack you did, made me think about making something similar for my server. Also I'm pretty sure that I'm gonna steal your chipset cooler idea, it's ridiculous how hot that chip gets.

I'm curious though about your choice to go Arch instead of something more enterprise focused like CentOS or Debian.
Aximous is offline  
post #20 of 30 (permalink) Old 04-13-2014, 06:49 AM - Thread Starter
New to Overclock.net
alpenwasser's Avatar
Join Date: Jul 2013
Location: Switzerland
Posts: 557
Rep: 41 (Unique: 36)
Originally Posted by Aximous View Post

Very nice build, I love the HDD rack you did, made me think about making something similar for my server.

Thanks, I appreciate the compliment! smile.gif
Originally Posted by Aximous View Post

Also I'm pretty sure that I'm gonna steal your chipset cooler idea, it's ridiculous how hot that chip gets.

Yes, ridiculous indeed. I noticed that when I started playing with my SR-2, the fan on
that thing was very busy. Good thing I now have a watercooler on that chipset, but I
didn't really feel like going W/C for this bulid. wink.gif
Originally Posted by Aximous View Post

I'm curious though about your choice to go Arch instead of something more enterprise focused like CentOS or Debian.

Two reasons primarily, the first one being my familiarity with it. I know there are people
who tend to go "Ah, Arch, bleeding edge, unstable!" and all that. But in all honesty, I've
been using it as my daily driver on several machines for three years now, and I've had
only one case of actual proper system breakage, and that was related to Gnome. And
even then, I just did a clean reinstall and was back up and running within about two
hours with all my settings and stuff from before.

I know my way around Arch well enough to feel comfortable with it and be efficient-ish
when needing to troubleshoot, which I can't say for Debian (or FreeBSD, which I actually
also considered at some point and did play around with on another machine for a while),
or other distros (I could learn, of course, but at the moment I'm a bit pressed for time with
college and all, I need this thing up and running sooner rather than later).

Secondly, ZFS support is very good on Arch, whereas I've read a few posts around some
forums which said that ZFS under Debian-based distros is... hinky. I haven't personally
tried it, so I can't speak from personal experience on that one though. I have been using
ZFS on Arch on another machine for about nine months now and it's been working very
well, so I thought I'd deploy it on this machine too.

So basically, "If it ain't broke, don't fix it." biggrin.gif
alpenwasser is offline  

Quick Reply

Register Now

In order to be able to post messages on the Overclock.net - An Overclocking Community forums, you must first register.
Please enter your desired user name, your email address and other required details in the form below.
User Name:
If you do not want to register, fill this field only and the name will be used as user name for your post.
Please enter a password for your user account. Note that passwords are case-sensitive.
Confirm Password:
Email Address
Please enter a valid email address for yourself.
Email Address:


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page

Forum Jump: 

Posting Rules  
You may post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off