Putting Together a 2P/4P Rig - From Wallet Emptying to Running bidadv
I've gotten a lot of positive comments from my build log and I wanted to make an "unofficial" how to thread instead of just going off a build log. This is in slight interference with Hoth17's multicpu discussion thread but as he is no longer with us I would like to update and expand upon his thread.
This is something as a team we need to get on as soon as humanely possible. As of right now I believe there are 10 of us who have either completed or very close to being completed 2p/4p rigs. H has an army of these things and EVGA just has a lot of folders. Multiple cpu rigs are the future of folding and the more people know the better
The first post will be detailed instructions on how to construct your own multiprocessor thread, along with different budget options that are available. The second post will be PPD benchmarks for various configurations. The third post will be a gallery.
Buying Parts
In all honesty, the best place to buy chips right now is eBay. It's just silly to pay $1240 for a chip that you could get for $450 on eBay. Yes we all have our issues with eBay, but it just is the most logical thing to do. Unless you just have cash to burn, but even then you could build 3 4P rigs for the price of one

For motherboards, Wiredzone is a good place. They have awesome prices and fast shipping. I ordered my board either Thursday night or Friday morning and it arrived to my door the following Monday.
For RAM, cooling, and PSUs there aren't any special places to order them from. Find the cheapest, most reliable place you can
CPUs (Click to show)
Budget: AMD 6128 8 Core Processor
Most common and best PPD/W: AMD 6174 12 Core Processor
Highest end 6180s
MC vs IL
The new Interlagos chips are based on BD...and we all know how that went. Fortunately the IL processors are actually faster then the MC processors. They just suck a helluva lot more power. Using 4x16core IL vs using 4x12core MC nets you approximately 15k more PPD [doing more research] but it will be sucking about 100-150 more watts from the wall. So if you can afford IL and don't pay your electricity bill then please be my guest and buy them

Quote:
Originally Posted by hoth17 
As for MC vs. IL, I'm still MC (I bought 4 6174s). I'm sure you have read the MC IL debate over at [H], if not, off you go, http://hardforum.com/showthread.php?t=1652213. However, I'm sure you have, so here is my take on it.
A 6272 will perform about 25 seconds faster than a 6174, that will be about 15k faster. Now, if you were planning on buying your chips retail regardless of price, IL all the way. However, because a 6174 can be had on ebay for ~450 (I just bought 4 [auctions] that averaged out to $450 each), you are saving $400 dollars over IL. That, and it will consume 100+ watts less, MC is the right choice. Even at $525, 15k PPD is not worth the extra watts, if you are paying for it.
Now, if you are not paying for your own electricity, and you don't want to wait for and 6174 auctions to come up, then IL is the way to go.
Having said all of that, there is a fourth option. Right now, I think it is the best option. Buy a 4p board and 4 x 6128s. This will cost about $1300. That is about the cost of a 2p board and two MC or IL chips (maybe slightly more). It will net you 260k PPD, which is far more than the 180ish you can expect form a 2p IL or MC setup. Then, over time, as auctions come up, you can upgrade to MC or IL. And better yet, used IL may start cropping up for < $400.

As for MC vs. IL, I'm still MC (I bought 4 6174s). I'm sure you have read the MC IL debate over at [H], if not, off you go, http://hardforum.com/showthread.php?t=1652213. However, I'm sure you have, so here is my take on it.
A 6272 will perform about 25 seconds faster than a 6174, that will be about 15k faster. Now, if you were planning on buying your chips retail regardless of price, IL all the way. However, because a 6174 can be had on ebay for ~450 (I just bought 4 [auctions] that averaged out to $450 each), you are saving $400 dollars over IL. That, and it will consume 100+ watts less, MC is the right choice. Even at $525, 15k PPD is not worth the extra watts, if you are paying for it.
Now, if you are not paying for your own electricity, and you don't want to wait for and 6174 auctions to come up, then IL is the way to go.
Having said all of that, there is a fourth option. Right now, I think it is the best option. Buy a 4p board and 4 x 6128s. This will cost about $1300. That is about the cost of a 2p board and two MC or IL chips (maybe slightly more). It will net you 260k PPD, which is far more than the 180ish you can expect form a 2p IL or MC setup. Then, over time, as auctions come up, you can upgrade to MC or IL. And better yet, used IL may start cropping up for < $400.
Motherboards (Click to show)
SuperMicro H8QGi+-F - WiredZone
SuperMicro H8QGL-iF+ - WiredZone
SuperMicro H8DGU - WiredZone
TYAN S8812WGM3NR - Newegg
TYAN S8236GM3NR - Newegg
Right now SuperMicro is the preferred manufacturer for boards as they are overclockable.
The board I'm using for my build is the second link, and also the newer board; it's also cheaper
. I would personally recommend it, unless you need more RAM slots and prefer the alternative layout for whatever reason, then you could go with the H8QGi+-F.RAM (Click to show)
With SuperMicro boards we can now use 1600Mhz ram effectively! 
Crucial Ballistix 4GB (2 x 2GB)
^this has been the most recommended ram so far.
If you want 4GB dimms then check out Crucial 8GB (2 x 4GB)

Crucial Ballistix 4GB (2 x 2GB)
^this has been the most recommended ram so far.
If you want 4GB dimms then check out Crucial 8GB (2 x 4GB)
Coolers (Click to show)
These chips do not run hot. A CM Hyper 212+ will cool them perfectly fine. They do however require custom mounting.
For custom mounting instructions scroll down on this post.
Here is a list of "native" g34 coolers.
Also I would like to bring it to attention that Koolance has launched a new screw set for their 360/370 block line! This is very exciting
For custom mounting instructions scroll down on this post.
Here is a list of "native" g34 coolers.
Also I would like to bring it to attention that Koolance has launched a new screw set for their 360/370 block line! This is very exciting

PSUs (Click to show)
Get solid built PSUs. DO NOT cheaper out here. Remember this will be running 24/7! These setups don't pull more then ~550w [Unless you get IL] from the wall but to maximize efficiency you want to buy a solid good brand name PSU. My personal recommendation for any 4p build is the Corsair AX1200.
More then like you will need this adapter as well. Most PSUs, including the AX1200 will only have 2 EPS 8 pin slots. I am using the same cable with my PSU and I've had no issues.
Also another adapter that has been recommended and overall might be a better one to use
Link
More then like you will need this adapter as well. Most PSUs, including the AX1200 will only have 2 EPS 8 pin slots. I am using the same cable with my PSU and I've had no issues.
Also another adapter that has been recommended and overall might be a better one to use
Cases (Click to show)
Go Full On custom like I am:

This is a custom prototype done by Rich Chomiczewski.
His website can be found here.
Quote:
Originally Posted by hoth17 
You have a few options.
If you want it rackmounted
NORCO RPC-470 Black 4U Rackmount Server Case
If not
Rosewill BLACKHAWK-ULTRA
COOLER MASTER HAF 932

You have a few options.
If you want it rackmounted
NORCO RPC-470 Black 4U Rackmount Server Case
If not
Rosewill BLACKHAWK-ULTRA
COOLER MASTER HAF 932
Other options
G34 Builds
~$1000 Budget (Click to show)
Part List:
2x AMD MC 6128 8 Core Processors - $100 each - eBay
Supermicro H8DGU - $350 or TYAN S8236GM3NR - $450
RAM - 2 Sets of these - $60 each
Cooling - 2xCM 212+s - $30 each These will for sure be mountable on the SuperMicro board, I need to do further research on the Tyan Board.
PSU - Anything above a 650W will be good. ~$150
Throw in an old HDD you have laying around and you're good to go.
Total: $880-$980, not including shipping.
This section still needs some more work as I haven't done a whole lot of research on 2P boards myself.
2x AMD MC 6128 8 Core Processors - $100 each - eBay
Supermicro H8DGU - $350 or TYAN S8236GM3NR - $450
RAM - 2 Sets of these - $60 each
Cooling - 2xCM 212+s - $30 each These will for sure be mountable on the SuperMicro board, I need to do further research on the Tyan Board.
PSU - Anything above a 650W will be good. ~$150
Throw in an old HDD you have laying around and you're good to go.
Total: $880-$980, not including shipping.
This section still needs some more work as I haven't done a whole lot of research on 2P boards myself.
~$1500 Budget (Click to show)
Part List::
4x AMD MC 6128 8 Core Processors - $100 each - eBay
/Supermicro H8QGL-IF - $640
RAM - 4 Sets of these - $60 each
Cooling - 4xCM 212+s - $30 each
PSU - Again your choice, just get a quality brand, I would recommend at least 850W. 1200W if you want to have the most efficiency possible.
HDD - Same thing, anything you have laying around works.
Total: $1400+PSU, not including shipping
Probably going to be a little bit above $1500, but for that budget this is the best you can build, it'd be worth it to stretch it another $100
4x AMD MC 6128 8 Core Processors - $100 each - eBay
/Supermicro H8QGL-IF - $640
RAM - 4 Sets of these - $60 each
Cooling - 4xCM 212+s - $30 each
PSU - Again your choice, just get a quality brand, I would recommend at least 850W. 1200W if you want to have the most efficiency possible.
HDD - Same thing, anything you have laying around works.
Total: $1400+PSU, not including shipping
Probably going to be a little bit above $1500, but for that budget this is the best you can build, it'd be worth it to stretch it another $100

~$2000 Budget (Click to show)
Part List:
Now here is where it gets a little awkward because it's between a 4p 6174 board and a 4p 6128 board. But it's right in the middle... Probably your best option is to take the $1500 build, and switch out 4 6128s for 2 6174s. You aren't going to get as much PPD, but this leaves great opportunities to expand to a full on 4p 6174 system as money becomes available.
Will work on other options for this build later
Now here is where it gets a little awkward because it's between a 4p 6174 board and a 4p 6128 board. But it's right in the middle... Probably your best option is to take the $1500 build, and switch out 4 6128s for 2 6174s. You aren't going to get as much PPD, but this leaves great opportunities to expand to a full on 4p 6174 system as money becomes available.
Will work on other options for this build later

~$3000 Budget (Click to show)
Part List:
4x AMD MC 6174 12 Core Processors - $450-$500each - eBay
/Supermicro H8QGL-IF - $640
RAM - 4 Sets of these - $60 each
Cooling - 4xCM 212+s - $30 each
PSU - Again your choice, just get a quality brand, I would recommend at least 850W. 1200W if you want to have the most efficiency possible.
HDD - Same thing, anything you have laying around works.
Total: $2800+PSU
4x AMD MC 6174 12 Core Processors - $450-$500each - eBay
/Supermicro H8QGL-IF - $640
RAM - 4 Sets of these - $60 each
Cooling - 4xCM 212+s - $30 each
PSU - Again your choice, just get a quality brand, I would recommend at least 850W. 1200W if you want to have the most efficiency possible.
HDD - Same thing, anything you have laying around works.
Total: $2800+PSU
>$3000 budget (Click to show)
Part List:
4x AMD MC 6180 12 Core Processors - $500-550each - eBay
/Supermicro H8QGL-IF - $640
RAM - 4 Sets of these - $60 each
Cooling - 4xCM 212+s - $30 each
PSU - Again your choice, just get a quality brand, I would recommend at least 850W. 1200W if you want to have the most efficiency possible.
HDD - Same thing, anything you have laying around works.
Total: $3200+PSU
4x AMD MC 6180 12 Core Processors - $500-550each - eBay
/Supermicro H8QGL-IF - $640
RAM - 4 Sets of these - $60 each
Cooling - 4xCM 212+s - $30 each
PSU - Again your choice, just get a quality brand, I would recommend at least 850W. 1200W if you want to have the most efficiency possible.
HDD - Same thing, anything you have laying around works.
Total: $3200+PSU
Server Rack Required Components (Click to show)
If you're wanting to build a 4p setup in a server rack your options become pretty limited if you want to avoid any "hardcore" modding.
H8QGi+-F
Noctua NH-U9DO-A3
NORCO RPC-470 Black 4U or similar
The rest is up to you

SR-2 Builds (Click to show)
New SR-2
Best places to buy a used SR-2 is either Fleabay or EVGA B-Stock
For chips, you're going to need to get dual hexacores if you want to be able to do the new Bigadv WUs. At least L5640s. You're also going to need a decent OC
@2.4Ghz they failed. @2.96Ghz they've completed them.Putting together a 4P - From box to powering on (Click to show)
1. Install CPUs & RAM

2. Check out musky's guide for custom mounting 212+ necessary hardware
3. Continue to look between my pictures and musky's because I'm a douche and didn't take all the necessary pictures
Note*my custom brackets are upside down compared to musky's... I did that on purpose. Not sure how he fitted his like that but mine would not fit in that orientation.






Actually I made a video, as soon as I edit I'll make up for the lack of pictures I took

4. Yay you're finished


Necessary steps to setup a 4P for folding (Click to show)
Basic Ubuntu Desktop Install
- This is about as simple as you are going to get. You can still download Ubuntu 10.10 HERE.
- Install Ubuntu Desktop 10.10 64 bit. It will install nicely from a USB drive with unetbootin. The only thing you should change is to use ext3 instead of the default ext4 filesystem when partitioning the drive. I think this is pretty straightforward to do from the installer. HERE are some screenshots of a pretty typical install.
- System - Administration - Update Manager. Click "Check" to let it find all available updates. Click "Install Updates" - you will be prompted for your password. Let it update.
- System - Administration - Update Manager. Click the "Settings" button and uncheck the "Check for update" box.
- Places - Home Folder. Create a new folder called fah (Right-click - Create Folder)
- Right-click on the new folder - Sharing Options - Check the Share This Folder checkbox - it will prompt you to install a package, let it install. It will want you to restart your session, don't yet. Cllick Cancel on the sharing box.
- Reboot the machine. When you log back in, go back to Places - Home Folder, right-click your fah folder and select Sharing options, and check all three checkboxes. You will get something about adding permissions, click OK.
- Open a terminal (Applications - Accessories - Terminal) and type:
Code:
cd ~/fah wget "http://www.stanford.edu/group/pandegroup/folding/release/FAH6.34-Linux64-SMP.exe" mv FAH6*.* fah6 chmod 755 fah6 - Run F@H config:
Code:
./fah6 -configonly - Enter all your stuff. I add the -smp -bigadv flags in the configuration to prevent having to make a shortcut with them, but to each his or her own.
- Run F@H:
Code:
./fah6
Install BFS - for all Intel systems I have tested. NOT for multi-socket AMD systems. (Click to show)
1. Open a terminal and type this:
Code:
sudo add-apt-repository ppa:chogydan/ppa && sudo apt-get update
Edit from 10e:
If you get a "command not found" error when running "sudo add-apt-repository ppa:chogydan/ppa && sudo apt-get update"
You need to do:
Code:
sudo apt-get install python-software-properties
2. Once it finishes, type this:
Code:
sudo apt-get install linux-image-generic-ck linux-headers-generic-ck
3. Still in the terminal, type;
Code:
sudo update-grub
Code:
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.35-28-generic-ck
Found initrd image: /boot/initrd.img-2.6.35-28-generic-ck
Found linux image: /boot/vmlinuz-2.6.35-28-generic
Found initrd image: /boot/initrd.img-2.6.35-28-generic
Found memtest86+ image: /boot/memtest86+.bin
done
Look for the ones that ends in -ck. If the first two entries end in "-generic-ck" like the example, you are finished with this step. If they are the third and fourth entries, type this:
Code:
sudo nano /etc/default/grub
Code:
GRUB_DEFAULT=0
Code:
GRUB_DEFAULT=2
Code:
sudo update-grub
If they are the fifth and sixth, you would set GRUB_DEFAULT=4, and so on. They should be either the first and second or third and fourth.
4. Now, reboot the system. Once it reboots, open a terminal and type:
Code:
uname -r
Install "The Kraken" wrapper - for all multi-socket systems
A big thanks to tear for this. The original guide is here.
*EDIT - 10/30/11 - Updated for version 0.6
*EDIT - 5/20/12 - Updated for version 0.7-pre10
*EDIT - 5/22/12 - Updated to version 0.7-pre11
*EDIT - 6/28/12 - Updated to version 0.7-pre15
*NOTE: To get the most use out of the kraken, you need to have NUMA enabled (SR-2s, other 2p Intel boards) or node interleaving disabled (G34/C32/Socket F AMD systems). If you have a multi-processor board, you should have a bios setting for one or the other.
1. First, make sure the client has downed a core. The wrapper wraps the actual FAH core, so it doesn't do anything until you have a core downloaded.
2. Download the package:
3. Untar it
4. Install it
5. The Kraken is now available at /usr/bin/thekraken. To wrap your existing A3/A5 cores, move to your fah directory and run The Kraken with the -i flag:
You should see something like this:
6. Start the client. From another terminal window, type:
If you top process looks something like this, "The Kraken" is working correctly:
A big thanks to tear for this. The original guide is here.
*EDIT - 10/30/11 - Updated for version 0.6
*EDIT - 5/20/12 - Updated for version 0.7-pre10
*EDIT - 5/22/12 - Updated to version 0.7-pre11
*EDIT - 6/28/12 - Updated to version 0.7-pre15
*NOTE: To get the most use out of the kraken, you need to have NUMA enabled (SR-2s, other 2p Intel boards) or node interleaving disabled (G34/C32/Socket F AMD systems). If you have a multi-processor board, you should have a bios setting for one or the other.
1. First, make sure the client has downed a core. The wrapper wraps the actual FAH core, so it doesn't do anything until you have a core downloaded.
2. Download the package:
Code:
cd ~
wget http://darkswarm.org/thekraken-0.7-pre15.tar.gz
Code:
tar xvf thekraken-0.7-pre15.tar.gz
Code:
cd thekraken-0.7-pre15
make
sudo make install
Code:
cd ~/fah
thekraken -i
Code:
dave@bulldawg:~/fah$ thekraken -i
thekraken: The Kraken 0.7-pre15 (compiled Tue May 22 05:22:45 CDT 2012 by dave@bulldawg)
thekraken: Processor affinity wrapper for Folding@Home
thekraken: The Kraken comes with ABSOLUTELY NO WARRANTY; licensed under GPLv2
thekraken: performing installation to .
thekraken: /home/dave/fah/FahCore_a5.exe: wrapper succesfully installed
thekraken: descend into /home/dave/fah/backup and all other subdirectories [Y/n]? Y
thekraken: finished installation, 1 out of 1 files processed
Code:
top
Code:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1984 dave 1 13 2789m 1.3g 3192 S 2384 22.8 16476:48 thekraken-FahCo
Start FAH on Boot (Click to show)
Code:
sudo nano /etc/rc.local
Code:
cd /home/[user]/fah && sudo -u [user] screen -d -m ./fah6
Code:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
cd /home/fah/fah && sudo -u fah screen -d -m ./fah6
exit 0
Code:
screen -r
Start FAH in Background screen (Click to show)
You need to make a small script to do this. Type
You should be looking at a blank file. Enter the following:
Control X then Y then Enter to save the file. Make your script executable by typing:
To run it, type
To make sure it started, attach to the new screen by typing
You should see the client running. To detach, hold down the Control key and type A the D.
Code:
cd ~/fah
nano start
Code:
#!/bin/sh
# Start FAH
screen -d -m ./fah6
Code:
chmod 755 start
Code:
./start
Code:
screen -r
HFM Install in Linux (Click to show)
HFM.net will run under Ubuntu and probably any other Linux GUI distro. The setup is a little more difficult, but it is not that bad.
Update sources list
Add these two lines to the end of the file:
Update the keys
Update everything
Update mono libraries and forms:
Make sure unzip is instaled
Get HFM.net
To run HFM type:
Setting up clients - example
Let's assume we have three remote machines with static IP addresses named larry (192.168.1.150), curly (192.168.1.155), and moe (192.168.1.160). These three machines all have a shared directory called "fah". You are logged into your local machine as user "shimp". First, make some share directories
Make sure smbfs is installed
Mount your shares
To mount the shares on boot, you need to update fstab:
Add these three lines to this file
Adding clients is similar to Windows. If you followed these instructions, the path to your remote clients will be /home/shimp/hfm/mounts/. If you have a local client, that path will be /home/shimp/fah/.
Install the bios OC (Click to show)Update sources list
Code:
sudo nano /etc/apt/sources.list
Code:
deb http://ppa.launchpad.net/directhex/monoxide/ubuntu maverick main
deb-src http://ppa.launchpad.net/directhex/monoxide/ubuntu maverick main
Code:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 23DC003A
Code:
sudo apt-get update
Code:
sudo apt-get install libmono-system-runtime2.0-cil
sudo apt-get install libmono-winforms2.0-cil
Code:
sudo apt-get install unzip
Code:
mkdir hfm
cd hfm
wget http://hfm-net.googlecode.com/files/HFM%20Release%200.6.2.336.zip
mv HFM*.* hfm.zip
unzip hfm.zip
Code:
cd ~/hfm
sudo mono HFM.exe
Setting up clients - example
Let's assume we have three remote machines with static IP addresses named larry (192.168.1.150), curly (192.168.1.155), and moe (192.168.1.160). These three machines all have a shared directory called "fah". You are logged into your local machine as user "shimp". First, make some share directories
Code:
mkdir /home/shimp/hfm/mounts/larry
mkdir /home/shimp/hfm/mounts/curly
mkdir /home/shimp/hfm/mounts/moe
Code:
sudo apt-get install smbfs
Code:
sudo smbmount //192.168.1.150/fah /home/shimp/hfm/mounts/larry -o guest
sudo smbmount //192.168.1.155/fah /home/shimp/hfm/mounts/curly -o guest
sudo smbmount //192.168.1.160/fah /home/shimp/hfm/mounts/moe -o guest
Code:
sudo nano /etc/fstab
Code:
//192.168.1.150/fah /home/shimp/hfm/mounts/larry smbfs guest 0 0
//192.168.1.155/fah /home/shimp/hfm/mounts/curly smbfs guest 0 0
//192.168.1.160/fah /home/shimp/hfm/mounts/moe smbfs guest 0 0
Adding clients is similar to Windows. If you followed these instructions, the path to your remote clients will be /home/shimp/hfm/mounts/. If you have a local client, that path will be /home/shimp/fah/.
Tear, of the 4P Supermicro Brotherhood, has posted an overclocking guide and link to a new OC BIOS HERE.
Salute to the mad Folders at [H]. - Thanks Linden
Salute to the mad Folders at [H]. - Thanks Linden
Other Useful info:
Other information I found very helpful but was hard to find--mostly over on [H]:
- Stopping your fah client can result in a bad checkpoint file and lost work. Here is how to backup your fah data automatically.
- It is bad practice to test your new build with live data. Here is how to setup a test fah configuration.
- The FAH client does not like IPv6. Here is how to disable it.
- Turion Power Control is a great way to monitor and tweak your build. Here is how to download and install it.
- Musky's FAQ from the main OC bios thread on [H] has some great tips like how to check for hypertransport retries and how to see what CPU speed was detected by Ubuntu.
- Here is how to see what refclock you currently have set.
Cheers TMTNET!
Future Build Info (Click to show)

Latest possible road map:

See the following posts for more info on PPD benchmarks of different systems along with how things are setup

Thank you for folding and even considering build such a monstrous rig dedicated to folding the proteins
Edited by derickwm - 8/27/12 at 1:58am




















