Overclock.net banner

Darkness, a Corsair 750D build (72TB)

14271 Views 61 Replies 5 Participants Last post by  janiirfan244
I mean who better than to keep my important data safe than Darkness?

Is there anything she is not willing to take for fun, I mean to protect her friends.



Also, I mean who else are you going to trust all your information to?




Yes for those wondering this is yet another character from konosuba as I felt it would be fun to have someone to go with Megumin and if we already got a system for the best girl we might as well have one for the second best as well.

Unlike my previous build I will not be going out of my way to make the actual build physically themed for Darkeness. It is just a server but at least I can have some fun with my Windows profile.

Onto the build!!

Specs:
Case: 750D Airflow​
CPU: RYZEN 3 1200​
M/B: X470 Taichi Ultimate​
RAM: 2x M391A1G43EB1-CPBQ (8GB ECC) (16GB ECC TOTAL)​
PSU: SSR-1000TR​
HBA Card: 2x AOC-SAS2LP-MV8​
HBA HT: 2x SLF-1 Ultra​
SSD Cache: 2x 970 EVO M.2 2280 250GB (One on each of the separate underline drives/arrays in a duplication pool)​
GPU: ZOTAC ZT-71304-20L (PCIe x1)​
HDDs:
4x 10TB WD White Label (removed from elements external case)
2x 8TB Deskstars
2x 4TB Toshiba
2x 4TB Deskstars


All the drives are ones I already have. I am moving them from an existing computer over to the new one.

--The END of HW based RAID--
One of the big changes in this build from my old one is removing my HW based RAID 6 array and replacing it with a software based solution similar to RAID 10.

I will be using DrivePool and storage spaces to accomplish this. The 2tb drives will be split into two pools which are then pooled together. This is so that backups can be done on the underline 4tb pools so that in the event of data loss only the section containing data loss will require recovery instead of the entire resultant large pool. As you can see raid 0 is used in pairs of the 2tb drives for increased sequential performance. This same method will be applied to the 4tb drives in the future as more are added.

For instructions on how to setup storage spaces to use RAID 0 see: Darkness, a Corsair 750D build (48TB)

[2TB -- RAID 0 -- 2TB]_[2TB -- RAID 0 -- 2TB]
----MIRROR POOL---------->Unified Pool of Pools<---------MIRROR POOL-----
[2TB -- RAID 0 -- 2TB]_[2TB -- RAID 0 -- 2TB]




The wiring for the HDDs is very deliberate to ensure that drives that are mirrors are never on the same power line or sata controller. Drives that are in the same raid 0 array share the same controller and power line. This is to prevent something like an HBA failure resulting in data not being accessible.

You will see that I have also used primocache with two 970 EVOs as a cache. This is to improve write performance to the system. I had to use two of them as drivepool with real time duplication enabled has a 2x read/write penalty to the cache as it is only a single target. So if you use the same cache for both drives/pools that are being duplicated you will get 2 of everything dumped to it. Thus you need a separate cache for each part. You can see how the cache is split up in the image above.

---ECC RAM---
One of the reasons I went with Ryzen for this build was that I would be able to use ECC ram. Now while ECC ram is not a requirement it is a clear choice to help avoid possible data corruption on a file server. The problem is that I still at the time of my build had not found ANY documented case of someone using ECC ram with Windows and having it work. So although I knew there was ECC ram listed on the QVL for my motherboard I was taking a risk ordering it to find out once and for all if ECC ram does fully work today on windows. As you can see from the image below it does and you can read more about it here: Darkness, a Corsair 750D build (48TB)


--10gbe--
The reason I chose this motherboard (in addition to the 8 sata ports) was the support for 10gbe. My main rig already has support for 5gbe and I wanted to ensure that this build was future proof. I already have a switch that supports 1/2.5/5/10gbe so there was nothing holding me back. The read cache helps me take advantage of some of the benefits but over time as I add more RAID 0 drives and performance improves the 10gb connection will give me plenty of headroom.

--acronis true image--
I managed to get 3 copies of this for free during an amazing newegg special! (after rebate) I am using the program to create daily backups of my server and other computers. The backups of the server drive are stored only on non RAID 0 volumes as in the event of a disaster I need easy access to the files and pulling them from a simple mirror between two drives will be much easier as I would only need to pull a single drive from the system and grab the files. (or I could just download the file from backblaze in a short time). Thanks to compression the backups of the OS are around 13GB and I can even download that for a restore from backblaze in a few min. This makes recovering from most major disasters pretty easy. In the event of a motherboard failure I suppose I would need to rebuild my storage spaces if the new motherboard is very different but honestly I am not sure how easy the recovery process is for those.

This build is highly inspired by the 750D build that fg2chase did here: Ok... So who needs a 50-66TB server??
He has been a big help in planning this build and if you have not please check out his thread.
21 - 40 of 62 Posts

· The Physical Manifestation of Typos
Joined
·
6,960 Posts
Or alternatively never sleep and sit next to the computer 24/7 so you can stop the reboot lol.
That's life advice right there :p
 

· Registered
Joined
·
623 Posts
Discussion Starter · #22 ·
All files migrated from my RAID 6 array. I took down the array and moved the drives over. I see now that it is easy for the power wire to get disconnected to one of the drives so I will swing by the store this weekend to grab an extender. Unfortunately by extender I mean splitter as it looks like they do not have any sata to sata power extensions from what I can see.

I did get the other drives powered up and testing and so far one came back as about to fail so I got to swap that out. (I got a spare)

Once all the drives can clear the surface scan I will build them into the array/pool and start moving the files over to it.

Once everything is settled I will try primocache again so that I can properly benchmark it and test every setting combination.
 

· Registered
Joined
·
623 Posts
Discussion Starter · #23 ·
It appears that I may have made a mistake going with the 250gb 970 evo as it appears to be too slow to be of use as a cache when used on a duplication pool due to the 2x W/R penalty. I will test a 1TB 960 EVO later to see how that goes.

Alternatively I could get a second 250gb 970 EVO and use it in my other m.2 slot if I also by a pcie1x gpu. Although that slot is only PCIE 2.0 x4 20Gb/s vs 32Gb/s but maybe that will not matter for the 250gb model.
 

· Registered
Joined
·
623 Posts
Discussion Starter · #27 · (Edited)
Remembered that I had an M500 lying around, I am thinking of replacing the HP 120GB ssd that the OS is on with that as it will be a lot faster. (I got the HP ssd for free with the motherboard, I might just take that and use it as the drive for a PfSense router)

Also, I am still waiting for the SSD to come in so I can install and setup the secondary write cache.

Lastly, here is a tip for installing a drive into a M.2 slot. As you may notice it is a PITA to get the dam screw in without it being launched from the M.2 catapult. To solve this simply place a 9v battery on the M.2 drive to weigh it down to make the install a lot easier.
 

· Registered
Joined
·
623 Posts
Discussion Starter · #29 ·
Decided to get them. For those interested these are the cables I was talking about: https://www.amazon.com/gp/product/B01IBA3ITK/ref=oh_aui_detailpage_o00_s01?ie=UTF8&psc=1

Now while I do not "need" them for that price it is worth it just to reduce the cable clutter caused by the 8 full size sata cables. You will notice they are basically the same as the cables from the HBA: https://www.newegg.com/Product/Product.aspx?item=N82E16812117625

I also picked up an ATX extension cable as discussed earlier in the thread. Figured if I was already going to be taking the left drive tower out might as well replace that wire as well so that I am done in that area for good.
 

· The Physical Manifestation of Typos
Joined
·
6,960 Posts
That's a decent price on those cables :thumb:
 

· Registered
Joined
·
623 Posts
Discussion Starter · #33 · (Edited)
I will get some pics up this weekend. Tomorrow I should be getting the cables installed and I can then finally put the right side panel on. With that last step the build will be physically complete and I can start to finalize the fan speed configuration.

I will preform about 10TB in writes per year on the SSDs just from running a daily 15GB backup of the OS. So that combined with normal traffic it looks like I will get between 10-15 years out of the SSD cache before I need to replace it. So you figure an annual cost of $10-16/year to have the SSD accelerated cache. But hey maybe in 10 years I could get 16GB ECC ram sticks on the cheep and replace the SSDs with that... well you can only dream. I would use a ram cache instead if I could store at least 32GB in it but today it would cost over $500 and that would be for 4x 8GB sticks so I would only have 24GB cache leaving 8GB for everything else.
 

· Registered
Joined
·
623 Posts
Discussion Starter · #34 ·
One thing of note in this build is the use of an Aquaero. I am using an AQ6 to control the fans and provide other remote monitoring abilities of the server. There is some limitations such as only being able to work with 3 inputs per virtual temp sensor and having a max of 4 of them. However by using only the temps of the drives in the left side it makes things easier. Logically the drives on the left side will always be hotter than the drives on the right side. By using the AQ I have been able to keep the fans runnign quiet and have them ramp up to a higher speed during heavy disk activity when additional cooling is required. I found the heat sink fans for the HBAs to be a bit loud when running at max speed so I limited them so that when running full tilt they will not run at their actual maximums. I found that even with this limitation I can easily keep them below 110F. I use the temperature probes from the AQ6 to monitor the HBA temperatures as there is no other way to get that information.

You may also notice some wiring labels that I use to make identifying what wires go to what. It makes things a lot easier to trace and debug if you ever have an issue. I have done the same for all my sata power and data lines so I know exactly what is connected to what. You can see the mapping for this in the excel sheet I posted earlier in the thread.
 

Attachments

· Registered
Joined
·
623 Posts
Discussion Starter · #35 ·
Photos from when it was time to move the RAID 6 over to the new system. The raid array was deleted and is now a sort of RAID 10 like system using the pods concept shown earlier in the thread. Here you can see the old mounting that those drives were in. You can also see the rats nest which was the old servers wiring.
 

Attachments

· Registered
Joined
·
623 Posts
Discussion Starter · #36 ·
Getting the heatsink onto the M.2 drive should not normally be an issue however the 0.5mm pad basically denigrated. Removing the plastic revealed another pice of plastic under it and the pad basically split in half. If you look at the image below you can see how half the pad is just gone after removing the plastic protector...
 

Attachments

· Registered
Joined
·
623 Posts
Discussion Starter · #37 ·
Luckily, I do not need to cool anything on the backside so I used what I had of the pad and mounted the heatsink and installed it. When using a heatsink on your m.2 drive you do not need to use the 9v battery trick to hold it down as the weight of the heatsink will do that for you.
 

Attachments

· Registered
Joined
·
623 Posts
Discussion Starter · #39 ·
Here you can see the new 24 pin extension installed. I do not have a good photo of the 24pin 90degree adapter but you can get the idea of how much space it took up in the second photo taken from the back side. It is pretty clear how much better using an extension here is than the right angle adapter.
 

Attachments

· Registered
Joined
·
623 Posts
Discussion Starter · #40 ·
Now onto the last objective which was to replace all those normal SATA data cables with nice low profile ones like the ones the HBA uses. Notice here how I label the cables so I know what is going where. I am pretty happy with how relatively clean the wiring ended up. Especially when you compare it to the mess of not only my old system but also compared to my earlier photos of how the wiring looked from page 2.

(Also yes I discovered when I plugged in the computer I forgot to plug in the power to the SSD even though I rechecked the connection on all the HDDs like 5 times. lol)
 

Attachments

21 - 40 of 62 Posts
Top