Overclock.net banner

1 - 20 of 60 Posts

·
Registered
Joined
·
602 Posts
Discussion Starter #1 (Edited)
I mean who better than to keep my important data safe than Darkness?

Is there anything she is not willing to take for fun, I mean to protect her friends.



Also, I mean who else are you going to trust all your information to?




Yes for those wondering this is yet another character from konosuba as I felt it would be fun to have someone to go with Megumin and if we already got a system for the best girl we might as well have one for the second best as well.

Unlike my previous build I will not be going out of my way to make the actual build physically themed for Darkeness. It is just a server but at least I can have some fun with my Windows profile.

Onto the build!!

Specs:
Case: 750D Airflow​
CPU: RYZEN 3 1200​
M/B: X470 Taichi Ultimate​
RAM: M391A1G43EB1-CPBQ (8GB ECC)​
PSU: SSR-1000TR​
HBA Card: 2x AOC-SAS2LP-MV8​
HBA HT: 2x SLF-1 Ultra​
SSD Cache: 2x 970 EVO M.2 2280 250GB (One on each of the separate underline drives/arrays in a duplication pool)​
GPU: ZOTAC ZT-71304-20L (PCIe x1)​
HDDs:
4x 10TB WD White Label (removed from elements external case)
2x 8TB Deskstars
2x 4TB Toshiba
2x 4TB Deskstars


All the drives are ones I already have. I am moving them from an existing computer over to the new one.

--The END of HW based RAID--
One of the big changes in this build from my old one is removing my HW based RAID 6 array and replacing it with a software based solution similar to RAID 10.

I will be using DrivePool and storage spaces to accomplish this. The 2tb drives will be split into two pools which are then pooled together. This is so that backups can be done on the underline 4tb pools so that in the event of data loss only the section containing data loss will require recovery instead of the entire resultant large pool. As you can see raid 0 is used in pairs of the 2tb drives for increased sequential performance. This same method will be applied to the 4tb drives in the future as more are added.

For instructions on how to setup storage spaces to use RAID 0 see: Darkness, a Corsair 750D build (48TB)

[2TB -- RAID 0 -- 2TB]_[2TB -- RAID 0 -- 2TB]
----MIRROR POOL-----_----->Unified Pool of Pools<-----_----MIRROR POOL-----
[2TB -- RAID 0 -- 2TB]_[2TB -- RAID 0 -- 2TB]




The wiring for the HDDs is very deliberate to ensure that drives that are mirrors are never on the same power line or sata controller. Drives that are in the same raid 0 array share the same controller and power line. This is to prevent something like an HBA failure resulting in data not being accessible.

You will see that I have also used primocache with two 970 EVOs as a cache. This is to improve write performance to the system. I had to use two of them as drivepool with real time duplication enabled has a 2x read/write penalty to the cache as it is only a single target. So if you use the same cache for both drives/pools that are being duplicated you will get 2 of everything dumped to it. Thus you need a separate cache for each part. You can see how the cache is split up in the image above.

---ECC RAM---
One of the reasons I went with Ryzen for this build was that I would be able to use ECC ram. Now while ECC ram is not a requirement it is a clear choice to help avoid possible data corruption on a file server. The problem is that I still at the time of my build had not found ANY documented case of someone using ECC ram with Windows and having it work. So although I knew there was ECC ram listed on the QVL for my motherboard I was taking a risk ordering it to find out once and for all if ECC ram does fully work today on windows. As you can see from the image below it does and you can read more about it here: Darkness, a Corsair 750D build (48TB)


--10gbe--
The reason I chose this motherboard (in addition to the 8 sata ports) was the support for 10gbe. My main rig already has support for 5gbe and I wanted to ensure that this build was future proof. I already have a switch that supports 1/2.5/5/10gbe so there was nothing holding me back. The read cache helps me take advantage of some of the benefits but over time as I add more RAID 0 drives and performance improves the 10gb connection will give me plenty of headroom.

--acronis true image--
I managed to get 3 copies of this for free during an amazing newegg special! (after rebate) I am using the program to create daily backups of my server and other computers. The backups of the server drive are stored only on non RAID 0 volumes as in the event of a disaster I need easy access to the files and pulling them from a simple mirror between two drives will be much easier as I would only need to pull a single drive from the system and grab the files. (or I could just download the file from backblaze in a short time). Thanks to compression the backups of the OS are around 13GB and I can even download that for a restore from backblaze in a few min. This makes recovering from most major disasters pretty easy. In the event of a motherboard failure I suppose I would need to rebuild my storage spaces if the new motherboard is very different but honestly I am not sure how easy the recovery process is for those.

This build is highly inspired by the 750D build that fg2chase did here: Ok... So who needs a 50-66TB server??
He has been a big help in planning this build and if you have not please check out his thread.
 

·
Registered
Joined
·
602 Posts
Discussion Starter #4
Here is with the drive cage cut off. I found that if I took the excess metal parts from where I cut and wiggled them back and forth a lot I could get them to break off along a crease line this is how I got it cut off so close using just a hacksaw and some pliers. Why no dremel? Well I could not find it but I did find my hacksaw...
 

Attachments

·
Registered
Joined
·
602 Posts
Discussion Starter #5 (Edited)
Now onto the actual computer part!

Using a RYZEN 3 1200 as this is just a file server so no need for anything faster. I just need ECC support which Ryzen offers and there is ECC ram listed on the QVL for my X470 Taichi Ultimate. Although the 10Gbps card in this will way outclass my 5Gbps that Megumin supports I am planning for the future here. I do have a 10Gb switch so I am already good there.

Some might wonder why not go with one of the Ryzen g series so that I do not need a GPU? Well the issue is that the built in gpu eats an entire 8x slot. By using an external GPU I can run it in 4x or 1x mode so that I can save slots for the HBA cards. I plan to just have 2 HBA cards and use the motherboard for the other 8; however, I may add a third card in the future if I start to use a lot of SSD drives.
 

Attachments

·
Registered
Joined
·
602 Posts
Discussion Starter #6 (Edited)
So a big question many have asked since Ryzen released is about ECC support under windows. We have all read that it works and detects under Linux; however, no one that I have found has posted results from Windows tests since Ryzens release. As a result many have avoided running ECC in Ryzen under windows thinking that it will not fully work. Today I can say that from my tests Ryzen ECC is 100% working under windows. I really wish reviewers would have revisited this a few months after Ryzen released as I am sure this was something that MS added support for later on. You can see from the images below that ECC is indeed working on Ryzen and even on the lowest end Ryzen chip.

Note the possible values for memoryerrorcorrection:
0 (0x0) Reserved
1 (0x1) Other
2 (0x2) Unknown
3 (0x3) None
4 (0x4) Parity
5 (0x5) Single-bit ECC
6 (0x6) Multi-bit ECC
7 (0x7) CRC
 

Attachments

·
Registered
Joined
·
602 Posts
Discussion Starter #7
After I first tested and verified that the HBA card was working it was time to install an upgraded heatsink. I used the SLF-1 Ultra which was a great fit. Also note that it does mount very tight, you really need to press those pins to get them in the first time. (you can see a photo of one of the pins on the back side which has a good mount.


You can see the process below:
 

Attachments

·
Typo Elemental
Joined
·
3,223 Posts
Subbed :D
 

·
Registered
Joined
·
602 Posts
Discussion Starter #9
So I see now that the second line of HDD cages need to be moved forward in the case for them to all fit with space for the 24pin connector. What I am doing is moving each subsequent cage forward a few mm by mounting them with standoffs to create spacing. This results in the last two having the required space to fit the HDDs and clear the 24pin connector. The consequence of this is that the case side panel can not fit on anymore. The solution that fg2chase used was to cut the eds of the HDD trays off to free up space. I am exploring a different approach. I do not need or care about having a window so why not abandon it. I can remove the window from the side panel and replace it with a simple piece of cardboard. I can then create some extra space in the area where the HDD cages stick out. If I really cared how it looks I suppose I could make a nice paper mache bump in that area and have everything look pretty nice but I guess I will see how the weather is later this week/weekend. I will likely just make the entire thing out of cardboard and spray paint it. Like I have said earlier in the thread the main directive here is function and if some cardboard gets the job done great.
 

·
Registered
Joined
·
602 Posts
Discussion Starter #10
Here you can see all the cadges mounted in. On the left side you can see that each cadge sticks out a few mm from the one below it.

From what I can see there is one drive slot who's mounting position is made difficult due to the position of the 24pin connector blocking easy access to its rear ports. It could be that this drive slot will need to use extensions or something so it can be wired before insertion. I will be taking as many spare hdds as I have and using them to test out the mounting before I move the real drives over. As the towers of drives are only supported at the bottom it is possible to shake them around a bit although I would imagine that the weight from the HDDs would cause them to become pretty solid once occupied. I do however plan to use some large sound proofing foam blocks that I have at the top of the cadges which will wedge them in place. This should help keep them sturdy and hopefully minimize any vibrations.
 

Attachments

·
Registered
Joined
·
602 Posts
Discussion Starter #11
So yesterday I checked the wiring and I am not finding a good way to deal with the two slots that get blocked by the 24pin. I could fit a 90d sata data connection on those drives but then for the one in the bottom of the top tray it hits the bottom and thus can not be pushed back in...

I already am using 1 of my sata ports for an SSD that is the boot drive and I planned to keep a second line open as a spare so I suppose I could just do it that way.

For anyone doing this build from scratch I would recommend making the holes for the wires behind the second tier of drives larger so that you can more easily place your sata power cables. The connections from my sata power splitters are fine but the oem connections are very tight and difficult to get connected to the drive.

I got another HBA card coming in today along with an exhaust fan. On Saturday the wiring will come in so hopefully on Sunday I will be up and running and then I can start testing HDDs and begin the migration.

Lastly I am going to look again at how the side panel fits if I just accept losing two mounting positions from the 24pin cable. I might be able to use the stock side panel after all. Also nothing stops me from going back and doing some mods to gain those two spaces back in the future.
 

·
Typo Elemental
Joined
·
3,223 Posts

·
Registered
Joined
·
602 Posts
Discussion Starter #13
Have you considered getting a low profile 24 pin extension to hopefully give you more space?

Either something like this: https://www.amazon.com/24Pin-female-adapter-desktop-supply/dp/B07BJ2QHM4
Or like this: http://www.performance-pcs.com/ppcs-low-profile-extension-cable-series-24pin-atx.html
Yes I tried to use one of those and got lucky it did not destroy my motherboard. I tested it afterword and found that over half of the pins did not work! I am currently waiting for a credit back from amazon once they process the return. After that I am a bit afraid about the quality of those things.

I am using a right angle adapter that takes up the same space as that low profile extension. What is nice about that extension though is that I could probably pass the wires though the 24pin wires to get it to fit then. I would likely attempt that in the future if I end up really needing those 2 extra slots.
 

·
Registered
Joined
·
602 Posts
Discussion Starter #14
I might be still a few days off from when I can actually start to configure the server but I have been planning my setup for quite some time. I wanted to give you all a bit more insight into how my drives will be configured in this system.

This configuration will be used for my 2TB drives that are currently in a RAID 6 array. Note that the NVME cache part will be implemented after I pickup a 970EVO that I hope to get a good black Friday sale on.

For those interested in more detail you can see a thread here where I started planning this out: https://community.covecube.com/index.php?/topic/3780-planning-for-drivepool/
 

Attachments

·
Registered
Joined
·
602 Posts
Discussion Starter #16 (Edited)
Update with some pics!

Very close to finishing now. The dust filters have come in and are installed. The second HBA is installed and all the wiring is installed. I have also installed the Aquaero but it will not be fully configured until all the hard drives are installed. You can see from the wiring that I am using it to also control the two HBA heatsink fans and I have temp probes going to them so I can spin those fans down a bit and easily maintain temps below 100F.

It took a few hours of tape but I finished the side panel. Yes it is made out of electrical tape and cardboard. Honestly, I do not think it looks that great in the pics but in person it actually has a nice low key look to it and I am happy how it turned out. I am hopeful that I will be able to place the other side panel on once all the drives are actually wired. Worst case I will make a side panel for that side as well.

Right now I am getting the last of the validation tests done before I get the 970evo and setup the L2 SSD cache. Once that is done I will start fully migrating data over from my existing build. Once I am able to clear everything off the RAID 6 array it will be removed and the 2tb drives will be moved over.

You can also see an example of my drive map chart for this build. One of the things that is nice about stable bit drive scanner is that it allows me to rename the drives in the program and label where they are located so they are easy to locate if there is any problem. However, I still like having a master excel sheet which I also use to help plan the wiring in advance as well.

You will notice that I have it set up so that drives or groups of drives that are mirrors of each other are placed parallel to each other. I have split the power and data lines to ensure that all the drives in a given duplication set are never all powered by the same line or connected to the same sata source or using the same wire. I also tried to make it so that drives in a raid 0 did share the same power source data cable and sata controller whenever possible. In this way a single cable failure or controller failure will not result in a loss of data access. Only multiple failures could result in an actual event where access to data would be cut off.
 

Attachments

·
Registered
Joined
·
602 Posts
Discussion Starter #17 (Edited)
Well it looks like drivepool does not support being stacked on top of raid 0. So I will need to use something else to do the same thing. I wish that they would have pointed that out in the thread I made asking about such limitations months ago...

At least the stablebit scanner works well.

EDIT: It appears that it will work if I use storage spaces to create the raid 0 array instead. Will try that out tomorrow.
 

·
Registered
Joined
·
602 Posts
Discussion Starter #18 (Edited)
Ok so I got a test run working but note that if your going to use Storage Spaces to create a RAID 0 you need to do the following:


1) Create a storage pool in the GUI but hit cancel when it asks to create a storage space
2) Rename the pool to something to identify this raid set.
3) Run the following command in PowerShell (run with admin power) editing as needed:
New-VirtualDisk -FriendlyName VirtualDriveName -StoragePoolFriendlyName NameOfPoolToUse -NumberOfColumns 2 -ResiliencySettingName simple -UseMaximumSize
 

·
Registered
Joined
·
602 Posts
Discussion Starter #19
So today I have been testing the different methods for using a write based cache. After testing multiple different methods and settings I have concluded the tech is just not there yet. First let me start off by what I was looking for:

A write cache that would act like a high speed buffer where data would go there first and then once the buffer fills it will overflow to the HDDs. Once the transfer finishes the buffer will clear out.

The closest thing to that is the way the write cache for PrimoCache works. Unfortunately, becuase of the way that DrivePool works I need to enable PrimoCache on the individual lower level volumes/drives. What this means is that a single write operation to the Duplication Pool becomes two W/R operations for the cache. Thus the performance of the cache is significantly crippled. As a result it is only somewhat worth the effort when acting as a cache for a simple pool and totally useless if your going to put it before your RAID 0 array that is within a pool. Now obviously a RAM based write cache gets around these issues but with 8GB ECC ram still costing $125 a stick I am going to stay away from that for now.

I also tried the write cache method for DrivePool and it was very fast as my writes all went directly to the SSD however if you transfer more than the volume size of the ssd your transfer fails so that is not an option. Also once it does fill up it is very slow to empty back out and it does not even push to your underline drives in parallel but instead dumping to the first and cascading from there. Thus the usage is very limited to home users messing around but your not going to deal with that nonsense in a production environment.

So write caching was worth a shot but in the end I am not seeing anything that is worth it as far as SSD write cache goes. RAM cache as always is there but is an expensive proposition that comes with its own risks.
 

·
Registered
Joined
·
602 Posts
Discussion Starter #20
PSA/Note to Self:

Pause Windows Updates whenever you start working on something important. Last night WinMerge which had been running for over 16 hours finally finished the directory compare so I could validate that I had my most critical files moved without issue. Then shortly thereafter windows rebooted to install updates and thus deleted the results when it closed WinMerge...

Or alternatively never sleep and sit next to the computer 24/7 so you can stop the reboot lol.
 
1 - 20 of 60 Posts
Top