[zeroibis]

Wanted to update that one of the HDD trays was defective as you can see in the pic.

I created an RMA and Corsair is sending a replacement.

 

[zeroibis]

Fixed some typos and updated the original post on the thread to add some more explanation about the build.

 

[zeroibis]

So I had a raid vanish from storage spaces due to some strange errors in windows. A simple reboot caused the storage spaces to reload them without issue. Drive pool then went to quickly verify and ensure everything was duplicated properly.

Seams to be an issue with parts of the old ATI gpu drivers running and crashing randomly. I disabled the service and am looking for additional problems.

sfc /scannow returned corruption which was fixed so hopefully that is the end of it.

I must say it is quite a bit easier working with software implementation vs hardware because it is much easier to follow logs of exactly what the system is upset about and much easier to try to identify the original source of the domino effect that leads to a problem.

 

[zeroibis]

Yesterday I copied some VMs over to the server and was excited to see transfer rates exceeding well over 500MB/s. Looks like that cache is continues to do its job well.

I am thinking that the origin of my recent HDD problem may have been due to using up all the CPU when my backup is running due to the heavy compression I have turned on. I will attempt to resolve this first by applying an OC to the CPU to see if that can add enough performance to avoid the problem. I could also just decrease the compression so it is not at maximum but I expect to be able to OC this CPU to 3.9-4GHz without much trouble.

 

[zeroibis]

Got the replacement tray in now so that is nice.

Also I have OCed the CPU to 3.8 GHz. I could run it higher but I turned the clocks down for ensured long term stability while keeping the voltage up at 1.35v using the stock cooler the CPU stays in the low 40s and under load below 75C in OCCT but in the mid 50s under real world full load conditions (VM compression which occurs daily).

System has been rock solid since the OC and I no longer have any crashes or drives that fall out during the daily backup/compression run.


Lastly, I should mention that I have gotten my new router setup as well. Thanks to the 10Gbe chip in the Darkness build I was able to free up my quad 1Gbe nic card for use in a PFsense router. Decided to call the build Wiz because who else are you going to trust for barrier defense. If it is good enough for the demon king it is good enough for my router.

At this point some are likely wondering if the next build I make will be Aqua, problem is could you ever trust an Aqua build to actually do any work...

 

[Simmons572]

Look forward to seeing the future builds mate :thumb:

 

[zeroibis]

Thanks to the holiday sales I have picked up two more 4TB drives (Deskstars) to add to the system.


I already have them installed and will be running tests on them for the next few weeks to validate them before putting them to use.



I will likely make them into a RAID 0 drive and them move over the data from the existing 4TB duplication pool and then convert those two drives into a RAID 0 and then merge the resulting drives to have a pool of 2 8TB raid 0 arrays. Given the 4TB drives are a lot faster than the old 2TB drives (due mainly to disk density improvements) I will move my slower access files over to the 8TB pool and move over the high speed access data over to the new 8tb mirror of raid 0 drives.

 

[reezin14]

So today I have been testing the different methods for using a write based cache. After testing multiple different methods and settings I have concluded the tech is just not there yet. First let me start off by what I was looking for:

A write cache that would act like a high speed buffer where data would go there first and then once the buffer fills it will overflow to the HDDs. Once the transfer finishes the buffer will clear out.

I know this is late or to late maybe but someone may be able to use the info. I was looking for the same function's in my server build too. Didn't want to go the Freenas ZIL or the unRAID route $$$. After much research I found Stablebit drive pool to fit the bill.

The closest thing to that is the way the write cache for PrimoCache works. Unfortunately, becuase of the way that DrivePool works I need to enable PrimoCache on the individual lower level volumes/drives. What this means is that a single write operation to the Duplication Pool becomes two W/R operations for the cache. Thus the performance of the cache is significantly crippled. As a result it is only somewhat worth the effort when acting as a cache for a simple pool and totally useless if your going to put it before your RAID 0 array that is within a pool. Now obviously a RAM based write cache gets around these issues but with 8GB ECC ram still costing $125 a stick I am going to stay away from that for now.

The SSD Optimizer plugin works 100 times better and replaces PrimoCache, which IMHO sucked. It is suggested(Drive Pool cache) that you run the same number cache drives for the number of dup's that you'll have on the pool/s. 2 dup's = 2 cache drives etc,etc... With raid 0 that might be different I don't know as of now.

I also tried the write cache method for DrivePool and it was very fast as my writes all went directly to the SSD however if you transfer more than the volume size of the ssd your transfer fails so that is not an option. Also once it does fill up it is very slow to empty back out and it does not even push to your underline drives in parallel but instead dumping to the first and cascading from there. Thus the usage is very limited to home users messing around but your not going to deal with that nonsense in a production environment.

I can see where transferring 10TB's of Data all at once wouldn't have suited your situation. But will you be doing this regularly? I broke my transfer of data into chunks 400gb's or less when transferring 3TB's+ of data. I know it's probably not as fast as all at once but... I use a 500gb drive for the cache because I don't think for now I'll be transferring more than that at one time. Plus the prices of 1TB ssd's are very reasonable now so. As far as filling up your volume(while transferring data). I think that if you had your balancing setting set to balance immediately that would have alleviated some of the congestion, although I never tried this myself. Parallel placement of files can be dictated in the file placement and Drive usage limiter tabs. The Disk Space equalizer plugin will distribute data evenly across drives in your pool(percentage or free space wise). This what I do and each drive is filled equally as far as unduplicated data, duplicated data I have going to specific drives. Which is only about 30gb's.

So write caching was worth a shot but in the end I am not seeing anything that is worth it as far as SSD write cache goes. RAM cache as always is there but is an expensive proposition that comes with its own risks.

I don't know how long ago you tried DrivePool out.But if you revisit the tech I sure you'll have a different outlook. I didn't use any storage space's pools when setting up my drive's, just straight DP software. BTW NICE BUILD.

 

[zeroibis]

I know this is late or to late maybe but someone may be able to use the info. I was looking for the same function's in my server build too. Didn't want to go the Freenas ZIL or the unRAID route $$$. After much research I found Stablebit drive pool to fit the bill.



The SSD Optimizer plugin works 100 times better and replaces PrimoCache, which IMHO sucked. It is suggested(Drive Pool cache) that you run the same number cache drives for the number of dup's that you'll have on the pool/s. 2 dup's = 2 cache drives etc,etc... With raid 0 that might be different I don't know as of now.



I can see where transferring 10TB's of Data all at once wouldn't have suited your situation. But will you be doing this regularly? I broke my transfer of data into chunks 400gb's or less when transferring 3TB's+ of data. I know it's probably not as fast as all at once but... I use a 500gb drive for the cache because I don't think for now I'll be transferring more than that at one time. Plus the prices of 1TB ssd's are very reasonable now so. As far as filling up your volume(while transferring data). I think that if you had your balancing setting set to balance immediately that would have alleviated some of the congestion, although I never tried this myself. Parallel placement of files can be dictated in the file placement and Drive usage limiter tabs. The Disk Space equalizer plugin will distribute data evenly across drives in your pool(percentage or free space wise). This what I do and each drive is filled equally as far as unduplicated data, duplicated data I have going to specific drives. Which is only about 30gb's.



I don't know how long ago you tried DrivePool out.But if you revisit the tech I sure you'll have a different outlook. I didn't use any storage space's pools when setting up my drive's, just straight DP software. BTW NICE BUILD.

I think you may have missed page 3. Those quotes were from when I was testing with a single cache drive.


Once I had a second drive transfer times no longer suffered the 2x W/R penalty.



The limits of SSD Optimizer are equal to the volume size of the cache which is limited by the physical disk size. You also still need two physical disks if your going to use real time duplication. Given that my cache size is 256GB and I daily preform copies of TBs worth of data this cache is not large enough to use with the SSD Optimizer. In my setup your 400GB chunks are way to bing to even be moved with the SSD Optimizer as the max I could move would be functionally about 200gb at a time.



So to recap, I have been using drivepool and have very much enjoyed it. I am also running primocache and it is working without issue. You can see more info on the setup for it on page one under: --The END of HW based RAID--

 

[reezin14]

I think you may have missed page 3. Those quotes were from when I was testing with a single cache drive.


Once I had a second drive transfer times no longer suffered the 2x W/R penalty.

I understood,and didn't see that you tested with x2 drives.But that's why I recommended x2 cache drives.


The limits of SSD Optimizer are equal to the volume size of the cache which is limited by the physical disk size. You also still need two physical disks if your going to use real time duplication. Given that my cache size is 256GB and I daily preform copies of TBs worth of data this cache is not large enough to use with the SSD Optimizer. In my setup your 400GB chunks are way to bing to even be moved with the SSD Optimizer as the max I could move would be functionally about 200gb at a time.

Again why I recommended x2 chache drives, and the statement breaking down your data movement was suggested because your cache drive/s where so small. I was just showing that moving smaller chunks of data might be feasible/better for you in that scenario. I was also suggesting that if the drives are set to balancing immediately that you may be able to transfer more than 200gb's of data. Being that the data is moved to the archive drive/s right away,so it could be possible to keep a steady stream of data going to the cache drives. As they are dumping data to the archive drives. I also stated that I never tested this so... There also was the suggestion of getting 1TB or bigger ssd's to alleviate this, seeing how the prices are reasonable now days.

So to recap, I have been using drivepool and have very much enjoyed it. I am also running primocache and it is working without issue. You can see more info on the setup for it on page one under: --The END of HW based RAID--

All in all happy you got it worked out. Quote is messing up sorry.

 

[zeroibis]

All in all happy you got it worked out. Quote is messing up sorry.

Thanks!


What is also nice about primocache is that is it a very flexible solution as well.



Now I just pray for the day that stable bit scanner offers better support for soft raid so I do not need to use a different program to pull SMART data from drives that are in a software raid.

 

[reezin14]

Thanks!


Now I just pray for the day that stable bit scanner offers better support for soft raid so I do not need to use a different program to pull SMART data from drives that are in a software raid.

Hmmm, I didn't know that! Thanks for the heads-up.

 

[zeroibis]

So today I have the results of a great exprement. What sort of performace change do you get going from 970 EVO to the 970 EVO Plus well what looks like a minor revision is a major change!


Same transfer and settings as here: https://www.overclock.net/forum/attachment.php?attachmentid=225308&d=1539954682


Time reduced from 9:13 to 7:27



So expect a 20% performance boost. Note that in the new test you can see that I use the exact same test folder with the same mixed content as before.



In conclusion those considering investing in higher end SSD drives and even the pro there is a lot of performance to be gained using top of the line drives as a cache.

 

[zeroibis]

Adding another 20TB but I plan to actually remove the 8x 2tb drives to reduce power and heat by replacing them all with just 2x 10tb drives. The 2tb drives have lived a long life but it is time for them to retire. I can use them as back up drives for 16tb of archive data so that is nice.



Over the next few months I will be testing the 10tb drives and getting ready for the migration. Also I will update this thread with my first shucking and 3.3v bypass for the new HDDS.

 

[zeroibis]

So I managed to get the 10TB drives out of the external case pretty easily. I ran a full surface scan before I did so and now I am doing a long run reformat on the two drives now installed internally. These things generate basically no heat compared to my pack of 7200rpm 2tb drives they will ultimately replace. Also very impressed with the ~250MB/s R/W sequential performance. For a drive storing archive data and other large files this is just great! In fact due to my ssd cache the write speeds do not even really matter and 250MB/s is great read speed for what I need.



Now to wait another 16 hours or so for the formatting to finish.

 

[zeroibis]

So have started migrating data to the new drives but after a day I checked on it and noticed how slow it was going. Then I remembered that by default drive pool does its stuff with a low priority so it does not effect performance of the drive. However, just like a raid rebuild it will maintain that same usage threshold regardless of if the drive is idle or not so that it is always in a state ready for load.



As I want it to finish faster I set it to high priority and now it is transferring at full speed just like I want.



Wish I would have remembered the full speed button a day ago...

 

[zeroibis]

Wanted to update that about 1 month ago I removed all the 2tb drives from the system and am now only on the two 10tb drives that replaced the 8 2tb drives.


The shucked 10tb drives have been working well and generating way less heat and noise.


I also took the time to move some of the drives around as there is a lot of freed up space now.

 

[leexgx]

I

thread.

are you sure that ECC is working (unsure if you monitor this thread as its tad old)

as your ECC should be 64 in first box and 72 in second box (if its 64 128 that should be no ECC as the second number is just a multiple of 64 its missing the extra 8 bits)

//ECC Memory DataWidth TotalWidth 64 72

//Non-ECC Memory DataWidth TotalWidth 64 64

quite sure 64 128 and ecc 6 is an error

ECC should be single bit > 5 as ecc udimms only support single bit error corrections not muti
multibit correction > 6 is only possible on Chip kill type buffered/Registered memory as they have double stacked ram chips so 18 chips per side + the buffer/Register chip in the middle

one way to find out is boot into linux and run the commands to check for actual ecc enabled (best just googling it)

or just overclock the ram a bit until it actually starts showing event viewer errors in Kernel-WHEA in windows when running a memtest from inside windows (Ryzen DRAM Calculator has a built mem test) if software memtest shows errors ECC is not working as ECC would have corrected them or halted the computer on a uncorrectable ECC error

just ordered some 2666 ECC ram 2x16gb for £100 (MTA18ADF2G72AZ-2G6) see what happens

my wmic MEMORYCHIP get DataWidth,TotalWidth report returns correct non ECC at the moment (dual channel so 2 rows of 64 and a non ecc > 3 on the MemoryErrorCorrection type)
64 64
64 64

if it works it should change
64 72
64 72
ecc type 5

will report back in 3-5 days once i get them (say hello from 12-31-1969 at 11:00 PM as that's the last time it said i was here lol, can't believe it's 12 years have passed from when i last did [email protected] heating the house up with 6x9800GX2 in 2 systems 2x 1200w PSUs just shy of 1KW been pulled from the wall each)

 

[zeroibis]

are you sure that ECC is working (unsure if you monitor this thread as its tad old)

as your ECC should be 64 in first box and 72 in second box (if its 64 128 that should be no ECC as the second number is just a multiple of 64 its missing the extra 8 bits)

//ECC Memory DataWidth TotalWidth 64 72

//Non-ECC Memory DataWidth TotalWidth 64 64

quite sure 64 128 and ecc 6 is an error

ECC should be single bit > 5 as ecc udimms only support single bit error corrections not muti
multibit correction > 6 is only possible on Chip kill type buffered/Registered memory as they have double stacked ram chips so 18 chips per side + the buffer/Register chip in the middle

one way to find out is boot into linux and run the commands to check for actual ecc enabled (best just googling it)

or just overclock the ram a bit until it actually starts showing event viewer errors in Kernel-WHEA in windows when running a memtest from inside windows (Ryzen DRAM Calculator has a built mem test) if software memtest shows errors ECC is not working as ECC would have corrected them or halted the computer on a uncorrectable ECC error

just ordered some 2666 ECC ram 2x16gb for £100 (MTA18ADF2G72AZ-2G6) see what happens

my wmic MEMORYCHIP get DataWidth,TotalWidth report returns correct non ECC at the moment (dual channel so 2 rows of 64 and a non ecc > 3 on the MemoryErrorCorrection type)
64 64
64 64

if it works it should change
64 72
64 72
ecc type 5

will report back in 3-5 days once i get them (say hello from 12-31-1969 at 11:00 PM as that's the last time it said i was here lol, can't believe it's 12 years have passed from when i last did [email protected] heating the house up with 6x9800GX2 in 2 systems 2x 1200w PSUs just shy of 1KW been pulled from the wall each)

Yes you should see in this post here: https://www.overclock.net/forum/221...ess-corsair-750d-build-48tb.html#post27646566


Note I am running windows, the results that I show in my images are the expected results if you are running ECC ram and it is fully functional in windows.


Note that memoryerrorcorrection is considered the final response, the output of memoryerrorcorrection is the authority of if windows is running under multi bit ECC, single bit ECC or not. Note that although total width is often 72 the expected value for ECC detection is simply >DataWidth so if DataWidth<TotalWidth ECC is present.

 

[zeroibis]

Ordered up 2 more 10TB drives bringing the total storage space to 72TB now. They are the same as the 10TB drives I added last year that have been running great.