Complicated: Low performance in RAID0 with 10Gb NIC - Overclock.net - An Overclocking Community
Forum Jump: 

Complicated: Low performance in RAID0 with 10Gb NIC

 
Thread Tools
post #1 of 8 (permalink) Old 09-24-2018, 04:14 PM - Thread Starter
Linux Lobbyist
 
elrompeplacas's Avatar
 
Join Date: Aug 2014
Posts: 78
Rep: 1 (Unique: 1)
Complicated: Low performance in RAID0 with 10Gb NIC

Hello,

I have two computers with a RAID0 in each one.

PC-1
HDD: Seagate Exos x10 hard drives in software RAID0. (10TBx4)
NIC: Asus XG-C100C 10Gbps ethernet card.
CPU: i7 4790K with Z97 chipset and 8GB of RAM.
SO: Windows 10 x64 Pro

PC-2
HDD: Seagate Ironwolf hard drives in software RAID0. (8TBx6)
NIC: Asus XG-C100C 10Gbps ethernet card.
CPU: i3 4160 with Z97 chipset and 8GB of RAM.
SO: Windows 10 x64 Pro

When I run the cristaldiskmark test I get good values:

PC1


PC2


But when I try to copy a file from PC-1 to PC-2 (or vice-versa), the copy speeds are much lower than the theoretical ones.

* PC-1 to PC-2: copy one file of 35GB I get a 530MB/s
* PC-2 to PC-1: copy one file of 35GB I get a 600MB/s

If I use RAMdisk software I get this speeds:

PC1


PC2


And when I try to copy a file I get the right speed:

* PC-1 to PC-2: copy one file of 5GB I get a 1.000MB/s
* PC-2 to PC-1: copy one file of 5GB I get a 1.000MB/s

Does anyone know why this happens? I do not understand why this happens!
I would like to get 1GB/s copy speeds from my RAID0 volumes between PC-1 and PC-2. Is this possible?

Thank you very much!

Last edited by elrompeplacas; 09-24-2018 at 04:24 PM.
elrompeplacas is offline  
Sponsored Links
Advertisement
 
post #2 of 8 (permalink) Old 09-24-2018, 04:27 PM
9 Cans of Ravioli
 
The Pook's Avatar
 
Join Date: May 2006
Posts: 17,733
Rep: 1267 (Unique: 689)
If I'm understanding everything right it's because the limitation of SATA 3. It tops out at 600MB/s / 6Gbps. CrystalDisk doesn't really move any files off the drives you're testing so it can go beyond that limitation, but when you're transferring files across the network, SATA 3 becomes a bottleneck.

Intel i9 9900K Coffee Lake @ X8 5309mhz (+47.5%)
AMD FX-8320 Vishera @ X4 5022mhz (+43.5%)
Intel i5 4690K Haswell @ X4 5013mhz (+43.2%)
AMD X4 960T Zosma @ X6 4870mhz (+62.3%)
Intel i7 6700 Skylake @ X4 4709mhz (+38.5%)
Intel i5 6400 Skylake @ X4 4588mhz (+69.9%)
2 OP pls nerf
(24 items)
NUC U
(10 items)
CPU
i9 9900K
Motherboard
AsRock Z390 Taichi
GPU
MSI GTX 1080 Ti
RAM
G.Skill DDR4 4133 32GB
Hard Drive
Mushkin Pilot NVMe 1TB
Hard Drive
HP EX950 NVMe 2TB
Hard Drive
Samsung 860 Evo 1TB
Hard Drive
Toshiba X300 5TB
Hard Drive
Western Digital EZRZ 3TB
Power Supply
Seasonic Focus Plus Gold 750w
Cooling
Corsair H115i Pro
Cooling
Raijintek Morpheus II
Case
Fractal Design Meshify S2
Operating System
Windows 10 Education x64
Monitor
Acer XF270HU 27" 1440p
Monitor
LG 24UD58 24" 4K
Keyboard
CoolerMaster MasterKeys Pro S
Mouse
Logitech G502
Mouse
Glorious Model D
Audio
Klipsch PowerGate + FiiO E10K
Audio
Micca RB42
Audio
Philips Fidelio X2
Audio
Philips SHP9500S
Audio
Audio Technica ATH-M50X
CPU
i3 8109U
Motherboard
Intel NUC8i3BEH
GPU
Iris Plus 655
RAM
Kingston 8GB DDR4 2400
Hard Drive
Dierka K1 120GB
Hard Drive
ADATA SU800 128GB M.2
Operating System
Windows 10 Professional
Operating System
MX Linux
Keyboard
Dierya DK63-BT 60% (Outemu Blue)
Mouse
Logitech G603 Wireless
CPU
i7 860
Motherboard
Advantech AIMB 280 ITX
GPU
EVGA GTX 950 2GB
RAM
Kingston 2GB DDR3 1333
Hard Drive
Seagate Momentus 2.5" 500GB
Hard Drive
Western Digital Blue 2.5" 320GB
Power Supply
Silverstone SFX-L Gold 500w
Cooling
Arctic Alpine 11 GT 2
Case
Silverstone Milo Z ML07B
Operating System
Windows XP x86 SP3
Keyboard
Velocifire TKL01 Outemu Browns
Mouse
Logitech M512
▲ hide details ▲


The Pook is offline  
post #3 of 8 (permalink) Old 09-24-2018, 04:30 PM - Thread Starter
Linux Lobbyist
 
elrompeplacas's Avatar
 
Join Date: Aug 2014
Posts: 78
Rep: 1 (Unique: 1)
Quote: Originally Posted by The Pook View Post
If I'm understanding everything right it's because the limitation of SATA 3. It tops out at 600MB/s / 6Gbps. CrystalDisk doesn't really move any files off the drives you're testing so it can go beyond that limitation, but when you're transferring files across the network, SATA 3 becomes a bottleneck.
But I am using 6 SATA ports not 1 port. The available speed should not be 6x600?

EDIT:

when I try to copy a file using RAMdisk and RAID0 I get this:

* PC-1 (RAID0) to PC-2 (RAMdisk): copy one file of 5GB I get a 630MB/s
* PC-2 (RAID0) to PC-1 (RAMdisk): copy one file of 5GB I get a 800MB/s

Obtaining 800MB/s seems to contradict your hypothesis.

Last edited by elrompeplacas; 09-24-2018 at 04:41 PM.
elrompeplacas is offline  
Sponsored Links
Advertisement
 
post #4 of 8 (permalink) Old 09-24-2018, 04:42 PM
New to Overclock.net
 
bucdan's Avatar
 
Join Date: Jan 2006
Location: San Diego, CA
Posts: 6,128
Rep: 202 (Unique: 181)
Are you using a dedicated RAID card, or the software/onboard raid?

Quote:
Originally Posted by holtzman go_quote.gif
Computer viruses are like herpes. You never think in a million years you'd get one... Till you do.
bucdan is offline  
post #5 of 8 (permalink) Old 09-24-2018, 04:47 PM - Thread Starter
Linux Lobbyist
 
elrompeplacas's Avatar
 
Join Date: Aug 2014
Posts: 78
Rep: 1 (Unique: 1)
Quote: Originally Posted by bucdan View Post
Are you using a dedicated RAID card, or the software/onboard raid?
Software RAID in the both computers.
I use dynamic disk RAID0 of windows 10 in the 2 computers.

With AHCI controller fixed in UEFI.

Last edited by elrompeplacas; 09-25-2018 at 01:36 AM.
elrompeplacas is offline  
post #6 of 8 (permalink) Old 09-24-2018, 04:58 PM
New to Overclock.net
 
bucdan's Avatar
 
Join Date: Jan 2006
Location: San Diego, CA
Posts: 6,128
Rep: 202 (Unique: 181)
Quote: Originally Posted by elrompeplacas View Post
Software RAID in the both computers.
I use dynamic disk RAID0 of windows 10 in the 2 computers.
As long as you're not using the old "Fake RAID", you're ok. The only thing I can think of is that hardware RAID will offer local RAM caching that will speed up the transfers, it's been a known given. I believe with Windows RAID, the operating system handles all of it, I'm not 100% on this last part.

Generally nowadays, the CPU and RAM are non issues for RAID since they are more than fast enough for the I/O, so the bottleneck is elsewhere, and I think it's the OS.

With your 800MB/s transfer, was that sustained over the whole copy? Or was it more of a burst? If it's bursting to 800MB/s, then dropping, then there is a bottleneck.

I'm 90% sure a dedicated RAID card will solve your problem. One that has built in cache.

EDIT: Of course, these enterprise level RAID cards are a lot of money, but you seem to care for data, just a thought.

Quote:
Originally Posted by holtzman go_quote.gif
Computer viruses are like herpes. You never think in a million years you'd get one... Till you do.

Last edited by bucdan; 09-24-2018 at 05:05 PM.
bucdan is offline  
post #7 of 8 (permalink) Old 09-28-2018, 08:01 AM - Thread Starter
Linux Lobbyist
 
elrompeplacas's Avatar
 
Join Date: Aug 2014
Posts: 78
Rep: 1 (Unique: 1)
Quote: Originally Posted by bucdan View Post
As long as you're not using the old "Fake RAID", you're ok. The only thing I can think of is that hardware RAID will offer local RAM caching that will speed up the transfers, it's been a known given. I believe with Windows RAID, the operating system handles all of it, I'm not 100% on this last part.

Generally nowadays, the CPU and RAM are non issues for RAID since they are more than fast enough for the I/O, so the bottleneck is elsewhere, and I think it's the OS.

With your 800MB/s transfer, was that sustained over the whole copy? Or was it more of a burst? If it's bursting to 800MB/s, then dropping, then there is a bottleneck.

I'm 90% sure a dedicated RAID card will solve your problem. One that has built in cache.

EDIT: Of course, these enterprise level RAID cards are a lot of money, but you seem to care for data, just a thought.
Thank you very much for your answer. I have done additional tests.

I send you captures of the copy graphics so you can analyze their stability.

I also think that the origin of the problem is likely to be the windows operating system.

Indeed my data is important and I do not mind spending some money. I'm going to buy a Fujitsu CP400I RAID controller (it has no cache memory). I will install the RAID controller on PC-1 and I will repeat the test. The problem I have is that I can not install a RAID controller on the PC-2 because it is a mini computer (mini ITX) and I only have one PCIe available that I need to use for the 10Gb NIC.

What is your opinion? with a single RAID controller can I get 1GB / s transfers? or do I need two RAID controllers?

Thank you very much!

Copy from RAID0-PC2 to RAID0-PC1:


Copy from M2-PC2 to RAID0-PC1:


Copy from Ramdisk-PC1 to RAID0-PC1 (internal copy):


Copy from Ramdisk-PC2 to RAID0-PC1:


Copy from Ramdisk-PC2 to Ramdisk-PC1:


Last edited by elrompeplacas; 09-28-2018 at 08:08 AM.
elrompeplacas is offline  
post #8 of 8 (permalink) Old 01-21-2020, 04:30 AM
New to Overclock.net
 
Join Date: Jan 2020
Posts: 4
Rep: 0
let me reply - while very very late (more than a year after you created this topic), it's better late than never.
Speeds you see are totally expected and you can't get more.



1) you only have FOUR disks on one side, in the best case (empty disk, outer tracks, sequential pattern) they transfer around 200MB/s. This speed HEAVILY decreases due to filesystem fragmentation and as it fills up (50% disk full, you will never achieve 200MB/s anymore, look how bandwidth graphs look like for ALL spinning disks - this is their nature. The more they are full, the slower they get)


2) you have old motherboards, weak CPUs. It's not primarily about CPU power, but it somehow plays role. Z97 chipset you used in both computers DOES NOT support PCI-Express v3.0, there only is v2.0 revision WITH HALF OF THE BANDWIDTH ! PCI-e 2.0 at 4x CAN'T AND WILL NOT transfer the same amount of data as PCI-e 3.0 4x can.


3) you will NOT achieve 10Gbit/s on 10Gbit network adapter, as you can see Ramdisk-to-Ramdisk gives you 8.4Gbit/s. Don't know what chip is used on this Acus adapter, but don't have any hopes for 9.95Gbit/s, that will never happen


4) OS optimization plays a role. Not in this case because you are limited by LOW AMOUNT of physical disks and speed they can achieve, but you would be surprised how many things can be fine-tuned in Windows (and much more in Linux).


5) Raid adapter change is not going to help you. Hardware or software doesn't matter, too.


6) you would be very surprised to hear metallic network adapters are SLOWER than fiber-optic based ones. Also, there is higher ping between two endpoints when not using fiber. Vendor-independent, this is not problem of Brocade, Intel nor anybody else.







What you achieved is totally within expectations. You can't hope for more.



FOUR physical disks in one PC is your primary limit, no matter what you do.


Six disks in second PC would be able to sustain 10Gbit/s bandwidth, but ONLY in some very specific circumstances. Almost empty, non-fragmented filesystem [!! head movement !!], liner/sequential transfers only (huge MKV file(s) like you copied here). When these disks will be full, meaning most inner tracks will be used, your hopes for 10Gbit are gone.
lubomirz is offline  
Reply

Quick Reply
Message:
Options

Register Now

In order to be able to post messages on the Overclock.net - An Overclocking Community forums, you must first register.
Please enter your desired user name, your email address and other required details in the form below.
User Name:
If you do not want to register, fill this field only and the name will be used as user name for your post.
Password
Please enter a password for your user account. Note that passwords are case-sensitive.
Password:
Confirm Password:
Email Address
Please enter a valid email address for yourself.
Email Address:

Log-in



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off