post #1 of 1
Thread Starter 
Reads do not scale. Writes do.
Read and Write cache must be OFF. However, Read cache doesn't seem to influence much if anything. ON or OFF.

The stripe size seems to only influence reads. 32K is best. It gets up to 900MB/s READ. 1024K yeilds only about 450 / ~1300

HARDWARE RAID
read / write
1 SSD = 420 / 210
2 SSD = 390 / 420
4 SSD = 700 / 780
6 SSD = 750 / ~1200
8 SSD = 950 / ~1400



1

Code:
dd if=/dev/zero of=/dev/sde bs=1M count=5000; hdparm -t /dev/sde ; hdparm -t /dev/sde; hdparm -t /dev/sde
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 24.6315 s, 213 MB/s

/dev/sde:
 Timing buffered disk reads: 624 MB in  3.00 seconds = 207.76 MB/sec

/dev/sde:
 Timing buffered disk reads: 1252 MB in  3.00 seconds = 416.85 MB/sec

/dev/sde:
 Timing buffered disk reads: 1252 MB in  3.00 seconds = 416.96 MB/sec


2
Code:
dd if=/dev/zero of=/dev/sdd bs=1M count=5000; hdparm -t /dev/sdd ; hdparm -t /dev/sdd; hdparm -t /dev/sdd
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 12.5462 s, 418 MB/s

/dev/sdd:
 Timing buffered disk reads: 540 MB in  3.01 seconds = 179.51 MB/sec

/dev/sdd:
 Timing buffered disk reads: 1180 MB in  3.00 seconds = 392.78 MB/sec

/dev/sdd:
 Timing buffered disk reads: 1178 MB in  3.00 seconds = 392.47 MB/sec




4
Code:
dd if=/dev/zero of=/dev/sdg bs=1M count=5000; hdparm -t /dev/sdg ; hdparm -t /dev/sdg; hdparm -t /dev/sdg 
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 6.69474 s, 783 MB/s

/dev/sdg:
 Timing buffered disk reads: 852 MB in  3.00 seconds = 283.75 MB/sec

/dev/sdg:
 Timing buffered disk reads: 2120 MB in  3.00 seconds = 706.25 MB/sec

/dev/sdg:
 Timing buffered disk reads: 2118 MB in  3.00 seconds = 705.68 MB/sec



6
Code:
dd if=/dev/zero of=/dev/sdd bs=1M count=5000; hdparm -t /dev/sdd ; hdparm -t /dev/sdd; hdparm -t /dev/sdd
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 4.70587 s, 1.1 GB/s

/dev/sdd:
 Timing buffered disk reads: 1378 MB in  3.00 seconds = 458.91 MB/sec

/dev/sdd:
 Timing buffered disk reads: 2242 MB in  3.00 seconds = 746.80 MB/sec

/dev/sdd:
 Timing buffered disk reads: 2244 MB in  3.00 seconds = 747.36 MB/sec

8
Code:
dd if=/dev/zero of=/dev/sdd bs=1M count=5000; hdparm -t /dev/sdd ; hdparm -t /dev/sdd; hdparm -t /dev/sdd
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 4.33591 s, 1.2 GB/s

/dev/sdd:
 Timing buffered disk reads: 1232 MB in  3.00 seconds = 410.34 MB/sec

/dev/sdd:
 Timing buffered disk reads: 2858 MB in  3.00 seconds = 952.60 MB/sec

/dev/sdd:
 Timing buffered disk reads: 2856 MB in  3.00 seconds = 951.79 MB/sec


MUCH better results with SOFTWARE RAID:

1024K stripe
SOFTWARE RAID:
read / write
8 SSD = 1500 / 1400

8
Code:
dd if=/dev/zero of=/dev/md/array bs=1M count=5000; hdparm -t /dev/md/array ; hdparm -t /dev/md/array; hdparm -t /dev/md/array 
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 4.32352 s, 1.2 GB/s

/dev/md/array:
 Timing buffered disk reads: 4520 MB in  3.00 seconds = 1506.42 MB/sec

/dev/md/array:
 Timing buffered disk reads: 4524 MB in  3.00 seconds = 1507.97 MB/sec

/dev/md/array:
 Timing buffered disk reads: 4518 MB in  3.00 seconds = 1506.00 MB/sec

Software RAID with 32K stripe only gets about 900MB READ (writes normal).

I expected over closer to 3GB/s READ.

What am I missing here?

I do not believe it is the RAID controller. I experienced similar and worse results using a different system with software RAID although that was SATA2 and over 2 different controllers so I should revisit that tomorrow.
workstation
(14 items)
 
  
CPUMotherboardGraphicsRAM
i5 2500K EVGA Z68 SLI GeForce 670 GTX G.Skill 32GB 1600Mhz 
Hard DriveCoolingOSMonitor
OCZ Agility 3 60GB Zalman arch x86_x64 U3011 + U3014 
PowerCaseMouseAudio
PCPowerCooling 860w Chenbro server G500 Asus D2X, Def Tech 5.3 
  hide details  
Reply
workstation
(14 items)
 
  
CPUMotherboardGraphicsRAM
i5 2500K EVGA Z68 SLI GeForce 670 GTX G.Skill 32GB 1600Mhz 
Hard DriveCoolingOSMonitor
OCZ Agility 3 60GB Zalman arch x86_x64 U3011 + U3014 
PowerCaseMouseAudio
PCPowerCooling 860w Chenbro server G500 Asus D2X, Def Tech 5.3 
  hide details  
Reply