Overclock.net › Forums › Components › Hard Drives & Storage › RAID Controllers and Software › PERC 5/i RAID Card: Tips and Benchmarks
New Posts  All Forums:Forum Nav:

PERC 5/i RAID Card: Tips and Benchmarks - Page 252

post #2511 of 7194
Quote:
Originally Posted by BLinux View Post
Hi, sorry, I don't know much about Vista. However, you need to put your PERC 5/I in a 8x lane PCI-E, otherwise you are limited to under 1GB/sec. Also, you are using 3x HDD in RAID5, which is *effectively* 2x HDD and you're getting 150~200MB/sec so that means each *effective* HDD is doing about 75~100MB/sec!! This is actually very good whenever you can get a SATA hard drive to push close to 100MB/sec, you are close to the limit of your hardware.

If you want something faster, you need to add more HDD. You also may want to move to a 8x lane PCI-E slot though that only matters if you are going to add more HDD.

As to answer your question, though in the context of my Linux server; i set the controller's read cache policy to "NRA", aka "No Read Ahead". Linux has its own read ahead caching mechanism, and doing it twice actually reduces speed by 5~10%. On the other hand, in Linux, i increased the read ahead cache from the default (256K) to 32MB with this command:

# blockdev --setra 32768 /path/to/your/raid/array/block/device

I've experimented with a few values here, from 8MB, 16MB, 32MB, 64MB. Somehow, 32MB seems to yield the best results. 64MB of read ahead cache seems to reduce read throughput of the iozone benchmark. Although I'm happy to have figured this value out, I can't explain *why* this is the best value.
Thanks for such a quick reply! I thought you had tuned your cache in the MegaCLI utility using some LSI command, obviously got a bit confused there.

Concerning the PCI-E 4x, I think I wont have a problem with this until I go over 6 hard drives, at the moment 4x bandwidth will be enough and not seeing any typical flat lines in graphs that coincide with bottlenecks.

I am still sure these drives should be running faster, I have seen a few people on here with 3 drive setups getting over 200mb/s, the first page of this thread shows 4x 1TB spinpoints getting ~350MB/sec read on average, this is incredible performance! Would you think that just by adding one more RE2 on to this could yield a huge performance gain? Benchmarking single 500gb RE2's are on par, if not faster than Spinpoint F1's.

The strange thing I have found is if I use no read ahead, the drives average out at about 68MB/sec which is incredibly bad.
post #2512 of 7194
Quote:
Originally Posted by Jimbwlah View Post
I am still sure these drives should be running faster, I have seen a few people on here with 3 drive setups getting over 200mb/s, the first page of this thread shows 4x 1TB spinpoints getting ~350MB/sec read on average, this is incredible performance! Would you think that just by adding one more RE2 on to this could yield a huge performance gain? Benchmarking single 500gb RE2's are on par, if not faster than Spinpoint F1's.

The strange thing I have found is if I use no read ahead, the drives average out at about 68MB/sec which is incredibly bad.
I think you need to consider what benchmarks you're comparing to, because each benchmark tool gives different results. So, if you're not running iozone, you can't really compare your numbers with mine. Not only that, different configurations will give different results (Vista vs Linux, NTFS vs XFS, etc.).

You have to think about what is theoretically possible. You have 3 HDD in RAID5, which makes it effectively 2 HDD. Your read performance will *never* go beyond that of 2 HDD, unless some caching gets really great cache hits, but then you're not measuring the speed of the RAID, you're measuring the speed of your cache. Your write performance will *never* exceed that of 2 HDD, and usually less because there is the overhead of the parity calculation for each block of writes. Again, if you write data that will fit inside your write cache and you have write-back cache policy, then you will see very fast writes, but that is measuring the speed of your cache, not the RAID array. This is also another reason why i say you can't compare numbers from one benchmark to another; one benchmark may measure the effects of caching, one may not, and another may mix the results.

If each of your HDD can do 125MB/sec, then your theoretical maximum is going to be 250MB/sec, no more. But, I haven't seen any 7200RPM SATA HDD that can sustain 125MB/sec... usually, the best I've seen is a little less than 90MB/sec.
TAIPEI
(10 items)
 
AURORA
(13 items)
 
 
MotherboardGraphicsRAMHard Drive
ASRock X99 Extreme11 EVGA GTX 980 Superclocked 32GB 8x4GB Corsair LPX Samsung XP941  
Hard DriveCoolingOSMonitor
Western Digital 3TB RE Noctua NH-D15 Fedora 21 Linux Samsung S27D590C 
PowerCase
Seasonic SS-1200XP Cooler Master Cosmos II 
CPUMotherboardGraphicsRAM
Dual Quad-core L5430 2.66Ghz 12mb cache Intel 5000 chipset ATI ES1000 64GB FBDIMM DDR2 PC2-5300 667Mhz 
Hard DriveOSPower
WD3000FYYZ PERC H700 w/ 512MB cache CentOS 7.2.1511 950W x2 
  hide details  
Reply
TAIPEI
(10 items)
 
AURORA
(13 items)
 
 
MotherboardGraphicsRAMHard Drive
ASRock X99 Extreme11 EVGA GTX 980 Superclocked 32GB 8x4GB Corsair LPX Samsung XP941  
Hard DriveCoolingOSMonitor
Western Digital 3TB RE Noctua NH-D15 Fedora 21 Linux Samsung S27D590C 
PowerCase
Seasonic SS-1200XP Cooler Master Cosmos II 
CPUMotherboardGraphicsRAM
Dual Quad-core L5430 2.66Ghz 12mb cache Intel 5000 chipset ATI ES1000 64GB FBDIMM DDR2 PC2-5300 667Mhz 
Hard DriveOSPower
WD3000FYYZ PERC H700 w/ 512MB cache CentOS 7.2.1511 950W x2 
  hide details  
Reply
post #2513 of 7194
I have not seen a RAID5 setup on one of these cards (or any other for that matter) go over 1GB/s anyway. So I doubt the PCIe x4 limit is really too much of a bottleneck. In any case, the PERC5 is actually a PCI-X card, with a PCIe bridge in it. This PCI-X part is already capped at just over 1GB/s, so using an x8 slot won't really do anything.
Edited by the_beast - 5/27/09 at 2:49am
post #2514 of 7194
Quote:
Originally Posted by RobiNet View Post
PERC 5/i - Intel IOP333 + LSISAS1068 controller IC
PERC 6/i - LSI SAS1078 ROC (Raid-on-Chip)
Quote:
Originally Posted by devsk View Post
BLinux, why do you think DDR2-667 vs. DDR2-400 is causing the bottleneck for perc5 for close to half of supposedly peak b/w? 3.2GB/s is definitely faster than 1GB/s.

I think its the IOP AND the SAS controller which are making the difference. Both are newer revisions. RAM integration in the board might have reduced some data paths as well.
Quote:
Originally Posted by the_beast View Post
I believe the Perc 6i has a 500MHz, PowerPC-like processor. I'm not sure about the 5i architecture or clockspeed.

The block diagram is from the PERC5/6 comparison document from Dell. It is easy to find from Google.

I think most of the difference between the 2 comes from the processor - as previouslt stated, the actual speeds on both are well below the bottlenecks mentioned on the specs.
So, all this got me thinking about the performance of the PERC 5/I and I decided to read into the technical documentation further on the IOP333 and LSISAS1068. After thinking about it further, I think RobiNet is correct in that the main bottleneck here is the PCI-X bus, but the Dell diagram between PERC 5 vs 6 is actually lacking a bit of detail that I found in the Intel docs on the IOP333. And based on all this, I *think* the theoretical maximum sequential write speed is capped at around 500MB/sec.

Here are the links to the Intel and LSI docs:

http://download.intel.com/design/iio...s/30543303.pdf

http://www.lsi.com/DistributionSyste...sas1068_pb.pdf

Here's the block diagram of the IOP333:



what you will notice that isn't clear in the Dell diagram is that there is a PCI-X bus between the PCIe x8 interface and the internal 2.7GB/sec bus. Meaning, the PCI-X bus isn't just between the IOP333 and the LSISAS1068, it's also between the IOP333 core and the PCIe x8 interface!

Furthermore, PCI-X 133Mhz is not only capped at around 1GB/sec, it's also a half-duplex bus. So, if we think about the scenario of a large sequential write and using write-back cache policy with 512MB of cache, this is what happens:

1) start writing 1GB of data to the controller, since that data is going thru the PCIe 8x and then the PCI-X bus to get to the XScale core and to write back cache, it's going to be limited to 1GB/sec initially.

2) if we assume we can write at full speed at 1GB/sec, after 0.5 seconds, the 512MB of cache is full. At this point, the controller has to start flushing that cache back to disk. But this data flush also has to traverse the same PCI-X bus which is a half-duplex bus which means it has to share that 1GB/sec with the incoming remainder 512MB that is still being written to the controller. In an "ideal" situation, you would use half the PCI-X to flush the cache and half to continue receiving data. So, the last 512MB will get to the cache at 0.5GB/sec so it will take about 1 sec.

So the total theoretical time to write 1GB of fresh data is 0.5+1 = 1.5 seconds. But if you exclude the initial burst into cache, the sustainable write speed is 0.5 GB/sec, hence about 500MB/sec.

So, my benchmark results of 423MB/sec is about 85% of the theoretical maximum which I think is really good because the theoretical doesn't take into account latencies / overhead of the filesystem/OS or anything else. So, I think realistically, the best you can ever get from the PERC 5/I in sequential writes is going to be in the 4xx MB/sec range.

Now, on the other hand, I don't know how sequential reads are handled. But since a read doesn't have to flush cache data, you may be able to better utilize the PCI-X/133 bus in a single direction so you should be able to exceed 0.5 GB/sec which my benchmarks did at 520MB/sec. But, I would *think* it could do better?

With 8x 7200RPM SATA HDD in RAID5, I have 7 effective disks. These are WD RE3 drives, which some benchmarks posted online suggest can do 90+ MB/sec reads, so with 7 drives I should be able to get about 630MB/sec out of my setup. Perhaps there is more room for tuning sequential reads?

Lastly, the Intel docs on the IOP333 memory controller show some interesting information:

- it supports DDR SDRAM @ 333Mhz (with 500Mhz or 667Mhz processors) or it supports DDR2 SDRAM @ 400Mhz (with 500Mhz or 800Mhz processors). Since we know the PERC 5 uses the 400Mhz variety, the IOP333 is either running at 500Mhz or 800Mhz. I don't know which speed it is running at, but if 500Mhz, then there is overclocking opportunity? it wouldn't help sequential reads as per the discussion above, but it might help with random I/O ops.

- it confirms the use of x8 or x16 memory chips.

- in the product brief (http://download.intel.com/design/iio...f/30658301.pdf) it indicates that the memory controller can support up to 2GB of memory. I know someone has posted that they tried 1GB DIMM and it didn't work, but I wonder if they used the proper x8/x16 components in single bank configuration? this sort of makes me curious to want to try a 1GB or 2GB DIMM....
LL
Edited by BLinux - 5/27/09 at 3:37am
TAIPEI
(10 items)
 
AURORA
(13 items)
 
 
MotherboardGraphicsRAMHard Drive
ASRock X99 Extreme11 EVGA GTX 980 Superclocked 32GB 8x4GB Corsair LPX Samsung XP941  
Hard DriveCoolingOSMonitor
Western Digital 3TB RE Noctua NH-D15 Fedora 21 Linux Samsung S27D590C 
PowerCase
Seasonic SS-1200XP Cooler Master Cosmos II 
CPUMotherboardGraphicsRAM
Dual Quad-core L5430 2.66Ghz 12mb cache Intel 5000 chipset ATI ES1000 64GB FBDIMM DDR2 PC2-5300 667Mhz 
Hard DriveOSPower
WD3000FYYZ PERC H700 w/ 512MB cache CentOS 7.2.1511 950W x2 
  hide details  
Reply
TAIPEI
(10 items)
 
AURORA
(13 items)
 
 
MotherboardGraphicsRAMHard Drive
ASRock X99 Extreme11 EVGA GTX 980 Superclocked 32GB 8x4GB Corsair LPX Samsung XP941  
Hard DriveCoolingOSMonitor
Western Digital 3TB RE Noctua NH-D15 Fedora 21 Linux Samsung S27D590C 
PowerCase
Seasonic SS-1200XP Cooler Master Cosmos II 
CPUMotherboardGraphicsRAM
Dual Quad-core L5430 2.66Ghz 12mb cache Intel 5000 chipset ATI ES1000 64GB FBDIMM DDR2 PC2-5300 667Mhz 
Hard DriveOSPower
WD3000FYYZ PERC H700 w/ 512MB cache CentOS 7.2.1511 950W x2 
  hide details  
Reply
post #2515 of 7194
Quote:
Originally Posted by BLinux View Post
Lastly, the Intel docs on the IOP333 memory controller show some interesting information:

- it supports DDR SDRAM @ 333Mhz (with 500Mhz or 667Mhz processors) or it supports DDR2 SDRAM @ 400Mhz (with 500Mhz or 800Mhz processors). Since we know the PERC 5 uses the 400Mhz variety, the IOP333 is either running at 500Mhz or 800Mhz. I don't know which speed it is running at, but if 500Mhz, then there is overclocking opportunity? it wouldn't help sequential reads as per the discussion above, but it might help with random I/O ops.

- it confirms the use of x8 or x16 memory chips.

- in the product brief (http://download.intel.com/design/iio...f/30658301.pdf) it indicates that the memory controller can support up to 2GB of memory. I know someone has posted that they tried 1GB DIMM and it didn't work, but I wonder if they used the proper x8/x16 components in single bank configuration? this sort of makes me curious to want to try a 1GB or 2GB DIMM....
I think:
DELL PERC 5/i = LSI MegaRAID SAS 8408E = Intel® RAID Controller SRCSAS18E

and from Intel SRCSAS18E spec:

Processor:
Intel® IOP333 I/O processor operating at 500MHz with hardware XOR
Memory:
Includes 256MB of Registered ECC DDR2 400MHz SDRAM (with support for up to 1GB)


and photos - PCB looks identically:

Intel SRCSAS18E:


LSI 8408E:


so there is question - is there 512mb limit for cache size on DELL PERC 5/i as result of DELL's implementation ? or - like BLinux noticed - only wrong memory modules was tested ?

maybe somebody who have Intel or LSI cards and can check this ... or ... maybe somebody will try to flash PERC 5/i with Intel's firmware
post #2516 of 7194
Hello again friends!

BLinux:
Looking at block diagram we can see two PCI-E to PCI-X bridges.
And here is where my problem goes.
Bridge A feels good on my system, while bridge B haven't enough DMA
resources? Any suggestions?

I have an Intel board with SMBus.
Tried to disable all onboard devices in BIOS, nothing helped.

By the way i'm interesting is this second bridge used only
when more than four HDD attached, if so, i should not worry.
post #2517 of 7194
hi guys

can someon please tell me where can I download the latest driver for Perc 5i for Windows XP 32bit ?

I upgraded the firmware to latest LSI one as the first page says but I cant install the driver, it does not work.

my current driver is 1.19.0.32

many thanks in advance
post #2518 of 7194


Raid0 - 4x36gb 15,000rpm Seagate Savvio 2.5" SAS drives - 64k stripe
FW - Dell 1.03.10-0216

Adaptive Read Ahead/Write Back etc

Test was done with 256k test size. The write graph is pretty much the same

Have 4 more drives to add once I get another hotswap bay.
post #2519 of 7194
Awesome, wish my 750 sammy's performed that well
post #2520 of 7194
Some company in the USA liquidated its stock of them. There's still quite a few available on ebay for about $50-60USD. Search for "ST936751SS"

Oh and I should also note, my PERC5/i is only connected to a PCI-e x4 slot (x16 physical)

Same as before but Raid5-64k

Read:


Write:

Edited by MetalPhreak - 5/29/09 at 7:30am
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: RAID Controllers and Software
Overclock.net › Forums › Components › Hard Drives & Storage › RAID Controllers and Software › PERC 5/i RAID Card: Tips and Benchmarks