Overclock.net › Forums › Components › Hard Drives & Storage › RAID Controllers and Software › PERC 5/i RAID Card: Tips and Benchmarks
New Posts  All Forums:Forum Nav:

PERC 5/i RAID Card: Tips and Benchmarks - Page 439

post #4381 of 7150
Quote:
Originally Posted by the_beast View Post
I know that what I posted works with the firmware level I had when I bought the card and using the CLI tools to set up the arrays, for the simple reason that I did it. However I didn't leave my card like that for long, simply because it doesn't suit what I use my arrays for.

I don't know whether the setup is possible with the current BIOS revisions, or if it ever was possible using the GUI tools (BIOS or OS based). I should probably have clarified this in my post. If I had a PERC not in active use at the moment I would check it again, although I don't have available hardware at the moment.

My apologies if my post is inaccurate or misleading. I will edit it shortly...
Do you remember if it was Dell firmware or LSI? I couldn't find on this forum if it's safe to flash perc 6 with LSI firmware, the only post I found was that flashing rendered card unusable. I can try do it with CLI.. U used dell open manage storage manager for CLI support?
post #4382 of 7150
Quote:
Originally Posted by BLinux View Post
I use Linux with the PERC 5/I. In the Linux world at least, the Linux block device read-ahead conflicted with the controller's read-ahead. I ended up disabling the read-ahead in the controller, and cranked up the read-ahead buffer in Linux block device driver to something like 8Mbytes or 16Mbytes. This improved my sequential read performance from 150MB/sec to about 400+ MB/sec.
Would you be able to assist with setting up the filesystem/stripe under Linux? I have 4x WD 640's with no read ahead, write through and the largest stripe size the firmware (Dell Pkg 5.1.1-0040, FW 1.03.10-0216) allowed (128kb). Was going to use ext3 and storing large files mostly (150mb upto 12gb) but there will also be ~30gb of photos (~3mb) which would be viewed infrequently. I'm not after extreme performance, but it would be stupid to not at least optimise my setup.

Code:
[root@localhost ~]# /opt/MegaRAID/MegaCli/MegaCli -LDInfo -Lall -aALL

Adapter 0 -- Virtual Drive Information:
Virtual Disk: 0 (Target Id: 0)
Name:storage
RAID Level: Primary-5, Secondary-0, RAID Level Qualifier-3
Size:1.744 TB
State: Optimal
Stripe Size: 128 KB
Number Of Drives:4
Span Depth:1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default
Encryption Type: None
After reading around, I should be creating my EXT3 filesystem with a stride size tuned to suit the stripe. This would be a stride of 32 for a 128k stripe yes? Would I be better off reducing the stripe size to 64Kb and using a stride of 16?

Do you have any advice?
post #4383 of 7150
Quote:
Originally Posted by IdleWild View Post
Do you remember if it was Dell firmware or LSI? I couldn't find on this forum if it's safe to flash perc 6 with LSI firmware, the only post I found was that flashing rendered card unusable. I can try do it with CLI.. U used dell open manage storage manager for CLI support?
It was the Dell firmware that came with my card when I bought it - I've had it for a while now, but I'm afraid I don't know the firmware version it shipped with and I have since updated it to the latest Dell. I wouldn't flash any LSI firmware onto the card.

I used DOS MegaCLI from LSI, and only did the setup to have a play with the settings. I never booted it into an OS, just messed with the options.
post #4384 of 7150
Quote:
Originally Posted by dave- View Post
Would you be able to assist with setting up the filesystem/stripe under Linux? I have 4x WD 640's with no read ahead, write through and the largest stripe size the firmware (Dell Pkg 5.1.1-0040, FW 1.03.10-0216) allowed (128kb). Was going to use ext3 and storing large files mostly (150mb upto 12gb) but there will also be ~30gb of photos (~3mb) which would be viewed infrequently. I'm not after extreme performance, but it would be stupid to not at least optimise my setup.

Code:
[root@localhost ~]# /opt/MegaRAID/MegaCli/MegaCli -LDInfo -Lall -aALL

Adapter 0 -- Virtual Drive Information:
Virtual Disk: 0 (Target Id: 0)
Name:storage
RAID Level: Primary-5, Secondary-0, RAID Level Qualifier-3
Size:1.744 TB
State: Optimal
Stripe Size: 128 KB
Number Of Drives:4
Span Depth:1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default
Encryption Type: None
After reading around, I should be creating my EXT3 filesystem with a stride size tuned to suit the stripe. This would be a stride of 32 for a 128k stripe yes? Would I be better off reducing the stripe size to 64Kb and using a stride of 16?

Do you have any advice?
yeah, i'd be happy to help... rarely ever get anyone asking Linux w/ PERC questions around here...

keep in mind, with 4x HDD in RAID-5, you can at most expect about ~240MBytes/sec sequential access. don't expect more than that, and probably a bit less; just so when you benchmark the speeds, you know what you're aiming for. (don't think you're going to get the 520MBytes/sec you see in my benchmarks which uses 8x HDD)

my first suggestion is forget ext3; just not a great file system for performance. i highly recommend XFS. along those lines, I'll explain how to tune XFS for your RAID. if you really want to use ext3, just ask and I can share with you what i know about ext3 too, but i don't tune ext3 as much as I do XFS.

if your stripe size = 128kbytes, that's equal to 256blocks because 512bytes per block.

in raid 5, with 4 disks, you have a 3 disk wide data stripe. since each stripe is 256blocks, your stripe width is 3x256=768blocks.

so, your sunit (stripe unit) = 256 blocks, your swidth (stripe width) = 768 blocks. this information will be for your data section of XFS.

for the log section, you need to match the stripe unit to the file system block size, which is limited by Linux's page size, which is 4kb = 8 blocks.

so, to create the XFS file system:

mkfs.xfs -d sunit=256,swidth=768 -l version=2,lazy-count=1,sunit=8 -b size=4k /dev/sdX

where /dev/sdX is your RAID5 device.

Edit: you'll probably want to mount the XFS file system with the following mount options:

rw,noatime,logbufs=8,logbsize=256k,nobarrier

End Edit.

after creating the file system, you should also tune the Linux block device read-ahead buffer. the buffer size that worked for me was 32MB (but you might try something else between 8MB-32MB). this is done with:

/sbin/blockdev --setra 32768 /dev/sdX

if you're using LVM, be sure to also change the read-ahead buffer for the logical device, in RHEL, it would be something like:

/sbin/blockdev --setra 32768 /dev/VolGroup00/Vol00

finally, you want to use the 'noop' scheduler (this is effective on PERC RAID cards, but NOT on ALL RAID cards... sometimes the 'deadline' scheduler is better for other RAID controllers like those from HP) on the RAID block device only:

echo noop > /sys/block/sdX/queue/scheduler

there are other parameters you can tune to get 1% more here or there depending on what you're doing (well, for file servers, you can gain 10% in transfer speeds by tuning the TCP/IP stack), but the above should get you 99% of the way to a pretty well tuned disk I/O sub-system for sequential access (not random I/O, but it isn't too bad at random I/O from the benchmarks I've done).

good luck.. let me know how it goes for you....
Edited by BLinux - 6/19/10 at 10:04am
TAIPEI
(10 items)
 
AURORA
(13 items)
 
 
MotherboardGraphicsRAMHard Drive
ASRock X99 Extreme11 EVGA GTX 980 Superclocked 32GB 8x4GB Corsair LPX Samsung XP941  
Hard DriveCoolingOSMonitor
Western Digital 3TB RE Noctua NH-D15 Fedora 21 Linux Samsung S27D590C 
PowerCase
Seasonic SS-1200XP Cooler Master Cosmos II 
CPUMotherboardGraphicsRAM
Dual Quad-core L5430 2.66Ghz 12mb cache Intel 5000 chipset ATI ES1000 64GB FBDIMM DDR2 PC2-5300 667Mhz 
Hard DriveOSPower
WD3000FYYZ PERC H700 w/ 512MB cache CentOS 7.2.1511 950W x2 
  hide details  
Reply
TAIPEI
(10 items)
 
AURORA
(13 items)
 
 
MotherboardGraphicsRAMHard Drive
ASRock X99 Extreme11 EVGA GTX 980 Superclocked 32GB 8x4GB Corsair LPX Samsung XP941  
Hard DriveCoolingOSMonitor
Western Digital 3TB RE Noctua NH-D15 Fedora 21 Linux Samsung S27D590C 
PowerCase
Seasonic SS-1200XP Cooler Master Cosmos II 
CPUMotherboardGraphicsRAM
Dual Quad-core L5430 2.66Ghz 12mb cache Intel 5000 chipset ATI ES1000 64GB FBDIMM DDR2 PC2-5300 667Mhz 
Hard DriveOSPower
WD3000FYYZ PERC H700 w/ 512MB cache CentOS 7.2.1511 950W x2 
  hide details  
Reply
post #4385 of 7150
Quote:
Originally Posted by BLinux View Post
yeah, i'd be happy to help... rarely ever get anyone asking Linux w/ PERC questions around here...
I'd have thought it would be a common scenario but seems not!

Quote:
Originally Posted by BLinux View Post
keep in mind, with 4x HDD in RAID-5, you can at most expect about ~240MBytes/sec sequential access.
No problems there, it will just be media storage for XMBC frontend machines (2 at a time max).

Quote:
Originally Posted by BLinux View Post
my first suggestion is forget ext3; just not a great file system for performance. i highly recommend XFS.
Ok, well I'm already using XFS for the MythTV recording buffer so that is fine. I just thought since it was random storage, going EXT3 would be the safer option. Before we move onto setting up the filesystem should I rebuild the array to use any read ahead or leave it turned off?

Quote:
Originally Posted by BLinux View Post
mkfs.xfs -d sunit=256,swidth=768 -l version=2,lazy-count=1,sunit=8 -b size=4k /dev/sdX
All making sense here. No need to fdisk /dev/sda first if I'm using the entire device as a single XFS F/S correct?

Quote:
Originally Posted by BLinux View Post
rw,noatime,logbufs=8,logbsize=256k,nobarrier
I've mounted my MythTV F/S with defaults,noatime,nodiratime,allocsize=512m based on their recommendations, are any of these options useful for the RAID drive?

Quote:
Originally Posted by BLinux View Post
good luck.. let me know how it goes for you....
Did some basic tests just to see how it all goes...
write: dd if=/dev/zero of=ddfile2 bs=8K count=500000
read: dd if=ddfile2 of=/dev/null bs=8k

My original EXT3 setup
4096000000 bytes (4.1 GB) copied, 82.043 s, 49.9 MB/s
4096000000 bytes (4.1 GB) copied, 22.4009 s, 183 MB/s
EXT3 with linux block device read ahead set to 32mb just for comparison
4096000000 bytes (4.1 GB) copied, 84.611 s, 48.4 MB/s
4096000000 bytes (4.1 GB) copied, 15.1946 s, 270 MB/s
XFS configured as per your post
4096000000 bytes (4.1 GB) copied, 50.793 s, 80.6 MB/s
4096000000 bytes (4.1 GB) copied, 12.7085 s, 322 MB/s

Not sure if there is any need to keep playing around with it, quite happy with that level of performance.
Edited by dave- - 6/19/10 at 7:32pm
post #4386 of 7150
Quote:
Originally Posted by dave- View Post
I just thought since it was random storage, going EXT3 would be the safer option. Before we move onto setting up the filesystem should I rebuild the array to use any read ahead or leave it turned off?
definitely leave the controllers read-ahead turned off. it conflicts with the Linux block device read-ahead and actually slows things down.

Quote:
Originally Posted by dave- View Post
All making sense here. No need to fdisk /dev/sda first if I'm using the entire device as a single XFS F/S correct?
yes, i believe so. if anything, there shouldn't be a need to re-partition. if there's an existing file system, mkfs.xfs might warn you about it, but you can override with -f. if you're going to create something larger than 2TB, you'll need to use 'parted' instead to create GPT tables.

Quote:
Originally Posted by dave- View Post
I've mounted my MythTV F/S with defaults,noatime,nodiratime,allocsize=512m based on their recommendations, are any of these options useful for the RAID drive?
nodiratime would probably be a fine addition. i don't know much about the allocsize option... most of these mount options are optimizations for the file system and not so much related to the underlying hardware. the connection between file system and hardware is taken care of so long as you specified the correct swidth and sunit during mkfs.xfs.

Quote:
Originally Posted by dave- View Post
Did some basic tests just to see how it all goes...
write: dd if=/dev/zero of=ddfile2 bs=8K count=500000
read: dd if=ddfile2 of=/dev/null bs=8k

My original EXT3 setup
4096000000 bytes (4.1 GB) copied, 82.043 s, 49.9 MB/s
4096000000 bytes (4.1 GB) copied, 22.4009 s, 183 MB/s
EXT3 with linux block device read ahead set to 32mb just for comparison
4096000000 bytes (4.1 GB) copied, 84.611 s, 48.4 MB/s
4096000000 bytes (4.1 GB) copied, 15.1946 s, 270 MB/s
XFS configured as per your post
4096000000 bytes (4.1 GB) copied, 50.793 s, 80.6 MB/s
4096000000 bytes (4.1 GB) copied, 12.7085 s, 322 MB/s

Not sure if there is any need to keep playing around with it, quite happy with that level of performance.
Well, it looks like you got a nice boost in performance there! However, theoretically, I would kind of expect more in the write performance, and expect a little less in the read performance.

Did you clear out the Linux system vm caches before doing the read test? Otherwise, when the file is written, it will be loaded into cache and your read test might just be pulling from cache. although I'm not 100% sure how dd interacts with the filesystem cache. you can clear the cache prior to the read test by:

# sync; echo 3 > /proc/sys/vm/drop_caches

I don't know your hardware, but what is the single drive write performance expected to be? Most of the 7200RPM SATA drives I've tested can do 50+ MBytes/sec, so with 3 drives I would expect around 150MB/sec writes...

EDIT: oh, i just noticed above you mentioned setting up the controller with "write-through" caching policy. this is only faster than "write-back" if your writes are typically larger than your controller cache size. if your typical sequential writes are smaller than your controller's cache size, try using "write-back" cache policy instead... that might be why your writes are not as fast as I would expect.

i've worked on massive file servers that need to write really large files (DVD ISOs, Blu-ray ISOs, etc.), and in those situations when the write fills up the cache, there's a big penalty and I/O blocking that happens to let the controller flush the cache to disk in "write-back". So only in these type of situations is "write-through" a good option where the cache is being sync'ed while the writes are taking place.

END EDIT.
Edited by BLinux - 6/19/10 at 8:45pm
TAIPEI
(10 items)
 
AURORA
(13 items)
 
 
MotherboardGraphicsRAMHard Drive
ASRock X99 Extreme11 EVGA GTX 980 Superclocked 32GB 8x4GB Corsair LPX Samsung XP941  
Hard DriveCoolingOSMonitor
Western Digital 3TB RE Noctua NH-D15 Fedora 21 Linux Samsung S27D590C 
PowerCase
Seasonic SS-1200XP Cooler Master Cosmos II 
CPUMotherboardGraphicsRAM
Dual Quad-core L5430 2.66Ghz 12mb cache Intel 5000 chipset ATI ES1000 64GB FBDIMM DDR2 PC2-5300 667Mhz 
Hard DriveOSPower
WD3000FYYZ PERC H700 w/ 512MB cache CentOS 7.2.1511 950W x2 
  hide details  
Reply
TAIPEI
(10 items)
 
AURORA
(13 items)
 
 
MotherboardGraphicsRAMHard Drive
ASRock X99 Extreme11 EVGA GTX 980 Superclocked 32GB 8x4GB Corsair LPX Samsung XP941  
Hard DriveCoolingOSMonitor
Western Digital 3TB RE Noctua NH-D15 Fedora 21 Linux Samsung S27D590C 
PowerCase
Seasonic SS-1200XP Cooler Master Cosmos II 
CPUMotherboardGraphicsRAM
Dual Quad-core L5430 2.66Ghz 12mb cache Intel 5000 chipset ATI ES1000 64GB FBDIMM DDR2 PC2-5300 667Mhz 
Hard DriveOSPower
WD3000FYYZ PERC H700 w/ 512MB cache CentOS 7.2.1511 950W x2 
  hide details  
Reply
post #4387 of 7150
Quote:
Originally Posted by BLinux View Post
nodiratime would probably be a fine addition. i don't know much about the allocsize option...
It was a recommendation on the mythtv wiki for tv recordings if you're interested.

Quote:
Originally Posted by BLinux View Post
Did you clear out the Linux system vm caches before doing the read test?
No but I'll try that later. I'm happy with the performance but will compare speed with the RAID disks and another exact drive running standalone on the mainboards sata controller.

Quote:
Originally Posted by BLinux View Post
EDIT: oh, i just noticed above you mentioned setting up the controller with "write-through" caching policy
I did this because I don't have a BBU installed. I do have a UPS but the general info seems to be not to use writeback unless you have a BBU?
post #4388 of 7150
Can this be done?? is it a firmware update or something that has to be done in the card bios??

Will this allow the card to run at its full potential as atm on write through with 4 F3's i am seein a poor performance on raid 5 with 128k stripe.
Gaming Rig
(12 items)
 
Main Server
(13 items)
 
 
CPUMotherboardGraphicsRAM
Intel i5 2400k Asus P8H77-M Onboard HDMI 16GB Gskill Ripjaw 1600 
Hard DriveOSMonitorPower
6x Sammy 2TB F4 (Perc 5i On LSI Firmware w/BBU) Windows Hyper-V 2008 R2 32" LG 1080p LCD via HDMI OCZ 700W Modular 
Case
Fractal Designs R3 
  hide details  
Reply
Gaming Rig
(12 items)
 
Main Server
(13 items)
 
 
CPUMotherboardGraphicsRAM
Intel i5 2400k Asus P8H77-M Onboard HDMI 16GB Gskill Ripjaw 1600 
Hard DriveOSMonitorPower
6x Sammy 2TB F4 (Perc 5i On LSI Firmware w/BBU) Windows Hyper-V 2008 R2 32" LG 1080p LCD via HDMI OCZ 700W Modular 
Case
Fractal Designs R3 
  hide details  
Reply
post #4389 of 7150
Does anyone need a PERC 5/6 simulation program? It acts just as the real perc BIOS, with 8 HDDs available to play with. Whoever manages the sticky could probably even add it there.
Edited by IdleWild - 6/23/10 at 5:05pm
post #4390 of 7150
1) anyone know somewhere I can get a cheap PERC5?

2) anyone have a PERC5 with 4x RaptorX in RAID 0+1?
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: RAID Controllers and Software
Overclock.net › Forums › Components › Hard Drives & Storage › RAID Controllers and Software › PERC 5/i RAID Card: Tips and Benchmarks