DELL PE860 & SAS 5/iR
DELL Power Edge 860 is a cheap 1U server. The price was a reason why we bought it with SAS 5/iR instead of PERC 5/i RAID controller. SAS 5/iR adapter is using chip SAS1068 from LSI. The adapter is supported by mptsas driver in Linux 2.6.18 kernel.
Monitoring of RAID status is possible by mpt-status program which is part of the Debian GNU/Linux distribution. The output of mpt-status is quite simple and easy to understand:
ocsp2:~# mpt-status ioc0 vol_id 0 type IM, 2 phy, 73 GB, state OPTIMAL, flags ENABLED ioc0 phy 1 scsi_id 32 ATA WDC WD800JD-75MS 1E04, 74 GB, state ONLINE, flags NONE ioc0 phy 0 scsi_id 1 ATA WDC WD800JD-75MS 1E04, 74 GB, state ONLINE, flags NONE
To get events from machine I installed dellomsa package which I got somewhere from the DELL support site. dellomsa depends on openipmi package. After installing both of them I started receiving events in daemon log file:
Jun 24 16:28:38 ocsp2 Administrator: Instrumentation Service EventID: 1000 Server Administrator starting Jun 24 16:28:38 ocsp2 Administrator: Instrumentation Service EventID: 1012 IPMI status Interface: OS Jun 24 16:28:38 ocsp2 Administrator: Instrumentation Service EventID: 1001 Server Administrator startup complete Jun 24 16:28:39 ocsp2 Administrator: Storage Service EventID: 2164 See readme.txt for a list of validated controller driver versions. Jun 24 16:30:52 ocsp2 Administrator: Storage Service EventID: 2199 The virtual disk cache policy has changed.: Virtual Disk 0 (Virtual Disk ) Controller 0 (SAS 5/iR Adapter) Jun 24 16:30:52 ocsp2 Administrator: Storage Service EventID: 2199 The virtual disk cache policy has changed.: Virtual Disk 0 (Virtual Disk ) Controller 0 (SAS 5/iR Adapter)
OMSA seems to be able to do much more than just receiving events from RAID adapter... to study this thing is on my TO-DO for quite a long time.
Another interesting utility is pediags Power Edge Diagnostic - it allows to run test on HDD, memory and on a few more HW parts of the server. Check out screen shot.
The most challenging part of setup was to get reasonable performance from SAS 5/iR adapter.
SAS 5/iR performance
By default SAS 5/iR RAID controller came with cache set to write through. That really horribly degrades performance:
ocsp2:~# time (dd if=/dev/zero of=bigfile count=10240 bs=1M; sync) 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 733.723 seconds, 14.6 MB/s real 13m24.367s user 0m0.008s sys 0m22.721s
The transfer rate is 10737418240 / (13 * 60 + 24.367) / 1024 / 1024 = 12.73 MiB/sec. Not very impressive, isn't it? The installation of Debian GNU/Linux on that system was boring, it took such a long time. It was even worse when I needed to restore the whole system from disk image backup, I had to write whole 74GB of data there, it took several hours.
During system restore I got enough time to google, I discovered that I'm not alone [1, 2, 3, 4], good to know but I was seeking for a solution. You have to read those URLs quite carefully to get answer. The tool you need is LSIUtil, it is inside Dell SAS 5/iR Adapter Driver. Cache tuning is not accessible via RAID controller BIOS. If you prefer you can take pre-compiled version of lsiutil-1.38 from my place.
Usage of lsiutil is simple:
ocsp2:~# ./mptlinux/lsiutil/lsiutil
LSI Logic MPT Configuration Utility, Version 1.38, July 6, 2005
1 MPT Port found
Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev
1. /proc/mpt/ioc0 LSI Logic SAS1068 B0 105 000a310
Select a device: [1-1 or 0 to quit] 1
1. Identify firmware, BIOS, and/or FCode
2. Download firmware (update the FLASH)
4. Download/erase BIOS and/or FCode (update the FLASH)
8. Scan for devices
10. Change IOC settings (interrupt coalescing)
13. Change SAS IO Unit settings
16. Display attached devices
20. Diagnostics
21. RAID actions
22. Reset bus
23. Reset target
30. Beacon on
31. Beacon off
97. Reset SAS phy
98. Reset SAS link
99. Reset port
Main menu, select an option: [1-99 or e for expert or 0 to quit] 21
1. Show volumes
2. Show physical disks
3. Get volume state
23. Replace physical disk
30. Create volume
31. Delete volume
32. Change volume settings
RAID actions menu, select an option: [1-99 or e for expert or 0 to quit] 32
Volume: [0-1 or RETURN to quit]
Volume 0 Settings: write caching disabled, auto configure, priority resync
Volume 0 draws from Hot Spare Pools:
Enable write caching: [Yes or No, default is No] yes
Offline on SMART data: [Yes or No, default is No]
Auto configuration: [Yes or No, default is Yes]
Priority resync: [Yes or No, default is Yes]
Hot Spare Pools (bitmask of pool numbers): [00 to FF, default is 01]
RAID actions menu, select an option: [1-99 or e for expert or 0 to quit]
Main menu, select an option: [1-99 or e for expert or 0 to quit]
Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev
1. /proc/mpt/ioc0 LSI Logic SAS1068 B0 105 000a310
Select a device: [1-1 or 0 to quit]
Setting seams to be persistent, it survives system reboot.
ocsp2:~# time (dd if=/dev/zero of=bigfile count=10240 bs=1M; sync) 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 252.835 seconds, 42.5 MB/s real 4m49.085s user 0m0.016s sys 0m23.889s
Write rate is now a lot bigger: 10737418240 / (4 * 60 + 49.085) / 1024 / 1024 = 35.42 MiB/sec.
Possible reason why LSI do not provide this feature inside adapter BIOS might be because this card doesn't have it's own battery backed up memory, adapter does rely on cache of HDDs. I'm afraid that in case of power failure during massive I/O, system could end with badly corrupted file system. What also worries me is that adapter is missing support for consistency checks (see feature list).
Conclusion
PE 860 is good not-so-expensive machine. If you can work just with one power supply and without hot plug hard drives, you will get a good value for your money.
Because of of poor performance of SAS 5/iR I almost rejected the very good machine as useless for my application. I would expect much better documentation from DELL, digging through Internet forums is time consuming. Essential utility hidden in package called "driver" is bad joke, without suggestion from others I would never open such package. Adapter was supported by vanilla kernel so why download anything else? I also hope that one day DELL will start support Debian GNU/Linux.
I tried to put all new info I learned into this article, if you find it useful or have any further comments to it, please let me know.