OpenIndiana 151a home server high disk load tuning

A week ago, I added a 30GB ZFS l2arc cache device to my storage array, resulting in tremendous system performance improvements. On a whim, I enabled the "Tracker" file indexer. Sadly, Tracker brought my system to its knees; my web browser became useless, and GNOME would shade out windows, indicating that they had become non-responsive.

I executed iostat -xcnCXTdz 5 and looked at the "%w" column, which indicates the percent time that there are disk transactions waiting for service. My %w column was pegged at 100% for one of my devices, indicating that the disk queue was bogged down with requests.

I did a little research, and I bumped across Princeton's Solaris troubleshooting guide for disk IO. The troubleshooting guide indicates that the sd_max_throttle parameter determines how many jobs can be queued up on a single host bus adapter, and is set to 256 by default.

I added set sd:sd_max_throttle=8 to /etc/system and rebooted. Not only did my system responsiveness improve, but zpool iostat showed an increase in throughput to my disk pools. My %w decreased to ~95% under heavy disk load.

I still notice times when %w==100 under heavy load. At these times, the system becomes unresponsive, and zpool iostat shows a drop in disk throughput. However, the system recovers quickly from these intermittent peaks.

I may further decrease sd_max_throttle and see how that affects system performance.

My system contains a mirrored array of three 1TB 4200RPM Western Digital "Green" hard disk drives and one WD Caviar 750MB drive. I'm using an AMD 760G mainboard with six on-board SATA ports; the SATA controller VID/PID is 1002/4391.