iostat -xcnCXTdz 5and looked at the "%w" column, which indicates the percent time that there are disk transactions waiting for service. My %w column was pegged at 100% for one of my devices, indicating that the disk queue was bogged down with requests.
I did a little research, and I bumped across Princeton's Solaris troubleshooting guide for disk IO. The troubleshooting guide indicates that the
sd_max_throttleparameter determines how many jobs can be queued up on a single host bus adapter, and is set to 256 by default.
I added set sd:sd_max_throttle=8 to /etc/system and rebooted. Not only did my system responsiveness improve, but
zpool iostatshowed an increase in throughput to my disk pools. My %w decreased to ~95% under heavy disk load.
I still notice times when %w==100 under heavy load. At these times, the system becomes unresponsive, and
zpool iostatshows a drop in disk throughput. However, the system recovers quickly from these intermittent peaks.
I may further decrease
sd_max_throttleand see how that affects system performance.
My system contains a mirrored array of three 1TB 4200RPM Western Digital "Green" hard disk drives and one WD Caviar 750MB drive. I'm using an AMD 760G mainboard with six on-board SATA ports; the SATA controller VID/PID is 1002/4391.