ZFS compression results in workload starvation, partially ameliorated by async_write_max_active

I have a particular dataset that consists of old backup files. Rather than storing them in .tar.gz archives, where the contents remain buried (and the files cannot be culled with useful tools such as fdupes), I elected to create a zfs dataset that uses gzip-9 compression.

However, on my system, I noticed that writing to the compression=gzip-9 zfs dataset resulted in starvation to other I/O processes  -- in particular, read processes -- rendering the system unusable.

In order to analyze the situation, I used this dtrace script to analyze different classes of I/O operations on my system.

See below for the sysctl tuneable that I modified.

vfs.zfs.vdev.async_write_max_active=10	vfs.zfs.vdev.async_write_max_active=3
Delete value ------------- Distribution ------------- count 256 | 0 512 |@@@ 2 1024 |@@@@ 3 2048 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 24 4096 | 0 Flush value ------------- Distribution ------------- count 512 | 0 1024 |@@@@ 4 2048 |@ 1 4096 |@@@@ 4 8192 |@@@@ 4 16384 |@@@ 3 32768 |@@@@@@@@@@@@@@@ 14 65536 |@@@@@@@@ 7 131072 | 0 Write value ------------- Distribution ------------- count 16 | 0 32 | 96 64 |@ 531 128 |@@ 735 256 |@@ 1067 512 |@@@@@@@@@@@@@@@@@@@@@@@@@@ 11639 1024 |@@@ 1573 2048 |@@ 878 4096 |@@ 785 8192 |@ 612 16384 | 183 32768 | 25 65536 | 16 131072 | 13 262144 | 3 524288 | 0 Read value ------------- Distribution ------------- count 32 | 0 64 | 12 128 | 48 256 |@ 124 512 |@@@@@@@@@@ 2046 1024 |@@@@ 906 2048 |@@@@@@ 1244 4096 |@@@@@@@ 1523 8192 |@@@@@@ 1228 16384 |@@@@ 774 32768 |@ 173 65536 | 26 131072 | 22 262144 | 5 524288 | 0	Delete value ------------- Distribution ------------- count 256 | 0 512 |@@ 1 1024 |@@@@@@@@@@@@@@@@@@@@ 12 2048 |@@@@@@@@ 5 4096 |@@ 1 8192 |@@@@@@@@ 5 16384 | 0 Flush value ------------- Distribution ------------- count 512 | 0 1024 |@@@@@@@@@ 5 2048 |@@@@ 2 4096 |@@@@@ 3 8192 |@@@@@@@@@ 5 16384 | 0 32768 |@@@@@@@ 4 65536 |@@@@@ 3 131072 | 0 Write value ------------- Distribution ------------- count 16 | 0 32 |@ 468 64 |@@@@@@ 3741 128 |@@@@@@ 3627 256 |@@@@@ 3234 512 |@@@@@@@@@@@@@@@@@ 10520 1024 |@ 916 2048 |@ 608 4096 |@ 476 8192 |@ 476 16384 | 246 32768 | 80 65536 | 54 131072 | 11 262144 | 5 524288 | 0 Read value ------------- Distribution ------------- count 32 | 0 64 | 18 128 |@ 70 256 |@ 153 512 |@@@@@@@@ 890 1024 |@@@@ 445 2048 |@@@@@ 594 4096 |@@@@@@ 667 8192 |@@@@@@ 709 16384 |@@@@@ 609 32768 |@@@ 381 65536 |@ 130 131072 | 26 262144 | 10 524288 | 0
avg latency stddev iops throughput Write 1892us 8057us 302/s 33416k/s Delete 2303us 732us 0/s 34k/s Read 7747us 15904us 135/s 17011k/s Flush 42742us 37072us 0/s 0k/s	avg latency stddev iops throughput Write 1521us 8316us 407/s 30107k/s Delete 3557us 3001us 0/s 41k/s Read 14255us 27405us 78/s 8844k/s Flush 25081us 32016us 0/s 0k/s

In particular, it seems that the average latency of I/O Flush dropped considerably.

A more exhaustive study under more carefully controlled conditions seems like a reasonable next step.

PC-BSD: FreeBSD 10.0 -> 10.1 upgrade using freebsd-update

I was having problems getting pc-sysupdate to work properly. For some reason, purging the pkg cache and forcing updates through pkg update -f resulted in package size mismatches, so I decided to try to upgrade my PC-BSD system manually. It took a while, but I think I figured out how to get it going.

First, I created a boot environment for my new upgrade:

beadm create fbsd_10.1_update
beadm mount fbsd_10.1_update /.updateStage


Next, I performed the freebsd-update on this new boot environment:

freebsd-update -b /.updateStage fetch
freebsd-update -b /.updateStage install


Eventually, the script finished its business. I then switched to the new environment:

beadm umount fbsd_10.1_update
beadm activate fbsd_10.1_update


All done, right? Here's what happened next. I rebooted the machine, and was dropped to a grub> prompt. What? beadm didn't handle grub gracefully?

Ugh. After digging, I figured out how to manhandle grub to boot my ZFS partition. The first trick, to save time, is to set an environment variable to the root filesystem. This saves a LOT of typing. Use "tab" completion to figure out what filesystems are available to you:

set P=(hd0,msdos2)/ROOT/fbsd_10.1_update/@/


Now, we can tell grub to load all the things (tab completion works with the $P expansion! brilliant!):

kfreebsd $P/boot/kernel/kernel
kfreebsd_loadenv $P/boot/device.hints
kfreebsd_module_elf $P/boot/kernel/opensolaris.ko
kfreebsd_module_elf $P/boot/kernel/zfs.ko
kfreebsd_module $P/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:tank/ROOT/fbsd_10.1_update


boot


Voila! System working again. But I still need to repair GRUB... after the system boots into the correct boot environment:

grub-mkconfig -o /boot/grub/grub.cfg

Unfortunately, that last step didn't actually fix my grub installation. I'm puzzled as to how to fix it. But for now, at least I can get the machine running.

ZFS Progress in FreeBSD

I've been watching the progress of my favorite filesystem, ZFS, in FreeBSD for a while now. ZFS v28 has been under test by the members of the freebsd-fs mailing list, and I have read multiple resources that have indicated that the performance of the ZFS implementation in FreeBSD is much better than in OpenSolaris; indeed, my Nexenta box only supports up to ZFS v26.

And now, ZFS v28 is now in FreeBSD-CURRENT:
http://lists.freebsd.org/pipermail/freebsd-fs/2011-February/010799.html

There is an excellent article in BSD magazine with some general information. The article is by Martin Matuška, who maintains a ZFS v28 patchset against FreeBSD-STABLE.

When I have some time, I'll be excited to migrate to FreeBSD, and to get away from the insanity that is (was) OpenSolaris.