ZFS compression results in workload starvation, partially ameliorated by async_write_max_active

I have a particular dataset that consists of old backup files. Rather than storing them in .tar.gz archives, where the contents remain buried (and the files cannot be culled with useful tools such as fdupes), I elected to create a zfs dataset that uses gzip-9 compression.

However, on my system, I noticed that writing to the compression=gzip-9 zfs dataset resulted in starvation to other I/O processes  -- in particular, read processes -- rendering the system unusable.

In order to analyze the situation, I used this dtrace script to analyze different classes of I/O operations on my system.

See below for the sysctl tuneable that I modified.

vfs.zfs.vdev.async_write_max_active=10	vfs.zfs.vdev.async_write_max_active=3
Delete value ------------- Distribution ------------- count 256 | 0 512 |@@@ 2 1024 |@@@@ 3 2048 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 24 4096 | 0 Flush value ------------- Distribution ------------- count 512 | 0 1024 |@@@@ 4 2048 |@ 1 4096 |@@@@ 4 8192 |@@@@ 4 16384 |@@@ 3 32768 |@@@@@@@@@@@@@@@ 14 65536 |@@@@@@@@ 7 131072 | 0 Write value ------------- Distribution ------------- count 16 | 0 32 | 96 64 |@ 531 128 |@@ 735 256 |@@ 1067 512 |@@@@@@@@@@@@@@@@@@@@@@@@@@ 11639 1024 |@@@ 1573 2048 |@@ 878 4096 |@@ 785 8192 |@ 612 16384 | 183 32768 | 25 65536 | 16 131072 | 13 262144 | 3 524288 | 0 Read value ------------- Distribution ------------- count 32 | 0 64 | 12 128 | 48 256 |@ 124 512 |@@@@@@@@@@ 2046 1024 |@@@@ 906 2048 |@@@@@@ 1244 4096 |@@@@@@@ 1523 8192 |@@@@@@ 1228 16384 |@@@@ 774 32768 |@ 173 65536 | 26 131072 | 22 262144 | 5 524288 | 0	Delete value ------------- Distribution ------------- count 256 | 0 512 |@@ 1 1024 |@@@@@@@@@@@@@@@@@@@@ 12 2048 |@@@@@@@@ 5 4096 |@@ 1 8192 |@@@@@@@@ 5 16384 | 0 Flush value ------------- Distribution ------------- count 512 | 0 1024 |@@@@@@@@@ 5 2048 |@@@@ 2 4096 |@@@@@ 3 8192 |@@@@@@@@@ 5 16384 | 0 32768 |@@@@@@@ 4 65536 |@@@@@ 3 131072 | 0 Write value ------------- Distribution ------------- count 16 | 0 32 |@ 468 64 |@@@@@@ 3741 128 |@@@@@@ 3627 256 |@@@@@ 3234 512 |@@@@@@@@@@@@@@@@@ 10520 1024 |@ 916 2048 |@ 608 4096 |@ 476 8192 |@ 476 16384 | 246 32768 | 80 65536 | 54 131072 | 11 262144 | 5 524288 | 0 Read value ------------- Distribution ------------- count 32 | 0 64 | 18 128 |@ 70 256 |@ 153 512 |@@@@@@@@ 890 1024 |@@@@ 445 2048 |@@@@@ 594 4096 |@@@@@@ 667 8192 |@@@@@@ 709 16384 |@@@@@ 609 32768 |@@@ 381 65536 |@ 130 131072 | 26 262144 | 10 524288 | 0
avg latency stddev iops throughput Write 1892us 8057us 302/s 33416k/s Delete 2303us 732us 0/s 34k/s Read 7747us 15904us 135/s 17011k/s Flush 42742us 37072us 0/s 0k/s	avg latency stddev iops throughput Write 1521us 8316us 407/s 30107k/s Delete 3557us 3001us 0/s 41k/s Read 14255us 27405us 78/s 8844k/s Flush 25081us 32016us 0/s 0k/s

In particular, it seems that the average latency of I/O Flush dropped considerably.

A more exhaustive study under more carefully controlled conditions seems like a reasonable next step.

dtrace newbie HOWTO: listing providers AND structures

One of the first things that I wanted to learn about dtrace was the answer to the question: "How do I figure out where I can put hooks to trace system activities?"

This is accomplished with 'dtrace -l', which lists all the possible providers to which dtrace scripts can be attached.

Once you find a provider of interest, e.g. io:::start(), you can again use the list (-l) option to take a peek into the kind of data that is accessible to you:

# dtrace -lvn io:::start

   ID   PROVIDER            MODULE                          FUNCTION NAME
62099         io            kernel                                   start

    Probe Description Attributes
        Identifier Names: Private
        Data Semantics:   Private
        Dependency Class: Unknown

    Argument Attributes
        Identifier Names: Private
        Data Semantics:   Private
        Dependency Class: ISA

    Argument Types
        args[0]: struct bio *
        args[1]: struct devstat *

The next logical question is, "What is in struct bio? What is in struct devstat?"

You could find the answer by grepping through /usr/include/sys. However, there's a much faster way to find out what you can look at. This is the critical piece of information that I couldn't find in other HOWTO manuals on the Internet:

# dtrace -qn 'io:::start{print(*args[0]); exit(0); }'

struct bio {
    uint8_t bio_cmd = 0x2
    uint8_t bio_flags = 0
    uint8_t bio_cflags = 0
    uint8_t bio_pflags = 0
    struct cdev *bio_dev = 0
    struct disk *bio_disk = 0xfffff80012144000
    off_t bio_offset = 0x116948cb000
    long bio_bcount = 0x1000
    caddr_t bio_data = 0xfffffe0005ada000
    struct vm_page **bio_ma = 0
    int bio_ma_offset = 0
    int bio_ma_n = 0
    int bio_error = 0
    long bio_resid = 0
    void (*)() bio_done = kernel`g_disk_done
    void *bio_driver1 = 0
    void *bio_driver2 = 0
    void *bio_caller1 = 0
    void *bio_caller2 = 0
    struct bio_queue = {
        struct bio *tqe_next = 0
        struct bio **tqe_prev = 0
    }
    const char *bio_attribute = 0
    struct g_consumer *bio_from = 0
    struct g_provider *bio_to = 0
    off_t bio_length = 0x1000
    off_t bio_completed = 0
    u_int bio_children = 0
    u_int bio_inbed = 0
    struct bio *bio_parent = 0xfffff8018178f2e8
    struct bintime bio_t0 = {
        time_t sec = 0x24d28
        uint64_t frac = 0x4543b859cc1368f0
    }
    bio_task_t *bio_task = 0
    void *bio_task_arg = 0
    void *bio_classifier1 = 0
    void *bio_classifier2 = 0
    daddr_t bio_pblkno = 0x8b4a4658
}

Now, THAT was easy!