Reduce userland CPU starvation from low-priority I/Os

[rottegift]

Once we have successfully read from disk we may do significant CPU-work on the data obtained, depending on features like record sizes, checksums, encryption, or compression. A successful write may cause significant CPU-work to be done for a subsequent zio.

Because our vdev_disk layer and its use of IOKit is fundamentally asynchronous, on some media modern
linearized scrubs and resilvers may "gang up" on bursts of interactive user I/Os.

Moreover, all zfs kernel threads are higher priority than the vast majority of userland threads, therefore the latter can be starved of CPU especially for a scrubbing pool which has a vdev count conmparable to the CPU core count and where data was wrtten using expensive checksums like sha256.

Practically all our IOKit I/Os are asynchronous, but significant work may be done on the taskq thread, possibly right to the entry into the vdev_disk_io_intr() callback function. We dispatch "background" I/Os into a lower thread-priority and lower thread-count taskq compared to other types of zio.

In the callback function itself, for these low-priority I/Os we kpreempt() before before calling
zio_delay_interrupt(). For writes, this may impose a system-load-dependent delay on notifying upper layers of zfs that IOKit has moved the buffer towards the physical device, generating backpressure on subsequent writes. For reads, this kpreempt() gives another thread in the system a chance to run before we do potentially heavy-CPU actions (such as checksumming or decyrption) on the data IOKit has obtained from the storage device.