zfs hang when read/write data on file system(ext4) formatted from zvol

[Vicente-Cheng]

I apologize in advance if this is a duplicate.

We got the zfs hang problem with following scenario:
1. Initiator format the exported volumes to file system(ext4).
2. Read/Write data on this file system (loading may be heavy).
3. The volumes often offline under 24 hours.

I got these code trace in server side.

Nov 21 00:08:25 ubuntu27 kernel: [480028.008998] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.009856] zvol/0          D ffff880627c14580     0  2369      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.009861]  ffff880c1ca89b70 0000000000000046 ffff880c1ca89fd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.009866]  ffff880c1ca89fd8 0000000000014580 ffff880c1bf42ee0 ffff880627c14e28
Nov 21 00:08:25 ubuntu27 kernel: [480028.009870]  ffff880576be8360 ffff880576be8390 0000000000000001 0000000000000002
Nov 21 00:08:25 ubuntu27 kernel: [480028.009875] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.009884]  [<ffffffff816ea87d>] io_schedule+0x9d/0x130
Nov 21 00:08:25 ubuntu27 kernel: [480028.009901]  [<ffffffffa036a1ad>] cv_wait_common+0x9d/0x1a0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.009907]  [<ffffffff81085580>] ? wake_up_atomic_t+0x30/0x30
Nov 21 00:08:25 ubuntu27 kernel: [480028.009916]  [<ffffffffa036a308>] __cv_wait_io+0x18/0x20 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.009954]  [<ffffffffa0d7af83>] zio_wait+0x103/0x1c0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.009977]  [<ffffffffa0cf98c3>] dmu_tx_count_write+0x663/0x6f0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.009985]  [<ffffffffa03610c6>] ? kmem_alloc_debug+0x96/0x3c0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.010009]  [<ffffffffa0cfd0e7>] ? dnode_hold_impl+0x437/0x5c0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.010015]  [<ffffffff8133501e>] ? freed_request+0x4e/0x80
Nov 21 00:08:25 ubuntu27 kernel: [480028.010038]  [<ffffffffa0cf9986>] dmu_tx_hold_write+0x36/0x50 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.010069]  [<ffffffffa0d8479a>] zvol_write+0x9a/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.010072]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.010081]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.010086]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.010090]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.010098]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.010104]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.010108]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.010113]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.010117]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.010120] INFO: task zvol/1:2370 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.010817] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.011669] zvol/1          D ffff880627c34580     0  2370      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.011672]  ffff880c19effbb0 0000000000000046 ffff880c19efffd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.011676]  ffff880c19efffd8 0000000000014580 ffff880c19fdaee0 ffff880c19d8d1d8
Nov 21 00:08:25 ubuntu27 kernel: [480028.011680]  ffff880c19d8d1dc ffff880c19fdaee0 00000000ffffffff ffff880c19d8d1e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.011684] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.011688]  [<ffffffff816eb4d9>] schedule_preempt_disabled+0x29/0x70
Nov 21 00:08:25 ubuntu27 kernel: [480028.011695]  [<ffffffff816e951f>] __mutex_lock_slowpath+0x13f/0x1c0
Nov 21 00:08:25 ubuntu27 kernel: [480028.011699]  [<ffffffff816e890f>] mutex_lock+0x1f/0x2f
Nov 21 00:08:25 ubuntu27 kernel: [480028.011732]  [<ffffffffa0d34791>] txg_delay+0x61/0x130 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.011759]  [<ffffffffa0d0dbc1>] dsl_pool_tempreserve_space+0xd1/0xe0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.011785]  [<ffffffffa0d0b5a7>] dsl_dir_tempreserve_space+0x137/0x200 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.011791]  [<ffffffff81019e89>] ? read_tsc+0x9/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.011814]  [<ffffffffa0cf8bce>] dmu_tx_assign+0x3ee/0x420 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.011845]  [<ffffffffa0d847a7>] zvol_write+0xa7/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.011848]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.011857]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.011861]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.011864]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.011872]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.011876]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.011880]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.011883]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.011887]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.011890] INFO: task zvol/2:2371 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.012649] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.034760] zvol/2          D ffff880627c54580     0  2371      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.034764]  ffff880c188e3bb0 0000000000000046 ffff880c188e3fd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.034768]  ffff880c188e3fd8 0000000000014580 ffff880c1d102ee0 ffff880c19d8d1d8
Nov 21 00:08:25 ubuntu27 kernel: [480028.034772]  ffff880c19d8d1dc ffff880c1d102ee0 00000000ffffffff ffff880c19d8d1e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.034776] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.034781]  [<ffffffff816eb4d9>] schedule_preempt_disabled+0x29/0x70
Nov 21 00:08:25 ubuntu27 kernel: [480028.034785]  [<ffffffff816e951f>] __mutex_lock_slowpath+0x13f/0x1c0
Nov 21 00:08:25 ubuntu27 kernel: [480028.034789]  [<ffffffff816e890f>] mutex_lock+0x1f/0x2f
Nov 21 00:08:25 ubuntu27 kernel: [480028.034822]  [<ffffffffa0d34791>] txg_delay+0x61/0x130 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.034849]  [<ffffffffa0d0dbc1>] dsl_pool_tempreserve_space+0xd1/0xe0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.034875]  [<ffffffffa0d0b5a7>] dsl_dir_tempreserve_space+0x137/0x200 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.034878]  [<ffffffff81019e89>] ? read_tsc+0x9/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.034901]  [<ffffffffa0cf8bce>] dmu_tx_assign+0x3ee/0x420 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.034932]  [<ffffffffa0d847a7>] zvol_write+0xa7/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.034935]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.034944]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.034948]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.034951]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.034959]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.034963]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.034967]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.034970]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.034974]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.034977] INFO: task zvol/3:2373 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.057364] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.080374] zvol/3          D ffff880627c74580     0  2373      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.080378]  ffff880c1c75dbb0 0000000000000046 ffff880c1c75dfd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.080382]  ffff880c1c75dfd8 0000000000014580 ffff880c1bfdc650 ffff880c19d8d1d8
Nov 21 00:08:25 ubuntu27 kernel: [480028.080386]  ffff880c19d8d1dc ffff880c1bfdc650 00000000ffffffff ffff880c19d8d1e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.080390] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.080394]  [<ffffffff816eb4d9>] schedule_preempt_disabled+0x29/0x70
Nov 21 00:08:25 ubuntu27 kernel: [480028.080399]  [<ffffffff816e951f>] __mutex_lock_slowpath+0x13f/0x1c0
Nov 21 00:08:25 ubuntu27 kernel: [480028.080402]  [<ffffffff816e890f>] mutex_lock+0x1f/0x2f
Nov 21 00:08:25 ubuntu27 kernel: [480028.080436]  [<ffffffffa0d34791>] txg_delay+0x61/0x130 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.080463]  [<ffffffffa0d0dbc1>] dsl_pool_tempreserve_space+0xd1/0xe0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.080490]  [<ffffffffa0d0b5a7>] dsl_dir_tempreserve_space+0x137/0x200 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.080494]  [<ffffffff81019e89>] ? read_tsc+0x9/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.080517]  [<ffffffffa0cf8bce>] dmu_tx_assign+0x3ee/0x420 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.080548]  [<ffffffffa0d847a7>] zvol_write+0xa7/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.080551]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.080560]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.080564]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.080567]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.080575]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.080579]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.080583]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.080586]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.080590]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.080593] INFO: task zvol/4:2374 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.103958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.128010] zvol/4          D ffff880627c94580     0  2374      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.128014]  ffff880c1c635a30 0000000000000046 ffff880c1c635fd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.128018]  ffff880c1c635fd8 0000000000014580 ffff880c1c610000 ffff880627c94e28
Nov 21 00:08:25 ubuntu27 kernel: [480028.128022]  ffff880913d99990 ffff880913d999c0 0000000000000001 0000000000000002
Nov 21 00:08:25 ubuntu27 kernel: [480028.128026] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.128030]  [<ffffffff816ea87d>] io_schedule+0x9d/0x130
Nov 21 00:08:25 ubuntu27 kernel: [480028.128041]  [<ffffffffa036a1ad>] cv_wait_common+0x9d/0x1a0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128044]  [<ffffffff81085580>] ? wake_up_atomic_t+0x30/0x30
Nov 21 00:08:25 ubuntu27 kernel: [480028.128052]  [<ffffffffa036a308>] __cv_wait_io+0x18/0x20 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128084]  [<ffffffffa0d7af83>] zio_wait+0x103/0x1c0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128102]  [<ffffffffa0ce59c7>] dbuf_read+0x2d7/0x880 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128120]  [<ffffffffa0ce6362>] __dbuf_hold_impl+0x3f2/0x4a0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128138]  [<ffffffffa0ce648b>] dbuf_hold_impl+0x7b/0xa0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128161]  [<ffffffffa0cf9627>] dmu_tx_count_write+0x3c7/0x6f0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128184]  [<ffffffffa0cf9986>] dmu_tx_hold_write+0x36/0x50 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128342]  [<ffffffffa0d8479a>] zvol_write+0x9a/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128346]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.128354]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128358]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.128361]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.128369]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.128380]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.128384]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.128387]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.128390]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.128393] INFO: task zvol/5:2375 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.152494] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.176851] zvol/5          D ffff880627cb4580     0  2375      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.176855]  ffff880c1cd81b20 0000000000000046 ffff880c1cd81fd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.176859]  ffff880c1cd81fd8 0000000000014580 ffff880c1cbc1770 ffff880c19d8d1d8
Nov 21 00:08:25 ubuntu27 kernel: [480028.176863]  ffff880c19d8d1dc ffff880c1cbc1770 00000000ffffffff ffff880c19d8d1e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.176867] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.176871]  [<ffffffff816eb4d9>] schedule_preempt_disabled+0x29/0x70
Nov 21 00:08:25 ubuntu27 kernel: [480028.176875]  [<ffffffff816e951f>] __mutex_lock_slowpath+0x13f/0x1c0
Nov 21 00:08:25 ubuntu27 kernel: [480028.176879]  [<ffffffff816e890f>] mutex_lock+0x1f/0x2f
Nov 21 00:08:25 ubuntu27 kernel: [480028.176890]  [<ffffffffa036a3c9>] __cv_timedwait_common+0xb9/0x210 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.176910]  [<ffffffffa0ce3b21>] ? dbuf_destroy+0x161/0x170 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.176918]  [<ffffffff8118de55>] ? __kmalloc+0x175/0x180
Nov 21 00:08:25 ubuntu27 kernel: [480028.176921]  [<ffffffff81085580>] ? wake_up_atomic_t+0x30/0x30
Nov 21 00:08:25 ubuntu27 kernel: [480028.176930]  [<ffffffffa036a533>] __cv_timedwait+0x13/0x20 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.176961]  [<ffffffffa0d3480f>] txg_delay+0xdf/0x130 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.176987]  [<ffffffffa0d0dbc1>] dsl_pool_tempreserve_space+0xd1/0xe0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.177013]  [<ffffffffa0d0b5a7>] dsl_dir_tempreserve_space+0x137/0x200 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.177017]  [<ffffffff81019e89>] ? read_tsc+0x9/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.177039]  [<ffffffffa0cf8bce>] dmu_tx_assign+0x3ee/0x420 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.177070]  [<ffffffffa0d847a7>] zvol_write+0xa7/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.177073]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.177082]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.177085]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.177089]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.177097]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.177101]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.177105]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.177108]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.177112]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.177115] INFO: task zvol/6:2376 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.201534] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.226420] zvol/6          D ffff880c3fc14580     0  2376      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.226424]  ffff880c1cbb9bb0 0000000000000046 ffff880c1cbb9fd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.226428]  ffff880c1cbb9fd8 0000000000014580 ffff880c1cbc0000 ffff880c19d8d1d8
Nov 21 00:08:25 ubuntu27 kernel: [480028.226432]  ffff880c19d8d1dc ffff880c1cbc0000 00000000ffffffff ffff880c19d8d1e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.226436] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.226441]  [<ffffffff816eb4d9>] schedule_preempt_disabled+0x29/0x70
Nov 21 00:08:25 ubuntu27 kernel: [480028.226445]  [<ffffffff816e951f>] __mutex_lock_slowpath+0x13f/0x1c0
Nov 21 00:08:25 ubuntu27 kernel: [480028.226448]  [<ffffffff816e890f>] mutex_lock+0x1f/0x2f
Nov 21 00:08:25 ubuntu27 kernel: [480028.226482]  [<ffffffffa0d34791>] txg_delay+0x61/0x130 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.226508]  [<ffffffffa0d0dbc1>] dsl_pool_tempreserve_space+0xd1/0xe0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.226534]  [<ffffffffa0d0b5a7>] dsl_dir_tempreserve_space+0x137/0x200 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.226538]  [<ffffffff81019e89>] ? read_tsc+0x9/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.226561]  [<ffffffffa0cf8bce>] dmu_tx_assign+0x3ee/0x420 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.226592]  [<ffffffffa0d847a7>] zvol_write+0xa7/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.226595]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.226604]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.226608]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.226611]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.226619]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.226623]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.226627]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.226630]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.226634]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.226637] INFO: task zvol/7:2377 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.251721] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.277500] zvol/7          D ffff880c3fc34580     0  2377      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.277503]  ffff880c1c57bbb0 0000000000000046 ffff880c1c57bfd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.277508]  ffff880c1c57bfd8 0000000000014580 ffff880c1bfe8000 ffff880c19d8d1d8
Nov 21 00:08:25 ubuntu27 kernel: [480028.277511]  ffff880c19d8d1dc ffff880c1bfe8000 00000000ffffffff ffff880c19d8d1e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.277516] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.277520]  [<ffffffff816eb4d9>] schedule_preempt_disabled+0x29/0x70
Nov 21 00:08:25 ubuntu27 kernel: [480028.277524]  [<ffffffff816e951f>] __mutex_lock_slowpath+0x13f/0x1c0
Nov 21 00:08:25 ubuntu27 kernel: [480028.277528]  [<ffffffff816e890f>] mutex_lock+0x1f/0x2f
Nov 21 00:08:25 ubuntu27 kernel: [480028.277562]  [<ffffffffa0d34791>] txg_delay+0x61/0x130 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.277588]  [<ffffffffa0d0dbc1>] dsl_pool_tempreserve_space+0xd1/0xe0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.277614]  [<ffffffffa0d0b5a7>] dsl_dir_tempreserve_space+0x137/0x200 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.277617]  [<ffffffff81019e89>] ? read_tsc+0x9/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.277640]  [<ffffffffa0cf8bce>] dmu_tx_assign+0x3ee/0x420 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.277671]  [<ffffffffa0d847a7>] zvol_write+0xa7/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.277674]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.277684]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.277687]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.277691]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.277699]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.277703]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.277707]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.277710]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.277714]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.277716] INFO: task zvol/8:2380 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.303859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.330217] zvol/8          D ffff880c3fc54580     0  2380      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.330220]  ffff880c1c6efb70 0000000000000046 ffff880c1c6effd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.330224]  ffff880c1c6effd8 0000000000014580 ffff880c1af35dc0 ffff880c3fc54e28
Nov 21 00:08:25 ubuntu27 kernel: [480028.330228]  ffff8800a40990b0 ffff8800a40990e0 0000000000000001 0000000000000002
Nov 21 00:08:25 ubuntu27 kernel: [480028.330232] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.330236]  [<ffffffff816ea87d>] io_schedule+0x9d/0x130
Nov 21 00:08:25 ubuntu27 kernel: [480028.330247]  [<ffffffffa036a1ad>] cv_wait_common+0x9d/0x1a0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330250]  [<ffffffff81085580>] ? wake_up_atomic_t+0x30/0x30
Nov 21 00:08:25 ubuntu27 kernel: [480028.330258]  [<ffffffffa036a308>] __cv_wait_io+0x18/0x20 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330290]  [<ffffffffa0d7af83>] zio_wait+0x103/0x1c0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330313]  [<ffffffffa0cf98c3>] dmu_tx_count_write+0x663/0x6f0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330322]  [<ffffffffa03610c6>] ? kmem_alloc_debug+0x96/0x3c0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330345]  [<ffffffffa0cfd0e7>] ? dnode_hold_impl+0x437/0x5c0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330349]  [<ffffffff8133501e>] ? freed_request+0x4e/0x80
Nov 21 00:08:25 ubuntu27 kernel: [480028.330372]  [<ffffffffa0cf9986>] dmu_tx_hold_write+0x36/0x50 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330403]  [<ffffffffa0d8479a>] zvol_write+0x9a/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330406]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.330414]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330418]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.330421]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.330429]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.330433]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.330437]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.330440]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.330444]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.330447] INFO: task zvol/9:2381 blocked for more than 120 seconds.
Nov 21 00:08:25 ubuntu27 kernel: [480028.356583] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 21 00:08:25 ubuntu27 kernel: [480028.383074] zvol/9          D ffff880c3fc74580     0  2381      2 0x00000000
Nov 21 00:08:25 ubuntu27 kernel: [480028.383078]  ffff880c1cf6fb70 0000000000000046 ffff880c1cf6ffd8 0000000000014580
Nov 21 00:08:25 ubuntu27 kernel: [480028.383082]  ffff880c1cf6ffd8 0000000000014580 ffff880c1a418000 ffff880c3fc74e28
Nov 21 00:08:25 ubuntu27 kernel: [480028.383086]  ffff8805e7985c20 ffff8805e7985c50 0000000000000001 0000000000000002
Nov 21 00:08:25 ubuntu27 kernel: [480028.383090] Call Trace:
Nov 21 00:08:25 ubuntu27 kernel: [480028.383094]  [<ffffffff816ea87d>] io_schedule+0x9d/0x130
Nov 21 00:08:25 ubuntu27 kernel: [480028.383105]  [<ffffffffa036a1ad>] cv_wait_common+0x9d/0x1a0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383108]  [<ffffffff81085580>] ? wake_up_atomic_t+0x30/0x30
Nov 21 00:08:25 ubuntu27 kernel: [480028.383116]  [<ffffffffa036a308>] __cv_wait_io+0x18/0x20 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383149]  [<ffffffffa0d7af83>] zio_wait+0x103/0x1c0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383172]  [<ffffffffa0cf98c3>] dmu_tx_count_write+0x663/0x6f0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383180]  [<ffffffffa03610c6>] ? kmem_alloc_debug+0x96/0x3c0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383203]  [<ffffffffa0cfd0e7>] ? dnode_hold_impl+0x437/0x5c0 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383207]  [<ffffffff8133501e>] ? freed_request+0x4e/0x80
Nov 21 00:08:25 ubuntu27 kernel: [480028.383230]  [<ffffffffa0cf9986>] dmu_tx_hold_write+0x36/0x50 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383260]  [<ffffffffa0d8479a>] zvol_write+0x9a/0x480 [zfs]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383264]  [<ffffffff816ea120>] ? __schedule+0x3b0/0x7e0
Nov 21 00:08:25 ubuntu27 kernel: [480028.383272]  [<ffffffffa03646c7>] taskq_thread+0x237/0x4b0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383276]  [<ffffffff81091420>] ? finish_task_switch+0x50/0xf0
Nov 21 00:08:25 ubuntu27 kernel: [480028.383279]  [<ffffffff81094950>] ? wake_up_state+0x20/0x20
Nov 21 00:08:25 ubuntu27 kernel: [480028.383287]  [<ffffffffa0364490>] ? taskq_cancel_id+0x1f0/0x1f0 [spl]
Nov 21 00:08:25 ubuntu27 kernel: [480028.383291]  [<ffffffff810847b0>] kthread+0xc0/0xd0
Nov 21 00:08:25 ubuntu27 kernel: [480028.383295]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
Nov 21 00:08:25 ubuntu27 kernel: [480028.383298]  [<ffffffff816f516c>] ret_from_fork+0x7c/0xb0
Nov 21 00:08:25 ubuntu27 kernel: [480028.383302]  [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120

pool/volume information:
6 disks format 1 raidz vdev
pool consist of 4 vdevs
volume export from this pool

are there some problems when dmu assign txg ?

or these may be caused by other reasons.

thanks for replying!

[GregorKopka]

What setting for PREEMT do you have in your kernel?

Should you have CONFIG_PREEMPT_NONE=y then try to set CONFIG_PREEMPT_VOLUNTARY=y instead, this fixed lockups (which iirc looked like yours) on my machines.

[Vicente-Cheng]

My PREEMT setting now is following:

# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y

Is this setting the same in your machines?

[GregorKopka]

@Vicente-Cheng yes, this is the setting that lead to my machines working without hanging on stacktraces similar to the one you posted. YMMV but i hope this helps.

[behlendorf]

Closing, setting CONFIG_PREEMPT_VOLUNTARY=y resolved the issue.