hung xfs on zvol

[ChessSpider]

System information

Type	Version/Name
Distribution Name	Ubuntu 18.10
Distribution Version
Linux Kernel	4.18.0-16-generic #17-Ubuntu SMP
Architecture	x86_64 x86_64 x86_64 GNU/Linux
ZFS Version	0.7.13-1
SPL Version	0.7.9-3ubuntu2 😱

Describe the problem you're observing

zfs fs hung, ssh still works

Describe how to reproduce the problem

xfs on zfs, bulk write to mongodb.

Include any warning/errors/backtraces from the system logs

[Apr14 01:03] INFO: task kswapd0:81 blocked for more than 120 seconds.
[  +0.001917]       Tainted: P           OE     4.18.0-16-generic #17-Ubuntu
[  +0.001909] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  +0.001954] kswapd0         D    0    81      2 0x80000000
[  +0.000004] Call Trace:
[  +0.000011]  __schedule+0x29e/0x840
[  +0.000004]  schedule+0x2c/0x80
[  +0.000004]  schedule_timeout+0x258/0x360
[  +0.000005]  ? blk_finish_plug+0x26/0x34
[  +0.000066]  ? _xfs_buf_ioapply+0x31d/0x480 [xfs]
[  +0.000008]  wait_for_completion+0xb7/0x140
[  +0.000009]  ? wake_up_q+0x80/0x80
[  +0.000047]  ? xfs_bwrite+0x24/0x60 [xfs]
[  +0.000045]  xfs_buf_submit_wait+0x7f/0x210 [xfs]
[  +0.000043]  xfs_bwrite+0x24/0x60 [xfs]
[  +0.000047]  xfs_reclaim_inode+0x32a/0x350 [xfs]
[  +0.000045]  xfs_reclaim_inodes_ag+0x1f5/0x380 [xfs]
[  +0.000005]  ? check_preempt_curr+0x79/0x90
[  +0.000003]  ? ttwu_do_wakeup+0x1e/0x140
[  +0.000003]  ? ttwu_do_activate+0x67/0x90
[  +0.000004]  ? try_to_wake_up+0x59/0x430
[  +0.000004]  ? wake_up_process+0x15/0x20
[  +0.000044]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[  +0.000045]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
[  +0.000004]  super_cache_scan+0x160/0x1b0
[  +0.000004]  shrink_slab.part.45+0x1a9/0x3b0
[  +0.000004]  shrink_slab+0x29/0x30
[  +0.000003]  shrink_node+0xf5/0x4b0
[  +0.000004]  balance_pgdat+0x1fb/0x350
[  +0.000007]  kswapd+0x228/0x410
[  +0.000006]  ? wait_woken+0x80/0x80
[  +0.000005]  kthread+0x120/0x140
[  +0.000005]  ? balance_pgdat+0x350/0x350
[  +0.000002]  ? kthread_bind+0x40/0x40
[  +0.000003]  ret_from_fork+0x35/0x40
[  +0.000011] INFO: task zvol:914 blocked for more than 120 seconds.
[  +0.001702]       Tainted: P           OE     4.18.0-16-generic #17-Ubuntu
[  +0.000988] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  +0.000991] zvol            D    0   914      2 0x80000000
[  +0.000003] Call Trace:
[  +0.000005]  __schedule+0x29e/0x840
[  +0.000003]  ? __wake_up_common+0x78/0x140
[  +0.000004]  schedule+0x2c/0x80
[  +0.000010]  cv_wait_common+0x101/0x140 [spl]
[  +0.000003]  ? wait_woken+0x80/0x80
[  +0.000008]  __cv_wait+0x15/0x20 [spl]
[  +0.000066]  txg_wait_open+0xb0/0x100 [zfs]
[  +0.000045]  dmu_tx_wait+0x3af/0x3c0 [zfs]
[  +0.000031]  dmu_tx_assign+0x161/0x460 [zfs]
[  +0.000020]  zvol_write+0x176/0x620 [zfs]
[  +0.000003]  taskq_thread+0x2ec/0x4d0 [spl]
[  +0.000001]  ? wake_up_q+0x80/0x80
[  +0.000002]  kthread+0x120/0x140
[  +0.000002]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[  +0.000001]  ? kthread_bind+0x40/0x40
[  +0.000001]  ret_from_fork+0x35/0x40
[  +0.000046] INFO: task WTJourn.Flusher:7571 blocked for more than 120 seconds.
[  +0.001320]       Tainted: P           OE     4.18.0-16-generic #17-Ubuntu
[  +0.001774] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  +0.000724] WTJourn.Flusher D    0  7571   6638 0x00000120
[  +0.000001] Call Trace:
[  +0.000002]  __schedule+0x29e/0x840

Additional info:

root@passdb:/home/john# free -m
              total        used        free      shared  buff/cache   available
Mem:          32133       27090         345           1        4698        4588
Swap:         32767         112       32655

root@passdb:/home/john#  cat /proc/sys/vm/swappiness
60

root@passdb:/home/john# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
passdb   928G   481G   447G         -    48%    51%  1.00x  ONLINE  -
zfs1    2.72T   936G  1.80T         -    25%    33%  1.00x  ONLINE  -

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.