Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix misuse of input argument in traverse_visitbp #3348

Closed
wants to merge 1 commit into from

Conversation

tuxoko
Copy link
Contributor

@tuxoko tuxoko commented Apr 26, 2015

In traverse_visitbp(), the input argument dnp is modified in the middle to
point to a temporary buffer. Originally this doesn't matter, because no user
of TRAVERSE_POST dereferences it. However, in fbeddd6 a piece of code is added
dereferencing dnp after the modification, creating a possible bug.

We fix this by creating a new local variable cdnp for the DMU_OT_DNODE case,
so we don't modify the input argument. Also we introduce different local
variables in the DMU_OT_OBJSET case to prevent confusion between the input
argument.

Signed-off-by: Chunwei Chen tuxoko@gmail.com

@tuxoko
Copy link
Contributor Author

tuxoko commented Apr 26, 2015

fix #2060

I edited it using github online editor, so I didn't even compile test it.
Better wait for the buildbot to test this.

@dweeezil
Copy link
Contributor

@tuxoko Good idea.

As I suspect you're aware, the code is identical in both illumos and FreeBSD. I'm still trying to get a grip on the actual bug here. Specifically, why it's bad that the modified dnp is used by the newly-added IO error handling code but is ok to use in the TRAVERSE_POST callback above.

@tuxoko
Copy link
Contributor Author

tuxoko commented Apr 26, 2015

@dweeezil
I could guess that illumos and FreeBSD have the same problem, but I've never looked into them.

It's not that it's ok to use in TRAVERSE_POST. It's that no user of TRAVERSE_POST use it.
As for the added code, I believe that the possibility of a bug showing is pretty low, because dn_indblkshift of the dnodes are most likely the same. However, if the dnode in DMU_OT_DNODE is very unfortunately evicted, this would be a access freed memory bug.

In traverse_visitbp(), the input argument dnp is modified in the middle to
point to a temporary buffer. Originally this doesn't matter, because no user
of TRAVERSE_POST dereferences it. However, in fbeddd6 a piece of code is added
dereferencing dnp after the modification, creating a possible bug.

We fix this by creating a new local variable cdnp for the DMU_OT_DNODE case,
so we don't modify the input argument. Also we introduce different local
variables in the DMU_OT_OBJSET case to prevent confusion between the input
argument.

Signed-off-by: Chunwei Chen <tuxoko@gmail.com>
@dweeezil
Copy link
Contributor

@tuxoko OK, that's clear. I was just pointed in the direction #2060 which for some reason I forgot about and has a similar line of discussion and which would likely also be fixed by your patch.

@ryao
Copy link
Contributor

ryao commented Apr 28, 2015

@tuxoko Nice find. I agree with your analysis of the issue. This looks good to me.

@behlendorf behlendorf added this to the 0.6.5 milestone Apr 28, 2015
@behlendorf
Copy link
Contributor

LGTM, merged as:

ecfb0b5 Fix misuse of input argument in traverse_visitbp

@behlendorf behlendorf closed this Apr 28, 2015
@dweeezil dweeezil mentioned this pull request Apr 28, 2015
@stevenh
Copy link
Contributor

stevenh commented Apr 28, 2015

Thanks for the heads up @kpande committed to FreeBSD as ref 282205.

@eolson78
Copy link

I can say that I have experienced symptoms of this issue as seen below

Apr 13 07:30:10 ip-172-26-0-157 kernel: INFO: task spl_system_task:2179 blocked for more than 120 seconds.
Apr 13 07:30:10 ip-172-26-0-157 kernel: Tainted: P O 3.14.4 #1
Apr 13 07:30:10 ip-172-26-0-157 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 13 07:30:10 ip-172-26-0-157 kernel: spl_system_task D ffffffff8181d000 0 2179 2 0x00000000
Apr 13 07:30:10 ip-172-26-0-157 kernel: ffff8800eba87648 0000000000000046 0000000000040000 ffff8800eba86010
Apr 13 07:30:10 ip-172-26-0-157 kernel: 0000000000014440 0000000000014440 ffff8800ebac8ef0 ffff880f05caaef0
Apr 13 07:30:10 ip-172-26-0-157 kernel: ffff8800eba87658 ffff880af695d630 ffff880af695d600 ffff8800eba87688
Apr 13 07:30:10 ip-172-26-0-157 kernel: Call Trace:
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] schedule+0x29/0x70
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] cv_wait_common+0x135/0x1c0 [spl]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? bit_waitqueue+0xe0/0xe0
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] __cv_wait+0x15/0x20 [spl]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_prefetcher+0xb5/0x170 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x2ce/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x56d/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x610/0x6f0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? sched_clock_cpu+0x98/0xc0
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] traverse_prefetch_thread+0x83/0xe0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? traverse_prefetch_thread+0xe0/0xe0 [zfs]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] taskq_thread+0x22d/0x4f0 [spl]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? try_to_wake_up+0x230/0x230
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? task_expire+0x120/0x120 [spl]
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] kthread+0xce/0xf0
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ret_from_fork+0x7c/0xb0
Apr 13 07:30:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:32:10 ip-172-26-0-157 kernel: INFO: task spl_system_task:2179 blocked for more than 120 seconds.
Apr 13 07:32:10 ip-172-26-0-157 kernel: Tainted: P O 3.14.4 #1
Apr 13 07:32:10 ip-172-26-0-157 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 13 07:32:10 ip-172-26-0-157 kernel: spl_system_task D ffffffff8181d000 0 2179 2 0x00000000
Apr 13 07:32:10 ip-172-26-0-157 kernel: ffff8800eba87648 0000000000000046 0000000000040000 ffff8800eba86010
Apr 13 07:32:10 ip-172-26-0-157 kernel: 0000000000014440 0000000000014440 ffff8800ebac8ef0 ffff880f05caaef0
Apr 13 07:32:10 ip-172-26-0-157 kernel: ffff8800eba87658 ffff880af695d630 ffff880af695d600 ffff8800eba87688
Apr 13 07:32:10 ip-172-26-0-157 kernel: Call Trace:
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] schedule+0x29/0x70
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] cv_wait_common+0x135/0x1c0 [spl]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? bit_waitqueue+0xe0/0xe0
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] __cv_wait+0x15/0x20 [spl]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_prefetcher+0xb5/0x170 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x2ce/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x56d/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x610/0x6f0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? sched_clock_cpu+0x98/0xc0
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] traverse_prefetch_thread+0x83/0xe0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? traverse_prefetch_thread+0xe0/0xe0 [zfs]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] taskq_thread+0x22d/0x4f0 [spl]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? try_to_wake_up+0x230/0x230
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? task_expire+0x120/0x120 [spl]
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] kthread+0xce/0xf0
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ret_from_fork+0x7c/0xb0
Apr 13 07:32:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:36:10 ip-172-26-0-157 kernel: INFO: task spl_system_task:2172 blocked for more than 120 seconds.
Apr 13 07:36:10 ip-172-26-0-157 kernel: Tainted: P O 3.14.4 #1
Apr 13 07:36:10 ip-172-26-0-157 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 13 07:36:10 ip-172-26-0-157 kernel: spl_system_task D ffffffff8181d000 0 2172 2 0x00000000
Apr 13 07:36:10 ip-172-26-0-157 kernel: ffff8800eba39648 0000000000000046 0000000000040000 ffff8800eba38010
Apr 13 07:36:10 ip-172-26-0-157 kernel: 0000000000014440 0000000000014440 ffff880f03adee70 ffff880f05cb2f30
Apr 13 07:36:10 ip-172-26-0-157 kernel: ffff8800eba39658 ffff880caff686f0 ffff880caff686c0 ffff8800eba39688
Apr 13 07:36:10 ip-172-26-0-157 kernel: Call Trace:
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] schedule+0x29/0x70
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] cv_wait_common+0x135/0x1c0 [spl]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? zio_wait_for_children+0x7a/0xa0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? bit_waitqueue+0xe0/0xe0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] __cv_wait+0x15/0x20 [spl]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_prefetcher+0xb5/0x170 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x2ce/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x56d/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x610/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? sched_clock_cpu+0x98/0xc0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_prefetch_thread+0x83/0xe0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? traverse_prefetch_thread+0xe0/0xe0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] taskq_thread+0x22d/0x4f0 [spl]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? try_to_wake_up+0x230/0x230
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? task_expire+0x120/0x120 [spl]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] kthread+0xce/0xf0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ret_from_fork+0x7c/0xb0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:36:10 ip-172-26-0-157 kernel: INFO: task spl_system_task:2175 blocked for more than 120 seconds.
Apr 13 07:36:10 ip-172-26-0-157 kernel: Tainted: P O 3.14.4 #1
Apr 13 07:36:10 ip-172-26-0-157 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 13 07:36:10 ip-172-26-0-157 kernel: spl_system_task D ffffffff8181d000 0 2175 2 0x00000000
Apr 13 07:36:10 ip-172-26-0-157 kernel: ffff8800ebabd648 0000000000000046 0000000000040000 ffff8800ebabc010
Apr 13 07:36:10 ip-172-26-0-157 kernel: 0000000000014440 0000000000014440 ffff880f038e4250 ffff880f05cbaf70
Apr 13 07:36:10 ip-172-26-0-157 kernel: ffff8800ebabd658 ffff88071ff04270 ffff88071ff04240 ffff8800ebabd688
Apr 13 07:36:10 ip-172-26-0-157 kernel: Call Trace:
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] schedule+0x29/0x70
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] cv_wait_common+0x135/0x1c0 [spl]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? bit_waitqueue+0xe0/0xe0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] __cv_wait+0x15/0x20 [spl]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_prefetcher+0xb5/0x170 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x2ce/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x56d/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x610/0x6f0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] traverse_prefetch_thread+0x83/0xe0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? traverse_prefetch_thread+0xe0/0xe0 [zfs]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] taskq_thread+0x22d/0x4f0 [spl]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? try_to_wake_up+0x230/0x230
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? task_expire+0x120/0x120 [spl]
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] kthread+0xce/0xf0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ret_from_fork+0x7c/0xb0
Apr 13 07:36:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:38:10 ip-172-26-0-157 kernel: INFO: task spl_system_task:2178 blocked for more than 120 seconds.
Apr 13 07:38:10 ip-172-26-0-157 kernel: Tainted: P O 3.14.4 #1
Apr 13 07:38:10 ip-172-26-0-157 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 13 07:38:10 ip-172-26-0-157 kernel: spl_system_task D ffffffff8181d000 0 2178 2 0x00000000
Apr 13 07:38:10 ip-172-26-0-157 kernel: ffff8800ebbdd748 0000000000000046 ffff8800ebbdd738 ffff8800ebbdc010
Apr 13 07:38:10 ip-172-26-0-157 kernel: 0000000000014440 0000000000014440 ffff8800ebb3c110 ffff880f05cc2fb0
Apr 13 07:38:10 ip-172-26-0-157 kernel: ffff8800ebbdd758 ffff88049e89a9b0 ffff88049e89a980 ffff8800ebbdd788
Apr 13 07:38:10 ip-172-26-0-157 kernel: Call Trace:
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] schedule+0x29/0x70
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] cv_wait_common+0x135/0x1c0 [spl]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? add_reference+0xf1/0x120 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? bit_waitqueue+0xe0/0xe0
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] __cv_wait+0x15/0x20 [spl]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_prefetcher+0xb5/0x170 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? arc_buf_remove_ref+0xe6/0x130 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x2ce/0x6f0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x610/0x6f0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? sched_clock_cpu+0x98/0xc0
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] traverse_prefetch_thread+0x83/0xe0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? traverse_prefetch_thread+0xe0/0xe0 [zfs]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] taskq_thread+0x22d/0x4f0 [spl]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? try_to_wake_up+0x230/0x230
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? task_expire+0x120/0x120 [spl]
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] kthread+0xce/0xf0
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ret_from_fork+0x7c/0xb0
Apr 13 07:38:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:40:10 ip-172-26-0-157 kernel: INFO: task spl_system_task:2178 blocked for more than 120 seconds.
Apr 13 07:40:10 ip-172-26-0-157 kernel: Tainted: P O 3.14.4 #1
Apr 13 07:40:10 ip-172-26-0-157 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 13 07:40:10 ip-172-26-0-157 kernel: spl_system_task D ffffffff8181d000 0 2178 2 0x00000000
Apr 13 07:40:10 ip-172-26-0-157 kernel: ffff8800ebbdd748 0000000000000046 ffff8800ebbdd738 ffff8800ebbdc010
Apr 13 07:40:10 ip-172-26-0-157 kernel: 0000000000014440 0000000000014440 ffff8800ebb3c110 ffff880f05cc2fb0
Apr 13 07:40:10 ip-172-26-0-157 kernel: ffff8800ebbdd758 ffff88049e89a9b0 ffff88049e89a980 ffff8800ebbdd788
Apr 13 07:40:10 ip-172-26-0-157 kernel: Call Trace:
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] schedule+0x29/0x70
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] cv_wait_common+0x135/0x1c0 [spl]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? add_reference+0xf1/0x120 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? bit_waitqueue+0xe0/0xe0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] __cv_wait+0x15/0x20 [spl]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_prefetcher+0xb5/0x170 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? arc_buf_remove_ref+0xe6/0x130 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x2ce/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x610/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? sched_clock_cpu+0x98/0xc0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_prefetch_thread+0x83/0xe0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? traverse_prefetch_thread+0xe0/0xe0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] taskq_thread+0x22d/0x4f0 [spl]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? try_to_wake_up+0x230/0x230
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? task_expire+0x120/0x120 [spl]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] kthread+0xce/0xf0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ret_from_fork+0x7c/0xb0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:40:10 ip-172-26-0-157 kernel: INFO: task spl_system_task:2179 blocked for more than 120 seconds.
Apr 13 07:40:10 ip-172-26-0-157 kernel: Tainted: P O 3.14.4 #1
Apr 13 07:40:10 ip-172-26-0-157 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 13 07:40:10 ip-172-26-0-157 kernel: spl_system_task D ffffffff8181d000 0 2179 2 0x00000000
Apr 13 07:40:10 ip-172-26-0-157 kernel: ffff8800eba87748 0000000000000046 0000000000000000 ffff8800eba86010
Apr 13 07:40:10 ip-172-26-0-157 kernel: 0000000000014440 0000000000014440 ffff8800ebac8ef0 ffff880f05caaef0
Apr 13 07:40:10 ip-172-26-0-157 kernel: ffff8800eba87758 ffff880e7ece5bf0 ffff880e7ece5bc0 ffff8800eba87788
Apr 13 07:40:10 ip-172-26-0-157 kernel: Call Trace:
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] schedule+0x29/0x70
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] cv_wait_common+0x135/0x1c0 [spl]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? zio_data_buf_free+0x30/0x30 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? bit_waitqueue+0xe0/0xe0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] __cv_wait+0x15/0x20 [spl]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_prefetcher+0xb5/0x170 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? arc_buf_remove_ref+0xe6/0x130 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x2ce/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x610/0x6f0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? sched_clock_cpu+0x98/0xc0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] traverse_prefetch_thread+0x83/0xe0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? traverse_prefetch_thread+0xe0/0xe0 [zfs]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] taskq_thread+0x22d/0x4f0 [spl]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? try_to_wake_up+0x230/0x230
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? task_expire+0x120/0x120 [spl]
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] kthread+0xce/0xf0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ret_from_fork+0x7c/0xb0
Apr 13 07:40:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:42:10 ip-172-26-0-157 kernel: INFO: task spl_system_task:2175 blocked for more than 120 seconds.
Apr 13 07:42:10 ip-172-26-0-157 kernel: Tainted: P O 3.14.4 #1
Apr 13 07:42:10 ip-172-26-0-157 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 13 07:42:10 ip-172-26-0-157 kernel: spl_system_task D ffffffff8181d000 0 2175 2 0x00000000
Apr 13 07:42:10 ip-172-26-0-157 kernel: ffff8800ebabd748 0000000000000046 ffff8800ebabd738 ffff8800ebabc010
Apr 13 07:42:10 ip-172-26-0-157 kernel: 0000000000014440 0000000000014440 ffff880f038e4250 ffff880f05cb2f30
Apr 13 07:42:10 ip-172-26-0-157 kernel: ffff8800ebabd758 ffff8806e527e2f0 ffff8806e527e2c0 ffff8800ebabd788
Apr 13 07:42:10 ip-172-26-0-157 kernel: Call Trace:
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] schedule+0x29/0x70
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] cv_wait_common+0x135/0x1c0 [spl]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? add_reference+0xf1/0x120 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? bit_waitqueue+0xe0/0xe0
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] __cv_wait+0x15/0x20 [spl]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_prefetcher+0xb5/0x170 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? arc_buf_remove_ref+0xe6/0x130 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? arc_buf_remove_ref+0x1/0x130 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x2ce/0x6f0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x458/0x6f0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_dnode+0x6c/0xf0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_visitbp+0x610/0x6f0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? sched_clock_cpu+0x98/0xc0
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? __switch_to+0x17d/0x5e0
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] traverse_prefetch_thread+0x83/0xe0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? traverse_prefetch_thread+0xe0/0xe0 [zfs]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] taskq_thread+0x22d/0x4f0 [spl]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? try_to_wake_up+0x230/0x230
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? task_expire+0x120/0x120 [spl]
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] kthread+0xce/0xf0
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ret_from_fork+0x7c/0xb0
Apr 13 07:42:10 ip-172-26-0-157 kernel: [] ? kthread_freezable_should_stop+0x70/0x70

I have spoke to @dweeezil and @ryao about this. We have applied this pull request and it appears to resolve the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants