Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ztest fails with "No such hold ... on refcount ..." in arc_free_data_impl->refcount_remove_many #7219

Closed
ofaaland opened this issue Feb 22, 2018 · 0 comments
Closed

ztest fails with "No such hold ... on refcount ..." in arc_free_data_impl->refcount_remove_many #7219

ofaaland opened this issue Feb 22, 2018 · 0 comments
Labels
Component: Test Suite Indicates an issue with the test framework or a test case

Comments

@ofaaland
Copy link
Contributor

System information

Type Version/Name
Distribution Name CentOS
Distribution Version 7 test builder
Linux Kernel
Architecture x86_64
ZFS Version dfcb779 on top of master 0d398b2
SPL Version master 378c6ed

Describe the problem you're observing

ztest failure with message "error: No such hold 0x7fab9a99f090 on refcount 7fabe6445590" and backtrace

http://build.zfsonlinux.org/builders/CentOS%207%20x86_64%20%28TEST%29/builds/1982/steps/shell_5/logs/1.ztest.gdb

Describe how to reproduce the problem

Occurred during automated testing of PR #7190

Include any warning/errors/backtraces from the system logs

#3  0x00007fabe5f41019 in panic (fmt=fmt@entry=0x7fabe610d290 "No such hold %p on refcount %llx") at kernel.c:789
#4  0x00007fabe5fd6651 in refcount_remove_many (rc=rc@entry=0x7fabe6445590 <ARC_anon+432>, number=number@entry=512, holder=holder@entry=0x7fab9a99f090) at ../../module/zfs/refcount.c:194
#5  0x00007fabe5f53d99 in arc_free_data_impl (hdr=0x7fab9a99f090, size=512, tag=0x7fab9a99f090) at ../../module/zfs/arc.c:5462
#6  0x00007fabe5f53e6f in arc_free_data_abd (hdr=<optimized out>, abd=0x7fab99929f90, size=<optimized out>, tag=<optimized out>) at ../../module/zfs/arc.c:5427
#7  0x00007fabe5f54614 in arc_hdr_free_abd (hdr=0x7fab9a99f090, free_rdata=B_TRUE) at ../../module/zfs/arc.c:3285
#8  0x00007fabe5f5da5b in arc_write (pio=pio@entry=0x7faba80c67f0, spa=0x1165c50, txg=txg@entry=349, bp=bp@entry=0x7fabd4069570, buf=buf@entry=0x7fabd4267400, l2arc=<optimized out>, zp=zp@entry=0x7fabda9860b0, ready=ready@entry=0x7fabe5f639e0 <dbuf_write_ready>, children_ready=0x0, physdone=physdone@entry=0x7fabe5f64b40 <dbuf_write_physdone>, done=done@entry=0x7fabe5f6d040 <dbuf_write_done>, private=private@entry=0x7fabd5cc1440, priority=priority@entry=ZIO_PRIORITY_ASYNC_WRITE, zio_flags=zio_flags@entry=0, zb=zb@entry=0x7fabda986090) at ../../module/zfs/arc.c:7032
#9  0x00007fabe5f667b6 in dbuf_write (dr=dr@entry=0x7fabd4069530, data=0x7fabd4267400, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:4160
#10 0x00007fabe5f6d9c7 in dbuf_sync_leaf (dr=0x7fabd4069530, tx=0x158c4d0) at ../../module/zfs/dbuf.c:3695
#11 0x00007fabe5f6e405 in dbuf_sync_list (list=list@entry=0x7fabd44d9430, level=<optimized out>, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3740
#12 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x7fabd44d9340, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#13 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x7fabd40d84b0, level=1, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#14 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x7fabd40d83c0, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#15 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x7fabd445f930, level=2, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#16 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x7fabd445f840, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#17 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x7fabd41e12d0, level=3, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#18 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x7fabd41e11e0, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#19 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x7fabd400b620, level=4, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#20 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x7fabd400b530, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#21 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x7fabd41d8830, level=5, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#22 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x7fabd41d8740, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#23 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x7fabd461a470, level=6, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#24 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x7fabd461a380, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#25 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x132bf60, level=7, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#26 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x132be70, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#27 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x133ac00, level=8, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#28 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x133ab10, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#29 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x14cfba0, level=9, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#30 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x14cfab0, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#31 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x14b66e0, level=10, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#32 0x00007fabe5f6e641 in dbuf_sync_indirect (dr=dr@entry=0x14b65f0, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3518
#33 0x00007fabe5f6e3e0 in dbuf_sync_list (list=list@entry=0x12cc3a0, level=11, tx=tx@entry=0x158c4d0) at ../../module/zfs/dbuf.c:3738
#34 0x00007fabe5f97487 in dnode_sync (dn=dn@entry=0x12cc210, tx=tx@entry=0x158c4d0) at ../../module/zfs/dnode_sync.c:760
#35 0x00007fabe5f7a224 in dmu_objset_sync_dnodes (list=list@entry=0x1420a50, tx=0x158c4d0) at ../../module/zfs/dmu_objset.c:1423
#36 0x00007fabe5f7a303 in sync_dnodes_task (arg=0x17ff470) at ../../module/zfs/dmu_objset.c:1493
#37 0x00007fabe5f41b44 in taskq_thread (arg=0x117aa70) at taskq.c:233
#38 0x00007fabe4fa3dc5 in start_thread () from /lib64/libpthread.so.0
#39 0x00007fabe4cd276d in clone () from /lib64/libc.so.6
@behlendorf behlendorf added the Component: Test Suite Indicates an issue with the test framework or a test case label Feb 22, 2018
behlendorf added a commit to behlendorf/zfs that referenced this issue Oct 8, 2018
@behlendorf
Add zfs_refcount_transfer_ownership_many()
a21a1f1 
When debugging is enabled and a zfs_refcount_t contains multiple holders
using the same key, but different ref_counts, the wrong reference_t may
be transferred.  Add a zfs_refcount_transfer_ownership_many() function,
like the existing zfs_refcount_*_many() functions, to match and transfer
the correct refcount_t;

This issue may occur when using encryption with refcount debugging
enabled.  An arc_buf_hdr_t can have references for both the
hdr->b_l1hdr.b_pabd and hdr->b_crypt_hdr.b_rabd which both use the
hdr as the reference holder.  When unsharing the buffer the p_abd
should be transferred.

This issue does not impact production builds because refcount holders
are not tracked.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#7219
behlendorf added a commit to behlendorf/zfs that referenced this issue Oct 8, 2018
@behlendorf
Fix arc_relase() refcount
4c4fb62 
Update arc_release to use arc_buf_size().  This hunk was accidentally
dropped when porting compressed send/recv, 2aa3438.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
TEST_ZTEST_TIMEOUT=3600
Issue openzfs#7219
@behlendorf behlendorf mentioned this issue Oct 8, 2018
Closed
13 tasks
GregorKopka pushed a commit to GregorKopka/zfs that referenced this issue Jan 7, 2019
@behlendorf
Add zfs_refcount_transfer_ownership_many()
05900e2 
When debugging is enabled and a zfs_refcount_t contains multiple holders
using the same key, but different ref_counts, the wrong reference_t may
be transferred.  Add a zfs_refcount_transfer_ownership_many() function,
like the existing zfs_refcount_*_many() functions, to match and transfer
the correct refcount_t;

This issue may occur when using encryption with refcount debugging
enabled.  An arc_buf_hdr_t can have references for both the
hdr->b_l1hdr.b_pabd and hdr->b_crypt_hdr.b_rabd both of which use
the hdr as the reference holder.  When unsharing the buffer the
p_abd should be transferred.

This issue does not impact production builds because refcount holders
are not tracked.

Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#7219
Closes openzfs#8000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Test Suite Indicates an issue with the test framework or a test case
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ARC refcount fixes behlendorf/zfs
2 participants
@behlendorf @ofaaland