-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PANIC at range_tree.c:368:range_tree_remove_impl() #11893
Comments
I have been investigating a bit this issue and came with a quick and maybe not so dirty fix. First, looking at the code, I was not able to track the code path the produces the trace above. Ie. I didn't find how and why Anyway, looking more closely at the error, it happens when an ASSERT detects an attempt to free a zero-sized segment. My impression is that the caller (which I was not able to identify), skipped a zero-size test and didnt prevent the useless call. As a temporary mitigation, I tested the following fix, which replaces the original ASSERT with a call to
As expected, when the situation occurs, I now have the following warning messages instead of the ASSERT induced PANIC
I think this fix may be acceptable because, the severity of the error does not seem to deserve such a strong response as a PANIC. Still it allows for tracking the issue since it still produces a trace in logs. This fix is still in operation (an will be as long as no better is found), do not hesitate to request action for investigating further. |
Seeing the same or similar panic on ZoL 0.8.3 on the CentOS 8 linux 4.18.0-147.5.1.el8_1.x86_64 kernel. Happens during a
|
I have implemented and deployed the small hack in my comment above and tested it for a few days. After this hack, I now have WARNINGs in logs instead of halting all ZFS operations. Here is a log of the WARNINGs a got in the recent days (with the small patch above). # dmesg -e | grep zfs
[Apr21 15:51] WARNING: zfs: freeing 0-sized segment (offset=618475339776 size=0)
[ +0.000012] WARNING: zfs: freeing 0-sized segment (offset=1013612331008 size=0)
[Apr21 19:41] WARNING: zfs: freeing 0-sized segment (offset=1219770712064 size=0)
[ +0.000014] WARNING: zfs: freeing 0-sized segment (offset=1340029796352 size=0)
[Apr23 04:46] WARNING: zfs: freeing 0-sized segment (offset=2199023505408 size=0)
[ +0.000013] WARNING: zfs: freeing 0-sized segment (offset=618475290624 size=0)
[Apr23 10:11] WARNING: zfs: freeing 0-sized segment (offset=2130303778816 size=0)
[ +0.000012] WARNING: zfs: freeing 0-sized segment (offset=2164663517184 size=0)
[Apr23 15:51] WARNING: zfs: freeing 0-sized segment (offset=2130303778816 size=0)
[ +0.000014] WARNING: zfs: freeing 0-sized segment (offset=2164663517184 size=0)
[Apr23 16:46] WARNING: zfs: freeing 0-sized segment (offset=618475290624 size=0)
[ +0.000015] WARNING: zfs: freeing 0-sized segment (offset=1889785806848 size=0)
[Apr24 18:16] WARNING: zfs: freeing 0-sized segment (offset=2336462209024 size=0)
[ +0.000013] WARNING: zfs: freeing 0-sized segment (offset=2302102470656 size=0)
[Apr24 21:26] WARNING: zfs: freeing 0-sized segment (offset=2405181685760 size=0)
[ +0.000023] WARNING: zfs: freeing 0-sized segment (offset=2164663517184 size=0)
[Apr25 08:16] WARNING: zfs: freeing 0-sized segment (offset=2903397892096 size=0)
[ +0.000018] WARNING: zfs: freeing 0-sized segment (offset=2370821947392 size=0)
[Apr25 09:56] WARNING: zfs: freeing 0-sized segment (offset=2800318676992 size=0)
[ +0.000074] WARNING: zfs: freeing 0-sized segment (offset=2834678841344 size=0)
[Apr25 14:15] WARNING: zfs: freeing 0-sized segment (offset=2920577925120 size=0)
[ +0.000014] WARNING: zfs: freeing 0-sized segment (offset=2989297238016 size=0)
[Apr25 21:31] WARNING: zfs: freeing 0-sized segment (offset=2869038153728 size=0)
[ +0.000013] WARNING: zfs: freeing 0-sized segment (offset=2765958938624 size=0)
[Apr25 22:26] WARNING: zfs: freeing 0-sized segment (offset=2869038153728 size=0)
[ +0.000020] WARNING: zfs: freeing 0-sized segment (offset=2765958938624 size=0)
[Apr25 23:25] WARNING: zfs: freeing 0-sized segment (offset=2869038153728 size=0)
[ +0.000015] WARNING: zfs: freeing 0-sized segment (offset=2765958938624 size=0)
[Apr26 08:10] WARNING: zfs: freeing 0-sized segment (offset=3075196583936 size=0)
[ +0.000021] WARNING: zfs: freeing 0-sized segment (offset=3401614327808 size=0)
[Apr26 12:31] WARNING: zfs: freeing 0-sized segment (offset=3539053117440 size=0)
[ +0.000014] WARNING: zfs: freeing 0-sized segment (offset=3332895277056 size=0)
[Apr27 01:36] WARNING: zfs: freeing 0-sized segment (offset=3075196829696 size=0)
[ +0.000015] WARNING: zfs: freeing 0-sized segment (offset=3401614327808 size=0)
[Apr27 09:11] WARNING: zfs: freeing 0-sized segment (offset=188978839552 size=0)
[ +0.000015] WARNING: zfs: freeing 0-sized segment (offset=3624952397824 size=0) Worth to mention: |
Hello, hitting this issue too - twice a month or more.
Scrubs are clean, the entire zfs system doesn't appear to be locked up certain operations involving the pool do not work, such as a rollback on the receiving pool is hung. However a send job from the original sending pool is working just fine to another off-site receiving pool. Edit: After a bit more testing, it looks like ZFS operations such as create, destroy, rollback, etc do not work on the local receiving pool. Local sending pool is acting normal. So I assume whatever is stuck here, its stuck on the receive side. Edit2: Looks like the receive process is still running, 100% CPU. Strace shows nothing at all on the process. Edit3: Last night I was able to reproduce this three times in a row, same send/receive job caused a panic. This morning I manually sent the remaining datasets to the receiving pool one at a time (instead of the recursive send) and they all went okay. Created a new recursive snapshot to try the entire job again and this time it sent normally. I'm not sure what that means, if there was an issue with the data you'd think that I would have hit it running the datasets one at a time but that didn't happen. Whatever this condition was, it persisted across reboots and it would panic at roughly the same time after I started the recursive send/receive job. |
I just hit this same problem with a zfs recv. Running proxmox 7.0, zfs 2.0.5. |
I am having this issue on zfs 2.0.6 on ubuntu 21.10, with receiving encrypted raw datasets. |
I don't know enough to conclude whether it is this exact one, but today while using syncoid I experienced a similar issue. Debian 11, dmesg
I was running two sync processes at once and indeed 2 cores are at 100% iowait |
I woke up to the same/similar issue this morning on Debian Bullseye running Linux
Unfortunately, I didn't have a serial console hooked up so no call trace :(. I do believe replication was in process when it happened. Anywhere between 1-4 datasets were probably being |
I had a lot of those at one point when I was using zfs in proxmox LXC containers and I did not pay attention to the fact that zfsutils-linux package in CTs did not match exactly the zfs kernel module in HV. I was convinced that the ioctl protocol between the zfs user-land command and kernel module would be able to detect and deal with such mismatch, but on contrary it seems very sensitive.
I first fixed it the hard way, by selecting and freezing a ZFS version by recompiling the whole zfs project by hand in the proxmox HV, and deploying the libs and user-land commands manually in CTs using bind mounts. It worked well, was very stable, but I was not happy with idea of not following ZFS updates.
I now reverted using the mainstream version and proxmox updates, which I manage to keep always identical in CTs and HV (using unattended-upgrades), and have not experienced the issue since then.
Although I now have another issue which I believe is also related to version mismatch: I have a RAIDZ1 pool suspended without any reported I/O fault occurring on vdevs, apparently due to to a txg timeout.
I think it may happen following automatic updates with delayed reboot (Ie zfs commands and libs get updated, but ZFS kernel is still using the old module version until next reboot, so we end up with a version mismatch between user-land and kernel). A reboot fixes it up, but I still have to find a way to hook an automatic reboot when zfs package is updated.
Olivier Dalle
Co-Fondateur
Inspeere
+33 (0) 603-92-19-14 <tel:+33 (0) 603-92-19-14>
***@***.*** ***@***.***>
https://inspeere.com <https://inspeere.com/>
24 boulevard du Grand Cerf, 86000 Poitiers (FR)
<https://www.linkedin.com/in/olivierdalle/>
… On 24 May 2022, at 23:08, Mathias Fredriksson ***@***.***> wrote:
I woke up to the same/similar issue this morning on Debian Bullseye running Linux 5.15.0-0.bpo.3-amd64 and ZFS 2.1.4-1~bpo11+1.
kernel:[377052.452950] VERIFY3(size != 0) failed (0 != 0)
kernel:[377052.457965] PANIC at range_tree.c:438:range_tree_remove_impl()
Unfortunately, I didn't have a serial console hooked up so no call trace :(. I do believe replication was in process when it happened. Anywhere between 1-4 datasets were probably being recvd concurrently on this machine via zrepl when it happened.
—
Reply to this email directly, view it on GitHub <#11893 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOPCEE7KEOAOFZ5SVG5JES3VLVANTANCNFSM4224KIZA>.
You are receiving this because you authored the thread.
|
I just ran into what I believe to be the same issue while sending an encrypted dataset to a different ZFS pool on the same machine:
This happened multiple times before.
|
I think I started seeing this after I configured automatic sends of an encrypted dataset to another pool on the same machine. I tried to enable debugging via
to
Unfortunately I'm quite unfamiliar with the internals of zfs, but I'm happy to run with patches that add more logging or debugging and report back.
Example stack trace without debugging
Example stack trace with debugging
|
I patched in a Full trace
Patchdiff --git a/module/zfs/zio.c b/module/zfs/zio.c
index 7b55450ca..9a259b785 100644
--- a/module/zfs/zio.c
+++ b/module/zfs/zio.c
@@ -1125,6 +1125,8 @@ zio_write(zio_t *pio, spa_t *spa, uint64_t txg, blkptr_t *bp,
zp->zp_copies > 0 &&
zp->zp_copies <= spa_max_replication(spa));
+ WARN_ON_ONCE(psize == 0);
+
zio = zio_create(pio, spa, txg, bp, data, lsize, psize, done, private,
ZIO_TYPE_WRITE, priority, flags, NULL, 0, zb,
ZIO_STAGE_OPEN, (flags & ZIO_FLAG_DDT_CHILD) ?
|
Checking back in the earlier messages, I didn't see anyone share the stuck task report (excluding the
|
I am hitting that bug as well on a backup server that receives an encrypted raw snapshot.
|
If we receive a DRR_FREEOBJECTS as the first entry in an object range, this might end up producing a hole if the freed objects were the only existing objects in the block. If the txg starts syncing before we've processed any following DRR_OBJECT records, this leads to a possible race where the backing arc_buf_t gets its psize set to 0 in the arc_write_ready() callback while still being referenced from a dirty record in the open txg. To prevent this, we insert a txg_wait_synced call if the first record in the range was a DRR_FREEOBJECTS that actually resulted in one or more freed objects. Signed-off-by: David Hedberg <david.hedberg@findity.com> Sponsored by: Findity AB Closes: openzfs#11893
I hit this issue on an ARM NAS while it is receiving an incremental snapshot (sent with
|
If we receive a DRR_FREEOBJECTS as the first entry in an object range, this might end up producing a hole if the freed objects were the only existing objects in the block. If the txg starts syncing before we've processed any following DRR_OBJECT records, this leads to a possible race where the backing arc_buf_t gets its psize set to 0 in the arc_write_ready() callback while still being referenced from a dirty record in the open txg. To prevent this, we insert a txg_wait_synced call if the first record in the range was a DRR_FREEOBJECTS that actually resulted in one or more freed objects. Signed-off-by: David Hedberg <david.hedberg@findity.com> Sponsored by: Findity AB Closes: openzfs#11893
If we receive a DRR_FREEOBJECTS as the first entry in an object range, this might end up producing a hole if the freed objects were the only existing objects in the block. If the txg starts syncing before we've processed any following DRR_OBJECT records, this leads to a possible race where the backing arc_buf_t gets its psize set to 0 in the arc_write_ready() callback while still being referenced from a dirty record in the open txg. To prevent this, we insert a txg_wait_synced call if the first record in the range was a DRR_FREEOBJECTS that actually resulted in one or more freed objects. Signed-off-by: David Hedberg <david.hedberg@findity.com> Sponsored by: Findity AB Closes: openzfs#11893
If we receive a DRR_FREEOBJECTS as the first entry in an object range, this might end up producing a hole if the freed objects were the only existing objects in the block. If the txg starts syncing before we've processed any following DRR_OBJECT records, this leads to a possible race where the backing arc_buf_t gets its psize set to 0 in the arc_write_ready() callback while still being referenced from a dirty record in the open txg. To prevent this, we insert a txg_wait_synced call if the first record in the range was a DRR_FREEOBJECTS that actually resulted in one or more freed objects. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: David Hedberg <david.hedberg@findity.com> Sponsored by: Findity AB Closes #11893 Closes #14358
If we receive a DRR_FREEOBJECTS as the first entry in an object range, this might end up producing a hole if the freed objects were the only existing objects in the block. If the txg starts syncing before we've processed any following DRR_OBJECT records, this leads to a possible race where the backing arc_buf_t gets its psize set to 0 in the arc_write_ready() callback while still being referenced from a dirty record in the open txg. To prevent this, we insert a txg_wait_synced call if the first record in the range was a DRR_FREEOBJECTS that actually resulted in one or more freed objects. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: David Hedberg <david.hedberg@findity.com> Sponsored by: Findity AB Closes openzfs#11893 Closes openzfs#14358
Is the fix for this issue shipped in any released zfs 2.1.* version? I am unable to find any reference to this issue or the fixing PR in the changelogs 😕 |
If we receive a DRR_FREEOBJECTS as the first entry in an object range, this might end up producing a hole if the freed objects were the only existing objects in the block. If the txg starts syncing before we've processed any following DRR_OBJECT records, this leads to a possible race where the backing arc_buf_t gets its psize set to 0 in the arc_write_ready() callback while still being referenced from a dirty record in the open txg. To prevent this, we insert a txg_wait_synced call if the first record in the range was a DRR_FREEOBJECTS that actually resulted in one or more freed objects. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: David Hedberg <david.hedberg@findity.com> Sponsored by: Findity AB Closes openzfs#11893 Closes openzfs#14358
If we receive a DRR_FREEOBJECTS as the first entry in an object range, this might end up producing a hole if the freed objects were the only existing objects in the block. If the txg starts syncing before we've processed any following DRR_OBJECT records, this leads to a possible race where the backing arc_buf_t gets its psize set to 0 in the arc_write_ready() callback while still being referenced from a dirty record in the open txg. To prevent this, we insert a txg_wait_synced call if the first record in the range was a DRR_FREEOBJECTS that actually resulted in one or more freed objects. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: David Hedberg <david.hedberg@findity.com> Sponsored by: Findity AB Closes #11893 Closes #14358
System information
Describe the problem you're observing
After printing a panic message, all zfs operations are stuck, the machine needs to be rebooted.
See below for panic message.
Note:
This is NOT the zfs version that comes with the distribution.
This is the locally recompiled zfs-0.8-release tagged version, but recompiled with NO error, using the default compilation settings.
Describe how to reproduce the problem
Hard to tell exactly when and why, but seems to happen under significant load, and most probably when trying to destroy snapshots.
After a reboot, it happens again, not immediately, but unpredictably. The stack trace when it reproduces is identical.
I haven't tried yet locally compiled earlier versions, but it seems that earlier versions that came with proxmox kernel 5.4.78 (zfs version 0.8.5) didnt produce this error. (proxmox 6.1.1 or 6.2.1 fresh install without apt update I guess).
Note that it MAY have started after I had once to destroy 13k snapshots that had been accumulating in a dataset. But I don't see why it and I am not totally sure about the timing. Maybe completely unrelated.
Scrub after occurrence completed without error, and it happened again after scrubbing.
Note that most datasets are encrypted. This is a backup server, mostly receiving encrypted snapshots sent in raw replication differential streams. Source streams are also produced using a locally recompiled 0.8 version (I had to downgrade to this version because of another issue on ZFS version 2.0.X that I reported in a separate issue that totally brake our backup scripts).
Include any warning/errors/backtraces from the system logs
The text was updated successfully, but these errors were encountered: