PANIC: zfs: accessing past end of object bf37/35f70 #8673
Comments
|
I have been hit with the same issue on Gentoo: ZFS 0.7.13 and Kernel 4.19.27-gentoo-r1. Backtrace looked similar (see below) and panic occurred during heavy I/O induced by one of the processes. At this point, its been triggered twice. |
It's reproducing constantly on one of our servers now, there is only a 75MB/sec async write via Several times (after reboots): Disks are connected via |
|
For some reason The stack dump looks always about the same, it comes from |
|
For now, I'm trying to serialize writes to the same file on the top level of the zfs stack with i_mutex; some comments in the code say it's not needed for ZFS, but I've looked in BTRFS in our RHEL6/OpenVZ kernel and it does the same. commit: vpsfreecz@3089152 Edit: this didn't help... maybe I need to wrap the mutex around more actions... |
|
Seeing this issue as well. 0.8.1 on Centos 7.6. This bug seems specific to ZFSoL, as we have not seen it on our similarly used FreeBSD ZFS fileservers. |
|
We're experiencing this one way too often. Anything I did didn't help me to get to the root cause of this. The only thing I know for sure is that it does this when reclaim kicks in - which does so because of the OpenVZ patchset even in OOM event inside the vz container - it invokes all the possible shrinkers registered anyway with little regard for what is whose... @behlendorf can someone more senior please help me debug this? We've been loosing members pretty fast because of this one, in a few weeks tens of people gone... |
|
|
Heck, after all, @martellini "trying to use it in production" would be a good way how to describe the 5-year-long journey I've had with ZoL so far ;) |
|
That is to say, I don't think there's any better alternative than ZFS for the use-case we have for it. I'd like to chase this one little ****er down rather than argue about who said what about what... |
|
Anyway, I'm now trying to roll back to 0.7.9, had to back-port project quotas, although we've never used them, just enabling them changed format on-disk, contrary to all other features. https://github.com/vpsfreecz/zfs/commits/vz-0.7.9 |
|
Is triggered somewhat consistently on my RT kernel during heavy IO and a non-io process being 'reniced' to -20 |
|
hit by the same issue, running zfs 0.7.13 on CentOS 7.6 kernel 3.10.0-957.27.2.el7 kernel:PANIC: zfs: accessing past end of object 1303/4ecce (size=27136 access=20000+26958) |
|
@atjegan 0.7.9 seems to be without this bug, running without a crash so far |
|
@snajpa thanks for the update. Will try the version 0.7.9. |
|
I've encounterd the same error on several different servers. but only with the version 0.7.13 and later.(not with 0.7.12) System Information
LogHow to reproduce in our site
TendencyWe've experienced/tested on various hardware/OS/zfs versions and the tendency is like this.
|
|
@behlendorf Considering your statements on #9034 this one is definately critical and seems to be (semi-)repeatable |
|
@Ornias1993 agreed. Based on the very helpful #8673 (comment) above I reviewed all of the changes from 0.7.12 to 0.7.13. The following commit, which addresses a rare deadlock condition, is the most likely cause. 98bb45e deadlock between mm_sem and tx assign in zfs_write() and page fault @kazunoriohki @snajpa if it wouldn't be too much trouble can you try reverting this commit from 0.7.13 and try to reproduce the issue. I'll see if I can reproduce the issue as well using the test case described above. |
|
@behlendorf Thanks, I'll try with 98bb45e. |
|
Same here :) |
|
Oh it's this patch. @behlendorf, I can't be sure because I didn't take notes at the time (and I should have), but I think I've ruled out that patch, because it only appeared in 0.7.13 release, but the original issue mentions 0.7.12. |
|
@behlendorf reverting 98bb45e resolved the issue at our environment! with 0.7.13(native), the error has been reproduced in 13times, 92times, 144times (1 job takes about 5min, and I've repeated 3 times for making sure), while the error never happend over 750times with reverting 98bb45e commit. |
|
We are experiencing this bug in production also. It appears in 0.8.2, 0.7.13, 0.7.12, 0.7.11. I'm pretty sure it didn't appear in 0.7.9, but now I have problems when downgrading to 0.7.9, probably should downgrade more dependencies. I'll try to migrate the workload to a server were we still use 0.7.9. The workload is tile rendering using Mapnik from PostgreSQL database with Postgis and some Shapefiles. |
|
Was toying with my gentoo-musl chroot lately, and managed to reproduce this 100% of the time, 4 times in a row: within the chroot, compile gentoo-sources kernel 5.4.12 with -j16. The first time it happened, I was emerging some packages with -j16 in MAKEOPTS, -j3 to emerge (machine with 8 cores/16 threads). First time when compiling kernel happened after ~6h uptime, 2nd and 3rd time were from cold boot (pressed reset button, remounted chroot, restarted compiling). The system was usable, except for writing to disk. The only out-of-tree module is ZFS. I will try to revert the patch mentioned above and post results. |
|
I did some additional digging (sorry, I almost forgot about this and then I tried to toy with the musl chroot again last evening...). After analyzing the code (git tag zfs-0.8.3) and the patch, I noticed that when dmu_write_uio_dbuf() is called from zfs_write (L#827) the operation can result in EFAULT in zfs_uio.c/uiomove_iov (L#92,98), returning early without touching uio->uio_loffset, uio->uio_resid, which in the end are used for issuing the next dmu_write_uio_dbuf() call in the zfs_write while() loop for the pending bytes. I am guessing two fixes should be possible, and both could be implemented:
|
|
Tentative (not very elegant) patch: https://pastebin.com/i5xnxii5 Seems to be working properly:
[edit]: spoke too soon, it took much longer for it to happen, but it happened again:
|
|
Did some more analysis and test patching, 2nd final patch (to be used with the first one above): https://pastebin.com/L1VStMDx There were two errors fixed by the 2nd patch in the zfs_write while(n > 0) loop.
I have left GCC compiling run in a loop overnight: 11 successful merges (and counting) with no sign of breakage. |
In zfs_write(), the loop continues to the next iteration without accounting for partial copies occurring in uiomove_iov when copy_from_user/__copy_from_user_inatomic return a non-zero status. This results in "zfs: accessing past end of object..." in the kernel log, and the write failing. Account for partial copies and update uio struct before returning EFAULT, leave a comment explaining the reason why this is done. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: ilbsmart <wgqimut@gmail.com> Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com> Closes openzfs#8673 Closes openzfs#10148
In zfs_write(), the loop continues to the next iteration without accounting for partial copies occurring in uiomove_iov when copy_from_user/__copy_from_user_inatomic return a non-zero status. This results in "zfs: accessing past end of object..." in the kernel log, and the write failing. Account for partial copies and update uio struct before returning EFAULT, leave a comment explaining the reason why this is done. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: ilbsmart <wgqimut@gmail.com> Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com> Closes openzfs#8673 Closes openzfs#10148
In zfs_write(), the loop continues to the next iteration without accounting for partial copies occurring in uiomove_iov when copy_from_user/__copy_from_user_inatomic return a non-zero status. This results in "zfs: accessing past end of object..." in the kernel log, and the write failing. Account for partial copies and update uio struct before returning EFAULT, leave a comment explaining the reason why this is done. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: ilbsmart <wgqimut@gmail.com> Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com> Closes openzfs#8673 Closes openzfs#10148
In zfs_write(), the loop continues to the next iteration without accounting for partial copies occurring in uiomove_iov when copy_from_user/__copy_from_user_inatomic return a non-zero status. This results in "zfs: accessing past end of object..." in the kernel log, and the write failing. Account for partial copies and update uio struct before returning EFAULT, leave a comment explaining the reason why this is done. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: ilbsmart <wgqimut@gmail.com> Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com> Closes openzfs#8673 Closes openzfs#10148
In zfs_write(), the loop continues to the next iteration without accounting for partial copies occurring in uiomove_iov when copy_from_user/__copy_from_user_inatomic return a non-zero status. This results in "zfs: accessing past end of object..." in the kernel log, and the write failing. Account for partial copies and update uio struct before returning EFAULT, leave a comment explaining the reason why this is done. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: ilbsmart <wgqimut@gmail.com> Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com> Closes #8673 Closes #10148
In zfs_write(), the loop continues to the next iteration without accounting for partial copies occurring in uiomove_iov when copy_from_user/__copy_from_user_inatomic return a non-zero status. This results in "zfs: accessing past end of object..." in the kernel log, and the write failing. Account for partial copies and update uio struct before returning EFAULT, leave a comment explaining the reason why this is done. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: ilbsmart <wgqimut@gmail.com> Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com> Closes openzfs#8673 Closes openzfs#10148
and others
System information
Describe the problem you're observing
Splunk was installed on a dataset. After sometime it crashed the whole hypervisor with a kernel panic. Had to manually reboot the hypervisor
Describe how to reproduce the problem
Seems to be related to the read/write heavy of Splunk. Unfortunately, can't reproduce accurately. Maybe the logs will help pinpoint the exact issue so we can reproduce.
Include any warning/errors/backtraces from the system logs
The text was updated successfully, but these errors were encountered: