arc_prune and arc_evict at 100% even with no disk activity #14005
Comments
|
I am also seeing this. I'd struggled with the issue from #9966 as well for months, even after setting these options After upgrading the server (AMD chips, 96-cores, 128GB) from 20.04 to 22.04 (and thus ZFS to 2.1.4) and reverting all these options to their defaults, the issue of having 96 Heres I've now tried reapplying the |
|
Back again - and I can confirm that forcing |
|
A few days later, happens again. I'm attaching the final 4 hours or so of |
|
When happening again, can you show the output of the following commanda?
|
|
With pleasure! I've caught it in the act for once so first here's the output of those commands while the machine is genuinely busy: Then I killed the application process which was using all the memory and hammering the disk - normally I don't spot it until the machine has been like this for a while, so this is a new step. The machine has no load running, several minutes pass. I don't know what I'm looking at with all this, but it feels like because I killed the job early the ZFS issue is now... simmering. The slightest disk activity causes the problem to surface. I need the machine back so drop caches, and we have: and I'll generate the same output the next time I fail to catch the job in time... |
|
Thanks for your reporting. The key point is that, until you dropped caches, your metadata were over limit ( For the next report, can you grep the following: |
|
Back again - this time I did not catch it in time. More diagnostics: Setting zfs_arc_meta_limit_percent to 100 made no difference, |
|
This happens to me regularly on a daily basis, usually after suspending the system to RAM. Changing |
|
@faceless2 @coot When happening, please report back the following data: |
|
Sorry - I removed these after the kernel upgrade but it looks like I didn't reboot. So those are the options I've been running with. Sorry I didn't notice The initial level for I have no particular reason for setting
I wonder if this could be a contributing factor? I've now set it to 8GB, half of the 16GB max value and will see if it makes a difference. |
|
Surely having zfs_arc_min set so high (relatively to ARC) can be problematic, lets see if the issue reappears. |
And after |
|
I don't know what has changed recently, but since a week ago (or maybe two) this has been happening to me now even with Borg (I haven't used Kopia since I tried it when I opened this issue). And I'm not 100% sure about this, but I think that compiling a big Java/Maven project the other day triggered it too. And quite frequently, once every 2 or 3 days. ZFS version is still 2.1.4-0ubuntu0.1. But the kernel is now Ubuntu's 5.15.0-53-generic. The arc meta cache keeps going bonkers, this time using more than twice the limit and even after several hours arc_prune and arc_evict haven't managed to bring it down. As usual, dropping the caches solves the problem temporarily. |
|
Yep, in both cases @behlendorf any idea on why |
|
It usually happens for when I start working. Sometimes it's just after I login, sometimes after some time when doing something, might be when compiling a large project, but I am not that sure if this is correlated. |
|
In my case, it used to happen only (but always) when running a Kopia backup job. I decided to keep using Borg and kind of forgot about the issue. But since a couple of weeks ago or so it's been happening while & after running Borg Backup. And I'm not sure, but I think that a big Java/Maven project compilation caused it the other day too. At least in my case, things seem to indicate that it's high disk activity what triggers the problem. |
|
If forgot to mention that with Borg it doesn't happen every time. Right now I have an uptime of almost a day and a half (with several cycles of RAM suspension). Borg has run 4 times during this time and everything is still fine: About 15 minutes ago the usage was 160....... So it's working fine, getting close to the limit and being reduced again. And a couple of days ago it happened after just 5 o 6 hours of uptime and I think it was the first time that Borg had run since booting up. So it doesn't seem to be related so much to the accumulated amount of disk reads but to a particularly high disk activity in a concrete point in time. |
|
My impression is that everything works as expected as long as 'arc_meta_used' is kept below the limit. But if for any reason the usage goes above the limit (is the limit a hard or soft limit?) the cache enters in some weird state from which it doesn't recover until it's reset by a flush. |
I think it is more an issue related to memory usage, both due to applications and vfs cache.
Yes, but when memory requirements become higher, |
|
Looks like reducing edit: I seem to have caught it in the "simmering" stage rather than finding it after it's boiled over - |
|
zfs version 2.1.5， kernel version 5.10.0-60.18.0.50.oe2203.x86_64 When using rsync to back up a large amount of data, the cache of arc cannot be released，and arc_prune at 100%，lasts for 50 minutes [root@nas ~]# ps -aux | grep D [root@nas ~]# arcstat 1 [root@nas ~]# free -h |
|
sometimes avail is negative
|
|
Commenting to note that a) I am still seeing this, but as I now have a cronjob that drops caches every hour I'm dealing with it, and b) I missed this question above
Oh yes. The process that triggers this creates about half a million files in 10 minutes while reading about 2 million. I should also note that we used to do this on more resource limited machine with 32 cores without problems - the same job took 45 minutes, and it was fine. It's when we upgraded to a much faster machine with more cores and NVMe disks that this started to happen. A case of "things happening faster than we can deal with them"? |
If possible, try updating to zfs 2.1.7 or 2.1.9 (.8 has a serious issue, so skip it) as a fix for this very same issue was merged. |
|
I set up a spank new server with arc_max as 8GB and arc_min 8GB - 1 and I also run repeatedly in this situation even with 2.1.9 so there is still something wired happening because the arc size goes slightly above the set max value. |
If you look on zfs_arc_overflow_shift, ARC is allowed to go slightly above the limit for a short time. And please, if you set some tunables to some weird values, don't complain if you see ARC behaves "weird". ARC eviction code was completely rewritten in upcoming ZFS 2.2. |
@amotin interesting, any information to share (or to point at) regarding this change? |
|
Hi all, Not sure if this is helping, but I had the same issue on older kernel 5.4.203 (proxmox) with zfs-2.0.7. It consistently happened after I received crypted dataset, loaded keys and then mounted the datasets. This triggered in turn several z_upgrade processes that started heavy I/Os. Which in turn started the arc_prune/arc_evict running full time. After reading this thread, I tried the drop cache trick and it worked (took some time), ie. z_upgrade was still working with heavy I/Os but arc_prune/arc_evict are gone. Hope this helps. |
|
This problem seems to still happen on zfs-2.1.13. I was waiting for 2.2.0 to be available in the RHEL dkms non testing repository on Rocky8 but I may have to try the testing version because I can't work when the storage is this slow. |
|
For me this problem was related to misconfiguration of |
|
Linux 6.5.9 with ZFS 2.2.0 Saw arc_prune and arc_evict showing up in my hud for a cpu usage tonight and remembered echoing I echoed 16gb ( |
System information
Describe the problem you're observing
I've been using ZFS for the root fs of my desktop, with ARC limited to 2GB, for several months without any issues until now. When I run disk intensive tasks, like borg-backup, duc, find, etc... I can see arc_prune and arc_evict working intermittently, with peaks of about 15% of a thread each, and they manage to keep the arc within limits:
And once the tasks are finished they both go to sleep. All normal so far.
But yesterday I decided to try another backup tool, Kopia, and I know what it does while doing a backup that makes ARC going out of control. Both arc_prune and arc_evict start using 100% of a CPU thread each. And despite that, they don't manage to keep the memory limits within range, only reaching some kind of balance at around 3.6GB.
But even after Kopia has finished or I have aborted it, the problem keeps going on indefinitely, even though there's no processes disk activity anymore (iostat and my system's disk led both show some continuous activity though, so it seems it's not just the CPU what they're using).
NOTES:
echo 3 > /proc/sys/vm/drop_cachesstops it, until I run a backup again.Describe how to reproduce the problem
Run a Kopia backup of a ZFS filesytem with many files and a low(ish) memory limit for ARC.
The text was updated successfully, but these errors were encountered: