Another ZFS deadlock with 0.6.3-1, sysrq-t included #2864
Comments
|
Two more interesting facts - spl_kmem_cache was spinning at 80-100% CPU, but what surprised me more, kswapd0/1 CPU usage was high as well, though I have no swap enabled on that machine. That one I would probably associate with OpenVZ VSwap functionality, which is probably unrelated, though. |
|
Might I dare to ask, are you guys @behlendorf or @ryao any closer to chasing this? As load on our machines progressively increases, we're running into these issues almost on daily basis now. Which is really, really bad :( What surprises me though is, that during the day, during the normal (heavy) workload it works all OK, but things change when the containers start running cron jobs during the night - it's loadavg > 1000 almost every night on at least one server. |
|
OK, I'll try applying those changes on 0.6.3 tag I can't move on to the HEAD, as it reports AIO as supported, but there's no direct IO support yet - this breaks mysql innodb engine, it tries to use both AIO and Direct IO, but only detects if AIO support is present. Effectively this nukes all MySQL instances and as we're OpenVZ based container host, people would kill me. |
|
Sounds like a good plan. I've opened #2872 for the MySQL issue you mentioned, if the addition of AIO support but lack of DIO support causes MySQL problems we may for forced to address that before the next tag. |
|
@behlendorf thanks for opening 2872. I've deployed patched versions of 0.6.3-1 (https://github.com/vpsfreecz/zfs) on all of our nodes but haven't rebooted them yet; I'm essentially waiting for a deadlock on a particular node to shoot it in the head, so the it will come up with the patched kmod-zfs. |
|
Sorry I've mistaken it for the other one where I've seen that happening pretty often (node5). Node6 is the one with this patch and it's got 2 and 1/2 days of uptime now. |
|
So the patch doesn't seem to make any difference. Stack traces again included from two nodes: node4 |
|
According to the stacks you've provided we've effectively deadlocked the system in memory reclaim. The patches proposed in #2918 should help with this and they'll likely be merged in the next week or so. |
|
So I've deployed HEAD on most troubled nodes, with a small patch disabling AIO, so that I don't break MySQL. https://github.com/vpsfreecz/zfs/commits/master I'll let you know how it goes, I either expect sh*t hitting the fan within one week or possibly(?) now never :) |
|
Well, it is better, however, I still get load spikes for too long - just now there's been a 10min long window, where any IO request was taking really long time to finish. I don't limit ARC size and the only deviations from default settings are zfs_txg_timeout=10s and zfs_dirty_data_sync=512M. I caught this in dmesg: |
|
Another interesting observation: when ARC is shrinking, kswapd0/1 spike up in CPU usage, IO seems to stall a bit and after reclaim is done, everything continues to normal. Funny thing is, that the system reports about 20G of absolutely free memory (of 256G total, ARC oscilates around 100G, no swap configured at all). |
|
Closing. This is believed to be resolved. |
https://gist.github.com/anonymous/718e6a1aa6c99bc21a1d
Setting spl_kmem_cache_slab_limit=16384 or spl_kmem_cache_reclaim=0 with ZFS already stuck didn't help.
The text was updated successfully, but these errors were encountered: