-
My system seems to have some similar sounds issues to those described in stratis-storage/stratisd#2814 Is there a suggested way to "kick" stratis to initialize resizing? is invoking
|
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 13 replies
-
Hi @erickj! I did a lot of digging into this and think I have an idea of what's going on. First, do you have a process to reproduce this? We want to make sure that we can put some kind of test in place to ensure that this doesn't happen in #2814. For some background, filesystem extension and thin pool event handling is done in the same method which will be changed in #2814. My understanding of how this worked is that the thin pool low water mark would be set to 2 GiB less than the current free space. This would trigger a devicemapper low water mark event every 2 GiB written to the pool. Filesystems are supposed to extend when there is only 8 GiB left of space so theoretically, the filesystem usage should always be checked and extended before it runs out of space. My guess at what is happening is that 2 GiB is not a huge amount of data to write. On faster machines and drives, it doesn't seem unreasonable that more than 2 GiB could be written or even flushed from the XFS cache between the time that the pool usage is determined and the time when then new low water mark is set. The low water mark only triggers an event when it is passed, not simply when the free space is less than the low water mark so if the pool has 5 GiB left, the next low water mark is determined to be 3 GiB left and then 2.5 GiB gets written and there is only 2.5 GiB of free space left before the 3 GiB low water mark is set, the devicemapper event would never fire again until the pool starts running out of space (which it is not in your case). This seems like the most likely issue and I would love to confirm this. If this is indeed the issue, #2814 should solve this. As for a solution in the meantime, stratisd only runs the filesystem check method on devicemapper events and start up. Restarting stratisd should cause the pools to run the checking method again. This should determine that the XFS filesystem needs to be extended from the usage and cause it to expand. It is unfortunately rather difficult to generate a devicemapper event so this seems like your best option. If you are interested in confirming that this is the problem before you fix it (which would help us a lot), here is an example output from
The 4th argument after Hopefully this helps some and if this doesn't resolve the issue, please do let us know! |
Beta Was this translation helpful? Give feedback.
-
unless "setup stratisd 3 years ago, and upgrade your workstation 6 times" is a viable repro, then unfortunately... no. I will say that I went through a recent rough patch of an upgrade from 34->35 resulting in an SELinux relabel to /. I have not idea if this is actually related to any recent stratis issues I've seen
here is the output for all drives listed above. I'm expecting both 7e18ddcd-9924-4c92-b926-100a7498630b and 8d86f3f68666490b99b0b5c6a5fc7986 to resize
|
Beta Was this translation helpful? Give feedback.
-
https://gist.github.com/erickj/c71578ade417c12448104f3216b95ad3 I noticed when looking at them that the update I mentioned above (from fedora 34 ->. 35)... I actually updated from 33 -> 34 -> 35 in 2 back-to-back reboot cycles. I see in the logs that stratisd upgraded from 2.1.0 -> 2.4.3. Could this direct upgrade have caused any backward compatibility issues? Other than that, no I have not made any modifications to any stratis storage structures, afaik. |
Beta Was this translation helpful? Give feedback.
-
@jbaublitz - so I just did a The volumes seemed to have resized on their own. This issue is solved from my perspective. Thank you
|
Beta Was this translation helpful? Give feedback.
@jbaublitz - so I just did a
dnf update
and stratis upgraded to 3.0.2.The volumes seemed to have resized on their own. This issue is solved from my perspective. Thank you