Garbage collection may take a long time #10971
Comments
It is quite interesting that StoreUpdate::commit can take more than 1s
|
github-merge-queue bot
pushed a commit
that referenced
this issue
Apr 19, 2024
Move garbage collection into a separate actor to prevent it from blocking synchronously inside client actor. Fixes #10971. For testing, `garbage_collection_intense.py` sends a lot of insertion and deletion transactions on the same key and make sure that nodes do not crash after gc is done. The change is also run on a mainnet node and so far it works fine. TODO: run all nayduck tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A validator reported that their node usually miss blocks and/or chunks at the beginning of an epoch. At some point in the middle of the epoch after gc finishes, the number of missed blocks and chunks starts to go down. As can be see in grafana, GC can take more than 0.3s, which could make the difference between missing a block/chunk vs. producing them on time.
A long term solution is to move GC to a different thread, as attempted in #3090. However, that could require some nontrivial work and a short term mitigation could be that we don't run garbage collection when block processing already takes a relatively long time.
The text was updated successfully, but these errors were encountered: