-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure that the reconcile queue tracks processing items #314
Conversation
@sambetts-cisco Not sure I understand, why that an item that was de-queued will be re-queued? |
The poller searching for items in specific states continues to run while the reconciler processes an item, which can result in the poller re-adding the item which is being processed to the end of the queue, because the queue doesn't know about this item any longer. By keeping track of the items which are being processed we can avoid readding these items and avoid an extra reconcile when we have a single reconciler, if we enable parallel reconcilers for some objects then its required to avoid two reconcilers trying to reconcile the same item. I original implemented this re-using the inqueue map, but that broke the behaviour I wanted for the "Has" function which is meant to check if something is in the queue, so I needed to maintain the list of processing items separately which is why I added the |
a3aee54
to
3815727
Compare
So the |
Dequeue removes the item from "inqueue" https://github.com/openclarity/vmclarity/pull/314/files#diff-c8aea9db7e7fbce1cfcb5e8ca00dabd75079958e9fa3409f2bbc9e5093f70e1cL86 so an item is only ever in "inqueue" if its in the queue slice. |
Maybe i'm missing something but if the item continue to be processed the |
face plam I must have changed that at some point during development and now I don't remember why... |
@FrimIdan I remembered why I track processing items separately! Its to prevent the inqueue map and q.queue's state getting out of sync. I'll add a comment. |
This commit adds some new state and functions to the orchestration queue so that it keeps track of Dequeued items until that item is marked as Done by the person who Dequeued it. This information is used to prevent items from being re-queued while the system is processing an item because the poller feeding the queue may pick up an item which is in the process of being updated. This also makes it safe to have multiple reconciler threads being fed from a single queue because we have a guarentee that the same item isn't in the queue twice and therefore we can't end up with mutiple reconcilers trying to process the same item.
3815727
to
465be88
Compare
This ensures that if there is an error during Dequeue that we don't try to Done() the item that is returned as a result because it should not be a real item in the queue and therefore could cause weird behaviour.
Description
This commit adds some new state and functions to the orchestration queue so that it keeps track of Dequeued items until that item is marked as Done by the person who Dequeued it.
This information is used to prevent items from being re-queued while the system is processing an item because the poller feeding the queue may pick up an item which is in the process of being updated.
This also makes it safe to have multiple reconciler threads being fed from a single queue because we have a guarentee that the same item isn't in the queue twice and therefore we can't end up with mutiple reconcilers trying to process the same item.
Type of Change
[ ] Bug Fix
[ ] New Feature
[ ] Breaking Change
[X] Refactor
[ ] Documentation
[ ] Other (please describe)
Checklist