-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CRD watch stream starts processing very old deleted resources #693
Comments
It's not the cause. I removed the second controller and this problem still persists. |
I'm not sure it's the same problem, though I think it can lead to that problem. As far as I can tell, Kubernetes watch is sending something like a compacted transaction log to the watcher. So, from the last point of compaction, you see all modifications to a given resource, in order. Since Kubernetes cares about eventual consistency, I suspect it assumes that any consumers of the watch will process the events in such a way that only the final state truly matters, not the intermediate states. Do you see DELETED after the ADDED? Or does it finish with ADDED? If it doesn't delete them, then yeah, there's a serious problem, because your application's state is now out of sync. |
Thanks for the reply @klarose
For resources that are still present in k8s, the last event_type is
OK, I think I understand why but not how are we supposed to deal with this in the controller? If the controller is adding and removing something based on these events, e.g. a database, or some configuration, then replaying the events (even in correct order) could be detrimental. In my example above, it's reprocessing events from days ago. If the same resources was added as name |
Hmmm. I'm not sure. Pretty fundamental to Kubernetes is the idea that you only care about eventual consistency. So if you're doing something non-idempotent with the results of the watch, you could be in for some pain. :) That said, how about this?
This is pretty clunky. It'd be better if the initial list would also return the maximum resource version across all resources. You could then just start the watch from that point... Is there an API to do that? |
Actually, maybe the API does do that?
Note how Unfortunately, the generated API docs aren't really clear on what is really returned here... |
In your example above, did you wait 5 mins after starting the |
Cool but, as you said, that's quite clunky. And from the examples I read, they don't implement this behaviour. E.g. https://github.com/karmab/samplecontroller/blob/master/controller.py from @karmab |
I'm not doing a watch. This is a simple list. So, the list is giving me the current state and the most recent resource version independent of the current state (I.e. it captures anything that has been deleted since the last updated to an existing resource). For example, imagine we had something like this: Add A: resource version 1. Delete C: resource version 4. Now, if we list the resource, we see: A: resource version 1. If we do a watch from 0, we see: 1: Add A However, if we do this:
Then we will never see versions 3 and 4, because we will see 1 and 2 from the list, then skip 3 and 4 when we start watching. Further, any resources that were added after we did the list will have a version greater than or equal to 5, meaning that we'll see them in the watch. |
Actually, maybe that is the fix for both the issues. :) https://github.com/kubernetes-client/python-base/blob/master/watch/watch.py#L135 Perhaps here, rather than tracking the resource version of each object, maybe it should be parsing out the resourceVersion from the result of func()? |
Isn't that what my code example is doing though? with these lines? if metadata['resourceVersion'] is not None:
resource_version = metadata['resourceVersion'] |
@max-rocket-internet That is the resourceVersion of the specific event, not the list itself. In my example, I get the resourceVersion of the list itself, not the items within the list. It is different. I tried to decode what was being returned here from the kubernetes docs, but they don't seem to have a definition for this API. The python client docs only describe the return type as an "object", sadly. Either way, here is what the result of the list returns right now for me:
Note how |
I suspect that this is the type we're seeing there: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/#listmeta-v1-meta It describes resourceVersion as:
The link in there has much more info on resourceVersions. It explains some things I've noticed about the resourceVersion I wasn't sure about (namely, it being a global value). There is one very important tidbit at the end:
I think this implies that my solution is the correct one: Start the watch with a list, then follow up with a watch from the list's resource version. |
Thanks for the replies @klarose
OK. Is there no examples of this though? Because the existing guitar one and the comments I found don't use this method. Would be nice to have an example with a confirmed correct approach! |
I'm not aware of any. Given that you can reproduce it, maybe you can be the one to give the example! :) |
I have posted a reproducing script in #700 |
fix posted: kubernetes-client/python-base#109 |
@juliantaylor so to be clear: when your PR is merged, then I will not have to worry about my watcher getting old resource versions? i.e. if I start watching NOW, I will not receive events for old resources? |
When the fix is merged the watch object will respect the It does not handle potential out of order objects which could again lead to receiving old objects. But I am not sure if that can happen nor can I reproduce that. (but it can probably happen: https://coreos.com/etcd/docs/latest/learning/api_guarantees.html) |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
This is resolved in latest releases for me. Thanks for the support! |
I have a basic controller running to watch
CustomObjectsApi
but about 2 minutes after starting it starts processing old resources that were already deleted days ago.Here's my code:
And the output is this:
The problem is that the resources
3070373
(app1) &3070374
(app2) were deleted from the cluster a long time ago. The don't show inkubectl get mycrd
.I have 2 controllers running watching the exact same CRD. One is running in the cluster and the one above is running on my workstation. Is the cause of this error the fact I'm running 2 controllers watching the same CRD? Or is there a mistake in my code?
Kubernetes 1.10.3
kubernetes-client/python: 8.0.0
The text was updated successfully, but these errors were encountered: