Possible race condition / inconsistency between CRUD operations

[nkvoll]

Create/Update/Delete operations are performed using non-caching clients and return before any caches are likely to have seen the changes. This means Get/List can return stale data for an undeterministic amount of time, could be milliseconds or could be longer based on latency/load on both client and apiserver.

Example:

1. List resource type X (see count: 0)
2. Create resource of type X (success)
3. List resource of type X (see count: 0)
4. wait
5. List resource of type X (see count: 1)
6. Update resource of type X (success)
7. Get resource of type X (see old object)
8. wait
9. Get resource of type X (see new object)
10. Delete resource of type X
11. Get resource of type X (see non-deleted object)
12. List resource of type X (see count: 1)
13. wait
14. Get resource of type X (see tombstoned object)
15. wait
16. List resource of type X (see count: 0)

This is something that seems likely to catch quite a few operator developers off guard and could cause a lot of resources to be created when missing if the identifier for the resource is non-deterministic/random.

This relates a little to #180 because the behavior comes from implicit caching.

[DirectXMan12]

That's intentional -- it's how Kubernetes works. If you write your controller correctly (i.e you set up signals for everything you care about, and you always try reconciles as trying to be declarative), you'll get the right behavior, because you'll get notified of updates as changes get noticed. Kubernetes is eventually consistent.

We could probably do a bit better in explaining this in the docs, but it's a bit of a core tenet of how Kubernetes controllers function.

[DirectXMan12]

(feel free to continue commenting, but I'm going to close this for bookkeeping purposes. I can always re-open it)

[ichekrygin]

@DirectXMan12 is there a some sort of "best practices" doc for writing a controller to get more info, or maybe you could point to a controller that could be used as an example?

If you write your controller correctly (i.e you set up signals for everything you care about, and you always try reconciles as trying to be declarative)