Manager default client does not provide strong read-after-write consistency

[harveyxia]

This is the default client that we're using in our skv2 generated clients. It does not provide strong read-after-write consistency, as detailed in this thread.

In SMH we've observed errors stemming from attempting to create objects that already exist, or updating objects with stale resource versions. We have reason to believe that these errors are due to the cache behavior—for a given client cache, reads are not guaranteed to return objects that have been written, even in a single threaded context.

In order to ensure correctness of our k8s controllers, we need a method to guarantee that CRUD operations to k8s objects are applied and not dropped. In the thread discussion linked above, one of the k8s contributors suggests the following:

(also, in case it wasn’t clear, while it’s fine (and generally preferable) to write tests expecting read-after-write consistency, you shouldn’t do that for controllers. Kubernetes favors a kind-of 2-phase approach -- do your reads, process, do some writes, and return, letting the requeuing deal with the next cycle of reads if they’re necessary, but don’t try to read an object after you’ve written it).

This is a fundamental problem that we need to sort out. We should discuss various approaches to this problem, including requeuing at the EventWatcher level.