Investigate K8s-go-client caching/load implications

[Kavindu-Dodan]

Flagd is designed to watch K8s CRD changes and update flag configurations on CRD changes (ex:- updates, removals). This is performed by K8s ISync implementation [1]

The Focus of this task is to know the impact of on-demand CRD fetching on the K8s API. This knowledge is a pre-requisite to remove any caching in-between sync providers and flag evaluations/grpc streams.

Research:

• Does K8s API cache its responses
• The impact of K8s API overloading (ex:- throttling, performance impact and stability)

Outcome:

Whether we require a cache OR whether we can rely on K8s API for high load scenarios

[1] - https://github.com/open-feature/flagd/blob/main/pkg/sync/kubernetes/kubernetes_sync.go

[Kavindu-Dodan]

Currently used K8s client of the K8s ISync implementation is not backed by a cache [1]

// New returns a new Client using the provided config and Options.
// The returned client reads and writes directly from the server
// (it doesn't use object caches). It understands how to work with
// normal types (both custom resources and aggregated/built-in resources),
// as well as unstructured types

• Source [2]

Next step, consider a caching client if exist.

[1] - https://github.com/open-feature/flagd/blob/main/pkg/sync/kubernetes/kubernetes_sync.go#L155
[2] - https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/client/client.go#L90-L99

[Kavindu-Dodan]

One option is to rely on the store backing the informer that we already use for eventing [1]. The downside is that we could be reading outdated changes.

[1] - https://github.com/open-feature/flagd/blob/main/pkg/sync/kubernetes/kubernetes_sync.go#L168

[Kavindu-Dodan]

Good reading resources

[1] https://kubernetes.io/docs/concepts/cluster-administration/flow-control/
[2] https://cloud.google.com/kubernetes-engine/quotas#limit_for_api_requests

Next steps, investigating improvements in our ISync implementation to reduce API impact and reuse resources.

[Kavindu-Dodan]

PR #443 improves K8s Sync provider internals to utilize the informer store. However, we will still have a fallback to API get if we have a cache mismatch.

This will allow us to avoid a caching layer for flagd server