Add VPA for admission-gcp deployment

[ialidzhikov]

How to categorize this PR?

/area auto-scaling
/kind task
/priority normal
/platform gcp

What this PR does / why we need it:
With the rollout of provider-gcp@v1.8.2, in some of the large landscapes we observed that the admission-gcp Pod was OOMKilled several times (with memory limit set to 200Mi) - probably admission-gcp now requires more memory after the introduction of #112 (new informers for Secrets and SecretBindings are added with a new webhook endpoint). I guess we could use a VerticalPodAutoscaler to minimize the manual request/limit adjustments in future.

Release note:

`admission-gcp` chart does now include a VerticalPodAutoscaler for the webhook deployment.

[timuthy]

probably admission-gcp now requires more memory after the introduction of #112

Independent from the changes introduced by this PR, can you please double check the impact of #112? I'm a bit disturbed that a "simple" webhooks requires that much memory. WDYT?

[rfranzke]

/lgtm
/assign @timuthy

[timuthy]

Tbh, I prefer to know the root cause of the higher memory consumption before we merge this PR. If the mem allocation was a spike because many requests hit the API server, would the VPA even help here? In our setup we have recommender-interval=1m0s.

[rfranzke]

Agreed that it would be helpful to know what causes the memory consumption, but IMO it's no prerequisite for merging this PR. Putting our components under auto-scaling means makes sense in general. Do you agree?

[timebertt]

probably admission-gcp now requires more memory after the introduction of #112

Independent from the changes introduced by this PR, can you please double check the impact of #112? I'm a bit disturbed that a "simple" webhooks requires that much memory. WDYT?

I think, with #112 admission-gcp watches Shoots, Secrets and SecretBindings, which it hasn't been doing before.
And because controller-runtime watches are not filtered, it will watch all instances of those resources, which obviously can be quite many in large gardener installations.
Would be nice to be able to filter for a specific spec.provider.type, though (ref kubernetes-sigs/controller-runtime#244)

FMPOV this explains, why the memory usage is then sometimes exceeding 200Mi.
WDYT?

[timuthy]

Putting our components under auto-scaling means makes sense in general. Do you agree?

Not if we do it by default and hope that the OOM issue above dissolves.

I'm mainly worried about downscaling actions performed by VPA. In operations, we observed cases for which a proper upscaling didn't happen after downscaling and in order to recover we needed to delete the VPA object. The admission component directly affects the availability of the Shoot API and thus I'm a bit critical. In addition, restarting the admission component isn't cheap any more since the introduction of #112 and the cache syncs. On our busiest landscapes I observed start-up times of ~2 mins.

What I'm trying to say is that adding VPA comes at a price:

• Only add VPA if it helps to improve our situation. I double checked @tim-ebert's statement and can confirm this partly but think in our case it's more related to getting Secrets via the Controller-Runtime client which is backed by a cache, i.e. all secrets are synced to the cache.
• We should think about the issue mentioned above. Either setting minAllowed to 200Mi or increasing the replica count as a safety buffer.

[timuthy]

@ialidzhikov let's increase the default replica count to 3 and decrease the memory footprint in another PR via #143.

[ialidzhikov]

/close