vpa-admission-controller: Add observability and alerting

[ialidzhikov]

How to categorize this issue?

/area auto-scaling
/kind enhancement

What would you like to be added:

• Configure prometheus to scrape the vpa-admission-controller metrics: Scrape vpa-admission-controller metrics with prometheus #10033
• Add plutono dashboards based on the metrics
• Add alerts

Why is this needed:
In a recent VPA related outage (ref kubernetes/autoscaler#6884), we noticed that we are missing observability for the vpa-admission-controller component. For example:

• we don't have a panel showing the response times of the component
• we don't have alerts if the vpa-admission-controller response time > 10s (the webhook's timeout).

[voelzmo]

/assign

[gardener-ci-robot]

The Gardener project currently lacks enough active contributors to adequately respond to all issues.
This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Mark this issue as rotten with /lifecycle rotten
• Close this issue with /close

/lifecycle stale

[voelzmo]

/remove-lifecycle stale

[voelzmo]

Given we contextified the admission-controller, such that response times >10s shouldn't be catastrophic anymore, I'm nor sure if we still need the alerting. @ialidzhikov WDYT?