☂️ Monitoring

[schrodit]

How to categorize this issue?

/area monitoring
/kind epic
/priority 3

Umbrella issue to expose landscaper and deployer metrics and tracing got the various reconcile loops.

General:

• Expose controller runtime metrics (https://book-v1.book.kubebuilder.io/beyond_basics/controller_metrics.html)

Landscaper:

• blueprint/component-descriptor artifact cache
• installations controller
  • add tracing through the reconcile flow
• executions controller
  • add tracing through the reconcile flow

The issue is open for other aspects we can monitor

[hendrikKahl]

Additionally, I feel we should expose basic controller metrics for all deployers as well.

In terms of further metrics: Maybe it would be interesting, if deployers would expose the connection time metrics?
Like, how long does it takes to fetch deploy items from a landscaper cluster as a histogram?

[schrodit]

Additionally, I feel we should expose basic controller metrics for all deployers as well.

In terms of further metrics: Maybe it would be interesting, if deployers would expose the connection time metrics?
Like, how long does it takes to fetch deploy items from a landscaper cluster as a histogram?

you mean how long does it take for a deploy item to be processed?
maybe thats something we can also include the deployer library. some of such kind of default metrics

[hendrikKahl]

you mean how long does it take for a deploy item to be processed?

That would be another valid option. What I had in mind was focusing more on traffic when there is a distributed setup with deployers and landscaper running in separate clusters.

[hendrikKahl]

metrics added to OCI cache - gardener-attic/component-cli#3

[hendrikKahl]

a first implementation has been merged with #110 . This enables the landscaper-controller collect metrics of the controller-runtime and its OCI cache(s).

Meaningful next steps would include:

• enable deployer-controllers to expose the same set of metrics
• write a meaningful prometheus scrape config and drop metrics which are not needed
• build a grafana dashboard to visualize the status of each controller

Additionally, it could be interesting to expand the cache metrics with a histogram that gives insight into the file sizes.

[gardener-robot]

@schrodit Label priority/normal does not exist.

[schrodit]

We should also switch to OpenTelementry as poen standard for metrics.
In addition open telemetry supports traces which could be a good way to monitor our reconcile loops.

Also the controller-runtime is migrating towards that approach: kubernetes-sigs/controller-runtime#305

[achimweigel]

@In-Ko closed because outdated and/or copied into internal project