Tinkerers-ci test suites are failing due to observability-platform

[QuantumEnigmaa]

The Tinkerers-ci test suites are failing on CAPV and CAPVCD due to the observability-platform, most likely due to the object storage issues we are facing right now with the recent migration of all CAPV and CAPVCD buckets CRs from glippy togaggle.

After the migration, the object-storage-operator is struggling to work correctly on gaggle so solving this would most likely help, if not solve, the tinkerers-ci issue as well.

See the related incident for more details : https://gigantic.slack.com/archives/C087RGMRPDZ

[QuantumEnigmaa]

After updating the observabiliy secrets on grouse so that they match the ones on gaggle which is now the MC managing the object-storage for CAPV & CAPVCD clusters, we got rid of the authentication errors.

Currently, there are 2 remaining issues on the MC :

• A lack of resources on the cluster which lead to several pods (mimir, loki and alloy ones) to be stuck in Pending state. This should be solved tomorrow with rocket adding an additional worker node.
• The prometheus-to-grafana-cloud pod is still having authentication issues so we need to go through the process for rolling the secret.

[QuantumEnigmaa]

So @giantswarm/team-rocket added an additional worker node today, which lead to some pods to schedule but unfortunately, most of the alloy-logs pods are still stuck in Pending state as the new node is already at maximum capacity concerning memory.

By trying to release some of the memory by lowering memory requests on loki-write, I also accidentaly put one of the replica in a Pending state.

[QuantumEnigmaa]

@QuentinBisson did some additional investigations and found out that alloy-metrics was requesting way more memory than what the vpa's recommender actually suggested, taking over almost entire nodes' memory.

As a temporary hot fix, Quentin created a PR to allow mimir-querier and mimir-distributor pods to schedule on control-plane nodes : https://github.com/giantswarm/shared-configs/pull/194

[QuentinBisson]

This workaround is sadly not accepted by the majority of team atlas

[QuantumEnigmaa]

Herve said he was ok (even though releuctant) in today's standup :)

[QuentinBisson]

@AverageMarcus the observability platform is running on grouse, closing for now, let's reopen if we still have issues