pluginChartTest Controller - performance improvement

[ibakshay]

Priority

(Low) Something is a little off

Description

The pluginChartTest controller does periodic testing for all the plugins that includes helm chart test. The pluginChartTest controller's work-queue keeps increasing when the plugin's test is failing as it is a blocking operation. This is because, when the test fails, there is a retry mechanism inside the test pod and the controller waits till this test pod finishes.

Acceptance Criteria

• Make sure that the work queue of the pluginChartTest controller is getting drained faster

Potential solutions:

• set MaxConcurrentReconciles to > 1. default is 1. link to a blog post

Example of an Alert:

[WARNING] [greenhouse-qa] GreenhouseOperatorWorkqueueNotDrained - Greenhouse Operator controller - pluginChartTest's backlog is not being drained.
:warning: The workqueue backlog of Greenhouse Operator controller - pluginChartTest is not getting drained.
[Prometheus](http://prometheus.greenhouse-qa.eu-nl-1.cloud.sap/graph?g0.expr=sum+by+%28name%29+%28rate%28workqueue_depth%7Bjob%3D%22greenhouse-controller-manager-metrics-service%22%7D%5B5m%5D%29%29+%3E+0&g0.tab=1)

Reproduction steps

1. Add a plugin that has a helm chart test defined. Example [Kube-monitoring plugin](https://github.com/cloudoperators/greenhouse-extensions/tree/main/kube-monitoring/charts/templates/tests)
2. tweak the helm chart of the test plugin in such a way that the helm chart test fails for this plugin

Manifests

No response

Screenshots

![DESCRIPTION](LINK.png)