[Bug] get version does not work with Azure Managed Grafana

[arminfelder]

Describe the bug

The Grafana Operator tries to get the version of the external Grafana instance from its /api/health endpoint, unfortunatly this does not work wit Azure Managed Grafana, as in this case, all endpoints, including the /api/health require authentication.

Secondly the Grafana Operator, seams not to check the HTTP status code (in my case 403) , but only if the request is successful.

Log:

2024-06-05T17:20:23Z    ERROR   GrafanaReconciler       failed to get version from external instance    {"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"external-grafana","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "external-grafana", "reconcileID": "79f3e706-0b32-4053-bdac-952e33464692", "error": "parsing health endpoint data: EOF"}
github.com/grafana/grafana-operator/v5/controllers.(*GrafanaReconciler).Reconcile
        github.com/grafana/grafana-operator/v5/controllers/grafana_controller.go:95
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:222

Version
5.9.2
To Reproduce

configure Azure Managed Grafana as external Grafana

Expected behavior
A clear and concise description of what you expected to happen.

Suspect component/Location where the bug might be occurring

grafana-operator/controllers/grafana_controller.go

Line 157 in f64b541

if err != nil {

Runtime (please complete the following information):

• OS: Linux
• Grafana Operator Version 5.9.2
• Environment: Azure Kubernetes Cluster
• Deployment type: kustomize

[theSuess]

@weisdd will check how authentication for this endpoint is implemented and we'll decide what to do from there

[weisdd]

@theSuess The health endpoint can be accessed using the same API key as for managing grafana instance:

So, I think the fix would be to simply use the same client as in various controllers.

[arminfelder]

@theSuess The health endpoint can be accessed using the same API key as for managing grafana instance:

So, I think the fix would be to simply use the same client as in various controllers.

this was also me thought, after debugging with curl, but I did not want to mess up your code

[DanielAtanasovski]

Getting a similar issue with AWS Managed Grafana on Version 5.9.2:

2024-06-13T04:33:13Z	ERROR	GrafanaReconciler	failed to get version from external instance	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"grafana-workspace","namespace":"grafana"}, "namespace": "grafana", "name": "grafana-workspace", "reconcileID": "e3efad02-5a15-4bc8-a527-2f30cc480e7b", "error": "parsing health endpoint data: invalid character 'B' looking for beginning of value"}
github.com/grafana/grafana-operator/v5/controllers.(*GrafanaReconciler).Reconcile
	github.com/grafana/grafana-operator/v5/controllers/grafana_controller.go:95
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:222

[weisdd]

@DanielAtanasovski I don't have an AWS environment at hand, so could you send an authenticated request (using the same credentials that you supply to the operator) to that endpoint and post the result here?

[bonclay7]

Is there a possibility to avoid that api endpoint by specifying the grafana version in configs?

[DanielAtanasovski]

@DanielAtanasovski I don't have an AWS environment at hand, so could you send an authenticated request (using the same credentials that you supply to the operator) to that endpoint and post the result here?

It might be worse with AWS as the api/health endpoint appears to be blocked. An authenticated request just responds with Bad Request: Not allowed.

Related to this issue:
aws/amazon-managed-grafana-roadmap#41

[VermaPriyanka]

@DanielAtanasovski Amazon Managed grafana supports this API endpoint: https://grafana.com/docs/grafana/latest/developers/http_api/other/#frontend-settings-api. Can this be used alternatively to get the version?