Grafana pod never created

[yogeek]

With this CR

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: monitoring-grafana
spec:
  ingress:
    enabled: False
  config:
    log:
      mode: "console"
      level: "warn"
    security:
      admin_user: "admin"
      admin_password: "admin"
    auth:
      disable_login_form: False
      disable_signout_menu: False
    auth.basic:
      enabled: True
    auth.anonymous:
      enabled: True
  dashboardLabelSelector:
    - matchExpressions:
        - {key: app, operator: In, values: [grafana]}

The grafana svc is present but the grafana pod is not created.
Here are the operator logs :

grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.359364,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:256","msg":"Phase: Create Config Files"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.3632724,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:452","msg":"Resource grafana-serviceaccount does not exist, creating now"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.3850758,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:286","msg":"Creating the grafana-dashboards resource"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.3909838,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:452","msg":"Resource grafana-dashboards does not exist, creating now"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.3993938,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:286","msg":"Creating the grafana-providers resource"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.4050183,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:452","msg":"Resource grafana-providers does not exist, creating now"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.4121027,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:286","msg":"Creating the grafana-datasources resource"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.4180439,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:452","msg":"Resource grafana-datasources does not exist, creating now"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.4305937,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:286","msg":"Creating the grafana-service resource"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.4351752,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:452","msg":"Resource grafana-service does not exist, creating now"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.4587262,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:293","msg":"Config files created"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.4808128,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:256","msg":"Phase: Create Config Files"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.5761788,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:286","msg":"Creating the grafana-dashboards resource"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.7659416,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:286","msg":"Creating the grafana-providers resource"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350508.965594,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:286","msg":"Creating the grafana-datasources resource"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350509.1760461,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:286","msg":"Creating the grafana-service resource"}
grafana-operator-c59c5b98c-vrbfd grafana-operator {"level":"info","ts":1576350509.3649647,"logger":"controller_grafana","caller":"grafana/grafana_controller.go:293","msg":"Config files created"}

[pb82]

Hey @yogeek , judging by the logs it looks like you're running an older version of the operator but the provided CR looks fine and should work. My guess is that the deployment is created, but fails to create the Grafana pod for some reason. So best to debug the deployment first. What's the output of:

kubectl get deployments -n <your namespace>

and

kubectl describe deployment <name of the grafana deployment from previous command> -n <your namespace>?

[yogeek]

About the version, I installed the last release : v3.0.1.
So you want me to test with the master branch instead ?

Currently, the deployment is not there, only the service is created :

# No grafana deployment
kubectl get deployments -n monitoring | grep grafana
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
grafana-operator      1/1     1            1           3m12s

# No grafana pod
k get po --all-namespaces| grep grafana    
monitoring     grafana-operator-6d49c98cd4-b2jth                                         1/1     Running     0          4m59s

# Grafana service is ok
k get svc --all-namespaces| grep grafana
monitoring     grafana-service          ClusterIP      10.108.156.254   <none>                                                                             3000/TCP                                     3m29s

[pb82]

3.0.1 is fine, but the logs you showed appear to be from an older version. So maybe there's something wrong with the operator deployment or the image? Can you confirm that the grafana-operator deployment points to the 3.0.1 image?

[yogeek]

Additionnal info : when I install the operator from OpertorHub like this :

curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.13.0/install.sh | bash -s 0.13.0

kubectl create -f https://operatorhub.io/install/grafana-operator.yaml

I works correctly !!!

What release has been published to OperatorHub ? It is not the last release of your github repo ?

[yogeek]

Installation from OperatorHub :

kubectl get deploy grafana-operator -o=jsonpath='{$.spec.template.spec.containers[:1].image}'
quay.io/integreatly/grafana-operator:v2.0.0

Installation from manifests :

kubectl get deploy grafana-operator -o=jsonpath='{$.spec.template.spec.containers[:1].image}'
quay.io/integreatly/grafana-operator:latest

[yogeek]

When I look at https://quay.io/repository/integreatly/grafana-operator?tab=tags, I have different tags :

docker images | grep quay.io/integreatly/grafana-operator
quay.io/integreatly/grafana-operator                                                    v3.0.1              73391e6c9576        4 days ago          45.8MB
quay.io/integreatly/grafana-operator                                                    v2.0.0              b9a8bae9ac63        2 months ago        47.2MB
quay.io/integreatly/grafana-operator                                                    latest              abaa4cf3c673        2 months ago        43MB

It seems that your operator.yaml points to latest tag which is different from the V2.0.0 of the OperatorHub and also different from the 3.0.1 you told me about.

[pb82]

Ah I see. The latest tag hasn't been updated, best to make sure you're selecting a specific tag in operor.yaml. Also, operatorhub is not yet updated with 3.0.1. You can find all the available tags here: https://quay.io/repository/integreatly/grafana-operator?tab=tags

Does it work when you install 3.0.1?

[yogeek]

@pb82 I can confirm that using the 3.0.1 tag is working ! Thank you for your help

I use kustomize to modify the release manifests for now but it would be great if you could either update the tag in the operator manifest or push the latest tag corresponding to the last image.

Plus, how is managed the publication to the operatorhub ? Each release is not deployed currently ?

[pb82]

@yogeek We are aiming to update the operatorhub release, just wanted to wait a bit to fix incoming bugs with the new version. I'll also update the latest tag.

[pb82]

I've updated the latest tag now. Closing this issue as the original problem has been resolved.

[yogeek]

@pb82 Thank you. Did you update it manually because I do not find the corresponding docker push in your travis CI file

[yogeek]

@pb82 is the latest tag up to date now please ? (cf. previous message)

[pb82]

Yes, I pushed it manually. It should be up to date now.

[yogeek]

Thank you 👍
Is it possible to add it in your .travis file to avoid you to do it manually ?