[BUG] Botkube 0.8.0 - crash loop - invalid memory address or nil pointer deference

[bilaloslo92]

Describe the bug
I have tried to deploy Botkube version 0.8.0 (which was recently released), I am using the deploy-all-in-one-tls.yaml to deploy it in my Kubernetes cluster.

To Reproduce
Steps to reproduce the behavior:

1. I am running mattermost server version: 5.11.0
2. Kubebot version 0.8.0
3. Deploy the deploy-all-in-one-tls.yaml with TLS (self-signed certificates)
4. See error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10a8e3c] 

goroutine 78 [running]:
github.com/infracloudio/botkube/pkg/bot.(*mmBot).Start.func1(0xc00010e150, 0xc0002af140) 
       /go/src/github.com/infracloudio/botkube/pkg/bot/mattermost.go:104 +0x5c 
created by github.com/infracloudio/botkube/pkg/bot.(*mmBot).Start 
      /go/src/github.com/infracloudio/botkube/pkg/bot/mattermost.go:101 +0x546​

[PrasadG193]

@bilaloslo92 Could you please post your configuration here (make sure you remove sensitive info like certs and tokens)?

[PrasadG193]

Are you encoding certificate before creating secret?
cat <YOUR_CERTIFICATE> | base64 -w 0

[bilaloslo92]

Hi @PrasadG193 ! Thanks for your reply. See attached config (removed credentials, clustername, and mattermost configuration).

---
# Configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: botkube-configmap
  labels:
    app: botkube
data:
  config.yaml: |
    ## Resources you want to watch
    resources:
      - name: pods                # Name of the resources e.g pods, deployments, ingresses, etc. (Resource name must be in plural form)
        namespaces:               # List of namespaces, "all" will watch all the namespaces
          - all
        events:                   # List of lifecycle events you want to receive, e.g create, update, delete, error OR all
          - create
          - delete
          - error
      - name: services
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: deployments
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: ingresses
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: nodes
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: namespaces
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: persistentvolumes
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: persistentvolumeclaims
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: secrets
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: configmaps
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: daemonsets
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: jobs
        namespaces:
          - all
        events:
          - create
          - update
          - delete
          - error
                  
      - name: roles
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: rolebindings
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: clusterroles
        namespaces:
          - all
        events:
          - create
          - delete
          - error
      - name: clusterrolebindings
        namespace:
          - all
        events:
          - create
          - delete
          - error
      - name: nodes
        namespaces:
          - all
        events:
          - create
          - delete
          - error
    # Check true if you want to receive recommendations
    # about the best practices for the created resource
    recommendations: true

    # Channels configuration
    communications:
      # Settings for Slack
      slack:
        enabled: false
        channel: 'SLACK_CHANNEL'
        token: 'SLACK_API_TOKEN'

      # Settings for Mattermost
      mattermost:
        enabled: true
        url: ''                # URL where Mattermost is running. e.g https://example.com:9243
        token: ''                   # Personal Access token generated by BotKube user
        team: ''                     # Mattermost Team to configure with BotKube
        channel: ''               # Mattermost Channel for receiving BotKube alerts

      # Settings for ELS
      elasticsearch:
        enable: false
        server: 'ELASTICSEARCH_ADDRESS'              # e.g https://example.com:9243
        username: 'ELASTICSEARCH_USERNAME'
        password: 'ELASTICSEARCH_PASSWORD'
        # ELS index settings
        index:
          name: botkube
          type: botkube-event
          shards: 1
          replicas: 0

    # Setting to support multiple clusters
    settings:
      # Cluster name to differentiate incoming messages
      clustername: cluster_name
      # Set true to enable kubectl commands execution
      allowkubectl: false
      # Set false to disable upgrade notification
      upgradeNotifier: true
---
# serviceaccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: botkube-sa
  labels:
    app: botkube
---
# Source: botkube/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: botkube-clusterrole
  labels:
    app: botkube
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "watch", "list"]
---
# clusterrolebinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: botkube-clusterrolebinding
  labels:
    app: botkube
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: botkube-clusterrole
subjects:
- kind: ServiceAccount
  name: botkube-sa
  namespace: botkube
---
# Secret
apiVersion: v1
kind: Secret
metadata:
  name: botkube-secret
  labels:
    app: botkube
data:
  ca-certificates.crt: 'ENCODED_CERTIFICATE '
---
# deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: botkube
  labels:
    component: controller
    app: botkube
spec:
  replicas: 1
  selector:
    matchLabels:
      component: controller
      app: botkube
  template:
    metadata:
      labels:
        component: controller
        app: botkube
    spec:
      serviceAccountName: botkube-sa
      containers:
        - name: botkube
          image: "infracloud/botkube:v0.8.0"
          imagePullPolicy: Always
          volumeMounts:
            - name: config-volume
              mountPath: "/config"
            - name: certs
              mountPath: "/etc/ssl/certs"
          env:
            - name: CONFIG_PATH
              value: "/config/"
            # set one of the log levels- info, warn, debug, error, fatal, panic
            - name: LOG_LEVEL
              value: "info"
            # set BotKube release version
            - name: BOTKUBE_VERSION
              value: v0.8.0
          resources:
            requests:
              cpu: 100m
              memory: "512Mi"
            limits:
              memory: "1024Mi"
      volumes:
        - name: config-volume
          configMap:
            name: botkube-configmap
        - name: certs
          secret:
            secretName: botkube-secret

[PrasadG193]

Just wanted to make sure:

1. Are you passing correct config in

      # Settings for Mattermost
      mattermost:
        enabled: true
        url: ''                # URL where Mattermost is running. e.g https://example.com:9243
        token: ''                   # Personal Access token generated by BotKube user
        team: ''                     # Mattermost Team to configure with BotKube
        channel: ''               # Mattermost Channel for receiving BotKube alerts

2. Are you passing encoded self signed certs in

# Secret
apiVersion: v1
kind: Secret
metadata:
  name: botkube-secret
  labels:
    app: botkube
data:
  ca-certificates.crt: 'ENCODED_CERTIFICATE '

[bilaloslo92]

1. Yes, I am passing Mattermost token, url, etc in that section which works.
2. Yes, I am passing the base64 encoded CA certificate.

Thanks for quick reply.

[PrasadG193]

@bilaloslo92 Are you creating BotKube in botkube namespace?

[PrasadG193]

Also, please verify if secret is getting populated properly with -
kubectl get secret botkube-secret -n botkube -oyaml

[bilaloslo92]

@PrasadG193 - I have tried to create BotKube in multiple namespaces. Is there any restrictions in which namespace it is created in?

I can confirm the secret is populated. BotKube is able to communicate with Mattermost and send notifications, however the pod is restarted because of the error in the initial post. Thats why I am also being spammed by the notification: ...and now my watch begins for cluster clustername!" in my mattermost channel.

[PrasadG193]

@bilaloslo92 If you want to deploy BotKube in a different namespace, you need to modify the namespace field in the ClusterRoleBinding resource accordingly.

The Pod crashing issue indeed looks strange to me. I am not able reproduce this on my setup. I'll keep looking into this.

[bilaloslo92]

@PrasadG193 I have changed the namespace in the ClusterRoleBinding accoundingly. It is very strange, I am also unsure why this happens. I have made as few modifications as nececcary. Hopefully you are able to find out the issue. Do you get any suggestion on what it could be based on the error?

[gangseok514]

Hello, I faced the same issue. I solved with adding event == nil check in mattermost.go.
In fact I don't know why event is nil. :)

[bilaloslo92]

Hi @gangseok514. Good that you managed to reproduce this issue. I hope this can be fixed in the next release, since I don't want to compile it from source. @PrasadG193 - When is next release comming up and could this be included?

[PrasadG193]

@gangseok514 Could you please raise the PR for the fix. Unless there are any surprises, we'll be releasing v0.9.0 by the end of this week or early next week

[bilaloslo92]

@gangseok514 - Could you please create an PR ? Since you managed to solve the issue. So the fix can be released in v0.9.0 :)