Enhanced monitoring of Argo deployments with Botkube and GitHub integration

[mszostok]

Overview

The goal is to integrate with Argo CD. Besides forwarding the original Argo CD notifications, we want to enhance monitoring of Argo deployments by integrating with Botkube and possibly GitHub.

Phases

Tasks

Beta Give feedback

1. Implement ArgoCD source plugin #1238
   enhancement +1
   
2. Dog-food ArgoCD plugin #1239
   community +1
   

Acceptance Criteria

• Create Argo source plugin
  • Allow to forward standard Argo CD events that are available OOTB by Argo itself.
• Install Argo plugin on our dev cluster, ping dev team to play around with it
• Get feedback from Kubefirst and Testkube team
• Implement enhancements after dog-fooding

Reason

• Users can manage platform secrets and notification in a single place instead of doing that per project, e.g., Kubernetes, Prometheus, Argo, etc.
• User has a more straight forward way to enable Argo notification
• User gain more visibility into Argo deployment process thanks to combining GitHub and Argo events and getting them directly on configured communication platform

[pkosiec]

Here is a quick summary after my short investigation:

Goals

• Have an useful plugin
• Facilitate configuration for user
• Make ArgoCD users feel at home
• Make sure we can keep maintaining our solution

Summary

Here are our options:

Viable and rather reasonable

1. Integrate with ArgoCD natively via webhook notification service
   • https://github.com/argoproj/notifications-engine/blob/master/pkg/services/webhook.go
2. Using Kubernetes API to watch for ArgoCD events, and make user configure such triggers
   - Essentially the same how Argo implements theirs REST API; we could use informers
   • Triggers are NOT based on K8s events. instead, they are based on Application resource properties.
   • This would be way different than on ArgoCD side, doesn't make sense IMO. It would be great to not reinvent the wheel.

Viable but terrible

• Using Kubernetes API and try to use ArgoCD's triggers code
  • looking at the code, it will involve a lot of copying; basically we'd need to run our own version of notification controller from notification-engine and mantain it
    • https://github.com/argoproj/argo-cd/blob/v2.8.1/notification_controller/controller/controller.go
    • https://github.com/argoproj/notifications-engine/blob/master/pkg/controller/controller.go#L250

Others

Contribute custom integration for notifications-engine (like webhook: https://github.com/argoproj/notifications-engine/blob/master/pkg/services/webhook.go) and use it to communicate with Botkube
- I don't see any benefit for this; webhook is fine, but maybe at some point it will be useful

Update after sync with @mszostok

After talking with @mszostok, it seems that the first option is the best. That is: "Integrate with ArgoCD natively via webhook notification service".

1. Argo source plugin configuration will look like more or less as described in the task:

on:
   - trigger: "on-created"
     customTemplate: "" # optional
     enabled: true
     apps: ['foo', 'bar']
     namespace:
       include: ["foo"]

1. User needs to provide trigger name. The trigger needs to be configured already in the cluster.
2. On plugin startup, we:
   • read the trigger name, duplicate it with our message template / custom template provided by the User
   • register our templates
   • create subscription by annotating Applications
     On plugin shutdown we could cleanup the resources.
3. Botkube Agent exposes a generic endpoint handling incoming requests
   • Based on path/header/something else we route the incoming requests to a proper source plugin via gRPC.
   • Source plugin can implement such handling mechanism. We'll use it for the very first time with Argo source plugin.

Tomorrow I'll start working on the first minimal implementation. In the meantime, I'd also like to consult this with @huseyinbabal.

[huseyinbabal]

I am also fine with that option, but we need to address following scenario;

1. Most of the time, Argo CD resources like ArgoApplications is tracked by Argo CD itself
2. If we try to modify a existing Argo resource, e.g. add annotation to subscribe notification, it will be overwritten by Argo
3. Should we encourage to add this subscription thing to their infra code, or it is ok to state this edge case and handle everything as you described above

[pkosiec]

Thank you @huseyinbabal, this is a valid point. I think we can handle that with plugin configuration properties to disable modifying Argo resources (e.g. annotating the app). I'll include that in the implementation 👍