Wait for reconciliation and promote changes across environments

[NasAmin]

Hi,

I am new to GitOps and Flux. Please forgive my ignorance.

I am following this page. In particular, I am looking at the Kubectl wait kustomization command.

I want to build a pipeline like this in GitHub Actions:

• When a pull request is merged to the main branch, I want to trigger a pipeline that updates the flux config for an application e.g. update image tag and push it to the config repo.
• I want to then wait for flux to reconcile those changes and may be use kubectl to check the pods are running successfully.
• I want to run tests against the staging environment via GitHub Actions.
• wait for the tests to finish and get the outcome of the tests.
• If the above conditions are successful then I want to provide a manual approval gate (via GitHub Actions) for production.
• Once manual approval is done, I want to then update production config for flux to reconcile the changes.
• then wait for flux to reconcile
• wait via kubectl to ensure the deployments are successful.
• If either tests fail or flux fails to reconcile, git revert the flux config for each environment.

What's the best way to achieve something like the above?

I'll really appreciate some guidance.

Thanks

[sladkoff]

@NasAmin Have you had any success implementing this?

---

Similar to this, I'm wondering if there is any way to achieve this in combination with Image Update Automation.

In the following example I'm using the following git repositories:

• application code repository (there could be multiple of these)
• cluster state repository

Suppose I have a git repository with my application code with the following CI pipeline:

1. Build
2. Push new docker image
   2.1. This triggers image automation updates to the dev-cluster state repo
3. Wait for new image to be deployed ❓
4. Run tests against dev cluster
5. Promote docker image to production (out of scope, this could be done by re-tagging the docker image...)

Suppose I have a Flux image automation in place that is triggered by step 2 that updates my dev-cluster state and in turn deploys the new image. Would there be any way to notify the CI pipeline of my app-code-repo about the deployment to trigger step 4? Maybe an event can be created and consumed for this purpose?

This promotion use-case seems quite interesting to me. I'm wondering if there's something wrong with my thinking in general? How do others solve this using Flux. Do you need to approach this differently?

Thanks!

Edit

I'm currently thinking that something can be done by organizing the git repositories differently. Sorry for the public brainstorming.

In my previous example the repository structure would have been:

• App A (code)
• App B (code)
• App N (code)
• Platform (flux state repo)
  • k8s deployments + update policies for all App images

When a commit is made to "Platform" (by user or by an image automation) a git commit status notification is sent to the git repo after successful reconciliation. I don't think that this commit status event can trigger any sort of build by itself but maybe a generic webhook event can be used to trigger tests+promotion.

As we're dealing with multiple apps in this case, we need to know which of those should be tested+promoted. With my current knowledge this is not possible with the above setup.

If we re-organize the repos:

• App A (code)
• Deployment A (flux Kustomization repo)
• App B (code)
• Deployment B (flux Kustomization repo)
• App N (code)
• Deployment N (flux Kustomization repo)
• Platform (flux state repo)
  • "the rest of the platform aka infrastructure"

By having per-app gitops repos, we can probably solve the issue of recognizing which service has been updated by an image update policy and trigger respective pipelines to test that version and create a new image for production (image automation on production takes care of the promotion...).

🤷‍♂️ I don't currently have a real flux cluster to test with so this is all only on paper. I'll leave these thoughts out here. Someone correct me if I'm missing something obvious. I'll update this thread if I get to test.