[proposal]: do not store project configuration as k8s secrets

[krancour]

Related to #1034

Currently we store project configuration as Kubernetes secrets. This was probably a convenient choice, as it introduced no dependency on a persistent datastore, however, it also presents certain complications:

• Along with events/builds, contributes to Brigade's "control plane" being unnecessarily constrained to running only in Kubernetes.

• Many end users (developers) who don't have any real Kubernetes experience (or permissions on a cluster, for that matter) could still make good use of Brigade as a CI/CD platform (especially once Declarative support? #1024 is addressed). So it is currently problematic that:

  • Creating/configuring projects requires permissions on the cluster that an end user may not otherwise require.

  • Creating/configuring projects requires more in-depth Kubernetes know how than an end user might otherwise require. (This depends a little on how you create projects-- Helm chart vs brig project create. Arguably brig project create doesn't require in-depth Kubernetes know how, but since it currently offers no ability to edit project configurations once they are created, the practical options are using Helm or directly editing project secrets-- both of which require Kubernetes know how.)

• Automated backup / restore of project configurations requires custom solution. (To be fair, it's not like options for backing up Kubernetes resources-- secrets in this case-- don't exist. They do. But how ubiquitous are they? How familiar are most operators with those options?) In contrast, out-of-the-box solutions exist for backup / restore or many different persistent datastores. In the case that a Brigade operator elects to use a compatible managed service as their datastore, backup / restore functionality is even "free."

[vdice]

Agreed on limitations with current approach. My thinking would be to maintain ability to store config as k8s secrets as one option even when/if other options are developed, if possible.

Indeed, I prefer managing my Brigade projects via the corresponding Helm chart and values files for reasons mentioned above: I can backup/restore from said values files (stored securely in a Keybase Git repo, for example) for easy migration from one cluster to the next. (I also have both the Git history and Helm release mgmt to offer traversing revision history, rollbacks, etc.)

Perhaps as inspiration from this workflow, we could consider adding a more Brigade-specific project description layer (maybe also represented in YAML 😬 ) that would be the canonical representation of a project that in the future would then be interpreted by the underlying storage driver (k8s secrets being the current) for a given Brigade installation. When a different project storage implementation arises, it would consume the same project description file for creation.

[krancour]

My thinking would be to maintain ability to store config as k8s secrets as one option even when/if other options are developed, if possible.

imho, that complicates matters a lot. I'd be fine if that were one possible underlying implementation, but I'd be hoping for it not to be exposed to end users in the way it is today.

Indeed, I prefer managing my Brigade projects via the corresponding Helm chart and values files for reasons mentioned above

I can see the allure in that... and yet that chart was deprecated in favor of brig project create, which is the right workflow if you're not a k8s person-- but it doesn't quite get you all the where to where you need to be in terms of automation or day 2 maintenance of your project.

Perhaps as inspiration from this workflow, we could consider adding a more Brigade-specific project description layer (maybe also represented in YAML 😬 ) that would be the canonical representation of a project that in the future would then be interpreted by the underlying storage driver (k8s secrets being the current) for a given Brigade installation. When a different project storage implementation arises, it would consume the same project description file for creation.

I ❤️ this @vdice. Yes x 💯 .

Let's have a simple project description / config "language" that is agnostic of the underlying datastore.

• Want to define a project? Pass the file to brig?

• Want to edit a project? Update the file and pass it to brig?

• Lost the file? Use brig to extract it from the underlying datastore.

☝️ Totally, 100% compatible with gitops.

[krancour]

This is well covered by the 2.0 proposal, which has been ratified and is now guiding the 2.0 development effort. It probably doesn't make sense to track this as a discrete issue anymore.