Vault and External Secrets Operator integration #98

schnatterer · 2022-11-10T10:04:09Z

Implements optional Vault and External Secrets Operator integration via a --vault=dev|prod parameter.
Initializes Vault with static secrets in dev mode via post-startup hook
Adds example stage-specific SecretStores to ArgoCD and ExternalSecrets to an example app (ArgoCD-only)
Mounts secret into example app, exposes via HTTP for demo purposes
Extends docs .Also creates a simpler diagram with all features at the top.
Note that the diagrams in README won't show until merged to master (because we use the plantuml proxy for rendering)

Can be tested with preview image: ghcr.io/cloudogu/gitops-playground:159954b

In order to implment this feature properly, some major refactorings needed to be done:

2eaee63
- Simplify package structure of Grovvy code. There were more packages than classes. Well, almost. 2eaee63
- modules were renamed to features to better reflect what they are to the GitOps playgrounds, e.g. monitoring and secrets managements are optional features
- Disentangled MetricsModule. Mailhog is always on. ArgoCD Notifications is always on. Only Prometheus Stack is enabled/disabled.
ef07e1c Mailhog and Prometheus Stack can now be deployed for all GitOps operators
49fe9bd Create a declarative default configuration in ApplicationConfigurator -> this holds a lot of constants that could be overwritten easily using a config file, no need to add dozens of CLI params.
b86fc65 CommandExecutor now fails on error, instead of silently continuing. A bit like set -o errexit. This lead to a couple of improvments for idempotence. e.g. where something failed in the past but was never recognized. Example: git push of control app.
b4559cd Make GitClient more generic: Remove control-app repo specifics, make explicitly stateful. Rename to ScmmRepo to better match what it is. A representation of a repostitory in SCMM, not a generic client
bfa2f87 Remove code for setting the SCMM default branch

Minor unrelated fixes

Jenkins upgrade 4d95f75
Avoid CVE-2022-37865 in ivy e5fe04e
Fix app helm-dependency for local deployment 3851813
Fix example app: nginx-helm-jenkins 0cd0567
Add missing README to control-app d9cd9a6
Add overview of dependencies 103ca34

- Rename project "monitoring" to more generic "system" - Move apps in "system" project to common folder in applications Avoid a bigger mess before adding ESO and vault operators to project.

In control-app and in repo. Prepares for more subfolders for ESO and vault, where there will be more sub folders/applications. webconfig-cm.yaml seems not to be needed.

- Simplify package structure - Rename module to feature - Rename metrics to argocd (because without the metrics module, the control-app repo would be empty!) - Make features tell if they are enabled or disabled. - Provide option for disabling features - Remove unnecessary classes: - Exception (we only need a simple error message here!) - ModuleRepository: What was it's purpose? Now part of Application - ApplicationTest: What did it test anyway? Log messages?!

Install via Helm directly (not ArgoCD), because - Difficult to implement apply of secret-store.yaml -> separate repo necessary? - This way ESO can be used with all operators, not only argoCD

E.g. when playground is applied a 2nd time, #idempotence

As part of the control app

Advantage: * Can be used from all GitOps operators in playground * Uniform way, sam method of deployment as for Jenkins, SCMM, Registry, External Secrets, Vault

Mainly to store e.g. helm chart version in single place. But also this allows for reading configuration from a file in the future. Introduce more generic terms monitoring and secrets in internal config. Can't ue application.yml, because it will not be bundled into static image.

E.g. when playground is applied a 2nd time, #idempotence Also extends command executor to also return stderr and exit code. Allow for suppressing failure. This was done because a different implementation for areChangesStagedForCommit() would be: "git update-index --refresh && git diff-index --exit-code HEAD --" which return exit code 1. A requirement such as ignoring failing commands might come back in the future so let's keep it.

- Install chart - Create Token secrets for secretStores

- Create external secrets - Integrate secret into example - Move example app repo creation from apply.sh to groovy - Make repo code init testable - Add missing ArgoCD.groovy tests -> Fail "GitClient" is stateful. Needs refactoring

4.2.9 is no longer working (even with default values) because of plugin dependency "Plugin workflow-aggregator ... depends on configuration-as-code:1559.v38a_b_2e3b_6b_b_7, but there is an older version defined on the top level"

The constructor now tells us, that it's bound to a repo and we need to use multiple instances for multiple repos.

This reflects more clearly what it is: It is not a client, it is a single repo (including the client) and has SCMM-specifics.

Newer versions of SCMM use "main" by default. We might want to bring this back when we migrate the SCMM API calls from bash to groovy. Or we use the SCMM client, or native Groovy HTTP methods.

Never finished syncing in argocd, because service remained on type LB, where no external IP can be received on local cluster.

Timing problem: Hook is executed before Vault HTTP is up. This leads to the hook failing, which leads to the pod restarting. Eventually we have a crash loop backoff. Solution: Wait for HTTP to become ready.

Staging app was missing, was pushed to wrong path in gitops repo.

External Secrets Operator seems to fail on properties containing minus, such as: property: nginx-secret

Group by repo. Easier to maintain, because there are a number of other repos waiting to be migrated from apply.sh

Avoid errors such as: Error: Service "vault-ui" is invalid: spec.ports[0].nodePort: Invalid value: 8200: provided port is not in the valid range. The range of valid ports is 30000-32767

So we hopefully can keep better track of what needs to be kept uptodate

Latest versions finally fix Jenkins builds on k8s nodes no longer work

🤷‍♂️ Exception in thread "main" com.oracle.svm.core.jdk.UnsupportedFeatureError: Runtime reflection is not supported for public abstract java.util.Iterator java.util.List.iterator() at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:89) at java.lang.reflect.Method.acquireMethodAccessor(Method.java:77) at java.lang.reflect.Method.invoke(Method.java:566) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035) at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:1017) at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1008) at org.codehaus.groovy.runtime.InvokerHelper.asIterator(InvokerHelper.java:609) at org.codehaus.groovy.runtime.DefaultGroovyMethods.removeAll(DefaultGroovyMethods.java:5171) at com.cloudogu.gitops.features.argocd.ArgoCD.removeObjectFromList(ArgoCD.groovy:117)

Because SCMM and Jenkins won't work without the fixes from this branch.

Bitnami removes all versions older than 6 months from their helm chart index. bitnami/charts#10833 To make sure we're not forced to update again, switch to full index.

# Conflicts: # scripts/jenkins/init-jenkins.sh

schnatterer · 2023-01-13T16:23:48Z

Merged Branch feature/argocd-2.5 into vault, Otherwise Jenkins and SCMM would no longer have worked. Cherry picking of 5 commits would have been to much for my taste. Also the argocd-2.5 branch has been battle-tested in a training.
Had to update NGINX Helm Charts, because Bitnami removes all versions older than 6 months from their helm chart index. To make sure we're not forced to update again, switch to full index.
Switched from using the root token to using k8s service accounts and a user account.
- For one this allows arbitrary --passwords. As the root token is also used as an ID, having special character in --password resulted in Error initializing Dev mode: failed to create root token with ID "<PW with special chars>".
- Not using the root token is much more realistic!
- If necessary, the root token can be found on the log

* For one this allows arbitrary `--password`s. As the `root` token is also used as an ID, having special character in `--password` resulted in `Error initializing Dev mode: failed to create root token with ID "<PW with special chars>"`. * Not using the root token is much closer to production! * If necessary, the root token can be found on the vault-0 pod log

And increased timeout. Hopefully builds will be less flaky in the future

transfer vault values.yaml into groovy class Signed-off-by: pmarkiewka <philipp.markiewka@cloudogu.com>

schnatterer

Did it just work without namespaceOverride?

…lib#23 cloudogu/gitops-build-lib#23 Signed-off-by: pmarkiewka <philipp.markiewka@cloudogu.com>

Signed-off-by: pmarkiewka <philipp.markiewka@cloudogu.com>

schnatterer added 30 commits October 25, 2022 12:35

Add TODO for notifications vs metrics

2423270

Implement parameter for vault

916c541

Control-app repo: Prepare restructuring.

405ff36

- Rename project "monitoring" to more generic "system" - Move apps in "system" project to common folder in applications Avoid a bigger mess before adding ESO and vault operators to project.

Create folder "system".

f8c59dd

In control-app and in repo. Prepares for more subfolders for ESO and vault, where there will be more sub folders/applications. webconfig-cm.yaml seems not to be needed.

Docker: Make sure maven caching works

d9b1a6b

Install ExternalSecretsOperator.

bdd0936

Install via Helm directly (not ArgoCD), because - Difficult to implement apply of secret-store.yaml -> separate repo necessary? - This way ESO can be used with all operators, not only argoCD

Fix GitClient.commitAndPush() fails if repo not empty.

ba372ab

E.g. when playground is applied a 2nd time, #idempotence

Secret Store: Deploy one per namespace via ArgoCD.

c0aee83

Deploy ArgoCD Dashboard with argoCD

517b0e4

As part of the control app

Deploy Mailhog and Prometheus stack imperatively

ef07e1c

Advantage: * Can be used from all GitOps operators in playground * Uniform way, sam method of deployment as for Jenkins, SCMM, Registry, External Secrets, Vault

Install and configure Hashicorp Vault

ff13e17

- Install chart - Create Token secrets for secretStores

Start implementing vault example.

f7ae9a8

- Create external secrets - Integrate secret into example - Move example app repo creation from apply.sh to groovy - Make repo code init testable - Add missing ArgoCD.groovy tests -> Fail "GitClient" is stateful. Needs refactoring

Upgrade Jenkins to Chart 4.2.12

4d95f75

4.2.9 is no longer working (even with default values) because of plugin dependency "Plugin workflow-aggregator ... depends on configuration-as-code:1559.v38a_b_2e3b_6b_b_7, but there is an older version defined on the top level"

Make GitClient explicitly stateful.

132b6a2

The constructor now tells us, that it's bound to a repo and we need to use multiple instances for multiple repos.

Rename GitClient to SCMM-Repo.

b4559cd

This reflects more clearly what it is: It is not a client, it is a single repo (including the client) and has SCMM-specifics.

Setting the SCMM default branch is no longer necessary

bfa2f87

Newer versions of SCMM use "main" by default. We might want to bring this back when we migrate the SCMM API calls from bash to groovy. Or we use the SCMM client, or native Groovy HTTP methods.

Fix app helm-dependency for local deployment

3851813

Never finished syncing in argocd, because service remained on type LB, where no external IP can be received on local cluster.

Fix vault not starting up.

4c3604e

Timing problem: Hook is executed before Vault HTTP is up. This leads to the hook failing, which leads to the pod restarting. Eventually we have a crash loop backoff. Solution: Wait for HTTP to become ready.

Fix example app: nginx-helm-jenkins

0cd0567

Staging app was missing, was pushed to wrong path in gitops repo.

Avoid CVE-2022-37865 in ivy for dev image

e5fe04e

Fix ExternalSecret not synced

97af581

External Secrets Operator seems to fail on properties containing minus, such as: property: nginx-secret

ArgoCD.groovy add missing features and tests

edcd698

Refactor ArgoCD.enable()

067dc56

Group by repo. Easier to maintain, because there are a number of other repos waiting to be migrated from apply.sh

Add missing README to control-app

d9cd9a6

Update README, add feature diagram

cbb499a

Vault: Set nodePort only for local cluster

25a9766

Avoid errors such as: Error: Service "vault-ui" is invalid: spec.ports[0].nodePort: Invalid value: 8200: provided port is not in the valid range. The range of valid ports is 30000-32767

Add overview of dependencies

c505390

So we hopefully can keep better track of what needs to be kept uptodate

schnatterer added 6 commits January 11, 2023 11:25

Upgrade Kubernetes plugin for Jenkins. Again.

5d9b8c1

Latest versions finally fix Jenkins builds on k8s nodes no longer work

Upgrade Dockerfile dependencies

0a80a07

Merge branch 'feature/argocd-2.5' into feature/vault

db5138f

Because SCMM and Jenkins won't work without the fixes from this branch.

Update NGINX helm charts.

fc71d7a

Bitnami removes all versions older than 6 months from their helm chart index. bitnami/charts#10833 To make sure we're not forced to update again, switch to full index.

Merge branch 'main' into feature/vault

1dbe375

# Conflicts: # scripts/jenkins/init-jenkins.sh

schnatterer force-pushed the feature/vault branch from 26e6ffb to 1dbe375 Compare January 11, 2023 15:00

schnatterer force-pushed the feature/vault branch 2 times, most recently from 1a3dbf8 to 46fabee Compare January 16, 2023 15:33

schnatterer added 5 commits January 16, 2023 17:28

Port logic for SCMM user+PW from apply.sh to groovy

bcc4f27

Upgrade trivy after some timed-out builds

7d38c28

And increased timeout. Hopefully builds will be less flaky in the future

README: Fix PlantUMLs

2e09054

README: Extend docs for secrets

159954b

schnatterer force-pushed the feature/vault branch from 46fabee to 159954b Compare January 16, 2023 16:29

schnatterer mentioned this pull request Jan 17, 2023

Feature/argocd 2.5 #103

Closed

schnatterer force-pushed the feature/vault branch from e2baf6d to a61b5aa Compare January 17, 2023 16:51

Polishing during review

1f197de

schnatterer force-pushed the feature/vault branch 2 times, most recently from a4afdbc to 06ef921 Compare January 18, 2023 08:14

Try to increase trivy scanning speed

42a48d9

schnatterer force-pushed the feature/vault branch from 06ef921 to 42a48d9 Compare January 18, 2023 08:22

Review polishing

e6fa263

transfer vault values.yaml into groovy class Signed-off-by: pmarkiewka <philipp.markiewka@cloudogu.com>

schnatterer commented Jan 25, 2023

View reviewed changes

pmarkiewka added 3 commits January 26, 2023 10:30

We need namespaceOverride because of this bug: cloudogu/gitops-build-…

8c6bfb8

…lib#23 cloudogu/gitops-build-lib#23 Signed-off-by: pmarkiewka <philipp.markiewka@cloudogu.com>

Fix indents

268a2dd

Signed-off-by: pmarkiewka <philipp.markiewka@cloudogu.com>

Fix indents

7e630c1

Signed-off-by: pmarkiewka <philipp.markiewka@cloudogu.com>

pmarkiewka merged commit 2cbd994 into main Jan 26, 2023

pmarkiewka deleted the feature/vault branch January 26, 2023 09:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vault and External Secrets Operator integration #98

Vault and External Secrets Operator integration #98

schnatterer commented Nov 10, 2022 •

edited

Loading

schnatterer commented Jan 13, 2023

schnatterer left a comment

Vault and External Secrets Operator integration #98

Vault and External Secrets Operator integration #98

Conversation

schnatterer commented Nov 10, 2022 • edited Loading

schnatterer commented Jan 13, 2023

schnatterer left a comment

Choose a reason for hiding this comment

schnatterer commented Nov 10, 2022 •

edited

Loading