TASK: Investigate and document method for testing locally-modified baseline helm charts in ArgoCD

[ewilkins-csi]

Description

We were testing changes to Helm charts under extensions-helm locally by changing the repository field to file:///path/to/modified/chart/, as noted in the extensions-helm README. With the move away from Tilt and to ArgoCD this technique will no longer work. We need an alternative approach.

DOD

• Find a technique for testing modifications to baseline charts prior to publishing
  • Note: We currently do have an approach that involves manually moving the chart tarball from the baseline project to the test project and committing/pushing to test project repo, but this is a bit cumbersome and can hard to tell/remember what version of the chart you're actually using
• Update the extensions-helm README with this approach

Test Strategy/Script

1. Make a change to extensions/extensions-helm/aissemble-kafka-chart/values.yaml that will be easy to verify (e.g. fullnameOverride)
2. Commit the change to a test branch and push to GitHub
3. Create a repo in your GitHub account called final-488
   • make the project public to avoid auth complications
4. Create a new project from the archetype

mvn archetype:generate '-DarchetypeGroupId=com.boozallen.aissemble' \
                       '-DarchetypeArtifactId=foundation-archetype' \
                       '-DarchetypeVersion=1.11.0-SNAPSHOT' \
                       '-DgroupId=org.test' \
                       '-Dpackage=org.test' \
                       '-DprojectGitUrl=<YOUR PROJECT URL>' \
                       '-DprojectName=Test Project' \
                       '-DartifactId=final-488'

5. Run the build ignoring any manual actions: mvn clean install
6. Follow the instructions in the extensions-helm README to point the kafka-cluster chart to your test branch
   • This should involve modifications to final-488-infrastructure/values.yaml and final-488-deploy/src/main/resources/apps/kafka-cluster/Chart.yaml
7. Add Chart.lock to .gitignore
8. Commit and push your changes

   git init
   git add .
   git commit -m'initial commit'
   git remote add origin <YOUR PROJECT URL>
   git branch -M main
   git push -u origin main

9. Deploy the project

   ./deploy.sh up

10. Navigate to ArgoCD at localhost:30080
11. Select the kafka deployment and verify the changed default value is present

[ewilkins-csi]

Definition of Done with @csun-cpointe and @carter-cundiff

[ewilkins-csi]

The most straight-forward answer to this issue would be to simply refer to the chart in the GitHub repo at the specific dev branch. Unfortunately, this is not supported by Helm out of the box. It has a storied history and is generally considered a good improvement to Helm, however the work has not landed. (It was "one of the longest standing" PRs in 2021.)

There has been some recent-ish work to revive the proposal. The linked PR and associated HIP do a good job of documenting the history, and also pointed out several Helm plugins that achieve this functionality. While I think it makes sense for us to keep an eye on the progress and perhaps provide help where we can, in the short term we can use one of the plugins mentioned.

Currently in the process of testing the most popular/frequently updated plugin now and updating docs accordingly.

[ewilkins-csi]

I got this working with the aslafy-z/helm-git plugin but there were a few "gotchas".

• The version of the ArgoCD chart we are on sets the cache and data directories for helm to the same location, which causes plugin install issues as the cached github download and the plugin are duplications
  • Work-around is to explicitly set the plugins directory to a different volume mount
• There is an existing issue with committing Chart.lock files when you're using an explicit repository vs a repository alias
  • Just don't check in Chart.lock 🙃

[ewilkins-csi]

As an outflow of this ticket, we probably want to add Chart.lock to the .gitignore file for all of the downstream charts. Because downstream projects are leveraging GitOps and not packaging/publishing their charts, it doesn't make sense to commit Chart.lock.

[ewilkins-csi]

@jaebchoi had luck with just pushing the helm chart tarball to his personal GH OCI repo with helm push. It's unclear why authentication wasn't an issue, however this approach doesn't satisfy concern noted in the DoD: can be hard to tell/remember what version of the chart you're actually using. So it's probably still better to use the plugin approach when iterating during development.

[ewilkins-csi]

OTS with @csun-cpointe and @carter-cundiff

[csun-cpointe]

Final test passed!!

[ewilkins-csi]

These changes were at least partially reverted in #513