Create a helm chart for Contour #2050

michmike · 2019-12-13T00:08:53Z

potentially publish to hub.kubeapps.com

jpeach · 2019-12-13T00:18:14Z

jpeach · 2019-12-19T01:56:57Z

Related #1190

stefanprodan · 2020-01-15T14:46:41Z

There is a contour chart in the helm/charts repo but it uses an older image and the deployment specs don't have the Envoy DaemonSet. Maybe contributing to that one instead of a creating something new would be better.

davecheney · 2020-01-15T23:40:08Z

@stefanprodan I was under the impression that helm was moving away from this central chart repository model. If this is not the cause then it is my overwhelming preference to continue to maintain the original chart.

jpeach · 2020-01-16T00:37:15Z

@pickledrick is the maintainer of the chart in the help repo. I've discussed this with him and we agreed that it makes sense to host the Contour char in the projectcontour Github org, given that the helm charts repo is deprecated.

I think that the next steps here are to agree on the broad outline of a transition plan and talk about how to manage the YAML without too much duplication.

carlisia · 2020-02-11T14:12:32Z

Hey y'all, you might benefit from the Velero's transition plan: vmware-tanzu/helm-charts#2

cortopy · 2020-03-29T18:50:01Z

has the new git repo been created?

After submitting #2374 I am familiar enough to do a first PR with a basic chart if community contributions are ok at this stage

jpeach · 2020-03-29T21:48:51Z

@pickledrick Are you able to work with @cortopy on this? I'm happy to shepherd and do any repository admin that's needed.

jkirklan · 2020-06-04T17:49:42Z

any updates on this?

jpeach · 2020-06-04T22:16:56Z

any updates on this?

Unfortunately no. We need a contributor to take this on. I'll be happy to shepherd changes from anyone who wants to tackle this.

borgoat · 2020-06-05T07:08:04Z

Since there's some interest on this, I can share here some work I've been doing, it's still based on 1.4.0 but I don't expect major changes, and I'm still in the process of making a public chart repository for our company so for now it's just the code here

https://github.com/skeeterhealth/helm-charts/tree/master/charts/contour

jpeach · 2020-06-09T23:57:08Z

@giorgioazzinnaro If you would like to help with an official helm chart for the project, I am happy to shepherd this work. I have a couple of other people (@georgegoh, @robinfoe) interested in helping too.

What I think needs to happen is:

basic agreement on the base chart and set of configuration values
- https://github.com/helm/charts/tree/master/stable/contour
- https://github.com/skeeterhealth/helm-charts/tree/master/charts/contour
some discussion and agreement on how to source the YAML for the charts (it's currently an example and not officially supported)
plan and get all the deployment, repositories and other machinery moving
figure out how to maintain the chart as the underlying YAML changes (hopefully some long-term maintainers)
documentation and user experience guides

borgoat · 2020-06-13T15:55:32Z

Absolutely, I'd be glad to help!
I think we could start from deciding in which repo we want to work so that we can bring all the discussions and code changes there?
Did I understand correctly that we could be using the VMware Tanzu Helm charts repo for this, or would you rather have one dedicated to Contour? (I'm unsure what the project governance looks like..)

jpeach · 2020-06-13T23:30:21Z

On Jun 14, 2020, at 1:55 AM, Giorgio Azzinnaro ***@***.***> wrote: Absolutely, I'd be glad to help! I think we could start from deciding in which repo we want to work so that we can bring all the discussions and code changes there? Did I understand correctly that we could be using the VMware Tanzu Helm charts repo for this, or would you rather have one dedicated to Contour? (I'm unsure what the project governance looks like..)

If we need a repo, I think that it’s best to create it within the projectcontour GitHub org. I can do that for you when you are ready.

georgegoh · 2020-07-01T02:51:26Z

@giorgioazzinnaro are you on kubernetes.slack.com? It would be great to continue the conversation there and get this onboard officially.

stefanprodan · 2020-07-15T07:46:31Z

FYI Bitnami maintains a Contour Helm chart here https://github.com/bitnami/charts/tree/master/bitnami/contour and they build their own container images https://github.com/bitnami/bitnami-docker-contour

I would personally prefer to use the official Contour/Envoy container images, but that chart seems to have all the options one would need to deploy Contour.

michmike · 2020-07-15T13:37:24Z

this is very cool. i didn't know that @stefanprodan . @jpeach , if the bitnami chart is adequate and sufficient, maybe we can start with that and over time help the bitnami team make that helm chart better (if that's necessary)

mkilchhofer · 2020-08-11T16:42:22Z

I use the bitnami chart and override the images and tags without issues. But I had to sync some changes back to their repo:

[bitnami/contour] Add ability to specify ingressClass and restart pods on config change bitnami/charts#3367
[bitnami/contour] Sync upstream changes and chart standardization bitnami/charts#3381

It would be nice, if someone keeps the changes in the manifests here in sync with the helm charts at that time the changes occur.

What I sometimes like about the container images from bitnami is that there are some debugtools in it since they based on minideb/debian. Official images variants with at least busybox in it would be nice - like the guys from fluent-bit for example. For all versions of their images it exists a -debug variant which helps troubleshooting when needed.

georgegoh · 2020-08-12T01:31:02Z

@mkilchhofer I forked the bitnami repo recently and synced up the charts with the canonical manifests as you pointed out.

It's here https://github.com/georgegoh/charts/commits/master, and I'll look into a PR later this week.

youngnick · 2020-08-12T04:48:21Z

@mkilchhofer thanks for the suggestion about debug images, could you log a separate issue for that please? It seems pretty reasonable to me, but I'd like everyone to have a chance to talk about it.

mkilchhofer · 2020-08-12T06:33:36Z

@mkilchhofer thanks for the suggestion about debug images, could you log a separate issue for that please? It seems pretty reasonable to me, but I'd like everyone to have a chance to talk about it.

Done :) #2789

@mkilchhofer I forked the bitnami repo recently and synced up the charts with the canonical manifests as you pointed out.

It's here https://github.com/georgegoh/charts/commits/master, and I'll look into a PR later this week.

Cool 👍 Maybe I then could close bitnami/charts#3381.

georgegoh · 2020-08-13T08:53:37Z

Cool 👍 Maybe I then could close bitnami/charts#3381.

Hi @mkilchhofer I think the changes you made are very close to what I have in my repo. A few differences:

I remove the contour-leaderelection role as it's no longer needed in templates/rbac.yaml.
General matching of the rbac.yaml to the rbacs defined in this project.

I also removed the crds from the templates directory since crd-install is no longer supported in Helm 3(does bitnami require compatibility with Helm 2?)

I could create a new PR after yours is merged to get those items in. What do you think?

stefanprodan · 2020-08-13T09:07:41Z

@georgegoh please take into account that Helm does not support CRDs upgrades when placed in crds dir. Upgrading will be broken if Contour comes with a CRD change, since Helm v3 ignores the crds dir on upgrades.

mkilchhofer · 2020-08-13T10:02:31Z

Hi @mkilchhofer I think the changes you made are very close to what I have in my repo. A few differences:

I remove the contour-leaderelection role as it's no longer needed in templates/rbac.yaml.

General matching of the rbac.yaml to the rbacs defined in this project.

I also removed the crds from the templates directory since crd-install is no longer supported in Helm 3(does bitnami require compatibility with Helm 2?)

I think they want compat to helm2, yes. But I am not an member of bitnami ;) - you need to ask them to be really sure.

I could create a new PR after yours is merged to get those items in. What do you think?

Sure, I then switch the PR back from "draft" to "Ready for review". In the meantime I went through the CRD changes again and dropped all the *.contour.heptio.com CRDs. Note that helm does not touch CRDs when they already exists. Maybe you could mention this bahavior with a reference to your guide Upgrading Contour in the README.md of your PR?

jpeach · 2020-08-13T21:36:35Z

@georgegoh please take into account that Helm does not support CRDs upgrades when placed in crds dir. Upgrading will be broken if Contour comes with a CRD change, since Helm v3 ignores the crds dir on upgrades.

How to helm-based deploys typically handle this then? Do operators need to upgrade CRDs manually?

https://helm.sh/docs/chart_best_practices/custom_resource_definitions/ 🤔

youngnick · 2020-08-14T04:40:38Z

From what I've seen reviewing that link, @jpeach and helm/helm#5781, looks like Helm will do minimal CRD management. To that end, I suggest we consider adding a single URL for just the CRDs for a given version of Contour, similarly to how we have the quickstart URL https://projectcontour.io/quickstart/contour.yaml. Maybe https://projectcontour.io/install/crds.yaml or something?

This would only be for CRDs, because of the ordering requirements for them. This means that the Helm chart will include the most recent version at the time it was cut, but upgrade instructions will include installing the CRDs from the quick location first.

jpeach · 2020-08-14T04:43:14Z

From what I've seen reviewing that link, @jpeach and helm/helm#5781, looks like Helm will do minimal CRD management. To that end, I suggest we consider adding a single URL for just the CRDs for a given version of Contour, similarly to how we have the quickstart URL https://projectcontour.io/quickstart/contour.yaml. Maybe https://projectcontour.io/install/crds.yaml or something?

I think that there's benefit in doing this.

This would only be for CRDs, because of the ordering requirements for them. This means that the Helm chart will include the most recent version at the time it was cut, but upgrade instructions will include installing the CRDs from the quick location first.

For Contour, we basically have an ongoing release cycle. The problem we need to solve is to make it easy and reliable for operators to stay on the upgrade cycle while also being able to customize their installs (at least a bit). So if we say that the upgrade is kubectl apply the latest CRD, then upgrade helm, that's not the best experience IMHO. You won't be able to just helm install on a new cluster, and helm upgrade becomes more complex and risky.

youngnick · 2020-08-14T06:18:33Z

I meant that we include the current version of the CRDs in the Helm chart, but, because of the Helm upgrade problem, we will need to have the upgrade instructions be "Reinstall the CRDs, then do helm upgrade". This is because once the CRDs are present at a particular apigroup and version - ie projectcontour.io/v1, Helm won't reinstall them, because of upgrade concerns.

We can be confident that reinstalling the CRDs is okay in this case, because we control the changes, and will only be adding things (by the GA API guarantee). I don't see any way around something like this for Helm, because of its CRD management behavior.

@georgegoh

As George Goh (@georgegoh) mentioned in issue projectcontour/contour#2050 the leaderelection role is not needed anymore: > 1. I remove the `contour-leaderelection` role as it's no longer needed > in `templates/rbac.yaml`. > 2. General matching of the rbac.yaml to the rbacs defined in this project. Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info>

mkilchhofer · 2020-08-23T12:22:16Z

We can be confident that reinstalling the CRDs is okay in this case, because we control the changes, and will only be adding things (by the GA API guarantee). I don't see any way around something like this for Helm, because of its CRD management behavior.

Another way to update the CRDs could be to do it programatically inside the contour source code. I am not sure if this is a good practice but I think I already saw this in some other applications.

georgegoh · 2020-08-24T01:25:41Z

I've been exploring some possibilities for the CRD upgrade in the contour helm chart, and one way forward could be to separate out CRDs into a sub-chart, and then use that as a dependency for the main contour CRD.

As a test, I created a contour_crds as a sub-chart:

Place untemplated CRD yamls in a sub-chart's templates directory.
Did a helm install.
Modified the annotations in the subchart - updated the version and added an extra field.
Ran helm upgrade.

kubectl get -o yaml crd shows updated annotations in the updated CRD.

Given that there's the GA API guarantee(only adding to the CRD instead of taking stuff away), this could be a possible way to handle CRD upgrades that can work in both Helm 2 and 3.

Another way to update the CRDs could be to do it programatically inside the contour source code. I am not sure if this is a good practice but I think I already saw this in some other applications.

youngnick · 2020-08-24T03:55:54Z

Hmm, that sounds pretty promising @georgegoh!

The corner cases I think are worth testing here are (doesn't seem like there could be any problems, but I've been surprised before):

jumping a few versions in one go for the CRDs
uninstall
we don't really support downgrading of the CRDs, but what would happen if someone downgraded the Contour chart? Would the CRDs want to downgrade as well?

jpeach · 2020-09-01T23:55:45Z

We can be confident that reinstalling the CRDs is okay in this case, because we control the changes, and will only be adding things (by the GA API guarantee). I don't see any way around something like this for Helm, because of its CRD management behavior.

Another way to update the CRDs could be to do it programatically inside the contour source code. I am not sure if this is a good practice but I think I already saw this in some other applications.

Yup, I expect that contour-operator might take that approach.

georgegoh · 2020-09-07T04:09:29Z

Hmm, that sounds pretty promising @georgegoh!

@youngnick sorry it took so long to get back.

The corner cases I think are worth testing here are (doesn't seem like there could be any problems, but I've been surprised before):

jumping a few versions in one go for the CRDs

I think this should work fine as the objects get replaced.

uninstall

This seems to be a problem here. My test steps were:

Install with sub-chart containing CRDs.
Create a httpproxy object.
Uninstall.

What I observed was 1) the CRD was removed even though the httpproxy object was not deleted, 2) the httpproxy object created earlier did not return after I reinstalled the (deleted) chart and the CRDs came back.

we don't really support downgrading of the CRDs, but what would happen if someone downgraded the Contour chart? Would the CRDs want to downgrade as well?

Based on what I seen I would expect that to happen. However, because there's no 'migration' taking place from newer to older versions, we would likely experience some loss of data/functionality.

youngnick · 2020-09-07T04:15:10Z

What I observed was 1) the CRD was removed even though the httpproxy object was not deleted, 2) the httpproxy object created earlier did not return after I reinstalled the (deleted) chart and the CRDs came back.

This is relatively expected behavior; when the CRD itself is removed, all objects of that type will also be removed.

I guess the version upgrade thing will rely on us keeping the API guarantees. :)

These upgrade concerns are particularly important across CRD object version boundaries. I'm worried about what will happen across the CRDv1alpha1 to CRDv1 boundary (when we move HTTPProxy there soon). However, it seems like this approach would work, assuming that the multiple-version upgrades work as they seem to.

georgegoh · 2020-09-07T04:37:52Z

I guess the version upgrade thing will rely on us keeping the API guarantees. :)

Yes - at least until there's an operator :-)

These upgrade concerns are particularly important across CRD object version boundaries. I'm worried about what will happen across the CRDv1alpha1 to CRDv1 boundary (when we move HTTPProxy there soon). However, it seems like this approach would work, assuming that the multiple-version upgrades work as they seem to.

I'll continue work and aim to submit a PR to the bitnami repo by end of week and hope to get some reviews there too, so that we don't get blindsided by other issues.

@georgegoh

) * Sync back most significant upstream projectcontour changes The most important change is the refactoring of the shutdown sidecar: projectcontour/contour@7cd9f4a Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info> * Delete removed CRD ingressroutes.contour.heptio.com > As a reminder, support for IngressRoute was officially dropped in v1.6. > If you haven’t already migrated to HTTPProxy, see the IngressRoute to > HTTPProxy migration guide for instructions on how to do so. Once you have > migrated, delete the IngressRoute and related CRDs. Ref: https://projectcontour.io/resources/upgrading/ Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info> * Sync HTTPProxy CRD Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info> * Sync TLSCertificateDelegation CRD * Drop old TLSCertificateDelegation CRD on API group contour.heptio.com API group contour.heptio.com is no longer supported since 1.6.x * Bump minor chart version as the change is no longer a patch Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info> * Attempt to fix contour tests by upgrading to latest github actions * Override envoy service type to ClusterIP to make tests happy I found a hint in PR #2721 that it should be possible to override values used in GH actions. * Revert "Attempt to fix contour tests by upgrading to latest github actions" This was requested during the review process by dani8art. This reverts commit 253a8ec. * Add bitnami/common as a dependency Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info> * Also use bitnami/common subchart for image * Implement extraVolumes and extraVolumeMounts on contour and envoy Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info> * Implement extraEnvVars on contour and envoy Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info> * Implement extraEnvVarsConfigMap and extraEnvVarsSecret * Implement initContainers for contour and envoy * Implement service.extraPorts on contour and envoy * Implement rolling tags helpers * Sync rbac with upstream As George Goh (@georgegoh) mentioned in issue projectcontour/contour#2050 the leaderelection role is not needed anymore: > 1. I remove the `contour-leaderelection` role as it's no longer needed > in `templates/rbac.yaml`. > 2. General matching of the rbac.yaml to the rbacs defined in this project. Signed-off-by: Marco Kilchhofer <marco@kilchhofer.info> * Update README.md * Fix default values for image repositories * Add quotes (`) * Fix whitespaces in table * Also use bitnami/common subchart for pullSecrets The subchart bitnami/common now supports this (version 0.6.2 and newer). See PR #3566 * Use same tpl functions for job Inside the Deployment we can use templating with 'tpl'. Since we use the same structure (affinity, nodeSelector and tolerations) for the Deployment and the Job, we should also use the same template functions. * Fix: affinity defined twice * Use more specific keyword antiAffinity -> antiAffinityPolicy * Bump chart major: 2.0.0 * Variant2: Implement certgen by using hooks (#2) * Implement certgen via hooks * Delete resources with helm hooks only if needed * Revert "Delete resources with helm hooks only if needed" This reverts commit 76449252d06b7bda0f16c490316478c3fb1004f1. As documented inside the helm docs, we should remove unneeded resources: ~~~ The resources that a hook creates are currently not tracked or managed as part of the release. Once Helm verifies that the hook has reached its ready state, it will leave the hook resource alone. Garbage collection of hook resources when the corresponding release is deleted may be added to Helm 3 in the future, so any hook resources that must never be deleted should be annotated with `helm.sh/resource-policy: keep`. Practically speaking, this means that if you create resources in a hook, you cannot rely upon helm uninstall to remove the resources. To destroy such resources, you need to either add a custom `helm.sh/hook-delete-policy` annotation to the hook template file, or set the time to live (TTL) field of a Job resource. ~~~ * Replace colons with dashes (standardize even more) This was requested during review by @dani8art * Clearify README regarding CRDs and helm v3 * Add small upgrading notes to README * Fix helm2 incompatibility due to PR #2961 Helm2 uses Sprig v2 and therefore the funtion "get" is not available there. Since we need to guarantee helm v2 support, we need to workaround this. * Use consistent component labels on certgen resources * Use bitnami/common for apiVersion of kind Deployment * Rename parameter `contour.createCustomResource` to `contour.installCRDs` * Do not allocate an IP address on metrics services Services for ServiceMonitor do not need to allocate an IP address. We therefore do not waste IPs from the services CIDR pool for this. * Split into subfolders

Helm does not perform any actions on CRDs when a chart is upgraded. Placing the CRDs in a subchart causes them to be processed when the contour chart is upgraded. Discussions about this approach - projectcontour/contour#2050 (comment)

* Place CRDs into a subchart of contour. Helm does not perform any actions on CRDs when a chart is upgraded. Placing the CRDs in a subchart causes them to be processed when the contour chart is upgraded. Discussions about this approach - projectcontour/contour#2050 (comment) * Bump version to 4.0.0 as there is a breaking change in how CRDs are installed and upgraded(see README). * Changed the repository location for contour-crds - this lives as top-level in charts. * Update Chart and add subchart. * Move contour-crds back inside contour chart. * Bump the chart version and fix trailing spaces. * Updated README to indicate notable changes and steps required for upgrading from 3.x to 4. * Move the 4.0.0 notes to the 'Upgrade' section. * Move out contour-crds subchart to manage CRDs in a separate `resources` directory. Add a `contour.manageCRDs` flag to indicate whether you want the chart to manage CRDs or not. * Remove the obsolete(Helm v2) CRD options from the README and values.yaml. Document the new `manageCRDs` option.

xaleeks · 2021-02-03T21:20:06Z

Is this still a priority given our efforts around the Contour operator. I see the operator design already covers both the Service API and non- Service APIs use case which is the most important thing as we in ease in adoption. What additional advantages does the helm chart provide here. It would just be another deployment we'd have to maintain as a repo under the Contour project

youngnick · 2021-02-04T04:20:35Z

The Helm chart that @georgegoh has built lives in Bitnami's chart repo. Since there is now no centralised chart repository, this gives us some Helm functionality at least.

I'd anticipate that as we make the operator stable, we can move the helm chart to use that instead.

…i#3665) * Place CRDs into a subchart of contour. Helm does not perform any actions on CRDs when a chart is upgraded. Placing the CRDs in a subchart causes them to be processed when the contour chart is upgraded. Discussions about this approach - projectcontour/contour#2050 (comment) * Bump version to 4.0.0 as there is a breaking change in how CRDs are installed and upgraded(see README). * Changed the repository location for contour-crds - this lives as top-level in charts. * Update Chart and add subchart. * Move contour-crds back inside contour chart. * Bump the chart version and fix trailing spaces. * Updated README to indicate notable changes and steps required for upgrading from 3.x to 4. * Move the 4.0.0 notes to the 'Upgrade' section. * Move out contour-crds subchart to manage CRDs in a separate `resources` directory. Add a `contour.manageCRDs` flag to indicate whether you want the chart to manage CRDs or not. * Remove the obsolete(Helm v2) CRD options from the README and values.yaml. Document the new `manageCRDs` option.

jpeach mentioned this issue Jan 8, 2020

Move the deployment YAML to kustomize #2088

Closed

jpeach mentioned this issue Feb 4, 2020

Add Contour Operator Project #2187

Closed

jpeach added the area/deployment Issues or PRs related to deployment tooling or infrastructure. label Feb 9, 2020

jpeach added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jun 4, 2020

jpeach added the lifecycle/accepted Denotes an issue that has been triaged and determined to be valid. label Jun 4, 2020

mkilchhofer mentioned this issue Aug 12, 2020

Provide debug variants of contour container images #2789

Closed

mkilchhofer mentioned this issue Aug 19, 2020

[bitnami/contour] Sync upstream changes and chart standardization bitnami/charts#3381

Merged

4 tasks

georgegoh mentioned this issue Sep 14, 2020

[bitnami/contour] Change CRDs approach to handle CRD upgrades bitnami/charts#3665

Merged

1 task

georgegoh mentioned this issue Nov 13, 2020

Add a Helm option to the Getting Started guide. #3125

Merged

Create a helm chart for Contour #2050

Create a helm chart for Contour #2050

Comments

michmike commented Dec 13, 2019

jpeach commented Dec 13, 2019

jpeach commented Dec 19, 2019

stefanprodan commented Jan 15, 2020

davecheney commented Jan 15, 2020

jpeach commented Jan 16, 2020 • edited Loading

carlisia commented Feb 11, 2020

cortopy commented Mar 29, 2020

jpeach commented Mar 29, 2020

jkirklan commented Jun 4, 2020

jpeach commented Jun 4, 2020

borgoat commented Jun 5, 2020

jpeach commented Jun 9, 2020 • edited Loading

borgoat commented Jun 13, 2020

jpeach commented Jun 13, 2020 via email

georgegoh commented Jul 1, 2020

stefanprodan commented Jul 15, 2020

michmike commented Jul 15, 2020

mkilchhofer commented Aug 11, 2020 • edited Loading

georgegoh commented Aug 12, 2020

youngnick commented Aug 12, 2020

mkilchhofer commented Aug 12, 2020

georgegoh commented Aug 13, 2020

stefanprodan commented Aug 13, 2020

mkilchhofer commented Aug 13, 2020 • edited Loading

jpeach commented Aug 13, 2020 • edited Loading

youngnick commented Aug 14, 2020

jpeach commented Aug 14, 2020 • edited Loading

youngnick commented Aug 14, 2020

mkilchhofer commented Aug 23, 2020

georgegoh commented Aug 24, 2020

youngnick commented Aug 24, 2020

jpeach commented Sep 1, 2020

georgegoh commented Sep 7, 2020

youngnick commented Sep 7, 2020

georgegoh commented Sep 7, 2020

xaleeks commented Feb 3, 2021

youngnick commented Feb 4, 2021

jpeach commented Jan 16, 2020 •

edited

Loading

jpeach commented Jun 9, 2020 •

edited

Loading

mkilchhofer commented Aug 11, 2020 •

edited

Loading

mkilchhofer commented Aug 13, 2020 •

edited

Loading

jpeach commented Aug 13, 2020 •

edited

Loading

jpeach commented Aug 14, 2020 •

edited

Loading