✨ Template defaults with envsubst #3270

wfernandes · 2020-06-29T21:41:22Z

What this PR does / why we need it:
This PR introduces support for default variables in the SimpleProcessor. It leverages the library https://github.com/drone/envsubst to parse and evaluate the yaml.

Since the library by default doesn't support ${ VAR}, ${ VAR }, ${VAR }, ${VA$R}, we've added backward compatibility for the first three, however we've dropped support for ${VA$R}.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3189

wfernandes · 2020-06-29T23:01:46Z

/test pull-cluster-api-e2e

cmd/clusterctl/client/yamlprocessor/simple_processor.go

vincepri · 2020-06-30T15:19:20Z

/milestone v0.3.x

fabriziopandini

@wfernandes the PR looks great to me (few small nits)
it will be great to update the clusterct provider contract adding a note about the legacy format (support will be dropped in future) and providing a table of supported syntax for variables (something similar to this but with a brief explanation for each function)

cmd/clusterctl/client/yamlprocessor/simple_processor.go

cmd/clusterctl/client/yamlprocessor/simple_processor_test.go

fabriziopandini

PR looks good, some nits on the doc

cmd/clusterctl/client/yamlprocessor/simple_processor.go

docs/book/src/clusterctl/provider-contract.md

CecileRobertMichon

Thanks for doing this @wfernandes! I will pull your branch and give this a try with some Azure var defaults

docs/book/src/clusterctl/provider-contract.md

CecileRobertMichon · 2020-07-02T22:31:20Z

@wfernandes to clarify, ${AZURE_VNET_NAME:-${CLUSTER_NAME}-vnet} is not supported syntax?

CecileRobertMichon · 2020-07-02T23:45:22Z

opened https://github.com/drone/envsubst/issues/21

fabriziopandini · 2020-07-06T08:52:07Z

/test pull-cluster-api-e2e

wfernandes · 2020-07-06T16:22:07Z

@wfernandes to clarify, ${AZURE_VNET_NAME:-${CLUSTER_NAME}-vnet} is not supported syntax?

@CecileRobertMichon I tested it out and I believe that will work. I built the binary locally and did the following.

$ export CLUSTER_NAME=foobar
$ echo "AzureVnetName: ${AZURE_VNET_NAME:-${CLUSTER_NAME}-vnet}" | ./envsubst
AzureVnetName: foobar-vnet

I can add a test case if that would help.

wfernandes · 2020-07-06T17:03:21Z

@CecileRobertMichon So it seems that I misspoke. As I was writing the test for the above using the library, the use case you mentioned above was failing.
Testing envsubst in bash as I did above doesn't produce the expected results as it evaluates the string before sending it into envsubst. 🙄
So your issue drone/envsubst#21 is still valid.

vincepri · 2020-07-06T19:23:56Z

That'd be a nice addition, let's see if we can add support for it — it should be as easy as having a commit merged to the library and vendoring it, or do we need other special code for it?

wfernandes · 2020-07-07T20:49:54Z

/hold Working on a fix to envsubst to support syntax like ${VAR:-${DEF_VAR}-suffix}

wfernandes · 2020-07-07T22:31:58Z

I created this PR drone/envsubst#23 to add support for the syntax discussed above.

However, I've also updated this PR with a go.mod replace directive to use my fork temporarily. Once upstream drone/envsubst has been merged and tagged, we can switch to that.
@vincepri Let me know if there are any problems with this or if we need to do something more stable for my fork.

@CecileRobertMichon Feel free to test this out and let me know if this works for you. 🙂

/hold cancel

vincepri · 2020-07-07T23:55:20Z

@wfernandes In general, we should wait for the library PR to merge, let's ping the envsubst maintainers to see if they can review and get it merged within the next two weeks

wfernandes · 2020-07-08T16:38:05Z

/hold
Adding this hold since we want to wait for the drone/envsubst library maintainers to get back to us on our PR drone/envsubst#23

vincepri · 2020-07-09T20:56:38Z

@fabriziopandini @CecileRobertMichon @wfernandes Do we want to try to get this in v0.3.7 or wait?

fabriziopandini · 2020-07-09T21:02:12Z

I'm +1 to get this merged with the current version of drone if the new won't get ready, because it will greatly simplify the usage of experimental features by introducing something like

- --feature-gates=ClusterResourceSet={$ENABLE_CLUSTER_RESOURCE_SET:false},MachinePools={$ENABLE_MACHINE_POOLS:false}

CecileRobertMichon · 2020-07-09T21:19:28Z

Just to confirm, this allows defining defaults for the provider components as well?

if so, +1 to getting this in from me as well to be able to test ClusterResourceSet more easily as Fabrizio points out.

vincepri · 2020-07-09T22:08:26Z

Does it? 🤔

wfernandes · 2020-07-09T22:20:04Z

This is a change to the SimpleProcessor which is used by default by the provider components so it should affect those templates. But I'll verify this manually and report back.

Also I got feedback on the drone/envsubst#23 PR so I'll update that and hopefully it will be merged soonish. 🤷

wfernandes · 2020-07-09T22:22:13Z

Also FYI, @vincepri Currently there is a replace directive pointing to my fork of envsubst so that others could pull this down and test it. If this is a "no-no" please let me "know-now". 😆

vincepri · 2020-07-09T22:28:56Z

If the PR doesn't get merged in the envsubst library tomorrow or early next week, we should probably remove the fork and add the support later

wfernandes · 2020-07-09T22:44:05Z

@CecileRobertMichon I did a quick test using the overrides functionality of clusterctl and verified that defaults work for infrastructure-components.yaml

CecileRobertMichon · 2020-07-09T22:45:38Z

Awesome!

envsubst PR just merged 🎉

wfernandes · 2020-07-09T22:45:52Z

@vincepri Lol...as I said that, it just got merged! drone/envsubst#23
OSS for the win!

- This is change to the SimpleProcessor so defaults will be available to cluster templates and provider components. - Create specific errMissingVariables error Since we are using a map, the order of the variables in the error message was changing. This would make the tests flaky since we were doing a fixed substring comparison. So the error which wraps the slice of missing vars is used - Use a specific sha of drone/envsubst until a tag is cut.

wfernandes · 2020-07-09T22:54:31Z

I squashed the commits...let me know if there is any other feedback!

wfernandes · 2020-07-09T22:54:57Z

/hold cancel

CecileRobertMichon · 2020-07-09T22:56:49Z

/lgtm

vincepri · 2020-07-09T23:10:24Z

docs/book/src/clusterctl/provider-contract.md

+as: `${ VAR }`, `${ VAR}`,`${VAR }`. However, these formats will be deprecated
+in the near future. e.g. v1alpha4.
+
+Formats such as `${VAR$FOO}` is not supported.


Isn't this supported now? @CecileRobertMichon @wfernandes

Good point, it is

This is still not supported. There has to be an expression character like =, :=, :- between the variables. This will result in an error.

vincepri

/approve
/milestone v0.3.7

k8s-ci-robot · 2020-07-10T02:10:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 29, 2020

k8s-ci-robot requested review from detiber and fabriziopandini June 29, 2020 21:41

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 29, 2020

wfernandes commented Jun 30, 2020

View reviewed changes

cmd/clusterctl/client/yamlprocessor/simple_processor.go Outdated Show resolved Hide resolved

k8s-ci-robot added this to the v0.3.x milestone Jun 30, 2020

fabriziopandini reviewed Jul 1, 2020

View reviewed changes

cmd/clusterctl/client/yamlprocessor/simple_processor.go Outdated Show resolved Hide resolved

cmd/clusterctl/client/yamlprocessor/simple_processor_test.go Outdated Show resolved Hide resolved

cmd/clusterctl/client/yamlprocessor/simple_processor_test.go Outdated Show resolved Hide resolved

fabriziopandini reviewed Jul 1, 2020

View reviewed changes

cmd/clusterctl/client/yamlprocessor/simple_processor.go Outdated Show resolved Hide resolved

docs/book/src/clusterctl/provider-contract.md Outdated Show resolved Hide resolved

docs/book/src/clusterctl/provider-contract.md Show resolved Hide resolved

wfernandes force-pushed the template-defaults-with-envsubst branch from 915d0db to 2f14a19 Compare July 1, 2020 22:49

CecileRobertMichon reviewed Jul 2, 2020

View reviewed changes

docs/book/src/clusterctl/provider-contract.md Outdated Show resolved Hide resolved

wfernandes force-pushed the template-defaults-with-envsubst branch from 2f14a19 to 5f9fe82 Compare July 6, 2020 17:06

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 7, 2020

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 7, 2020

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 8, 2020

wfernandes force-pushed the template-defaults-with-envsubst branch from 0501ce8 to c1f6c15 Compare July 9, 2020 22:31

wfernandes force-pushed the template-defaults-with-envsubst branch from c1f6c15 to d4907ef Compare July 9, 2020 22:53

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 9, 2020

k8s-ci-robot assigned CecileRobertMichon Jul 9, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 9, 2020

vincepri reviewed Jul 9, 2020

View reviewed changes

vincepri approved these changes Jul 10, 2020

View reviewed changes

k8s-ci-robot modified the milestones: v0.3.x, v0.3.7 Jul 10, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 10, 2020

k8s-ci-robot merged commit e020060 into kubernetes-sigs:master Jul 10, 2020

This was referenced Jul 10, 2020

🌱 Add variables for featuregates #3325

Merged

Make Tilt to work with envsubst style variables #3326

Closed

wfernandes deleted the template-defaults-with-envsubst branch July 10, 2020 14:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Template defaults with envsubst #3270

✨ Template defaults with envsubst #3270

wfernandes commented Jun 29, 2020

wfernandes commented Jun 29, 2020

vincepri commented Jun 30, 2020

fabriziopandini left a comment •

edited

Loading

fabriziopandini left a comment

CecileRobertMichon left a comment

CecileRobertMichon commented Jul 2, 2020

CecileRobertMichon commented Jul 2, 2020

fabriziopandini commented Jul 6, 2020

wfernandes commented Jul 6, 2020

wfernandes commented Jul 6, 2020

vincepri commented Jul 6, 2020

wfernandes commented Jul 7, 2020

wfernandes commented Jul 7, 2020

vincepri commented Jul 7, 2020

wfernandes commented Jul 8, 2020

vincepri commented Jul 9, 2020

fabriziopandini commented Jul 9, 2020

CecileRobertMichon commented Jul 9, 2020

vincepri commented Jul 9, 2020

wfernandes commented Jul 9, 2020

wfernandes commented Jul 9, 2020

vincepri commented Jul 9, 2020

wfernandes commented Jul 9, 2020

CecileRobertMichon commented Jul 9, 2020

wfernandes commented Jul 9, 2020

wfernandes commented Jul 9, 2020

wfernandes commented Jul 9, 2020

CecileRobertMichon commented Jul 9, 2020

vincepri Jul 9, 2020

CecileRobertMichon Jul 9, 2020

wfernandes Jul 10, 2020 •

edited

Loading

vincepri left a comment

k8s-ci-robot commented Jul 10, 2020

✨ Template defaults with envsubst #3270

✨ Template defaults with envsubst #3270

Conversation

wfernandes commented Jun 29, 2020

wfernandes commented Jun 29, 2020

vincepri commented Jun 30, 2020

fabriziopandini left a comment • edited Loading

Choose a reason for hiding this comment

fabriziopandini left a comment

Choose a reason for hiding this comment

CecileRobertMichon left a comment

Choose a reason for hiding this comment

CecileRobertMichon commented Jul 2, 2020

CecileRobertMichon commented Jul 2, 2020

fabriziopandini commented Jul 6, 2020

wfernandes commented Jul 6, 2020

wfernandes commented Jul 6, 2020

vincepri commented Jul 6, 2020

wfernandes commented Jul 7, 2020

wfernandes commented Jul 7, 2020

vincepri commented Jul 7, 2020

wfernandes commented Jul 8, 2020

vincepri commented Jul 9, 2020

fabriziopandini commented Jul 9, 2020

CecileRobertMichon commented Jul 9, 2020

vincepri commented Jul 9, 2020

wfernandes commented Jul 9, 2020

wfernandes commented Jul 9, 2020

vincepri commented Jul 9, 2020

wfernandes commented Jul 9, 2020

CecileRobertMichon commented Jul 9, 2020

wfernandes commented Jul 9, 2020

wfernandes commented Jul 9, 2020

wfernandes commented Jul 9, 2020

CecileRobertMichon commented Jul 9, 2020

vincepri Jul 9, 2020

Choose a reason for hiding this comment

CecileRobertMichon Jul 9, 2020

Choose a reason for hiding this comment

wfernandes Jul 10, 2020 • edited Loading

Choose a reason for hiding this comment

vincepri left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 10, 2020

fabriziopandini left a comment •

edited

Loading

wfernandes Jul 10, 2020 •

edited

Loading