Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: specify env values from the parent to the nested state #622

Merged
merged 3 commits into from
May 29, 2019

Conversation

mumoshu
Copy link
Collaborator

@mumoshu mumoshu commented May 29, 2019

Adds the helmfiles[].environment.values that accepts a mix of file pathes and inline dicts:

helmfiles:
- path: path/to/nested/helmfile.yaml
  environment:
    values:
    - key1: val1
    - values.yaml

The values files are loaded in the context of the parent state file. For example, in case the above state file is located at /path/to/helmfile.yaml,
values.yaml is located at /path/to/values.yaml instead of /path/to/nested/values.yaml.

Resolves #523


I've also included the fix for #615 and the another fix that fixes the regression due to that. I've included all hese 3 commits into this one PR, because all of them correlate to each other(I think).

Adds the `helmfiles[].environment.values` that accepts a mix of file pathes and inline dictes:

```yaml
helmfiles:
- path: path/to/nested/helmfile.yaml
  environment:
    values:
    - key1: val1
    - values.yaml
```

The values files are loaded in the context of the parent state file. For example, in case the above state file is located at `/path/to/helmfile.yaml`,
`values.yaml` is located at `/path/to/values.yaml` instead of `/path/to/nested/values.yaml`.

Resolves #523
The latest commit broke any state files like the below to NOT pass env value overrides at all:

```
helmfiles:
- path: nested/state.yaml
  environment:
    values:
    - overrides.yaml
```

This fixes the issue.
@mumoshu mumoshu merged commit 1226ea6 into master May 29, 2019
@mumoshu mumoshu deleted the env-propagation-and-fix-env-merging branch May 29, 2019 10:09
@skaji
Copy link

skaji commented May 29, 2019

# dev.yaml
foo:
  bar: 1
# helmfile.yaml
environments:
  dev:
    values:
      - dev.yaml

Then:

❯ ./helmfile --version
helmfile version v0.68.0

❯ ./helmfile -e dev diff
panic: json: unsupported type: map[interface {}]interface {} [recovered]
	panic: json: unsupported type: map[interface {}]interface {}

goroutine 1 [running]:
github.com/urfave/cli.HandleAction.func1(0xc0000d2de0)
	/go/pkg/mod/github.com/urfave/cli@v0.0.0-20160620154522-6011f165dc28/app.go:474 +0x287
panic(0x14583a0, 0xc00005e390)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/roboll/helmfile/environment.Environment.DeepCopy(0x7ffeefbff6fe, 0x3, 0xc0000742a0, 0x1319a7c, 0xc000099650, 0xff)
	/home/circleci/workspace/helmfile/environment/environment.go:18 +0x12b
github.com/roboll/helmfile/environment.(*Environment).Merge(0xc0001c66a8, 0x0, 0x14d9408, 0x17, 0xc0000d1c90)
	/home/circleci/workspace/helmfile/environment/environment.go:38 +0x142
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).twoPassRenderTemplateToYaml(0xc00008a060, 0xc00017c020, 0x0, 0x14d1ccc, 0x1, 0xc000180020, 0x14, 0xc00017a000, 0x32, 0x232, ...)
	/home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:86 +0x133
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderTemplatesToYamlWithEnv(...)
	/home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:62
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderTemplatesToYaml(0xc00008a060, 0x14d1ccc, 0x1, 0xc000180020, 0x14, 0xc00017a000, 0x32, 0x232, 0xc00017c000, 0x1, ...)
	/home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:58 +0xd2
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderAndLoad(0xc00008a060, 0x0, 0x0, 0x14d1ccc, 0x1, 0xc000176002, 0xd, 0xc00017a000, 0x32, 0x232, ...)
	/home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:170 +0x6d3
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).loadFileWithOverrides(0xc00008a060, 0x0, 0x0, 0x14d1ccc, 0x1, 0x14d52bc, 0xd, 0x401, 0x1b0bd00, 0x203000, ...)
	/home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:110 +0x1f4
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).Load(0xc00008a060, 0x14d52bc, 0xd, 0x18a6328, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:59 +0x473
github.com/roboll/helmfile/pkg/app.(*App).loadDesiredStateFromYaml(0xc0000c35e0, 0x14d52bc, 0xd, 0xc0000d24c0, 0x1, 0x1, 0xc0000d2338, 0xc000000000, 0xc0000d2390)
	/home/circleci/workspace/helmfile/pkg/app/app.go:140 +0x1c6
github.com/roboll/helmfile/pkg/app.(*App).VisitDesiredStates.func1(0x14d52bc, 0xd, 0xc000178040, 0x37, 0x0, 0x0)
	/home/circleci/workspace/helmfile/pkg/app/app.go:147 +0xf3
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles.func1(0x0, 0xc0000d2640)
	/home/circleci/workspace/helmfile/pkg/app/app.go:111 +0x9a
github.com/roboll/helmfile/pkg/app.(*App).within(0xc0000c35e0, 0x14d1ccc, 0x1, 0xc000172000, 0xc0000d2838, 0x2)
	/home/circleci/workspace/helmfile/pkg/app/app.go:53 +0x722
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles(0xc0000c35e0, 0x0, 0x0, 0xc000090720, 0x10, 0x1475180)
	/home/circleci/workspace/helmfile/pkg/app/app.go:105 +0x29f
github.com/roboll/helmfile/pkg/app.(*App).VisitDesiredStates(0xc0000c35e0, 0x0, 0x0, 0x18a6328, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/circleci/workspace/helmfile/pkg/app/app.go:146 +0xd4
github.com/roboll/helmfile/pkg/app.(*App).VisitDesiredStatesWithReleasesFiltered(0xc0000c35e0, 0x0, 0x0, 0xc0000b3340, 0x0, 0x0)
	/home/circleci/workspace/helmfile/pkg/app/app.go:235 +0x10e
github.com/roboll/helmfile/cmd.FindAndIterateOverDesiredStatesUsingFlagsWithReverse(0xc0000c4f00, 0xc000095400, 0xc0000954c0, 0x8, 0xc0000b32e0)
	/home/circleci/workspace/helmfile/cmd/cmd.go:77 +0xe6
main.findAndIterateOverDesiredStatesUsingFlags(...)
	/home/circleci/workspace/helmfile/main.go:633
main.main.func3(0xc0000c4f00, 0x0, 0x0)
	/home/circleci/workspace/helmfile/main.go:183 +0x66
reflect.Value.call(0x14464e0, 0x14efdc0, 0x13, 0x14d21a8, 0x4, 0xc0000d2d80, 0x1, 0x1, 0xc000092040, 0x1011bc3, ...)
	/usr/local/go/src/reflect/value.go:447 +0x461
reflect.Value.Call(0x14464e0, 0x14efdc0, 0x13, 0xc0000d2d80, 0x1, 0x1, 0xc0000f4c30, 0xc0000f4c98, 0x140)
	/usr/local/go/src/reflect/value.go:308 +0xa4
github.com/urfave/cli.HandleAction(0x14464e0, 0x14efdc0, 0xc0000c4f00, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli@v0.0.0-20160620154522-6011f165dc28/app.go:483 +0x1ff
github.com/urfave/cli.Command.Run(0x14d22e8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x14e67c1, 0x35, 0x0, ...)
	/go/pkg/mod/github.com/urfave/cli@v0.0.0-20160620154522-6011f165dc28/command.go:186 +0x8d1
github.com/urfave/cli.(*App).Run(0xc000156180, 0xc000092000, 0x4, 0x4, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli@v0.0.0-20160620154522-6011f165dc28/app.go:237 +0x601
main.main()
	/home/circleci/workspace/helmfile/main.go:592 +0x21e6

@sgandon
Copy link
Contributor

sgandon commented May 29, 2019

I am sorry I don't understand what this does ?

@sgandon
Copy link
Contributor

sgandon commented May 29, 2019

ha I guess you meant this

- path: path/to/nested/helmfile.yaml
  environment:
    default:    #or any other environment
      values:
      - key1: val1
      - values.yaml

@skaji
Copy link

skaji commented May 30, 2019

I mean helmfile v0.68.0 introduces a panic.
The above helmfile.ymal is a minimal example that panics.

Here is a more practical example:

❯ cat dev.yaml
foo:
  bar: 1

❯ cat helmfile.yaml
environments:
  dev:
    values:
      - dev.yaml

releases:
  - name: kube2iam
    namespace: kube-system
    chart: stable/kube2iam

❯ ./helmfile --version
helmfile version v0.68.0

❯ ./helmfile -e dev template
panic: json: unsupported type: map[interface {}]interface {} [recovered]
	panic: json: unsupported type: map[interface {}]interface {}

goroutine 1 [running]:
github.com/urfave/cli.HandleAction.func1(0xc0000d2de0)
	/go/pkg/mod/github.com/urfave/cli@v0.0.0-20160620154522-6011f165dc28/app.go:474 +0x287
panic(0x14583a0, 0xc000060390)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/roboll/helmfile/environment.Environment.DeepCopy(0x7ffeefbff6eb, 0x3, 0xc0000742a0, 0x1319a7c, 0xc000099650, 0xff)
	/home/circleci/workspace/helmfile/environment/environment.go:18 +0x12b
github.com/roboll/helmfile/environment.(*Environment).Merge(0xc0001c26a8, 0x0, 0x14d9408, 0x17, 0xc0000d1c90)
	/home/circleci/workspace/helmfile/environment/environment.go:38 +0x142
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).twoPassRenderTemplateToYaml(0xc00008a060, 0xc000176020, 0x0, 0x14d1ccc, 0x1, 0xc00017c020, 0x14, 0xc000174000, 0x86, 0x286, ...)
	/home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:86 +0x133
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderTemplatesToYamlWithEnv(...)
	/home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:62
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderTemplatesToYaml(0xc00008a060, 0x14d1ccc, 0x1, 0xc00017c020, 0x14, 0xc000174000, 0x86, 0x286, 0xc000176000, 0x1, ...)
	/home/circleci/workspace/helmfile/pkg/app/two_pass_renderer.go:58 +0xd2
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).renderAndLoad(0xc00008a060, 0x0, 0x0, 0x14d1ccc, 0x1, 0xc000172020, 0xd, 0xc000174000, 0x86, 0x286, ...)
	/home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:170 +0x6d3
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).loadFileWithOverrides(0xc00008a060, 0x0, 0x0, 0x14d1ccc, 0x1, 0x14d52bc, 0xd, 0x401, 0x1b0b900, 0x203000, ...)
	/home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:110 +0x1f4
github.com/roboll/helmfile/pkg/app.(*desiredStateLoader).Load(0xc00008a060, 0x14d52bc, 0xd, 0x18a6328, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/circleci/workspace/helmfile/pkg/app/desired_state_file_loader.go:59 +0x473
github.com/roboll/helmfile/pkg/app.(*App).loadDesiredStateFromYaml(0xc0000c3680, 0x14d52bc, 0xd, 0xc0000d24c0, 0x1, 0x1, 0xc0000d2338, 0xc000000000, 0xc0000d2390)
	/home/circleci/workspace/helmfile/pkg/app/app.go:140 +0x1c6
github.com/roboll/helmfile/pkg/app.(*App).VisitDesiredStates.func1(0x14d52bc, 0xd, 0xc000170040, 0x37, 0x0, 0x0)
	/home/circleci/workspace/helmfile/pkg/app/app.go:147 +0xf3
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles.func1(0x0, 0xc0000d2640)
	/home/circleci/workspace/helmfile/pkg/app/app.go:111 +0x9a
github.com/roboll/helmfile/pkg/app.(*App).within(0xc0000c3680, 0x14d1ccc, 0x1, 0xc000093580, 0xc0000d2838, 0x2)
	/home/circleci/workspace/helmfile/pkg/app/app.go:53 +0x722
github.com/roboll/helmfile/pkg/app.(*App).visitStateFiles(0xc0000c3680, 0x0, 0x0, 0xc000090720, 0x10, 0x1475180)
	/home/circleci/workspace/helmfile/pkg/app/app.go:105 +0x29f
github.com/roboll/helmfile/pkg/app.(*App).VisitDesiredStates(0xc0000c3680, 0x0, 0x0, 0x18a6328, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/circleci/workspace/helmfile/pkg/app/app.go:146 +0xd4
github.com/roboll/helmfile/pkg/app.(*App).VisitDesiredStatesWithReleasesFiltered(0xc0000c3680, 0x0, 0x0, 0xc0000b3320, 0x0, 0x0)
	/home/circleci/workspace/helmfile/pkg/app/app.go:235 +0x10e
github.com/roboll/helmfile/cmd.FindAndIterateOverDesiredStatesUsingFlagsWithReverse(0xc0000c4f00, 0xc000095400, 0xc000095480, 0x8, 0xc0000b32c0)
	/home/circleci/workspace/helmfile/cmd/cmd.go:77 +0xe6
main.findAndIterateOverDesiredStatesUsingFlags(...)
	/home/circleci/workspace/helmfile/main.go:633
main.main.func4(0xc0000c4f00, 0x0, 0x0)
	/home/circleci/workspace/helmfile/main.go:225 +0x66
reflect.Value.call(0x14464e0, 0x14efdc8, 0x13, 0x14d21a8, 0x4, 0xc0000d2d80, 0x1, 0x1, 0xc000092040, 0x1011bc3, ...)
	/usr/local/go/src/reflect/value.go:447 +0x461
reflect.Value.Call(0x14464e0, 0x14efdc8, 0x13, 0xc0000d2d80, 0x1, 0x1, 0xc0000faa90, 0xc0000faad8, 0x140)
	/usr/local/go/src/reflect/value.go:308 +0xa4
github.com/urfave/cli.HandleAction(0x14464e0, 0x14efdc8, 0xc0000c4f00, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli@v0.0.0-20160620154522-6011f165dc28/app.go:483 +0x1ff
github.com/urfave/cli.Command.Run(0x14d3843, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x14e821d, 0x3d, 0x0, ...)
	/go/pkg/mod/github.com/urfave/cli@v0.0.0-20160620154522-6011f165dc28/command.go:186 +0x8d1
github.com/urfave/cli.(*App).Run(0xc000152180, 0xc000092000, 0x4, 0x4, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli@v0.0.0-20160620154522-6011f165dc28/app.go:237 +0x601
main.main()
	/home/circleci/workspace/helmfile/main.go:592 +0x21e6

@mumoshu
Copy link
Collaborator Author

mumoshu commented May 30, 2019

@skaji Thanks for reporting! It seems like the same issue as #623. I'll fix it shortly.

@mumoshu
Copy link
Collaborator Author

mumoshu commented May 30, 2019

@sgandon I've intentionally omitted the environment name from the config syntax, because requiring the env name to override makes it harder to "just override some values of the selected env".

With the config syntax introduced via this change, you don't need to write something like the below to do that:

- path: path/to/nested/helmfile.yaml
  environment:
    {{ .Environment.Name }}:
      values:
      - key1: val1
      - values.yaml

Instead you write:

- path: path/to/nested/helmfile.yaml
  environment:
    values:
    - key1: val1
    - values.yaml

@sgandon
Copy link
Contributor

sgandon commented May 30, 2019

@mumoshu I am not sure that I get it, to what environment the values key1: val1 will be applied ?

@yurrriq
Copy link
Contributor

yurrriq commented May 31, 2019

I am not a fan of this, I wanted to mention. Perhaps I'm missing something, but this broke my existing setup.

in helmfile.d/subhelmfile.yaml: failed to read subhelmfile.yaml: environment values file matching "../environments/{{ .Environment.Name }}.yaml" does not exist in "."

@mumoshu
Copy link
Collaborator Author

mumoshu commented Jun 1, 2019

@yurrriq Hey! Thanks for reporting. The only reason you don't love this feature is that it breaks your existing setup, right?

This isn't intended to break anything. If it broke, it is a bug.

Would you mind sharing your detailed setup incl. helmfile version, so that I can try to reproduce and see what's wrong?

I had no prob with this:

$ tail -n +1 */**
==> environments/default.yaml <==
foo: FOO

==> helmfile.d/subhelmfile.yaml <==
environments:
  default:
    values:
    - "../environments/{{ .Environment.Name }}.yaml"

releases:
- name: {{ .Environment.Values.foo }}
  chart: stable/nginx

==> helmfile.d/subhelmfile.yaml~ <==
environments:
  default:
    values:
    - "../environments/{{ .Environment.Name }}.yaml"
$ helmfile -v
helmfile version v0.68.1
$ helmfile -f helmfile.yaml diff

@mumoshu
Copy link
Collaborator Author

mumoshu commented Jun 1, 2019

I am not sure that I get it, to what environment the values key1: val1 will be applied ?

@sgandon Hey! It overrides the values for the current environment (set via --environment, defaults to default). That's basically the same as calling the subhelmfile with --set key1=val1 as explained in #361 (comment)

mumoshu added a commit that referenced this pull request Jun 1, 2019
…-helmfiles)

We added envvals overrides in the state file via #622 two days ago:

```
helmfiles:
- name: sub.helmfile.yaml
  environment:
    values:
    - mykey: myvalue
```

This change removes the `environment` level in the above cofig, so that it looks like:

```
helmfiles:
- name: sub.helmfile.yaml
  values:
  - mykey: myvalue
``

This is an inevitable breaking change towards #361. But I wanted to break it earlier so that less folks are affected.`

Ref #361 (comment)
mumoshu added a commit that referenced this pull request Jun 1, 2019
…-helmfiles) (#635)

We added envvals overrides in the state file via #622 two days ago:

```
helmfiles:
- name: sub.helmfile.yaml
  environment:
    values:
    - mykey: myvalue
```

This change removes the `environment` level in the above cofig, so that it looks like:

```
helmfiles:
- name: sub.helmfile.yaml
  values:
  - mykey: myvalue
``

This is an inevitable breaking change towards #361. But I wanted to break it earlier so that less folks are affected.`

Ref #361 (comment)
@yurrriq
Copy link
Contributor

yurrriq commented Jun 3, 2019

I don't really understand or have a use for this future, at least at this time, and it broke my old configs, which looked more like the following.

==> environments/baz.yaml <==
foo: FOO

==> helmfile.d/subhelmfile.yaml <==
{{ readFile "../include/environments.yaml" }}

==> include/environments.yaml <==
environments:
  default: &default_environment
    values:
    - "../environments/{{ .Environment.Name }}.yaml"
  baz: *default_environment

This worked fine with helmfile 0.54.0 (and 0.64.1 IIRC) and broke when I switched to 0.68.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: Allow environment values to be overrode when referring sub-helmfile
4 participants