Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent rendering with shared secrets files #251

Closed
bfin opened this issue Aug 29, 2018 · 12 comments
Closed

Inconsistent rendering with shared secrets files #251

bfin opened this issue Aug 29, 2018 · 12 comments
Labels
bug

Comments

@bfin
Copy link

bfin commented Aug 29, 2018

Hi!

I'm seeing inconsistent rendering in my releases when using shared secrets files.

Probably related:
#149 (sync race condition)
#167 (cleaning up shared secrets files)
#250 (PR addressing #167)

Given a helmfile like the following, chart templates in my releases are missing values from some of the secrets.

...
releases:

  - name: api
    ...
    secrets:
      - values/broker/secrets.yaml
      - values/db/secrets.yaml
      - values/global/secrets.yaml

  - name: broker
    ...
    secrets:
      - values/broker/secrets.yaml

  - name: db
    ...
    secrets:
      - values/db/secrets.yaml

  - name: dbpooler
    ...
    secrets:
      - values/db/secrets.yaml

  - name: frontend
    ...
    secrets:
      - values/global/secrets.yaml

  - name: scheduler
    ...
    secrets:
      - values/broker/secrets.yaml

  - name: taskmonitor
    ...
    secrets:
      - values/broker/secrets.yaml
      - values/global/secrets.yaml

  - name: worker
    ...
    secrets:
      - values/broker/secrets.yaml
      - values/db/secrets.yaml
      - values/global/secrets.yaml

Furthermore, within a single helmfile sync (or helmfile diff) command, some releases will receive the secret values and others will not, and either all chart templates in a release receive the secret values or all of them do not receive them. I wish I knew more about go so I could troubleshoot it myself, but it looks like an issue with decryption parallelism. Here is some (sanitized) sample output from several runs of helmfile diff for the above helmfile. Note the inconsistent rendering and multiple attempts to decrypt the same files.

Decrypting secret values/db/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/broker/secrets.yaml
Decrypting secret values/db/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/broker/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting values/broker/secrets.yaml

Comparing scheduler ********/scheduler
Decrypting values/global/secrets.yaml

Decrypting secret values/broker/secrets.yaml
application, scheduler, Secret (v1) has changed:

-   broker_password: ********
+   broker_password:

Decrypting values/global/secrets.yaml

Comparing frontend ********/frontend
Decrypting values/global/secrets.yaml

Decrypting secret values/broker/secrets.yaml
Decrypting values/broker/secrets.yaml

Comparing broker stable/rabbitmq
Decrypting values/global/secrets.yaml

Decrypting secret values/db/secrets.yaml
Decrypting values/db/secrets.yaml

Comparing dbpooler ********/dbpooler
application, broker, Secret (v1) has changed:

-   password: ********
+   password: 
-   erlang-cookie: ********
+   erlang-cookie: 

Decrypting values/broker/secrets.yaml

Decrypting secret values/db/secrets.yaml
Decrypting values/db/secrets.yaml

Comparing db ********/db
application, db, Secret (v1) has changed:

-   username: ********
+   username:

Decrypting values/db/secrets.yaml

Decrypting secret values/broker/secrets.yaml
Decrypting values/broker/secrets.yaml

Comparing taskmonitor ********/taskmonitor
application, taskmonitor, Secret (v1) has changed:

-   broker_password: ********
+   broker_password:

Decrypting values/db/secrets.yaml

Comparing api ********/api
application, api, Secret (v1) has changed:

-   broker_password: ********
+   broker_password:

Decrypting values/broker/secrets.yaml

Comparing worker ********/worker
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory

Another run...

Decrypting secret values/broker/secrets.yaml
Decrypting secret values/db/secrets.yaml
Decrypting secret values/db/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/broker/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting values/broker/secrets.yaml

Comparing broker stable/rabbitmq
Decrypting values/global/secrets.yaml

Decrypting secret values/db/secrets.yaml
Decrypting values/broker/secrets.yaml

Comparing scheduler ********/scheduler
Decrypting values/db/secrets.yaml

Comparing dbpooler ********/dbpooler
Decrypting values/global/secrets.yaml

Decrypting secret values/broker/secrets.yaml
Decrypting values/db/secrets.yaml

Comparing db ********/db
Decrypting values/global/secrets.yaml

Comparing frontend ********/frontend
Decrypting values/global/secrets.yaml

Decrypting secret values/broker/secrets.yaml
Decrypting values/db/secrets.yaml

Decrypting secret values/broker/secrets.yaml
Decrypting values/broker/secrets.yaml

Comparing taskmonitor ********/taskmonitor
Decrypting values/broker/secrets.yaml

Decrypting values/broker/secrets.yaml

Decrypting secret values/db/secrets.yaml
Comparing worker ********/worker
application, worker, Secret (v1) has changed:

-   database_username: ********
-   database_password: ********
+   database_username:
+   database_password:

Decrypting values/db/secrets.yaml

Comparing api ********/api
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory

Another run...

Decrypting secret values/broker/secrets.yaml
Decrypting secret values/broker/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/db/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/db/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting secret values/global/secrets.yaml
Decrypting values/db/secrets.yaml

Comparing dbpooler ********/dbpooler
Decrypting values/broker/secrets.yaml

Comparing scheduler ********/scheduler
Decrypting values/broker/secrets.yaml

Comparing broker stable/rabbitmq
Decrypting values/global/secrets.yaml

Decrypting secret values/db/secrets.yaml
Decrypting values/global/secrets.yaml

Decrypting values/global/secrets.yaml

Decrypting secret values/broker/secrets.yaml
Comparing frontend ********/frontend
application, broker, Secret (v1) has changed:

-   password: ********
+   password: 
-   erlang-cookie: ********
+   erlang-cookie: 

Decrypting values/db/secrets.yaml

Comparing db ********/db
Decrypting values/global/secrets.yaml

Decrypting secret values/broker/secrets.yaml
Decrypting values/broker/secrets.yaml

Decrypting values/db/secrets.yaml

Decrypting secret values/db/secrets.yaml
Decrypting secret values/broker/secrets.yaml
Decrypting values/broker/secrets.yaml

Comparing taskmonitor ********/taskmonitor
Decrypting values/broker/secrets.yaml

Comparing worker ********/worker
Decrypting values/db/secrets.yaml

Comparing api ********/api
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory
err: remove values/db/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory
err: remove values/broker/secrets.yaml.dec: no such file or directory
err: remove values/global/secrets.yaml.dec: no such file or directory

Perhaps an alternative would be to use sops directly instead of the helm secrets plugin? This would allow you to specify randomized output filenames (helm secrets only outputs secrets.yaml.dec) and have the added benefit of freeing users to be able to use other filenames (instead of secrets.yaml), not that that is nearly as important as consistent template rendering!

I should note that when I sync the above helmfile – but with the secrets already decrypted and passed in as values files instead of secrets files – all templates in all releases receive the correct values.

Thanks for your help!
@mumoshu
Copy link
Collaborator

mumoshu commented Aug 30, 2018

@bfin Hey! Thanks a lot for your detailed report.

I was wondering if this was an unexpected side-effect of #167.
As #250 is already merged and has been released as v0.25.1, would you mind trying again with the new release?

@mumoshu mumoshu added the bug label Aug 30, 2018
@sstarcher
Copy link
Contributor

I would suspect that #250 fixes this issue.

@bfin
Copy link
Author

bfin commented Aug 30, 2018

@mumoshu @sstarcher Thanks for the quick replies! (I should mention that it's partly because of your attentiveness to this project that I'm using helmfile over tools written in languages I actually understand...grin.)

I should have mentioned that I tested with #250 without success. This is the relevant output (which I thought I was receiving because I didn't properly build the binary from the PR):

Decrypting secret values/broker/development/secrets.yaml
Decrypting secret values/global/development/secrets.yaml
Decrypting secret values/db/development/secrets.yaml
Decrypting secret values/db/development/secrets.yaml
Decrypting secret values/global/development/secrets.yaml
Decrypting secret values/broker/development/secrets.yaml
Decrypting secret values/global/development/secrets.yaml
Decrypting secret values/global/development/secrets.yaml

Decrypting values/global/development/secrets.yaml
Decrypting values/db/development/secrets.yaml
Decrypting values/db/development/secrets.yaml
Decrypting values/global/development/secrets.yaml
Decrypting values/global/development/secrets.yaml
Decrypting values/broker/development/secrets.yaml
Decrypting values/broker/development/secrets.yaml
Decrypting values/global/development/secrets.yaml

err: release "frontend" in "development.application.yaml" failed: rename values/global/development/secrets.yaml.dec /*********/secret883345779: cross-device link
err: release "dbpooler" in "development.application.yaml" failed: rename values/db/development/secrets.yaml.dec /*********/secret157679670: cross-device link
err: release "db" in "development.application.yaml" failed: rename values/db/development/secrets.yaml.dec /*********/secret610119709: cross-device link
err: release "taskmonitor" in "development.application.yaml" failed: rename values/global/development/secrets.yaml.dec /*********/secret160019672: cross-device link
err: release "api" in "development.application.yaml" failed: rename values/global/development/secrets.yaml.dec /*********/secret066611287: cross-device link
err: release "broker" in "development.application.yaml" failed: rename values/broker/development/secrets.yaml.dec /*********/secret187728033: cross-device link
err: release "scheduler" in "development.application.yaml" failed: rename values/broker/development/secrets.yaml.dec /*********/secret125184970: cross-device link
err: release "worker" in "development.application.yaml" failed: rename values/global/development/secrets.yaml.dec /*********/secret715448204: cross-device link

I just tested with the newest version of helmfile and got the same result.

This is promising in that each secrets file was intended to be renamed to a unique filename in the temp directory, but I'm not sure about the "cross-device link" error. Perhaps because my "projects" directory is on a separate partition from my operating system partition (OS is macOS v10.13.6 BTW)?

@mumoshu
Copy link
Collaborator

mumoshu commented Aug 30, 2018

Ah, might be the case.

@sstarcher Should be cp to override the temp file just to use the temp file name, and the remove the original .dec file instead?

@mumoshu
Copy link
Collaborator

mumoshu commented Aug 30, 2018

@bfin cross-device link error sounds like more about creating a hard-link of the file over another filesystem, as I believe that mv doesn't create a hard-link. But would it be your case? Please inform me anything you think relevant. Thanks!

mumoshu added a commit to mumoshu/helmfile that referenced this issue Aug 30, 2018
@mumoshu
fix: avoid "cross-device link" errors while decrypting secrets
d5d64b6 
ref roboll#251 (comment)
@mumoshu mumoshu mentioned this issue Aug 30, 2018
Merged
mumoshu added a commit that referenced this issue Aug 30, 2018
@mumoshu
fix: avoid "cross-device link" errors while decrypting secrets (#260)
79f0e70 
ref #251 (comment)
@mumoshu
Copy link
Collaborator

mumoshu commented Aug 30, 2018

@bfin Would you mind trying again with the new v0.25.2?

@bfin
Copy link
Author

bfin commented Aug 30, 2018

Progress!

Releases now sync with the correct values from secrets files, but the decrypted files remain on disk afterwards. So in my case I'm left with something like...

values/broker/development/secrets.yaml.dec
values/broker/development/secrets.yaml
values/db/development/secrets.yaml.dec
values/db/development/secrets.yaml
values/global/development/secrets.yaml.dec
values/global/development/secrets.yaml

...where each secrets.yaml is encrypted and each secrets.yaml.dec is decrypted.

@bfin
Copy link
Author

bfin commented Aug 30, 2018

I don't know if it helps, but sops (i.e., the basis of the helm-secrets plugin) can be used as a go library (https://godoc.org/go.mozilla.org/sops/decrypt). Maybe this would provide more flexible access to encrypted/decrypted data?

mumoshu added a commit to mumoshu/helmfile that referenced this issue Aug 30, 2018
@mumoshu
fix: follow up for the relative paths improvement
e9fc922 
Fixes for the bugs that are introduced by roboll#261, that is values.yaml files specified in `values:` have redundant base path in their prefixes, and remaining .dec files after secrets decryption(roboll#251 (comment))
@mumoshu mumoshu mentioned this issue Aug 30, 2018
Merged
mumoshu added a commit that referenced this issue Aug 30, 2018
@mumoshu
fix: follow up for the relative paths improvement (#262)
bb3b44e 
Fixes for the bugs that are introduced by #261, that is values.yaml files specified in `values:` have redundant base path in their prefixes, and remaining .dec files after secrets decryption(#251 (comment))
@mumoshu
Copy link
Collaborator

mumoshu commented Aug 30, 2018

@bfin Thanks for testing it out ☺️ I've just release v0.25.3 with the fix for that. Could you try once again?

@bfin
Copy link
Author

bfin commented Aug 30, 2018

@mumoshu I've been stalking your progress on this...grin. Just finished testing: success!

Thanks so much for your quick work with these fixes!

@bfin bfin closed this as completed Aug 30, 2018
@mumoshu
Copy link
Collaborator

mumoshu commented Aug 30, 2018

@bfin Thanks a lot for your quick comments and the whole support! Good luck ☺️

@sstarcher
Copy link
Contributor

@mumoshu thanks for fixing that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mumoshu @sstarcher @bfin