Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

inconsistent results of annotation: ...-comand-${secret} #137

Closed
mrproper opened this issue May 20, 2020 · 2 comments
Closed

inconsistent results of annotation: ...-comand-${secret} #137

mrproper opened this issue May 20, 2020 · 2 comments

Comments

@mrproper
Copy link

command does not seem to execute consistently

Using the following annotations on the master build of vault-k8s (needed for the run-as-user|group)

Annotations

      annotations:
        vault.hashicorp.com/log-level: "debug"
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/role: "logging-elasticsearch"
        vault.hashicorp.com/ca-cert: "/vault/tls/cert.ca"
        vault.hashicorp.com/tls-secret: "vault-ca-cert"
        vault.hashicorp.com/agent-run-as-user: "1000"
        vault.hashicorp.com/agent-run-as-group: "1000"
        vault.hashicorp.com/secret-volume-path: "/usr/share/elasticsearch/config/certs"
        vault.hashicorp.com/agent-inject-secret-ca.crt: "certificates/svc/logging/elasticsearch"
        vault.hashicorp.com/agent-inject-template-ca.crt: '{{- with secret "certificates/svc/logging/elasticsearch" -}}{{ .Data.data.Cert }}{{- end -}}'
        vault.hashicorp.com/agent-inject-secret-elasticsearch.crt: "certificates/svc/logging/elasticsearch"
        vault.hashicorp.com/agent-inject-template-elasticsearch.crt: '{{- with secret "certificates/svc/logging/elasticsearch" -}}{{ .Data.data.Cert }}{{- end -}}'
        vault.hashicorp.com/agent-inject-secret-elasticsearch.key: "certificates/svc/logging/elasticsearch"
        vault.hashicorp.com/agent-inject-template-elasticsearch.key: '{{- with secret "certificates/svc/logging/elasticsearch" -}}{{ .Data.data.CertKey }}{{- end -}}'
        vault.hashicorp.com/secret-volume-path-elastic_password: "/usr/share/elasticsearch/config/env"
        vault.hashicorp.com/agent-inject-secret-elastic_password: "credentials/logging/elasticsearch"
        vault.hashicorp.com/agent-inject-template-elastic_password: '{{- with secret "credentials/logging/elasticsearch" -}}{{ .Data.data.ELASTIC_PASSWORD }}{{- end -}}'
        vault.hashicorp.com/agent-inject-command-elastic_password: "chmod 400 /usr/share/elasticsearch/config/env/elastic_password"

Applying this to a statefulset with 3 pods.
Sometimes the chmod works properly other times it does not:

First one is fine

 kubectl --namespace logging exec elasticsearch-0 -c elasticsearch -- ls -lah /usr/share/elasticsearch/config/env/
total 4.0K
drwxrwxrwt. 2 root          root           60 May 20 05:28 .
drwxrwxr-x. 4 elasticsearch root          202 May 20 05:28 ..
-r--------. 1 elasticsearch elasticsearch  32 May 20 05:28 elastic_password

Second one is not:

$ kubectl --namespace logging exec elasticsearch-1 -c elasticsearch -- ls -lah /usr/share/elasticsearch/config/env/
error: unable to upgrade connection: container not found ("elasticsearch")

$ kubectl --namespace logging logs -f elasticsearch-1 -c elasticsearch
ERROR: File /usr/share/elasticsearch/config/env/elastic_password from ELASTIC_PASSWORD_FILE must have file permissions 400 or 600, but actually has: 644

Third one is fine:

$ kubectl --namespace logging exec elasticsearch-2 -c elasticsearch -- ls -lah /usr/share/elasticsearch/config/env/
total 4.0K
drwxrwxrwt. 2 root          root           60 May 20 05:28 .
drwxrwxr-x. 4 elasticsearch root          202 May 20 05:28 ..
-r--------. 1 elasticsearch elasticsearch  32 May 20 05:28 elastic_password

Relevant debug log for vault-agent-init on elasticsearch-0

2020/05/20 05:28:00.896364 [INFO] (runner) creating watcher
2020/05/20 05:28:00.896415 [INFO] (runner) starting
2020/05/20 05:28:00.896420 [DEBUG] (runner) running initial templates
2020/05/20 05:28:00.896422 [DEBUG] (runner) initiating run
2020/05/20 05:28:00.896427 [DEBUG] (runner) checking template a67610cc9bf8ce377d5af235dd0f5224
2020/05/20 05:28:00.898286 [DEBUG] (runner) was not watching 1 dependencies
2020/05/20 05:28:00.898321 [DEBUG] (watcher) adding vault.read(certificates/svc/logging/elasticsearch)
2020/05/20 05:28:00.898334 [DEBUG] (runner) checking template 2dc29a64ac68d68930bf8a401684b35d
2020/05/20 05:28:00.898460 [DEBUG] (runner) was not watching 1 dependencies
2020/05/20 05:28:00.898470 [DEBUG] (watcher) adding vault.read(credentials/logging/elasticsearch)
2020/05/20 05:28:00.898475 [DEBUG] (runner) checking template e8f9c46166e1c9a86eac76cb09aa9531
2020/05/20 05:28:00.898569 [DEBUG] (runner) missing data for 1 dependencies
2020/05/20 05:28:00.898576 [DEBUG] (runner) diffing and updating dependencies
2020/05/20 05:28:00.898581 [DEBUG] (runner) watching 2 dependencies
2020-05-20T05:28:00.911Z [INFO]  auth.handler: renewed auth token
2020/05/20 05:28:01.003492 [DEBUG] (runner) receiving dependency vault.read(credentials/logging/elasticsearch)
2020/05/20 05:28:01.003514 [DEBUG] (runner) initiating run
2020/05/20 05:28:01.003538 [DEBUG] (runner) checking template a67610cc9bf8ce377d5af235dd0f5224
2020/05/20 05:28:01.003731 [DEBUG] (runner) missing data for 1 dependencies
2020/05/20 05:28:01.003740 [DEBUG] (runner) checking template 2dc29a64ac68d68930bf8a401684b35d
2020/05/20 05:28:01.003876 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.004072 [INFO] (runner) rendered "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.004091 [DEBUG] (runner) appending command "chmod 400 /usr/share/elasticsearch/config/env/elastic_password" from "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.004102 [DEBUG] (runner) checking template e8f9c46166e1c9a86eac76cb09aa9531
2020/05/20 05:28:01.004244 [DEBUG] (runner) missing data for 1 dependencies
2020/05/20 05:28:01.004255 [DEBUG] (runner) diffing and updating dependencies
2020/05/20 05:28:01.004265 [DEBUG] (runner) vault.read(certificates/svc/logging/elasticsearch) is still needed
2020/05/20 05:28:01.004268 [DEBUG] (runner) vault.read(credentials/logging/elasticsearch) is still needed
2020/05/20 05:28:01.004273 [INFO] (runner) executing command "chmod 400 /usr/share/elasticsearch/config/env/elastic_password" from "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.004353 [INFO] (child) spawning: chmod 400 /usr/share/elasticsearch/config/env/elastic_password
2020/05/20 05:28:01.005022 [DEBUG] (runner) watching 2 dependencies
2020/05/20 05:28:01.011057 [DEBUG] (runner) receiving dependency vault.read(certificates/svc/logging/elasticsearch)
2020/05/20 05:28:01.011071 [DEBUG] (runner) initiating run
2020/05/20 05:28:01.011077 [DEBUG] (runner) checking template a67610cc9bf8ce377d5af235dd0f5224
2020/05/20 05:28:01.011250 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.crt"
2020/05/20 05:28:01.011380 [INFO] (runner) rendered "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.crt"
2020/05/20 05:28:01.011390 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/ca.crt"
2020/05/20 05:28:01.011439 [INFO] (runner) rendered "(dynamic)" => "/usr/share/elasticsearch/config/certs/ca.crt"
2020/05/20 05:28:01.011448 [DEBUG] (runner) checking template 2dc29a64ac68d68930bf8a401684b35d
2020/05/20 05:28:01.011560 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.011582 [DEBUG] (runner) checking template e8f9c46166e1c9a86eac76cb09aa9531
2020/05/20 05:28:01.011701 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.key"
2020/05/20 05:28:01.011768 [INFO] (runner) rendered "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.key"
2020/05/20 05:28:01.011780 [DEBUG] (runner) diffing and updating dependencies
2020/05/20 05:28:01.011785 [DEBUG] (runner) vault.read(certificates/svc/logging/elasticsearch) is still needed
2020/05/20 05:28:01.011788 [DEBUG] (runner) vault.read(credentials/logging/elasticsearch) is still needed
2020/05/20 05:28:01.011790 [DEBUG] (runner) watching 2 dependencies
2020/05/20 05:28:01.011793 [DEBUG] (runner) all templates rendered
2020/05/20 05:28:01.011802 [INFO] (runner) stopping
2020/05/20 05:28:01.011809 [DEBUG] (runner) stopping watcher
2020/05/20 05:28:01.011811 [DEBUG] (watcher) stopping all views
2020-05-20T05:28:01.011Z [INFO]  template.server: template server stopped
2020/05/20 05:28:01.011841 [INFO] (runner) received finish

Relevant debug log for vault-agent-init on elasticsearch-1

2020/05/20 05:28:00.899145 [INFO] (runner) creating watcher
2020/05/20 05:28:00.899207 [INFO] (runner) starting
2020/05/20 05:28:00.899216 [DEBUG] (runner) running initial templates
2020/05/20 05:28:00.899219 [DEBUG] (runner) initiating run
2020/05/20 05:28:00.899225 [DEBUG] (runner) checking template a67610cc9bf8ce377d5af235dd0f5224
2020/05/20 05:28:00.901645 [DEBUG] (runner) was not watching 1 dependencies
2020/05/20 05:28:00.901676 [DEBUG] (watcher) adding vault.read(certificates/svc/logging/elasticsearch)
2020/05/20 05:28:00.901690 [DEBUG] (runner) checking template e8f9c46166e1c9a86eac76cb09aa9531
2020/05/20 05:28:00.901792 [DEBUG] (runner) missing data for 1 dependencies
2020/05/20 05:28:00.901802 [DEBUG] (runner) checking template 2dc29a64ac68d68930bf8a401684b35d
2020/05/20 05:28:00.901879 [DEBUG] (runner) was not watching 1 dependencies
2020/05/20 05:28:00.901890 [DEBUG] (watcher) adding vault.read(credentials/logging/elasticsearch)
2020/05/20 05:28:00.901901 [DEBUG] (runner) diffing and updating dependencies
2020/05/20 05:28:00.901913 [DEBUG] (runner) watching 2 dependencies
2020-05-20T05:28:00.916Z [INFO]  auth.handler: renewed auth token
2020/05/20 05:28:01.013642 [DEBUG] (runner) receiving dependency vault.read(certificates/svc/logging/elasticsearch)
2020/05/20 05:28:01.013658 [DEBUG] (runner) initiating run
2020/05/20 05:28:01.013663 [DEBUG] (runner) checking template a67610cc9bf8ce377d5af235dd0f5224
2020/05/20 05:28:01.013829 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/ca.crt"
2020/05/20 05:28:01.013984 [INFO] (runner) rendered "(dynamic)" => "/usr/share/elasticsearch/config/certs/ca.crt"
2020/05/20 05:28:01.013995 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.crt"
2020/05/20 05:28:01.014052 [INFO] (runner) rendered "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.crt"
2020/05/20 05:28:01.014061 [DEBUG] (runner) checking template e8f9c46166e1c9a86eac76cb09aa9531
2020/05/20 05:28:01.014203 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.key"
2020/05/20 05:28:01.014270 [INFO] (runner) rendered "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.key"
2020/05/20 05:28:01.014278 [DEBUG] (runner) checking template 2dc29a64ac68d68930bf8a401684b35d
2020/05/20 05:28:01.014409 [DEBUG] (runner) missing data for 1 dependencies
2020/05/20 05:28:01.014419 [DEBUG] (runner) diffing and updating dependencies
2020/05/20 05:28:01.014424 [DEBUG] (runner) vault.read(certificates/svc/logging/elasticsearch) is still needed
2020/05/20 05:28:01.014428 [DEBUG] (runner) vault.read(credentials/logging/elasticsearch) is still needed
2020/05/20 05:28:01.014430 [DEBUG] (runner) watching 2 dependencies
2020/05/20 05:28:01.016927 [DEBUG] (runner) receiving dependency vault.read(credentials/logging/elasticsearch)
2020/05/20 05:28:01.016940 [DEBUG] (runner) initiating run
2020/05/20 05:28:01.016944 [DEBUG] (runner) checking template a67610cc9bf8ce377d5af235dd0f5224
2020/05/20 05:28:01.017140 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/ca.crt"
2020/05/20 05:28:01.017179 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.crt"
2020/05/20 05:28:01.017204 [DEBUG] (runner) checking template e8f9c46166e1c9a86eac76cb09aa9531
2020/05/20 05:28:01.017328 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/certs/elasticsearch.key"
2020/05/20 05:28:01.017357 [DEBUG] (runner) checking template 2dc29a64ac68d68930bf8a401684b35d
2020/05/20 05:28:01.017469 [DEBUG] (runner) rendering "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.017591 [INFO] (runner) rendered "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.017603 [DEBUG] (runner) appending command "chmod 400 /usr/share/elasticsearch/config/env/elastic_password" from "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.017612 [DEBUG] (runner) diffing and updating dependencies
2020/05/20 05:28:01.017618 [DEBUG] (runner) vault.read(certificates/svc/logging/elasticsearch) is still needed
2020/05/20 05:28:01.017620 [DEBUG] (runner) vault.read(credentials/logging/elasticsearch) is still needed
2020/05/20 05:28:01.017625 [INFO] (runner) executing command "chmod 400 /usr/share/elasticsearch/config/env/elastic_password" from "(dynamic)" => "/usr/share/elasticsearch/config/env/elastic_password"
2020/05/20 05:28:01.017681 [INFO] (child) spawning: chmod 400 /usr/share/elasticsearch/config/env/elastic_password
2020/05/20 05:28:01.017737 [INFO] (runner) stopping
2020/05/20 05:28:01.017749 [DEBUG] (runner) stopping watcher
2020/05/20 05:28:01.017752 [DEBUG] (watcher) stopping all views
2020-05-20T05:28:01.017Z [INFO]  template.server: template server stopped```
@jasonodonnell
Copy link
Contributor

Hi @mrproper,

This is actually a bug in Consul Template which was fixed. I'm trying to get this update into Vault 1.4.2 release, however, if it's rejected then it will be released in 1.5.

@jasonodonnell
Copy link
Contributor

This fix made it into 1.4.2 and should be released soon. Keep an eye out for the new version of Vault!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mrproper @jasonodonnell