Add file and file_perms parameter to job's vault stanza

[grembo]

At the moment, secrets/vault_token is written using the nomad
process' umask, which - by default - is 0022, which results in
writing the vault_token world-readable (0644 / rw-r--r--).

This patch aims to address this by allowing to change the
permissions used from the default 666 (before umask).

The name file_perms was chosen to distinguish this from the
parameter perms in the template stanza (as templates always
operate on files, while Vault tokens are about permissions as
well, which would've been confusing otherwise).

Example of use:

vault {
    policies = ["nomad-tls-policy"]
    change_mode = "noop"
    file_perms = "600"
}

The patch also adds the file parameter, which controls if
secrets/vault_token is written at all.

The Vault token is always written to <task_dir>/private/vault_token,
private being a new directory to hold private secrets data not to be
shared with the chroot and also not to be shared with Nomad UI.

If file is set to true (the default), it is also written to
<task_dir>/secrets/vault_token, using the permissions configured
in file_perms.

There are two rationales for this design:

1. Have one authoritative place where vault_token exists.
2. Don't read vault_token from a place that operates on a lower
   security level on Nomad restart. Ultimately, secrets/vault_token
   could have been altered in unfortunate or malicious ways from within
   the chroot. This was the primary reason to add an additional write
   operation instead of just modifying tokenPath depending on the
   value of file.

Resolves #11900

[grembo]

A few force-pushes later the only failing test seems unrelated (and is marked FLAKY since Jan 12th):

TestAutopilot_RollingUpdate
1 test failed  out of 2304

[hashicorp-cla]

All committers have signed the CLA.

[grembo]

[![CLA assistant check]
Have you signed the CLA already but the status is still pending? Recheck it.

Hi @hashicorp-cla @lgfa29 @tgross,

I already signed the CLA in early January as part of #11783, so I don't think signing it again should be necessary.

Please see here for when I signed it.

Screenshot from #11783:

The HashiCorp OAuth app still shows as authorized in my github settings:

[tgross]

Sorry about that @grembo. The CLA bot went a bit wild over the weekend across all our repos. 😊 Should be fixed now.

[grembo]

@tgross Any updates on this?

In practice we ended up only using file = false, so we never store the vault token issued based on the vault stanza inside of the container. If the container requires a token, we issue it using a consul template stanza in the job description, which gives us full control over file permissions.

Based on this approach, we can issue all kinds of credentials (including vault tokens) to the container, without giving anything within the container the power to issue additional credentials itself, which is quite important to us.

So if it helps to move things forward, I could reduce this PR to only contain file (bool), which would still scratch out itch.

I can also share more about our setup, in case there are open questions.

[tgross]

@grembo sorry this never got marked for triage. I've marked it as such so that someone will take a look. At first glance I'm not sure this is the right approach. Why make the value configurable at all if you're putting the token in its own directory? I feel like this is handing operators a footgun.

[grembo]

@grembo sorry this never got marked for triage. I've marked it as such so that someone will take a look. At first glance I'm not sure this is the right approach. Why make the value configurable at all if you're putting the token in its own directory? I feel like this is handing operators a footgun.

@tgross Just to re-iterate the motivation: The goal is to store the token in a separate directory that is not accessible from inside the container.

If file is configured to true (the default), it is additionally written to /secrets inside of the container - the only reason I did this is so that the behavior of nomad would be backwards compatible, as this reflects the status quo of nomad, see also [0] for related thoughts on env.

If payload backwards compatibility is not an issue - ultimately, job definitions can probably be altered to mimic the current behavior - I would be totally happy to change the patch so that it only consists of nomad always writing the vault token to <task_dir>/private/vault_token without adding any configuration options to also copy it into the container, reducing the size of the patch considerably.

If some compatibility is required, changing file's (and potentially env's) default to false might present some middle-ground.

---

[0] Detour: This is also in line with nomad writing VAULT_TOKEN into the tasks environment, which has to be explicitly disabled by setting env = false in the vault stanza. With the current iteration if the patch, this is how our vault stanza looks like:

      vault {
        policies = ["tls-policy", "nomad-job-policy"]
        change_mode = "noop"
        env = false
        file = false
      }

which allows us to do things like:

      template {
        data = <<-EOH
          {{ with secret "pki_int/issue/nomad-task"
          "common_name=my.service.consul" "ttl=90m"
          "alt_names=localhost" "ip_sans=127.0.0.1"}}
          {{ .Data.certificate }}
          {{ .Data.private_key }}
          {{ .Data.issuing_ca }}
          {{ end }}
        EOH
        destination = "${NOMAD_SECRETS_DIR}/my.crt"
        change_mode = "noop"
        perms = "600"
      }

[tgross]

@grembo sorry I didn't notice there was already an extensive discussion in #11900 about this. This goal

The goal is to store the token in a separate directory that is not accessible from inside the container.

doesn't seen to be in evidence in that discussion though? Let's move this discussion about the overall design over to the issue though. I'll let @schmichael and @lgfa29 take it from there.

[grembo]

@grembo sorry I didn't notice there was already an extensive discussion in #11900 about this. This goal

The goal is to store the token in a separate directory that is not accessible from inside the container.

doesn't seen to be in evidence in that discussion though? Let's move this discussion about the overall design over to the issue though. I'll let @schmichael and @lgfa29 take it from there.

@tgross Yeah, originally #11900 was about changing /secrets/vault_token's file permissions, I only realized later that moving the file completely out of the way is a much cleaner solution and actually what we require. In the discussion in #11900 we also stumbled over the fact that task directories are world-readable (including files in any task's /secrets directory), which might have come to the surprise of some and introduced an additional topic into the discussion.

Hence my idea to split this PR/the issue and reduce the scope it handles, so it becomes single-topic and actionable.

[grembo]

Closing this pull request in favor of #13343, which is smaller in scope.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.