Optional meta should really be optional

[tino]

Nomad version

0.6.3

Operating system and Environment details

Ubuntu 16.04

Issue

Omitting a meta_optiona parameter leads to the verbatim name of the variable in the output:

⌘ nomad logs ee6a
before ${NOMAD_META_optional:-} after

I would expect: before after

Reproduction steps

1. nomad run jobfile.nomad
2. nomad job dispatch -meta command=echo test_meta_optional
3. nomad logs <alloc-id>

Job file (if appropriate)

job "test_meta_optional" {
  datacenters = ["NL1"]
  type = "batch"

  parameterized {
    payload       = "forbidden"
    meta_required = ["command"]
    meta_optional = ["optional"]
  }

  group "test_meta_optional" {

    # Fail hard, immediately
    restart {
      attempts = 0
      mode = "fail"
    }

    task "test_meta_optional" {
      driver = "exec"
      config {
        command = "${NOMAD_META_command}"
        args = ["before", "${NOMAD_META_optional:-}", "after"]
      }
    }
  }
}

[dadgar]

Hey you can mimic this behavior by just adding the default value as follows:

job "test_meta_optional" {
  datacenters = ["dc1"]
  type        = "batch"

  parameterized {
    payload       = "forbidden"
    meta_required = ["command"]
    meta_optional = ["optional"]
  }

  group "test_meta_optional" {
    # Fail hard, immediately
    restart {
      attempts = 0
      mode     = "fail"
    }

    task "test_meta_optional" {
      driver = "raw_exec"

      meta {
        "optional" = "-"
      }

      config {
        command = "${NOMAD_META_command}"
        args    = ["before", "${NOMAD_META_optional}", "after"]
      }
    }
  }
}

[tino]

Hmm, okay. Not really that intuitive though, and undocumented as far as I can see?

Wait, that doesn't work, now the output is always before - after, even when I pass the it in the dispatch as such: nomad job dispatch -meta command=echo -meta optional=between test_meta_optional

I would expect an optional, non provided variable to become "", as I am used to with other scripts and template languates ☺️

[sevagh]

So how are meta optional variables meant to work?

parametrized {
    meta_optional = ["CANONICAL_PATH"]
}

meta {
        "CANONICAL_PATH" = ""
      }

env {
    CANONICAL_PATH="${NOMAD_META_CANONICAL_PATH}",
}

This is literally passing "${NOMAD_META_CANONICAL_PATH}" to my Dockerfile's entrypoint.sh:

$ cat entrypoint.sh
echo "Value of canonical path: ${CANONICAL_PATH}"
$ nomad <alloc_id> fs alloc/logs/build.stdout.0
Value of canonical path: ${NOMAD_META_CANONICAL_PATH}

My meta_required are all behaving normally.

[nickethier]

Hi @tino, I spent some time looking into this issue and wanted to let you know what I've found thus far.

In the jobfile reported I see "${NOMAD_META_optional:-}" as one of the args. The :- syntax is bash (possibly other shells) parameter expansion. This syntax is not supported in nomad's variable interpolation and will also not work when passing directly to a command. If you want this functionality then you will need to do so in a bash script and use that to initialize your workload.

The documentation for the meta stanza can be found here.

I tried reproducing the behavior @sevagh described with the following jobspec on nomad 0.8.3, but could not replicate what was experienced. If you have a full jobspec and steps to repeat please let us know.

$> cat meta.nomad
job "meta" {
  datacenters = ["dc1"]
  parameterized {
    meta_required = ["meta_req"]
    meta_optional = ["meta_opt"]
  }

  type = "batch"

  group "g1" {
    task "sleep" {
      meta {
        meta_opt = ""
      }

      env {
        META_OPT="${NOMAD_META_meta_opt}"
        META_REQ="${NOMAD_META_meta_req}"
      }
      driver = "raw_exec"

      config {
        command = "env"
      }

      resources {
        cpu    = 50
        memory = 32
      }
    }
  }
}

$> nomad run meta.nomad
Job registration successful

$> nomad job dispatch -meta meta_req=bar meta
Dispatched Job ID = meta/dispatch-1525400516-8321d574
Evaluation ID     = 3baa66a7

==> Monitoring evaluation "3baa66a7"
    Evaluation triggered by job "meta/dispatch-1525400516-8321d574"
    Allocation "6512c5e6" created: node "b6656bfd", group "g1"
    Allocation "6512c5e6" status changed: "pending" -> "complete"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "3baa66a7" finished with status "complete"

>$ nomad alloc logs 651 | grep --ignore-case meta
NOMAD_ALLOC_NAME=meta/dispatch-1525400516-8321d574.g1[0]
NOMAD_META_META_OPT=
META_OPT=
NOMAD_META_meta_opt=
NOMAD_JOB_NAME=meta/dispatch-1525400516-8321d574
NOMAD_META_META_REQ=bar
NOMAD_META_meta_req=bar
META_REQ=bar

[nickethier]

After chatting with @dadgar about this one it seems like the root of the issue is that optional metadata interpolation (like all other variable interpolation) returns the token as the value if unset. For optional metadata it is much more expected that an empty string is returned.

This something we'll be working on fixing.

[dadgar]

Unfortunately the original PR didn't fix this. The correct fix which we will get in for 0.9.0, is to attached the parameterized configuration to the dispatched job but to add a boolean field that specifies that it is the dispatched instance. Then we can use that new field to determine if the alloc is an instance of a dispatched job and keep all the changes from the original PR.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.