Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic when referencing ${env["VAR"]} in Docker task args #9956

Closed
manicminer opened this issue Feb 3, 2021 · 8 comments · Fixed by #10326
Closed

Panic when referencing ${env["VAR"]} in Docker task args #9956

manicminer opened this issue Feb 3, 2021 · 8 comments · Fixed by #10326
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. theme/cli theme/crash theme/hcl type/bug

Comments

@manicminer
Copy link
Member

For reporting security vulnerabilities please refer to the website.

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

Nomad v1.0.3 (08741d9f2003ec26e44c72a2c0e27cdf0eadb6ee)

Operating system and Environment details

Linux nomad 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Issue

Trying to reference an environment variable in the args attribute of a Docker task. Get a panic when running nomad job run foo.nomad. I'm not sure if the task syntax is entirely correct but I figured a panic is undesirable either way?

Reproduction steps

  1. Use the job spec below
  2. nomad job run foo.nomad
  3. See panic

Job file (if appropriate)

job "http-echo" {
  datacenters = ["home"]
  group "echo" {
    count = 1

    network {
      port "http" {}
    }

    service {
      name = "echo"
      port = "http"
      tags = [
        "http",
        "proxy",
      ]

      check {
        type     = "http"
        path     = "/"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo:latest"
        args  = [
          "-listen", ":${env["BLAH"]}",
          "-text", "Echo service on port ${env["BLAH"]}",
        ]
        ports = ["http"]
      }

      resources {
        cpu    = 100
        memory = 64
      }
    }
  }
}

Panic trace

panic: interface conversion: hcl.Traverser is hcl.TraverseIndex, not hcl.TraverseAttr

goroutine 1 [running]:
github.com/hashicorp/hcl/v2.Traversal.toStringValue(0xc0002f8160, 0x2, 0x2, 0x3, 0x52e3240)
	github.com/hashicorp/hcl/v2@v2.7.1-0.20201020204811-68a97f93bb48/traversal.go:140 +0x566
github.com/hashicorp/hcl/v2.Traversal.TraverseAbs(0xc0002f8160, 0x2, 0x2, 0xc0002f9580, 0x2be1060, 0x1, 0x40, 0xc0003e2300, 0xc0000520e9, 0x7f10b59cc701, ...)
	github.com/hashicorp/hcl/v2@v2.7.1-0.20201020204811-68a97f93bb48/traversal.go:94 +0x9e5
github.com/hashicorp/hcl/v2/hclsyntax.(*ScopeTraversalExpr).Value(0xc0002c7740, 0xc0002f9580, 0x1, 0x1, 0x0, 0x0, 0x2be0e20, 0xc000143910, 0x0)
	github.com/hashicorp/hcl/v2@v2.7.1-0.20201020204811-68a97f93bb48/hclsyntax/expression.go:109 +0x65
github.com/hashicorp/hcl/v2/hclsyntax.(*TemplateExpr).Value(0xc0002c7ce0, 0xc0002f9580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	github.com/hashicorp/hcl/v2@v2.7.1-0.20201020204811-68a97f93bb48/hclsyntax/expression_template.go:33 +0x157
github.com/hashicorp/hcl/v2/hclsyntax.(*TupleConsExpr).Value(0xc000530aa0, 0xc0002f9580, 0xc0001da0e0, 0xc000305b8c, 0x52ac520, 0x41985b, 0xc0001da0e0, 0xc000249968, 0x10)
	github.com/hashicorp/hcl/v2@v2.7.1-0.20201020204811-68a97f93bb48/hclsyntax/expression.go:712 +0xf0
github.com/hashicorp/nomad/jobspec2.decodeInterface(0x7f108ca60458, 0xc000530aa0, 0xc0002f9580, 0x2be0e01, 0x2ac0d80, 0xc000123cc0, 0x2d3cb80, 0xc0001da0e0)
	github.com/hashicorp/nomad/jobspec2/parse_map.go:65 +0x59
github.com/hashicorp/nomad/jobspec2.(*walker).Map(0xc0002f95a0, 0x2dd0640, 0xc0000e5638, 0x195, 0xc0002f95a0, 0x2e60701)
	github.com/hashicorp/nomad/jobspec2/parse_map.go:52 +0x20a
github.com/mitchellh/reflectwalk.walkMap(0x2dd0640, 0xc0000e5638, 0x195, 0x2e39b40, 0xc0002f95a0, 0x3483700, 0xc0005dec08)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:205 +0x586
github.com/mitchellh/reflectwalk.walk(0x2dd0640, 0xc0000e5638, 0x195, 0x2e39b40, 0xc0002f95a0, 0x0, 0x2dd0640)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:182 +0x43a
github.com/mitchellh/reflectwalk.walkStruct(0x34836e0, 0xc0000e5600, 0x199, 0x2e39b40, 0xc0002f95a0, 0x100, 0xc0005dedf0)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:386 +0x3ed
github.com/mitchellh/reflectwalk.walk(0x31590a0, 0xc000010530, 0x196, 0x2e39b40, 0xc0002f95a0, 0xc000010500, 0x196)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:188 +0x4e5
github.com/mitchellh/reflectwalk.walkSlice(0x2b58020, 0xc0001ea310, 0x197, 0x2e39b40, 0xc0002f95a0, 0x345b600, 0x0)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:284 +0x211
github.com/mitchellh/reflectwalk.walk(0x2b58020, 0xc0001ea310, 0x197, 0x2e39b40, 0xc0002f95a0, 0x0, 0x2b58020)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:185 +0x3ab
github.com/mitchellh/reflectwalk.walkStruct(0x345b600, 0xc0001ea2d0, 0x199, 0x2e39b40, 0xc0002f95a0, 0x100, 0x0)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:386 +0x3ed
github.com/mitchellh/reflectwalk.walk(0x3159160, 0xc000010418, 0x196, 0x2e39b40, 0xc0002f95a0, 0xc000010400, 0x196)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:188 +0x4e5
github.com/mitchellh/reflectwalk.walkSlice(0x2b58120, 0xc0003dc080, 0x197, 0x2e39b40, 0xc0002f95a0, 0x34a2300, 0x0)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:284 +0x211
github.com/mitchellh/reflectwalk.walk(0x2b58120, 0xc0003dc080, 0x197, 0x2e39b40, 0xc0002f95a0, 0x0, 0x2b58120)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:185 +0x3ab
github.com/mitchellh/reflectwalk.walkStruct(0x34a22e0, 0xc0003dc000, 0x199, 0x2e39b40, 0xc0002f95a0, 0x100, 0x0)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:386 +0x3ed
github.com/mitchellh/reflectwalk.walk(0x2a97c40, 0xc0001fad30, 0x16, 0x2e39b40, 0xc0002f95a0, 0x10cb89cc99259d00, 0x0)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:188 +0x4e5
github.com/mitchellh/reflectwalk.Walk(0x2a97c40, 0xc0001fad30, 0x2e39b40, 0xc0002f95a0, 0x2dd0640, 0x3a33d60)
	github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:92 +0x18f
github.com/hashicorp/nomad/jobspec2.decodeMapInterfaceType(0x2a97c40, 0xc0001fad30, 0xc0002f9580, 0x0, 0x0, 0x0)
	github.com/hashicorp/nomad/jobspec2/parse_map.go:21 +0x7f
github.com/hashicorp/nomad/jobspec2.decode(0xc0001fad20, 0xc0001fad20, 0x34df1d5)
	github.com/hashicorp/nomad/jobspec2/parse.go:108 +0x71b
github.com/hashicorp/nomad/jobspec2.ParseWithConfig(0xc0002424d0, 0xc0002424d0, 0x1a, 0xc000010250)
	github.com/hashicorp/nomad/jobspec2/parse.go:44 +0xa8
github.com/hashicorp/nomad/command.(*JobGetter).ApiJobWithArgs(0xc000140590, 0x7fff743fb757, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	github.com/hashicorp/nomad/command/helpers.go:455 +0x41f
github.com/hashicorp/nomad/command.(*JobRunCommand).Run(0xc0001404e0, 0xc0000d0030, 0x1, 0x1, 0xc00024d6c0)
	github.com/hashicorp/nomad/command/job_run.go:197 +0x6ef
github.com/mitchellh/cli.(*CLI).Run(0xc000570c80, 0xc000570c80, 0xc0001a7140, 0x37)
	github.com/mitchellh/cli@v1.1.0/cli.go:260 +0x1cf
main.RunCustom(0xc0000d0010, 0x3, 0x3, 0xc0000a2058)
	github.com/hashicorp/nomad/main.go:142 +0x4a5
main.Run(...)
	github.com/hashicorp/nomad/main.go:87
main.main()
	github.com/hashicorp/nomad/main.go:83 +0x65
@tgross tgross self-assigned this Feb 3, 2021
@tgross
Copy link
Member

tgross commented Feb 3, 2021

Thanks for opening this issue @manicminer! We definitely don't want to have a panic here.

If we use the -hcl1 flag we get an allocation failure instead:

Recent Events:
Time                       Type               Description
2021-02-03T10:21:11-05:00  Killing            Sent interrupt. Waiting 5s before force killing
2021-02-03T10:21:09-05:00  Alloc Unhealthy    Unhealthy because of failed task
2021-02-03T10:21:09-05:00  Not Restarting     Error was unrecoverable
2021-02-03T10:21:09-05:00  Failed Validation  3 errors occurred:
        * failed to parse config:
        * Invalid index: The given key does not identify an element in this collection value.
        * Invalid index: The given key does not identify an element in this collection value.
2021-02-03T10:21:09-05:00  Task Setup         Building Task Directory
2021-02-03T10:21:09-05:00  Received           Task received by client

As for what you're trying to do, env is not a function in Nomad's HCL2 implementation, so depending on where you want to source the environment from, you either want to use HCL2 variables or just use the task's own shell interpolation like $BLAH.

@tgross tgross added theme/hcl type/bug stage/accepted Confirmed, and intend to work on. No timeline committment though. labels Feb 3, 2021
@tgross tgross removed their assignment Feb 3, 2021
@mwild1
Copy link

mwild1 commented Feb 5, 2021

I think I'm running into the same issue. The relevant part of my config is (inside a task stanza):

      template {
        data        = <<EOH  
APP_DOMAIN={{ env "NOMAD_META_DOMAIN" }}
APP_ADMIN_EMAIL={{ env "NOMAD_META_ADMIN_EMAIL" }}
EOH
        destination = "local/environment"
        env         = true
      }

This was derived from the example at: https://www.nomadproject.io/docs/job-specification/template#inline-template

I've tried various alternatives, I'm not sure what the correct way to do this is.

@mwild1
Copy link

mwild1 commented Feb 5, 2021

Sorry, scratch the above comment. The config snippet I pasted is in fact not the problem.

Elsewhere in the file I had an example that looked just like the original reporter's ${env["foo"]}. I replaced that with ${foo} and the panic has gone away. Was tricky to debug without a line number :)

@tgross tgross added this to Needs Triage in Nomad - Community Issues Triage via automation Feb 12, 2021
@tgross tgross moved this from Needs Triage to Needs Roadmapping in Nomad - Community Issues Triage Feb 12, 2021
@tgross tgross removed this from Needs Roadmapping in Nomad - Community Issues Triage Feb 18, 2021
@tgross tgross self-assigned this Feb 18, 2021
@tgross tgross added this to Needs Triage in Nomad - Community Issues Triage via automation Feb 18, 2021
@tgross tgross moved this from Needs Triage to In Progress in Nomad - Community Issues Triage Feb 18, 2021
@tgross
Copy link
Member

tgross commented Feb 18, 2021

Did a little more debugging. This more minimal jobspec throws the same panic:

variable "env" {
  type = map(string)
  default = {
    "PORT" = "8000"
  }
}

job "http-echo" {
  datacenters = ["home"]
  group "echo" {
    task "server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo:latest"
        args = [
          "-listen", ":${env["PORT"]}",
        ]
      }
    }
  }
}

This variant passes validation and doesn't panic, but then throws an error into nomad alloc status on placement. I think the map declaration is wrong here but I can't figure out why yet.

variable "env" {
  type = map(string)
  default = {
    "PORT" = "8000"
  }
}

job "http-echo" {
  datacenters = ["home"]
  group "echo" {
    task "server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo:latest"
        args = [
          "-listen", ":${env.PORT}",
        ]
      }
    }
  }
}
2021-02-18T10:31:37-05:00  Failed Validation  2 errors occurred:
        * failed to parse config:
        * Missing map element: This map does not have an element with the key "PORT".

Unfortunately this jobspec passes validation (which we probably don't want) but then results in the same error after placement:

job "http-echo" {
  datacenters = ["home"]
  group "echo" {
    task "server" {
      driver = "docker"

      config {
        image = "hashicorp/http-echo:latest"
        args = [
          "-listen", ":${env.PORT}",
        ]
      }
    }
  }
}

@tgross
Copy link
Member

tgross commented Feb 18, 2021

So it looks like there's a bit of an architectural problem here. When we implemented HCL2 we pushed as much of the processing as possible down into the gohcl layer so that we didn't have to write a whole bunch of custom decoders, but it looks like that lightweight approach makes it challenging to recover from this kind of error as well. Because this is entirely client side and there's a workaround (not passing invalid HCL), I'm not going to try to hold up Nomad 1.0.4 for it... I'm going to circle up with some of my colleagues and try to figure out what we can do about this sort of problem for 1.1.

@tgross tgross moved this from In Progress to Needs Roadmapping in Nomad - Community Issues Triage Feb 18, 2021
@tgross tgross removed their assignment Feb 18, 2021
@jesper-sjovall
Copy link

We have the same issue as above, is this any known workarounds yet?

@mwild1
Copy link

mwild1 commented Mar 2, 2021

@jesper-sjovall the issue only occurs when you supply invalid configuration in a job. Therefore the workaround is to fix your job specification.

The problem is that it can be hard to debug without a proper error message. Double-check your variables, especially any usage of env (which is what hit me and the original reporter).

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. theme/cli theme/crash theme/hcl type/bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants