Nomad job plan errors with the below logs.

[blmhemu]

Nomad version

1.3.4

Operating system and Environment details

Debian ARM64

Issue

When doing a nomad job plan, I am getting the following error
Error submitting job: Put "http://100.66.210.90:4646/v1/jobs?namespace=default&region=home": EOF
The plan was working if change_mode is NOT set to script.

Reproduction steps

Added change_mode="script" and change_script to the job template stanza.

Expected Result

Plan works as expected.

Actual Result

Error submitting job: Put "http://100.66.210.90:4646/v1/jobs?namespace=default&region=home": EOF

Job file (if appropriate)

Here is the snippet

          "Templates": [
            {
              "SourcePath": "",
              "DestPath": "local/Caddyfile",
              "EmbeddedTmpl": "<REDACTED>",
              "ChangeMode": "script",
              "ChangeSignal": "",
              "ChangeScript": null,
              "Splay": 5000000000,
              "Perms": "0644",
              "Uid": null,
              "Gid": null,
              "LeftDelim": "[[",
              "RightDelim": "]]",
              "Envvars": false,
              "VaultGrace": 0,
              "Wait": null
            }
          ],

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Aug 28 19:24:36 rpi0 nomad[4032]:     2022-08-28T19:24:36.784+0530 [ERROR] http: http: panic serving 100.111.170.15:53910: runtime error: invalid memor>
Aug 28 19:24:36 rpi0 nomad[4032]: goroutine 2003 [running]:
Aug 28 19:24:36 rpi0 nomad[4032]: net/http.(*conn).serve.func1()
Aug 28 19:24:36 rpi0 nomad[4032]:         net/http/server.go:1850 +0xb8
Aug 28 19:24:36 rpi0 nomad[4032]: panic({0x2161220, 0x3e68400})
Aug 28 19:24:36 rpi0 nomad[4032]:         runtime/panic.go:890 +0x260
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad/structs.(*Template).Validate(0x4000794600)
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/structs/structs.go:7825 +0x30c
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad/structs.(*Task).Validate(0x40017ee6e0, 0x400195a1b0, {0x400220e0d6, 0x6}, {0x400283c>
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/structs/structs.go:7297 +0x13c8
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad/structs.(*TaskGroup).Validate(0x4000bc07e0, 0x4000c42b40)
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/structs/structs.go:6517 +0x21a0
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad/structs.(*Job).Validate(0x4000c42b40)
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/structs/structs.go:4446 +0x1cdc
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad.jobValidate.Validate({}, 0x4000c42b40)
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:248 +0x38
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad.(*Job).admissionValidators(0x400083cc30, 0x46daf0?)
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:104 +0xf8
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad.(*Job).admissionControllers(0x40001161c0?, 0x2673a55?)
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/job_endpoint_hooks.go:71 +0x44
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad.(*Job).Plan(0x400083cc30, 0x4001182a80, 0x4000783ce0)
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/job_endpoint.go:1606 +0xc4
Aug 28 19:24:36 rpi0 nomad[4032]: reflect.Value.call({0x4000862900?, 0x40001105c0?, 0x4e40?}, {0x266bf83, 0x4}, {0x40022493e8, 0x3, 0x4000783ce0?})
Aug 28 19:24:36 rpi0 nomad[4032]:         reflect/value.go:584 +0x688
Aug 28 19:24:36 rpi0 nomad[4032]: reflect.Value.Call({0x4000862900?, 0x40001105c0?, 0x251ea00?}, {0x40022493e8?, 0x251ea00?, 0x22a7c20?})
Aug 28 19:24:36 rpi0 nomad[4032]:         reflect/value.go:368 +0x90
Aug 28 19:24:36 rpi0 nomad[4032]: net/rpc.(*service).call(0x40001163c0, 0x2c3bd28?, 0x40022202c0?, 0x0, 0x40000b9e00, 0x257d540?, {0x251ea00?, 0x400118>
Aug 28 19:24:36 rpi0 nomad[4032]:         net/rpc/server.go:382 +0x20c
Aug 28 19:24:36 rpi0 nomad[4032]: net/rpc.(*Server).ServeRequest(0x4000f31578?, {0x2c3bd28, 0x40022202c0})
Aug 28 19:24:36 rpi0 nomad[4032]:         net/rpc/server.go:503 +0x110
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/nomad.(*Server).RPC(0x400099c380, {0x2673a55, 0x8}, {0x251ea00?, 0x4001182900}, {0x22a7c20>
Aug 28 19:24:36 rpi0 nomad[4032]:         github.com/hashicorp/nomad/nomad/server.go:1891 +0xf8
Aug 28 19:24:36 rpi0 nomad[4032]: github.com/hashicorp/nomad/command/agent.(*Agent).RPC(0x40011bf1d0?, {0x2673a55?, 0x4000cf6000?}, {0x251ea00?, 0x4001>

[blmhemu]

I think i updated only the remote agents and not my local agent 🤦

[tgross]

Hi @blmhemu we almost certainly shouldn't be panicking here even in that scenario. The panic we're hitting is in the validation function for templates in structs.go#L7825. cc'ing @pkazmierczak @lgfa29 as I think y'all worked on this?

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.