-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Template Stanza single quote and HCL2 cause parser error #9838
Comments
Thanks for reporting this @idrennanvmware! |
Hi @idrennanvmware ! Thanks for raising this issue, and we'll look into how to address this behavior change. I believe the issue to be hcl2 attempting variable expansion, so the workaround would be to to escape We'll document this gotcha in the upgrade guide for now. Also, we'll consider having a way to disable variable interpolation in a heredoc, potentially by quoting the delimiter like in bash (e.g. using |
Hi @notnoop - thanks for the fast reply (and work around). For now we can definitely add that work around but it raises a bigger problem for us in that our config files are now "hcl aware" and that may not always be the case. In fact, often it's not at all. I can see users wanting both types of options (inject the variables for HCL2 etc) but there are also definitely cases we would want the file to be "as is" Interesting problem, thanks for taking the time to hear us out :) For now we'll add the Also, we'll consider having a way to disable variable interpolation in a heredoc, potentially by quoting the delimiter like in bash (e.g. using <<'EOH' in your sample to disable HCL2 variable interpolation). I like it :) |
So I just checked - it would be a massive change impacting multiple project scripts for us to do the |
Disabling variable interpolation is really needed as of now behaviour is tricky. i.e. template {
data = <<-EOH
${var.pause_mem} interpolated to var value
${NOMAD_UPSTREAM_ADDR_hasura} not interpolated and same as $${NOMAD_UPSTREAM_ADDR_hasura}
{{ env "NOMAD_UPSTREAM_ADDR_hasura" }} interpolated
${1} interpolated to 1
${meta.connect.proxy_concurrency} not interpolated
{{ env "meta.connect.proxy_concurrency" }} interpolated
EOH
destination = "/local/hasura/example.tpl"
} Result:
Like in bash i.e. |
Agreed @istarkov - we really are between a rock and a hard place and can't begin to practically migrate with out this ability We're stuck on HC1 for the foreseeable future or until we can escape |
Support the new post-1.0.0 job spec fields in the HCLv1 parser. The HCLv1 parser is still the default (or only!) parser in many downstream tools, e.g. [Levant](https://github.com/hashicorp/levant/blob/e48c439f14def83b0cbe24c2553e90e959474f16/template/render.go#L13-L30), and [terraform-provider-nomad](https://github.com/hashicorp/terraform-provider-nomad/blob/bce32a783176c7943ac662a3eee48b76d92c4d6a/nomad/resource_job.go#L735-L743) . While we initially intended to deprecate HCLv1 parser in 1.0.0, we never communicated that publicly. We did not fully anticipate the public usage of `jobspec` package (we assumed it's a private package), or the friction that HCLv2 introduced in some cases (e.g. #10777, #9838). So moving forward we intend to ensure that new job spec fields are honored in both the HCLv1 and HCLv2 parser until we solidify the migration path and communicate it properly.
Just was taking a look at this briefly because of @42wim's comment here #20195 (comment) and decided to do some archaeology on the HCL repo. It looks like the |
Another thought I had was that we parse the HCL (ref |
I did a little more poking at this and we can't even parse the AST first and then manipulate it before doing evaluation, because the heredoc sigil isn't included in the AST. Ex. a heredoc like this: data = <<EOH
{{ env SOME_ENV }}
EOH has an AST node shaped like this (via
|
Oh, interesting... if I parse just the heredoc with the
That doesn't parse in a whole jobspec. But there might be some support for this in the parser we're just not taking advantage of... will keep digging. Edit: it parses that as a literal but returns diagnostic errors, that's why. I also tried adding a |
I did take a look at what it would take to get heredoc literals added to the HCL syntax, and unfortunately that in itself seems like it'd be a much larger lift than I expected, when compared to the decoder tweaks that Nomad has (ref hashicorp/hcl#620). But given the decoding step is where evaluation happens, there might be an opportunity to introduce some other behavior in the Nomad-specific decoding tweaks. In the meantime, the test below exercises some workarounds using either the jobspec2/parse_test.gofunc TestHeredocInterpolation(t *testing.T) {
templateHeredoc := `
variable "port" {
type = number
default = 8001
}
job "example" {
group "group" {
task "task" {
template {
data = <<EOH
PORT=${var.port}
EOH
}
}
}
}
`
out, err := ParseWithConfig(&ParseConfig{
Path: "input.hcl",
Body: []byte(templateHeredoc),
ArgVars: nil,
AllowFS: false,
})
must.NoError(t, err)
must.Eq(t, "PORT=8001\n", *out.TaskGroups[0].Tasks[0].Templates[0].EmbeddedTmpl)
tmpdir := t.TempDir()
path := filepath.Join(tmpdir, "foo.tpl")
os.WriteFile(path, []byte("PORT=${var.port}\n"), 0777)
literalFromFileContent := fmt.Sprintf(`variable "port" {
type = number
default = 8001
}
job "example" {
group "group" {
task "task" {
template {
data = file("%s")
}
}
}
}
`, path)
out, err = ParseWithConfig(&ParseConfig{
Path: "input.hcl",
Body: []byte(literalFromFileContent),
ArgVars: nil,
AllowFS: true,
})
must.NoError(t, err)
must.Eq(t, "PORT=${var.port}\n", *out.TaskGroups[0].Tasks[0].Templates[0].EmbeddedTmpl)
literalFromVariable := `variable "template" {
type = string
}
job "example" {
group "group" {
task "task" {
template {
data = var.template
}
}
}
}
`
out, err = ParseWithConfig(&ParseConfig{
Path: "input.hcl",
Body: []byte(literalFromVariable),
ArgVars: []string{"template=PORT=${var.port}\n"},
AllowFS: true,
})
must.NoError(t, err)
must.Eq(t, "PORT=${var.port}\n", *out.TaskGroups[0].Tasks[0].Templates[0].EmbeddedTmpl)
} |
Nomad 1.0.2
Issue
We make pretty extensive usage of file content rendering in our template stanza's but we have noticed that if the file contents have
''
in them then the HCL renderer gives us this kind of messageThe problem is that if we then replace the
''
in the file with""
we then get a different error because we also have}
and that error isHere's a snippet from the job file
and the respective file it's trying loading in (redacted)
and with the single quote replacement
Neither of which are ok in the HCL2 parser. However, if we run this with the
-hcl1
flag for deployment then it all works as expectedThe text was updated successfully, but these errors were encountered: