[Improvement] Add an API endpoint to render HCL as JSON job

[stevenscg]

We like the expressiveness and readability of the HCL format for our job files, but run into issues when trying to automate job submissions to the Nomad cluster via API because only the JSON job format is accepted.

Currently, we have deployment scripts that have to shell out to the nomad agent cli and echo job contents in HCL to nomad run -output.

This often occurs after templating basic variables into the HCL from outside sources.

We then typically fetch the currently running job, pull out the jobId, and sync up the task counts before submitting the job via POST /v1/job/:jobId.

It would be extremely useful to have access to the JSON-formatted job contents via API prior to submittal without having to have the nomad binary available on the job submitter's workstation or within a CI service.

We've run across this use case several times now. The first was a simple CI/CD workflow. The most recent is when bootstrapping a new nomad cluster or datacenter with Ansible or Terraform.

Based on feedback on the Gitter room, others have encountered this as well.

[stevenscg]

While we're at it, could we pass in a collection of variables and their values that will be templated into the JSON job that is returned?

What about interpolated or from Consul KV?

[mengesb]

What about interpolated or from Consul KV?

Wouldn't that be consul-template?

Sounds like an interesting situation, however I've (for the most part) abandoned HCL for most things and have gone pure JSON, especially where we can use terraform's jsonencode() interpolation

[stevenscg]

@mengesb I've thought of going straight to json as well. Might happen yet.

The interpolation I mentioned in the comment would be for the job file. I've found that I have some per-invocation data that always needs to be templated into a job file (container tag or commit SHA) that would not likely be in consul.

However, things like the target nomad region or the region-local container registry url or the expected task counts could all be stored in consul. These don't really change per-invocation. I'm currently fetching them from consul myself and templating them into the job file prior to submission.

[sprutner]

+1. HCL is so nice but we can't even use it for this reason.

[lord2800]

I have a ci/cd workflow that could benefit from this as well. I currently maintain a hand-encoded json file as well as the hcl file.

[adragoset]

I have yet to get a working job using the json api even copying the json from inspect after submitting via hcl throws errors when posting the job back to the rest api.

[dadgar]

@adragoset Just did the following and it worked:

nomad init
nomad run example.nomad
nomad inspect example > job.json
(can stop the job so you can see it gets re-registered)
curl -XPUT -d @job.json http://127.0.0.1:4646/v1/job/example

[adragoset]

I finally managed to get it working after i tried your method of piping the command output to a file. From there i was able to take the valid json created from the hcl and modify it into a more complex job template that i run through jinja first to add stuff that i wanted outside of what consul template can replace inside the job definition.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.