Accept JSON via nomad job run

[ccakes]

Nomad version

Nomad v0.9.6

Operating system and Environment details

Mac + Linux

Issue

From the Nomad CLI tool I want to be able to output a job spec and then "run" that output.

For example I have a script which stops a running tasks, applies changes to an underlying service, and then restarts it. nomad job inspect outputs JSON which I can submit back via the Job HTTP API but then I'm forced to pass Nomad and Vault tokens via CURL and it's a bit messy. I have an environment setup where the nomad CLI works and it makes sense to be able to ingest the same format as is output

Reproduction steps

$ nomad job inspect my-task > job.json
$ nomad job stop my-task
==> Monitoring evaluation "f5ac2705"
    Evaluation triggered by job "my-task"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "f5ac2705" finished with status "complete"
$ nomad job run job.json
Error getting job struct: Error parsing job file from job.json: 1 error(s) occurred:

* invalid key: Job

I'm open to any alternatives suggested, but basically I want to be able to interact with a job (ie stop + start) from an environment with access to Nomad, but without access to the original jobspec files.

[tgross]

Hi @ccakes! You're right that the command line doesn't accept the JSON version of the jobspec directly, but the HTTP API does. In fact, the command line client converts the HCL to JSON before POSTing it to the HTTP API itself.

So you're right that the best way to do this is to use curl to send the JSON body directly like so:

▶ nomad job init -short
▶ nomad job run ./example.nomad
==> Monitoring evaluation "6d962c2e"
    Evaluation triggered by job "example"
    Allocation "ea878e56" created: node "707a29ca", group "cache"
    Evaluation within deployment: "8b08b848"
    Allocation "ea878e56" status changed: "pending" -> "running" (Tasks are running)
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "6d962c2e" finished with status "complete"

▶ nomad job inspect example > example.json

▶ nomad job stop example
==> Monitoring evaluation "17721706"
    Evaluation triggered by job "example"
    Evaluation within deployment: "8b08b848"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "17721706" finished with status "complete"

▶ curl -XPOST -H "Content-Type: application/json" \
    -d @example.json http://localhost:4646/v1/jobs
{"EvalCreateIndex":27,"EvalID":"0320367b-cdd8-dd4f-bedc-77aafcc5ec02","Index":27,"JobModifyIndex":26,"KnownLeader":false,"LastContact":0,"Warnings":""}%

▶ nomad job status example
ID            = example
Name          = example
Submit Date   = 2019-11-22T08:25:19-05:00
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
cache       0       0         1        0       1         0

Latest Deployment
ID          = 114691ea
Status      = running
Description = Deployment is running

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
cache       1        1       0        0          2019-11-22T13:35:19Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created    Modified
157bdc9a  707a29ca  cache       2        run      running   8s ago     7s ago
ea878e56  707a29ca  cache       0        stop     complete  1m19s ago  53s ago

I hope this helps!

[ccakes]

Thanks for the reply! The intent is to run this from within a Nomad job where the environment is already setup nicely to call the nomad CLI with all the appropriate tokens set. If I use curl then I need to pass in NOMAD_TOKEN and VAULT_TOKEN via the request and make sure they're not logged.

Definitely do-able, but not ideal 😉

Was intending this to be a bit of a feature request to create a way for the CLI to output a format which can be read back in.

[tgross]

Was intending this to be a bit of a feature request to create a way for the CLI to output a format which can be read back in.

It'd be a nice quality-of-life improvement for sure to allow for some kind of round-tripping of the jobspec with the CLI. Marking this as an enhancement.

[briansorahan]

This behavior is very surprising and disappointing.
nomad run needs to accept what nomad job inspect writes.

[briansorahan]

Also, since nomad job inspect outputs JSON and not HCL, what is the point of the -json flag?
Wouldn't it be better to have an -hcl flag that would write something to stdout that I could save and send back to nomad with nomad run?

[shishir-a412ed]

@tgross What would be a good way to approach this?

Introduce a boolean CLI flag -json in the nomad job run command.
$$ nomad job run -json example.json

If this flag is set, you can either do (1) or (2):

1. Since nomad job JSON is pretty complex, with multiple nested structs. It cannot be directly unmarshalled into the job struct. One way would be to unmarshal the JSON into a generic interface.

var f interface{}
json.NewDecoder(r).Decode(&f)

where r is your io.Reader for the input job JSON.

and use this generic interface f.(map[string]interface{}) to traverse into the job structure and unmarshal nested elements again into generic interfaces, until you are able to decode to atomic types like int, bool, string. Not sure if there is an easier way to unmarshal it directly?

c.JobGetter.ApiJob(args[0]) could take that boolean flag json (as an extra arguement) and unmarshal the JSON into job struct and return.

Then you can just use that job struct to submit the job

OR

2. Take the JSON from CLI and post to nomad server endpoint /v1/jobs, get the response
   and post back to the CLI.

[silasb]

I'm new to Nomad and this lack of consistency is frustrating. Ideally you could nomad job inspect --output hcl|json and nomad run could take in any version of hcl or json in.

[luckymike]

+1 that this is a surprising omission, and one that feels like a bug given the HCL Readme (linked from the Nomad job spec docs):

This allows configuration to be provided as a mixture of human-authored configuration files in the native syntax and machine-generated files in JSON.

Would it be reasonable for the CLI to detect JSON and just skip any of the parsing?

[tgross]

I wanted to circle back to this because there was an internal discussion that generated some confusion around the command line vs the HTTP API, "round tripping", and what is and is not currently supported. A few important notes for the discussion here and any future implementations:

• HCL is currently parsed entirely in the CLI. This is required for HCL2's ability to get environment variables and local files from the CLI's environment.
• The nomad job run command currently accepts only HCL. There's no particular documentation that says otherwise, but I've opened docs: clarify HCL is parsed in CLI #10141 to make it explicit.
• The output of nomad job inspect :jobid is the JSON jobs output. This is round-trippable to the Create Job HTTP API. (There's an integration test demonstrating this behavior as far back as e13f868 in 2017).
• The output of the nomad job inspect -json :jobid command is the Read Job HTTP API output. This is not round-trippable to the Create Job HTTP API. This flag was added back in Support nomad CLI output with JSON and template format  #1503 but it's not super clear what this gets you over leaving it out.
• The only difference between the two is the Job wrapper in the JSON. See the examples below for a very simple workaround which should be feasible for any context in which you're manipulating JSON anyways.

---

ReadJob HTTP API is not round-trippable

$ curl -s "localhost:4646/v1/job/example" | jq . > ReadJob.json
$ curl -d @ReadJob.json "http://localhost:4646/v1/jobs"
Job must be specified

{
  "Affinities": null,
  "AllAtOnce": false,
  "Constraints": null,
  "ConsulToken": "",
  "CreateIndex": 10,
  "Datacenters": [
    "dc1"
  ],
  "Dispatched": false,
  "ID": "example",
  "JobModifyIndex": 10,
  "Meta": null,
  "ModifyIndex": 17,
  "Multiregion": null,
  "Name": "example",
  "Namespace": "default",
  "NomadTokenID": "",
  "ParameterizedJob": null,
  "ParentID": "",
  "Payload": null,
  "Periodic": null,
  "Priority": 50,
  "Region": "global",
  "Spreads": null,
  "Stable": true,
  "Status": "running",
  "StatusDescription": "",
  "Stop": false,
  "SubmitTime": 1615305947060406800,
  "TaskGroups": [
    {
      "Affinities": null,
      "Constraints": null,
      "Count": 1,
      "EphemeralDisk": {
        "Migrate": false,
        "SizeMB": 300,
        "Sticky": false
      },
      "Meta": null,
      "Migrate": {
        "HealthCheck": "checks",
        "HealthyDeadline": 300000000000,
        "MaxParallel": 1,
        "MinHealthyTime": 10000000000
      },
      "Name": "cache",
      "Networks": [
        {
          "CIDR": "",
          "DNS": null,
          "Device": "",
          "DynamicPorts": [
            {
              "HostNetwork": "default",
              "Label": "db",
              "To": 6379,
              "Value": 0
            }
          ],
          "IP": "",
          "MBits": 0,
          "Mode": "",
          "ReservedPorts": null
        }
      ],
      "ReschedulePolicy": {
        "Attempts": 0,
        "Delay": 30000000000,
        "DelayFunction": "exponential",
        "Interval": 0,
        "MaxDelay": 3600000000000,
        "Unlimited": true
      },
      "RestartPolicy": {
        "Attempts": 2,
        "Delay": 15000000000,
        "Interval": 1800000000000,
        "Mode": "fail"
      },
      "Scaling": null,
      "Services": null,
      "ShutdownDelay": null,
      "Spreads": null,
      "StopAfterClientDisconnect": null,
      "Tasks": [
        {
          "Affinities": null,
          "Artifacts": null,
          "CSIPluginConfig": null,
          "Config": {
            "image": "redis:3.2",
            "ports": [
              "db"
            ]
          },
          "Constraints": null,
          "DispatchPayload": null,
          "Driver": "docker",
          "Env": null,
          "KillSignal": "",
          "KillTimeout": 5000000000,
          "Kind": "",
          "Leader": false,
          "Lifecycle": null,
          "LogConfig": {
            "MaxFileSizeMB": 10,
            "MaxFiles": 10
          },
          "Meta": null,
          "Name": "redis",
          "Resources": {
            "CPU": 500,
            "Devices": null,
            "DiskMB": 0,
            "IOPS": 0,
            "MemoryMB": 256,
            "Networks": null
          },
          "RestartPolicy": {
            "Attempts": 2,
            "Delay": 15000000000,
            "Interval": 1800000000000,
            "Mode": "fail"
          },
          "ScalingPolicies": null,
          "Services": null,
          "ShutdownDelay": 0,
          "Templates": null,
          "User": "",
          "Vault": null,
          "VolumeMounts": null
        }
      ],
      "Update": {
        "AutoPromote": false,
        "AutoRevert": false,
        "Canary": 0,
        "HealthCheck": "checks",
        "HealthyDeadline": 300000000000,
        "MaxParallel": 1,
        "MinHealthyTime": 10000000000,
        "ProgressDeadline": 600000000000,
        "Stagger": 30000000000
      },
      "Volumes": null
    }
  ],
  "Type": "service",
  "Update": {
    "AutoPromote": false,
    "AutoRevert": false,
    "Canary": 0,
    "HealthCheck": "",
    "HealthyDeadline": 0,
    "MaxParallel": 1,
    "MinHealthyTime": 0,
    "ProgressDeadline": 0,
    "Stagger": 30000000000
  },
  "VaultNamespace": "",
  "VaultToken": "",
  "Version": 0
}

Read Job HTTP API with minimal manipulation is round-trippable

$ jq '{Job: .}' ReadJob.json | curl -d @- "http://localhost:4646/v1/jobs"
{"EvalCreateIndex":42,"EvalID":"09b2333c-5cac-cba8-e34a-23f0b38df1d0","Index":42,"JobModifyIndex":10,"KnownLeader":false,"LastContact":0,"Warnings":""

nomad inspect -json is not round-trippable

$ nomad job inspect -json example > InspectJsonFlag.json
$ curl -d @InspectJsonFlag.json "http://localhost:4646/v1/jobs"
Job must be specified

{
    "Affinities": null,
    "AllAtOnce": false,
    "Constraints": null,
    "ConsulToken": "",
    "CreateIndex": 10,
    "Datacenters": [
        "dc1"
    ],
    "Dispatched": false,
    "ID": "example",
    "JobModifyIndex": 10,
    "Meta": null,
    "Migrate": null,
    "ModifyIndex": 17,
    "Multiregion": null,
    "Name": "example",
    "Namespace": "default",
    "NomadTokenID": "",
    "ParameterizedJob": null,
    "ParentID": "",
    "Payload": null,
    "Periodic": null,
    "Priority": 50,
    "Region": "global",
    "Reschedule": null,
    "Spreads": null,
    "Stable": true,
    "Status": "running",
    "StatusDescription": "",
    "Stop": false,
    "SubmitTime": 1615305947060406757,
    "TaskGroups": [
        {
            "Affinities": null,
            "Constraints": null,
            "Count": 1,
            "EphemeralDisk": {
                "Migrate": false,
                "SizeMB": 300,
                "Sticky": false
            },
            "Meta": null,
            "Migrate": {
                "HealthCheck": "checks",
                "HealthyDeadline": 300000000000,
                "MaxParallel": 1,
                "MinHealthyTime": 10000000000
            },
            "Name": "cache",
            "Networks": [
                {
                    "CIDR": "",
                    "DNS": null,
                    "Device": "",
                    "DynamicPorts": [
                        {
                            "HostNetwork": "default",
                            "Label": "db",
                            "To": 6379,
                            "Value": 0
                        }
                    ],
                    "IP": "",
                    "MBits": 0,
                    "Mode": "",
                    "ReservedPorts": null
                }
            ],
            "ReschedulePolicy": {
                "Attempts": 0,
                "Delay": 30000000000,
                "DelayFunction": "exponential",
                "Interval": 0,
                "MaxDelay": 3600000000000,
                "Unlimited": true
            },
            "RestartPolicy": {
                "Attempts": 2,
                "Delay": 15000000000,
                "Interval": 1800000000000,
                "Mode": "fail"
            },
            "Scaling": null,
            "Services": null,
            "ShutdownDelay": null,
            "Spreads": null,
            "StopAfterClientDisconnect": null,
            "Tasks": [
                {
                    "Affinities": null,
                    "Artifacts": null,
                    "Config": {
                        "image": "redis:3.2",
                        "ports": [
                            "db"
                        ]
                    },
                    "Constraints": null,
                    "DispatchPayload": null,
                    "Driver": "docker",
                    "Env": null,
                    "KillSignal": "",
                    "KillTimeout": 5000000000,
                    "Kind": "",
                    "Leader": false,
                    "Lifecycle": null,
                    "LogConfig": {
                        "MaxFileSizeMB": 10,
                        "MaxFiles": 10
                    },
                    "Meta": null,
                    "Name": "redis",
                    "Resources": {
                        "CPU": 500,
                        "Devices": null,
                        "DiskMB": 0,
                        "IOPS": 0,
                        "MemoryMB": 256,
                        "Networks": null
                    },
                    "RestartPolicy": {
                        "Attempts": 2,
                        "Delay": 15000000000,
                        "Interval": 1800000000000,
                        "Mode": "fail"
                    },
                    "ScalingPolicies": null,
                    "Services": null,
                    "ShutdownDelay": 0,
                    "Templates": null,
                    "User": "",
                    "Vault": null,
                    "VolumeMounts": null
                }
            ],
            "Update": {
                "AutoPromote": false,
                "AutoRevert": false,
                "Canary": 0,
                "HealthCheck": "checks",
                "HealthyDeadline": 300000000000,
                "MaxParallel": 1,
                "MinHealthyTime": 10000000000,
                "ProgressDeadline": 600000000000,
                "Stagger": 30000000000
            },
            "Volumes": null
        }
    ],
    "Type": "service",
    "Update": {
        "AutoPromote": false,
        "AutoRevert": false,
        "Canary": 0,
        "HealthCheck": "",
        "HealthyDeadline": 0,
        "MaxParallel": 1,
        "MinHealthyTime": 0,
        "ProgressDeadline": 0,
        "Stagger": 30000000000
    },
    "VaultNamespace": "",
    "VaultToken": "",
    "Version": 0
}

nomad inspect -json with minimal manipulation is round-trippable

$ jq '{Job: .}' InspectJsonFlag.json | curl -d @- "http://localhost:4646/v1/jobs"
{"EvalCreateIndex":37,"EvalID":"dc72b6ac-2062-5747-1eca-e24f02131081","Index":37,"JobModifyIndex":10,"KnownLeader":false,"LastContact":0,"Warnings":""}%

nomad inspect is round-trippable

$ nomad job inspect example > InspectNoJsonFlag.json
$ curl -d @InspectNoJsonFlag.json "http://localhost:4646/v1/jobs"
{"EvalCreateIndex":30,"EvalID":"2138a16f-953e-0074-a6ea-3de5a7221605","Index":30,"JobModifyIndex":10,"KnownLeader":false,"LastContact":0,"Warnings":""}

{
    "Job": {
        "Affinities": null,
        "AllAtOnce": false,
        "Constraints": null,
        "ConsulToken": "",
        "CreateIndex": 10,
        "Datacenters": [
            "dc1"
        ],
        "Dispatched": false,
        "ID": "example",
        "JobModifyIndex": 10,
        "Meta": null,
        "Migrate": null,
        "ModifyIndex": 17,
        "Multiregion": null,
        "Name": "example",
        "Namespace": "default",
        "NomadTokenID": "",
        "ParameterizedJob": null,
        "ParentID": "",
        "Payload": null,
        "Periodic": null,
        "Priority": 50,
        "Region": "global",
        "Reschedule": null,
        "Spreads": null,
        "Stable": true,
        "Status": "running",
        "StatusDescription": "",
        "Stop": false,
        "SubmitTime": 1615305947060406757,
        "TaskGroups": [
            {
                "Affinities": null,
                "Constraints": null,
                "Count": 1,
                "EphemeralDisk": {
                    "Migrate": false,
                    "SizeMB": 300,
                    "Sticky": false
                },
                "Meta": null,
                "Migrate": {
                    "HealthCheck": "checks",
                    "HealthyDeadline": 300000000000,
                    "MaxParallel": 1,
                    "MinHealthyTime": 10000000000
                },
                "Name": "cache",
                "Networks": [
                    {
                        "CIDR": "",
                        "DNS": null,
                        "Device": "",
                        "DynamicPorts": [
                            {
                                "HostNetwork": "default",
                                "Label": "db",
                                "To": 6379,
                                "Value": 0
                            }
                        ],
                        "IP": "",
                        "MBits": 0,
                        "Mode": "",
                        "ReservedPorts": null
                    }
                ],
                "ReschedulePolicy": {
                    "Attempts": 0,
                    "Delay": 30000000000,
                    "DelayFunction": "exponential",
                    "Interval": 0,
                    "MaxDelay": 3600000000000,
                    "Unlimited": true
                },
                "RestartPolicy": {
                    "Attempts": 2,
                    "Delay": 15000000000,
                    "Interval": 1800000000000,
                    "Mode": "fail"
                },
                "Scaling": null,
                "Services": null,
                "ShutdownDelay": null,
                "Spreads": null,
                "StopAfterClientDisconnect": null,
                "Tasks": [
                    {
                        "Affinities": null,
                        "Artifacts": null,
                        "Config": {
                            "image": "redis:3.2",
                            "ports": [
                                "db"
                            ]
                        },
                        "Constraints": null,
                        "DispatchPayload": null,
                        "Driver": "docker",
                        "Env": null,
                        "KillSignal": "",
                        "KillTimeout": 5000000000,
                        "Kind": "",
                        "Leader": false,
                        "Lifecycle": null,
                        "LogConfig": {
                            "MaxFileSizeMB": 10,
                            "MaxFiles": 10
                        },
                        "Meta": null,
                        "Name": "redis",
                        "Resources": {
                            "CPU": 500,
                            "Devices": null,
                            "DiskMB": 0,
                            "IOPS": 0,
                            "MemoryMB": 256,
                            "Networks": null
                        },
                        "RestartPolicy": {
                            "Attempts": 2,
                            "Delay": 15000000000,
                            "Interval": 1800000000000,
                            "Mode": "fail"
                        },
                        "ScalingPolicies": null,
                        "Services": null,
                        "ShutdownDelay": 0,
                        "Templates": null,
                        "User": "",
                        "Vault": null,
                        "VolumeMounts": null
                    }
                ],
                "Update": {
                    "AutoPromote": false,
                    "AutoRevert": false,
                    "Canary": 0,
                    "HealthCheck": "checks",
                    "HealthyDeadline": 300000000000,
                    "MaxParallel": 1,
                    "MinHealthyTime": 10000000000,
                    "ProgressDeadline": 600000000000,
                    "Stagger": 30000000000
                },
                "Volumes": null
            }
        ],
        "Type": "service",
        "Update": {
            "AutoPromote": false,
            "AutoRevert": false,
            "Canary": 0,
            "HealthCheck": "",
            "HealthyDeadline": 0,
            "MaxParallel": 1,
            "MinHealthyTime": 0,
            "ProgressDeadline": 0,
            "Stagger": 30000000000
        },
        "VaultNamespace": "",
        "VaultToken": "",
        "Version": 0
    }
}

[luckymike]

Is there any way to extract a JSON job spec locally from the CLI? The current behavior makes it impossible to interact with both an HCL spec and the API directly, because you need to round-trip to a Nomad server to get JSON (which is of course only possible if you already have a valid job spec).

I think what's maybe being missed here by the Nomad team is that there are very strong use cases for working with a JSON job spec prior to submitting the job. For example:

• Linting the job for expected configuration
• Programmatic generation of a job spec
• Direct submission via the API

It would be nice if using JSON did not lock you out of the CLI.

[jippi]

Is there any way to extract a JSON job spec locally from the CLI

nomad run -output $JOB_FILE

[tgross]

I think what's maybe being missed here by the Nomad team is that there are very strong use cases for working with a JSON job spec prior to submitting the job.

We're not missing it. This issue is still open and marked as an enhancement, after all.

But your examples of manipulating the JSON job spec, programmatically generating JSON, or submitting JSON via the API don't seem to the problem you have here? That's the motivation for why you want to convert from HCL to JSON. But you can already do this via nomad run -output $JOB_FILE if you want to do it locally and get the HCL2 interpolation done for you (as @jippi has suggested), or you can use the Parse Job HTTP API if you don't care about local interpolation or mind talking to the server.

Once you're at that point, it's true that you can't submit that JSON via nomad job run... that's what this issue is and it's marked as an enhancement. But given that you're already working in a programmatic context, submitting via the HTTP API (or using a library wrapper like the api package, or the Java SDK) seem like it's right in front of you.

[luckymike]

Thanks @jippi @tgross, that's indeed a big help. I didn't realize that was under the run command.

[blaggacao]

@tgross The nomad CLI has two assets that are valuable, even if tooling is providing json output:

• Waiting for healthchecks to pass
• Rendering the diffs from /v1/job/$id/plan nicely

Maybe the internal discussion can be enhanced with those arguments and proper support for lingua franca json will be added to nomad job run.

(even french people learn english now a days 😄 )

---

We want to add first class nomad support to https://github.com/fluidattacks/makes, a nix based task runner. My current research is that implementing json (a first class nix output) is the only thing that's currently blocking this.

/cc @kamadorueda

EDIT: committing to maintaining wrapper tooling is not an option just to add first class support for nomad.

[yaroot]

Hi, some recent experience, I'm using dhall with https://github.com/seatgeek/dhall-nomad to generate json job files (as json is technically HCL2), and submit the results via nomad run. Other than double escaping in string interpolation it works pretty well.

[maxadamo]

@tgross the problem with HCL arises is when it comes to tooling, and using third parties solutions, and HCL is kind of esoteric. The way around, JSON and YAML are lingua franca and there are plenty of libraries for plenty of programming languages.
One example for all: there is a nice module for Puppet to configure Nomad because the configuration can be done using JSON, but it doesn't exist (and I have also tried myself) a module to handle the Nomad jobs.
Puppet has to_json, to_json.pretty, and to_yaml functions and you can easily translate the code in your manifest to JSON or YAML.
It's likely to be the same with Ansible or any other tool.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.