nomad plan for periodic tasks

[holtwilkins]

Nomad version 0.5.0.

For a periodic job, nomad plan seems to always exit with exitcode 1, indicating pending changes, even when there are no changes pending.

This output is when the job doesn't exist, this is expected

[centos@ip-10-2-211-31 ~]$ nomad plan hello-periodic.hcl
+ Job: "hello-periodic"
+ Task Group: "hello-periodic" (1 create)
  + Task: "hello-periodic" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.
- If submitted now, next periodic launch would be at 11/18/16 23:10:00 UTC (2m22s from now).

Job Modify Index: 0
To submit the job with version verification run:

nomad run -check-index 0 hello-periodic.hcl

When running the job with the check-index flag, the job will only be run if the
server side version matches the job modify index returned. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
[centos@ip-10-2-211-31 ~]$ echo $?
1

However, I expect that once the job is running, that subsequent plans are noops, but that's not what happens.

[centos@ip-10-2-211-31 ~]$ nomad run hello-periodic.hcl
Job registration successful
Approximate next launch time: 11/18/16 23:10:00 UTC (1m2s from now)
[centos@ip-10-2-211-31 ~]$ nomad plan hello-periodic.hcl
+/- Job: "hello-periodic"
Datacenters {
  Datacenters: "us-east-1"
}
Task Group: "hello-periodic" (1 create)
  Task: "hello-periodic"

Scheduler dry-run:
- All tasks successfully allocated.
- If submitted now, next periodic launch would be at 11/18/16 23:10:00 UTC (57s from now).

Job Modify Index: 13593
To submit the job with version verification run:

nomad run -check-index 13593 hello-periodic.hcl

When running the job with the check-index flag, the job will only be run if the
server side version matches the job modify index returned. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
[centos@ip-10-2-211-31 ~]$ echo $?
1

EDIT: Including the jobfile as an attachment, but I renamed it to hello-periodic.txt to make github happy.
hello-periodic.txt

[holtwilkins]

Also of note, if a batch type job is brand new, nomad plan exits with exitcode 0, whereas it should be 1, like it is for service.

[dadgar]

Thanks for the report!

[clinta]

I also found this when doing some config managemetnt for nomad via the API. This is because the nomad plan API call always returns Place: 1 in the DesiredTGUpdates for batch jobs.

[dadgar]

Just ran nomad plan with a new batch job and it exits with code 1 which is correct:

$ nomad plan example.nomad
+ Job: "test"
+ Task Group: "server" (1 create)
  + Task: "server" (forces create)

Scheduler dry-run:
- WARNING: Failed to place all allocations.
  Task Group "server" (failed to place 1 allocation):
    * Resources exhausted on 1 nodes
    * Dimension "memory exhausted" exhausted on 1 nodes

Job Modify Index: 0
To submit the job with version verification run:

nomad run -check-index 0 example.nomad

When running the job with the check-index flag, the job will only be run if the
server side version matches the job modify index returned. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
$ echo $?
1

@clinta I don't believe that is true. That depends on the count of the task group!

[clinta]

@dadgar Here's how to reproduce, with v0.5.6. Use the following batch job:

{
    "Job": {
        "ID": "test",
        "Name": "test",
        "Type": "batch",
        "Datacenters": [
            "dc0"
        ],
        "TaskGroups": [
            {
                "Name": "test",
                "Count": 1,
                "Constraints": [],
                "Tasks": [
                    {
                        "Name": "test",
                        "Driver": "exec",
                        "Config": {
                            "command": "echo",
                            "args": [
                                "foobar"
                            ]
                        }
                    }
                ]
            }
        ],
        "Periodic": {
            "Enabled": true,
            "Spec": "1 * * * * *",
            "SpecType": "cron",
            "ProhibitOverlap": true
        }
    }
}

Run the job:

# curl -X PUT http://127.0.0.1:4646/v1/job/test --upload-file test.json
{"EvalID":"","EvalCreateIndex":0,"JobModifyIndex":34193,"Index":0,"LastContact":0,"KnownLeader":false}%    

Then run plan. Place should be 0, as this job is unchanged.

# curl -X PUT http://127.0.0.1:4646/v1/job/test/plan --upload-file test.json
{"Annotations":{"DesiredTGUpdates":{"test":{"Ignore":0,"Place":1,"Migrate":0,"Stop":0,"InPlaceUpdate":0,"DestructiveUpdate":0}}},"FailedTGAllocs":null,"JobModifyIndex":34193,"CreatedEvals":null,"Diff":null,"NextPeriodicLaunch":"2017-05-04T14:01:00Z","Index":34193}% 

[dadgar]

@clinta Ah! Sorry I thought you meant that plan was always returning Place:1 regardless of the count of the actual task group.

[tgross]

Doing some issue cleanup and I've re-verified this against current versions of Nomad:

job "example" {
  datacenters = ["dc1"]
  type        = "batch"

  periodic {
    cron             = "*/10 * * * * *"
    prohibit_overlap = true
  }

  group "group" {

    task "task" {
      driver = "exec"

      config {
        command = "/bin/sh"
        args    = ["-c", "echo $(date)"]
      }
    }
  }
}

Initial plan:

$ nomad plan example.nomad
+ Job: "example"
+ Task Group: "group" (1 create)
  + Task: "task" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.
- If submitted now, next periodic launch would be at 2021-07-07T19:40:00Z (22s from now).

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 example.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

$ echo $?
1

Run the job:

$ nomad job run ./example.nomad
Job registration successful
Approximate next launch time: 2021-07-07T19:50:00Z (9m41s from now)

Plan again:

$ nomad plan example.nomad
+/- Job: "example"
+/- Task Group: "group" (1 create)
  +/- Task: "task" (forces in-place update)

Scheduler dry-run:
- All tasks successfully allocated.
- If submitted now, next periodic launch would be at 2021-07-07T19:50:00Z (9m32s from now).

Job Modify Index: 10
To submit the job with version verification run:

nomad job run -check-index 10 example.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
$ echo $?
1

$ nomad job inspect example > example.json

$ curl -d @example.json "http://127.0.0.1:4646/v1/job/example/plan"
{"Annotations":{"DesiredTGUpdates":{"group":{"Canary":0,"DestructiveUpdate":0,"Ignore":0,"InPlaceUpdate":0,"Migrate":0,"Place":1,"Preemptions":0,"Stop":0}},"PreemptedAllocs":null},"CreatedEvals":null,"Diff":null,"FailedTGAllocs":null,"Index":10,"JobModifyIndex":10,"NextPeriodicLaunch":"2021-07-07T19:50:00Z","Warnings":""}%

[lgfa29]

Fixed in #14492

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.