Forcing periodic jobs results in one pending and one running

[thetooth]

Nomad version

Nomad v1.0.1 (c9c68aa)

Issue

When forcing a periodic job to run, a mystery pending job appears and hangs around forever, stopping the running instance also results in the entire job getting garbage collected(job is going from running->dead status when stopping myjob/periodic-xxx), preventing the actual scheduled launch of the job.

Reproduction steps

nomad run myjob.hcl
Job registration successful
Approximate next launch time: 2021-01-13T06:00:00+10:30 (17h35m2s from now)

nomad job periodic force myjob
==> Monitoring evaluation "5e6a0dd5"
    Evaluation triggered by job "livesale-streamer/periodic-1610416639"
    Allocation "1124a587" created: node "a6a13bca", group "streamer"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "5e6a0dd5" finished with status "complete"

nomad job status livesale-streamer
ID                   = livesale-streamer
Name                 = livesale-streamer
Submit Date          = 2021-01-12T12:24:58+10:30
Type                 = batch
Priority             = 50
Datacenters          = mtc1
Namespace            = default
Status               = running
Periodic             = true
Parameterized        = false
Next Periodic Launch = 2021-01-13T06:00:00+10:30 (17h29m25s from now)

Children Job Summary
Pending  Running  Dead
1        1        0

Previously Launched Jobs
ID                                     Status
livesale-streamer/periodic-1610416639  running

nomad job stop livesale-streamer/periodic-1610416639
==> Monitoring evaluation "f1c849d0"
    Evaluation triggered by job "livesale-streamer/periodic-1610416639"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "f1c849d0" finished with status "complete"

nomad job status livesale-streamer
ID                   = livesale-streamer
Name                 = livesale-streamer
Submit Date          = 2021-01-12T12:24:58+10:30
Type                 = batch
Priority             = 50
Datacenters          = mtc1
Namespace            = default
Status               = dead (stopped)
Periodic             = true
Parameterized        = false
Next Periodic Launch = none (job stopped)

Children Job Summary
Pending  Running  Dead
1        0        1

Previously Launched Jobs
ID                                     Status
livesale-streamer/periodic-1610416639  dead

Job file (if appropriate)

job "livesale-streamer" {
  datacenters = ["mtc1"]
  type = "batch"

  periodic {
    cron = "0 6 * * WED"
    time_zone = "Australia/Adelaide"
  }

  reschedule {
    unlimited      = true
    delay          = "5s"
    delay_function = "constant"
    attempts       = 0
  }

  group "streamer" {
    restart {
      attempts = 5
      delay    = "1s"
    }
    task "head" {
      constraint {
        attribute = "${meta.capturecard}"
        operator  = "="
        value     = "yes"
      }

      driver = "raw_exec"

      config {
        command = "/bin/bash"
        args    = [
                  "-c",
                  "cd /opt && /opt/head.sh"
        ]
      }

      resources {
        cpu    = 6500 # MHz
        memory = 1024 # MB
      }
    }
  }
}

[tgross]

Thanks for reporting this @thetooth. Looks like this might be another case of #8692, and that it should be fixed by @drewbailey's PR: #9768

[thetooth]

Excellent, I did try to find an issue for this but must have missed it. Good to see we have a fix already, if I correct my application to return 0 on termination the parent job does not enter the dead state, also it can be brought back from the dead by simply running it again before GC. Periodic jobs are indeed running when they should.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.