Dispatch batch job summary reporting negative values

[lgfa29]

Nomad version

Nomad v1.0.4 (9294f35f9aa8dbb4acb6e85fa88e3e2534a3e41a)

Operating system and Environment details

MacOS 10.15.7

Issue

When using parameterized batch jobs, the job summary returned by /v1/job/<job>summary miscounts the number of running children, keeping it always at 0.

This results in negative values for Running when the dispatched job finishes.

The problem doesn't happen in Nomad v1.0.3 and git bisect shows that the problem started in this commit: 85129bb

Reproduction steps

1. Start nomad agent

$ sudo nomad agent -dev

2. Run sample job

$ nomad run counter.nomad

3. Dispatch job

$ nomad job dispatch -meta start=0 counter

4. Check job summary

$ curl http://localhost:4646/v1/job/counter/summary | jq .Children
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
100   214  100   214    0     0  42800      0 --:--:-- --:--:-- --:--:-- 53500
{
"Pending": 0,
"Running": 0,
"Dead": 0
}

5. Wait for job to finish and check summary again

$ curl http://localhost:4646/v1/job/counter/summary | jq .Children
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
100   215  100   215    0     0  71666      0 --:--:-- --:--:-- --:--:-- 71666
{
"Pending": 0,
"Running": -1,
"Dead": 1
}

Expected Result

In step 4 there should've been 1 job running, and 0 in step 5.

Actual Result

0 children running in step 4 and -1 in step 5.

Job file

job "counter" {
  datacenters = ["dc1"]
  type        = "batch"

  parameterized {
    meta_required = ["start"]
  }

  group "counter" {
    task "counter" {
      driver = "docker"

      config {
        image   = "alpine:3.13"
        command = "count.sh"
        volumes = [
          "local/count.sh:/usr/bin/count.sh",
        ]
      }

      template {
        data        = <<EOF
#!/bin/sh

count=${NOMAD_META_start}
while [ $count -lt 100000 ]
do
  echo $count
  count=$((count + 1))
done
        EOF
        destination = "local/count.sh"
        perms       = "777"
      }
    }
  }
}

[notnoop]

This seems like a regression in 1.0.4. It's very similar to #10145, in that the initial status is not properly set but in a different path.

[tgross]

Some other issues that look related to this one: #13519 #4731 #10338 #13897.