Autoscaler Policy Overrides Count each time a job is deployed

[idrennanvmware]

Output from nomad version
1.0.2

Issue

First, apologies if this is in the wrong repo. I wasn't sure if this fell under Nomad, or the Nomad autoscaler.

We have been experimenting with the scaling stanza in preparation for some work around the Autoscaling feature(s) in Nomad. In our work we found some behavior we didn't expect around the scaling stanza - we found that that our CI/CD pipeline would deploy Nomad jobs (due to other changes) but the count of the job would always revert to the MIN of the scaling stanza. We tried omitting the count at the group level as well as setting it - both scenarios had undesirable results. If we set a count for the group then the job would always get set to that, regardless of any scaling that had been applied. We weren't sure if this was due to us having enabled = false (which is actually what we want as we're using the UI to manually scale groups while we experiment.

Reproduction steps

1. Create a job with a scaling stanza (disabled so the UI can

    scaling {
      enabled = false
      min = 1
      max = 5
      policy {
      }
    }

2. Deploy the job

3. Scale the job group to 2 in the UI

4. Deploy the original job file (again)

Job scales back to the original count of 1.

[jrasell]

Hi @idrennanvmware.

When count is omitted from a task group specification, Nomad sets a default value of 1. What I believe is happening is when you register the job again in step 4, the count is being defaulted to 1 and therefore updates the running count of the job from the scaled value. In order to avoid this, and instead keep the count as described by the Nomad cluster, you can use the preserve count api or run CLI parameter.

Testing this locally I got the behaviour you desire with the following steps:

1. create the example Nomad init job and add the scaling stanza from your comment
2. deploy the job and wait for it to complete successfully
3. scale the job to 2
4. modify the Docker image tag
5. deploy the job again using the following command nomad job run -preserve-counts example.nomad
6. observe the completed deployment produces 2 running allocations

I hope this helps. Please let me know if you have any follow up questions or comments.

[idrennanvmware]

@jrasell - thank you! I had no idea about the -preserve-counts flag. I'm sure that will do the trick. I do have some ideas I'd like to discuss like being able to preserve counts at a group level (but I'll save that for the discuss forum). Closing this for now.

Thanks again

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.