rolling upgrades / gradual rollout for batch type jobs

[anapsix]

Nomad version

• Nomad v0.12.9
• Nomad v1.0.1

Issue

It appears there is no way to use "update" stanza on "batch" type job to stagger. Only on "service" and "system" type jobs.
In other words, rolling upgrades seem to be unsupported for "batch" type jobs.
Unfortunately, "service" type job isn't flexible enough for my use case, and using "system" job feels wrong.
Executed script could be updated to use a locking mechanism, making it wait until the lock is available before proceeding. But that feels like over engineering the process.
Is there some other Nomad way to achieve a gradual rollout / execution of "batch" job?

Use case: using raw_exec to execute script performing complex deployment on hosts gradually, ensuring such execution will not cause an outage by making application unavailable on all targeted hosts.

/cc @tgross

Reproduction steps

• save job file example as batch-test.nomad
• run nomad job validate batch-test.nomad

Job file

job "deploy-MyApp-297209" {
  datacenters = ["DC1"]
  type        = "batch"

  # make sure to run on windows hosts only
  constraint {
    attribute = "${attr.kernel.name}"
    value     = "windows"
  }

  # run on hosts designated for specific application
  constraint {
    attribute = "${meta.application}"
    operator  = "regexp"
    value     = "(?i)MyApp"
  }

  # do not run more than one on the same node
  constraint {
    distinct_hosts = true
  }

  # rolling updates, run one server at a time
  update {
   max_parallel = 1
   stagger = "60s"
  }

  group "deploy" {
    count = 4

    # retry failed jobs
    restart {
      attempts = 3
      delay    = "10s"
      mode     = "fail"
    }

    # updates shared scripts
    task "update_scripts" {
      driver = "raw_exec"

      config {
        command = "powershell"
        args    = [
          "-command",
          <<-EOS
          .. do prestart things ..
          EOS
        ]
      }

      lifecycle {
        hook    = "prestart"
        sidecar = false
      }
    }

    # runs commands to copy site from artifact destination
    # directory to permanent site's root and performs deployment tasks
    task "deploy_artifacts" {
      driver = "raw_exec"

      # site artifact
      artifact {
        source      = ".. some source .."
        destination = "local/"
      }

      config {
        command = "powershell"
        args    = [
          "-command",
          <<-EOS
          .. do deploy things ..
          EOS
          ]
      }
    }
  }
}

Nomad Client logs

$ nomad job validate batch-test.nomad
Job validation errors:
1 error occurred:
	* Task group deploy validation failed: 1 error occurred:
	* Job type "batch" does not allow update block

[tgross]

Hi @anapsix! Thanks for opening this issue. It looks like you're running into a feature that's been requested a few different times in different ways.

This is sort of like the "Schedule a job depending on another job" issue #545, and partly like the Airflow integration discussed in #419 (comment). One of the ideas we decided not to implement for lifecycle was to allow someone to create a DAG of tasks. So we definitely feel your pain here but don't have a timeline for when this sort of thing can be done "natively" in Nomad.

You may be able to workaround this one of two ways:

• Have each task obtain an external lock (Consul has great primitives for this)
• Make the job parameterized and have each task dispatch the next task via the API.

For both of those workarounds, if you don't want to embed this in the task itself, you could do it as a prestart and poststop task.

[anapsix]

Thank you for a quick response @tgross. Appreciate the workaround suggestion as well. Prestart and poststop could be very useful for this indeed.
I don't think my use case is as complicated, since it's mostly about staggering a job with a single task (disregard the "update_scripts" prestart in my example, it not really relevant to the context of the question).
Since "system" type jobs do support the functionality I'm looking for, I can't help but wonder why it would be a bad idea to allow the same "update" stanza capability on "batch" type jobs?

[tgross]

Since "system" type jobs do support the functionality I'm looking for

Just in case it's helpful, we do have a systembatch job type in the works in #9160. We didn't manage to land that in 1.0 but I expect we intend to ship it in 1.1?

I can't help but wonder why it would be a bad idea to allow the same "update" stanza capability on "batch" type jobs?

I don't think it's a "bad" idea so much as one we just haven't gotten onto the roadmap. Just as an experiment, I just cooked up a build which removes the validation that creates the Job type "batch" does not allow update block error. Then I ran this job, and I think the results illustrate some of the semantic hurdles:

job "example" {
  datacenters = ["dc1"]

  type = "batch"

  group "worker" {

    count = 10

    update {
      max_parallel = 2
      stagger      = "3s"
    }

    task "worker" {
      driver = "docker"

      config {
        image   = "busybox:1"
        command = "/bin/sh"
        args    = ["-c", "echo 'this looks like work'; sleep 10"]
      }

      resources {
        cpu    = 128
        memory = 64
      }
    }
  }
}

If we run that job, we immediately get 10 running allocations because this is the first time we're running it. And after a little while, if we check the job status we see all the tasks in the complete state:

$ nomad job status example
...
Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
worker      10       10      0        0          2021-01-07T20:22:51Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created  Modified
057bf7c0  be8b90b0  worker      0        run      complete  18s ago  4s ago
08927084  be8b90b0  worker      0        run      complete  18s ago  4s ago
2b9cea83  be8b90b0  worker      0        run      complete  18s ago  4s ago
36ea0c94  be8b90b0  worker      0        run      complete  18s ago  3s ago
84eaa606  be8b90b0  worker      0        run      complete  18s ago  4s ago
97fb3e15  be8b90b0  worker      0        run      complete  18s ago  4s ago
a059acd6  be8b90b0  worker      0        run      complete  18s ago  4s ago
ab3d96e3  be8b90b0  worker      0        run      complete  18s ago  3s ago
c2bf7865  be8b90b0  worker      0        run      complete  18s ago  3s ago
d76936d1  be8b90b0  worker      0        run      complete  18s ago  4s ago

If we change something in the job and run it again, we get 10 new allocations, because there are no running allocations to replace in the deployment! If we add update to batch jobs, we'll need to think a bit about how that changes the existing way we do deployments, which needs to check against the status of running tasks from the previous version.

In any case, I suspect what you really want here is a staggered rollout even for the first deployment, and not just on updates. That's why one of the other approaches like a dispatch job might be a reasonable workaround for you for now.

[anapsix]

Just as an experiment, I just cooked up a build which removes the validation...

Oh, thanks for trying that

If we run that job, we immediately get 10 running allocations because this is the first time we're running it

In my case, I'd generate a unique job id (e.g. "deploy-MyApp-297209") each time new application version is deployed. So every time the deployment job runs it would be unique. And all would be executed at once, as your experiment shown.

Going to consider generating a single job per node, or using locking mechanism we've talked about.. And will play with "sysbatch" jobs, then those become available.

...
I was thinking.. perhaps, I could work around it by using a "service" type job, which would run all the same tasks, except it would start a dummy service on some alternative port (and maybe even serve application health data, or deployed version), which could be used for a health_check.. or maybe, I could configure health check to use actual application port..
Either way, "service" type job supports "update" stanza, and can still contain "raw_exec" tasks to do whatever complicated stuff I need them to do in order to perform deployment. That would do the trick, wouldn't it?

[tgross]

I'd steer away from trying to bend service jobs into this role. If a service task completes, Nomad is going to try to restart/reschedule it. Anything you can do to workaround that is likely to end up being more complicated than doing some of the other approaches you're considering.

[tgross]

Going to mark this as a feature request and get it into the roadmapping discussion.