[Feedback] Nomad 0.11 lifecycle insufficient for 3 phase deploys

[spuder]

Nomad 0.11 introduced lifecycle stanza which is a much needed resource. According to the blogpost you are looking for feedback

One use case this doeesn't cover is when you have 3 steps

• postgres server
• seed database
• start web server

Here is an example job

https://gist.github.com/spuder/54f4b54a86d4690a76bfc4ed8eecc415

I've set the following

• postgres server

    task "postgres" {
      driver = "docker"
      config {
        image = "postgres:9.6-alpine"
      }
      lifecycle {
        sidecar = true # <= calling this a sidecar doesn't really seem right. It does ensure it will keep running
        hook = "prestart"
      }

• seed database

    task "seed_database" {
      driver = "docker"
      lifecycle {
        sidecar = false  # <= this allows the job to run then exit
        hook = "prestart"
      }
      config {
        image = "redash/redash:8.0.0.b32245"
        command = "create_db"
      }
...

• start web server

While this mostly works, and the seed database job always runs before the web server, it doesn't guarantee that postgres will be up before the seed database job.

There needs to be a way to allow for 3 services to be chained together

[DhashS]

There’s a good amount of discussion here #419 - but the short of it is that nomad does not currently support this. We ended up using terraform to implement this

[yishan-lin]

We are planning on adding a PostStop hook to the Nomad 0.12 series. Would this be helpful?

[Legogris]

As a separate one, multiple prestarts which need to be run sequentially/in a particular order.

I can see two alternatives:

1. A sequence/order/suitable name - like init.d runlevels
2. depends_on a la systemd services

[DhashS]

@yishan-lin adding more hooks just solves it for an N stage deploy, where N is the amount of hooks you have. One can imagine that a 5-phase deploy where only the last phase is the main container needing a postprepreprestart hook or something heinous like that.

Docker-compose does this with the depends_on clause

[yishan-lin]

Heard loud and clear. We’ll be introducing PostStop and PostStart hooks into the 0.12 series but know that doesn’t address the underlying limitations of such a design.

We’ll look to our airflow integration targeted in the fall to provide this functionality in a comprehensive manner to Nomad.

[tino]

I see that lifecycle works on task level. I would like it to work on a group level. Why?

Right now we run some auxiliary apps that need a redis database. Preferably I would put redis in a different group, so it can be spawned on a different node if necessary. Also I might need a higher count of workers than redis dbs. However now that means I have to put redis either in the same group so I can refer to its address with S{NOMAD_ADDR_redis_redis}, or put it in a separate nomad file and launch it first.

If I'm able to say: this group should be up before the other group, then we can register the redis task in consul, and use the service consul-template tag in the worker's env template to pass it the address.

[Aposhian]

Also currently blocked by this. I'm concerned that it seems the plan to support this simple use-case is to outsource to Apache Airflow. I have no interest in Airflow: I just need to specify a basic dependency chain, which is already possible in docker-compose or systemd.