Provide for dependencies between tasks in a group

[mfischer-zd]

Tasks in a group sometimes need to be ordered to start up correctly.

For example, to support the Ambassador pattern, proxy containers (P[n]) used for outbound request routing by a dependent application may be started only after the dependent application (A) is started. This is because Docker needs to know the name of A to configure shared-container networking when launching P[n].

In the first approximation of the solution, ordering can be simple, e.g., by having the task list in a group be an array.

[ChrisHines]

+1

Batch jobs need task dependencies to manage data-flow between tasks.

I would expand this to allow dependencies between tasks in a job rather than just a task group. Data flow is not constrained to a single node.

[suyash-repo]

Any tentative date to add this enhancement feature in Nomad?

[dadgar]

@suyash1983: we don't have any dates yet but it will most likely be something that comes in 0.4

[blalor]

This is really going to be core to a number of use cases, like allowing EBS volumes to be mounted on a host before the application starts.

[c4milo]

or pulling down certificates and secrets from Vault before starting a given service.

[vrenjith]

+1
This will be a great addition for large enterprise cluster deployments

[steve-jansen]

+1

Running consul-template alongside an app...

[dadgar]

@steve-jansen You can run consul-template currently. We do this in our Nomad deployment. Essentially you have a script that runs consul-template which produces your config, and then runs your binary

[steve-jansen]

@dadgar nice, thanks for sharing. It's always great to hear how a company uses their own products. I imagine consul-template creates the env vars in the job config...

One catch for us, we aspire to have consul-template rewrite the config in response to changes in a Consul k/v, or more importantly, when Vault rotates a secret. We're trying to have consul-template signal the co-scheduled task to reload its config.

Indeed, my need for task dependencies is very narrow. In a perfect world for me, Nomad would have integration with Consul K/V and/or Vault. My need is config that updates dynamically for tasks. That would eliminate my need for task dependencies.

[dadgar]

@steve-jansen consul-template can supervise the process and restart/signal it whenever the config changes. @sethvargo to verify

[sethvargo]

Hi @steve-jansen

Vault currently does not support blocking queries, which is what Consul (and CT) use to give you that "real-time" trigger when something changes. CT will renew secrets at lease_duration/2.0, but it's not currently possible to trigger a change in Vault and have that immediately notify another process. There is an open issue in Vault for blocking queries. Please note this was an intentional design decision in Vault for performance reasons, and not a bug in CT or Vault.

CT will start the process, but it is not a supervisor or monitor. CT has the expectation that the given command will return within 30s, typically by delegating to some supervisor. If you're running in pure Docker, CT can optionally cleanup PIDs and act like PID 1, but it doesn't monitor the process itself (which is best monitored by the scheduler anyway).

[OferE]

+1 - any updates on this?

[nugend]

I'd also like to see this land. I'm evaluating nomad for service orchestration and it would be very nice to be able to explicitly express dependencies.

Alternatively, or as a stop gap, maybe a recipe for doing this with the existing APIs could be written up (#1065 mentions this is possible)

[mbravorus]

Currently, the use cases which require dependencies and cron-like control tend to push me towards airbnb's Chronos which implies Mesos. I would be delighted if I could just use Nomad.

Recurring jobs are possible (starting from #540, ultimately described in https://www.nomadproject.io/docs/job-specification/periodic.html), but in a very common scenario where periodic tasks need interlocking dependency control, there is no way to do it with Nomad currently (or I didn't manage to find it)

[nugend]

Given that this doesn't seem to be a high priority, could someone from the project briefly explain the suggestion about how to achieve this with existing APIs as mentioned in #1065?

[dadgar]

@nugend You would essentially need to put a wrapper around the task that uses the allocation endpoint to determine if the task it should wait for has finished successfully and then start itself. It is non-trivial amount of work though

[RobertAtomic]

+1 for DAG support... It's quite important for my use case.

[tduffield]

I too would love to see this.

[SunSparc]

+1 for task sequence/dependency

[nugend]

For anyone really pining for this, there is a slightly easier method provided you are registering the upstream tasks as services: use a template stanza to create a run script and make that the command of your task. Then you can use a conditional dependent on the service being available to either launch the actual process or sleep forever.

Works pretty well, though it does require a bit of scripting (and I’m not sure how to do it in Windows)

[vrenjith]

Or use containerpilot if you are using docker driver of nomad. https://github.com/joyent/containerpilot

…

On Mon, Oct 23, 2017 at 6:32 PM, Daniel Nugent ***@***.***> wrote: For anyone really pining for this, there is a slightly easier method provided you are registering the upstream tasks as services: use a template stanza to create a run script and make that the command of your task. Then you can use a conditional dependent on the service being available to either launch the actual process or sleep forever. Works pretty well, though it does require a bit of scripting (and I’m not sure how to do it in Windows) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#419 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB_wuV8EgzlMZFkFFd_XrQ_pplUPndUrks5svI5kgaJpZM4Gie-j> .

-- Regards, Renjith

[abohne]

Is there any update on where this might fall on the roadmap?

[kcwong-verseon]

@keith6014 I'm sure you know Airflow is a very different beast than Nomad. They have completely different primary objective. They may be able to work together, however.

[keith6014]

@keith6014 Is Airflow now suitable for service graphs as well as task pipelines? Or were you referring to using Nomad for building task pipelines?

for building task pipelines. Data flow, ETL.

[ValFadeev]

Something that might cater for the use case: https://github.com/ValFadeev/rundeck-nomad-plugin
Basically, using Rundeck as an enhanced UI for Nomad and taking advantage of its workflow-building and time scheduling functionality.
Cons:

• adding Rundeck as another moving piece;
• project needs updating for more recent versions;
  Pros:
• still running jobs on Nomad;

[yishan-lin]

Hi everyone - thank you for the patience.

We are working on implementing native task dependencies now and are exploring a potential Airflow integration.

Would love support in adding feedback + your interest in this ticket to the Apache Airflow committee so they may understand the demand. Ideally, we'd like to optimize the experience by providing a first-class integration, rather than a maintained fork.

https://issues.apache.org/jira/browse/AIRFLOW-5633

cc @jazzyfresh

[CarlosDomingues]

@yishan-lin a Nomad executor for Airflow would be absolutely brilliant.

[sfs77]

in watching, and expect

[sagarrakshe]

I faced the similar issue in our deployments, so I created a tool.
https://github.com/sagarrakshe/nomad-dtree

[DhashS]

We needed this enough that we implemented it ourselves. We have an AST for nomad jobs and interpret it to figure out which consul health checks to watch, wait for their success/fail timeout, and add the unblocked jobs to the work queue.

[recursionbane]

Agreed, we could not wait, either.

We ended up writing a DAG parser to evaluate eligibility of a node based on complex boolean dependencies, only exposing eligible nodes to Nomad for scheduling.

Not ideal, since we are now reliant on a single-threaded process for scheduling, but we are able to schedule several thousand jobs per minute this way. This might pay off in the long term, since it is unlikely Nomad's dependency roadmap includes boolean/complex dependencies.

[yishan-lin]

Hey all, for those that missed our Nomad Virtual Day livestream last week - task dependencies is coming in Nomad 0.11, which folks will hear more about it in the coming weeks.

Here is a recording of the wonderful demo and presentation for reference that @jazzyfresh did on the feature - https://www.hashicorp.com/resources/preview-of-nomad-0-11-task-dependencies

For more complex dependencies as @recursionbane mentioned, we are targeting an integration with Apache Airflow to support such functionality.

[eigengrau]

That’s great news. @jazzyfresh I have a question related to this issue: I presume if we wanted to have a database server up and running before the main task, we would declare it as a pre-start, sidecar task in Nomad v0.11. Does the new lifecycle-hook mechanism observe the Consul health-check of the database service before moving on with the main lifecycle phase? Or would we need to leverage Apache AirFlow for this?

[DhashS]

@yishan-lin that's awesome! Prestart and Poststop hooks are definitely not just a nice-to-have, and i'm super happy that you added them.

However, i don't think that those hooks count as "task dependencies". Consider a group with 5 containers, one that needs to run before (prestart), one that needs to run after (poststop), and the other three containers need to be brought up in sequence.
Prestart and poststop partition the scheduling space into 3 chunks, not N chunks like a true "task dependencies" addition would.

An example of this is how we bring up ZK/Kafka in our software (we run them on nomad with host volumes). We have to submit two different jobs since there's no way to have "generic" task dependencies, so we're forced to wait until ZK's health check comes back before submitting the kafka job. True task dependencies would allow us to coalesce them into one job.

[yishan-lin]

Hey Dhash - you and I synced on this offline but recapping it here for visibility for all. The 5 container group example you mentioned is the kind of DAG functionality that I'd look for our Apache Airflow integration to cover, which is on our roadmap and coming soon!

[DhashS]

Our use case has been worked around well by the use of consul_service_health and nomad_job in terraform.

We now use terraform to submit all our nomad jobs, and the wait_for parameter in the consul_service_health allows the data dependency to the next nomad job to not be fulfilled until all checks are passing

[evandam]

Hey @yishan-lin, I was just curious if there are any updates on the Airflow integration? We would love to see a Nomad executor!

[retarpt]

Hi, does anyone here have experience using Nomad for scheduling Airflow tasks (or vice-versa)? I am looking to constrain resources of individual tasks within an Airflow DAG by isolating them with cgroups and namespaces provided by Nomad's exec driver. Any help, resources, or advice would be so very much appreciated! Thank you, all.

[Oloremo]

Interested in that as well

[ahmedwonolo]

Any update on this?

[tgross]

Nomad lifecycle hooks have shipped for a while now. There's an open issue still for cross-job dependencies #545 that covers the other use cases described here. That's a bigger project and one we've had some discussions about, but it's not on our immediate roadmap either.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.