[Discussion] Managing template source files with container-based drivers

[stevenscg]

The following is a summary from the nomad/Lobby Gitter room starting around March 1, 2017 11:50am.

I posed the question:

Question for those using the docker driver and templating with Nomad:
How did you deal with requirement for template source files to exist on the nomad worker prior to job launch?
I think I've heard the following:
a) Use the artifact stanza to fetch the template source files,
b) Embed consul-template within the container image,
c) Some out-of-band process to fetch the source files,
d) Something else entirely.
e) Embed the template source directly in the job file with the data key. (suggested by @schmichael).

My point was: I really like the nomad template functionality, but if I'm going to do containers, I'd really like to have a legit artifact that I can manage.

Michael also summarized the internal flow, which was very helpful:

To put it in Nomad internal terms: we currently do the following:
(1) download artifacts, (2) render templates, (3) driver.Prestart() downloads image, (4) driver.Start() starts container.

What we need is: (1) artifacts, (2) driver.Prestart() downloads image and creates container, (3) render templates, (4) start.

Some of this may be in the works already, but @schmichael wanted to capture it as an issue.

Members of the community have developed various solutions that may be of benefit to the project and will hopefully capture those concepts here.

[schmichael]

Thanks for writing this up @stevenscg! @dbresson suggested a solution that would allow arbitrary template sidecar tasks to work:

Allow Nomad to restart tasks when a file changes so that a sidecar could write the alloc/ and Nomad would signal other tasks to reload.

The Pro being it allows for a signalling mechanism within a task group and supports alternative templating engines.

The Con is that it's still more involved than just rendering a template out of an image.

There's no technical reason both solutions couldn't be implemented.

[blalor]

I think there's already an issue open for task dependencies; I think pulling the template out of the container image is an awkward model, but having another task that's responsible for doing setup for other tasks would solve this problem.

I personally use the rkt driver and embed the template source directly into the jobspec. I do find myself in template hell, however: does this template variable get replaced by Terraform, Nomad, or consul-template (embedded in Nomad)? I've been wondering if base64 encoding the template payload would be more or less messy…

[stevenscg]

I mentioned this later on in the Gitter discussion, but will capture it here as well...

I'm leaning right now toward one of the following while this discussion plays out:
a) Use the artifact stanza to fetch the template source files,
b) Embed consul-template within the container image,

The artifact stanza approach breaks my ideal scenario of having the single container as the build-time artifact, but it is what is supported right now and I see a number of folks using it.

I'm also concerned that if we fetch artifacts from the Github repo for each project that we will have to keep the job specification updated to reference the same commit as the container. This almost guarantees needing to template the job specification file on deploy as well, though we were considering that separate from this issue.

Embedding consul-template in the container is intriguing especially since the exec functionality has been available for several releases now. If I was going to run an init or process supervisor anyway, why not have it be consul-template. I think this pattern might also let me run with or without the container and with or without nomad (should the need arise).

[rokka-n]

Artifact works well for us, after we abandoned idea of templates managed externally to a container (with consul-template, scripts and so on)

If artifact is not enough - container can simply download internally whatever required from consul.

[stevenscg]

FYI, nomad can now fetch artifacts via Git and Mercurial.
See: #2386

[rokka-n]

I have a simple scenario: CI builds docker image, tags it with git hash or something, pushes to registry.

How nomad job can be composed automatically in this case to deploy new version of service?

[dadgar]

@rokka-n I would just have your CI use envsubst to change the docker image in the nomad job file and run that job.

[FlorinAsavoaie]

Having the templates in the containers would be the best design in my opinion.

As said in the first reply of this issue, technically this could be implemented relatively easy, with a slight change from what the colleague in the first reply said:

If both artifact downloading and template rendering is moved out of TaskRunner.prestart and ran after driver's PreStart (eg. in startTask() function), then it can have access to the image of the container, as the container was created (but not ran).

All that's needed after this would be to get the templates using the archive endpoint of docker. More info about it here. Basically meaning that the first step would be to implement a "docker-container" protocol in go-getter.

@dadgar , any idea if something like this could have any chance of being prioritized?

[FlorinAsavoaie]

Well, maybe the idea to implement the docker-container protocol in go-getter is not so good. However, we could add a stanza to the Docker driver that exports templates as CreatedResources during PreStart.