Ability to run the command directly from a mounted hosted volume

[wojciak]

Proposal

I'd like to use an fsx mount as a hosted volume for my nomad cluster, the issue is that I cannot run any apps from it, because the volume is not mounted by the time of the app execution.

The proposal is to make sure (or add a lifecycle toggle) that when the config.command stanza is run, all the hosted volume mounts are resolved.

Use-cases

This is very useful for hotfix deployment and any ci/cd pipeline

Attempted Solutions

Prestart lifecycle hook with a different task, poststart lifecycle hook - as all tasks have their own allocations this still won't make the volume available when the config.command stanza is run

[wojciak]

Ok I did some digging and it is possible with some workarounds see:

• https://discuss.hashicorp.com/t/running-a-command-from-a-mounted-hosted-volume/33927/2
• Support running executables from mounted paths in exec driver (like docker driver) #8410

Would be cool to have this behavior build-in somehow (like disable_checker = true on the exec driver)

[DerekStrickland]

Hi @wojciak

Thanks for all your work and thoughts on this. We'll review this suggestion, and think about feasibility, etc. From there we'll update this issue with next steps.

Derek

[tgross]

Hi @wojciak effectively you're running into the exact same issue I commented on here #11690 (comment):

you're running into a consequence of the order in which the filesystem for a task is built. If you take a look at the filesystem docs you'll see that the task gets started with the volume mount in a single step at the end. That happens in the task driver by calling out to libcontainer. So you can't use binaries from a volume mount for the exec task driver.

As a workaround, you might be able to get away with using /bin/sh as the command and have it exec into your binary, because it won't be started until after the volume is mounted. Something like:

commands = "/bin/sh"
args = ["-c", "exec /foo/var"]

The source of this issue is arising from libcontainer and not something we have control over in Nomad. It might be possible to do in some future task drivers not based on libcontainer but right now that's virtually all of them. I'm going to keep this issue open but I want to be honest with you that I would not expect to be implemented anytime soon.

Give the workaround I've proposed here as try for your use case.

[tgross]

PR up for this in #14851. As it turns out, it's not a libcontainer limitation but something we were doing around setting binaries as executable.

[tgross]

#14851 has been merged and will ship in the next regular release of Nomad (likely 1.4.3)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.