-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[potential-bug] raw_exec artifact not set as executable #4809
Comments
@jrasell from the following reproduction steps it looks like go-getter does not make the file executable. Nomad does not directly change file permissions in the artifact stanza.
|
@schmichael pointed out that our executor should mark the file as executable if its not already ( nomad/client/driver/executor/executor.go Line 284 in e7f6419
|
@preetapan thanks for the quick responses; i'll run it again tomorrow and capture any relevant logs from the client to add here. |
I was able to reproduce this with the following job spec file, adapted from https://github.com/hashicorp/nomad/blob/community-call-0/community/specs/hello.nomad but using
|
After playing around with the above spec file some more, got it to work after changing the command to the following change:
That's the relative path to the task directory, use that instead of ${NOMAD_TASK_DIR} and it works. If that doesn't work for you, please re-open this. |
We can make a fully interpolated path like `${NOMAD_TASK_DIR}/binary' work with a small patch to nomad/client/driver/executor/executor.go Line 506 in e7f6419
|
Follow up to #4809, so we only return bin path if it's inside task dir; otherwise, use old mechanism. We should aim priortize bin path inside task dir, otherwise, we'd be marking the wrong file as executable in later steps.
This is a follow up work to #4813 to fix #4809 and fix a regression introduced in 0.9 in marking files in libcontainer executable. #4809 bug is that `lookupBin` uses `exec.LookPath` when not inspecting task dir files. `exec.LookPath` only returns a file if it's already marked as an executable path in https://github.com/golang/go/blob/go1.12.1/src/os/exec/lp_unix.go#L24-L27 . This affects raw exec as if passed an absolute path to file, `lookupBin` returns an error if file isn't already an executable. This explains why the error manifests when an absolute interpolated path is used (e.g. `${NOMAD_TASK_DIR}/hellov1`) but not when using a task rel dir (e.g. `local/hellov1`) in the above examples used in ticket. PR #4813 remedied this problem for raw exec but inadvertably broke libcontainer executor, as it made `lookupBin` returns the paths to host files rather than ones found inside chroot. This PR reorders the evaluation, so we go back to 0.8 behavior of looking up task directories first, but then check for host paths before using `exec.LookPath`. This PR is broken into three commits to illustrate evolution and confirming hypothesis: 1. 9adab75 : Adding a test illustrating how libcontainer executor fails at marking processes as executable in https://travis-ci.org/hashicorp/nomad/jobs/514942694 - note that the test doesn't depend on artifacts or interpolated paths 2. d441cdd: reverting PR #4809 and showing the test fail now with raw_exec case (as expected) in https://travis-ci.org/hashicorp/nomad/jobs/514944065 2. 244544b: in where we add the check in appropriate place next to `exec.LookPath(...)` for absolute paths and have a green job in https://travis-ci.org/hashicorp/nomad/jobs/514945024 ## Future work Inspecting `lookupBin` in 0.8 and 0.9 case, we have a bug in using `exec.LookPath` for the libcontainer executor case. We should be looking up paths based on the container chroot and container PATH rather than the host's. However, this is not a 0.9.0 regression and was present in 0.8; so punting to fix it post 0.9.
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.8.3 (c85483da3471f4bd3a7c3de112e95f551071769f)
Operating system and Environment details
Issue
When using the raw_exec driver and downloading an artifact binary, the binary is not getting set as executable.
The file permissions:
The job error message:
As far as I am aware when using the raw_exec driver and downloading a binary, nomad/go-getter should be setting the file as executable.
Job file (if appropriate)
The text was updated successfully, but these errors were encountered: