Failure to launch process with a materialized arg0 on rust 1.49.0 and OSX

[stuhood]

On rust 1.49.0:

Exception: Failed to execute: Process {
    argv: [
        "./find_binary.sh",
        "bash",
    ],
    env: {
        "PATH": "/usr/bin:/bin:/usr/local/bin",
    },
    working_directory: None,
    input_files: Digest(
        Fingerprint<124495cfdd1160d88a797da1b5de65f0b9d57642b553d7ff5195d5375411b492>,
        91,
    ),
    output_files: {},
    output_directories: {},
    timeout: None,
    execution_slot_variable: None,
    description: "Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin",
    level: Debug,
    append_only_caches: {},
    jdk_home: None,
    platform_constraint: None,
    is_nailgunnable: false,
    cache_scope: PerRestart,
}

Error launching process: Os { code: 2, kind: NotFound, message: "No such file or directory" }

[jsirois]

How do you know? It could be /usr/bin/env is not found:

$ cat script.sh 
#!/usr/bin/env bash2

^jsirois@gill ~/dev/pantsbuild/pants (master) $ ./script.sh 
/usr/bin/env: ‘bash2’: No such file or directory

[stuhood]

After the run, with --no-process-execution-cleanup-local-dirs, the following works:

$ /private/var/folders/bg/_r10hqp14kjcpv68yzdk5svc0000gn/T/process-executionvrX4qr/find_binary.sh bash
+ command -v which
+ command which -a bash
+ which -a bash
/bin/bash

[tdyas]

Also I see that working_directory is None. Shouldn't that be set to the directory where find_binary.sh was written?

[tdyas]

(Or is that a relative working directory for the Process request?)

[stuhood]

(Or is that a relative working directory for the Process request?)

Yea, it's for the request. The process itself is always launched inside the sandbox, and the extra working directory is a subdirectory of the sandbox.

[stuhood]

Poked this for a few minutes over the weekend: at the time of spawning, the executable does exist on disk. Next might be looking into any changes to the Process/Command APIs (noting that we use tokio's wrappers) that might be causing issues with relative files, or looking into whether the error is related to the kernel disliking the shebang contents somehow (perhaps due to changed handling of env).

[Eric-Arellano]

This will need to be fixed to unblock building Pants with Apple Silicon machines. Co-assigning to me.

[Eric-Arellano]

or looking into whether the error is related to the kernel disliking the shebang contents somehow (perhaps due to changed handling of env).

How come you think that might be a possibility? This seems to pretty confidently be a regression between Rust 1.48 vs. 1.49. Using the exact same environment, e.g. same OS, this code works with 1.48 but breaks on 1.49. The file also seems to be the same contents between both versions, per --no-process-execution-cleanup-local-dirs.

That is, Rust seems to have changed something, but I don't think something like how the shebang is interpreter by the kernel would be in play.

[tdyas]

rust-lang/rust#80819 seems to be this issue with a work-around fix in rust-lang/rust#80537. Basically a bug on macOS with a certain syscall used by the Rust std library.

[Eric-Arellano]

To clarify, the workaround is to absolutify the path, right?

[tdyas]

To clarify, the workaround is to absolutify the path, right?

Seems like that. Which you could do by not relying on the shebang but instead run /bin/bash ./find_binary.sh as the program.

[Eric-Arellano]

Which you could do by not relying on the shebang but instead run /bin/bash ./find_binary.sh as the program.

That would fix ./find_binary.sh, but then we get errors in other places like calling ./scc or ./pytest.pex. We use relpaths in lots of places - I think the fix needs to be more permanent in Rust-land to ~ os.path.join(cwd, rel_path).

[tdyas]

That would fix ./find_binary.sh, but then we get errors in other places like calling ./scc or ./pytest.pex. We use relpaths in lots of places - I think the fix needs to be more permanent in Rust-land to ~ os.path.join(cwd, rel_path).

Which will work in local execution, but not in remote execution (since you currently cannot rely on a remote execution server to normalize paths). So your proposal should fix the issue locally in the short term. Long term I've thought about proposing a change to REAPI to do that, but that would require much thought. Although given REAPI servers only run on Linux currently, maybe it is fine to just keep the path relative on every platform other than macOS?