No way to run action with a modified version of default_shell_env

[Reflexe]

First of all, I started writing this issue as a bug since a recent commit broke my code but then I thought it could be also a feature request. If you think it is more of a FR than a bug, please tell me and I will edit it to match the correct format.

Description of the problem / feature request:

Since 0f5679e, ctx.configuration.default_shell_env would return an empty dictionary on Linux; Bazel no longer inherits environment variables from the compiling host in the analysis phase but only in execution time (according to @ulfjack 's comments on BazelConfiguration.java#135). However, that make it impossible to append environment variable to the default shell environment.

I can't think of a good way to solve this problem without a magic value that get changed on execution time (similar to cmake's generator expressions). Another way is to maybe transparently return a unique UncookedDefaultEnv from ctx.configuration.default_shell_env that will behave like a regular dictionary and would transparently forward null keys to the action, to be replaced in execution time.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

example.bzl:

def _impl(ctx):
    # Add 'TheAnswer' to the default shell environment.
    environment_vars = ctx.configuration.default_shell_env
    environment_vars['TheAnswer'] = '42'

    ctx.actions.run_shell(
        outputs = [ctx.outputs.dummy],
        command='env && touch dummy',
        env = environment_vars,
    )

example = rule(
    implementation=_impl, 
    outputs = {
        'dummy':'dummy'
    }
)

BUILD:

load(':example.bzl', 'example')
example(name='example')

Example output with bazel 0.11.1

...
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl
...

Example output for the current master:

PWD=/home/reflexe/.cache/bazel/_bazel_reflexe/95fc1ed1afdc0650f9a0db1c7a4b7c34/sandbox/9014715435859486979/execroot/__main__
TMPDIR=/tmp
TheAnswer=42
SHLVL=1
_=/usr/bin/env

Result: you should see that the output of env does not contain the default PATH.

What operating system are you running Bazel on?

Linux - ArchLinux

What's the output of bazel info release?

development version

If bazel info release returns "development version" or "(@Non-Git)", tell us how you built Bazel.

$ git clone github://bazelbuild/bazel
$ bazel build //src:bazel

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/bazelbuild/bazel
bffa2db380cb3ca2fd9262ac5a45d02518376e03
bffa2db380cb3ca2fd9262ac5a45d02518376e03

And I need to say that every time, thank you very much for this amazing thing called Bazel. It feels so right to use it; and the best and somehow the simplest build system I ever had to use. Just thanks :)

[jmmv]

@ulfjack I see you did the original change referenced in this bug and this is assigned to you. Any thoughts on what the right behavior should be here?

[ulfjack]

The problem is that we need to merge the environment set on the command-line with --action_env and --test_env with the environment specified by the Skylark rule in some way. Which of these sources should have precedence over the other is a critical question here. If we allow whatever, then rules will almost certainly end up inconsistent wrt. environment handling, which makes it a pain for users.

My personal preference is that the --action_env flag overrides any per-rule defaults, possibly with a more fine-grained action_env spec, e.g., based on mnemonics like --strategy. Maybe also allow rules to error out if the env is conflicting with rule-specific requirements. The built-in rules currently do their own thing, although most of them don't set any additional env variables, I think.

[ulfjack]

Note also that default_shell_env is a misnomer. We probably need to enforce that --action_env is taken into account during Skylark action creation, and not make it optional.

[Globegitter]

We have been facing similar issues and it would be great if it was possible to somehow merge env variables defined by the rule, as well as variables passed in via --action_env and also allow a more fine-grained --action_env spec.

Somewhat related: #7364 & #6011

[flaub]

This is the exact problem I'm running into as well. The ctx.configuration.default_shell_env is empty, but I need the host's default shell environment vars so that I can append my own for a particular rule.

[tillahoffmann]

This workaround doesn't properly inherit values, but it allows for setting defaults that can be copied from the host environment.

# .env.bazelrc: this file is like a .env file but in bazel CLI style.
build --action_env FOO=bar --action_env BAR=foo

# .bazelrc: the configuration simply imports the environment if available.
try-import %workspace%/.env.bazelrc

# example.bzl: example implementation showing that environment variables are available.
def _impl(ctx):
    print(ctx.configuration.default_shell_env)

example = rule(
    implementation=_impl,
)

# BUILD: example invocation.
load("//:example.bzl", "example")
example(name="example")

bazel build //:example
# DEBUG: /path/to/BUILD: {"FOO": "bar", "BAR": "foo"}

This is of course not as nice as being able to use the host environment directly, but copying the relevant environment variables to .env.bazelrc isn't too cumbersome.

[burdiyan]

Got hit by this too.

I'm using --incompatible_strict_action_env because normally I don't want any host env vars to be leaked into the build actions. But for one specific rule I want to inherit a variable from host. So I configure it as --action_env=MYVAR in the .bazelrc file. I was expecting to get it inside ctx.configuration.default_shell_env and then pass it manually to my custom build action, but it is empty. Although if I invoke Bazel and manually pass the actual value like --actions_env=MYVAR=$MYVAR" then it works. But it can't be configured within .bazelrc this way, so I have to use a wrapper script, which is annoying.

Can someone confirm if this is actually a bug or an expected behavior?

[jesses-canva]

Is this fixed by #19317?

[fmeum]

Is this fixed by #19317?

Only partially. You can now choose to inherit the --action_env via use_default_shell_env and still set environment variables via env, but it's not possible to modify the inherited values short of fully replacing them.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.