Runtime environment variables

[jromero]

Summary

As a developer, I'd like to be able to provide environment variables that would be used at runtime environment when executing a build.

Details

Currently, pack supports providing build environment variables as a method of configuring buildpacks. Runtime environment variables would enable scenarios where it would be prefered for environment variables to be stored within the container image. This would be analogous to the Dockerfile's ENV instruction but scoped to only applying at runtime.

Furthermore, it would be ideal if the provided environment variables could have "operations" as build-time env vars do now in certain contexts.

Functionality

End-User interactions

Some example extension points to provide this functionality to end-users are as follows:

1. project.toml
2. pack build
3. /platform/env.launch/

project.toml

# ...

[[run.env]] # → applicable to all process types
name="<KEY>"
value="<VALUE>"
operation="append|override|..." # optional, default=override

[[run.<process-type>.env]] # → applicable to only <process-type> (most specific wins)
name="<KEY>"
value="<VALUE>"
operation="append|override|..."

# ...

pack build

pack build ... --runtime-env <KEY>=<VALUE> --runtime-env-file <path/to/file>

NOTES:

• Pack would simply use the /platform/env.launch/ contract below to provide configuration to the lifecycle.
• Env vars provided via Pack CLI arguments would default to override operations for all process types.
  • For more advances usage, users can provide a project.toml file.

/platform/env.launch/

.
└── /platform/env.launch/ → applicable to all process types
    ├── KEY.override
    └── <process-type>/ → applicable to only <process-type> (most specific wins)
        ├── KEY1.override
        ├── KEY2.append
        └── ...

Lifecycle implementation

Configuration

• At build-time, the configuration should be read from <platform>/env.launch/ but stored in some other format (possibly in the <layers>/env.launch/ directory).
• At runtime, the configuration should be read from <platform>/env.launch/ and applied to the running environment.

Order of precedence

This is a proposed order of precedence (higher takes precedence):

1. Container Environment Variables (ie. docker run --env, podspec env vars)
2. Image Config Environment Variables
   • Ultimately applied as Container Environment Variables
3. Configuration from <platform>/env.launch/ (provided at runtime)
4. Configuration from <layers>/env.launch/ (persisted from build-time)
5. Configuration from <layers>/<layer>/env.launch/ (provided by buildpacks from build-time)

Alternatives

• Buildpacks can already be used to provide this functionality (see Paketo's buildpack)
  • There are multiple downsides to this current approach:
    1. The buildpack must be compatible with a multitude of stacks or each stack author needs to provide a buildpack to enable this functionality.
    2. The interface for adding the env vars aren't as nice (dependant on buildpack implementation) or discoverable.

Additional Thoughts

1. BREAKING: We should align env directories with Buildpack API contract
   • <platform>/env - environment variables for both runtime and build-time (currently this is defined as build-time only)
   • <platform>/env - remove use to prevent unintentionally leaking built-time only env vars
   • <platform>/env.build - build-time environment variables
   • <platform>/env.launch - runtime environment variables for all process types (this proposal)
   • <platform>/env.launch/<process-type>/ - runtime environment variables for a specific process type (this proposal)

References

• Related: Environment variables not accessible at Runtime pack#914 (comment)
• Slack conversation (join): https://buildpacks.slack.com/archives/CCP60GJLS/p1613076546013700

[jromero]

Sounds like there's some desire to get this done. This would need to be updated to project descriptor 0.2 and make an RFC.

I've created an issue (buildpacks/rfcs#164) to track this work. Open to contributions. :)

[sclevine]

@jromero I agree that runtime env vars in project.toml would be a useful feature.

Why would we expose them in /platform? That directory provides read-only configuration to buildpacks, and buildpacks don't need access to runtime env vars.

[jromero]

Think of use cases where project.toml isn't in use. For example, a platform such as pack can provide additional configuration pack build --run-env MY_VAR=adf .... The platform in this case needs an interface for providing runtime environment variables similar to how build time environment variables can be provided via /platform/env.