- Name: Build/Run image configs MUST contain env.PATH
- Start Date: 2020-08-21
- Author(s): @aemengo @ameyer-pivotal @mvalliath @TisVictress @micahyoung
- Status: Implemented
- RFC Pull Request: rfcs#114
- CNB Pull Request: (leave blank)
- CNB Issue: buildpacks/spec#150, buildpacks/docs#223
- Supersedes: N/A
Windows build & run stack images need to set a PATH
environment variable in the image config to avoid Docker's alternative mechanism for env var PATH
-loading from the Windows registry, which typically contains the value of PATH
for Windows images. Relying on the Windows registry creates inconsistency between Windows and Linux CNB mechanisms which causes Windows app images to be unrunnable in some circumstances lifecycle#384.
I propose we change the spec to codify the current de-facto Linux image behavior -- to require a complete config.Env.PATH
to be set on all build & run stack images, for Windows or Linux. When a config.Env.PATH
is set on a Windows image, they will always behave like Linux images and avoid triggering Docker's Windows registry-based loading of the PATH
environment variable. Linux stack images get additional validation but generally will never need to change because they already have a PATH
set from the base image.
See spec-changes for details.
- Windows registry: Very common key-value store for Windows OS, used to store environment variable values among other things
- Windows registry hives delta: Files stored in each OCI image layer, containing Windows registry values that are aggregated and exposed to processes in a running container. The format is not documented by Docker but is bcd-formatted.
-
Why should we do this?
- Fix lifecycle#384
- Make Windows and Linux images load CNB-required env var data consistently from the same place:
config.Env
- Using two sources of data for Windows
PATH
env var would be confusing - Reading Windows registry-
PATH
directly from the image would require parsing and aggregating Windows registryHives/*_delta
files from layers, which are undocumented nor guaranteed stable - Still allows other env var values to come either
config.Env
or from the Windows registry - Windows stack authors can set this once, with low effort
- Validation tooling can expose the correct value from the runtime environment
- Better bug reports, since Windows registry
PATH
values are hard to access - No effort for Linux stack authors, and no backward compatibility concerns as all existing images must have had a
config.Env.PATH
set to function.
-
What use cases does it support?
- Stack authors using their build and/or run images with a platform (i.e.
pack create-builder
)
- Stack authors using their build and/or run images with a platform (i.e.
-
What is the expected outcome?
- Any Windows or Linux build run images missing a
config.Env.PATH
which would cause the platform to fail with error message.
- Any Windows or Linux build run images missing a
This provides a high level overview of the feature.
-
Define any new terminology.
- N/A, similar image env vars are already spec'd in platforms
-
Define the target persona: buildpack author, buildpack user, platform operator, platform implementor, and/or project contributor.
- Stack Authors
-
Explaining the feature largely in terms of examples.
- As a Stack Author making a stack image from scratch:
- Create a Windows stack image from a new
Dockerfile
without aPATH
- Attempt to use the stack image with their platform (i.e.
pack create-builder
) - See a failure
- Add a
ENV PATH C:\\Windows\\System32;C:\\stack-specific-dir
- Re-attempt to use the image
- See no failure
- Create a Windows stack image from a new
- As a Stack Author making a stack image from samples:
- Copy existing sample stack (which already contains
ENV PATH C:\\Windows\\System32
) - Attempt to use the stack image with their platform (i.e.
pack create-builder
) - See no failure
- Add stack-image specific directory to
PATH
:ENV PATH C:\\Windows\\System32;C:\\stack-specific-dir
- Attempt to use the stack image with their platform (i.e.
pack create-builder
) - See no failure
- Copy existing sample stack (which already contains
- As a Stack Author making a stack image from scratch:
-
If applicable, provide sample error messages, deprecation warnings, or migration guidance.
- Sample error message (based on existng
pack
error message):
ERROR: failed to create builder: invalid build-image: adding image labels to builder: image cnbs/sample-stack-build:nanoserver-1809 missing required env var PATH
- Sample error message (based on existng
-
If applicable, describe the differences between teaching this to existing users and new users.
- This is best taught the same way, along-side other required platform env vars.
The spec changes are minimal: change platform.md
to always require run images and build images to have a config.Env
entry for PATH
. It is also valid to set to empty PATH=
for stack images that do not want to use a PATH
. If config.Env.PATH
were unset, a platform would fail.
This change relies on the current behavior of Docker and containerd for Windows which is to set each container runtime env var from only one of two places in the image: preferring config.Env
variables that are set, then falling back to values set in the Windows registry hive layer data. Docker's behavior is intended to align with Windows conventions of apps setting all environment variables through the Windows registry but still allowing for config.Env
to override them.
Windows base images de-facto always leave the config.Env.PATH
unset so that Docker will load the env var PATH
(an any other unset env vars) from the Windows registry.
$ crane config mcr.microsoft.com/windows/nanoserver:1809-amd64 | jq '.config.Env'
null
$ docker run mcr.microsoft.com/windows/nanoserver:1809-amd64 cmd /c echo %PATH%
C:\Windows\system32;C:\Windows; # From the registry hives value
On the other hand, Linux base images - and build & run stack images by inheritance - de-facto always set config.Env.PATH
to /usr/bin:/bin:<etc≥
because there is no alternative mechanism for setting the initial env var PATH
.
$ crane config alpine | jq '.config.Env'
[
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
$ crane config cnbs/sample-stack-run:alpine | jq '.config.Env'
[
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"CNB_USER_ID=1000",
"CNB_GROUP_ID=1001"
]
$ docker run alpine sh -c 'echo $PATH'
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
$ docker run cnbs/sample-stack-run:alpine sh -c 'echo $PATH'
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
The difference in Windows and Linux base image conventions pass-through to the respective run images which results in Windows apps images with incomplete config.Env.PATH
, containing only lifecycle CNB paths (i.e. c:\cnb\process
). Docker never attempts to load the Windows registry PATH
value since it prefers the config.Env.PATH
.
To fix this, platforms would ensure that config.Env.PATH
is set on a build or run image configs, just like they ensure config.Labels
and other config.Env
values are set(ex CNB_USER_ID
).
Stack authors would set config.Env.PATH
to the complete value they need for the image (perhaps just the shell's PATH
), either by adding ENV PATH=...
to their stack image Dockerfiles or directly setting in the image config with the equivalent of config.Env += ["PATH=..."]
for their build and run images to pass validation. They would choose an empty path with config.Env = ["PATH="]
.
In practice (not spec'd), the config.Env.PATH
will typically contain, at minimum, a path to the directory of the shell. Also, if unset, the platform could fail with a helpful error message, perhaps how to set a config.Env.PATH
and perhaps how to find a recommended value: docker run <base image> cmd /c echo %PATH%
or something similar.
The result of all this is that once a config.Env.PATH
is set, Docker Windows (and containerd) will never attempt to load the PATH
value from the registry, even with config.Env.PATH=
. This ensures consistent behavior in lifecycle and app image containers, for Windows and Linux -- that the runtime value for PATH
is always config.Env.PATH
, and if a command is needed, a Stack Author can add it to the config.Env.PATH
.
- Stack Authors may be surprised that stack images require
config.Env.PATH
, whereas typical Windows base images do not.- This can be mitigated with appropriate platform error messaging when it is unset.
- Stack Authors will not be able to use the Windows-specific workaround syntax for Dockerfile
PATH
augmentation (i.e.setx PATH %PATH%;c:\mystuff
). They'll always need to provide the full value.- I feel this is not a major concern, as Docker's workarounds are not documented and are problematic in the first place. Docker itself only documents setting the entire env vars through
ENV PATH=...
.
- I feel this is not a major concern, as Docker's workarounds are not documented and are problematic in the first place. Docker itself only documents setting the entire env vars through
- Stack Authors will need to correctly duplicate their Windows registry value of
PATH
, which may lead to accidental mis-matches.- No good option for mitigation but some possibilities considered:
- Change lifecycle or launcher to compare the running container's
PATH
with the Windows registry value. A warning message could be given to the user, but it could not be fatal since only a Stack Author could not fix the issue. - Change platforms to attempt to read an image's
Hives/*_Delta
, though the format is undocumented and potentially unstable and the value would need to be aggregated from every layer and hive delta file. - Change platforms to create a one-off container that would return Docker's runtime value of
PATH
.
- Change lifecycle or launcher to compare the running container's
- No good option for mitigation but some possibilities considered:
- What other designs have been considered?
- An alternative approach I tried was to change lifecycle and launcher to always merge the runtime env var
PATH
(fromconfig.Env.PATH
) and the Windows registryPATH
values before running subprocesses but it was somewhat complex, led to very platform-specific differences, and didn't handle thePATH
forENTRYPOINT
s (beforelifecycle
was run). - Reading a the
PATH
directly from an image'sHives/*_Delta
files, whose format is undocumented and potentially unstable, is not feasible. If the value could retrieved, it would still need to be aggregated across all hives and layers.
- An alternative approach I tried was to change lifecycle and launcher to always merge the runtime env var
- Why is this proposal the best?
- It unifies Linux and Windows env var behavior while still allowing Windows env vars to come from either an image's
config.Env
or the Windows registry. - It allows for Docker to come up with a better Dockerfile solution to augment
config.Env
vars, similar to Linux. - It imposes only a one-time, minor burden on Stack Authors.
- It unifies Linux and Windows env var behavior while still allowing Windows env vars to come from either an image's
- What is the impact of not doing this?
- Lifecycle will continue to generate unrunnable app images.
Docker's abandoned attempt at Dockerfile-based PATH
augmentaton:
- Initial PR: moby/moby#29048
- End of discussion: moby/buildkit#74
Another abandoned Dockerfile ENV
approach: ENV --lazy-expand
- What parts of the design do you expect to be resolved before this gets merged?
- Should the spec require
PATH
to contain at least the shell?
- Should the spec require
- What parts of the design do you expect to be resolved through implementation of the feature?
- Implementers would provide helpful error messages to end users, much like other spec'd env vars
- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC?
- Changing lifecycle on Windows to validate the runtime values of env vars contain the Windows registry values (when Stack Authors mis-matched the
confiv.Env.PATH
values and the Windows registryPATH
value)
- Changing lifecycle on Windows to validate the runtime values of env vars contain the Windows registry values (when Stack Authors mis-matched the
Proposed spec diff: https://github.com/buildpacks/spec/compare/main...micahyoung:spec-stack-images-config-env-path-must-exist