S2i injection feature is broken on stock Ubuntu 20.04 LTS systems

[jperville]

Is this a feature request or bug?

/kind bug

What went wrong?

On Ubuntu 20.04 system, I cannot build a docker image which uses the injection feature.
It used to work on Ubuntu < 20.04 and still works on CentOS 7.x systems.

This bug has already been reported as #1006 but was closed for lifecycle/rotten.
This bug report should have all the information to reproduce, workaround and fix the issue.

Steps to reproduce:

Run this on fresh Ubuntu 20.04 system with docker installed (sudo apt-get install docker.io).
Attention: do not tweak the default value of the fs.protected_regular sysctl, which should be 1.

1. docker pull quay.io/centos7/ruby-27-centos7
2. echo hello > file.txt
3. s2i build https://github.com/openshift/ruby-hello-world.git quay.io/centos7/ruby-27-centos7 my-hello-world:xxx --inject $PWD/file.txt:/bug.txt (fails)
4. echo $? (displays 1)

Expected results:

I expect the image to build successfully.

Actual results:

The image fails to build, with the following error message at the end:

---> Cleaning up unused ruby gems ...
Running `bundle clean --verbose` with bundler 2.1.4
Frozen, using resolution from the lockfile
truncate: cannot open '/tmp/rm-injections' for writing: Permission denied
Build failed
ERROR: An error occurred: non-zero (13) exit code from quay.io/centos7/ruby-27-centos7

Version:

s2i: s2i v1.3.1
docker:

Client:
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.2-0ubuntu1~20.04.2
 Built:             Tue Mar 30 21:24:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.2-0ubuntu1~20.04.2
  Built:            Mon Mar 29 19:10:09 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.3
  GitCommit:        
 runc:
  Version:          spec: 1.0.2-dev
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

Additional info:

The bug happens on stock Ubuntu 20.04 LTS system.
It does not happen on Centos 7.x or on Ubuntu 14.04 systems that I have access to.

The reason is that on Ubuntu 20.04 LTS sudo sysctl fs.protected_regular=1 is the default,
which (according to https://unix.stackexchange.com/questions/503111/group-permissions-for-root-not-working-in-tmp ) restricts the permissions under /tmp and in particular prevents s2i from truncating the /tmp/rm-injections file inside the container despite that file being writable by everyone (chmod 0666).

At the end of the s2i assemble phase, s2i will try to truncate the /tmp/rm-injections file, which has mode 0666 (writable by everyone). This file is owned by user 1000:1000 while the s2i assemble user has uid 1001:1001. The assemble user is then denied the right to truncate the /tmp/rm-injections file, despite that file being world-writable.

Here is to reproduce the problem with a simple busybox container:

sudo sysctl fs.protected_regular=1
docker run -d --name=permbug -u 1001:1001 --entrypoint=/bin/sh busybox:latest -c 'sleep 10000'
docker exec -u 0 permbug /bin/sh -c 'echo blah > /tmp/xxx ; chown 1000:1000 /tmp/xxx ; chmod 666 /tmp/xxx'
docker exec -u 1001:1001 permbug /bin/sh -c 'truncate -s0 /tmp/xxx && echo "WORKS" || echo "BUG"'
docker rm -f permbug

We see that if the run the same command creating the xxx file in /var instead of /tmp it works:

sudo sysctl fs.protected_regular=1
docker run -d --name=permbug -u 1001:1001 --entrypoint=/bin/sh busybox:latest -c 'sleep 10000'
docker exec -u 0 permbug /bin/sh -c 'echo blah > /var/xxx ; chown 1000:1000 /var/xxx ; chmod 666 /var/xxx'
docker exec -u 1001:1001 permbug /bin/sh -c 'truncate -s0 /var/xxx && echo "WORKS" || echo "BUG"'
docker rm -f permbug

If we force sudo sysctl fs.protected_regular=0 in the first example it works again:

sudo sysctl fs.protected_regular=0
docker run -d --name=permbug -u 1001:1001 --entrypoint=/bin/sh busybox:latest -c 'sleep 10000'
docker exec -u 0 permbug /bin/sh -c 'echo blah > /tmp/xxx ; chown 1000:1000 /tmp/xxx ; chmod 666 /tmp/xxx'
docker exec -u 1001:1001 permbug /bin/sh -c 'truncate -s0 /tmp/xxx && echo "WORKS" || echo "BUG"'
docker rm -f permbug

To workaround this issue I had to:

• sudo sysctl fs.protected_regular=0 (in a /etc/sysctl.d file) (best workaround)
• hack the s2i binary with sed -i -e s:tmp/rm-injections:var/rm-injections:g (giant hack)

To fix the issue properly, I suggest either:

• add a command-line option to allow creating the /tmp/rm-injections file outside of /tmp
• hardcode the value of rmInjectionsScript in

  source-to-image/pkg/build/strategies/sti/sti.go

  Line 35 in 30d81a9

  rmInjectionsScript = "/tmp/rm-injections"

  to a path that is not tmp-like, for example to /var/rm-injections.
• add a FAQ entry in the project README.md advising to create a /etc/sysctl.d entry with fs.protected_regular=0 on Ubuntu 20.04 and more recent systems.

[adambkaplan]

• hardcode the value of rmInjectionsScript in

  source-to-image/pkg/build/strategies/sti/sti.go

  Line 35 in 30d81a9

  rmInjectionsScript = "/tmp/rm-injections"

  
  to a path that is not tmp-like, for example to /var/rm-injections.

This is the most viable solution.

Given that this is a bug that happens on Ubuntu, we should probably wire up a Github action to test this (our tests currently run in RHEL/UBI8 containers iirc).

[adambkaplan]

PS thank you @jperville for your thoroughly detailed bug report!

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[jperville]

/lifecycle frozen

[ricoms]

This bug is happening to me today.

» s2i version 
s2i v1.3.9

My builder Dockerfile:

FROM python:3.11-slim as base

ARG S2IDIR="/s2i"
ARG APPDIR="/opt/program"

LABEL io.openshift.s2i.scripts-url="image://$S2IDIR/bin" \
    io.openshift.s2i.destination="/tmp"
ENV PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    POETRY_PATH=/opt/poetry \
    VENV_PATH=/opt/venv \
    POETRY_VERSION=1.5.0 \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1
ENV PATH="$POETRY_PATH/bin:$VENV_PATH/bin:$APPDIR:$PATH"
WORKDIR $APPDIR

FROM base as builder
RUN python3 -m pip install "poetry==$POETRY_VERSION" \
    && python3 -m venv $VENV_PATH
COPY poetry.lock pyproject.toml $APPDIR
RUN poetry export --without-hashes -o /tmp/requirements.txt \
    && $VENV_PATH/bin/pip install -r /tmp/requirements.txt
COPY . .
RUN poetry build && $VENV_PATH/bin/pip install dist/*.whl

FROM base as final
COPY .s2i $S2IDIR
COPY --from=builder $VENV_PATH $VENV_PATH

the error happens when I'm trying to run s2i build within GitHub Actions, although it does not happen when running s2i build command locally (same machine I've built the builder image.

error message is the same as OP:

truncate: cannot open '/tmp/rm-injections' for writing: Permission denied
Build failed

Sorry, but why @coreydaley's fix from October last year was not merged?

I'm still trying to figure out solutions to this issue, I tried:

1. defining USER 0 and io.openshift.s2i.assemble-user="0".
2. chmod /tmp folder as suggested here
3. changing io.openshift.s2i.destination to other folders.
4. I'm not being able to use the workarounds OP suggested, like sudo sysctl fs.protected_regular=0 on my docker image.

[ricoms]

A successful workaround for me was to include the following line by the end of my s2i assemble script:

sed -i '/rm-injections/d' /tmp/rm-injections

This removes inplace the line truncate -s0 "/tmp/rm-injections" from the /tmp/rm-injections.