Caching problem with multistage build

[mhubig]

Actual behavior
I have a problem with a docker image containing a python cli script. While tracking down the problem I found that kaniko inappropriately uses a cached version of the RUN command which is responsible for installing the cli tool which is build in a prior stage of the multistage Dockerfile.

FROM python:3.9-slim as base

ENV PYTHONFAULTHANDLER=1 \
  PYTHONHASHSEED=random \
  PYTHONUNBUFFERED=1

RUN set -ex; \
  apt-get update -y; \
  apt-get install -y --no-install-recommends gnupg curl parallel; \
  \
  echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list ;\
  curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - ;\
  apt-get update -y; \
  apt-get install -y google-cloud-sdk; \
  \
  apt-get clean; \
  rm -rf /var/lib/apt/lists;

WORKDIR /app

FROM base as builder

ENV PIP_DEFAULT_TIMEOUT=100 \
  PIP_DISABLE_PIP_VERSION_CHECK=1 \
  PIP_NO_CACHE_DIR=1 \
  POETRY_VERSION=1.1.7

COPY pyproject.toml poetry.lock ./
RUN set -ex; \
  pip install "poetry==$POETRY_VERSION"; \
  poetry config virtualenvs.create false; \
  poetry install --no-interaction --no-ansi

COPY . .
RUN poetry build

FROM base as final

COPY --from=builder /app/dist/*.whl ./
RUN pip install *.whl

The last line of this Dockerfile ist cached by kaniko, so in the resulting image I always have the old cli tool version. Here ist the gitlab-ci job which I use to build the image with kaniko (version gcr.io/kaniko-project/executor:v1.6.0-debug):

Build Image:
  stage: build
  image: registry.gitlab.dm-drogeriemarkt.com/mythos/docker/kaniko:latest
  script:
    - /kaniko/executor
      --cache=true
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}"

Here is a snippet of the kaniko log output which also states that a cached version is used:

INFO[0051] Pushed image to 1 destinations               
INFO[0051] Saving file app/dist/cleanup-0.1.0-py3-none-any.whl for later use 
INFO[0051] Deleting filesystem...                       
INFO[0053] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0053] Executing 0 build triggers                   
INFO[0054] Checking for cached layer registry.gitlab.site.com/mythos/hybris/hybris-backup/cache:49cdb016213ce38e45199ab5f02246092fabe2659381c707c26ac923de44253f... 
INFO[0054] Using caching version of cmd: RUN pip install *.whl 
INFO[0054] Unpacking rootfs as cmd COPY --from=builder /app/dist/*.whl ./ requires it. 
INFO[0070] COPY --from=builder /app/dist/*.whl ./       
INFO[0070] Resolving srcs [/app/dist/*.whl]...          
INFO[0070] Taking snapshot of files...                  
INFO[0070] RUN pip install *.whl                        
INFO[0070] Found cached layer, extracting to filesystem 
INFO[0071] Pushing image to registry.gitlab.site.com/mythos/hybris/hybris-backup:5fe242f8b8a38d2ebcb82b4052005b9a972401d2 
108INFO[0072] Pushed image to 1 destinations 

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[cbartz]

Possibly related to #589 (comment) , try with kaniko v1.3.0 as it appears that a regression has happened.

[montyben]

I was having a similar issue building docker images with Gitlab CI. Downgrading kaniko to v1.3.0 solved the problem.

[hoang-innomize]

I also faced this issue and downgraded to v1.3.0 worked.

[gilbsgilbs]

FYI, I opened #1735 which should fix this issue if anybody wants to give a look.