pack build fails to export image if extension run image not previously pulled

[cormacpayne]

Description

When using pack build to create an application image using a builder containing various platform buildpacks and their corresponding build and run extensions, it seems that if the new run image base (from the run extension) wasn't previously pulled down on the machine, then the application image cannot be exported.

Below are the verbose logs from the export phase where the new run image base has not been pulled down onto the machine:

===> EXPORTING
Running the 'exporter' on OS 'linux' with:
Container Settings:
Args: '/cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -log-level debug -app /workspace -cache-dir /cache -stack /layers/stack.toml <registry>/cbl-mariner-aca:run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615132919112539'
System Envs: 'CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.10'
Image: 'buildpacksio/lifecycle:0.15.2'
User: 'root'
Labels: 'map[author:pack]'
Host Settings:
Binds: 'pack-cache-cbl-mariner-aca_run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615132919112539-f07c32078b84.build:/cache /var/run/docker.sock:/var/run/docker.sock pack-cache-cbl-mariner-aca_run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615132919112539-f07c32078b84.launch:/launch-cache pack-layers-jjteiimkuq:/layers pack-app-vfbuhpafuu:/workspace'
Network Mode: ''
[exporter] Using runImage from analyzed metadata 'mcr.microsoft.com/dotnet/aspnet:7.0.5-cbl-mariner2.0'
[exporter] ERROR: failed to export: get run image top layer SHA: image "<registry>/cbl-mariner-aca:run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615132919112539" has no layers

If I then use docker pull to pull down the new run image specified in the extension, the application image is exported successfully:

===> EXPORTING
Running the 'exporter' on OS 'linux' with:
Container Settings:
Args: '/cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -log-level debug -app /workspace -cache-dir /cache -stack /layers/stack.toml <registry>/cbl-mariner-aca:run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615133552173374'
System Envs: 'CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.10'
Image: 'buildpacksio/lifecycle:0.15.2'
User: 'root'
Labels: 'map[author:pack]'
Host Settings:
Binds: 'pack-cache-cbl-mariner-aca_run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615133552173374-0a165b2ebdc2.build:/cache /var/run/docker.sock:/var/run/docker.sock pack-cache-cbl-mariner-aca_run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615133552173374-0a165b2ebdc2.launch:/launch-cache pack-layers-lpvxlnglpr:/layers pack-app-iexgxlzvee:/workspace'
Network Mode: ''
[exporter] Using runImage from analyzed metadata 'mcr.microsoft.com/dotnet/aspnet:7.0.5-cbl-mariner2.0'
[exporter] Reading buildpack directory: /layers/capps_dotnet-7.0
[exporter] Reading buildpack directory item: launch.toml
[exporter] Reading buildpack directory item: run-script
[exporter] Reading buildpack directory item: run-script.toml
[exporter] Processing buildpack directory: /layers/capps_dotnet-7.0
[exporter] Processing launch layer: /layers/capps_dotnet-7.0/run-script
[exporter] Reusing tarball for layer "capps/dotnet-7.0:run-script" with SHA: sha256:26093b5f94f9c6a4314497485f643f2bb72363d32991d9b3176be5cc3887a2de
[exporter] Adding layer 'capps/dotnet-7.0:run-script'
[exporter] Layer 'capps/dotnet-7.0:run-script' SHA: sha256:26093b5f94f9c6a4314497485f643f2bb72363d32991d9b3176be5cc3887a2de
[exporter] Found SBOM of type launch for at /layers/sbom/launch
[exporter] Reusing tarball for layer "launch.sbom" with SHA: sha256:36f9de0748c420980a265697f38f76b0f9ae9b7862d3467b76b638706e637425
[exporter] Adding layer 'launch.sbom'
[exporter] Layer 'launch.sbom' SHA: sha256:36f9de0748c420980a265697f38f76b0f9ae9b7862d3467b76b638706e637425
[exporter] Layer 'slice-1' SHA: sha256:0babcc029d5f7df191832a5db4c2eb34548eabdd058a9ac31b378117331d16d2
[exporter] Adding 1/1 app layer(s)
[exporter] Reusing tarball for layer "launcher" with SHA: sha256:6c749df2cfd63eaf8d564c60026df199deccb92225c279b7e858e7cae4dfbe14
[exporter] Adding layer 'launcher'
[exporter] Layer 'launcher' SHA: sha256:6c749df2cfd63eaf8d564c60026df199deccb92225c279b7e858e7cae4dfbe14
[exporter] Reusing tarball for layer "config" with SHA: sha256:7987d808359d7236349aae784ec2f913f451d30a1241fcbca454392c6e8d5bde
[exporter] Adding layer 'config'
[exporter] Layer 'config' SHA: sha256:7987d808359d7236349aae784ec2f913f451d30a1241fcbca454392c6e8d5bde
[exporter] Reusing tarball for layer "process-types" with SHA: sha256:e7adcf8438e4e7fd9efd6b7ec714bc7031e2fba9bcf42d5450a5c88957f2f391
[exporter] Adding layer 'process-types'
[exporter] Layer 'process-types' SHA: sha256:e7adcf8438e4e7fd9efd6b7ec714bc7031e2fba9bcf42d5450a5c88957f2f391
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] Setting CNB_LAYERS_DIR=/layers
[exporter] Setting CNB_APP_DIR=/workspace
[exporter] Setting CNB_PLATFORM_API=0.10
[exporter] Setting CNB_DEPRECATION_MODE=quiet
[exporter] Prepending /cnb/process and /cnb/lifecycle to PATH
[exporter] Setting WORKDIR: '/workspace'
[exporter] Setting default process type 'run-script'
[exporter] Setting ENTRYPOINT: '/cnb/process/run-script'
[exporter] Saving <registry>/cbl-mariner-aca:run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615133552173374...
[exporter] *** Images (d689d05de34f):
[exporter]       <registry>/cbl-mariner-aca:run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615133552173374
[exporter]
[exporter] *** Image ID: d689d05de34f7ec9bbcffff5b9d13020307874927364159de97d08368cb3af24
[exporter] Reading buildpack directory: /layers/capps_dotnet-7.0
[exporter] Reading buildpack directory item: launch.toml
[exporter] Reading buildpack directory item: run-script
[exporter] Reading buildpack directory item: run-script.toml

Proposed solution

Using the same daemon that can pull down the buildpacksio/lifecycle, builder and stack images if not present on the machine, the new run image defined in the extension would also be pulled down if not present.

It appears that the exporter already has context about what this new run image is via the logs above:

[exporter] Using runImage from analyzed metadata 'mcr.microsoft.com/dotnet/aspnet:7.0.5-cbl-mariner2.0'

So I was hoping the daemon could check if this image is present locally first before proceeding.

Describe alternatives you've considered

We could manually pull down the run image before running pack build, but we'd have to ensure that all possible run images bundled as extensions within the builder are present since we won't know which platform a user is targeting when using the builder until the buildpacks decide.

Additional context

• This feature should be documented somewhere

[natalieparellano]

@cormacpayne - thanks for this. This appears to be the same issue as #1686, which should be fixed on pack 0.30.0-pre.2 - can you confirm the version you are using?

[cormacpayne]

@natalieparellano shoot, not sure how I missed the existing issue (GitHub search filters strike again 😢) -- I just updated to the pack version you mentioned and the latest buildpacksio/lifecycle available (0.17.0-rc.1) and am unfortunately still seeing this error, with slightly different verbiage prior to it:

===> EXPORTING
Running the 'exporter' on OS 'linux' with:
Container Settings:
Args: '/cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -log-level debug -app /workspace -cache-dir /cache -stack /layers/stack.toml <registry>/cbl-mariner-aca:run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615142620621473'
System Envs: 'CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.10'
Image: 'index.docker.io/buildpacksio/lifecycle:0.17.0-rc.1'
User: 'root'
Labels: 'map[author:pack]'
Host Settings:
Binds: 'pack-cache-cbl-mariner-aca_run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615142620621473-230f7014d610.build:/cache /var/run/docker.sock:/var/run/docker.sock pack-cache-cbl-mariner-aca_run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615142620621473-230f7014d610.launch:/launch-cache pack-layers-lkuzqarmrk:/layers pack-app-nxwcafucvm:/workspace'

Network Mode: ''
[exporter] Starting exporter...
[exporter] Parsing inputs...
[exporter] Ensuring privileges...
[exporter] Executing command...
[exporter] ERROR: failed to export: get run image top layer SHA: image "<registry>/cbl-mariner-aca:run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615142620621473" has no layers

I'm happy to provide any additional information or try any other versions of the tools involved!

[natalieparellano]

Ah, apologies @cormacpayne - I may have steered you wrong. It appears that pack 0.30.0-pre2 may be broken with lifecycle 0.17.0-rc.1.

I suspect the output from above may still be from an older version of pack, because I see System Envs: 'CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.10' - newer pack would try to use CNB_PLATFORM_API=0.12. I was trying to repro this myself but instead I got this error:

===> ANALYZING
Running the analyzer on OS linux with:
Container Settings:
  Args: /cnb/lifecycle/analyzer -log-level debug -daemon -stack /layers/stack.toml -run-image cnbs/sample-stack-run:alpine -launch-cache /launch-cache hello-extensions
  System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1001 CNB_PLATFORM_API=0.12
  Image: buildpacksio/lifecycle:0.17.0-rc.1
  User: root
  Labels: map[author:pack]
Host Settings:
  Binds: /var/run/docker.sock:/var/run/docker.sock pack-cache-library_hello-extensions_latest-605c1c945967.launch:/launch-cache pack-layers-rnnzoquprm:/layers pack-app-rgjtjvruhv:/workspace
  Network Mode: host
[analyzer] flag provided but not defined: -stack
...
ERROR: failed to build: executing lifecycle: failed with status code: 2

This particular bug will be fixed in #1789 and together with a couple of other fixes (cc @jjbustamante) we hope to cut a new pre-release in the next few days.

In the meantime, I tried to repro the above using lifecycle 0.17.0-pre.2, and I did see the image being pulled:

• Using the runext/update-samples branch of the sample repo
• Updated builders/alpine/builder.toml to contain:

[lifecycle]
version = "0.17.0-pre.2"

• (before the build)$ docker image ls has:

REPOSITORY               TAG           IMAGE ID       CREATED        SIZE
registry                 2             4bb5ea59f8e0   33 hours ago   24MB
cnbs/sample-stack-run    alpine        76c07164e3cb   6 months ago   8.11MB
buildpacksio/lifecycle   0.17.0-rc.1   77123df190b0   43 years ago   32.5MB

• (note localhost:5000/run-image-curl is NOT present)
• $ pack build hello-extensions --builder localhost:5000/extensions-builder --env BP_EXT_DEMO=1 --env BP_REQUIRES=curl --path $PWD/apps/java-maven --pull-policy always --network host --verbose (BP_REQUIRES=curl invokes the extension that does run image switching)
• The output has:

[detector] Found a run.Dockerfile from extension 'samples/curl' setting run image to 'localhost:5000/run-image-curl' 
...
===> BUILDING
...
Pulling image localhost:5000/run-image-curl
latest: Pulling from run-image-curl
...
Status: Downloaded newer image for localhost:5000/run-image-curl:latest
...
===> EXPORTING
Running the exporter on OS linux with:
Container Settings:
  Args: /cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -log-level debug -app /workspace -cache-dir /cache -run /layers/run.toml hello-extensions
  System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1001 CNB_PLATFORM_API=0.12
...
Successfully built image hello-extensions

• (note CNB_PLATFORM_API=0.12)
• (after the build)$ docker image ls has:

REPOSITORY                          TAG            IMAGE ID       CREATED          SIZE
localhost:5000/run-image-curl       latest         b674d18e2cf7   12 minutes ago   22.7MB
registry                            2              4bb5ea59f8e0   33 hours ago     24MB
cnbs/sample-stack-run               alpine         76c07164e3cb   6 months ago     8.11MB
localhost:5000/extensions-builder   latest         927794559adb   43 years ago     55.1MB
buildpacksio/lifecycle              0.17.0-pre.2   16ea1bbc6247   43 years ago     32.3MB
hello-extensions                    latest         b65cc3ab2b77   43 years ago     25.1MB
buildpacksio/lifecycle              0.17.0-rc.1    77123df190b0   43 years ago     32.5MB

• (note localhost:5000/run-image-curl IS present)

Is it possible to check if you have Pulling image <registry>/cbl-mariner-aca:run-dotnet-aspnet-7.0.7-cbl-mariner2.0-20230615142620621473 in your logs? There are a couple of places where this could go wrong, but barring the pack version being older it's possible we don't have the right creds to pull the run image.

[cormacpayne]

@natalieparellano thanks for the detailed response -- it is greatly appreciated!

The CNB_PLATFORM_API value is an interesting note -- when running pack --version, I see 0.30.0-pre2+git-b56634e.build-4601, which is the Windows pack executable I pulled from the latest release on GitHub. I just tried using the previous release (0.30.0-pre1+git-18662b2.build-4267) and see the same API version in the logs. Is it possible the configuration of my builder/buildpacks is snapping to 0.10 rather than 0.12?

I just cleared the Docker images and cache on my system and did a fresh pack build using the 0.30.0-pre2 version of the CLI and 0.17.0-pre.2 version of the lifecycle and wasn't able to see the Pulling image ... message mentioned above. I'll drop the full logs below in case they may help in any way:

Logs

  Builder <registry>/capps/builder:cbl-mariner-2.0 is untrusted
  As a result, the phases of the lifecycle which require root access will be run in separate trusted ephemeral containers.
  For more information, see https://medium.com/buildpacks/faster-more-secure-builds-with-pack-0-11-0-4d0c633ca619
  Pulling image <registry>/capps/builder:cbl-mariner-2.0
  cbl-mariner-2.0: Pulling from capps/builder
  163c883f165a: Pull complete
  17aed9e3f33c: Pull complete
  db8308e67a25: Pull complete
  1de23eee9ca7: Pull complete
  ce0e64b2d539: Pull complete
  70e0bea64ce8: Pull complete
  def02eae9adc: Pull complete
  9bced1db4414: Pull complete
  06acce2dc482: Pull complete
  836b54c2eebd: Pull complete
  5b8ca5752afa: Pull complete
  4150bcd15456: Pull complete
  50ca732467a5: Pull complete
  5f04a5f82530: Pull complete
  35428f7cc8cd: Pull complete
  b32773bb684e: Pull complete
  46773417d952: Pull complete
  72eda6eddfb5: Pull complete
  4f4fb700ef54: Pull complete
  Digest: sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae
  Status: Downloaded newer image for <registry>/capps/builder:cbl-mariner-2.0
  Selected run image <registry>/capps/stacks/run:cbl-mariner-base-core-2.0
  Pulling image <registry>/capps/stacks/run:cbl-mariner-base-core-2.0
  cbl-mariner-base-core-2.0: Pulling from capps/stacks/run
  163c883f165a: Already exists
  17aed9e3f33c: Already exists
  db8308e67a25: Already exists
  1de23eee9ca7: Already exists
  Digest: sha256:f5d80df9005d11131d54a601cbe36c9916c213ead1b13780762adb2c25ced109
  Status: Downloaded newer image for <registry>/capps/stacks/run:cbl-mariner-base-core-2.0
  Pulling image index.docker.io/buildpacksio/lifecycle:0.17.0-pre.2
  0.17.0-pre.2: Pulling from buildpacksio/lifecycle
  a7ca0d9ba68f: Pull complete
  fe5ca62666f0: Pull complete
  b02a7525f878: Pull complete
  fcb6f6d2c998: Pull complete
  e8c73c638ae9: Pull complete
  1e3d9b7d1452: Pull complete
  4aa0ea1413d3: Pull complete
  7c881f9ab25e: Pull complete
  5627a970d25e: Pull complete
  717c0c26cd51: Pull complete
  Digest: sha256:29715c22a65eb8c3f9d4d896813103cf49ec8fe65863f171cf5bd7fdb3e50ea4
  Status: Downloaded newer image for buildpacksio/lifecycle:0.17.0-pre.2
  Creating builder with the following buildpacks:
  -> capps/dotnet-7.0@0.0.1
  -> capps/node-18@0.0.1
  -> capps/python-3.9@0.0.1
  Using build cache volume pack-cache-cbl-mariner_app_1234-75e32be0e7d7.build
  ===> ANALYZING
  Running the analyzer on OS linux with:
  Container Settings:
    Args: /cnb/lifecycle/analyzer -log-level debug -daemon -stack /layers/stack.toml -run-image <registry>/capps/stacks/run:cbl-mariner-base-core-2.0 -launch-cache /launch-cache <registry>/cbl-mariner/app:1234
    System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.10
    Image: index.docker.io/buildpacksio/lifecycle:0.17.0-pre.2
    User: root
  Labels: map[author:pack]
  Host Settings:
    Binds: /var/run/docker.sock:/var/run/docker.sock pack-cache-cbl-mariner_app_1234-75e32be0e7d7.launch:/launch-cache pack-layers-pkrnbokdhz:/layers pack-app-grssqtusjt:/workspace
    Network Mode:
  [analyzer] Image with name "<registry>/cbl-mariner/app:1234" not found
  [analyzer] Ignoring image "<registry>/cbl-mariner/app:1234" because it was corrupt
  [analyzer] Found image with identifier "662627d28adbaa12561efe5bb822fbe45ebcdecb8722a2ba851dfcf6953c9f83"
  ===> DETECTING
  Running the detector on OS linux with:
  Container Settings:
    Args: /cnb/lifecycle/detector -app /workspace -log-level debug
    System Envs: CNB_EXPERIMENTAL_MODE=warn CNB_PLATFORM_API=0.10
    Image: pack.local/builder/6365657163666f696a69:latest
    User:
    Labels: map[author:pack]
  Host Settings:
    Binds: pack-layers-pkrnbokdhz:/layers pack-app-grssqtusjt:/workspace
    Network Mode:
  [detector] Warning: Platform requested experimental feature 'Dockerfiles'
  [detector] ======== Output: capps/dotnet-7.0@0.0.1 ========
  [detector] Found .NET 7.0 applcation via .csproj file in provided source.
  [detector] ======== Results ========
  [detector] pass: capps/install-dotnet-7.0@0.0.1
  [detector] pass: capps/run-dotnet-7.0@0.0.1
  [detector] pass: capps/dotnet-7.0@0.0.1
  [detector] Resolving plan... (try #1)
  [detector] capps/install-dotnet-7.0 0.0.1
  [detector] capps/run-dotnet-7.0     0.0.1
  [detector] capps/dotnet-7.0         0.0.1
  [detector] Running generate for extension capps/install-dotnet-7.0@0.0.1
  [detector] Looking up extension
  [detector] Finding plan
  [detector] Invoking command
  [detector] Creating plan directory
  [detector] Preparing paths
  [detector] Running generate command
  [detector] Reading output files
  [detector] Found '1' Dockerfiles for processing
  [detector] Finished running generate for extension capps/install-dotnet-7.0@0.0.1
  [detector] Running generate for extension capps/run-dotnet-7.0@0.0.1
  [detector] Looking up extension
  [detector] Finding plan
  [detector] Invoking command
  [detector] Creating plan directory
  [detector] Preparing paths
  [detector] Running generate command
  [detector] Reading output files
  [detector] Found '1' Dockerfiles for processing
  [detector] Finished running generate for extension capps/run-dotnet-7.0@0.0.1
  [detector] Validating Dockerfiles
  [detector] Copying Dockerfiles
  [detector] Checking for new run image
  [detector] Found a run.Dockerfile configuring image 'mcr.microsoft.com/dotnet/aspnet:7.0.5-cbl-mariner2.0' from extension with id 'capps/run-dotnet-7.0'
  [detector] Finished build, selected runImage 'mcr.microsoft.com/dotnet/aspnet:7.0.5-cbl-mariner2.0'
  [detector] Updating analyzed metdata with new runImage
  [detector] Loaded existing analyzed metdata from '/layers/analyzed.toml'
  [detector] Updated analyzed metadata with new runImage 'mcr.microsoft.com/dotnet/aspnet:7.0.5-cbl-mariner2.0'
  ===> RESTORING
  Running the restorer on OS linux with:
  Container Settings:
    Args: /cnb/lifecycle/restorer -cache-dir /cache -build-image <registry>/capps/builder:cbl-mariner-2.0 -log-level debug
    System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_REGISTRY_AUTH=<redacted> CNB_PLATFORM_API=0.10
    Image: index.docker.io/buildpacksio/lifecycle:0.17.0-pre.2
    User: root
    Labels: map[author:pack]
  Host Settings:
    Binds: pack-cache-cbl-mariner_app_1234-75e32be0e7d7.build:/cache pack-cache-cbl-mariner_app_1234-75e32be0e7d7.build:/kaniko pack-layers-pkrnbokdhz:/layers pack-app-grssqtusjt:/workspace
    Network Mode:
  [restorer] Pulling builder image metadata...
  [restorer] Reading buildpack directory: /layers/capps_dotnet-7.0
  [restorer] Reading buildpack directory: /layers/capps_dotnet-7.0
  ===> EXTENDING (BUILD)
  Running the extender on OS linux with:
  Container Settings:
    Args: /cnb/lifecycle/extender -app /workspace -log-level debug
    System Envs: CNB_EXPERIMENTAL_MODE=warn CNB_PLATFORM_API=0.10
    Image: pack.local/builder/6365657163666f696a69:latest
    User: root
    Labels: map[author:pack]
  Host Settings:
    Binds: pack-cache-cbl-mariner_app_1234-75e32be0e7d7.build:/kaniko pack-layers-pkrnbokdhz:/layers pack-app-grssqtusjt:/workspace
    Network Mode:
  [extender (build)] Extending <registry>/capps/builder@sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae
  [extender (build)] Found build Dockerfile for extension 'capps/install-dotnet-7.0'
  [extender (build)] Applying the Dockerfile at /layers/generated/build/capps_install-dotnet-7.0/Dockerfile...
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Found sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae in local cache"
  [extender (build)] time="2023-06-16T16:23:33Z" level=error msg="Error while retrieving image from cache: base@sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae getting image from path: read /kaniko/cache/base/sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae: is a directory"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Built cross stage deps: map[]"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Found sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae in local cache"
  [extender (build)] time="2023-06-16T16:23:33Z" level=error msg="Error while retrieving image from cache: base@sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae getting image from path: read /kaniko/cache/base/sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae: is a directory"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Executing 0 build triggers"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Building stage 'base@sha256:2a39ba63cbab53f61e46be9a68f2143aa4a830d0b1130ffa7ba24453d4c542ae' [idx: '0', base-idx: '-1']"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Cmd: USER"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Checking for cached layer oci:/kaniko/cache/layers/cached:b0d248d57d91d63e14cfbca520f4a8083101c3e14dbe7b96f859af96a870ff28..."
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Using caching version of cmd: RUN tdnf -y update &&     tdnf -y install dotnet-sdk-7.0 &&     tdnf clean all"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Checking for cached layer oci:/kaniko/cache/layers/cached:fd73b8ffe734e9da1326acce55da9c78c78bc2bc744f953e8dee0cbd3f9866dd..."
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Using caching version of cmd: RUN tdnf -y install ca-certificates-microsoft &&     tdnf clean all &&     curl --retry 3 --retry-all-errors --create-dirs --output /tmp/certs/nuget.crt http://www.microsoft.com/pkiops/certs/Microsoft%20Azure%20TLS%20Issuing%20CA%2001%20-%20xsign.crt &&     openssl x509 -inform DER -outform PEM -in /tmp/certs/nuget.crt -out /etc/pki/ca-trust/source/anchors/nuget.pem &&     update-ca-trust extract &&     openssl verify /etc/pki/ca-trust/source/anchors/nuget.pem"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Skipping unpacking as no commands require it."
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="USER root"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Cmd: USER"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="No files changed in this command, skipping snapshotting."
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="RUN tdnf -y update &&     tdnf -y install dotnet-sdk-7.0 &&     tdnf clean all"
  [extender (build)] time="2023-06-16T16:23:33Z" level=info msg="Found cached layer, extracting to filesystem"
  [extender (build)] time="2023-06-16T16:23:38Z" level=info msg="RUN tdnf -y install ca-certificates-microsoft &&     tdnf clean all &&     curl --retry 3 --retry-all-errors --create-dirs --output /tmp/certs/nuget.crt http://www.microsoft.com/pkiops/certs/Microsoft%20Azure%20TLS%20Issuing%20CA%2001%20-%20xsign.crt &&     openssl x509 -inform DER -outform PEM -in /tmp/certs/nuget.crt -out /etc/pki/ca-trust/source/anchors/nuget.pem &&     update-ca-trust extract &&     openssl verify /etc/pki/ca-trust/source/anchors/nuget.pem"
  [extender (build)] time="2023-06-16T16:23:38Z" level=info msg="Found cached layer, extracting to filesystem"
  [extender (build)] Starting build
  [extender (build)] Running build for buildpack capps/dotnet-7.0@0.0.1
  [extender (build)] Looking up buildpack
  [extender (build)] Finding plan
  [extender (build)] Creating plan directory
  [extender (build)] Preparing paths
  [extender (build)] Running build command
  [extender (build)] Using Azure Container Apps .NET 7.0 Buildpack
  [extender (build)]
  [extender (build)] Using .NET 7.0 version:
  [extender (build)] 7.0.304
  [extender (build)]
  [extender (build)] Attempting to build project file '/workspace/HelloWorldApp.csproj'
  [extender (build)]
  [extender (build)]
  [extender (build)] Welcome to .NET 7.0!
  [extender (build)] ---------------------
  [extender (build)] SDK Version: 7.0.304
  [extender (build)]
  [extender (build)] Telemetry
  [extender (build)] ---------
  [extender (build)] The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.
  [extender (build)]
  [extender (build)] Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
  [extender (build)]
  [extender (build)] ----------------
  [extender (build)] Installed an ASP.NET Core HTTPS development certificate.
  [extender (build)] To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
  [extender (build)] Learn about HTTPS: https://aka.ms/dotnet-https
  [extender (build)] ----------------
  [extender (build)] Write your first app: https://aka.ms/dotnet-hello-world
  [extender (build)] Find out what's new: https://aka.ms/dotnet-whats-new
  [extender (build)] Explore documentation: https://aka.ms/dotnet-docs
  [extender (build)] Report issues and find source on GitHub: https://github.com/dotnet/core
  [extender (build)] Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  [extender (build)] --------------------------------------------------------------------------------------
  [extender (build)] MSBuild version 17.6.3+07e294721 for .NET
  [extender (build)]   Determining projects to restore...
  [extender (build)]   Restored /workspace/HelloWorldApp.csproj (in 1.6 sec).
  [extender (build)]   HelloWorldApp -> /workspace/bin/release/net7.0/linux-x64/HelloWorldApp.dll
  [extender (build)]   HelloWorldApp -> /workspace/buildpack-output/
  [extender (build)]
  [extender (build)] Running .NET application with .dll file 'HelloWorldApp.dll'
  [extender (build)]
  [extender (build)] No PORT environment variable detected, defaulting to '8080'
  [extender (build)]
  [extender (build)] Processing layers
  [extender (build)] Updating environment
  [extender (build)] Reading output files
  [extender (build)] Updating buildpack processes
  [extender (build)] Updating process list
  [extender (build)] Finished running build for buildpack capps/dotnet-7.0@0.0.1
  [extender (build)] Copying SBOM files
  [extender (build)] Creating SBOM files for legacy BOM
  [extender (build)] Listing processes
  [extender (build)] Finished build
  ===> EXPORTING
  Running the exporter on OS linux with:
  Container Settings:
    Args: /cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -log-level debug -app /workspace -cache-dir /cache -stack /layers/stack.toml <registry>/cbl-mariner/app:1234
    System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.10
    Image: index.docker.io/buildpacksio/lifecycle:0.17.0-pre.2
    User: root
    Labels: map[author:pack]
  Host Settings:
    Binds: pack-cache-cbl-mariner_app_1234-75e32be0e7d7.build:/cache /var/run/docker.sock:/var/run/docker.sock pack-cache-cbl-mariner_app_1234-75e32be0e7d7.launch:/launch-cache pack-layers-pkrnbokdhz:/layers pack-app-grssqtusjt:/workspace
    Network Mode:
  [exporter] ERROR: failed to export: get run image top layer SHA: image "<registry>/cbl-mariner/app:1234" has no layers
  ERROR: failed to build: executing lifecycle: failed with status code: 62

[natalieparellano]

@cormacpayne - that's very interesting. If you run pack builder inspect on your builder, what Platform API versions do you see under Lifecycle, e.g.:

Lifecycle:
  Version: 0.15.2
...
  Platform APIs:
    Deprecated: (none)
    Supported: 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10

Perhaps the CNB_PLATFORM_API though is a red herring - I'll grab a Windows environment to see if I could reproduce your issue there, it might have to do with the filepaths that are generated when copying files out of a container. We grab <layers>/analyzed.toml out of the detect container to determine if the run image changed.

[cormacpayne]

@natalieparellano ah ha, I found the culprit thanks to the pack builder inspect command -- the Created By field told me the builder was created via a 0.29.0 version of pack, so it looks like I was using the correct version of the CLI, but I just hadn't refreshed my builder to be created by it.

I just ran through this scenario with the updated builder and we're all good now -- the extension run image is being pulled down after a wipe of the Docker images and cache!

Sorry for the false alarm, and thanks again for all of the help! 😃

[natalieparellano]

That's great to hear! 🙌🏼