pack inspect on an image created using an extension doesn't display swapped run-time base image

[cormacpayne]

Description

I have a set of buildpacks that use the same skeleton base image as the build and run-time stack, but depending on the platform found to be used by the provided source (i.e., .NET, node, Python, etc.), the platform's buildpack will use a corresponding extension to swap out the run-time base image to one based on that platform.

For example, the run-time stack specified in the builder is skeleton-run-image:123, and when our .NET buildpack is used, it will leverage an extension that swaps out this run-image with dotnet-run-image:123. However, when I run pack inspect on the image produced from pack build, I see the following:

Inspecting image: final-sample-app:123

REMOTE:

Stack:

Base Image:
  Reference: <large string omitted>
  Top Layer: sha256:<another large string omitted>

Run Images:
  skeleton-run-image:123

<rest of output omitted>

Proposed solution

Using the above example, I would've expected the swapped out run-image, dotnet-run-image:123 to be the displayed "Run Images" when calling pack inspect.

Inspecting image: final-sample-app:123

REMOTE:

Stack:

Base Image:
  Reference: <large string omitted>
  Top Layer: sha256:<another large string omitted>

Run Images:
  dotnet-run-image:123

<rest of output omitted>

Describe alternatives you've considered

I'm not sure of other routes to take when attempting to determine the run-time image used for an image produced by pack build -- I'm not sure if this "Run Images" field is determined by the static run-time image provided either within the builder or with the value provided to the --run-image argument. I was hoping there would be a way for the builder to parse the new run-time image from the extension and update the metadata used by pack inspect to reflect the swap.

Additional context

• This feature should be documented somewhere

Please feel free to let me know if you need any additional information from me! Thanks in advance!

[natalieparellano]

Hi @cormacpayne - thanks for reporting this issue. Just to confirm, is pack inspect just showing the wrong thing, or is the run image not actually getting switched as you expect? You could see the run image output by the lifecycle by running something like

docker inspect <image-name> | jq -r .[0].Config.Labels.\"io.buildpacks.lifecycle.metadata\" | jq .runImage.image

[natalieparellano]

One more question - is it possible to confirm your lifecycle version? Looking at the pack code, this seems to be reading from "old" metadata that might not getting updated in lifecycle 0.16.x but will be updated in lifecycle 0.17.x.

[cormacpayne]

@natalieparellano -- Hey Natalie, thanks for the prompt response!

I'm seeing the run-time image being swapped as expected when running the application (i.e., my .NET app is able to run in a container even though the statically-defined run-time image in the builder that inspect says is the base does not have .NET binaries baked with it, so it's definitely using the corresponding extension to swap out the run-time base to a .NET image), it's just the metadata from inspect that's specifying the wrong run-time image base.

From right before the ANALYZING step during pack build, it's showing the lifecycle as follows:

0.15.2: Pulling from buildpacksio/lifecycle
Digest: sha256:e9e6012cf38bf4904f15c0059fe8de9dc79a87279c9ae9b4f4b7a15e81103d7b
Status: Image is up to date for buildpacksio/lifecycle:0.15.2

I'll upgrade to a newer version of the lifecycle and see if the same inspect results holds true.

[cormacpayne]

Just a quick follow-up: I tested with the 0.16.4 and 0.17.0-pre.2 tags for the buildpacksio/lifecycle image and found the same results with pack inspect.

Happy to provide any additional information you may require 🙂

[natalieparellano]

Ah, apologies - this is fixed in lifecycle on main but not in the pre-release. We should have lifecycle 0.17.0-rc.1 soon to verify, but in my local testing it's functioning as expected :)

[cormacpayne]

Perfect! Thanks for the update -- feel free to let me know when the latest tag containing the fix is available and I'll be happy to test it out on my side as well!

[cormacpayne]

@natalieparellano Hey Natalie, just a quick update on this item: I've been trying to use a builder created with the 0.17.0-rc.1 version of the lifecycle and the 0.30.0-pre2+git-b56634e.build-4601 version of pack (confirmed values from the pack builder inspect command), but I'm running into the following error when running pack build using this builder:

...
Pulling image buildpacksio/lifecycle:0.17.0-rc.1
0.17.0-rc.1: Pulling from buildpacksio/lifecycle
Digest: sha256:fa8f50586378aa1baa6dd5af727c022055b7918258fa2e95a3649434e7dcbef9
Status: Image is up to date for buildpacksio/lifecycle:0.17.0-rc.1
Creating builder with the following buildpacks:
-> capps/dotnet-7.0@0.0.1
-> capps/node-18@0.0.1
-> capps/python-3.9@0.0.1
Using build cache volume pack-cache-cbl-mariner_app_1234-75e32be0e7d7.build
===> ANALYZING
Running the analyzer on OS linux with:
Container Settings:
  Args: /cnb/lifecycle/analyzer -log-level debug -daemon -stack /layers/stack.toml -run-image <registry>/capps/stacks/run:cbl-mariner-base-core-2.0 -launch-cache /launch-cache <registry>/cbl-mariner/app:1234
  System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.12
  Image: buildpacksio/lifecycle:0.17.0-rc.1
  User: root
  Labels: map[author:pack]
Host Settings:
  Binds: /var/run/docker.sock:/var/run/docker.sock pack-cache-cbl-mariner_app_1234-75e32be0e7d7.launch:/launch-cache pack-layers-dyezxblgch:/layers pack-app-knruticsxi:/workspace
  Network Mode:
[analyzer] flag provided but not defined: -stack
[analyzer] Usage of lifecycle:
[analyzer]   -analyzed string
[analyzer]      path to analyzed.toml (default "<layers>/analyzed.toml")
[analyzer]   -cache-image string
[analyzer]      cache image tag name
[analyzer]   -daemon
[analyzer]      export to docker daemon
[analyzer]   -gid int
[analyzer]      GID of user's group in the stack's build and run images (default 1000)
[analyzer]   -launch-cache string
[analyzer]      path to launch cache directory
[analyzer]   -layers string
[analyzer]      path to layers directory (default "/layers")
[analyzer]   -layout
[analyzer]      export to OCI layout format on disk
[analyzer]   -layout-dir string
[analyzer]      path to output directory for images in OCI layout format
[analyzer]   -log-level string
[analyzer]      logging level (default "info")
[analyzer]   -no-color
[analyzer]      disable color output
[analyzer]   -previous-image string
[analyzer]      reference to previous image
[analyzer]   -run string
[analyzer]      path to run.toml (default "/cnb/run.toml")
[analyzer]   -run-image string
[analyzer]      reference to run image
[analyzer]   -skip-layers
[analyzer]      do not provide layer metadata to buildpacks
[analyzer]   -tag value
[analyzer]      additional tags
[analyzer]   -uid int
[analyzer]      UID of user in the stack's build and run images (default 1000)
[analyzer]   -version
[analyzer]      show version
ERROR: failed to build: executing lifecycle: failed with status code: 2

What's interesting is that if I remove the [lifecycle] section in my builder.toml that was used to snap to 0.17.0-rc.1 and create a new builder that's based on 0.17.0-pre.2, I see the same command call to the analyzer, but no error:

...
Pulling image buildpacksio/lifecycle:0.17.0-pre.2
0.17.0-pre.2: Pulling from buildpacksio/lifecycle
Digest: sha256:29715c22a65eb8c3f9d4d896813103cf49ec8fe65863f171cf5bd7fdb3e50ea4
Status: Image is up to date for buildpacksio/lifecycle:0.17.0-pre.2
Creating builder with the following buildpacks:
-> capps/dotnet-7.0@0.0.1
-> capps/node-18@0.0.1
-> capps/python-3.9@0.0.1
Using build cache volume pack-cache-cbl-mariner_app_1234-75e32be0e7d7.build
===> ANALYZING
Running the analyzer on OS linux with:
Container Settings:
  Args: /cnb/lifecycle/analyzer -log-level debug -daemon -stack /layers/stack.toml -run-image <registry>/capps/stacks/run:cbl-mariner-base-core-2.0 -launch-cache /launch-cache <registry>/cbl-mariner/app:1234
  System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1000 CNB_PLATFORM_API=0.12
  Image: buildpacksio/lifecycle:0.17.0-pre.2
  User: root
  Labels: map[author:pack]
Host Settings:
  Binds: /var/run/docker.sock:/var/run/docker.sock pack-cache-cbl-mariner_app_1234-75e32be0e7d7.launch:/launch-cache pack-layers-mwvkrhbbgg:/layers pack-app-leiyjjgsvr:/workspace
  Network Mode:
[analyzer] Found image with identifier "7f66946e6bd1935e17c1c9b3591b7bc7c8b4f5114b1b70c54601fe65a21594d5"
[analyzer] Restoring data for SBOM from previous image
[analyzer] Retrieving previous image SBOM layer for "sha256:ba8c81eb0b0d7ca3c54e892d5b3c47cb874378597f32deeb9fae81bdacf764c0"
[analyzer] Found image with identifier "315a1fc4a60694b462d2d1ae879c79f2ebc9775f31e1c82b062c4f10bd8e5c2f"
...

I was hoping to use pack build to spin up a quick image to test pack inspect against using this new version of the lifecycle, but am a little stuck with this error at the moment. I wasn't sure if this was something that you've seen or could assist with -- as always, I'm happy to provide any additional context you may need!

[daniv-msft]

@natalieparellano Thanks for your help with this! On top of @cormacpayne's question, I would like to know if you have visibility on when these changes would be released?
Again, thank you for all the help!

[AidanDelaney]

This has been referenced in a community discussion at buildpacks/community#223

[natalieparellano]

@cormacpayne @daniv-msft thanks for your question, and apologies for the slow reply - the notification was filtered from my inbox somehow.

flag provided but not defined: -stack indicates that the pack version is too old, as this has been fixed on main but isn't part of pre2. Sadly the lifecycle also had a bug where the pre-release version erroneously accepted -stack on the newer platform API; this was fixed in the lifecycle release candidate but unfortunately the fix broke earlier versions of pack.

All that is to say that, to your point, we need to get a new release of pack out so that this can be tested end-to-end. We have been hoping to have a new pre-release (or even a release candidate) out "soon" for a few weeks now, but unfortunately this has been difficult to achieve due to the platform team being a bit stretched & the complexity of some of the open PRs. If you're feeling ambitious to try a dev version of pack, this action run has artifacts that could be used for testing.

At this point all of the issues in the milestone now have a PR that is ready for review, so I think it's safe to assume that we're getting pretty close now. I'll follow up in the next couple of days when we have a better estimate.

[cormacpayne]

@natalieparellano Hey Natalie, thanks for the informative response as always -- I just tested our flow with the pack artifact from the linked GitHub Action run and the 0.17.0-rc.2 lifecycle. The pack build command was successful in creating the image and when running pack inspect on this image, I could see that the run image was now the one specified in our run extension:

Inspecting image: sample/test/app:123

REMOTE:
(not present)

LOCAL:

Stack:

Base Image:
  Reference: 1e5bffb9778d093b0d02db4e1154dd561dda02d42e40e70cc0362d2b3f2ba19e
  Top Layer: sha256:e3a7d46308cb738e9892aae0e8ebcf62a58d15d64c663e3bf19912e7708bb7a2

Run Images:
  mcr.microsoft.com/dotnet/aspnet:7.0.7-cbl-mariner2.0

It looks like we should be good to go whenever this version of pack is officially released in the future

Thanks for all of the help!

[jjbustamante]

Hi @cormacpayne

Regarding the pack release cadence, we don’t have a time-based release cadence anymore, now we do a feature-based release cadence for minor releases, so the time between releases could be longer. We are aiming for time-based patch versions with bug fixes and dependency bumps (but we haven’t been great at sticking to this).

We are trying to grow the platform team and have more things done, but it's been a little tricky. In any case, we are very close to releasing pack 0.30.0

cc @hone @jkutner

[jjbustamante]

This issue will close once pack 0.30.0 is released

[jjbustamante]

Closing after pack 0.30.0 released