pack build fails with untrusted builders on Apple Silicon

[briandealwis]

Summary

pack build from an Apple M1 will fail when using an untrusted builder. With untrusted builders, the pack library will not use the builder's embedded lifecycle binary and instead attempt to run the lifecycle from a separate image:

pack/build.go

Lines 329 to 343 in e53adb9

	if !opts.TrustBuilder {
	if lifecycleImageSupported(imgOS, lifecycleVersion) {
	lifecycleImageName := opts.LifecycleImage
	if lifecycleImageName == "" {
	lifecycleImageName = fmt.Sprintf("%s:%s", internalConfig.DefaultLifecycleImageRepo, lifecycleVersion.String())
	}
	lifecycleImage, err := c.imageFetcher.Fetch(
	ctx,
	lifecycleImageName,
	true,
	opts.PullPolicy,
	)
	if err != nil {
	return errors.Wrap(err, "fetching lifecycle image")
	}

Recent lifecycle images (e.g., buildpacksio/lifecycle:0.11.1) point to a manifest list with images for linux/amd64 and windows/amd64, and so the attempt to fetch an image for linux/arm64 fails.

I guess this is somewhat related to buildpacks/lifecycle#435 except that it's actually useful to be able to use Docker Desktop for M1's preconfigured binfmt_exec emulation to build linux/amd64 images. Perhaps pack should attempt to pull down the lifecycle image for the same os/arch as the builder image.

---

Reproduction

Steps

On an Apple M1 device, try to build using the gcr.io/buildpacks/builder:latest builder, which is not considered a trusted builder.

1. pack build --builder gcr.io/buildpacks/builder <image-name>

Current behavior

$ pack build --builder gcr.io/buildpacks/builder test
latest: Pulling from buildpacks/builder
Digest: sha256:5f109c6becf6ecd70d7201de44775f21902e1b027ee68eb982bd55787eb25165
Status: Downloaded newer image for gcr.io/buildpacks/builder:latest
v1: Pulling from buildpacks/gcp/run
Digest: sha256:13535b83b3cfb66571907c574bca1cc1852ac6cf5b3096d3e654cd2d95bbecd6
Status: Image is up to date for gcr.io/buildpacks/gcp/run:v1
0.11.1: Pulling from buildpacksio/lifecycle
ERROR: failed to build: fetching lifecycle image: no matching manifest for linux/arm64/v8 in the manifest list entries

Expected behavior

Hoped for the build to happen.

Workarounds

Indicate that the builder is trusted, so that pack will use the lifecycle embedded in the builder image.

---

Environment

pack info

$ pack version
0.18.1+git-b5c1a96.build-2373

docker info

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 32
  Running: 1
  Paused: 0
  Stopped: 31
 Images: 66
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.942GiB
 Name: docker-desktop
 ID: 4TO7:A6P3:ZM5L:BMIK:VTON:N6PA:VRHC:DBKI:Z5GO:5SFG:U4T3:N3L6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[jromero]

The proposal, "Perhaps pack should attempt to pull down the lifecycle image for the same os/arch as the builder image.", makes a lot of sense to me.

It looks like we'd need to propagate some information about the builder image to these locations:

Perhaps pack should attempt to pull down the lifecycle image for the same os/arch as the builder image.

pack/internal/image/fetcher.go

Lines 65 to 89 in e53adb9

	func (f *Fetcher) fetchDaemonImage(name string) (imgutil.Image, error) {
	image, err := local.NewImage(name, f.docker, local.FromBaseImage(name))
	if err != nil {
	return nil, err
	}
	if !image.Found() {
	return nil, errors.Wrapf(ErrNotFound, "image %s does not exist on the daemon", style.Symbol(name))
	}
	return image, nil
	}
	func (f *Fetcher) fetchRemoteImage(name string) (imgutil.Image, error) {
	image, err := remote.NewImage(name, authn.DefaultKeychain, remote.FromBaseImage(name))
	if err != nil {
	return nil, err
	}
	if !image.Found() {
	return nil, errors.Wrapf(ErrNotFound, "image %s does not exist in registry", style.Symbol(name))
	}
	return image, nil
	}

We'd accept a contribution :)

[natalieparellano]

Just to connect some dots, we've had some comments on buildpacks/lifecycle#435 indicating that this issue is causing errors when trying to use Hashicorp Waypoint on M1 Macs (see buildpacks/lifecycle#435 (comment)).

[jromero]

@briandealwis a fix for this by @aemengo has been merged to main. Could you verify that it address the issue?

[briandealwis]

Apologies, I missed your request @jromero: it works great with pack 0.20.0. Thanks all!