Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ARM #435

Open
ekcasey opened this issue Oct 9, 2020 · 35 comments
Open

Support ARM #435

ekcasey opened this issue Oct 9, 2020 · 35 comments

Comments

@ekcasey
Copy link
Member

ekcasey commented Oct 9, 2020

GOAL: Support buildpack builds for ARM images

** WARNING** Not yet ready to implement, may require RFCs, spec changes and/or imgutil changes.

Lifecycle Behavior

analyzer

  1. If the previous image is a multiarch image, select the child manifest that matches the runtime architecture of the build environment.
  2. Warn and proceed without analyzing the previous image if there the image exists but has the wrong architecture.

Because analyzer may run in a lifecycle image rather than in the build environment the runtime architecture of the analyze phase may not match the build environment. We need an RFC to either:

  1. Record the information in detector and pass it to analyzer.
  2. Accept an -arch flag and fail if the previous image is a multiarch image and the -arch flag is not provided by the platform

exporter

  1. If the run-image is a multiarch image, select the child manifest that matches the runtime architecture of the build environment.
  2. Fail if there is no run-image with the correct architecture
  3. If there was a previous image, validate that the previous image architecture and the run image architecture match.

Because exporter may run in a lifecycle image rather than in the build environment the runtime architecture of the analyze phase may not match the build environment. We need an RFC to either:

  1. Record the information in a previous phase and pass it to analyzer.
  2. Accept an -arch flag and fail if the previous image is a multiarch image and the -arch flag is not provided by the platform

rebaser

  1. If the run-image is a multiarch image, select the child manifest that matches the app image architecture
  2. If the run-image is not a multiarch image rebaser should validate that the architecture matches the app-image before rebasing.

Lifecycle Artifacts

Publish ARM lifecycle artifacts

Add the following new Makefile targets:

  • make build-linux-x86-64
  • make build-linux-arm64
  • make build-windows-x86-64
  • make build-windows-arm64
  • make package-linux-x86-64
  • make package-linux-arm64
  • make package-windows-x86-64
  • make package-windows-arm64
    Binaries should end up in out/<os>/<arch>.

And update the existing targets:

  • make build-linux -> make build-linux-x86-64 && make build-linux-arm64
  • make build-windows -> make build-windows-x86-64 && make build-windows-arm64
  • make package-linux -> make package-linux-x86-64 && make package-linux-arm64
  • make package-windows -> make package-windows-x86-64 && make package-windows-arm64

Local ARM image builds

Add the following new Makefile targets:

  • make build-image-linux-amd64
  • make build-image-linux-arm64
  • make build-image should create a multiarch image containing a windows image and both linux architectures.

Docker desktop should support both building for multiple architectures. I do not know if a daemon running on a x86 linux machine will allow us to load arm64 images.

Note We cannot publish arm windows images becausemcr.microsoft.com/windows/servercore:ltsc2019 does not include an arm manifest (unlikemcr.microsoft.com/windows/nanoserver:1809 which we recently moved away from).

Publish ARM images

Instead of publish buildpacksio/lifecycle:<commit>-linux images for each build, publish

  • buildpacksio/lifecycle:<commit>-linux-amd64
  • buildpacksio/lifecycle:<commit>-linux-arm64

The buildpacksio/lifecycle:<commit> multiarch image should contain the windows image and both linux architectures.

When we release a new lifecycle version we should retag appropriate images with

  • buildpacksio/lifecycle:<version>-linux-amd64
  • buildpacksio/lifecycle:<version>-linux-arm64
  • buildpacksio/lifecycle:latest-linux-amd64
  • buildpacksio/lifecycle:latest-linux-arm64

And ensure that both buildpacksio/lifecycle:latest and buildpacksio/lifecycle:<version> include both linux architectures.

@dfreilich
Copy link
Member

This issue focuses on support for arm64. There's an issue open by pack (buildpacks/pack#907) re: arm64 and armv7 support – should I open a separate issue for armv7, or can we tack that on to this?

@dfreilich
Copy link
Member

Also, for clarity sake, what RFC(s) would you imagine we would need to drive this through?

@natalieparellano
Copy link
Member

Possibly relevant exploration around building ARM images on x86: https://github.com/buildpacks/samples/pull/60/files

@dfreilich
Copy link
Member

Would we need separate lifecycle images for darwin/arm64 as well? If so, should I open up a new issue for that? Related: buildpacks/pack#1003

@tibeer
Copy link

tibeer commented Mar 30, 2021

Ran into this issue today. Just for your information: I do not use lifecycle directly but rather by trying to use Hashicorps waypoint to spin up a Kubernetes deployment (of any container) on my local Kubernetes cluster. I am working on a M1 Mac.

@ryuheechul
Copy link

👍🏼 for this issue and I'm pretty much repeating what @tibeer said above.

Just to report a use-case, currently https://learn.hashicorp.com/tutorials/waypoint/get-started-docker?in=waypoint/get-started-docker would not work on M1 laptops with this error.

I can see there was this issue, buildpacks/pack#1003 (in another repo) relating to m1 was resolved but that seems to be for the native binary building not container image building. And this, buildpacks/pack#1171 also to seems be a related issue.

❯ waypoint up

» Building...
Creating new buildpack-based image using builder: heroku/buildpacks:18
✓ Creating pack client
❌ Building image
 │ 2021/06/10 21:32:23.990480 DEBUG:  Pulling image index.docker.io/heroku/buildpac
 │ ks:18

...

 │ 2021/06/10 21:32:27.168139 DEBUG:  -> heroku/nodejs-function-invoker@0.1.5
 │ 2021/06/10 21:32:27.168142 DEBUG:  -> heroku/nodejs-npm@0.4.3
 │ 2021/06/10 21:32:27.168145 DEBUG:  -> heroku/nodejs-typescript@0.2.2
 │ 2021/06/10 21:32:27.168150 DEBUG:  -> heroku/nodejs-engine@0.7.3
 │ 2021/06/10 21:32:27.168154 DEBUG:  -> evergreen/fn@0.3.0
 │ 2021/06/10 21:32:28.220344 DEBUG:  Pulling image buildpacksio/lifecycle:0.11.3
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries

@imjasonh
Copy link
Member

I looked into this a bit out of curiosity, and it looked like building the lifecycle binary into an image would be fairly straightforward to accomplish with ko, not just for linux/arm64 but also more exotic architectures like linux/ppc64le -- basically anything go build supports, plus what you can find a base image for -- and after ko-build/ko#374 for Windows as well.

Essentially, ko does what your image build process already does: it go builds the binary for the desired output os+architecture, tars it up, appends it to a base image, and either puts it in the local daemon or pushes it to a registry.

The only hitch seems to be the use of cgo in priv/user_linux.go. If there's any way to avoid that, I think you could get multi-arch lifecycle images for lifecycle relatively easily with ko.

buildpacksio/pack doesn't have this issue, so you could get a multi-arch image for that even easier.

I'd be happy to help push this forward if this seems like a reasonable path. I understand the use of cgo might be unavoidable, or that changing the release process might be undesirable.

@micahyoung
Copy link
Member

I feel like it's a great suggestion @imjasonh. Having more standardization and tooling around generating these foundational images would be a big win (selfishly, particularly for the Windows images). I'm not sure how maintainers might feel about migrating to ko, but if you're up for it, I think starting a separate issue here would be enough to kick off the discussion. I'll gladly add my +1.

@natalieparellano
Copy link
Member

@ryuheechul it looks like your issue most closely relates to buildpacks/pack#1197. Just to confirm, are you trying to build a linux/amd64 image? If so, I think a change in pack will unblock you. Is there any way to configure Waypoint to trust the provided builder?

There seem to be several threads on this issue, I wonder if it's worth breaking them out:

@natalieparellano
Copy link
Member

Just wanted to let anyone interested know that there is an #arm channel in buildpacks Slack. This thread provides a somewhat mind-boggling matrix of possible supported scenarios. Please feel free to add your thoughts here!

@ryuheechul
Copy link

@natalieparellano Thanks for your comment. Btw, I'm not trying to build anything, all the logs are triggered by Waypoint itself and based on the error logs at the bottom,

 │ 2021/06/10 21:32:28.220344 DEBUG:  Pulling image buildpacksio/lifecycle:0.11.3
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries

I think it's simply because the docker image is not built for linux/armX as I can see it here, https://hub.docker.com/r/buildpacksio/lifecycle/tags.

But then, I'm only guessing at this point so I might be wrong.

@warlockdn
Copy link

warlockdn commented Jun 20, 2021

Do you guys have anything planned for arm architecture ?
I also came across this issue while using waypoint.

@natalieparellano
Copy link
Member

@willianpaixao I believe it's fixed by buildpacks/pack#1216. Does upgrading resolve your issue? Please let us know!

@cihantas
Copy link

@natalieparellano It appears like the issue isn't resolved. Using v0.11.4 on an M1 Mac still results in the same error message.

@natalieparellano
Copy link
Member

natalieparellano commented Aug 9, 2021

@cihantas just confirming you are using pack v0.20.0 (v0.11.4 is the lifecycle version).

Edit: also just to confirm - are you trying to build an amd64 image?

@cihantas
Copy link

cihantas commented Aug 9, 2021

@natalieparellano Yes, I can confirm I'm using v0.11.4.

I'm using pack in the context of Hashicorp Waypoint. Not sure if I'm building the amd64 version. Is there a way for me to verify this?

@aemengo
Copy link
Contributor

aemengo commented Aug 10, 2021

@cihantas Thanks for calling this out. This is a great find.

It seems that there's been some confusion (myself included) about how Hashicorp Waypoint works. Upgrading the pack CLI on your local machine won't have any effect because Waypoint is using pack via its golang API library. As of right now, Waypoint's pack version is pinned to v0.18.1.

There's the quicker fix which is to make a PR bumping Waypoint's pinned pack version to v0.20.0, like @natalieparellano suggested.

But I think you've highlighted a nascent problem with the pack Golang API library: the "trusted builder code path" isn't exercised at all. For trusted builders (like their default heroku/buildpacks:18), the lifecycle shouldn't even be downloaded. And more importantly, it means that the lifecycle's newer creator workflow isn't getting exercised.

--

If no one else wants to have the fun, I'll make the appropriate PRs in Waypoint, and pack, GitHub repositories - within the next couple days.


EDIT:

@aemengo
Copy link
Contributor

aemengo commented Aug 23, 2021

@cihantas @natalieparellano and other interested folks,

Just wanted to highlight that hashicorp released waypoint v0.5.0+ which should have the architecture fix. The following error should no longer be present:

 # ...
 │ 0.11.3: Pulling from buildpacksio/lifecycle
! fetching lifecycle image: no matching manifest for linux/arm64/v8 in the
  manifest list entries

@imjasonh
Copy link
Member

buildpacksio/lifecycle is still not released with arm64 support: https://explore.ggcr.dev/?image=buildpacksio%2Flifecycle

#680 should fix this (after another release)

@donnyv12
Copy link

another M1 Mac user who would love to use buildpacks here 👋

@imjasonh
Copy link
Member

imjasonh commented Sep 10, 2021

As of v0.12.0-rc.1, the lifecycle image includes a linux/arm64 build -- the image is buildpacksio/lifecycle:0.12.0-rc.1.

For Waypoint users on M1, I'm not sure how to configure Waypoint to use this image, or if it's even configurable. If someone can manage to figure that out and try out the 0.12 RC and report back, that would be really helpful.

edit: Or will this require pack to bump its dep on lifecycle, and waypoint to bump its dep on pack, and do a release? 🤷

@aemengo
Copy link
Contributor

aemengo commented Sep 10, 2021

@imjasonh The Waypoint issue is orthogonal to this "Support ARM" issue (and should be resolved with the latest Waypoint release). Let's please open another issue about Waypoint if someone is still finding issue.

@agracey
Copy link

agracey commented May 17, 2022

What's the status of this? I see a lot of arm related build steps and artifacts in the makefile. If I build a builder with the arm binaries and ran it on a cluster that's arm only, would it fail?

@natalieparellano
Copy link
Member

@agracey thanks for the feedback - we should probably close this issue or update it to include only the work yet to be done. If you have a builder where the base image, lifecycle, and buildpacks are all based on arm, and you run it on arm, it should produce an arm image.

The work still to be done is of the nature of edge cases that may arise when e.g., the lifecycle is running on one architecture but the desired output image is of another architecture. There is a matrix of possibilities that we need to evaluate to determine if they can be supported.

@xlf12
Copy link

xlf12 commented May 18, 2022

We have started migrating our development boxes from x86 to Apple Silicon. A packing on arm towards x86 would be welcome ;-)

... just to put 2 cent in this discussion ;-)

@agracey
Copy link

agracey commented May 18, 2022

That's good to hear! Thank you for the update.

@vnobo
Copy link

vnobo commented Sep 13, 2023

How is it going?

@dlion
Copy link
Member

dlion commented Sep 15, 2023

@vnobo @jjbustamante is actually working on this topic, you can follow this tracking issue buildpacks/rfcs#294 to follow the progress

@jjbustamante
Copy link
Member

I want to summarize some stuffs because this issue contains a lot of different threads and things to be done.
I think the original idea for this ticket was completed, the latest version of lifecycle and pack released till date are multi-arch images.

> crane manifest buildpacksio/lifecycle:0.17.0 | jq '.manifests[].platform'
{
  "architecture": "arm64",
  "os": "linux"
}
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "amd64",
  "os": "windows",
  "os.version": "10.0.17763.4737"
}
> crane manifest buildpacksio/pack:0.30.0 | jq '.manifests[].platform'
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "arm64",
  "os": "linux"
}

With that been said, I like to point out this comment from @natalieparellano

If you have a builder where the base image, lifecycle, and buildpacks are all based on arm, and you run it on arm, it should produce an arm image.

We already know you can run pack build on ARM and it works correctly. @dmikusa and @dashaun have these incredible repos that are very helpful

What is missing?
We want to deliver some tooling and capabilities to the CNB ecosystem that allow Buildpack authors to package and distribute their Builders and Buildpacks supporting multi-arch and that's the goal being tracked by the issue mentioned above from @dlion, also we want to actually add support in pack to run pack build --platform, or something like that, to output an application image for different os/arch (check last kubecon 2022 talk) from @AidanDelaney, but we are still not there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests