Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quickstart: PipeCD control plane does not work on Arm architecture cluster #4896

Closed
ponkio-o opened this issue Apr 24, 2024 · 7 comments · Fixed by #5420
Closed

Quickstart: PipeCD control plane does not work on Arm architecture cluster #4896

ponkio-o opened this issue Apr 24, 2024 · 7 comments · Fixed by #5420
Assignees
@ponkio-o
Labels
area/server dependencies Pull requests that update a dependency file good first issue Good for newcomers kind/enhancement New feature or request

Comments

@ponkio-o
Copy link
Contributor

What happened:
Quickstart doesn't work on Arm architecture kubernetes cluster.

related: #4116

What you expected to happen:
Working pipecd correctly on kubernetes cluster using arm arch.

How to reproduce it:
Run pipectl quickstart against a Kubernetes cluster configured with only Arm architecture worker nodes.

My k8s cluster configured with only arm64 nodes.

$ k version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

$ k get node -o go-template='{{range .items}}{{.metadata.name}}  {{index .metadata.labels "kubernetes.io/arch"}}{{"\n"}}{{end}}'
instance-20211220-2249  arm64
instance-20211220-2250  arm64
instance-20211222-2325  arm64
instance-20211222-2332  arm64

Run pipectl quickstart

$ pipectl quickstart --version v0.47.0
Installing the controlplane in quickstart mode...
Release "pipecd" does not exist. Installing it now.
NAME: pipecd
LAST DEPLOYED: Wed Apr 24 11:29:25 2024
NAMESPACE: pipecd
STATUS: deployed
REVISION: 1
TEST SUITE: None

Intalled the controlplane successfully!

Waiting for PipeCD control plane to be ready...
PipeCD control plane status: ContainerCreating 3 Running 1 Init:0/1 2
PipeCD control plane status: ContainerCreating 1 CrashLoopBackOff 2 Running 1 Init:0/1 2
PipeCD control plane status: ContainerCreating 1 CrashLoopBackOff 2 Running 1 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: Running 3 Init:0/1 2 Error 1
PipeCD control plane status: Running 3 Init:0/1 2 Error 1
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: Running 2 Init:0/1 2 Error 2
PipeCD control plane status: CrashLoopBackOff 1 Running 2 Init:0/1 2 Error 1
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
PipeCD control plane status: CrashLoopBackOff 2 Running 2 Init:0/1 2
Failed to expose PipeCD control plane service!!
github.com/pipe-cd/pipecd/pkg/app/pipectl/cmd/quickstart.(*command).run
        /home/runner/work/pipecd/pipecd/pkg/app/pipectl/cmd/quickstart/quickstart.go:121
github.com/pipe-cd/pipecd/pkg/cli.runWithContext
        /home/runner/work/pipecd/pipecd/pkg/cli/cmd.go:95
github.com/pipe-cd/pipecd/pkg/cli.WithContext.func1
        /home/runner/work/pipecd/pipecd/pkg/cli/cmd.go:51
github.com/spf13/cobra.(*Command).execute
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950
github.com/spf13/cobra.(*Command).Execute
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
github.com/pipe-cd/pipecd/pkg/cli.(*App).Run
        /home/runner/work/pipecd/pipecd/pkg/cli/app.go:69
main.main
        /home/runner/work/pipecd/pipecd/cmd/pipectl/main.go:48
runtime.main
        /opt/hostedtoolcache/go/1.20.5/x64/src/runtime/proc.go:250
2024/04/24 11:41:28 failed while waiting for server to be ready

Some components do not start properly.

$ k get pod -n pipecd
NAME                              READY   STATUS     RESTARTS      AGE
pipecd-cache-56c7c65ddc-qngvf     1/1     Running    0             35s
pipecd-gateway-58589b55f9-8jj9z   0/1     Error      2 (28s ago)   35s
pipecd-minio-677999d5bb-v2727     0/1     Error      2 (28s ago)   35s
pipecd-mysql-6db6688bd4-gnd82     1/1     Running    0             35s
pipecd-ops-667d6f6b74-5vvcg       0/1     Init:0/1   0             35s
pipecd-server-9bcf9f7fd-kvwb4     0/1     Init:0/1   0             35s

This is due to the fact that the container image used doesn't support arm arch.
The following is an example for envoyproxy/envoy-alpine, but other Pods are similar.
https://hub.docker.com/r/envoyproxy/envoy-alpine/tags/?page=&page_size=&ordering=&name=v1.18.3
CleanShot 2024-04-24 at 11 33 54

$ k logs pipecd-gateway-58589b55f9-8jj9z
exec /usr/local/bin/envoy: exec format error

$ k get pod pipecd-gateway-58589b55f9-8jj9z -o json | jq .spec.containers[].image
"envoyproxy/envoy-alpine:v1.18.3"

Upon investigation, the following container images do not support arm.

The pipe-cd/pipecd container image is built for arm64, but the pipecd binary inside the container appears to be built for amd64.

$ docker inspect ghcr.io/pipe-cd/pipecd:v0.47.0 | jq .[].Architecture
"arm64"

$ docker run --entrypoint sh --rm -it ghcr.io/pipe-cd/pipecd:v0.47.0
/ # which pipecd
/usr/local/bin/pipecd

$ docker ps
CONTAINER ID    NAME             IMAGE             TAG        STATE        STATUS           PUBLISHED PORTS
------------    -------------    --------------    -------    ---------    -------------    ---------------
dbfa86dacef5    goofy_ganguly    pipe-cd/pipecd    v0.47.0    [RUNNING]    Up 12 seconds
$ docker cp dbfa86dacef5:/usr/local/bin/pipecd ./

$ file pipecd
pipecd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=WUGpoqiNmaaNRUBLNkNh/JdKvMBjyQMmzFXHw0YM5/9e34ewlbEHqbW1yVSxxv/ZC6HmlDZ9SWAo1MGHTTC, not stripped

Environment:

  • piped version: v0.47.0
  • control-plane version: v0.47.0
  • Others:
@ponkio-o ponkio-o added the kind/bug Something isn't working label Apr 24, 2024
@khanhtc1202 khanhtc1202 added good first issue Good for newcomers area/server dependencies Pull requests that update a dependency file labels Apr 24, 2024
@khanhtc1202
Copy link
Member

Hi @ponkio-o, Thanks for reporting this 👍 Basically, this can be renamed as PipeCD control plane does not work on Arm architecture cluster.

Also, the idea of PipeCD architecture does not require users to install the control plane in the same cluster as the applications cluster; only the piped agent should be installed in the same cluster with the application. So I think this should be only a nice-to-help enhancement. WDYT?

Of course, I understand that in some cases, users may only have Arm architecture clusters to install pipecd, but please understand my point of recommending not forcing the pipecd control plane in the same cluster as the applications cluster.

ref: https://pipecd.dev/docs-v0.47.x/concepts/

@ponkio-o ponkio-o changed the title Does not work properly on Arm architecture PipeCD control plane does not work on Arm architecture cluster Apr 24, 2024
@ponkio-o
Copy link
Contributor Author

@khanhtc1202 Thank you for your response!

Also, the idea of PipeCD architecture does not require users to install the control plane in the same cluster as the applications cluster; only the piped agent should be installed in the same cluster with the application. So I think this should be only a nice-to-help enhancement. WDYT?

Does this mean that PipeCD is not designed to run the control plane and applications on the same cluster? Also, I feel like the issue of not supporting the Arm architecture and the issue of where the control plane operates are separate problems.

@khanhtc1202
Copy link
Member

khanhtc1202 commented Apr 24, 2024 •
edited
Loading

Does this mean PipeCD is not designed to run the control plane and applications on the same cluster?

I mean, the PipeCD architecture doesn't force users to install its Control Plane in the same cluster as the application cluster, which is a good point since some applications require special specs for their infra. You can install PipeCD control plane in the same cluster with your application, but keeping ControlPlane separated from your applications cluster is the key point for PipeCD to support multi-platform deployment.

Also, I feel like the issue of not supporting the Arm architecture and the issue of where the control plane operates are separate problems.

I agree with this, too. What I want to mention here is instead of treating this issue as a bug, we should tag it as an enhancement issue 👍 I mention this point here in case you're starting with PipeCD but only have an Arm cluster; you can prepare another cluster for running the PipeCD control plane while installing the Piped agent to your arm arch application running cluster for testing in the meantime. That was my point 😄

@ponkio-o
Copy link
Contributor Author

I agree with this, too. What I want to mention here is instead of treating this issue as a bug, we should tag it as an enhancement issue 👍

Thank you. If that's the case, I think it's okay for me to handle it as an enhancement as well 👍
However, when I tried it, it did not work correctly with the pipectl quickstart command, so it might be a good idea to include a note in the documentation until full support for ARM is achieved.

@khanhtc1202 khanhtc1202 added kind/enhancement New feature or request and removed kind/bug Something isn't working labels Apr 25, 2024
@khanhtc1202
Copy link
Member

@ponkio-o I updated the label for the issue from bug -> enhancement. Thank you 👍
Also, do you want to propose a fix for this issue? 😄

@ponkio-o
Copy link
Contributor Author

Thank you for change the label! I'm going to investigate this issue.

@ffjlabo
Copy link
Member

ffjlabo commented Apr 26, 2024

@ponkio-o Thank you so much. assigned you 👍

@t-kikuc t-kikuc changed the title PipeCD control plane does not work on Arm architecture cluster Quickstart: PipeCD control plane does not work on Arm architecture cluster Oct 4, 2024
@ponkio-o ponkio-o mentioned this issue Dec 6, 2024
Merged
@ponkio-o ponkio-o mentioned this issue Dec 14, 2024
Merged
@ponkio-o ponkio-o mentioned this issue Dec 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ponkio-o ponkio-o

Labels
area/server dependencies Pull requests that update a dependency file good first issue Good for newcomers kind/enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: added arm64 build ponkio-o/pipecd
3 participants
@ponkio-o @khanhtc1202 @ffjlabo