Support NetAPP trident on ARM architecture

[khatrig]

Describe the solution you'd like
NetApp Trident doesn't support ARM architecture today. This is a big limitation in choosing Netapp trident as a CSI solution as most kubernetes CSI providers today support ARM.

Describe alternatives you've considered
Most alternative CSI providers support ARM today.

Additional context
https://scaleoutsean.github.io/2021/02/24/netapp-trident-on-arm64.html

[mreamy]

I would also like to see support for ARM as well as other architectures.
We have multiple development environments running Red Hat Openshift on ARM, IBM Power, and IBM S390 where we would like to use NetrApp Trident for provisioning.

[gnarl]

Hi @khatrig and @mreamy,

Trident can be built from source to work on the ARM architecture and is known to work.

We will keep this issue open and update it once Trident officially supports ARM architecture.

[vveeregandha]

I have tried following steps to build the trident image from the source for ARM

1. Downloaded trident v20.01.0 source code archive, made a change in variable GOARCH to arm64 as suggested in Support for kubernetes on arm64/aarch64  #298, and did a make trident_build
2. Downloaded csi-node-driver-registrar v2.1.0 source code archive and made a change in BUILD_PLATFORMS="linux arm64" as sugested in here - https://github.com/kubernetes-csi/node-driver-registrar/blob/master/release-tools/build.make#L134
   and also multi-platform is supported from the v2.0 version only as per Multi-Arch build kubernetes-csi/node-driver-registrar#48 (comment)

After building and pushing the images to our registry, tested them by creating a new daemon set for arm servers with new images. The trident pod is coming up fine.
But when trying a test job in which the pod tries to mount the trident PVC it is failing with this error -

Normal SuccessfulAttachVolume 20s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-*"
Warning FailedMount 2s (x6 over 19s) kubelet MountVolume.SetUp failed for volume "pvc-" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = error mounting NFS volume 10.7.240.244:/trident_pvc_*** on mountpoint /var/lib/kubelet/pods/ea842b1b-3534-43e5-8a83-efcef63d41b0/volumes/kubernetes.io~csi/pvc-/mount: fork/exec /netapp/mount: exec format error

These are errors from trident container:

level=error msg="could not read transactions" error="Unable to find key"
level=info msg="Listening for GRPC connections." name=/plugin/csi.sock net=unix
level=error msg="Error gathering initiator names."
level=error msg="Could not get iSCSI initiator name." error="fork/exec /netapp/cat: exec format error"
level=warning msg="Mkdir failed." error="fork/exec /netapp/mkdir: exec format error"

Any quick pointers on what the issue might be or if something is missed?

[crossond]

The trident error messages are sometimes occurring during startup and node discovery, (ie not too concerning if Trident pods are Running.)
Quick search on the mountpoint warning "level=warning msg="Mkdir failed." error="fork/exec /netapp/mkdir: exec format error" results show it could be caused by an old Docker or container runtime version, or processor architecture mismatch.

[andymain72]

Is there any timeline by which Trident will provide ARM support?

[scaleoutsean]

mountpoint /var/lib/kubelet/pods/ea842b1b-3534-43e5-8a83-efcef63d41b0/volumes/kubernetes.io~csi/pvc-/mount: fork/exec /netapp/mount: exec format error

You're clearly using an x86 executable.

As mentioned here you need to replace all amd64 base with arm64 base images. Which is easy to check with docker image inspect after the images have been built (or downloaded if you get them from elsewhere).

sean@k2:~$ docker images | grep trident
trident                                              22.01.0-custom    cc27cecc98e2   3 months ago    169MB
k1:5000/trident                                      v22.01.0-custom   cc27cecc98e2   3 months ago    169MB
netapp/trident-autosupport                           22.01             1156a97fc139   5 months ago    79.7MB
scaleoutsean/trident-arm64                           v21.07.0-custom   614576a19f02   11 months ago   131MB
k1:5000/trident                                      v21.07.0-custom   614576a19f02   11 months ago   131MB
k1:5000/v2/trident                                   v21.07.0-custom   614576a19f02   11 months ago   131MB

sean@k2:~$ docker image inspect cc27cecc98e2 | grep Arch
        "Architecture": "arm64",

sean@k2:~$ docker image inspect 1156a97fc139 | grep Arch
        "Architecture": "amd64",

^ This (the second image inspect) is also why I eliminate trident-autosupport from deployment (by using custom deployment YAML) - as I said in the blog post it won't impact anything but it can't work so no need to install it on ARM64 workers.

If you have a test environment where you can use untrusted images, try to pull my older image from Docker Hub and use custom deploy to deploy it. I've been using Trident on ARM64 for close to 18 months now, on my home ARM64 cluster.

[gnarl]

Hi @khatrig and @andymain72,

We are looking into how to best qualify ARM support in Trident. We do not have a timeline for when this support might be available at this time.

[vveeregandha]

mountpoint /var/lib/kubelet/pods/ea842b1b-3534-43e5-8a83-efcef63d41b0/volumes/kubernetes.io~csi/pvc-/mount: fork/exec /netapp/mount: exec format error

You're clearly using an x86 executable.

As mentioned here you need to replace all amd64 base with arm64 base images. Which is easy to check with docker image inspect after the images have been built (or downloaded if you get them from elsewhere).

sean@k2:~$ docker images | grep trident
trident                                              22.01.0-custom    cc27cecc98e2   3 months ago    169MB
k1:5000/trident                                      v22.01.0-custom   cc27cecc98e2   3 months ago    169MB
netapp/trident-autosupport                           22.01             1156a97fc139   5 months ago    79.7MB
scaleoutsean/trident-arm64                           v21.07.0-custom   614576a19f02   11 months ago   131MB
k1:5000/trident                                      v21.07.0-custom   614576a19f02   11 months ago   131MB
k1:5000/v2/trident                                   v21.07.0-custom   614576a19f02   11 months ago   131MB

sean@k2:~$ docker image inspect cc27cecc98e2 | grep Arch
        "Architecture": "arm64",

sean@k2:~$ docker image inspect 1156a97fc139 | grep Arch
        "Architecture": "amd64",

^ This (the second image inspect) is also why I eliminate trident-autosupport from deployment (by using custom deployment YAML) - as I said in the blog post it won't impact anything but it can't work so no need to install it on ARM64 workers.

If you have a test environment where you can use untrusted images, try to pull my older image from Docker Hub and use custom deploy to deploy it. I've been using Trident on ARM64 for close to 18 months now, on my home ARM64 cluster.

@scaleoutsean I followed the steps from here and tried a build, but the arch is still showing up as amd64.

Steps I followed:

1. wget https://github.com/NetApp/trident/archive/v21.01.1.tar.gz
2. build using: sudo GOOS=linux GOARCH=arm64 make trident_build
3. Also tried changing the Makefile to GOARCH ?= arm64, but no luck.

$ docker image inspect 7dfa4ef9c642 | grep Arch
"Architecture": "amd64",

Not sure if there is a prerequisite I am missing before the build.

[scaleoutsean]

@vveeregandha are you building on x86_64 or ARM64 (natively)? On x86_64 you need a functioning cross-patform build toolkit.

Secondly, have you got that suggestion in the steps to hard-code ARM64 into the Dockerfiles? I did that for all Dockerfiles in Trident source tree:

$ find . -name "Dockerfile"
./contrib/docker/plugin/Dockerfile
./operator/Dockerfile
./Dockerfile

$ cat Dockerfile 
# FROM gcr.io/distroless/static:a9b705d9ff06e9c60f6ceb2d2ef99dab084c17c0
# replace the above with something like this:
FROM gcr.io/distroless/static:latest-arm64

(Note: it goes without saying, for best effect you should use the same version, but for arm64, if you can)

And then for offline install you need to pull an ARM64 version of 3rd party images (depending on your version, see the Trident docs; for old docs here's an example for v21.01).

I used to build on ARM64, but now I've tried on x86_64. I've downloaded the source, replaced Dockerfiles as per above, and built (without sudo, because my user is a member of docker group).

sean@scaleoutSean  /tmp/trident-21.01.1  uname -a
Linux scaleoutSean 5.15.0-43-generic #46~20.04.1-Ubuntu SMP Thu Jul 14 15:20:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

sean@scaleoutSean  /tmp/trident-21.01.1  GOOS=linux GOARCH=arm64 make trident_build
fatal: not a git repository (or any of the parent directories): .git
Unable to find image 'golang:1.14' locally
1.14: Pulling from library/golang
0ecb575e629c: Pull complete 
7467d1831b69: Pull complete 
feab2c490a3c: Pull complete 
f15a0f46f8c3: Pull complete 
1517911a35d7: Pull complete 
48bbd1746d63: Pull complete 
944903612fdd: Pull complete 
Digest: sha256:1a7173b5b9a3af3e29a5837e0b2027e1c438fd1b83bbee8f221355087ad416d6
Status: Downloaded newer image for golang:1.14
go: downloading k8s.io/client-go v0.20.1
go: downloading github.com/google/uuid v1.1.4
go: downloading github.com/container-storage-interface/spec v1.3.0
go: downloading github.com/cenkalti/backoff v2.2.1+incompatible
go: downloading github.com/mitchellh/hashstructure/v2 v2.0.1
go: downloading k8s.io/apimachinery v0.20.1
go: downloading github.com/RoaringBitmap/roaring v0.5.5
go: downloading github.com/cenkalti/backoff/v4 v4.1.0
go: downloading github.com/golang/protobuf v1.4.3
go: downloading google.golang.org/grpc v1.34.1
go: downloading github.com/Azure/azure-sdk-for-go v50.0.0+incompatible
go: downloading github.com/Azure/go-autorest v14.2.0+incompatible
go: downloading github.com/Azure/go-autorest/autorest v0.11.15
go: downloading github.com/sirupsen/logrus v1.7.0
go: downloading k8s.io/api v0.20.1
go: downloading github.com/mitchellh/copystructure v1.0.0
go: downloading github.com/gogo/protobuf v1.3.1
go: downloading github.com/google/gofuzz v1.1.0
go: downloading google.golang.org/genproto v0.0.0-20201110150050-8816d57aaa9a
go: downloading github.com/prometheus/client_golang v1.9.0
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.0.2
go: downloading k8s.io/klog/v2 v2.4.0
go: downloading golang.org/x/sys v0.0.0-20210112091331-59c308dcf3cc
go: downloading google.golang.org/protobuf v1.25.0
go: downloading golang.org/x/net v0.0.0-20201110031124-69a78807bb2b
go: downloading golang.org/x/oauth2 v0.0.0-20210112200429-01de73cf58bd
go: downloading github.com/Azure/go-autorest/tracing v0.6.0
go: downloading gopkg.in/inf.v0 v0.9.1
go: downloading github.com/Azure/go-autorest/logger v0.2.0
go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.8
go: downloading k8s.io/kube-openapi v0.0.0-20201113171705-d219536bb9fd
go: downloading github.com/json-iterator/go v1.1.10
go: downloading github.com/go-logr/logr v0.2.0
go: downloading github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32
go: downloading golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad
go: downloading gopkg.in/yaml.v2 v2.3.0
go: downloading github.com/mitchellh/reflectwalk v1.0.0
go: downloading github.com/gorilla/mux v1.8.0
go: downloading k8s.io/utils v0.0.0-20201110183641-67b214c5f920
go: downloading github.com/zcalusic/sysinfo v0.0.0-20201228191146-22fe1fd10b0f
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading cloud.google.com/go v0.65.0
go: downloading github.com/imdario/mergo v0.3.5
go: downloading golang.org/x/text v0.3.4
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/Azure/go-autorest/autorest/azure/auth v0.5.5
go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
go: downloading github.com/prometheus/common v0.15.0
go: downloading github.com/vishvananda/netlink v1.1.0
go: downloading github.com/googleapis/gnostic v0.4.1
go: downloading github.com/Azure/go-autorest/autorest/azure/cli v0.4.2
go: downloading github.com/docker/go-plugins-helpers v0.0.0-20200102110956-c9a8a2d92ccc
go: downloading github.com/form3tech-oss/jwt-go v3.2.2+incompatible
go: downloading github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df
go: downloading github.com/cespare/xxhash v1.1.0
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369
go: downloading github.com/prometheus/procfs v0.2.0
go: downloading github.com/cespare/xxhash/v2 v2.1.1
go: downloading github.com/hashicorp/golang-lru v0.5.1
go: downloading golang.org/x/term v0.0.0-20201117132131-f5c789dd3221
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/modern-go/reflect2 v1.0.1
go: downloading github.com/dimchansky/utfbom v1.1.0
go: downloading github.com/google/go-cmp v0.5.4
go: downloading sigs.k8s.io/yaml v1.2.0
go: downloading k8s.io/apiextensions-apiserver v0.20.1
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96
go: downloading github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e
go: downloading github.com/evanphx/json-patch v4.9.0+incompatible
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/docker/go-connections v0.4.0
go: downloading github.com/Azure/go-autorest/autorest/to v0.4.0
go: downloading github.com/Azure/go-autorest/autorest/validation v0.3.1
go: downloading github.com/tinylib/msgp v1.1.0
go: downloading github.com/glycerine/go-unsnap-stream v0.0.0-20181221182339-f9677308dec2
go: downloading github.com/golang/snappy v0.0.1
go: downloading github.com/philhofer/fwd v1.0.0
fatal: not a git repository (or any of the parent directories): .git
go: downloading github.com/olekukonko/tablewriter v0.0.4
go: downloading github.com/dustin/go-humanize v1.0.0
go: downloading github.com/spf13/cobra v1.1.1
go: downloading github.com/go-logfmt/logfmt v0.5.0
go: downloading github.com/mattn/go-runewidth v0.0.7
fatal: not a git repository (or any of the parent directories): .git
cp /tmp/trident-21.01.1/bin/trident_orchestrator /tmp/trident-21.01.1/bin/tridentctl .
chwrap/make-tarball.sh /tmp/trident-21.01.1/bin/chwrap chwrap.tar
docker build --build-arg PORT=8000 --build-arg BIN=trident_orchestrator --build-arg CLI_BIN=tridentctl --build-arg K8S="" -t trident:21.01.1-custom --rm .
[+] Building 4.1s (9/9) FINISHED                                                                                           
 => [internal] load build definition from Dockerfile                                                                  0.0s
 => => transferring dockerfile: 592B                                                                                  0.0s
 => [internal] load .dockerignore                                                                                     0.0s
 => => transferring context: 2B                                                                                       0.0s
 => [internal] load metadata for gcr.io/distroless/static:latest-arm64                                                1.6s
 => [1/4] FROM gcr.io/distroless/static:latest-arm64@sha256:98beb44a80e05bded5076a1c57d4866f09f5ae34eb3337f5edc588cb  0.6s
 => => resolve gcr.io/distroless/static:latest-arm64@sha256:98beb44a80e05bded5076a1c57d4866f09f5ae34eb3337f5edc588cb  0.0s
 => => sha256:98beb44a80e05bded5076a1c57d4866f09f5ae34eb3337f5edc588cb2f1d4122 426B / 426B                            0.0s
 => => sha256:66be18d6c7f33f53d1886789be832c9d6de5fcea3d0237a3f9d5799db61c89ea 462B / 462B                            0.0s
 => => sha256:a6f41b961ab0c6744fe9133b4fc2df373903f18ee5bf90f4ca00ff1a8c989173 801.01kB / 801.01kB                    0.4s
 => => extracting sha256:a6f41b961ab0c6744fe9133b4fc2df373903f18ee5bf90f4ca00ff1a8c989173                             0.2s
 => [internal] load build context                                                                                     0.5s
 => => transferring context: 83.34MB                                                                                  0.5s
 => [2/4] COPY trident_orchestrator /                                                                                 0.1s
 => [3/4] COPY tridentctl /bin/                                                                                       0.8s
 => [4/4] ADD chwrap.tar /                                                                                            0.2s
 => exporting to image                                                                                                0.4s
 => => exporting layers                                                                                               0.4s
 => => writing image sha256:0f9b6d4c6afcc6f66c4a488291430ef1986dabfd8f2f1904a6e4e5697c7ab3db                          0.0s
 => => naming to docker.io/library/trident:21.01.1-custom                                                             0.0s
rm trident_orchestrator tridentctl


sean@scaleoutSean  /tmp/trident-21.01.1  docker images | grep trident
trident                                         21.01.1-custom               0f9b6d4c6afc   10 seconds ago   85.7MB
scaleoutsean/trident-arm64                      v22.01.0-custom              cc27cecc98e2   5 months ago     169MB
k1:5000/trident                                 v22.01.0-custom              cc27cecc98e2   5 months ago     169MB


sean@scaleoutSean  /tmp/trident-21.01.1  docker inspect 0f9b6d4c6afc | grep Arch
        "Architecture": "arm64",

[gnarl]

With the Trident v23.04 release, the ARM64 architecture is now supported.

[vveeregandha]

@gnarl I could not find v23.04 in releases. I could only see [v23.01.1] as the latest. When is this expected to be released?

[clintonk]

@gnarl I could not find v23.04 in releases. I could only see [v23.01.1] as the latest. When is this expected to be released?

23.04 == April 2023. So very soon!

[khatrig]

With the Trident v23.04 release, the ARM64 architecture is now supported.

That's great. Thanks.