etcd cannot run on arm64

[Zhang-zhixiong]

Hi,
I have installed the etcd3.3.12 (etcd-v3.3.12-linux-arm64.tar.gz)on the Intranet.
I'm trying to start the etcd.service but the Job for etcd.service failed and I have seen “journalctl -xe” for details, and then I got:
etcd[20836]: etcd on unsupported platform without ETCD_UNSUPPORTED_ARCH=arm64 set.
systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start Etcd Server.
Then, I have added the variable by export ETCD_UNSUPPORTED_ARCH=arm64 and I've got the same result.
When I enter the command etcd & , I got this:

2019-04-25 19:46:55.131947 W | etcdmain: running etcd on unsupported architecture "arm64" since ETCD_UNSUPPORTED_ARCH is set
2019-04-25 19:46:55.132284 I | etcdmain: etcd Version: 3.3.12
2019-04-25 19:46:55.132298 I | etcdmain: Git SHA: d57e8b8
2019-04-25 19:46:55.132308 I | etcdmain: Go Version: go1.10.8
2019-04-25 19:46:55.132318 I | etcdmain: Go OS/Arch: linux/arm64

How can I solve it?
May I consider that my operating system is incompatible with etcd3.3.12?
Thanks very much

[jingyih]

How can I solve it?

Are you referring to the warning message?

It is a warning message for not officially supported platform. Please see [1].

Ref:
[1] https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/supported-platform.md#32-bit-and-other-unsupported-systems.

[Zhang-zhixiong]

Thanks.

I've set the environment variable

[root@master ~]# export
declare -x DISPLAY="localhost:10.0"
declare -x ETCD_UNSUPPORTED_ARCH="arm64"
declare -x GOARCH="arm64"

I think it's not a warning, an error occurred while starting the etcd.service

-- Unit etcd.service has begun starting up
Apr 25 19:24:36 master etcd[20836]: etcd on unsupported platform without ETCD_UNSUPPORTED_ARCH=arm64 set. 
Apr 25 19:24:36 master systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE 
Apr 25 19:24:36 master systemd[1]: Failed to start Etcd Server. 
-- Subject: Unit etcd.servcie has failed 
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel 
--
-- Unit etcd.service has failed.

I'm verifying kubernetes on a new operating system and need the etcd server, I wonder is there any way to prove that the operating system is not compatible with it.

[jingyih]

Could you share the etcd log?

[Zhang-zhixiong]

sorry, because we are an Intranet, so the full log cannot be copied out.
I copied some of it, this logs are looped out：

Apr 26 13:28:11 master systemd[1]: Starting Etcd Server...
Apr 26 13:28:11 master systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE 
Apr 26 13:28:11 master systemd[1]: Failed to start Etcd Server. 
Apr 26 13:28:11 master systemd[1]: Unit etcd.service entered failed state.
Apr 26 13:28:11 master systemd[1]: etcd.service failed.
Apr 26 13:28:11 master systemd[1]: etcd.service holdoff time over, scheduling restart.

Is it possible that there is something wrong with my system setup？

[jingyih]

These are not etcd logs. Can you check journald?
https://github.com/etcd-io/etcd/tree/master/contrib/systemd/etcd3-multinode#check-logs

[lubinsz]

@jingyih @mkumatag
Hi, what are the block issues affecting stable on arm64 platform?
I want to remove arm64 from the UNSUPPORTED list, just like ppc64le.
Thanks.

[Zhang-zhixiong]

I restarted the etcd service.

[root@master ~]# systemctl status my-etcd-1.service -l --no-pager
• etcd.service - Etcd Server
    Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
     Active: failed (Result: start-limit) since Sun 2019-04-28 13:37:58 CST; 27s ago
  Process: 61645 ExecStart=/usr/bin/etcd $ETCD_OPTION (code=exited, status=1/FAILURE)
Main PID: 61645 (code=exited, status=1/FAILURE)

Apr 28 13:37:58 master systemd[1]: Failed to start Etcd Server.
Apr 28 13:37:58 master systemd[1]: Unit etcd.service entered failed state.
Apr 28 13:37:58 master systemd[1]: etcd.service failed.
Apr 28 13:37:58 master systemd[1]: etcd.service holdoff time over, scheduling restart.
Apr 28 13:37:58 master systemd[1]: start request repeated too quickly for etcd.service
Apr 28 13:37:58 master systemd[1]: Failed to start Etcd Server.
Apr 28 13:37:58 master systemd[1]: Unit etcd.service entered failed state.
Apr 28 13:37:58 master systemd[1]: etcd.service failed.

And then

[root@master ~]# journalctl - f -u etcd.service --since "2019-04-28 13:30:00" 
-- Logs begin at Sat 2019-04-20 15:51:35 CST. --
Apr 28 13:37:56 master systemd[1]: Starting Etcd Server...
Apr 28 13:37:56 master systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Apr 28 13:37:56 master systemd[1]: Failed to start Etcd Server.
Apr 28 13:37:56 master systemd[1]: Unit etcd.service entered failed state.
Apr 28 13:37:56 master systemd[1]: etcd.service failed.
Apr 28 13:37:57 master systemd[1]: etcd.service holdoff time over, scheduling restart.
Apr 28 13:37:57 master systemd[1]: Starting Etcd Server...
Apr 28 13:37:57 master systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Apr 28 13:37:57 master systemd[1]: Failed to start Etcd Server.
Apr 28 13:37:57 master systemd[1]: Unit etcd.service entered failed state.
Apr 28 13:37:57 master systemd[1]: etcd.service failed.
Apr 28 13:37:57 master systemd[1]: etcd.service holdoff time over, scheduling restart.
Apr 28 13:37:57 master systemd[1]: Starting Etcd Server...
Apr 28 13:37:57 master systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Apr 28 13:37:57 master systemd[1]: Failed to start Etcd Server.
Apr 28 13:37:57 master systemd[1]: Unit etcd.service entered failed state.
Apr 28 13:37:57 master systemd[1]: etcd.service failed.
Apr 28 13:37:57 master systemd[1]: etcd.service holdoff time over, scheduling restart.
Apr 28 13:37:57 master systemd[1]: Starting Etcd Server...
Apr 28 13:37:57 master systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Apr 28 13:37:57 master systemd[1]: Failed to start Etcd Server.
Apr 28 13:37:57 master systemd[1]: Unit etcd.service entered failed state.
Apr 28 13:37:57 master systemd[1]: etcd.service failed.
Apr 28 13:37:58 master systemd[1]: etcd.service holdoff time over, scheduling restart.
Apr 28 13:37:58 master systemd[1]: Starting Etcd Server...
Apr 28 13:37:58 master systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Apr 28 13:37:58 master systemd[1]: Failed to start Etcd Server.
Apr 28 13:37:58 master systemd[1]: Unit etcd.service entered failed state.
Apr 28 13:37:58 master systemd[1]: etcd.service failed.
Apr 28 13:37:58 master systemd[1]: etcd.service holdoff time over, scheduling restart.
Apr 28 13:37:58 master systemd[1]: start request repeated too quickly for etcd.service
Apr 28 13:37:58 master systemd[1]: Failed to start Etcd Server.
Apr 28 13:37:58 master systemd[1]: Unit etcd.service entered failed state.
Apr 28 13:37:58 master systemd[1]: etcd.service failed.

There is no warning message about "unsupported platform... arm64..." in the logs, now I think there is something wrong with the etcd.service file, but I used the same etcd.servcie file on another amd64 platform, and it worked.

[lubinsz]

@Zhang-zhixiong
Can you run the following command in your Arm board ?

etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.iLLHi4BiDj --listen-client-urls http://127.0.0.1:2379 --debug

I have tested it with kubernetes, and the result is OK.
My log:

root@entos-d05:~#   which etcd
/root/go/src/k8s.io/kubernetes/third_party/etcd/etcd
root@entos-d05:~#   etcd --version
2019-04-28 06:33:30.545580 W | etcdmain: running etcd on unsupported architecture "arm64" since ETCD_UNSUPPORTED_ARCH is set
etcd Version: 3.3.12
Git SHA: d57e8b8
Go Version: go1.10.8
Go OS/Arch: linux/arm64
root@entos-d05:~#   uname -m
aarch64
root@entos-d05:~#   etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.iLLHi4BiDj --listen-client-urls http://127.0.0.1:2379 --debug
2019-04-28 06:39:28.901214 W | etcdmain: running etcd on unsupported architecture "arm64" since ETCD_UNSUPPORTED_ARCH is set
2019-04-28 06:39:28.901624 I | etcdmain: etcd Version: 3.3.12
2019-04-28 06:39:28.901639 I | etcdmain: Git SHA: d57e8b8
2019-04-28 06:39:28.901651 I | etcdmain: Go Version: go1.10.8
2019-04-28 06:39:28.901663 I | etcdmain: Go OS/Arch: linux/arm64
2019-04-28 06:39:28.901675 I | etcdmain: setting maximum number of CPUs to 64, total number of available CPUs is 64
2019-04-28 06:39:28.901778 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2019-04-28 06:39:28.902276 I | embed: listening for peers on http://localhost:2380
2019-04-28 06:39:28.902322 I | embed: listening for client requests on 127.0.0.1:2379
2019-04-28 06:39:28.903013 I | etcdserver: name = default
2019-04-28 06:39:28.903030 I | etcdserver: data dir = /tmp/tmp.iLLHi4BiDj
2019-04-28 06:39:28.903043 I | etcdserver: member dir = /tmp/tmp.iLLHi4BiDj/member
2019-04-28 06:39:28.903055 I | etcdserver: heartbeat = 100ms
2019-04-28 06:39:28.903066 I | etcdserver: election = 1000ms
2019-04-28 06:39:28.903077 I | etcdserver: snapshot count = 100000
2019-04-28 06:39:28.903105 I | etcdserver: advertise client URLs = http://127.0.0.1:2379
2019-04-28 06:39:28.904827 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 8
2019-04-28 06:39:28.905231 I | raft: 8e9e05c52164694d became follower at term 4
2019-04-28 06:39:28.905263 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 4, commit: 8, applied: 0, lastindex: 8, lastterm: 4]
2019-04-28 06:39:28.927253 W | auth: simple token is not cryptographically signed
2019-04-28 06:39:28.943630 I | etcdserver: starting server... [version: 3.3.12, cluster version: to_be_decided]
2019-04-28 06:39:28.944976 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2019-04-28 06:39:28.945143 N | etcdserver/membership: set the initial cluster version to 3.3
2019-04-28 06:39:28.945198 I | etcdserver/api: enabled capabilities for version 3.3
2019-04-28 06:39:30.205607 I | raft: 8e9e05c52164694d is starting a new election at term 4
2019-04-28 06:39:30.205640 I | raft: 8e9e05c52164694d became candidate at term 5
2019-04-28 06:39:30.205691 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 5
2019-04-28 06:39:30.205725 I | raft: 8e9e05c52164694d became leader at term 5
2019-04-28 06:39:30.205742 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 5
2019-04-28 06:39:30.223518 I | etcdserver: published {Name:default ClientURLs:[http://127.0.0.1:2379]} to cluster cdf818194e3a8c32
2019-04-28 06:39:30.223562 I | embed: ready to serve client requests
2019-04-28 06:39:30.223724 E | etcdmain: forgot to set Type=notify in systemd service file?
INFO: 2019/04/28 06:39:30 dialing to target with scheme: ""
INFO: 2019/04/28 06:39:30 could not get resolver for scheme: ""
2019-04-28 06:39:30.224442 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

[Zhang-zhixiong]

I run the following command:

etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /var/lib/etcd --listen-client-urls http://127.0.0.1:2379 --debug

and the result is OK, the log is same to you

[root@master ~]# netstat -tnlp | grep etcd
tcp          0          0  127.0.0.1:2379          0.0.0.0:*          LISTEN        58478/etcd
tcp          0          0  127.0.0.1:2380          0.0.0.0:*          LISTEN        58478/etcd

the following is my etcd systemd service files:

[root@master ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/usr/bin/etcd --name my-etcd \
    --data-dir /var/lib/etcd \
    --listen-client-urls http://122.71.221.39:2379,http://127.0.0.1:2379 \
    --advertise-client-urls http://122.71.221.39:2379,http://127.0.0.1:2379 \
    --listen-peer-urls http://122.71.221.39:2380 \
    --initial-advertise-peer-urls http://122.71.221.39:2380

[Install]
WantedBy=multi-user.target

but it still couldn't start the etcd.service

[spzala]

@jingyih @mkumatag
Hi, what are the block issues affecting stable on arm64 platform?
I want to remove arm64 from the UNSUPPORTED list, just like ppc64le.
Thanks.

@lubinsz a related checklist here - https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/supported-platform.md#supporting-a-new-system-platform Also a related issue tracking checklist #10318

[Vae1997]

I run the following command:

etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /var/lib/etcd --listen-client-urls http://127.0.0.1:2379 --debug

and the result is OK, the log is same to you

[root@master ~]# netstat -tnlp | grep etcd
tcp          0          0  127.0.0.1:2379          0.0.0.0:*          LISTEN        58478/etcd
tcp          0          0  127.0.0.1:2380          0.0.0.0:*          LISTEN        58478/etcd

the following is my etcd systemd service files:

[root@master ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/usr/bin/etcd --name my-etcd \
    --data-dir /var/lib/etcd \
    --listen-client-urls http://122.71.221.39:2379,http://127.0.0.1:2379 \
    --advertise-client-urls http://122.71.221.39:2379,http://127.0.0.1:2379 \
    --listen-peer-urls http://122.71.221.39:2380 \
    --initial-advertise-peer-urls http://122.71.221.39:2380

[Install]
WantedBy=multi-user.target

but it still couldn't start the etcd.service

Hello! Just now,I face with the same problem when I try to install etcd on arm64.
I also try to export ETCD_UNSUPPORTED_ARCH=arm64 ,but failed.
However,I try to make file in /etc/etcd/etcd.conf and write ETCD_UNSUPPORTED_ARCH=arm64 ,saved.
Then,I change /etc/systemd/system/etcd.service by adding EnvironmentFile=/etc/etcd/etcd.conf at [Service] section,reboot.
Nice,now it work well!

[struxoje]

In my case I changed /etc/default/etcd and added ETCD_UNSUPPORTED_ARCH=arm to the end of the file. I also had to grant etcd account permissions to folder
sudo mkdir -p /var/lib/etcd/default
sudo chown -R etcd:etcd /var/lib/etcd

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[vorpalhex]

I was able to repro this on arm64.

root@apollo:/home# declare -x ETCD_UNSUPPORTED_ARCH="arm64"
root@apollo:/home# apt install -y etcd

● etcd.service - etcd - highly-available key value store
     Loaded: loaded (/lib/systemd/system/etcd.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2020-07-17 20:06:36 UTC; 31ms ago
       Docs: https://github.com/coreos/etcd
             man:etcd
    Process: 88390 ExecStart=/usr/bin/etcd $DAEMON_ARGS (code=exited, status=1/FAILURE)
   Main PID: 88390 (code=exited, status=1/FAILURE)

Jul 17 20:06:36 apollo systemd[1]: Starting etcd - highly-available key value store...
Jul 17 20:06:36 apollo etcd[88390]: etcd on unsupported platform without ETCD_UNSUPPORTED_ARCH=arm64 set.
Jul 17 20:06:36 apollo systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Jul 17 20:06:36 apollo systemd[1]: etcd.service: Failed with result 'exit-code'.
Jul 17 20:06:36 apollo systemd[1]: Failed to start etcd - highly-available key value store.

Confirm that the env exists:

root@apollo:/home# export | grep ARCH
declare -x ETCD_UNSUPPORTED_ARCH="arm64"

[ptabor]

FYI: Bolt data format does not handle endianess properly, i.e. data written on little-endian architecture will not be accessible on big-endian architectures. Make sure you don't transfer data between architectures or use your ARM in little-endian mode.

[JamesCHub]

This may be an extreme edge case, but it tripped me up so sharing if useful to even one other person. I have many pi systems of different vintages, using different images. I picked three to build out an etcd cluster but neglected to do a uname -a on each to confirm that they were each running the same arch. (It didn't help that online tutorials also neglected this step and just made assumptions, but still, my oversight)

All the way from the Go install you have to follow slightly different paths for aarch64/arm64 and arm7/32-bit, which of course sounds obvious in retrospect. The slight trap is that it actually almost/sorta/kinda works when you do it wrong.

aarch64/arm64

Before you even download Go, update .bashrc with

export GOOS="linux"
export GOARCH="arm64"
export CGO_ENABLED=0
export ETCD_UNSUPPORTED_ARCH=arm64

Source that, then, use a script like below to install Go

#!/bin/bash

VERSION=1.21.1

## Download the latest version of Golang
echo "Downloading Go $VERSION"
wget https://dl.google.com/go/go$VERSION.linux-arm64.tar.gz
echo "Downloading Go $VERSION completed"

## Extract the archive
echo "Extracting..."
tar -C ~/.local/share -xzf go$VERSION.linux-arm64.tar.gz
echo "Extraction complete"

## Detect the user's shell and add the appropriate path variables
SHELL_TYPE=$(basename "$SHELL")

if [ "$SHELL_TYPE" = "zsh" ]; then
    echo "Found ZSH shell"
    SHELL_RC="$HOME/.zshrc"
elif [ "$SHELL_TYPE" = "bash" ]; then
    echo "Found Bash shell"
    SHELL_RC="$HOME/.bashrc"
elif [ "$SHELL_TYPE" = "fish" ]; then
    echo "Found Fish shell"
    SHELL_RC="$HOME/.config/fish/config.fish"
else
    echo "Unsupported shell: $SHELL_TYPE"
    exit 1
fi

echo 'export GOPATH=$HOME/.local/share/go' >> "$SHELL_RC"
echo 'export PATH=$HOME/.local/share/go/bin:$PATH' >> "$SHELL_RC"

## Verify the installation
if [ -x "$(command -v go)" ]; then
    INSTALLED_VERSION=$(go version | awk '{print $3}')
    if [ "$INSTALLED_VERSION" == "go$VERSION" ]; then
        echo "Go $VERSION is installed successfully."
    else
        echo "Installed Go version ($INSTALLED_VERSION) doesn't match the expected version (go$VERSION)."
    fi
else
    echo "Go is not found in the PATH. Make sure to add Go's bin directory to your PATH."
fi

## Clean up
rm go$VERSION.linux-arm64.tar.gz

Then do your basic etcd source clone and build

$ git clone -b v3.4.16 https://github.com/etcd-io/etcd.git
$ cd etcd
$ ./build

arm71

Update .bashrc with

export GOOS="linux"
export GOARCH="arm"
export CGO_ENABLED=0
export GOARM=6
export ETCD_UNSUPPORTED_ARCH=arm

Source that, then, use a script like below to install Go

#!/bin/bash

VERSION=1.21.1

## Download the latest version of Golang
echo "Downloading Go $VERSION"
wget https://dl.google.com/go/go$VERSION.linux-armv6l.tar.gz
echo "Downloading Go $VERSION completed"

## Extract the archive
echo "Extracting..."
tar -C ~/.local/share -xzf go$VERSION.linux-armv6l.tar.gz
echo "Extraction complete"

## Detect the user's shell and add the appropriate path variables
SHELL_TYPE=$(basename "$SHELL")

if [ "$SHELL_TYPE" = "zsh" ]; then
    echo "Found ZSH shell"
    SHELL_RC="$HOME/.zshrc"
elif [ "$SHELL_TYPE" = "bash" ]; then
    echo "Found Bash shell"
    SHELL_RC="$HOME/.bashrc"
elif [ "$SHELL_TYPE" = "fish" ]; then
    echo "Found Fish shell"
    SHELL_RC="$HOME/.config/fish/config.fish"
else
    echo "Unsupported shell: $SHELL_TYPE"
    exit 1
fi

echo 'export GOPATH=$HOME/.local/share/go' >> "$SHELL_RC"
echo 'export PATH=$HOME/.local/share/go/bin:$PATH' >> "$SHELL_RC"

## Verify the installation
if [ -x "$(command -v go)" ]; then
    INSTALLED_VERSION=$(go version | awk '{print $3}')
    if [ "$INSTALLED_VERSION" == "go$VERSION" ]; then
        echo "Go $VERSION is installed successfully."
    else
        echo "Installed Go version ($INSTALLED_VERSION) doesn't match the expected version (go$VERSION)."
    fi
else
    echo "Go is not found in the PATH. Make sure to add Go's bin directory to your PATH."
fi

## Clean up
rm go$VERSION.linux-armv6l.tar.gz

etcd build is actually the same - now that we have our Go env vars in place.

$ git clone -b v3.4.16 https://github.com/etcd-io/etcd.git
$ cd etcd
$ ./build

Do whatever you want with the binaries - move them to /usr/bin, /usr/local/bin, leave them as is and add to path - whatever.

The really odd thing is that if you do this wrong, you get this maddening error:
etcd on unsupported platform without ETCD_UNSUPPORTED_ARCH=arm set. which, at least in my case, could be resolved by doing the export in an interactive session, but which could not be cleared in any other way. Not in a service file, not in a crontab, not in a shellscript.

Anyway - it's a weird case - hope I help one person one day.