"failed to verify flags" Error occurs with --listen-peer-urls option

[UedaTakeyuki]

Hello,

I'm trying to make a etcd cluster with this tutorial with following script:

# for etcd cluster
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=parsifal
NAME_2=d
NAME_3=c
HOST_1=${NAME_1}.uedasoft.com
HOST_2=${NAME_2}.uedasoft.com
HOST_3=${NAME_3}.uedasoft.com
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For this node
THIS_NAME=$(hostname)
THIS_IP=${THIS_NAME}.uedasoft.com

# start this node
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
        --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

When I run this on the server parsifal, error occurs as follows:

ueda@parsifal:~/etcd$ ./etcd
{"level":"info","ts":"2023-11-08T13:53:45.317738+0900","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--data-dir=data.etcd","--name","parsifal","--initial-advertise-peer-urls","http://parsifal.uedasoft.com:2380","--listen-peer-urls","http://parsifal.uedasoft.com:2380","--advertise-client-urls","http://parsifal.uedasoft.com:2379","--listen-client-urls","http://parsifal.uedasoft.com:2379","--initial-cluster","parsifal=http://parsifal.uedasoft.com:2380,d=http://d.uedasoft.com:2380,c=http://c.uedasoft.com:2380","--initial-cluster-state","new","--initial-cluster-token","token-01"]}
{"level":"warn","ts":"2023-11-08T13:53:45.318111+0900","caller":"etcdmain/etcd.go:75","msg":"failed to verify flags","error":"expected IP in URL for binding (http://parsifal.uedasoft.com:2380)"}

But, when I comment out --listen-peer-urls option, it work well

How should I fix my script. All suggestions are welcome! Thank you!

[ahrtr]

@UedaTakeyuki Can you write a simple golang program to parse http://parsifal.uedasoft.com:2380 using https://pkg.go.dev/net/url#Parse?

[UedaTakeyuki]

Hi ahrtr, thank you for your feedback.

I've tried as follow:

package main

import (
	"fmt"
	"log"
	"net/url"
)

func main() {
	u, err := url.Parse("http://parsifal.uedasoft.com:2380")
	if err != nil {
		log.Fatal(err)
	}
	rel, err := u.Parse("/foo")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(rel)
	_, err = u.Parse(":foo")
	if _, ok := err.(*url.Error); !ok {
		log.Fatal(err)
	}
}

The result is as follow:

ueda@parsifal:~/parse$ go run main.go
http://parsifal.uedasoft.com:2380/foo

[ahrtr]

--listen-peer-urls and --listen-client-urls will reject domain name for the network interface binding.

References:

• https://etcd.io/docs/v3.5/op-guide/clustering/#bootstrap-the-etcd-cluster-using-dns
• Validation error: expected IP in URL for binding etcd#9575

--listen-client-http-urls and --listen-metrics-urls also reject domain name for the network interface binding; but it isn't covered by the doc. So we need to update the doc

[UedaTakeyuki]

Thank you ahrtr!

I've changed parsifal.uedasoft.com to 163.172.156.111, then error is changed as follow:

{"level":"fatal","ts":"2023-11-08T20:27:45.860446+0900","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"listen tcp 163.172.156.111:2380: bind: cannot assign requested address","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:250"}

But, it seems that there are no blocking process of port 2380 as follow:

ueda@parsifal:~/etcd$ lsof -i:8380 -P 

Is there still something wrong with my step? Thank you for your pointing.

[ahrtr]

listen tcp 163.172.156.111:2380: bind: cannot assign requested address

Please ensure,

• The IP is correct;
• The port isn't used by other process

[UedaTakeyuki]

The global IP address is correct.

ueda@parsifal:~/etcd$ curl globalip.me
163.172.156.111

Port isn't used by other process.

ueda@parsifal:~/etcd$ lsof -i:2380 -P 
ueda@parsifal:~/etcd$ 

[ahrtr]

Can you write a simple program to listen on 163.172.156.111:2380 using net.Listen ?

FYI. https://github.com/etcd-io/etcd/blob/649b3659379c780ffa8f44723ac3410e1b922cfd/client/pkg/transport/listener.go#L109

[UedaTakeyuki]

Hi, I've written as follows:

package main

import(
	"net"
	"log"
)

func main(){
	_, err := net.Listen("tcp", ":2380")
	if err != nil {
		log.Println(err)
	} else {
		log.Println("listen")
	}
}

The Listen seems working as follows:

ueda@parsifal:~/parse$ go run main2.go
2023/11/09 07:46:41 listen

[UedaTakeyuki]

Hi, good news!

Finally, after trial and error, I've succeeded in running well with the following settings:

• --listen-peer-urls: INADDR_ANY
• --listen-client-urls: INADDR_ANY
• --advertise-client-urls: own global address

A reference script is as follow:

#!/bin/bash

# for etcd cluster
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=parsifal
NAME_2=d
NAME_3=c
HOST_1=${NAME_1}.uedasoft.com
HOST_2=${NAME_2}.uedasoft.com
HOST_3=${NAME_3}.uedasoft.com
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For this node
THIS_NAME=$(hostname)
THIS_IP=$(curl globalip.me)
INADDR_ANY=0.0.0.0

# start this node
etcd --data-dir=data.etcd --name ${THIS_NAME} --log-level info \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
        --listen-peer-urls http://${INADDR_ANY}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${INADDR_ANY}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

[ahrtr]

_, err := net.Listen("tcp", ":2380")

I asked you to try 163.172.156.111:2380 instead of ":2380", to double confirm that you will fail on 163.172.156.111:2380.

INADDR_ANY=0.0.0.0

Yes, it works. But it may not the best practice. You will expose your service on all interface, we should try to minimize the attack surface. Of course, it's fine if you are sure it's safe & expected. Usually it's recommended to use addresses something like --listen-client-urls http://10.0.1.10:2379,http://127.0.0.1:2379.

FYI. https://etcd.io/docs/v3.5/op-guide/clustering/

[UedaTakeyuki]

Thank you for your kind response.

With 163.172.156.111:2380 as

package main

import(
	"net"
	"log"
)

func main(){
	_, err := net.Listen("tcp", "163.172.156.111:2380")
	if err != nil {
		log.Println(err)
	} else {
		log.Println("listen")
	}
}

It returns same error as:

ueda@parsifal:~/parse$ go run main2.go
2023/11/09 17:01:14 listen tcp 163.172.156.111:2380: bind: cannot assign requested address

Also thanks for teaching me about best practices. I noticed that using INADDR_ANY for client listing is so dangerous. --listen-client-urls should be local loopback (127.0.0.1:2379).
Is it ok to use INADDR_ANY for --listen-peer-urls? I guess --listen-peer-urls is open only for the peers who are nodes of this cluster.

[ahrtr]

Is it ok to use INADDR_ANY for --listen-peer-urls?

Not really. Minimizing the attack surface should be a generic rule, no matter it's for client or peer communication.

[jmhbnz]

Hey @ahrtr - I note we state:

Since 3.2 (3.1 prints warnings) --listen-peer-urls and --listen-client-urls will reject domain name for the network interface binding.

In https://etcd.io/docs/v3.5/op-guide/clustering/#bootstrap-the-etcd-cluster-using-dns.

So it looks like this is actually covered by the documentation. Can we close this issue?

[ahrtr]

Yes, closing...