3.4.10 behavioral change breaking data-dir breaks existing docs

[howardjohn]

The breaking change in 3.4.10 related to --data-dir permissions breaks the getting started documentation found on https://github.com/etcd-io/etcd/releases/tag/v3.4.10.

$ rm -rf /tmp/etcd-data.tmp && mkdir -p /tmp/etcd-data.tmp && docker rmi gcr.io/etcd-development/etcd:v3.4.10 || true && docker run -p 2379:2379 -p 2380:2380 --mount type=bind,source=/tmp/etcd-data.tmp,destination=/etcd-data --name etcd-gcr-v3.4.10 gcr.io/etcd-development/etcd:v3.4.10 /usr/local/bin/etcd --name s1 --data-dir /etcd-data --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://0.0.0.0:2379 --listen-peer-urls http://0.0.0.0:2380 --initial-advertise-peer-urls http://0.0.0.0:2380 --initial-cluster s1=http://0.0.0.0:2380 --initial-cluster-token tkn --initial-cluster-state new --log-level info --logger zap --log-outputs stderr
{"level":"info","ts":"2020-07-18T14:48:37.698Z","caller":"etcdmain/etcd.go:134","msg":"server has been already initialized","data-dir":"/etcd-data","dir-type":"member"}
{"level":"info","ts":"2020-07-18T14:48:37.698Z","caller":"embed/etcd.go:117","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2020-07-18T14:48:37.699Z","caller":"embed/etcd.go:127","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2020-07-18T14:48:37.699Z","caller":"embed/etcd.go:301","msg":"starting an etcd server","etcd-version":"3.4.10","git-sha":"18dfb9cca","go-version":"go1.12.17","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"s1","data-dir":"/etcd-data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/etcd-data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://0.0.0.0:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://0.0.0.0:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":false,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":""}
{"level":"info","ts":"2020-07-18T14:48:37.699Z","caller":"embed/etcd.go:362","msg":"closing etcd server","name":"s1","data-dir":"/etcd-data","advertise-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2020-07-18T14:48:37.699Z","caller":"embed/etcd.go:366","msg":"closed etcd server","name":"s1","data-dir":"/etcd-data","advertise-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://0.0.0.0:2379"]}
{"level":"fatal","ts":"2020-07-18T14:48:37.699Z","caller":"etcdmain/etcd.go:271","msg":"discovery failed","error":"cannot access data directory: directory \"/etcd-data\",\"drwxr-x---\" exist without desired file permission \"-rwx------\".","stacktrace":"go.etcd.io/etcd/etcdmain.startEtcdOrProxyV2\n\t/tmp/etcd-release-3.4.10/etcd/release/etcd/etcdmain/etcd.go:271\ngo.etcd.io/etcd/etcdmain.Main\n\t/tmp/etcd-release-3.4.10/etcd/release/etcd/etcdmain/main.go:46\nmain.main\n\t/tmp/etcd-release-3.4.10/etcd/release/etcd/main.go:28\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}

A similar kubernetes deployment that also broke on upgrade:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: etcd
spec:
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      name: etcd
      labels:
        app: etcd
    spec:
      containers:
      - name: etcd
        image: gcr.io/etcd-development/etcd:v3.4.10
        args:
        - etcd
        - --data-dir=/var/lib/etcd
        - --listen-client-urls=http://0.0.0.0:2379
        - --advertise-client-urls=http://0.0.0.0:2379
        volumeMounts:
          - mountPath: /var/lib/etcd
            name: etcd-data
      volumes:
      - emptyDir: {}
        name: etcd-data

It seems this change is documented under https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.4.md#breaking-changes so I think this is just a doc issue

[howardjohn]

For anyone running into this, I added this to get pod spec to get things working:

      initContainers:
      - name: volume-mount-hack
        image: busybox
        command: ["sh", "-c", "chmod 0700 /var/lib/etcd"]
        volumeMounts:
        - name: etcd-data
          mountPath: /var/lib/etcd

[gyuho]

/cc @spzala

[spzala]

@howardjohn thank you for reporting! Yes, we should fix the broken install doc/command with docker which I think can be fixed by modifying the command in few different ways, 1) set the desired permission when we create dir i.e. rm -rf /tmp/etcd-data.tmp && mkdir -p -m700 /tmp/etcd-data.tmp 2) don't specify the --data-dir and let etcd use the default one 3)specify a --data-dir that is not already created by docker i.e. something other than --data-dir /etcd-data. Per some testing that I did all of these works. I am leaning to the first solution that is to specify permission (mkdir -p -m700 /tmp/etcd-data.tmp). It would be great if you and @gyuho can let me know your thoughts. Thanks!

[prabhakarp-dev]

Just wondering if its expected to have breaking changes within a patch release?

[spzala]

Just wondering if its expected to have breaking changes within a patch release?

afaik, yes, e.g. https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#breaking-changes-2

[spzala]

@howardjohn the release page is updated, so if you can please verify the updated command that would be great. Thanks!

[howardjohn]

@spzala new docs LGTM, feel free to close if this was the only change needed

[spzala]

@howardjohn awesome, thanks much for quickly verifying it!

[jingyih]

For reference, this issue is fixed by #12242