Got panic in raft.go

[allencloud]

Hi, All,

Today my colleague ran into a panic of docker daemon, I think it is related to swarmkit and etcd/raft.

Here is the log message of docker daemon:

Nov 02 19:04:52 10-10-237-29 dockerd[28586]: panic: runtime error: index out of range
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: goroutine 3396 [running]:
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: panic(0x1a90660, 0xc82000e020)
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: github.com/coreos/etcd/raft.(raft).maybeCommit(0xc821e28240, 0xc822018990)
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/coreos/etcd/raft/raft.go:382 +0x302
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: github.com/coreos/etcd/raft.(raft).removeNode(0xc821e28240, 0x49b7a3e5811c3b72)
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/coreos/etcd/raft/raft.go:842 +0x59
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: github.com/coreos/etcd/raft.(*node).run(0xc821f1a550, 0xc821e28240)
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/coreos/etcd/raft/node.go:330 +0xca4
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: created by github.com/coreos/etcd/raft.RestartNode
Nov 02 19:04:52 10-10-237-29 dockerd[28586]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/coreos/etcd/raft/node.go:215 +0x2e4

Oh in addition, my colleague told me that docker version is 1.12.2.

Here is my thought:

func (r *raft) maybeCommit() bool {
	// TODO(bmizerany): optimize.. Currently naive
	mis := make(uint64Slice, 0, len(r.prs))
	for id := range r.prs {
		mis = append(mis, r.prs[id].Match)
	}
	sort.Sort(sort.Reverse(mis))
	mci := mis[r.quorum()-1]
	return r.raftLog.maybeCommit(mci, r.Term)
}

when len(r.prs) == 0, then mis[r.quorum()-1] got panic.

[tonistiigi]

#1353 updated etcd to v3 that should contain the fix etcd-io/etcd#5366

[aaronlehmann]

Thanks for tracking down the upstream fix, @tonistiigi. Since this has been vendored, I'll close the issue.