Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

member panic when adding to a existing cluster #9720

Closed
C0reFast opened this issue May 11, 2018 · 2 comments
Closed

member panic when adding to a existing cluster #9720

C0reFast opened this issue May 11, 2018 · 2 comments

Comments

@C0reFast
Copy link

we use a modified etcd-operator, when we trying to restore a cluster from a backup, the seed member is running and online,then the operator trying to add a member to the cluster, than the new member crashed, this is happening when the old cluster is auth enabled,if the old cluster's auth is disabled, the backup works fine.
here is the crash logs:

2018-05-11 02:08:01.070911 W | rafthttp: lost the TCP streaming connection with peer 22611789f2c6080c (stream Message reader)
2018-05-11 02:08:01.070933 E | rafthttp: failed to read 22611789f2c6080c on stream Message (context canceled)
2018-05-11 02:08:01.070942 I | rafthttp: peer 22611789f2c6080c became inactive
2018-05-11 02:08:01.070955 I | rafthttp: stopped streaming with peer 22611789f2c6080c (stream Message reader)
2018-05-11 02:08:01.070974 I | rafthttp: stopped peer 22611789f2c6080c
2018-05-11 02:08:01.070994 I | rafthttp: removed peer 22611789f2c6080c
2018-05-11 02:08:01.071003 I | etcdserver: finished removing old peers from network
2018-05-11 02:08:01.071017 I | etcdserver: adding peers from new cluster configuration into network...
2018-05-11 02:08:01.071044 I | rafthttp: starting peer 22611789f2c6080c...
2018-05-11 02:08:01.071068 I | rafthttp: started HTTP pipelining with peer 22611789f2c6080c
2018-05-11 02:08:01.072324 I | rafthttp: started streaming with peer 22611789f2c6080c (writer)
2018-05-11 02:08:01.072734 I | etcdserver: finished closing old backend
2018-05-11 02:08:01.074462 I | rafthttp: started peer 22611789f2c6080c
2018-05-11 02:08:01.074518 I | rafthttp: added peer 22611789f2c6080c
2018-05-11 02:08:01.074536 I | etcdserver: finished adding peers from new cluster configuration into network...
2018-05-11 02:08:01.074556 I | etcdserver: finished applying incoming snapshot at index 0
2018-05-11 02:08:01.074659 I | rafthttp: started streaming with peer 22611789f2c6080c (writer)
2018-05-11 02:08:01.074847 I | rafthttp: started streaming with peer 22611789f2c6080c (stream MsgApp v2 reader)
2018-05-11 02:08:01.075240 I | rafthttp: started streaming with peer 22611789f2c6080c (stream Message reader)
2018-05-11 02:08:01.076213 I | etcdserver: published {Name:test-etcd-g6n7p2hrt4 ClientURLs:[http://test-etcd-g6n7p2hrt4.test-etcd.test.svc:2379]} to cluster a7af61399c989d31
2018-05-11 02:08:01.076305 I | embed: ready to serve client requests
2018-05-11 02:08:01.077354 N | embed: serving insecure client requests on 0.0.0.0:2379, this is strongly discouraged!
2018-05-11 02:08:01.078375 I | rafthttp: peer 22611789f2c6080c became active
2018-05-11 02:08:01.078420 I | rafthttp: established a TCP streaming connection with peer 22611789f2c6080c (stream MsgApp v2 reader)
2018-05-11 02:08:01.079031 I | rafthttp: established a TCP streaming connection with peer 22611789f2c6080c (stream Message reader)
2018-05-11 02:08:01.173180 I | rafthttp: established a TCP streaming connection with peer 22611789f2c6080c (stream Message writer)
2018-05-11 02:08:01.173622 I | rafthttp: established a TCP streaming connection with peer 22611789f2c6080c (stream MsgApp v2 writer)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9220c7]

goroutine 177 [running]:
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*simpleTokenTTLKeeper).addSimpleToken(0x0, 0xc42012cfe0, 0x13)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/simple_token.go:60 +0xa7
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*tokenSimple).assignSimpleTokenToUser(0xc420285a80, 0xc4202ac1f8, 0x4, 0xc42012cfe0, 0x13)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/simple_token.go:127 +0x113
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*tokenSimple).assign(0xc420285a80, 0x7f2b1dcd9930, 0xc420290390, 0xc4202ac1f8, 0x4, 0x3, 0xc420290390, 0xc420553360, 0x40d7e8, 0xe4a420)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/simple_token.go:191 +0x26f
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth.(*authStore).Authenticate(0xc42020c780, 0x7f2b1dcd9930, 0xc420290390, 0xc4202ac1f8, 0x4, 0xc4202ac260, 0x10, 0x0, 0x0, 0x0)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/auth/store.go:275 +0x180
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*applierV3backend).Authenticate(0xc4201ae0e8, 0xc420290300, 0xc4205829c0, 0xc42001dad0, 0xc420553558)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:584 +0x259
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*quotaApplierV3).Authenticate(0xc42018e020, 0xc420290300, 0xe7fc80, 0xb6bf01, 0xc420290330)
	<autogenerated>:201 +0x53
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*authApplierV3).Authenticate(0xc420582f80, 0xc420290300, 0xc42001d900, 0xc42001d900, 0xc4201981a0)
	<autogenerated>:93 +0x53
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*applierV3backend).Apply(0xc4201ae0e8, 0xc420ffa0e0, 0xc4200168c0)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:112 +0xc8d
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*quotaApplierV3).Apply(0xc42018e020, 0xc420ffa0e0, 0xc420582fa0)
	<autogenerated>:198 +0x53
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*authApplierV3).Apply(0xc420582f80, 0xc420ffa0e0, 0x0)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply_auth.go:56 +0xed
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntryNormal(0xc4200d58c0, 0xc420553798)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1396 +0x26e
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).apply(0xc4200d58c0, 0xc420214360, 0x1, 0x4, 0xc420012a00, 0x0, 0xc420553840, 0x45ae43)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1319 +0x3d5
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntries(0xc4200d58c0, 0xc420012a00, 0xc420210630)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:957 +0xc6
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyAll(0xc4200d58c0, 0xc420012a00, 0xc420210630)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:819 +0xb1
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).run.func6(0x7f2b1dd192a0, 0xc420582240)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:777 +0x3c
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule.(*fifo).run(0xc420190180)
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:158 +0xdd
created by github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule.NewFIFOScheduler
	/tmp/etcd-release-3.2.19/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:71 +0x1ac
@gyuho
Copy link
Contributor

gyuho commented May 14, 2018

It panics here

https://github.com/coreos/etcd/blob/8a9b3d53857554df347e8b80a42961c88f3dc14f/auth/simple_token.go#L119-L124

This should have been fixed via #8695, which is not backported to v3.2.

Can you try v3.3+?

@C0reFast
Copy link
Author

yeah v3.3.4 works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants