TTL=0 too soon #6978
Assignees
Labels
Comments
|
It's a cluster of 3 etcd machines, here are the logs for the three machines at the time this happened: Server 3/3 application logs: Server 3/3 etcd logs: Server 2/3 etcd logs: Server 1/3 etcd logs: |
|
For reference, the three servers are located in different EC2 regions. They have GP SSD drives. I haven't had time yet to work out why I am getting warnings about apply entries taking too long, I'm not an etcd expert. Not sure if this information was relevant. |
|
@immesys https://github.com/coreos/etcd/blob/master/Documentation/faq.md#what-does-the-etcd-warning-apply-entries-took-too-long-mean explains why you see |
|
@immesys what timezone are your other etcd servers live in? are they in same timezone? by default, etcd server side logging uses local timezone. |
|
@immesys could you possibly provide a simpler script that I can reproduce the issue locally? |
|
Sure I can try make one. I haven't had a chance to triple check my timezones but I usually keep all my servers in UTC so I would be surprised if the logs did not line up. |
|
I was able to reproduce this with the stock 3-member goreman cluster on both 3.0.15 and master using tmpfs: package main
import (
"fmt"
"github.com/coreos/etcd/clientv3"
"golang.org/x/net/context"
)
func stress(s string) {
cfg := clientv3.Config{Endpoints: []string{s}}
cli, err := clientv3.New(cfg)
if err != nil {
fmt.Println(err)
return
}
i := 0
for {
resp, gerr := cli.Grant(context.TODO(), 60)
if gerr != nil {
panic(gerr)
}
_, kerr := cli.KeepAliveOnce(context.TODO(), resp.ID)
if kerr != nil {
panic(kerr)
}
cli.Revoke(context.TODO(), resp.ID)
fmt.Println(s, i)
i++
}
}
func main() {
for i := 0; i < 10; i++ {
go stress("http://localhost:2379")
}
stress("http://localhost:2379")
}I could not reproduce it with a single node cluster. |
|
@heyitsanthony Do you know if I could switch to a different version in the meantime? Is this present in the 3.1 rc? |
|
@immesys nope, this case looks all around broken, sorry :( Now, this isn't a true fix, but since the cause may take some time to figure out: is there a reason why that code must immediately call KeepAliveOnce? I think this is a race isolated to creating renew streams. Another way to structure the loop without the KeepAliveOnce: kactx, kcancel := context.WithCancel()
defer kcancel()
if _, err := cli.KeepAlive(kactx, leaseId); err != nil { panic(err) }
for _, ev := range resp.Kvs { ....}
cli.Revoke(context.Background(), leaseId)This is a lot cheaper because it'll use a single renew stream per client and it keeps a timer so it doesn't have to send a keep alive message for every loop iteration. Even if it doesn't patch over the problem (it seems to mask it in the stress test), it'll still save some bandwidth. |
|
@heyitsanthony Thanks! I'll change my code to do that. |
|
I added some server side debugging for this issue by running @heyitsanthony's script. The problem only occurs if the client talks to a follower not a leader. In that way, the follower must redirect client (url: http://localhost:22379): Server: |
|
The issue does not just affect the |
fanminshi
added a commit
to fanminshi/etcd
that referenced
this issue
Dec 15, 2016
…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978
fanminshi
added a commit
to fanminshi/etcd
that referenced
this issue
Dec 15, 2016
…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978
fanminshi
added a commit
to fanminshi/etcd
that referenced
this issue
Dec 15, 2016
…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978
fanminshi
added a commit
to fanminshi/etcd
that referenced
this issue
Dec 16, 2016
…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978
fanminshi
added a commit
to fanminshi/etcd
that referenced
this issue
Dec 16, 2016
…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978
fanminshi
added a commit
to fanminshi/etcd
that referenced
this issue
Dec 16, 2016
…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978
fanminshi
added a commit
to fanminshi/etcd
that referenced
this issue
Dec 16, 2016
…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978
fanminshi
added a commit
to fanminshi/etcd
that referenced
this issue
Dec 22, 2016
…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978
|
@immesys I pushed out a fix for this issue. If you checkout the latest version from etcd master branch, it should solve this problem. Please reach out if this fix doesn't solve this problem on your side. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
via irc,
Version: 3.0.15
Code: https://github.com/immesys/wdd/blob/dev/actionRunner.go#L237-L253
Client-side logs:
So it's triggering the TTL == 0 path in the client to report the lease is gone. However, given that there's a minute-length TTL, the TTL shouldn't be 0 so soon.
About 1/1000 chance of happening.
The text was updated successfully, but these errors were encountered: