Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

One of three nodes revision inconsistency cause the node compaction abnormal #16106

Closed
2 of 4 tasks
xrkCode opened this issue Jun 19, 2023 · 4 comments
Closed
2 of 4 tasks

Comments

@xrkCode
Copy link

xrkCode commented Jun 19, 2023

Bug report criteria

What happened?

1、one node db size is full, achieve 2g, the log is
_ util.go:129: Warning: failed to apply request "header:<ID:3842561065849928906 username:"" auth_revision:1 > put:<key:"" value_size:313 lease:3842561065759985782 >" with response "size:8" took (147.544µs) to execute, err is etcdserver: no space_

2、the compact is abnormal,the log is
util.go:129: Warning: failed to apply request "header:<ID:3842561065849944060 > compaction:<revision:385317169 > " with response "" took (9.894µs) to execute, err is mvcc: required revision is a future revision

3、the revision of three node is inconsistency,two of the three is same, the remaining abnormal node is different,In addition, the mod_revision of the same key is different.

What did you expect to happen?

the three node revision is the same, the compact is normal, db size is not full.

How can we reproduce it (as minimally and precisely as possible)?

No triggers have been found so far.
In addition, etcd does not restart during the problem include abnormal node

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here
VERSION:
	3.4.10
$ etcdctl version
# paste output here

VERSION:
3.4.10
API VERSION:
3.4

Etcd configuration (command line flags or environment variables)

paste your configuration here

auto-compaction-mode: 'periodic'
auto-compaction-retention: '30m'
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

@jmhbnz
Copy link
Member

jmhbnz commented Jun 19, 2023

Hey @xrkCode - Thanks for raising this issue. Can you please clarify what you mean by No triggers have been found so far. Are you saying that you cannot reproduce this issue?

Please provide clear steps we can follow to reproduce the issue.

@xrkCode
Copy link
Author

xrkCode commented Jun 20, 2023

Hey @xrkCode - Thanks for raising this issue. Can you please clarify what you mean by No triggers have been found so far. Are you saying that you cannot reproduce this issue?

Please provide clear steps we can follow to reproduce the issue.

Hey, Thank you for your reply.
Currently, this problem occurs only once. We are trying to construct an exception to reproduce the problem. However, the problem has not been reproduced yet.

@tjungblu
Copy link
Contributor

@xrkCode I don't want to dismiss this issue, but 3.4.10 seems fairly old. Did you try upgrading to a newer version?

IIRC #13766 was only in 3.5, #11651 might be applicable - are you using auth?

@jmhbnz
Copy link
Member

jmhbnz commented Jul 26, 2023

Hey @xrkCode - Thank you for raising this. Until we can validate a bug is present in a supported version this support issue will be moved to our Discussion Forums.

We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to requests. Members of the etcd community use Discussion Forums to manage support requests.

Again, thanks for using etcd and raising this discussion.

The etcd team

@jmhbnz jmhbnz closed this as completed Jul 26, 2023
@jmhbnz jmhbnz converted this issue into discussion #16304 Jul 26, 2023

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Development

No branches or pull requests

3 participants