atomic Compare-and-Swap PrevNoExist, the operation failed,but the key was stored

[yorkart]

etcd: v2.3.6

there is a demo https://github.com/yorkart/etcd-demo
when loop invoke atomic set and delete
after some times get error :

• set error (but value of the key has stored in it)
  set key error: 105: Key already exists (/demo/a) [1533028]
• delete (indeed key has been deleted )
  delete key error: 100: Key not found (/demo/a) [1530634]

[xiang90]

Can you format your code? Also it would be helpful if you can provide the full code block.

[heyitsanthony]

@yorkart I tried to reproduce this but no luck. What is opts? Are you sure that Delete isn't failing so that Set sometimes operates on a key that wasn't deleted?

[yorkart]

@xiang90 @heyitsanthony I have edited issue and submitted the code. Run a few times, can see the above two kinds of errors

[heyitsanthony]

@yorkart I still can't reproduce this bug after looping a few hundred times with that code. What do you mean by "run a few times"? Do the errors show up immediately when the program starts or does it start giving errors in the middle of a run?

[yorkart]

giving errors in the middle of a run

in addition, cluster log frequently prompted

failed to send out heartbeat on time (deadline exceeded for 646.014151ms)
server is likely overloaded

heartbeat config:

ETCD_HEARTBEAT_INTERVAL=1000
ETCD_ELECTION_TIMEOUT=5000

I guess the timeout has led to the operation of the result is not consistent, the cluster has been processed, but to return to the failure to the client

[heyitsanthony]

@yorkart that wouldn't lead to loss of consistency. Are the Set/Delete requests returning time-out errors? etcd shouldn't return success unless the request is committed. Can you please provide the full server log?

[yorkart]

@heyitsanthony no time-out error. As the demo code , sequential execute set , delete
get error only when set is Key already exists, delet is Key not found

I have push the server log
etcd-144
etcd-147
etcd-148

[heyitsanthony]

@yorkart that is a very unhealthy cluster; it's doing a leader election several times a minute. I'll see if I can reproduce under similar conditions. Do you see the same behavior with 3.0?

[yorkart]

v3 is ok . Using the same test logic, only timed out error v3 demo
rpc error: code = 13 desc = etcdserver: request timed out, possibly due to previous leader failure

at this time, cluster get log

raft.node: 7ab00e9f791aa00a lost leader 17047805852fad33 at term 12678
raft.node: 7ab00e9f791aa00a elected leader 85ed6313fba477e3 at term 12678

in addition, when the demo is running, cluster always print log

apply entries took too long [164.283544ms for 1 entries]
avoid queries with large range/delete range!

#5871 - I have the same problem when use v3 client

[heyitsanthony]

@yorkart your etcd cluster has very high latencies which is why it's triggering leader elections. You'll need to increase the ETCD_ELECTION_TIMEOUT and ETCD_HEARTBEAT_INTERVAL to stop the frequent leader elections. The apply entry warning is hardcoded to 10ms; we'll probably change that to a less aggressive value soon.