Silenced entries can fail to be deleted if there are many for an entity

[echlebek]

Expected Behavior

Expired silenced entries will be deleted, no matter how many there are.

Current Behavior

{"component":"store","error":"internal error: etcdserver: too many operations in txn request","level":"error","msg":"error deleting expired silenced entries","time":"2021-08-10T20:57:55Z"}

It seems that when the number of silenced entries for an entity pass a certain threshold, trying to delete all of them in a transaction produces an error.

Possible Solution

Batch the operations in smaller amounts.

[portertech]

Seems max operations per transaction is configurable, see etcd-io/etcd#10048

[portertech]

We probably want to increase the default value, when running with embedded etcd.

[tarcinil]

We've noticed this error too.

However, ours is different in that it can sometime cause the actual sensu-backend service to fail starting. We believe that this was caused by someone accidentally silencing everything. We noted that we had over 800 keys in the /sensu.io/silenced keyspace.

You have to carefully increase the etcd --max-txn-ops flag, because if it is too high, you can start reaching context deadline exceeded territory due to transactions taking too long, I noticed this when initializing keepalives for namespace/node.

Caveats:

• We are running an 3 node external etcd 3.5.0 (chef managed).
• We are running a community build at 6.4.0 (chef managed).

[echlebek]

@tarcinil yeah this is an unfortunate side effect of trying to batch silenced operations into transactions. Sensu needs to be patched to use multiple batches instead of trying to do everything in a single txn.