gazctl: apply and edit take optional --max-txn-size

[jgraettinger]

When applying very large updates, we run into Etcd's limit on the number of maximum keys in a single transaction.

That's because an apply is performed as a single Etcd transaction, meaning that either the entire change is made, or none of it is.

Etcd does expose this setting as a flag, and on the one hand the operator could elect to ramp this up to a sufficiently large value. That leaves intact the transactionality of gazctl apply.

However, this is awkward and there will be reasonable YAML specs which violate this limit no matter how large it is.

Another option is for gazctl apply to break up the total set of changes into a number of smaller RPCs / etcd transactions.

I think we let the user decide, by exposing a flag argument to control this behavior. If left at zero, the default, gazctl should attempt to apply the entire change as a single transaction, and return a contextualized etcd error if that violates the current server limit. Otherwise, gazctl should apply changes iteratively with at most --max-txn-size specs per RPC.

[jskelcy]

I am thinking about this option and the case where there are some set of changes lined up and a change is landed in between them, which should be evident because the revision number has jumped by more than 1, and in this case, the apply/edit is abandoned. In this scenario we tell the user they need to attempt the command again after verifying that they still want the change. Does that seem too cumbersome to anyone?

[jgraettinger]

which should be evident because the revision number has jumped by more than 1,

We can't assume anything at all about this case. It could be a completely unrelated update, or a valid reaction to the apply "chunk" that just completed (eg, replication factor was increased).

I don't believe the apply should be abandoned unless a chunk fails to apply. Each apply chunk already encodes the expected ModRevision values of each of the keys it's updating. If any have been updated in the meantime, the chunk will fail.