proxy: V system leasing

[heyitsanthony]

Suppose a client starts spamming the etcd cluster with Puts to a key with many watchers. This will cause a lot of network traffic and considerable server load. Likewise, suppose a client keeps requesting the same serialized ranges / txns for keys that it just wrote out, also causing unnecessary network traffic. What to do?

V system leasing reduces the communication overhead of these situations. This can be implemented on top of etcd3 through a proxy. A sketch and some remarks:

All clients go through a local proxy. When a client writes to a fresh key the proxy will grant itself a lease with an attached lease key. The presence of the lease key locks the other proxies from immediately sending updates to the key to their clients (updates are still posted to all clients connected to the proxy that owns the lease). When the key is updated, the proxy will keep-alive the lease. If the lease expires, the proxy loses its claim on that key. When a proxy services a write to a key with a foreign lease, it will transactionally delete the foreign lease key and create a lease key for itself. Starvation is avoided by periodically refreshing any cached keys / updating gated watchers.

Watch optimizations:

1. Watchers on rev=0 that have not yet received an event can get the revision after the write spam. (this can only be done per-stream because the create header revision can't be faked easily)
2. Watchers that have a set revision still have to receive the spam events, but gating the writes until after the lease expires will makes it possible to benefit from event batching. Alternatively, there could be a "coalescing" flag for watchers where it can drop events on a key in favor of the most recent one.

Serialized Range caching:
When a proxy fetches a key, it also fetches its lease key (or creates a new lease key, if none exists). The key is then used for the rev=0 serialized range cache. If the lease key expires, the key is discarded as stale.

/cc @HardySimpson

[heyitsanthony]

Fixed by #8341