Fix etcd3 locks to handle improper shutdown

[marshallbrekka]

Problem: Etcd3 locks are not released if shutdown improperly.

Example

After an instance has improperly shutdown, its lock will remain in Etcd

$ ETCDCTL_API=3 /tmp/test-etcd/etcdctl  get /vault/core/lock/ --keys-only=true --prefix=true

/vault/core/lock/49075afbd30ce3c9

/vault/core/lock/7e055af811b4bb53

/vault/core/lock/7e055afbd3165b61

Inspecting them individually shows that two (the alive instances) pending locks have leases (results trimmed for brevity).

$ ETCDCTL_API=3 /tmp/test-etcd/etcdctl get /vault/core/lock/49075afbd30ce3c9 --write-out json

{  
   "kvs":[  
      {  
         "key":"L3ZhdWx0L2NvcmUvbG9jay80OTA3NWFmYmQzMGNlM2M5",
         "lease":5262274727229842377
      }
   ]
}

$ ETCDCTL_API=3 /tmp/test-etcd/etcdctl get /vault/core/lock/7e055afbd3165b61 --write-out json

{  
   "kvs":[  
      {  
         "key":"L3ZhdWx0L2NvcmUvbG9jay83ZTA1NWFmYmQzMTY1YjYx",
         "lease":9080764261287222113
      }
   ]
}

While the third (the dead instance) owns the lock, but does not have a lease applied.
$ ETCDCTL_API=3 /tmp/test-etcd/etcdctl get /vault/core/lock/7e055af811b4bb53 --write-out json

{  
   "kvs":[  
      {  
         "key":"L3ZhdWx0L2NvcmUvbG9jay83ZTA1NWFmODExYjRiYjUz",
         "value":"MDgxZWNlZDctOThiMC01ZmRmLTZlODMtMzE4NTNiZjdkYzI5"
      }
   ]
}

Solution: Write the lock item with the lease.

Looking at the internals of the Mutex.Lock method shows that all that is required is to add the lease with the put operation

[jefferai]

Pinging @xiang90 for a review

[xiang90]

@jefferai This pr changes the default lock timeout from 60s to 15s. Is this consistent with other backends? If yes, then I am ok with it.

[xiang90]

etcd default lease duration is 60s. set to 15s for faster recovery.

[xiang90]

etcd3LockTimeoutInSeconds

[xiang90]

LGTM after addressing the minor issues.

[jefferai]

@xiang90 Making it shorter makes sense to me if it's generally stable.

[xiang90]

@jefferai OK. Then 15 seconds should be good enough.

[jefferai]

Seems this is ready to merge?

[xiang90]

LGTM. I cannot click the button. :P

[jefferai]

@xiang90 Hah, just wanted to make sure that you were fine with the changes. Probably should have just requested a review :-) Thanks!