refresh a ttl node

[defunctzombie]

Reference issue: #385 (comment)

I am working on a service registry module on top of etcd and one of the things I really like from etcd is the TTL feature. It seems that there should also be a way to refresh the TTL without updating the value, This would allow the registered service to continue announcing its presence. Right now I can accomplish this using a PUT /v2/keys/foobar?value=<old value>&prevExist=true but that requires that I continue specifying the old value. It would be nice to be able to just refresh a key.

Some apis that might accomplish this:

Simple refresh with no TTL change.

PUT /v2/keys/foobar?refresh=true

Reset the TTL to a new value

PUT /v2/keys/foobar?ttl=30&refresh=true

These would require that the key already exists and would imply prevExist=true.

[ajaygulati]

Thanks for fixing this. This is super useful.
Will the refresh operation fire a watch that is set on the key? It will be good if it doesn't, so that other nodes can watch a key for being alive as long as the value doesn't change.

[ajaygulati]

Also is it possible to know the time by which this feature may be available?

[philips]

@defunctzombie We do not store the original TTL in etcd. The TTL is transformed into a time stamp in the future. So, the API would need to be:

PUT /v2/keys/foobar?refresh=true&ttl=30

We would not be able to support leaving off the TTL.

[defunctzombie]

I assumed the TTL was stored since I seem to get it back when I do a request for a given key.

Would be fine if that had to be specified.

[willejs]

+1 Really want this to happen sooner though!

[nickethier]

👍

[cnelson]

Is it expected that this new operation would or would not affect the modifiedIndex?

I'd prefer not to have all my watchers get triggered when I'm just refreshing a TTL and not changing data. As it stands now, my watchers have to be much more complex than they need to be, as their watches fire everytime I heartbeat a node and update it's ttl.

[ghost]

This is a very exciting feature. I'm looking for a solution for heart beating service in etcd and this is the best one I believe.
I can hardly wait to use this api in v3.0.

[abourget]

Any update on this one ?

[MartinPodval]

Also giving +1 as using ETCD as service registry generates many updates because of TTL refreshing which needs to by filtered as "uninteresting" configuration changes.

[bitglue]

It might be insightful to describe how Zookeeper solves this problem. Zookeeper has ephemeral nodes, which instead of having a TTL, have a lifetime scoped to the TCP connection that created them. When that connection is closed or times out, all the ephemeral nodes it created are destroyed.

This seems pretty elegant, mapping pretty well to the use case of a service registry. I do wonder though what time out semantics Zookeeper guarantees, if any: depending on the host's TCP stack settings, connections might take a very long time to time out. And of course, there's the issue that Etcd in common usage may not involve any persistent connection since that wouldn't be very curlable.

Perhaps it would still be useful to have some other object to which a similar concept of ephemeral nodes can be bound? Maybe my service registration agent could look like this:

PUT some key to register myself as a unique client
    (specify options about timeout)
    (save the auto-generated, guaranteed unique result)
for each service I know about:
    create key(s) registering that service, with ?ephemeral=true
loop forever:
    heartbeat the key I made initially
    sleep for some fraction of the timeout

When the registration agent stops heartbeating and the timeout expires, or explicitly removes the client key, all the ephemeral nodes associated with it are deleted.

A nice advantage here is that etcd client libraries can handle the client key creation and heartbeating orthogonally, so a service registration implementation may not need to actually have any heartbeating logic. If that's not possible (maybe you are just using curl), then the necessary logic isn't much more complicated than the currently necessary logic of updating the service keys before the TTL expires, except that it's clear that watchers won't be notified because you aren't actually updating the service key in any way, and if you have multiple service keys that should go away when the registration agent dies, you don't need to heartbeat more than one key.

[xiang90]

@bitglue

I do not quite like the idea to attach a session to a TCP connection. That means only one TCP connection can maintain that session. The ownership of the session is kind of too fixed. It limites mechanism for ownership transfer and session maintenance by multiple entities.

We do have the plan to make this elegant in etcd.

[mwitkow]

Any update on this one?

We're using Etcd for service discovery, and we have our services periodically refresh their registrations. The same registrations are watched by SkyDNS, and other discovery software for updates.

We currently use PUT /v2/keys/foobar?value=<old value>&prevExist=true&ttl=30 for refreshing the registrations, but a compareAndSet causes all watchers to be notified and reestablish. On large clusters, this can cause really significant load on Etcd (reestablishing watches).

It would be a really nice solution to have PUT /v2/keys/foobar?refresh=true&ttl=30, which would up the TTL of the Node but would not cause watchers to be notified. any chance of getting that in Etcd 2.x? :)

[YanXiaoping]

Any update?

[shafreeck]

I really need this feature on an heavy online envrionment. I'd like to implement it. @xiang90 Is there any docs telling me where can I start?

I noticed there is a branch named dev-2.2. I think this maybe the next release version. Am I right ? I indeed can not wait for v3.0.0-maybe

[adohe-zz]

@shafreeck @YanXiaoping I am afraid this feature is still not ok. And the latest code is on the master branch.

[xiang90]

@adohe is correct.
@shafreeck This will only be added into v3 api.

[shafreeck]

@adohe Are you working on this ? How is the progress now ? I am going to keep watching on this feature, @adohe @xiang90 Could you guys supply some documents about this feature ?

I am willing to get some details about the implementation of this feature, then I can make a self-using patch based on dev-2.2 and keep compatible with the v3 api

I am sad to hear that the api will only be added into v3 api. Maybe I should work around this using the "refresh directory" way at least now.

[adohe-zz]

@xiang90 any progress on this? I am afraid it is a little bit difficult for people make self-using patch and keep compatible with the v3 api

[shafreeck]

@adohe

I am afraid it is a little bit difficult for people make self-using patch and keep compatible with the v3 api

I don't understand why ? The API has big changes ?

[cchamplin]

For people looking for this in 2.x, I've put together a preliminary implementation.

Adding refresh=true to PUT/POST options when updating a TTL will no longer fire a watch notification (unless a value change occurs with the TTL update)

Commit:
cchamplin@4f68152
Patch File
https://github.com/cchamplin/etcd/commit/4f6815207752dcfd4dc2e9a50cf0a3c1b2fa6179.patch

Example:

curl http://127.0.0.1:2379/v2/keys/foo -XPUT -d value=bar -d ttl=5 -d refresh=true -d prevExist=true

I've included tests, and done some local testing but if anyone wants to do some testing that would be great.

I doubt that this would get included in the upstream 2.x branch at this point, but if it's possible I'll go ahead and create a pull request after additional testing and finishing some TODOs

TODO: Add support for this to the client and etcdctl

[oldCoderException]

Adding our +1 to implementing this in the client API as soon as possible. The TTL is a fantastic feature, but for service discovery and keep-alive in a large microservice environment the load so many real updates would levy on Etcd is of great concern. Having some kind of "I'm still OK" capability without updating indexes and watchers would be awesome. Loving it otherwise guys! Great work.

[raoofm]

+1 really need this for service discovery

[danielzheng]

+1 Wish this feature could be implemented in the release version ASAP.

[sherlockhua]

+1

[bearrito]

+1

[jwclark]

+1

[adohe-zz]

@xiang90 are we going to replace TTL feature with lease in v3?

[xiang90]

@adohe Yes.

[adohe-zz]

@xiang90 how about the lease in v3? complete? I have not seen etcd code for a long time, guess can I have a try about lease?

[xiang90]

I am closing this issue since:

1. We solved this in v3 by adding a new concept called lease. Client can attach keys to a lease. TTL is attached to lease. Updating/KeepAlive TTL on lease will not send notification on keys. The code is alrady merged in master.
2. For v2, we got a pr at store/httpapi: support refresh ttl without firing watch #4164. We are reviewing it. It seems to be fine since it is an incremental change without breaking any existing API or guarantees.

[YanXiaoping]

Sorry, may I know when this "v3" will be available? (or where could I find this info?)
Does v3 here means etcd server 3.x and a client using v3 version of api?

[jonboulle]

@YanXiaoping you can experiment with the V3 API using the --experimental-v3demo=true flag to recent versions of etcd. On the client side, there's currently a Go package and a command line tool:
https://github.com/coreos/etcd/tree/master/clientv3
https://github.com/coreos/etcd/tree/master/etcdctlv3

[YanXiaoping]

@jonboulle , ok, thank you!