[autoscaler] Allow more than 5s from node creation to first heartbeat #3385
Conversation
|
Test FAILed. |
python/ray/autoscaler/autoscaler.py
Outdated
| self.provider.internal_ip(node_id), 0) | ||
| key = self.provider.internal_ip(node_id) | ||
| if key not in self.load_metrics.last_heartbeat_time_by_ip: | ||
| self.load_metrics.last_heartbeat_by_ip = time.time() |
There was a problem hiding this comment.
This line should read self.load_metrics.last_heartbeat_time_by_ip[key] = time.time() (attribute name, and you need the subscript)
|
Test FAILed. |
|
Test FAILed. |
|
Test FAILed. |
ericl
added
the
tests-ok
| key = self.provider.internal_ip(node_id) | ||
| if key not in self.load_metrics.last_heartbeat_time_by_ip: | ||
| self.load_metrics.last_heartbeat_time_by_ip[key] = time.time() | ||
| last_heartbeat_time = self.load_metrics.last_heartbeat_time_by_ip[key] |
There was a problem hiding this comment.
So this used to default to 0, which means that delta (below) would be enormous and then the autoscaler would attempt to restart the node, right?
|
@ericl |
|
Sure, if those can be surfaced to python.
…On Mon, Nov 26, 2018, 5:26 PM Robert Nishihara ***@***.***> wrote:
@ericl <https://github.com/ericl> monitor.cc already makes decisions
about when a node should be considered dead. Wouldn't it make sense to just
use those decisions? Instead of also having the autoscaler make that
decision?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3385 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAA6SqgOPcrOy6FQy7rw4SjzgnBhpQEsks5uzJSpgaJpZM4Yuzr1>
.
|
What do these changes do?
I think this is due to a race condition between when we first mark a node as active and check for restarts. It's possible a node state changes in the background between these two checks, which would result in a spurious restart.
Related issue number
Closes #3361