You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When a model job fails a health check, it moves immediately into a FailedState and hence the gateway never sends the request to shut the job down. We should send the shutdown request first. Celery logs as follows:
[2023-05-01 20:59:58,426: WARNING/ForkPoolWorker-3] The model is healthy
[2023-05-03 18:42:23,284: WARNING/ForkPoolWorker-3] Model health verification error:: HTTPConnectionPool(host='172.17.8.109', port=43537): Max retries exceeded with url: /health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb5c7984b80>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-05-03 18:42:23,285: WARNING/ForkPoolWorker-3] [2023-05-03 18:42:23,285] ERROR in models: Health check for active model OPT-6.7B failed
[2023-05-03 18:42:23,285: ERROR/ForkPoolWorker-3] Health check for active model OPT-6.7B failed
[2023-05-03 18:42:23,429: ERROR/ForkPoolWorker-3] Task tasks.verify_model_instance_health[94ca39cf-8249-46de-ae48-605aae58ec48] raised unexpected: InvalidStateError("Invalid operation for model instance state: <class 'models.FailedState'>")
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/app/gateway_service.py", line 55, in call
return self.run(*args, **kwargs)
File "/app/tasks.py", line 11, in verify_model_instance_health
model_instance.shutdown()
File "/app/models.py", line 302, in shutdown
self._state.shutdown()
File "/app/models.py", line 200, in shutdown
raise InvalidStateError(self)
errors.InvalidStateError: Invalid operation for model instance state: <class 'models.FailedState'>
The text was updated successfully, but these errors were encountered:
Describe the bug
When a model job fails a health check, it moves immediately into a FailedState and hence the gateway never sends the request to shut the job down. We should send the shutdown request first. Celery logs as follows:
[2023-05-01 20:59:58,426: WARNING/ForkPoolWorker-3] The model is healthy
[2023-05-03 18:42:23,284: WARNING/ForkPoolWorker-3] Model health verification error:: HTTPConnectionPool(host='172.17.8.109', port=43537): Max retries exceeded with url: /health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb5c7984b80>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-05-03 18:42:23,285: WARNING/ForkPoolWorker-3] [2023-05-03 18:42:23,285] ERROR in models: Health check for active model OPT-6.7B failed
[2023-05-03 18:42:23,285: ERROR/ForkPoolWorker-3] Health check for active model OPT-6.7B failed
[2023-05-03 18:42:23,429: ERROR/ForkPoolWorker-3] Task tasks.verify_model_instance_health[94ca39cf-8249-46de-ae48-605aae58ec48] raised unexpected: InvalidStateError("Invalid operation for model instance state: <class 'models.FailedState'>")
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/app/gateway_service.py", line 55, in call
return self.run(*args, **kwargs)
File "/app/tasks.py", line 11, in verify_model_instance_health
model_instance.shutdown()
File "/app/models.py", line 302, in shutdown
self._state.shutdown()
File "/app/models.py", line 200, in shutdown
raise InvalidStateError(self)
errors.InvalidStateError: Invalid operation for model instance state: <class 'models.FailedState'>
The text was updated successfully, but these errors were encountered: