directorv2 fails to reconnect to new scheduler and remains "waiting for cluster"

[pcrespov]

Is there an existing issue for this?

• I have searched the existing issues

Which deploy/s?

development (master)

Current Behavior

When dask-scheduler restarts and a computational service is run, the directorv2 notices and raises https://monitoring.osparc-master.speag.com/graylog/messages/graylog_115/dd0866a5-b38d-11ee-baa3-0242ac130007

The Tasks are set back to WAITING_FOR_CLUSTER state until scheduler comes back but the scheduler somehow does not "come back" and hangs forever

Expected Behavior

After some time, dv2 should resolve that there is a new scheduler in place. I guess it needs to resync pending tasks list?

Steps To Reproduce

1. open sleepers project
2. in portainer restart dask scheduler
3. open graylog and set filter dv2
4. run slepper project

Anything else?

No response