♻️On-demand clusters: remove dask-gateway

[sanderegg]

What do these changes do?

Clusters-keeper

• the clusters-keeper deployed machines now start a dask-scheduler/dask-worker directly instead of osparc-dask-gateway
  • deploys a docker-compose.yml stack in the machine,
  • the dask-sidecar is set as global
• removed dask-gateway/dask-gateway-server dependencies in clusters-keeper
• moved dask-scheduler fixture to pytest-simcore (shared between dv-2 and clusters-keeper, and soon autoscaling)
• @mguidon the name of the machine created on EC2 was modified to be more user-friendly, it now goes along: osparc-computational-cluster-manager-{app_settings.SWARM_STACK_NAME}-user_id:{user_id}-wallet_id:{wallet_id}
• add a distributed lock mechanism to prevent creating multiple clusters for the same user_id/wallet_id pair if the RPC is called in a short burst
• fixed pipeline cannot be stopped when waiting for cluster

dask-sidecar

• scheduler now also shows a logo on start
• fixed issue in devel mode where /home/scu/.config/dask/distributed.yml was not writable

director-v2

• adapted code to change from dask-gateway to dask-scheduler

Related issue/s

How to test

DevOps Checklist

[codecov]

Codecov Report

Merging #4760 (6a43354) into master (a2b046c) will increase coverage by 0.2%.
The diff coverage is 73.2%.

@@           Coverage Diff            @@
##           master   #4760     +/-   ##
========================================
+ Coverage    86.8%   87.0%   +0.2%     
========================================
  Files        1115    1138     +23     
  Lines       46511   47509    +998     
  Branches     1021    1021             
========================================
+ Hits        40392   41356    +964     
- Misses       5888    5922     +34     
  Partials      231     231             

Flag	Coverage Δ
integrationtests	65.0% <77.7%> (-0.1%)	⬇️
unittests	84.7% <73.2%> (+0.2%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...r/src/simcore_service_clusters_keeper/utils/ec2.py	100.0% <ø> (ø)
...ecar/src/simcore_service_dask_sidecar/scheduler.py	0.0% <0.0%> (ø)
...ore_service_director_v2/modules/clusters_keeper.py	50.0% <0.0%> (ø)
...e_service_director_v2/modules/dask_clients_pool.py	93.4% <ø> (ø)
...rc/simcore_service_clusters_keeper/rpc/clusters.py	53.5% <23.0%> (-2.0%)	⬇️
...-sidecar/src/simcore_service_dask_sidecar/_meta.py	92.3% <83.3%> (-7.7%)	⬇️
.../simcore_service_clusters_keeper/utils/clusters.py	67.5% <92.8%> (+9.4%)	⬆️
...ls_library/rpc_schemas_clusters_keeper/clusters.py	100.0% <100.0%> (ø)
...eeper/src/simcore_service_clusters_keeper/_meta.py	92.0% <100.0%> (+0.6%)	⬆️
...c/simcore_service_clusters_keeper/core/settings.py	100.0% <100.0%> (ø)
... and 7 more

... and 24 files with indirect coverage changes

[pcrespov]

Except for the "unfancy" banner ;-) ... just some minor comments

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[codeclimate]

Code Climate has analyzed commit 6a43354 and detected 0 issues on this pull request.

View more on Code Climate.