✨Enable on-demand clusters based on groups extra properties (⚠️ devops)

[sanderegg]

What do these changes do?

• based on groups extra properties, asks director-v2 to run in on-demand clusters mode
• pass ENV variables to clusters-keeper
• return ETA of cluster in the logs passed to the frontend
• log rabbit message now contains optional node id
• ENVs are set in the .env-devel file

Related issue/s

• requires ✨Computational backend: connect dv2 to clusters keeper for on-demand clusters (🗃️ + ⚠️devops) #4703
• requires ✨New groups extra properties to use on demand clusters in DB (🗃️) #4736

How to test

DevOps Checklist

new ENV variables to be defined

CLUSTERS_KEEPER_MAX_MISSED_HEARTBEATS_BEFORE_CLUSTER_TERMINATION=5
CLUSTERS_KEEPER_TASK_INTERVAL=60

the following ENV are only necessary in case clusters-keeper is enabled:

CLUSTERS_KEEPER_EC2_ACCESS_KEY_ID=XXXXXXXXXX
CLUSTERS_KEEPER_EC2_ENDPOINT=ec2.amazonaws.com
CLUSTERS_KEEPER_EC2_INSTANCES_ALLOWED_TYPES="[\"g4dn.xlarge\"]"
CLUSTERS_KEEPER_EC2_INSTANCES_AMI_ID=XXXXXXXXXXX
CLUSTERS_KEEPER_EC2_INSTANCES_KEY_NAME=XXXXXXXXXXXXXX
CLUSTERS_KEEPER_EC2_INSTANCES_SECURITY_GROUP_IDS="[\"XXXXXXXXXXXXX\"]"
CLUSTERS_KEEPER_EC2_INSTANCES_SUBNET_ID=XXXXXXXXXXXXX
CLUSTERS_KEEPER_EC2_REGION_NAME=us-east-1
CLUSTERS_KEEPER_EC2_SECRET_ACCESS_KEY=XXXXXXXXXXXXX

[codecov]

Codecov Report

Merging #4738 (31f0212) into master (ab5fa9a) will increase coverage by 1.9%.
The diff coverage is 82.3%.

@@            Coverage Diff            @@
##           master   #4738      +/-   ##
=========================================
+ Coverage    86.8%   88.7%    +1.9%     
=========================================
  Files        1112     509     -603     
  Lines       46198   26330   -19868     
  Branches     1013     190     -823     
=========================================
- Hits        40107   23376   -16731     
+ Misses       5864    2907    -2957     
+ Partials      227      47     -180     

Flag	Coverage Δ
integrationtests	65.2% <76.4%> (+1.4%)	⬆️
unittests	86.9% <82.3%> (+2.2%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...imcore_service_director_v2/utils/comp_scheduler.py	100.0% <ø> (ø)
...ore_service_director_v2/modules/clusters_keeper.py	56.0% <50.0%> (+3.6%)	⬆️
...-v2/src/simcore_service_director_v2/core/errors.py	77.9% <100.0%> (ø)
...rector_v2/modules/comp_scheduler/base_scheduler.py	90.9% <100.0%> (+<0.1%)	⬆️
.../src/simcore_service_director_v2/utils/rabbitmq.py	100.0% <100.0%> (ø)
...simcore_service_webserver/director_v2/_handlers.py	95.6% <100.0%> (+1.9%)	⬆️

... and 624 files with indirect coverage changes

[codeclimate]

Code Climate has analyzed commit 31f0212 and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication