✨ Comp backend: allow multiple amis/bootscripts/pre-pull images

[sanderegg]

What do these changes do?

This PR makes the autoscaling service to define an AMI, specific boot scripts, pre-pull images per EC2 instance type instead of one for all.

This means the following ENVs are deprecated in Autoscaling service:
EC2_INSTANCES_AMI_ID
EC2_INSTANCES_CUSTOM_BOOT_SCRIPTS
EC2_INSTANCES_PRE_PULL_IMAGES
EC2_INSTANCES_PRE_PULL_IMAGES_CRON_INTERVAL

and following ENVs are deprecated in Clusters-keeper service:
WORKERS_EC2_INSTANCES_AMI_ID
WORKERS_EC2_INSTANCES_AMI_ID

The ENV EC2_INSTANCES_ALLOWED_TYPES and WORKERS_EC2_INSTANCES_ALLOWED_TYPES are now a dictionary and may be created like so

json.dumps(
  {
    "t2.micro": 
      {
        "ami_id": AMISTRING, 
        "custom_boot_scripts": Optional[list[COMMAND]], 
        "pre_pull_images": Optional[list[IMAGETAG]], 
        "pre_pull_images_cron_interval": Optional[timedelta]
     }
  }
)

Bonus

• partially check the syntax of the custom scripts (at least finds weird quoting), partially fixes Autoscaling: Give feedback in the logs in case something wrong happens with a starting EC2 #3910
• define all the ENVs variables in docker-compose.yml/.env-devel as required by our new best practices (a corresponding PR in osparc-ops-environment will follow) see Autoscaling: remove all overriden ENVs osparc-ops-environments#438
• fixes wrong ENV variable set for clusters-keeper service in docker-compose (see 🚀 Pre-release master -> staging_sevenPeaks2 #5029 (comment))

Related issue/s

• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/148
• Autoscaling: remove all overriden ENVs osparc-ops-environments#438
• fixes Autoscaling: settings for AMI and Custom boot script shall be linked and made into a list #4781

How to test

Dev Checklist

multiple changes in ENVs for autoscaling and clusters-keeper
documented in:

• Autoscaling: remove all overriden ENVs osparc-ops-environments#438
• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/148
• No ENV changes or I properly updated ENV (read the instruction)

DevOps Checklist

[codecov]

Codecov Report

Merging #5033 (89fd705) into master (5477e3d) will increase coverage by 0.9%.
The diff coverage is 78.6%.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #5033     +/-   ##
========================================
+ Coverage    86.2%   87.2%   +0.9%     
========================================
  Files        1059    1252    +193     
  Lines       45012   51451   +6439     
  Branches      835    1108    +273     
========================================
+ Hits        38834   44879   +6045     
- Misses       5993    6333    +340     
- Partials      185     239     +54     

Flag	Coverage Δ
integrationtests	64.9% <ø> (+<0.1%)	⬆️
unittests	85.0% <78.6%> (+0.8%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...kages/settings-library/src/settings_library/ec2.py	100.0% <ø> (ø)
...ling/src/simcore_service_autoscaling/api/health.py	100.0% <100.0%> (ø)
...e_service_autoscaling/modules/auto_scaling_core.py	94.9% <100.0%> (-0.1%)	⬇️
...ore_service_autoscaling/utils/auto_scaling_core.py	94.2% <100.0%> (-0.1%)	⬇️
.../simcore_service_clusters_keeper/utils/clusters.py	74.0% <100.0%> (+2.2%)	⬆️
...g/src/simcore_service_autoscaling/core/settings.py	97.8% <86.6%> (-2.2%)	⬇️
...c/simcore_service_clusters_keeper/core/settings.py	97.5% <77.7%> (-2.5%)	⬇️
packages/aws-library/src/aws_library/ec2/models.py	87.1% <66.6%> (ø)

... and 197 files with indirect coverage changes

[mrnicegyu11]

tldr thx

[GitHK]

👍

[pcrespov]

🎉 spotless

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[codeclimate]

Code Climate has analyzed commit 89fd705 and detected 0 issues on this pull request.

View more on Code Climate.