Clusters-keeper: WORKERS_EC2_INSTANCES_ALLOWED_TYPE is not passing 16KB limit for EC2 instances

[sanderegg]

When the primary machine is created via Clusters-Keeper, it passes a bunch of text via the EC2 User-Data field.
This field is limited to 16KB which is too small for what we use it for.
Currently that field contains a bash script which:

• pre-pulls images
• runs custom script as defined in PRIMARY_EC2_INSTANCES_ALLOWED_TYPES for the primary machine type
• create a docker swarm
• transmits a docker-compose file
• start a docker-compose file with a bunch of ENV variables + among other WORKERS_EC2_INSTANCES_ALLOWED_TYPES which is typically huge

All in all the complete resulting text file is > 16KB and that is not sustainable.

Tasks

Preview Give feedback

1. add validation step to know beforehand when the 16KB is reached
2. most of the custom scripts are the same, maybe they could be baked in the AMI
3. static data could be pulled in the image instead of passed over

[sanderegg]

fixed together with #5294

The new AMI allows to only call a pre-baked script that takes care of most of the work. only the pre-pull images have to be transfered, thus reducing the size by a huge amount.
In need even the docker-compose files could be pre-baked.