Dask backend: run 15600 sleepers

[sanderegg]

in order to test the dask-gateway here are the steps:

• need a new sleeper version that needs less than 2 Gb (probably 100Mb is enough)
• create a separate cluster in ITIS account
  • create a dask-gateway-manager node
  • create 25-50 worker nodes t2.medium instances (@Surfict we need enough vpcus for that)
• test a few sleepers run on the separate cluster from the GUI
• run 15600 jobs with a sleeper through the osparc API

--> get results and observations/problems

[sanderegg]

• using resource overrides, the sleepers:2.0.2 was made to use 0.1 CPU and 100MB
• created 1 dask-gateway-manager node
• created 21 dask-gateway worker nodes (t2.large)
• tested running 10, 20, 200 sleepers
• delete job projects POC
• test with 2000 sleepers
• test with 16000 sleepers
• check get results

[sanderegg]

• disabling dask-gateway auto-adapt mode unleashes issue-free running of 2200 jobs on 51 machines
• director-v2 now asks for a large amount of workers (10000) to ensure the number of machines is maximized

This shows the status on the gateway while running 2200 jobs

Adaptive scaling seems to be evil

remaining issues:

• throughput of director-v2 to send the tasks could be increased

[sanderegg]

closing this. outdated.