✨ Store computational services progress in comp_tasks table, decouple webserver from api-server 🗃️

[sanderegg]

What do these changes do?

• add progress column to comp_tasks table
• adapt director-v2 to store last progress in comp_tasks table
• adapt director-v2 to return the progress from comp_tasks table
• adapt director-v2 to return overall pipeline progress (based on pipeline each node progress)
• dask-sidecar: uncontrolled progress value from comp. services gets forcibly set between 0 and 1
• api-server now returns progress as well (PublicAPI: Correctly retrieve progress of jobs #2444)

NOTE: Next PR should then remove the progress changes in the DB done by the webserver. This should completely decouple the webserver from the api-server run pipelines.

Bonus:

• fix flaky with migration service (remove service instead of killing the task which would trigger a double restart)
• some sqlalchemy 2.0 refactoring (use Row._asdict() to convert to a dict)

Related issue/s

• resolves PublicAPI: Correctly retrieve progress of jobs #2444
• relates to Dask backend: run 15600 sleepers #3991
• relates to M1-12 Maintenance and DevOps osparc-issues#675
• relates to Flaky tests/integration/02/test_dynamic_sidecar_nodeports_integration.py #3820
• relates to Flaky tests/integration/02/test_mixed_dynamic_sidecar_and_legacy_project.py::test_legacy_and_dynamic_sidecar_run #3756

How to test

DevOps Checklist

[codecov]

Codecov Report

Merging #4197 (07b4d8d) into master (4cee078) will increase coverage by 2.5%.
The diff coverage is 97.6%.

@@            Coverage Diff            @@
##           master   #4197      +/-   ##
=========================================
+ Coverage    84.0%   86.6%    +2.5%     
=========================================
  Files         956     558     -398     
  Lines       41489   27527   -13962     
  Branches      960     547     -413     
=========================================
- Hits        34884   23842   -11042     
+ Misses       6400    3554    -2846     
+ Partials      205     131      -74     

Flag	Coverage Δ
integrationtests	63.6% <91.0%> (+2.6%)	⬆️
unittests	84.7% <91.7%> (+2.3%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ector_v2/modules/db/repositories/comp_pipelines.py	94.2% <ø> (ø)
...e_director_v2/modules/db/repositories/comp_runs.py	94.8% <ø> (ø)
...ce_director_v2/modules/db/repositories/projects.py	100.0% <ø> (ø)
...or_v2/modules/db/repositories/projects_networks.py	52.1% <0.0%> (ø)
..._director_v2/modules/db/repositories/comp_tasks.py	96.9% <92.3%> (+0.2%)	⬆️
...brary/src/servicelib/long_running_tasks/_models.py	97.0% <100.0%> (+<0.1%)	⬆️
...e_service_director_v2/models/domains/comp_tasks.py	100.0% <100.0%> (ø)
...rector_v2/modules/comp_scheduler/base_scheduler.py	91.9% <100.0%> (ø)
...rector_v2/modules/comp_scheduler/dask_scheduler.py	93.8% <100.0%> (+<0.1%)	⬆️
...ce_director_v2/modules/db/repositories/clusters.py	94.7% <100.0%> (+<0.1%)	⬆️
... and 2 more

... and 430 files with indirect coverage changes

[pcrespov]

Great work!
Please check test_solvers_jobs_api.py check on the progress ... that seems suspicious to me

[matusdrobuliak66]

👍

[codeclimate]

Code Climate has analyzed commit 07b4d8d and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication