✨ Computational backend: connect to resource tracking via RabbitMQ (🗃️, ⚠️)

[sanderegg]

What do these changes do?

This PR let now the director-v2 send messages when computational services are started/stopped and while they are running (heartbeat concept)

• passes correct simcore_user_agent to instrumentation RabbitMQ messages when running computational services
• publishes Resource Usage Tracker messages events when start/stop computational services
• publishes Resource Usage Tracker heartbeat events when running computational services
• (⚠️Devops) new ENV SERVICE_TRACKING_HEARTBEAT in date time format, defines the interval with which a heartbeat is sent into RabbitMQ while a computational service is running defaults to 60 seconds, this ENV is global as it will be used in other services (such as RUTH)
• 🗃️comp_tasks table enhanced with last_heartbeat, created, modified timestamps
• added tests to check for correct publication of the RabbitMQ messages
• 🗃️ new RabbitMQ exchange to send messages about started,stopped and heartbeat messages is used from the computational scheduler

various:

• removed RunningState.RETRY state as unused and confusing
• fixed running postgres-database with external database

Related issue/s

• solves Comp backend: collect metrics and send in RabbitMQ #4523

How to test

DevOps Checklist

[codecov]

Codecov Report

Merging #4570 (3a296eb) into master (74636e0) will decrease coverage by 5.1%.
The diff coverage is 71.4%.

@@           Coverage Diff            @@
##           master   #4570     +/-   ##
========================================
- Coverage    84.5%   79.4%   -5.1%     
========================================
  Files        1040     903    -137     
  Lines       44269   37500   -6769     
  Branches     1013     960     -53     
========================================
- Hits        37416   29802   -7614     
- Misses       6635    7491    +856     
+ Partials      218     207     -11     

Flag	Coverage Δ
integrationtests	38.9% <ø> (-0.1%)	⬇️
unittests	78.9% <71.4%> (-5.5%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...odels-library/src/models_library/projects_state.py	98.4% <ø> (-0.1%)	⬇️
...ackages/service-library/src/servicelib/rabbitmq.py	94.1% <ø> (ø)
...ls-library/src/models_library/rabbitmq_messages.py	90.2% <50.0%> (-0.8%)	⬇️
...ckages/models-library/src/models_library/docker.py	89.4% <66.6%> (-1.3%)	⬇️
...src/simcore_postgres_database/models/comp_tasks.py	100.0% <100.0%> (ø)

... and 242 files with indirect coverage changes

[mrnicegyu11]

tldr

green light for the ops changes thanks a lot for the nice summary in the PR text :--)

[matusdrobuliak66]

Looking good! 👍

[YuryHrytsuk]

Thinking aloud

If we want to be fully transparent with env and avoid implicit defaults (in code), then we want to have no defaults in code and provide values transparently via .env files 🤔

*I am still not sure if we want to go in this direction

[GitHK]

Please consider my comments regarding the moving of some code.

[codeclimate]

Code Climate has analyzed commit 3a296eb and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
14.0% Duplication

Catch issues before they fail your Quality Gate with our IDE extension SonarLint