♻️ Locking dy-service start per user and per project

[GitHK]

What do these changes do?

Addresses the following:

• 🐛 Fixed multiple race conditions causing issues. Added a distributed Redis lock to address the issue of limiting the amount of services to start. SEE services/web/server/src/simcore_service_webserver/projects/projects_api.py
• 🐛 Test checking above was broken. Was also fixed to take into account the delay when mocking. SEE services/web/server/tests/unit/with_dbs/02/test_projects_nodes_handler.py

Repo wide Redis modules refactor and centralization:

• ♻️ Redis configuration, it was outdated and very hard to maintain. Now it is more readable.
• ♻️ Refactor servicelib's RedisClientSDK module. It is now the de facto standard used by all module which require modules.
  • lock_context can now be blocking
  • added RedisClientsManager which keeps track of multipl Redis clients, useful for applications that require access to more than one DB.
  • changed improperly named error AlreadyLockedError -> CouldNotAcquireLockError
  • lock release and require can raise redis.exceptions.LockError which are now suppressed
• ♻️ Replaced werbserver's redid client with RedisClientSDK and RedisClientsManager
• ♻️ Refactor services/director-v2/src/simcore_service_director_v2/modules/node_rights.py to use RedisClientSDK
• ♻️ using correct pattern to cancel background tasks packages/service-library/src/servicelib/background_task.py

Extra:

• ✨ new configuration for pylance added by default
• ⬆️ redis==4.5.4 in services that use it
• ♻️ reduces test flakiness tests/integration/02/test_dynamic_sidecar_nodeports_integration.py::test_nodeports_integration

Related issue/s

• created Replace all usage of reds locks with lock_context #4112
• impacts Failed services seem to count towards (maximum) started services #3726

How to test

• covered by unit tests and integration tests

[codecov]

Codecov Report

Merging #4055 (2c73726) into master (2f9cce2) will increase coverage by 3.3%.
The diff coverage is 90.6%.

@@           Coverage Diff            @@
##           master   #4055     +/-   ##
========================================
+ Coverage    81.3%   84.7%   +3.3%     
========================================
  Files         953     953             
  Lines       41392   41408     +16     
  Branches      957     962      +5     
========================================
+ Hits        33684   35104   +1420     
+ Misses       7503    6086   -1417     
- Partials      205     218     +13     

Flag	Coverage Δ
integrationtests	61.6% <53.4%> (?)
unittests	82.3% <90.6%> (+1.0%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ges/settings-library/src/settings_library/redis.py	0.0% <0.0%> (ø)
...ing/src/simcore_service_autoscaling/core/errors.py	100.0% <ø> (ø)
...s/dynamic_sidecar/scheduler/_core/_events_utils.py	56.9% <40.0%> (+0.3%)	⬆️
packages/service-library/src/servicelib/redis.py	96.2% <93.6%> (-1.9%)	⬇️
.../web/server/src/simcore_service_webserver/redis.py	97.3% <94.1%> (+4.2%)	⬆️
.../service-library/src/servicelib/background_task.py	97.7% <100.0%> (+6.6%)	⬆️
...brary/src/servicelib/long_running_tasks/_errors.py	100.0% <100.0%> (ø)
...library/src/servicelib/long_running_tasks/_task.py	95.2% <100.0%> (ø)
...s/service-library/src/servicelib/retry_policies.py	100.0% <100.0%> (+100.0%)	⬆️
...ling/src/simcore_service_autoscaling/api/health.py	100.0% <100.0%> (ø)
... and 8 more

... and 66 files with indirect coverage changes

[codeclimate]

Code Climate has analyzed commit 2c73726 and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.1% Duplication