🐛 Reduce DB connection locking in storage

[GitHK]

What do these changes do?

It is not necessary to expand directories (access S3) while holding a transaction in Postgres.

This will free up more connections.

Related issue/s

• should help with List file metadata DB query too slow #4971

How to test

DevOps Checklist

[codecov]

Codecov Report

Merging #4984 (27e1400) into master (420f177) will decrease coverage by 20.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##           master   #4984      +/-   ##
=========================================
- Coverage    85.4%   65.4%   -20.1%     
=========================================
  Files        1239     554     -685     
  Lines       51029   28309   -22720     
  Branches     1080     195     -885     
=========================================
- Hits        43612   18516   -25096     
- Misses       7180    9744    +2564     
+ Partials      237      49     -188     

Flag	Coverage Δ
integrationtests	63.5% <ø> (-1.4%)	⬇️
unittests	90.7% <100.0%> (+5.9%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...rage/src/simcore_service_storage/simcore_s3_dsm.py	94.3% <100.0%> (+<0.1%)	⬆️

... and 981 files with indirect coverage changes

[sanderegg]

I am not sure I understand how you fixed the issue here. I would like to better understand that and I have some proposals for you to change.

[GitHK]

I am not sure I understand how you fixed the issue here. I would like to better understand that and I have some proposals for you to change.

@sanderegg

There is an open connection/transaction to the database while making calls to S3 to expand the directory (listing all files inside). It is not required to list the contents of the directory while holding the transaction opened.

[sanderegg]

still have questions, and want to discuss

[sanderegg]

thank you very much for the additional changes!

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.1% Duplication

[codeclimate]

Code Climate has analyzed commit 27e1400 and detected 0 issues on this pull request.

View more on Code Climate.