✨ Maintenance/add consistency scripts

[sanderegg]

What do these changes do?

• adds a script for checking consistency between the file entries in the projects/file_meta_data tables in the database and what is available on S3

The scripts does the following:
pre-requisite:

• local docker volume containing a database from a deployment (see make import-db-from-docker-volume in /packages/postgres-database)

1. from the local DB parses the projects table for all the node UUIDs available used by some non-GUEST user
2. from the local DB parses the file_meta_data table for all file entries corresponding to the node UUIDs from 1.
3. from the corresponding S3 endpoint parses all the file corresponding to the node UUIDs from 1.
4. Compare files parsed in DB to the ones in S3
5. create a report as csv file containing

Related issue/s

How to test

Checklist

[codecov]

Codecov Report

Merging #2531 (5ab4637) into master (6aa8d03) will decrease coverage by 0.0%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master   #2531     +/-   ##
========================================
- Coverage    76.5%   76.5%   -0.1%     
========================================
  Files         610     610             
  Lines       23397   23397             
  Branches     2294    2294             
========================================
- Hits        17917   17909      -8     
- Misses       4873    4880      +7     
- Partials      607     608      +1     

Flag	Coverage Δ
integrationtests	66.4% <ø> (-0.1%)	⬇️
unittests	70.6% <ø> (-0.1%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
..._director_v2/modules/dynamic_sidecar/client_api.py	76.6% <0.0%> (-5.6%)	⬇️
...tor_v2/modules/dynamic_sidecar/scheduler/events.py	93.6% <0.0%> (-3.8%)	⬇️
..._director_v2/modules/dynamic_sidecar/docker_api.py	85.8% <0.0%> (-0.7%)	⬇️
.../director/src/simcore_service_director/producer.py	61.2% <0.0%> (-0.3%)	⬇️
...ce_webserver/resource_manager/garbage_collector.py	73.9% <0.0%> (+0.8%)	⬆️

[pcrespov]

Super nice and useful scripts!

A few suggestion:

• dump data in csv files with headers (can be opened and explored in excel-like)
• in s3_missing_files.csv add columns with file size and last modification date (see below). This helps to explore the consistency
• add docker pull mino/cl before doing requests to s3. otherwise captured stdout is polluted with log message from pulling instead of the client responses
• EXTRA: prompt passwords/secret instead of passing them as arguments. This way they do not remain in the history.
• EXTRA: option to use dumped files as cache instead of re-running requests

[pcrespov]

nce touch

[mguidon]

real nice