Replace Datacube Explorer Database Test Infrastructure

[omad]

Continuing on from work started by @Ariana-B in #479 and myself in opendatacube/odc-tools#524

This PR completely redoes the way database tests work for Datacube Explorer.

Temporary Docker based PostgreSQL server

• If Docker is available, a temporary PostGIS server is run for the duration of the tests. (Pytest Fixture postgresql_server, scope = module)
  • If the database fixture detects it is running within Docker, and that ODC Database access is configured already, it does nothing, allowing the existing testing infrastructure using docker to continue working.
• An ODC Database is created and destroyed per test module, by a new fixture called odc_db of scope module. It creates a new database within the postgres server, named after the test module, and configures ODC to access it via environment variables and temporary configuration files.
• A new fixture called odc_test_db, of module scope, initialises the ODC database structure within the temporary database. It also removes the database after the module is finished.

Test fixture data loading

Previously, test data was loaded a little bit all over the place, using a function which was copied into multiple test modules.

This has been replaced by a new fixture called auto_odc_db of module scope, which looks for module variables (named METADATA_TYPES, PRODUCTS, and DATASETS), and loads them into the current database. This allows each test module to specify exactly which data it expects to be available.

Benefits

In combination, this new setup:

• Allows the running of all the integration tests without any database setup, assuming docker is available.
• Allows running any test individually, with the correct fixtures being used to setup the database and test data.
• Simplifies and makes explicit which test data is used by which test module.
• Makes it possible to run multiple instances of pytest simultaneously (due to unique docker postgres instances), and multiple test modules simultaneously using pytest-xdist, due to unique databases being created per module.

These new fixtures have been implemented in a new cubedash.testutils.database module, which is loaded as a pytest plugin. My intention, once proven, is to move this test module into datacube-core to be available for all downstream ODC based software.

---

📚 Documentation preview 📚: https://datacube-explorer--490.org.readthedocs.build/en/490/

[codecov]

Codecov Report

Base: 86.60% // Head: 86.04% // Decreases project coverage by -0.56% ⚠️

Coverage data is based on head (c0ec914) compared to base (60daf44).
Patch coverage: 82.40% of modified lines in pull request are covered.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #490      +/-   ##
===========================================
- Coverage    86.60%   86.04%   -0.57%     
===========================================
  Files           23       24       +1     
  Lines         3017     3188     +171     
===========================================
+ Hits          2613     2743     +130     
- Misses         404      445      +41     

Impacted Files	Coverage Δ
cubedash/summary/_summarise.py	96.05% <ø> (ø)
cubedash/testutils/database.py	68.68% <68.68%> (ø)
cubedash/_stac.py	86.53% <85.26%> (-5.11%)	⬇️
cubedash/_api.py	95.91% <93.33%> (-1.23%)	⬇️
cubedash/_audit.py	92.95% <100.00%> (+1.78%)	⬆️
cubedash/_filters.py	89.75% <100.00%> (+0.31%)	⬆️
cubedash/_utils.py	89.18% <100.00%> (+1.98%)	⬆️
cubedash/summary/_stores.py	92.16% <100.00%> (+0.01%)	⬆️
cubedash/warmup.py	65.64% <100.00%> (+0.80%)	⬆️
cubedash/summary/_schema.py	80.67% <0.00%> (-0.85%)	⬇️
... and 5 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[omad]

Switching back to Draft PR.

I haven't explained the testing changes anywhere near well enough, or documented the new pytest fixtures I've added. I just got excited that all the tests were passing.

[pindge]

Please update this too https://github.com/opendatacube/datacube-explorer#contributing-to-integration-test