- This directory wasn't being checked for formatting and linting issues in the Nox sessions before this commit, so there are some changes to files outside of the `util/realistic_data/`

…not be made

- Files are now not stored into directories by day - this fits with the ingestion echo script format

- This can now be done because EPAC Data Sim can be specified as a dependency in Poetry (where it couldn't be done previously because its a GitHub repo, had some issues with that)

…ngoimport file

…c-test-data

…current datetime

…c-test-data

…c-test-data

- This is so we can exclude it if we don't have the correct SSH permissions to access/clone the repo. For example, the CI cannot clone the repo

- This is so the old Gemini test data still works

- This seemed to cause issues when mass uploading HDF files which contained multiple images in each. Uploading images sequentially seemed to fix the issue

- This has also updated the lock file for lots of other dependencies...

…ents

- For single day experiments, the start date was 23:59 and the end date was 00:00 - an experiment ended before it even started!

- This seems to help with ingesting large HDF files

…c-test-data

…-realistic-test-data

- This isn't needed, logging config location is controlled by the constant called `LOG_CONFIG_LOCATION`

- This doesn't work on GitHub Actions so needs investigating at a later date - just want to get GitHub Actions working for right now

…c-test-data

- I've restored this functionality as I now understand why I was having issues with it. The previous implementation creates as many threads as their were images. When you have 69 images (and therefore 69 threads), it's now obvious that my dev VM simply ran out of memory trying to ingest a HDF file with that number of images. I've made the number of threads configurable via the `upload_image_threads` option
- This change has tested to be more reliable and has sped up ingestion of the 14TB of simulated data quite a bit. Image processing looks to be a bottleneck when ingesting HDF files containing many images and I think further performance improvements can be made, but this is to be explored in the future.

- This approach is a little safer, returning `None` if no message is in the dictionary, rather than throwing a `KeyError`. I had this happen when I exceeded the quota through our S3 storage
- Also removed a TODO that I noticed isn't relevant, the bucket name is replaced in CI

- This reduces the amount of size required to store them using the simulated data we now have access too

- This will be useful when the script is run on local machines where SSH is an issue (e.g. GitHub Actions)
- Also included a couple of linting fixes in this PR

…c-test-data

…c-test-data

- Experiments are stored using ObjectIDs, not custom identifers

these caused an error where none of the database collections where dropped
when using
poetry run python util/realistic_data/ingest_echo_data.py

…st-data