Optimize ingest_zarr_archive task

[jjnesbitt]

This optimization is achieved in two main ways:

1. Using a priority queue (heap) in the ZarrChecksumModificationQueue class, instead of a normal queue
2. Storing all s3 files in the queue before doing any processing (see below for memory usage).

The main benefit this provides is being able to update the checksum tree strictly from the bottom up, removing duplicate file updates.

Previously, a batch of files were fetched, immediately processed, and the next batch fetched, etc. This resulted in all parent checksum files being updated (up to the root checksum file) in every batch. The optimized method only updates each checksum file once.

Results

The outcome of this optimization is an almost 2x speed improvement. For a zarr archive with ~37k files (52.5GB), I observed an average runtime of ~83s, compared to a previous runtime of ~147s.

In regards to memory usage, it seems the max usage was ~411kb for the same zarr with ~37k files, decreasing as the ingestion goes on. This was the largest zarr I was able to test on, since this is the largest zarr in staging. Also, to clarify, this testing was done by using a local shell pointed at the staging database and s3 bucket, so these times should be representative of what we would observe in production.

[mvandenburgh]

Very cool, some minor suggestions but this LGTM

[jjnesbitt]

FYI it seems that django-stubs was updated last night, and so caused a small mypy error which 9848539 addresses. It's a pretty minor change, so I've included it here.

[dandibot]

🚀 PR was released in v0.3.8 🚀