fix: error on collection with no items TDE-1216 #1000
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amfage
requested review from
l0b0,
blacha,
MDavidson17,
paulfouquet and
dwsilk
and removed request for
l0b0 and
MDavidson17
blacha
approved these changes
Jul 29, 2024
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
Sep 12, 2024
🤖 I have created a release *beep* *boop* --- ## [4.10.0](v4.9.0...v4.10.0) (2024-09-12) ### Features * add derived_from links to item TDE-1251 ([#1043](#1043)) ([077f14f](077f14f)) * determine start and end datetimes when item include derived TDE-1258 ([#1052](#1052)) ([62d571a](62d571a)) * make start_datetime and end_datetime optional when includeDerived is True TDE-1258 ([#1065](#1065)) ([57f088c](57f088c)) * More types ([#989](#989)) ([3f7cd57](3f7cd57)) ### Bug Fixes * capture area winding order TDE-1205 ([#1008](#1008)) ([61be096](61be096)) * error on collection with no items TDE-1216 ([#1000](#1000)) ([d351d85](d351d85)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The
create-collectionstandardising Argo workflow task that callscollection-from-items.pydoes not fail if there are no items added to the collection, and creates an invalidcollection.jsonfile that fails STAC Validation (it has no item links or extents).This happened recently when a standardising workflow was retried, creating a new collection ID, however the target scratch directory already had standardised images and STAC. We do not overwrite exisiting standardised files, so the collection ID did not match between the new collection and the existing items.
We want to know if the collection has no items and the
create-collectionworkflow task should fail with an error.Modifications
collection.jsonfile may exist.Examples of logging/Exception:
If the Item Collection ID and Collection ID do not match:
{"file": "afage/collection_from_items_id_mismatch/AZ34_5000_0402.json", "action": "collection_from_items", "reason": "skip", "level": 40, "time": 1721257664445, "v": 1, "pid": 1, "hostname": "d5df8d86499f", "id": "01J31F777EZD0M0WN8HZY1BVZ6", "msg": "skipping: 01J1XDYXSC1GC9ASDBXNT3A93T and 01J1XG0GJF2AK110287P265HJM do not match"}If the collection has no items:
{"level": 50, "time": 1721258802092, "v": 1, "pid": 1, "hostname": "1807f10cfe48", "id": "01J31G9XHEFA79K827P0VQBYP9", "msg": "Collection 01J1XG0GJF2AK110287P265HJM has no items"} Traceback (most recent call last): File "/app/scripts/collection_from_items.py", line 187, in <module> main() File "/app/scripts/collection_from_items.py", line 172, in main raise Exception(f"Collection {collection_id} has no items") Exception: Collection 01J1XG0GJF2AK110287P265HJM has no itemsIf there is a non-item file in the source directory:
{"file": "afage/collection_from_items_id_mismatch/collection.json", "action": "collection_from_items", "reason": "skip", "level": 40, "time": 1721258457687, "v": 1, "pid": 1, "hostname": "2dbdb92c7ac4", "id": "01J31FZDP3AVF0B4SVTW3MGJBE", "msg": "skipping: not a STAC item"}Verification
collection-from-items.pyshould be refactored to be able to create unit tests for individual functions, including this functionality. This is outside the scope of this Pull Request.Testing was done by running the
collection-from-items.pycommand with a matching, and a mismatching ID.Example command (for ID mismatch, change
--collection-idto something like0123456789):❯ docker build -t topo-imagery . && docker run -v ${HOME}/tmp/:/tmp/:rw topo-imagery python /app/scripts/collection_from_items.py --uri /home/dev/collection_from_items_id_mismatch/ --collection-id 01J1XG0GJF2AK110287P365HJM --category urban-aerial-photos --region hawkes-bay --gsd 1m --start-date 2023-09-20 --end-date 2023-12-21 --lifecycle ongoing --producer "Placeholder" --licensor "Placeholder" --concurrency 25