Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corrupt Archive handling #10

Merged
merged 5 commits into from
Nov 29, 2023
Merged

Corrupt Archive handling #10

merged 5 commits into from
Nov 29, 2023

Conversation

griembauer
Copy link
Member

@griembauer griembauer commented Nov 27, 2023

This PR adapts SADASADAM as follows:

  • Solves Error in post processing #5: Corrupt .SAFE.zip/.SAFE/.tar.gz scene archives are removed and downloaded again (with up to three tries)
  • The download_dir does not have to be defined in the eodag-conf file, it is enough to define it in the SADASADAM config file
  • A bug is fixed that only a subset of all found scenes is downloaded. The search function of eodag provides search results by page, thus only downloading the first page, if the results are passed to the download_all-method. This occurs only for downloads of a large number of scenes. The search_all method is used instead

@griembauer griembauer changed the title [DRAFT] Corrupt Archive handling Corrupt Archive handling Nov 28, 2023
Copy link
Member

@mmacata mmacata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for that!

The try block in extract_and_delete_tar_gz_files seems really large, so that no detailed exception can be catched but only a general one. And it might be a bit unintuitive that the return value is corrupt_files. But overall it seems logical and it is clear why, so just take these as hints and feel free to merge :)

@griembauer
Copy link
Member Author

Thanks! I agree to your points - the try block is really large because we don't really know in what ways downloaded files can be corrupted and this way it is made sure they are removed and the code carries on even if an not anticipated error occurs during archive extraction.

@griembauer griembauer merged commit a9f11ad into main Nov 29, 2023
8 checks passed
@griembauer griembauer deleted the corrupt_zip branch November 29, 2023 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants