Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hash checking for extractor.*.skip #530

Closed
panhartstuff opened this issue Dec 23, 2019 · 1 comment
Closed

Hash checking for extractor.*.skip #530

panhartstuff opened this issue Dec 23, 2019 · 1 comment

Comments

@panhartstuff
Copy link

It would be nice to have hash checking options for the extractor.*.skip configuration
Very useful in a scenario where a module gets updated with a better downloader (i.e. grabs better image quality)

Some suggestions:

  • "hashoverwrite": Downloads the file again, then compare the hash with the older version of the file. If they mismatch, overwrite the older version
  • "hashenumerate": Downloads the file again, then compare the hash with the older version of the file. If they mismatch, add an enumeration index to the beginning of the filename extension (file.1.ext, file.2.ext, etc.)
@mikf
Copy link
Owner

mikf commented Jan 18, 2020

This functionality is now implemented as a post processor module: 254f7c3
(It's a lot easier to do it like that)

It requires extractor.skip to be set to false, so it re-downloads the file even though it was downloaded before, and downloader.part set to true (the default), so it writes the downloaded file to a temporary location first.

It will compare old and new versions of a file by comparing their size and then their content, and replace/enumerate them on mismatch. No hashing, because that's at least twice as slow than a direct comparison.

Is that (roughly) what you had in mind?

@mikf mikf closed this as completed Dec 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants