Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kemonoparty] improve hash extraction #3531

Merged
merged 1 commit into from
Jan 16, 2023
Merged

Conversation

ClosedPort22
Copy link
Contributor

No description provided.

@mikf
Copy link
Owner

mikf commented Jan 14, 2023

Why not just use | in the regular expression? Seems a lot simpler.

diff --git a/gallery_dl/extractor/kemonoparty.py b/gallery_dl/extractor/kemonoparty.py
index 63e30841..f5b0e8bd 100644
--- a/gallery_dl/extractor/kemonoparty.py
+++ b/gallery_dl/extractor/kemonoparty.py
@@ -41,7 +41,10 @@ class KemonopartyExtractor(Extractor):
         self._find_inline = re.compile(
             r'src="(?:https?://(?:kemono|coomer)\.party)?(/inline/[^"]+'
             r'|/[0-9a-f]{2}/[0-9a-f]{2}/[0-9a-f]{64}\.[^"]+)').findall
-        find_hash = re.compile("/[0-9a-f]{2}/[0-9a-f]{2}/([0-9a-f]{64})").match
+        find_hash = re.compile(r"/(?:"
+            r"[0-9a-f]{2}/[0-9a-f]{2}/([0-9a-f]{64})|"
+            r"attachments/\w+/\w+/\w+/([0-9a-f]{32}))"
+        ).match
         generators = self._build_file_generators(self.config("files"))
         duplicates = self.config("duplicates")
         comments = self.config("comments")
@@ -88,7 +91,7 @@ class KemonopartyExtractor(Extractor):
 
                 match = find_hash(url)
                 if match:
-                    file["hash"] = hash = match.group(1)
+                    file["hash"] = hash = match.group(1) or match.group(2)
                     if hash in hashes and not duplicates:
                         self.log.debug("Skipping %s (duplicate)", url)
                         continue

or even

diff --git a/gallery_dl/extractor/kemonoparty.py b/gallery_dl/extractor/kemonoparty.py
index 63e30841..b6105f21 100644
--- a/gallery_dl/extractor/kemonoparty.py
+++ b/gallery_dl/extractor/kemonoparty.py
@@ -41,7 +41,9 @@ class KemonopartyExtractor(Extractor):
         self._find_inline = re.compile(
             r'src="(?:https?://(?:kemono|coomer)\.party)?(/inline/[^"]+'
             r'|/[0-9a-f]{2}/[0-9a-f]{2}/[0-9a-f]{64}\.[^"]+)').findall
-        find_hash = re.compile("/[0-9a-f]{2}/[0-9a-f]{2}/([0-9a-f]{64})").match
+        find_hash = re.compile(
+            r"/(?:[0-9a-f]{2}/[0-9a-f]{2}|attachments/\w+/\w+/\w+)"
+            r"/([0-9a-f]{32,})").match
         generators = self._build_file_generators(self.config("files"))
         duplicates = self.config("duplicates")
         comments = self.config("comments")

@ClosedPort22
Copy link
Contributor Author

Yeah that's a lot simpler. Can't believe I forgot to use alternation first 🤦‍♂️

- extract MD5 hash from URLs
- extract MD5 and SHA256 hash from Discord URLs (kemono.party only)
- minor optimization (do not call 'hashes.add' when 'duplicates' is
  true)
- update tests accordingly

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
@ClosedPort22 ClosedPort22 marked this pull request as ready for review January 15, 2023 04:04
@mikf mikf merged commit 35a3049 into mikf:master Jan 16, 2023
@ClosedPort22 ClosedPort22 deleted the kemono-md5 branch January 16, 2023 14:35
@AlttiRi
Copy link

AlttiRi commented Jan 23, 2023

I think it's just a coincidence that the filename can be the same as MD5 of the file.

@mikf
Copy link
Owner

mikf commented Jan 24, 2023

That seems to be the case. The majority of old attachment URLs from https://kemono.party/subscribestar/user/alcorart have regular, non-MD5 filenames:

https://kemono.party/data/attachments/subscribestar/alcorart/175364/100.png
...
https://kemono.party/data/attachments/subscribestar/alcorart/171196/page_13.png
https://kemono.party/data/attachments/subscribestar/alcorart/166970/page_12.png
https://kemono.party/data/attachments/subscribestar/alcorart/162778/page_10-33.png

with the test URL being the odd one out and having an MD5 hash in it by coincidence. The same seems to be true for other subscribestar artists as well.

@ClosedPort22
Copy link
Contributor Author

I discovered that URL by pure chance while I was working on #3532. What are the odds of that? 🤔

@AlttiRi
Copy link

AlttiRi commented Jan 24, 2023

I checked my files, only 80 files of 190k have filename is matched MD5 pattern.

Most likely, these authors upload an artwork to a booru site first, then download it from the site with the right click and upload it to another site (patreon, fanbox, subscribestar).

More over the r"attachments/\w+/\w+/\w+/([0-9a-f]{32}))" regex is suited only for the old Kemono URLs, which I think are less than 1 % (or maybe even less than 0.1 %) now.

Additionally, filename may have a wrong MD5 value, or something that is looked like this [0-9a-f]{32}.
(for example, 11111111111111111111111111111111111.png)

@ClosedPort22
Copy link
Contributor Author

Additionally, filename may have a wrong MD5 value, or something that is looked like this [0-9a-f]{32}.

Yeah, that's the biggest issue. I didn't know these old-styled URLs existed.

mikf added a commit that referenced this pull request Jan 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants