[kemono.party]Corrupted PNGs when downloading using version 1.24.3

[FriedGenera]

The pngs downloaded after upgrading to 1.24.3 all seem to be corrupted. I downgraded to previous version and it works fine.

[kattjevfel]

473bd38 seems to be at fault, I have a build of the commit prior to it and it works fine.

[mikf]

gallery-dl/gallery_dl/extractor/kemonoparty.py

Lines 202 to 203 in 473bd38

	return (response.headers["content-length"] != "9" and
	response.content != b"not found")

This "and" should have been an "or" and almost all bugs this is causing now would have been avoided. Accessing response.content here for basically every file causes the first 16 bytes to be duplicated:

00000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452  .PNG........IHDR
00000010: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452  .PNG........IHDR
00000020: 0000 0b4e 0000 0ffd 0806 0000 00c7 9db6  ...N............

I ran all extractor result tests and even added a new one. They all passed so I thought it was OK like this. Oh well. Sorry. Guess I'll be pushing another release this evening.

[mikf]

Fixed in commit 85bd1cb.
New release at https://github.com/mikf/gallery-dl/releases/tag/v1.24.4.

[reyaz006]

What's the safe way to fix the issues with broken files if I use archive (don't wan to re-download something I removed earlier)? I can't just wipe latest data inside the .sqlite3 file?

[Hrxn]

What's the safe way to fix the issues with broken files if I use archive (don't wan to re-download something I removed earlier)? I can't just wipe latest data inside the .sqlite3 file?

Yes, you can quite easily actually: https://sqlitebrowser.org/

[reyaz006]

I mean there is no column for date. How do I sort by date, or select entries created on specific date and remove them?

[Hrxn]

Uhh.. not sure.
By checking the first bytes of the local files on the filesystem then? Aren't they identical for every corrupted file?

[reyaz006]

They are identical for each individual binary file (and not just PNG) only. I found them all by creation date, but I'll need to make some script that would properly find such duped bytes and remove them.

[Hrxn]

Well, if you also save the ID inside the filename, and not just in archive-format, we'd known the relevant IDs for the archive file..

[reyaz006]

Yes, there are ~4000 files with ids in their names. I guess I could list, find them inside the .sqlite3 and remove. But that seems like too much work compared to a script.

[kattjevfel]

If you can identify all the bad files, say with file (it should show up as PNG image data, -1991225785 x 218765834, 0-bit grayscale), you can then move them somewhere and remove the first 16 bytes, I'd do something like this:

for f in ./*.png; do dd if="$f" of="$f"_fixed.png ibs=16 skip=1; done

This will remove the first 16 bytes of all pngs in the folder and output them to filename.png_fixed.png

[reyaz006]

Solved the problem using sfk.exe and a batch script:

@echo off
move %1 tempfile
sfk hexdump -raw -offlen 0 16 tempfile >1.txt
sfk hexdump -raw -offlen 16 16 tempfile >2.txt
sfk md5 1.txt 2.txt
if %ERRORLEVEL% equ 0 goto :process
echo ABORT
goto :skip

:process
move tempfile !._16broken
sfk partcopy !._16broken -allfrom 16 !._16fixed -yes
move !._16fixed %1
echo DONE
goto :skip2

:skip
move tempfile %1

:skip2

Usage for each file: dothething.bat "D:\path to broken file\file.zip"