CRC Calculations Are Ocassionally Incorrect

[barnstar]

The CRC calculation in the Data.process(...) is incorrect. In the case where the loop continues without stream.src_size getting set to zero, another pass with the same datablock is done on the CRC, causing it to be incorrect. compression_stream_process does not necessarily consume all the bytes.

unzip utilities on macOS are forgiving on incorrect CRCs (and ZIPFoundation itself lets you ignore them), but many other libraries will error out if the stored CRCs in the ZIP file don't match the CRC of the decoded file/bytes.

The correct action for COMPRESSION_STREAM_ENCODE is to run the CRC pass only when a new block of data is read from the provider.

[weichsel]

Thanks for making me aware of this.

Do you have a file at hand that showcases this behavior?

[andrewchang-bird]

I can repro this consistently when using ZIPFoundation to archive build products:

% unzip Mockingbird.zip
Archive:  Mockingbird.zip
  inflating: Mockingbird/LICENSE.md  
   creating: Mockingbird/Libraries/
  inflating: Mockingbird/Libraries/lib_InternalSwiftSyntaxParser.dylib  
  inflating: Mockingbird/Libraries/LICENSE.txt  
  inflating: Mockingbird/mockingbird   bad CRC 0e8ab0ad  (should be c9189cde)

Attached the file here: Mockingbird.zip

And here’s how we’re calling ZIPFoundation in our build pipeline: https://github.com/birdrides/mockingbird/blob/master/Sources/MockingbirdAutomationCli/Commands/Build.swift#L52-L55

You can repro it locally by checking out Mockingbird and running $ Sources/MockingbirdAutomationCli/buildAndRun.sh build cli --archive Mockingbird.zip

[weichsel]

Opened a PR here: #243

@andrewchang-bird Does this fix your issue?

[andrewchang-bird]

Yep, looks like bugfix/invalidCRC32 fixes it.

[weichsel]

@andrewchang-bird Thanks for confirming. Merged - will be part of next release.

[marcussc]

@weichsel thanks for this nice project and helping to get this issue fixed. Do you have an estimation when the next version with the fix will go live?

[weichsel]

Hi @marcussc
No ETA yet. Currently looking into another issue that still needs some time.