jar: skip corrupt jar files

[frostmar]

Some types of jar file corruption don't cause errors as soon as the file is opened with archive/zip, but only later when the scanner looks for specific zip entries. Currently the error causes the indexing to fail completely.

So: Log and skip any jar files where archive/zip errors with "not a zip file"

---

Example of a weirdly corrupt jar:
https://github.com/IBM/db2-samples/blob/master/repl/xmlpubtk/loadqueue/LoadQueue.jar

# file /tmp/LoadQueue.jar
/tmp/LoadQueue.jar: Java archive data (JAR)

# jar tf /tmp/LoadQueue.jar
java.util.zip.ZipException: invalid END header (bad central directory offset)
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:247)
	at java.util.zip.ZipFile.<init>(ZipFile.java:172)
	at java.util.zip.ZipFile.<init>(ZipFile.java:143)
	at sun.tools.jar.Main.list(Main.java:1127)
	at sun.tools.jar.Main.run(Main.java:305)
	at sun.tools.jar.Main.main(Main.java:1300)

[codecov]

Codecov Report

Patch coverage is 60.00% of modified lines.

Files Changed	Coverage
java/jar/jar.go	60.00%

📢 Thoughts on this report? Let us know!.

[crozzy]

Great stuff, thanks for the example jar. The change LGTM, it'd be create to have a test case around it. I'm thinking of either:

• If a public c/image exists somewhere with this jar we could add it to the packagescanner_test.TestScan testcases
• We could include the LoadQueue.jar in java/testdata and build a layer with it during the test and just check it no longer fails (gobin/gobin_test.go does something similar to embed a binary in a layer).

Thoughts?

[frostmar]

Yup, very reasonable to have a test, I only didn't include one as I couldn't see any existing ones to extend covering error scenarios.

I'm unsure if there's a freely available image containing the corrupt LoadQueue.jar, I'll go searching... if not, I'll check it's licencing and try the second approach to build a layer to scan

[frostmar]

I've added an integration test in the existing pattern.
The DB2 layer containing a corrupt file is ~2Gb (unpacked), so it may take some time to fetch - is that OK?

Passes for me locally, and fails without the code fix.

[crozzy]

I've added an integration test in the existing pattern. The DB2 layer containing a corrupt file is ~2Gb (unpacked), so it may take some time to fetch - is that OK?

Passes for me locally, and fails without the code fix.

Great, thanks for adding that, it looks like the average CI time is no slower so LGTM, just going to check this flaky test isn't indicative of something in this patch...

[hdonnay]

Why not figure out how the jar is corrupted and create a jar in-test that's suitably mangled?

[frostmar]

Why not figure out how the jar is corrupted and create a jar in-test that's suitably mangled?

If you'd like that, I'd appreciate assistance as I fear it may take some time to pick through the zip specs and fiddling to get an equivalent manufactured recreate. My team has several other urgent priorities, so the time I can invest right now is limited.

As the jar example in the description is a public DB2 sample, I think it could be used without licencing concerns, which may make reverse engineering to create an equivalent unnecessary.

[frostmar]

Is there anything needed from me to make this mergable? I'd like to get into the next release, as we have users affected by this

[hdonnay]

So I opened up the offending jar in a hex editor (after finding it, the comment in the test is wrong) and found that the offset for the central directory footer somehow has the wrong offset: It's off by 3 bytes (for 3 entries, so I think the producer has an off-by-one error somewhere).

I'll knock together some code to generate a similarly broken zip in the test.

Here's an IPS patch to fix if you're interested:
LoadQueue.patch.gz
(gzipped to placate GitHub). It's literally swapping one byte.

[hdonnay]

Should be good to go.

[crozzy]

LGTM, nice work debugging the corruption, just one question