Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalize timestamps and file ordering in jars, making the outputs reproducible #6471

Merged
merged 1 commit into from
May 28, 2024

Conversation

DPUkyle
Copy link
Contributor

@DPUkyle DPUkyle commented May 22, 2024

Fixes #4488

The changes I added to the Java convention plugin will remove the common sources of non-determinism from the project's jar files, namely file creation/modification dates stored in zip catalog metadata.

The net result is that for future releases, a user would be able to checkout a specific tag, rebuild the code locally and produce jars that are bitwise identical (matching checksums) to those releases available on Maven Central.

As for testing: I was able to test this by building the project twice in distinct folders, then spot-checking about ten of the jars (sdk, api, and context libraries plus their -javadoc and -source jars), and the opentelemetry-sdk-trace-shaded-deps-1.39.0-SNAPSHOT-all.jar. All had identical SHA1 checksums, which is the goal.

Please let me know how else we can validate that these changes can pass tests, I'm eager to help.

@DPUkyle DPUkyle requested a review from a team May 22, 2024 18:57
Copy link

linux-foundation-easycla bot commented May 22, 2024

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: DPUkyle / name: Kyle Moore (aa81c85)

Copy link

codecov bot commented May 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.87%. Comparing base (a764419) to head (aa81c85).

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #6471   +/-   ##
=========================================
  Coverage     90.87%   90.87%           
  Complexity     6169     6169           
=========================================
  Files           678      678           
  Lines         18507    18507           
  Branches       1818     1818           
=========================================
  Hits          16819    16819           
  Misses         1153     1153           
  Partials        535      535           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@jkwatson jkwatson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Copy link
Member

@jack-berg jack-berg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is indeed the advice to make builds reproducible from gradle docs: https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives

I verified that on the current main branch, builds are not reproducible: if you run the build twice, the resulting jars will not have the same md5 checksum result. On this branch, if you run the build twice, the resulting jars have matching md5 checksums.

Thanks @DPUkyle!

@jack-berg jack-berg merged commit a1c72d1 into open-telemetry:main May 28, 2024
18 checks passed
@hboutemy
Copy link

hboutemy commented Jun 8, 2024

here is the result on the 1.39.0 release: https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/content/io/opentelemetry/java/README.md

there is an improvement but 2 issues remain in generated MANIFEST.MF https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/content/io/opentelemetry/java/opentelemetry-sdk-1.39.0.diffoscope :

  1. Built-By: is it really useful to record the user who built?
  2. Built-JDK: is there a way to enforce a precise JDK patch version when using official Docker image? https://hub.docker.com/_/gradle

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

please make OpenTelemetry Java build reproducible
4 participants