Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Maven build reproducible #2543

Merged
merged 1 commit into from
Nov 18, 2023
Merged

Make Maven build reproducible #2543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
cache: 'maven'
- name: Build with Maven
# This also runs javadoc:jar to detect any issues with the Javadoc generated during release
run: mvn --batch-mode --update-snapshots --no-transfer-progress verify javadoc:jar
run: mvn --batch-mode --no-transfer-progress verify javadoc:jar

native-image-test:
name: "GraalVM Native Image test"
Expand All @@ -42,4 +42,29 @@ jobs:
- name: Build and run tests
# Only run tests in `graal-native-image-test` (and implicitly build and run tests in `gson`),
# everything else is covered already by regular build job above
run: mvn test --batch-mode --update-snapshots --no-transfer-progress --activate-profiles native-image-test --projects graal-native-image-test --also-make
run: mvn test --batch-mode --no-transfer-progress --activate-profiles native-image-test --projects graal-native-image-test --also-make

verify-reproducible-build:
name: "Verify reproducible build"
Comment on lines +47 to +48
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have included this as separate job (instead of including it in the steps of the build above) because it requires running mvn ... install which is normally not needed for a regular build, and also to keep it independent from the 'real' build setup above.

However, this makes the GitHub workflow slightly slower though.

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- name: "Set up JDK 17"
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: 17
cache: 'maven'

- name: "Verify no plugin issues"
run: mvn artifact:check-buildplan --batch-mode --no-transfer-progress

- name: "Verify reproducible build"
# See https://maven.apache.org/guides/mini/guide-reproducible-builds.html#how-to-test-my-maven-build-reproducibility
run: |
mvn clean install --batch-mode --no-transfer-progress -Dproguard.skip -DskipTests
# Run with `-Dbuildinfo.attach=false`; otherwise `artifact:compare` fails because it creates a `.buildinfo` file which
# erroneously references the existing `.buildinfo` file (respectively because it is overwriting it, a file with size 0)
# See https://issues.apache.org/jira/browse/MARTIFACT-57
mvn clean verify artifact:compare --batch-mode --no-transfer-progress -Dproguard.skip -DskipTests -Dbuildinfo.attach=false
66 changes: 66 additions & 0 deletions ReleaseProcess.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,72 @@ This section was borrowed heavily from [Doclava release process](https://code.go

See [OSSRH Publish Guide](https://central.sonatype.org/publish/publish-guide/).

## Testing Maven release workflow locally

The following describes how to perform the steps of the release locally to verify that they work as desired.

**Warning:** Be careful with this, these steps might be outdated or incomplete. Doublecheck that you are working on a copy of your local Gson Git repository and make sure you have followed all steps. To be safe you can also temporarily turn off your internet connection to avoid accidentally pushing changes to the real remote Git or Maven repository.\
Comment on lines +40 to +44
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This setup is a bit hacky and I am not sure if there is a proper better alternative to this, therefore I added this "Warning". But it does allow you to test the behavior of the release workflow quite accurately, which is why I added it.

As an alternative to the steps described below you can instead [perform a dry run](https://maven.apache.org/maven-release/maven-release-plugin/usage.html#do-a-dry-run), though this might not behave identical to a real release.

1. Make a copy of your local Gson Git repository and only work with that copy
2. Make sure you are on the `main` branch
3. Create a temp directory outside the Gson directory\
In the following steps this will be called `#gson-remote-temp#`; replace this with the actual absolute file path of the directory, using only forward slashes. For example under Windows `C:\my-dir` becomes `C:/my-dir`.
4. Create the directory `#gson-remote-temp#/git-repo`
5. In that directory run

```sh
git init --bare --initial-branch=main .
```

6. Create the directory `#gson-remote-temp#/maven-repo`
7. Edit the root `pom.xml` of Gson
1. Change the `<developerConnection>` to

```txt
scm:git:file:///#gson-remote-temp#/git-repo
```

2. Change the `<url>` of the `<distributionManagement>` to

```txt
file:///#gson-remote-temp#/maven-repo
```

3. If you don't want to use GPG, remove the `maven-gpg-plugin` entry from the 'release' profile.\
There is also an entry under `<pluginManagement>`; you can remove that as well.
8. Commit the changes using Git
9. Change the remote repository of the Git project

<!-- Uses `txt` instead of `sh` to avoid the `#` being highlighted in some way -->
```txt
git remote set-url origin file:///#gson-remote-temp#/git-repo
```

10. Push the changes

```sh
git push origin main
```

Now you can perform the steps of the release:

1. ```sh
mvn release:clean
```

2. ```sh
mvn release:prepare
```

3. ```sh
mvn release:perform
```

4. Verify that `#gson-remote-temp#/git-repo` and `#gson-remote-temp#/maven-repo` contain all the desired changes
5. Afterwards delete all Gson files under `${user.home}/.m2/repository/com/google/code/gson` which have been installed in your local Maven repository during the release.\
Otherwise Maven might not download the real Gson artifacts with these version numbers, once they are released.

## Running Benchmarks or Tests on Android

* Download vogar
Expand Down
6 changes: 6 additions & 0 deletions extras/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@
<name>Gson Extras</name>
<description>Google Gson grab bag of utilities, type adapters, etc.</description>

<properties>
<!-- Make the build reproducible, see root `pom.xml` -->
<!-- This is duplicated here because that is recommended by `artifact:check-buildplan` -->
<project.build.outputTimestamp>2023-01-01T00:00:00Z</project.build.outputTimestamp>
</properties>

<licenses>
<license>
<name>Apache-2.0</name>
Expand Down
8 changes: 5 additions & 3 deletions graal-native-image-test/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,7 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

Expand All @@ -27,6 +25,10 @@
<artifactId>graal-native-image-test</artifactId>

<properties>
<!-- Make the build reproducible, see root `pom.xml` -->
<!-- This is duplicated here because that is recommended by `artifact:check-buildplan` -->
<project.build.outputTimestamp>2023-01-01T00:00:00Z</project.build.outputTimestamp>

<!-- GraalVM is JDK >= 17, however for build with regular JDK these tests
are also executed with JDK 11, so for them exclude JDK 17 specific tests -->
<maven.compiler.testRelease>11</maven.compiler.testRelease>
Expand Down
4 changes: 4 additions & 0 deletions gson/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@
</licenses>

<properties>
<!-- Make the build reproducible, see root `pom.xml` -->
<!-- This is duplicated here because that is recommended by `artifact:check-buildplan` -->
<project.build.outputTimestamp>2023-01-01T00:00:00Z</project.build.outputTimestamp>

<excludeTestCompilation>**/Java17*</excludeTestCompilation>
</properties>

Expand Down
6 changes: 6 additions & 0 deletions metrics/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@
<name>Gson Metrics</name>
<description>Performance Metrics for Google Gson library</description>

<properties>
<!-- Make the build reproducible, see root `pom.xml` -->
<!-- This is duplicated here because that is recommended by `artifact:check-buildplan` -->
<project.build.outputTimestamp>2023-01-01T00:00:00Z</project.build.outputTimestamp>
</properties>

<licenses>
<license>
<name>Apache-2.0</name>
Expand Down
34 changes: 29 additions & 5 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,7 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"
child.project.url.inherit.append.path="false">
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd" child.project.url.inherit.append.path="false">
<modelVersion>4.0.0</modelVersion>

<groupId>com.google.code.gson</groupId>
Expand All @@ -40,6 +39,10 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.release>7</maven.compiler.release>
<maven.compiler.testRelease>11</maven.compiler.testRelease>

<!-- Make the build reproducible, see https://maven.apache.org/guides/mini/guide-reproducible-builds.html -->
<!-- Automatically updated by Maven Release Plugin -->
<project.build.outputTimestamp>2023-01-01T00:00:00Z</project.build.outputTimestamp>
</properties>

<!-- These attributes specify that the URLs should be inherited by the modules as is, to avoid constructing
Expand Down Expand Up @@ -152,8 +155,8 @@
<!-- For Markdown files removing trailing whitespace causes issues for hard line breaks,
which use two trailing spaces. However, the trailing spaces are difficult to notice anyway;
prefer a trailing `\` instead of two spaces. -->
<trimTrailingWhitespace/>
<endWithNewline/>
<trimTrailingWhitespace />
<endWithNewline />
<indent>
<spaces>true</spaces>
<!-- This seems to mostly (or only?) affect the suggested fix in case code contains tabs -->
Expand All @@ -175,10 +178,29 @@
<reorderImports>true</reorderImports>
<formatJavadoc>true</formatJavadoc>
</googleJavaFormat>
<formatAnnotations/> <!-- Puts type annotations immediately before types. -->
<formatAnnotations /> <!-- Puts type annotations immediately before types. -->
</java>
</configuration>
</plugin>

<!-- Attaches a `.buildinfo` file which contains information for reproducing the build,
such as OS, JDK version, ...
Since this is a multi-module Maven project, only one aggregated file will be created for
the last module, see the note on https://maven.apache.org/plugins/maven-artifact-plugin/usage.html#recording-buildinfo-file -->
<!-- The other goals of this plugin are run by the GitHub workflow to verify that
the build is reproducible (see `artifact:...` usage in the workflow) -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-artifact-plugin</artifactId>
Comment on lines +186 to +194
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This includes information about the build environment to make it easier to reproduce the build. I am not sure if you would consider some of this information sensitive (e.g. OS name and version or Java vendor and version).

For example for me the generated file looks like this:

gson-2.10.2.buildinfo (click to expand) 
# https://reproducible-builds.org/docs/jvm/
buildinfo.version=1.0-SNAPSHOT

name=Gson Parent
group-id=com.google.code.gson
artifact-id=gson-parent
version=2.10.2

# source information
source.scm.uri=scm:git:https://github.com/google/gson.git
source.scm.tag=gson-parent-2.10.2

# build instructions
build-tool=mvn

# effective build environment information
java.version=17.0.6
java.vendor=Eclipse Adoptium
os.name=Windows 10

# Maven rebuild instructions and effective environment
mvn.version=3.9.1
mvn.aggregate.artifact-id=gson

# aggregated output

outputs.0.coordinates=com.google.code.gson:gson-parent

outputs.0.0.groupId=com.google.code.gson
outputs.0.0.filename=gson-parent-2.10.2.pom
outputs.0.0.length=20999
outputs.0.0.checksums.sha512=36a916aea7d2e261e62a9287d0d9a2fcf9cdd94f1658bfba616dc3e9a08121b3ff080e5b574f32102311b8c6d286db977dd5b7038c5b4841c0fd63a164fecd93

outputs.1.coordinates=com.google.code.gson:gson

outputs.1.0.groupId=com.google.code.gson
outputs.1.0.filename=gson-2.10.2.pom
outputs.1.0.length=11602
outputs.1.0.checksums.sha512=e16df3d3a571fee3d6145007eb7eb20dcef0a7c9707d7ea347341cf3708ea73f197550076336dc3bb29c842ee4d00ad2ce5a9ae10302d98ac55d4437d3d7949c

outputs.1.1.groupId=com.google.code.gson
outputs.1.1.filename=gson-2.10.2.jar
outputs.1.1.length=307320
outputs.1.1.checksums.sha512=300dc667e042cb8dcc65fcf97b169b2b15bec96d4728164649f2e9dff2cc1a8b4b1f1053f12875fe59395fb42bac8abed49a5d410ecfcae8ca2287f9f8e9ada1

outputs.1.2.groupId=com.google.code.gson
outputs.1.2.filename=gson-2.10.2-sources.jar
outputs.1.2.length=202441
outputs.1.2.checksums.sha512=aa94a042ee8e3a985329eb421853668a2660ef5f1b70c5fdbcfa0b6f2cbb203ae07275ccaba02fb418773fc89ba36984e1671ee832ebe8eb31c97436e84761c9

<version>3.5.0</version>
<executions>
<execution>
<goals>
<goal>buildinfo</goal>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This causes a warning during the build:

[WARNING] SCM source tag in buildinfo source.scm.tag=HEAD does not permit rebuilders reproducible source checkout

But I guess that can be safely ignored because during release source.scm.tag will be temporarily changed by the Maven Release Plugin.

</goals>
</execution>
</executions>
</plugin>
</plugins>

<pluginManagement>
Expand Down Expand Up @@ -350,6 +372,7 @@
</execution>
</executions>
</plugin>

<!-- Plugin for checking source and binary compatibility; used by GitHub workflow -->
<plugin>
<groupId>com.github.siom79.japicmp</groupId>
Expand Down Expand Up @@ -381,6 +404,7 @@
</parameter>
</configuration>
</plugin>

<!-- Plugin for checking compatibility with Android API -->
<!-- Note: For now this is not part of a normal Maven build but instead executed only by a
GitHub workflow because the Animal Sniffer signature files use Java Serialization, so they
Expand Down
6 changes: 6 additions & 0 deletions proto/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@
<name>Gson Protobuf Support</name>
<description>Gson support for Protobufs</description>

<properties>
<!-- Make the build reproducible, see root `pom.xml` -->
<!-- This is duplicated here because that is recommended by `artifact:check-buildplan` -->
<project.build.outputTimestamp>2023-01-01T00:00:00Z</project.build.outputTimestamp>
</properties>

<licenses>
<license>
<name>Apache-2.0</name>
Expand Down
8 changes: 5 additions & 3 deletions shrinker-test/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,7 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
Expand All @@ -28,6 +26,10 @@
<artifactId>shrinker-test</artifactId>

<properties>
<!-- Make the build reproducible, see root `pom.xml` -->
<!-- This is duplicated here because that is recommended by `artifact:check-buildplan` -->
<project.build.outputTimestamp>2023-01-01T00:00:00Z</project.build.outputTimestamp>

<maven.compiler.release>8</maven.compiler.release>
</properties>

Expand Down