Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] repeated runs of mvn package without clean lead to missing spark-rapids-jni-version-info.properties in dist jar #5777

Closed
gerashegalov opened this issue Jun 7, 2022 · 2 comments · Fixed by #5779
Assignees
Labels
bug Something isn't working build Related to CI / CD or cleanly building

Comments

@gerashegalov
Copy link
Collaborator

gerashegalov commented Jun 7, 2022

Describe the bug
If the user invokes repeatedly mvn package on dist without clean, the dist jar ends up with a missing spark-rapids-jni-version-info.properties and the Plugin refuses to initialize.

Steps/Code to reproduce bug
Clean just once:

mvn clean

Run package for dist module using

mvn package -pl dist -am -Dbuildver=321 -DskipTests

Check the resulting jar has the build info for spark-rapids-jni

jar tvf dist/target/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar  | grep spark-rapids-jni-version-info.properties
   170 Tue Jun 07 14:28:34 PDT 2022 spark-rapids-jni-version-info.properties

Re-run package for dist module using the same command:

mvn package -pl dist -am -Dbuildver=321 -DskipTests

Recheck the build info for spark-rapids-jni, and observe it's missing

jar tvf dist/target/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar  | grep spark-rapids-jni-version-info.properties
echo $?
1

Expected behavior
mvn package with the same parameters should be reproducible and result in equivalent artifacts without a clean

As a workaround use mvn clean package/install ...

Environment details (please complete the following information)
Any

Additional context
Maybe related to #5603
h/t @NVnavkumar @anthony-chang

@gerashegalov gerashegalov added bug Something isn't working ? - Needs Triage Need team to review and classify build Related to CI / CD or cleanly building labels Jun 7, 2022
@res-life
Copy link
Collaborator

res-life commented Jun 8, 2022

Yes, it's related to #5603

For the second running mvn package -pl dist -am -Dbuildver=321 -DskipTest,
the unpack-spark-rapids-jni-and-ucx execution in dist pom stops unpacking the JNI and the UCX jars to dist/target/parallel-world, and it prints: already unpacked

[INFO] --- maven-dependency-plugin:2.8:unpack (unpack-spark-rapids-jni-and-ucx) @ rapids-4-spark_2.12 ---
[INFO] Configured Artifact: com.nvidia:spark-rapids-jni:cuda11:?:jar
[INFO] Configured Artifact: org.openucx:jucx:?:jar
[INFO] spark-rapids-jni-22.08.0-SNAPSHOT-cuda11.jar already unpacked.
[INFO] jucx-1.12.1.jar already unpacked.

I'm curious where mvn records the information.

The dist/target/parallel-world directory is deleted when building dist, so the JNI and the UCX are not exist in the dist.jar

@res-life
Copy link
Collaborator

res-life commented Jun 8, 2022

ls -lh rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar

// first time
377M  6  8 10:41 rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar

// second time
7.2M  6  8 10:44 rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar

@res-life res-life self-assigned this Jun 8, 2022
@gerashegalov gerashegalov changed the title [BUG] repeated runs mvn package without clean lead to missing spark-rapids spark-rapids-jni-version-info.properties in dist jar [BUG] repeated runs of mvn package without clean lead to missing spark-rapids-jni-version-info.properties in dist jar Jun 8, 2022
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Jun 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working build Related to CI / CD or cleanly building
Projects
None yet
3 participants