Fix multi-release jar problem #11185
Conversation
Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
liurenjie1024
requested review from
jlowe,
revans2,
tgravescs,
GaryShen2008,
NvTimLiu and
gerashegalov
as code owners
|
build |
jenkins/Jenkinsfile-blossom.premerge
Outdated
| @@ -197,7 +197,8 @@ git --no-pager diff --name-only HEAD \$BASE -- ${PREMERGE_DOCKERFILE} || true""" | |||
| execPattern : '**/target/jacoco.exec', | |||
| classPattern : 'target/jacoco_classes/', | |||
| sourceInclusionPattern: '**/*.java,**/*.scala', | |||
| sourcePattern : sourcePattern | |||
| sourcePattern : sourcePattern, | |||
| exclude : '*/META-INF/versions/*', | |||
There was a problem hiding this comment.
cc @NvTimLiu Could you help to confirm this part?
There was a problem hiding this comment.
the exclude param should be exclusionPattern, let me double conform
Also exclude only not cp the dup classes under target/jacoco_classes/*/META-INF/versions/* dir, the due classes still exist in the dist jar
There was a problem hiding this comment.
The goal of this pr is not removing due classes, and they are in fact not duplicated classes, they are shim layer of jdk 11.
There was a problem hiding this comment.
@liurenjie1024
exclusionPattern : 'target/jacoco_classes/*/META-INF/versions/*', does not work here,
Looks like [Jacoco plugin] run below steps
1, firstly tries to gather all classes together;
2 ,then it excludes classes defined by exclusionPattern
The first step fails due to the dup ArraysShim.class
16:06:54 java.lang.IllegalStateException: Can't add different class with same name: com/nvidia/shaded/spark/org/roaringbitmap/ArraysShim
16:06:54 at org.jacoco.core.analysis.CoverageBuilder.visitCoverage(CoverageBuilder.java:106)
16:06:54 at org.jacoco.core.analysis.Analyzer$1.visitEnd(Analyzer.java:99)
16:06:54 at org.objectweb.asm.ClassVisitor.visitEnd(ClassVisitor.java:377)
16:06:54 at org.jacoco.core.internal.flow.ClassProbesAdapter.visitEnd(ClassProbesAdapter.java:100)
16:06:54 at org.objectweb.asm.ClassReader.accept(ClassReader.java:725)
16:06:54 at org.objectweb.asm.ClassReader.accept(ClassReader.java:401)
16:06:54 at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:116)
16:06:54 at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:132)
16:06:54 Caused: java.io.IOException: Error while analyzing /var/jenkins/jobs/rapids_premerge-github/builds/9756/jacoco/classes/spark-shared/com/nvidia/shaded/spark/org/roaringbitmap/ArraysShim.class.
16:06:54 at org.jacoco.core.analysis.Analyzer.analyzerError(Analyzer.java:162)
16:06:54 at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:134)
16:06:54 at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:157)
I guess this is why instead of with param exclusionPattern for [Jacoco Plugin],
we remove folders in advance here: https://github.com/NVIDIA/spark-rapids/blob/branch-24.08/jenkins/spark-premerge-build.sh#L95-L96
BTW, the pre-merge CI failed, because the path of ArraysShim.class in sparkxxx/MATE-INFO has been changed with this PR:
spark320/META-INF/versions/11/org/roaringbitmap/ -- > /META-INF/versions/11/com/nvidia/shaded/spark/org/roaringbitmap/
So please help to update the line?
https://github.com/NVIDIA/spark-rapids/blob/branch-24.08/jenkins/spark-premerge-build.sh#L95
rm -rf com/nvidia/shaded/ org/openucx/ spark${SPK_VER}/META-INF/versions/* spark-shared/com/nvidia/spark/udf/ spark${SPK_VER}/com/nvidia/spark/udf/
There was a problem hiding this comment.
Thanks for update, I've fixed the script.
|
please also help revert the TODO (spark${SPK_VER}/META-INF/versions/*/org/roaringbitmap/) at |
Done, PTAL. cc @pxLi |
|
build |
|
CI still failed.
|
|
build |
Close #11175 .
Multi-release-jar is introduced in jdk to support conditional compilation. This fix includes two things:
See apache/pulsar#22964 as an example.