Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disable Spark 3.4.0 premerge for 22.08 and enable for 22.10 #5809

Merged
merged 5 commits into from
Jun 14, 2022
Merged

Disable Spark 3.4.0 premerge for 22.08 and enable for 22.10 #5809

merged 5 commits into from
Jun 14, 2022

Conversation

tgravescs
Copy link
Collaborator

@tgravescs tgravescs commented Jun 13, 2022 •
edited
Loading

fixes #5808

Disable the Spark 3.4 build for premerge since its not stable and changing frequently. We will re-enable in 22.10 when its closer to release.
Signed-off-by: Thomas Graves tgraves@nvidia.com

@tgravescs tgravescs added the build Related to CI / CD or cleanly building label Jun 13, 2022
@tgravescs tgravescs added this to the Jun 6 - Jun 17 milestone Jun 13, 2022
@tgravescs tgravescs requested a review from jlowe as a code owner June 13, 2022 19:32
@tgravescs tgravescs self-assigned this Jun 13, 2022
@tgravescs
Copy link
Collaborator Author

build

revans2
revans2 previously approved these changes Jun 13, 2022
View reviewed changes
jlowe
jlowe previously approved these changes Jun 13, 2022
View reviewed changes
@tgravescs tgravescs dismissed stale reviews from jlowe and revans2 via c43b62e June 13, 2022 21:09
@tgravescs
Copy link
Collaborator Author

build

@gerashegalov
Copy link
Collaborator

gerashegalov commented Jun 13, 2022 •
edited
Loading

It's still trying to include 340 because it's part of snapshot.buildvers in dist/pom.xml but now it came from cache (we either need to update the property and file an issue for a single source of truth to avoid running into this) :

[2022-06-13T22:10:21.501Z] [ERROR] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-4907/dist/maven-antrun/build-parallel-worlds.xml:163: exec check-shims-revisions.sh failed, exit code is 1, error msg is Check Failed: git revisions between shims are not equal
...
Error: 6-13T22:10:21.501Z] [ERROR] Version file /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-4907/dist/target/parallel-world/spark330/rapids4spark-version-info.properties, revision is 65f2d79e1cc6fda420ffe5a0716da54d9b58c1d0
Error: 6-13T22:10:21.501Z] [ERROR] Version file /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-4907/dist/target/parallel-world/spark340/rapids4spark-version-info.properties, revision is 165ec8127f829ede29c69cff729fcee82bcbd8e2

@pxLi
Copy link
Collaborator

pxLi commented Jun 14, 2022

It's still trying to include 340 because it's part of snapshot.buildvers in dist/pom.xml but now it came from cache (we either need to update the property and file an issue for a single source of truth to avoid running into this) :

[2022-06-13T22:10:21.501Z] [ERROR] /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-4907/dist/maven-antrun/build-parallel-worlds.xml:163: exec check-shims-revisions.sh failed, exit code is 1, error msg is Check Failed: git revisions between shims are not equal
...
Error: 6-13T22:10:21.501Z] [ERROR] Version file /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-4907/dist/target/parallel-world/spark330/rapids4spark-version-info.properties, revision is 65f2d79e1cc6fda420ffe5a0716da54d9b58c1d0
Error: 6-13T22:10:21.501Z] [ERROR] Version file /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-4907/dist/target/parallel-world/spark340/rapids4spark-version-info.properties, revision is 165ec8127f829ede29c69cff729fcee82bcbd8e2

Yes, also nightly build would also be blocked by the issue.
https://github.com/NVIDIA/spark-rapids/blob/branch-22.08/jenkins/version-def.sh#L52

There is an issue #5086 for tracking, it would nice to use maybe pom file to be the source of truth for all build/test scripts

@tgravescs
Copy link
Collaborator Author

yeah I forgot we put it in all the builds, I'll remove everywhere and we can create a totally separate nightly for spark 3.4

@tgravescs
Copy link
Collaborator Author

filed #5823 to add separate nightly for 3.4.
filed #5824 to make sure we add 3.4 back into build in 22.10

@tgravescs
Copy link
Collaborator Author

build

@tgravescs tgravescs merged commit 4b9438a into NVIDIA:branch-22.08 Jun 14, 2022
@tgravescs tgravescs deleted the premergespark34 branch June 14, 2022 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gerashegalov gerashegalov gerashegalov left review comments

@abellina abellina abellina approved these changes

@jlowe jlowe jlowe left review comments

@revans2 revans2 revans2 left review comments

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@NvTimLiu NvTimLiu Awaiting requested review from NvTimLiu NvTimLiu is a code owner

@zhanga5 zhanga5 Awaiting requested review from zhanga5

Assignees

@tgravescs tgravescs

Labels
build Related to CI / CD or cleanly building
Projects
None yet
Milestone
Jun 6 - Jun 17
Development

Successfully merging this pull request may close these issues.

Disable Spark 3.4 for premerge build
6 participants
@tgravescs @gerashegalov @pxLi @jlowe @abellina @revans2