[SPARK-45506][CONNECT] Add ivy URI support to SparkConnect addArtifact #43354

vsevolodstep-db · 2023-10-12T19:02:41Z

What changes were proposed in this pull request?

This PR extends existing SparkConnect SparkSession.addArtifact API to support Apache Ivy URIs in extend to the existing local .jar & .class file support. It leverages the existing support of Apache Ivy which was previously a part of spark-code (SparkSubmitUtils)

This PR contains the following changes:

Refactoring SparkSubmitUtils and moving Ivy-related parts to commons module. This results in adding two small dependencies to common (apache ivy and commons-io)
Extracting some parts of Utils & TestUtils to common
Extending SparkConnect SparkSession.addArtifact to support Ivy URIs

Refactoring changes done in 1-2 are not altering code functionality. These changes are mostly about moving code between files & packages and adjusting some dependencies

Why are the changes needed?

It's possible to add maven artifacts to Spark, but SparkConnect currently lacks this functionality

Does this PR introduce any user-facing change?

Yes: it extends the existing SparkConnect SparkSession.addArtifact API to support more URI types

How was this patch tested?

Existing tests covering Ivy resolution functionality;
New test in ArtifactSuite testing ivy support in ArtifactManager
New E2E test in ReplE2ESuite testing adding maven library and using it from UDF

Was this patch authored or co-authored using generative AI tooling?

No

common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala

hvanhovell · 2023-10-16T17:49:10Z

common/utils/pom.xml

@@ -51,6 +51,20 @@
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-text</artifactId>
    </dependency>
+    <dependency>


How much deps do we add to the client by doing this?

~400 Kb from commons-io and ~1Mb from Ivy

hvanhovell · 2023-10-16T17:55:40Z

common/utils/pom.xml

+      <artifactId>ivy</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>oro</groupId>


BTW oro is pretty much dead. Why does ivy need it?

glob matchers (ivySettings.getMatcher("glob")) require oro https://ant.apache.org/ivy/history/latest-milestone/concept.html

Also if I remove this dep then tests start failing:

[error] Uncaught exception when running org.apache.spark.util.MavenUtilsSuite: java.lang.NoClassDefFoundError: org/apache/oro/text/regex/MalformedPatternException [error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: org/apache/oro/text/regex/MalformedPatternException [error] at org.apache.ivy.plugins.matcher.GlobPatternMatcher.newMatcher(GlobPatternMatcher.java:57) [error] at org.apache.ivy.plugins.matcher.AbstractPatternMatcher.getMatcher(AbstractPatternMatcher.java:44) [error] at org.apache.ivy.plugins.matcher.MatcherHelper.matches(MatcherHelper.java:34) [error] at org.apache.ivy.plugins.matcher.MatcherHelper.matches(MatcherHelper.java:39) [error] at org.apache.ivy.plugins.matcher.MatcherHelper.matches(MatcherHelper.java:49) [error] at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.doesExclude(DefaultModuleDescriptor.java:783) [error] at org.apache.ivy.core.resolve.IvyNode.directlyExcludes(IvyNode.java:439) [error] at org.apache.ivy.core.resolve.IvyNode.doesExclude(IvyNode.java:421) [error] at org.apache.ivy.core.resolve.IvyNode.isDependencyModuleExcluded(IvyNode.java:411) [error] at org.apache.ivy.core.resolve.IvyNode.getDependencies(IvyNode.java:361) [error] at org.apache.ivy.core.resolve.VisitNode.getDependencies(VisitNode.java:305) [error] at org.apache.ivy.core.resolve.ResolveEngine.doFetchDependencies(ResolveEngine.java:797) [error] at org.apache.ivy.core.resolve.ResolveEngine.fetchDependencies(ResolveEngine.java:729) [error] at org.apache.ivy.core.resolve.ResolveEngine.getDependencies(ResolveEngine.java:607) [error] at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:250) [error] at org.apache.ivy.Ivy.resolve(Ivy.java:522) [error] at org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:456)

hvanhovell · 2023-10-17T01:38:16Z

common/utils/src/main/scala/org/apache/spark/util/SparkTestUtils.scala

+  }
+
+  /** Creates a compiled class with the source file. Class file will be placed in destDir. */
+  def createCompiledClass(


Where is this used?

IvyTestUtils.createLocalRepository -> IvyTestUtils.createJavaClass to populate a local repository with some JARs.
Also in SparkSubmitSuite and a bunch of others in spark-core

hvanhovell · 2023-10-17T01:40:51Z

...or/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/ArtifactSuite.scala

+
+  test("resolve ivy") {
+    val artifacts =
+      Artifact.newIvyArtifacts(URI.create("ivy://org.apache.hive:hive-storage-api:2.7.0"))


Is there a risk that we hammer an external repo here?

Probably, but many tests in SparkContextSuite also do this, though this is not a strong supporting argument

I'm wondering if we should add ?repos=... parameter (or something similar) to the Ivy URI to support providing additional private / local repositories? Spark-core does support this by using Spark confs, but we can't do this in connect.

This will add some useful functionality and will allow us to test this without relying on remote repositories

hvanhovell

Looks good overall. I left a few comments.

common/utils/src/test/scala/org/apache/spark/util/MavenUtilsSuite.scala

hvanhovell

LGTM

…tests to use local repo instead of remote maven

hvanhovell · 2023-10-26T01:34:18Z

@vsevolodstep-db can you retrigger tests? I am not able to do so.

# Conflicts: # common/utils/src/main/scala/org/apache/spark/util/SparkStreamUtils.scala

hvanhovell · 2023-10-30T19:43:14Z

@HyukjinKwon are the current python failures a known problem?

HyukjinKwon · 2023-10-30T23:47:13Z

Yeah, the test failure at https://github.com/vsevolodstep-db/spark/actions/runs/6691537121/job/18179083580 seems unrelated.

HyukjinKwon · 2023-10-30T23:47:21Z

Merged to master.

LuciferYang · 2023-11-16T03:11:34Z

@vsevolodstep-db

I found that after moving MavenUtilsSuite.scala to the common-utils module, it cannot pass the test. Do you know why? The current GA does not test this case (this issue will be fixed later), and local reproduction can be executed by build/sbt "common-utils/test".

then

[info] MavenUtilsSuite:
[info] - incorrect maven coordinate throws error (8 milliseconds)
[info] - create repo resolvers (24 milliseconds)
[info] - create additional resolvers (3 milliseconds)
:: loading settings :: url = jar:file:/Users/yangjie01/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/ivy/ivy/2.5.1/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
[info] - add dependencies works correctly (35 milliseconds)
[info] - excludes works correctly (2 milliseconds)
[info] - ivy path works correctly (3 seconds, 759 milliseconds)
[info] - search for artifact at local repositories *** FAILED *** (2 seconds, 833 milliseconds)
[info]   java.lang.RuntimeException: [unresolved dependency: my.great.lib#mylib;0.1: java.text.ParseException: [[Fatal Error] ivy-0.1.xml.original:22:18: XML document structures must start and end within the same entity. in file:/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/cache/my.great.lib/mylib/ivy-0.1.xml.original
[info] ]]
[info]   at org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:459)
[info]   at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$25(MavenUtilsSuite.scala:173)
[info]   at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$25$adapted(MavenUtilsSuite.scala:172)
[info]   at org.apache.spark.util.IvyTestUtils$.withRepository(IvyTestUtils.scala:373)
[info]   at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$18(MavenUtilsSuite.scala:172)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info]   at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info]   at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:333)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Suite.run(Suite.scala:1114)
[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info]   at org.apache.spark.util.MavenUtilsSuite.org$scalatest$BeforeAndAfterAll$$super$run(MavenUtilsSuite.scala:36)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.util.MavenUtilsSuite.run(MavenUtilsSuite.scala:36)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:833)
[info] - dependency not found throws RuntimeException (2 seconds, 274 milliseconds)
[info] - neglects Spark and Spark's dependencies (1 second, 934 milliseconds)
[info] - exclude dependencies end to end (953 milliseconds)
:: loading settings :: file = /Users/yangjie01/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/ivysettings.xml
[info] - load ivy settings file *** FAILED *** (167 milliseconds)
[info]   java.lang.RuntimeException: [unresolved dependency: my.great.lib#mylib;0.1: java.text.ParseException: [[Fatal Error] ivy-0.1.xml.original:22:18: XML document structures must start and end within the same entity. in file:/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/cache/my.great.lib/mylib/ivy-0.1.xml.original
[info] ]]
[info]   at org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:459)
[info]   at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$40(MavenUtilsSuite.scala:260)
[info]   at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$40$adapted(MavenUtilsSuite.scala:259)
[info]   at org.apache.spark.util.IvyTestUtils$.withRepository(IvyTestUtils.scala:373)
[info]   at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$39(MavenUtilsSuite.scala:259)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info]   at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info]   at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:333)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Suite.run(Suite.scala:1114)
[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info]   at org.apache.spark.util.MavenUtilsSuite.org$scalatest$BeforeAndAfterAll$$super$run(MavenUtilsSuite.scala:36)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.util.MavenUtilsSuite.run(MavenUtilsSuite.scala:36)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:833)
[info] - SPARK-10878: test resolution files cleaned after resolving artifact (880 milliseconds)
[info] - SPARK-34624: should ignore non-jar dependencies (145 milliseconds)
[info] Run completed in 13 seconds, 493 milliseconds.
[info] Total number of tests run: 13
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 11, failed 2, canceled 0, ignored 0, pending 0
[info] *** 2 TESTS FAILED ***
[error] Failed tests:
[error]         org.apache.spark.util.MavenUtilsSuite

also cc @HyukjinKwon @hvanhovell

LuciferYang · 2023-11-16T09:26:51Z

common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala

@@ -230,12 +226,11 @@ private[deploy] object IvyTestUtils {
      val inside = deps.map(ivyArtifactWriter).mkString("\n")
      "\n  <dependencies>\n" + inside + "\n  </dependencies>"
    }.getOrElse("")
-    content += "\n</ivy-module>"


Why do we need to delete this line? It will make the content of ivy.xml incomplete...

Give a fix: #43834

…ction and make `common-utils` module can run tests on GitHub Action ### What changes were proposed in this pull request? This PR mainly does two things: 1. It revert a line of code in `IvyTestUtils.scala` that was mistakenly deleted in SPARK-45506 | #43354 to ensure that the `ivy.xml` file generated by `IvyTestUtils#createIvyDescriptor` is complete. Before this PR, the generated `ivy.xml` file would missing the `</ivy-module>` end tag, which would cause two test cases in `MavenUtilsSuite` to fail. We can reproduce the problem by executing the `build/sbt "common-utils/test"` command: ``` [info] MavenUtilsSuite: [info] - incorrect maven coordinate throws error (8 milliseconds) [info] - create repo resolvers (24 milliseconds) [info] - create additional resolvers (3 milliseconds) :: loading settings :: url = jar:file:/Users/yangjie01/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/ivy/ivy/2.5.1/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml [info] - add dependencies works correctly (35 milliseconds) [info] - excludes works correctly (2 milliseconds) [info] - ivy path works correctly (3 seconds, 759 milliseconds) [info] - search for artifact at local repositories *** FAILED *** (2 seconds, 833 milliseconds) [info] java.lang.RuntimeException: [unresolved dependency: my.great.lib#mylib;0.1: java.text.ParseException: [[Fatal Error] ivy-0.1.xml.original:22:18: XML document structures must start and end within the same entity. in file:/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/cache/my.great.lib/mylib/ivy-0.1.xml.original [info] ]] [info] at org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:459) [info] at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$25(MavenUtilsSuite.scala:173) [info] at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$25$adapted(MavenUtilsSuite.scala:172) [info] at org.apache.spark.util.IvyTestUtils$.withRepository(IvyTestUtils.scala:373) [info] at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$18(MavenUtilsSuite.scala:172) [info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) [info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) [info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) [info] at org.scalatest.Transformer.apply(Transformer.scala:22) [info] at org.scalatest.Transformer.apply(Transformer.scala:20) [info] at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) [info] at org.scalatest.TestSuite.withFixture(TestSuite.scala:196) [info] at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195) [info] at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) [info] at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) [info] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) [info] at scala.collection.immutable.List.foreach(List.scala:333) [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) [info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) [info] at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) [info] at org.scalatest.Suite.run(Suite.scala:1114) [info] at org.scalatest.Suite.run$(Suite.scala:1096) [info] at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:535) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) [info] at org.apache.spark.util.MavenUtilsSuite.org$scalatest$BeforeAndAfterAll$$super$run(MavenUtilsSuite.scala:36) [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] at org.apache.spark.util.MavenUtilsSuite.run(MavenUtilsSuite.scala:36) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [info] at java.base/java.lang.Thread.run(Thread.java:833) [info] - dependency not found throws RuntimeException (2 seconds, 274 milliseconds) [info] - neglects Spark and Spark's dependencies (1 second, 934 milliseconds) [info] - exclude dependencies end to end (953 milliseconds) :: loading settings :: file = /Users/yangjie01/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/ivysettings.xml [info] - load ivy settings file *** FAILED *** (167 milliseconds) [info] java.lang.RuntimeException: [unresolved dependency: my.great.lib#mylib;0.1: java.text.ParseException: [[Fatal Error] ivy-0.1.xml.original:22:18: XML document structures must start and end within the same entity. in file:/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/cache/my.great.lib/mylib/ivy-0.1.xml.original [info] ]] [info] at org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:459) [info] at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$40(MavenUtilsSuite.scala:260) [info] at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$40$adapted(MavenUtilsSuite.scala:259) [info] at org.apache.spark.util.IvyTestUtils$.withRepository(IvyTestUtils.scala:373) [info] at org.apache.spark.util.MavenUtilsSuite.$anonfun$new$39(MavenUtilsSuite.scala:259) [info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) [info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) [info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) [info] at org.scalatest.Transformer.apply(Transformer.scala:22) [info] at org.scalatest.Transformer.apply(Transformer.scala:20) [info] at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) [info] at org.scalatest.TestSuite.withFixture(TestSuite.scala:196) [info] at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195) [info] at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) [info] at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) [info] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) [info] at scala.collection.immutable.List.foreach(List.scala:333) [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) [info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) [info] at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) [info] at org.scalatest.Suite.run(Suite.scala:1114) [info] at org.scalatest.Suite.run$(Suite.scala:1096) [info] at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:535) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) [info] at org.apache.spark.util.MavenUtilsSuite.org$scalatest$BeforeAndAfterAll$$super$run(MavenUtilsSuite.scala:36) [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] at org.apache.spark.util.MavenUtilsSuite.run(MavenUtilsSuite.scala:36) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [info] at java.base/java.lang.Thread.run(Thread.java:833) [info] - SPARK-10878: test resolution files cleaned after resolving artifact (880 milliseconds) [info] - SPARK-34624: should ignore non-jar dependencies (145 milliseconds) [info] Run completed in 13 seconds, 493 milliseconds. [info] Total number of tests run: 13 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 11, failed 2, canceled 0, ignored 0, pending 0 [info] *** 2 TESTS FAILED *** [error] Failed tests: [error] org.apache.spark.util.MavenUtilsSuite ``` 2. Added `sbt_test_goals` to the `utils` module in `modules.py` to ensure that the unit tests in the `common-utils` module will be verified by GitHub Action. ### Why are the changes needed? Fix the failed test cases in `MavenUtilsSuite` and let GitHub Action verify the unit tests in the `common-utils` module. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Monitor GA, the unit tests in the common-utils module should be run, and `MavenUtilsSuite` should test successfully. ### Was this patch authored or co-authored using generative AI tooling? No Closes #43834 from LuciferYang/utils-ga-test. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

cloud-fan · 2023-12-15T01:10:10Z

common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala


-private[deploy] object IvyTestUtils {
+private[spark] object IvyTestUtils {


this file should be in src/test instead of src/main

nchammas · 2023-12-18T06:54:50Z

core/src/main/scala/org/apache/spark/internal/config/package.scala

-    ConfigBuilder("spark.jars.ivySettings")
+    ConfigBuilder(MavenUtils.JAR_IVY_SETTING_PATH_KEY)


Isn't it more appropriate to have MavenUtils refer to this config entry and get the key name?

It seems weird and breaks with the remaining configs for this config to get its key name from somewhere else.

…test` directory ### What changes were proposed in this pull request? This pr move `IvyTestUtils` back to `src/test` directory because it has been in the `src/test` directory before the refactoring work of #43354. Meanwhile, in order to make the `core` and `connect-client-jvm` module use `IvyTestUtils` in the tests, this pr has added the corresponding Maven dependencies in the respective `pom.xml` files. ### Why are the changes needed? Move `IvyTestUtils` back to `src/test` directory ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #44440 from LuciferYang/mv-IvyTestUtils-to-test-dir. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>

vsevolodstep-db added 2 commits October 11, 2023 17:42

Support Ivy in SparkConnect addArtifact

291dc8c

Fix tests

ac5e48a

github-actions bot added SQL BUILD CORE CONNECT labels Oct 12, 2023

vsevolodstep-db added 2 commits October 12, 2023 12:59

scalafmt

d39199e

scalafmt

4cc803a

HyukjinKwon reviewed Oct 16, 2023

View reviewed changes

common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala Show resolved Hide resolved

hvanhovell reviewed Oct 16, 2023

View reviewed changes

hvanhovell reviewed Oct 17, 2023

View reviewed changes

Undo scalafmt changes for lines not changed

18fa50c

HyukjinKwon reviewed Oct 17, 2023

View reviewed changes

common/utils/src/test/scala/org/apache/spark/util/MavenUtilsSuite.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Oct 17, 2023

View reviewed changes

common/utils/src/test/scala/org/apache/spark/util/MavenUtilsSuite.scala Outdated Show resolved Hide resolved

vsevolodstep-db added 3 commits October 17, 2023 13:59

Undo scalafmt changes for lines not changed

587ae1f

fix

e6bc941

Scalafmt

17ed7a1

hvanhovell approved these changes Oct 17, 2023

View reviewed changes

Add support for custom repositories in ivy url; rewrite SparkConnect …

85837d4

…tests to use local repo instead of remote maven

vsevolodstep-db requested a review from HyukjinKwon October 20, 2023 12:56

vsevolodstep-db added 2 commits October 26, 2023 13:10

Merge branch 'master' into SPARK-45506/add-artifact-ivy

4bcd755

# Conflicts: # common/utils/src/main/scala/org/apache/spark/util/SparkStreamUtils.scala

Merge branch 'master' into SPARK-45506/add-artifact-ivy

845738f

hvanhovell closed this Oct 30, 2023

hvanhovell reopened this Oct 30, 2023

HyukjinKwon closed this in 2ad79e1 Oct 30, 2023

LuciferYang reviewed Nov 16, 2023

View reviewed changes

LuciferYang mentioned this pull request Nov 16, 2023

[SPARK-45950][INFRA][CORE] Fix IvyTestUtils#createIvyDescriptor function and make common-utils module can run tests on GitHub Action #43834

Closed

cloud-fan reviewed Dec 15, 2023

View reviewed changes

nchammas reviewed Dec 18, 2023

View reviewed changes

LuciferYang mentioned this pull request Dec 21, 2023

[SPARK-46476][BUILD][CORE][CONNECT] Move IvyTestUtils back to src/test directory #44440

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-45506][CONNECT] Add ivy URI support to SparkConnect addArtifact #43354

[SPARK-45506][CONNECT] Add ivy URI support to SparkConnect addArtifact #43354

vsevolodstep-db commented Oct 12, 2023

hvanhovell Oct 16, 2023

vsevolodstep-db Oct 17, 2023

hvanhovell Oct 16, 2023

vsevolodstep-db Oct 17, 2023

hvanhovell Oct 17, 2023

vsevolodstep-db Oct 17, 2023

hvanhovell Oct 17, 2023

vsevolodstep-db Oct 17, 2023

hvanhovell left a comment

hvanhovell left a comment

hvanhovell commented Oct 26, 2023

hvanhovell commented Oct 30, 2023

HyukjinKwon commented Oct 30, 2023

HyukjinKwon commented Oct 30, 2023

LuciferYang commented Nov 16, 2023 •

edited

Loading

LuciferYang Nov 16, 2023

LuciferYang Nov 16, 2023

cloud-fan Dec 15, 2023

LuciferYang Dec 21, 2023

nchammas Dec 18, 2023


		private[deploy] object IvyTestUtils {
		private[spark] object IvyTestUtils {

		ConfigBuilder("spark.jars.ivySettings")
		ConfigBuilder(MavenUtils.JAR_IVY_SETTING_PATH_KEY)

[SPARK-45506][CONNECT] Add ivy URI support to SparkConnect addArtifact #43354

[SPARK-45506][CONNECT] Add ivy URI support to SparkConnect addArtifact #43354

Conversation

vsevolodstep-db commented Oct 12, 2023

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvanhovell left a comment

Choose a reason for hiding this comment

hvanhovell left a comment

Choose a reason for hiding this comment

hvanhovell commented Oct 26, 2023

hvanhovell commented Oct 30, 2023

HyukjinKwon commented Oct 30, 2023

HyukjinKwon commented Oct 30, 2023

LuciferYang commented Nov 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LuciferYang commented Nov 16, 2023 •

edited

Loading