[Bug] Spark Doris Connector build faild with spark3

[lordk911]

Search before asking

• I had searched in the issues and found no similar issues.

Version

apache-doris-0.15.0-incubating-src

What's Wrong?

I'm using apache-doris-0.15.0-incubating-src to build Spark Doris Connector build faild with spark3 by using:
sh build.sh 3

but got error:

[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisSourceProvider.scala:24: error: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] import org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
[ERROR]                                     ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisSourceProvider.scala:25: error: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] import org.apache.spark.sql.sources.v2.{DataSourceOptions, StreamWriteSupport}
[ERROR]                                     ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisSourceProvider.scala:37: error: not found: type StreamWriteSupport
[ERROR] private[sql] class DorisSourceProvider extends DataSourceRegister with RelationProvider with CreatableRelationProvider with StreamWriteSupport with Logging {
[ERROR]                                                                                                                             ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisSourceProvider.scala:126: error: not found: type StreamWriter
[ERROR]   override def createStreamWriter(queryId: String, structType: StructType, outputMode: OutputMode, dataSourceOptions: DataSourceOptions): StreamWriter = {
[ERROR]                                                                                                                                           ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisSourceProvider.scala:126: error: not found: type DataSourceOptions
[ERROR]   override def createStreamWriter(queryId: String, structType: StructType, outputMode: OutputMode, dataSourceOptions: DataSourceOptions): StreamWriter = {
[ERROR]                                                                                                                       ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:23: error: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] import org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
[ERROR]                                     ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:35: error: not found: type StreamWriter
[ERROR] class DorisStreamWriter(settings: SparkSettings) extends StreamWriter {
[ERROR]                                                          ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:24: error: object v2 is not a member of package org.apache.spark.sql.sources
[ERROR] import org.apache.spark.sql.sources.v2.writer.{DataWriter, DataWriterFactory, WriterCommitMessage}
[ERROR]                                     ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:50: error: not found: type DataWriterFactory
[ERROR] case class DorisStreamWriterFactory(settings: SparkSettings) extends DataWriterFactory[Row] {
[ERROR]                                                                      ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:38: error: not found: type WriterCommitMessage
[ERROR]   override def commit(l: Long, writerCommitMessages: Array[WriterCommitMessage]): Unit = {}
[ERROR]                                                            ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:40: error: not found: type WriterCommitMessage
[ERROR]   override def abort(l: Long, writerCommitMessages: Array[WriterCommitMessage]): Unit = {}
[ERROR]                                                           ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:51: error: not found: type DataWriter
[ERROR]   override def createDataWriter(partitionId: Int, attemptNumber: Int): DataWriter[Row] = {
[ERROR]                                                                        ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:68: error: not found: type DataWriter
[ERROR] class DorisStreamDataWriter(settings: SparkSettings) extends DataWriter[Row] {
[ERROR]                                                              ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:60: error: not found: type WriterCommitMessage
[ERROR] case object DorisWriterCommitMessage extends WriterCommitMessage
[ERROR]                                              ^
[ERROR] /root/apache-doris-0.15.0-incubating-src/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamWriter.scala:89: error: not found: type WriterCommitMessage
[ERROR]   override def commit(): WriterCommitMessage = {
[ERROR]                          ^
[ERROR] 15 errors found

What You Expected?

how to build it ?

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[lordk911]

v2 package is not in spark3

[chovy-3012]

v2 package is not in spark3

Which verison of the source code do you buiild ?

I just checked the latest release version 0.15.0-rc04 , PR #6956 was't merge into the latest release.
v2 package was removed in branch master.
https://github.com/apache/incubator-doris/blob/master/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisSourceProvider.scala#L25

You can build doris-spark-connector using branch master code. It is compatible with Doris after version 0.14 .

[lordk911]

v2 package is not in spark3

Which verison of the source code do you buiild ?

I just checked the latest release version 0.15.0-rc04 , PR #6956 was't merge into the latest release. v2 package was removed in branch master. https://github.com/apache/incubator-doris/blob/master/extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisSourceProvider.scala#L25

You can build doris-spark-connector using branch master code. It is compatible with Doris after version 0.14 .

thanks, my source code was the latest release, maybe this is the reason.