[Bug] spark3.4.3 java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils$.toArrowSchema

[liuchunhua]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master f720167

What's Wrong?

类名冲突

java.lang.NoSuchMethodError: org.apache.spark.sql.util.ArrowUtils$.toArro
wSchema(Lorg/apache/spark/sql/types/StructType;Ljava/lang/String;)Lorg/apache/arrow/vector/types/pojo/Schema;
        at org.apache.spark.sql.execution.arrow.ArrowConverters$ArrowBatchIterator.<init>(ArrowConverters.scala:82)
        at org.apache.spark.sql.execution.arrow.ArrowConverters$.toBatchIterator(ArrowConverters.scala:203)
        at org.apache.spark.sql.Dataset.$anonfun$toArrowBatchRdd$1(Dataset.scala:4263)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:891)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:891)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
        at org.apache.spark.scheduler.Task.run(Task.scala:139)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

What You Expected?

org.apache.spark.sql.util.ArrowUtils 最好更改Doris里一下包名

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct