Skip to content

Commit 1930116

Browse files
committed
[SPARK-38040][BUILD] Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules
### What changes were proposed in this pull request? We don't currently run binary compatibility check in below modules: ``` [info] spark-parent: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-network-common: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-tags: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-unsafe: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-network-shuffle: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-kvstore: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-tools: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-token-provider-kafka-0-10: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-streaming-kafka-0-10-assembly: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-catalyst: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-repl: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-avro: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-sql-kafka-0-10: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-hive: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-assembly: mimaPreviousArtifacts not set, not analyzing binary compatibility [info] spark-examples: mimaPreviousArtifacts not set, not analyzing binary compatibility ``` However, there are some APIs under these modules. For example, https://github.com/apache/spark/blob/master/external/avro/src/main/scala/org/apache/spark/sql/avro/functions.scala for Avro, https://github.com/apache/spark/tree/master/common/kvstore/src/main/java/org/apache/spark/util/kvstore for KVStore (to be API), and https://github.com/apache/spark/tree/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector for Catalyst ### Why are the changes needed? To detect binary compatibility. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually tested via running `dev/mima`. Closes apache#35339 from HyukjinKwon/SPARK-38040. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 6e64e92 commit 1930116

File tree

2 files changed

+8
-2
lines changed

2 files changed

+8
-2
lines changed

project/MimaExcludes.scala

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,12 @@ object MimaExcludes {
6666
ProblemFilters.exclude[Problem]("org.apache.spark.sql.catalyst.*"),
6767
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
6868
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
69+
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
70+
// DSv2 catalog and expression APIs are unstable yet. We should enable this back.
71+
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
72+
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
73+
// Avro source implementation is internal.
74+
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
6975

7076
// [SPARK-34848][CORE] Add duration to TaskMetricDistributions
7177
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),

project/SparkBuild.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -376,8 +376,8 @@ object SparkBuild extends PomBuild {
376376

377377
val mimaProjects = allProjects.filterNot { x =>
378378
Seq(
379-
spark, hive, hiveThriftServer, catalyst, repl, networkCommon, networkShuffle, networkYarn,
380-
unsafe, tags, tokenProviderKafka010, sqlKafka010, kvstore, avro
379+
spark, hive, hiveThriftServer, repl, networkCommon, networkShuffle, networkYarn,
380+
unsafe, tags, tokenProviderKafka010, sqlKafka010
381381
).contains(x)
382382
}
383383

0 commit comments

Comments
 (0)