Skip to content

Commit 094fa3d

Browse files
LuciferYangHyukjinKwon
authored andcommittedJun 25, 2022
[SPARK-39595][BUILD] Upgrade rocksdbjni to 7.3.1
### What changes were proposed in this pull request? This PR aims to upgrade RocksDB JNI library from 7.2.2 to 7.3.1. ### Why are the changes needed? This will bring some bug fix of RocksDB JNI. - https://github.com/facebook/rocksdb/releases/tag/v7.3.1 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GA - The benchmark result : **Before 7.2.2** ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.350 0.278 0.535 0.535 dbCreation 4 75.780 3.486 294.500 294.500 naturalIndexCreateIterator 1024 0.005 0.002 1.403 0.007 naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.061 0.007 naturalIndexDescendingIteration 1024 0.006 0.004 0.244 0.008 naturalIndexIteration 1024 0.006 0.004 0.060 0.008 randomDeleteIndexed 1024 0.026 0.019 0.310 0.035 randomDeletesNoIndex 1024 0.015 0.013 0.044 0.017 randomUpdatesIndexed 1024 0.081 0.034 30.300 0.082 randomUpdatesNoIndex 1024 0.035 0.031 0.530 0.040 randomWritesIndexed 1024 0.115 0.034 51.592 0.120 randomWritesNoIndex 1024 0.038 0.033 1.612 0.042 refIndexCreateIterator 1024 0.004 0.004 0.016 0.006 refIndexDescendingCreateIterator 1024 0.003 0.003 0.027 0.005 refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008 refIndexIteration 1024 0.007 0.005 0.051 0.009 sequentialDeleteIndexed 1024 0.021 0.017 0.094 0.026 sequentialDeleteNoIndex 1024 0.015 0.013 0.045 0.017 sequentialUpdatesIndexed 1024 0.043 0.038 0.752 0.049 sequentialUpdatesNoIndex 1024 0.040 0.031 0.748 0.049 sequentialWritesIndexed 1024 0.048 0.042 1.910 0.054 sequentialWritesNoIndex 1024 0.038 0.032 2.311 0.041 ``` **After 7.3.1** ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.340 0.274 0.509 0.509 dbCreation 4 76.287 3.469 296.518 296.518 naturalIndexCreateIterator 1024 0.005 0.002 1.425 0.007 naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.061 0.007 naturalIndexDescendingIteration 1024 0.006 0.004 0.260 0.008 naturalIndexIteration 1024 0.006 0.004 0.060 0.008 randomDeleteIndexed 1024 0.026 0.019 0.283 0.033 randomDeletesNoIndex 1024 0.015 0.012 0.044 0.017 randomUpdatesIndexed 1024 0.081 0.032 30.606 0.084 randomUpdatesNoIndex 1024 0.034 0.031 0.545 0.036 randomWritesIndexed 1024 0.116 0.033 51.262 0.124 randomWritesNoIndex 1024 0.040 0.034 1.515 0.046 refIndexCreateIterator 1024 0.005 0.005 0.019 0.006 refIndexDescendingCreateIterator 1024 0.003 0.002 0.027 0.004 refIndexDescendingIteration 1024 0.006 0.005 0.041 0.008 refIndexIteration 1024 0.007 0.005 0.061 0.009 sequentialDeleteIndexed 1024 0.021 0.017 0.101 0.026 sequentialDeleteNoIndex 1024 0.016 0.013 0.041 0.018 sequentialUpdatesIndexed 1024 0.042 0.038 0.752 0.049 sequentialUpdatesNoIndex 1024 0.041 0.031 0.714 0.050 sequentialWritesIndexed 1024 0.048 0.039 1.887 0.055 sequentialWritesNoIndex 1024 0.039 0.031 2.305 0.042 ``` From the benchmark results, there is no significant performance difference between these two versions. Closes apache#36979 from LuciferYang/rocksdb-731. Authored-by: yangjie01 <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent ba87d7e commit 094fa3d

File tree

3 files changed

+3
-3
lines changed

3 files changed

+3
-3
lines changed
 

‎dev/deps/spark-deps-hadoop-2-hive-2.3

+1-1
Original file line numberDiff line numberDiff line change
@@ -235,7 +235,7 @@ pickle/1.2//pickle-1.2.jar
235235
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
236236
py4j/0.10.9.5//py4j-0.10.9.5.jar
237237
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
238-
rocksdbjni/7.2.2//rocksdbjni-7.2.2.jar
238+
rocksdbjni/7.3.1//rocksdbjni-7.3.1.jar
239239
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
240240
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
241241
scala-library/2.12.16//scala-library-2.12.16.jar

‎dev/deps/spark-deps-hadoop-3-hive-2.3

+1-1
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,7 @@ pickle/1.2//pickle-1.2.jar
224224
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
225225
py4j/0.10.9.5//py4j-0.10.9.5.jar
226226
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
227-
rocksdbjni/7.2.2//rocksdbjni-7.2.2.jar
227+
rocksdbjni/7.3.1//rocksdbjni-7.3.1.jar
228228
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
229229
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
230230
scala-library/2.12.16//scala-library-2.12.16.jar

‎pom.xml

+1-1
Original file line numberDiff line numberDiff line change
@@ -678,7 +678,7 @@
678678
<dependency>
679679
<groupId>org.rocksdb</groupId>
680680
<artifactId>rocksdbjni</artifactId>
681-
<version>7.2.2</version>
681+
<version>7.3.1</version>
682682
</dependency>
683683
<dependency>
684684
<groupId>${leveldbjni.group}</groupId>

0 commit comments

Comments
 (0)
Please sign in to comment.