Skip to content

Commit a3f84ac

Browse files
committedMay 8, 2022
[SPARK-39124][BUILD] Upgrade rocksdbjni to 7.1.2
### What changes were proposed in this pull request? This PR aims to upgrade RocksDB JNI library from 7.0.3 to 7.1.2. ### Why are the changes needed? This will bring improvements. Note that 7.2.0 JNI is not released yet. - https://github.com/facebook/rocksdb/releases/tag/v7.1.1 - https://github.com/facebook/rocksdb/releases/tag/v7.1.2 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. The benchmark result is here. **Before (7.0.3)** ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.500 0.356 0.856 0.856 dbCreation 4 433.785 195.266 1159.003 1159.003 naturalIndexCreateIterator 1024 0.019 0.002 6.221 0.055 naturalIndexDescendingCreateIterator 1024 0.009 0.006 1.259 0.008 naturalIndexDescendingIteration 1024 0.021 0.005 1.786 0.097 naturalIndexIteration 1024 0.017 0.006 0.765 0.033 randomDeleteIndexed 1024 0.050 0.025 3.664 0.079 randomDeletesNoIndex 1024 0.019 0.016 0.118 0.021 randomUpdatesIndexed 1024 0.118 0.035 26.061 0.357 randomUpdatesNoIndex 1024 0.031 0.026 0.799 0.033 randomWritesIndexed 1024 0.169 0.041 59.242 0.346 randomWritesNoIndex 1024 0.055 0.032 9.539 0.052 refIndexCreateIterator 1024 0.009 0.007 0.200 0.010 refIndexDescendingCreateIterator 1024 0.008 0.003 0.669 0.016 refIndexDescendingIteration 1024 0.019 0.006 0.432 0.061 refIndexIteration 1024 0.018 0.007 1.134 0.044 sequentialDeleteIndexed 1024 0.039 0.023 1.110 0.038 sequentialDeleteNoIndex 1024 0.019 0.016 0.382 0.021 sequentialUpdatesIndexed 1024 0.063 0.032 2.218 0.102 sequentialUpdatesNoIndex 1024 0.045 0.029 1.737 0.064 sequentialWritesIndexed 1024 0.075 0.037 16.837 0.115 sequentialWritesNoIndex 1024 0.065 0.031 16.540 0.055 ``` **After (7.1.2)** ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.891 0.380 1.286 1.286 dbCreation 4 411.247 195.950 1058.678 1058.678 naturalIndexCreateIterator 1024 0.017 0.002 6.792 0.011 naturalIndexDescendingCreateIterator 1024 0.011 0.008 0.882 0.010 naturalIndexDescendingIteration 1024 0.014 0.005 1.541 0.009 naturalIndexIteration 1024 0.020 0.006 1.347 0.036 randomDeleteIndexed 1024 0.075 0.026 4.560 0.256 randomDeletesNoIndex 1024 0.020 0.017 0.054 0.022 randomUpdatesIndexed 1024 0.117 0.036 24.286 0.363 randomUpdatesNoIndex 1024 0.034 0.026 2.126 0.033 randomWritesIndexed 1024 0.168 0.041 59.470 0.371 randomWritesNoIndex 1024 0.057 0.030 11.681 0.049 refIndexCreateIterator 1024 0.006 0.004 0.201 0.006 refIndexDescendingCreateIterator 1024 0.005 0.003 0.407 0.004 refIndexDescendingIteration 1024 0.015 0.006 1.498 0.017 refIndexIteration 1024 0.020 0.007 0.993 0.021 sequentialDeleteIndexed 1024 0.035 0.023 1.605 0.038 sequentialDeleteNoIndex 1024 0.025 0.017 1.706 0.023 sequentialUpdatesIndexed 1024 0.058 0.031 2.765 0.051 sequentialUpdatesNoIndex 1024 0.052 0.031 3.404 0.058 sequentialWritesIndexed 1024 0.075 0.035 17.668 0.093 sequentialWritesNoIndex 1024 0.065 0.031 16.220 0.054 ``` Closes apache#36480 from dongjoon-hyun/SPARK-39124. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 1ebf3e5 commit a3f84ac

File tree

3 files changed

+3
-3
lines changed

3 files changed

+3
-3
lines changed
 

‎dev/deps/spark-deps-hadoop-2-hive-2.3

+1-1
Original file line numberDiff line numberDiff line change
@@ -235,7 +235,7 @@ pickle/1.2//pickle-1.2.jar
235235
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
236236
py4j/0.10.9.5//py4j-0.10.9.5.jar
237237
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
238-
rocksdbjni/7.0.3//rocksdbjni-7.0.3.jar
238+
rocksdbjni/7.1.2//rocksdbjni-7.1.2.jar
239239
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
240240
scala-compiler/2.12.15//scala-compiler-2.12.15.jar
241241
scala-library/2.12.15//scala-library-2.12.15.jar

‎dev/deps/spark-deps-hadoop-3-hive-2.3

+1-1
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,7 @@ pickle/1.2//pickle-1.2.jar
224224
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
225225
py4j/0.10.9.5//py4j-0.10.9.5.jar
226226
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
227-
rocksdbjni/7.0.3//rocksdbjni-7.0.3.jar
227+
rocksdbjni/7.1.2//rocksdbjni-7.1.2.jar
228228
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
229229
scala-compiler/2.12.15//scala-compiler-2.12.15.jar
230230
scala-library/2.12.15//scala-library-2.12.15.jar

‎pom.xml

+1-1
Original file line numberDiff line numberDiff line change
@@ -675,7 +675,7 @@
675675
<dependency>
676676
<groupId>org.rocksdb</groupId>
677677
<artifactId>rocksdbjni</artifactId>
678-
<version>7.0.3</version>
678+
<version>7.1.2</version>
679679
</dependency>
680680
<dependency>
681681
<groupId>${leveldbjni.group}</groupId>

0 commit comments

Comments
 (0)
Please sign in to comment.