Support fast merge in the storage plugin interface. #60
Assignees
Labels
enhancement
New feature or request
Comments
jealous
added a commit
that referenced
this issue
Oct 9, 2019
Allow the plugin developer to implement a fast merge which will be invoked when: * Encryption is disabled. * Compression is disabled or the compression codec supports oncatenation of serialized streams. * The `spark.shuffle.unsafe.fastMergeEnabled` option is true. * The plugin supports fast merge. Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills. The sample implementation of fast merge used in the shared file system plugin is migrated from the original Spark's `UnsafeShuffleWriter.mergeSpillsWithTransfer` function. Other storage plugins could implement their own fast merge. The new fast merge API is by default disabled by the plugin.
jealous
added a commit
that referenced
this issue
Oct 9, 2019
Allow the plugin developer to implement a fast merge which will be invoked when: * Encryption is disabled. * Compression is disabled or the compression codec supports oncatenation of serialized streams. * The `spark.shuffle.unsafe.fastMergeEnabled` option is true. * The plugin supports fast merge. Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills. The sample implementation of fast merge used in the shared file system plugin is migrated from the original Spark's `UnsafeShuffleWriter.mergeSpillsWithTransfer` function. Other storage plugins could implement their own fast merge. The new fast merge API is by default disabled by the plugin.
jealous
added a commit
that referenced
this issue
Oct 11, 2019
Allow the plugin developer to implement a fast merge which will be invoked when: * Encryption is disabled. * Compression is disabled or the compression codec supports oncatenation of serialized streams. * The `spark.shuffle.unsafe.fastMergeEnabled` option is true. * The plugin supports fast merge. Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills. The sample implementation of fast merge used in the shared file system plugin is migrated from the original Spark's `UnsafeShuffleWriter.mergeSpillsWithTransfer` function. Other storage plugins could implement their own fast merge. The new fast merge API is by default disabled by the plugin.
jealous
added a commit
that referenced
this issue
Nov 11, 2019
Allow the plugin developer to implement a fast merge which will be invoked when: * Encryption is disabled. * Compression is disabled or the compression codec supports oncatenation of serialized streams. * The `spark.shuffle.unsafe.fastMergeEnabled` option is true. * The plugin supports fast merge. Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills. The sample implementation of fast merge used in the shared file system plugin is migrated from the original Spark's `UnsafeShuffleWriter.mergeSpillsWithTransfer` function. Other storage plugins could implement their own fast merge. The new fast merge API is by default disabled by the plugin.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Allow the plugin developer to implement a fast merge which will be invoked when:
spark.shuffle.unsafe.fastMergeEnabledoption is true.Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills.
The text was updated successfully, but these errors were encountered: