1 Release published by 1 person

• v1.3.0-preview

  published Jan 7, 2025

45 Pull requests merged by 22 people

• [GLUTEN-8319][VL] Support date_format Spark function

  #8323 merged Jan 13, 2025

• [DOC] Update README.md

  #8444 merged Jan 13, 2025

• [GLUTEN-8497][VL] A bad test case that fails columnar table cache query

  #8498 merged Jan 13, 2025

• [GLUTEN-6887][VL] Daily Update Velox Version (2025_01_12)

  #8508 merged Jan 12, 2025

• [GLUTEN-6887][VL] Daily Update Velox Version (2025_01_11)

  #8507 merged Jan 12, 2025

• [GLUTEN-8471][VL] Fix usage of uninitialized variables

  #8470 merged Jan 11, 2025

• [GLUTEN-8503][VL] Fix macro parenthesis CVE

  #8504 merged Jan 10, 2025

• [GLUTEN-8476][VL] Fix allocate and free memory

  #8477 merged Jan 10, 2025

• [GLUTEN-6887][VL] Daily Update Velox Version (2025_01_10)

  #8489 merged Jan 10, 2025

• [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250110)

  #8490 merged Jan 10, 2025

• [DOC][VL] Fix typo in microbenchmark.md

  #8495 merged Jan 10, 2025

• [GLUTEN-8483][CORE] A stable and universal way to find component files

  #8486 merged Jan 10, 2025

• [Core] Bump version to 1.4.0-SNAPSHOT

  #8452 merged Jan 10, 2025

• [GLUTEN-8481][VL] Clean up shuffle reader cpp code

  #8482 merged Jan 10, 2025

• [GLUTEN-8343][CH]Fix cast number to decimal and improve performance of it

  #8351 merged Jan 10, 2025

• [GLUTEN-8397][CH][Part-2] Fix statica_cast failed on macos

  #8485 merged Jan 9, 2025

• [GLUTEN-8414][VL] Override doCanonicalize in ColumnarPartialProjectEx…

  #8415 merged Jan 9, 2025

• [GLUTEN-8462][CH] Fixed the loading of Components and Backend

  #8464 merged Jan 9, 2025

• [CORE] Refactor columnar noop write rule

  #8422 merged Jan 9, 2025

• [GLUTEN-6887][VL] Daily Update Velox Version (2025_01_09)

  #8472 merged Jan 9, 2025

• [GLUTEN-8455][VL] Fallback Scan for Encrypted Parquet Files

  #8456 merged Jan 9, 2025

• [GLUTEN-8465] [VL] Bump Celeborn to 0.5.3

  #8467 merged Jan 9, 2025

• [VL] Refactor Velox.md

  #8478 merged Jan 9, 2025

• [GLUTEN-8453][VL] Follow-up to #8454 to add a ensureVeloxBatch API for limited use cases

  #8463 merged Jan 9, 2025

• [GLUTEN-8462][CORE] Raise a meaningful error when no component is found from classpath

  #8468 merged Jan 9, 2025

• [VL] Update document of build gluten in Docker

  #8459 merged Jan 9, 2025

• [GLUTEN-6887][VL] Daily Update Velox Version (2025_01_08)

  #8457 merged Jan 8, 2025

• [CH] Add tools to dump ActionsDAG into tree graph

  #8461 merged Jan 8, 2025

• [GLUTEN-8453] [VL] Allow Heavy Batch to be Processed by ColumnarCachedBatchSerializer

  #8454 merged Jan 8, 2025

• Revert "[CH] Disable gluten arm ci"

  #8460 merged Jan 8, 2025

• [GLUTEN-8385][VL] Support write compatible-hive bucket table for Spark3.4 and Spark3.5.

  #8386 merged Jan 8, 2025

• [VL] Fix wrong warning of "Memory overhead is set to ..." under default Spark config settings

  #8448 merged Jan 8, 2025

• [GLUTEN-8327][CORE][Part-3] Introduce the ConfigEntry to gluten config

  #8431 merged Jan 7, 2025

• [VL] Remove an out-of-date warning message

  #8447 merged Jan 7, 2025

• [GLUTEN-8375][CH][MINOR] Fix USE_EMBEDDED_COMPILER

  #8441 merged Jan 7, 2025

• [VL] Gluten-it: New option --collect-sql-metrics=execution-time to collect slowest plan nodes into benchmark report

  #8445 merged Jan 7, 2025

• [GLUTEN-7641][VL] Minor fixup for the perf analysis script

  #8430 merged Jan 7, 2025

• [GLUTEN-6887][VL] Daily Update Velox Version (2025_01_07)

  #8439 merged Jan 7, 2025

• [GLUTEN-8307][VL] Support Int64 Timestamp in parquet reader

  #8308 merged Jan 7, 2025

• [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250107)

  #8440 merged Jan 7, 2025

• [GLUTEN-8408][CH] Fix compile failures on ARM

  #8413 merged Jan 7, 2025

• [GLUTEN-8183][CORE] Prune unused column in project operator

  #8295 merged Jan 7, 2025

• [GLUTEN-8304][CORE] Add an optimization rule to collapse nested get_json_object functions

  #8305 merged Jan 7, 2025

• [GLUTEN-6887][VL] Daily Update Velox Version (2025_01_06)

  #8427 merged Jan 6, 2025

• [GLUTEN-8397][CH][Part-1]: Disable hdfs while compiling clickhouse backend on macOS

  #8400 merged Jan 6, 2025

16 Pull requests opened by 12 people

• [GLUTEN-8432][CH]Remove duplicate output attributes of aggregate's child

  #8450 opened Jan 7, 2025

• [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250108)

  #8458 opened Jan 7, 2025

• [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250109)

  #8473 opened Jan 8, 2025

• [GLUTEN_8475][VL] Fix C-style casts to C++-style

  #8474 opened Jan 8, 2025

• [GLUTEN-8479][CORE][Part-1] Remove unnecessary config

  #8480 opened Jan 9, 2025

• [GLUTEN-7544][CORE] Add Qualification Tool

  #8484 opened Jan 9, 2025

• [GLUTEN-8499][VL] Fix the logic of cast expression

  #8500 opened Jan 10, 2025

• [GLUTEN-8455][VL] Port encrypted file checks to shim layer

  #8501 opened Jan 10, 2025

• [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250111)

  #8505 opened Jan 10, 2025

• [VL] fix security volation

  #8506 opened Jan 10, 2025

• [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250112)

  #8509 opened Jan 11, 2025

• [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250113)

  #8511 opened Jan 12, 2025

• [WIP][VL] Test simple UDAF interface decimal average

  #8512 opened Jan 13, 2025

• [GLUTEN-8487][VL] adding JDK11 based Centos8 image

  #8513 opened Jan 13, 2025

• [GLUTEN-8516][CH]Optimize `and` filter by move equal conditions ahead

  #8517 opened Jan 13, 2025

• [WIP] Support range functions

  #8518 opened Jan 13, 2025

22 Issues closed by 14 people

• [VL] Scalar function name not registered: format_datetime, called with arguments: (TIMESTAMP, VARCHAR)..

  #8319 closed Jan 13, 2025

• [VL] Security Voliations

  #8471 closed Jan 11, 2025

• [VL] Security Violation Macro replacement lists should be parenthesized

  #8503 closed Jan 10, 2025

• [VL] Security Voliations Allocate and free memory in the same module

  #8476 closed Jan 10, 2025

• [VL] Security Violation NULL Pointer Dereference

  #8493 closed Jan 10, 2025

• [CH] Illegal type Nullable(Decimal(22,2)) of argument for aggregate function PartialMerge suffix must be AggregateFunction

  #8365 closed Jan 10, 2025

• [CORE] Do not rely on system property `java.class.path` to discover components

  #8483 closed Jan 10, 2025

• [CH] Diff when cast `Number` to `Decimal`

  #8343 closed Jan 10, 2025

• ColumnarPartialProjectExec has special filed but not implement doCanonicalize method

  #8414 closed Jan 9, 2025

• [CH] Gluten failed to load META-INF, causing abnormal execution

  #8462 closed Jan 9, 2025

• [VL] Velox Runtime Error on Encrypted Parquet Files

  #8455 closed Jan 9, 2025

• [VL] Why do not use sort-based shuffle for single output partition?

  #8363 closed Jan 9, 2025

• [CELEBORN] Bump Celeborn to 0.5.3

  #8465 closed Jan 9, 2025

• Columnar Cached Batch Serializer Fails for Heavy Batch

  #8453 closed Jan 8, 2025

• [VL] Support write compatible-hive bucket table for Spark3.4 and Spark3.5.

  #8385 closed Jan 8, 2025

• [VL] Support Int64 Timestamp in parquet reader

  #8307 closed Jan 7, 2025

• [CH] The result of `full jion` is wrong

  #8443 closed Jan 7, 2025

• [CH]current code base compile failed for ARM

  #8408 closed Jan 7, 2025

• [CH] Duplicated expression evaluation in project before aggregate operator due to PullOutPreProject Rule

  #8183 closed Jan 7, 2025

• [CH] Optimize get_json_object nested function calls

  #8304 closed Jan 7, 2025

• [VL] Link openssl dynamicly for FIPS

  #8232 closed Jan 7, 2025

• [VL] PrestoPage header is invalid

  #8437 closed Jan 6, 2025

16 Issues opened by 13 people

• [CH]Optimize and filter conditions by move equals ahead

  #8516 opened Jan 13, 2025

• SparkFloorFunction for the AVX2 Benchmarking

  #8515 opened Jan 13, 2025

• Performance Discrepancy: Velox Accelerates Spark on x86 but Not on ARM

  #8514 opened Jan 13, 2025

• [VL] Result mismatch of `try_cast`

  #8499 opened Jan 10, 2025

• [VL] Use of columnar table cache causes error when data emitted from the cached plan is in vanilla columnar format

  #8497 opened Jan 10, 2025

• Gluten Build Is Failing In X86 machine

  #8496 opened Jan 10, 2025

• [CH] sum with filter bad performance compared to vanilla spark

  #8492 opened Jan 10, 2025

• [VL] Fallback summary not populated if InMemoryTableScan is present in spark plan

  #8488 opened Jan 9, 2025

• [VL] Version tag for Docker image

  #8487 opened Jan 9, 2025

• [VL] Clean up unused code in shuffle

  #8481 opened Jan 9, 2025

• [CORE] Split the configuration into individual modules that belong to each other.

  #8479 opened Jan 9, 2025

• [VL] Security Voliations Do Not Cast

  #8475 opened Jan 8, 2025

• [VL] Understanding the Role of builder_helper_functions.sh in Architecture Detection and AVX Impact

  #8469 opened Jan 8, 2025

• [VL] Gluten runs slower for some TPC-H queries than Vanilla Spark

  #8466 opened Jan 8, 2025

• Usage of Gluten Functions At Query Level

  #8451 opened Jan 7, 2025

• [CH] Diff on date comparison

  #8446 opened Jan 7, 2025

31 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [CORE][CH] Support MicroBatchScanExec with KafkaScan in batch mode

  #8321 commented on Jan 13, 2025 • 5 new comments

• [CORE] Bump iceberg version of spark 3.3 to 1.5.0

  #8418 commented on Jan 13, 2025 • 4 new comments

• [GLUTEN-3620][VL] Support Range operator for Velox Backend

  #8161 commented on Jan 13, 2025 • 3 new comments

• [GLUTEN-8379][VL] Support query trace

  #8380 commented on Jan 13, 2025 • 2 new comments

• [GLUTEN-8434][CH] Function bloomFilterContains process improvement

  #8435 commented on Jan 10, 2025 • 0 new comments

• [VL] Support casting varchar type to timestamp type

  #8357 commented on Jan 13, 2025 • 0 new comments

• [GLUTEN-8340][VL] Enable from_json function

  #8320 commented on Jan 9, 2025 • 0 new comments

• [GLUTEN-8018][VL] Support adjusting stage resource profile dynamically

  #8209 commented on Jan 13, 2025 • 0 new comments

• [GLUTEN-8070][CORE] Generate gluten configuration doc and check it

  #8071 commented on Jan 13, 2025 • 0 new comments

• [WIP][GLUTEN-8055][VL] Support read Iceberg equality delete file MOR table

  #8056 commented on Jan 13, 2025 • 0 new comments

• [VL][GLUTEN-8020] Remove the libhdfs3 installation script required for static linking

  #8013 commented on Jan 13, 2025 • 0 new comments

• [GLUTEN-6227][VL] Add unit tests in unix_timestamp for Spark legacy parser policy

  #7975 commented on Jan 7, 2025 • 0 new comments

• [GLUTEN-7750][VL]: store unsafe batches data

  #7944 commented on Jan 11, 2025 • 0 new comments

• [GLUTEN-7706][CORE] Support Spark-344

  #7789 commented on Jan 10, 2025 • 0 new comments

• DNM: Debug

  #6482 commented on Jan 11, 2025 • 0 new comments

• [VL] Fallback happens to vanilla spark if there is a cast from varchar to timestamp

  #8401 commented on Jan 13, 2025 • 0 new comments

• [VL] Need Clarification on Java/Scala to Native C++ Handover and Profiling Native C++ Files

  #8419 commented on Jan 11, 2025 • 0 new comments

• [VL] Umbrella issue for sort-based shuffle

  #6896 commented on Jan 10, 2025 • 0 new comments

• TPCDS q72 uses gluten calculation much worse than spark

  #8417 commented on Jan 10, 2025 • 0 new comments

• [VL] Shuffle write size larger than vanilla spark

  #3267 commented on Jan 10, 2025 • 0 new comments

• [VL] Unsupported spark function list [please leave a comment if you plan to pick some]

  #4039 commented on Jan 10, 2025 • 0 new comments

• [CH][UMBRELLA] Support building on MacOS

  #8397 commented on Jan 9, 2025 • 0 new comments

• Gluten 2025 Roadmap

  #8226 commented on Jan 9, 2025 • 0 new comments

• [CORE][umbrella] Configuration enhancements

  #8326 commented on Jan 9, 2025 • 0 new comments

• [VL] Use multiple threads in the same executor

  #7810 commented on Jan 8, 2025 • 0 new comments

• UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

  #8207 commented on Jan 8, 2025 • 0 new comments

• [VL] Array condition filtering is not supported

  #8336 commented on Jan 8, 2025 • 0 new comments

• [VL] Fallback happens if child node of GetStructField is Scalar Function Node

  #8306 commented on Jan 8, 2025 • 0 new comments

• [CH] Support CH backend with Spark 3.5.x

  #6067 commented on Jan 7, 2025 • 0 new comments

• [CH] Fully Support writing parquet and mergetree in spark 3.5.x with delta protocol

  #7028 commented on Jan 7, 2025 • 0 new comments

• Builds of Velox fail in system with ARM-SVE but Builds of Gluten do not

  #8423 commented on Jan 6, 2025 • 0 new comments