48 Pull requests merged by 25 people

• chore(deps): update sqllogictest requirement from 0.23.0 to 0.24.0

  #13902 merged Dec 26, 2024

• Preserve constant values across union operations

  #13805 merged Dec 25, 2024

• Introduce UserDefinedLogicalNodeUnparser for User-defined Logical Plan unparsing

  #13880 merged Dec 25, 2024

• Changed the url for downloading IMDB dataset from benchmark - Fixed Issue #13896

  #13903 merged Dec 25, 2024

• fix case_column_or_null with nullable when conditions

  #13886 merged Dec 25, 2024

• Support unparsing implicit lateral UNNEST plan to SQL text

  #13824 merged Dec 25, 2024

• Prepare for 44.0.0 release: version and changelog

  #13882 merged Dec 25, 2024

• Minor: Avoid emitting empty batches in partial sort

  #13895 merged Dec 25, 2024

• Fix visibility of swap_hash_join to be pub

  #13899 merged Dec 24, 2024

• Fix recursive-protection feature flag

  #13887 merged Dec 24, 2024

• Minor: change visibility of hash join utils

  #13893 merged Dec 24, 2024

• Minor: change the sort merge join emission as incremental

  #13894 merged Dec 24, 2024

• Support (order by / sort) for DataFrameWriteOptions

  #13874 merged Dec 24, 2024

• Support 1 or 3 arg in generate_series() UDTF

  #13856 merged Dec 24, 2024

• minor: fix typos in comments / structure names

  #13879 merged Dec 23, 2024

• chore: Consolidate Example: simplify_udwf_expression.rs into advanced_udwf.rs

  #13883 merged Dec 23, 2024

• Improve error messages for incorrect zero argument signatures

  #13881 merged Dec 23, 2024

• Restore DocBuilder::new() to avoid breaking API change

  #13870 merged Dec 22, 2024

• Consolidate Example: dataframe_output.rs into dataframe.rs

  #13877 merged Dec 22, 2024

• Minor: remove unused async-compression futures-io feature

  #13875 merged Dec 22, 2024

• [minor] make recursive package dependency optional

  #13778 merged Dec 22, 2024

• Support unicode character for initcap function

  #13752 merged Dec 22, 2024

• Add documentation for SHOW FUNCTIONS

  #13868 merged Dec 21, 2024

• Minor: improve error message when ARRAY literals can not be planned

  #13859 merged Dec 21, 2024

• Minor: fix: Include FetchRel when producing LogicalPlan from Sort

  #13862 merged Dec 21, 2024

• feat(substrait): modular substrait consumer

  #13803 merged Dec 21, 2024

• Fix build use of undeclared type ShowStatementFilter

  #13869 merged Dec 21, 2024

• Update bzip2 requirement from 0.4.3 to 0.5.0

  #13740 merged Dec 21, 2024

• Implement SHOW FUNCTIONS

  #13799 merged Dec 21, 2024

• Minor: Unify downcast_arg method

  #13865 merged Dec 21, 2024

• Improve SortPreservingMerge::enable_round_robin_repartition docs

  #13826 merged Dec 20, 2024

• feat(function): add least function

  #13786 merged Dec 20, 2024

• Minor: Use resize instead of extend for static values in SMJ logic

  #13861 merged Dec 20, 2024

• Upgrade to sqlparser 0.53.0

  #13767 merged Dec 20, 2024

• feat: support normalized expr in CSE

  #13315 merged Dec 20, 2024

• ImproveSignature and comparison_coercion documentation

  #13840 merged Dec 20, 2024

• Add configurable normalization for configuration options and preserve case for S3 paths

  #13576 merged Dec 20, 2024

• fix: enable DF's nested_expressions feature by in datafusion-substrait tests to make them pass

  #13857 merged Dec 20, 2024

• replace CASE expressions in predicate pruning with boolean algebra

  #13795 merged Dec 20, 2024

• Preserve ordering equivalencies on with_reorder

  #13770 merged Dec 20, 2024

• Replace execution_mode with emission_type and boundedness

  #13823 merged Dec 20, 2024

• Support n-ary monotonic functions in ordering equivalence

  #13841 merged Dec 20, 2024

• MINOR: typo -- remove extra "`" interfering with doc formatting

  #13847 merged Dec 19, 2024

• typo: remove extraneous "`" in doc comment, fix header

  #13848 merged Dec 19, 2024

• Update substrait requirement from 0.49 to 0.50

  #13808 merged Dec 19, 2024

• Add example of interacting with a remote catalog

  #13722 merged Dec 19, 2024

• [bugfix] ScalarFunctionExpr does not preserve the nullable flag on roundtrip

  #13830 merged Dec 19, 2024

• Rename TypeSignature::NullAry --> TypeSignature::Nullary and improve comments

  #13817 merged Dec 19, 2024

8 Pull requests opened by 8 people

• chore: Migration Guide

  #13849 opened Dec 19, 2024

• WIP: ParquetSink should be aware of arrow schema encoding for the file metadata.

  #13866 opened Dec 21, 2024

• Require all zero argument UDFs use `Signature::Nullary`, improve error messages

  #13871 opened Dec 21, 2024

• ci improvements

  #13876 opened Dec 21, 2024

• doc-gen: migrate builtin scalar functions documentation to attribute based

  #13878 opened Dec 22, 2024

• Add substrait tpch round trip tests from sql query

  #13888 opened Dec 23, 2024

• chore(deps): update parquet requirement from 53.3.0 to 54.0.0

  #13892 opened Dec 24, 2024

• Implement maintains_input_order for AggregateExec

  #13897 opened Dec 24, 2024

31 Issues closed by 6 people

• Preserve constant values in union operations

  #13804 closed Dec 25, 2024

• Support unparsing `LogicalPlan::Extension` to SQL tesxt

  #13753 closed Dec 25, 2024

• Downloading IMDB dataset for benchmarks gives 404 Not Found

  #13896 closed Dec 25, 2024

• Incorrect CASE WHEN + ELSE NULL behavior

  #13885 closed Dec 25, 2024

• Support unparsing implicit lateral `UNNEST` plan to SQL text

  #13793 closed Dec 25, 2024

• Test DataFusion 44.0.0 with Comet

  #13835 closed Dec 24, 2024

• `swap_hash_join` is no longer public so comet doesn't compile

  #13898 closed Dec 24, 2024

• Making the `recursive` dependency an optional feature

  #13766 closed Dec 24, 2024

• inner join involving hive-partitioned parquet dataset and filters on LHS and RHS causes panic

  #9797 closed Dec 24, 2024

• Support (order by / sort) for DataFrameWriteOptions

  #13873 closed Dec 24, 2024

• Support 1 or 3 arg in `generate_series()` UDTF

  #13615 closed Dec 24, 2024

• Panic when querying a hive-partitioned parquet dataset created with wrong column name

  #10020 closed Dec 23, 2024

• Consolidate Example: simplify_udwf_expression.rs into advanced_udwf.rs #13842

  #13843 closed Dec 23, 2024

• Make migration to `Signature::nullary` in 44.0.0 easier / less confusing

  #13763 closed Dec 23, 2024

• Make `DocBuilder` migration in `44.0.0` easier

  #13764 closed Dec 22, 2024

• Consolidate Example: dataframe_output.rs into dataframe.rs

  #13844 closed Dec 22, 2024

• Test DataFusion 44.0.0 with delta.rs

  #13834 closed Dec 22, 2024

• Regression in 43.0.0: coalesce no longer works between Utf8 and Utf8View columns

  #13568 closed Dec 22, 2024

• Support unicode character for `initcap` function

  #13711 closed Dec 22, 2024

• Rename / simplify `BuiltInWindowExpr` / `BuiltInWindowFunctionExpr`

  #13473 closed Dec 21, 2024

• Substrait roundtrip fails for Sort with a fetch

  #13860 closed Dec 21, 2024

• List available functions (`SHOW FUNCTIONS`)

  #12144 closed Dec 21, 2024

• Add `greatest(T,...)` and `least(T,...)` SQL functions

  #6531 closed Dec 20, 2024

• Support per-option value normalization

  #11650 closed Dec 20, 2024

• Proposal: Restructure DataFusion site

  #1821 closed Dec 20, 2024

• substrait_integration integration tests are failing

  #13854 closed Dec 20, 2024

• Preserve ordering equivalencies on `with_reorder`

  #13769 closed Dec 20, 2024

• Support n-ary monotonic functions in ordering equivalence

  #13839 closed Dec 20, 2024

• support make_interval function

  #6951 closed Dec 19, 2024

• Provide an example of using a remote catalog

  #13714 closed Dec 19, 2024

• ScalarFunctionExpr does not preserve the nullable flag on serialization roundtrip

  #13829 closed Dec 19, 2024

15 Issues opened by 9 people

• SELECT * FROM subquery ignores ordering

  #13904 opened Dec 25, 2024

• [substrait] customizable producer

  #13901 opened Dec 24, 2024

• Find a way to communicate the ordering of a file back with the existing listing table implementation

  #13891 opened Dec 24, 2024

• Functionality of `array_repeat` udf

  #13872 opened Dec 21, 2024

• Replace `BufferBuilder<u8>` with `Vec<u8>`

  #13867 opened Dec 21, 2024

• [substrait] refactor consumer.rs

  #13864 opened Dec 20, 2024

• [substrait] more abstract SubstraitConsumer API

  #13863 opened Dec 20, 2024

• Test DataFusion 44.0.0 with Sail

  #13855 opened Dec 20, 2024

• datafusion-substrait API docs on docs.rs are broken

  #13853 opened Dec 19, 2024

• FFI Execution Plans that spawn threads panic

  #13851 opened Dec 19, 2024

• Support multiply and divide on intervals

  #13850 opened Dec 19, 2024

• Contemplate stop CI testing on intel mac

  #13846 opened Dec 19, 2024

• Improve efficiency of CI checks (so we can add MORE!)

  #13845 opened Dec 19, 2024

• Consolidate Example: simplify_udaf_expression.rs into advanced_udaf.rs

  #13842 opened Dec 19, 2024

• [DISCUSS] Single Source `ExecutionPlan` Across All `TableProviders`

  #13838 opened Dec 19, 2024

50 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Introduce LogicalPlan invariants, begin automatically checking them

  #13651 commented on Dec 25, 2024 • 24 new comments

• Add snapshot testing to CLI & set up AWS mock

  #13672 commented on Dec 24, 2024 • 6 new comments

• feat: add `AsyncCatalogProvider` helpers for asynchronous catalogs

  #13800 commented on Dec 19, 2024 • 4 new comments

• Deprecate ScalarUDFImpl::return_type

  #13717 commented on Dec 20, 2024 • 4 new comments

• WIP Upgrade to arrow-rs/parquet `54.0.0`

  #13663 commented on Dec 21, 2024 • 2 new comments

• [substrait] Add support for ExtensionTable

  #13772 commented on Dec 24, 2024 • 1 new comment

• Cannot create a `List` of `FixedSizedList` in SQL

  #13819 commented on Dec 24, 2024 • 0 new comments

• Introduce a way to represent constrained statistics / bounds on values in Statistics

  #8078 commented on Dec 24, 2024 • 0 new comments

• Datafusion binary size has been getting bigger

  #13816 commented on Dec 24, 2024 • 0 new comments

• Refactor `TableFunctionImpl` to a separate module

  #13613 commented on Dec 25, 2024 • 0 new comments

• Release DataFusion `44.0.0`

  #13334 commented on Dec 25, 2024 • 0 new comments

• OOM in `GroupedHashAggregateStream::group_aggregate_batch()`

  #13831 commented on Dec 25, 2024 • 0 new comments

• Support Null aware anti join by HashJoin

  #10584 commented on Dec 21, 2024 • 0 new comments

• Adding node_id to ExecutionPlanProperties

  #12186 commented on Dec 19, 2024 • 0 new comments

• feat: scalar regex match physical expr

  #12270 commented on Dec 21, 2024 • 0 new comments

• feat: support inner iejoin

  #12754 commented on Dec 23, 2024 • 0 new comments

• Implement predicate pruning for `like` expressions (prefix matching)

  #12978 commented on Dec 23, 2024 • 0 new comments

• Fix DISTINCT ON expressions match ORDER BY expressions check - normal…

  #13039 commented on Dec 22, 2024 • 0 new comments

• feat: Add regexp_split_to_array function

  #13110 commented on Dec 26, 2024 • 0 new comments

• feat: Add ConfigOptions to ScalarFunctionArgs

  #13527 commented on Dec 23, 2024 • 0 new comments

• Update hashbrown requirement from 0.14.5 to 0.15.2

  #13557 commented on Dec 25, 2024 • 0 new comments

• Support specific `GroupsAccumulator` for `median`

  #13681 commented on Dec 21, 2024 • 0 new comments

• Document SQL dialect guidance

  #13706 commented on Dec 19, 2024 • 0 new comments

• Add sum statistics and PhysicalExpr::column_statistics

  #13736 commented on Dec 21, 2024 • 0 new comments

• Feature scalar regexp match benchmark

  #13789 commented on Dec 23, 2024 • 0 new comments

• Epic: Better / Improved Documentation, Tutorials and Examples

  #7013 commented on Dec 19, 2024 • 0 new comments

• [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB)

  #13525 commented on Dec 19, 2024 • 0 new comments

• Running tests uses 50.1GB of disk space on Ubuntu

  #11105 commented on Dec 19, 2024 • 0 new comments

• Compute ScalarFunction properties including `return_type` and `nullable` on creation

  #13825 commented on Dec 19, 2024 • 0 new comments

• Building project takes a *long* time (esp compilation time for `datafusion` core crate)

  #13814 commented on Dec 19, 2024 • 0 new comments

• Blog / Example of how to compile DataFusion to WASM

  #13715 commented on Dec 19, 2024 • 0 new comments

• Browser-accessible official DataFusion playground

  #13818 commented on Dec 19, 2024 • 0 new comments

• Add version checking to FFI crate

  #13827 commented on Dec 19, 2024 • 0 new comments

• [EPIC] Additional Date/Time related open issues

  #8282 commented on Dec 19, 2024 • 0 new comments

• Add H2O.ai Database-like Ops benchmark to `dfbench`

  #7209 commented on Dec 19, 2024 • 0 new comments

• Arrow schema is missing from the parquet metadata, for files written by ParquetSink.

  #11770 commented on Dec 20, 2024 • 0 new comments

• [Discuss] Release cadence / patch releases / Long Term Supported (lts) minor releases

  #5269 commented on Dec 20, 2024 • 0 new comments

• Dec 13, 2024: This week(s) in DataFusion

  #13760 commented on Dec 20, 2024 • 0 new comments

• [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade)

  #13648 commented on Dec 20, 2024 • 0 new comments

• [EPIC] A collection of items to improve developer / CI speed

  #13813 commented on Dec 20, 2024 • 0 new comments

• CI: Windows flow takes 1.5h

  #13726 commented on Dec 21, 2024 • 0 new comments

• 2gb parquet file takes 100s to process, even on second attempt (on main)

  #13785 commented on Dec 21, 2024 • 0 new comments

• [DISCUSSION] Make it easier and faster to query remote files (S3, iceberg, etc)

  #13456 commented on Dec 21, 2024 • 0 new comments

• parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect

  #13821 commented on Dec 22, 2024 • 0 new comments

• Regression: `Invalid comparison operation: Utf8 == Utf8View` error during LEFT ANTI JOIN

  #13510 commented on Dec 22, 2024 • 0 new comments

• [Epic] A Collection of Additional UTF8View support tickets

  #13504 commented on Dec 22, 2024 • 0 new comments

• Why does `PruningPredicate` reference a `row_count` for each column?

  #13836 commented on Dec 22, 2024 • 0 new comments

• Complete / integrate sqlite sqllogictest test scripts integrattion

  #13812 commented on Dec 23, 2024 • 0 new comments

• [EPIC] Improve examples to make them easier to navigate

  #11172 commented on Dec 23, 2024 • 0 new comments

• sql result discrepency with sqlite, postgres and duckdb bug #3

  #13784 commented on Dec 24, 2024 • 0 new comments