Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow passing schema argument to scan_parquet() #11853

Closed
yonil7 opened this issue Oct 19, 2023 · 0 comments · Fixed by #20999
Closed

Allow passing schema argument to scan_parquet() #11853

yonil7 opened this issue Oct 19, 2023 · 0 comments · Fixed by #20999
Labels
enhancement New feature or an improvement of an existing feature

Comments

@yonil7
Copy link

yonil7 commented Oct 19, 2023 •
edited
Loading

Description

Similar to the schema argument passed to pyarrow.dataset.dataset() - "Optionally provide the Schema for the Dataset, in which case it will not be inferred from the source."

If I understand correctly, the current implementation of scan_parquet() reads the schema from the file. This can be very expensive (I/O) when working with many parquet files.
@yonil7 yonil7 added the enhancement New feature or an improvement of an existing feature label Oct 19, 2023
@lukemanley lukemanley mentioned this issue Jan 30, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or an improvement of an existing feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: Add tests for resolved issues lukemanley/polars
1 participant
@yonil7