Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to constrain arrow target type in infer_schema #1799

Open
AdamGS opened this issue Jan 3, 2025 · 0 comments
Open

Allow users to constrain arrow target type in infer_schema #1799

AdamGS opened this issue Jan 3, 2025 · 0 comments

Comments

@AdamGS
Copy link
Contributor

AdamGS commented Jan 3, 2025

Right now we infer every string/binary column in files to view types, and canonicalize all data into them. That generates extra work and might just not be what the user wants, we should allow them to control this behavior, either by hinting to the compressor or adding some metadata to the file's footer. (See #1736 for the issue/impact).

@AdamGS AdamGS changed the title Allow to hint arrow target type in infer_schema Allow users to constrain arrow target type in infer_schema Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant