Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can we use the arrow PyCapsule/C api when ingesting from pyarrow.Table? #2729

Closed
timkpaine opened this issue Aug 30, 2024 · 1 comment
Closed

Can we use the arrow PyCapsule/C api when ingesting from pyarrow.Table? #2729

timkpaine opened this issue Aug 30, 2024 · 1 comment
Labels
enhancement Feature requests or improvements

Comments

@timkpaine
Copy link
Member

we have a relatively long-winded ingestion from a pyarrow.Table:

perspective/rust/perspective-python/src/client/python.rs

Line 166 in 768ca40

fn to_arrow_bytes<'py>(

Even though it would start as a different format, can we save time by going through the pycapsule/c interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html#consume-a-pycapsule

xref: #1157
@timkpaine timkpaine added the enhancement Feature requests or improvements label Aug 30, 2024
@texodus
Copy link
Member

texodus commented Feb 2, 2025

Converting to a discussion - definitely interested in higher performance Arrow ingestion!

@finos finos locked and limited conversation to collaborators Feb 2, 2025
@texodus texodus converted this issue into discussion #2911 Feb 2, 2025

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Assignees
No one assigned
Labels
enhancement Feature requests or improvements
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@texodus @timkpaine