Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an example of using a remote catalog #13714

Closed
Tracked by #7013 ...
alamb opened this issue Dec 10, 2024 · 1 comment · Fixed by #13722
Closed
Tracked by #7013 ...

Provide an example of using a remote catalog #13714

alamb opened this issue Dec 10, 2024 · 1 comment · Fixed by #13722
Assignees
Labels
enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented Dec 10, 2024

Is your feature request related to a problem or challenge?

Quoting from @westonpace on #13582

Many catalogs are remote (and/or disk based) and offer only asynchronous APIs. For example, Polaris, Unity, and Hive. Integrating with this catalogs is impossible since something like ctx.sql("SELECT * FROM db.schm.tbl") first enters an async context (sql) then a synchronous context (calling the catalog provider to resolve db) and then we need to go into an asynchronous context to interact with the catalog and this async -> sync -> async path is generally forbidden.

This also came up in

I believe it is possible to interact with remote catalogs with DataFusion's non async CatalogAPIs but it is not obvious how to do so

Describe the solution you'd like

I would like a clear well documented example of a DataFusion catalog that interacts with a remote catalog

Describe alternatives you've considered

Another approach that is taken by the SessionContext::sql Is:

Does an initial pass through the parse tree to find all references (non async)
Then fetch all references (can be async)
Then does the planning (non async) with all the relevant references
I don't think this is particularly well documented

Additional context

No response

@alamb
Copy link
Contributor Author

alamb commented Dec 10, 2024

Here is a proposed example: #13722

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant