feat: Integrate Trino as a data source using sfu-db/connector-x#405
Open
vcknorket wants to merge 10 commits intoroapi:mainfrom
Open
feat: Integrate Trino as a data source using sfu-db/connector-x#405vcknorket wants to merge 10 commits intoroapi:mainfrom
vcknorket wants to merge 10 commits intoroapi:mainfrom
Conversation
This commit introduces support for Trino as a data source by updating
the connector-x dependency and integrating its Trino capabilities.
Key changes include:
1. **Dependency Update:**
* Switched `connectorx` in `columnq/Cargo.toml` from the forked
`roapi/connector-x` to the official `sfu-db/connector-x`
(version `0.4.4-alpha.1`).
* Updated `connectorx` features to `["fptr", "src_trino", "dst_arrow"]`.
2. **Core Logic for Trino:**
* Added a `Trino` variant to the `DatabaseLoader` enum in
`columnq/src/table/database.rs`.
* Updated conditional compilation flags in `columnq/src/table/database.rs`
to include `database-trino`.
* In `columnq/src/table/mod.rs`:
* Added `Trino` to the `Extension` enum and updated its `From`
and `TryFrom` implementations.
* Added a `trino { table: Option<String> }` variant to the
`TableLoadOption` enum.
* Updated `TableLoadOption::extension()` method.
* Updated `TableSource::parse_option()` to recognize "trino" URIs.
* Updated the main `load()` function to handle Trino.
* Modified table name extraction in `columnq/src/table/database.rs`
to support `TableLoadOption::trino`.
3. **Cargo Feature Flag:**
* Added `database-trino = ["connectorx/src_trino"]` to the
`[features]` section in `columnq/Cargo.toml`.
* Included `database-trino` in the `database` feature group.
4. **Testing:**
* Added a new test file `columnq/tests/table_trino_test.rs`.
* This includes a basic compile-time and structural test for the
Trino integration, conditional on the `database-trino` feature.
* A comment in the test notes that full integration testing requires
a live Trino instance and appropriate environment setup.
This integration allows you to specify Trino as a data source via
URI or configuration, leveraging the capabilities of the underlying
`sfu-db/connector-x` library.
houqp
reviewed
Jun 15, 2025
| // but avoid pulling in too much if it complicates things without a live server. | ||
|
|
||
| #[tokio::test] | ||
| async fn test_trino_loader_construction() { |
Member
There was a problem hiding this comment.
this test doesn't seem to do anything? I am okay with merging without a test if it's too much work to setup trio in the CI.
Member
|
thanks for the contribution, but looks like the build is broken. |
rusqlite version
rusqlite version update
Update Cargo.toml
Update get_arrow call to match connectorx API"
Enable Arrow 54 use 55
Revert the changes made
Update arrow version to be specific
Author
Member
|
@vcknorket you will need to send an upstream PR to connector to bump their version of arrow to 55 first. |
Author
|
Seems someone already asked the upstream but they said...it cannot be done but suggest that it should still work
…________________________________
From: QP Hou ***@***.***>
Sent: June 27, 2025 2:47 AM
To: roapi/roapi ***@***.***>
Cc: Venkata Chandra (VC) ***@***.***>; Mention ***@***.***>
Subject: Re: [roapi/roapi] feat: Integrate Trino as a data source using sfu-db/connector-x (PR #405)
[https://avatars.githubusercontent.com/u/670302?s=20&v=4]houqp left a comment (roapi/roapi#405)<#405 (comment)>
@vcknorket<https://github.com/vcknorket> you will need to send an upstream PR to connector to bump their version of arrow to 55 first.
—
Reply to this email directly, view it on GitHub<#405 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWCT64STYLLUBJHQDMTLJML3FTSG3AVCNFSM6AAAAAB7JE2GWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMJRHA4TGNBZG4>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Member
|
@vcknorket you should be able to upgrade, I have don't that a few times in the past. Could you link me to the discussion? |
|
Hey @vcknorket do you continue working on it? I'm using roapi for some scenarios here and now. Trino supports it, which is essential for us. If you aren't working on it, I'll clone this branch, add test scenarios, and validate it before merging this new feature into the product |
Author
|
Will be great if you can help with this..and make the needed changes |
|
Nice, thank you for the faster reply haha I'll work on it and return if there's news, thanks, guys! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit introduces support for Trino as a data source by updating the connector-x dependency and integrating its Trino capabilities.
Key changes include:
Dependency Update:
connectorxincolumnq/Cargo.tomlfrom the forkedroapi/connector-xto the officialsfu-db/connector-x(version
0.4.4-alpha.1).connectorxfeatures to["fptr", "src_trino", "dst_arrow"].Core Logic for Trino:
Trinovariant to theDatabaseLoaderenum incolumnq/src/table/database.rs.columnq/src/table/database.rsto include
database-trino.columnq/src/table/mod.rs:Trinoto theExtensionenum and updated itsFromand
TryFromimplementations.trino { table: Option<String> }variant to theTableLoadOptionenum.TableLoadOption::extension()method.TableSource::parse_option()to recognize "trino" URIs.load()function to handle Trino.columnq/src/table/database.rsto support
TableLoadOption::trino.Cargo Feature Flag:
database-trino = ["connectorx/src_trino"]to the[features]section incolumnq/Cargo.toml.database-trinoin thedatabasefeature group.Testing:
columnq/tests/table_trino_test.rs.Trino integration, conditional on the
database-trinofeature.a live Trino instance and appropriate environment setup.
This integration allows you to specify Trino as a data source via URI or configuration, leveraging the capabilities of the underlying
sfu-db/connector-xlibrary.