Lazy pagination iterator for catalog list methods#3416
Lazy pagination iterator for catalog list methods#3416GayathriSrividya wants to merge 2 commits into
Conversation
|
@GayathriSrividya To credit the original author, it would be good to cherry-pick his work |
7b6744e to
7e1538e
Compare
|
Addressed the CI failures with follow-up commits and pushed fixes to this PR branch. |
|
@GayathriSrividya was this based off of #2172? If it was, we should cherry-pick in the commits so that the original author shows up in this PR as well. |
Replace the collect-then-return approach with proper generator functions that yield results page by page. Extract per-page fetch logic into dedicated helper methods (_fetch_tables_page, _fetch_views_page, _fetch_namespaces_page) decorated with @Retry so authentication retries work correctly per page. Co-authored-by: Yuya Ebihara <ebyhry@gmail.com>
aa88377 to
54d8f64
Compare
Thanks for checking on this. Yes, this PR is based on @jayceslesar’s earlier work in #2172. I restructured the branch to preserve attribution clearly:
I also updated the PR description to explicitly mention that this builds on @jayceslesar’s work. Let me know if there’s a better way you’d prefer me to handle attribution here. |
Hi @rambleraptor and @Fokko — yes, PR #3416 was based on @jayceslesar's work from #2172. I've restructured the branch to credit them properly: Commit 1 (authored by @jayceslesar): Changes abstract method signatures and adds iter() wrapping for all non-REST catalogs (BigQuery, DynamoDB, Glue, Hive, NoOp, SQL) |
Closes #3365
This PR builds on @jayceslesar's earlier work in #2172 and has been restructured to credit the original author.
Changes
Commit 1 — by @jayceslesar (rebased from #2172):
list_tables,list_namespaces,list_viewsreturn types fromlist[Identifier]toIterator[Identifier]iter()of their resultsCommit 2 — by @GayathriSrividya:
_fetch_tables_page,_fetch_namespaces_page,_fetch_views_page) decorated with@retryso auth retry logic works correctly per page