Skip to content

[RFC][Connectors]: Add native s2 input and output connector support in Feldera #5728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mrhs121 wants to merge 9 commits into
base: main
Choose a base branch
from
Open

[RFC][Connectors]: Add native s2 input and output connector support in Feldera #5728

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
35b3e6a
support s2 native connector
Mrhs121 Mar 2, 2026
3956c0d
support insert_delete for s2 connector
Mrhs121 Mar 2, 2026
8ec5634
support retry connect for s2 connector
Mrhs121 Mar 2, 2026
f853b23
replace manual retry logic with S2 SDK built-in RetryConfig
Mrhs121 Mar 6, 2026
b34d580
support s2_output for s2 connector
Mrhs121 Mar 12, 2026
35388a2
fix s2_output buffer size limit to respect S2 1MB per-record constraint
Mrhs121 Mar 12, 2026
215f527
downgrade s2 input per-record log from info to trace
Mrhs121 Mar 18, 2026
1c18523
move raw format envelope mismatch check to error path to avoid double…
Mrhs121 Mar 18, 2026
feedd68
upgrade s2-sdk from 0.24.4 to 0.26.0 to fix internal dependency conflict
Mrhs121 Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix s2_output buffer size limit to respect S2 1MB per-record constraint
  • Loading branch information
@Mrhs121
Mrhs121 committed Mar 23, 2026
commit 35388a2ebcaa01a162b6096cf94c8e0b73a28860
4 changes: 3 additions & 1 deletion crates/adapters/src/transport/s2/output.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,9 @@ impl OutputEndpoint for S2OutputEndpoint {
}

fn max_buffer_size_bytes(&self) -> usize {
usize::MAX
// S2 enforces a 1MB limit per AppendRecord.
// Use a conservative limit to leave room for framing overhead.
1_000_000
}

Copy link
Copy Markdown

@mythical-fred mythical-fred Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

push_buffer submits and immediately awaits the ack before returning — one round-trip per buffer. For real throughput: pipeline submits and wait for acks at flush/disconnect. Not a hard blocker for a first pass, but add a TODO so it isn't forgotten.

fn push_buffer(&mut self, buffer: &[u8]) -> AnyResult<()> {
Expand Down