[adapters] Tolerate transient Kafka errors in FT input tests#6586
Closed
ryzhyk wants to merge 1 commit into
Closed
[adapters] Tolerate transient Kafka errors in FT input tests#6586ryzhyk wants to merge 1 commit into
ryzhyk wants to merge 1 commit into
Conversation
Transient librdkafka connectivity blips ("AllBrokersDown", "N/M brokers
are down") are reported to the consumer as non-fatal errors, and the
connector recovers on its own. DummyInputConsumer recorded every such
error in the strict call sequence that expect() checks, so a blip
anywhere in a test failed it, e.g.:
thread 'transport::kafka::ft::test::multiple_input' panicked at
crates/adapters/src/transport/kafka/ft/test.rs:463:22:
assertion `left == right` failed
left: [Extended { num_records: 90, ... }]
right: [Error(false)]
Record only fatal errors. No test expects ConsumerCall::Error, so no
expectation changes.
Validated by bouncing the broker mid-test to inject the blip: with this
change multiple_input rides out the errors; without it, the test
reproduces the CI failure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mythical-fred
approved these changes
Jul 3, 2026
mythical-fred
left a comment
There was a problem hiding this comment.
Approve. Right fix: non-fatal errors are advisory and should not enter the strict call-sequence expectation. Matches the MockInputConsumer convention (on_error(|,| {})) and the validation table nicely isolates the flake with a broker bounce. info! on the non-fatal path keeps the signal in logs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
transport::kafka::ft::test::multiple_inputfailed in the merge queue (run 28634798524) and kicked out an unrelated PR (#6584):The job log shows the cause: librdkafka reported a momentary connectivity blip ("1/2 brokers are down", "AllBrokersDown").
refine_kafka_errorclassifies such errors as non-fatal, and the connector recovered on its own: the expectedExtended { num_records: 90 }call arrived 100 ms after the panic.The bug is in the test harness:
DummyInputConsumer::errorrecords every error, fatal or not, in the strict call sequence thatexpect()checks, so a transient blip anywhere intest_inputortest_input_partitionfails the test.Fix
Record only fatal errors; log non-fatal ones. No test expects
ConsumerCall::Error, so no expectation changes. This matches theMockInputConsumer-based Kafka tests, which installon_error(|_, _| {}). Same flake class as #1830.Validation
Reproduced the blip locally by bouncing a dedicated Redpanda broker mid-test (
docker restart -t 1), which produces the exact CI error signature (Message consumption error: AllBrokersDown ...,N/M brokers are down):right: [Error(false)], same as CIkafka::ft::testmodule: 39 passed, 0 failed🤖 Generated with Claude Code