test(aibridge): reproduce Bedrock SigV4 failure when a proxy mutates X-Forwarded-For#26018
Draft
blinkagent[bot] wants to merge 3 commits into
Draft
test(aibridge): reproduce Bedrock SigV4 failure when a proxy mutates X-Forwarded-For#26018blinkagent[bot] wants to merge 3 commits into
blinkagent[bot] wants to merge 3 commits into
Conversation
…en a proxy mutates X-Forwarded-For
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This is a reproduction-only PR. It adds a single new test,
TestBlockingInterception_BedrockProxyHeadersBreakSigV4, that demonstrates a production failure mode reported by a customer whose AI Bridge deployment fronts AWS Bedrock from behind the Coder app proxy.The customer hit the following AWS error on every Bedrock request:
The canonical request returned by AWS included
x-forwarded-for,x-forwarded-host, andx-forwarded-protoin theSignedHeaderslist, which is the smoking gun.Hypothesis under test
X-Forwarded-*headers populated by the upstream Coder app proxy.intercept.PrepareClientHeaders/intercept.BuildUpstreamHeaders(inaibridge/intercept/client_headers.go) preserves every non-auth, non-hop-by-hop client header on the outbound request to the upstream provider, includingX-Forwarded-*.X-Forwarded-*end up in the canonical request and theSignedHeaderslist.bedrock-runtime.<region>.amazonaws.comthat mutates one of those signed headers (an outbound HTTP proxy that appends its own hop toX-Forwarded-Foris the most common offender) causes the SigV4 signature AWS recomputes from received headers to differ from the one aibridge sent, producingSignatureDoesNotMatch.How the test reproduces this
The test spins up two
httptestservers:A mock Bedrock server that performs real SigV4 verification on every incoming request:
Authorizationheader to extract the credential scope andSignedHeaderslist.SignedHeaders.aws/aws-sdk-go-v2/aws/signer/v4.Authorizationto what came in. Mismatch -> 403 with an AWS-shapeddoes not matcherror; match -> 200 with a non-streaming Anthropic response.A mock proxy that sits between aibridge and the mock Bedrock and appends
, 10.0.0.42to the request'sX-Forwarded-Forheader before forwarding. This mirrors what an outbound HTTP proxy does in a typical enterprise network path.The interceptor is invoked twice:
baseline_direct_to_bedrock_passes-- aibridge -> mock Bedrock directly. TheX-Forwarded-*headers reach the mock unchanged and SigV4 verification succeeds. This proves the verifier agrees with the SDK's signer on a clean path.mutating_proxy_breaks_signature-- aibridge -> mock proxy -> mock Bedrock. The mock proxy mutatesX-Forwarded-For, the verifier's recomputed signature diverges, and the mock returns a 403 in the AWS shape.Both subtests pass against
main, confirming the hypothesis.What this PR does NOT do
Suggested follow-up (not in this PR)
Add proxy-injected headers (at minimum
X-Forwarded-For,X-Forwarded-Host,X-Forwarded-Proto,Forwarded,X-Real-Ip,Via,True-Client-Ip,Cf-*) to the strip list inintercept.PrepareClientHeaders. They are never meaningful to an upstream LLM provider and they break SigV4 the moment any hop touches them.The chatd path was fixed via a similar mechanism in #24718 (fork the Anthropic SDK to strip Anthropic-only headers in the Bedrock middleware before signing).
Created on behalf of @ssncferreira.