Skip to content

test(aibridge): reproduce Bedrock SigV4 failure when a proxy mutates X-Forwarded-For#26018

Draft
blinkagent[bot] wants to merge 3 commits into
mainfrom
blink/bedrock-sigv4-proxy-headers-repro
Draft

test(aibridge): reproduce Bedrock SigV4 failure when a proxy mutates X-Forwarded-For#26018
blinkagent[bot] wants to merge 3 commits into
mainfrom
blink/bedrock-sigv4-proxy-headers-repro

Conversation

@blinkagent
Copy link
Copy Markdown
Contributor

@blinkagent blinkagent Bot commented Jun 3, 2026

What this PR does

This is a reproduction-only PR. It adds a single new test, TestBlockingInterception_BedrockProxyHeadersBreakSigV4, that demonstrates a production failure mode reported by a customer whose AI Bridge deployment fronts AWS Bedrock from behind the Coder app proxy.

The customer hit the following AWS error on every Bedrock request:

403 Forbidden { "message": "The request signature we calculated does not match the signature you provided. ..." }

The canonical request returned by AWS included x-forwarded-for, x-forwarded-host, and x-forwarded-proto in the SignedHeaders list, which is the smoking gun.

Hypothesis under test

  1. The original client request arrives at aibridge carrying X-Forwarded-* headers populated by the upstream Coder app proxy.
  2. intercept.PrepareClientHeaders / intercept.BuildUpstreamHeaders (in aibridge/intercept/client_headers.go) preserves every non-auth, non-hop-by-hop client header on the outbound request to the upstream provider, including X-Forwarded-*.
  3. The Anthropic SDK's Bedrock middleware signs the outgoing request with SigV4. The AWS SDK v2 signer signs every request header by default (the unsigned-headers denylist is small), so X-Forwarded-* end up in the canonical request and the SignedHeaders list.
  4. Anything between aibridge and bedrock-runtime.<region>.amazonaws.com that mutates one of those signed headers (an outbound HTTP proxy that appends its own hop to X-Forwarded-For is the most common offender) causes the SigV4 signature AWS recomputes from received headers to differ from the one aibridge sent, producing SignatureDoesNotMatch.

How the test reproduces this

The test spins up two httptest servers:

  • A mock Bedrock server that performs real SigV4 verification on every incoming request:

    • Parses the Authorization header to extract the credential scope and SignedHeaders list.
    • Rebuilds a verification request containing only the headers in SignedHeaders.
    • Re-signs with the same access key + secret using aws/aws-sdk-go-v2/aws/signer/v4.
    • Compares the resulting Authorization to what came in. Mismatch -> 403 with an AWS-shaped does not match error; match -> 200 with a non-streaming Anthropic response.
    • This is how AWS verifies SigV4 on its side: independently rebuild the canonical request from received headers, derive the signing key from the scope, and compare against the supplied signature byte-for-byte.
  • A mock proxy that sits between aibridge and the mock Bedrock and appends , 10.0.0.42 to the request's X-Forwarded-For header before forwarding. This mirrors what an outbound HTTP proxy does in a typical enterprise network path.

The interceptor is invoked twice:

  1. baseline_direct_to_bedrock_passes -- aibridge -> mock Bedrock directly. The X-Forwarded-* headers reach the mock unchanged and SigV4 verification succeeds. This proves the verifier agrees with the SDK's signer on a clean path.
  2. mutating_proxy_breaks_signature -- aibridge -> mock proxy -> mock Bedrock. The mock proxy mutates X-Forwarded-For, the verifier's recomputed signature diverges, and the mock returns a 403 in the AWS shape.

Both subtests pass against main, confirming the hypothesis.

What this PR does NOT do

  • It does not change production code paths. No fix is included here on purpose; we want to land the reproduction first and discuss the right remediation separately.

Suggested follow-up (not in this PR)

Add proxy-injected headers (at minimum X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, Forwarded, X-Real-Ip, Via, True-Client-Ip, Cf-*) to the strip list in intercept.PrepareClientHeaders. They are never meaningful to an upstream LLM provider and they break SigV4 the moment any hop touches them.

The chatd path was fixed via a similar mechanism in #24718 (fork the Anthropic SDK to strip Anthropic-only headers in the Bedrock middleware before signing).

Created on behalf of @ssncferreira.

@github-actions github-actions Bot added the community Pull Requests and issues created by the community. label Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community Pull Requests and issues created by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants