fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation by pabloDeputter · Pull Request #7050 · getsentry/sentry-python

pabloDeputter · 2026-08-05T14:51:07Z

Description

Summary of issue

pre-4.12.0 ddtrace basically behaved like Sentry before: propagation headers were added by the HTTP client after botocore already had calculate the SigV4 signature. So baggage was not included in SignedHeaders. Any later modifications to the value did not invalidate the request.
4.12.0 moves the header injection to botocore's before-sign event. It adds baggage, ... and x-datadog-* before signing. Any later modifications to the value DO invalidate the request, thus later HTTP-client injection is suppressed to avoid duplicate headers.
How failure happens:
1. incoming FastAPI request contains Sentry or other W3C baggage
2. datadog extracts that baggage into its active context
3. boto3 request is created
4. datadog’s before-sign handler writes the baggage to the AWS request
5. botocore includes baggage in the SigV4 signature
6. sentry’s httplib runs afterwards and updates or adds another baggage value
7. AWS receives a different baggage value from the one that was signed. Because request was "tampered" with after signing, AWS rejects the request with 403 Forbidden or SignatureDoesNotMatch.

Changes

inject sentry propagation through before-sign handler, so final baggage and sentry-trace values are created before SigV4 signing.
existing third-party baggage is merged into one header.
http.client propagation is delayed until endheaders(), when the complete request headers and SigV4 SignedHeaders are available. Existing baggage header is never mutated after it already was signed.
propagation is skipped for presigned requests, since adding headers could require a future caller to reproduce these exact headers.

Issues

Resolves: #7031 & PY-2667

Reminders

Please add tests to validate your changes, and lint your code using uv run ruff.
Add GH Issue ID & Linear ID (if applicable)
PR title should use conventional commit style (feat:, fix:, ref:, meta:)
For external contributors: CONTRIBUTING.md, Sentry SDK development docs, Discord community

- merge Sentry baggage with existing vendor (e.g. Datadog) baggage in botocore's`before-sign` hook; avoiding post-sign header tampering that invalidates the SigV4 signature. - Skip propagation for presigned requests Fixes: #7031 & PY-2667

github-actions · 2026-08-05T15:04:10Z

Codecov Results 📊

✅ 99283 passed | ⏭️ 6494 skipped | Total: 105777 | Pass Rate: 93.86% | Execution Time: 340m 44s

📊 Comparison with Base Branch

Metric	Change
Total Tests	📈 +140
Passed Tests	📈 +126
Failed Tests	📉 -1
Skipped Tests	📈 +15

➖ Removed Tests (1)

View removed tests

test_continuous_profiler_auto_start_and_stop_sampled_span_streaming[non-experiment-thread]
- File: tests.profiler.test_continuous_profiler

All tests are passing successfully.

✅ Patch coverage is 94.81%. Project has 2503 uncovered lines.
✅ Project coverage is 89.99%. Comparing base (base) to head (head).

Files with missing lines (2)

File	Patch %	Lines
sentry_sdk/integrations/stdlib.py	94.12%	⚠️ 3 Missing and 4 partials
sentry_sdk/integrations/boto3.py	96.15%	⚠️ 1 Missing and 1 partials

Coverage diff

@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
+ Coverage    89.99%    89.99%        —%
==========================================
  Files          193       193         —
  Lines        24946     25016       +70
  Branches      9000      9034       +34
==========================================
+ Hits         22451     22513       +62
- Misses        2495      2503        +8
- Partials      1435      1437        +2

Generated by Codecov Action

- trace-header injection is delayed until `endheaders()` after all request headers are available. - avoid duplicate propagation headers. - no tampering/modifying of already signed baggage Fixes: #7031 & PY-2667

Refs: #7031 & PY-2667

# Refs: #7031 & PY-2667

alexander-alderman-webb · 2026-08-06T11:20:09Z

        return rv

+    def endheaders(self: "HTTPConnection", *args: "Any", **kwargs: "Any") -> "Any":
+        trace_headers = getattr(self, "_sentrysdk_trace_headers", ())


Does calling sentry_sdk.get_current_scope().iter_trace_propagation_headers() here instead of in putrequest() work?
It would be best to avoid stashing stuff on the HTTPConnection instance if we can help it.

Good suggestion :) header generation is moved into endheaders() instead of being stored on the connection.

Refs: #7031 & PY-2667

… SigV4 headers Refs: #7031 & PY-2667

sentry-warden · 2026-08-06T15:34:26Z

                _complete_span(span)

    HTTPConnection.putrequest = putrequest  # type: ignore[method-assign]
+    HTTPConnection.endheaders = endheaders  # type: ignore[method-assign]


Eager .format() in endheaders crashes on URLs or header values with braces

The endheaders patch eagerly interpolates user-controlled real_url and header_value with .format(), raising KeyError and aborting the HTTP request when either contains curly braces.

Evidence

In sentry_sdk/integrations/stdlib.py, the endheaders function (defined at line 211) calls logger.debug("...{real_url}...".format(key=header_name, value=header_value, real_url=real_url)) at line 236.

real_url is constructed from the url argument to putrequest (line 192), which may contain literal braces such as /api/{id}.

header_value comes from iter_trace_propagation_headers (line 224) and may include baggage or trace values with braces.

Python evaluates the .format() eagerly before logger.debug is invoked, so a KeyError propagates out of the try block even when debug logging is disabled, aborting the HTTP request.

_{Identified by Warden · code-review · 4DB-4BC}

pabloDeputter added 5 commits August 5, 2026 17:46

fix(stdlib): Preserve signed headers during trace propagation

78f1ee6

- trace-header injection is delayed until `endheaders()` after all request headers are available. - avoid duplicate propagation headers. - no tampering/modifying of already signed baggage Fixes: #7031 & PY-2667

test(boto3): add tests covering trace propagation + SigV4 signing

88fdd7d

Refs: #7031 & PY-2667

test(boto3): inline span_streaming logic

a1e5f9c

Refs: #7031 & PY-2667

test(stdlib): cover httplib SigV4 trace header preservation

4ba8901

Refs: #7031 & PY-2667

test(boto3): Add server_close() in test cleanup

92672ca

# Refs: #7031 & PY-2667

pabloDeputter marked this pull request as ready for review August 6, 2026 11:12

pabloDeputter requested a review from a team as a code owner August 6, 2026 11:12

sentry Bot reviewed Aug 6, 2026

View reviewed changes

Comment thread sentry_sdk/integrations/stdlib.py Outdated

alexander-alderman-webb reviewed Aug 6, 2026

View reviewed changes

pabloDeputter added 2 commits August 6, 2026 14:41

fix(stdlib): clarify comment

37fa9ff

Refs: #7031 & PY-2667

fix(stdlib): Move trace header generation into endheaders()

e675673

Refs: #7031 & PY-2667

sentry-warden Bot reviewed Aug 6, 2026

View reviewed changes

Comment thread sentry_sdk/integrations/stdlib.py Outdated

fix(stdlib): Replace return set() with continue to avoid skipping…

14bad29

… SigV4 headers Refs: #7031 & PY-2667

sentry-warden Bot reviewed Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation - #7050

fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation#7050
pabloDeputter wants to merge 9 commits into
masterfrom
pablo/fix-botocore-sigv4-trace-propagation

pabloDeputter commented Aug 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

alexander-alderman-webb Aug 6, 2026

Uh oh!

pabloDeputter Aug 6, 2026

Uh oh!

Uh oh!

sentry-warden Bot Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

pabloDeputter commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Summary of issue

Changes

Issues

Reminders

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Results 📊

📊 Comparison with Base Branch

➖ Removed Tests (1)

Uh oh!

Uh oh!

alexander-alderman-webb Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

pabloDeputter Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sentry-warden Bot Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pabloDeputter commented Aug 5, 2026 •

edited

Loading

github-actions Bot commented Aug 5, 2026 •

edited

Loading