Skip to content

One NaN or inf value silently drops the whole batch of logs, spans or metrics #8066

Closed

Description

@4ktLuffy

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.72.0 (latest on PyPI at the time of writing; also on master 0f7acd5)

Steps to Reproduce

Self-contained, only needs pip install sentry-sdk. It starts a local DSN sink on 127.0.0.1 and
counts the log records it receives. A plain stdlib logging call with a NaN argument is enough.

# Repro: one NaN attribute value drops the whole batch of logs
import json, logging, sys, threading
from http.server import BaseHTTPRequestHandler, HTTPServer

import sentry_sdk

received = []

class Sink(BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_POST(self):
        body = self.rfile.read(int(self.headers["Content-Length"]))
        import gzip
        if self.headers.get("Content-Encoding") == "gzip": body = gzip.decompress(body)
        lines = body.split(b"\n")
        for ln in lines[1:]:
            try:
                d = json.loads(ln)
            except Exception:
                continue
            if "items" in d: received.append(len(d["items"]))
        self.send_response(200); self.send_header("Content-Length", "2"); self.end_headers(); self.wfile.write(b"{}")

srv = HTTPServer(("127.0.0.1", 0), Sink)
threading.Thread(target=srv.serve_forever, daemon=True).start()

value = float("nan") if sys.argv[1] == "nan" else 0.5
sentry_sdk.init(dsn="http://k@127.0.0.1:%d/1" % srv.server_port, enable_logs=True)

for i in range(10):
    sentry_sdk.logger.info("innocent log %d" % i)
log = logging.getLogger("app"); log.setLevel(logging.INFO)
log.info("loss=%s", value)                                 # plain stdlib logging call, NaN as a %s argument
sentry_sdk.flush()
print(sys.argv[1], "-> log records delivered:", sum(received), "(sent 11)")

Run python repro.py ok and python repro.py nan.

Expected Result

The 10 innocent logs are delivered. At most the one record with the non-finite value is dropped
(or the value is sent as a string), and the drop is recorded in a client report.

Actual Result

ok  -> log records delivered: 11 (sent 11)
nan -> log records delivered: 0 (sent 11)

All 11 logs are lost, with no client report; only debug=True shows the error. The same happens
with span streaming: one span with a NaN attribute loses the whole trace (6 of 6 spans in my
test), and with buffered metrics: one sentry_sdk.metrics.gauge("loss", float("nan")) next to
10 counters, and 0 of 11 metrics arrive (11 of 11 with 0.5).

Cause: json_dumps in sentry_sdk/utils.py:125 uses allow_nan=False. format_attribute
(utils.py:2159) keeps non-finite floats as they are, and serialization only happens later in
the transport worker, so the ValueError loses the whole envelope. The event path already
handles this in serializer.py:286.

Possible fix: turn non-finite floats into strings (or drop that one attribute) in
format_attribute, and record a client report if an item still fails to serialize. Happy to
open a PR if that sounds right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions