fix: bound the decoder adapter caches - #191

What changed

PydanticDecoder and MsgspecDecoder each keep a dict from model to adapter (decoders/_caching.py). Until now that dict never evicted and held every model strongly, so code that builds a model per request grew it, and kept the classes alive, for the life of the decoder.

Each cache now holds at most _MAX_ENTRIES = 4096 entries. When an insert would go past the cap, the cache is cleared first and then the new entry is stored. That covers the class/union key path and the _lookup_by_repr fallback for unhashable models: both now insert through one _store helper. The cap is a module constant, not public config.

Reproduction (before the fix)

10,000 models decoded once each through one decoder, each model in its own process run:

After the fix, the same run ends with 1,808 entries in each cache (10,000 minus two full clears of 4,096), and the first model is collected (weakref is dead). Peak RSS growth falls to 51.5 MB (pydantic) and 17.3 MB (msgspec). The peak still includes a full cache of 4,096 entries.

Why clear-on-overflow and why 4096

Clearing keeps the cache-hit path exactly as it was: a hit runs the same _key + dict.get as before, and the cap check only runs on a miss. An LRU would need bookkeeping on every hit, and the lookup runs on every decoded request. A WeakKeyDictionary won't work, because the cached adapter holds the model strongly.

An entry is one distinct type expression per decoder, so User, list[User] and User | None take three slots. Even a client for a large API with a few hundred response models, plus their list and optional forms, stays well under 4,096, so a real app should never hit the clear. Measured on trivial one-field models, an entry costs about 9 KB for pydantic and 2.3 KB for msgspec, so a full cache tops out around 37 MB and 10 MB for such models. Apps above the cap pay a periodic rebuild. Apps that make models per request now have bounded memory and get their adapters rebuilt after each clear.

Thread safety

Unchanged: concurrent first calls may build twice and the last write wins. dict.get, len, clear and item assignment are each atomic on a dict, including under free threading, so a lookup racing a clear sees either the old entry for its own key or a miss and rebuilds. It never raises and never returns another model's adapter. Under a race the cache can briefly go past the cap by about the number of threads inserting at once. A new stress test drives 16 threads through a cache with a tiny cap and checks that every lookup returns its own model's entry.

Benchmarks

The cache-hit bytecode of _lookup is unchanged; only the miss branch moved into _store. Interleaved A/B in one process (old and new _caching modules side by side, best of 9 runs of 500k calls), on a machine with a load average around 40:

The differences go both ways from run to run and are noise. A full mocked Client.get(..., response_model=User) measured 178 us before and 172 us after in back-to-back runs on the same loaded machine, also noise.

Tests

tests/test_decoders_caching.py patches the cap to 8. It checks that cap+1 create_model models leave the pydantic cache bounded with the first model garbage collected, and that cap+1 defstruct models leave the msgspec cache bounded with the first model no longer in it. It also checks that the repr-keyed path is bounded, that a model decodes after a clear, and the threaded race above. The three bound tests failed before the fix.

The msgspec test checks only our cache, not a weakref. msgspec 0.19 (a dependency floor) runs MsgpackDecoder(typing.Tuple[self.types]) inside msgspec.inspect.type_info, and typing caches subscriptions in an lru_cache of 128, so the last 128 inspected models stay alive outside httpware (found with gc.get_referrers in a floors venv). msgspec 0.22 uses builtin tuple[...], which typing does not cache, and the weakref check passes there. That retention is capped at 128, so it is not a leak.

Docs

docs/decoders.md gets a short section saying response models should be long-lived, and that models created per request get rebuilt once the cache fills.

Closes #174