feat(webapp,redis-worker): add global mollifier gate mode#3837
Conversation
The per-env trip rate-limits each environment independently and cannot bound the aggregate trigger rate hitting the primary database. Global mode rate-limits the fleet-wide aggregate via a single shared Redis counter (mollifier:rate:{global}), ignoring per-env contributions. Behind config, default unchanged (per_env).
- redis-worker: MollifierBuffer.evaluateTripGlobal reuses the trip Lua against hash-tagged global keys
- trip evaluator gains a mode option; global mode reports reason "global_rate"
- separate TRIGGER_MOLLIFIER_GATE_MODE + TRIGGER_MOLLIFIER_GLOBAL_* env vars
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 3a47cc7 The changes in this PR will be included in the next version bump. This PR includes changesets to release 25 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
📜 Recent review details⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (21)
🧰 Additional context used📓 Path-based instructions (11)**/*.{ts,tsx}📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
{packages/core,apps/webapp}/**/*.{ts,tsx}📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
**/*.{ts,tsx,js,jsx}📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
**/*.ts📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)
Files:
apps/webapp/**/*.{ts,tsx}📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
Files:
apps/webapp/**/*.server.ts📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md)
Files:
**/*.{js,ts,tsx,jsx,css,json,md}📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.{test,spec}.{ts,tsx}📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
apps/webapp/**/*.test.{ts,tsx}📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
Files:
**/*.test.{ts,tsx}📄 CodeRabbit inference engine (CLAUDE.md)
Files:
**/*.test.{js,ts,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
🧠 Learnings (46)📓 Common learnings📚 Learning: 2026-03-22T13:26:12.060ZApplied to files:
📚 Learning: 2026-03-22T19:24:14.403ZApplied to files:
📚 Learning: 2026-05-18T08:21:27.694ZApplied to files:
📚 Learning: 2026-05-18T08:21:27.694ZApplied to files:
📚 Learning: 2026-03-29T19:16:28.864ZApplied to files:
📚 Learning: 2026-05-05T09:38:02.512ZApplied to files:
📚 Learning: 2026-05-12T21:04:05.815ZApplied to files:
📚 Learning: 2026-05-14T08:21:07.614ZApplied to files:
📚 Learning: 2026-06-04T18:16:35.386ZApplied to files:
📚 Learning: 2026-06-01T12:05:44.112ZApplied to files:
📚 Learning: 2026-06-01T15:01:35.175ZApplied to files:
📚 Learning: 2026-03-02T12:43:43.173ZApplied to files:
📚 Learning: 2026-01-12T17:18:09.451ZApplied to files:
📚 Learning: 2026-05-15T08:05:57.683ZApplied to files:
📚 Learning: 2026-03-03T13:08:03.862ZApplied to files:
📚 Learning: 2026-03-02T12:43:43.173ZApplied to files:
📚 Learning: 2026-05-12T06:43:12.346ZApplied to files:
📚 Learning: 2026-03-03T13:07:33.177ZApplied to files:
📚 Learning: 2026-05-04T19:14:58.481ZApplied to files:
📚 Learning: 2026-05-14T08:21:10.439ZApplied to files:
📚 Learning: 2026-05-07T12:25:18.271ZApplied to files:
📚 Learning: 2026-05-28T20:02:10.647ZApplied to files:
📚 Learning: 2026-05-18T14:40:02.173ZApplied to files:
📚 Learning: 2026-04-23T13:26:31.290ZApplied to files:
📚 Learning: 2026-03-02T12:43:43.173ZApplied to files:
📚 Learning: 2026-05-26T13:46:37.556ZApplied to files:
📚 Learning: 2026-05-15T08:05:54.659ZApplied to files:
📚 Learning: 2026-06-01T11:37:12.623ZApplied to files:
📚 Learning: 2026-05-22T15:14:11.190ZApplied to files:
📚 Learning: 2026-06-01T11:37:08.569ZApplied to files:
📚 Learning: 2026-05-20T17:21:18.543ZApplied to files:
📚 Learning: 2026-04-15T15:39:31.575ZApplied to files:
📚 Learning: 2026-03-02T12:42:47.652ZApplied to files:
📚 Learning: 2026-04-16T14:19:16.330ZApplied to files:
📚 Learning: 2026-04-15T15:39:31.575ZApplied to files:
📚 Learning: 2026-05-19T13:33:07.551ZApplied to files:
📚 Learning: 2026-03-27T18:11:57.032ZApplied to files:
📚 Learning: 2026-04-16T13:45:22.317ZApplied to files:
📚 Learning: 2026-03-02T12:43:25.254ZApplied to files:
📚 Learning: 2026-04-07T14:12:18.946ZApplied to files:
📚 Learning: 2026-05-18T14:40:18.886ZApplied to files:
📚 Learning: 2026-05-04T19:14:44.097ZApplied to files:
📚 Learning: 2026-03-02T12:43:43.173ZApplied to files:
📚 Learning: 2026-05-18T14:40:02.173ZApplied to files:
📚 Learning: 2026-03-06T14:44:55.489ZApplied to files:
🔇 Additional comments (13)
WalkthroughThis pull request introduces a fleet-wide rate-limiting mode for the Mollifier trigger gate. Previously, rate limits were enforced per environment using separate Redis counters. The change adds ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
|
Summary
The mollifier trip gate is per-env: each environment is rate-limited independently. That structurally cannot bound the aggregate trigger rate hitting shared infrastructure (the primary database) — N environments each sitting just under their per-env threshold sum to N× the intended ceiling. This adds an opt-in global gate mode that rate-limits the fleet-wide aggregate via a single shared counter. It's behind config and off by default (existing
per_envbehavior unchanged).Design
MollifierBuffer.evaluateTripGlobal(opts)reuses the existing fixed-window trip Lua but against fixedmollifier:rate:{global}/mollifier:tripped:{global}keys (hash-tagged so the two-key script stays on one slot under Redis Cluster). Every trigger increments the same counter, so the gate sees aggregate load and ignores per-env contributions entirely — by design, so a global cap can't be sidestepped by spreading load across envs.The trip evaluator gains a
mode: "per_env" | "global"option; in global mode it callsevaluateTripGlobaland reportsreason: "global_rate"(a distinct metric label fromper_env_rate).Tuning is kept in separate env vars so the two regimes never share values:
TRIGGER_MOLLIFIER_GATE_MODE(defaultper_env), plusTRIGGER_MOLLIFIER_GLOBAL_TRIP_WINDOW_MS,TRIGGER_MOLLIFIER_GLOBAL_TRIP_THRESHOLD,TRIGGER_MOLLIFIER_GLOBAL_HOLD_MS.Test plan
evaluateTripGlobaltrips once the global counter exceeds threshold, ignoring envholdMs > windowMs);INCRatomic under 100 concurrent callsper_envmode; switching to global mid-flight starts the counter cold and is isolated from per-env countersreason: "global_rate"propagates throughevaluateGatetorecordDecisionfor both shadow-log and mollify outcomesManual verification (local)
Ran against a local webapp + Redis with
GATE_MODE=global:mollifier:rate:{global}/mollifier:tripped:{global}(no per-env keys), every decision loggedreason: "global_rate", and the count-vs-threshold boundary was exact (count > thresholdtrips).TRIGGER_MOLLIFIER_ENABLED=0): triggers passed through with zeromollifier:*keys created and no decision logs — the kill-switch short-circuits before the mode is read, so this holds identically for bothper_envandglobal.