Skip to content

feat: Full Stack v12 — safe traffic ramp-up, variation preview & experiment exclusion - #56

Merged
abbaseya merged 39 commits into
mainfrom
feat/fullstack-v12
Jul 24, 2026
Merged

feat: Full Stack v12 — safe traffic ramp-up, variation preview & experiment exclusion#56
abbaseya merged 39 commits into
mainfrom
feat/fullstack-v12

Conversation

@abbaseya

Copy link
Copy Markdown
Collaborator

Full Stack v12 — next major SDK release

This branch delivers three major capabilities, shipped consistently across all Full Stack SDKs. All three are inert until activated on the backend/UI side, so this deploys safely in any order.

1. Anchored (ramping) bucketing layout

Makes changing an experiment's total traffic allocation safe on a running experiment.

  • Each variation is pinned at its own fixed offset (cumulative-weight anchor); its range width equals its allocation. Raising the total grows every arm at its own edge (even growth); lowering ejects evenly; nobody already bucketed gets reshuffled.
  • Gated on the experience's existing serving version property: > 11 → anchored pass (contract v12); ≤ 11 / missing → packed pass, bit-identical to today. No schema change, zero migration.
  • Hash, seed, stored-decision guard, forced path, and events are untouched. Still stateless, deterministic, and identical across every SDK — enforced by shared cross-SDK golden vectors.
  • Changing variation weights mid-flight stays unsupported (unsafe in every layout); the split lock is unchanged.

2. Experiment preview

Lets users preview any variation — including drafts and paused ones — in their own environment, with zero trace.

  • New debugToken config option (24h QA token) unlocks the full config, including draft/paused experiences.
  • Stateless preview links (web) and deep links (mobile) via convert_preview={expId}.{varId} force a single variation with no side effects: no tracking events, no visitor-state writes, no impact on reporting.
  • Rides the existing public serving primitive — no backend serving changes required.

3. Mutual exclusion / inclusion audience rule

Lets teams control which experiments a visitor can be in at the same time, through the normal audience-targeting rules.

  • New audience rule bucketed_into_experience_key, resolved against SDK-stored bucketing state.
  • negated: true → "NOT in experiment X" (mutual exclusion); default → "in experiment X" (inclusion).
  • No new application inputs; evaluates before bucketing and composes identically with both packed and anchored layouts. Old SDKs fail closed.

Notes

  • Backend registration of the new rule type and the version 12 stamp, plus the related convert-ui work (preview link/QR button, rule picker) and shared docs, are tracked and coordinated separately.
  • Full existing test suites remain green; the anchored pass adds the shared golden-vector acceptance tests.

abbaseya and others added 30 commits July 4, 2026 02:55
… fixture

Byte-for-byte copy from javascript-sdk feat/anchored-bucketing-layout
(packages/bucketing/tests/cross-sdk-bucketing-vectors.json).
sha256 9993d0e6c5719cb17a4dcbcc73af9c0cb14ee0258e7fd29cc88f44b61083dd1b
59 vectors, versions {11,12}. The fixture IS the cross-SDK contract.

Beads: ai-driven-product-dev-padw

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TDD Phase 1: pytest AC1-AC9 + 59-vector parameterized golden runner.
RED confirmed: 29 failed / 51 passed / 1 collection error — ImportError/
AttributeError on not-yet-existing anchored selectors + v12 value mismatches;
v11 packed vectors pass (AC6 lock intact).

Beads: ai-driven-product-dev-k1vf

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…EEN)

TDD Phase 2: anchored layout gated on experience.version > 11.
- bucketing.py: build_bucket_ranges + select_bucket_anchored (anchor scale
  reuses DEFAULT_MAX_TRAFFIC; half-open anchor<=v<anchor+width; inactive arms
  keep weight, zero width).
- experiences.py: _is_anchored_layout (float() coercion, '12' routes anchored
  like JS Number('12')>11) + _build_variation_allocations (full ordered list
  incl. inactive arms); gate wired into select_experience. Packed path
  (_build_buckets + select_bucket) byte-for-byte untouched (AC6).
- Fixed one RED test bug: (11.9, False) -> (11.9, True), grounded on the JS
  oracle's literal Number(version)>11 (no flooring).

Gates: full suite 949 passed; golden 59/59; mypy strict clean; ruff clean.

Beads: ai-driven-product-dev-k1vf

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…unt + conftest type

Documentation-only (code review R1, conf 88%): rewrote the TDD RED-phase module
docstrings in the 3 new test files to describe current GREEN state; corrected the
pair-count figure 18 -> 8 (verified against the fixture); fixed conftest _load
return-type annotation to Union[dict, list] for the bare-list anchored fixture.
No source/fixture/assertion changes. Full suite 949 passed; ruff clean.

Beads: ai-driven-product-dev-k77j

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tests/parity/README.md + TESTING.md: list the anchored golden-vector fixture
(anchored_bucketing_vectors.json, bare list, imported verbatim from the cross-SDK
source) and its parity test alongside the MurmurHash3 parity proof; note it is not
written by the fixture generator. Part of PR #52.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BREAKING CHANGE: Experiences on bucketing contract v12 (experience version above 11) bucket visitors with the anchored layout, where each variation holds a fixed position derived from its cumulative traffic allocation. Changing an experience's total allocation now grows or shrinks each variation at its own boundary instead of shifting every variation range, so visitor-to-variation assignments under contract v12 differ from the earlier packed layout for the same configuration.
feat: anchored bucketing layout (contract v12) — Python SDK pass
Beads: ai-driven-product-dev-cope
19 new tests fail (missing debug_token field); 34 pre-existing pass. ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-cope
SDKConfig.debug_token: Optional[str] = field(repr=False) with strict validation.
Config route carries debug_token + forced _conv_low_cache=1; never on track route.
Full suite 968 passed, coverage 96.81% (eval 98%), mypy --strict + ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-rq0l
Pure helper convert_sdk.parse_preview_param; '123.456'->('123','456'), malformed->None
(strict numeric-segment check, JS/Ruby sibling parity, decision I6). 18 tests, full
suite 986 passed, cov 96.84% (preview.py 100%), mypy+ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-lgao
11 tests fail (get_preview_decision absent). Locks full-bypass matrix incl draft/paused/
non-running/zero-traffic variation + unknown-id->None. ruff clean.
NOTE: experience-status & environment bypass cases are forward-compat locks — Python's
normal path does not gate on those today (decision I9; flagged for story owner).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…REEN)

Beads: ai-driven-product-dev-lgao
get_preview_decision(experience, variation_id) in evaluation/experiences.py: reuses
_find_variation (no status/traffic filter) -> full bypass; normal ExperienceResult shape;
unknown variation->None; pure/no-side-effects. 11 tests, full suite 997 passed, eval cov 98%.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-5801
8 failed + 11 errored (missing fetch_config_by_experience + module memo globals).
Locks: exp=<id>&_conv_low_cache=1 (+debug_token) unconditional; 60s wall-clock TTL;
thread-safe single fetch under 6-thread barrier; cross-tenant isolation; never-DataStore;
failed fetch not memoized (JS parity). ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-5801
HttpxTransport.fetch_config_by_experience + _build_experience_route/query: exp=<id>&
_conv_low_cache=1 (+debug_token), no environment (spec Resolution clause, I16). Module-level
thread-safe 60s wall-clock memo keyed sdk_key:experience_id, never DataStore, failed fetch
not memoized. Test-only isolation fixture added (I18). Full suite 1016 passed, cov 96.90%/eval 98%.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-1b47
15 fail (Context.set_preview absent) + 1 diagnose regression-lock passes. Locks AC3
(?exp=-only delivery), AC4 (full-bypass matrix incl different stored decision), AC6
(isolation: concurrent non-preview context normal), AC7 (inert: unknown exp/var, fetch
503 -> warning + normal). Transport kept a public port (I22 -> capability protocol in GREEN).
ruff clean; 1016 pre-existing pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-1b47
Context.set_preview(experience_id, variation_id): eager resolution (local snapshot ->
?exp= fetch via SupportsPreviewFetch capability protocol -> get_preview_decision),
per-context state, precedence over stored/bucketed decision for target only, other
experiences normal, inert+warn on bad input. run_experiences synthesizes fetch-only
target (I27 test locks it). Transport stays a non-breaking public port. 18 targeted,
full suite 1034 passed, cov 96.81%/eval 98%, mypy+ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-oq4q
5/6 fail (spies see track POSTs + store writes that must be zero under preview); 1 AC10
regression lock passes (normal path unaffected). Full-lifecycle proof across core.flush()
+ atexit hook; AC6 isolation via shared Core; all persist/track sites swept (I34).
transport + _SpyDataStore spies. ruff clean; PY-5 suite 18/18 untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-oq4q
Preview context leaves zero trace: track_bucketing skipped in run_experience/run_experiences
(overrides enable_tracking=True), track_conversion unconditional no-op (returns DEDUPLICATED,
I35), _persist_visitor_state no-ops -> zero DataStore writes. Gated at Context call sites (P2),
per-context only (AC6 isolation). Full lifecycle incl flush()+atexit. Full suite 1040 passed,
cov 96.82%/eval 98%, mypy+ruff clean, generated-pyi OK.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on write

Beads: ai-driven-product-dev-cszc
Sweep-on-write inside _CONFIG_BY_EXPERIENCE_LOCK: closes unbounded-growth path keyed by
link-param experience_ids; completes TTL-60s semantics. AC8/never-DataStore/failed-fetch-
not-memoized/thread-safety preserved. ruby-sdk#41 parity (I36). 13 memo tests, full suite
1042 passed, cov 96.83%/eval 98%, mypy+ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An experience dict with a missing or null "id" stringified to "None" and
would falsely match a lookup for the literal experience_id "None"
(str(None) == "None"). Reject a None id before the string comparison so
malformed/id-less experiences never match — honoring the helper's
documented "never mismatch on malformed bodies" contract. Adds 5
parametrized regression cases. Addresses Gemini code-review finding on #53.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-mm0v
18 new tests fail (field/method/envelope absent); 56 pre-existing pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-mm0v
ContextState.bucketing field + with_bucketing (key-merge, carried through
with_attributes/with_segments); 3-key envelope persist; hydrate w/ legacy
2-key + plain-attributes backward compat (empty bucketing). No eval/hash change.
Full suite 1065 passed, 96.85% cov; ruff + mypy --strict clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-v0d1
5 new tests fail (sticky_bucketing param absent); 10 pre-existing pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-v0d1
select_experience gains pure sticky_bucketing param; short-circuits between
qualifies() and get_bucket_value_for_visitor when stored vid resolves via
_find_variation. Fresh+sticky returns share _build_experience_result helper
(value-neutral). None-arg byte-identical. AC2/AC4/AC8/AC9.
Full suite 1070 green, 96.87% cov; eval floor 98%; parity 214 green (AC9);
ruff+mypy strict clean. Parity fixtures untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-av5a
6 integration tests (AC1/2/3/5/6/8) over exp-a/exp-b 8-row fixture fail
(enable_storage/read/write wiring absent). AC4 row4 added next (GREEN dispatch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-av5a
enable_storage=True kwarg on run_experience/run_experiences (threaded via
_evaluate_and_record); sticky read wires self._state.bucketing into
select_experience (non-preview branch); gated write-after-hash
(result and enable_storage and preview is None and bucketing.get(exp_id)!=vid)
-> with_bucketing + _persist_visitor_state. AC1/2/3/4/5/6/8.
Full suite 1077 green 96.87% cov; eval floor 98%; parity 214 (AC9);
ruff+mypy strict clean. AC6 asserts filtered to state envelope (dedup marker
is orthogonal tracking artifact). set_attributes/set_segments untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oseout

Beads: ai-driven-product-dev-rudj
AC7: shared DataStore across two Cores -> sticky via hydration (zero hash on
Core B); negative control: unshared per-instance stores do NOT share. No src
change (PY-1..PY-3 mechanism already complete). AC9: parity 214 green.
Full suite 1079 green 96.87% cov; eval floor 98%; mypy strict (core +
serving_config_contract) + pyi-header gate clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-g4pc
6 new tests (AC11 conversion attribution x2, AC12 feature x2, AC13 diagnostics x2)
fail w/ substantive assertion mismatches (sibling paths re-hash fresh); 9 pre-existing pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-g4pc
Thread self._state.bucketing (pure read-only) through every variation-resolution
path: run_feature/run_features (features.py), conversion attribution
(conversions.py _compute_bucketing_assignments + create_conversion, tracker.py
track), diagnose_experience/diagnose_feature (context.py). Read-only consumers
mint no decision; write stays experience-run-only. Fixed stale conversions.py
docstring. AC11 (attribution sticky both paths)/AC12 (feature sticky)/AC13
(diagnostics sticky).
Full suite 1085 green 96.87% cov; eval floor 98%; parity 214 (AC9); mypy strict
+ serving contract + pyi-header clean. Hash contract + fixtures untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tr-coerce rehydration

Beads: ai-driven-product-dev-93nf
Gemini code-assist (PR #54): bucketing param + _hydrate_visitor_state return +
stored_bucketing local -> Optional[Mapping[str,str]]; rehydration coerces keys+
values to str (defensive vs a DataStore persisting non-string IDs). +1 test.
Type-only + one coercion; no write-path/behavior change. mypy strict clean,
full suite 1086 green, parity 214.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
abbaseya and others added 8 commits July 15, 2026 22:27
Beads: ai-driven-product-dev-rloj
Agent: sdk-python
Additive/inert: RULE_TYPE_BUCKETED_INTO_EXPERIENCE_KEY + log_mutual_exclusion_target_not_found; no behavior change (AC7).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…data only)

Beads: ai-driven-product-dev-xt69
Agent: sdk-python
NamedTuple 8-row table (verbatim spec values) + build_mutual_exclusion_config(); 5 genuine sanity invariants.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-nz93
Agent: sdk-python
20 failed / 1 passed (default-ANY regression guard). Genuine RED via get_experience_by_key spy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…udiences + bucketed_into_experience_key

Beads: ai-driven-product-dev-nz93
Agent: sdk-python
Audience-level exclusion resolver (read-only, presence-only, negation-last, AC8 warn) + matching_options
ALL/ANY (default ANY). is_rule_matched + generic walk byte-identical (AC7 proven). 21 new pass, full 1114,
cov 96.71%, eval floor 97%, parity 214.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Beads: ai-driven-product-dev-b3sb
Agent: sdk-python
15 tests: 3 generic match_types (equals/contains/less) bit-identical; matching_options ALL/ANY boundary
(default-ANY == pre-seam); mixed-tree fails-closed guard. src untouched. Full 1129, cov 96.76%, eval 97%.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(preview): experiment preview — debug_token + set_preview (qs-02)
feat: sticky-bucketing persistence (qs-03) — per-visitor decision map + full read-back parity
feat: mutual-exclusion audience rule bucketed_into_experience_key (qs-04)
@abbaseya abbaseya self-assigned this Jul 16, 2026
@abbaseya
abbaseya requested a review from usmanabbas7 July 16, 2026 16:27

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements several key features for the Convert Python SDK, including the anchored bucketing layout (contract v12), experiment preview capabilities with zero-trace suppression and process-wide memoization, sticky-bucketing persistence via a 3-key envelope, and mutual-exclusion audience rules (bucketed_into_experience_key). It also includes comprehensive integration and parity tests. Feedback on the changes highlights a robustness issue in the _sole_rule_item helper, which lacks defensive type-checking for nested configuration structures and could crash on malformed inputs; implementing the suggested type-validation block will ensure the SDK fails closed gracefully.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/convert_sdk/evaluation/rules.py
Comment thread tests/integration/test_context_preview_zero_trace.py
Comment thread tests/test_config_by_experience_memo.py
Comment thread src/convert_sdk/ports/transport.py
Comment thread tests/integration/test_context_preview_zero_trace.py
Comment thread tests/integration/test_context_preview_zero_trace.py
Comment thread tests/integration/test_context_preview_zero_trace.py
Comment thread tests/integration/test_context_preview_zero_trace.py
Comment thread tests/integration/test_context_preview_zero_trace.py
Comment thread tests/integration/test_context_preview_zero_trace.py
Comment thread tests/integration/test_context_set_preview.py
@abbaseya abbaseya changed the title Full Stack v12: safe traffic ramp-up, variation preview & experiment exclusion feat: Full Stack v12 — safe traffic ramp-up, variation preview & experiment exclusion Jul 16, 2026
@abbaseya abbaseya closed this Jul 16, 2026
@abbaseya abbaseya reopened this Jul 16, 2026
Wire qs-02 preview into demo/run_demo.py via a new CONVERT_DEMO_PREVIEW
env var ("{experienceId}.{variationId}"): parses with parse_preview_param,
calls context.set_preview, and prints the forced decision. Adds a dedicated
paused (non-running) preview-target experience to config_fixture.json with
numeric ids (900210001 / 900210101 / 900210102) so a real convert_preview
link parses and the demo proves the two headline guarantees — a paused
experience's variation is force-rendered (bypassing status + bucketing),
and the preview context is zero-trace (track_conversion with
force_multiple=True still yields tracked=False, distinguishing real
suppression from ordinary dedup). A different experience on the same
context is unaffected; malformed/unresolvable ids degrade to the AC7
inert-with-warning path.

demo/README.md gains a "Testing preview links" section and .env.example
documents the new vars. 5 TestPreviewFixture tests added. demo/ + fixture
+ docs + tests only — no SDK source change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@usmanabbas7

Copy link
Copy Markdown
Collaborator

Final release review — sign-off ✅ (with one non-blocking follow-up)

Reviewed the consolidated branch first-hand in an isolated worktree at 0d4dd38, focusing on how the three capabilities compose (each PR was already approved in isolation; the interactions across the overlapping files — bucketing.py, experiences.py, context.py, rules.py — were not).

Verdict: approve for merge.

Quality gates (CI-pinned deps)

Check Result
Full test suite 1,134 passed
Ruff (E/W/F/B/SIM/RUF) clean
mypy --strict (shippable package) clean, 47 files
Coverage 97% total; bucketing 100%, experiences 99%, rules 95%
Merge integrity clean linear stack anchored → preview → sticky → exclusion; no conflict markers
JS anchored-bucketing parity (72 vectors via real select_experience seam) pass

Capability behavior (empirically probed, not assumed)

  • Safe Traffic Ramp-Up — stopping an arm in a 3-way split leaves the other arms' visitors exactly in place (A→A, C→C, stopped→idle), distribution byte-identical pre/post. The old packed v11 path reshuffles the same scenario. The version > 11 gate routes legacy configs to the old path and v12 to the safe one. ✔
  • Variation Preview — zero-trace suppression is comprehensive and per-context: bucketing events, conversions (track_conversion short-circuit), and visitor-state persistence all suppressed; forced decision bypasses qualification/bucketing/exclusion by construction; a concurrent non-preview context sharing the tracker/store is unaffected. ✔
  • Experiment Exclusion — works on the default path; qualification checked before sticky read-back (can't resurrect an unqualified visitor); read-only against stored bucketing state. ✔

One finding — non-blocking, recommend follow-up

run_experiences(enable_storage=False) silently disables mutual exclusion — returns ['exp-a', 'exp-b'] where the default returns ['exp-a']. Cause: exclusion reads the accumulated self._state.bucketing map, but that map is only populated by the write-after-hash in _evaluate_and_record, which is gated by the same enable_storage flag. enable_tracking=False is unaffected.

  • Severity: low-medium. Default path is safe; the break needs an explicit opt-out of persistence while relying on exclusion in the same call, and cross-request exclusion requires a DataStore anyway.
  • Not covered by tests — integration tests only exercise two sequential singular run_experience() calls with default storage.
  • Parity note: the port cites parity with JS BucketingAttributes.enableStorage, but that field isn't present in the JS repo on main/fetched branches; JS _retrieveBucketing writes the in-memory bucketing map unconditionally on a fresh bucket (separating in-request accumulation from persistence in a way this port conflates). Could not verify the exact enableStorage branch semantics against the JS oracle.

Suggested follow-up (post-merge): add a test for run_experiences(enable_storage=False) + exclusion, and confirm intended semantics against JS — likely fix is to decouple the in-request exclusion-input map from the persistence flag.

🤖 Generated with Claude Code

@abbaseya
abbaseya merged commit 1f4bd82 into main Jul 24, 2026
27 checks passed
@abbaseya
abbaseya deleted the feat/fullstack-v12 branch July 24, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants