epic-3/story-5: Establish JavaScript parity fixtures for state and evaluation behavior by usmanabbas7 · Pull Request #37 · convertcom/python-sdk

usmanabbas7 · 2026-06-08T09:28:04Z

Story 3.5 — JavaScript Parity Fixtures (NFR20, FR30)

Formalizes and completes the cross-SDK parity validation harness. Test-and-tooling only — zero src/convert_sdk/ production changes (git diff src/ is empty). Part of sprint sprint/2026-04-06-convert-python-sdk. This is the last story in epic-3.

What was built

scripts/generate_parity_fixtures.py — Python generator (single documented entry point, outside src/, no runtime dep added) that machine-derives ALL four fixture families from the sibling ../javascript-sdk. It spawns dependency-free Node subprocess helpers under scripts/js_reference/ (byte-faithful ports of the JS bucketing/rules/comparisons/data-manager/segments source) — chosen because the sibling JS SDK has no installed node_modules/dist. Deterministic: --check passes clean. The superseded generate_bucketing_vectors.js bootstrap was removed.
Fixtures under tests/parity/fixtures/ (external checked-in JSON, never hand-authored, every file carrying generated_from with js_sdk_commit: 34f0a7a4): confirmed/unified bucketing_vectors.json; NEW rule_vectors.json, feature_vectors.json, state_vectors.json.
Parity test modules under tests/parity/ (parametrized, diagnostic, offline, JS-runtime-free at test time): NEW test_js_rule_parity.py, test_js_feature_parity.py, test_js_state_parity.py; conftest loaders for each fixture family. Confirmed existing test_js_bucketing_parity.py.
tests/parity/README.md — maintainer regeneration workflow, failure-reading guide, generated_from provenance convention; links tracking-payload parity (NFR21/Story 2.2) and CI ownership (Story 5.1) without authoring them.

Scope discipline

No production changes; if a vector fails it's a drift signal owned by the offending evaluation story (1.4/1.5/3.3/3.4), not a license to rewrite. State vectors encode the Story-3.4 None/empty no-match form (NOT the FR50 typed-reason object — that's Story 4.2). Descoped (not authored): tracking_payloads.json (Story 2.2), CI YAML (Story 5.1), FR50 taxonomy (Story 4.2).
F-016 (audit): Story 1.4's File List did not confirm parity files shipped — driver verified the bucketing baseline DOES exist on-branch, so it was confirmed/extended, not duplicated.

Tests — zero regressions

494 → 543 (+49 parity tests). Parity suite = 123 (bucketing 72 + 2 contract, rule 27, feature 4, state 18). Full prior suite stayed green (NFR22). pytest tests/parity/ is runnable standalone for the NFR20 release gate.

Beads

Epic ai-driven-product-dev-m3wi; tasks -pd1e (PAR-1 baseline+loaders), -xual (PAR-2 generator+Node ports), -vmiu (PAR-3 fixtures), -v6sh (PAR-4 parity tests), -b85x (PAR-5 README+e2e) — all closed.

Readiness & review

Readiness PASS 9.0/10 (round 2; round 1 7.5/10 on generator drive-strategy ambiguity), 4 auto-delegations in sprint mode — see conductor's assessment report. Code review clean round 1, one non-blocking note (feature fixtures use a single 100%-traffic variation; bucketing-distribution parity is covered by the 72-vector bucketing suite).

Stacked on epic-3/story-4 (#36 → #35 → #34 → #33 chain).

🤖 Generated with Claude Code

gemini-code-assist

Code Review

This pull request introduces a comprehensive cross-SDK parity validation harness to ensure the Python SDK behaves identically to the JavaScript reference implementation. It adds a Python generator script to drive dependency-free Node helper scripts for bucketing, rules, features, and state/entity lookup, alongside corresponding parity tests and documentation. The review feedback points out a redundancy where session-scoped fixtures are defined and passed to tests but never used, as the tests load the JSON vectors directly at the module level for parametrization. Removing these unused fixtures and assertions will prevent redundant JSON parsing and simplify the test setup.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-08T09:31:02Z

+@pytest.mark.parametrize("vector", _VECTORS, ids=[v["id"] for v in _VECTORS])
+def test_rule_evaluation_matches_js_reference(vector, rule_vectors):
+    """Python ``is_rule_matched`` must equal the JS ``RuleManager`` reference."""
+    # The conftest loader is consumed so no test inlines fixture data; the
+    # module-level _VECTORS only drives parametrize ids/cases.
+    assert isinstance(rule_vectors, list) and rule_vectors
+
+    result = is_rule_matched(vector["data"], vector["rule"])
+    assert result == vector["expected"], (
+        f"rule parity divergence in {_FIXTURE} [{vector['id']}]: "
+        f"data={vector['data']!r} -> python={result!r} != js={vector['expected']!r}"
+    )


The rule_vectors fixture is passed as an argument and asserted to be a list, but its data is never actually used in the test. Instead, the test evaluates vector, which is loaded directly from the JSON file at the module level for parametrization.

This results in the JSON file being read and parsed twice (once at module import time and once via the session-scoped fixture), which is redundant and inefficient. Since module-level loading is required for @pytest.mark.parametrize, the rule_vectors fixture should be removed from the test signature and the redundant assertion deleted.

Suggested change

@pytest.mark.parametrize("vector", _VECTORS, ids=[v["id"] for v in _VECTORS])

def test_rule_evaluation_matches_js_reference(vector, rule_vectors):

"""Python ``is_rule_matched`` must equal the JS ``RuleManager`` reference."""

# The conftest loader is consumed so no test inlines fixture data; the

# module-level _VECTORS only drives parametrize ids/cases.

assert isinstance(rule_vectors, list) and rule_vectors

result = is_rule_matched(vector["data"], vector["rule"])

assert result == vector["expected"], (

f"rule parity divergence in {_FIXTURE} [{vector['id']}]: "

f"data={vector['data']!r} -> python={result!r} != js={vector['expected']!r}"

)

@pytest.mark.parametrize("vector", _VECTORS, ids=[v["id"] for v in _VECTORS])

def test_rule_evaluation_matches_js_reference(vector):

"""Python is_rule_matched must equal the JS RuleManager reference."""

result = is_rule_matched(vector["data"], vector["rule"])

assert result == vector["expected"], (

f"rule parity divergence in {_FIXTURE} [{vector['id']}]: "

f"data={vector['data']!r} -> python={result!r} != js={vector['expected']!r}"

)

gemini-code-assist · 2026-06-08T09:31:02Z

+@pytest.mark.parametrize("vector", _VECTORS, ids=[v["id"] for v in _VECTORS])
+def test_feature_resolution_matches_js_reference(vector, feature_vectors):
+    """Python ``resolve_feature`` must equal the JS feature-resolution reference."""
+    assert isinstance(feature_vectors, list) and feature_vectors
+
+    snapshot = load_snapshot(vector["config"])
+    result = resolve_feature(
+        vector["feature_key"],
+        snapshot,
+        visitor_id=vector["visitor_id"],
+    )
+    actual = _result_to_comparable(result)
+    assert actual == vector["expected"], (
+        f"feature parity divergence in {_FIXTURE} [{vector['id']}]: "
+        f"feature={vector['feature_key']!r} visitor={vector['visitor_id']!r} -> "
+        f"python={actual!r} != js={vector['expected']!r}"
+    )


The feature_vectors fixture is passed as an argument and asserted to be a list, but its data is never actually used in the test. Instead, the test evaluates vector, which is loaded directly from the JSON file at the module level for parametrization.

This results in the JSON file being read and parsed twice (once at module import time and once via the session-scoped fixture), which is redundant and inefficient. Since module-level loading is required for @pytest.mark.parametrize, the feature_vectors fixture should be removed from the test signature and the redundant assertion deleted.

@pytest.mark.parametrize("vector", _VECTORS, ids=[v["id"] for v in _VECTORS]) def test_feature_resolution_matches_js_reference(vector): """Python resolve_feature must equal the JS feature-resolution reference.""" snapshot = load_snapshot(vector["config"]) result = resolve_feature( vector["feature_key"], snapshot, visitor_id=vector["visitor_id"], ) actual = _result_to_comparable(result) assert actual == vector["expected"], ( f"feature parity divergence in {_FIXTURE} [{vector['id']}]: " f"feature={vector['feature_key']!r} visitor={vector['visitor_id']!r} -> " f"python={actual!r} != js={vector['expected']!r}" )

gemini-code-assist · 2026-06-08T09:31:02Z

+@pytest.mark.parametrize("vector", _VECTORS, ids=[v["id"] for v in _VECTORS])
+def test_state_and_entity_lookup_matches_js_reference(vector, state_vectors):
+    """Python state/entity-lookup + segment output must equal the JS reference."""
+    assert isinstance(state_vectors, list) and state_vectors
+
+    actual = _run_operation(vector)
+    assert actual == vector["expected"], (
+        f"state parity divergence in {_FIXTURE} [{vector['id']}]: "
+        f"op={vector['operation']!r} -> python={actual!r} != js={vector['expected']!r}"
+    )


The state_vectors fixture is passed as an argument and asserted to be a list, but its data is never actually used in the test. Instead, the test evaluates vector, which is loaded directly from the JSON file at the module level for parametrization.

This results in the JSON file being read and parsed twice (once at module import time and once via the session-scoped fixture), which is redundant and inefficient. Since module-level loading is required for @pytest.mark.parametrize, the state_vectors fixture should be removed from the test signature and the redundant assertion deleted.

@pytest.mark.parametrize("vector", _VECTORS, ids=[v["id"] for v in _VECTORS]) def test_state_and_entity_lookup_matches_js_reference(vector): """Python state/entity-lookup + segment output must equal the JS reference.""" actual = _run_operation(vector) assert actual == vector["expected"], ( f"state parity divergence in {_FIXTURE} [{vector['id']}]: " f"op={vector['operation']!r} -> python={actual!r} != js={vector['expected']!r}" )

gemini-code-assist · 2026-06-08T09:31:02Z

+@pytest.fixture(scope="session")
+def rule_vectors() -> list:
+    """Golden rule-evaluation vectors derived from the JS ``RuleManager`` reference.
+
+    Each entry pairs a ``data`` mapping + a ``rule`` set with the JS reference
+    ``expected`` boolean, exercised through the Python ``is_rule_matched`` surface
+    (Story 1.4 ``evaluation/rules.py``).
+    """
+    return _load("rule_vectors.json")["vectors"]
+
+
+@pytest.fixture(scope="session")
+def feature_vectors() -> list:
+    """Golden feature-resolution vectors derived from the JS feature path.
+
+    Each entry pairs a config + visitor inputs with the JS reference expected
+    resolution (status + cast variables, or a ``None`` miss), exercised through
+    the Python ``resolve_feature`` surface (Story 1.5/1.6 ``evaluation/features.py``).
+    """
+    return _load("feature_vectors.json")["vectors"]
+
+
+@pytest.fixture(scope="session")
+def state_vectors() -> list:
+    """Golden Epic-3 state / entity-lookup + segment vectors from the JS reference.
+
+    Each entry pairs a config + a lookup/segment operation with the JS reference
+    expected result, mirroring the ``DataManager.getEntity``/``getEntityById``/
+    ``getEntities`` and ``SegmentsManager`` surfaces. Encodes the Story-3.4
+    ``null`` -> ``None``/empty no-match contract (NOT the FR50 typed reason).
+    """
+    return _load("state_vectors.json")["vectors"]


Since the new test files (test_js_rule_parity.py, test_js_feature_parity.py, and test_js_state_parity.py) load their vectors directly at the module level for @pytest.mark.parametrize, these session-scoped fixtures are completely unused and redundant. Removing them simplifies the test setup and avoids parsing the JSON files twice.

…eature/state conftest loaders Beads: ai-driven-product-dev-pd1e Agent: fullstack-sdk-dev Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…cy-free Node reference ports Beads: ai-driven-product-dev-xual Agent: fullstack-sdk-dev scripts/generate_parity_fixtures.py is the single documented entry point; spawns faithful Node ports (scripts/js_reference/*) of the JS bucketing/rules/data/segments reference, emits all 4 fixture families with per-file generated_from, exits non-zero on helper failure. Maintainer tooling outside src/ — no runtime dependency added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…unified bucketing provenance Beads: ai-driven-product-dev-vmiu Agent: fullstack-sdk-dev rule_vectors.json (27), feature_vectors.json (4), state_vectors.json (18), and bucketing_vectors.json provenance unified — each with top-level generated_from naming js_sdk_commit 34f0a7a. State vectors encode the Story-3.4 None/empty no-match contract (skip-unknown multi), NOT the deferred FR50 typed reason. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…stic, offline) Beads: ai-driven-product-dev-v6sh Agent: fullstack-sdk-dev Parametrized over the checked-in fixtures via conftest loaders; exercise the REAL Python surfaces (is_rule_matched, resolve_feature, Context.get_config_entity*, select_custom_segments). Diagnostic asserts name fixture/entry/expected-JS/actual. Offline + JS-runtime-free. 49 new parity tests pass byte/value-exact (123 total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…n, full suite green Beads: ai-driven-product-dev-b85x Agent: fullstack-sdk-dev tests/parity/README.md documents run/regenerate/read-failure/provenance and the tracking(2.2)/CI(5.1)/FR50(4.2) descopes. Removed the superseded standalone generate_bucketing_vectors.js — generate_parity_fixtures.py is the single entry point. Forced-drift sanity check confirmed diagnostic messaging (reverted, no perturbed fixture committed). Full suite 543 green, zero src/convert_sdk changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…rovenance) Beads: ai-driven-product-dev-b85x Agent: fullstack-sdk-dev Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… (UTF-8) — F-063 propagation The parity "JS reference" (scripts/js_reference/bucketing.js) re-implemented the hash with charCodeAt & 0xff over UTF-16 code units — co-buggy with the original SDK bug, so it machine-derived a WRONG golden value (用户123@9999 = 531569922) and the suite passed against a wrong oracle. - js_reference/bucketing.js: replace the hand port with the real npm `murmurhash` package (require('murmurhash').v3); keep getBucketValueForVisitor/selectBucket. - js_reference/package.json (new): pin murmurhash@^2.0.1 as the regen-time oracle. - emit_bucketing.js: add café/🎯emoji multi-byte + astral hardening inputs. - generate_parity_fixtures.py: correct the false "dependency-free faithful port" provenance to the real npm oracle (UTF-8 / TextEncoder). - Regenerated all four fixtures from the real oracle: bucketing_vectors.json now 用户123@9999 = 3859151469 (80 vectors); rule/feature/state unchanged for ASCII. Verified: 12 non-ASCII vectors all byte-exact against real npm murmurhash@2.0.1. Parity suite 131 passed; full suite 551 passed; generator --check up to date. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

usmanabbas7 · 2026-06-15T11:28:27Z

F-066 propagation (rebase onto remediated 3-3)

Rebased onto the remediated stack (3-3 f2d8916 → 3-4 → this branch). Does not modify evaluation/segments.py; the latch fix is inherited cleanly (segments.py/test_segments.py byte-identical to remediated 3-3). One already-upstream commit was skipped by rebase (normal cherry-pick dedup), no conflicts.

uv run pytest → 553 passed
CI gate: no .github/workflows/ on this branch → no-ci.

abbaseya · 2026-06-18T16:31:15Z

Superseded — all commits already in main (bc76b64). Closing without merge as part of post-sprint cleanup.

usmanabbas7 self-assigned this Jun 8, 2026

usmanabbas7 requested a review from clllaur June 8, 2026 09:28

gemini-code-assist Bot reviewed Jun 8, 2026

View reviewed changes

This was referenced Jun 8, 2026

epic-4/story-1: Add Production-Safe Diagnostic Logging #38

Closed

epic-4/story-2: Surface Clear Errors and Diagnosable No-Result Outcomes #39

Closed

usmanabbas7 force-pushed the epic-3/story-4-add-entity-lookup-helpers branch from 456e2fc to 9d4e32d Compare June 14, 2026 16:59

usmanabbas7 force-pushed the epic-3/story-5-establish-javascript-parity-fixtures-for-state-and-evaluation-behavior branch from 023cd99 to 9598cad Compare June 14, 2026 17:06

usmanabbas7 force-pushed the epic-3/story-4-add-entity-lookup-helpers branch from 9d4e32d to 663c22b Compare June 15, 2026 11:26

usmanabbas7 and others added 7 commits June 15, 2026 16:27

wf(wf-3p5par): [PAR-1] confirm bucketing parity baseline + add rule/f…

dcdfa75

…eature/state conftest loaders Beads: ai-driven-product-dev-pd1e Agent: fullstack-sdk-dev Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wf(wf-3p5par): [PAR-5] add parity harness README (regen/run/failure/p…

d2ee33a

…rovenance) Beads: ai-driven-product-dev-b85x Agent: fullstack-sdk-dev Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

usmanabbas7 force-pushed the epic-3/story-5-establish-javascript-parity-fixtures-for-state-and-evaluation-behavior branch from 9598cad to 78d7ca6 Compare June 15, 2026 11:27

abbaseya closed this Jun 18, 2026

abbaseya deleted the epic-3/story-5-establish-javascript-parity-fixtures-for-state-and-evaluation-behavior branch June 18, 2026 16:31

Navigation Menu

Uh oh!

epic-3/story-5: Establish JavaScript parity fixtures for state and evaluation behavior - #37

epic-3/story-5: Establish JavaScript parity fixtures for state and evaluation behavior#37
usmanabbas7 wants to merge 7 commits into
epic-3/story-4-add-entity-lookup-helpersfrom
epic-3/story-5-establish-javascript-parity-fixtures-for-state-and-evaluation-behavior

usmanabbas7 commented Jun 8, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jun 8, 2026

Uh oh!

gemini-code-assist Bot Jun 8, 2026

Uh oh!

gemini-code-assist Bot Jun 8, 2026

Uh oh!

gemini-code-assist Bot Jun 8, 2026

Uh oh!

usmanabbas7 commented Jun 15, 2026

Uh oh!

abbaseya commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

usmanabbas7 commented Jun 8, 2026

Story 3.5 — JavaScript Parity Fixtures (NFR20, FR30)

What was built

Scope discipline

Tests — zero regressions

Beads

Readiness & review

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

usmanabbas7 commented Jun 15, 2026

F-066 propagation (rebase onto remediated 3-3)

Uh oh!

abbaseya commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants