epic-1/story-4: Run Local Experience Evaluations by usmanabbas7 · Pull Request #26 · convertcom/python-sdk

usmanabbas7 · 2026-06-07T08:55:35Z

Summary

Implements BMAD story 1.4 (1-4-run-local-experience-evaluations) for the Convert Python SDK — local, network-free experience evaluation against the immutable config snapshot.

Pure-Python MurmurHash3-32 bucketing (evaluation/bucketing.py): murmurhash3_32(value, seed=9999) returning unsigned 32-bit, byte-exact with the npm murmurhash v3 output the JS SDK uses (audit finding F-009 — no mmh3/third-party hashing dep; httpx remains the only runtime dependency). Parity verified against 72 JS-derived vectors in tests/parity/fixtures/bucketing_vectors.json (generated via scripts/generate_bucketing_vectors.js).
Audience/location rule evaluation (evaluation/rules.py): OR/AND/OR_WHEN rule tree + JS comparison operators for current FullStack config shapes.
Typed results + experience selection (domain/results.py, evaluation/experiences.py): typed variation results; normal misses return None/empty collections, never exceptions or raw dicts.
Visitor-scoped Context (context.py, core.create_context): run_experience() / run_experiences() with request-time attribute overlays that never mutate stored visitor state. (Story 1.3's Context was absent on this branch lineage — minimal foundation built here, logged in conductor audit trail.)

Tests: 170 passing (53 baseline + 117 new). Wheel builds clean.

Traceability

Part of sprint sprint/2026-04-06-convert-python-sdk
Stacked on epic-1/story-2: Support sdkKey and Direct-Config Initialization #25 (epic-1/story-2) → Epic 1 Story 1: Scaffold the publishable SDK foundation #24 (epic-1/story-1)
Beads: epic ai-driven-product-dev-cwp1; tasks -lm55 (bucketing+parity), -scbd (rules), -ranl (typed results/selection), -84o7 (Context + run_experience/s) — all closed
Conductor work dir: ai-driven-product-dev/work/2026-06-06-run-local-experience-evaluations/

Review notes

Code review: clean round 1, no warnings.
Readiness gate scored 8.5/10 (PASS); 4 ambiguities auto-delegated in sprint mode (murmurhash byte semantics → JS UTF-16 code-unit ground truth; parity fixture sourcing → machine-generated with metadata; missing Story 1.3 Context → minimal foundation built; MVP rule subset). See readiness-assessment.md in the conductor work dir.

🤖 Generated with Claude Code

Beads: ai-driven-product-dev-lm55 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pure-Python murmurhash3_32 (seed 9999, unsigned 32-bit) byte-exact with npm murmurhash v3 (charCodeAt&0xff UTF-16 semantics). 72 JS-derived parity vectors. Beads: ai-driven-product-dev-lm55 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Beads: ai-driven-product-dev-scbd Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

is_rule_matched (OR/AND/OR_WHEN tree + JS comparison operators) and qualifies() (matchRulesByField MVP policy). Added audience id/key indexes to ConfigSnapshot. Beads: ai-driven-product-dev-scbd Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Beads: ai-driven-product-dev-ranl Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ation (GREEN) ExperienceResult frozen dataclass (read-only variation payload) and select_experience (qualify -> build buckets -> deterministic select), mirroring JS _retrieveBucketing minus storage/tracking. Beads: ai-driven-product-dev-ranl Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Beads: ai-driven-product-dev-84o7 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…EEN) Minimal visitor-scoped Context (immutable stored attributes, ephemeral request-time overlay), Core.create_context, run_experience/run_experiences (local-only, no network/tracking). ExperienceResult exported additively. Beads: ai-driven-product-dev-84o7 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

gemini-code-assist

Code Review

This pull request implements local experience evaluation (Story 1.4) for the Convert Python SDK, introducing visitor-scoped evaluation contexts, a pure-Python MurmurHash3-32 implementation with JS parity, and nested rule qualification. The reviewer's feedback focuses on enhancing robustness and defensive programming, specifically by ensuring consistent case-insensitivity in comparisons, wrapping comparison operators in try-except blocks to prevent TypeErrors, and adding type guards to handle potentially malformed configurations safely.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-07T08:57:35Z

+def _equals(value: Any, test_against: Any) -> bool:
+    if isinstance(value, (list, tuple)):
+        return test_against in value
+    if isinstance(value, Mapping):
+        return str(test_against) in {str(k) for k in value.keys()}
+    return str(value).lower() == str(test_against).lower()


The _equals operator is case-insensitive for scalar string comparisons, but performs case-sensitive comparisons when checking elements within a list/tuple or keys within a mapping. To ensure consistent evaluation behavior, case-insensitivity should be applied uniformly across all supported data structures.

def _equals(value: Any, test_against: Any) -> bool:\n if isinstance(value, (list, tuple)):\n return str(test_against).lower() in {str(item).lower() for item in value}\n if isinstance(value, Mapping):\n return str(test_against).lower() in {str(k).lower() for k in value.keys()}\n return str(value).lower() == str(test_against).lower()

References

Ensure consistent behavior and case-insensitivity across all data structures during comparison operations.

gemini-code-assist · 2026-06-07T08:57:35Z

+def _less(value: Any, test_against: Any) -> bool:
+    v = _to_number(value) if _is_numeric(value) else value
+    t = _to_number(test_against) if _is_numeric(test_against) else test_against
+    if type(v) is not type(t):
+        return False
+    return v < t


Comparing non-comparable types of the same class (such as None < None or dictionary comparisons) will raise a TypeError in Python 3. To fulfill the contract of never raising exceptions during rule evaluation, wrap the comparison in a try...except block to safely return False on incompatible types.

def _less(value: Any, test_against: Any) -> bool:\n try:\n v = _to_number(value) if _is_numeric(value) else value\n t = _to_number(test_against) if _is_numeric(test_against) else test_against\n if type(v) is not type(t):\n return False\n return v < t\n except (TypeError, ValueError):\n return False

References

Defensive programming: wrap operations that can raise TypeError or ValueError in try-except blocks to prevent runtime crashes.

gemini-code-assist · 2026-06-07T08:57:35Z

+def _less_equal(value: Any, test_against: Any) -> bool:
+    v = _to_number(value) if _is_numeric(value) else value
+    t = _to_number(test_against) if _is_numeric(test_against) else test_against
+    if type(v) is not type(t):
+        return False
+    return v <= t


Comparing non-comparable types of the same class (such as None <= None or dictionary comparisons) will raise a TypeError in Python 3. Wrap the comparison in a try...except block to safely return False on incompatible types.

def _less_equal(value: Any, test_against: Any) -> bool:\n try:\n v = _to_number(value) if _is_numeric(value) else value\n t = _to_number(test_against) if _is_numeric(test_against) else test_against\n if type(v) is not type(t):\n return False\n return v <= t\n except (TypeError, ValueError):\n return False

References

Defensive programming: wrap operations that can raise TypeError or ValueError in try-except blocks to prevent runtime crashes.

gemini-code-assist · 2026-06-07T08:57:35Z

+def _process_rule_item(data: Optional[Mapping[str, Any]], rule: Mapping[str, Any]) -> bool:
+    """Evaluate a single rule item against a key-value ``data`` mapping."""
+    matching = rule.get("matching") or {}


If the rule parameter is malformed and passed as a non-mapping type (e.g., a list or None), calling rule.get will raise an AttributeError. Adding an explicit isinstance check ensures robust handling of malformed configurations.

def _process_rule_item(data: Optional[Mapping[str, Any]], rule: Mapping[str, Any]) -> bool:\n \"\"\"Evaluate a single rule item against a key-value ``data`` mapping.\"\"\"\n if not isinstance(rule, Mapping):\n return False\n matching = rule.get(\"matching\") or {}

References

Defensive programming: ensure appropriate guards exist before object property accesses or operations on potentially nullable/malformed inputs to prevent runtime crashes.

gemini-code-assist · 2026-06-07T08:57:35Z

+def _process_or_when(data: Optional[Mapping[str, Any]], rules_subset: Mapping[str, Any]) -> bool:
+    """OR_WHEN: any rule item true -> true."""
+    items: Sequence[Mapping[str, Any]] = rules_subset.get("OR_WHEN") or []


If rules_subset is malformed and passed as a non-mapping type, calling rules_subset.get will raise an AttributeError. Adding an explicit isinstance check ensures robust handling of malformed configurations.

def _process_or_when(data: Optional[Mapping[str, Any]], rules_subset: Mapping[str, Any]) -> bool:\n \"\"\"OR_WHEN: any rule item true -> true.\"\"\"\n if not isinstance(rules_subset, Mapping):\n return False\n items: Sequence[Mapping[str, Any]] = rules_subset.get(\"OR_WHEN\") or []

References

Defensive programming: ensure appropriate guards exist before object property accesses or operations on potentially nullable/malformed inputs to prevent runtime crashes.

gemini-code-assist · 2026-06-07T08:57:35Z

+def _process_and(data: Optional[Mapping[str, Any]], rules_subset: Mapping[str, Any]) -> bool:
+    """AND: every OR_WHEN block must be true."""
+    blocks: Sequence[Mapping[str, Any]] = rules_subset.get("AND") or []


If rules_subset is malformed and passed as a non-mapping type, calling rules_subset.get will raise an AttributeError. Adding an explicit isinstance check ensures robust handling of malformed configurations.

def _process_and(data: Optional[Mapping[str, Any]], rules_subset: Mapping[str, Any]) -> bool:\n \"\"\"AND: every OR_WHEN block must be true.\"\"\"\n if not isinstance(rules_subset, Mapping):\n return False\n blocks: Sequence[Mapping[str, Any]] = rules_subset.get(\"AND\") or []

References

Defensive programming: ensure appropriate guards exist before object property accesses or operations on potentially nullable/malformed inputs to prevent runtime crashes.

gemini-code-assist · 2026-06-07T08:57:36Z

+    for variation in experience.get("variations", []) or []:
+        if not _is_running(variation):


If the variations list contains malformed non-mapping elements, calling _is_running(variation) will raise an AttributeError. Adding an explicit isinstance check ensures robust handling of malformed configurations.

for variation in experience.get(\"variations\", []) or []:\n if not isinstance(variation, Mapping):\n continue\n if not _is_running(variation):

References

Defensive programming: ensure appropriate guards exist before object property accesses or operations on potentially nullable/malformed inputs to prevent runtime crashes.

gemini-code-assist · 2026-06-07T08:57:36Z

+    for variation in experience.get("variations", []) or []:
+        if str(variation.get("id")) == str(variation_id):


If the variations list contains malformed non-mapping elements, calling variation.get will raise an AttributeError. Adding an explicit isinstance check ensures robust handling of malformed configurations.

for variation in experience.get(\"variations\", []) or []:\n if not isinstance(variation, Mapping):\n continue\n if str(variation.get(\"id\")) == str(variation_id):

References

Defensive programming: ensure appropriate guards exist before object property accesses or operations on potentially nullable/malformed inputs to prevent runtime crashes.

…P parity — F-063 The pure-Python MurmurHash3-32 hashed UTF-16 code units (charCodeAt & 0xff) and mixed in the code-unit count, diverging from the JS SDK (npm `murmurhash` v3 → `new TextEncoder().encode()`, UTF-8) and PHP (`unpack('C*')`, UTF-8) for every non-ASCII visitor id — so non-ASCII visitors bucketed to a DIFFERENT variation in Python than in JS/PHP. - bucketing.py: hash value.encode("utf-8"); mix in UTF-8 byte length; delete _utf16_code_units and the false "UTF-16 byte-exact" docstring claim. - scripts/generate_bucketing_vectors.js: replace the co-buggy charCodeAt hand port with the REAL npm `murmurhash` package as the oracle; add café/🎯emoji hardening inputs; fix the false UTF-16 provenance metadata. - tests/parity/fixtures/bucketing_vectors.json: regenerated from the real npm oracle (用户123@9999 531569922 → 3859151469; +8 multi-byte/astral vectors). - tests/test_bucketing.py: correct the co-buggy 用户123 unit assertion to the real-oracle value. Verified: murmurhash3_32("用户123",9999)==3859151469 (real npm) and would fail the old UTF-16 531569922. Parity suite 82 passed; full suite 178 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

abbaseya · 2026-06-18T16:31:55Z

Superseded — all commits already in main (bc76b64). Closing without merge as part of post-sprint cleanup.

usmanabbas7 and others added 8 commits June 6, 2026 20:16

wf(s14eval): [SDK-1] bucketing + parity vectors — tests (RED)

e8a248d

Beads: ai-driven-product-dev-lm55 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wf(s14eval): [SDK-2] audience/location rules — tests (RED)

909451e

Beads: ai-driven-product-dev-scbd Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wf(s14eval): [SDK-3] typed results + experience selection — tests (RED)

8cb6945

Beads: ai-driven-product-dev-ranl Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wf(s14eval): [SDK-4] Context + run_experience(s) — tests (RED)

8f61a70

Beads: ai-driven-product-dev-84o7 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

usmanabbas7 self-assigned this Jun 7, 2026

usmanabbas7 requested a review from clllaur June 7, 2026 08:55

gemini-code-assist Bot reviewed Jun 7, 2026

View reviewed changes

abbaseya closed this Jun 18, 2026

abbaseya deleted the epic-1/story-4-run-local-experience-evaluations branch June 18, 2026 16:31

Navigation Menu

Uh oh!

epic-1/story-4: Run Local Experience Evaluations - #26

epic-1/story-4: Run Local Experience Evaluations#26
usmanabbas7 wants to merge 9 commits into
epic-1/story-2-support-sdkkey-and-direct-config-initializationfrom
epic-1/story-4-run-local-experience-evaluations

usmanabbas7 commented Jun 7, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Uh oh!

abbaseya commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		for variation in experience.get("variations", []) or []:
		if not _is_running(variation):

		for variation in experience.get("variations", []) or []:
		if str(variation.get("id")) == str(variation_id):

Uh oh!

Conversation

usmanabbas7 commented Jun 7, 2026

Summary

Traceability

Review notes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

abbaseya commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants