Skip to content

feat(python-sdk): add in-memory config evaluation managers - #2

Merged
usmanabbas7 merged 1 commit into
mainfrom
codex/python-sdk-evaluation-layer
Mar 26, 2026
Merged

feat(python-sdk): add in-memory config evaluation managers#2
usmanabbas7 merged 1 commit into
mainfrom
codex/python-sdk-evaluation-layer

Conversation

@usmanabbas7

Copy link
Copy Markdown
Collaborator

Summary

This PR adds the in-memory evaluation layer for the Python SDK on top of the deterministic core from PR1. It ports the JS SDK’s static-config evaluation flow so Python code can resolve experiences, features, and segments from loaded config data without introducing the public SDK entrypoints or HTTP layer yet.

What’s Included

  • Added DataManager for:

    • config storage and validation
    • entity and sub-entity lookup helpers
    • in-memory visitor bucketing storage
    • optional datastore handoff points
    • matched experience and variation resolution
  • Added DataStoreManager as a thin datastore adapter abstraction

  • Added SegmentsManager for:

    • storing/retrieving visitor segments
    • selecting custom segments using the existing rule engine
  • Added ExperienceManager for:

    • listing and fetching experiences by key/id
    • selecting bucketed variations for a visitor
    • returning all matched variations across experiences
  • Added FeatureManager for:

    • listing and fetching features by key/id
    • resolving enabled/disabled features from bucketed variations
    • returning feature variables
    • type casting feature variable values
  • Added type-casting utilities for feature variables

  • Added shared config fixtures and manager-level unit tests for:

    • data manager behavior
    • segment selection
    • experience selection
    • feature resolution

Verification

  • Ran the Python test suite successfully:
    • .venv/bin/pytest -q
    • 56 passed

Out of Scope

This PR intentionally does not include:

  • ConvertSDK
  • Context
  • config fetching from CDN
  • tracking/event queue
  • conversion API calls
  • background refresh behavior

Those will come in the next PRs once the in-memory evaluation layer is locked down.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes a foundational in-memory evaluation layer for the Python SDK. It enables the SDK to efficiently resolve experiences, features, and segments directly from loaded configuration data, mirroring the capabilities of its JavaScript counterpart. This enhancement is crucial for building out the SDK's core functionality, providing a robust mechanism for deterministic evaluation without immediate reliance on external network calls or public SDK entrypoints.

Highlights

  • In-Memory Evaluation Layer: Introduced a new in-memory evaluation layer for the Python SDK, porting the static-config evaluation flow from the JS SDK.
  • New Manager Classes: Added several new manager classes: DataManager, DataStoreManager, SegmentsManager, ExperienceManager, and FeatureManager, each responsible for specific aspects of configuration and evaluation.
  • Type-Casting Utilities: Implemented utilities for type-casting feature variables to ensure correct data handling.
  • Comprehensive Testing: Included shared configuration fixtures and extensive unit tests for the new managers, covering data management, segment selection, experience selection, and feature resolution.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@usmanabbas7
usmanabbas7 merged commit 9778742 into main Mar 26, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a significant refactoring and expansion of the SDK's core logic, adding new managers for data, experiences, features, and segments, along with utility functions for object merging and type casting. It also includes comprehensive unit tests and a new fixture configuration. However, several critical issues and improvement opportunities have been identified: the test_config.json fixture contains invalid JSON, which will cause parsing errors; the DataManager.is_valid_config_data method has a logical bug, incorrectly validating configurations with an "error" key; in DataManager.match_rules_by_field, segments are incorrectly retrieved using the audiences field; FeatureManager.run_features silently drops bucketing errors; FeatureManager.run_feature and run_feature_by_id have inconsistent return types; the cast_type function in type_utils.py has ambiguous behavior when casting invalid JSON; the config parameter is passed to several manager constructors but is immediately deleted or only partially used, indicating a potential API design inconsistency; and the _retrieve_bucketing and _get_bucketing_by_field methods in DataManager also have inconsistent return types.

Comment thread tests/unit/conftest.py
@pytest.fixture
def config():
fixture_path = Path(__file__).resolve().parents[1] / "fixtures" / "test_config.json"
return json.loads(fixture_path.read_text())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The config fixture attempts to load test_config.json. Due to the invalid JSON structure in test_config.json (missing brace for a feature variable, see comment in test_config.json), this line will raise a json.JSONDecodeError, causing all tests relying on this fixture to fail.

"type": "float"
},
{
"key": "link",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The test_config.json file contains invalid JSON. Specifically, there is a missing opening curly brace { for a dictionary within the features array, making the entry for the "link" variable malformed. This will cause parsing errors when the configuration is loaded.

          {
            "key": "link",
            "type": "string"
          },

Comment on lines +43 to +44
data.get("error")
or (data.get("account_id") and (data.get("project") or {}).get("id"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The is_valid_config_data method incorrectly considers a configuration valid if it contains an "error" key with a truthy value. A configuration with an "error" key typically indicates an invalid state, not a valid one. The logic should likely check for the absence of an error.

            not data.get("error")
            and data.get("account_id")
            and (data.get("project") or {}).get("id")

else:
audiences_matched = not audiences

segments = self.get_items_by_ids(experience.get("audiences") or [], "segments")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In match_rules_by_field, the line segments = self.get_items_by_ids(experience.get("audiences") or [], "segments") appears to be a logical error. It attempts to retrieve segments using the audiences field of the experience object. Audiences and segments are distinct entities, and experience objects typically have an audiences field, but not a segments field that would contain segment IDs. This will likely result in incorrect segment matching.

Comment on lines +139 to +140
if isinstance(variation, dict):
bucketed_variations.append(variation)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The run_features method silently drops potential errors returned by _data_manager.get_bucketing. If get_bucketing returns a BucketingError or RuleError enum member (as it's designed to do), these errors are not propagated or handled, leading to features not being evaluated without any clear indication of why. This can make debugging difficult.

Suggested change
if isinstance(variation, dict):
bucketed_variations.append(variation)
variation = self._data_manager.get_bucketing(visitor_id, experience.get("key"), attributes)
if isinstance(variation, dict):
bucketed_variations.append(variation)
elif isinstance(variation, (BucketingError, RuleError)):
# Log the error or handle it appropriately
# For now, we'll just skip this experience if it results in an error
pass

data_manager: DataManager,
rule_manager: RuleManager,
) -> None:
del config

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The config parameter is passed to the __init__ method but is immediately deleted using del config. This indicates that the parameter is not used within the method. It's better practice to prefix unused parameters with an underscore (e.g., _config) to clearly signal intent and avoid unnecessary operations like del.

Suggested change
del config
_config: Mapping[str, Any] | None = None,

try:
return json.loads(str(value))
except Exception:
return str(value)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In the cast_type function, when kind is "json", if json.loads(str(value)) fails due to an invalid JSON string, the function falls back to returning str(value). This can lead to ambiguous behavior where a caller might expect a JSON object (dict/list) or an error, but instead receives an unparsed string. It would be clearer to either raise an exception or return None to explicitly indicate a failure to cast to JSON.

Suggested change
return str(value)
try:
return json.loads(str(value))
except json.JSONDecodeError:
return None
except Exception:
return str(value)

*,
data_store: Any = None,
) -> None:
del config

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The config parameter is passed to the __init__ method but is immediately deleted using del config. This indicates that the parameter is not used within the method. It's better practice to prefix unused parameters with an underscore (e.g., _config) to clearly signal intent and avoid unnecessary operations like del.

Suggested change
del config
_config: Mapping[str, Any] | None = None,

else {"experienceId": str(experience.get("id"))},
)
if not bucketing:
return BucketingError.VARIAION_NOT_DECIDED

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The methods _retrieve_bucketing and _get_bucketing_by_field (line 355) can return either a dictionary (representing a bucketing result) or a BucketingError/RuleError enum member. This inconsistency in return types can make error handling more complex for callers. Consider raising exceptions for error conditions or returning None consistently and letting the caller interpret the absence of a result as an error.

Comment thread tests/unit/conftest.py
Comment on lines +29 to +39
config,
bucketing_manager=bucketing_manager,
rule_manager=rule_manager,
)
segments_manager = SegmentsManager(
config,
data_manager=data_manager,
rule_manager=rule_manager,
)
experience_manager = ExperienceManager(config, data_manager=data_manager)
feature_manager = FeatureManager(config, data_manager=data_manager)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The config object is passed to the constructors of DataManager, SegmentsManager, ExperienceManager, and FeatureManager. However, these managers then immediately discard or ignore the config parameter (e.g., using del config or by only extracting sub-parts of it and not using the full object). This creates an inconsistent API where config is required in the signature but not fully utilized, which can be misleading. Consider refactoring the constructors to only accept the specific configuration parts they need, or to use a common base class that handles the config extraction if it's a shared pattern.

@abbaseya
abbaseya deleted the codex/python-sdk-evaluation-layer branch June 18, 2026 16:37
abbaseya added a commit that referenced this pull request Aug 8, 2026
All eight open Dependabot alerts on this repo sit in `yarn.lock` — the
dev-only semantic-release tooling. None of them reach the published
wheel/sdist (httpx is still the only runtime dependency).

Three of the four highs were already reachable inside the existing semver
ranges; the fourth (sigstore GHSA-52v5-jr5w-gjxr, `certificateOIDs`
verification constraints silently dropped) needed sigstore >= 4.1.1, which
only arrives through a major bump:

  semantic-release 24.2.9 -> 25.0.9
    @semantic-release/npm 12.0.2 -> 13.1.5
      npm 10.9.8 -> 11.19.0
        libnpmpublish 10 -> 11.2.0  }-> sigstore ^3 -> ^4  (4.1.1)
        pacote        19 -> 21.5.1  }   @sigstore/core 2.0.0 -> 3.2.1
        tar 7.5.16 -> 7.5.22

`@semantic-release/github` moves to ^12 to match what semantic-release 25
depends on — leaving it at ^11 would hoist the older copy to the project
root and shadow the one core resolves.

Resolved (4 high, 4 medium — no criticals were open):

  #14 high    ip-address       10.2.0 -> 10.4.0   (needs >= 10.3.1)
  #9  high    brace-expansion  2.1.1  -> 5.0.9    (needs >= 2.1.2)
  #7  high    js-yaml          4.2.0  -> 4.3.1    (needs >= 4.3.0)
  #2  high    sigstore         3.1.0  -> 4.1.1    (needs >= 4.1.1)
  #13 medium  ip-address       (same bump as #14)
  #12 medium  ip-address       (same bump as #14)
  #4  medium  tar              7.5.16 -> 7.5.22   (needs >= 7.5.18)
  #1  medium  @sigstore/core   2.0.0  -> 3.2.1    (needs >= 3.2.1)

Node: semantic-release 25 requires ^22.14.0 || >= 24.10.0. release.yml
installs `lts/*`, currently Node 24.19.0 — satisfied, and every future LTS
line stays above the floor.

Verified: `yarn install --immutable` (what release.yml runs) passes against
the regenerated lockfile with the lockfile format unchanged (__metadata
version 10), and `yarn npm audit --all --recursive` reports no suggestions.
A `semantic-release --dry-run` against this branch loads all four configured
plugins, passes verifyConditions for both exec and github (GitHub
authentication + push permission), and analyzes commits to "no release" —
correct, since `chore` is a non-releasing type in release.config.mjs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant