Skip to content

Make e2e tests deterministic for provider-proxy cache #7248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AntoineToussaint wants to merge 6 commits into from
Closed

Make e2e tests deterministic for provider-proxy cache #7248

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9a6feab
Make e2e tests deterministic for provider-proxy cache
AntoineToussaint Apr 8, 2026
02a5a18
Revert embeddings cache tests to use random inputs
AntoineToussaint Apr 8, 2026
98afdd0
Fix embeddings cache tests: use write_only instead of random inputs
AntoineToussaint Apr 9, 2026
39952f2
Fix image URL tests and bypass raw.githubusercontent.com
AntoineToussaint Apr 9, 2026
0fbe2ab
Fix embeddings cache test and image server port conflicts
AntoineToussaint Apr 9, 2026
9d94b58
Revert cache test changes to split into a separate PR
AntoineToussaint Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert embeddings cache tests to use random inputs 
These tests need random inputs to avoid hitting the internal Valkey
cache from previous runs. The provider-proxy cache will need body
sanitization to handle these — they can't be made deterministic
without breaking the cache-testing logic.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
  • Loading branch information
@AntoineToussaint @claude
AntoineToussaint and claude committed Apr 8, 2026
commit 02a5a1802a5368cdf3dbee64b93f1416f7898227
7 changes: 6 additions & 1 deletion crates/tensorzero-core/tests/e2e/raw_response/embeddings.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,12 @@ async fn test_embeddings_raw_response_batch() {

#[tokio::test]
async fn test_embeddings_raw_response_with_cache() {
let input_text = "This is a cache test for embeddings raw_response - deterministic-cache-test";
// Random input ensures we don't hit the internal (Valkey) cache from a previous run.
// The provider-proxy cache will need body sanitization to handle this.
let input_text = format!(
"This is a cache test for embeddings raw_response - {}",
rand::random::<u32>()
);

// First request: populate cache with raw_response enabled
let payload = json!({
Expand Down
9 changes: 7 additions & 2 deletions crates/tensorzero-python/tests/test_embeddings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"""

import asyncio
import random

import pytest

Expand Down Expand Up @@ -216,7 +217,9 @@ async def test_embeddings_consistency(async_openai_client):
@pytest.mark.asyncio
async def test_embeddings_cache_with_float_encoding(async_openai_client):
"""Test that caching works correctly with float encoding format"""
input_text = "Cache test with float encoding - deterministic"
# Random input ensures we don't hit the internal (Valkey) cache from a previous run.
# The provider-proxy cache will need body sanitization to handle this.
input_text = f"Cache test with float encoding - {random.randint(0, 1000000)}"

# First request with float encoding and cache enabled
result1 = await async_openai_client.embeddings.create(
Expand Down Expand Up @@ -252,7 +255,9 @@ async def test_embeddings_cache_with_float_encoding(async_openai_client):
@pytest.mark.asyncio
async def test_embeddings_cache_with_base64_encoding(async_openai_client):
"""Test that caching works correctly with base64 encoding format"""
input_text = "Cache test with base64 encoding - deterministic"
# Random input ensures we don't hit the internal (Valkey) cache from a previous run.
# The provider-proxy cache will need body sanitization to handle this.
input_text = f"Cache test with base64 encoding - {random.randint(0, 1000000)}"

# First request with base64 encoding and cache enabled
result1 = await async_openai_client.embeddings.create(
Expand Down
Loading