Is your feature request related to a problem? Please describe.
Our current test organization relies on pytest markers to separate unit and integration tests, but this system is failing:
test-python-unit runs pytest sdk/python/tests (the entire tests directory) and filters by markers — any integration test missing @pytest.mark.integration silently runs as a "unit" test
- At least 3 integration test files (
test_mcp_feature_server.py, test_hybrid_online_store.py, test_dbt_integration.py) are missing the marker and currently run during unit CI
- Unit tests import heavyweight dependencies (ray, pyspark) that slow down the entire suite and require special environment setup
- The
conftest.py marker-filtering logic in pytest_collection_modifyitems is a house of cards — markers are opt-in and people forget them
Describe the solution you'd like
Replace marker-based filtering with a folder-based 4-tier layout:
tests/
unit/ → Fast, no heavy deps, pure logic
universal/ → Cross-backend parameterized tests (same test, many backends)
component/ → Backend-specific tests (needs ray/spark/etc installed)
integration/ → Needs external services (K8s, Docker, Keycloak, cloud)
Key benefits:*
pytest tests/unit — path-based, no marker gymnastics, no leakage
pytest tests/component/ray — runs only ray-specific tests
pytest tests/universal --config=duckdb — runs universal tests against DuckDB
- The folder is the categorization — no markers to forget
- Ray CI =
component/ray + universal(ray config). Clean and composable.
Sub-tasks
Migration Principles
- Each sub-task is a self-contained PR that can be merged and tested independently
- No functional changes to tests — only file moves and import path updates
- Old Makefile targets should remain as aliases during transition (deprecated, then removed)
- Run the relevant CI after each PR to verify nothing broke
Thanks to @tokoko for idea and kicking off discussion.
Is your feature request related to a problem? Please describe.
Our current test organization relies on pytest markers to separate unit and integration tests, but this system is failing:
test-python-unitrunspytest sdk/python/tests(the entire tests directory) and filters by markers — any integration test missing@pytest.mark.integrationsilently runs as a "unit" testtest_mcp_feature_server.py,test_hybrid_online_store.py,test_dbt_integration.py) are missing the marker and currently run during unit CIconftest.pymarker-filtering logic inpytest_collection_modifyitemsis a house of cards — markers are opt-in and people forget themDescribe the solution you'd like
Replace marker-based filtering with a folder-based 4-tier layout:
Key benefits:*
pytest tests/unit— path-based, no marker gymnastics, no leakagepytest tests/component/ray— runs only ray-specific testspytest tests/universal --config=duckdb— runs universal tests against DuckDBcomponent/ray+universal(ray config). Clean and composable.Sub-tasks
feature_repos/component/ray/component/spark/universal/offline_store/universal/online_store/universal/conftest.py— move fixtures, remove marker filteringMigration Principles
Thanks to @tokoko for idea and kicking off discussion.