Unit test installs kserve into the live interpreter, causing nondeterministic failures in unrelated tests #6732

Expected Behavior

Running the unit suite leaves the environment it ran in unchanged, and a test's outcome does not depend on what other tests are doing concurrently.

Current Behavior

sdk/python/tests/unit/infra/test_dependency_conflicts.py installs a package into the interpreter that is running the suite:

command = [sys.executable, "-m", "pip", "install", "kserve==0.15.2"]

There is no isolation and no cleanup, so this mutates the environment every other test is using. Two consequences.

Unrelated tests fail nondeterministically in the same run. pyproject.toml pins psutil==5.9.0; kserve 0.15.2 requires psutil<6.0.0,>=5.9.6. pip therefore cannot leave the installed version in place — it uninstalls psutil, then installs 5.9.8:

Attempting uninstall: psutil
  Found existing installation: psutil 5.9.0
  Uninstalling psutil-5.9.0:
    Successfully uninstalled psutil-5.9.0
Installing collected packages: … psutil …

make test-python-unit runs pytest -n 8, so eight workers share one environment. Any test running during that window which imports psutil, directly or through a subprocess, fails. feast/metrics.py imports psutil at module scope, and feast/cli/cli.py reaches it through feast/__init__.py, so every test that shells out to the CLI is exposed.

Observed on an unrelated docs PR, in unit-test-python (3.12, ubuntu-latest):

FAILED sdk/python/tests/unit/cli/test_cli.py::test_3rd_party_providers
AssertionError: Expected <b'Traceback (most recent call last):
  File ".../feast/cli/cli.py", line 27, in <module>
    from feast import utils
  ...
  File ".../feast/metrics.py", line 57, in <module>
    import psutil
ModuleNotFoundError: No module named \'psutil\'
'> to contain item <b"Provider \'feast123\' is not implemented">

In that job test_install_kserve_with_feast completed at 12:25:54 and the failure landed at 12:29:15. Because it depends on worker timing, a re-run usually passes, so it presents as flake and the blame lands on whichever PR happens to be running.

A second consecutive local run fails. kserve also pulls protobuf down to 4.25.x, and the installed grpcio-health-checking ships protobuf 6.x generated code that imports google.protobuf.runtime_version. The first run still passes, because pytest imports every module during collection before the mid-run install lands; the next run fails at collection with ImportError: cannot import name 'runtime_version' from 'google.protobuf'. CI does not see this because its environments are ephemeral.

Steps to reproduce

make test-python-unit    # passes
make test-python-unit    # collection errors on runtime_version

For the race, run the suite with -n 8 and observe that any test shelling out to the CLI can fail while the kserve install is between uninstall and install.

Specifications

• Version: master
• Platform: any; observed on ubuntu-latest Python 3.12 in CI and locally on macOS

Possible Solution

The test checks that kserve and feast can be resolved together. pip can answer that without installing anything, using --dry-run. Verified both directions:

• no conflict: exits 0, performs full resolution (Would install … psutil-5.9.8 …), and leaves the environment untouched — psutil stays at 5.9.0 and kserve is not importable afterwards
• real conflict: pip install --dry-run kserve==0.15.2 psutil==5.9.0 exits 1 with Cannot install kserve==0.15.2 and psutil==5.9.0 because these package versions have conflicting dependencies

So the detection is unchanged and the mutation disappears. Happy to open a PR for this.

Separately, the current assertion looks inverted:

conflict_occured = "dependency conflicts" in err and "ERROR" not in err

That is only true when pip reports conflicts without an error, so a run that fails loudly sets it to False. The exit_code == 0 assertion above it is what actually catches failures today.