This directory contains tests for the audio-separator project.

• Core functionality tests: test_cli.py, test_stft.py
• Remote API tests: test_remote_api_client.py, test_remote_cli.py

• CLI integration tests: test_cli_integration.py
• Audio output validation: test_separator_output_integration.py
• Remote API integration tests: test_remote_api_integration.py

The project includes comprehensive tests for the remote API functionality:

• 61 tests covering all remote API components
• Unit tests with mocked HTTP responses
• Integration tests with a mock HTTP server
• End-to-end workflow tests

Key features tested:

• Multiple model support (upload once, process with multiple models)
• Full parameter compatibility with local CLI
• Asynchronous processing with job polling
• Error handling and edge cases
• File upload/download workflows

See tests/remote/README.md for detailed documentation on remote API tests.

The integration tests now include validation of output audio files by comparing waveform and spectrogram images with reference images. This helps ensure that the audio separation results remain consistent across different runs and code changes.

1. Reference waveform and spectrogram images are generated from expected output files
2. During test execution, the same images are generated for the actual output files
3. The images are compared using Structural Similarity Index (SSIM) to measure perceptual similarity
4. If the images differ significantly, the test fails, indicating a change in the audio output

The tests use Structural Similarity Index Measure (SSIM) to compare images, which is more robust than pixel-by-pixel comparison:

• SSIM considers structural information in the images
• It's more resilient to small spatial shifts or offsets
• It better matches human perception of image similarity
• It works well across different environments (local machines vs CI servers)

The comparison uses a minimum similarity threshold (0.0-1.0):

• Higher values (closer to 1.0) require images to be more similar
• Lower values (closer to 0.0) are more permissive
• A value of 0.99 requires 99% similarity between images
• A value of 0.0 would consider any images to match

The default threshold is set to 0.999, which is quite strict. However, model-specific thresholds can be set to accommodate different models' behavior.

Some models inherently produce slightly different outputs on different runs, even with the same input. To accommodate these models, you can set model-specific thresholds in the MODEL_SIMILARITY_THRESHOLDS dictionary:

MODEL_SIMILARITY_THRESHOLDS = {
    "htdemucs_6s.yaml": 0.990,  # Demucs models need a lower threshold
    # Add other models that need custom thresholds here
}

This allows you to maintain a high threshold for most models while being more flexible with models that naturally exhibit more variation.

To generate or update the reference images, use the script provided:

python tests/integration/generate_reference_images.py

This script will create waveform and spectrogram images for all expected output files and store them in the tests/inputs/reference directory.

If you need to skip the audio validation (e.g., when you're intentionally changing the output), you can set the environment variable SKIP_AUDIO_VALIDATION=1:

SKIP_AUDIO_VALIDATION=1 pytest tests/integration/test_cli_integration.py

When adding a new model to the tests:

1. Add the model and its expected output files to the MODEL_PARAMS list in test_cli_integration.py
2. Run the integration test to generate the output files
3. Run the generate_reference_images.py script to create the reference images
4. Run the tests again to validate the output files
5. If necessary, add a custom similarity threshold for the new model in MODEL_SIMILARITY_THRESHOLDS

To see detailed validation results, run pytest with the -sv flag:

pytest tests/integration/test_cli_integration.py -sv

This will show the similarity scores for each comparison and whether they passed or failed.

To run all tests:

pytest

To run specific tests:

pytest tests/unit/
pytest tests/integration/

To run remote API tests:

# All remote API tests (unit + integration)
pytest tests/unit/test_remote*.py tests/integration/test_remote*.py -v

# Only unit tests (fast, no HTTP server)
pytest tests/unit/test_remote*.py -v

# Only integration tests (with mock HTTP server)
pytest tests/integration/test_remote*.py -v

To run with coverage:

pytest --cov=audio_separator

# Coverage for remote API specifically
pytest tests/unit/test_remote*.py tests/integration/test_remote*.py --cov=audio_separator.remote --cov-report=html