SWIFT: Speech and Waveform Investigative Forensic Toolkit

SWIFT is an enterprise-grade, real-time deepfake audio detection system engineered specifically for telephony and live-call environments. It is designed to evaluate bidirectional telecommunication streams and differentiate between authentic human speech and high-fidelity synthetic AI voices generated by modern text-to-speech engines or voice cloning vocoders.

The platform utilizes a high-performance Python backend for concurrent acoustic feature extraction and quantized neural inference, integrating directly into telephony providers via WebSockets.

The following diagram illustrates the complete data pipeline, from telephonic audio ingestion through quantized inference and automated policy enforcement.

graph TD
    subgraph Layer1 [1. Ingestion Layer]
        A[SignalWire / Telephony Media Stream] -->|WebSocket 8kHz μ-law| B[Ingestion Receiver]
    end
    
    subgraph Layer2 [2. Buffer Management]
        B -->|Resample to 16kHz PCM| C[Sliding Window Buffer]
        C -->|32k samples / 2.0s window| D{500ms Hop Stride}
    end
    
    subgraph Layer3 [3. Feature Extraction]
        D --> E[Concurrent Extraction Pipeline]
        E -->|Micro-Acoustic Metrics| F(PhysioSpecNet Tensor)
        E -->|Biomechanical Dynamics| F
        E -->|Forensic Spectrograms| F
    end
    
    subgraph Layer4 [4. Inference Engine]
        F -->|Cross-Channel Attention| G[PhysioSpecNet]
        G -->|EfficientNet-B0| H[INT8 ONNX Runtime]
        H -->|Evaluation| I{Synthetic Probability Index}
    end
    
    subgraph Layer5 [5. Policy Engine and HUD]
        I -->|SPI < 0.30| J[HUD: AUTHENTIC]
        I -->|0.30 <= SPI < 0.70| K[HUD: MONITORING ANOMALY]
        I -->|SPI >= 0.70| L[HUD: SYNTHETIC DETECTED]
        L -->|REST API Trigger| M[Automated Mitigation / Call Severance]
    end

• Audio Ingestion: Ingests live dual-channel or mono phone audio via SignalWire Media Streams (or WebRTC) over a secure WebSocket (wss://) at 8 kHz μ-law.
• Resampling and Normalization: Standardizes and upsamples the incoming byte stream into linear 16-bit 16 kHz PCM arrays using optimal Fourier decimation to prevent high-frequency aliasing.

• Sliding Window Buffer: Maintains an in-memory ring buffer holding a rolling 2.0-second window (32,000 samples).
• Hop Stride: Executes inference on a 500 ms rolling interval to enable sub-second temporal overlap and low-latency continuous monitoring.
• Algorithmic VAD: Integrates Voice Activity Detection via Peak and Root Mean Square energy gating to bypass inference during pure silence, preserving compute resources.

Standard audio models compress high frequencies to replicate human hearing, destroying the microscopic synthetic artifacts left by AI generators. SWIFT resolves this by projecting the 2.0-second buffer across six distinct acoustic dimensions:

• Micro-Acoustic Metrics: Evaluates Spectral Flatness, Phase Entropy, and Spectral Centroid Variance to detect vocoder anomalies. Includes an Instantaneous Phase Unwrapping Map to capture phase incoherence, a strict signature of generative AI vocoders.
• Biomechanical Vocal Dynamics: Measures Pitch Jitter, Shimmer, and the Glottal-to-Noise Excitation (GNE) spectrum to assess natural vocal cord micro-irregularities. Humans exhibit natural micro-tremors, whereas synthetic cloning outputs are mathematically perfect.
• Deep Forensic Representations: Utilizes Linear Frequency Cepstral Coefficients (LFCC) alongside high-resolution Formant Trajectory Maps to expose high-frequency synthesis anomalies and ensure physical consistency in resonant frequencies.

• Feature Fusion (Cross-Channel Attention): Aggregates the acoustic metrics and spectral representations into a dense (6, 224, 224) tensor. The physiological channels dynamically attend to and modulate the primary LFCC channel, highlighting inconsistencies between acoustic features and physical spectral data.
• Quantized Inference: Processes the modulated tensor through the EfficientNet-B0 backbone of our custom PhysioSpecNet model. The network is exported and executed via an INT8-quantized deep learning pipeline on ONNX Runtime for strict latency adherence.
• Output Metric: Generates a continuous Synthetic Probability Index (SPI) calibrated from 0.00 (Organic Human) to 1.00 (Synthetic or Cloned).

The evaluated SPI dictates programmatic action within the telephony environment.

• Nominal State (SPI < 0.30): Verified authentic stream; session integrity remains authenticated.
• Elevated Risk State (0.30 ≤ SPI < 0.70): Triggers continuous telemetry polling and escalates session logging for post-call audit.
• High-Confidence Threat (SPI ≥ 0.70): Authenticates a spoofing event and executes automated mitigation protocols. This triggers automated call severance or programmatic mute via the SignalWire REST API.

• Low-Latency Telemetry Channel: Broadcasts per-window forensic payloads over bidirectional WebSockets (wss://telemetry/session-id) with under 50 ms transmission latency.
• Floating In-Call Status Widget: Displays a non-intrusive heads-up monitoring widget on the agent interface featuring:
  • Dynamic Integrity Gauge: Live circular or linear risk meter tracking real-time SPI score fluctuations.
  • Spectral Confidence Breakdown: Visual micro-bars displaying metric weights (Acoustic Jitter, Phase Coherence, LFCC Anomaly).
  • Real-Time Forensic Banner: Dynamic status badge (AUTHENTIC | MONITORING ANOMALY | SYNTHETIC DETECTED).
  • Incident Action Controls: Inline programmatic action triggers permitting one-click session isolation, secondary challenge-response prompt injection, or instant call severance.

SWIFT/
├── src/                    # Core Python package
│   ├── features/           # LFCC and 6-Channel Spectrogram extraction logic
│   ├── models/             # PhysioSpecNet architecture and Attention mechanisms
│   ├── realtime_stream.py  # AudioRingBuffer and VAD engine
│   └── config.py           # Hyperparameters
├── public/                 # Web dashboard frontend (HUD logic)
├── checkpoints/            # Pre-trained PyTorch weights
├── scripts/                # Evaluation and benchmarking scripts
├── tests/                  # Unit tests for the pipeline
├── data/                   # Datasets (git-ignored)
├── docs/                   # Convergence graphs and metrics
├── server.py               # Main API and WebSocket server
├── evaluate_and_train.py   # Primary fine-tuning and evaluation script
└── requirements.txt        # Python dependencies

• Python 3.9+
• CUDA-compatible GPU (recommended) or CPU
• ONNX Runtime / PyTorch 2.0+

Clone the repository and install the dependencies:

git clone https://github.com/codebreaker77/SWIFT.git
cd SWIFT
pip install -r requirements.txt

SWIFT includes a dedicated forensic testing utility (scripts/test_model_realtime.py) to test PhysioSpecNet directly across multiple operational modes:

Synthesizes speech on-the-fly using state-of-the-art neural cloud TTS engines (Azure / OpenAI / ElevenLabs architectures) and injects the waveform directly into the inference pipeline in-memory:

# Test with default male neural voice (GuyNeural)
python scripts/test_model_realtime.py --tts "This is an urgent call regarding your financial accounts. Please state your passcode."

# Test with female expressive neural voice (JennyNeural)
python scripts/test_model_realtime.py --tts "Good morning, thank you for calling customer support. How may I assist you?" --voice en-US-JennyNeural

# Test with conversational neural voice (RyanNeural)
python scripts/test_model_realtime.py --tts "Hey there, I wanted to follow up on the proposal we discussed." --voice en-US-RyanNeural

Listens directly to your local system microphone with sub-second sliding window analysis and prints a real-time risk meter in the terminal:

python scripts/test_model_realtime.py --mic

Evaluates any recorded WAV or MP3 audio file:

python scripts/test_model_realtime.py --file public/samples/modern_tts/tts_male_deep_1.wav
python scripts/test_model_realtime.py --file public/samples/real_human_1.wav

Runs a quick automated verification across all local authentic human samples and AI voice models:

python scripts/test_model_realtime.py

To train PhysioSpecNet on genuine multi-speaker data and modern neural cloud TTS vocoders:

Generate 25+ high-fidelity multi-speaker neural samples (male, female, deep, conversational, expressive) across Azure/OpenAI neural voices:

python scripts/generate_modern_tts_samples.py

Trains PhysioSpecNet using Cosine Annealing, G.711 $\mu$-law transcoding augmentation, and balanced real human vs neural TTS speech pools:

python scripts/finetune_modern_tts.py

For large-scale multi-speaker training (2,500+ samples, 3-way ablation study, 300 DPI publication plots, LaTeX tables, minDCF, and EER): Open and run notebooks/PhysioSpecNet_Colab_Training.ipynb in Google Colab with GPU acceleration.

Start the backend WebSocket server and telemetry dashboard with SignalWire telephony:

python -m uvicorn signalwire_server:app --host 0.0.0.0 --port 8001

• Navigate to http://localhost:8001 to access the live Heads-Up Display (HUD).
• Configure your SignalWire phone number to route incoming voice calls to your public URL (via ngrok): https://<your-ngrok-domain>/signalwire/voice