Live Demo — novikov-rocket-lab.streamlit.app Adjust physics sliders and watch the LightGBM model classify trajectories in real time.

A production-grade ML pipeline that classifies rocket types from radar-tracked 3D flight trajectories. Given a variable-length sequence of (traj_ind, x, y, z, time_stamp) radar observations, the system extracts physics-derived features — enriched with domain-specific salvo and rebel-group context — and predicts the rocket class (0, 1, or 2).

Standard accuracy is the wrong metric here. Class 2 comprises only 7.1% of training data (2,339 of 32,741 trajectories), yet the evaluation metric is minimum per-class recall — the system is scored by whichever class it handles worst. A model that perfectly identifies classes 0 and 1 but misses every class 2 rocket scores 0.0.

This metric demands that every design decision — feature engineering, class weighting, objective function, and post-hoc threshold tuning — be oriented toward equalising recall across all classes, deliberately sacrificing majority-class precision where necessary to protect minority-class recall.

Result: Global OOB min-recall of 1.000000 — 0 misclassified trajectories out of 32,741.

Achieved via proximity-based salvo consensus: the 2 OOB misses (1 class-0, 1 class-1) were rockets in tight salvos (dist ≈ 0 m, Δt < 12 s), each surrounded by same-class neighbours. Mode voting within each salvo group corrects every borderline prediction. Group purity = 100%, n_broken = 0.

No mountains or valleys — z is absolute altitude with no terrain correction required. This means altitude features (initial_z, final_z, apogee_relative) are physically interpretable and comparable across trajectories.

Rockets follow standard ballistic flight modelled as a frictionless point mass. Under this assumption, vertical acceleration is constant at −g; any deviation from that baseline is attributable solely to motor burn. Kinematic features (velocity, acceleration, jerk) therefore capture the thrust profile directly, with no confounding aerodynamic terms.

Assumptions 1 and 2 together establish that physics-derived features are meaningful — the signal is real, not artefact. Assumption 3 provides three business sub-assumptions from a domain expert, each directly driving a feature engineering decision.

Different launcher types can fire a different maximum number of rockets. This means the largest salvo ever observed from a rebel base reveals which launcher type they operate. Feature: group_max_salvo_size.

Rockets are typically fired together or with a short delay. Spatiotemporal DBSCAN clustering on (launch_x, launch_y, launch_time_s) identifies co-fired rockets and produces four features:

salvo_time_rank is the most discriminative of these, ranking 12th in model feature importance — rockets fired later in a salvo carry a detectable kinematic imprint.

There are few rebel groups, each operating from a fixed geographic area. Critically, each group purchases its own rocket supply independently — they are not organised under a single umbrella organisation. This means every group stocks one rocket type. Pure-spatial DBSCAN on (launch_x, launch_y) (eps=0.25, min_samples=3) produces group-level features:

Empirical finding: DBSCAN (on StandardScaler-normalised (launch_x, launch_y), eps=0.25) finds 3 geographic clusters. KDE contour analysis reveals a nested structure: Class 1 and Class 2 each occupy their own concentrated cluster, while Class 0 appears both in a separate adjacent cluster and as inner pockets at the core of each of the other two clusters. Consistent with assumption 3c (each group stocks one type), the inner Class 0 pockets represent independent Class 0 groups operating in geographic proximity to the Class 1 and Class 2 groups — not the same group firing multiple types. Individual exact-coordinate launch sites are class-exclusive (8,016 unique sites, zero cross-class overlap). The group-level aggregate features carry near-zero SHAP importance — the raw launch coordinates (launch_x, launch_y) already capture this signal directly (SHAP ranks 2 and 5).

Raw radar pings are aggregated into 32 scalar features per trajectory — 25 kinematic features from finite-difference physics (assumptions 1 & 2), plus 7 salvo and rebel-group features from DBSCAN clustering (assumption 3). These 32 survived automated backward elimination during training.

Why these features discriminate between rocket classes:

Why 32? Automated backward elimination on the full training set started from 83 candidates (76 kinematic + 7 salvo/group) and dropped 51 features that did not improve min_class_recall — they either encoded redundant information or introduced noise. The production code now only computes the 25 kinematic features that survived.

Note on model inputs: At inference, 3 rebel-group class-prior columns are appended to the 32 selected features, giving the model 35 total inputs. These priors (global training class distribution: 68.6%/24.3%/7.1%) are fixed constants that substitute for the fold-specific priors used during training — their feature importance is near-zero, making this approximation negligible.

The production model is LightGBM, trained via GPU-accelerated Optuna search (stops when post-consensus OOB = 1.0):

Threshold biases: [0.000000, -0.253165, 1.265823] — shifts decision boundaries to compensate for the 69%/24%/7% class imbalance (downweights class 1, upweights class 2). Found via coarse→fine grid search on OOB predictions.

class_weight="balanced" in LightGBM, which applies inverse-frequency weights (w_i = N / (K * N_j)) internally. Preferred over SMOTE because synthesizing trajectory feature vectors produces physically implausible combinations — a trajectory cannot have high jerk but zero acceleration.

LightGBM outputs calibrated per-class probabilities (multiclass objective). Per-class log-probability biases are then optimised on all OOB predictions simultaneously from 10-fold CV to maximise the min-recall metric via a coarse→fine grid search, producing [0.000000, -0.253165, 1.265823] for the current model.

Three layers:

1. GroupKFold on traj_ind — all radar pings from one trajectory stay in the same fold.
2. Per-fold NaN imputation — column medians computed from the training fold only.
3. OOB threshold tuning — biases optimised on OOB predictions only.

The problem is tabular classification on physics-engineered features with cross-trajectory context. LightGBM is the right choice for four structural reasons:

1. The feature set is inherently tabular, not sequential. The discriminative signal lives in aggregate statistics — muzzle velocity, apogee shape, salvo rank, launch position. These are scalar features per trajectory, not temporal sequences. Sequence models (CNN, Transformer) are designed to learn representations from raw time series; here the representations are already hand-crafted from physics. Asking a neural network to re-derive initial_speed from raw pings adds no information but introduces variance and training complexity.

2. Salvo and group features are cross-trajectory by nature. salvo_time_rank, group_max_salvo_size, and the other 5 salvo/group features (assumption 3) require aggregating information across multiple trajectories via DBSCAN clustering. A per-trajectory sequential model cannot access this information at all — it would need to be provided as an additional scalar input anyway. LightGBM treats all 32 features uniformly regardless of how they were derived.

3. Calibrated probabilities are essential for threshold tuning. The evaluation metric (min_class_recall) is optimised post-hoc by tuning per-class log-probability biases on OOB predictions. This requires the model to output well-calibrated class probabilities. LightGBM's multiclass objective with softmax produces probabilities suitable for this; raw neural network logits require additional calibration steps.

4. Leaf-wise growth concentrates capacity on hard cases. With a severe class imbalance (69%/24%/7%) and a metric that penalises the worst-performing class, the model must focus on the small number of class-2 trajectories that sit near class boundaries. LightGBM's leaf-wise growth builds asymmetric trees that allocate splits where the gain is highest — precisely the borderline minority-class examples. Level-wise trees (Random Forest, default XGBoost) waste capacity on already-well-separated majority-class regions.

Empirical validation via AutoML. The algorithm choice was confirmed — not assumed — through a 100-trial Optuna search over LightGBM, XGBoost, CatBoost, and a neural TabMLP with entity embeddings, all on the same 32-feature set with the same 10-fold GroupKFold evaluation. LightGBM dominated from the first trial: XGBoost scored 0.9253 raw OOB while LightGBM scored 0.9996 in the very next trial — a gap that never closed. Every top-5 Optuna trial across all runs was LightGBM.

A separate experiment tested a Transformer encoder operating on raw radar sequences with the same 7 salvo/group features as additional context input (salvo_dim=64, 4 layers, 8 heads). Best OOB min-recall after 10 Optuna trials: 0.9110 — confirming that the hand-crafted aggregate features used by LightGBM encode the discriminative signal far more effectively than learned temporal representations on this dataset.

Raw radar pings (traj_ind, x, y, z, t)
    │
    ▼
Kinematic Feature Engineering ──► 25 physics features per trajectory
    │
    ▼
Salvo/Group Feature Engineering ──► +7 features via DBSCAN (assumptions 3a-3c)
    │                                  (32 total)
    ▼
LightGBM trained on 32 features (25 kinematic + 7 salvo/group)
    │
    ▼
LightGBM + Optuna (GPU, stops when OOB consensus = 1.0) ──► hyperparameters
    │
    ▼
10-fold GroupKFold OOB Threshold Tuning ──► biases [0.000000, -0.253165, 1.265823]
    │
    ▼
Proximity Consensus Validation ──► group purity 100%, n_broken 0, OOB = 1.0
    │
    ▼
artifacts/model.lgb  ·  artifacts/train_medians.npy  ·  artifacts/threshold_biases.npy

cache/cache_test_features.feather   (pre-computed 32-feature matrix, Arrow IPC)
    │
    ▼
Select 32 production features + impute NaN with train_medians.npy
Append global class priors [0.686, 0.243, 0.071] → 35-column model input
    │
    ▼
artifacts/model.onnx  ──►  ONNX Runtime (AVX2, all cores, JIT pre-warmed)
         ↑ requires make export-model; falls back to artifacts/model.lgb if absent
    │
    ▼
log(proba) + biases [0.000000, -0.253165, 1.265823]  ──►  argmax
    │
    ▼
Proximity consensus: group by launch position + 60 s window → mode vote
    │
    ▼
output/submission.csv

An oracle threshold test tunes the per-class log-probability biases directly on the validation labels for each CV fold (cheating — labels are used to find the best threshold for that specific fold). If this oracle cannot reach 1.0 on a fold, then no threshold, feature, or model can fix that fold's errors — this is Bayes error.

Result: oracle = 1.0 on all 10 folds. The model already assigns higher probability to the correct class for every trajectory in every fold. The signal exists; the barrier was the global threshold constraint.

The 2 OOB misses (1 class-0, 1 class-1) were rockets in tight salvos. Diagnostic analysis showed each miss had same-class neighbours at dist ≈ 0 m, Δt < 12 s — borderline predictions in salvos fired from the same launcher within seconds of correctly-classified siblings.

Domain invariant (assumption 3b + 3c): all rockets fired from the same launcher within a salvo window share a rebel group → share a procurement → share a rocket type.

Algorithm: group trajectories by launch position (rounded to 0.01 precision) + 60-second time window. Within each group of size ≥ 2, apply strict-majority mode voting to the model's predictions.

Validation on training data:

• Group class purity: 100% (every proximity group is class-pure — no risk of consensus introducing errors)
• n_broken: 0 (consensus never worsened a correct prediction)
• OOB score: 0.999874 → 1.000000

This is not overfitting. Groups are formed from input features only (launch position, launch time) — labels play no role. The thresholds (60 s, position precision) were chosen from the physical definition of a salvo, not optimised against OOB labels. Purity was measured after the fact to confirm correctness, not used to select the parameters.

All runtimes measured on a Windows 11 machine (Intel Core i5, 4 cores, 8,185 test trajectories).

1. Thread saturation (empirical)

All 4 physical cores are in use. Thread sweep (30 runs each, ONNX inference only):

Adding threads beyond cpu_count() yields no further gain — there are no more physical cores.

2. Memory-bandwidth bound (theoretical)

The irreducible work is 8,185 samples × 1108 trees × depth 9 = 81.6M comparisons across the ONNX model.

Theoretical compute floor: 81.6M ops / 4 cores / 10⁹ ops/s  ≈  20ms
Measured inference:                                            ~0.60s
Overhead factor:                                               ~30x

This overhead is explained entirely by cache miss cost: the 5.8 MB model does not fit in L1/L2 cache (256 KB / 1 MB per core). Each tree traversal follows random pointers through L3 and main memory. Effective memory bandwidth for irregular-access tree walks is ~1–5 GB/s vs the raw L3 peak of ~100 GB/s.

3. Algorithm irreducibility

The 1108 trees were determined by Optuna to be the minimum that achieves 1.0 global OOB min-recall with proximity consensus. Reducing tree count would degrade accuracy below the operational threshold. Traversing every tree for every sample is not optional.

4. Backend optimality

ONNX Runtime with model_opt.onnx (graph optimization applied once at export time) applies:

• AVX2/AVX512 SIMD vectorisation for operator kernels
• Graph-level operator fusion (baked into model_opt.onnx)
• Memory arena pre-allocation (enable_mem_pattern = True)

The remaining non-inference overhead (~0.4s: session init + I/O + Python startup) cannot be eliminated in a single-shot batch process.

5. Persistent-service mode

For a long-running server that loads the model once and handles repeated requests, the model_opt.onnx init overhead (~0.25s) is paid only once. Steady-state per-batch latency drops to ~0.6s (pure inference + I/O).

Install uv:

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

git clone https://github.com/IlyaNovikov-RD/rocket_classifier.git
cd rocket_classifier
uv sync

# Full pipeline — downloads ~107 MB from GitHub Release, runs in ~1.0s
make pipeline
# Output: output/submission.csv  +  updated assets/

# Step by step:
make download-all    # artifacts/ + cache/ + data/ (test.csv, sample_submission.csv)
make run             # → output/submission.csv
make interpret       # → assets/shap_summary.png
make visualize       # → assets/demo.png

# After a model update — regenerate ONNX (requires: uv pip install onnxmltools)
make export-model

make demo            # streamlit demo (localhost:8501)

# Top-level
make all              # full validation: setup → quality → train → test → run → analysis (~20 min)
make all-full         # all + cold Docker rebuild + Streamlit smoke test (~30 min)

# Setup                                          ~1s
make install          # uv sync --group dev
make lock             # uv lock — regenerate uv.lock from pyproject.toml
make clean            # remove output/, cache/, artifacts/ for a fresh cold start

# Quality                                        ~1s
make lint             # ruff check
make format           # ruff format

# Training                                       ~17 min
make train            # full training pipeline (Optuna + consensus → artifacts/)
make export-model     # convert model.lgb → model.onnx (~48s)
make test             # full test suite — 130 tests (~16s)

# Inference                                      ~1s hot, ~3 min cold
make run              # inference pipeline → output/submission.csv

# Analysis                                       ~1 min
make interpret        # regenerate SHAP assets after model update (~60s)
make visualize        # regenerate assets/demo.png (~3s)

# Deploy                                         ~9 min cold build
make docker           # build + run Docker image → output/submission.csv
make docker-clean     # remove image and rebuild from scratch
make demo             # streamlit demo (localhost:8501)

# Data (download pre-built artifacts instead of training)  ~30s
make download-models  # fetch artifacts/ from GitHub Release
make download-all     # + cache/ parquet caches + data/ (test.csv, sample_submission.csv)
make pipeline         # download-all + run + interpret (full end-to-end, ~1.5 min)

# Release
make release TAG=v1.x.0 NOTES="..."  # create GitHub Release with all artifacts

Runtimes measured on Windows 11, 4-core CPU, 6 GB RAM. Training and Docker vary by hardware.

The image is self-contained — model artifacts, feature caches, and test.csv are baked in at build time.

docker build -t rocket_classifier .
docker run --rm rocket_classifier          # writes submission.csv inside the container

To retrieve the output file:

docker run --rm -v $(pwd)/output:/app/output rocket_classifier

rocket_classifier/              # Production inference package
├── __init__.py
├── features.py                 # 25 kinematic + 7 salvo/group features (single source of truth)
├── model.py                    # RocketClassifier — loads LightGBM, applies biases
├── schema.py                   # Pydantic v2 data contracts (TrajectoryPoint)
├── main.py                     # Inference pipeline: features → predict → submission.csv
└── app.py                      # Streamlit interactive demo

scripts/
├── download_models.py         # Download model artifacts from GitHub Release
└── export_fast_models.py       # Export model.lgb → model.onnx + benchmark all backends

requirements.txt               # pip-compatible export of production deps (uv export --no-dev)
training_report.json           # Authoritative record of latest training run: OOB scores, biases, best hyperparameters (tracked in git, uploaded to every release)

research/                       # R&D scripts (GPU training)
├── train.py                            # Full training pipeline → 1.0 OOB + artifacts
├── interpret.py                        # SHAP interpretability — run via `make interpret`
└── visualize.py                        # Feature visualization — run via `make visualize`

tests/
├── conftest.py                 # Shared fixtures (StubModel, synthetic trajectory builder)
├── test_app.py                 # Streamlit demo unit tests
├── test_consensus.py           # Proximity consensus unit tests
├── test_docs.py                # CI guard: documented test counts match actual
├── test_features.py            # Feature engineering unit tests
├── test_main.py                # Inference pipeline unit tests
├── test_model.py               # RocketClassifier + min_class_recall unit tests
└── test_schema.py              # Schema validation unit tests

assets/                           # Generated visualizations (git-tracked)
├── demo.png                    # Trajectory physics plot (make visualize)
├── shap_summary.png            # SHAP feature importance (make interpret)
└── interpretation_report.txt   # Human-readable model interpretation (make interpret)

artifacts/                        # Model artifacts — gitignored, from GitHub Release
├── model_opt.onnx              # Pre-graph-optimized ONNX (~0.3s faster load)
├── model.onnx                  # ONNX format — fastest inference (run: make export-model)
├── model.lgb                   # Native LightGBM — present after training
├── train_medians.npy           # 32-feature NaN imputation medians
└── threshold_biases.npy        # Per-class log-probability biases [0.000000, -0.253165, 1.265823]

cache/                          # Feature caches — gitignored
├── cache_train_features.parquet   # 25 kinematic features + label (salvo/group recomputed by DBSCAN at training time)
├── cache_train_features.feather   # Arrow IPC sidecar — fast reads
├── cache_test_features.parquet    # 32-feature matrix (25 kinematic + 7 salvo/group)
└── cache_test_features.feather

The automation in this project is not boilerplate — each choice directly serves a goal of the assignment.

research/interpret.py computes exact SHAP values via TreeExplainer. Top discriminators:

• Launch position (launch_x, launch_y, initial_z) — geography clusters by rebel group; altitude encodes terrain and launch geometry
• Horizontal speed (v_horiz_median) — muzzle velocity is propellant-charge dependent
• Salvo timing (salvo_time_rank) — launch order within a salvo carries a kinematic imprint (assumption 3b)
• Acceleration (acc_horiz_min, acc_mag_min) — propulsion physics separate the classes
• Ballistic arc (apogee_relative, delta_z_total) — arc shape differs between rocket families

make interpret   # regenerates assets/shap_summary.png and assets/interpretation_report.txt

The current pipeline is designed for batch inference — it classifies a complete set of pre-collected trajectories. The operational requirement is to classify rockets as early as possible after detection, before they reach apogee. This section outlines the architectural path from batch to real-time.

The model's 32 features fall into two categories with fundamentally different latency profiles:

A real-time system cannot compute salvo_time_rank or salvo_size for a rocket until sibling rockets in the same firing event have also been detected — they arrive asynchronously across different radars. This motivates a two-phase progressive classification architecture:

Radar pings arrive
       │
       ▼  Phase 1 — immediate (< 100ms after first ping)
Kinematic features only
  → ONNX inference (salvo features at training medians)
  → Preliminary class estimate broadcast to operators
       │
       ▼  Phase 2 — salvo-confirmed (~5–30s after launch cluster detected)
Salvo context materialises via streaming DBSCAN
  → Re-run inference with full 32 features
  → Updated high-confidence prediction replaces preliminary estimate

Phase 1 already satisfies the "as early as possible" requirement from the assignment at ~0.9997 quality — operators receive a threat assessment within the first second. Phase 2 upgrades to 1.000000 as the salvo context arrives and consensus is applied, without requiring any architectural change to the model itself.

Radar stations (N radars across the operational theatre)
        │  raw pings (x, y, z, t, radar_id)
        ▼
   Apache Kafka  ·  topic: radar-pings
        │
        ▼
  Apache Flink (stream processor)
   ├─ Trajectory assembler
   │   Groups pings by traj_ind within a sliding time window
   │
   ├─ Kinematic feature extractor ─────────────────────────┐
   │   Partial trajectory (≥ 3 pings) · Phase 1 (< 100ms)  │
   │                                                       │
   └─ Salvo coordinator ───────────────────────────────────┤
       Online DBSCAN · Phase 2 (salvo confirmed, ~5–30s)   │
                                                           ▼
                                                           ONNX serving endpoint
                                                           model.onnx · < 2ms/trajectory
                                                           │
                                                           ▼
                                                           Prediction store (Redis)
                                                           Keyed by traj_ind
                                                           Updated by both phases
                                                           │
                                                           ▼
                                                           Operator dashboard  ──  Alert system

DBSCAN as currently implemented runs in batch (O(N²) for large clusters). In a streaming context, the salvo coordinator would use an online spatial index (e.g. R-tree or ball tree over a sliding 60-second window of recent launches) to assign incoming rockets to existing salvos or open a new salvo group. The salvo DBSCAN parameters (eps=0.5, min_samples=2) and group DBSCAN parameters (eps=0.25, min_samples=3) remain unchanged — only the execution model shifts from batch to incremental.

The assignment identifies few rebel groups each buying independently (assumption 3c). If a new group acquires a new rocket type, production recall for that class will degrade. Operationally, this requires:

• Recall monitoring per class — not overall accuracy. The evaluation metric (min_class_recall) maps directly to the production alert threshold. When any class recall drops below an agreed SLA, retraining is triggered.
• Active labelling pipeline — intercepted rockets provide confirmed labels. These feed the next training run.
• GroupKFold preservation — retraining must continue to group all pings from one trajectory into the same fold to prevent leakage from partial-trajectory data collected mid-flight.

A ballistic rocket travelling at typical class-1 speeds takes 60–90 seconds from launch to apogee. Both phases complete well within the available threat-assessment window.

Ilya Novikov