A single-protocol, controlled comparison of prior-injection mechanisms for safe reinforcement learning on a 3D quadrotor (Crazyflie 2.X), under explicit model–plant mismatch. The model-based prior is a nonlinear MPC (acados NMPC); the simulator is gym-pybullet-drones / PyBullet at 30 Hz under PYB_GND_DRAG_DW physics.

The MPC's internal model is always the nominal CF2X; only the simulated plant is perturbed so the prior is suboptimal by construction. This is what non-expert prior means here.

A trained BC+PPO policy flying the perturbed plant to the target at [2, 2, 2] (red marker), recorded from the PyBullet GUI:

1. Safe-exploration gains. Under model–plant mismatch, how much does a non-expert MPC prior reduce the safety cost incurred during learning (recoverable vs. catastrophic, reported separately) at matched final task performance?
2. Break-even boundaries. As mismatch increases, at what severity does each prior-injection mechanism stop helping crossing below the no-prior baseline (negative transfer) or below the static MPC-only reference?
3. Prior degradation & failure modes. Which mechanism best tolerates a degrading prior, and which collapses (e.g. BC-pretrain via catastrophic forgetting)?
4. Zero-shot generalization. Does a policy trained safely under a non-expert prior transfer zero-shot to an online-discovered course (sequential gate reveal) with no retraining? Does the safety advantage persist under that distribution shift?
5. Soft priors vs. hard shields. A soft prior reduces but cannot eliminate catastrophic violations; does an MPC runtime safety filter (shield) drive them toward ~0, and at what cost to sample efficiency and final performance?

The objective is competent task performance while minimizing cumulative cost incurred during learning (safe exploration). Recoverable and catastrophic costs are tracked separately throughout. The safety classifier is measurement-only: it never enters the optimizer.

Mismatch is injected by two wrappers (plant only; MPC sees the nominal model):

• PerturbedPlantWrapper : mass multiplier, aerodynamic drag (the plant has GND_DRAG_DW drag; the MPC model is drag-free), first-order actuator lag.
• SensorNoiseWrapper : per-episode constant position bias (sampled once, zero derivative → no spurious velocity/rate bias) plus small zero-mean Gaussian noise on position, attitude, and linear/angular velocity. Both the learned policy and the MPC prior receive only the noisy observation; the privileged true state is used exclusively to compute the safety metric.

Method arms (PPO with identical SB3-default hyperparameters, wrappers, budget, and seeds across all trained arms, no per-task tuning):

MPC prior. Nonlinear MPC (acados; SQP-RTI with partial-condensing HPIPM, linear-least-squares cost) over a 12-state rigid-body model with four per-motor thrusts; hand-tuned weights; solver warm-start reset to a hover trim at every episode.

Observations. Ego-centric goal encoding: the first three obs components are the goal-relative position error (p_t − p) instead of absolute world position; attitude and linear/angular velocity unchanged. Required for the zero-shot multi-gate generalization study and standard for goal-reaching RL.

Evaluation. Reported separately for recoverable and catastrophic: (i) training-time cumulative cost (the safe-exploration headline), (ii) eval-time unsafe-step rate of the fixed final policy, (iii) per-episode ever-unsafe and catastrophe rates. Episodes truncate only on catastrophe. Aggregates use IQM with stratified-bootstrap 95% CIs (rliable); seed is the unit of replication.

MPC-Safe-Exploration/
├── README.md
├── .gitignore
├── figs/                                # paper figures (committed)
├── gym-pybullet-drones/                 # absorbed fork  project source
│   └── gym_pybullet_drones/
│       ├── control/MPCControl.py        # acados NMPC controller
│       ├── control/c_generated_code/    # acados-generated C (binaries gitignored)
│       ├── envs/                        # BaseRLAviary, HoverAviary (modified)
│       └── src/                         # method arms + wrappers + callbacks
├── acados/                              # NOT in git  install separately
├── results/                             # NOT in git  training/eval outputs
└── local_folder/                        # NOT in git  scratch / personal notes

python3.10 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

MPCControl.py imports acados_template and compiles an OCP solver into gym_pybullet_drones/control/c_generated_code/. Install acados into the acados/ directory at the repo root:

git clone https://github.com/acados/acados.git
cd acados
git submodule update --recursive --init
mkdir -p build && cd build
cmake -DACADOS_WITH_QPOASES=ON ..
make install -j4
cd ..
pip install -e interfaces/acados_template

Then export the runtime environment variables (add to your shell profile):

export ACADOS_SOURCE_DIR="$PWD/acados"
export LD_LIBRARY_PATH="$PWD/acados/lib:$LD_LIBRARY_PATH"

acados also fetches t_renderer on first solver build; allow it network access or place the binary in acados/bin/.

cd gym-pybullet-drones
pip install -e .

This installs the package in editable mode along with its dependencies (pybullet, gymnasium, stable-baselines3, casadi, scipy, ...).

Run scripts from inside gym-pybullet-drones/ so package imports and the relative results/ output paths resolve correctly:

cd gym-pybullet-drones

# MPC-only rollout (static-prior reference)
python gym_pybullet_drones/src/MPC_only.py

# PPO from scratch (no-prior reference)
python gym_pybullet_drones/src/PPO_from_Scratch.py

# Residual RL on top of the MPC
python gym_pybullet_drones/src/residualMpc.py

# BC pre-training + PPO fine-tune (MPC as expert)
python gym_pybullet_drones/src/bc_ppo_mpc.py

# MPC safety filter (shield)
python gym_pybullet_drones/src/MPCShield.py

# Evaluate a trained shield-protected policy
python gym_pybullet_drones/src/eval_shield.py

Models and evaluation logs are written to results/ not tracked in git.

Model-based scaffolding of RL is well established: guided policy search with an MPC teacher (Zhang & Levine, 2016); residual policy learning (Silver et al., 2018; Johannink et al., 2019); Jump-Start RL (Uchendu et al., 2023); BC + RL fine-tuning as a common initialization scheme. Each prior-injection failure mode is documented separately RL with an inaccurate model (Abbeel & Ng, 2006), RL under model mismatch (Roy et al., 2017), negative transfer in continual RL (Reset & Distill, 2024), fine-tuning as a forgetting problem (Wolczyk et al., 2024) but on different systems and protocols. The projects Simulation is slightly modified Pybullet (https://pybullet.org/wordpress/). We provide a single-protocol comparison and failure-boundary map on a realistic 3D quadrotor under an explicitly non-expert prior. Safe-exploration metrics follow Safety Gym (Ray et al., 2019) and safe-control-gym (Yuan et al., 2022); statistics follow rliable (Agarwal et al., 2021).

• A controlled, single-protocol comparison of four prior-injection mechanisms (no-prior, static MPC, residual, BC-pretrain) on a realistic 3D quadrotor injection mechanism is the only variable.
• An empirical failure-boundary map: the mismatch severity at which each mechanism crosses below the no-prior baseline or the static-prior reference.
• A zero-shot generalization study of safely-trained policies on an online-revealed multi-gate course.