This directory contains benchmarks comparing diff-diff against equivalent R packages for validation and performance testing.

# Generate synthetic data (required before first run)
python run_benchmarks.py --generate-data-only

# Run all benchmarks
python run_benchmarks.py --all

# Run specific estimator
python run_benchmarks.py --estimator callaway

Note: Synthetic data files and results are not committed to the repository. Run --generate-data-only first to create the test datasets.

• diff-diff (this package)
• numpy, pandas, scipy

Install R and required packages:

# Install R (macOS)
brew install r

# Install R packages
Rscript R/requirements.R

Required R packages:

• did - Callaway & Sant'Anna (2021)
• synthdid - Synthetic DiD (Arkhangelsky et al. 2021)
• HonestDiD - Rambachan & Roth (2023)
• fixest - Fast fixed effects estimation
• jsonlite - JSON interchange
• data.table - Fast data manipulation

A few goldens are anchored against Stata where no runnable R reference exists. Stata is node-locked single-user, so — like the R arm — the goldens are committed and CI never needs Stata. Generators live in stata/ and are run headless:

# macOS, StataSE 19 (binary is not on PATH by default)
STATA=/Applications/Stata/StataSE.app/Contents/MacOS/stata-se
$STATA -b do benchmarks/stata/requirements.do            # one-time SSC install
$STATA -b do benchmarks/stata/generate_lpdid_ra_golden.do
$STATA -b do benchmarks/stata/generate_imputation_loo_golden.do
$STATA -b do benchmarks/stata/generate_etwfe_cs_golden.do
$STATA -b do benchmarks/stata/generate_reghdfe_kref_golden.do

The LPDiD arm uses only native Stata commands (teffects), pinned by version 19. The ImputationDiD arm depends on SSC packages (did_imputation/reghdfe/ftools/require), the ETWFE/CS arm on drdid/csdid/jwdid/hdfe, and the reghdfe K_reference arm on reghdfe; version 19 does NOT pin SSC packages (SSC has no version history) — install them once via requirements.do (the generators do not auto-install) and each golden records version/checksum metadata for drift detection. See stata/README.md.

benchmarks/
├── README.md                 # This file
├── run_benchmarks.py         # Main benchmark orchestrator
├── compare_results.py        # Result comparison utilities
├── R/
│   ├── requirements.R        # R package installation
│   ├── benchmark_did.R       # Callaway-Sant'Anna
│   ├── benchmark_synthdid.R  # Synthetic DiD
│   ├── benchmark_honest.R    # HonestDiD
│   └── benchmark_fixest.R    # Basic DiD / TWFE
├── stata/
│   ├── README.md                          # Stata arm docs
│   ├── requirements.do                    # one-time SSC install (did_imputation etc.)
│   ├── generate_lpdid_ra_golden.do        # LPDiD RA SE vs teffects ra
│   ├── generate_imputation_loo_golden.do  # ImputationDiD LOO SE vs did_imputation leaveout
│   ├── generate_etwfe_cs_golden.do        # ETWFE/CS vs jwdid + csdid (+ subsample ladder)
│   └── generate_reghdfe_kref_golden.do    # clustered CR1 K_reference vs reghdfe (disconnected panel)
├── python/
│   ├── utils.py              # Common utilities
│   ├── benchmark_callaway.py # CallawaySantAnna
│   ├── benchmark_synthdid.py # SyntheticDiD
│   ├── benchmark_honest.py   # HonestDiD
│   └── benchmark_basic.py    # Basic DiD / TWFE
├── data/
│   ├── synthetic/            # Generated test data
│   └── real/                 # Public datasets
└── results/
    ├── accuracy/             # Numerical comparison results
    └── performance/          # Timing results

Note: HonestDiD benchmark scripts exist but are not yet integrated into the main runner.

• ATT difference: < 1e-4 (absolute) or < 1% (relative)
• SE difference: < 10% (relative)
• CI overlap: Confidence intervals must contain each other's point estimates

• Wall clock time (seconds)
• Memory usage (MB) - optional
• Scaling behavior

Results are saved as JSON files:

{
  "estimator": "diff_diff.CallawaySantAnna",
  "overall_att": 2.0123,
  "overall_se": 0.1234,
  "timing": {
    "estimation_seconds": 0.456,
    "total_seconds": 0.789
  },
  "metadata": {
    "n_units": 200,
    "n_periods": 8,
    "n_obs": 1600
  }
}

1. Create R script in R/benchmark_<name>.R
2. Create Python script in python/benchmark_<name>.py
3. Add to run_benchmarks.py
4. Update documentation

See docs/benchmarks.rst for full methodology and results.