Skip to content

4-atari-hard: PPO+RND on Montezuma's Revenge + benchmark#130

Merged
dnddnjs merged 5 commits into
masterfrom
ai/montezuma-rnd
Jun 6, 2026
Merged

4-atari-hard: PPO+RND on Montezuma's Revenge + benchmark#130
dnddnjs merged 5 commits into
masterfrom
ai/montezuma-rnd

Conversation

@dnddnjs

@dnddnjs dnddnjs commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

What

Makes the 4-atari-hard PPO + RND reproducible and resumable, and adds a Montezuma's Revenge benchmark to the README.

Result

  • Final mean per-game return ~3120 @ 65M agent steps, ~3.4h, single seed, Mac Studio (M4 Max).
  • Above the paper PPO baseline (2497) (+623).
  • 512 parallel envs are the key lever: the first key is found reliably (~327k steps), then the extrinsic value bootstraps around 10M steps. With 128 envs the same code never scored in 50M steps — parallel breadth cracks the first-key bottleneck.
  • Still far below RND's headline 8152 (128–1024 envs × 1.97B frames, ~30× more experience). Single seed → no ± std; a 3-seed run is the next step.

Changes

  • 4-atari-hard/1-ppo-rnd.py: reproducible + resumable training (seed, run dir, metrics.jsonl, atomic periodic/milestone/best checkpoints, --resume restoring normalizer + optimizer + frame counter). Runs standalone.
  • 4-atari-hard/env.py: optional CLI flags + a shared RunLogger (run-directory logging / checkpoints / resume).
  • README.md: Montezuma's Revenge (PPO + RND) row in Benchmarks.

W&B: https://wandb.ai/rlcode/rl-atari-hard-ppo-rnd

dnddnjs and others added 4 commits May 24, 2026 12:08
New chapter for hard-exploration Atari. PPO with Random Network
Distillation (Burda et al., 2018) as the curiosity bonus.

- env.py: ALE/MontezumaRevenge-v5 (and pitfall, private_eye) with the
  standard Atari preprocessing, no FireResetEnv, no LifeLossTerminalEnv
  (uninterrupted episodes so intrinsic returns can chain across deaths).
- 1-ppo-rnd.py: two-value-head ActorCritic, RND target/predictor with
  LeakyReLU, single-frame normalized input clipped to [-5, 5], obs RMS
  seeded by 50 rollouts of a random agent, intrinsic reward scaled by
  running std of discounted intrinsic returns, dual GAE (extrinsic
  episodic + intrinsic non-episodic), predictor updated on 25% of each
  minibatch, combined advantage A = 2*A_ext + 1*A_int.

Not run end-to-end yet. Sanity-checked static shapes and module wiring.
PPO+RND made reproducible and resumable. Shared run plumbing (seed,
metrics.jsonl, periodic/milestone/best checkpoints, resume, final summary)
lives in env.py's RunLogger, keeping the algorithm file focused. 512 parallel
envs crack the first-key bottleneck (128 envs never scored in 50M); final mean
per-game return ~3120 @ 65M steps, single seed (M4 Max), above the paper PPO
baseline (2497). Adds a README benchmark row. Count-based exploration is
deferred to a later PR (not yet trained/benchmarked).
@dnddnjs dnddnjs merged commit e590444 into master Jun 6, 2026
@dnddnjs dnddnjs deleted the ai/montezuma-rnd branch June 6, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants