RL Snake

A full journey through modern reinforcement learning on Snake — from tabular Q-learning through ConvNets — with interactive notebooks, real training logs, and honest empirical results.

PPO agent mastering 10×10 Snake after curriculum training

Final PPO agent after the 5×5 → 10×10 curriculum

Interactive Notebooks

Curiosity Killed the Snake

Does the Intrinsic Curiosity Module (Pathak et al., 2017) actually help? A 2×3×2×3 ablation across DQN, PPO, three reward modes, two boards, three seeds. Includes the death-oversampling trap and the terminal-mask fix.

ConvNet vs Feature DQN

Can raw grid pixels beat 24 hand-crafted features? A 30-trial W&B hyperparameter sweep, architecture walkthrough, and RND exploration ablation. ConvNet v3 reaches mean 12.32 on 10×10 vs Feature DQN's 5.50.

Results

10×10 board · 10k games · 3 seeds

ConvNet v3

12.32

ConvNet+RND

11.96

Feature DQN

5.50

ConvNet v1

1.49

16×16 board · 16k games

ConvNet v3

10.83

Feature DQN

5.92

ConvNet v1

0.20

Experiment History

1 · Zero to Hero — the Curriculum Journey

Watch the gameplay replay →

Phase	Algorithm	Board	Max Score	Notes
0	Tabular Q-Learning	5×5	24 ✓	Perfect on small board
1	Double Q-Learning	5×5	24	Stable convergence
2	Behaviour Cloning + PPO	8×8	46	Human demos bootstrap policy
3	PPO + Curriculum (5→10×10)	10×10	64	Progressive board growth

2 · DQN Ablations (8×8, 4k games, n=3)

Algorithm	Mean Score	Notes
DQN + Dueling + 3-step	4.61	Pilot A baseline
DQN + PER	4.48	Pilot B — prioritised replay
DQN + PER + ICM (unmasked)	4.46	Pilot C — ICM poisons terminal transitions
DQN + PER + ICM (terminal mask)	4.60	Pilot D — fix recovers baseline

3 · ICM Investigation (10×10, 5k games, n=3)

Algorithm	Reward mode	Mean Score	Notes
PPO	dense	0.12	PPO struggles with shaped reward
PPO + ICM	dense	0.14	ICM doesn't rescue PPO + dense
PPO	pure sparse	5.36	PPO shines without dense shaping
PPO + ICM	pure sparse	6.63	+24% — ICM's real contribution

4 · Curiosity Summary

Question	Answer
Does ICM help DQN with dense reward?	No — \|Δ\| < 0.05
Does ICM rescue DQN under sparse reward?	No — \|Δ\| < 0.15
Does dense shaping matter for DQN?	Yes — ~15% score drop without it
Does ICM help PPO under sparse reward?	Yes — +24% on pure_sparse 10×10
Does ICM increase state coverage?	No — both saturate at ~99%

5 · ConvNet Sweep Findings

n_steps=1 universally better with PER — appeared in every top sweep config; multi-step returns hurt with prioritised replay
ch1=64 / ch2=128 — larger second conv layer consistently wins over ch1=32/ch2=64
epsilon_end=0.01, gamma=0.995 — exploit harder, value distant food more
RND null result — saturation on sparse boards; curiosity helps weak baselines, not well-tuned agents

Running Locally

git clone https://github.com/Saheb/rl-snake.git
cd rl-snake
uv sync
uv run marimo edit notebooks/curiosity.py
uv run marimo edit notebooks/convnet.py