ICML 2026Seoul

Active Curriculum Refinement for Reinforcement Learning

Treat the curriculum as a structured space. Explore it with diverse paths, then actively refine the regions where learning has not yet saturated.

Zhenya LiuYuxin ChenThe University of ChicagoContactzhenya@uchicago.edu

Paper on arXiv alphaXiv Code

terrain geometry

dynamics

task complexity

Curriculum space · live

Problem

A curriculum is not only an ordering. It is a space of possible paths.

Real environment families vary along several difficulty axes: terrain roughness, gap width, stair height, obstacle count, or observable game statistics. Small increases along one coordinate induce prerequisite relations and a curriculum graph over environments.

PATH changes the unit of curriculum selection from isolated environments to trajectories through this graph.

Interactive mechanism

From broad random exploration to active refinement

The visualization is a conceptual rendering of the paper's coverage model. Drag to rotate the space; select a stage to replay it.

terrain geometry

dynamics

task complexity

1 path

01 · Difficulty space

A monotone path is a curriculum.

Each coordinate is a difficulty axis. A valid path advances through local parameter increments or complexity-increasing edits.

Algorithm

PATH combines two allocation regimes.

A lightweight switch preserves fast early exploration and reserves the more expensive active allocation for the residual regime.

PATH:Random

Grow coverage with diverse curriculum paths.

Maintain parallel pointers in the curriculum graph. Sample pointers uniformly, train on their current environments, and advance through a randomly chosen successor once the mastery threshold is met.

Selection: Uniform over path pointers
Role: Low-overlap early coverage

PATH:Active

Concentrate paths where learning potential remains.

Replace uniform allocation with positive-value-loss weighting. When a prioritized pointer becomes mastered and advances, launch additional pointers into its successor region and evict low-weight entries from the fixed-size buffer.

Selection: Positive-value-loss replay
Role: Residual-region refinement

Acquisition signal

Positive value loss prioritizes a productive frontier.

PATH uses the positive value loss score from PLR as a practical measure of current learning potential. A large score means that a rollout was better than the critic predicted across several consecutive steps, so PATH raises that pointer's replay priority. A separate mastery event then triggers progression: once recent return crosses the threshold, PATH advances the pointer and launches paths through its successor region. The score is not raw GAE: it retains the positive part of each GAE estimate and averages it across the rollout.

\begin{aligned} \delta_t^\theta &= r_t^\theta+\gamma V(s_{t+1}^\theta)-V(s_t^\theta),\\[4pt] S(\theta) &= \frac1H\sum_{t=0}^{H-1} \left[ \underbrace{\sum_{\ell=0}^{H-t-1}(\gamma\lambda)^\ell \delta_{t+\ell}^{\theta}}_{\widehat A_t^{\mathrm{GAE}}(\theta)} \right]_+,\qquad [x]_+=\max\{x,0\}. \end{aligned}

\begin{aligned} S(\theta)\ \text{large} &\quad\Longrightarrow\quad \text{raise replay priority},\\ \overline R_\theta\ge\varepsilon_{\mathrm{es}} &\quad\Longrightarrow\quad \text{advance and expand into }\mathrm{Succ}_{\mathcal G}(\theta). \end{aligned}

Detect useful surprise

The positive part isolates better-than-predicted outcomes, making a sudden improvement visible without treating every large negative error as useful.

Use multistep evidence

GAE combines temporally adjacent TD residuals through $γλ$ , so the score reflects more than a single noisy reward while retaining local responsiveness.

Reuse PPO computation

The estimator is already produced during PPO training. It therefore acts as a lightweight acquisition score rather than requiring a separate teacher, uncertainty model, or environment embedding.

This is a replay priority rather than a calibrated uncertainty estimate. PATH uses it to rank where more training is currently promising; the mastery threshold, not a high score by itself, releases the successor tree. Formula follows positive value loss in PLR; GAE follows Schulman et al.

Coverage view

Why random paths eventually saturate

A visited node can generalize to an unknown subset of its local graph neighborhood. Early paths encounter largely new neighborhoods. Later paths overlap what has already been mastered, so their expected marginal coverage declines.

Path coverage

C(P) = ⋃_u∈S(P) G(u)

Conceptual coverage versus budget curves for PATH and its two stages — Conceptual comparison from the paper. PATH switches from random exploration to active allocation at $T$ .

Empirical evidence

Robustness and held-out generalization

Experiments cover sparse-reward discrete control, dense-reward continuous control, and robustness across distinct families of held-out environments.

Animated blue MiniGrid agent navigating a fixed 15 by 15 wall-block level toward a green goal — **MiniGrid.** A scripted shortest route through an illustrative 15×15 training-family layout. The page exposes the full grid; the policy receives only a 5×5 egocentric observation. The DAG shows one small, non-exhaustive slice of the full layout graph.MiniGrid

Animated BipedalWalker agent traversing repeated pit obstacles in a Hardcore terrain — **BipedalWalker.** A fixed-seed heuristic rollout traverses a Hardcore course with visible pit obstacles rather than flat terrain. The DAG illustrates two of the environment’s difficulty axes with the terrain seed fixed.Gymnasium environment

These clips illustrate the tasks and are not PATH evaluation rollouts. The 15×15 clip uses MiniGrid rendering and a scripted shortest route through a fixed wall-block level from the training family; the Hardcore rollout was rendered from the official Gymnasium environment with a fixed seed and its built-in heuristic controller. Gymnasium is distributed under the MIT License, and MiniGrid under Apache 2.0.

BipedalWalker robustness return curves across five terrain classes — BipedalWalker robustness tests during training. Curves show mean ± standard error over five seeds, evaluated every 1k PPO updates.

MiniGrid sampling complexity and training return curves — MiniGrid sampling complexity and training return. PATH advances steadily through increasingly complex layouts; curves summarize five training seeds.

Radar plots of BipedalWalker terrain coverage — Terrain parameters for solved BipedalWalker environments from one run of each method.

Held-out tasks

Generalization at the final checkpoint

Values reproduce Tables 1, 2, and 6 of the current paper.

Benchmark	Metric	DR	PLR	ACCEL	PATH
MiniGrid	Solved rate	0.950 ± 0.0	0.977 ± 0.0	0.976 ± 0.0	0.994 ± 0.0
BipedalWalker	Mean return	9.94 ± 5.75	15.39 ± 2.84	18.14 ± 5.31	42.95 ± 1.31
BipedalWalker	Solved rate	0.01 ± 0.00	0.02 ± 0.00	0.04 ± 0.00	0.11 ± 0.01
Procgen Leaper	Solved rate	0.57 ± 0.13	0.45 ± 0.08	0.51 ± 0.16	0.66 ± 0.04

MiniGrid: solve rate over 1,000 uniformly sampled held-out layouts after 20k PPO updates.

BipedalWalker: deterministic evaluation on 1,000 held-out terrains sampled from the continuous parameter space, reported as mean return and solved rate after 20k updates.

Procgen Leaper: solve rate over 100 held-out environments after 1,000 PPO updates. Every entry is a mean ± standard error over five training seeds.

Takeaway

Explore the curriculum broadly while paths are informative. Refine it actively once random exploration begins to repeat itself.

Active path acquisitionCurriculum DAGsRobust reinforcement learning