Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
docs: add Q67 sample report under example-reports/ (#21)
Jordan Gibbs28e574fda16e
diff --git a/example-reports/rl-exploration-trajectory-planning.md b/example-reports/rl-exploration-trajectory-planning.mdnew file mode 100644index 0000000..9401be3--- /dev/null+++ b/example-reports/rl-exploration-trajectory-planning.md@@ -0,0 +1,465 @@+# Sample report — RL exploration under sparse rewards and constraints++> **Prompt** (verbatim):+>+> > Summarize recent research progress in reinforcement learning focused on enabling agents to explore efficiently and proactively under conditions of sparse rewards and constraints, respectively. Additionally, analyze and discuss the potential implications and insights this research provides for trajectory planning problems.+>+> **Generated by** `/hyperresearch` running the full 16-step pipeline. Score: **58.3 / 100** on DeepResearch-Bench RACE (partner-aware Gemini-2.5-Pro pairwise judge). Wall-clock: ~2 hours. The agent fetched and indexed 80+ academic sources across Semantic Scholar, arXiv, OpenAlex; ran two parallel loci-analysts to identify the dialectical questions where evidence forks; spawned K depth-investigators that each committed to a position; reconciled their committed positions into a tensions document; drafted three angle-specific reports in parallel; synthesized them; ran four adversarial Opus critics; applied their findings as surgical Edit hunks via a Read+Edit-locked patcher; and finished with hygiene + readability passes.+>+> **Citations** are numbered `[N]` inline with a `## Sources` section at the end. Some sources resolve to titles only because their URLs were fetched via direct file pulls (academic PDFs from arXiv/JMLR/etc.) rather than canonical landing pages.++---++# Reinforcement Learning Exploration under Sparse Rewards and Constraints — and What It Means for Trajectory Planning++## 1. Framing: Two Exploration Regimes Joined by One Word++The prompt couples "efficient" with "sparse rewards" and "proactive" with "constraints," and the adverbial pairing carries more weight than rhetoric suggests. **Efficient exploration under sparse rewards is a coverage problem.** When extrinsic feedback is rare, the dominant question is how to reach states the agent has never seen, since the reward signal cannot do that work itself. The literature's response has been to manufacture artificial signal — intrinsic motivation, novelty bonuses, world-model uncertainty, hindsight relabeling — so that local optimization remains informative even when the extrinsic reward sits at zero for thousands of steps [1] [2] [3].++**Proactive exploration under constraints is a representation problem.** When the agent must satisfy hard constraints from the start, the dominant question is how to anticipate the constraint surface before colliding with it. The literature's response has been to inject geometric and dual-variable structure — Lagrange multipliers, control barrier functions (CBFs, smooth functions whose level sets enforce forward invariance of a safe set), projection layers, and manifold-constrained samplers — so that exploratory actions are shaped by constraint geometry rather than punished after the fact [4] [5] [6] [7].++The two problems share a vocabulary but their failure modes diverge. Sparse-reward methods fail at the noisy TV — a stochastic transition the predictor can never master because its irreducible aleatoric entropy (the conditional entropy H[X_{t+1} | x_t, a_t] of the next observation given the current state-action) is positive [1] [8]. Constrained methods fail when integral-only Lagrangian dynamics oscillate around the constraint boundary [5] or when the policy class itself cannot represent the bifurcated map a non-simply-connected obstacle field requires [9]. Yet at the algorithmic level the two streams are converging on the same primitives: world models, ensemble or distributional uncertainty, optimism-on-reward paired with pessimism-on-cost, and policy classes — increasingly diffusion-based or mixture-based — that absorb multi-modality and constraint-respecting structure into the action distribution itself [10] [11] [12].++The thesis this report defends is that the joint maturity of these two literatures has reshaped what trajectory planning should look like. Continuous-policy model-free safe RL — the workhorse Constrained Markov Decision Process (CMDP, an MDP augmented with cost constraints) family of CPO, PPO-Lagrangian, and PID-Lagrangian — is doubly inadequate for safety-critical trajectory planning. The design space the corpus actually supports is a regime-conditioned composition: a multi-modal proposer (a diffusion planner or a mixture policy) layered above a geometric projector (Acting on the Tangent space of the Constraint Manifold (ATACOM) for first-order analytic constraints, or joint denoising over both a model-free score and a model-based safety score for second-order or learned constraints), with end-to-end pure-RL planners (CarPlanner, CaRL) visibly closing aggregate metrics under controlled compute but persistently trailing on safety sub-metrics by 0.7-1.5 points. The rest of the report builds the case.++## 2. Efficient Exploration under Sparse Rewards++### 2.1. Why the canonical baselines fail++Five years of work treat one benchmark — Montezuma's Revenge — as the stress test for hard exploration. Standard ε-greedy and entropy-bonus exploration cannot beat human scores on it [1] [2]. The field has been organized by two complementary surveys:++- **Ladosz–Weng** identifies seven categories in deep RL — count-based, prediction-error, diverse-behavior, goal-based, probabilistic, imitation-based, and random-based — and notes that long-horizon credit assignment and global-versus-local exploration remain unresolved [2].+- **Aubret** collapses intrinsic motivation into three families: surprise (rewarding prediction error), novelty (rewarding low visitation density), and skill-learning (rewarding mutual information between latent skills and trajectories) [3].++Every subsequent advance is judged on whether it cracks a hard sparse-reward task without breaking on stochasticity.++### 2.2. Prediction-error intrinsic motivation: RND, ICM, BYOL-Explore++Random Network Distillation (RND) provides the cleanest prediction-error mechanism *of its era*. The intrinsic reward is the mean-squared error between a trainable predictor network and a frozen randomly-initialized target network on the current observation [1]. By construction the target is a deterministic function of its inputs, neutralizing the noise floor that traps stochastic forward models. RND scores 8,152 on Montezuma's Revenge against the 4,753 average human score, exploring 22 of 24 first-level rooms — the first method to exceed human performance without demonstrations or game-state access. The architectural details matter: dual value heads (γ_I = 0.99 for intrinsic, γ_E = 0.999 for extrinsic), non-episodic intrinsic returns, and 128 parallel environments. PreND [13] later diagnosed two structural RND failure modes — weak correlation (~0.39) between RND prediction error and actual state-embedding distance, meaning the bonus is poorly aligned with semantic novelty; and rapid intrinsic-reward collapse from predictor over-fitting to the frozen target — and showed that ResNet-50 pre-trained backbones plus a slower predictor learning rate outperform RND in low-data regimes (Atari Boxing/Riverraid). FARCuriosity [14] separately showed that RND's intrinsic reward for *the same starting observation* in Jamesbond *increases* across 100M frames, identifying within-episode catastrophic forgetting in the curiosity module itself.++Intrinsic Curiosity Module (ICM) preceded RND with an inverse-dynamics feature space that filters stochasticity-only changes and uses forward-model error as the curiosity signal [15]. BYOL-Explore unifies world-representation, dynamics, and exploration policy into one bootstrapped latent loss; it solves the majority of DM-HARD-8 tasks where prior methods required human demonstrations and reaches superhuman scores on the ten hardest Atari exploration games [16]. The architectural progression — from prediction-error on raw observations to latent prediction error to imagined-future disagreement — is a steady reduction in dependence on hand-engineered signals.++### 2.3. World-model exploration: Plan2Explore, DreamerV3, Optimistic World Models++A parallel program treats sparse rewards by training a world model (a learned latent simulator of dynamics and rewards) and letting the agent imagine. The lineage starts with PlaNet [17] (latent-state planning via Cross Entropy Method over a learned RSSM) and DreamerV1 [18], the first Dreamer agent to solve long-horizon tasks from images purely by latent imagination — establishing the imagination-based actor-critic that DreamerV3, Plan2Explore, OWMs, DreamerV3-XP, and InDRiVE all inherit. MAX (Model-based Active eXploration, Shyam et al.) is the historical precursor to Plan2Explore that proposed utility-driven trajectory sampling against an ensemble-disagreement signal; we do not cover MAX directly but flag it as the spiritual predecessor of the Plan2Explore latent-ensemble formulation. Plan2Explore uses prospective latent disagreement — expected future novelty estimated by an ensemble of one-step predictors — to drive exploration without any extrinsic reward; on 20 DMControl tasks it nearly matches a reward-supervised oracle in zero- and few-shot transfer [19].++DreamerV3 [10] is the load-bearing workhorse: a single Recurrent State Space Model (RSSM) with fixed hyperparameters outperforms domain-specific algorithms on more than 150 tasks across seven domains (Atari, ProcGen, DMLab, BSuite, Minecraft, proprioceptive control, visual control), and is the first algorithm to collect Minecraft diamonds from scratch under sparse rewards. Larger models score higher and require fewer environment steps; the scaling is predictable. DreamerV3 itself notes, however, that its standard entropy-based exploration is *insufficient* in the hardest sparse-reward settings.++Three augmentations have been needed to close that gap. Optimistic World Models (OWMs) translate the classical Reward-Biased Maximum Likelihood Estimation (RBMLE) principle from 1980s adaptive control into deep RL via a gradient-based optimistic dynamics loss, achieving 152.68% mean Human-Normalized Score on Atari100K versus DreamerV3's 97.45% — a 55% relative improvement, plug-and-play with both DreamerV3 and STORM [20]. DreamerV3-XP layers prioritized replay (return + VAE error + critic error) on an ensemble-disagreement intrinsic reward [21]. CEE-US replaces the monolithic world model with an object-centric Graph Neural Network (GNN) ensemble, achieving zero-shot stacking, flipping, pick-and-place, and throwing through Model-Predictive Control over the learned model — confirming that the right inductive bias converts intrinsic-motivation exploration into a compositional capability rather than a free-play wander [22]. Curious Replay raises DreamerV3's Crafter score from 14.5 to 19.4 (+34%) by replacing uniform replay with curiosity-prioritized replay, identifying uniform replay as the root cause of Dreamer's adaptability failures [23].++### 2.4. The noisy-TV three-way fight++The residual stochasticity problem produced a structurally important confrontation. **Aleatoric Mavor-Parker Augmentations (AMA)** directly address action-dependent noisy TVs by learning a heteroscedastic prediction network that outputs both mean and variance, subtracting the trace of the variance from the intrinsic reward [24]. **Curiosity-in-Hindsight (CiH)** goes further: it proves that prediction-error curiosity converges to the conditional entropy H[X_{t+1} | x_t, a_t] rather than to zero, and adds a hindsight representation that drives the intrinsic reward to zero in the limit [8]. **BYOL-Hindsight**, the CiH instantiation, achieves state-of-the-art on sticky-action Montezuma's Revenge.++Learning Progress Monitoring (LPM) attacks the same problem from the opposite direction by sidestepping aleatoric estimation entirely. LPM rewards model *improvement* across update iterations, not error magnitude. For an unlearnable transition the model cannot improve, so the reward collapses to zero immediately, no Gaussian-noise estimator required. LPM dominates aleatoric methods on every directly comparable domain, with a 12.4% stochastic-coverage drop on continuous MountainCar versus 33-86% for AMA/IDF/ensemble/EDT baselines, ~150 vs ~400 steps to converge intrinsic reward to zero on Noisy MNIST, 1,347.6 average MiniWorld visited states (best, +95.3 over second-best), and 20M-step Montezuma's Revenge to meaningful extrinsic reward where RND requires 50M and NGU fails entirely; under Atari action noise, LPM shows 3.9-4.7% drops while EME — the strongest clean performer — collapses 100% [25]. LPM is the most sample-efficient noisy-TV fix on the discrete/low-dim continuous benchmarks it directly tested.++But its theoretical guarantee assumes i.i.d. Gaussian observation noise, which does not hold in autonomous-driving sensor regimes (lidar dropouts, multimodal returns, temporally correlated noise); CiH's causal-model framework provides architectural generality (drop-in for any prediction-error method, any representation space), an information-theoretic identification of the hindsight target Z_{t+1}, and is the only baseline in the corpus achieving SOTA on sticky-action Montezuma while preserving non-sticky performance — operative reasons why the high-dim winner is not yet decided. The LPM group's own CDRM companion paper acknowledges AMA's diagonal-Gaussian assumption is broken for multimodal sensor distributions, achieving AUROCs of 0.8876 (aleatoric) / 0.9981 (epistemic) on a mixed test set with arbitrary output distributions including multimodal ones — a structural admission that the LPM/AMA family's noise model is wrong for AD-class sensors. **The fight is structurally unsettled at the high-dimensional continuous-control scale that trajectory planning lives in** — neither LPM nor CiH has been head-to-head benchmarked there.++### 2.5. Goal-conditioning, archives, skill discovery, and the foundation-model frontier++Hindsight Experience Replay (HER) relabels failed trajectories with achieved goals; sparse binary rewards with HER outperform shaped rewards without HER, and DDPG+HER reaches near-100% success on robotic tasks where vanilla DDPG fails entirely, with five-of-five sim-to-real trials [26]. Go-Explore identifies *detachment* (forgetting how to reach productive frontiers) and *derailment* (stochastic perturbations preventing reliable return) as the failure modes of intrinsic-motivation methods, resolving both with an explicit archive plus deterministic return-then-explore — scoring above 43,000 on Montezuma's Revenge (about four times prior SOTA) and being the first algorithm to score above zero on Pitfall [27] [28]. Go-Explore's Phase-1 deterministic "return-to-archived-state" assumption is unrealistic for real-robot trajectory planning (no emulator save/load); the Nature follow-up [28] added goal-conditioned return policies for stochastic environments, but real-world transferability of the archive paradigm to trajectory planning remains substantially weaker than the headline numbers suggest. Intelligent Go-Explore replaces all heuristic components with foundation-model queries and beats Reflexion on multiple benchmarks [29].++Diversity Is All You Need (DIAYN) learns walking, jumping, and backflipping in MuJoCo without any extrinsic reward by maximizing mutual information between latent skills and visited states [30]; Contrastive Intrinsic Control (CIC) achieves 1.79× over prior competence-based methods on the Unsupervised RL Benchmark [31]. Two further canonical methods divide the skill-discovery axis: APT (Active Pre-Training, Liu & Abbeel) maximizes particle-based KNN entropy of state visitations as the unsupervised-pretraining objective, and SMM (State Marginal Matching, Lee et al.) frames a unifying objective behind the family — match a target state-marginal distribution rather than maximize MI directly. The three carve the design space into mutual-information (DIAYN), entropy-maximization (APT), and marginal-matching (SMM) — the corpus's CIC and CeSD notes survey these as substring references; we do not cite them directly. DISCERN (Discriminative Embedding Reward Networks, Warde-Farley et al.) is a goal-conditioned counterpart to HER, using a discriminative reward learned over goal embeddings rather than HER's relabeling — also outside the scope of this review; we name it as a representative method the surveyed work does not reach.++The foundation-model frontier has added a layer of abstraction. LLM-Driven Intrinsic Motivation outperforms VAE-only on MiniGrid DoorKey-8x8 by combining VAE KL-divergence with LLM direct-prompting reward in A2C [32]. IMAGINE brings trajectory-aware sequence-level intrinsic rewards to RL fine-tuning of LLM reasoning, achieving +22.23% on AIME 2024 with strongest gains on the hardest benchmarks [33]. The frontier is real, but its load-bearing relevance for trajectory planning is currently low: success domains are reasoning, code, and discrete games — not continuous-control safety. Decision Transformer reframes offline RL as conditional sequence modeling, autoregressively generating actions conditioned on a desired return-to-go with no temporal-difference learning; trained only on random walks on a directed graph, it recovers optimal shortest-path trajectories at test time when conditioned on a high return-to-go [34]. The architecture is, in effect, a trajectory-conditional planner — suggestive but not yet validated on continuous-control trajectory planning, since the shortest-path result is on a synthetic directed graph with TD-free conditional generation; the boundary between "learn a policy" and "plan a trajectory" is dissolving from the policy side as well as the diffusion side, but the empirical anchor for trajectory planning specifically is not yet in the corpus (consistent with 5.6).++### 2.6. The transfer caveat that matters for trajectory planning++The intrinsic-motivation-pretraining literature has a structural reliability problem any naive trajectory-planning recommendation must surface. Plan2Explore on DMControl, CIC and CeSD on URLB [35], and InDRiVE on CARLA [36] all transfer cleanly because pretraining and task environments are structurally homologous. But Change-Based Intrinsic Motivation (CBET) layered onto DreamerV3 is *detrimental* in MiniGrid while beneficial in Crafter [37]; Plan2Explore in Curious Replay's object-interaction assay fails to re-orient toward novel objects the way animals do, due to uniform replay diluting the novelty signal.++Intrinsic-motivation pretraining is reliable only when (i) pretraining and task environments share dynamics, (ii) the world-model architecture encodes causally relevant structure — operationally, monolithic RSSM world models (DreamerV3-class, InDRiVE-class) lack the relational structure that CEE-US's object-centric GNN provides, and this matters for any AD planner reasoning about agent-object interactions; this is the most operationally relevant predictor for AD trajectory planning — and (iii) replay is curiosity-prioritized. Outside this envelope, transfer is contingent and can actively harm performance. InDRiVE achieves 64.52% success rate in Town01 versus 64.06% in Town02 zero-shot, and 96-100% few-shot success at 51× fewer fine-tuning steps than DreamerV3 [36] — strong numbers *within* the structural envelope (sim-to-sim across CARLA towns sharing dynamics and architecture), but they cannot be promoted to a general sim-to-real claim.++## 3. Proactive Exploration under Constraints++### 3.1. The CMDP framework and its two structural axes++Constrained safe RL is dominated by the CMDP formalism: maximize expected return subject to expected cumulative cost being below a threshold [4] [38]. The two structural axes echo Garcia & Fernandez's earlier safe-RL taxonomy [39] of optimality-criterion modification versus exploration-process modification. The architectural choices fan out along two axes that together determine which algorithm wins under which conditions. The **policy-form axis** runs from continuous Gaussian policies (CPO, PPO-Lagrangian, PID-Lagrangian, RCPO, CRPO) through state-dependent Lagrangians (FAC, SRMPC) to multi-modal policies (Multimodal Policy Optimization (MUPO)) to action-projection layers (ATACOM, Safety Layer, CBF-QP) to full diffusion planners with safety scores (SafeDiffuser, JM2D, RAIL). The **dynamics-knowledge axis** runs from pure model-free (CPO, Lagrangian methods) through learned models with epistemic uncertainty (LAMBDA, CERL, DOPE) to known dynamics (CBF-QP, RMPS, ATACOM with analytical constraints).++These axes compound. The continuous + model-free quadrant fails representationally because no continuous map exists across distinct homotopy classes (equivalence classes of paths under continuous deformation) of obstacle-free space — the Policy Bifurcation theorem [9] — and it fails guarantee-class because no model-free CMDP method can promote expectation-bounded constraint satisfaction to almost-sure per-step satisfaction — the SCPO impossibility [40]. Both failure modes are theoretically rigorous, independent, and empirically anchored. They are the load-bearing argument for Section 5.++Risk-sensitive and CVaR-based RL constitute a third constraint regime distinct from CMDPs and CBFs: rather than bounding expected cost (CMDP) or enforcing forward invariance (CBF), the optimization penalizes tail risk. CPPO [41] formulates safe RL as max E[return] subject to −CVaR_α(−D(π)) ≥ β, achieving higher cumulative reward and stronger robustness than CMDP baselines on MuJoCo and providing a Variance-of-Future-Returns CVaR robustness bound; this is the cleanest in-corpus instance of probabilistic constraint formulation, with mechanistic ties to D-ATACOM's chance-constraint enforcement P(k(s_t)>0) ≤ η_c (Section 3.6). The corpus's coverage of risk-sensitive RL at the trajectory-planning interface is shallow (one direct paper plus chance-constraint extensions of D-ATACOM and the Safaoui drone planner); this is named explicitly as an open problem in Section 6. Constrained policy optimization with chance constraints is the probabilistic generalization of CPO that bounds P(constraint violation) ≤ α rather than E[cost] ≤ d, and bridges directly to chance-constrained MPC in trajectory planning (Section 5).++### 3.2. CPO versus Lagrangian, and where formal theory pays off++Within the continuous + model-free quadrant, the field's most-litigated comparison is Constrained Policy Optimization (CPO)'s trust-region machinery against simpler Lagrangian variants with PID damping. CPO provides a per-iteration near-constraint guarantee with worst-case violation bounded by sqrt(2·δ·γ·ε)/(1−γ)² [4]. Reward Constrained Policy Optimization (RCPO) handles both discounted-sum and mean-value constraints via multi-timescale Lagrangian updates with almost-sure convergence under the assumption that every local minimum of the constraint function is feasible — an assumption the paper itself flags as a hidden weakness in non-convex settings [42].++Standard Lagrangian dual ascent acts as integral-only control on the constraint, producing 90-degree phase-shift oscillations the algorithm only recovers from at convergence [5]. PID-Lagrangian fixes this by adding proportional and derivative terms to the multiplier update. Empirically, simple PPO-Lagrangian and TRPO-Lagrangian with PID damping match or beat CPO across the Safety Gym and GUARD benchmarks [43] [44]. CUP [45] is a tighter-bound, first-order alternative to CPO/PCPO/FOCOPS that extends the surrogate bounds to GAE, derives tighter difference bounds, and avoids Fisher-information-matrix inversion via standard first-order optimizers. The Spoor et al. systematic study (2,480 runs across 8 Safety-Gymnasium tasks, ~57,000 CPU core-hours) goes further: no single update mechanism consistently satisfies constraints, and on PointGoal1 the PID update yields cost 27.47 (violating the limit of 10.0) while a fixed-λ baseline achieves 8.49 [46]. CPO's trust-region machinery is a research artifact whose theoretical guarantees do not translate into proportionate empirical wins; the design budget for trajectory planning should not be spent re-implementing it.++### 3.3. State-wise constraints, the impossibility result, and the "zero violation" disentangling++State-wise Constrained Policy Optimization (SCPO) is the formal indictment of expectation-based safety [40]. SCPO addresses *instantaneous* (per-step) constraints rather than cumulative ones via a Maximum Markov Decision Process augmentation tracking running maximum cost. Its most consequential claim is the impossibility result: it is impossible to guarantee hard state-wise safety during training without prior knowledge of the dynamic model. SCPO itself acknowledges that even its own theorems are "valid only in expectation."++This is doubly damning when read alongside the "zero violation" claims of AutoCost, the proactive variant of PCPO, and CRLLK. AutoCost's evolutionary search for an intrinsic cost function reports "zero violation across all Safety-Gym tasks" — but only as expected-cumulative-cost = 0 at convergence [47]. AutoCost itself defends the framing by claiming "strictly satisfying a zero cost limit in expectation is actually equivalent to state-wise safety," which is mathematically correct only under the conjunction of three conditions — C(s,a) ≥ 0 (indicator costs are non-negative), E[C(s,a)] = 0 holds at every state under the *stationary* distribution, and the equivalence is asymptotic — under which C(s,a) = 0 almost surely under the stationary distribution. Neither condition is checked by the benchmark; the equivalence is operationally weaker than its rhetoric implies. CRLLK reports J_c_coll = 0.05 ± 0.01 in 100 test episodes and uses "zero violation" as a *cost-limit specification*, not an empirical outcome [48]. PCPO's preemptive log-barrier penalty proves a cumulative violation bound V_P(T) ≤ V_L(T) − Δ(T) growing to infinity versus Lagrangian — but this is again expectation-asymptotic, not almost-sure per-step [49].++The Wachi-Shen-Sui survey makes the architecture explicit: T-class algorithms (safety guaranteed *during training*, almost-surely) require instantaneous constraints, and "virtually no existing open-source implementations support" the relevant problem classes [38]. Every method labeled "zero violation" lives in the C-class (convergence-only, expectation-bounded) world. The Eckel et al. critique completes the picture by attacking the metric itself: expected cumulative cost is the wrong measure, and the Expected Maximum Consecutive Cost (EMCC) ranks algorithms differently because two trajectories with equal cumulative cost can have radically different exploration risk profiles [50]. The "zero violation" rhetoric should be re-read as "expected cumulative cost approximately zero at convergence under the stationary distribution" — defensible technically, misleading operationally.++### 3.4. Empirical anchors: Safety Gym, Safety Gymnasium, OmniSafe, GUARD++The benchmarks make the theoretical results visible as numbers. **Safety Gym** defined the canonical CMDP benchmark with 18 high-dimensional continuous-control environments and randomized episode layouts; PointGoal2 cost rate is 5-20% of timesteps for unconstrained PPO and CPO normalized cost on SG2 is 0.428 (failing) [43]. **Safety Gymnasium** extends to 54 environments, confirming that on Level-2 navigation tasks 9 of 40 have optimal policies that violate constraints for the majority of algorithms; unconstrained PPO's episodic cost on AntVelocity is 38.33× the cost limit while PPO-Lagrangian holds it to 0.22× [51]. **OmniSafe** reports SafetyPointGoal2-v0 unconstrained cost of 204.96 against a cost limit of 25 — eight times over [52]. **GUARD** tests 8 algorithms across 11 robots and confirms that current safe RL algorithms are hard to drive to zero-violation performance even in benchmark environments [44]. The metric called "safety" is satisfied in expectation across simply-connected, low-density geometries and *systematically violated* on the dense, non-trivial geometries that real trajectory planning must handle.++### 3.5. The optimism-pessimism design template: DOPE, LSVI-AE, CERL, COX-Q++A distinct theoretical strand exploits one shared uncertainty estimate deployed in opposite directions: optimistic on reward, pessimistic on cost. Doubly Optimistic and Pessimistic Exploration (DOPE) uses an optimistic model confidence set plus an optimistic reward bonus, plus a pessimistic constraint penalty added to constraint cost, solved via a joint linear program. DOPE achieves Õ(|S|·sqrt(|A|K)) regret with zero violations, improving the prior OptPessLP bound by sqrt(|S|) [7]. LSVI-AE (Linear State-Value Iteration with Adaptive penalty Exploration) proves a counterintuitive theorem: aggressive early exploration *reduces* later constraint violations because better state coverage yields more accurate cost-function estimation. LSVI-AE achieves first order-optimal regret Õ(H·sqrt(dK)) for hard instantaneous constraints without prior safe-action knowledge [53]. COX-Q is the off-policy match — the first off-policy safe-RL method to achieve constraint satisfaction during data collection comparable to on-policy methods, by combining cost-constrained optimistic exploration via Policy-MGDA gradient-conflict resolution with truncated-quantile critics, applied on top of the Optimistic Actor-Critic (OAC) [54] upper-confidence-bound exploration mechanism [55].++CERL layers a Bayesian world model with Log-Barrier Stochastic Gradient Descent, maintaining safety from training step 1 while baselines (CPO, LAMBDA) require approximately 1.5M training steps to satisfy constraints [11]. LAMBDA was the predecessor: Bayesian world models plus Augmented Lagrangian, beating CPO and PPO/TRPO-Lagrangian on Safety Gym SG6 [56]. MASE (Meta-Algorithm for Safe Exploration) [57] is a fourth optimism-pessimism instance whose Theorem 3.1 unifies cumulative, state-wise, and instantaneous CMDP problems into a single Generalized Safe Exploration form with time-varying instantaneous threshold b_h — a uniqueness result that bears directly on the Section 3.3 distinction between expectation-based and per-step constraints — and which couples any unconstrained RL algorithm with an uncertainty quantifier that penalizes the agent *before* violations occur, achieving no safety violations during training on Safety Gym. Safe-Dreamer [58] integrates Lagrangian safety into the DreamerV3 world model with two variants — OSRP using Constrained Cross-Entropy planning at each timestep and BSRP-Lag using a background safe actor — and achieves nearly zero-cost performance on Safety-Gymnasium where every model-free Lagrangian baseline fails. Both extend the optimism-pessimism template into the explicitly model-based safe-RL regime.++Two readings deserve precise calibration. The LSVI-AE "aggressive exploration reduces violations" result is a *time-horizon* argument: it accepts non-zero violations during early-exploration phases (consistent with SCPO impossibility) in exchange for asymptotically tighter constraint satisfaction (still expectation-bounded, not almost-sure). It cannot be repurposed as a deployment safety claim. For trajectory planning where every training rollout in a real robot has finite irreversible cost, CERL-style pessimism from step 1 is the operationally correct choice; LSVI-AE-style aggressive optimism is the right choice only when training is simulator-only and early violations cost only compute.++Second, the optimism-pessimism algorithms are *not* mechanistically unified at the architectural level — they share one uncertainty estimate but deploy it in computationally separate signals. DOPE constructs four objects: an optimistic model confidence set P_k from empirical Bernstein, an optimistic reward bonus that subtracts exploration incentive from the objective (Eq. 9), a pessimistic constraint penalty that adds uncertainty to the constraint budget (Eq. 8), and an LP over their Cartesian product. CERL's Eq. 4 has the form max_π J(π, p_θ) s.t. max_{p_θ_i ∈ P} J_c(π, p_θ_i) ≤ 0 — reward evaluated under one model draw (optimistic), constraint evaluated under the worst-case draw (pessimistic) — combined by a log-barrier gradient ∇J + η · ∇J_c,P / (−J_c,P). LSVI-AE's action-selection rule is argmax{Q − Z · g_+} with the dual Z and the UCB on g adapting from independent sources via Z_h^{k+1} = max(Z_h^k + (g_h^k)_+, η_h^k). The "doubly optimistic and pessimistic" framing is a unification of *signal*, not of *architecture*. Whether a single-mechanism alternative exists is a frontier question (Section 6).++### 3.6. Geometric mechanisms: ATACOM, CBF-RL, RMPS++A different lineage absorbs the constraint into the action geometry rather than the loss. ATACOM projects actions onto the tangent space of the constraint manifold defined by c(q) = 0 [6]. Any action sampled from this tangent space keeps the trajectory on the manifold by construction — equality and inequality constraints handled simultaneously, no manual backup policy required. The mechanism is shared with IMACS/AtlasRRT at the linear-algebra level: both compute the orthonormal basis of the constraint Jacobian's null space via SVD or QR — ATACOM calls this N_c (with J_c · N_c = 0, equation 4 of the ATACOM paper), IMACS calls it Φ(q) = ker(J(q)) — and both embed sampled tangent coordinates back to configuration space via the same linear map (ATACOM: [q̇, μ̇]^T = N_c · α; IMACS: q_u = q + Φ(q) · u). This shared null-space embedding is the load-bearing identity behind the ATACOM-IMACS tension in Section 4.4. The algorithm is compatible with any model-free RL backend (PPO, TRPO, DDPG, TD3, SAC) and requires C2-differentiable constraint functions plus either an accurate invertible dynamics model or a well-performing tracking controller. **Among the corpus's safe-exploration mechanisms, ATACOM is the only one that achieves *genuine* mechanistic unification of exploration and constraint satisfaction** — the constraint is absorbed into the action representation, leaving the RL objective unmodified.++D-ATACOM extends ATACOM with learnable constraints from data plus distributional RL for chance-constraint enforcement P(k(s_t) > 0) ≤ η_c, with a safety value function providing predictive estimation of future constraint satisfaction over entire trajectories [59]. The ATACOM TRO extended paper formally defines the Singular Set Y, proving it is disjoint from the Constraint Manifold M for first-order systems, so agents on M never encounter singularities; singular points are saddle points of the Lyapunov function rather than violations [60]. The crucial caveat from the same paper: **ATACOM does NOT guarantee safety for second-order systems with position-based constraints without modifying the constraint** — a load-bearing scope limitation that shapes the two-case split in Section 5.++Control Barrier Functions enforce forward invariance via the Lie-derivative condition L_f h + L_g h · u ≥ −α(h(x)) implemented as a quadratic program safety filter; the 2024 survey on learning CBFs catalogues the model-dependence problem and surveys data-driven, GP-based, SVM-based, and neural-network-based methods to synthesize CBFs from data [61]. CBF-RL introduces dual-mode training: a CBF safety filter applied during training rollouts plus a barrier-inspired reward; the policy internalizes safety and deploys without a runtime filter. On a Unitree G1 humanoid the dual approach achieves 92.7% obstacle-avoidance success without a runtime filter versus 38.7% for filter-only training, with zero-shot sim-to-real transfer [62]. Cohen and Belta propose Lyapunov-like CBFs that decouple safety from learning convergence: the CBF safeguarding controller guarantees forward invariance at all times, independent of value-function convergence [63]. Robust Model Predictive Shielding (RMPS) uses tube-based robust nonlinear MPC as a backup controller — the first framework handling both nonlinearity and stochasticity simultaneously [64].++Recovery RL [65] is the canonical example of *separating* the task and safety policies into two heads: a task policy plus a recovery policy that takes over when a safety critic Q_risk (initialized from offline constraint-violation data) exceeds a threshold; it outperforms five prior safe-RL baselines (2-20× efficiency in simulation, 3× on a da Vinci surgical robot). The mechanism inherits the Safety Critic (SQRL) formulation [66] of pre-training a task-agnostic Q_safe(s,a) that constrains fine-tuning. This contrasts with ATACOM's geometric absorption (constraint absorbed into action representation) and CBF-RL's internalization (constraint compiled into the policy via training-time filter): Recovery RL keeps task and safety as architecturally separate policies, the synthesized-shield template (Alshiekh et al.-style shielding from logical specs) translated into a learned safety critic. The CBF-RL result deserves precise calibration: it demonstrates that the runtime filter becomes optional *on tasks where the policy can internalize a single CBF*. The Unitree G1 task involves continuous obstacles; the multi-homotopy obstacle case (urban driving with central obstructions) has not yet been tested without a runtime filter.++## 4. Where the Two Streams Meet: Tensions and Convergences++### 4.1. Shared infrastructure, shared failure modes, shared design templates++Both literatures end at world models. The sparse-reward stream uses them for imagination-based exploration (Plan2Explore, DreamerV3, BYOL-Explore, CEE-US); the constrained stream uses them for safe Bayesian planning (LAMBDA, CERL, Cohen & Belta). Three convergences follow:++- **No-single-method generalization:** Spoor et al. show no Lagrangian update mechanism consistently satisfies constraints across tasks, echoing the world-model exploration finding that no single intrinsic-motivation method works across all environments.+- **Shared noisy-TV failure:** curiosity-driven exploration and confidence-set safety methods alike collapse when irreducible aleatoric noise masquerades as epistemic uncertainty.+- **Expectation-vs-trajectory dichotomy:** both streams share the same gap between expectation-based theoretical convenience and trajectory-level operational reality.++The Wachi-Shen-Sui T-class versus C-class taxonomy is the single best lens for understanding both literatures simultaneously [38]. Sparse-reward exploration is almost universally C-class — the agent accumulates failures during training in exchange for asymptotic competence. Constrained exploration is almost universally C-class as well; the celebrated "zero violation" claims of AutoCost, PCPO, and CRLLK are convergence-only properties. The few T-class algorithms — CERL "safe from training step 1," ATACOM by-construction safety (first-order systems with C2 analytic constraints only; see 5.3), CBF-RL with runtime filter active during training (its no-runtime-filter deployment claim is what earns its T-class status), MASE with no training-time violations on Safety Gym [57], Safe-Dreamer [58] — are the ones with any shot at deployment-grade trajectory planning. The taxonomy does not just describe the literature; it predicts which methods will transfer.++### 4.2. Hindsight and reachability are dual++HER relabels failed sparse-reward trajectories with achieved goals, bootstrapping a goal-conditioned policy from data the original policy generated by failing. RAIL (Reachability-Aided Imitation Learning) projects a pre-trained imitation-learning policy into a reachable safe set computed via polynomial zonotope swept volumes [67]. The two operations are formally dual: hindsight rewrites the labels to make the data informative; reachability rewrites the actions to make the policy safe. RAIL achieves 0% collision rate on all tested tasks while exposing that highest-performing imitation policies sometimes achieve their headline numbers *only by violating constraints* — a finding that retrospectively casts doubt on any unconstrained imitation-learning success metric.++### 4.3. Composition matters: joint denoising beats sequential post-hoc projection++A distinct architectural finding crosses both streams. JM2D (Joint Model-Based Model-Free Diffusion) directly benchmarks the *composition* of a diffusion proposer and a safety projector. Two architectures were compared: sequential (run diffusion to completion, then post-hoc project) and joint (jointly denoise over both score functions in a single coupled process). On D3IL-Avoiding Cluttered, joint denoising achieves SSucc = 0.75 with 0.32 violations; SafeDiffuser (sequential CBF projection) achieves SSucc = 0.01 with 22.92 violations; DPCC achieves SSucc = 0.0 with 6.52 violations [68] [69] [70]. Joint denoising delivers a 75× success-rate advantage and 72× fewer violations on this benchmark. The mechanism: sequential projection causes "trap behavior" because the diffusion proposer commits globally to a trajectory that collapses into infeasible regions, and the projection operator is local while the proposer's commitment was global. The 75×/72× figures come from a single CoRL 2025 paper on one task family; the structural prescription should be read as "embed safety in the denoising loop, not after it" — corroborated mechanism-differently by DDAT [71], which uses partial polytopic projections *during* diffusion for dynamic admissibility on a Unitree GO2 — rather than as a claim that JM2D's specific score-coupling architecture is the unique answer. Section 5 makes the prescription concrete.++### 4.4. Seven cross-cutting tensions++Seven tensions cut across the literature and bear directly on the recommendation in Section 5. Each is named explicitly with a committed resolution.++**Tension 1 — Continuous-policy safe RL: representational adequacy versus structural inadequacy.** *Side A:* CPO/PPO-Lagrangian/PID-Lagrangian are widely deployed and report strong empirical safety on Safety-Gym/GUARD; the field treats them as the practical standard [4] [43] [44]. *Side B:* Policy Bifurcation proves no continuous policy can satisfy constraints in non-simply-connected configuration spaces (the default geometry around any central obstacle) [9]; SCPO proves model-free CMDP cannot guarantee almost-sure per-step safety during training without a dynamics model [40]; CRLLK's own collision cost J_c_coll = 0.05 (not zero) confirms the expectation-bounded reading [48]. *Resolution:* Side B is the load-bearing reading for safety-critical trajectory planning. The two failure modes are independent; fixing one does not fix the other. The slope-of-progress observation — that stochastic Gaussian policy *samples* span homotopy classes even when expectations do not — is an active research thread, not a refutation: the safety guarantee in CMDP literature is about expected cost, which inherits the bifurcation problem. RLTT [72] is direct empirical evidence that the homotopy-class-enumeration workaround is field-validated on autonomous-vehicle motion planning (19.9% distance-cost reduction).++**Tension 2 — ATACOM convergence from independent paths.** *Side A (implementation viability):* ATACOM's null-space tangent projection is formally identical to AtlasRRT/IMACS continuation-based sampling — both compute ker(J(q)) via SVD/QR — and ATACOM's constraint functions plug directly into OMPL's Constraint interface [6] [73]. *Side B (architectural cleanliness):* Among DOPE/LSVI-AE/CERL/ATACOM, only ATACOM achieves single-mechanism unification; the others share an uncertainty estimate but deploy two computationally separate signals. *Resolution:* Convergence from independent paths. ATACOM is the first-case answer in the two-case split, with the proviso that C2-differentiability excludes learned non-differentiable constraints, second-order systems with position constraints require constraint modification [60], and no paper has yet implemented ATACOM inside OMPL and benchmarked the composition.++**Tension 3 — CPO trust-region machinery vs simple Lagrangian damping.** *Side A:* CPO has formal per-iteration constraint guarantees [4] [42]. *Side B:* Empirically, PPO-Lagrangian and TRPO-Lagrangian with PID damping match or beat CPO across Safety-Gym and GUARD; CPO has practical infeasibility-fallback failures [43] [5] [46]. *Resolution:* Side B has the empirical edge. The design budget should be spent on the topology and model-knowledge problems, not on re-implementing trust-region machinery.++**Tension 4 — Diffusion planning vs diffusion policy: domain split, not a winner.** *Side A:* Diffusion-as-planner systematically outperforms diffusion-as-policy on sparse-reward long-horizon tasks per the 6,000-model Diffusion Veteran study [74]. *Side B:* Diffusion Policy achieves +46.9% over prior SOTA on 15 manipulation tasks [12]. *Resolution:* Both correct under different scope. Planning-level wins for sparse-reward long-horizon (long credit-assignment chains, where compounding single-step errors dominate); policy-level wins for dense-feedback short-horizon manipulation. AD trajectory planning → diffusion planner; visuomotor manipulation → diffusion policy.++**Tension 5 — DV overturns Diffuser/AdaptDiffuser conventions.** *Side A:* Joint state-action diffusion with classifier guidance and U-Net backbones is the standard recipe per Janner Diffuser and AdaptDiffuser [75] [76]. *Side B:* The Diffusion Veteran 6,000-model systematic study overturns five conventions: Monte Carlo Sample Selection beats classifier guidance and classifier-free guidance with near-optimal data; Transformer beats U-Net on 8 of 9 sub-tasks; jump-step planning (stride > 1) is almost always better; inverse-dynamics action generation beats joint state-action; no scaling law (deeper not always better) [74]. *Resolution:* Side B is load-bearing because it rests on a much larger systematic ablation. Practitioners building diffusion-based trajectory planners in 2025-2026 should default to the DV recipe.++**Tension 6 — Hybrid RL+classical vs pure end-to-end RL.** *Side A:* Hybrid stacks systematically beat pure end-to-end RL or pure MPC for safe trajectory generation; Section 5.4 anchors this with SRMPC, SMART, HYPE, and the PDM-Closed family. *Side B:* CarPlanner is the first RL-based planner to surpass IL- and rule-based SOTAs on nuPlan; CaRL achieves CLS-NR = 91.3 vs PDM-Closed 92.8 with 500M samples; CBF-RL achieves zero-shot sim-to-real with 92.7% success without a runtime filter; under nuPlan-R's reactive agents, PDM-Closed drops 7.86 points on Test14-Hard while learning planners hold ground [77] [78] [62] [79]. *Resolution:* Pure RL is closing aggregate metrics under controlled compute but persistently losing safety sub-metrics by 0.7-1.5 points. PLUTO and Diffusion Planner only beat PDM-Closed by incorporating its rule-based post-processing [80] [81]. The hybrid stack remains the load-bearing recipe for safety-critical deployment; Side B's scaling argument is a real and unresolved counter-trend that may force re-evaluation within 24 months.++**Tension 7 — LSVI-AE "aggressive exploration reduces violations" vs CERL "safe from training step 1."** *Side A:* LSVI-AE proves first order-optimal regret with aggressive optimism; DOPE achieves Õ(|S|·sqrt(|A|K)) regret with zero violations [53] [7]. *Side B:* CERL maintains safety throughout training via Bayesian pessimistic world model and log-barrier SGD; baselines need ~1.5M steps to satisfy constraints [11]. *Resolution:* Different time horizons, not contradictions. LSVI-AE accepts early violations to learn the cost function then enforces in expectation later; CERL is a deployment-grade per-step argument. For real-robot training, CERL-style pessimism is operationally correct; for simulator-only training, LSVI-AE-style aggressive exploration is sample-efficiency-optimal. The two camps agree on the unifying template DOPE, LAMBDA, and CERL all instantiate: optimism-on-reward + pessimism-on-cost, simultaneously, in one optimization.++The seven tensions resolve to one architecture. Continuous-policy model-free safe RL fails along two independent axes. The cleanest single-mechanism fix is geometric projection, but it cannot enumerate homotopy classes alone. The Lagrangian family is empirically usable but theoretically inadequate. Diffusion planning is the right multi-modal proposer for sparse-reward long-horizon trajectory tasks, and the DV recipe specifies its design. The hybrid stack is the dominant architecture for safety-critical trajectory generation. The optimism-pessimism design template supplies the training-time mechanism that lets the proposer learn its constraints without breaking the robot.++## 5. Implications and Insights for Trajectory Planning++### 5.1. The two indictments compound — what falls out of the safety-critical stack++Standard CPO/PPO-Lagrangian/PID-Lagrangian uses a continuous Gaussian policy. Policy Bifurcation proves: when the obstacle-free configuration space is non-simply-connected (which it is whenever any obstacle creates a topological hole), no continuous policy is feasible across homotopy classes. The proof is topological, not algorithmic; it applies independent of learning algorithm, hyperparameters, or compute budget. The autonomous-vehicle motivating example is decisive — a continuous policy must interpolate between "steer left around the obstacle" and "steer right around the obstacle," and that interpolation is a direct collision at the boundary initial condition; the Geekplus M200 real-robot experiment confirmed this, with central initialization producing collision under continuous policy control [9].++Safety Gym geometry is non-simply-connected by construction: a bounded 2D arena minus circular hazards is topologically equivalent to a plane with punctures, containing loops that cannot be contracted to a point, with layouts re-randomized per episode to produce a *distribution* over topologies. Many initializations therefore trigger the bifurcation condition — which is precisely why the Section 3.4 violation-rate numbers look the way they do. The Policy Bifurcation paper itself notes that the failure manifold is narrow — initial states leading to violations have position deviations at sub-centimeter scale, on the order of ~1 cm wide around the obstacle centerline — so random per-episode initialization undersamples the manifold and the empirical *frequency* of bifurcation-triggering configurations on Safety-Gym Level-1/2 has not been directly measured; the indictment is structural (random sampling does not erase the failure mode) rather than a frequency claim, and for deployed planners that actively pursue central trajectories the tail-risk profile is not measure-zero. Add SCPO's impossibility, and the indictments compound:++- Adopting MUPO's mixture policy fixes topology but not model-knowledge.+- Adopting a Bayesian world model fixes model-knowledge but not topology. **Standard CPO-family methods are research artifacts that establish design intuition but cannot be lifted as the *sole* deployed motion planner without additional structure**, regardless of which benchmark scores they post — though RLTT [72] demonstrates a fourth architectural escape (developed in 5.2) in which a separate Topological Path Search enumerates homotopy classes and continuous-policy CRL optimizes inside each corridor, sidestepping the bifurcation condition without requiring multi-modal policies. This is the report's first load-bearing recommendation.++### 5.2. The three architectural escapes — complementary, not competitive++Policy Bifurcation names three resolutions: mixture/bifurcated policies (MUPO with Gaussian mixture), action-projection layers (ATACOM, CBF-QP, Safety Layer), and diffusion planners (multi-homotopy-class trajectory distributions with inference-time selection). A *fourth* escape — implicit in the hybrid recipe of 5.4 — is external topological decomposition: a separate planner enumerates homotopy classes and continuous-policy RL optimizes inside the chosen corridor. RLTT [72] is the in-corpus instantiation, with Topological Path Search providing macro-scale homotopy enumeration and constrained RL optimizing within each topological corridor, reducing distance cost by 19.9% on autonomous-vehicle motion planning; HYPE's MCTS proposal layer over discrete trajectory modes [82] is the AD-domain analogue.++These four are not alternatives. They handle different failure modes:++- **Multi-modal proposer** enumerates discrete homotopy classes — for a vehicle that must choose between "left around the obstacle" and "right around the obstacle," the proposer produces both candidates and an inference-time selector picks one.+- **Geometric projector** refines each candidate onto the constraint manifold for continuous constraint surfaces (joint limits, distance-to-obstacle inequalities).+- **Runtime filter** (CBF-QP, RMPS shielding) provides deployment-grade per-step safety for the irreducible online stochasticity that no proposer can perfectly anticipate. The MUPO paper concedes that diffusion is theoretically a superior multimodal approximator but criticizes its inference complexity — an objection rapidly weakening as DDIM and flow-matching reduce diffusion to under 10 denoising steps and Habi distills diffusion planners to over 800 Hz on laptop CPUs [83]; the Habi performance-preservation claim is on D4RL offline benchmarks (a posterior-to-prior variational distillation, fundamentally lossy by construction), not yet on closed-loop safety-critical AD evaluation, so the inference-cost objection is weakening *in benchmarks*, not yet definitively in production AD. No single mechanism resolves the topological problem; the practical recipe combines a multi-modal proposer with a geometric projector and a runtime filter.++### 5.3. The two-case split for the geometric projector layer++The geometric-projector layer admits a two-case split.++**Case 1: first-order systems with analytic C2 constraints — use ATACOM-style tangent-space projection.** Two independent lines of analysis land here. The implementation path: Kingston, Moll & Kavraki's IMACS framework decouples motion planner choice from constraint adherence method, supports projection-based methods (CBiRRT2) and continuation-based methods (AtlasRRT, TB-RRT) as orthogonal axes, and is implemented in OMPL's ConstrainedStateSpace, AtlasStateSpace, and TangentBundleStateSpace [73]. Tangent-space sampling in IMACS computes the orthonormal basis for the null space of the constraint Jacobian — exactly the operation ATACOM performs (J_c · N_c = 0, with N_c via SVD or QR). ATACOM's constraint functions plug directly into OMPL's Constraint interface, making BIT*/RRT*/PRM* with ATACOM constraints *architecturally* implementable today — but the slack-variable Jacobian J_c(q,μ) may lose rank at μ=0 (the constraint boundary itself), which is exactly the IMACS singularity failure mode AtlasStateSpace warns about; the bridge is available in principle but its numerical viability at safety-critical configurations is unverified, and the IiwaAirHockey μ=0 region is the natural test case (see Section 6). The architectural-cleanliness path — single-mechanism unification — already established in Section 3.6, points to the same recipe. McVAMP demonstrates that projection-based variants scale to 36-DoF Digit humanoids at 12 ms average, vectorizing the Newton projection step for 100-1000× speed-up over OMPL projection-based RRT-Connect [84] — projection-based handles the Jacobian rank issues that atlas-based runs into.++**Case 2: second-order systems, position-based constraints, or learned non-differentiable constraints — use joint denoising over both score functions à la JM2D.** ATACOM's TRO extended paper formally proves safety for first-order systems but shows the mechanism *fails for second-order systems with position-based constraints without modifying the constraint* [60]. D-ATACOM partially relaxes the differentiability requirement via projection-based IMACS but loses atlas-RRT compatibility. For these regimes, the JM2D finding from Section 4.3 is the structural prescription: jointly denoise over the model-free trajectory score and the model-based safety score, weighted by an interaction potential that prevents collapse into infeasible regions. SafeDiffuser's QP-per-denoising-step at O(q³) complexity per solve is the right *direction* — embedding safety in the denoising loop rather than after it — but JM2D's joint-distribution formulation is the right *implementation* of that direction.++### 5.4. The hybrid recipe in autonomous driving — and the closing-gap counter-trend++The closed-loop autonomous-driving benchmarks make the recipe falsifiable. PDM-Closed is the canonical hybrid rule-based baseline, achieving CLS-NR = 92 and CLS-R = 93 on nuPlan Val14 [81]. CaRL is pure PPO with 500M nuPlan samples and a simple route-completion reward, achieving CLS-NR = 91.3 versus PDM-Closed 92.8 and Col. = 97.4 versus PDM-Closed 98.1 [78]. CarPlanner is the first RL-based planner to surpass both IL- and rule-based SOTAs on nuPlan with CLS-NR = 94.07 [77]; PLUTO and Diffusion Planner only surpass PDM-Closed *with* its rule-based post-processing, trailing by 3-4 points without it [80] — Diffusion Planner reports NR=89.87 pure versus 94.26 with refinement [85]. RITP [86] is the cleanest in-corpus instance of the proposer-plus-numerical-refinement recipe: an RL action sampler generates S candidate trajectories, a TD3 critic evaluates them, and the RL-planned trajectory seeds an optimization-based refinement. CaRL's central methodological finding — that PPO fails to optimize complex shaped rewards as mini-batch size increases because the gradient becomes a noisy mix of trade-off terms, while a simple route-completion reward with multiplicative infraction penalties scales gracefully — is a constraint on RL design, not evidence that RL has solved trajectory planning.++The decisive reading: under controlled equal-compute, the hybrid stack retains approximately 1.5 CLS / 0.7 collision-metric advantage over pure end-to-end RL. HYPE integrates multimodal learned trajectory proposals into Monte Carlo Tree Search refinement with ego-conditioned occupancy prediction, achieving NR Collision = 0.020 and R Collision = 0.005 — the lowest of all evaluated planners on nuPlan dynamic scenarios [82]. SRMPC bridges the same recipe to motion control: a Safe RL policy generates a reference trajectory for LTV-MPC linearization at each step, enabling MPC to escape its linearization neighborhood and find different homotopy classes — over 30% better safety than RL alone, over 65% better than MPC alone in light traffic [87]. SRMPC's state-dependent Lagrange-multiplier head is inherited from Feasible Actor-Critic (FAC) [88], the first model-free CRL method to enforce per-state rather than expectation-only feasibility. SMART scales to 90 vehicles on 100m × 100m maps with hierarchical RL+convex-optimization [89]. Beyond Conservative AD couples MPC and RL in Highway-Env and reduces collisions 21% versus pure MPC, while showing that pure PPO transfers poorly while MPC variants generalize [90].++The Safaoui drone planner uses single-agent RL with a convex chance-constraint safety filter, enabling planner-agnostic deployment — RL or RRT can swap in [91]. UAV / aerial trajectory generation is the third application domain the prompt names; the layered-stack recipe maps to flight regimes asymmetrically — low-altitude obstacle-dense flight inherits the multi-homotopy reasoning of urban driving (proposer + projector + filter), while high-altitude long-horizon planning leans more on the diffusion-planner branch. The Safaoui chance-constraint filter is the in-corpus instance; the broader UAV-RL literature (Champion-level Drone Racing / Swift, Kaufmann et al. 2023, RL-MPC quadrotor stacks) is named here as adjacent context the recipe is designed to absorb. The Iterative Reward Prediction (IRP) CARLA study reports a 60.17% collision-rate reduction and 30.82× reward increase versus a control-command RL baseline, without an external safety filter, by using Kalman-filter-based uncertainty propagation and Minkowski-sum probabilistic collision bounds [92].++The pure-RL counter-trend deserves explicit acknowledgment. CaRL's scaling result, CarPlanner's headline win, and CBF-RL's "no runtime filter" Unitree G1 demonstration are real and not artifacts. V-Max [93] argues that a chunk of nuPlan's 2023 RL underperformance was an infrastructure gap rather than a structural RL limitation, providing the standardized RL framework the AD-RL community had been missing. The slope of progress visible in 2024-2026 is that pure end-to-end RL with internalized safety is closing the gap with hybrid systems faster than the cautious-conservative reading anticipated. For safety-critical deployment in *single-homotopy* regimes (continuous obstacle fields, no central obstructions creating discrete route choices), CBF-RL has already demonstrated that pure end-to-end RL with training-time CBF integration can deploy without a runtime filter on real hardware (Unitree G1, 92.7% success, zero-shot sim-to-real); the hybrid-recipe recommendation is therefore regime-conditioned on multi-homotopy or non-internalizable constraint geometry, not universal. The safety sub-metric advantage of hybrid stacks remains the load-bearing finding for safety-critical deployment in multi-homotopy regimes today, with the scaling counter-trend a real and unresolved threat to that recommendation within 24 months. The decisive experiment — a CaRL-class system at 5B samples with CBF-RL-style training-time safety integration, evaluated on nuPlan-R against PDM-Closed — has not yet been run.++| Trajectory-planning condition | Recommended composition | Why |+|---|---|---|+| Real-robot deployment, first-order dynamics, analytic C2 constraints | ATACOM tangent-space projection over a CBF-RL-trained policy that internalizes the constraint | Single-mechanism unification + eliminated runtime filter |+| Real-robot deployment, second-order dynamics, position-based constraints | D-ATACOM with learned distributional safety value + RMPS tube-based shielding | ATACOM does not guarantee safety here; D-ATACOM safety value provides multi-step lookahead; RMPS provides PAC backup |+| Non-simply-connected geometry (urban driving with central obstacles) | Multi-modal proposer (diffusion planner via DV recipe OR MUPO) + ATACOM-style projection per candidate + CBF-QP runtime shield | Continuous policies cannot span homotopy classes; multi-modal proposer enumerates options, projector refines, CBF guarantees forward invariance |+| Sparse-reward long-horizon offline RL (D4RL Maze2D / AntMaze / Kitchen) | Diffusion planner via DV recipe (state-only + inverse-dynamics action + Transformer + jump-step + MCSS) | DV's 6,000-model study shows planning-level diffusion dominates this regime |+| Dense-feedback short-horizon visuomotor manipulation | Diffusion Policy at the action level | Native multi-modal action distribution avoids mode-averaging |+| Multi-agent (drones, V2X driving fleets) | Single-agent RL + chance-constraint safety filter that handles coordination | Decouples coordination from policy training; planner-agnostic |+| Sparse-reward AD with closed-loop reactive evaluation | Hybrid: multi-modal RL/IL proposer + PDM-Closed-style rule-based post-processing OR HYPE-style MCTS+occupancy | CarPlanner / CaRL close aggregate gap but lose 1.5 CLS / 0.7 collision sub-metrics; PLUTO and Diffusion Planner only beat PDM-Closed *with* its post-processing |+| Simulator-only training, sample-efficiency-bound | LSVI-AE-style aggressive-optimism exploration | Aggressive exploration accelerates cost-function learning, reduces later violations |+| Real-robot online training (every rollout has irreversible cost) | CERL Bayesian world model + log-barrier SGD; safe from training step 1 | CPO/LAMBDA need ~1.5M steps to satisfy constraints; CERL holds safety throughout |+| High-DoF humanoid whole-body motion planning | McVAMP-style SIMD-parallelized projection-based RRT-Connect | 100-1000× speed-up over OMPL; <1ms for most plans, 12ms on 36-DoF Digit |++### 5.5. The diffusion split mapped to applications++The diffusion planning vs diffusion policy split maps onto application domains cleanly. For sparse-reward long-horizon trajectory tasks — autonomous-driving lane changes, multi-stage manipulation, goal-conditioned navigation — practitioners should default to a diffusion *planner* using the DV recipe: state-only diffusion with inverse-dynamics action generation, Transformer backbone, jump-step planning with stride 4-8, MCSS guidance [74]. Several diffusion-planner variants extend the baseline:++- **Decision Diffuser** [94] is the load-bearing classifier-free-guidance baseline these later systems extend, and is uniquely the corpus instance demonstrating training-time *constraint conditioning* that composes at test time — training with a single constraint or skill enables test-time composition over multiple constraints simultaneously, directly relevant to the 5.3 joint-denoising prescription.+- **AdaptDiffuser** self-evolves through reward-gradient guidance plus discriminator filtering, outperforming Diffuser by 20.8% on Maze2D, 7.5% on MuJoCo locomotion, and 27.9% on unseen KUKA pick-and-place [76].+- **Diffusion-ES** generalizes to gradient-free evolutionary search with truncated-diffusion mutation, reaching SOTA on nuPlan with LLM-shaped reward functions [95].+- **Hierarchical Diffuser** adds high-level Sparse Diffuser jumpy subgoals plus low-level Dense Diffuser segments for an additional +12.0% on Maze2D and +9.2% on MuJoCo over flat Diffuser [96].+- **PG-Diffuser** replaces the Gaussian initial prior with a learnable parameterized distribution optimized via behavior-regularized objective, enabling single-trajectory inference [97]. For dense-feedback short-horizon manipulation, practitioners should default to a diffusion *policy* at the action level — Diffusion Policy achieves +46.9% over prior SOTA on 15 manipulation tasks [12].++### 5.6. What transfers, and what is RL-specific scaffolding++Concrete transfer mechanisms with defensible cross-domain stories:++- **ATACOM tangent-space projection** ↔ AtlasRRT/IMACS continuation-based sampling at the linear-algebra level.+- **World-model imagination** ↔ MPC over a learned dynamics model.+- **HER goal relabeling** ↔ reachability projection (RAIL).+- **Intrinsic motivation as exploration heuristic** ↔ sampling-based planner exploration heuristics.++Diffusion planners are arguably best understood not as "RL with diffusion" but as a *generalization of sampling-based planners that learns the proposal distribution from data*: a diffusion planner conditioned on goal and current state samples trajectory candidates, scores them, selects the best — functionally a learned RRT. The DV finding that "unconditional sampling with selection" beats classifier guidance precisely when the dataset has near-optimal samples is exactly the regime in which a sampling-based planner's "build a tree from samples" strategy works.++RL-specific scaffolding that does NOT transfer cleanly:++- **Noisy-TV fixes at the high-dimensional continuous-control scale:** LPM and CiH validated only on discrete and low-dim continuous environments.+- **IM pretraining outside the structural-homology envelope:** CBET hurts in MiniGrid while helping in Crafter; Plan2Explore fails the object-interaction novelty assay.+- **Expectation-bounded "zero violation" claims** interpreted as deployment safety.+- **Decision-Transformer-class sequence models** without trajectory-planning empirical anchors in the corpus. Classical task-and-motion planning (PDDLStream-style hybrid discrete/continuous planners) is a natural integration target for the layered-stack recipe — the discrete TAMP backbone selecting symbolic operators, continuous trajectory refinement via diffusion + ATACOM — although the corpus does not directly cover TAMP work and the integration is named here as an open direction.++The trajectory-planning synthesis: the literature supports a regime-conditioned composition rather than a single best practice.++- **Safety-critical deployment, non-simply-connected geometry, second-order dynamics:** the load-bearing recipe is multi-modal proposer + geometric projector + runtime CBF/MPC shield, with JM2D-style joint denoising outperforming sequential post-hoc projection by 75× in cluttered environments.+- **Simulator-only training, first-order analytic constraints:** ATACOM is implementable as a sampling kernel inside OMPL today and is the cleanest single-mechanism answer available.++Reading the literature normatively — "use this method" — is wrong; the right read is structural — "under these conditions, this composition wins" — and the marginal research effort is best spent on the open empirical gaps named in Section 6 rather than on architectural maximalism in either direction.++## 6. Open Problems and Outlook++Seven concrete problems and one cultural problem demand the next two years of attention, ranked below by leverage and tagged by category (Empirical / Theoretical / Methodological / Cultural). The single highest-leverage open project is the **ATACOM-OMPL bridge (rank #1)**.++**First, the ATACOM-OMPL bridge has not been built. (Empirical gap, leverage: HIGH — rank #1.)** The formal equivalence of ATACOM tangent-space projection and AtlasRRT continuation-based sampling is mathematically clean and mechanically demonstrable. No paper in the corpus has implemented and benchmarked the composition. The specific falsification condition: the slack-variable Jacobian J_c(q,μ) may lose full rank in the slack subspace at μ=0 (the constraint boundary itself), causing the IMACS atlas-chart creation to fail — a known failure mode for continuation-based methods at boundaries. This is the single highest-leverage open empirical project — a controlled head-to-head of (BIT* + ATACOM constraints in OMPL) versus (ATACOM + SAC) on the IiwaAirHockey μ=0 region (the natural test case) or PlanarAirHockey would settle whether the architectural recommendation in Section 5 actually delivers in practice. McVAMP's vectorized projection suggests the projection-based variant is the right starting point because it sidesteps the singularity issues that plague atlas-based methods on second-order systems. Note also the architectural compromise for learned constraints: D-ATACOM's distributional safety value can drive ProjectedStateSpace/CBiRRT but not AtlasStateSpace, because neural-network Jacobians are typically poorly conditioned at constraint boundaries.++**Second, ATACOM's second-order safety gap must be closed. (Theoretical gap, leverage: HIGH.)** The TRO extended paper formally proves the Singular Set is disjoint from the Constraint Manifold for first-order systems but shows ATACOM does not guarantee safety for second-order systems with position-based constraints without modifying the constraint. The fix is non-obvious. CBF-QP handles second-order systems natively via the Lie-derivative condition — so the layered stack may need a hybrid first-order (ATACOM) / second-order (CBF-QP) split. CBF-RL's dual-mode training suggests how this can be folded into an end-to-end policy that internalizes both types of constraint.++**Third, the noisy-TV fix at high-dimensional continuous-control scale is unresolved. (Empirical gap, leverage: HIGH.)** LPM dominates aleatoric methods on every directly comparable domain, but its theoretical proof assumes i.i.d. Gaussian observation noise — which does not hold in autonomous-driving sensor regimes (lidar dropouts, multimodal returns, temporally correlated noise). CiH's causal-model framework provides stronger theoretical coverage for non-Gaussian noise but has not been tested in continuous control. A head-to-head benchmark of LPM versus BYOL-Hindsight on a high-dimensional continuous-control task with realistic non-Gaussian sensor noise (CARLA, MuJoCo with injected sensor noise) is the most decision-relevant exploration experiment in this design space. InDRiVE — the only autonomous-driving paper using disagreement-based MBRL exploration — is structurally vulnerable to this gap because it uses ensemble disagreement, the family that both LPM and CiH outperform under stochasticity.++**Fourth, intrinsic-motivation pretraining transfer requires structural-alignment diagnostics. (Methodological gap, leverage: MEDIUM.)** Curious Replay shows that uniform replay in Plan2Explore is the root cause of failure to re-orient toward novel objects; CEE-US shows that object-centric GNN ensembles substantially outperform monolithic RSSM world models on manipulation. Together these suggest the right transfer protocol is (i) match the world-model architecture's inductive bias to the task family, (ii) use curiosity-prioritized replay throughout, and (iii) verify structural homology between pretraining and deployment environments before claiming transfer. The autonomous-driving community has not yet adopted this protocol.++**Fifth, "zero violation" rhetoric must be reformed. (Methodological gap, leverage: MEDIUM.)** Wachi's T-class versus C-class taxonomy is a sharp tool the field has not adopted as standard reporting. AutoCost / PCPO / CRLLK's "zero violation" claims are C-class, not T-class. The Eckel et al. EMCC metric is the right operational fix for benchmarks: report Expected Maximum Consecutive Cost rather than expected cumulative cost, distinguishing infrequent severe violations from frequent mild ones. The next safe-RL benchmark generation should adopt T-class versus C-class labelling and EMCC reporting as standard.++**Sixth, the joint-denoising architecture suggested by JM2D needs broader empirical validation. (Empirical gap, leverage: MEDIUM.)** JM2D is a single CoRL 2025 paper; the 75× success-rate advantage and 72× fewer violations versus SafeDiffuser/DPCC are striking but represent one experiment in one benchmark family. Natural follow-ups: joint denoising with ATACOM-derived constraint functions for first-order systems; joint denoising with CBF-QP for second-order systems; joint denoising with reachability-based safety à la RAIL but coupled rather than sequential. Each is a falsifiable architectural variant; none has been benchmarked. JM2D's I(I+1)/2 score-evaluation cost versus standard diffusion's I also raises a real-time question: whether the joint composition scales to AD planning under the <100ms requirement is open, and Habi-style distillation may absorb this but the composition has not been tested.++**Seventh, risk-sensitive / CVaR RL at the trajectory-planning interface is under-explored. (Empirical gap, leverage: MEDIUM.)** CPPO [41] is the only direct corpus instance; whether tail-risk objectives produce systematically different deployment behavior than CMDP expectation constraints in autonomous driving and aerial regimes (where heavy-tailed disturbances dominate) is open. Joint chance-constrained policy optimization linking CPPO to chance-constrained MPC is the most natural research direction.++**The cultural open question sits at the hybrid-vs-pure-RL convergence point. (Cultural / methodological gap, leverage: HIGH.)** Whether a 5B-sample pure-RL planner closes the safety sub-metric gap, or whether the gap is structural (i.e., requires the rule-based post-processing PDM-Closed exploits), determines the long-run direction of AD trajectory planning research. The corpus has insufficient data to predict which it is.++The forward-looking implication is large. The next 12-24 months of trajectory-planning research will see the layered stack — multi-modal proposer plus geometric projector plus runtime filter — become the standard reference architecture for safety-critical deployment. CPO-family methods will not disappear; they will be reframed as research-grade exploration drives that supply gradient signal during training, while the deployment-time architecture will be hybrid (RL or diffusion proposer + classical safety enforcer). The autonomous-driving community is already most of the way there: every nuPlan-leading planner in 2025 is a hybrid (PDM-Closed, CaRL with rule-based post-processing, HYPE with MCTS+occupancy, PLUTO+post-processing). The robotic-manipulation community is following: SMART, RAIL, JM2D, McVAMP all instantiate the same recipe.++If the pure-RL counter-trend matures faster than expected — a 5B-sample CaRL with CBF-RL-style internalized safety closing the safety sub-metric gap — the field will bifurcate into "scale and internalize" versus "compose and project" paradigms, and recipe-style recommendations will become regime-conditioned defaults rather than universal prescriptions. The single largest cultural risk is the persistence of "zero violation" rhetoric and the failure to distinguish C-class from T-class guarantees, leaving practitioners to deploy expectation-bounded algorithms into safety-critical contexts where almost-sure guarantees are the operational requirement. The two streams have matured to the point where their joint output is no longer a survey but a prescription — regime-conditioned, with the regimes knowable in advance, and the practitioner who reads the conditions can pick the composition that wins.++## Sources++[1] EXPLORATION BY RANDOM NETWORK DISTILLATION. https://arxiv.org/abs/1810.12894++[2] Exploration in Deep Reinforcement Learning: A. https://arxiv.org/abs/2205.00824++[3] An information-theoretic perspective on intrinsic motivation in reinforcement. https://arxiv.org/abs/2209.08890++[4] Constrained Policy Optimization. https://arxiv.org/abs/1705.10528++[5] Responsive Safety in Reinforcement Learning by PID Lagrangian Methods. https://arxiv.org/abs/2007.03964++[6] Robot Reinforcement Learning on the Constraint. https://proceedings.mlr.press/v164/liu22c/liu22c.pdf++[7] DOPE: Doubly Optimistic and Pessimistic. https://arxiv.org/abs/2112.00885++[8] Curiosity in Hindsight:. https://arxiv.org/abs/2211.10515++[9] Policy Bifurcation in Safe Reinforcement Learning. https://arxiv.org/abs/2403.12847++[10] Mastering Diverse Domains through World Models. https://arxiv.org/abs/2301.04104++[11] Safe Exploration Using Bayesian World Models and. https://arxiv.org/abs/2405.05890++[12] Diffusion Policy: Visuomotor Policy. https://arxiv.org/abs/2303.04137++[13] PreND: Enhancing Intrinsic Motivation in. https://arxiv.org/abs/2410.01745++[14] Neuro-Inspired Fragmentation and Recall to. https://arxiv.org/abs/2310.17537++[15] Curiosity-driven Exploration by Self-supervised Prediction. https://arxiv.org/abs/1705.05363++[16] BYOL-Explore:. https://arxiv.org/abs/2206.08332++[17] Learning Latent Dynamics for Planning from Pixels. https://arxiv.org/abs/1811.04551++[18] Published as a conference paper at ICLR 2020. https://arxiv.org/abs/1912.01603++[19] Planning to Explore via Self-Supervised World Models. https://arxiv.org/abs/2005.05960++[20] Optimistic World Models:. https://arxiv.org/abs/2602.10044++[21] DreamerV3-XP: Optimizing exploration through. https://arxiv.org/abs/2510.21418++[22] Curious Exploration via Structured World Models. https://arxiv.org/abs/2206.11403++[23] Curious Replay for Model-based Adaptation. https://arxiv.org/abs/2306.15934++[24] How to Stay Curious while avoiding Noisy TVs. https://arxiv.org/abs/2102.04399++[25] Published as a conference paper at ICLR 2026. https://arxiv.org/abs/2509.25438++[26] Hindsight Experience Replay. https://arxiv.org/abs/1707.01495++[27] Go-Explore: a New Approach for Hard-Exploration. https://arxiv.org/abs/1901.10995++[28] First return, then explore. https://arxiv.org/abs/2004.12919++[29] Published as a conference paper at ICLR 2025. https://arxiv.org/abs/2405.15143++[30] DIVERSITY IS ALL YOU NEED:. https://arxiv.org/abs/1802.06070++[31] CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery. https://arxiv.org/abs/2202.00161++[32] LLM-Driven Intrinsic Motivation for Sparse Reward. https://arxiv.org/abs/2508.18420++[33] Navigate the Unknown: Enhancing LLM Reasoning with. https://arxiv.org/abs/2505.17621++[34] Decision Transformer: Reinforcement. https://arxiv.org/abs/2106.01345++[35] Constrained Ensemble Exploration for Unsupervised Skill Discovery. https://arxiv.org/abs/2405.16030++[36] This work has been submitted to the IEEE for possible publication. Copyright. https://arxiv.org/abs/2503.05573++[37] World Model Agents with Change-Based Intrinsic Motivation. https://arxiv.org/abs/2503.21047++[38] arXiv:2402.02025v2 [cs.LG] 8 May 2024. https://arxiv.org/abs/2402.02025++[39] Journal of Machine Learning Research 16 (2015) 1437-1480. https://www.jmlr.org/papers/volume16/garcia15a/garcia15a.pdf++[40] Published in Transactions on Machine Learning Research (04/2024). https://arxiv.org/abs/2306.12594++[41] Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk. https://arxiv.org/abs/2206.04436++[42] Published as a conference paper at ICLR 2019. https://arxiv.org/abs/1805.11074++[43] Benchmarking Safe Exploration in Deep. https://cdn.openai.com/safexp-short.pdf++[44] Published in Transactions on Machine Learning Research (03/2024). https://arxiv.org/abs/2305.13681++[45] CUP: A Conservative Update Policy Algorithm for Safe. https://arxiv.org/abs/2202.07565++[46] Towards a Practical Understanding of Lagrangian. https://arxiv.org/abs/2510.17564++[47] AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning. https://arxiv.org/abs/2301.10339++[48] CRLLK: Constrained Reinforcement Learning for Lane Keeping. https://arxiv.org/abs/2503.22248++[49] Proactive Constrained Policy Optimization with. https://arxiv.org/abs/2508.01883++[50] Revisiting Safe Exploration in Safe Reinforcement. https://arxiv.org/abs/2409.01245++[51] Safety-Gymnasium: A Unified Safe Reinforcement. https://proceedings.neurips.cc/paper_files/paper/2023/file/3c557a3d6a48cc99444f85e924c66753-Paper-Datasets_and_Benchmarks.pdf++[52] OmniSafe: An Infrastructure for Accelerating. https://arxiv.org/abs/2305.09304++[53] Safe Reinforcement Learning with Instantaneous Constraints: The Role of. https://arxiv.org/abs/2312.14470++[54] Better Exploration with Optimistic Actor-Critic. https://arxiv.org/abs/1910.12807++[55] Published as a conference paper at ICLR 2026. https://arxiv.org/abs/2603.23889++[56] Published as a conference paper at ICLR 2022. https://arxiv.org/abs/2201.09802++[57] Safe Exploration in Reinforcement Learning:. https://arxiv.org/abs/2310.03225++[58] Published as a conference paper at ICLR 2024. https://arxiv.org/abs/2307.07176++[59] Handling Long-Term Safety and Uncertainty in Safe. https://arxiv.org/abs/2409.12045++[60] SUBMITTED TO IEEE TRANSACTIONS ON ROBOTICS. https://www.ias.informatik.tu-darmstadt.de/uploads/Team/PuzeLiu/TRO_ATACOM_submitted.pdf++[61] arXiv:2404.16879v1 [cs.LG] 22 Apr 2024. https://arxiv.org/abs/2404.16879++[62] CBF-RL: Safety Filtering Reinforcement Learning in Training with. https://arxiv.org/abs/2510.14959++[63] Safe Exploration in Model-based Reinforcement Learning. https://arxiv.org/abs/2104.08171++[64] Robust Model Predictive Shielding for Safe Reinforcement Learning. https://arxiv.org/abs/1910.10885++[65] IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED FEBRUARY,. https://arxiv.org/abs/2010.15920++[66] Learning to be Safe: Deep RL with a Safety Critic. https://arxiv.org/abs/2010.14603++[67] RAIL: Reachability-Aided Imitation Learning for Safe Policy Execution. https://arxiv.org/abs/2409.19190++[68] Joint Model-based Model-free Diffusion. https://arxiv.org/abs/2509.08775++[69] SafeDiffuser: Safe Planning with Diffusion Probabilistic Models. https://arxiv.org/abs/2306.00148++[70] Proceedings of Machine Learning Research vol 283:1–14, 2025. https://arxiv.org/abs/2412.09342++[71] Robotics: Science and Systems 2025. https://arxiv.org/abs/2502.15043++[72] Constrained Reinforcement Learning for Vehicle Motion Planning with Topological. https://www.mdpi.com/2218-6581/11/4/81++[73] Exploring Implicit Spaces for. https://www.kavrakilab.org/publications/kingston2019exploring-implicit-spaces-for-constrained.pdf++[74] Published as a conference paper at ICLR 2025. https://arxiv.org/abs/2503.00535++[75] Planning with Diffusion for Flexible Behavior Synthesis. https://arxiv.org/abs/2205.09991++[76] AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners. https://arxiv.org/abs/2302.01877++[77] CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale. https://arxiv.org/abs/2502.19908++[78] CaRL: Learning Scalable Planning. https://www.cvlibs.net/publications/Jaeger2025CORL.pdf++[79] nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous. https://arxiv.org/abs/2511.10403++[80] PLUTO: Pushing the Limit of Imitation. https://arxiv.org/abs/2404.14327++[81] Parting with Misconceptions about. https://arxiv.org/abs/2306.07962++[82] HYPE: Hybrid Planning with Ego Proposal-Conditioned Predictions. https://arxiv.org/abs/2510.12733++[83] Habitizing Diffusion Planning for Efficient and Effective Decision Making. https://arxiv.org/abs/2502.06401++[84] Vectorizing Projection in Manifold-Constrained Motion Planning for. https://arxiv.org/abs/2604.13323++[85] Published as a conference paper at ICLR 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/5c0cd43063f087eb32e572d040a19cd2-Paper-Conference.pdf++[86] Reinforced Imitative Trajectory Planning for Urban. https://arxiv.org/abs/2410.15607++[87] ©2023 IEEE. Personal use of this material is permitted. Permission from IEEE. https://arxiv.org/abs/2512.03774++[88] Feasible Actor-Critic: Constrained Reinforcement Learning. https://arxiv.org/abs/2105.10682++[89] SMART: Scalable Multi-Agent Reasoning and Trajectory Planning. https://arxiv.org/abs/2509.15737++[90] Beyond Conservative Automated Driving in Multi-Agent Scenarios. https://arxiv.org/abs/2604.13891++[91] Safe multi-agent motion planning under uncertainty for drones using. https://arxiv.org/abs/2311.00063++[92] Trajectory Planning for Autonomous Vehicle Using. https://arxiv.org/abs/2404.12079++[93] V-Max: A RL Framework for Autonomous Driving. https://rlj.cs.umass.edu/2025/papers/RLJ_RLC_2025_295.pdf++[94] Published as a conference paper at ICLR 2023. https://arxiv.org/abs/2211.15657++[95] Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving. https://arxiv.org/abs/2402.06559++[96] SIMPLE HIERARCHICAL PLANNING WITH DIFFUSION. https://arxiv.org/abs/2401.02644++[97] Prior-Guided Diffusion Planning for. https://arxiv.org/abs/2505.10881