Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

58.2 KB
Raw Blame
//! Routes a request to the Laya checkpoint best suited to it. Ports `laya/router.py`.
//!
//! Three checkpoints, measured on a shared benchmark (17,416 questions, one T4, identical
//! questions per model):
//!
//! ```text
//! english          convaiinnovations/laya                421M  ModernBERT-large, 512 tokens
//! multilingual     convaiinnovations/laya-multilingual   322M  mmBERT-base, 1024 tokens, 100+ langs
//! typed-decisions  convaiinnovations/laya-typed-decisions 421M  ModernBERT-large, 1024 tokens,
//!                                                                fine-tuned on the typed-decisions
//!                                                                workflows
//! ```
//!
//! The English checkpoint does not gently degrade off English, it collapses: on 20-option
//! MASSIVE intent it scores 0.100 on Hindi and 0.103 on Korean, against 0.050 for random
//! guessing -- and it reports high confidence while doing so. Script detection is therefore the
//! primary routing signal; see [`crate::language_detection`].
//!
//! `typed-decisions` is never selected automatically unless the caller opts in with
//! [`LayaRouterOptions::auto_task_detection`] or passes `task = Some("typed_decisions")`: it is
//! fine-tuned on four specific synthetic workflows and should not be a silent default.
//!
//! ## Eviction safety
//!
//! Python relies on garbage collection: an evicted `Agent` is simply dropped from `self._agents`
//! and freed once nothing else references it, including an in-flight `system_one` call. Rust has
//! no GC, so every loaded engine here is held as `Arc<dyn LayaPredictor>`: [`Self::predict`]
//! clones the `Arc` out from under the router lock before running inference, so an eviction that
//! happens mid-predict only drops the router's own reference — the engine itself is freed only
//! once that in-flight predict's clone is also dropped. Inference therefore never runs while
//! holding the router's lock, matching Python's `Agent.system_one` being called outside
//! `self._lock`.
//!
//! Building an engine (unlike inference) *does* happen under the lock, exactly as Python's
//! `load()` holds `self._lock` for the whole `Agent(...)` construction: this is what makes
//! concurrent `Load` of the same checkpoint build the engine exactly once (every later caller
//! blocks on the lock, then sees the checkpoint already resident).

use crate::answers::LayaResult;
use crate::batch_prediction::BatchOptions;
use crate::confidence::apply_confidence_gate;
use crate::error::{LayaError, Result};
use crate::language_detection::{LanguageAnalysis, LanguageDetection, prefix_chars, py_strip};
use crate::laya_engine::{LayaEngine, refuse_null_states};
use crate::laya_options::{LayaCheckpoint, LayaOptions};
use crate::long_prediction::LongOptions;
use crate::predict_options::PredictOptions;
use crate::questions::QuestionSet;
use serde_json::Value;
use std::collections::{HashMap, HashSet};
use std::sync::{Arc, Mutex};

/// Answers typed questions about a piece of state in one forward pass. Implemented by
/// [`LayaEngine`] and [`LayaRouter`] so `laya_shortlist` (and any other caller) can work over
/// either without knowing which one it has — the Rust equivalent of Python's `_call_predict`
/// duck-typing (`getattr(agent, "predict", None) or getattr(agent, "system_one", None)`).
///
/// Takes `state: Value` rather than `impl Into<Value>` (unlike [`LayaEngine::predict`]) so the
/// trait stays object-safe: `LayaRouter` stores its loaded checkpoints as `Arc<dyn
/// LayaPredictor>`, and a generic method cannot be part of a trait object's vtable. Callers with
/// an `impl Into<Value>` state can convert with `.into()` at the call site.
pub trait LayaPredictor: Send + Sync {
    /// Answer every question in `questions` about `state`.
    fn predict(&self, state: Value, questions: &QuestionSet) -> Result<LayaResult>;

    /// [`Self::predict`] with the per-call `lang`, `max_len` and `head_max_len` the router
    /// forwards. The default drops `lang`, as Python's router does for an attached agent whose
    /// `system_one` takes none, and refuses a token-budget override, which Python passes on and
    /// such an agent rejects with a `TypeError`; [`LayaEngine`] and [`LayaRouter`] honour all
    /// three. `options.min_confidence` is never set: the router applies the gate itself.
    fn predict_with_options(
        &self,
        state: Value,
        questions: &QuestionSet,
        options: &PredictOptions,
    ) -> Result<LayaResult> {
        if options.max_len.is_some() || options.head_max_len.is_some() {
            return Err(LayaError::NotSupported(
                "this predictor does not accept max_len or head_max_len".to_string(),
            ));
        }
        self.predict(state, questions)
    }

    /// Whether this predictor carries per-language temperatures, which is when the router groups
    /// a batch by language (Python's `getattr(agent, "lang_temperatures", None)`). The default is
    /// `false`.
    fn has_lang_temperatures(&self) -> bool {
        false
    }

    /// This predictor's scan capability, if it has one. The default is `None`, so implementing
    /// [`LayaPredictor`] elsewhere keeps compiling; [`LayaEngine`] and [`LayaRouter`] return
    /// themselves. [`LayaRouter::predict_long`] needs it: Rust has no run-time interface check on
    /// a `dyn LayaPredictor`, and a predictor without it must fail loudly rather than silently
    /// truncate a long state.
    fn as_long_predictor(&self) -> Option<&dyn LayaLongPredictor> {
        None
    }

    /// This predictor's batch capability, if it has one. The default is `None`, so implementing
    /// [`LayaPredictor`] elsewhere keeps compiling; [`LayaEngine`] and [`LayaRouter`] return
    /// themselves. [`LayaRouter::predict_batch`] needs it.
    fn as_batch_predictor(&self) -> Option<&dyn LayaBatchPredictor> {
        None
    }
}

/// A [`LayaPredictor`] that can score many states against the same questions in shared forward
/// passes. The Rust counterpart of Python's router calling `agent.predict_batch`.
pub trait LayaBatchPredictor: LayaPredictor {
    /// Answer `questions` about every state, in input order. See [`LayaEngine::predict_batch`].
    fn predict_batch(
        &self,
        states: &[Value],
        questions: &QuestionSet,
        options: &BatchOptions,
    ) -> Result<Vec<LayaResult>>;
}

/// A [`LayaPredictor`] that can also scan a state longer than one sequence. The Rust counterpart
/// of Python's router checking `getattr(agent, "predict_long", None)`.
pub trait LayaLongPredictor: LayaPredictor {
    /// Scan overlapping state-token windows and select one answer per question. See
    /// [`LayaEngine::predict_long`].
    fn predict_long(
        &self,
        state: Value,
        questions: &QuestionSet,
        options: &LongOptions,
    ) -> Result<LayaResult>;
}

impl LayaPredictor for LayaEngine {
    fn predict(&self, state: Value, questions: &QuestionSet) -> Result<LayaResult> {
        LayaEngine::predict(self, state, questions)
    }

    fn predict_with_options(
        &self,
        state: Value,
        questions: &QuestionSet,
        options: &PredictOptions,
    ) -> Result<LayaResult> {
        LayaEngine::predict_with_options(self, state, questions, options)
    }

    fn has_lang_temperatures(&self) -> bool {
        !self.lang_temperatures().is_empty()
    }

    fn as_long_predictor(&self) -> Option<&dyn LayaLongPredictor> {
        Some(self)
    }

    fn as_batch_predictor(&self) -> Option<&dyn LayaBatchPredictor> {
        Some(self)
    }
}

impl LayaBatchPredictor for LayaEngine {
    fn predict_batch(
        &self,
        states: &[Value],
        questions: &QuestionSet,
        options: &BatchOptions,
    ) -> Result<Vec<LayaResult>> {
        LayaEngine::predict_batch(self, states, questions, options)
    }
}

impl LayaLongPredictor for LayaEngine {
    fn predict_long(
        &self,
        state: Value,
        questions: &QuestionSet,
        options: &LongOptions,
    ) -> Result<LayaResult> {
        LayaEngine::predict_long(self, state, questions, options)
    }
}

/// Question-id signatures of the four typed-decisions workflows, used only when
/// [`LayaRouterOptions::auto_task_detection`] is enabled.
const TYPED_DECISION_WORKFLOWS: &[(&str, &[&str])] = &[
    (
        "agent_trace_observability",
        &["action", "needs_review", "outcome", "risk", "urgency"],
    ),
    (
        "customer_service",
        &["action", "category", "churn_risk", "needs_human", "urgency"],
    ),
    (
        "invoice_processing",
        &[
            "discrepancy_severity",
            "disposition",
            "duplicate",
            "matches_order",
            "urgency",
        ],
    ),
    (
        "security_incidents",
        &[
            "credential_compromise",
            "disposition",
            "severity",
            "true_positive",
            "urgency",
        ],
    ),
];

/// The state a [`LangGuess`] callback is handed, exactly as the caller gave it to the router: a
/// JSON value from `route`, `predict` and `predict_long`, the original bytes from `route_bytes`
/// (Python passes the `bytes` object itself, invalid UTF-8 included).
#[derive(Debug, Clone, Copy)]
pub enum GuessInput<'a> {
    /// A JSON state.
    Value(&'a Value),
    /// A `bytes` state, untouched.
    Bytes(&'a [u8]),
}

/// The callback form of [`LangGuess`]: reads the state, answers with a language code or `None`.
type GuessFn = dyn Fn(GuessInput<'_>) -> Option<String> + Send + Sync;

/// A caller-supplied answer to "can the English checkpoint read this state?", Python's
/// `lang_guess`: a language code, or a callback taking the state. Any code that is not English
/// routes to the multilingual checkpoint; a code that names no language (blank, `C`, `POSIX`,
/// `C.UTF-8`, `und`, `zxx`, `mul`) or a callback returning `None` abstains, and routing falls
/// through to detection.
#[derive(Clone)]
#[non_exhaustive]
pub enum LangGuess {
    /// A language code such as `"en"`, `"en-US"`, `"en_US.UTF-8"` or `"pt-BR"`.
    Code(String),
    /// A function of the state that returns a language code, or `None` to abstain (for example
    /// a language-identification model that is not confident).
    Callback(Arc<GuessFn>),
}

impl LangGuess {
    /// A hint that reads the state and answers with a code, or `None` to abstain.
    pub fn callback(f: impl Fn(GuessInput<'_>) -> Option<String> + Send + Sync + 'static) -> Self {
        Self::Callback(Arc::new(f))
    }

    /// `_resolve_hint`: `Some(true)` when the hint says English, `Some(false)` when it names
    /// another language, `None` when it identifies nothing.
    fn resolve(&self, state: GuessInput<'_>) -> Option<bool> {
        match self {
            Self::Code(code) => english_from_code(code),
            Self::Callback(f) => f(state).and_then(|code| english_from_code(&code)),
        }
    }
}

impl std::fmt::Debug for LangGuess {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            Self::Code(code) => f.debug_tuple("Code").field(code).finish(),
            Self::Callback(_) => f.write_str("Callback(..)"),
        }
    }
}

impl From<&str> for LangGuess {
    fn from(code: &str) -> Self {
        Self::Code(code.to_string())
    }
}

impl From<String> for LangGuess {
    fn from(code: String) -> Self {
        Self::Code(code)
    }
}

/// Codes that are valid `$LANG` values but name no language, so they answer nothing about the
/// state: `C`, `POSIX` and `C.UTF-8` (what minimal images ship) and the ISO 639-2 special codes
/// `und` (undetermined), `zxx` (no linguistic content) and `mul` (multiple languages).
const LANGUAGE_AGNOSTIC_CODES: [&str; 5] = ["c", "posix", "und", "zxx", "mul"];

/// `_english_from_code`: `Some(true)` for an English code, `Some(false)` for any other language
/// code, `None` when the code identifies nothing.
///
/// Accepts the forms a caller is likely to have to hand: `"en"`, `"EN"`, `"en-US"`, the POSIX
/// `"en_US"` (which `$LANG` holds), and `"en_US.UTF-8"`.
pub(crate) fn english_from_code(value: &str) -> Option<bool> {
    let code = py_strip(value).to_lowercase();
    if code.is_empty() {
        return None;
    }
    let code = code.split('.').next().unwrap_or("");
    let primary = code.replace('_', "-");
    let primary = primary.split('-').next().unwrap_or("");
    if primary.is_empty() || LANGUAGE_AGNOSTIC_CODES.contains(&primary) {
        return None;
    }
    Some(matches!(primary, "en" | "eng" | "english"))
}

/// The routing outcome: which checkpoint, why, and what was detected. Serializes into
/// `LayaResult::routing` (`answers.rs`). Python's `RouteDecision` also carries a `repo` field
/// (the resolved HuggingFace repo/subfolder string); it is intentionally not ported, since this
/// crate's [`LayaRouterOptions`] already resolves artifacts per checkpoint through the same
/// [`crate::model_artifacts::ModelArtifacts`] path every other engine uses, so there is no
/// separate "which repo string" question to answer here.
#[derive(Debug, Clone, PartialEq)]
pub struct RouteDecision {
    /// The chosen checkpoint.
    pub model: LayaCheckpoint,
    /// Human-readable reason, byte-identical to Python's (including `%r`-style quoting and
    /// `%.0f%%` percentages) so the two are directly comparable in logs.
    pub reason: String,
    /// The script/language detection this decision was based on, or `None` when the model was
    /// chosen without looking at the state at all (an explicit `model`/`task`, or an opted-in
    /// typed-decisions workflow match).
    pub detection: Option<LanguageAnalysis>,
    /// The typed-decisions workflow the question ids matched, if any — populated whenever a
    /// match exists, even if [`LayaRouterOptions::auto_task_detection`] was off and the match
    /// therefore did not decide the routing.
    pub workflow: Option<String>,
}

impl std::fmt::Display for RouteDecision {
    /// Matches Python's `RouteDecision.__repr__`.
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(
            f,
            "RouteDecision(model={}, reason={})",
            python_repr_str(self.model.subdir()),
            python_repr_str(&self.reason)
        )
    }
}

/// One request of [`LayaRouter::route_batch`] / [`LayaRouter::predict_batch`]: a state, its
/// questions and the same optional routing overrides [`LayaRouter::route_with_lang_guess`] takes.
/// Python passes these as a request dict with the keys `state`, `questions`, `model`, `task`,
/// `lang` and `lang_guess`.
///
/// A request may also carry its own `max_len` and `head_max_len`, which override the checkpoint's
/// token budget for that request alone (requests that ask for different budgets, or different
/// languages when the engine has per-language temperatures, do not share a forward pass). Build
/// one with [`Self::new`] and the chained setters.
#[derive(Debug, Clone)]
#[non_exhaustive]
pub struct BatchRequest {
    /// What to decide about. A `null` state is refused, as Python's `TypeError`.
    pub state: Value,
    /// The questions to answer. Shared by `Arc`, so requests that ask the same questions need not
    /// each hold a copy; requests with equal question schemas share a forward pass either way.
    pub questions: Arc<QuestionSet>,
    /// Route to this checkpoint (a name or alias), skipping detection.
    pub model: Option<String>,
    /// Route by task, such as `"typed_decisions"`.
    pub task: Option<String>,
    /// A language code that decides English or not, ahead of detection. It also selects the
    /// per-language temperatures when the engine has any, falling back to the detected language.
    pub lang: Option<String>,
    /// A per-request language hint; see [`LayaRouter::route_with_lang_guess`].
    pub lang_guess: Option<LangGuess>,
    /// Override the checkpoint's `max_len` for this request; see [`PredictOptions::max_len`].
    pub max_len: Option<i64>,
    /// Override the checkpoint's `head_max_len` for this request; see
    /// [`PredictOptions::head_max_len`].
    pub head_max_len: Option<i64>,
}

impl BatchRequest {
    /// A request with no routing overrides.
    pub fn new(state: impl Into<Value>, questions: impl Into<Arc<QuestionSet>>) -> Self {
        Self {
            state: state.into(),
            questions: questions.into(),
            model: None,
            task: None,
            lang: None,
            lang_guess: None,
            max_len: None,
            head_max_len: None,
        }
    }

    /// Set [`Self::model`].
    #[must_use]
    pub fn model(mut self, model: impl Into<String>) -> Self {
        self.model = Some(model.into());
        self
    }

    /// Set [`Self::task`].
    #[must_use]
    pub fn task(mut self, task: impl Into<String>) -> Self {
        self.task = Some(task.into());
        self
    }

    /// Set [`Self::lang`].
    #[must_use]
    pub fn lang(mut self, lang: impl Into<String>) -> Self {
        self.lang = Some(lang.into());
        self
    }

    /// Set [`Self::lang_guess`].
    #[must_use]
    pub fn lang_guess(mut self, lang_guess: impl Into<LangGuess>) -> Self {
        self.lang_guess = Some(lang_guess.into());
        self
    }

    /// Set [`Self::max_len`].
    #[must_use]
    pub fn max_len(mut self, max_len: i64) -> Self {
        self.max_len = Some(max_len);
        self
    }

    /// Set [`Self::head_max_len`].
    #[must_use]
    pub fn head_max_len(mut self, head_max_len: i64) -> Self {
        self.head_max_len = Some(head_max_len);
        self
    }
}

/// Configuration for [`LayaRouter`]. `engine_options` is the template every loaded checkpoint's
/// [`LayaOptions`] is built from (provider, threads, download settings, cache, HF token); its own
/// `checkpoint` field is ignored, since each load overrides it with the checkpoint being built.
pub struct LayaRouterOptions {
    /// Template engine options shared by every checkpoint the router loads. Only the
    /// `checkpoint` field varies per load; every other field (execution provider, thread counts,
    /// `allow_download`, cache directory, HF token) is copied as-is.
    pub engine_options: LayaOptions,
    /// How many checkpoints may stay resident at once before the least-recently-used one is
    /// evicted. Always at least 1. Defaults to 2, as Python's `Router(max_loaded=2)` does, so
    /// alternating two languages does not reload a checkpoint on every request.
    pub max_loaded: usize,
    /// Checkpoint used when [`LayaRouter::route`] cannot find any letters in the state at all.
    /// Also used for undecided Latin text. Multilingual by default, matching Python 0.4.0;
    /// mostly-English deployments may explicitly select English.
    pub default: LayaCheckpoint,
    /// Whether a question-id set that exactly matches one of the four typed-decisions workflow
    /// signatures should automatically route to `TypedDecisions`. Off by default: that checkpoint
    /// is fine-tuned on four specific synthetic workflows and should never be a silent default.
    pub auto_task_detection: bool,
    /// Build and load every checkpoint immediately, so no request ever pays a cold load. See
    /// [`LayaRouter::preload`].
    pub preload: bool,
}

impl Default for LayaRouterOptions {
    fn default() -> Self {
        Self {
            engine_options: LayaOptions::default(),
            max_loaded: 2,
            default: LayaCheckpoint::Multilingual,
            auto_task_detection: false,
            preload: false,
        }
    }
}

/// Builds the [`LayaPredictor`] for one checkpoint. The production path
/// ([`LayaRouter::new`]) always builds a real [`LayaEngine`]; tests substitute a factory that
/// returns a fake predictor with no ONNX weights via [`LayaRouter::with_factory`].
type Factory =
    Box<dyn Fn(LayaCheckpoint, &LayaOptions) -> Result<Arc<dyn LayaPredictor>> + Send + Sync>;

struct RouterState {
    loaded: HashMap<LayaCheckpoint, Arc<dyn LayaPredictor>>,
    /// Least-recently-used first.
    order: Vec<LayaCheckpoint>,
    max_loaded: usize,
}

/// Lazily loads Laya checkpoints and sends each request to the right one. See the module docs.
///
/// ```no_run
/// use laya::laya_router::{LayaRouter, LayaRouterOptions};
///
/// let router = LayaRouter::new(LayaRouterOptions::default())?;
/// // router.predict(state, &questions, None, None, None)?;  // routes, then answers
/// # Ok::<(), laya::error::LayaError>(())
/// ```
pub struct LayaRouter {
    engine_options: LayaOptions,
    default: LayaCheckpoint,
    auto_task_detection: bool,
    lang_guess: Option<LangGuess>,
    factory: Factory,
    state: Mutex<RouterState>,
}

impl LayaRouter {
    /// Build a router whose checkpoints are real [`LayaEngine`]s, built with
    /// [`LayaEngine::create`] on first use.
    pub fn new(options: LayaRouterOptions) -> Result<Self> {
        Self::with_factory(options, |checkpoint, opts| {
            let mut opts = opts.clone();
            opts.checkpoint = checkpoint;
            let engine = LayaEngine::create(opts)?;
            Ok(Arc::new(engine) as Arc<dyn LayaPredictor>)
        })
    }

    /// A test-only engine-factory seam: builds a router whose checkpoints come from `factory`
    /// instead of a real [`LayaEngine`], so LRU/attach/preload/unload tests can run with fake
    /// predictors and no ~1.3-1.7 GB weights. `factory` receives the checkpoint being loaded and
    /// the (already checkpoint-substituted) engine options template.
    ///
    /// `#[doc(hidden)]` rather than crate-private: Rust has no visibility scoped to "this crate's
    /// own integration tests", and the integration test suite is a separate crate that links
    /// against this one, so the seam must be `pub` to reach it.
    #[doc(hidden)]
    pub fn with_factory(
        options: LayaRouterOptions,
        factory: impl Fn(LayaCheckpoint, &LayaOptions) -> Result<Arc<dyn LayaPredictor>>
        + Send
        + Sync
        + 'static,
    ) -> Result<Self> {
        let router = Self {
            engine_options: options.engine_options,
            default: options.default,
            auto_task_detection: options.auto_task_detection,
            lang_guess: None,
            factory: Box::new(factory),
            state: Mutex::new(RouterState {
                loaded: HashMap::new(),
                order: Vec::new(),
                max_loaded: options.max_loaded.max(1),
            }),
        };
        if options.preload {
            router.preload(None)?;
        }
        Ok(router)
    }

    /// Install a language hint that applies to every request, as Python's
    /// `Router(lang_guess=...)` does: checked after an explicit `lang` and a per-call hint, and
    /// before the built-in detection. See [`Self::route_with_lang_guess`].
    #[must_use]
    pub fn with_lang_guess(mut self, lang_guess: impl Into<LangGuess>) -> Self {
        self.lang_guess = Some(lang_guess.into());
        self
    }

    // ------------------------------------------------------------------ name resolution

    /// Aliases people are likely to type, resolved to the canonical checkpoint name before
    /// matching. Mirrors Python's `_ALIASES`.
    fn alias(key: &str) -> Option<&'static str> {
        match key {
            "en" | "laya" | "default" => Some("english"),
            "multi" | "ml" | "laya-multilingual" => Some("multilingual"),
            "typed" | "typed_decisions" | "laya-typed-decisions" | "decisions" => {
                Some("typed-decisions")
            }
            _ => None,
        }
    }

    /// Normalise a model name (case-insensitive, whitespace-trimmed, with aliases resolved) to a
    /// [`LayaCheckpoint`]. Ports `normalise_name`.
    pub fn normalise_name(name: &str) -> Result<LayaCheckpoint> {
        let key = name.trim().to_lowercase();
        let key = Self::alias(&key).unwrap_or(key.as_str());
        match key {
            "english" => Ok(LayaCheckpoint::English),
            "multilingual" => Ok(LayaCheckpoint::Multilingual),
            "typed-decisions" => Ok(LayaCheckpoint::TypedDecisions),
            _ => Err(LayaError::InvalidConfig(format!(
                "unknown model {}; choose one of ['english', 'multilingual', 'typed-decisions'] \
                 (or an alias: ['decisions', 'default', 'en', 'laya', 'laya-multilingual', \
                 'laya-typed-decisions', 'ml', 'multi', 'typed', 'typed_decisions'])",
                python_repr_str(name)
            ))),
        }
    }

    /// Name of the typed-decisions workflow whose question ids these are, else `None`. Requires
    /// an exact id-set match, so an unrelated schema that happens to contain `"urgency"` is never
    /// captured. Ports `match_typed_decisions_workflow`.
    pub fn match_typed_decisions_workflow(questions: &QuestionSet) -> Option<String> {
        let ids: HashSet<&str> = questions.ids().collect();
        for (workflow, signature) in TYPED_DECISION_WORKFLOWS {
            let sig: HashSet<&str> = signature.iter().copied().collect();
            if ids == sig {
                return Some((*workflow).to_string());
            }
        }
        None
    }

    // ------------------------------------------------------------------ loading

    /// Return the predictor for `name`, downloading and building it on first use. Concurrent
    /// callers requesting the same checkpoint share a single build (see the module docs).
    pub fn load(&self, name: &str) -> Result<Arc<dyn LayaPredictor>> {
        let key = Self::normalise_name(name)?;
        self.load_checkpoint(key)
    }

    fn load_checkpoint(&self, key: LayaCheckpoint) -> Result<Arc<dyn LayaPredictor>> {
        let mut state = self.state.lock().expect("router state mutex poisoned");
        if let Some(engine) = state.loaded.get(&key) {
            let engine = Arc::clone(engine);
            Self::touch(&mut state, key);
            return Ok(engine);
        }
        // The lock is held across the build itself (see the module docs): a second caller asking
        // for the same checkpoint blocks here rather than racing a duplicate build.
        let engine = (self.factory)(key, &self.engine_options)?;
        state.loaded.insert(key, Arc::clone(&engine));
        state.order.push(key);
        Self::evict(&mut state);
        Ok(engine)
    }

    fn touch(state: &mut RouterState, key: LayaCheckpoint) {
        state.order.retain(|&k| k != key);
        state.order.push(key);
    }

    fn evict(state: &mut RouterState) {
        while state.order.len() > state.max_loaded {
            let victim = state.order.remove(0);
            state.loaded.remove(&victim);
        }
    }

    /// Register an already-built predictor under `name` instead of loading a second copy. Raises
    /// `max_loaded` to fit it if needed, so attaching never immediately evicts what was just
    /// attached.
    pub fn attach(
        &self,
        name: &str,
        predictor: Arc<dyn LayaPredictor>,
    ) -> Result<Arc<dyn LayaPredictor>> {
        let key = Self::normalise_name(name)?;
        let mut state = self.state.lock().expect("router state mutex poisoned");
        state.loaded.insert(key, Arc::clone(&predictor));
        Self::touch(&mut state, key);
        state.max_loaded = state.max_loaded.max(state.loaded.len());
        Ok(predictor)
    }

    /// Download and build checkpoints up front so no request ever pays a model load. `names`
    /// defaults to every checkpoint (`english`, `multilingual`, `typed-decisions`, in that
    /// order). `max_loaded` is raised first, before any of the loads run, so the loop's own
    /// incremental loads never trigger a premature eviction of what preload is about to build.
    ///
    /// Unlike Python (whose single re-entrant lock covers the whole preload call), each
    /// checkpoint's load here takes and releases the router's lock separately — `std::sync::
    /// Mutex` is not re-entrant, so holding it across a loop of `self.load_checkpoint` calls
    /// (which each need the lock themselves) would deadlock. This does not change the
    /// observable end state: `load_checkpoint` already treats an already-resident checkpoint as
    /// a touch rather than a rebuild, so a preload racing another thread's load/attach still
    /// converges to the same result, just interleaved more finely than Python's.
    pub fn preload(&self, names: Option<&[&str]>) -> Result<()> {
        let keys: Vec<LayaCheckpoint> = match names {
            Some(names) => names
                .iter()
                .map(|n| Self::normalise_name(n))
                .collect::<Result<_>>()?,
            None => vec![
                LayaCheckpoint::English,
                LayaCheckpoint::Multilingual,
                LayaCheckpoint::TypedDecisions,
            ],
        };
        {
            let mut state = self.state.lock().expect("router state mutex poisoned");
            state.max_loaded = state.max_loaded.max(keys.len()).max(state.loaded.len());
        }
        for key in keys {
            self.load_checkpoint(key)?;
        }
        Ok(())
    }

    /// Free one checkpoint, or every checkpoint when `name` is `None`.
    pub fn unload(&self, name: Option<&str>) -> Result<()> {
        let mut state = self.state.lock().expect("router state mutex poisoned");
        match name {
            None => {
                state.loaded.clear();
                state.order.clear();
            }
            Some(name) => {
                let key = Self::normalise_name(name)?;
                state.loaded.remove(&key);
                state.order.retain(|&k| k != key);
            }
        }
        Ok(())
    }

    /// Currently-resident checkpoints, least-recently-used first.
    pub fn loaded(&self) -> Vec<LayaCheckpoint> {
        self.state
            .lock()
            .expect("router state mutex poisoned")
            .order
            .clone()
    }

    // ------------------------------------------------------------------ routing

    /// Decide which checkpoint to use, without loading or running anything.
    ///
    /// Precedence: explicit `model` > explicit `task` > detected workflow (opt-in) > explicit
    /// `lang` > `lang_guess` (see [`Self::route_with_lang_guess`]) > detected script/language >
    /// [`LayaRouterOptions::default`].
    pub fn route(
        &self,
        state: &Value,
        questions: Option<&QuestionSet>,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
    ) -> Result<RouteDecision> {
        self.route_with_lang_guess(state, questions, model, task, lang, None)
    }

    /// [`Self::route_with_lang_guess`] for a `bytes` state, which Python's `lang.analyse` accepts:
    /// the bytes are read as strict UTF-8 and invalid UTF-8 counts as no text. A `lang_guess`
    /// callback receives the original bytes as [`GuessInput::Bytes`], as Python's receives the
    /// `bytes` object.
    pub fn route_bytes(
        &self,
        state: &[u8],
        questions: Option<&QuestionSet>,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
        lang_guess: Option<&LangGuess>,
    ) -> Result<RouteDecision> {
        let text = Value::String(std::str::from_utf8(state).unwrap_or("").to_string());
        self.route_inner(
            GuessInput::Bytes(state),
            &text,
            questions,
            model,
            task,
            lang,
            lang_guess,
        )
    }

    /// [`Self::route`] with Python's per-call `lang_guess` hint.
    ///
    /// `lang_guess` is an opt-in hint -- a language code or a callback taking the state --
    /// checked after an explicit `lang` and before the built-in detection. It only answers "can
    /// the English checkpoint read this?", so any non-English code routes to the multilingual
    /// checkpoint. A hint that resolves to nothing (a blank code, `C`, `POSIX`, `und`, ... or a
    /// callback returning `None`) falls through to detection, which lets a language-identification
    /// model abstain. A hint installed with [`Self::with_lang_guess`] applies to every request,
    /// after this per-call one.
    pub fn route_with_lang_guess(
        &self,
        state: &Value,
        questions: Option<&QuestionSet>,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
        lang_guess: Option<&LangGuess>,
    ) -> Result<RouteDecision> {
        self.route_inner(
            GuessInput::Value(state),
            state,
            questions,
            model,
            task,
            lang,
            lang_guess,
        )
    }

    /// `Router._route`: `given` is the state as the caller passed it (what a hint callback sees),
    /// `state` the JSON text detection reads.
    #[allow(clippy::too_many_arguments)]
    fn route_inner(
        &self,
        given: GuessInput<'_>,
        state: &Value,
        questions: Option<&QuestionSet>,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
        lang_guess: Option<&LangGuess>,
    ) -> Result<RouteDecision> {
        if let Some(model) = model {
            let key = Self::normalise_name(model)?;
            return Ok(RouteDecision {
                model: key,
                reason: format!("explicit model={}", python_repr_str(model)),
                detection: None,
                workflow: None,
            });
        }

        if let Some(task) = task {
            let normalized = task.to_lowercase().replace('-', "_");
            let target = if normalized == "typed_decisions" {
                "typed-decisions"
            } else {
                task
            };
            let key = Self::normalise_name(target)?;
            return Ok(RouteDecision {
                model: key,
                reason: format!("explicit task={}", python_repr_str(task)),
                detection: None,
                workflow: None,
            });
        }

        let workflow = questions.and_then(Self::match_typed_decisions_workflow);
        if let Some(wf) = &workflow
            && self.auto_task_detection
        {
            return Ok(RouteDecision {
                model: LayaCheckpoint::TypedDecisions,
                reason: format!(
                    "question ids match the {} typed-decisions workflow",
                    python_repr_str(wf)
                ),
                detection: None,
                workflow: Some(wf.clone()),
            });
        }

        if let Some(lang) = lang {
            // An explicit `lang` is decisive only when the code names a language. Blank or
            // whitespace, and `C`/`POSIX`/`und`/`zxx`/`mul`, resolve to no usable hint, so they
            // fall through to `lang_guess`/detection exactly as an abstaining hint does.
            if let Some(english) = english_from_code(lang) {
                return Ok(RouteDecision {
                    model: if english {
                        LayaCheckpoint::English
                    } else {
                        LayaCheckpoint::Multilingual
                    },
                    reason: format!("explicit lang={}", python_repr_str(lang)),
                    detection: None,
                    workflow,
                });
            }
        }

        // Caller-supplied hint, per-call first, then the one installed on the router. Only a hint
        // that actually answers the question routes here; anything else falls through.
        let hints = [
            ("lang_guess", lang_guess),
            ("Router(lang_guess=...)", self.lang_guess.as_ref()),
        ];
        for (source, hint) in hints {
            if let Some(english) = hint.and_then(|h| h.resolve(given)) {
                return Ok(RouteDecision {
                    model: if english {
                        LayaCheckpoint::English
                    } else {
                        LayaCheckpoint::Multilingual
                    },
                    reason: format!(
                        "{source}: the caller identified this as {} text",
                        if english { "English" } else { "non-English" }
                    ),
                    detection: None,
                    workflow,
                });
            }
        }

        let det = LanguageDetection::analyse(state);
        let (key, reason) = if det.script == "unknown" {
            (
                self.default,
                format!(
                    "no letters detected in state; using default ({})",
                    self.default.subdir()
                ),
            )
        } else if det.script != "latin" {
            (
                LayaCheckpoint::Multilingual,
                format!(
                    "non-Latin script ({}, {:.0}% of letters); the English checkpoint cannot read it",
                    det.script,
                    100.0 * det.non_latin_fraction
                ),
            )
        } else if !det.is_english {
            if let (Some(lang), Some(mixed)) = (&det.language, det.mixed_segment.as_deref())
                && !mixed.is_empty()
            {
                (
                    LayaCheckpoint::Multilingual,
                    format!(
                        "Latin script, mostly English, but a line or field reads as {} ({}); \
                         the English checkpoint cannot read it",
                        python_repr_str(lang),
                        python_repr_str(prefix_chars(mixed, 60))
                    ),
                )
            } else if let Some(lang) = &det.language {
                (
                    LayaCheckpoint::Multilingual,
                    format!(
                        "Latin script but language looks like {}, not English",
                        python_repr_str(lang)
                    ),
                )
            } else {
                (
                    LayaCheckpoint::Multilingual,
                    format!(
                        "Latin script, language not identified but {:.0}% non-English letters; \
                         not safe for the English checkpoint",
                        100.0 * det.diacritic_rate
                    ),
                )
            }
        } else if det.language_undecided {
            (
                self.default,
                format!(
                    "Latin script, language not identified and no non-English letters; \
                     using default ({})",
                    self.default.subdir()
                ),
            )
        } else {
            (LayaCheckpoint::English, "English Latin text".to_string())
        };

        Ok(RouteDecision {
            model: key,
            reason,
            detection: Some(det),
            workflow,
        })
    }

    // ------------------------------------------------------------------ running

    /// Route, then answer every question in one forward pass on the chosen checkpoint. The
    /// result's [`LayaResult::routing`] records the decision.
    pub fn predict(
        &self,
        state: impl Into<Value>,
        questions: &QuestionSet,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
    ) -> Result<LayaResult> {
        self.predict_with_lang_guess(state, questions, model, task, lang, None)
    }

    /// [`Self::predict`] with a per-call `lang_guess` hint; see [`Self::route_with_lang_guess`].
    pub fn predict_with_lang_guess(
        &self,
        state: impl Into<Value>,
        questions: &QuestionSet,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
        lang_guess: Option<&LangGuess>,
    ) -> Result<LayaResult> {
        self.predict_with_options(
            state,
            questions,
            model,
            task,
            lang,
            lang_guess,
            &PredictOptions::default(),
        )
    }

    /// [`Self::predict_with_lang_guess`] with per-call [`PredictOptions`]. Ports
    /// `Router.predict(min_confidence=...)`: a `null` state is refused first, then the threshold is
    /// validated ([`LayaError::InvalidMinConfidence`]), both before anything is routed or loaded.
    /// The gate runs on the router's own result, so it also covers an attached predictor that
    /// knows nothing of it.
    ///
    /// The predictor is handed `lang` (else the language the routing detection found, as
    /// Python's `effective_lang`) and `options.max_len` / `options.head_max_len`;
    /// `options.lang` is not read here, the `lang` argument being the one that also routes.
    #[allow(clippy::too_many_arguments)]
    pub fn predict_with_options(
        &self,
        state: impl Into<Value>,
        questions: &QuestionSet,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
        lang_guess: Option<&LangGuess>,
        options: &PredictOptions,
    ) -> Result<LayaResult> {
        let state = state.into();
        // Python's `Router.predict` refuses `None` before it routes or loads anything.
        refuse_null_states(std::slice::from_ref(&state))?;
        let min_confidence = options.min_confidence.as_ref();
        if let Some(min_confidence) = min_confidence {
            min_confidence.validate()?;
        }
        let decision =
            self.route_with_lang_guess(&state, Some(questions), model, task, lang, lang_guess)?;
        let engine = self.load_checkpoint(decision.model)?;
        // Inference deliberately runs after the `Arc<dyn LayaPredictor>` clone above and outside
        // any lock: see the module docs on eviction safety.
        // Python passes `lang` always, and the token budgets only when set; the gate is the
        // router's own, so the predictor is not handed it.
        let forwarded = PredictOptions {
            min_confidence: None,
            lang: effective_lang(lang, &decision),
            max_len: options.max_len,
            head_max_len: options.head_max_len,
        };
        let mut results = [engine.predict_with_options(state, questions, &forwarded)?];
        apply_confidence_gate(&mut results, min_confidence);
        let [result] = results;
        Ok(result.with_routing(decision))
    }

    /// Route a heterogeneous request batch without loading any checkpoint. Ports
    /// `Router.route_batch`: each request is routed as [`Self::route_with_lang_guess`] routes it
    /// from its own `model`, `task`, `lang` and `lang_guess`, and the decisions come back in
    /// input order.
    ///
    /// A request whose state is `null` is refused with [`LayaError::InvalidState`], naming the
    /// request's index as Python does; the first failing request fails the call.
    pub fn route_batch(&self, requests: &[BatchRequest]) -> Result<Vec<RouteDecision>> {
        requests
            .iter()
            .enumerate()
            .map(|(index, request)| {
                if request.state.is_null() {
                    return Err(LayaError::InvalidState(format!(
                        "request {index} 'state' must not be None; pass a string, dict, or list"
                    )));
                }
                self.route_with_lang_guess(
                    &request.state,
                    Some(&request.questions),
                    request.model.as_deref(),
                    request.task.as_deref(),
                    request.lang.as_deref(),
                    request.lang_guess.as_ref(),
                )
            })
            .collect()
    }

    /// Route and answer a heterogeneous request batch with minimal model churn. Ports
    /// `Router.predict_batch`: requests are routed first (nothing loads if routing fails), grouped
    /// by checkpoint in order of first appearance, and within a checkpoint by question schema
    /// (order-sensitive: ids, types, instructions and options), so the states of one group share
    /// forward passes through [`LayaEngine::predict_batch`]. Each result carries its request's
    /// [`LayaResult::routing`] and the results come back in input order.
    ///
    /// `options` goes to every group: `batch_size` caps states per forward pass and
    /// `sort_by_length` reorders within a group only when that group clears the engine's own
    /// `1 < batch_size < group size` gate. No requests give an empty list. The first failing group
    /// fails the call. Returns [`LayaError::NotSupported`] when a chosen predictor (for example
    /// one registered with [`Self::attach`]) has no batch capability.
    ///
    /// `options.min_confidence` is validated before anything is routed
    /// ([`LayaError::InvalidMinConfidence`]) and gates every result of the call, as one batch-wide
    /// threshold: it takes no part in the grouping, and each result is gated here, by the router,
    /// so an attached predictor needs no support for it.
    ///
    /// Each request's `max_len` and `head_max_len` join the question schema in the grouping key,
    /// since one forward pass carries one budget; so does the request's `lang` (else the detected
    /// language), raw and unnormalised, but only when the predictor has per-language temperatures
    /// ([`LayaPredictor::has_lang_temperatures`]). The group's values are what the predictor is
    /// handed. As in Python, which has no such arguments on `Router.predict_batch`, a batch-wide
    /// `options.lang`, `options.max_len` or `options.head_max_len` is refused with
    /// [`LayaError::NotSupported`] (`... got an unexpected keyword argument '<name>'`); put them on
    /// the requests.
    pub fn predict_batch(
        &self,
        requests: &[BatchRequest],
        options: &BatchOptions,
    ) -> Result<Vec<LayaResult>> {
        // Python's `Router.predict_batch` takes no batch-wide language or token budget: each
        // request names its own, so the call itself is a `TypeError` before anything is read.
        for (given, name) in [
            (options.lang.is_some(), "lang"),
            (options.max_len.is_some(), "max_len"),
            (options.head_max_len.is_some(), "head_max_len"),
        ] {
            if given {
                return Err(LayaError::NotSupported(format!(
                    "Router.predict_batch() got an unexpected keyword argument '{name}'"
                )));
            }
        }
        let min_confidence = options.min_confidence.as_ref();
        if let Some(min_confidence) = min_confidence {
            min_confidence.validate()?;
        }
        let decisions = self.route_batch(requests)?;
        if decisions.is_empty() {
            return Ok(Vec::new());
        }

        // Insertion order keeps cache effects deterministic and collapses an interleaved
        // workload to one load per routed checkpoint for this call.
        let mut groups: Vec<(LayaCheckpoint, Vec<usize>)> = Vec::new();
        for (index, decision) in decisions.iter().enumerate() {
            match groups
                .iter_mut()
                .find(|(model, _)| *model == decision.model)
            {
                Some((_, indices)) => indices.push(index),
                None => groups.push((decision.model, vec![index])),
            }
        }

        // The gate is the router's: the predictors score ungated, whatever the caller asked for.
        // The language and budgets are each request's own, set per group below.
        let ungated = BatchOptions {
            min_confidence: None,
            lang: None,
            max_len: None,
            head_max_len: None,
            ..options.clone()
        };
        let mut results: Vec<Option<LayaResult>> = Vec::with_capacity(requests.len());
        results.resize_with(requests.len(), || None);
        for (model, indices) in groups {
            let predictor = self.load_checkpoint(model)?;
            let batcher = predictor.as_batch_predictor().ok_or_else(|| {
                LayaError::NotSupported(format!(
                    "the predictor for '{}' has no predict_batch, so a batch cannot share \
                     forward passes",
                    model.subdir()
                ))
            })?;

            // Order-sensitive at every level: options are positional, so two equal schemas with
            // different orders must not share a group.
            // Requests also split on their token budgets, since one forward pass carries one, and
            // on language, but only for an engine with per-language temperatures: the language is
            // otherwise unused, and splitting on it would break up a group that shares a pass. The
            // language is the request's own, else the one detected, compared as written.
            let has_lang_temperatures = predictor.has_lang_temperatures();
            type GroupKey = (String, Option<i64>, Option<i64>, Option<String>);
            let mut question_groups: Vec<(GroupKey, Vec<usize>)> = Vec::new();
            let mut last: Option<(&Arc<QuestionSet>, String)> = None;
            for &index in &indices {
                let request = &requests[index];
                let questions = &request.questions;
                let schema = match &last {
                    Some((seen, key)) if Arc::ptr_eq(seen, questions) => key.clone(),
                    _ => questions.schema_key(),
                };
                last = Some((questions, schema.clone()));
                let lang_key = if has_lang_temperatures {
                    request.lang.clone().or_else(|| {
                        decisions[index]
                            .detection
                            .as_ref()
                            .and_then(|detection| detection.language.clone())
                    })
                } else {
                    None
                };
                let key = (schema, request.max_len, request.head_max_len, lang_key);
                match question_groups.iter_mut().find(|(known, _)| *known == key) {
                    Some((_, items)) => items.push(index),
                    None => question_groups.push((key, vec![index])),
                }
            }

            for ((_, max_len, head_max_len, lang), items) in question_groups {
                let grouped = BatchOptions {
                    lang,
                    max_len,
                    head_max_len,
                    ..ungated.clone()
                };
                let states: Vec<Value> = items
                    .iter()
                    .map(|&index| requests[index].state.clone())
                    .collect();
                let batch =
                    batcher.predict_batch(&states, &requests[items[0]].questions, &grouped)?;
                if batch.len() != items.len() {
                    return Err(LayaError::Onnx(format!(
                        "internal error: predict_batch returned {} results for {} states",
                        batch.len(),
                        items.len()
                    )));
                }
                for (&index, result) in items.iter().zip(batch) {
                    results[index] = Some(result.with_routing(decisions[index].clone()));
                }
            }
        }

        let mut results = results
            .into_iter()
            .map(|result| {
                result.ok_or_else(|| {
                    LayaError::Onnx(
                        "internal error: batch execution did not produce every result".to_string(),
                    )
                })
            })
            .collect::<Result<Vec<_>>>()?;
        apply_confidence_gate(&mut results, min_confidence);
        Ok(results)
    }

    /// Route once, on the original state, then scan the whole state on the chosen checkpoint with
    /// [`LayaEngine::predict_long`]. The result's [`LayaResult::routing`] records the decision.
    ///
    /// Returns [`LayaError::NotSupported`] when the chosen predictor (for example one registered
    /// with [`Self::attach`]) has no scan capability, rather than truncating the state.
    ///
    /// The scan is handed `lang`, else the language the routing detection found; `options.lang` is
    /// not read here.
    pub fn predict_long(
        &self,
        state: impl Into<Value>,
        questions: &QuestionSet,
        options: &LongOptions,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
    ) -> Result<LayaResult> {
        self.predict_long_with_lang_guess(state, questions, options, model, task, lang, None)
    }

    /// [`Self::predict_long`] with a per-call `lang_guess` hint; see
    /// [`Self::route_with_lang_guess`].
    #[allow(clippy::too_many_arguments)]
    pub fn predict_long_with_lang_guess(
        &self,
        state: impl Into<Value>,
        questions: &QuestionSet,
        options: &LongOptions,
        model: Option<&str>,
        task: Option<&str>,
        lang: Option<&str>,
        lang_guess: Option<&LangGuess>,
    ) -> Result<LayaResult> {
        let state = state.into();
        // Python's `Router.predict` refuses `None` before it routes or loads anything.
        refuse_null_states(std::slice::from_ref(&state))?;
        let decision =
            self.route_with_lang_guess(&state, Some(questions), model, task, lang, lang_guess)?;
        let engine = self.load_checkpoint(decision.model)?;
        // Python hands the scan the explicit language, else the one detected.
        let options = LongOptions {
            lang: effective_lang(lang, &decision),
            ..options.clone()
        };
        let scanner = engine.as_long_predictor().ok_or_else(|| {
            LayaError::NotSupported(format!(
                "the predictor for '{}' has no predict_long, so a state longer than its window \
                 cannot be scanned",
                decision.model.subdir()
            ))
        })?;
        let result = scanner.predict_long(state, questions, &options)?;
        Ok(result.with_routing(decision))
    }
}

impl LayaPredictor for LayaRouter {
    /// [`LayaRouter::predict`] with every override left at its default (no explicit model, task
    /// or lang) — the entry point `laya_shortlist`'s generic `&dyn LayaPredictor` path uses.
    fn predict(&self, state: Value, questions: &QuestionSet) -> Result<LayaResult> {
        LayaRouter::predict(self, state, questions, None, None, None)
    }

    /// [`LayaRouter::predict_with_options`] with `options.lang` as the explicit language.
    fn predict_with_options(
        &self,
        state: Value,
        questions: &QuestionSet,
        options: &PredictOptions,
    ) -> Result<LayaResult> {
        LayaRouter::predict_with_options(
            self,
            state,
            questions,
            None,
            None,
            options.lang.as_deref(),
            None,
            options,
        )
    }

    fn as_long_predictor(&self) -> Option<&dyn LayaLongPredictor> {
        Some(self)
    }

    fn as_batch_predictor(&self) -> Option<&dyn LayaBatchPredictor> {
        Some(self)
    }
}

impl LayaBatchPredictor for LayaRouter {
    /// [`LayaRouter::predict_batch`] with every override left at its default: each state is
    /// routed on its own, as [`LayaRouter::predict`] routes one.
    fn predict_batch(
        &self,
        states: &[Value],
        questions: &QuestionSet,
        options: &BatchOptions,
    ) -> Result<Vec<LayaResult>> {
        let questions = Arc::new(questions.clone());
        let requests: Vec<BatchRequest> = states
            .iter()
            .map(|state| {
                let mut request = BatchRequest::new(state.clone(), Arc::clone(&questions));
                request.lang = options.lang.clone();
                request.max_len = options.max_len;
                request.head_max_len = options.head_max_len;
                request
            })
            .collect();
        // The language and budgets became each request's own above.
        let options = BatchOptions {
            lang: None,
            max_len: None,
            head_max_len: None,
            ..options.clone()
        };
        LayaRouter::predict_batch(self, &requests, &options)
    }
}

impl LayaLongPredictor for LayaRouter {
    /// [`LayaRouter::predict_long`] with every override left at its default.
    fn predict_long(
        &self,
        state: Value,
        questions: &QuestionSet,
        options: &LongOptions,
    ) -> Result<LayaResult> {
        LayaRouter::predict_long(
            self,
            state,
            questions,
            options,
            None,
            None,
            options.lang.as_deref(),
        )
    }
}

/// The language a call hands the predictor: the explicit `lang`, else the language the routing
/// detection found (`router.py`'s `effective_lang`). An empty detected language counts as none.
fn effective_lang(lang: Option<&str>, decision: &RouteDecision) -> Option<String> {
    lang.map(str::to_string).or_else(|| {
        decision
            .detection
            .as_ref()
            .and_then(|detection| detection.language.clone())
            .filter(|language| !language.is_empty())
    })
}

/// Python `repr()` for a string: single-quoted unless the string itself contains a `'` and no
/// `"`, in which case double quotes are used instead; backslash, the chosen quote character, and
/// the control characters are escaped, as is every character Python does not consider printable
/// (`\xNN`, `\uNNNN`, `\UNNNNNNNN`). The routing reasons embed caller text in this form (the
/// `lang` code and the mixed-language line), so the quoting has to agree with Python's.
pub(crate) fn python_repr_str(s: &str) -> String {
    let quote = if s.contains('\'') && !s.contains('"') {
        '"'
    } else {
        '\''
    };
    let mut out = String::with_capacity(s.len() + 2);
    out.push(quote);
    for c in s.chars() {
        match c {
            '\\' => out.push_str("\\\\"),
            c if c == quote => {
                out.push('\\');
                out.push(c);
            }
            '\n' => out.push_str("\\n"),
            '\r' => out.push_str("\\r"),
            '\t' => out.push_str("\\t"),
            c if !python_is_printable(c) => {
                let cp = c as u32;
                if cp < 0x100 {
                    out.push_str(&format!("\\x{cp:02x}"));
                } else if cp < 0x10000 {
                    out.push_str(&format!("\\u{cp:04x}"));
                } else {
                    out.push_str(&format!("\\U{cp:08x}"));
                }
            }
            c => out.push(c),
        }
    }
    out.push(quote);
    out
}

/// `str.isprintable()` for one character: everything except the "Other" and "Separator"
/// categories (control, format, surrogate, private use, unassigned, line/paragraph/space
/// separators), with the ASCII space as the one printable separator.
fn python_is_printable(c: char) -> bool {
    use unicode_general_category::{GeneralCategory as G, get_general_category};
    c == ' '
        || !matches!(
            get_general_category(c),
            G::Control
                | G::Format
                | G::Surrogate
                | G::PrivateUse
                | G::Unassigned
                | G::LineSeparator
                | G::ParagraphSeparator
                | G::SpaceSeparator
        )
}