AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
systemone: decision cache flywheel — write-back, source, human corrections (#1061)
xerj-org5ca08dc3b63d
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.mdindex b75aeb902..d60b2f535 100644--- a/docs/DECISIONS.md+++ b/docs/DECISIONS.md@@ -67,6 +67,58 @@ is `history` or `local`, and a local answer carries`model: "xerj-decide-local-1"` — the same never-a-Jev-name discipline as thevote's own id.+## The flywheel: every tier-2+ answer is cached (#1061)++The ladder's answers are not spent when served. Every tier-2 (local head)+answer — and every tier-3 answer, when that tier exists — is written back to+the `[decisions]` index as an ordinary document, so the answers a node+computes become the history that answers the next request:++- **What is written**: the configured `text_field` (the payload as decided),+ the configured `label_field` (the winning label), and the fixed `p`+ (probability served), `source` (the tier that produced it) and `ts` (RFC+ 3339) fields. Because the text and label ride the configured fields, a+ cached answer is retrievable and votable exactly like a seeded example.+- **When**: after the response is complete. The write-back is spawned, never+ awaited by the request, and can neither delay nor fail it — an index that+ cannot be created or written is a log line, not an error the caller sees.+ A configured-but-missing index is created by the write-back itself, which+ is how a node armed with the local tier bootstraps its own history.+- **What is not written**: history-tier answers (they are the index already —+ re-writing them would double their vote), and `/_decide` abstains (an+ abstain was not an answer; caching it would seed the history with a doubt).+ A node with no `[decisions] index` at all has nowhere to cache.+- **`source` on the wire**: every answer names its tier — `/_decide` at the+ top level, `/v1/systemone` per-question in `decisions.evidence.*.source`+ (the same string as `tier`, under the flywheel's name, and the value+ written into the cached document).++No replay claim is made yet: the Banking77 replay gate in the issue (after+2,000 cached hosted answers, ≥ 80 % of later traffic answered by history at+≥ 0.8 confidence and ≥ 0.97 accuracy) is a release-time measurement, and the+TypeSafe/Jev API-terms question on retaining provider outputs must be+answered before a tier-3 write-back ships.++## Human corrections: `human: true`, weighted `decisions.human_weight`×++A correction is an ordinary document with one extra field:++```sh+curl -X PUT localhost:9200/judgements/_doc/fix-42 -H 'content-type: application/json' \+ -d '{"text": "the message that was mislabelled", "label": "refund", "human": true}'+```++No new endpoint — the ordinary indexing path is the corrections path. In the+vote, a `human: true` neighbour weighs `decisions.human_weight`× (default+2.0, the issue's ≥ 2x floor; 1.0 makes corrections ordinary neighbours) its+reciprocal rank, so one correction outranks the cached answers it corrects.+Everything else keeps the 1/rank arithmetic the published measurements used.+Only a JSON boolean `true` earns the weight — `"true"`, `1`, and a missing+field are ordinary history — and `decisions.human_weight` must be a positive+number or the config refuses to boot (0 would erase a correction from the+vote, a negative value would invert it). The boosted weight is the weight the+vote used, so `/_decide` shows it in each neighbour's `weight`.+## Set it up```toml@@ -78,6 +130,7 @@ label_field = "label"text_field = "text"positive_label = "true" # the label whose share a noul answersmin_confidence = 0.0 # /_decide abstains below this+human_weight = 2.0 # vote-weight multiplier for history docs carrying human: true```The history index is ordinary documents: one per example, with the text the@@ -141,9 +194,9 @@ name are `choice` answers, not query vocabulary.Response: `{model, answers, usage}` with `answers` keyed exactly by thequestion ids sent, each answer carrying only its documented fields (clientsvalidate strictly). XERJ's extras ride at the top level in `decisions` — the-index, k, the requested model, and per-question evidence (winning label, its-support, how many neighbours were found and voted) — where every verified-client ignores them.+index, k, the requested model, and per-question evidence (the tier and source+that answered, winning label, its support, how many neighbours were found and+voted) — where every verified client ignores them.Two deliberate breaks from the hosted API, both tested so they cannot regress:@@ -169,11 +222,14 @@ tier is armed), `question` (required), `k` (default the configured 10,clamped 1..100), `positive_label` (default the configured one). The responsereturns the label, its confidence, the verdict (`abstain` below`decisions.min_confidence`, or when no labelled neighbour exists), the **tier**-that answered, and the **neighbours** — each with `_id`, `label`, `_score`-(the engine's own BM25 score), `weight` (1/rank), and the text — so every-answer can be checked against the evidence that produced it. A local-tier-answer has no neighbours (having none is what tier 2 means) and carries-`tier: "local"`, `model: "xerj-decide-local-1"`.+and **source** that answered (the same tier name under both fields), and the+**neighbours** — each with `_id`, `label`, `_score` (the engine's own BM25+score), `weight` (1/rank, × `decisions.human_weight` for a `human: true`+document), and the text — so every answer can be checked against the evidence+that produced it. A local-tier answer has no neighbours (having none is what+tier 2 means) and carries `tier: "local"`, `source: "local"`,+`model: "xerj-decide-local-1"`; when it did not abstain and the request named+an index, it is cached into that index (the flywheel above).```shcurl -s localhost:9200/_decide -H 'content-type: application/json' \diff --git a/engine/crates/xerj-api/src/systemone_api.rs b/engine/crates/xerj-api/src/systemone_api.rsindex 38f870306..c1d7db857 100644--- a/engine/crates/xerj-api/src/systemone_api.rs+++ b/engine/crates/xerj-api/src/systemone_api.rs@@ -39,6 +39,26 @@//! as before: 503 with no `[decisions]` index, 422 `no_support` / on a//! history that cannot answer.//!+//! # The decision cache flywheel (#1061)+//!+//! Every tier-2 (and, when it is built, tier-3) answer is written back to+//! the `[decisions]` index as an ordinary document — the configured+//! `text_field`/`label_field` plus `p`, `source`, `ts` — so the answers a+//! node computes once become the history that answers the next request.+//! History-tier answers are not written: they ARE the index already, and+//! double-writing them would double-count their vote. The write-back is+//! spawned, never awaited by the request, and cannot fail it: a missing or+//! unwritable index is a log line, not an error the caller sees. Every+//! answer names its tier in a `source` field (`/_decide` top level,+//! `/v1/systemone` per-question in `decisions.evidence.*.source` — the same+//! string as `tier`, under the flywheel's name).+//!+//! Human corrections ride the same index: a history document carrying+//! `human: true` (written through the ordinary indexing API) is weighted+//! [`DecisionsConfig::human_weight`]× (default 2.0, the issue's ≥ 2x floor)+//! in the vote — see [`neighbour_weight`]. The weight is read in the vote+//! aggregation, so a correction outranks the cached answers it corrects.+//!//! # The wire contract, and where we deliberately break it//!//! Request: `{ state, model, questions: { id: { type, instructions, criteria } } }`.@@ -172,6 +192,118 @@ pub(crate) fn resolve_tier(support: &HistorySupport, local_enabled: bool) -> Opt}}+impl DecideTier {+ /// The tier's wire name — the `tier`/`source` field value on both+ /// surfaces and the `source` written into every cached answer, so all+ /// three name the same thing by construction.+ pub(crate) fn as_str(self) -> &'static str {+ match self {+ DecideTier::History => "history",+ DecideTier::Local => "local",+ DecideTier::Hosted => "hosted",+ }+ }+}++// ─────────────────────────────────────────────────────────────────────────────+// The decision cache flywheel (#1061)+// ─────────────────────────────────────────────────────────────────────────────++/// One answer bound for the decisions index: the payload as decided, the+/// label it won, the probability that was served, and the tier that produced+/// it. History-tier answers never appear here — they are the index already,+/// and writing them back would double their vote on the next request.+struct CachedAnswer {+ /// The text that was decided (the question's vote text, clipped).+ text: String,+ /// The winning label.+ label: String,+ /// The probability served for that label.+ p: f64,+ /// The tier that produced the answer — [`DecideTier::as_str`].+ source: &'static str,+}++/// The document one cached answer writes back as. Pure, and unit-tested: the+/// field names ARE the flywheel's contract — the configured text/label fields+/// (so the cached answer is retrievable and votable exactly like a seeded+/// example) plus the fixed `p`, `source`, `ts` of issue #1061.+fn write_back_doc(answer: &CachedAnswer, cfg: &DecisionsConfig, ts: &str) -> Value {+ // Built by hand, not json!: two of the keys are configuration, not+ // syntax, and the json! macro only takes literal keys.+ let mut doc = serde_json::Map::new();+ doc.insert(cfg.text_field.clone(), json!(answer.text));+ doc.insert(cfg.label_field.clone(), json!(answer.label));+ doc.insert("p".into(), json!(round6(answer.p)));+ doc.insert("source".into(), json!(answer.source));+ doc.insert("ts".into(), json!(ts));+ Value::Object(doc)+}++/// Cache `answers` into the configured decisions index — the flywheel's+/// write-back. Spawned and never awaited by the request, so it can neither+/// delay nor fail it: every error (index cannot be created or reached,+/// document rejected) is a log line naming the index, and the answer the+/// caller already holds stands. Skipped outright when no index is configured+/// — a node with no `[decisions] index` has nowhere to cache, which is the+/// documented state the local tier answers for.+fn spawn_write_back(state: &AppState, cfg: &DecisionsConfig, answers: Vec<CachedAnswer>) {+ if answers.is_empty() || cfg.index.is_empty() {+ return;+ }+ let state = state.clone();+ let cfg = cfg.clone();+ tokio::spawn(async move {+ // The same write path PUT /{index}/_doc takes, including+ // auto-creating a configured-but-missing index: that is how a node+ // armed with the local tier bootstraps its own history.+ let idx = match state.engine.get_or_create_index(&cfg.index) {+ Ok(idx) => idx,+ Err(e) => {+ tracing::warn!(+ index = %cfg.index, error = %e,+ "decide flywheel: could not open the decisions index for write-back; the \+ answer was served but not cached"+ );+ return;+ }+ };+ for answer in &answers {+ let ts = chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Millis, true);+ match idx+ .index_document(None, write_back_doc(answer, &cfg, &ts))+ .await+ {+ Ok(_) => state.metrics.record_doc_indexed(&cfg.index),+ Err(e) => tracing::warn!(+ index = %cfg.index, error = %e,+ "decide flywheel: one cached answer was not written; the answer was served"+ ),+ }+ }+ });+}++/// One neighbour's vote weight: reciprocal rank, scaled by+/// [`DecisionsConfig::human_weight`] when the history document carries+/// `human: true` — a correction outranks the answers it corrects. Pure, and+/// unit-tested: this is the flywheel's correction arithmetic.+fn neighbour_weight(rank: usize, human: bool, human_weight: f64) -> f64 {+ let base = 1.0 / (rank as f64 + 1.0);+ if human {+ base * human_weight+ } else {+ base+ }+}++/// Whether a history document is a human correction. Strict — `human: true`+/// as a JSON boolean, exactly — because a correction's weight is trust, and+/// "true"/1/yes must not silently earn it.+fn is_human_correction(source: &Value) -> bool {+ source.get("human") == Some(&Value::Bool(true))+}+// ─────────────────────────────────────────────────────────────────────────────// DecideSettings — the node's ladder configuration// ─────────────────────────────────────────────────────────────────────────────@@ -503,7 +635,8 @@ pub async fn systemone(evidence.insert(id.clone(),json!({- "tier": "history",+ "tier": DecideTier::History.as_str(),+ "source": DecideTier::History.as_str(),"label": best_label,"support": round6(best_weight / total),"neighbours": labelled.len(),@@ -538,6 +671,11 @@ pub async fn systemone(// from.#[cfg_attr(not(feature = "decide-local"), allow(unused_mut))]let mut local_answered = 0usize;+ // The flywheel's cargo (#1061): every answer tiers 2+ produced here is+ // written back to the decisions index after the response is built. The+ // hosted tier, when it is built, pushes into the same vector.+ #[cfg_attr(not(feature = "decide-local"), allow(unused_mut))]+ let mut cached: Vec<CachedAnswer> = Vec::new();#[cfg(feature = "decide-local")]if !local_needed.is_empty() {let Some(local) = state.decide.local_tier() else {@@ -618,17 +756,29 @@ pub async fn systemone(evidence.insert(id.clone(),json!({- "tier": "local",+ "tier": DecideTier::Local.as_str(),+ "source": DecideTier::Local.as_str(),"model": LOCAL_MODEL_ID,"label": best_label,"support": round6(*best_score as f64),"hypotheses": Value::Object(by_label),}),);+ cached.push(CachedAnswer {+ text: question.vote_text.clone(),+ label: best_label.clone(),+ p: *best_score as f64,+ source: DecideTier::Local.as_str(),+ });local_answered += 1;}}+ // The flywheel's write-back (#1061): cache what tiers 2+ answered, after+ // the answer is complete and without ever holding the response for it.+ // History-tier answers are not in `cached` — they are the index already.+ spawn_write_back(&state, &cfg, cached);+let index_echo = if cfg.index.is_empty() {LOCAL_MODEL_ID.to_string()} else {@@ -768,7 +918,7 @@ pub async fn decide(let Some(local) = local.as_ref() else {unreachable!("resolve_tier names Local only when the tier is armed");};- return decide_local(&state, local, &question, &positive, &index, k, started).await;+ return decide_local(&state, local, &question, &cfg, started).await;}}let neighbours = match neighbours {@@ -834,7 +984,8 @@ pub async fn decide("confidence": round6(confidence),"abstain": abstain,"neighbours": neighbour_list,- "tier": "history",+ "tier": DecideTier::History.as_str(),+ "source": DecideTier::History.as_str(),"took_ms": started.elapsed().as_millis() as u64,});if let Some(r) = reason {@@ -846,18 +997,22 @@ pub async fn decide(/// The local tier's `/_decide` answer: the head's probabilities over the/// positive label and its negation, tier-tagged, abstaining by the same/// `decisions.min_confidence` rule the history vote uses. `neighbours` is-/// empty — having none is exactly what tier 2 means here.+/// empty — having none is exactly what tier 2 means here. A non-abstaining+/// answer over a named index is cached into it (the flywheel, #1061); an+/// abstain is not — it was not an answer, and caching it would seed the+/// history with a doubt.#[cfg(feature = "decide-local")]async fn decide_local(state: &AppState,local: &LocalTier,question: &str,- positive: &str,- index: &str,- k: usize,+ cfg: &DecisionsConfig,started: Instant,) -> axum::response::Response {- let defaults = &state.config.decisions;+ // `cfg` is the caller's per-request view: its index is the index THIS+ // request named (empty when it named none), so the flywheel caches into+ // the index the question was asked of — never a different one.+ let (index, k, positive) = (&cfg.index, cfg.k, cfg.positive_label.as_str());let labels = [positive.to_string(), format!("not {positive}")];let requests = vec![xerj_ai::decide::ScoreRequest {premise: clip(question, MAX_VOTE_TEXT_CHARS),@@ -876,7 +1031,19 @@ async fn decide_local(} else {(labels[1].clone(), scores[1] as f64)};- let abstain = confidence < defaults.min_confidence;+ let abstain = confidence < cfg.min_confidence;+ if !abstain {+ spawn_write_back(+ state,+ cfg,+ vec![CachedAnswer {+ text: clip(question, MAX_VOTE_TEXT_CHARS),+ label: label.clone(),+ p: confidence,+ source: DecideTier::Local.as_str(),+ }],+ );+ }state.metrics.record_query(LOCAL_MODEL_ID,"decide_local",@@ -890,14 +1057,15 @@ async fn decide_local("confidence": round6(confidence),"abstain": abstain,"neighbours": [],- "tier": "local",+ "tier": DecideTier::Local.as_str(),+ "source": DecideTier::Local.as_str(),"model": LOCAL_MODEL_ID,"took_ms": started.elapsed().as_millis() as u64,});if abstain {resp["reason"] = json!(format!("confidence {:.3} below decisions.min_confidence {:.3}",- confidence, defaults.min_confidence+ confidence, cfg.min_confidence));}Json(resp).into_response()@@ -1077,7 +1245,13 @@ async fn vote_neighbours(let mut out = Vec::with_capacity(result.hits.len());for (i, hit) in result.hits.into_iter().enumerate() {if let Some(label) = hit.source.get(&cfg.label_field).and_then(Value::as_str) {- out.push((1.0 / (i as f64 + 1.0), label.to_string(), hit));+ // The flywheel's one refinement to the raw vote (#1061): a+ // document the operator marked `human: true` — a correction,+ // indexed through the ordinary write path — weighs+ // `decisions.human_weight`× its reciprocal rank. Everything else+ // keeps the 1/rank arithmetic the published measurements used.+ let w = neighbour_weight(i, is_human_correction(&hit.source), cfg.human_weight);+ out.push((w, label.to_string(), hit));}}Ok(out)@@ -1370,4 +1544,117 @@ mod tests {};assert_eq!(labels_for(&choice, &cfg), vec!["billing", "tech"]);}++ // ─────────────────────────────────────────────────────────────────────────+ // The flywheel (#1061)+ // ─────────────────────────────────────────────────────────────────────────++ /// The issue's arithmetic, exactly: at the default 2× a correction's+ /// vote weight doubles, and doubling flips a concrete two-label vote the+ /// 1/rank arithmetic settles the other way. Ranks are BM25 ranks; the+ /// label sums are what `/_decide` and `/v1/systemone` aggregate.+ #[test]+ fn two_x_human_weight_changes_the_vote() {+ // Rank 0 is a plain `billing` neighbour; ranks 1 and 2 are human+ // corrections saying `tech`.+ let billing = neighbour_weight(0, false, 2.0);+ let tech_at_1 = neighbour_weight(1, true, 2.0);+ let tech_at_2 = neighbour_weight(2, true, 2.0);+ // Sanity: plain weights are the reciprocal-rank arithmetic the+ // published measurements used, untouched.+ assert_eq!(neighbour_weight(0, false, 9.9), 1.0);+ assert_eq!(neighbour_weight(1, false, 9.9), 0.5);+ // With the boost disabled (weight 1.0) billing wins: 1.0 against+ // 0.5 + 1/3.+ let tech_unboosted = neighbour_weight(1, true, 1.0) + neighbour_weight(2, true, 1.0);+ assert!(+ billing > tech_unboosted,+ "at weight 1.0 corrections are ordinary neighbours: {billing} vs {tech_unboosted}"+ );+ // At the default 2× the corrections win: 0.5·2 + (1/3)·2 > 1.0.+ let tech_boosted = tech_at_1 + tech_at_2;+ assert!(+ tech_boosted > billing,+ "at weight 2.0 corrections outrank the vote they correct: {tech_boosted} vs {billing}"+ );+ // And the default IS 2.0 — the issue's ≥ 2x floor.+ assert_eq!(DecisionsConfig::default().human_weight, 2.0);+ }++ /// The weight is configuration: `[decisions] human_weight` parses, and a+ /// value that would erase or invert a correction's vote is refused at+ /// config load, not discovered in a wrong answer.+ #[test]+ fn human_weight_is_configurable_and_range_checked() {+ let cfg = xerj_common::config::Config::from_toml_str(+ "[decisions]\nindex = \"judgements\"\nhuman_weight = 3.5\n",+ )+ .expect("toml");+ assert_eq!(cfg.decisions.human_weight, 3.5);+ assert_eq!(+ cfg.decisions.index, "judgements",+ "the rest of the section parses unchanged"+ );+ for bad in ["0.0", "-2.0", "nan"] {+ let err = xerj_common::config::Config::from_toml_str(&format!(+ "[decisions]\nhuman_weight = {bad}\n"+ ))+ .expect_err(bad);+ assert!(err.to_string().contains("human_weight"), "{bad}: {err}");+ }+ }++ /// The cached answer's document is the flywheel's contract: the+ /// configured text/label fields (so it votes like any seeded example) and+ /// the fixed p / source / ts the issue names. Nothing else — a cached+ /// answer is NOT marked human, or it would outrank itself forever.+ #[test]+ fn write_back_doc_carries_the_flywheel_fields() {+ let cfg = DecisionsConfig {+ text_field: "message".into(),+ label_field: "intent".into(),+ ..DecisionsConfig::default()+ };+ let doc = write_back_doc(+ &CachedAnswer {+ text: "refund my subscription".into(),+ label: "refund".into(),+ p: 0.8712345678,+ source: DecideTier::Local.as_str(),+ },+ &cfg,+ "2026-09-30T00:00:00.123Z",+ );+ assert_eq!(doc["message"], "refund my subscription");+ assert_eq!(doc["intent"], "refund");+ assert_eq!(doc["p"], 0.871235, "probability rounded like the wire");+ assert_eq!(doc["source"], "local");+ assert_eq!(doc["ts"], "2026-09-30T00:00:00.123Z");+ assert_eq!(+ doc.as_object().map(|o| o.len()),+ Some(5),+ "exactly the five fields: {doc}"+ );+ }++ /// Only a JSON boolean `true` earns the correction weight — "true", 1,+ /// and a missing field are ordinary history.+ #[test]+ fn only_a_boolean_true_marks_a_human_correction() {+ assert!(is_human_correction(&json!({"human": true})));+ assert!(!is_human_correction(&json!({"human": "true"})));+ assert!(!is_human_correction(&json!({"human": 1})));+ assert!(!is_human_correction(&json!({"human": false})));+ assert!(!is_human_correction(&json!({"text": "no human field"})));+ assert!(!is_human_correction(&Value::Null));+ }++ /// Every tier's wire name is stable — it is `tier`, `source`, and the+ /// cached document's `source` all at once.+ #[test]+ fn tier_names_are_the_source_names() {+ assert_eq!(DecideTier::History.as_str(), "history");+ assert_eq!(DecideTier::Local.as_str(), "local");+ assert_eq!(DecideTier::Hosted.as_str(), "hosted");+ }}diff --git a/engine/crates/xerj-api/tests/decide_flywheel.rs b/engine/crates/xerj-api/tests/decide_flywheel.rsnew file mode 100644index 000000000..a76b53468--- /dev/null+++ b/engine/crates/xerj-api/tests/decide_flywheel.rs@@ -0,0 +1,524 @@+//! End-to-end checks for the decision cache flywheel (issue #1061): every+//! tier-2+ answer written back to the `[decisions]` index, every answer+//! naming its tier in a `source` field, and human corrections weighted+//! `decisions.human_weight`× (default 2.0) in the vote.+//!+//! Three of the contracts here hold in ANY build (they are the history+//! tier's, and the default node is history-only), so they are not+//! feature-gated:+//!+//! - the `source` field on both surfaces (`/_decide` top level,+//! `/v1/systemone` per-question in `decisions.evidence.*.source`), without+//! touching the strictly-parsed answer objects — the wire compatibility+//! #1072 established;+//! - a history document carrying `human: true`, indexed through the ordinary+//! `PUT /{index}/_doc` path, is weighted by the configured `human_weight`+//! in the vote — at 1.0 the plain vote wins, at the default 2.0 the+//! corrections flip it;+//! - history-tier answers are never written back (the write-back tests below+//! assert the negative half of that on the same node).+//!+//! The write-back itself needs an answer produced by tier 2, so those tests+//! are gated on `decide-local` and arm the same tiny untrained fixture as+//! `systemone_local_decide.rs` — the probabilities are reproducible+//! arithmetic, not judgement quality, and nothing here measures accuracy.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use serde_json::{json, Value};+use tower::ServiceExt;++use xerj_api::state::AppState;++// ─────────────────────────────────────────────────────────────────────────────+// The node+// ─────────────────────────────────────────────────────────────────────────────++struct Node {+ native: axum::Router,+ app: axum::Router,+ _dir: tempfile::TempDir,+}++/// A node whose `[decisions]` block names `index`, with `human_weight`+/// overrideable (the flywheel's one new setting). No local tier: the+/// ungated tests are the history tier's contract.+async fn node_with_decisions(index: &str, human_weight: f64, min_confidence: f64) -> Node {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ config.decisions.index = index.to_string();+ config.decisions.human_weight = human_weight;+ config.decisions.min_confidence = min_confidence;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = AppState::new(config, engine, metrics);+ Node {+ native: xerj_api::router::build_native_router(state.clone()),+ app: xerj_api::router::build_es_compat_router(state),+ _dir: dir,+ }+}++impl Node {+ async fn call(+ &self,+ router: &axum::Router,+ method: &str,+ path: &str,+ body: Value,+ ) -> (StatusCode, Value) {+ let response = router+ .clone()+ .oneshot(+ Request::builder()+ .method(method)+ .uri(path)+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .expect("request"),+ )+ .await+ .expect("response");+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .expect("body");+ (+ status,+ serde_json::from_str(&String::from_utf8_lossy(&bytes)).unwrap_or(Value::Null),+ )+ }++ async fn systemone(&self, body: Value) -> (StatusCode, Value) {+ self.call(&self.native.clone(), "POST", "/v1/systemone", body)+ .await+ }+ async fn decide(&self, body: Value) -> (StatusCode, Value) {+ self.call(&self.app.clone(), "POST", "/_decide", body).await+ }+ async fn put_doc(&self, index: &str, id: &str, doc: Value) {+ let (st, b) = self+ .call(+ &self.app.clone(),+ "PUT",+ &format!("/{index}/_doc/{id}"),+ doc,+ )+ .await;+ assert!(st.is_success(), "index {index}/{id}: {st} {b}");+ }+ async fn create_index(&self, index: &str) {+ let (st, b) = self+ .call(+ &self.app.clone(),+ "PUT",+ &format!("/{index}"),+ json!({"mappings": {"properties": {+ "text": {"type": "text"},+ "label": {"type": "keyword"}+ }}}),+ )+ .await;+ assert!(st.is_success(), "create {index}: {st} {b}");+ }+ async fn refresh(&self, index: &str) {+ let (st, b) = self+ .call(+ &self.app.clone(),+ "POST",+ &format!("/{index}/_refresh"),+ json!({}),+ )+ .await;+ assert!(st.is_success(), "refresh {index}: {st} {b}");+ }+ /// Every document's `_source` in the index, refreshed first.+ #[cfg_attr(not(feature = "decide-local"), allow(dead_code))]+ async fn docs(&self, index: &str) -> Vec<Value> {+ self.refresh(index).await;+ self.search_all(index).await+ }+ /// `_source` of every document, no refresh.+ #[cfg_attr(not(feature = "decide-local"), allow(dead_code))]+ async fn search_all(&self, index: &str) -> Vec<Value> {+ let (st, r) = self+ .call(+ &self.app.clone(),+ "POST",+ &format!("/{index}/_search"),+ json!({"query": {"match_all": {}}, "size": 100}),+ )+ .await;+ assert!(st.is_success(), "search {index}: {st} {r}");+ r["hits"]["hits"]+ .as_array()+ .map(|hits| hits.iter().map(|h| h["_source"].clone()).collect())+ .unwrap_or_default()+ }+ /// Poll until the index holds at least `n` documents — the write-back is+ /// a spawned task, so the test yields until it lands (and says so, with+ /// what it saw, if it never does). Tolerates the index not existing YET:+ /// in the bootstrap case the write-back itself creates it.+ #[cfg_attr(not(feature = "decide-local"), allow(dead_code))]+ async fn wait_for_docs(&self, index: &str, n: usize) -> Vec<Value> {+ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);+ loop {+ let (rst, _) = self+ .call(+ &self.app.clone(),+ "POST",+ &format!("/{index}/_refresh"),+ json!({}),+ )+ .await;+ if rst.is_success() {+ let docs = self.search_all(index).await;+ if docs.len() >= n {+ return docs;+ }+ }+ if std::time::Instant::now() > deadline {+ panic!("write-back did not land: wanted {n} docs in `{index}`");+ }+ tokio::time::sleep(std::time::Duration::from_millis(25)).await;+ }+ }+}++// ─────────────────────────────────────────────────────────────────────────────+// The source field — every build, both surfaces, wire-compatible+// ─────────────────────────────────────────────────────────────────────────────++/// Every answer names the tier that produced it: `/_decide` at the top+/// level, `/v1/systemone` per-question in the evidence — and the strictly+/// parsed answer objects are untouched, exactly as #1072 left them (extras+/// ride in `decisions`, which every verified client ignores).+#[tokio::test]+async fn every_answer_names_its_source_on_both_surfaces() {+ let node = node_with_decisions("history", 2.0, 0.0).await;+ node.create_index("history").await;+ node.put_doc(+ "history",+ "h1",+ json!({"text": "refund refund refund asked for money back", "label": "true"}),+ )+ .await;+ node.put_doc(+ "history",+ "h2",+ json!({"text": "refund policy says money back", "label": "true"}),+ )+ .await;+ node.refresh("history").await;++ // /_decide: the audit surface, top level.+ let (st, r) = node+ .decide(json!({"index": "history", "question": "refund asked for money back"}))+ .await;+ assert!(st.is_success(), "{st} {r}");+ assert_eq!(r["tier"], "history", "{r}");+ assert_eq!(r["source"], "history", "source names the tier: {r}");++ // /v1/systemone: per question, in the evidence — and the answer object+ // itself still carries ONLY its documented fields.+ let (st, r) = node+ .systemone(json!({+ "state": {"documents": {"doc_0": "refund asked for money back"}},+ "questions": {"d0": {"type": "noul", "instructions": "Is `documents.doc_0` about refunds?"}}+ }))+ .await;+ assert!(st.is_success(), "{st} {r}");+ assert_eq!(r["decisions"]["evidence"]["d0"]["tier"], "history", "{r}");+ assert_eq!(+ r["decisions"]["evidence"]["d0"]["source"], "history",+ "source names the tier: {r}"+ );+ let answer = r["answers"]["d0"].as_object().expect("answer: {r}");+ let mut keys = answer.keys().collect::<Vec<_>>();+ keys.sort();+ assert_eq!(+ keys,+ ["noul", "type"],+ "the answer object is unchanged — extras stay in `decisions`: {r}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// Human corrections — ordinary write path, weighted ≥ 2x in the vote+// ─────────────────────────────────────────────────────────────────────────────++/// The flywheel's correction arithmetic over the real vote. One history+/// document says `billing` and ranks first (it matches every query term);+/// two HUMAN corrections say `tech` and rank below it. At `human_weight = 1.0`+/// (corrections are ordinary neighbours) billing wins; at the DEFAULT 2.0 the+/// corrections outweigh it — the same documents, the same ranks, only the+/// weight differs, so the flip is the weight's and nothing else's.+#[tokio::test]+async fn a_two_x_human_correction_flips_the_vote_the_plain_arithmetic_settles() {+ for (human_weight, expected) in [(1.0, "billing"), (2.0, "tech")] {+ let node = node_with_decisions("history", human_weight, 0.0).await;+ node.create_index("history").await;+ node.put_doc(+ "history",+ "plain",+ json!({"text": "refund money back guarantee", "label": "billing"}),+ )+ .await;+ // The corrections are written through the ORDINARY indexing path —+ // `human: true` is a field on a document, not a new API.+ node.put_doc(+ "history",+ "human1",+ json!({"text": "refund money", "label": "tech", "human": true}),+ )+ .await;+ node.put_doc(+ "history",+ "human2",+ json!({"text": "refund policy", "label": "tech", "human": true}),+ )+ .await;+ node.refresh("history").await;++ let (st, r) = node+ .decide(json!({"index": "history", "question": "refund money back guarantee"}))+ .await;+ assert!(st.is_success(), "{st} {r}");+ assert_eq!(r["label"], expected, "human_weight {human_weight}: {r}");+ assert!(!r["abstain"].as_bool().unwrap_or(true), "{r}");+ assert_eq!(r["source"], "history", "{r}");+ }+}++/// The boosted weight is the weight the vote USED, so the audit surface must+/// show it: every neighbour's `weight` is its reciprocal rank, ×+/// `human_weight` exactly when the document is a correction. Asserted+/// per-position in the returned (rank-ordered) neighbour list, so the test+/// does not depend on predicting BM25's ordering — only on reading it.+#[tokio::test]+async fn the_audit_surface_shows_the_boosted_weight() {+ let human_weight = 3.0;+ let node = node_with_decisions("history", human_weight, 0.0).await;+ node.create_index("history").await;+ node.put_doc(+ "history",+ "plain",+ json!({"text": "refund money back guarantee", "label": "billing"}),+ )+ .await;+ node.put_doc(+ "history",+ "human1",+ json!({"text": "refund money", "label": "tech", "human": true}),+ )+ .await;+ node.refresh("history").await;++ let (st, r) = node+ .decide(json!({"index": "history", "question": "refund money back guarantee"}))+ .await;+ assert!(st.is_success(), "{st} {r}");+ let neighbours = r["neighbours"].as_array().expect("neighbours: {r}");+ for (rank, n) in neighbours.iter().enumerate() {+ let id = n["_id"].as_str().expect("id: {r}");+ let expected = if id == "human1" {+ human_weight / (rank as f64 + 1.0)+ } else {+ 1.0 / (rank as f64 + 1.0)+ };+ let got = n["weight"].as_f64().expect("weight: {r}");+ assert!(+ (got - expected).abs() < 1e-9,+ "{id} at rank {rank}: weight {got}, expected {expected}: {r}"+ );+ }+}++// ─────────────────────────────────────────────────────────────────────────────+// The write-back — needs tier 2, so `decide-local`+// ─────────────────────────────────────────────────────────────────────────────++#[cfg(feature = "decide-local")]+mod write_back {+ use super::*;++ struct LocalNode {+ node: Node,+ _model_dir: tempfile::TempDir,+ }++ /// The same armed node as `systemone_local_decide.rs`: a tiny untrained+ /// fixture model, `[decisions] index` set to `index`.+ async fn local_node(index: &str, min_confidence: f64) -> LocalNode {+ let dir = tempfile::tempdir().expect("tempdir");+ let model_dir = tempfile::tempdir().expect("model tempdir");+ xerj_ai::decide::testing::write_fixture(model_dir.path())+ .unwrap_or_else(|e| panic!("write decide fixture: {e:#}"));++ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ config.decisions.index = index.to_string();+ config.decisions.min_confidence = min_confidence;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let mut state = AppState::new(config, engine, metrics);+ // The injection seam, not the environment.+ state.decide = std::sync::Arc::new(xerj_api::systemone_api::DecideSettings::local(+ model_dir.path().to_path_buf(),+ ));+ LocalNode {+ node: Node {+ native: xerj_api::router::build_native_router(state.clone()),+ app: xerj_api::router::build_es_compat_router(state),+ _dir: dir,+ },+ _model_dir: model_dir,+ }+ }++ /// THE flywheel test. One request, two questions: history answers d0+ /// (seeded refund neighbours), the local head answers d1 (payload unlike+ /// anything seeded). Exactly ONE document is written back — d1's, with+ /// the text that was decided, the label and probability served, the+ /// source that produced them, and a timestamp — and d0's history answer+ /// is NOT written again. Then the same request again: d1 is now answered+ /// by HISTORY, from the cached answer, which is the entire point.+ #[tokio::test]+ async fn local_answers_are_cached_once_and_then_answer_the_next_request() {+ let ln = local_node("history", 0.0).await;+ let node = &ln.node;+ node.create_index("history").await;+ node.put_doc(+ "history",+ "h1",+ json!({"text": "refund refund refund asked for money back", "label": "true"}),+ )+ .await;+ node.put_doc(+ "history",+ "h2",+ json!({"text": "refund policy says money back", "label": "true"}),+ )+ .await;+ node.refresh("history").await;+ assert_eq!(node.docs("history").await.len(), 2, "the seeded history");++ let request = json!({+ "state": {"documents": {+ "doc_0": "refund asked for money back",+ "doc_1": "zzzqqq nothing like the history"+ }},+ "questions": {+ "d0": {"type": "noul", "instructions": "Is `documents.doc_0` about refunds?"},+ "d1": {"type": "noul", "instructions": "Is `documents.doc_1` about refunds?"}+ }+ });+ let (st, r) = node.systemone(request.clone()).await;+ assert!(st.is_success(), "{st} {r}");+ assert_eq!(r["decisions"]["evidence"]["d0"]["tier"], "history", "{r}");+ assert_eq!(r["decisions"]["evidence"]["d1"]["tier"], "local", "{r}");+ let served_p = r["decisions"]["evidence"]["d1"]["support"]+ .as_f64()+ .expect("p: {r}");++ // The write-back lands: exactly one new document.+ let docs = node.wait_for_docs("history", 3).await;+ assert_eq!(+ docs.len(),+ 3,+ "history answers are not double-written: {docs:?}"+ );+ let cached: Vec<&Value> = docs.iter().filter(|d| d.get("source").is_some()).collect();+ assert_eq!(cached.len(), 1, "exactly one cached answer: {docs:?}");+ let cached = cached[0];+ assert_eq!(+ cached["text"], "zzzqqq nothing like the history",+ "{cached}"+ );+ assert!(+ cached["label"].is_string(),+ "the label the head served: {cached}"+ );+ assert_eq!(cached["source"], "local", "{cached}");+ assert!(+ (cached["p"].as_f64().unwrap_or(-1.0) - served_p).abs() < 1e-9,+ "p is the probability that was served ({served_p}): {cached}"+ );+ let ts = cached["ts"].as_str().expect("ts: {cached}");+ assert!(+ ts.ends_with('Z') && ts.len() == 24,+ "an RFC 3339 millisecond timestamp: {cached}"+ );+ // The flywheel closes: the SAME request again has d1 answered by+ // history — the cached answer is the labelled neighbour now.+ let (st, r2) = node.systemone(request).await;+ assert!(st.is_success(), "{st} {r2}");+ assert_eq!(+ r2["decisions"]["evidence"]["d1"]["tier"], "history",+ "the cached answer answered it: {r2}"+ );+ assert_eq!(+ r2["decisions"]["evidence"]["d1"]["source"], "history",+ "{r2}"+ );+ // And nothing was written back for it: still three documents.+ tokio::time::sleep(std::time::Duration::from_millis(100)).await;+ assert_eq!(+ node.docs("history").await.len(),+ 3,+ "a history-tier answer is never cached again"+ );+ }++ /// `/_decide` caches its local answers too — and an abstain is NOT an+ /// answer, so it is never cached. Two nodes, the same fixture and+ /// question, differing only in `min_confidence`: one answers and writes,+ /// the other abstains and writes nothing (not even the index).+ #[tokio::test]+ async fn decide_caches_answered_local_answers_but_never_an_abstain() {+ // Answers: min_confidence 0 → the answer stands, the write-back+ // creates the configured-but-never-created index itself.+ let answered = local_node("history", 0.0).await;+ let (st, r) = answered+ .node+ .decide(json!({"index": "history", "question": "urgent prize claim call now"}))+ .await;+ assert!(st.is_success(), "{st} {r}");+ assert_eq!(r["tier"], "local", "{r}");+ assert_eq!(r["source"], "local", "{r}");+ let docs = answered.node.wait_for_docs("history", 1).await;+ assert_eq!(docs.len(), 1, "{docs:?}");+ assert_eq!(docs[0]["text"], "urgent prize claim call now", "{docs:?}");+ assert_eq!(docs[0]["source"], "local", "{docs:?}");+ let confidence = r["confidence"].as_f64().expect("confidence: {r}");+ assert!(+ (docs[0]["p"].as_f64().unwrap_or(-1.0) - confidence).abs() < 1e-9,+ "p is the served confidence ({confidence}): {docs:?}"+ );++ // Abstains: min_confidence 2.0 is above any probability, so the same+ // question abstains — and nothing is written, not even the index.+ let abstained = local_node("history", 2.0).await;+ let (st, r) = abstained+ .node+ .decide(json!({"index": "history", "question": "urgent prize claim call now"}))+ .await;+ assert!(st.is_success(), "{st} {r}");+ assert_eq!(r["tier"], "local", "{r}");+ assert!(r["abstain"].as_bool().unwrap_or(false), "{r}");+ tokio::time::sleep(std::time::Duration::from_millis(200)).await;+ let (st, r) = abstained+ .node+ .call(+ &abstained.node.app.clone(),+ "GET",+ "/history/_count",+ json!({}),+ )+ .await;+ assert_eq!(st, StatusCode::NOT_FOUND, "no index was created: {st} {r}");+ }+}diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 3215f318e..06c114846 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1,6 +1,6 @@//! xerj configuration system.//!-//! Configuration is intentionally minimal: **128 settings** versus+//! Configuration is intentionally minimal: **129 settings** versus//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible//! production-ready default. The format is TOML, loaded from a single file.//!@@ -98,12 +98,12 @@ pub struct Config {/// Second-stage reranking provider — 3 settings. Inert until a key is set.pub rerank: RerankProviderConfig,/// Typed decisions answered from a labelled-history index by- /// nearest-neighbour vote — 6 settings. Inert until `decisions.index`+ /// nearest-neighbour vote — 7 settings. Inert until `decisions.index`/// names an index that exists.pub decisions: DecisionsConfig,}-// 23 sub-configs, 128 leaf settings in total. Do not maintain that sum by hand+// 23 sub-configs, 129 leaf settings in total. Do not maintain that sum by hand// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a// serialised `Config::default()` and fails if this comment and the module// header stop matching. `Default` is derived: every field is a sub-config that@@ -334,6 +334,20 @@ impl Config {));}+ // Decisions: a human_weight that is not a positive finite number would+ // not scale a correction's vote — 0 erases it, a negative value+ // inverts it, NaN compares false against everything — so the config+ // refuses to boot instead of silently mis-weighting the one document+ // class the operator marked as authoritative.+ if !self.decisions.human_weight.is_finite() || self.decisions.human_weight <= 0.0 {+ return Err(XerjError::config(format!(+ "decisions.human_weight must be a positive number (got {}); it multiplies the \+ vote weight of history documents carrying human: true, and 1.0 disables the \+ boost",+ self.decisions.human_weight+ )));+ }+self.engine.validate()?;// Cluster: fail closed rather than expose an unauthenticated Raft@@ -470,7 +484,7 @@ impl Config {}// ═════════════════════════════════════════════════════════════════════════════-// Sub-configs (128 user-facing settings total; counted by+// Sub-configs (129 user-facing settings total; counted by// `journey_zero_config`, not by hand)// ═════════════════════════════════════════════════════════════════════════════@@ -2349,6 +2363,11 @@ pub struct DecisionsConfig {/// `/_decide` abstains below this vote share (0 = never abstain; the/// `/v1/systemone` wire has no abstain field and errors instead).pub min_confidence: f64,+ /// How much more a human correction weighs in the vote than a cached or+ /// seeded example, when the history document carries `human: true`+ /// (issue #1061). 2.0 — the issue's ≥ 2x floor for corrections — is the+ /// default; 1.0 makes corrections count exactly like any other example.+ pub human_weight: f64,}impl Default for DecisionsConfig {@@ -2360,6 +2379,7 @@ impl Default for DecisionsConfig {text_field: "text".to_string(),positive_label: "true".to_string(),min_confidence: 0.0,+ human_weight: 2.0,}}}@@ -3170,7 +3190,7 @@ mod tests {("lifecycle", 1),("wal_tap", 10),("rerank", 3),- ("decisions", 6),+ ("decisions", 7),];/// Count the settings by *counting them*.@@ -3213,7 +3233,7 @@ mod tests {"the section table must sum to the whole config");assert_eq!(- total, 128,+ total, 129,"the total settings count changed. It is quoted in this module's \header, in xerj-common/src/lib.rs, in engine/README.md, in \xerj.default.toml and in EXPECTED_SETTINGS in \diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rsindex a5b1b18d8..bfd20a0e3 100644--- a/engine/crates/xerj-common/src/lib.rs+++ b/engine/crates/xerj-common/src/lib.rs@@ -7,13 +7,13 @@//! ## Design philosophy//!//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes-//! **128 settings**, each meaningful and production-tested. Every default+//! **129 settings**, each meaningful and production-tested. Every default//! is chosen so that a fresh deployment with zero configuration changes performs//! well for the majority of workloads.//!//! ## Modules//!-//! - [`config`] — TOML-based configuration (128 settings)+//! - [`config`] — TOML-based configuration (129 settings)//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`//! - [`error`] — Unified error type ([`XerjError`])//! - [`types`] — Core domain types (documents, fields, IDs)diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rsindex 4f47e24f6..6e4d98051 100644--- a/engine/crates/xerj-engine/tests/product_experience.rs+++ b/engine/crates/xerj-engine/tests/product_experience.rs@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {/// Settings in a default `Config`, measured by `count_settings`. Quoted in/// `xerj-common/src/config.rs`; keep the two in step.-const EXPECTED_SETTINGS: usize = 128;+const EXPECTED_SETTINGS: usize = 129;#[tokio::test]async fn journey_zero_config() {diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex 74fdb0c87..d0c5821e0 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -1,6 +1,6 @@# xerj Default Configuration# ============================================================-# This file sets 55 of XERJ's 128 user-facing settings to their+# This file sets 55 of XERJ's 129 user-facing settings to their# production-ready defaults, and describes the rest in comments.# Every value here IS the default, so copying this file changes# nothing until you edit it (a test enforces that). The complete@@ -655,3 +655,4 @@ access_log = false# text_field = "text"# positive_label = "true" # the label a noul's probability is the share of# min_confidence = 0.0 # /_decide abstains below this+# human_weight = 2.0 # vote-weight multiplier for history docs carrying human: true