AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
console: honest numbers on the knowledge surface — dominant fields first, semantic row without invented vectors (#1098, #1099)
xerj-org829e5ae97fb6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex baabe827f..c970f3a35 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -59,7 +59,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0reworded into the graph layer of the corpus rather than a notes-flavoredmode, and `xerj mcp` help now says 13 tools (it said 11). Pinned onpayloads by `xerj-console-api/tests/knowledge_surface.rs` (5) and- `xerj-ux/test/knowledge-surface.test.mjs` (7).+ `xerj-ux/test/knowledge-surface.test.mjs` (7). Two honest-claims fixes+ from that PR's screenshot review, riding the same branch (issues+ [#1098](https://github.com/xerj-org/xerj/issues/1098) and+ [#1099](https://github.com/xerj-org/xerj/issues/1099)): the CORPUS fields+ table now leads with the corpus's DOMINANT fields, ranked by the+ catalog's own coverage × avg_len with autoindex plumbing last (a+ type-first rank made a 96%-code corpus open with 0.6%-coverage email+ fields and pushed its `code` field out of the table), unmeasured+ coverage renders as "—" on both card paths (never a defaulted 100%), and+ the whole table is one ranked list through the ALL-FIELDS disclosure; and+ the capability strip gained a server-computed `semantic` entry carrying+ the node's own facts — field, companion vector field, dims, similarity+ and the embedder's own label ("lexical feature-hash (built-in, 384-dim,+ non-neural)" on a default node, with the `--embed-mode neural` pointer)+ — with NO vector count, because the engine exposes none on any stats+ surface; an index whose elected semantic field carries no embedding says+ so in as many words.- **`xerj-decide-v1`: the open tier-2 decide model — training/export harness,measured eval card, upload-ready bundle([#1064](https://github.com/xerj-org/xerj/issues/1064), PRdiff --git a/engine/crates/xerj-console-api/src/knowledge.rs b/engine/crates/xerj-console-api/src/knowledge.rsindex 6e7ecb8fe..66af2daab 100644--- a/engine/crates/xerj-console-api/src/knowledge.rs+++ b/engine/crates/xerj-console-api/src/knowledge.rs@@ -18,7 +18,12 @@//! 3. **What can I do with this now?** — a capability strip computed from//! real facts on the node (catalog present, brains present), each entry//! carrying the real console route, CLI command, or HTTP endpoint it-//! names. No capability is listed that this tree does not ship.+//! names. No capability is listed that this tree does not ship. The+//! `semantic` entry (#1099) is the node's own posture — the elected+//! field, its companion vector field, dims, similarity and the+//! embedder's own label, read from the live schema — and it NEVER+//! carries a vector count: no engine stats surface exposes one, and a+//! number the node cannot back must not reach the UI.//!//! This is the server side of the Console's Corpus home (the landing view).//! It exists as ONE endpoint rather than a client-side fan-out because the@@ -188,6 +193,11 @@ fn parse_fields(fields_json: Option<&Value>) -> Vec<Value> {"cardinality_overflow": overflow,"null_ratio": get("null_ratio").and_then(Value::as_f64),"coverage": get("coverage").and_then(Value::as_f64),+ // The spec's measured average value length — the corpus's content+ // lives in its LONG fields, so the SPA ranks the fields a person+ // reads first by coverage × avg_len (the dominant fields), not by+ // type. Not rendered as a claim; a ranking input only.+ "avg_len": get("avg_len").and_then(Value::as_f64).unwrap_or(0.0),"examples": examples.iter().take(MAX_EXAMPLES)@@ -362,10 +372,67 @@ async fn brains_listing(engine: &Engine, may_read: bool) -> ConsoleResult<Vec<VaOk(out)}+/// The corpus's semantic posture, from the live engine — the facts behind+/// the `semantic` capability entry (#1099). Two honest shapes, nothing else:+///+/// - `Semantic` — the catalog elected a semantic field AND the index's live+/// schema carries its embedding config: dims/similarity/embedder exactly as+/// the query and ingest paths resolve them (`Index::semantic_field_facts`).+/// No vector count is included, because no engine surface exposes one — a+/// number the node cannot back must not reach the UI.+/// - `ElectedButMissing` — the catalog's last run elected a semantic field,+/// but this index has no embedding for it (it was rebuilt or mapped another+/// way): the entry then says plainly that there are no vectors on this node+/// for that field.+///+/// `None` (no catalog, or no dataset elected a semantic field) → no entry.+enum SemanticCapability {+ Semantic(xerj_engine::index::SemanticFieldFacts),+ ElectedButMissing { field: String, index: String },+}++/// Render the semantic capability's blurb from the facts above. Every phrase+/// is grounded: dims/similarity/embedder come from the live schema and the+/// embedder's own label; the lexical case says it is NOT neural and how to+/// get neural; the missing case says there is nothing to match.+fn semantic_blurb(sem: &SemanticCapability) -> String {+ match sem {+ SemanticCapability::Semantic(f) => {+ let dims = if f.dims_from_schema {+ format!("{}-D", f.dims)+ } else {+ format!("{}-D (the engine's built-in default)", f.dims)+ };+ if f.embedder_active {+ format!(+ "Semantic and hybrid over `{}` → `{}`: {dims} · {} · {} — one vector per \+ embedded document, embedded by that same backend at ingest.",+ f.field, f.target_field, f.similarity, f.embedder+ )+ } else {+ format!(+ "Semantic and hybrid over `{}` → `{}`: {dims} · {} · {} — one vector per \+ embedded document. That embedder is NOT neural; start the node with \+ `--embed-mode neural` (or `proxy` / `onnx-experimental`) for neural \+ semantics.",+ f.field, f.target_field, f.similarity, f.embedder+ )+ }+ }+ SemanticCapability::ElectedButMissing { field, index } => format!(+ "The catalog's last run elected `{field}` as the semantic field, but index \+ `{index}` carries no embedding for it on this node — there are no vectors to \+ match, and semantic/hybrid queries on that field are refused. Re-run \+ `xerj autoindex` over the source (or start with `--embed-mode neural`) to \+ give it one.",+ ),+ }+}+/// The capability strip. Every entry names a shipped surface — console route/// (`href`), CLI command (`command`), or HTTP endpoint (`endpoint`) — and/// availability is computed from facts on the node, not from a feature list.-fn capabilities(catalog: bool, brains: &[Value]) -> Vec<Value> {+fn capabilities(catalog: bool, brains: &[Value], semantic: Option<&SemanticCapability>) -> Vec<Value> {let mut caps = vec![json!({"id": "search",@@ -374,14 +441,26 @@ fn capabilities(catalog: bool, brains: &[Value]) -> Vec<Value> {"href": "#/discover","kind": "console",}),- json!({- "id": "read",- "title": "Read the documents",- "blurb": "Page through records with highlights and the per-record link list.",- "href": "#/reader",- "kind": "console",- }),];+ // The semantic row is computed from the node's own schema and embedder+ // (#1099): dims, similarity and the embedder's label come from the engine,+ // and no vector count is ever claimed — the engine backs none.+ if let Some(sem) = semantic {+ caps.push(json!({+ "id": "semantic",+ "title": "Search it semantically",+ "blurb": semantic_blurb(sem),+ "href": "#/discover",+ "kind": "console",+ }));+ }+ caps.push(json!({+ "id": "read",+ "title": "Read the documents",+ "blurb": "Page through records with highlights and the per-record link list.",+ "href": "#/reader",+ "kind": "console",+ }));if let Some(brain) = brains.first() {let name = brain.get("name").and_then(Value::as_str).unwrap_or_default();let links = brain.get("links").and_then(Value::as_u64).unwrap_or(0);@@ -523,9 +602,32 @@ pub async fn knowledge(rank(b).cmp(&rank(a))});- // 6. Brains (operator tier) + the capability strip.+ // 6. The corpus's semantic posture — largest dataset first (the corpus's+ // dominant one), from the LIVE index schema, never from a feature list.+ // See `SemanticCapability` for the two honest shapes.+ let mut semantic: Option<SemanticCapability> = None;+ for d in &datasets {+ let Some(field) = str_of(d.get("semantic_field")) else {+ continue;+ };+ let index = d["index"].as_str().unwrap_or_default();+ let facts = match state.engine.get_index(index) {+ Ok(idx) => idx.semantic_field_facts(&field).await,+ Err(_) => None,+ };+ semantic = Some(match facts {+ Some(f) => SemanticCapability::Semantic(f),+ None => SemanticCapability::ElectedButMissing {+ field,+ index: index.to_string(),+ },+ });+ break;+ }++ // 7. Brains (operator tier) + the capability strip.let brains = brains_listing(&state.engine, graph::may_read_brains(&sess.user)).await?;- let caps = capabilities(catalog, &brains);+ let caps = capabilities(catalog, &brains, semantic.as_ref());let mut payload = Map::new();payload.insert("catalog".into(), json!(catalog));diff --git a/engine/crates/xerj-console-api/tests/knowledge_surface.rs b/engine/crates/xerj-console-api/tests/knowledge_surface.rsindex e847a1bc8..ee64702ce 100644--- a/engine/crates/xerj-console-api/tests/knowledge_surface.rs+++ b/engine/crates/xerj-console-api/tests/knowledge_surface.rs@@ -20,6 +20,10 @@//! catalog exists, the graph entry only for a role that may read brains//! AND only when a brain exists, and every entry names a real console//! route, CLI command, or endpoint;+//! - the `semantic` entry (#1099) is the node's own posture — dims,+//! similarity and the embedder's label from the live schema — and never+//! a vector count (the engine exposes none) nor a READY claim; an index+//! that lost its embedding says so in as many words;//! - a node with nothing indexed is the ordinary empty state (`catalog://! false`), not an error;//! - the endpoint is session-authorized: no cookie is a 401.@@ -30,7 +34,7 @@ use serde_json::{json, Value};use tempfile::TempDir;use tower::ServiceExt;use xerj_common::config::Config;-use xerj_common::types::{FieldConfig, FieldType, Schema};+use xerj_common::types::{EmbeddingConfig, FieldConfig, FieldType, Schema};use xerj_console_api::{auth::{sessions, store},state::ClusterMode,@@ -150,7 +154,13 @@ fn fields_json() -> String {/// a catalog describing two datasets (one larger, one smaller), one/// key-overlap correlation, a brain over the first dataset, and one user/// index the catalog does NOT describe.-async fn boot() -> TestApp {+///+/// `embed_semantic` decides whether ax-mail's `body` carries the embedding+/// config an autoindex run installs (the `semantic_text` mapping): `true` is+/// the ordinary post-`xerj brain` node, `false` is an index that was rebuilt+/// or mapped another way — the catalog still ELECTS `body`, but the index+/// has nothing to embed it with (#1099's "no vectors on this node" case).+async fn boot_with(embed_semantic: bool) -> TestApp {let dir = TempDir::new().unwrap();let mut cfg = Config::default();cfg.server.data_dir = dir.path().to_str().unwrap().to_string();@@ -169,8 +179,22 @@ async fn boot() -> TestApp {// The dataset indices the catalog describes — 3 real documents in// ax-mail (the LIVE count the endpoint must join on), 1 in ax-pdfs.+ // ax-mail's `body` carries the embedding config (dims/similarity) the+ // es-compat layer installs for autoindex's `semantic_text` mapping.+ let mut mail_schema = Schema::empty();+ let mut body = FieldConfig::new("body", FieldType::Text);+ if embed_semantic {+ body.options.dimensions = Some(384);+ body.options.similarity = Some("cosine".into());+ body.embedding = Some(EmbeddingConfig {+ endpoint: None,+ model: None,+ target_field: None,+ });+ }+ mail_schema.add_field(body).expect("add body");engine- .create_index("ax-mail", schema(&[("body", FieldType::Text)]))+ .create_index("ax-mail", mail_schema).expect("create ax-mail");engine.get_index("ax-mail").unwrap().index_documents_batched(vec![@@ -308,6 +332,12 @@ async fn boot() -> TestApp {}}+/// The ordinary post-`xerj brain` node: the elected semantic field carries+/// its embedding config.+async fn boot() -> TestApp {+ boot_with(true).await+}+/// A node with NOTHING indexed — no catalog, no user index.async fn boot_empty() -> TestApp {let dir = TempDir::new().unwrap();@@ -411,6 +441,10 @@ async fn the_payload_carries_the_catalogs_own_facts() {let body = by_name("body");assert_eq!(body["semantic"].as_bool(), Some(true), "the semantic_text field is flagged");assert_eq!(body["type"].as_str(), Some("semantic_text"));+ // the spec's measured average value length round-trips — the SPA ranks+ // the card's fields by coverage × avg_len, so the corpus's DOMINANT+ // fields lead the table (#1098)+ assert_eq!(body["avg_len"].as_f64(), Some(412.5));let page = by_name("page");assert_eq!(page["null_ratio"].as_f64(), Some(0.75));assert_eq!(page["coverage"].as_f64(), Some(0.25));@@ -486,6 +520,71 @@ async fn the_capability_strip_is_grounded_in_facts() {assert!(!vids.contains(&"graph"), "viewer must not see the graph capability");}+/// #1099 — the semantic capability is the NODE's own posture, computed from+/// the live schema and the embedder, and it never claims a vector count the+/// engine cannot back (the screenshot defect: "6,546,091 BODY VECTORS ·+/// 384-D · COSINE — READY" on a node reporting no such number anywhere).+#[tokio::test]+async fn the_semantic_capability_is_the_nodes_own_facts_with_no_invented_count() {+ let app = boot().await;+ let (_, v) = get(&app, Some(&app.owner_cookie)).await;+ let caps = data(&v)["capabilities"].as_array().unwrap();+ let sem = caps+ .iter()+ .find(|c| c["id"].as_str() == Some("semantic"))+ .expect("a semantic_text field with an embedding config earns the semantic entry");+ // the facts are the schema's and the embedder's own: field, companion,+ // dims (from the mapping here), similarity, and the embedder's label+ let blurb = sem["blurb"].as_str().unwrap();+ assert!(blurb.contains("`body`"), "names the elected field: {blurb}");+ assert!(blurb.contains("`body_vector`"), "names the companion: {blurb}");+ assert!(blurb.contains("384-D"), "dims from the mapping: {blurb}");+ assert!(blurb.contains("cosine"), "similarity from the mapping: {blurb}");+ assert!(+ blurb.contains("lexical feature-hash (built-in, 384-dim, non-neural)"),+ "the embedder's own honesty label, verbatim: {blurb}"+ );+ assert!(blurb.contains("NOT neural"), "the lexical case says so plainly: {blurb}");+ assert!(blurb.contains("--embed-mode neural"), "and how to get neural: {blurb}");+ // the honest-claims line: no vector count and no READY claim — the node+ // exposes no per-field vector count on any stats surface, so none may+ // appear here, whatever the client does with the payload+ assert!(!blurb.contains("VECTOR"), "no vector count: {blurb}");+ assert!(!blurb.contains("READY"), "no readiness claim: {blurb}");+ for key in sem.as_object().unwrap().keys() {+ assert!(!key.contains("count"), "no count key may ride the entry: {key}");+ }+ // it is not brain-gated — it describes the corpus, not the operator's role+ let (_, vv) = get(&app, Some(&app.viewer_cookie)).await;+ let viewer_has = data(&vv)["capabilities"]+ .as_array()+ .unwrap()+ .iter()+ .any(|c| c["id"].as_str() == Some("semantic"));+ assert!(viewer_has, "the viewer sees the node's semantic posture too");+}++/// #1099's other shape — the catalog's last run ELECTED a semantic field,+/// but the live index carries no embedding for it (rebuilt / mapped another+/// way): the entry must say plainly that there are no vectors to match,+/// never a derived count and never READY.+#[tokio::test]+async fn an_elected_semantic_field_the_index_does_not_carry_is_stated_plainly() {+ let app = boot_with(false).await;+ let (_, v) = get(&app, Some(&app.owner_cookie)).await;+ let caps = data(&v)["capabilities"].as_array().unwrap();+ let sem = caps+ .iter()+ .find(|c| c["id"].as_str() == Some("semantic"))+ .expect("the catalog elected body; the mismatch must be visible, not omitted");+ let blurb = sem["blurb"].as_str().unwrap();+ assert!(blurb.contains("`body`"), "names the elected field: {blurb}");+ assert!(blurb.contains("no embedding"), "states the gap: {blurb}");+ assert!(blurb.contains("no vectors to match"), "states it in those words: {blurb}");+ assert!(!blurb.contains("READY"), "no readiness claim: {blurb}");+ assert!(!blurb.contains("VECTOR"), "no vector count: {blurb}");+}+#[tokio::test]async fn nothing_indexed_is_the_empty_state_not_an_error() {let app = boot_empty().await;@@ -504,5 +603,6 @@ async fn nothing_indexed_is_the_empty_state_not_an_error() {.collect();assert!(!ids.contains(&"map"), "no catalog → no map/ask capability");assert!(!ids.contains(&"ask"), "no catalog → no ask capability");+ assert!(!ids.contains(&"semantic"), "no dataset elected a semantic field → no semantic claim");assert!(ids.contains(&"search"), "search is always real");}diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 2eae5965b..5102a3e4c 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -7783,6 +7783,30 @@ pub struct FieldEncodingInfo {pub distinct_values: usize,}+/// What the node runs for semantic search over one field — see+/// [`Index::semantic_field_facts`]. Every value is read from the live schema+/// and embedder; none is derived, and there is intentionally no vector count.+#[derive(Debug, Clone, Serialize)]+pub struct SemanticFieldFacts {+ /// The concrete field (alias-resolved) carrying the embedding config.+ pub field: String,+ /// Companion field the vectors live in (`<field>_vector` by default).+ pub target_field: String,+ /// Effective dimensionality — the schema's own, or the engine's built-in+ /// default when the mapping did not pin one (`dims_from_schema` says which).+ pub dims: usize,+ /// `true` when `dims` came from the mapping, `false` when it is the+ /// engine default the query/ingest paths fall back to.+ pub dims_from_schema: bool,+ /// Effective similarity metric (`cosine` unless the mapping overrode it).+ pub similarity: String,+ /// The embedder's own honesty label, e.g.+ /// `"lexical feature-hash (built-in, 384-dim, non-neural)"`.+ pub embedder: &'static str,+ /// `false` only for the built-in lexical embedder (non-neural).+ pub embedder_active: bool,+}+/// Statistics about an index.#[derive(Debug, Clone, Serialize, Deserialize)]pub struct IndexStats {@@ -24817,6 +24841,57 @@ impl Index {self.schema.read().await.schema.clone()}+ /// What this node actually runs for semantic search over `field` — the+ /// facts behind any "semantic" claim a console or agent surface renders.+ ///+ /// Resolved EXACTLY the way the query and ingest paths resolve it (same+ /// alias handling, same `dimensions.unwrap_or(DEFAULT_DIMS)` fallback,+ /// the same embedder instance), so a claim built from these facts cannot+ /// drift from what a query actually does:+ ///+ /// - `Some(facts)` — `field` carries an [`crate::FieldConfig::embedding`]+ /// config (an ES `semantic_text` field): the companion `target_field`+ /// holds one vector per embedded document, `dims`/`similarity` are the+ /// effective ones, and `embedder` is the backend's own label (e.g.+ /// `"lexical feature-hash (built-in, 384-dim, non-neural)"`).+ /// - `None` — no embedding config: there is no stored vector to match,+ /// and the honest answer is "none on this node", never a count derived+ /// from terms or tokens.+ ///+ /// The facts deliberately carry NO vector count: the engine exposes no+ /// per-field vector count on any stats surface, and a number it cannot+ /// back must not exist (the honest-claims rule binds UIs, not just docs).+ pub async fn semantic_field_facts(&self, field: &str) -> Option<SemanticFieldFacts> {+ let guard = self.schema.read().await;+ let resolved = resolve_field_alias(&guard.schema, field);+ let fc = guard.schema.field(&resolved)?;+ let emb = fc.embedding.as_ref()?;+ let target_field = emb+ .target_field+ .clone()+ .unwrap_or_else(|| format!("{resolved}_vector"));+ let (dims, dims_from_schema) = match fc.options.dimensions {+ Some(d) => (d, true),+ None => (xerj_ai::local::DEFAULT_DIMS, false),+ };+ let similarity = fc+ .options+ .similarity+ .clone()+ .unwrap_or_else(|| "cosine".to_string());+ drop(guard);+ let embedder = self.embedder.read().await;+ Some(SemanticFieldFacts {+ field: resolved,+ target_field,+ dims,+ dims_from_schema,+ similarity,+ embedder: embedder.describe(),+ embedder_active: embedder.is_active(),+ })+ }+/// Add a field to the schema.pub async fn add_field(self: &Arc<Self>, field: FieldConfig) -> Result<()> {self.add_fields(vec![field]).awaitdiff --git a/xerj-ux/src/data/catalog.js b/xerj-ux/src/data/catalog.jsindex a2f768681..9c015341c 100644--- a/xerj-ux/src/data/catalog.js+++ b/xerj-ux/src/data/catalog.js@@ -28,9 +28,14 @@ function parseFields(fieldsJson) {name: f.name,type: str(f.es_type || f.type || 'object'),semantic: !!f.semantic,- coverage: Number.isFinite(Number(f.coverage)) ? Number(f.coverage) : 1,+ // Unmeasured coverage is null → "—", never 1 → "100%": a field the+ // catalog did not measure must not read as fully covered (#1098).+ // (`Number(null)` is 0, so null has to be checked before the finite+ // test — a JSON null means "not measured", not "measured zero".)+ coverage: f.coverage == null || !Number.isFinite(Number(f.coverage)) ? null : Number(f.coverage),cardinality: num(f.cardinality_est),- nullRatio: Number.isFinite(Number(f.null_ratio)) ? Number(f.null_ratio) : null,+ nullRatio: f.null_ratio == null || !Number.isFinite(Number(f.null_ratio)) ? null : Number(f.null_ratio),+ avgLen: Number.isFinite(Number(f.avg_len)) ? Number(f.avg_len) : 0,examples: Array.isArray(f.examples) ? f.examples.slice(0, 3).map(str) : [],}));}@@ -84,20 +89,26 @@ export function parseCatalogHits(hits) {return cards;}-/** The fields worth showing on a card: the semantic field first, then the- * best-covered keyword/date/text fields; autoindex plumbing (`ax_*`) last. */+/** The card's fields in DISPLAY ORDER: the corpus's DOMINANT fields first —+ * ranked by the share of the corpus's content they hold (the catalog's own+ * coverage × avg_len), tie-broken by coverage — with autoindex plumbing+ * (`ax_*`) last. Type must not lead the ranking (#1098): a type-first rank+ * put every keyword/date field above the text field that IS the corpus, so+ * a 96%-code corpus opened with 0.6%-coverage email fields and its `code`+ * field was pushed into the disclosure. The semantic field keeps its accent+ * styling but no ranking pin — on a notes corpus it dominates on its own. */+export function rankedFields(card) {+ const plumbing = (f) => (f.name.startsWith('ax_') ? 1 : 0);+ const content = (f) => (f.coverage == null ? 0 : f.coverage) * (f.avgLen || 0);+ const cov = (f) => (f.coverage == null ? -1 : f.coverage);+ return [...(card.fields || [])].sort(+ (a, b) => (plumbing(a) - plumbing(b)) || (content(b) - content(a)) || (cov(b) - cov(a)) || a.name.localeCompare(b.name),+ );+}++/** The fields worth showing as a card's top chips. */export function topFields(card, n = 8) {- const rank = (f) => {- if (f.name.startsWith('ax_')) return 9;- if (f.semantic || f.name === card.semanticField) return 0;- if (f.type === 'date') return 1;- if (f.type === 'keyword') return 2;- if (f.type === 'text' || f.type === 'semantic_text') return 3;- return 5;- };- return [...(card.fields || [])]- .sort((a, b) => (rank(a) - rank(b)) || (b.coverage - a.coverage) || a.name.localeCompare(b.name))- .slice(0, n);+ return rankedFields(card).slice(0, n);}// ── the knowledge surface ───────────────────────────────────────────@@ -116,9 +127,10 @@ const knowledgeField = (f) => ({name: str(f.name),type: str(f.type || 'object'),semantic: f.semantic === true,- coverage: Number.isFinite(Number(f.coverage)) ? Number(f.coverage) : null,+ coverage: f.coverage == null || !Number.isFinite(Number(f.coverage)) ? null : Number(f.coverage),cardinality: Number.isFinite(Number(f.cardinality)) ? Number(f.cardinality) : null,- nullRatio: Number.isFinite(Number(f.null_ratio)) ? Number(f.null_ratio) : null,+ nullRatio: f.null_ratio == null || !Number.isFinite(Number(f.null_ratio)) ? null : Number(f.null_ratio),+ avgLen: Number.isFinite(Number(f.avg_len)) ? Number(f.avg_len) : 0,examples: Array.isArray(f.examples) ? f.examples.slice(0, 3).map(str) : [],});diff --git a/xerj-ux/src/ux/corpus-render.js b/xerj-ux/src/ux/corpus-render.jsindex 487f8839d..662ed83f9 100644--- a/xerj-ux/src/ux/corpus-render.js+++ b/xerj-ux/src/ux/corpus-render.js@@ -17,7 +17,7 @@import { h } from './safe-dom.js';import { readerHref } from './reader-render.js';-import { topFields, sampleQueryToSearch, fmtBytes, fmtCount, timeSpan } from '../data/catalog.js';+import { rankedFields, sampleQueryToSearch, fmtBytes, fmtCount, timeSpan } from '../data/catalog.js';export const EMPTY_COMMAND = 'xerj brain <folder>';@@ -57,17 +57,18 @@ function fieldChip(f) {}/** The fields block: the ranked top fields as chips, then EVERY field in a- * disclosure when there are more. A corpus's structure is the one thing a- * person came to see — nothing is capped away. */+ * disclosure when there are more — in the SAME display order, so the table+ * reads as one ranked list, top chips first. A corpus's structure is the+ * one thing a person came to see — nothing is capped away. */function fieldsBlock(card) {- const all = card.fields || [];- if (!all.length) return null;- const top = topFields(card, 8);- const rest = all.filter((f) => !top.includes(f));+ const ranked = rankedFields(card);+ if (!ranked.length) return null;+ const top = ranked.slice(0, 8);+ const rest = ranked.slice(8);return h('div', { class: 'cp-fields' },top.map(fieldChip),rest.length ? h('details', { class: 'cp-fields__all' },- h('summary', null, `ALL ${all.length} FIELDS`),+ h('summary', null, `ALL ${ranked.length} FIELDS`),h('div', { class: 'cp-fields' }, rest.map(fieldChip))) : null);}diff --git a/xerj-ux/test/knowledge-surface.test.mjs b/xerj-ux/test/knowledge-surface.test.mjsindex 90a4a4f07..7c69ee77d 100644--- a/xerj-ux/test/knowledge-surface.test.mjs+++ b/xerj-ux/test/knowledge-surface.test.mjs@@ -14,7 +14,7 @@ function json(status, body) {return { status, ok: status >= 200 && status < 300, json: async () => body, text: async () => JSON.stringify(body) };}-const { parseKnowledge } = await import('../src/data/catalog.js');+const { parseKnowledge, parseCatalogHits, topFields } = await import('../src/data/catalog.js');const { renderCorpusCard, renderRelations, renderCapabilities } = await import('../src/ux/corpus-render.js');const { corpus } = await import('../src/dashboards/corpus.js');const { textOf, findAll } = await import('../src/ux/safe-dom.js');@@ -172,3 +172,137 @@ test('liveCorpus reads ONE endpoint and keeps the never-mock contract', async ()assert.deepEqual(err.data.datasets, []);assert.deepEqual(err.data.relations, []);});++// ── #1098: the FIELDS table must be the corpus the catalog measured ────+//+// The screenshot defect: a 95.8%-code corpus opened with `email_date (date)+// 100% · email_subject (keyword) 100% · email_from (keyword) 59% · …` —+// coverage that matches nothing in fields_json — while `code` (0.958) was+// pushed out of the table entirely. The 100/59/57/43 pattern is coverage+// within the corpus's mbox-format SUBSET; these tests pin that neither the+// ranking nor the numbers can present anything but the catalog's own+// field-for-field, number-for-number facts, whichever path served the card.+const CODE_CORPUS_FIELDS = [+ { name: 'code', type: 'text', semantic: false, coverage: 0.958, cardinality: 15436, null_ratio: 0.042, avg_len: 912.0, examples: ['fn short_passage(i: usize) -> String'] },+ { name: 'title', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 500, null_ratio: 0.0, avg_len: 34.0, examples: ['index.rs'] },+ { name: 'body', type: 'semantic_text', semantic: true, coverage: 0.039, cardinality: 602, null_ratio: 0.961, avg_len: 412.5, examples: ['term sheet attached'] },+ { name: 'labels', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 4, null_ratio: 0.0, avg_len: 9.0, examples: ['rust'] },+ { name: 'defs', type: 'text', semantic: false, coverage: 0.032, cardinality: 208, null_ratio: 0.968, avg_len: 64.0, examples: ['pub fn parse_request'] },+ { name: 'mtime', type: 'date', semantic: false, coverage: 1.0, cardinality: 491, null_ratio: 0.0, avg_len: 0.0, examples: [] },+ { name: 'email_date', type: 'date', semantic: false, coverage: 0.0064, cardinality: 99, null_ratio: 0.9936, avg_len: 0.0, examples: [] },+ { name: 'email_subject', type: 'keyword', semantic: false, coverage: 0.0064, cardinality: 97, null_ratio: 0.9936, avg_len: 31.0, examples: ['Re: contract'] },+ // the mbox-subset figures from the screenshot — 59/57/43 must never be+ // able to stand in for the corpus coverage of ANY field+ { name: 'email_from', type: 'keyword', semantic: false, coverage: 0.0064, cardinality: 61, null_ratio: 0.9936, avg_len: 22.0, examples: ['[email protected]'] },+ { name: 'email_to', type: 'keyword', semantic: false, coverage: 0.0033, cardinality: 55, null_ratio: 0.9967, avg_len: 22.0, examples: ['[email protected]'] },+ { name: 'email_cc', type: 'keyword', semantic: false, coverage: 0.0026, cardinality: 37, null_ratio: 0.9974, avg_len: 20.0, examples: [] },+ { name: 'ax_path', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 500, null_ratio: 0.0, avg_len: 48.0, examples: [] },+ { name: 'ax_format', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 6, null_ratio: 0.0, avg_len: 4.0, examples: [] },+ { name: 'ax_source', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 1, null_ratio: 0.0, avg_len: 5.0, examples: [] },+ { name: 'ax_kind', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 2, null_ratio: 0.0, avg_len: 4.0, examples: [] },+ // a field the catalog did not measure — "—", never a fabricated 100%+ { name: 'unmeasured', type: 'long', semantic: false, coverage: null, cardinality: null, null_ratio: null, avg_len: 0.0, examples: [] },+];++const chipsOf = (card) => findAll(card, (n) => (n.attrs.class || '').split(' ').includes('cp-field'));++test('#1098 the FIELDS table leads with the corpus\'s dominant fields at the catalog\'s own coverage', () => {+ const st = parseKnowledge({ data: { ...PAYLOAD().data, datasets: [{ ...PAYLOAD().data.datasets[0], fields: CODE_CORPUS_FIELDS }] } });+ const card = renderCorpusCard(st.datasets[0], {});+ const chips = chipsOf(card);+ // the top chips (before the ALL-FIELDS disclosure) are the first 8+ assert.equal(chips.length, CODE_CORPUS_FIELDS.length, 'every field renders, in the disclosure when not top');+ const names = chips.map((c) => textOf(findAll(c, (n) => (n.attrs.class || '').includes('cp-field__name'))[0]));+ // the corpus IS its content: code first, body and defs ahead of every+ // email_* field, autoindex plumbing last+ assert.equal(names[0], 'code', `dominant field first: ${names.join(', ')}`);+ for (const dominant of ['code', 'body', 'defs']) {+ for (const email of names.filter((n) => n.startsWith('email_'))) {+ assert.ok(names.indexOf(dominant) < names.indexOf(email), `${dominant} must outrank ${email}: ${names.join(', ')}`);+ }+ }+ const lastThree = names.slice(-3).sort();+ assert.deepEqual(lastThree, ['ax_format', 'ax_kind', 'ax_source'], `plumbing last: ${names.join(', ')}`);+ // number-for-number: each chip shows ITS OWN measured coverage, rounded+ const byName = new Map(CODE_CORPUS_FIELDS.map((f) => [f.name, f]));+ const chipText = (name) => textOf(chips[names.indexOf(name)]);+ for (let i = 0; i < chips.length; i++) {+ const f = byName.get(names[i]);+ const expected = f.coverage == null ? '—' : `${Math.round(f.coverage * 100)}%`;+ assert.ok(textOf(chips[i]).includes(expected), `${names[i]} must show ${expected}: ${textOf(chips[i])}`);+ }+ // the screenshot's mbox-subset mirage is dead: no email field reads 100%,+ // 59/57/43 appear nowhere, and the dominant coverage is on the table+ const text = textOf(card);+ assert.ok(!chipText('email_date').includes('100%'), `email_date renders its own coverage: ${chipText('email_date')}`);+ assert.ok(!chipText('email_subject').includes('100%'), `email_subject renders its own coverage: ${chipText('email_subject')}`);+ for (const bogus of ['59%', '57%', '43%']) assert.ok(!text.includes(bogus), `${bogus} must not render: no field measures it`);+ assert.ok(chipText('code').includes('96%'), 'code 0.958 leads at 96%'); // code 0.958+ // unmeasured coverage renders as "—", never as 100%+ assert.match(chipText('unmeasured'), /—/);+ assert.doesNotMatch(chipText('unmeasured'), /100%/);+});++test('#1098 both card paths agree: the catalog-search fallback ranks and numbers identically', () => {+ // The legacy path parses fields_json (the autoindex catalog document);+ // the knowledge path parses the endpoint payload. The screenshot showed a+ // table that disagreed with BOTH — pin that whichever path serves the+ // card, the same fields lead with the same numbers.+ const specs = CODE_CORPUS_FIELDS.map((f) => ({+ name: f.name, es_type: f.type, semantic: f.semantic === true ? 'lexical-hash-384' : undefined,+ cardinality_est: f.cardinality, null_ratio: f.null_ratio, coverage: f.coverage,+ avg_len: f.avg_len, examples: f.examples,+ }));+ const hits = [{ _id: 'ds:code', _source: {+ doc_kind: 'dataset', index_name: 'ax-docs', record_count: 15436, file_count: 500,+ bytes: 16496072, semantic_field: 'body', fields_json: JSON.stringify(specs),+ } }];+ const legacy = parseCatalogHits(hits)[0];+ const st = parseKnowledge({ data: { ...PAYLOAD().data, datasets: [{ ...PAYLOAD().data.datasets[0], index: 'ax-docs', fields: CODE_CORPUS_FIELDS }] } });+ const knowledgeCard = st.datasets[0];+ const rank = (card) => topFields(card, 8).map((f) => f.name);+ assert.deepEqual(rank(legacy), rank(knowledgeCard), 'both paths lead with the same fields in the same order');+ assert.equal(rank(legacy)[0], 'code');+ // an unmeasured field is null on BOTH paths — the legacy default of 1+ // (which rendered "100%") is gone+ const legacyUnmeasured = legacy.fields.find((f) => f.name === 'unmeasured');+ assert.equal(legacyUnmeasured.coverage, null);+});++// ── #1099: the semantic capability states what the node runs — no invented vectors ──++test('#1099 the semantic row renders the node\'s own facts and never a vector count', () => {+ const lexical = { data: { ...PAYLOAD().data, capabilities: [+ ...PAYLOAD().data.capabilities,+ { id: 'semantic', title: 'Search it semantically',+ blurb: 'Semantic and hybrid over `body` → `body_vector`: 384-D · cosine · lexical feature-hash (built-in, 384-dim, non-neural) — one vector per embedded document. That embedder is NOT neural; start the node with `--embed-mode neural` (or `proxy` / `onnx-experimental`) for neural semantics.',+ href: '#/discover', kind: 'console' },+ ] } };+ const strip = renderCapabilities(parseKnowledge(lexical));+ const text = textOf(strip);+ assert.match(text, /Search it semantically/);+ assert.match(text, /lexical feature-hash/); // the embedder's own label+ assert.match(text, /NOT neural/);+ assert.match(text, /--embed-mode neural/);+ // the honest-claims line: no count of vectors, no READY claim — the node+ // backs neither ("6,546,091 BODY VECTORS · 384-D · COSINE — READY" was+ // computed by nothing the node reports)+ assert.doesNotMatch(text, /\d[\d,.]*\s+VECTORS?/i);+ assert.doesNotMatch(text, /VECTOR COUNT/i);+ assert.doesNotMatch(text, /—\s*READY\b/);+ // and when the node reports no semantic surface, no semantic row renders+ const none = renderCapabilities(parseKnowledge(PAYLOAD()));+ assert.doesNotMatch(textOf(none), /Search it semantically/);+});++test('#1099 a catalog that elected a semantic field the index does not carry says so plainly', () => {+ const missing = { data: { ...PAYLOAD().data, capabilities: [+ ...PAYLOAD().data.capabilities,+ { id: 'semantic', title: 'Search it semantically',+ blurb: "The catalog's last run elected `body` as the semantic field, but index `ax-mail` carries no embedding for it on this node — there are no vectors to match, and semantic/hybrid queries on that field are refused.",+ href: '#/discover', kind: 'console' },+ ] } };+ const text = textOf(renderCapabilities(parseKnowledge(missing)));+ assert.match(text, /no vectors to match/);+ assert.doesNotMatch(text, /\d[\d,.]*\s+VECTORS?/i);+});