Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

console: honest numbers on the knowledge surface — dominant fields first, semantic row without invented vectors (#1098, #1099)

xerj-org829e5ae97fb6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index baabe827f..c970f3a35 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -59,7 +59,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
reworded into the graph layer of the corpus rather than a notes-flavored
mode, and `xerj mcp` help now says 13 tools (it said 11). Pinned on
payloads by `xerj-console-api/tests/knowledge_surface.rs` (5) and
- `xerj-ux/test/knowledge-surface.test.mjs` (7).
+ `xerj-ux/test/knowledge-surface.test.mjs` (7). Two honest-claims fixes
+ from that PR's screenshot review, riding the same branch (issues
+ [#1098](https://github.com/xerj-org/xerj/issues/1098) and
+ [#1099](https://github.com/xerj-org/xerj/issues/1099)): the CORPUS fields
+ table now leads with the corpus's DOMINANT fields, ranked by the
+ catalog's own coverage × avg_len with autoindex plumbing last (a
+ type-first rank made a 96%-code corpus open with 0.6%-coverage email
+ fields and pushed its `code` field out of the table), unmeasured
+ coverage renders as "—" on both card paths (never a defaulted 100%), and
+ the whole table is one ranked list through the ALL-FIELDS disclosure; and
+ the capability strip gained a server-computed `semantic` entry carrying
+ the node's own facts — field, companion vector field, dims, similarity
+ and the embedder's own label ("lexical feature-hash (built-in, 384-dim,
+ non-neural)" on a default node, with the `--embed-mode neural` pointer)
+ — with NO vector count, because the engine exposes none on any stats
+ surface; an index whose elected semantic field carries no embedding says
+ so in as many words.
- **`xerj-decide-v1`: the open tier-2 decide model — training/export harness,
measured eval card, upload-ready bundle
([#1064](https://github.com/xerj-org/xerj/issues/1064), PR
diff --git a/engine/crates/xerj-console-api/src/knowledge.rs b/engine/crates/xerj-console-api/src/knowledge.rs
index 6e7ecb8fe..66af2daab 100644
--- a/engine/crates/xerj-console-api/src/knowledge.rs
+++ b/engine/crates/xerj-console-api/src/knowledge.rs
@@ -18,7 +18,12 @@
//! 3. **What can I do with this now?** — a capability strip computed from
//! real facts on the node (catalog present, brains present), each entry
//! carrying the real console route, CLI command, or HTTP endpoint it
-//! names. No capability is listed that this tree does not ship.
+//! names. No capability is listed that this tree does not ship. The
+//! `semantic` entry (#1099) is the node's own posture — the elected
+//! field, its companion vector field, dims, similarity and the
+//! embedder's own label, read from the live schema — and it NEVER
+//! carries a vector count: no engine stats surface exposes one, and a
+//! number the node cannot back must not reach the UI.
//!
//! This is the server side of the Console's Corpus home (the landing view).
//! It exists as ONE endpoint rather than a client-side fan-out because the
@@ -188,6 +193,11 @@ fn parse_fields(fields_json: Option<&Value>) -> Vec<Value> {
"cardinality_overflow": overflow,
"null_ratio": get("null_ratio").and_then(Value::as_f64),
"coverage": get("coverage").and_then(Value::as_f64),
+ // The spec's measured average value length — the corpus's content
+ // lives in its LONG fields, so the SPA ranks the fields a person
+ // reads first by coverage × avg_len (the dominant fields), not by
+ // type. Not rendered as a claim; a ranking input only.
+ "avg_len": get("avg_len").and_then(Value::as_f64).unwrap_or(0.0),
"examples": examples
.iter()
.take(MAX_EXAMPLES)
@@ -362,10 +372,67 @@ async fn brains_listing(engine: &Engine, may_read: bool) -> ConsoleResult<Vec<Va
Ok(out)
}
+/// The corpus's semantic posture, from the live engine — the facts behind
+/// the `semantic` capability entry (#1099). Two honest shapes, nothing else:
+///
+/// - `Semantic` — the catalog elected a semantic field AND the index's live
+/// schema carries its embedding config: dims/similarity/embedder exactly as
+/// the query and ingest paths resolve them (`Index::semantic_field_facts`).
+/// No vector count is included, because no engine surface exposes one — a
+/// number the node cannot back must not reach the UI.
+/// - `ElectedButMissing` — the catalog's last run elected a semantic field,
+/// but this index has no embedding for it (it was rebuilt or mapped another
+/// way): the entry then says plainly that there are no vectors on this node
+/// for that field.
+///
+/// `None` (no catalog, or no dataset elected a semantic field) → no entry.
+enum SemanticCapability {
+ Semantic(xerj_engine::index::SemanticFieldFacts),
+ ElectedButMissing { field: String, index: String },
+}
+
+/// Render the semantic capability's blurb from the facts above. Every phrase
+/// is grounded: dims/similarity/embedder come from the live schema and the
+/// embedder's own label; the lexical case says it is NOT neural and how to
+/// get neural; the missing case says there is nothing to match.
+fn semantic_blurb(sem: &SemanticCapability) -> String {
+ match sem {
+ SemanticCapability::Semantic(f) => {
+ let dims = if f.dims_from_schema {
+ format!("{}-D", f.dims)
+ } else {
+ format!("{}-D (the engine's built-in default)", f.dims)
+ };
+ if f.embedder_active {
+ format!(
+ "Semantic and hybrid over `{}` → `{}`: {dims} · {} · {} — one vector per \
+ embedded document, embedded by that same backend at ingest.",
+ f.field, f.target_field, f.similarity, f.embedder
+ )
+ } else {
+ format!(
+ "Semantic and hybrid over `{}` → `{}`: {dims} · {} · {} — one vector per \
+ embedded document. That embedder is NOT neural; start the node with \
+ `--embed-mode neural` (or `proxy` / `onnx-experimental`) for neural \
+ semantics.",
+ f.field, f.target_field, f.similarity, f.embedder
+ )
+ }
+ }
+ SemanticCapability::ElectedButMissing { field, index } => format!(
+ "The catalog's last run elected `{field}` as the semantic field, but index \
+ `{index}` carries no embedding for it on this node — there are no vectors to \
+ match, and semantic/hybrid queries on that field are refused. Re-run \
+ `xerj autoindex` over the source (or start with `--embed-mode neural`) to \
+ give it one.",
+ ),
+ }
+}
+
/// The capability strip. Every entry names a shipped surface — console route
/// (`href`), CLI command (`command`), or HTTP endpoint (`endpoint`) — and
/// availability is computed from facts on the node, not from a feature list.
-fn capabilities(catalog: bool, brains: &[Value]) -> Vec<Value> {
+fn capabilities(catalog: bool, brains: &[Value], semantic: Option<&SemanticCapability>) -> Vec<Value> {
let mut caps = vec![
json!({
"id": "search",
@@ -374,14 +441,26 @@ fn capabilities(catalog: bool, brains: &[Value]) -> Vec<Value> {
"href": "#/discover",
"kind": "console",
}),
- json!({
- "id": "read",
- "title": "Read the documents",
- "blurb": "Page through records with highlights and the per-record link list.",
- "href": "#/reader",
- "kind": "console",
- }),
];
+ // The semantic row is computed from the node's own schema and embedder
+ // (#1099): dims, similarity and the embedder's label come from the engine,
+ // and no vector count is ever claimed — the engine backs none.
+ if let Some(sem) = semantic {
+ caps.push(json!({
+ "id": "semantic",
+ "title": "Search it semantically",
+ "blurb": semantic_blurb(sem),
+ "href": "#/discover",
+ "kind": "console",
+ }));
+ }
+ caps.push(json!({
+ "id": "read",
+ "title": "Read the documents",
+ "blurb": "Page through records with highlights and the per-record link list.",
+ "href": "#/reader",
+ "kind": "console",
+ }));
if let Some(brain) = brains.first() {
let name = brain.get("name").and_then(Value::as_str).unwrap_or_default();
let links = brain.get("links").and_then(Value::as_u64).unwrap_or(0);
@@ -523,9 +602,32 @@ pub async fn knowledge(
rank(b).cmp(&rank(a))
});
- // 6. Brains (operator tier) + the capability strip.
+ // 6. The corpus's semantic posture — largest dataset first (the corpus's
+ // dominant one), from the LIVE index schema, never from a feature list.
+ // See `SemanticCapability` for the two honest shapes.
+ let mut semantic: Option<SemanticCapability> = None;
+ for d in &datasets {
+ let Some(field) = str_of(d.get("semantic_field")) else {
+ continue;
+ };
+ let index = d["index"].as_str().unwrap_or_default();
+ let facts = match state.engine.get_index(index) {
+ Ok(idx) => idx.semantic_field_facts(&field).await,
+ Err(_) => None,
+ };
+ semantic = Some(match facts {
+ Some(f) => SemanticCapability::Semantic(f),
+ None => SemanticCapability::ElectedButMissing {
+ field,
+ index: index.to_string(),
+ },
+ });
+ break;
+ }
+
+ // 7. Brains (operator tier) + the capability strip.
let brains = brains_listing(&state.engine, graph::may_read_brains(&sess.user)).await?;
- let caps = capabilities(catalog, &brains);
+ let caps = capabilities(catalog, &brains, semantic.as_ref());
let mut payload = Map::new();
payload.insert("catalog".into(), json!(catalog));
diff --git a/engine/crates/xerj-console-api/tests/knowledge_surface.rs b/engine/crates/xerj-console-api/tests/knowledge_surface.rs
index e847a1bc8..ee64702ce 100644
--- a/engine/crates/xerj-console-api/tests/knowledge_surface.rs
+++ b/engine/crates/xerj-console-api/tests/knowledge_surface.rs
@@ -20,6 +20,10 @@
//! catalog exists, the graph entry only for a role that may read brains
//! AND only when a brain exists, and every entry names a real console
//! route, CLI command, or endpoint;
+//! - the `semantic` entry (#1099) is the node's own posture — dims,
+//! similarity and the embedder's label from the live schema — and never
+//! a vector count (the engine exposes none) nor a READY claim; an index
+//! that lost its embedding says so in as many words;
//! - a node with nothing indexed is the ordinary empty state (`catalog:
//! false`), not an error;
//! - the endpoint is session-authorized: no cookie is a 401.
@@ -30,7 +34,7 @@ use serde_json::{json, Value};
use tempfile::TempDir;
use tower::ServiceExt;
use xerj_common::config::Config;
-use xerj_common::types::{FieldConfig, FieldType, Schema};
+use xerj_common::types::{EmbeddingConfig, FieldConfig, FieldType, Schema};
use xerj_console_api::{
auth::{sessions, store},
state::ClusterMode,
@@ -150,7 +154,13 @@ fn fields_json() -> String {
/// a catalog describing two datasets (one larger, one smaller), one
/// key-overlap correlation, a brain over the first dataset, and one user
/// index the catalog does NOT describe.
-async fn boot() -> TestApp {
+///
+/// `embed_semantic` decides whether ax-mail's `body` carries the embedding
+/// config an autoindex run installs (the `semantic_text` mapping): `true` is
+/// the ordinary post-`xerj brain` node, `false` is an index that was rebuilt
+/// or mapped another way — the catalog still ELECTS `body`, but the index
+/// has nothing to embed it with (#1099's "no vectors on this node" case).
+async fn boot_with(embed_semantic: bool) -> TestApp {
let dir = TempDir::new().unwrap();
let mut cfg = Config::default();
cfg.server.data_dir = dir.path().to_str().unwrap().to_string();
@@ -169,8 +179,22 @@ async fn boot() -> TestApp {
// The dataset indices the catalog describes — 3 real documents in
// ax-mail (the LIVE count the endpoint must join on), 1 in ax-pdfs.
+ // ax-mail's `body` carries the embedding config (dims/similarity) the
+ // es-compat layer installs for autoindex's `semantic_text` mapping.
+ let mut mail_schema = Schema::empty();
+ let mut body = FieldConfig::new("body", FieldType::Text);
+ if embed_semantic {
+ body.options.dimensions = Some(384);
+ body.options.similarity = Some("cosine".into());
+ body.embedding = Some(EmbeddingConfig {
+ endpoint: None,
+ model: None,
+ target_field: None,
+ });
+ }
+ mail_schema.add_field(body).expect("add body");
engine
- .create_index("ax-mail", schema(&[("body", FieldType::Text)]))
+ .create_index("ax-mail", mail_schema)
.expect("create ax-mail");
engine.get_index("ax-mail").unwrap()
.index_documents_batched(vec![
@@ -308,6 +332,12 @@ async fn boot() -> TestApp {
}
}
+/// The ordinary post-`xerj brain` node: the elected semantic field carries
+/// its embedding config.
+async fn boot() -> TestApp {
+ boot_with(true).await
+}
+
/// A node with NOTHING indexed — no catalog, no user index.
async fn boot_empty() -> TestApp {
let dir = TempDir::new().unwrap();
@@ -411,6 +441,10 @@ async fn the_payload_carries_the_catalogs_own_facts() {
let body = by_name("body");
assert_eq!(body["semantic"].as_bool(), Some(true), "the semantic_text field is flagged");
assert_eq!(body["type"].as_str(), Some("semantic_text"));
+ // the spec's measured average value length round-trips — the SPA ranks
+ // the card's fields by coverage × avg_len, so the corpus's DOMINANT
+ // fields lead the table (#1098)
+ assert_eq!(body["avg_len"].as_f64(), Some(412.5));
let page = by_name("page");
assert_eq!(page["null_ratio"].as_f64(), Some(0.75));
assert_eq!(page["coverage"].as_f64(), Some(0.25));
@@ -486,6 +520,71 @@ async fn the_capability_strip_is_grounded_in_facts() {
assert!(!vids.contains(&"graph"), "viewer must not see the graph capability");
}
+/// #1099 — the semantic capability is the NODE's own posture, computed from
+/// the live schema and the embedder, and it never claims a vector count the
+/// engine cannot back (the screenshot defect: "6,546,091 BODY VECTORS ·
+/// 384-D · COSINE — READY" on a node reporting no such number anywhere).
+#[tokio::test]
+async fn the_semantic_capability_is_the_nodes_own_facts_with_no_invented_count() {
+ let app = boot().await;
+ let (_, v) = get(&app, Some(&app.owner_cookie)).await;
+ let caps = data(&v)["capabilities"].as_array().unwrap();
+ let sem = caps
+ .iter()
+ .find(|c| c["id"].as_str() == Some("semantic"))
+ .expect("a semantic_text field with an embedding config earns the semantic entry");
+ // the facts are the schema's and the embedder's own: field, companion,
+ // dims (from the mapping here), similarity, and the embedder's label
+ let blurb = sem["blurb"].as_str().unwrap();
+ assert!(blurb.contains("`body`"), "names the elected field: {blurb}");
+ assert!(blurb.contains("`body_vector`"), "names the companion: {blurb}");
+ assert!(blurb.contains("384-D"), "dims from the mapping: {blurb}");
+ assert!(blurb.contains("cosine"), "similarity from the mapping: {blurb}");
+ assert!(
+ blurb.contains("lexical feature-hash (built-in, 384-dim, non-neural)"),
+ "the embedder's own honesty label, verbatim: {blurb}"
+ );
+ assert!(blurb.contains("NOT neural"), "the lexical case says so plainly: {blurb}");
+ assert!(blurb.contains("--embed-mode neural"), "and how to get neural: {blurb}");
+ // the honest-claims line: no vector count and no READY claim — the node
+ // exposes no per-field vector count on any stats surface, so none may
+ // appear here, whatever the client does with the payload
+ assert!(!blurb.contains("VECTOR"), "no vector count: {blurb}");
+ assert!(!blurb.contains("READY"), "no readiness claim: {blurb}");
+ for key in sem.as_object().unwrap().keys() {
+ assert!(!key.contains("count"), "no count key may ride the entry: {key}");
+ }
+ // it is not brain-gated — it describes the corpus, not the operator's role
+ let (_, vv) = get(&app, Some(&app.viewer_cookie)).await;
+ let viewer_has = data(&vv)["capabilities"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .any(|c| c["id"].as_str() == Some("semantic"));
+ assert!(viewer_has, "the viewer sees the node's semantic posture too");
+}
+
+/// #1099's other shape — the catalog's last run ELECTED a semantic field,
+/// but the live index carries no embedding for it (rebuilt / mapped another
+/// way): the entry must say plainly that there are no vectors to match,
+/// never a derived count and never READY.
+#[tokio::test]
+async fn an_elected_semantic_field_the_index_does_not_carry_is_stated_plainly() {
+ let app = boot_with(false).await;
+ let (_, v) = get(&app, Some(&app.owner_cookie)).await;
+ let caps = data(&v)["capabilities"].as_array().unwrap();
+ let sem = caps
+ .iter()
+ .find(|c| c["id"].as_str() == Some("semantic"))
+ .expect("the catalog elected body; the mismatch must be visible, not omitted");
+ let blurb = sem["blurb"].as_str().unwrap();
+ assert!(blurb.contains("`body`"), "names the elected field: {blurb}");
+ assert!(blurb.contains("no embedding"), "states the gap: {blurb}");
+ assert!(blurb.contains("no vectors to match"), "states it in those words: {blurb}");
+ assert!(!blurb.contains("READY"), "no readiness claim: {blurb}");
+ assert!(!blurb.contains("VECTOR"), "no vector count: {blurb}");
+}
+
#[tokio::test]
async fn nothing_indexed_is_the_empty_state_not_an_error() {
let app = boot_empty().await;
@@ -504,5 +603,6 @@ async fn nothing_indexed_is_the_empty_state_not_an_error() {
.collect();
assert!(!ids.contains(&"map"), "no catalog → no map/ask capability");
assert!(!ids.contains(&"ask"), "no catalog → no ask capability");
+ assert!(!ids.contains(&"semantic"), "no dataset elected a semantic field → no semantic claim");
assert!(ids.contains(&"search"), "search is always real");
}
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 2eae5965b..5102a3e4c 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -7783,6 +7783,30 @@ pub struct FieldEncodingInfo {
pub distinct_values: usize,
}
+/// What the node runs for semantic search over one field — see
+/// [`Index::semantic_field_facts`]. Every value is read from the live schema
+/// and embedder; none is derived, and there is intentionally no vector count.
+#[derive(Debug, Clone, Serialize)]
+pub struct SemanticFieldFacts {
+ /// The concrete field (alias-resolved) carrying the embedding config.
+ pub field: String,
+ /// Companion field the vectors live in (`<field>_vector` by default).
+ pub target_field: String,
+ /// Effective dimensionality — the schema's own, or the engine's built-in
+ /// default when the mapping did not pin one (`dims_from_schema` says which).
+ pub dims: usize,
+ /// `true` when `dims` came from the mapping, `false` when it is the
+ /// engine default the query/ingest paths fall back to.
+ pub dims_from_schema: bool,
+ /// Effective similarity metric (`cosine` unless the mapping overrode it).
+ pub similarity: String,
+ /// The embedder's own honesty label, e.g.
+ /// `"lexical feature-hash (built-in, 384-dim, non-neural)"`.
+ pub embedder: &'static str,
+ /// `false` only for the built-in lexical embedder (non-neural).
+ pub embedder_active: bool,
+}
+
/// Statistics about an index.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct IndexStats {
@@ -24817,6 +24841,57 @@ impl Index {
self.schema.read().await.schema.clone()
}
+ /// What this node actually runs for semantic search over `field` — the
+ /// facts behind any "semantic" claim a console or agent surface renders.
+ ///
+ /// Resolved EXACTLY the way the query and ingest paths resolve it (same
+ /// alias handling, same `dimensions.unwrap_or(DEFAULT_DIMS)` fallback,
+ /// the same embedder instance), so a claim built from these facts cannot
+ /// drift from what a query actually does:
+ ///
+ /// - `Some(facts)` — `field` carries an [`crate::FieldConfig::embedding`]
+ /// config (an ES `semantic_text` field): the companion `target_field`
+ /// holds one vector per embedded document, `dims`/`similarity` are the
+ /// effective ones, and `embedder` is the backend's own label (e.g.
+ /// `"lexical feature-hash (built-in, 384-dim, non-neural)"`).
+ /// - `None` — no embedding config: there is no stored vector to match,
+ /// and the honest answer is "none on this node", never a count derived
+ /// from terms or tokens.
+ ///
+ /// The facts deliberately carry NO vector count: the engine exposes no
+ /// per-field vector count on any stats surface, and a number it cannot
+ /// back must not exist (the honest-claims rule binds UIs, not just docs).
+ pub async fn semantic_field_facts(&self, field: &str) -> Option<SemanticFieldFacts> {
+ let guard = self.schema.read().await;
+ let resolved = resolve_field_alias(&guard.schema, field);
+ let fc = guard.schema.field(&resolved)?;
+ let emb = fc.embedding.as_ref()?;
+ let target_field = emb
+ .target_field
+ .clone()
+ .unwrap_or_else(|| format!("{resolved}_vector"));
+ let (dims, dims_from_schema) = match fc.options.dimensions {
+ Some(d) => (d, true),
+ None => (xerj_ai::local::DEFAULT_DIMS, false),
+ };
+ let similarity = fc
+ .options
+ .similarity
+ .clone()
+ .unwrap_or_else(|| "cosine".to_string());
+ drop(guard);
+ let embedder = self.embedder.read().await;
+ Some(SemanticFieldFacts {
+ field: resolved,
+ target_field,
+ dims,
+ dims_from_schema,
+ similarity,
+ embedder: embedder.describe(),
+ embedder_active: embedder.is_active(),
+ })
+ }
+
/// Add a field to the schema.
pub async fn add_field(self: &Arc<Self>, field: FieldConfig) -> Result<()> {
self.add_fields(vec![field]).await
diff --git a/xerj-ux/src/data/catalog.js b/xerj-ux/src/data/catalog.js
index a2f768681..9c015341c 100644
--- a/xerj-ux/src/data/catalog.js
+++ b/xerj-ux/src/data/catalog.js
@@ -28,9 +28,14 @@ function parseFields(fieldsJson) {
name: f.name,
type: str(f.es_type || f.type || 'object'),
semantic: !!f.semantic,
- coverage: Number.isFinite(Number(f.coverage)) ? Number(f.coverage) : 1,
+ // Unmeasured coverage is null → "—", never 1 → "100%": a field the
+ // catalog did not measure must not read as fully covered (#1098).
+ // (`Number(null)` is 0, so null has to be checked before the finite
+ // test — a JSON null means "not measured", not "measured zero".)
+ coverage: f.coverage == null || !Number.isFinite(Number(f.coverage)) ? null : Number(f.coverage),
cardinality: num(f.cardinality_est),
- nullRatio: Number.isFinite(Number(f.null_ratio)) ? Number(f.null_ratio) : null,
+ nullRatio: f.null_ratio == null || !Number.isFinite(Number(f.null_ratio)) ? null : Number(f.null_ratio),
+ avgLen: Number.isFinite(Number(f.avg_len)) ? Number(f.avg_len) : 0,
examples: Array.isArray(f.examples) ? f.examples.slice(0, 3).map(str) : [],
}));
}
@@ -84,20 +89,26 @@ export function parseCatalogHits(hits) {
return cards;
}
-/** The fields worth showing on a card: the semantic field first, then the
- * best-covered keyword/date/text fields; autoindex plumbing (`ax_*`) last. */
+/** The card's fields in DISPLAY ORDER: the corpus's DOMINANT fields first —
+ * ranked by the share of the corpus's content they hold (the catalog's own
+ * coverage × avg_len), tie-broken by coverage — with autoindex plumbing
+ * (`ax_*`) last. Type must not lead the ranking (#1098): a type-first rank
+ * put every keyword/date field above the text field that IS the corpus, so
+ * a 96%-code corpus opened with 0.6%-coverage email fields and its `code`
+ * field was pushed into the disclosure. The semantic field keeps its accent
+ * styling but no ranking pin — on a notes corpus it dominates on its own. */
+export function rankedFields(card) {
+ const plumbing = (f) => (f.name.startsWith('ax_') ? 1 : 0);
+ const content = (f) => (f.coverage == null ? 0 : f.coverage) * (f.avgLen || 0);
+ const cov = (f) => (f.coverage == null ? -1 : f.coverage);
+ return [...(card.fields || [])].sort(
+ (a, b) => (plumbing(a) - plumbing(b)) || (content(b) - content(a)) || (cov(b) - cov(a)) || a.name.localeCompare(b.name),
+ );
+}
+
+/** The fields worth showing as a card's top chips. */
export function topFields(card, n = 8) {
- const rank = (f) => {
- if (f.name.startsWith('ax_')) return 9;
- if (f.semantic || f.name === card.semanticField) return 0;
- if (f.type === 'date') return 1;
- if (f.type === 'keyword') return 2;
- if (f.type === 'text' || f.type === 'semantic_text') return 3;
- return 5;
- };
- return [...(card.fields || [])]
- .sort((a, b) => (rank(a) - rank(b)) || (b.coverage - a.coverage) || a.name.localeCompare(b.name))
- .slice(0, n);
+ return rankedFields(card).slice(0, n);
}
// ── the knowledge surface ───────────────────────────────────────────
@@ -116,9 +127,10 @@ const knowledgeField = (f) => ({
name: str(f.name),
type: str(f.type || 'object'),
semantic: f.semantic === true,
- coverage: Number.isFinite(Number(f.coverage)) ? Number(f.coverage) : null,
+ coverage: f.coverage == null || !Number.isFinite(Number(f.coverage)) ? null : Number(f.coverage),
cardinality: Number.isFinite(Number(f.cardinality)) ? Number(f.cardinality) : null,
- nullRatio: Number.isFinite(Number(f.null_ratio)) ? Number(f.null_ratio) : null,
+ nullRatio: f.null_ratio == null || !Number.isFinite(Number(f.null_ratio)) ? null : Number(f.null_ratio),
+ avgLen: Number.isFinite(Number(f.avg_len)) ? Number(f.avg_len) : 0,
examples: Array.isArray(f.examples) ? f.examples.slice(0, 3).map(str) : [],
});
diff --git a/xerj-ux/src/ux/corpus-render.js b/xerj-ux/src/ux/corpus-render.js
index 487f8839d..662ed83f9 100644
--- a/xerj-ux/src/ux/corpus-render.js
+++ b/xerj-ux/src/ux/corpus-render.js
@@ -17,7 +17,7 @@
import { h } from './safe-dom.js';
import { readerHref } from './reader-render.js';
-import { topFields, sampleQueryToSearch, fmtBytes, fmtCount, timeSpan } from '../data/catalog.js';
+import { rankedFields, sampleQueryToSearch, fmtBytes, fmtCount, timeSpan } from '../data/catalog.js';
export const EMPTY_COMMAND = 'xerj brain <folder>';
@@ -57,17 +57,18 @@ function fieldChip(f) {
}
/** The fields block: the ranked top fields as chips, then EVERY field in a
- * disclosure when there are more. A corpus's structure is the one thing a
- * person came to see — nothing is capped away. */
+ * disclosure when there are more — in the SAME display order, so the table
+ * reads as one ranked list, top chips first. A corpus's structure is the
+ * one thing a person came to see — nothing is capped away. */
function fieldsBlock(card) {
- const all = card.fields || [];
- if (!all.length) return null;
- const top = topFields(card, 8);
- const rest = all.filter((f) => !top.includes(f));
+ const ranked = rankedFields(card);
+ if (!ranked.length) return null;
+ const top = ranked.slice(0, 8);
+ const rest = ranked.slice(8);
return h('div', { class: 'cp-fields' },
top.map(fieldChip),
rest.length ? h('details', { class: 'cp-fields__all' },
- h('summary', null, `ALL ${all.length} FIELDS`),
+ h('summary', null, `ALL ${ranked.length} FIELDS`),
h('div', { class: 'cp-fields' }, rest.map(fieldChip))) : null);
}
diff --git a/xerj-ux/test/knowledge-surface.test.mjs b/xerj-ux/test/knowledge-surface.test.mjs
index 90a4a4f07..7c69ee77d 100644
--- a/xerj-ux/test/knowledge-surface.test.mjs
+++ b/xerj-ux/test/knowledge-surface.test.mjs
@@ -14,7 +14,7 @@ function json(status, body) {
return { status, ok: status >= 200 && status < 300, json: async () => body, text: async () => JSON.stringify(body) };
}
-const { parseKnowledge } = await import('../src/data/catalog.js');
+const { parseKnowledge, parseCatalogHits, topFields } = await import('../src/data/catalog.js');
const { renderCorpusCard, renderRelations, renderCapabilities } = await import('../src/ux/corpus-render.js');
const { corpus } = await import('../src/dashboards/corpus.js');
const { textOf, findAll } = await import('../src/ux/safe-dom.js');
@@ -172,3 +172,137 @@ test('liveCorpus reads ONE endpoint and keeps the never-mock contract', async ()
assert.deepEqual(err.data.datasets, []);
assert.deepEqual(err.data.relations, []);
});
+
+// ── #1098: the FIELDS table must be the corpus the catalog measured ────
+//
+// The screenshot defect: a 95.8%-code corpus opened with `email_date (date)
+// 100% · email_subject (keyword) 100% · email_from (keyword) 59% · …` —
+// coverage that matches nothing in fields_json — while `code` (0.958) was
+// pushed out of the table entirely. The 100/59/57/43 pattern is coverage
+// within the corpus's mbox-format SUBSET; these tests pin that neither the
+// ranking nor the numbers can present anything but the catalog's own
+// field-for-field, number-for-number facts, whichever path served the card.
+const CODE_CORPUS_FIELDS = [
+ { name: 'code', type: 'text', semantic: false, coverage: 0.958, cardinality: 15436, null_ratio: 0.042, avg_len: 912.0, examples: ['fn short_passage(i: usize) -> String'] },
+ { name: 'title', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 500, null_ratio: 0.0, avg_len: 34.0, examples: ['index.rs'] },
+ { name: 'body', type: 'semantic_text', semantic: true, coverage: 0.039, cardinality: 602, null_ratio: 0.961, avg_len: 412.5, examples: ['term sheet attached'] },
+ { name: 'labels', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 4, null_ratio: 0.0, avg_len: 9.0, examples: ['rust'] },
+ { name: 'defs', type: 'text', semantic: false, coverage: 0.032, cardinality: 208, null_ratio: 0.968, avg_len: 64.0, examples: ['pub fn parse_request'] },
+ { name: 'mtime', type: 'date', semantic: false, coverage: 1.0, cardinality: 491, null_ratio: 0.0, avg_len: 0.0, examples: [] },
+ { name: 'email_date', type: 'date', semantic: false, coverage: 0.0064, cardinality: 99, null_ratio: 0.9936, avg_len: 0.0, examples: [] },
+ { name: 'email_subject', type: 'keyword', semantic: false, coverage: 0.0064, cardinality: 97, null_ratio: 0.9936, avg_len: 31.0, examples: ['Re: contract'] },
+ // the mbox-subset figures from the screenshot — 59/57/43 must never be
+ // able to stand in for the corpus coverage of ANY field
+ { name: 'email_from', type: 'keyword', semantic: false, coverage: 0.0064, cardinality: 61, null_ratio: 0.9936, avg_len: 22.0, examples: ['[email protected]'] },
+ { name: 'email_to', type: 'keyword', semantic: false, coverage: 0.0033, cardinality: 55, null_ratio: 0.9967, avg_len: 22.0, examples: ['[email protected]'] },
+ { name: 'email_cc', type: 'keyword', semantic: false, coverage: 0.0026, cardinality: 37, null_ratio: 0.9974, avg_len: 20.0, examples: [] },
+ { name: 'ax_path', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 500, null_ratio: 0.0, avg_len: 48.0, examples: [] },
+ { name: 'ax_format', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 6, null_ratio: 0.0, avg_len: 4.0, examples: [] },
+ { name: 'ax_source', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 1, null_ratio: 0.0, avg_len: 5.0, examples: [] },
+ { name: 'ax_kind', type: 'keyword', semantic: false, coverage: 1.0, cardinality: 2, null_ratio: 0.0, avg_len: 4.0, examples: [] },
+ // a field the catalog did not measure — "—", never a fabricated 100%
+ { name: 'unmeasured', type: 'long', semantic: false, coverage: null, cardinality: null, null_ratio: null, avg_len: 0.0, examples: [] },
+];
+
+const chipsOf = (card) => findAll(card, (n) => (n.attrs.class || '').split(' ').includes('cp-field'));
+
+test('#1098 the FIELDS table leads with the corpus\'s dominant fields at the catalog\'s own coverage', () => {
+ const st = parseKnowledge({ data: { ...PAYLOAD().data, datasets: [{ ...PAYLOAD().data.datasets[0], fields: CODE_CORPUS_FIELDS }] } });
+ const card = renderCorpusCard(st.datasets[0], {});
+ const chips = chipsOf(card);
+ // the top chips (before the ALL-FIELDS disclosure) are the first 8
+ assert.equal(chips.length, CODE_CORPUS_FIELDS.length, 'every field renders, in the disclosure when not top');
+ const names = chips.map((c) => textOf(findAll(c, (n) => (n.attrs.class || '').includes('cp-field__name'))[0]));
+ // the corpus IS its content: code first, body and defs ahead of every
+ // email_* field, autoindex plumbing last
+ assert.equal(names[0], 'code', `dominant field first: ${names.join(', ')}`);
+ for (const dominant of ['code', 'body', 'defs']) {
+ for (const email of names.filter((n) => n.startsWith('email_'))) {
+ assert.ok(names.indexOf(dominant) < names.indexOf(email), `${dominant} must outrank ${email}: ${names.join(', ')}`);
+ }
+ }
+ const lastThree = names.slice(-3).sort();
+ assert.deepEqual(lastThree, ['ax_format', 'ax_kind', 'ax_source'], `plumbing last: ${names.join(', ')}`);
+ // number-for-number: each chip shows ITS OWN measured coverage, rounded
+ const byName = new Map(CODE_CORPUS_FIELDS.map((f) => [f.name, f]));
+ const chipText = (name) => textOf(chips[names.indexOf(name)]);
+ for (let i = 0; i < chips.length; i++) {
+ const f = byName.get(names[i]);
+ const expected = f.coverage == null ? '—' : `${Math.round(f.coverage * 100)}%`;
+ assert.ok(textOf(chips[i]).includes(expected), `${names[i]} must show ${expected}: ${textOf(chips[i])}`);
+ }
+ // the screenshot's mbox-subset mirage is dead: no email field reads 100%,
+ // 59/57/43 appear nowhere, and the dominant coverage is on the table
+ const text = textOf(card);
+ assert.ok(!chipText('email_date').includes('100%'), `email_date renders its own coverage: ${chipText('email_date')}`);
+ assert.ok(!chipText('email_subject').includes('100%'), `email_subject renders its own coverage: ${chipText('email_subject')}`);
+ for (const bogus of ['59%', '57%', '43%']) assert.ok(!text.includes(bogus), `${bogus} must not render: no field measures it`);
+ assert.ok(chipText('code').includes('96%'), 'code 0.958 leads at 96%'); // code 0.958
+ // unmeasured coverage renders as "—", never as 100%
+ assert.match(chipText('unmeasured'), /—/);
+ assert.doesNotMatch(chipText('unmeasured'), /100%/);
+});
+
+test('#1098 both card paths agree: the catalog-search fallback ranks and numbers identically', () => {
+ // The legacy path parses fields_json (the autoindex catalog document);
+ // the knowledge path parses the endpoint payload. The screenshot showed a
+ // table that disagreed with BOTH — pin that whichever path serves the
+ // card, the same fields lead with the same numbers.
+ const specs = CODE_CORPUS_FIELDS.map((f) => ({
+ name: f.name, es_type: f.type, semantic: f.semantic === true ? 'lexical-hash-384' : undefined,
+ cardinality_est: f.cardinality, null_ratio: f.null_ratio, coverage: f.coverage,
+ avg_len: f.avg_len, examples: f.examples,
+ }));
+ const hits = [{ _id: 'ds:code', _source: {
+ doc_kind: 'dataset', index_name: 'ax-docs', record_count: 15436, file_count: 500,
+ bytes: 16496072, semantic_field: 'body', fields_json: JSON.stringify(specs),
+ } }];
+ const legacy = parseCatalogHits(hits)[0];
+ const st = parseKnowledge({ data: { ...PAYLOAD().data, datasets: [{ ...PAYLOAD().data.datasets[0], index: 'ax-docs', fields: CODE_CORPUS_FIELDS }] } });
+ const knowledgeCard = st.datasets[0];
+ const rank = (card) => topFields(card, 8).map((f) => f.name);
+ assert.deepEqual(rank(legacy), rank(knowledgeCard), 'both paths lead with the same fields in the same order');
+ assert.equal(rank(legacy)[0], 'code');
+ // an unmeasured field is null on BOTH paths — the legacy default of 1
+ // (which rendered "100%") is gone
+ const legacyUnmeasured = legacy.fields.find((f) => f.name === 'unmeasured');
+ assert.equal(legacyUnmeasured.coverage, null);
+});
+
+// ── #1099: the semantic capability states what the node runs — no invented vectors ──
+
+test('#1099 the semantic row renders the node\'s own facts and never a vector count', () => {
+ const lexical = { data: { ...PAYLOAD().data, capabilities: [
+ ...PAYLOAD().data.capabilities,
+ { id: 'semantic', title: 'Search it semantically',
+ blurb: 'Semantic and hybrid over `body` → `body_vector`: 384-D · cosine · lexical feature-hash (built-in, 384-dim, non-neural) — one vector per embedded document. That embedder is NOT neural; start the node with `--embed-mode neural` (or `proxy` / `onnx-experimental`) for neural semantics.',
+ href: '#/discover', kind: 'console' },
+ ] } };
+ const strip = renderCapabilities(parseKnowledge(lexical));
+ const text = textOf(strip);
+ assert.match(text, /Search it semantically/);
+ assert.match(text, /lexical feature-hash/); // the embedder's own label
+ assert.match(text, /NOT neural/);
+ assert.match(text, /--embed-mode neural/);
+ // the honest-claims line: no count of vectors, no READY claim — the node
+ // backs neither ("6,546,091 BODY VECTORS · 384-D · COSINE — READY" was
+ // computed by nothing the node reports)
+ assert.doesNotMatch(text, /\d[\d,.]*\s+VECTORS?/i);
+ assert.doesNotMatch(text, /VECTOR COUNT/i);
+ assert.doesNotMatch(text, /—\s*READY\b/);
+ // and when the node reports no semantic surface, no semantic row renders
+ const none = renderCapabilities(parseKnowledge(PAYLOAD()));
+ assert.doesNotMatch(textOf(none), /Search it semantically/);
+});
+
+test('#1099 a catalog that elected a semantic field the index does not carry says so plainly', () => {
+ const missing = { data: { ...PAYLOAD().data, capabilities: [
+ ...PAYLOAD().data.capabilities,
+ { id: 'semantic', title: 'Search it semantically',
+ blurb: "The catalog's last run elected `body` as the semantic field, but index `ax-mail` carries no embedding for it on this node — there are no vectors to match, and semantic/hybrid queries on that field are refused.",
+ href: '#/discover', kind: 'console' },
+ ] } };
+ const text = textOf(renderCapabilities(parseKnowledge(missing)));
+ assert.match(text, /no vectors to match/);
+ assert.doesNotMatch(text, /\d[\d,.]*\s+VECTORS?/i);
+});