Skip to content

Open-source manufacturing ERP, MES and QMS. Quoting, MRP, inventory, shop floor, quality and lot/serial traceability on one Postgres schema, with a REST API and MCP server. Self-host or use Carbon Cloud.

refactor(mcp): rank tool search inside zbsearch's sortBy

Sidwebworksd346e3beffaa
diff --git a/.claude/rules/mcp-tools-reference.md b/.claude/rules/mcp-tools-reference.md
index cd43c96fa1..4eb538dc5a 100644
--- a/.claude/rules/mcp-tools-reference.md
+++ b/.claude/rules/mcp-tools-reference.md
@@ -201,15 +201,18 @@ the manifest.
issue). `expandQueryTerm` adds each word's inflection stem to
the QUERY, never the index. The in-app agent's `search_docs`
(`doc-search.ts`) shares it.
-- BM25 only gathers candidates (top `RERANK_POOL`, 200). `rerankByName` orders
- them by the tool NAME: query words it covers, then words matched in full
- (an `ACTION_SYNONYMS` word such as create → insert, or every word of an
- alias), then READ first when the query names no action, then fewer extra name
- words, then a database-paged tool before a `get*List` that reads the whole
- table, then BM25. An exact tool name, with or without its module, is put
- first. BM25 alone ranked `inventory_getInventoryItems` tenth for its own name.
- `stemInflection` leaves "materials" unstemmed, so `sameWord` also accepts a
- plain -s/-es plural.
+- BM25 finds the matches; `createNameComparator` orders them, applied as
+ zbsearch's `sortBy` to every hit before `limit`/`offset`, so paging and the
+ total stay native. Order: the exact tool name (with or without its module),
+ then query words the NAME covers, then words matched in full (an
+ `ACTION_SYNONYMS` word such as create → insert, or every word of an alias),
+ then READ first when the query names no action, then fewer extra name words,
+ then a database-paged tool before a `get*List` that reads the whole table,
+ then the BM25 score. BM25 alone ranked `inventory_getInventoryItems` tenth for
+ its own name, and no `relevance` (k, b) or `threshold` setting gets close:
+ measured 22/34 at best against 33/34. Pinning cannot hold the exact-name rule
+ because it matches the expanded term, not the query. `stemInflection` leaves
+ "materials" unstemmed, so `sameWord` also accepts a plain -s/-es plural.
- `module` keeps substring semantics; filter-only calls bypass the index.
- One line per tool: `name [READ] (requiredParams, +N optional)`
(`formatParamSummary`, `describe-format.ts`), plus the description's first
diff --git a/apps/erp/app/routes/api+/mcp+/lib/catalog-search.test.ts b/apps/erp/app/routes/api+/mcp+/lib/catalog-search.test.ts
index 178e4ee9ab..666bb1ebdd 100644
--- a/apps/erp/app/routes/api+/mcp+/lib/catalog-search.test.ts
+++ b/apps/erp/app/routes/api+/mcp+/lib/catalog-search.test.ts
@@ -6,8 +6,8 @@ import type { ManifestEntry } from "@carbon/api";
import {
collectFieldNames,
createCatalogSearch,
+ createNameComparator,
expandQueryTerm,
- rerankByName,
splitWords
} from "@carbon/ee/mcp";
import { describe, expect, test } from "vitest";
@@ -220,7 +220,7 @@ describe("ranking by name", () => {
});
});
-describe("rerankByName", () => {
+describe("createNameComparator", () => {
const entry = (
name: string,
classification: ManifestEntry["classification"] = "READ"
@@ -228,19 +228,25 @@ describe("rerankByName", () => {
const actions = new Set(["get", "delete", "insert"]);
test("prefers the tighter name when coverage ties", () => {
- const ranked = rerankByName(
- "job",
- [entry("production_getJobOperations"), entry("production_getJob")],
- actions
- );
+ const compare = createNameComparator("job", actions)!;
+ const ranked = [
+ entry("production_getJobOperations"),
+ entry("production_getJob")
+ ].sort(compare);
expect(ranked.map((t) => t.name)).toEqual([
"production_getJob",
"production_getJobOperations"
]);
});
- test("keeps the BM25 order when the query has no usable words", () => {
- const candidates = [entry("a_getX"), entry("b_getY")];
- expect(rerankByName("the of", candidates, actions)).toBe(candidates);
+ test("puts an exact name first, with or without its module", () => {
+ const compare = createNameComparator("getJob", actions)!;
+ expect(
+ compare(entry("production_getJob"), entry("production_getJobs"))
+ ).toBeLessThan(0);
+ });
+
+ test("leaves the BM25 order when the query has no usable words", () => {
+ expect(createNameComparator("the of", actions)).toBe(null);
});
});
diff --git a/packages/ee/src/mcp/catalog-search.ts b/packages/ee/src/mcp/catalog-search.ts
index fce14ad281..114705b7dd 100644
--- a/packages/ee/src/mcp/catalog-search.ts
+++ b/packages/ee/src/mcp/catalog-search.ts
@@ -196,34 +196,37 @@ export function expandQueryTerm(query: string): string {
}
/**
- * Re-rank BM25 candidates by how well each tool's NAME answers the query. BM25
- * finds the right neighbourhood but scores repeated common words ("items",
- * "get") over the one exact tool: `inventory_getInventoryItems` ranked tenth
- * for its own name. Order, highest first:
+ * Order search hits by how well each tool's NAME answers the query. zbsearch
+ * applies it as `sortBy` to every BM25 hit before paging. BM25 alone scores
+ * repeated common words ("items", "get") over the one exact tool:
+ * `inventory_getInventoryItems` ranked tenth for its own name, and no
+ * `relevance` setting fixes that. Order, highest first:
*
- * 1. query terms the name covers. A term is a query word, or a whole
+ * 1. the exact tool name, with or without its module ("getInventoryItems");
+ * 2. query terms the name covers. A term is a query word, or a whole
* PHRASE_ALIASES phrase ("work orders"), which then stands in for its words;
- * 2. of those, terms matched in full: the word itself, an ACTION_SYNONYMS
+ * 3. of those, terms matched in full: the word itself, an ACTION_SYNONYMS
* word ("create" → insert), every word of its alias ("po" → purchase order)
* or its phrase. A stem or part of an alias covers but is not full;
- * 3. with no action word in the query, READ tools first (a bare noun is a
+ * 4. with no action word in the query, READ tools first (a bare noun is a
* lookup);
- * 4. fewer name words the query did not ask for (getJob before
+ * 5. fewer name words the query did not ask for (getJob before
* getJobOperations for "job");
- * 5. a tool that pages in the database before a `get*List` that reads the
+ * 6. a tool that pages in the database before a `get*List` that reads the
* whole table on every call ("list customers" wants getCustomers, not
- * getCustomersList);
- * 6. the BM25 order.
+ * getCustomersList).
+ *
+ * Ties are left at 0 for the caller to break by BM25 score. Returns null when
+ * the query has no words to rank by, so the BM25 order stands.
*
* `actions` is the set of first words of the catalog's function names (get,
* insert, post, …), so a query counts as naming an action only when the
* catalog has one.
*/
-export function rerankByName(
+export function createNameComparator(
query: string,
- candidates: ManifestEntry[],
actions: ReadonlySet<string>
-): ManifestEntry[] {
+): ((a: ManifestEntry, b: ManifestEntry) => number) | null {
let words = [
...new Set(
splitWords(query)
@@ -243,13 +246,14 @@ export function rerankByName(
phrases.push(aliases);
words = words.filter((word) => !keyWords.includes(word));
}
- if (words.length === 0 && phrases.length === 0) return candidates;
+ if (words.length === 0 && phrases.length === 0) return null;
+ const needle = query.trim().toLowerCase();
const hasAction = words.some(
(word) => actions.has(word) || word in ACTION_SYNONYMS
);
- const scored = candidates.map((tool, bm25Rank) => {
+ const rank = (tool: ManifestEntry) => {
const separator = tool.name.indexOf("_");
const tokens = splitWords(tool.name).split(" ");
const tokenSet = new Set(tokens);
@@ -294,24 +298,43 @@ export function rerankByName(
const moduleWords = new Set(
splitWords(tool.name.slice(0, Math.max(separator, 0))).split(" ")
);
- const extra = tokens.filter(
- (token) => !used.has(token) && !moduleWords.has(token) && token !== "get"
- ).length;
- const readFirst = !hasAction && tool.classification === "READ" ? 1 : 0;
- const fullSet = !tool.paginates && tool.name.endsWith("List") ? 1 : 0;
- return { tool, covered, full, readFirst, extra, fullSet, bm25Rank };
- });
-
- scored.sort(
- (a, b) =>
- b.covered - a.covered ||
- b.full - a.full ||
- b.readFirst - a.readFirst ||
- a.extra - b.extra ||
- a.fullSet - b.fullSet ||
- a.bm25Rank - b.bm25Rank
- );
- return scored.map((entry) => entry.tool);
+ const name = tool.name.toLowerCase();
+ return {
+ exact: name === needle || name.slice(separator + 1) === needle ? 1 : 0,
+ covered,
+ full,
+ readFirst: !hasAction && tool.classification === "READ" ? 1 : 0,
+ extra: tokens.filter(
+ (token) =>
+ !used.has(token) && !moduleWords.has(token) && token !== "get"
+ ).length,
+ fullSet: !tool.paginates && tool.name.endsWith("List") ? 1 : 0
+ };
+ };
+
+ // A sort compares each hit many times; rank each tool once.
+ const ranks = new Map<string, ReturnType<typeof rank>>();
+ const rankOf = (tool: ManifestEntry) => {
+ let cached = ranks.get(tool.name);
+ if (!cached) {
+ cached = rank(tool);
+ ranks.set(tool.name, cached);
+ }
+ return cached;
+ };
+
+ return (a, b) => {
+ const x = rankOf(a);
+ const y = rankOf(b);
+ return (
+ y.exact - x.exact ||
+ y.covered - x.covered ||
+ y.full - x.full ||
+ y.readFirst - x.readFirst ||
+ x.extra - y.extra ||
+ x.fullSet - y.fullSet
+ );
+ };
}
/**
@@ -325,9 +348,6 @@ function sameWord(a: string, b: string): boolean {
return long === `${short}s` || long === `${short}es`;
}
-/** How many BM25 hits the name re-rank reorders. */
-const RERANK_POOL = 200;
-
export type CatalogSearchOptions = {
query?: string;
module?: string;
@@ -383,15 +403,6 @@ async function buildIndex(tools: ManifestEntry[]) {
export function createCatalogSearch(tools: ManifestEntry[]): CatalogSearch {
const byName = new Map(tools.map((tool) => [tool.name, tool]));
- const byLowerName = new Map(
- tools.map((tool) => [tool.name.toLowerCase(), tool])
- );
- // "getJobs" without its module: every tool whose function part is that name.
- const byFunction = new Map<string, ManifestEntry[]>();
- for (const tool of tools) {
- const fn = tool.name.slice(tool.name.indexOf("_") + 1).toLowerCase();
- byFunction.set(fn, [...(byFunction.get(fn) ?? []), tool]);
- }
const actions = new Set(
tools.map(
(tool) =>
@@ -439,6 +450,8 @@ export function createCatalogSearch(tools: ManifestEntry[]): CatalogSearch {
}
if (classification) where.classification = { eq: classification };
+ const compare = createNameComparator(query!, actions);
+ const toolOf = (document: { name: string }) => byName.get(document.name)!;
const params = {
term,
properties: ["name", "tokens", "description", "fields"] as (
@@ -449,8 +462,16 @@ export function createCatalogSearch(tools: ManifestEntry[]): CatalogSearch {
)[],
boost: { name: 4, tokens: 3, description: 1.5, fields: 0.5 },
...(Object.keys(where).length > 0 ? { where } : {}),
- limit: Math.max(RERANK_POOL, offset + limit),
- offset: 0
+ ...(compare
+ ? {
+ sortBy: (
+ [, scoreA, a]: [unknown, number, { name: string }],
+ [, scoreB, b]: [unknown, number, { name: string }]
+ ) => compare(toolOf(a), toolOf(b)) || scoreB - scoreA
+ }
+ : {}),
+ limit,
+ offset
};
const db = await getIndex();
@@ -461,31 +482,11 @@ export function createCatalogSearch(tools: ManifestEntry[]): CatalogSearch {
results = await search(db, { ...params, tolerance: 1 });
}
- const passesFilters = (tool: ManifestEntry) =>
- (!modules || modules.includes(tool.module)) &&
- (!classification || tool.classification === classification);
-
- // An exact tool name, with or without its module, always comes first.
- const needle = query!.trim().toLowerCase();
- const exactHits = (
- byLowerName.has(needle)
- ? [byLowerName.get(needle)!]
- : (byFunction.get(needle) ?? [])
- ).filter(passesFilters);
-
- const hits = results.hits
- .map((hit) => byName.get((hit.document as { name: string }).name))
- .filter((tool): tool is ManifestEntry => tool !== undefined);
- const ranked = [
- ...exactHits,
- ...rerankByName(query!, hits, actions).filter(
- (tool) => !exactHits.includes(tool)
- )
- ];
-
return {
- matches: ranked.slice(offset, offset + limit),
- total: Math.max(results.count, exactHits.length)
+ matches: results.hits.map((hit) =>
+ toolOf(hit.document as { name: string })
+ ),
+ total: results.count
};
}