Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

fix(switchdash): stop splitting a hyphenated query into separate terms (CHOO-1423)

Christian McDermott3015ccdc4c3a
diff --git a/dash/apps/switchdash-desktop/src/main/core/search/search-service.db.test.ts b/dash/apps/switchdash-desktop/src/main/core/search/search-service.db.test.ts
index f18ee9a47..4dbdfc57a 100644
--- a/dash/apps/switchdash-desktop/src/main/core/search/search-service.db.test.ts
+++ b/dash/apps/switchdash-desktop/src/main/core/search/search-service.db.test.ts
@@ -218,15 +218,45 @@ describe('searchService', () => {
expect(result.items.filter((i) => i.kind === 'session')).toHaveLength(0);
});
- // The trigram index matches any 3-char substring anywhere, so it happily
- // returns "Manage storage quotas" for "age" and "migration-bot" for "rat".
- // Those are hits the index likes and a person does not.
- it('does not match mid-word, however happy the index is to', () => {
+ it('matches mid-word, so a term inside a compound name is still found', () => {
searchService.initialize();
- expect(titles(searchService.search({ query: 'rat' }).items)).toEqual([]);
- expect(titles(searchService.search({ query: 'iewer' }).items)).toEqual([]);
- expect(titles(searchService.search({ query: 'ion' }).items)).toEqual([]);
+ expect(titles(searchService.search({ query: 'iewer' }).items)).toEqual([
+ 'Fix the reviewer crash',
+ 'reviewer-bot',
+ ]);
+ expect(titles(searchService.search({ query: 'rat' }).items)).toEqual(['migration-bot']);
+ });
+
+ /**
+ * A hyphen is part of a name here, not a separator. Splitting on it turned
+ * `test-tt` into the terms `test` and `tt`, dropped `tt` for being too short
+ * to index, and searched for `test` alone — so every name containing "test"
+ * came back and the query looked far broader than it was.
+ */
+ it('treats a hyphenated query as one string, not two terms', () => {
+ searchService.initialize();
+ for (const [id, name] of [
+ ['a-want', 'test-tt'],
+ ['a-noise', 'co-test'],
+ ['a-noise2', 'test-agent-tt'],
+ ]) {
+ agentEvents._emit('agent:created', agentRecord({ id, name, locationId: 'loc-1' }));
+ }
+
+ expect(titles(searchService.search({ query: 'test-tt' }).items)).toEqual(['test-tt']);
+ });
+
+ // The short term still has to be present, it just cannot be asked of the
+ // index — dropping it silently is what widened the query.
+ it('enforces a term too short to index rather than discarding it', () => {
+ searchService.initialize();
+ agentEvents._emit(
+ 'agent:created',
+ agentRecord({ id: 'a-cc', name: 'reviewer cc', locationId: 'loc-1' })
+ );
+
+ expect(titles(searchService.search({ query: 'reviewer cc' }).items)).toEqual(['reviewer cc']);
});
it('matches a word after a separator, so a suffixed name is still findable', () => {
diff --git a/dash/apps/switchdash-desktop/src/main/core/search/search-service.ts b/dash/apps/switchdash-desktop/src/main/core/search/search-service.ts
index c6f528fc4..a8f8f7aad 100644
--- a/dash/apps/switchdash-desktop/src/main/core/search/search-service.ts
+++ b/dash/apps/switchdash-desktop/src/main/core/search/search-service.ts
@@ -28,31 +28,52 @@ type FtsRow = {
};
/**
- * Rows pulled from the index before the word-boundary filter, and rows returned
- * after it. The index over-matches (see `matchQuality`), so the filter discards
- * a share of what it returns — asking for more candidates than the caller wants
- * keeps a precise query from coming back thin just because a lot of mid-word
- * noise sorted above the real hits.
+ * Rows pulled from the index before filtering, and rows returned after it. The
+ * index answers more loosely than the filter does, so asking for more
+ * candidates than the caller wants keeps a precise query from coming back thin
+ * because loose matches sorted above the real ones.
*/
const FTS_CANDIDATE_LIMIT = 120;
const RESULT_LIMIT = 30;
+/** The shortest term the trigram tokenizer can index. */
+const MIN_INDEXABLE_TERM = 3;
+
/** Ranking tiers, best first: the title starts with the term, a word in the
- * title does, or only a keyword does. */
+ * title does, the title contains it somewhere, or only a keyword does. */
const TIER_TITLE_PREFIX = 0;
const TIER_TITLE_WORD = 1;
-const TIER_KEYWORD = 2;
+const TIER_TITLE_SUBSTRING = 2;
+const TIER_KEYWORD = 3;
/** Wider than any BM25 magnitude, so the tier decides before the score does. */
const TIER_STRIDE = 1_000_000;
/**
- * The worst tier across all terms, or null when any term fails to begin a word
- * in either the title or the keywords.
+ * Split a query into the terms every result must contain.
+ *
+ * **Whitespace only.** Splitting on `-` and `_` as well is what made
+ * `test-tt` return `co-test`: it became the terms `test` and `tt`, `tt` was
+ * dropped for being too short to index, and the search silently degraded to
+ * `test` — which every one of those names contains. A hyphen is part of a name
+ * here (`reviewer-bot`, `test-tt`), not a separator between two things the user
+ * wants ANDed.
+ */
+function queryTerms(query: string): string[] {
+ return query.trim().split(/\s+/).filter(Boolean);
+}
+
+/**
+ * The worst tier across all terms, or null when any term is absent from both
+ * the title and the keywords.
+ *
+ * This is the real filter. The index is trigram-tokenised and answers a phrase
+ * more loosely than it looks — and a term too short to index is not sent to it
+ * at all — so a row coming back is not evidence that it contains what was
+ * typed. Checking the text directly is.
*
- * Judged on the worst term so that a query only ranks as a title match when
- * *every* word of it is one — "reviewer bot" matching an item whose title holds
- * one word and whose description holds the other is a keyword-grade hit, not a
- * name hit.
+ * Judged on the worst term so a query only ranks as a title match when *every*
+ * word of it is one: "reviewer bot" matching an item whose title holds one word
+ * and whose description holds the other is a keyword-grade hit, not a name hit.
*/
function matchTier(row: FtsRow, terms: string[]): number | null {
let worst = TIER_TITLE_PREFIX;
@@ -63,9 +84,11 @@ function matchTier(row: FtsRow, terms: string[]): number | null {
? TIER_TITLE_PREFIX
: inTitle === 'word'
? TIER_TITLE_WORD
- : matchQuality(row.keywords, term)
- ? TIER_KEYWORD
- : null;
+ : inTitle === 'substring'
+ ? TIER_TITLE_SUBSTRING
+ : matchQuality(row.keywords, term)
+ ? TIER_KEYWORD
+ : null;
if (tier === null) return null;
worst = Math.max(worst, tier);
}
@@ -99,20 +122,21 @@ class SearchService {
search({ query, context }: CommandPaletteQuery): SearchResult {
if (!query.trim()) return { items: this.recents(context), status: 'recents' };
- // Trigram tokenizer requires each term to be at least 3 characters, so
- // shorter terms are dropped. When nothing survives there is no query to
- // run: recents are returned, but reported as recents so the palette can
- // say so rather than presenting them as matches.
- const terms = query
- .trim()
- .split(/[\s\-_]+/)
- .filter((t) => t.length >= 3);
+ const terms = queryTerms(query);
- if (terms.length === 0) {
+ // The tokenizer cannot index anything shorter than a trigram, so a query of
+ // only short terms has nothing to ask the index. Recents are returned, but
+ // reported as recents so the palette can say so rather than presenting them
+ // as matches.
+ const indexable = terms.filter((t) => t.length >= MIN_INDEXABLE_TERM);
+ if (indexable.length === 0) {
return { items: this.recents(context), status: 'query-too-short' };
}
- const ftsQuery = terms.map((t) => `"${t}"`).join(' AND ');
+ // Only the indexable terms narrow the candidate set; `matchTier` then holds
+ // every row to *all* the terms, short ones included. Dropping a short term
+ // outright is what silently widened `test-tt` into `test`.
+ const ftsQuery = indexable.map((t) => `"${t.replace(/"/g, '""')}"`).join(' AND ');
let rows: FtsRow[];
try {
diff --git a/dash/apps/switchdash-desktop/src/renderer/features/command-palette/search-utils.ts b/dash/apps/switchdash-desktop/src/renderer/features/command-palette/search-utils.ts
index 0e6f554b4..17ddd52db 100644
--- a/dash/apps/switchdash-desktop/src/renderer/features/command-palette/search-utils.ts
+++ b/dash/apps/switchdash-desktop/src/renderer/features/command-palette/search-utils.ts
@@ -23,12 +23,13 @@ export function applyContextAffinity(
const RENDERER_RESULT_LIMIT = 8;
/**
- * Rank one renderer-matched candidate, or null when the query does not match.
+ * Rank one renderer-matched candidate, or null when the text does not contain
+ * the query.
*
- * Held to the same word-boundary rule as the indexed kinds (see `matchQuality`),
- * so the two populations agree on what counts as a match — a palette where
- * rooms match mid-word and agents do not would be incoherent to use, whichever
- * rule is the better one.
+ * Graded the same way as the indexed kinds (see `matchQuality`) so the two
+ * populations agree on what a match is and on which matches are better — a
+ * palette where rooms and agents disagree about that would be incoherent to
+ * use, whichever rule is the better one.
*
* `score` orders these against each other and nothing else. It is an ordinal,
* not a BM25 rank, so renderer-matched kinds render in their own groups rather
@@ -46,6 +47,8 @@ function rank(haystack: string, query: string): number | null {
return -haystack.length;
case 'word':
return 1000 + haystack.length;
+ case 'substring':
+ return 2000 + haystack.length;
default:
return null;
}
diff --git a/dash/apps/switchdash-desktop/src/renderer/tests/palette-search-utils.test.ts b/dash/apps/switchdash-desktop/src/renderer/tests/palette-search-utils.test.ts
index a44011dde..f0341db17 100644
--- a/dash/apps/switchdash-desktop/src/renderer/tests/palette-search-utils.test.ts
+++ b/dash/apps/switchdash-desktop/src/renderer/tests/palette-search-utils.test.ts
@@ -43,23 +43,34 @@ describe('matchRooms', () => {
// tokenizer's three-character floor that the indexed kinds are.
// No trigram tokenizer here, so there is no three-character floor.
it('answers a one- and two-character query', () => {
- expect(matchRooms(ROOMS, 'se').map((r) => r.id)).toEqual(['r1']);
- // 'Workforce', not 's(w)itchdash'.
- expect(matchRooms(ROOMS, 'w').map((r) => r.id)).toEqual(['r2']);
+ expect(matchRooms(ROOMS, 'se').map((r) => r.id)).toEqual(['r1', 'r3']);
});
- // "relea(se)" is a substring match a person did not ask for. Matching only at
- // word starts is what keeps the result set honest.
- it('does not match mid-word', () => {
- expect(matchRooms(ROOMS, 'lease')).toEqual([]);
- expect(matchRooms(ROOMS, 'ork')).toEqual([]);
+ it('matches mid-word, so a term inside a compound name is still found', () => {
+ expect(matchRooms(ROOMS, 'dash').map((r) => r.id)).toEqual(['r1']);
+ expect(matchRooms(ROOMS, 'lease').map((r) => r.id)).toEqual(['r3']);
});
- it('still matches a word after a separator', () => {
+ it('matches a word after a separator', () => {
expect(matchRooms(ROOMS, 'bar').map((r) => r.id)).toEqual(['r1']);
expect(matchRooms(ROOMS, 'hub').map((r) => r.id)).toEqual(['r2']);
});
+ // The whole query is one substring: a hyphen is part of a name here, not a
+ // separator between two independently-matched terms.
+ it('treats a hyphenated query as one string, not two terms', () => {
+ const hosts = [room({ id: 'want', name: 'test-tt' }), room({ id: 'noise', name: 'co-test' })];
+ expect(matchRooms(hosts, 'test-tt').map((r) => r.id)).toEqual(['want']);
+ });
+
+ it('ranks a name that starts with the query above one that contains it', () => {
+ const hits = matchRooms(
+ [room({ id: 'mid', name: 'the search bar' }), room({ id: 'start', name: 'search bar' })],
+ 'search'
+ );
+ expect(hits.map((h) => h.id)).toEqual(['start', 'mid']);
+ });
+
it('ranks a name that starts with the query above one that merely contains it', () => {
const hits = matchRooms(
[
diff --git a/dash/apps/switchdash-desktop/src/shared/core/search.ts b/dash/apps/switchdash-desktop/src/shared/core/search.ts
index 81bfe9028..ed5d5d200 100644
--- a/dash/apps/switchdash-desktop/src/shared/core/search.ts
+++ b/dash/apps/switchdash-desktop/src/shared/core/search.ts
@@ -46,36 +46,36 @@ export interface SearchResult {
}
/**
- * Where a term matched, best first. `null` means "not usefully matched".
+ * Where a term matched, best first. `null` means the text does not contain the
+ * term at all.
*
- * The FTS index is trigram-tokenised, which matches any substring of three or
- * more characters anywhere in a column. That is far broader than anyone means
- * when they type: "age" matches *Man*age* and *stor*age*, "ion" matches
- * migrat*ion*-bot and Sess*ion*, "rat" matches mig*rat*ion. Every one of those
- * is a hit the index is happy with and a person is not.
+ * A term matches if it appears **anywhere** in the text, so "dash" finds
+ * `switchdash`. That is the recall a substring search is for, and the grade
+ * exists to order results rather than to exclude them: an item whose name
+ * begins with what you typed should outrank one that merely contains it
+ * somewhere in the middle, but both are real answers.
*
- * So the index is treated as a candidate generator and the results are held to
- * a stricter rule: a term must begin a word. That keeps "bot" finding
- * `reviewer-bot` and "ses" finding `New Session`, and drops the mid-word noise.
- *
- * The cost is deliberate and worth stating: a term that starts mid-word no
- * longer matches, so "dash" no longer finds `switchdash`. Precision is bought
- * with recall — there is no setting of this dial that gives both.
+ * Filtering is done by the term being present, not by where — see
+ * `search-service`, where the index's own matching is too loose to be trusted
+ * as the filter.
*/
-export type MatchQuality = 'prefix' | 'word' | null;
+export type MatchQuality = 'prefix' | 'word' | 'substring' | null;
-/** Characters that separate words for matching. Covers the shapes names take
+/** Characters that separate words for ranking. Covers the shapes names take
* here: `reviewer-bot`, `gpu_box`, `switch.local`, `host:port`, paths, URLs. */
const WORD_SEPARATORS = /[\s\-_./:@\\]+/;
/** How well `term` matches `text`: at its start, at the start of a word inside
- * it, or not well enough to offer. */
+ * it, elsewhere inside it, or not at all. */
export function matchQuality(text: string, term: string): MatchQuality {
const haystack = text.toLowerCase();
const needle = term.toLowerCase();
if (!needle) return null;
if (haystack.startsWith(needle)) return 'prefix';
- return haystack.split(WORD_SEPARATORS).some((word) => word.startsWith(needle)) ? 'word' : null;
+ if (!haystack.includes(needle)) return null;
+ return haystack.split(WORD_SEPARATORS).some((word) => word.startsWith(needle))
+ ? 'word'
+ : 'substring';
}
export interface CommandPaletteQuery {