Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
fix(search): start typo tolerance at four characters, not three
Elian Doranc0431e6d15f2
diff --git a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.html b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.htmlindex b0894389ee..aef518d2b0 100644--- a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.html+++ b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.html@@ -411,8 +411,9 @@ class="table">matches a value that is a close whole-word variant of your term. <code>~*</code> (fuzzy-contains)matches when your term appears anywhere inside the value, either as a fragmentor as a near-miss. Both work on note properties such as <code>note.title</code> and <code>note.content</code>,- and on labels (<code>#label</code>). Fuzzy operators require at least 3- characters.</p>+ and on labels (<code>#label</code>). Fuzzy operators accept terms of at least+ 3 characters, though how many typos each term tolerates depends on its length+ — see <em>Fuzzy tolerance</em> below.</p><p>The examples assume a note titled <code>Books</code> carrying the label <code>#author=Tolkien</code>,and a note whose content is <code>learn programming today</code>.</p><figure@@ -466,10 +467,11 @@ class="table"></tbody></table></figure>-<h3>Fuzzy tolerance (AUTO)</h3>+<h3>Fuzzy tolerance</h3><p><strong>Rule:</strong> how many typos are tolerated depends on the <strong>length</strong> of- your search term (the "AUTO" scheme popularized by Elasticsearch). Short- terms must match almost exactly to avoid noise; longer terms tolerate more.</p>+ your search term. Short terms must match exactly, because a single edit is+ enough to turn one short word into an unrelated one; longer terms tolerate+ more.</p><figureclass="table"><table>@@ -481,15 +483,15 @@ class="table"></thead><tbody><tr>- <td>1–2 characters</td>+ <td>1–3 characters</td><td>0 (exact only)</td></tr><tr>- <td>3–5 characters</td>+ <td>4–6 characters</td><td>1</td></tr><tr>- <td>6+ characters</td>+ <td>7+ characters</td><td>2</td></tr></tbody>@@ -513,8 +515,18 @@ class="table"><td>3</td><td><code>a bright red car</code></td>+ <td>No</td>+ <td>No edits are allowed below 4 characters, so <code>cat</code> does not+ reach <code>car</code></td>+ </tr>+ <tr>+ <td><code>carr</code>+ </td>+ <td>4</td>+ <td><code>a blue debit card</code>+ </td><td>Yes</td>- <td>1 edit is within budget for 3–5 character terms</td>+ <td>1 edit is within budget for 4–6 character terms</td></tr><tr><td><code>ceck</code>@@ -534,7 +546,7 @@ class="table"></td><td>Yes</td><td><code>combinef</code>→<code>combined</code> is 1 edit; up to 2 are allowed- at 6+ characters</td>+ at 7+ characters</td></tr></tbody></table>@@ -865,14 +877,14 @@ class="table"><td><code>MIN_FUZZY_TOKEN_LENGTH</code></td><td>3</td>- <td>Minimum characters for fuzzy matching</td>+ <td>Minimum characters the <code>~=</code> and <code>~*</code> operators accept</td></tr><tr><td><code>MAX_EDIT_DISTANCE</code></td><td>2</td>- <td>Ceiling on character changes, reached only by 6+ character terms; shorter- terms allow fewer. See the <em>Fuzzy tolerance (AUTO)</em> section above</td>+ <td>Ceiling on character changes, reached only by 7+ character terms; shorter+ terms allow fewer. See the <em>Fuzzy tolerance</em> section above</td></tr><tr><td><code>RESULT_SUFFICIENCY_THRESHOLD</code>@@ -895,5 +907,5 @@ class="table">issues</li><li>Notes exceeding this limit will still be included in title and attributesearches</li>- <li>Fuzzy matching requires tokens of at least 3 characters</li>+ <li>Terms of 3 characters or fewer are matched exactly; typo tolerance starts at 4 characters</li></ul>\ No newline at end of filediff --git a/docs/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.md b/docs/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.mdindex 93c418ae9e..2f4d3eee4d 100644--- a/docs/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.md+++ b/docs/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.md@@ -133,7 +133,7 @@ The examples below assume four notes: _Austria_ (`#capital=Vienna`), _Somewhere_### Fuzzy operators (`~=` and `~*`)-**Rule:** the fuzzy operators tolerate typos. `~=` (fuzzy-equals) matches a value that is a close whole-word variant of your term. `~*` (fuzzy-contains) matches when your term appears anywhere inside the value, either as a fragment or as a near-miss. Both work on note properties such as `note.title` and `note.content`, and on labels (`#label`). Fuzzy operators require at least 3 characters.+**Rule:** the fuzzy operators tolerate typos. `~=` (fuzzy-equals) matches a value that is a close whole-word variant of your term. `~*` (fuzzy-contains) matches when your term appears anywhere inside the value, either as a fragment or as a near-miss. Both work on note properties such as `note.title` and `note.content`, and on labels (`#label`). Fuzzy operators accept terms of at least 3 characters, though how many typos each term tolerates depends on its length — see _Fuzzy tolerance_ below.The examples assume a note titled `Books` carrying the label `#author=Tolkien`, and a note whose content is `learn programming today`.@@ -144,21 +144,22 @@ The examples assume a note titled `Books` carrying the label `#author=Tolkien`,| `note.content ~* progr` | `learn programming today` | Yes | `progr` is a fragment of `programming` || `note.content ~* programing` | `learn programming today` | Yes | `programing` is one edit from `programming` |-### Fuzzy tolerance (AUTO)+### Fuzzy tolerance-**Rule:** how many typos are tolerated depends on the **length** of your search term (the "AUTO" scheme popularized by Elasticsearch). Short terms must match almost exactly to avoid noise; longer terms tolerate more.+**Rule:** how many typos are tolerated depends on the **length** of your search term. Short terms must match exactly, because a single edit is enough to turn one short word into an unrelated one; longer terms tolerate more.| Term length | Edits allowed || --- | --- |-| 1–2 characters | 0 (exact only) |-| 3–5 characters | 1 |-| 6+ characters | 2 |+| 1–3 characters | 0 (exact only) |+| 4–6 characters | 1 |+| 7+ characters | 2 || Query | Term length | Example note content | Matches? | Why || --- | --- | --- | --- | --- |-| `cat` | 3 | `a bright red car` | Yes | 1 edit is within budget for 3–5 character terms |+| `cat` | 3 | `a bright red car` | No | No edits are allowed below 4 characters, so `cat` does not reach `car` |+| `carr` | 4 | `a blue debit card` | Yes | 1 edit is within budget for 4–6 character terms || `ceck` | 4 | `the latest tech trends` | No | `ceck`→`tech` needs 2 edits; only 1 is allowed at this length |-| `combinef` | 8 | `the values were combined together` | Yes | `combinef`→`combined` is 1 edit; up to 2 are allowed at 6+ characters |+| `combinef` | 8 | `the values were combined together` | Yes | `combinef`→`combined` is 1 edit; up to 2 are allowed at 7+ characters |### Relevance ranking@@ -364,8 +365,8 @@ You can open Trilium and automatically trigger a search by including the search| Parameter | Value | Description || --- | --- | --- |-| `MIN_FUZZY_TOKEN_LENGTH` | 3 | Minimum characters for fuzzy matching |-| `MAX_EDIT_DISTANCE` | 2 | Ceiling on character changes, reached only by 6+ character terms; shorter terms allow fewer. See the _Fuzzy tolerance (AUTO)_ section above |+| `MIN_FUZZY_TOKEN_LENGTH` | 3 | Minimum characters the `~=` and `~*` operators accept |+| `MAX_EDIT_DISTANCE` | 2 | Ceiling on character changes, reached only by 7+ character terms; shorter terms allow fewer. See the _Fuzzy tolerance_ section above || `RESULT_SUFFICIENCY_THRESHOLD` | 5 | Minimum exact results before fuzzy fallback || `MAX_CONTENT_SIZE` | 10MB | Maximum note content size for search processing |@@ -373,4 +374,4 @@ You can open Trilium and automatically trigger a search by including the search* Searched note content is limited to 10MB per note to prevent performance issues* Notes exceeding this limit will still be included in title and attribute searches-* Fuzzy matching requires tokens of at least 3 characters\ No newline at end of file+* Terms of 3 characters or fewer are matched exactly; typo tolerance starts at 4 characters\ No newline at end of filediff --git a/packages/trilium-core/src/services/search/docs_examples.spec.ts b/packages/trilium-core/src/services/search/docs_examples.spec.tsindex d6ff8f978d..b0c72fe838 100644--- a/packages/trilium-core/src/services/search/docs_examples.spec.ts+++ b/packages/trilium-core/src/services/search/docs_examples.spec.ts@@ -207,23 +207,30 @@ describe("Search documentation examples", () => {});});- describe('Fuzzy tolerance (AUTO): token length decides allowed edits', () => {- // Docs: Search.md § "Fuzzy tolerance (AUTO)", length-3-to-5 tier (1 edit)- it("`cat` (length 3) finds `car`: one edit is allowed for 3-5 character tokens", () => {+ describe('Fuzzy tolerance: token length decides allowed edits', () => {+ // Docs: Search.md § "Fuzzy tolerance", length-1-to-3 tier (0 edits)+ it("`cat` (length 3) does NOT find `car`: no edits are allowed below 4 characters", () => {const car = contentNote("Vehicle", "a bright red car");- expect(rank(search("cat"), car.noteId)).toBeGreaterThanOrEqual(0);+ expect(rank(search("cat"), car.noteId)).toEqual(-1);});- // Docs: Search.md § "Fuzzy tolerance (AUTO)", length-3-to-5 tier rejects 2 edits+ // Docs: Search.md § "Fuzzy tolerance", length-4-to-6 tier (1 edit)+ it("`carr` (length 4) finds `card`: one edit is allowed for 4-6 character tokens", () => {+ const card = contentNote("Wallet", "a blue debit card");++ expect(rank(search("carr"), card.noteId)).toBeGreaterThanOrEqual(0);+ });++ // Docs: Search.md § "Fuzzy tolerance", length-4-to-6 tier rejects 2 editsit("`ceck` (length 4) does NOT find `tech`: two edits exceed the 1-edit budget", () => {const tech = contentNote("News", "the latest tech trends");expect(rank(search("ceck"), tech.noteId)).toEqual(-1);});- // Docs: Search.md § "Fuzzy tolerance (AUTO)", length-6+ tier (2 edits)- it("`combinef` (length 8) finds `combined`: two edits are allowed for 6+ character tokens", () => {+ // Docs: Search.md § "Fuzzy tolerance", length-7+ tier (2 edits)+ it("`combinef` (length 8) finds `combined`: two edits are allowed for 7+ character tokens", () => {const combined = contentNote("Build", "the values were combined together");expect(rank(search("combinef"), combined.noteId)).toBeGreaterThanOrEqual(0);diff --git a/packages/trilium-core/src/services/search/utils/text_utils.spec.ts b/packages/trilium-core/src/services/search/utils/text_utils.spec.tsindex 36e779a9b5..e37f88bd27 100644--- a/packages/trilium-core/src/services/search/utils/text_utils.spec.ts+++ b/packages/trilium-core/src/services/search/utils/text_utils.spec.ts@@ -112,29 +112,40 @@ describe('Fuzzy Search Core', () => {});describe('getAutoMaxEditDistance', () => {- it('scales the allowed edit distance by token length (Elasticsearch AUTO-style)', () => {- // 0-2 chars: no fuzzy.+ it('scales the allowed edit distance by token length', () => {+ // 0-3 chars: no fuzzy.expect(getAutoMaxEditDistance(0)).toBe(0);expect(getAutoMaxEditDistance(1)).toBe(0);expect(getAutoMaxEditDistance(2)).toBe(0);- // 3-5 chars: 1 edit.- expect(getAutoMaxEditDistance(3)).toBe(1);+ expect(getAutoMaxEditDistance(3)).toBe(0);+ // 4-6 chars: 1 edit.expect(getAutoMaxEditDistance(4)).toBe(1);expect(getAutoMaxEditDistance(5)).toBe(1);- // 6+ chars: 2 edits.- expect(getAutoMaxEditDistance(6)).toBe(2);+ expect(getAutoMaxEditDistance(6)).toBe(1);+ // 7+ chars: 2 edits.+ expect(getAutoMaxEditDistance(7)).toBe(2);expect(getAutoMaxEditDistance(8)).toBe(2);expect(getAutoMaxEditDistance(20)).toBe(2);});- it('rejects distance-2 typos for short (<=5 char) tokens', () => {+ it('gives a three-character token no edit budget at all', () => {+ // One edit reaches a different word rather than correcting a typo at this length,+ // so "for" must not match "fox" nor "not" match "now".+ expect(getAutoMaxEditDistance(3)).toBe(0);+ expect(fuzzyMatchWord('for', 'fox')).toBe(false);+ expect(fuzzyMatchWord('not', 'now')).toBe(false);+ // The token still matches itself, so an exact hit is unaffected.+ expect(fuzzyMatchWord('for', 'for')).toBe(true);+ });++ it('rejects distance-2 typos for short (<=6 char) tokens', () => {// "sync" (4) vs "send": d=2 > 1 -> no fuzzy match (was a false positive).expect(fuzzyMatchWord('sync', 'send')).toBe(false);// "ceck" (4) vs "tech": d=2 > 1 -> no fuzzy match.expect(fuzzyMatchWord('ceck', 'tech')).toBe(false);});- it('allows distance-2 typos for longer (6+ char) tokens', () => {+ it('allows distance-2 typos for longer (7+ char) tokens', () => {// "combinef" (8) vs "combined": d=1 <= 2 -> fuzzy match.expect(fuzzyMatchWord('combinef', 'combined')).toBe(true);});diff --git a/packages/trilium-core/src/services/search/utils/text_utils.ts b/packages/trilium-core/src/services/search/utils/text_utils.tsindex 1bfbbcc3e1..d8574f3ed1 100644--- a/packages/trilium-core/src/services/search/utils/text_utils.ts+++ b/packages/trilium-core/src/services/search/utils/text_utils.ts@@ -280,16 +280,19 @@ function escapeRegExp(string: string): string {}/**- * Maximum edit distance allowed for a fuzzy match, scaled by token length the way- * Elasticsearch's `fuzziness: AUTO` does: 0 for 1-2 characters, 1 for 3-5, and- * `FUZZY_SEARCH_CONFIG.MAX_EDIT_DISTANCE` beyond that. A flat distance of 2 is loose- * enough to match "sync" against "send".+ * Maximum edit distance allowed for a fuzzy match, scaled by token length: 0 for 1-3 characters,+ * 1 for 4-6, and `FUZZY_SEARCH_CONFIG.MAX_EDIT_DISTANCE` beyond that. A flat distance of 2 is+ * loose enough to match "sync" against "send".+ *+ * The thresholds are the ones Typesense and Algolia use for search-as-you-type. One edit spans+ * too much of a three-letter word to correct a typo rather than reach a different word: "for"+ * would match "fox", "not" would match "now" and "got".*/export function getAutoMaxEditDistance(tokenLength: number): number {- if (tokenLength <= 2) {+ if (tokenLength <= 3) {return 0;}- if (tokenLength <= 5) {+ if (tokenLength <= 6) {return 1;}return FUZZY_SEARCH_CONFIG.MAX_EDIT_DISTANCE;