Skip to content

Build your personal knowledge base with Trilium Notes

fix(search): start typo tolerance at four characters, not three

Elian Doranc0431e6d15f2
diff --git a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.html b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.html
index b0894389ee..aef518d2b0 100644
--- a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.html
+++ b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.html
@@ -411,8 +411,9 @@ class="table">
matches a value that is a close whole-word variant of your term. <code>~*</code> (fuzzy-contains)
matches when your term appears anywhere inside the value, either as a fragment
or as a near-miss. Both work on note properties such as <code>note.title</code> and <code>note.content</code>,
- and on labels (<code>#label</code>). Fuzzy operators require at least 3
- characters.</p>
+ and on labels (<code>#label</code>). Fuzzy operators accept terms of at least
+ 3 characters, though how many typos each term tolerates depends on its length
+ — see <em>Fuzzy tolerance</em> below.</p>
<p>The examples assume a note titled <code>Books</code> carrying the label <code>#author=Tolkien</code>,
and a note whose content is <code>learn programming today</code>.</p>
<figure
@@ -466,10 +467,11 @@ class="table">
</tbody>
</table>
</figure>
-<h3>Fuzzy tolerance (AUTO)</h3>
+<h3>Fuzzy tolerance</h3>
<p><strong>Rule:</strong> how many typos are tolerated depends on the <strong>length</strong> of
- your search term (the "AUTO" scheme popularized by Elasticsearch). Short
- terms must match almost exactly to avoid noise; longer terms tolerate more.</p>
+ your search term. Short terms must match exactly, because a single edit is
+ enough to turn one short word into an unrelated one; longer terms tolerate
+ more.</p>
<figure
class="table">
<table>
@@ -481,15 +483,15 @@ class="table">
</thead>
<tbody>
<tr>
- <td>1–2 characters</td>
+ <td>1–3 characters</td>
<td>0 (exact only)</td>
</tr>
<tr>
- <td>3–5 characters</td>
+ <td>4–6 characters</td>
<td>1</td>
</tr>
<tr>
- <td>6+ characters</td>
+ <td>7+ characters</td>
<td>2</td>
</tr>
</tbody>
@@ -513,8 +515,18 @@ class="table">
<td>3</td>
<td><code>a bright red car</code>
</td>
+ <td>No</td>
+ <td>No edits are allowed below 4 characters, so <code>cat</code> does not
+ reach <code>car</code></td>
+ </tr>
+ <tr>
+ <td><code>carr</code>
+ </td>
+ <td>4</td>
+ <td><code>a blue debit card</code>
+ </td>
<td>Yes</td>
- <td>1 edit is within budget for 3–5 character terms</td>
+ <td>1 edit is within budget for 4–6 character terms</td>
</tr>
<tr>
<td><code>ceck</code>
@@ -534,7 +546,7 @@ class="table">
</td>
<td>Yes</td>
<td><code>combinef</code>→<code>combined</code> is 1 edit; up to 2 are allowed
- at 6+ characters</td>
+ at 7+ characters</td>
</tr>
</tbody>
</table>
@@ -865,14 +877,14 @@ class="table">
<td><code>MIN_FUZZY_TOKEN_LENGTH</code>
</td>
<td>3</td>
- <td>Minimum characters for fuzzy matching</td>
+ <td>Minimum characters the <code>~=</code> and <code>~*</code> operators accept</td>
</tr>
<tr>
<td><code>MAX_EDIT_DISTANCE</code>
</td>
<td>2</td>
- <td>Ceiling on character changes, reached only by 6+ character terms; shorter
- terms allow fewer. See the <em>Fuzzy tolerance (AUTO)</em> section above</td>
+ <td>Ceiling on character changes, reached only by 7+ character terms; shorter
+ terms allow fewer. See the <em>Fuzzy tolerance</em> section above</td>
</tr>
<tr>
<td><code>RESULT_SUFFICIENCY_THRESHOLD</code>
@@ -895,5 +907,5 @@ class="table">
issues</li>
<li>Notes exceeding this limit will still be included in title and attribute
searches</li>
- <li>Fuzzy matching requires tokens of at least 3 characters</li>
+ <li>Terms of 3 characters or fewer are matched exactly; typo tolerance starts at 4 characters</li>
</ul>
\ No newline at end of file
diff --git a/docs/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.md b/docs/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.md
index 93c418ae9e..2f4d3eee4d 100644
--- a/docs/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.md
+++ b/docs/User Guide/User Guide/Basic Concepts and Features/Navigation/Search.md
@@ -133,7 +133,7 @@ The examples below assume four notes: _Austria_ (`#capital=Vienna`), _Somewhere_
### Fuzzy operators (`~=` and `~*`)
-**Rule:** the fuzzy operators tolerate typos. `~=` (fuzzy-equals) matches a value that is a close whole-word variant of your term. `~*` (fuzzy-contains) matches when your term appears anywhere inside the value, either as a fragment or as a near-miss. Both work on note properties such as `note.title` and `note.content`, and on labels (`#label`). Fuzzy operators require at least 3 characters.
+**Rule:** the fuzzy operators tolerate typos. `~=` (fuzzy-equals) matches a value that is a close whole-word variant of your term. `~*` (fuzzy-contains) matches when your term appears anywhere inside the value, either as a fragment or as a near-miss. Both work on note properties such as `note.title` and `note.content`, and on labels (`#label`). Fuzzy operators accept terms of at least 3 characters, though how many typos each term tolerates depends on its length — see _Fuzzy tolerance_ below.
The examples assume a note titled `Books` carrying the label `#author=Tolkien`, and a note whose content is `learn programming today`.
@@ -144,21 +144,22 @@ The examples assume a note titled `Books` carrying the label `#author=Tolkien`,
| `note.content ~* progr` | `learn programming today` | Yes | `progr` is a fragment of `programming` |
| `note.content ~* programing` | `learn programming today` | Yes | `programing` is one edit from `programming` |
-### Fuzzy tolerance (AUTO)
+### Fuzzy tolerance
-**Rule:** how many typos are tolerated depends on the **length** of your search term (the "AUTO" scheme popularized by Elasticsearch). Short terms must match almost exactly to avoid noise; longer terms tolerate more.
+**Rule:** how many typos are tolerated depends on the **length** of your search term. Short terms must match exactly, because a single edit is enough to turn one short word into an unrelated one; longer terms tolerate more.
| Term length | Edits allowed |
| --- | --- |
-| 1–2 characters | 0 (exact only) |
-| 3–5 characters | 1 |
-| 6+ characters | 2 |
+| 1–3 characters | 0 (exact only) |
+| 4–6 characters | 1 |
+| 7+ characters | 2 |
| Query | Term length | Example note content | Matches? | Why |
| --- | --- | --- | --- | --- |
-| `cat` | 3 | `a bright red car` | Yes | 1 edit is within budget for 3–5 character terms |
+| `cat` | 3 | `a bright red car` | No | No edits are allowed below 4 characters, so `cat` does not reach `car` |
+| `carr` | 4 | `a blue debit card` | Yes | 1 edit is within budget for 4–6 character terms |
| `ceck` | 4 | `the latest tech trends` | No | `ceck`→`tech` needs 2 edits; only 1 is allowed at this length |
-| `combinef` | 8 | `the values were combined together` | Yes | `combinef`→`combined` is 1 edit; up to 2 are allowed at 6+ characters |
+| `combinef` | 8 | `the values were combined together` | Yes | `combinef`→`combined` is 1 edit; up to 2 are allowed at 7+ characters |
### Relevance ranking
@@ -364,8 +365,8 @@ You can open Trilium and automatically trigger a search by including the search
| Parameter | Value | Description |
| --- | --- | --- |
-| `MIN_FUZZY_TOKEN_LENGTH` | 3 | Minimum characters for fuzzy matching |
-| `MAX_EDIT_DISTANCE` | 2 | Ceiling on character changes, reached only by 6+ character terms; shorter terms allow fewer. See the _Fuzzy tolerance (AUTO)_ section above |
+| `MIN_FUZZY_TOKEN_LENGTH` | 3 | Minimum characters the `~=` and `~*` operators accept |
+| `MAX_EDIT_DISTANCE` | 2 | Ceiling on character changes, reached only by 7+ character terms; shorter terms allow fewer. See the _Fuzzy tolerance_ section above |
| `RESULT_SUFFICIENCY_THRESHOLD` | 5 | Minimum exact results before fuzzy fallback |
| `MAX_CONTENT_SIZE` | 10MB | Maximum note content size for search processing |
@@ -373,4 +374,4 @@ You can open Trilium and automatically trigger a search by including the search
* Searched note content is limited to 10MB per note to prevent performance issues
* Notes exceeding this limit will still be included in title and attribute searches
-* Fuzzy matching requires tokens of at least 3 characters
\ No newline at end of file
+* Terms of 3 characters or fewer are matched exactly; typo tolerance starts at 4 characters
\ No newline at end of file
diff --git a/packages/trilium-core/src/services/search/docs_examples.spec.ts b/packages/trilium-core/src/services/search/docs_examples.spec.ts
index d6ff8f978d..b0c72fe838 100644
--- a/packages/trilium-core/src/services/search/docs_examples.spec.ts
+++ b/packages/trilium-core/src/services/search/docs_examples.spec.ts
@@ -207,23 +207,30 @@ describe("Search documentation examples", () => {
});
});
- describe('Fuzzy tolerance (AUTO): token length decides allowed edits', () => {
- // Docs: Search.md § "Fuzzy tolerance (AUTO)", length-3-to-5 tier (1 edit)
- it("`cat` (length 3) finds `car`: one edit is allowed for 3-5 character tokens", () => {
+ describe('Fuzzy tolerance: token length decides allowed edits', () => {
+ // Docs: Search.md § "Fuzzy tolerance", length-1-to-3 tier (0 edits)
+ it("`cat` (length 3) does NOT find `car`: no edits are allowed below 4 characters", () => {
const car = contentNote("Vehicle", "a bright red car");
- expect(rank(search("cat"), car.noteId)).toBeGreaterThanOrEqual(0);
+ expect(rank(search("cat"), car.noteId)).toEqual(-1);
});
- // Docs: Search.md § "Fuzzy tolerance (AUTO)", length-3-to-5 tier rejects 2 edits
+ // Docs: Search.md § "Fuzzy tolerance", length-4-to-6 tier (1 edit)
+ it("`carr` (length 4) finds `card`: one edit is allowed for 4-6 character tokens", () => {
+ const card = contentNote("Wallet", "a blue debit card");
+
+ expect(rank(search("carr"), card.noteId)).toBeGreaterThanOrEqual(0);
+ });
+
+ // Docs: Search.md § "Fuzzy tolerance", length-4-to-6 tier rejects 2 edits
it("`ceck` (length 4) does NOT find `tech`: two edits exceed the 1-edit budget", () => {
const tech = contentNote("News", "the latest tech trends");
expect(rank(search("ceck"), tech.noteId)).toEqual(-1);
});
- // Docs: Search.md § "Fuzzy tolerance (AUTO)", length-6+ tier (2 edits)
- it("`combinef` (length 8) finds `combined`: two edits are allowed for 6+ character tokens", () => {
+ // Docs: Search.md § "Fuzzy tolerance", length-7+ tier (2 edits)
+ it("`combinef` (length 8) finds `combined`: two edits are allowed for 7+ character tokens", () => {
const combined = contentNote("Build", "the values were combined together");
expect(rank(search("combinef"), combined.noteId)).toBeGreaterThanOrEqual(0);
diff --git a/packages/trilium-core/src/services/search/utils/text_utils.spec.ts b/packages/trilium-core/src/services/search/utils/text_utils.spec.ts
index 36e779a9b5..e37f88bd27 100644
--- a/packages/trilium-core/src/services/search/utils/text_utils.spec.ts
+++ b/packages/trilium-core/src/services/search/utils/text_utils.spec.ts
@@ -112,29 +112,40 @@ describe('Fuzzy Search Core', () => {
});
describe('getAutoMaxEditDistance', () => {
- it('scales the allowed edit distance by token length (Elasticsearch AUTO-style)', () => {
- // 0-2 chars: no fuzzy.
+ it('scales the allowed edit distance by token length', () => {
+ // 0-3 chars: no fuzzy.
expect(getAutoMaxEditDistance(0)).toBe(0);
expect(getAutoMaxEditDistance(1)).toBe(0);
expect(getAutoMaxEditDistance(2)).toBe(0);
- // 3-5 chars: 1 edit.
- expect(getAutoMaxEditDistance(3)).toBe(1);
+ expect(getAutoMaxEditDistance(3)).toBe(0);
+ // 4-6 chars: 1 edit.
expect(getAutoMaxEditDistance(4)).toBe(1);
expect(getAutoMaxEditDistance(5)).toBe(1);
- // 6+ chars: 2 edits.
- expect(getAutoMaxEditDistance(6)).toBe(2);
+ expect(getAutoMaxEditDistance(6)).toBe(1);
+ // 7+ chars: 2 edits.
+ expect(getAutoMaxEditDistance(7)).toBe(2);
expect(getAutoMaxEditDistance(8)).toBe(2);
expect(getAutoMaxEditDistance(20)).toBe(2);
});
- it('rejects distance-2 typos for short (<=5 char) tokens', () => {
+ it('gives a three-character token no edit budget at all', () => {
+ // One edit reaches a different word rather than correcting a typo at this length,
+ // so "for" must not match "fox" nor "not" match "now".
+ expect(getAutoMaxEditDistance(3)).toBe(0);
+ expect(fuzzyMatchWord('for', 'fox')).toBe(false);
+ expect(fuzzyMatchWord('not', 'now')).toBe(false);
+ // The token still matches itself, so an exact hit is unaffected.
+ expect(fuzzyMatchWord('for', 'for')).toBe(true);
+ });
+
+ it('rejects distance-2 typos for short (<=6 char) tokens', () => {
// "sync" (4) vs "send": d=2 > 1 -> no fuzzy match (was a false positive).
expect(fuzzyMatchWord('sync', 'send')).toBe(false);
// "ceck" (4) vs "tech": d=2 > 1 -> no fuzzy match.
expect(fuzzyMatchWord('ceck', 'tech')).toBe(false);
});
- it('allows distance-2 typos for longer (6+ char) tokens', () => {
+ it('allows distance-2 typos for longer (7+ char) tokens', () => {
// "combinef" (8) vs "combined": d=1 <= 2 -> fuzzy match.
expect(fuzzyMatchWord('combinef', 'combined')).toBe(true);
});
diff --git a/packages/trilium-core/src/services/search/utils/text_utils.ts b/packages/trilium-core/src/services/search/utils/text_utils.ts
index 1bfbbcc3e1..d8574f3ed1 100644
--- a/packages/trilium-core/src/services/search/utils/text_utils.ts
+++ b/packages/trilium-core/src/services/search/utils/text_utils.ts
@@ -280,16 +280,19 @@ function escapeRegExp(string: string): string {
}
/**
- * Maximum edit distance allowed for a fuzzy match, scaled by token length the way
- * Elasticsearch's `fuzziness: AUTO` does: 0 for 1-2 characters, 1 for 3-5, and
- * `FUZZY_SEARCH_CONFIG.MAX_EDIT_DISTANCE` beyond that. A flat distance of 2 is loose
- * enough to match "sync" against "send".
+ * Maximum edit distance allowed for a fuzzy match, scaled by token length: 0 for 1-3 characters,
+ * 1 for 4-6, and `FUZZY_SEARCH_CONFIG.MAX_EDIT_DISTANCE` beyond that. A flat distance of 2 is
+ * loose enough to match "sync" against "send".
+ *
+ * The thresholds are the ones Typesense and Algolia use for search-as-you-type. One edit spans
+ * too much of a three-letter word to correct a typo rather than reach a different word: "for"
+ * would match "fox", "not" would match "now" and "got".
*/
export function getAutoMaxEditDistance(tokenLength: number): number {
- if (tokenLength <= 2) {
+ if (tokenLength <= 3) {
return 0;
}
- if (tokenLength <= 5) {
+ if (tokenLength <= 6) {
return 1;
}
return FUZZY_SEARCH_CONFIG.MAX_EDIT_DISTANCE;