Skip to content

Build your personal knowledge base with Trilium Notes

Merge remote-tracking branch 'origin/main' into feat/ocr-scanned-pdfs

Elian Doran33c8a84ad021
diff --cc apps/server/src/assets/doc_notes/en/User Guide/User Guide/Advanced Usage/Text Extraction (OCR).html
index 8fb65f552e,a68f9a30e5..4f60f7581b
--- a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Advanced Usage/Text Extraction (OCR).html
+++ b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Advanced Usage/Text Extraction (OCR).html
@@@ -31,104 -31,98 +31,106 @@@
<li>The underlying technology is Tesseract.js.</li>
</ul>
<h3>PDFs</h3>
-<p>Currently only text extraction is supported and not OCR.</p>
+<p>The PDF extraction supports:</p>
<ul>
- <li>This means that the PDF needs to have proper text information in it (i.e.
- the text can be selected in a PDF viewer), whereas scanned documents are
- not yet supported.</li>
- <li>There are plans to integrate the same OCR-based recognition for PDFs used
- for images, but this is not yet implemented.</li>
+ <li>PDFs with text information (e.g. a PDF created by a browser or a word
+ processing application), in which the text is extracted.</li>
+ <li>Scanned PDFs, in which the text is extracted through an OCR process.</li>
+ <li>Mixed PDFs, containing both text and images with text content.</li>
</ul>
+<p>Note that OCR extraction is limited to 50 pages per PDF, pages containing
+ text information (not as an image) are not subject to this limitation.</p>
+<aside
+class="admonition note">
- <p>Text extraction for scanned PDFs using OCR was introduced in v0.104.0.
++ <p>Text extraction for scanned PDFs using OCR was introduced in v0.106.0.
+ If a scanned PDF was already processed by an earlier version, reprocess
+ it individually via <em>Process OCR</em>; batch processing only handles
- files that haven't been OCR'd yet.</p>
++ files that haven&apos;t been OCR&apos;d yet.</p>
+ </aside>
- <h3>Office documents</h3>
- <p>The text will be extracted from the following file formats:</p>
+ <h3>Office documents</h3>
+ <p>The text will be extracted from the following file formats:</p>
+ <ul>
+ <li>Microsoft Word documents</li>
+ <li>Microsoft Excel documents
+ <ul>
+ <li>Only the raw text information, the cell structure is not maintained.</li>
+ <li>Values are extracted raw, so searching for dates will not work. The OpenDocument
+ alternative will actually extract the properly formatted value.</li>
+ </ul>
+ </li>
+ <li>Microsoft PowerPoint documents</li>
+ <li>The OpenDocument alternatives to the previous formats (Text, Spreadsheet,
+ Presentation), created by editors such as LibreOffice and OpenOffice.</li>
+ <li><a href="https://en.wikipedia.org/wiki/Rich_Text_Format">Rich Text Format (RTF)</a>,
+ since v0.104.0.</li>
+ <li><a href="https://en.wikipedia.org/wiki/EPUB">EPUB</a>, since v0.104.1.</li>
+ </ul>
+ <h2>Configuring and triggering OCR</h2>
+ <p>The OCR can be configured by going to&nbsp;<a class="reference-link" href="#root/_help_4TIF1oA4VQRO">Options</a>&nbsp;→&nbsp;
+ <a
+ class="reference-link" href="#root/_hidden/_options/_optionsMedia">Media</a>&nbsp;and looking for the <em>Text Extraction (OCR)</em> section.</p>
+ <p>There are three ways to trigger the OCR:</p>
+ <ul>
+ <li>By enabling <em>Auto-process new files</em> which will process only the
+ notes or attachments created after enabling the option, existing files
+ will remain unprocessed.</li>
+ <li>By pressing <em>Start Batch Processing</em> which will process all the existing
+ notes.</li>
+ <li>By manually requesting for an image or file to have its text extracted,
+ regardless of whether the automatic processing is enabled or not.</li>
+ </ul>
+ <h3>Minimum confidence</h3>
+ <p>When extracting text from an image, there is a certain level of confidence
+ which indicates whether the extracted text appears relevant.</p>
+ <p>When the minimum confidence is set to a low percentage, the text extraction
+ can interpret symbols and drawings incorrectly resulting in garbled text.</p>
+ <p>If the extracted text for a note or an attachment quality is lower than
+ the minimum confidence, the OCR is disregarded.</p>
+ <h2>Language management</h2>
+ <p>OCR needs to be aware of the language of the content in order for it to
+ work correctly. The reason is that each language has its own data which
+ needs to be downloaded, and accents or other symbols will not be supported
+ by the default language.</p>
+ <p>To configure the languages that are supported by the OCR, simply go to&nbsp;
+ <a
+ class="reference-link" href="#root/_help_4TIF1oA4VQRO">Options</a>&nbsp;→&nbsp;<a class="reference-link" href="#root/_hidden/_options/_optionsLocalization">Language &amp; Region</a>&nbsp;and
+ adjust the <em>Content languages</em>.</p>
+ <p>When there are no content languages defined, the user interface <em>Language</em> is
+ used instead.</p>
+ <p>After making this change, the automatic processing or manual reprocessing
+ will take into consideration the new languages.</p>
+ <p>To enforce the detection in a particular language for a given note, use
+ the <code spellcheck="false">language</code> <a href="#root/_help_zEY4DaJG4YT5">attribute</a>,
+ similar to <a href="#root/_help_veGu4faJErEM">text content language</a>. For&nbsp;
+ <a
+ class="reference-link" href="#root/_help_0vhv7lsOLy82">Attachments</a>, it's not possible to manually adjust the language.</p>
+ <aside
+ class="admonition note">
+ <p>The trained data for each language is not packaged with Trilium, as that
+ would require a significant amount of space that might not be otherwise
+ needed. As such, when the trained data will be downloaded automatically
+ via <a href="https://github.com/naptha/tesseract.js/">Tesseract.js</a>.</p>
+ <p>The downloaded trained data is located in the&nbsp;<a class="reference-link"
+ href="#root/_help_tAassRL4RSQL">Data directory</a>, in the <code spellcheck="false">ocr-cache</code> directory.</p>
+ </aside>
+ <h2>Viewing extracted content for a single note</h2>
+ <p>To access the extracted content of a note:</p>
<ul>
- <li>Microsoft Word documents</li>
- <li>Microsoft Excel documents
- <ul>
- <li>Only the raw text information, the cell structure is not maintained.</li>
- <li>Values are extracted raw, so searching for dates will not work. The OpenDocument
- alternative will actually extract the properly formatted value.</li>
- </ul>
- </li>
- <li>Microsoft PowerPoint documents</li>
- <li>The OpenDocument alternatives to the previous formats (Text, Spreadsheet,
- Presentation), created by editors such as LibreOffice and OpenOffice.</li>
- <li>Rich Text Format (RTF), since v0.104.0.</li>
+ <li>For&nbsp;<a class="reference-link" href="#root/_help_W8vYD3Q1zjCR">File</a>&nbsp;notes,
+ go to the&nbsp;<a class="reference-link" href="#root/_help_8YBEPzcpUgxw">Note buttons</a>&nbsp;→ <em>Advanced</em> → <em>View OCR Text</em>.</li>
+ <li>For&nbsp;<a class="reference-link" href="#root/_help_0vhv7lsOLy82">Attachments</a>&nbsp;(e.g.&nbsp;
+ <a
+ class="reference-link" href="#root/_help_mT0HEkOsz6i1">Images</a>&nbsp;in&nbsp;<a class="reference-link" href="#root/_help_iPIMuisry3hd">Text</a>&nbsp;notes),
+ double-click the attachment to view the details, press the […] button at
+ the left and select <em>View extracted text (OCR)</em>.</li>
</ul>
- <h2>Configuring and triggering OCR</h2>
- <p>The OCR can be configured by going to&nbsp;<a class="reference-link" href="#root/_help_4TIF1oA4VQRO">Options</a>&nbsp;→&nbsp;
- <a
- class="reference-link" href="#root/_hidden/_options/_optionsMedia">Media</a>&nbsp;and looking for the <em>Text Extraction (OCR)</em> section.</p>
- <p>There are three ways to trigger the OCR:</p>
+ <p>This section allows:</p>
<ul>
- <li>By enabling <em>Auto-process new files</em> which will process only the
- notes or attachments created after enabling the option, existing files
- will remain unprocessed.</li>
- <li>By pressing <em>Start Batch Processing</em> which will process all the existing
- notes.</li>
- <li>By manually requesting for an image or file to have its text extracted,
- regardless of whether the automatic processing is enabled or not.</li>
- </ul>
- <h3>Minimum confidence</h3>
- <p>When extracting text from an image, there is a certain level of confidence
- which indicates whether the extracted text appears relevant.</p>
- <p>When the minimum confidence is set to a low percentage, the text extraction
- can interpret symbols and drawings incorrectly resulting in garbled text.</p>
- <p>If the extracted text for a note or an attachment quality is lower than
- the minimum confidence, the OCR is disregarded.</p>
- <h2>Language management</h2>
- <p>OCR needs to be aware of the language of the content in order for it to
- work correctly. The reason is that each language has its own data which
- needs to be downloaded, and accents or other symbols will not be supported
- by the default language.</p>
- <p>To configure the languages that are supported by the OCR, simply go to&nbsp;
- <a
- class="reference-link" href="#root/_help_4TIF1oA4VQRO">Options</a>&nbsp;→&nbsp;<a class="reference-link" href="#root/_hidden/_options/_optionsLocalization">Language &amp; Region</a>&nbsp;and
- adjust the <em>Content languages</em>.</p>
- <p>When there are no content languages defined, the user interface <em>Language</em> is
- used instead.</p>
- <p>After making this change, the automatic processing or manual reprocessing
- will take into consideration the new languages.</p>
- <p>To enforce the detection in a particular language for a given note, use
- the <code spellcheck="false">language</code> <a href="#root/_help_zEY4DaJG4YT5">attribute</a>,
- similar to <a href="#root/_help_veGu4faJErEM">text content language</a>. For&nbsp;
- <a
- class="reference-link" href="#root/_help_0vhv7lsOLy82">Attachments</a>, it's not possible to manually adjust the language.</p>
- <aside
- class="admonition note">
- <p>The trained data for each language is not packaged with Trilium, as that
- would require a significant amount of space that might not be otherwise
- needed. As such, when the trained data will be downloaded automatically
- via <a href="https://github.com/naptha/tesseract.js/">Tesseract.js</a>.</p>
- <p>The downloaded trained data is located in the&nbsp;<a class="reference-link"
- href="#root/_help_tAassRL4RSQL">Data directory</a>, in the <code spellcheck="false">ocr-cache</code> directory.</p>
- </aside>
- <h2>Viewing extracted content for a single note</h2>
- <p>To access the extracted content of a note:</p>
- <ul>
- <li>For&nbsp;<a class="reference-link" href="#root/_help_W8vYD3Q1zjCR">File</a>&nbsp;notes,
- go to the&nbsp;<a class="reference-link" href="#root/_help_8YBEPzcpUgxw">Note buttons</a>&nbsp;→ <em>Advanced</em> → <em>View OCR Text</em>.</li>
- <li>For&nbsp;<a class="reference-link" href="#root/_help_0vhv7lsOLy82">Attachments</a>&nbsp;(e.g.&nbsp;
- <a
- class="reference-link" href="#root/_help_mT0HEkOsz6i1">Images</a>&nbsp;in&nbsp;<a class="reference-link" href="#root/_help_iPIMuisry3hd">Text</a>&nbsp;notes),
- double-click the attachment to view the details, press the […] button at
- the left and select <em>View extracted text (OCR)</em>.</li>
- </ul>
- <p>This section allows:</p>
- <ul>
- <li>Viewing the extracted text, which can be copied elsewhere if needed or
- just to check the quality of the extraction.</li>
- <li>If the note has not been extracted yet, pressing <em>Process OCR</em> will
- process it in the background. If the extraction confidence is lower than
- the minimum confidence, there will be a notification.</li>
- <li>Similarly, if the minimum confidence was changed in settings, it is possible
- to press the <em>Process OCR</em> button again to extract the text again.</li>
- </ul>
+ <li>Viewing the extracted text, which can be copied elsewhere if needed or
+ just to check the quality of the extraction.</li>
+ <li>If the note has not been extracted yet, pressing <em>Process OCR</em> will
+ process it in the background. If the extraction confidence is lower than
+ the minimum confidence, there will be a notification.</li>
+ <li>Similarly, if the minimum confidence was changed in settings, it is possible
+ to press the <em>Process OCR</em> button again to extract the text again.</li>
+ </ul>
diff --cc apps/server/src/services/ocr/processors/pdf_processor.ts
index 70ccd8ccd7,1ca486dc7e..ef84c2b8b4
--- a/apps/server/src/services/ocr/processors/pdf_processor.ts
+++ b/apps/server/src/services/ocr/processors/pdf_processor.ts
@@@ -1,41 -1,9 +1,39 @@@
import { getLog } from "@triliumnext/core";
- import { Jimp } from "jimp";
- import { extractImages, extractText, getDocumentProxy } from 'unpdf';
+
import { OCRProcessingOptions, OCRResult } from '../ocr_service.js';
+import recognizer from '../tesseract_recognizer.js';
import { FileProcessor } from './file_processor.js';
- type PdfDocument = Awaited<ReturnType<typeof getDocumentProxy>>;
++type PdfDocument = Awaited<ReturnType<typeof import('unpdf').getDocumentProxy>>;
+
+/** A page whose embedded text is shorter than this (after trimming) is treated as scanned and sent to OCR. */
+const MIN_EMBEDDED_PAGE_CHARS = 16;
+
+/**
+ * Upper bound on how many scanned pages a single PDF may OCR. Rasterizing and
+ * recognizing a page costs seconds and non-trivial memory, so an unbounded scan
+ * of a large document could stall the OCR queue. Pages beyond the cap keep
+ * whatever embedded text they have (usually none) and are logged, never silently
+ * dropped.
+ */
+const MAX_OCR_PAGES = 50;
+
+/** An embedded-text page is exact, so it is reported at the same high confidence the previous text-only path used. */
+const EMBEDDED_TEXT_CONFIDENCE = 0.99;
+
/**
- * PDF processor for extracting embedded text from PDF files using unpdf.
+ * Embedded images smaller than this on either side are skipped: scanned pages are
+ * full-page raster, whereas tiny images are almost always icons, bullets or rules
+ * that hold no recognizable text and would only waste an OCR pass.
+ */
+const MIN_OCR_IMAGE_DIM = 50;
+
+/**
+ * PDF processor. Prefers the PDF's embedded text layer (fast and exact) and
+ * falls back to OCR for scanned, image-only pages by extracting each page's
+ * embedded images and running them through the shared Tesseract recognizer.
+ * Detection is per page, so mixed PDFs (some real-text pages, some scans) are
+ * handled correctly.
*/
export class PDFProcessor extends FileProcessor {
@@@ -50,59 -18,17 +48,61 @@@
async extractText(buffer: Buffer, options: OCRProcessingOptions = {}): Promise<OCRResult> {
getLog().info('Starting PDF text extraction...');
+ const language = options.language || "eng";
+ // Dynamically imported so unpdf only loads when a PDF is actually processed.
+ const { extractText, getDocumentProxy } = await import('unpdf');
const pdf = await getDocumentProxy(new Uint8Array(buffer));
- const { totalPages, text } = await extractText(pdf, { mergePages: true });
- const trimmed = text.trim();
+ const { totalPages, text: pageTexts } = await extractText(pdf, { mergePages: false });
+
+ const pageResults: string[] = [];
+ const pageConfidences: number[] = [];
+ let ocrPagesProcessed = 0;
+ let ocrPagesSkipped = 0;
+
+ for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
+ const embedded = (pageTexts[pageNum - 1] ?? "").trim();
+
+ if (embedded.length >= MIN_EMBEDDED_PAGE_CHARS) {
+ pageResults.push(embedded);
+ pageConfidences.push(EMBEDDED_TEXT_CONFIDENCE);
+ continue;
+ }
+
+ if (ocrPagesProcessed >= MAX_OCR_PAGES) {
+ ocrPagesSkipped++;
+ if (embedded.length > 0) {
+ pageResults.push(embedded);
+ pageConfidences.push(EMBEDDED_TEXT_CONFIDENCE);
+ }
+ continue;
+ }
+
+ ocrPagesProcessed++;
+ const ocr = await this.ocrPage(pdf, pageNum, language);
+ if (ocr.text.length > 0) {
+ pageResults.push(ocr.text);
+ pageConfidences.push(ocr.confidence);
+ } else if (embedded.length > 0) {
+ // Nothing OCR-able on the page; keep whatever little embedded text it had.
+ pageResults.push(embedded);
+ pageConfidences.push(EMBEDDED_TEXT_CONFIDENCE);
+ }
+ }
+
+ if (ocrPagesSkipped > 0) {
+ getLog().info(`PDF OCR page cap reached: OCR'd ${MAX_OCR_PAGES} page(s), skipped ${ocrPagesSkipped} further scanned page(s).`);
+ }
+
+ const text = pageResults.join("\n\n").trim();
+ const confidence = text.length === 0 || pageConfidences.length === 0
+ ? 0
+ : pageConfidences.reduce((sum, c) => sum + c, 0) / pageConfidences.length;
return {
- text: trimmed,
- confidence: trimmed.length > 0 ? 0.99 : 0,
+ text,
+ confidence,
extractedAt: new Date().toISOString(),
- language: options.language || "eng",
+ language,
pageCount: totalPages
};
}
@@@ -111,90 -37,4 +111,93 @@@
return 'pdf';
}
+ /**
+ * OCR a single scanned page by recognizing each embedded image it paints and
+ * concatenating the results. Failures on an individual page are logged and
+ * treated as "no text" so one bad page never aborts the whole document.
+ */
+ private async ocrPage(pdf: PdfDocument, pageNum: number, language: string): Promise<{ text: string; confidence: number }> {
+ try {
++ const { extractImages } = await import('unpdf');
+ const images = await extractImages(pdf, pageNum);
+ if (images.length === 0) {
+ return { text: "", confidence: 0 };
+ }
+
+ const parts: string[] = [];
+ const confidences: number[] = [];
+
+ for (const image of images) {
+ if (image.width < MIN_OCR_IMAGE_DIM || image.height < MIN_OCR_IMAGE_DIM) {
+ continue;
+ }
+ const png = await toPngBuffer(image);
+ const { text, confidence } = await recognizer.recognize(png, language);
+ if (text.length > 0) {
+ parts.push(text);
+ confidences.push(confidence);
+ }
+ }
+
+ return {
+ text: parts.join("\n"),
+ confidence: confidences.length > 0
+ ? confidences.reduce((sum, c) => sum + c, 0) / confidences.length
+ : 0
+ };
+ } catch (error) {
+ getLog().error(`PDF OCR failed for page ${pageNum}: ${error}`);
+ return { text: "", confidence: 0 };
+ }
+ }
+}
+
+interface ExtractedImage {
+ data: Uint8ClampedArray;
+ width: number;
+ height: number;
+ // Typed as a plain number rather than `1 | 3 | 4`: the value is whatever unpdf
+ // hands us at runtime, so it is validated below rather than trusted here.
+ channels: number;
+}
+
+/**
+ * Encode a raw image extracted from a PDF into a PNG buffer that Tesseract can
+ * decode. unpdf returns 1- (grayscale), 3- (RGB) or 4-channel (RGBA) pixel data;
+ * Jimp bitmaps are always RGBA, so narrower formats are expanded here. An
+ * unexpected channel count throws rather than silently producing garbled pixels.
+ */
+async function toPngBuffer(image: ExtractedImage): Promise<Buffer> {
+ const { data, width, height, channels } = image;
+ const pixelCount = width * height;
+
+ let rgba: Buffer;
+ if (channels === 4) {
+ // Already RGBA — reuse the underlying bytes instead of copying pixel by pixel.
+ rgba = Buffer.from(data.buffer, data.byteOffset, data.byteLength);
+ } else if (channels === 3 || channels === 1) {
+ rgba = Buffer.alloc(pixelCount * 4);
+ for (let i = 0; i < pixelCount; i++) {
+ const out = i * 4;
+ if (channels === 1) {
+ const value = data[i];
+ rgba[out] = value;
+ rgba[out + 1] = value;
+ rgba[out + 2] = value;
+ } else {
+ const src = i * 3;
+ rgba[out] = data[src];
+ rgba[out + 1] = data[src + 1];
+ rgba[out + 2] = data[src + 2];
+ }
+ rgba[out + 3] = 255;
+ }
+ } else {
+ throw new Error(`Unsupported image channel count: ${channels}`);
+ }
+
++ // Dynamically imported so jimp only loads when a scanned page is actually rasterized.
++ const { Jimp } = await import("jimp");
+ const jimpImage = Jimp.fromBitmap({ data: rgba, width, height });
+ return jimpImage.getBuffer("image/png");
}
diff --cc apps/server/src/services/ocr/tesseract_recognizer.spec.ts
index f0c179efdf,0000000000..39734ce0a4
mode 100644,000000..100644
--- a/apps/server/src/services/ocr/tesseract_recognizer.spec.ts
+++ b/apps/server/src/services/ocr/tesseract_recognizer.spec.ts
@@@ -1,251 -1,0 +1,294 @@@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+
+// Mock Tesseract.js so no real OCR model is ever loaded.
+const mockWorker = {
+ recognize: vi.fn(),
+ terminate: vi.fn().mockResolvedValue(undefined)
+};
+
+const mockTesseract = {
+ createWorker: vi.fn()
+};
+
+vi.mock('tesseract.js', () => ({
+ default: mockTesseract
+}));
+
+// Avoid touching the real filesystem for the worker cache directory.
+vi.mock('fs', () => ({
+ default: {
+ mkdirSync: vi.fn()
+ }
+}));
+
+vi.mock('../data_dir.js', () => ({
+ default: {
+ OCR_CACHE_DIR: '/tmp/trilium-ocr-test-cache'
+ }
+}));
+
+const mockOptions = {
+ getOption: vi.fn()
+};
+
+const mockLog = {
+ info: vi.fn(),
+ error: vi.fn()
+};
+
+vi.mock('@triliumnext/core', async (importOriginal) => {
+ const actual = await importOriginal<typeof import('@triliumnext/core')>();
+ return {
+ ...actual,
+ options: mockOptions,
+ getLog: () => mockLog
+ };
+});
+
+// A fresh singleton is imported per test so worker state never leaks between them.
+let recognizer: typeof import('./tesseract_recognizer.js').default;
+
+beforeEach(async () => {
+ vi.resetModules();
+ vi.clearAllMocks();
+ mockOptions.getOption.mockReturnValue('0');
+ mockTesseract.createWorker.mockResolvedValue(mockWorker);
+ recognizer = (await import('./tesseract_recognizer.js')).default;
+});
+
+afterEach(() => {
+ vi.restoreAllMocks();
+});
+
+const image = Buffer.from('fake-image');
+
++/**
++ * A recognition result in the shape tesseract.js actually answers with when the `blocks` output
++ * format is requested: the words are nested under blocks → paragraphs → lines, and there is no
++ * top-level `words` array to read them from.
++ */
++function pageWithLines(confidence: number, lines: [ text: string, confidence: number ][][]) {
++ return {
++ text: lines.map(line => line.map(([ text ]) => text).join(' ')).join('\n'),
++ confidence,
++ blocks: [ {
++ paragraphs: [ {
++ lines: lines.map(line => ({
++ text: line.map(([ text ]) => text).join(' '),
++ words: line.map(([ text, wordConfidence ]) => ({ text, confidence: wordConfidence }))
++ }))
++ } ]
++ } ]
++ };
++}
++
++/** What a recognition with no readable text answers with: text, a score, and `blocks: null`. */
++function pageWithoutBlocks(text: string, confidence: number) {
++ return { text, confidence, blocks: null };
++}
++
+describe('TesseractRecognizer', () => {
+ it('recognizes text and reports overall confidence when no threshold is set', async () => {
+ mockWorker.recognize.mockResolvedValue({
- data: { text: ' hello world ', confidence: 88, words: [] }
++ data: pageWithoutBlocks(' hello world ', 88)
+ });
+
+ const result = await recognizer.recognize(image, 'eng');
+
+ expect(result.text).toBe('hello world');
+ expect(result.confidence).toBeCloseTo(0.88);
+ expect(mockTesseract.createWorker).toHaveBeenCalledWith(
+ 'eng',
+ 1,
+ expect.objectContaining({ cachePath: '/tmp/trilium-ocr-test-cache' })
+ );
+ });
+
++ it('asks tesseract for the per-word breakdown the confidence filter needs', async () => {
++ mockWorker.recognize.mockResolvedValue({ data: pageWithLines(80, [ [ [ 'x', 80 ] ] ]) });
++
++ await recognizer.recognize(image, 'eng');
++
++ // Without this output format the result carries the plain text alone, and every image is
++ // judged by its mean confidence instead of word by word.
++ expect(mockWorker.recognize).toHaveBeenCalledWith(image, {}, { blocks: true });
++ });
++
+ it('reuses the worker for the same language and recreates it when the language changes', async () => {
+ mockWorker.recognize.mockResolvedValue({
- data: { text: 'a', confidence: 50, words: [] }
++ data: pageWithLines(50, [ [ [ 'a', 50 ] ] ])
+ });
+
+ await recognizer.recognize(image, 'eng');
+ await recognizer.recognize(image, 'eng');
+ expect(mockTesseract.createWorker).toHaveBeenCalledTimes(1);
+ expect(mockWorker.terminate).not.toHaveBeenCalled();
+
+ await recognizer.recognize(image, 'deu');
+ expect(mockWorker.terminate).toHaveBeenCalledTimes(1);
+ expect(mockTesseract.createWorker).toHaveBeenCalledTimes(2);
+ });
+
+ it('serializes overlapping calls so a language switch cannot tear down an in-flight worker', async () => {
+ // Hold the first recognition open so the second call is issued while it runs.
+ let releaseFirst: (value: unknown) => void = () => {};
+ const firstRecognition = new Promise((resolve) => { releaseFirst = resolve; });
+ mockWorker.recognize
+ .mockReturnValueOnce(firstRecognition)
- .mockResolvedValue({ data: { text: 'second', confidence: 50, words: [] } });
++ .mockResolvedValue({ data: pageWithLines(50, [ [ [ 'second', 50 ] ] ]) });
+
+ const p1 = recognizer.recognize(image, 'eng');
+ const p2 = recognizer.recognize(image, 'deu');
+
+ // Let microtasks settle while the first job is still running.
+ await new Promise((resolve) => setTimeout(resolve, 0));
+
+ // The second (different-language) job must not have started: no second worker
+ // was created and the in-flight worker was not terminated mid-recognition.
+ expect(mockTesseract.createWorker).toHaveBeenCalledTimes(1);
+ expect(mockWorker.terminate).not.toHaveBeenCalled();
+
- releaseFirst({ data: { text: 'first', confidence: 50, words: [] } });
++ releaseFirst({ data: pageWithLines(50, [ [ [ 'first', 50 ] ] ]) });
+ const [r1, r2] = await Promise.all([p1, p2]);
+
+ expect(r1.text).toBe('first');
+ expect(r2.text).toBe('second');
+ // Only after the first job finished did the language switch tear down and rebuild.
+ expect(mockWorker.terminate).toHaveBeenCalledTimes(1);
+ expect(mockTesseract.createWorker).toHaveBeenCalledTimes(2);
+ });
+
+ it('invokes the recognizing-text logger callback', async () => {
+ mockWorker.recognize.mockResolvedValue({
- data: { text: 'a', confidence: 50, words: [] }
++ data: pageWithLines(50, [ [ [ 'a', 50 ] ] ])
+ });
+
+ await recognizer.recognize(image, 'eng');
+
+ const config = mockTesseract.createWorker.mock.calls[0][2];
+ config.logger({ status: 'recognizing text', progress: 0.5 });
+ config.logger({ status: 'loading', progress: 0.1 });
+
+ expect(mockLog.info).toHaveBeenCalledWith(
+ expect.stringContaining('OCR progress')
+ );
+ });
+
+ it('passes an errorHandler that logs worker errors instead of rethrowing them', async () => {
+ mockWorker.recognize.mockResolvedValue({
- data: { text: 'a', confidence: 50, words: [] }
++ data: pageWithLines(50, [ [ [ 'a', 50 ] ] ])
+ });
+
+ await recognizer.recognize(image, 'eng');
+
+ // Without an errorHandler, tesseract.js turns job failures into uncaught
+ // exceptions, which surface as Electron's "JavaScript error" dialog (#9754).
+ const config = mockTesseract.createWorker.mock.calls[0][2];
+ expect(() => config.errorHandler('Error attempting to read image.')).not.toThrow();
+ expect(mockLog.error).toHaveBeenCalledWith(
+ 'Tesseract worker error: Error attempting to read image.'
+ );
+ });
+
+ it('propagates recognition errors to the caller', async () => {
+ mockWorker.recognize.mockRejectedValue(new Error('recognize failed'));
+
+ await expect(recognizer.recognize(image, 'eng')).rejects.toThrow('recognize failed');
+ });
+
+ describe('confidence filtering', () => {
- it('keeps only words above the configured threshold', async () => {
++ it('keeps the words above the threshold on the lines they were read on', async () => {
+ mockOptions.getOption.mockReturnValue('0.8');
+ mockWorker.recognize.mockResolvedValue({
- data: {
- text: 'good bad good',
- confidence: 70,
- words: [
- { text: 'good', confidence: 90 },
- { text: 'bad', confidence: 50 },
- { text: 'good', confidence: 95 }
- ]
- }
++ data: pageWithLines(70, [
++ [ [ 'good', 90 ], [ 'bad', 50 ] ],
++ [ [ 'also', 95 ], [ 'good', 85 ] ]
++ ])
+ });
+
+ const result = await recognizer.recognize(image, 'eng');
+
- expect(result.text).toBe('good good');
- expect(result.confidence).toBeCloseTo((0.9 + 0.95) / 2);
++ expect(result.text).toBe('good\nalso good');
++ expect(result.confidence).toBeCloseTo((0.9 + 0.95 + 0.85) / 3);
+ });
+
- it('returns empty confidence when no words pass the threshold', async () => {
- mockOptions.getOption.mockReturnValue('0.99');
++ it('keeps well-read words on an image whose mean confidence is below the threshold', async () => {
++ // The mean is dragged under the line by the marks Tesseract misread as text, which is
++ // exactly what the per-word filter is for — judging the image by that mean instead
++ // would throw away every word on it.
++ mockOptions.getOption.mockReturnValue('0.75');
+ mockWorker.recognize.mockResolvedValue({
- data: {
- text: 'low',
- confidence: 10,
- words: [{ text: 'low', confidence: 10 }]
- }
++ data: pageWithLines(60, [ [ [ 'Invoice', 96 ], [ '®', 0 ], [ 'total', 94 ] ] ])
+ });
+
+ const result = await recognizer.recognize(image, 'eng');
+
- expect(result.text).toBe('');
- expect(result.confidence).toBe(0);
++ expect(result.text).toBe('Invoice total');
+ });
+
- it('handles an empty word array with a threshold set', async () => {
- mockOptions.getOption.mockReturnValue('0.5');
++ it('leaves no blank line where every word of one was dropped', async () => {
++ mockOptions.getOption.mockReturnValue('0.8');
++ mockWorker.recognize.mockResolvedValue({
++ data: pageWithLines(70, [
++ [ [ 'kept', 90 ] ],
++ [ [ 'noise', 10 ] ],
++ [ [ 'also-kept', 90 ] ]
++ ])
++ });
++
++ const result = await recognizer.recognize(image, 'eng');
++
++ expect(result.text).toBe('kept\nalso-kept');
++ });
++
++ it('returns empty text and no confidence when no word passes the threshold', async () => {
++ mockOptions.getOption.mockReturnValue('0.99');
+ mockWorker.recognize.mockResolvedValue({
- data: { text: 'ignored', confidence: 80, words: [] }
++ data: pageWithLines(10, [ [ [ 'low', 10 ] ] ])
+ });
+
+ const result = await recognizer.recognize(image, 'eng');
+
+ expect(result.text).toBe('');
+ expect(result.confidence).toBe(0);
+ });
+
- it('falls back to overall confidence when there is no word-level data', async () => {
++ it('falls back to overall confidence when nothing was broken down into words', async () => {
+ mockOptions.getOption.mockReturnValue('0.5');
+ mockWorker.recognize.mockResolvedValue({
- data: { text: ' whole text ', confidence: 80, words: undefined }
++ data: pageWithoutBlocks(' whole text ', 80)
+ });
+
+ const result = await recognizer.recognize(image, 'eng');
+
+ expect(result.text).toBe('whole text');
+ expect(result.confidence).toBeCloseTo(0.8);
+ });
+
+ it('drops all text via the fallback when overall confidence is too low', async () => {
+ mockOptions.getOption.mockReturnValue('0.9');
+ mockWorker.recognize.mockResolvedValue({
- data: { text: 'whole text', confidence: 40, words: undefined }
++ data: pageWithoutBlocks('whole text', 40)
+ });
+
+ const result = await recognizer.recognize(image, 'eng');
+
+ expect(result.text).toBe('');
+ expect(result.confidence).toBeCloseTo(0.4);
+ expect(mockLog.info).toHaveBeenCalledWith(
+ expect.stringContaining('Entire text filtered out')
+ );
+ });
+
+ it('defaults the threshold to 0 when the option is null', async () => {
+ mockOptions.getOption.mockReturnValue(null);
+ mockWorker.recognize.mockResolvedValue({
- data: { text: 'kept', confidence: 30, words: [{ text: 'kept', confidence: 30 }] }
++ data: pageWithLines(30, [ [ [ 'kept', 30 ] ] ])
+ });
+
+ const result = await recognizer.recognize(image, 'eng');
+
+ expect(result.text).toBe('kept');
+ });
+ });
+});
diff --cc apps/server/src/services/ocr/tesseract_recognizer.ts
index 850e71d2a3,0000000000..145b93dc5b
mode 100644,000000..100644
--- a/apps/server/src/services/ocr/tesseract_recognizer.ts
+++ b/apps/server/src/services/ocr/tesseract_recognizer.ts
@@@ -1,161 -1,0 +1,186 @@@
+import { getLog, options } from '@triliumnext/core';
+import fs from 'fs';
+import Tesseract from 'tesseract.js';
+
+import dataDirs from '../data_dir.js';
+
+export interface RecognitionResult {
+ /** Recognized text after per-word confidence filtering. */
+ text: string;
+ /** Overall confidence in the range [0, 1] of the kept text. */
+ confidence: number;
+}
+
- /** The subset of Tesseract's page result this recognizer relies on. */
- interface RecognizedData {
- text: string;
- /** Overall page confidence, 0–100. */
- confidence: number;
- /** Per-word data; absent in some Tesseract output configurations. */
- words?: Array<{ text: string; confidence: number }>;
- }
-
+/**
+ * Owns a single long-lived Tesseract.js worker and the confidence filtering
+ * shared by every OCR path that recognizes raster images — standalone image
+ * notes/attachments and, page by page, scanned PDFs. Keeping one worker (rather
+ * than one per processor) avoids paying the model-load cost repeatedly and lets
+ * an image and a PDF page reuse the same loaded language data.
+ */
+class TesseractRecognizer {
+ private worker: Tesseract.Worker | null = null;
+ private currentLanguage: string | null = null;
+ // Serializes recognition jobs. The worker is a single shared resource and a
+ // language change tears it down and rebuilds it, so two overlapping calls (e.g.
+ // a manual reprocess arriving mid-batch) could otherwise terminate a worker that
+ // another call is mid-recognition with. Chaining also bounds OCR to one CPU-heavy
+ // job at a time.
+ private queue: Promise<unknown> = Promise.resolve();
+
+ /**
+ * Recognize text in an encoded image buffer (PNG/JPEG/etc.) for the given
+ * Tesseract language code(s), applying the configured confidence filtering.
+ * Calls are queued so only one recognition runs at a time.
+ */
+ recognize(image: Buffer, language: string): Promise<RecognitionResult> {
+ const result = this.queue.then(async () => {
+ const worker = await this.ensureWorker(language);
- const { data } = await worker.recognize(image);
++ // The per-word breakdown the confidence filter runs on is an opt-in output format:
++ // by default `recognize` answers with the plain text and nothing else, leaving the
++ // filter with only the whole image's mean confidence to judge by.
++ const { data } = await worker.recognize(image, {}, { blocks: true });
+ return this.filterTextByConfidence(data);
+ });
+ // Keep the chain going regardless of this job's outcome, without swallowing
+ // the result/error the caller receives.
+ this.queue = result.catch(() => {});
+ return result;
+ }
+
+ /**
+ * Ensures a Tesseract worker is ready for the given language.
+ * Creates a new worker if none exists or if the language has changed.
+ */
+ private async ensureWorker(language: string): Promise<Tesseract.Worker> {
+ if (this.worker && this.currentLanguage === language) {
+ return this.worker;
+ }
+
+ if (this.worker) {
+ await this.worker.terminate();
+ this.worker = null;
+ }
+
+ fs.mkdirSync(dataDirs.OCR_CACHE_DIR, { recursive: true });
+
+ getLog().info(`Initializing Tesseract worker for language(s): ${language}`);
+ const worker = await Tesseract.createWorker(language, 1, {
+ cachePath: dataDirs.OCR_CACHE_DIR,
+ // Without an errorHandler, tesseract.js rethrows job failures (e.g. undecodable
+ // images) from its worker message handler as uncaught exceptions — in the desktop
+ // app that surfaces as Electron's blocking "JavaScript error" dialog (#9754).
+ // The job promise still rejects, so callers handle the failure normally.
+ errorHandler: (error: unknown) => {
+ getLog().error(`Tesseract worker error: ${error}`);
+ },
+ logger: (m: { status: string; progress: number }) => {
+ if (m.status === 'recognizing text') {
+ getLog().info(`OCR progress (${language}): ${Math.round(m.progress * 100)}%`);
+ }
+ }
+ });
+ this.worker = worker;
+ this.currentLanguage = language;
+ return worker;
+ }
+
+ /**
- * Filter text based on minimum confidence threshold
++ * Filter text based on minimum confidence threshold.
++ *
++ * Word by word, because the words Tesseract is least sure of are usually not words at all —
++ * a border read as punctuation, a speck read as a comma — and it is those that drag the mean
++ * for the image down. Judging the image by that mean throws away every well-read word on it
++ * along with them, so what is kept is decided per word and reassembled line by line, leaving
++ * the text laid out as it is on the picture.
+ */
- private filterTextByConfidence(data: RecognizedData): RecognitionResult {
++ private filterTextByConfidence(data: Tesseract.Page): RecognitionResult {
+ const minConfidence = this.getMinConfidenceThreshold();
+
+ // If no minimum confidence set, return original text
+ if (minConfidence <= 0) {
+ return {
+ text: data.text.trim(),
+ confidence: data.confidence / 100
+ };
+ }
+
- const filteredWords: string[] = [];
- const validConfidences: number[] = [];
++ const lines = getRecognizedLines(data);
+
- // Tesseract provides word-level data
- if (data.words && Array.isArray(data.words)) {
- for (const word of data.words) {
- const wordConfidence = word.confidence / 100; // Convert to decimal
-
- if (wordConfidence >= minConfidence) {
- filteredWords.push(word.text);
- validConfidences.push(wordConfidence);
- }
- }
- } else {
- // Fallback: if word-level data not available, use overall confidence
++ // Nothing was broken down into words — an image nothing could be read from. There is only
++ // the one score to go by, so the text stands or falls as a whole.
++ if (lines.length === 0) {
+ const overallConfidence = data.confidence / 100;
+ if (overallConfidence >= minConfidence) {
+ return {
+ text: data.text.trim(),
+ confidence: overallConfidence
+ };
+ }
+ getLog().info(`Entire text filtered out due to low confidence ${overallConfidence} (below threshold ${minConfidence})`);
+ return {
+ text: '',
+ confidence: overallConfidence
+ };
+ }
+
++ const keptLines: string[] = [];
++ const keptConfidences: number[] = [];
++ let totalWords = 0;
++
++ for (const line of lines) {
++ const keptWords: string[] = [];
++
++ for (const word of line.words ?? []) {
++ totalWords++;
++ const wordConfidence = word.confidence / 100; // Convert to decimal
++
++ if (wordConfidence >= minConfidence) {
++ keptWords.push(word.text);
++ keptConfidences.push(wordConfidence);
++ }
++ }
++
++ // A line every word of which was dropped leaves no blank behind: the gap would read as
++ // spacing on the picture that isn't there.
++ if (keptWords.length > 0) {
++ keptLines.push(keptWords.join(' '));
++ }
++ }
++
+ // Calculate average confidence of accepted words
- const averageConfidence = validConfidences.length > 0
- ? validConfidences.reduce((sum, conf) => sum + conf, 0) / validConfidences.length
++ const averageConfidence = keptConfidences.length > 0
++ ? keptConfidences.reduce((sum, conf) => sum + conf, 0) / keptConfidences.length
+ : 0;
+
- const filteredText = filteredWords.join(' ').trim();
-
- getLog().info(`Filtered OCR text: ${filteredWords.length} words kept out of ${data.words?.length || 0} total words (min confidence: ${minConfidence})`);
++ getLog().info(`Filtered OCR text: ${keptConfidences.length} words kept out of ${totalWords} total words (min confidence: ${minConfidence})`);
+
+ return {
- text: filteredText,
++ text: keptLines.join('\n').trim(),
+ confidence: averageConfidence
+ };
+ }
+
+ /**
+ * Get minimum confidence threshold from options
+ */
+ private getMinConfidenceThreshold(): number {
+ const minConfidence = options.getOption('ocrMinConfidence') ?? 0;
+ return parseFloat(minConfidence);
+ }
+}
+
+export default new TesseractRecognizer();
++
++/**
++ * The words Tesseract read, in the lines it read them on.
++ *
++ * They sit three levels down the page it describes, and that description is only present when the
++ * `blocks` output format was asked for — see {@link TesseractRecognizer.recognize}. Without it, and
++ * for an image nothing was read from, this is empty.
++ */
++function getRecognizedLines(data: Tesseract.Page): Tesseract.Line[] {
++ return (data.blocks ?? []).flatMap(
++ block => (block.paragraphs ?? []).flatMap(paragraph => paragraph.lines ?? [])
++ );
++}
diff --cc docs/User Guide/User Guide/Advanced Usage/Text Extraction (OCR).md
index 5a18cfd006,b45f484795..14f0faac78
--- a/docs/User Guide/User Guide/Advanced Usage/Text Extraction (OCR).md
+++ b/docs/User Guide/User Guide/Advanced Usage/Text Extraction (OCR).md
@@@ -27,16 -27,10 +27,16 @@@ OCR in Trilium supports the following f
### PDFs
-Currently only text extraction is supported and not OCR.
+The PDF extraction supports:
-* This means that the PDF needs to have proper text information in it (i.e. the text can be selected in a PDF viewer), whereas scanned documents are not yet supported.
-* There are plans to integrate the same OCR-based recognition for PDFs used for images, but this is not yet implemented.
+* PDFs with text information (e.g. a PDF created by a browser or a word processing application), in which the text is extracted.
+* Scanned PDFs, in which the text is extracted through an OCR process.
+* Mixed PDFs, containing both text and images with text content.
+
+Note that OCR extraction is limited to 50 pages per PDF, pages containing text information (not as an image) are not subject to this limitation.
+
+> [!NOTE]
- > Text extraction for scanned PDFs using OCR was introduced in v0.104.0. If a scanned PDF was already processed by an earlier version, reprocess it individually via _Process OCR_; batch processing only handles files that haven't been OCR'd yet.
++> Text extraction for scanned PDFs using OCR was introduced in v0.106.0. If a scanned PDF was already processed by an earlier version, reprocess it individually via _Process OCR_; batch processing only handles files that haven't been OCR'd yet.
### Office documents