Skip to content

Build your personal knowledge base with Trilium Notes

feat(llm): let a model read web pages when it searches through a search provider

Elian Doran7c037b22fb40
diff --git a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/AI.html b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/AI.html
index c158e4c028..2e61bf6978 100644
--- a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/AI.html
+++ b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/AI.html
@@ -136,6 +136,11 @@
<p>In the chat, a search run through a search provider shows that provider's
logo at the start of its line instead of the <span class="tn-icon bx bx-search"></span> magnifier;
hover the logo to see the provider's name.</p>
+<p>With a search provider, the model can also read the pages it finds, or
+ a link you give it, as text: Trilium fetches the page itself and hands
+ the model up to its first 20,000 characters. Addresses on your own machine
+ or network are refused. In the browser version most sites don't allow being
+ read this way, so reading a page there usually fails.</p>
<h3>Thinking</h3>
<p>Some models reason before they answer. While the model is thinking, its
reasoning is shown in full under a spinner; once it is done, it folds into
diff --git a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/AI/Privacy.html b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/AI/Privacy.html
index 7c66b01c0b..63f425fc4f 100644
--- a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/AI/Privacy.html
+++ b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/AI/Privacy.html
@@ -78,6 +78,10 @@
search terms reach the provider's own search infrastructure, or the search
provider chosen in the <em>Tools</em> menu (Brave Search, Tavily, Exa or
your SearXNG instance).</li>
+ <li><strong>The pages the model reads</strong>, when web search goes through
+ a search provider: Trilium fetches each page the model asks for from the
+ site itself, so the site sees a request from the machine running Trilium.
+ The page's text is then sent to the model's provider.</li>
</ul>
<h2>The in-editor AI assistant</h2>
<p>A run of the&nbsp;<a class="reference-link" href="#root/_help_VRoP0g1uHHc2">In-editor AI assistant</a>&nbsp;sends
diff --git a/docs/User Guide/User Guide/AI.md b/docs/User Guide/User Guide/AI.md
index a69c4a7b52..2b10fbedf3 100644
--- a/docs/User Guide/User Guide/AI.md
+++ b/docs/User Guide/User Guide/AI.md
@@ -95,6 +95,8 @@ Whether web search is on is kept per chat; the search source is shared by all ch
In the chat, a search run through a search provider shows that provider's logo at the start of its line instead of the <span class="tn-icon bx bx-search"></span> magnifier; hover the logo to see the provider's name.
+With a search provider, the model can also read the pages it finds, or a link you give it, as text: Trilium fetches the page itself and hands the model up to its first 20,000 characters. Addresses on your own machine or network are refused. In the browser version most sites don't allow being read this way, so reading a page there usually fails.
+
### Thinking
Some models reason before they answer. While the model is thinking, its reasoning is shown in full under a spinner; once it is done, it folds into a collapsible _Thought process_ line above the reply.
diff --git a/docs/User Guide/User Guide/AI/Privacy.md b/docs/User Guide/User Guide/AI/Privacy.md
index e24565792b..38f007fa0d 100644
--- a/docs/User Guide/User Guide/AI/Privacy.md
+++ b/docs/User Guide/User Guide/AI/Privacy.md
@@ -31,6 +31,7 @@ This is the reason the provider list is grouped this way, and why Trilium pre-se
* It decides what to fetch; there is no per-note permission system, so the reachable scope is your whole tree.
* **Attachments and mentions**, when you add them: images, PDFs and text files are sent to the provider, and an `@` mention makes that note fetchable.
* **Your search query**, when web search is enabled, the model's search terms reach the provider's own search infrastructure, or the search provider chosen in the _Tools_ menu (Brave Search, Tavily, Exa or your SearXNG instance).
+* **The pages the model reads**, when web search goes through a search provider: Trilium fetches each page the model asks for from the site itself, so the site sees a request from the machine running Trilium. The page's text is then sent to the model's provider.
## The in-editor AI assistant
diff --git a/packages/trilium-core/src/services/llm/providers/base_provider.ts b/packages/trilium-core/src/services/llm/providers/base_provider.ts
index 52edad6059..88aff485dd 100644
--- a/packages/trilium-core/src/services/llm/providers/base_provider.ts
+++ b/packages/trilium-core/src/services/llm/providers/base_provider.ts
@@ -13,7 +13,7 @@ import { buildNoteHint } from "../note_hint.js";
import { buildSystemPrompt as composeSystemPrompt } from "../system_prompt.js";
import { allToolRegistries } from "../tools/index.js";
import type { LlmProvider, LlmProviderConfig, ModelInfo, ModelPricing, StreamResult } from "../types.js";
-import { createWebSearchTool } from "../web_search.js";
+import { createReadWebPageTool, createWebSearchTool } from "../web_search.js";
import MODEL_PRICES_JSON from "./model_prices.json" with { type: "json" };
const DEFAULT_MAX_TOKENS = 8096;
@@ -409,6 +409,7 @@ export abstract class BaseProvider implements LlmProvider {
if (config.enableWebSearch) {
if (config.webSearch) {
tools.web_search = createWebSearchTool(config.webSearch);
+ tools.read_web_page = createReadWebPageTool();
} else {
this.addWebSearchTool(tools);
}
diff --git a/packages/trilium-core/src/services/llm/providers/google.spec.ts b/packages/trilium-core/src/services/llm/providers/google.spec.ts
index 240d875f32..59cb0d3d08 100644
--- a/packages/trilium-core/src/services/llm/providers/google.spec.ts
+++ b/packages/trilium-core/src/services/llm/providers/google.spec.ts
@@ -143,6 +143,8 @@ describe("GoogleProvider tool handling", () => {
expect(googleSearchMock).toHaveBeenCalledOnce();
const opts = streamTextMock.mock.calls[0][0] as any;
expect(opts.tools.google_search).toEqual({ kind: "google_search" });
+ // The built-in search reads pages its own way.
+ expect(opts.tools.read_web_page).toBeUndefined();
expect(opts.toolChoice).toBe("auto");
});
@@ -174,6 +176,7 @@ describe("GoogleProvider tool handling", () => {
const opts = streamTextMock.mock.calls[0][0] as any;
expect(opts.tools.google_search).toBeUndefined();
expect(opts.tools.web_search).toBeDefined();
+ expect(opts.tools.read_web_page).toBeDefined();
expect(opts.tools.search_notes).toBeDefined();
expect(opts.system).not.toContain("web search is unavailable in this turn");
});
diff --git a/packages/trilium-core/src/services/llm/web_search.spec.ts b/packages/trilium-core/src/services/llm/web_search.spec.ts
index 28beee4aca..8e099fb2a2 100644
--- a/packages/trilium-core/src/services/llm/web_search.spec.ts
+++ b/packages/trilium-core/src/services/llm/web_search.spec.ts
@@ -3,7 +3,13 @@ import { afterEach, describe, expect, it, vi } from "vitest";
const fetchMock = vi.hoisted(() => vi.fn());
vi.mock("./providers/fetch.js", () => ({ llmFetch: fetchMock }));
-import { createWebSearchTool, searchWeb } from "./web_search.js";
+const fetchResourceMock = vi.hoisted(() => vi.fn());
+vi.mock("../request.js", async (importOriginal) => {
+ const actual = await importOriginal<typeof import("../request.js")>();
+ return { ...actual, default: { ...actual.default, fetchResource: fetchResourceMock } };
+});
+
+import { createReadWebPageTool, createWebSearchTool, readWebPage, searchWeb } from "./web_search.js";
function respond(body: unknown, status = 200) {
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify(body), { status }));
@@ -20,6 +26,7 @@ const PAGE = { title: "Trilium", url: "https://triliumnotes.org" };
afterEach(() => {
fetchMock.mockReset();
+ fetchResourceMock.mockReset();
});
describe("searchWeb", () => {
@@ -94,3 +101,48 @@ describe("createWebSearchTool", () => {
expect(await run(tool, "trilium")).toEqual({ error: expect.stringContaining("HTTP 403") });
});
});
+
+describe("readWebPage", () => {
+ function serve(body: string, contentType = "text/html", status = 200) {
+ fetchResourceMock.mockResolvedValueOnce({ status, ok: status < 400, contentType, bytes: new TextEncoder().encode(body) });
+ }
+
+ it("reads a page's main content as Markdown, headed by its title, without its furniture", async () => {
+ serve(`<html><head><title>Trilium</title><script>alert(1)</script></head><body>
+ <nav><a href="/">Home</a></nav>
+ <main><h2>Notes</h2><p>A <a href="https://triliumnotes.org/docs">hierarchical</a> note app.</p><style>p{}</style></main>
+ <footer>Copyright</footer>
+ </body></html>`);
+ const page = await readWebPage("https://triliumnotes.org");
+
+ // Fetched under the policy for addresses from content, with a ceiling on the body.
+ expect(fetchResourceMock).toHaveBeenCalledExactlyOnceWith("https://triliumnotes.org/", expect.objectContaining({ maxBytes: 5 * 1024 * 1024 }));
+ expect(page).toBe("# Trilium\n\n## Notes\n\nA [hierarchical](https://triliumnotes.org/docs) note app.");
+ });
+
+ it("passes plain text through, and cuts a long page off with a note saying so", async () => {
+ serve(" just text ", "text/plain");
+ expect(await readWebPage("https://a.example/t.txt")).toBe("just text");
+
+ serve("x".repeat(25_000), "text/plain");
+ const long = await readWebPage("https://a.example/long.txt");
+ expect(long.startsWith("x".repeat(20_000) + "\n\n[The page continues")).toBe(true);
+ });
+
+ it("refuses what isn't a web page: another scheme, an error status, a binary type", async () => {
+ await expect(readWebPage("file:///etc/passwd")).rejects.toThrow();
+ expect(fetchResourceMock).not.toHaveBeenCalled();
+
+ serve("Not found", "text/html", 404);
+ await expect(readWebPage("https://a.example/missing")).rejects.toThrow("HTTP 404");
+ serve("%PDF", "application/pdf");
+ await expect(readWebPage("https://a.example/doc.pdf")).rejects.toThrow("application/pdf");
+ });
+
+ it("hands the model a refused address as an error to report", async () => {
+ fetchResourceMock.mockRejectedValueOnce(new Error("URLs pointing to private/internal networks are not allowed"));
+ const tool = createReadWebPageTool();
+ expect(await tool.execute?.({ url: "http://169.254.169.254/" }, { toolCallId: "1", messages: [], context: {} }))
+ .toEqual({ error: "URLs pointing to private/internal networks are not allowed" });
+ });
+});
diff --git a/packages/trilium-core/src/services/llm/web_search.ts b/packages/trilium-core/src/services/llm/web_search.ts
index d74be07c4b..f3f5da0bf0 100644
--- a/packages/trilium-core/src/services/llm/web_search.ts
+++ b/packages/trilium-core/src/services/llm/web_search.ts
@@ -1,12 +1,16 @@
/**
* Web search through a search provider the user configured (`kind: "search"` in the
* `llmProviders` option), offered to the model as a `web_search` tool in place of the model's
- * built-in search.
+ * built-in search, with a `read_web_page` tool to read what it finds.
*/
import { tool } from "ai";
+import { parse as parseHtml } from "node-html-parser";
import { z } from "zod";
+import markdownExport from "../export/markdown.js";
+import request, { validateFetchableUrl } from "../request.js";
+import { decodeUtf8 } from "../utils/binary.js";
import { llmFetch } from "./providers/fetch.js";
/** A configured search provider, as stored in the `llmProviders` option. */
@@ -99,6 +103,74 @@ export async function searchWeb(setup: WebSearchSetup, query: string): Promise<W
}
}
+const MAX_PAGE_BYTES = 5 * 1024 * 1024;
+const MAX_PAGE_CHARS = 20_000;
+const HTML_TYPES = new Set([ "text/html", "application/xhtml+xml" ]);
+const TEXT_TYPES = new Set([ "text/plain", "text/markdown" ]);
+/** Elements that are page furniture rather than content. */
+const NON_CONTENT = "script, style, noscript, template, svg, iframe, nav, header, footer, aside, form";
+
+/**
+ * The `read_web_page` tool. Its result is the page as Markdown, which the chat previews under the
+ * tool's line; a failure is returned as `{ error }` so the model can tell the user about it.
+ */
+export function createReadWebPageTool() {
+ return tool({
+ description: "Read a web page, such as a web search result or a link the user gave, as Markdown. Long pages are cut off.",
+ inputSchema: z.object({
+ url: z.string().describe("The http(s) URL of the page")
+ }),
+ execute: async ({ url }) => {
+ try {
+ return await readWebPage(url);
+ } catch (e) {
+ return { error: e instanceof Error ? e.message : String(e) };
+ }
+ }
+ });
+}
+
+/**
+ * The page at `url` as Markdown, cut off at {@link MAX_PAGE_CHARS}.
+ *
+ * The URL is the model's to choose, and a note or a search result can steer it, so the page is
+ * fetched with `fetchResource()`: the policy for addresses that arrive in content, which refuses
+ * private networks, vets each redirect and caps the body.
+ */
+export async function readWebPage(url: string): Promise<string> {
+ const response = await request.fetchResource(validateFetchableUrl(url).toString(), {
+ maxBytes: MAX_PAGE_BYTES,
+ headers: { "Accept": "text/html, application/xhtml+xml, text/plain;q=0.9" }
+ });
+ if (!response.ok) {
+ throw new Error(`The page answered HTTP ${response.status}.`);
+ }
+ const contentType = response.contentType || "text/html";
+ let content: string;
+ if (HTML_TYPES.has(contentType)) {
+ content = htmlToMarkdown(decodeUtf8(response.bytes));
+ } else if (TEXT_TYPES.has(contentType)) {
+ content = decodeUtf8(response.bytes).trim();
+ } else {
+ throw new Error(`The page is ${contentType}, which can't be read as text.`);
+ }
+ return content.length > MAX_PAGE_CHARS
+ ? `${content.slice(0, MAX_PAGE_CHARS)}\n\n[The page continues; only its first ${MAX_PAGE_CHARS} characters are shown.]`
+ : content;
+}
+
+/** The content of a page as Markdown, headed by its title: its `main` or `article` where it has one. */
+function htmlToMarkdown(html: string): string {
+ const root = parseHtml(html);
+ const title = root.querySelector("title")?.textContent.trim();
+ for (const element of root.querySelectorAll(NON_CONTENT)) {
+ element.remove();
+ }
+ const content = root.querySelector("main, article, [role=main]") ?? root.querySelector("body") ?? root;
+ const markdown = markdownExport.toMarkdown(content.innerHTML, { headerlessTables: "emptyHeader" }).trim();
+ return title ? `# ${title}\n\n${markdown}` : markdown;
+}
+
async function requestJson(url: string, init: RequestInit): Promise<unknown> {
const response = await llmFetch(url, { ...init, signal: AbortSignal.timeout(SEARCH_TIMEOUT_MS) });
if (!response.ok) {