Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
feat(clipper): offer the whole page beside the article (closes #9611)
Elian Doran9e35303f642f
diff --git a/.claude/skills/developing-web-clipper/SKILL.md b/.claude/skills/developing-web-clipper/SKILL.mdindex c90bfb6966..2b36acef5f 100644--- a/.claude/skills/developing-web-clipper/SKILL.md+++ b/.claude/skills/developing-web-clipper/SKILL.md@@ -18,7 +18,7 @@ repository.| `entrypoints/background/index.ts` | service worker (MV3) / background page (MV2) | context menus, keyboard commands, every save, image download, toasts || `entrypoints/background/trilium_server_facade.ts` | background | finds Trilium, authenticates, `callService()` || `entrypoints/content/index.ts` | every page (`<all_urls>`) | selection, page description, crop overlay, toast UI |-| `entrypoints/readable-page/index.ts` | the tab, injected on demand | Defuddle extraction of the whole page; its `main()` value is the `executeScript()` result |+| `entrypoints/readable-page/index.ts` | the tab, injected on demand | Defuddle extraction of the page, as an article and as a whole; its `main()` value is the `executeScript()` result || `entrypoints/popup/main.tsx` | the toolbar popup | Preact `Popup`: capture buttons, link-with-note form, connection status || `entrypoints/options/main.tsx` | the options page | Preact `Options`: desktop port, server login || `entrypoints/offscreen/` | MV3 only (Chrome) | crops screenshots on a canvas, since an MV3 service worker has no DOM |@@ -38,11 +38,11 @@ Everything goes through `browser.runtime.sendMessage` / `browser.tabs.sendMessag(`message`, `noteId`, `tabIds`).- **background → offscreen:** `{ type: "CROP_IMAGE" }`, the one message keyed by `type`.-The readable page is not a message: the popup and the background call `extractReadablePage(tabId)`+The readable page is not a message: the popup and the background call `extractPage(tabId)`(`utils.ts`), which injects `/readable-page.js` with `browser.scripting.executeScript()` (the`scripting` permission, in both builds), so Defuddle stays out of the content script that loadson every page. Specs mock it with `Object.assign(fakeBrowser, { scripting: { executeScript } })`,-answering `[ { result: page } ]`.+answering `[ { result: { article, wholePage } } ]` (`ExtractedPage`).Every user action in the background runs inside `showFailures()`, which turns an exception into atoast. A new action belongs inside it too, or it fails silently. A toast cannot show on browser@@ -120,9 +120,16 @@ against older versions of the other.** Prefer changes that need nothing new frommade with `cloneNode()` does not have. `url` makes links and images absolute.- Its HTML is parsed with `DOMParser`, a document that loads no image and runs no code, before`getImages()` swaps the sources for placeholders.+ - `readPage()` parses twice: the article, and the whole page (`WHOLE_PAGE_OPTIONS`:+ `contentSelector: "body"` with scoring and the clutter selectors off, hidden-element removal+ and standardization kept). Defuddle drops a grid of cards with little text under each heading+ as "related articles" (`isCardGrid()`) and any `.footer`, which on a gallery or test page is the+ content (#9611). The popup offers both (`Article` / `Whole page`); the shortcut and the menu+ save the article, or the whole page when there is none.- A page without an article comes back as `<body></body>`, which `readPage()` turns into- `undefined` (the popup's bookmark mode). Defuddle finds an article on more pages than- Readability did, home pages included.+ `article: undefined`; only a page whose whole page is empty too returns `undefined` (the+ popup's bookmark mode). Defuddle's own retries relax its rules on a very short page ("possible+ index page"), so a spec fixture of a card grid needs a paragraph of text to stay above that.- Use `parse()`, never `parseAsync()`: the async extractors fetch third-party APIs.- **happy-dom drops lazy images** (`data-src` behind a `data:image/svg+xml` placeholder) thatChromium and linkedom promote to `src`; test such Defuddle behavior in a real browser, notdiff --git a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Installation & Setup/Web Clipper.html b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Installation & Setup/Web Clipper.htmlindex 7ff90f8bbc..7299973ef3 100644--- a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Installation & Setup/Web Clipper.html+++ b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Installation & Setup/Web Clipper.html@@ -38,16 +38,22 @@<ul><li><em>Selection</em>, when text is selected on the page: <em>Save selection</em> savesit into the note of that page's clippings, as the context menu does.</li>- <li><em>Page</em>, when the page has an article: <em>Save page to Trilium</em> saves- the article's text and images. The author the page names shows under the- description, marked by the <span class="tn-icon bx bx-user"></span> icon.- It can be edited or cleared before saving, or written in when the page- names none, and is saved in the <code>#pageAuthor</code> label of the note.- When the page states when it was published, that date shows under the author,- marked by the <span class="tn-icon bx bx-calendar"></span> icon, and is saved- in the <code>#publishedDate</code> label of the note. When it states when- it was last modified, that date shows below, marked by the <span class="tn-icon bx bx-history"></span> icon,- and is saved in the <code>#modifiedDate</code> label.</li>+ <li><em>Page</em>: <em>Save page to Trilium</em> saves the page's text and images.+ A smaller switch at the top of the preview chooses how much of the page+ to keep: <em>Article</em> keeps only the page's main text, without its menus,+ sidebars and footers, and <em>Whole page</em> keeps everything the page shows. <em>Whole page</em> is+ for pages that are not an article (such as a gallery or a page of cards),+ where <em>Article</em> can leave out part of the content; the preview shows+ what each one keeps. <em>Article</em> is unavailable on a page where Trilium+ finds no article. The author the page names shows under the description,+ marked by the <span class="tn-icon bx bx-user"></span> icon. It can be edited+ or cleared before saving, or written in when the page names none, and is+ saved in the <code>#pageAuthor</code> label of the note. When the page states+ when it was published, that date shows under the author, marked by the <span+ class="tn-icon bx bx-calendar"></span> icon, and is saved in the <code>#publishedDate</code> label+ of the note. When it states when it was last modified, that date shows+ below, marked by the <span class="tn-icon bx bx-history"></span> icon, and+ is saved in the <code>#modifiedDate</code> label.</li><li><em>Bookmark</em>, always: a text box for a note about the page replacesthe preview, and <em>Save bookmark</em> (or <kbd>Ctrl</kbd>+<kbd>Enter</kbd>)saves a link to the page with that note. Left empty, the title stays the@@ -60,10 +66,12 @@of the note. Clippings saved with the keyboard shortcuts or the contextmenu, screenshots and images included, get the description as the pagegives it. Each clipping that brings a description updates the label.</p>- <p>The popup opens on the selection if there is one, otherwise on the article.- On a page with no article and no selection, it opens on <em>Bookmark</em> and- says the page has no article. The keyboard shortcuts and the context menu- still save right away, without a preview.</p>+ <p>The popup opens on the selection if there is one, otherwise on <em>Page</em>,+ showing the article, or the whole page when the page has no article. On+ a page that shows nothing to save, it opens on <em>Bookmark</em> and says+ so. The keyboard shortcuts and the context menu still save right away,+ without a preview: saving the whole page that way saves the article, or+ the whole page when the page has no article.</p></li><li><p>Pages the extension cannot access (the browser's own pages, the extensiondiff --git a/apps/web-clipper/entrypoints/background/index.spec.ts b/apps/web-clipper/entrypoints/background/index.spec.tsindex d6b934e417..b069ab1576 100644--- a/apps/web-clipper/entrypoints/background/index.spec.ts+++ b/apps/web-clipper/entrypoints/background/index.spec.ts@@ -56,9 +56,9 @@ let tabs: Tab[];let tabMessageHandler: (message: { name: string }) => unknown;const tabsSendMessage = vi.fn((_tabId: number, message: { name: string }) => Promise.resolve(tabMessageHandler(message)));-/** What `readable-page.js` returns in the active tab; no article unless a test says so. */-let readablePage: unknown;-const executeScript = vi.fn(async () => [ { result: readablePage } ]);+/** What `readable-page.js` returns in the active tab; nothing to save unless a test says so. */+let extractedPage: unknown;+const executeScript = vi.fn(async () => [ { result: extractedPage } ]);const contextMenusCreate = vi.fn();const contextMenusRemoveAll = vi.fn(async () => {});const getContexts = vi.fn(async (): Promise<unknown[]> => []);@@ -76,7 +76,7 @@ describe("background", () => {fakeBrowser.reset();tabs = [ ACTIVE_TAB ];tabMessageHandler = () => undefined;- readablePage = undefined;+ extractedPage = undefined;facade.callService.mockResolvedValue({ noteId: "saved" });vi.stubGlobal("fetch", fetchMock);vi.spyOn(console, "log").mockImplementation(() => {});@@ -281,9 +281,9 @@ describe("background", () => {});describe("saving the whole page", () => {- it("saves the readable page from the shortcut, the menu and the popup", async () => {+ it("saves the article from the shortcut, the menu and the popup", async () => {const page = { title: "Article", content: "<p>Body</p>", images: [], clipType: "page" };- readablePage = page;+ extractedPage = { article: page, wholePage: { ...page, content: "<nav>Menu</nav><p>Body</p>" } };await onCommand("saveWholePage");await onContextMenuClicked({ menuItemId: "trilium-save-page" });@@ -301,14 +301,24 @@ describe("background", () => {expect(await lastToast()).toMatchObject(FAILURE_TOAST);});- it("saves nothing from a page without an article or out of reach", async () => {+ it("saves the whole page from the shortcut when the page has no article", async () => {+ const wholePage = { title: "Gallery", content: "<h2>Card</h2>", images: [], clipType: "page" };+ extractedPage = { wholePage };++ await onCommand("saveWholePage");++ expect(facade.callService).toHaveBeenCalledWith("POST", "notes", wholePage);+ expect(await lastToast()).toMatchObject({ message: "Page has been saved to Trilium.", noteId: "saved" });+ });++ it("saves nothing from a page that shows nothing or is out of reach", async () => {await onCommand("saveWholePage");expect(await lastToast()).toMatchObject({...GENERIC_FAILURE_TOAST,- message: "Saving to Trilium failed: Trilium could not find an article on this page."+ message: "Saving to Trilium failed: Trilium found nothing to save on this page."});- readablePage = { title: "Article", content: "<p>Body</p>", images: [] };+ extractedPage = { wholePage: { title: "Article", content: "<p>Body</p>", images: [] } };tabs = [ { title: "Tab without id" } ];await onCommand("saveWholePage");diff --git a/apps/web-clipper/entrypoints/background/index.ts b/apps/web-clipper/entrypoints/background/index.tsindex 666232d36c..29bc141612 100644--- a/apps/web-clipper/entrypoints/background/index.ts+++ b/apps/web-clipper/entrypoints/background/index.ts@@ -1,4 +1,4 @@-import { extractReadablePage, randomString, type Rect } from "@/utils";+import { extractPage, randomString, type Rect } from "@/utils";import setupContextMenu from "./context_menu";import TriliumServerFacade, { TriliumError } from "./trilium_server_facade";@@ -331,7 +331,10 @@ export default defineBackground(() => {toast(savedMessage("Image", failedImages), resp.noteId);}- /** Saves the readable page, or the page the popup already extracted and the user edited. */+ /**+ * Saves the page the popup already extracted and the user edited, or else the article, or the+ * whole page when it has none.+ */async function saveWholePage(page?: object) {const payload = page ?? await readActivePage();@@ -344,11 +347,11 @@ export default defineBackground(() => {async function readActivePage() {const activeTab = await getActiveTab();- const page = activeTab.id === undefined ? undefined : await extractReadablePage(activeTab.id);+ const page = activeTab.id === undefined ? undefined : await extractPage(activeTab.id);if (!page) {- throw new Error("Trilium could not find an article on this page.");+ throw new Error("Trilium found nothing to save on this page.");}- return page;+ return page.article ?? page.wholePage;}async function saveLinkWithNote(title: string, content: string, description?: string) {diff --git a/apps/web-clipper/entrypoints/popup/main.tsx b/apps/web-clipper/entrypoints/popup/main.tsxindex ebdbb6a4b3..909a91cc9c 100644--- a/apps/web-clipper/entrypoints/popup/main.tsx+++ b/apps/web-clipper/entrypoints/popup/main.tsx@@ -24,7 +24,7 @@ import type { ComponentChildren } from "preact";import { Fragment, render } from "preact";import { useEffect, useRef, useState } from "preact/hooks";-import { extractReadablePage, type ReadablePage } from "@/utils";+import { type ExtractedPage, extractPage, type ReadablePage } from "@/utils";import type { TriliumSearchNoteStatus, TriliumSearchStatus } from "../background/trilium_server_facade";@@ -52,8 +52,8 @@ interface SelectedContent {labels?: Record<string, string>;}-/** The current page: its readable version, or why there is none. */-type PageState = ReadablePage | "unreadable" | "inaccessible";+/** The current page: its article and its whole page, or why there is nothing to save. */+type PageState = ExtractedPage | "unreadable" | "inaccessible";/** What the active tab offers to save. */interface TabContent {@@ -68,14 +68,18 @@ interface TabContent {}/**- * What the card saves: the selection, the readable page (`clipType: "page"`), or a note about the- * page with a link to it (`clipType: "note"`), which the popup calls a bookmark.+ * What the card saves: the selection, the page (`clipType: "page"`), or a note about the page with a+ * link to it (`clipType: "note"`), which the popup calls a bookmark.*/type ClipMode = "selection" | "page" | "note";+/** Which reading of the page the `page` mode saves. */+type PageVariant = keyof ExtractedPage;+/** What the user has chosen and typed in the card, kept by `Popup` so a reconnection does not lose it. */interface Draft {mode?: ClipMode;+ variant?: PageVariant;/** `undefined` until the user edits the title. */title?: string;/** `undefined` until the user edits the page's description. */@@ -124,6 +128,8 @@ const PREVIEW_STYLE = `const MODE_LABELS: Record<ClipMode, string> = { selection: "Selection", page: "Page", note: "Bookmark" };+const VARIANT_LABELS: Record<PageVariant, string> = { article: "Article", wholePage: "Whole page" };+/** The save button of each mode, with the command whose shortcut it shows, or its own shortcut. */const SAVE_BUTTONS: Record<ClipMode, { icon: IconDefinition, label: string, command: string, shortcut?: string }> = {selection: { icon: Highlight, label: "Save selection", command: "saveSelection" },@@ -339,18 +345,20 @@ function PagePreview({ tab, disabled, shortcuts, draft, onDraftChange }: {draft: Draft;onDraftChange: (draft: Draft) => void;}) {- const { mode: chosenMode, title, description, author, text } = draft;+ const { mode: chosenMode, variant: chosenVariant, title, description, author, text } = draft;const textAreaRef = useRef<HTMLTextAreaElement>(null);- const article = typeof tab?.page === "object" ? tab.page : undefined;+ const page = typeof tab?.page === "object" ? tab.page : undefined;+ const variant = chosenVariant && page?.[chosenVariant] ? chosenVariant : page?.article ? "article" : "wholePage";+ const pageClip = page?.[variant];const selection = tab?.selection ?? undefined;- const modes = tab ? availableModes(article, selection) : [];+ const modes = tab ? availableModes(pageClip, selection) : [];const mode = chosenMode && modes.includes(chosenMode) ? chosenMode : modes[0];- const clip = mode === "selection" ? selection : mode === "page" ? article : undefined;- const shownTitle = title ?? (article ?? selection)?.title ?? tab?.title ?? "";+ const clip = mode === "selection" ? selection : mode === "page" ? pageClip : undefined;+ const shownTitle = title ?? (pageClip ?? selection)?.title ?? tab?.title ?? "";const shownDescription = description ?? tab?.description ?? "";const savedDescription = shownDescription.replace(/\s+/g, " ").trim();- const shownAuthor = author ?? article?.labels.pageAuthor ?? "";+ const shownAuthor = author ?? pageClip?.labels.pageAuthor ?? "";const savedAuthor = shownAuthor.replace(/\s+/g, " ").trim();useEffect(() => {@@ -396,7 +404,7 @@ function PagePreview({ tab, disabled, shortcuts, draft, onDraftChange }: {}const pageUrl = mode === "note" ? tab?.url : clip?.pageUrl;- const pageLabels = mode === "page" ? article?.labels : undefined;+ const pageLabels = mode === "page" ? pageClip?.labels : undefined;const published = pageLabels?.publishedDate;const modified = pageLabels?.modifiedDate;const button = SAVE_BUTTONS[mode ?? "page"];@@ -432,7 +440,7 @@ function PagePreview({ tab, disabled, shortcuts, draft, onDraftChange }: {)}{mode === "note" && modes.length === 1 && (- <div className="clip-hint">This page has no article; save it as a bookmark.</div>+ <div className="clip-hint">This page has nothing to save; save it as a bookmark.</div>)}<div className="page-body">@@ -507,6 +515,21 @@ function PagePreview({ tab, disabled, shortcuts, draft, onDraftChange }: {onKeyDown={saveNoteOnCtrlEnter}/>)}+ {mode === "page" && page && (+ <div className="page-variant" role="group" aria-label="How to read the page">+ {(Object.keys(VARIANT_LABELS) as PageVariant[]).map((option) => (+ <button+ key={option}+ aria-pressed={option === variant}+ disabled={!page[option]}+ title={page[option] ? undefined : "Trilium found no article on this page."}+ onClick={() => onDraftChange({ ...draft, variant: option })}+ >+ {VARIANT_LABELS[option]}+ </button>+ ))}+ </div>+ )}{clip && (<iframeclassName="page-content"@@ -556,11 +579,11 @@ function fitToContent(frame: HTMLIFrameElement) {}).observe(root);}-/** The modes the page offers, the default first: the selection, then the article, then a note. */-function availableModes(article: ReadablePage | undefined, selection: SelectedContent | undefined): ClipMode[] {+/** The modes the page offers, the default first: the selection, then the page, then a note. */+function availableModes(page: ReadablePage | undefined, selection: SelectedContent | undefined): ClipMode[] {const modes: ClipMode[] = [];if (selection) modes.push("selection");- if (article) modes.push("page");+ if (page) modes.push("page");modes.push("note");return modes;}@@ -622,18 +645,18 @@ async function readActiveTab(): Promise<TabContent> {if (tab?.id === undefined) return { page: "inaccessible", selection: null };const [ page, selection, description ] = await Promise.all([- extractPage(tab.id), readSelection(tab.id), readDescription(tab.id)+ readPage(tab.id), readSelection(tab.id), readDescription(tab.id)]);return { page, selection, title: tab.title, url: tab.url, description };}/**- * `extractReadablePage()` answers `undefined` when Readability cannot parse the page, and fails- * where the extension cannot run scripts.+ * `extractPage()` answers `undefined` when the page shows nothing, and fails where the extension+ * cannot run scripts.*/-async function extractPage(tabId: number): Promise<PageState> {+async function readPage(tabId: number): Promise<PageState> {try {- return await extractReadablePage(tabId) ?? "unreadable";+ return await extractPage(tabId) ?? "unreadable";} catch {return "inaccessible";}diff --git a/apps/web-clipper/entrypoints/popup/popup.css b/apps/web-clipper/entrypoints/popup/popup.cssindex 107c8dfc47..6d489810aa 100644--- a/apps/web-clipper/entrypoints/popup/popup.css+++ b/apps/web-clipper/entrypoints/popup/popup.css@@ -100,15 +100,12 @@ body {border-radius: 8px;background: var(--card-background-color);- .clip-mode {+ .clip-mode,+ .page-variant {display: flex;gap: 2px;- padding: 2px;- border-radius: 8px;- background: var(--input-background-color);button {- flex: 1;padding: 3px 8px;border: 0;border-radius: 6px;@@ -117,10 +114,15 @@ body {font-size: 0.9em;cursor: pointer;- &:hover {+ &:hover:not(:disabled) {color: var(--main-text-color);}+ &:disabled {+ opacity: 0.5;+ cursor: default;+ }+&[aria-pressed="true"] {background: var(--cmd-button-background-color);box-shadow: 1px 1px 1px var(--cmd-button-shadow-color);@@ -129,6 +131,32 @@ body {}}+ .clip-mode {+ padding: 2px;+ border-radius: 8px;+ background: var(--input-background-color);++ button {+ flex: 1;+ }+ }++ .page-variant {+ flex: none;+ position: sticky;+ top: 0;+ z-index: 1;+ justify-content: flex-end;+ padding: 4px 6px;+ border-bottom: 1px solid var(--main-border-color);+ background: var(--main-background-color);++ button {+ padding: 1px 8px;+ font-size: 0.8em;+ }+ }+.page-body {display: flex;flex-direction: column;diff --git a/apps/web-clipper/entrypoints/popup/popup.spec.tsx b/apps/web-clipper/entrypoints/popup/popup.spec.tsxindex d8d08ad611..43cf4e6cba 100644--- a/apps/web-clipper/entrypoints/popup/popup.spec.tsx+++ b/apps/web-clipper/entrypoints/popup/popup.spec.tsx@@ -27,8 +27,9 @@ const PAGE = {clipType: "page",labels: { pageAuthor: "Jane Doe", publishedDate: "2024-05-01", modifiedDate: "2024-06-12" }};+const WHOLE_PAGE = { ...PAGE, content: `<nav>Menu</nav><p>Body</p><img src="i1">` };/** What `readable-page.js` returns in the tab, as `browser.scripting.executeScript()` answers. */-const executeScript = vi.fn(async (): Promise<unknown> => [ { result: PAGE } ]);+const executeScript = vi.fn(async (): Promise<unknown> => extracted(PAGE));const tabsQuery = vi.fn(async (): Promise<{ id?: number, title?: string, url?: string }[]> => [ { id: 7 } ]);const SELECTION = {title: "Page title",@@ -230,10 +231,47 @@ describe("popup", () => {expect(frame?.getAttribute("sandbox")).toBe("allow-same-origin");expect(frame?.getAttribute("srcdoc")).toBe(previewDocument(PAGE));expect([ ...container.querySelector(".page-body")?.children ?? [] ].map((part) => part.className))- .toEqual([ "page-fields", "page-content" ]);+ .toEqual([ "page-fields", "page-variant", "page-content" ]);expect(button("Save page to Trilium")?.disabled).toBe(false);});+ it("switches the page between its article and the whole page, and keeps the choice", async () => {+ expect(variants()).toEqual([ [ "Article", "true", false ], [ "Whole page", "false", false ] ]);+ expect(container.querySelector(".page-variant")?.getAttribute("aria-label")).toBe("How to read the page");++ await click("Whole page");+ expect(variants()).toEqual([ [ "Article", "false", false ], [ "Whole page", "true", false ] ]);+ expect(frameDocument()).toBe(previewDocument(WHOLE_PAGE));++ await click("Bookmark");+ expect(container.querySelector(".page-variant")).toBeNull();+ await receive({ name: "trilium-search-status", triliumSearch: { status: "not-found" } });+ await receive({ name: "trilium-search-status", triliumSearch: { status: "found-desktop", port: 37840, url: "" } });+ await flush();+ await click("Page");+ expect(variants()).toEqual([ [ "Article", "false", false ], [ "Whole page", "true", false ] ]);++ await type(".page-title", "Gallery");+ await click("Save page to Trilium");+ expect(sendMessage).toHaveBeenLastCalledWith({ name: "save-whole-page", page: { ...WHOLE_PAGE, title: "Gallery" } });++ await click("Article");+ expect(frameDocument()).toBe(previewDocument(PAGE));+ });++ it("opens a page without an article on the whole page", async () => {+ executeScript.mockResolvedValueOnce(extracted(undefined));+ await rerender();+ expect(modes()).toEqual([ [ "Page", "true" ], [ "Bookmark", "false" ] ]);+ expect(variants()).toEqual([ [ "Article", "false", true ], [ "Whole page", "true", false ] ]);+ expect(button("Article")?.title).toBe("Trilium found no article on this page.");+ expect(frameDocument()).toBe(previewDocument(WHOLE_PAGE));+ expect(container.querySelector<HTMLInputElement>(".page-title")?.value).toBe("An article");++ await click("Save page to Trilium");+ expect(sendMessage).toHaveBeenLastCalledWith({ name: "save-whole-page", page: WHOLE_PAGE });+ });+it("saves the extracted page with the edited title, and closes", async () => {await type(".page-title", " My title ");await click("Save page to Trilium");@@ -246,7 +284,7 @@ describe("popup", () => {});it("leaves out the dates of a page that states none", async () => {- executeScript.mockResolvedValueOnce([ { result: { ...PAGE, labels: {} } } ]);+ executeScript.mockResolvedValueOnce(extracted({ ...PAGE, labels: {} }));await rerender();expect(container.querySelector(".page-meta")?.textContent).toBe("example.com");expect(container.querySelectorAll(".page-fields > .page-field")).toHaveLength(2);@@ -264,17 +302,17 @@ describe("popup", () => {expect(cardParts()).toEqual([ "page-body", "btn btn-primary primary-action" ]);expect(button("Save page to Trilium")?.disabled).toBe(true);- resolvePage([ { result: PAGE } ]);+ resolvePage(extracted(PAGE));await flush();expect(container.querySelector(".page-body iframe")).not.toBeNull();});- it("opens a page with no article on a bookmark, and keeps every other action", async () => {+ it("opens a page with nothing to save on a bookmark, and keeps every other action", async () => {executeScript.mockResolvedValueOnce([ { result: null } ]);await rerender();expect(container.querySelector(".no-items")).toBeNull();expect(container.querySelector(".clip-mode")).toBeNull();- expect(container.querySelector(".clip-hint")?.textContent).toBe("This page has no article; save it as a bookmark.");+ expect(container.querySelector(".clip-hint")?.textContent).toBe("This page has nothing to save; save it as a bookmark.");expect(document.activeElement).toBe(container.querySelector(".page-body .note-text"));expect(container.querySelector(".page-meta")).toBeNull();expect(button("Save bookmark")?.disabled).toBe(false);@@ -325,7 +363,7 @@ describe("popup", () => {expect(closeWindow).toHaveBeenCalledTimes(2);});- it("saves the selection of a page that has no article, and ignores an empty one", async () => {+ it("saves the selection of a page with nothing else to save, and ignores an empty one", async () => {selection = SELECTION;executeScript.mockResolvedValueOnce([ { result: null } ]);await rerender();@@ -493,7 +531,7 @@ describe("popup", () => {expect(author()).toBeNull();selection = undefined;- executeScript.mockResolvedValueOnce([ { result: { ...PAGE, labels: {} } } ]);+ executeScript.mockResolvedValueOnce(extracted({ ...PAGE, labels: {} }));await rerender();expect(author()?.value).toBe("");await type(".page-author", "Written by hand");@@ -800,6 +838,17 @@ function modes() {.map((option) => [ option.textContent, option.getAttribute("aria-pressed") ]);}+/** The page variant switch's options, each with whether it is the current one and whether it is disabled. */+function variants() {+ return [ ...container.querySelectorAll<HTMLButtonElement>(".page-body > .page-variant button") ]+ .map((option) => [ option.textContent, option.getAttribute("aria-pressed"), option.disabled ]);+}++/** What `executeScript()` answers for a page with `article`, whose whole page is `WHOLE_PAGE`. */+function extracted(article: object | undefined) {+ return [ { result: { article, wholePage: WHOLE_PAGE } } ];+}+function frameDocument() {const frame = container.querySelector(".page-body iframe.page-content");expect(frame).not.toBeNull();@@ -815,7 +864,7 @@ function cardParts() {function captureButtons() {return [ ...container.querySelectorAll<HTMLButtonElement>(".capture-actions button") ]- .filter((button) => !button.closest(".clip-mode"));+ .filter((button) => !button.closest(".clip-mode, .page-variant"));}function status() {diff --git a/apps/web-clipper/entrypoints/readable-page/index.spec.ts b/apps/web-clipper/entrypoints/readable-page/index.spec.tsindex 3e93508a3f..0e5856654c 100644--- a/apps/web-clipper/entrypoints/readable-page/index.spec.ts+++ b/apps/web-clipper/entrypoints/readable-page/index.spec.ts@@ -1,7 +1,7 @@import type { Window as HappyDOMWindow } from "happy-dom";import { beforeEach, describe, expect, it } from "vitest";-import type { ReadablePage } from "@/utils";+import type { ExtractedPage } from "@/utils";import readablePage from "./index";@@ -37,7 +37,7 @@ describe("readable page", () => {</article>`;const pageBefore = document.documentElement.outerHTML;- const page = readPage();+ const page = readPage()?.article;expect(page).toMatchObject({title: "Article title",@@ -74,12 +74,12 @@ describe("readable page", () => {document.head.innerHTML = `<title>Post title</title>${structuredData("2025-12-24T08:00:00Z")}<meta property="article:published_time" content="">`;document.body.innerHTML = article;- expect(readPage()?.labels).toEqual({ publishedDate: "2025-12-24" });+ expect(readPage()?.article?.labels).toEqual({ publishedDate: "2025-12-24" });document.head.innerHTML = `<title>Post title</title>${structuredData("next week")}<meta property="article:modified_time" content="not a date">`;document.body.innerHTML = article;- expect(readPage()?.labels).toEqual({});+ expect(readPage()?.article?.labels).toEqual({});});it("keeps the day the page states, whatever its offset from UTC", () => {@@ -88,23 +88,69 @@ describe("readable page", () => {<meta property="article:published_time" content="2022-02-09T00:30:00+01:00"><meta property="article:modified_time" content="2022-02-10T23:30:00-05:00">`;document.body.innerHTML = `<article><p>${PARAGRAPH.repeat(5)}</p></article>`;- expect(readPage()?.labels).toEqual({ publishedDate: "2022-02-09", modifiedDate: "2022-02-10" });+ expect(readPage()?.article?.labels).toEqual({ publishedDate: "2022-02-09", modifiedDate: "2022-02-10" });document.head.innerHTML = "<title>Post title</title>";document.body.innerHTML = `<article><p>${PARAGRAPH.repeat(5)}</p><p>Published <time>February 8, 2022</time></p></article>`;- expect(readPage()?.labels).toEqual({});+ expect(readPage()?.article?.labels).toEqual({});});- it("keeps an article that is only a picture, and returns nothing for a page without one", () => {+ it("keeps an article that is only a picture, and the whole page of a page without one", () => {document.body.innerHTML = `<article><img src="photo.jpg" width="800" height="600"></article>`;- expect(readPage()?.images.map(({ src }) => src)).toEqual([ "https://example.com/blog/photo.jpg" ]);+ expect(readPage()?.article?.images.map(({ src }) => src)).toEqual([ "https://example.com/blog/photo.jpg" ]);document.body.innerHTML = "<nav><a href=\"/\">Home</a> <a href=\"/archive\">Archive</a></nav>";+ const page = readPage();+ expect(page?.article).toBeUndefined();+ expect(page?.wholePage.content).toContain("href=\"https://example.com/archive\"");++ document.body.innerHTML = "<div> </div>";expect(readPage()).toBeUndefined();});++ it("keeps every card of a card grid in the whole page, without hidden elements", () => {+ document.head.innerHTML = `<title>Image formats</title><meta name="description" content="Test page">`;+ // The page of issue #9611, with enough text that Defuddle does not read it as an index page.+ const cards = [ "png", "jpg", "webp", "gif" ].map((format) => `+ <div class="card"><h2>${format.toUpperCase()} image test</h2>+ <img src="images/sample.${format}" alt="${format.toUpperCase()} Test" />+ <strong>Format:</strong> ${format.toUpperCase()} <code>.${format}</code></div>`);+ document.body.innerHTML = `+ <h1>Image formats</h1>+ <div class="desc">+ <p>This page tests how the web clipper saves each image format.</p>+ <p>${PARAGRAPH}</p>+ <p>Test method:</p>+ <ol><li>Open this page</li><li>Save the page with the web clipper</li>+ <li>Check that Trilium saved every image</li></ol>+ </div>+ <div class="grid">${cards.join("")}</div>+ <div class="footer"><h3>What to check</h3>+ <ul><li>Whether every image is saved as an attachment</li>+ <li>Whether the GIF stays animated</li></ul></div>+ <div style="display: none">Cookie banner</div>+ <script>window.tracked = true;</script>`;++ const { article, wholePage } = readPage() ?? {};++ expect(article?.images.length).toBeLessThan(4);+ expect(article?.content).not.toContain("What to check");+ expect(wholePage).toMatchObject({+ title: "Image formats",+ pageUrl: "https://example.com/blog/post.html?id=1",+ clipType: "page",+ labels: { pageDescription: "Test page" }+ });+ expect(wholePage?.images.map(({ src }) => src)).toEqual([ "png", "jpg", "webp", "gif" ]+ .map((format) => `https://example.com/blog/images/sample.${format}`));+ expect(wholePage?.content).toContain("This page tests how the web clipper saves each image format.");+ expect(wholePage?.content).toContain("What to check");+ expect(wholePage?.content).not.toContain("Cookie banner");+ expect(wholePage?.content).not.toContain("window.tracked");+ });});function readPage() {- return readablePage.main() as ReadablePage | undefined;+ return readablePage.main() as ExtractedPage | undefined;}diff --git a/apps/web-clipper/entrypoints/readable-page/index.ts b/apps/web-clipper/entrypoints/readable-page/index.tsindex 81b5529ed2..4a97d27d74 100644--- a/apps/web-clipper/entrypoints/readable-page/index.ts+++ b/apps/web-clipper/entrypoints/readable-page/index.ts@@ -1,27 +1,59 @@import Defuddle from "defuddle";-import { descriptionLabels, getImages, getPageLocationOrigin, type ReadablePage } from "@/utils";+import { descriptionLabels, type ExtractedPage, getImages, getPageLocationOrigin, type ReadablePage } from "@/utils";/**- * Extracts the readable page with Defuddle. `extractReadablePage()` injects it into the tab only- * when a page is read, so Defuddle does not load with the content script on every page. The value- * `main()` returns is the result of `executeScript()`.+ * Reads the page with Defuddle, as an article and as a whole. `extractPage()` injects it into the+ * tab only when a page is read, so Defuddle does not load with the content script on every page. The+ * value `main()` returns is the result of `executeScript()`.*/export default defineUnlistedScript(readPage);-function readPage(): ReadablePage | undefined {+/**+ * Keeps all of `<body>`: Defuddle still removes hidden elements, makes links absolute and converts+ * embeds, but none of what it scores as clutter, which on a page without an article is the content.+ */+const WHOLE_PAGE_OPTIONS = {+ contentSelector: "body",+ removeLowScoring: false,+ removeExactSelectors: false,+ removePartialSelectors: false,+ removeContentPatterns: false,+ removeSmallImages: false+};++function readPage(): ExtractedPage | undefined {// `parse()` works on a copy of the document and reads styles and image sizes from the page,// so it is given the live document.const article = new Defuddle(document, { url: location.href }).parse();+ const wholePage = new Defuddle(document, { url: location.href, ...WHOLE_PAGE_OPTIONS }).parse();- // A document of its own loads none of the article's images and runs none of its code.- const body = new DOMParser().parseFromString(article.content, "text/html").body;+ const page = {+ title: article.title,+ pageUrl: getPageLocationOrigin() + location.pathname + location.search,+ clipType: "page" as const,+ labels: pageLabels(article)+ };+ const whole = withContent(page, wholePage.content);+ if (!whole) {+ return undefined;+ }+ return { article: withContent(page, article.content), wholePage: whole };+}++/** The clip of `html`, its images replaced by placeholders, or `undefined` when it shows nothing. */+function withContent(page: Omit<ReadablePage, "content" | "images">, html: string): ReadablePage | undefined {+ // A document of its own loads none of the page's images and runs none of its code.+ const body = new DOMParser().parseFromString(html, "text/html").body;if (!body.textContent.trim() && !body.querySelector("img")) {return undefined;}const images = getImages(body);+ return { ...page, content: body.innerHTML, images };+}+function pageLabels(article: { author: string, published: string }) {const labels = descriptionLabels();const author = article.author.replace(/\s+/g, " ").trim();if (author) {@@ -35,15 +67,7 @@ function readPage(): ReadablePage | undefined {if (modifiedDate) {labels['modifiedDate'] = modifiedDate;}-- return {- title: article.title,- content: body.innerHTML,- images,- pageUrl: getPageLocationOrigin() + location.pathname + location.search,- clipType: 'page',- labels- };+ return labels;}function metaContent(property: string) {diff --git a/apps/web-clipper/utils.spec.ts b/apps/web-clipper/utils.spec.tsindex d25028a255..882e869459 100644--- a/apps/web-clipper/utils.spec.ts+++ b/apps/web-clipper/utils.spec.ts@@ -2,7 +2,7 @@ import type { Window as HappyDOMWindow } from "happy-dom";import { afterEach, describe, expect, it, vi } from "vitest";import { fakeBrowser } from "wxt/testing/fake-browser";-import { createLink, extractReadablePage, getPageLocationOrigin, randomString } from "./utils";+import { createLink, extractPage, getPageLocationOrigin, randomString } from "./utils";describe("randomString", () => {it("returns alphanumeric strings of the requested length", () => {@@ -41,7 +41,7 @@ describe("createLink", () => {});-describe("extractReadablePage", () => {+describe("extractPage", () => {afterEach(() => {Reflect.deleteProperty(fakeBrowser, "scripting");});@@ -54,10 +54,10 @@ describe("extractReadablePage", () => {.mockResolvedValueOnce([]);Object.assign(fakeBrowser, { scripting: { executeScript } });- expect(await extractReadablePage(7)).toBe(page);+ expect(await extractPage(7)).toBe(page);expect(executeScript).toHaveBeenCalledWith({ target: { tabId: 7 }, files: [ "/readable-page.js" ] });- expect(await extractReadablePage(7)).toBeUndefined();- expect(await extractReadablePage(7)).toBeUndefined();+ expect(await extractPage(7)).toBeUndefined();+ expect(await extractPage(7)).toBeUndefined();});});diff --git a/apps/web-clipper/utils.ts b/apps/web-clipper/utils.tsindex 7ffcc8daa6..d07f29b848 100644--- a/apps/web-clipper/utils.ts+++ b/apps/web-clipper/utils.ts@@ -3,7 +3,7 @@ export type Rect = { x: number, y: number, width: number, height: number };/** An image of a clipping; the clipping's `content` references it by `imageId`. */export type ClippedImage = { imageId: string, src: string };-/** The readable version of a page, as `entrypoints/readable-page/index.ts` extracts it. */+/** A page clip, as `entrypoints/readable-page/index.ts` extracts it. */export interface ReadablePage {title: string;content: string;@@ -13,17 +13,22 @@ export interface ReadablePage {labels: Record<string, string>;}+/** The two ways to clip a page: its article, when Defuddle finds one, and all of it. */+export interface ExtractedPage {+ article?: ReadablePage;+ wholePage: ReadablePage;+}+/*** Runs `entrypoints/readable-page/index.ts` in the tab and returns the page it read, or `undefined`- * when the page has no article. Throws where the extension cannot run scripts, such as browser- * pages.+ * when the page shows nothing. Throws where the extension cannot run scripts, such as browser pages.*/-export async function extractReadablePage(tabId: number): Promise<ReadablePage | undefined> {+export async function extractPage(tabId: number): Promise<ExtractedPage | undefined> {const [ injection ] = await browser.scripting.executeScript({target: { tabId },files: [ "/readable-page.js" ]});- return (injection?.result ?? undefined) as ReadablePage | undefined;+ return (injection?.result ?? undefined) as ExtractedPage | undefined;}export function randomString(len: number) {diff --git a/docs/User Guide/User Guide/Installation & Setup/Web Clipper.md b/docs/User Guide/User Guide/Installation & Setup/Web Clipper.mdindex 4529613219..babdf4ae64 100644--- a/docs/User Guide/User Guide/Installation & Setup/Web Clipper.md+++ b/docs/User Guide/User Guide/Installation & Setup/Web Clipper.md@@ -32,12 +32,12 @@ Clicking the extension's button in the browser toolbar opens its popup:* The popup shows what will be saved from the current page: the note's title, which can be edited before saving, the website, and below them a preview, which starts with the page's description. A switch above the preview chooses what to save, among what the page offers:* _Selection_, when text is selected on the page: _Save selection_ saves it into the note of that page's clippings, as the context menu does.- * _Page_, when the page has an article: _Save page to Trilium_ saves the article's text and images. The author the page names shows under the description, marked by the <span class="tn-icon bx bx-user"></span> icon. It can be edited or cleared before saving, or written in when the page names none, and is saved in the `#pageAuthor` label of the note. When the page states when it was published, that date shows under the author, marked by the <span class="tn-icon bx bx-calendar"></span> icon, and is saved in the `#publishedDate` label of the note. When it states when it was last modified, that date shows below, marked by the <span class="tn-icon bx bx-history"></span> icon, and is saved in the `#modifiedDate` label.+ * _Page_: _Save page to Trilium_ saves the page's text and images. A smaller switch at the top of the preview chooses how much of the page to keep: _Article_ keeps only the page's main text, without its menus, sidebars and footers, and _Whole page_ keeps everything the page shows. _Whole page_ is for pages that are not an article (such as a gallery or a page of cards), where _Article_ can leave out part of the content; the preview shows what each one keeps. _Article_ is unavailable on a page where Trilium finds no article. The author the page names shows under the description, marked by the <span class="tn-icon bx bx-user"></span> icon. It can be edited or cleared before saving, or written in when the page names none, and is saved in the `#pageAuthor` label of the note. When the page states when it was published, that date shows under the author, marked by the <span class="tn-icon bx bx-calendar"></span> icon, and is saved in the `#publishedDate` label of the note. When it states when it was last modified, that date shows below, marked by the <span class="tn-icon bx bx-history"></span> icon, and is saved in the `#modifiedDate` label.* _Bookmark_, always: a text box for a note about the page replaces the preview, and _Save bookmark_ (or <kbd>Ctrl</kbd>+<kbd>Enter</kbd>) saves a link to the page with that note. Left empty, the title stays the page's title.The description, marked by the <span class="tn-icon bx bx-align-left"></span> icon, comes from the page's `description` meta tag, or the one it gives social sites. It can be edited or cleared before saving, or written in when the page has none, and is saved in the `#pageDescription` label of the note. Clippings saved with the keyboard shortcuts or the context menu, screenshots and images included, get the description as the page gives it. Each clipping that brings a description updates the label.- The popup opens on the selection if there is one, otherwise on the article. On a page with no article and no selection, it opens on _Bookmark_ and says the page has no article. The keyboard shortcuts and the context menu still save right away, without a preview.+ The popup opens on the selection if there is one, otherwise on _Page_, showing the article, or the whole page when the page has no article. On a page that shows nothing to save, it opens on _Bookmark_ and says so. The keyboard shortcuts and the context menu still save right away, without a preview: saving the whole page that way saves the article, or the whole page when the page has no article.* Pages the extension cannot access (the browser's own pages, the extension stores, and pages that were open before the extension was installed until they are reloaded) cannot be saved or captured, so the popup says so in place of the preview, and only _Tabs_ remains available there.* Below the preview, a row of buttons offers the other actions: _Crop_ (a screenshot of an area you select), _Screenshot_ (of the visible part of the page) and _Tabs_ (saves the links of every tab in the current window as a list). Hovering over a button shows its full name and, if it has one, its keyboard shortcut.* If the current page was already clipped, a notice at the top offers to open the note in Trilium.