Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
client/backend log: add support for highlighting SQL queries
Adorian Doran5ca811abb76f
diff --git a/apps/server/src/assets/help/help_content.json b/apps/server/src/assets/help/help_content.jsonindex 458b834c67..f80e5443a1 100644--- a/apps/server/src/assets/help/help_content.json+++ b/apps/server/src/assets/help/help_content.json@@ -188,7 +188,7 @@"_help_wy8So3yZZlH9": "<p>Go to <a href=\"https://github.com/TriliumNext/Trilium/issues\">Issues · TriliumNext/Trilium</a> and press “New issue”.</p><p>If you are reporting a bug, select “Bug Report” and fill in the details.</p>","_help_x59R8J8KV5Bp": "<p><img src=\"{{helpAssets}}/User%20Guide/Troubleshooting/Anonymized%20Database_image.jpg\"></p><p>In certain scenarios, understanding the structure of a database is crucial for troubleshooting issues. However, sharing your actual <a href=\"#root/_help_wX4HbRucYSDD\">database</a> file with personal notes is not advisable. To address this, Trilium offers a feature to anonymize the database. This feature can be accessed via Menu -> Options -> Advanced tab.</p><p>This feature creates a copy of your database with all sensitive data removed. Specifically, it strips out note titles, contents, revisions, history, and some non-system attributes while retaining the overall structure and metadata, such as modification dates. After anonymization, the database undergoes a <a href=\"https://sqlite.org/lang_vacuum.html\">vacuuming process</a> to ensure no sensitive data remnants remain in the file. The anonymized database is saved in the <code>anonymized</code> directory within the <a href=\"#root/_help_tAassRL4RSQL\">data directory</a>, making it safe to share with bug reports.</p><p>This will create a copy of your document and remove all sensitive data (currently note titles, contents, revisions, history and some of the options, and non-system attributes) while leaving all structure and metadata (e.g. date of last change). After this is done, the database is <a href=\"https://sqlite.org/lang_vacuum.html\">VACUUMed</a> to make sure there's no stale sensitive data in the document file. The resulting file is stored in <code>anonymized</code> directory (placed in the <a href=\"#root/_help_tAassRL4RSQL\">data directory</a>). You can safely attach it to a bug report.</p><h2>Command Line Anonymization</h2><p>If your <a href=\"#root/_help_wX4HbRucYSDD\">database</a> is corrupted to the point where Trilium cannot start, the anonymization process can still be executed via the command line:</p><pre><code class=\"language-text-x-trilium-auto\">node src/anonymize.js</code></pre><p>Run this command from the directory containing Trilium's source files, typically found in the <code>resources/app</code> directory for desktop builds.</p>","_help_qzNzp9LYQyPT": "<p>It's important to provide all available error logs together with bug reports.</p><p>There are two types of error logs, both of which are useful when reporting bugs.</p><ul><li><a class=\"reference-link\" href=\"#root/_help_bnyigUA2UK7s\">Backend (server) logs</a></li><li><a class=\"reference-link\" href=\"#root/_help_9yEHzMyFirZR\">Frontend logs</a></li></ul><h2>Providing sensitive data</h2><p>If you don't feel comfortable attaching the logs or anything sensitive to the public GitHub issues, feel free to contact the devs in our Matrix <a href=\"https://github.com/TriliumNext/Trilium#-discuss-with-us\">support channel</a>.</p><p>Use this email to also provide anything which could assist in analysing the bug - e.g. files/images/ZIPs being imported or <a href=\"#root/_help_x59R8J8KV5Bp\">anonymized database</a>.</p><h3>Exporting note subtree for reproduction</h3><p>Often times, bugs manifest themselves in specific notes and having them would greatly ease reproduction and fixing.</p><p>In such case, please export the relevant note subtree by right-clicking it on the left tree, choosing Export - HTML as ZIP:</p><p><img src=\"{{helpAssets}}/User%20Guide/Troubleshooting/Error%20logs_error-logs-export-subtree.png\"></p>",- "_help_bnyigUA2UK7s": "<h2>Accessing via the backend log</h2><p>In the <a class=\"reference-link\" href=\"#root/_help_x3i7MxGccDuM\">Global menu</a>, go to <em>Advanced</em> → <em>Show backend log</em>. This will display the current backend log (i.e. today's), with the historical information available only on disk (see below).</p><p>Interaction since v0.104.0:</p><ul><li>The file can be downloaded as a text file using the <span class=\"tn-icon bx bx-download\"></span> button in the <a class=\"reference-link\" href=\"#root/_help_8YBEPzcpUgxw\">Note menu</a> area.</li><li>The file can also be searched just like a normal <a class=\"reference-link\" href=\"#root/_help_6f9hih2hXXZk\">Code</a> note.</li></ul><h2>Location on the disk</h2><p>Backend logs are stored on the file system. To find them, open the <a class=\"reference-link\" href=\"#root/_help_tAassRL4RSQL\">Data directory</a>, go to the <code>log</code> subdirectory and find the latest log file, e.g. <code>trilium-2022-12-14.log</code>. </p><h2>In the terminal</h2><p>The server also prints every log entry to its standard output, which is what <code>docker logs</code> or <code>journalctl</code> show. When that output is a terminal, each entry starts with its time and is colored the same way as the backend log in the app: request status codes by class, slow SQL queries, and errors in red. The startup banner with the Trilium logo and version information is colored too. Output to a file, a pipe or a container log stays plain text without the time, since those tools add their own.</p><p>To turn the colors off in a terminal, set the <code>NO_COLOR</code> environment variable; to force them on elsewhere, set <code>FORCE_COLOR=1</code>.</p><h2>Reporting backend bugs</h2><p>You can attach the whole file to the bug report (preferable) or open it and copy-paste only the last lines / lines you believe are relevant.</p><h2>Customizing the retention of logs</h2><p>The backend logs are fully managed by the Trilium server. By default the last 90 days worth of logs are kept; the logs older than that are deleted in order to reduce the space consumption.</p><p>It's possible to change the retention period by modifying the <a class=\"reference-link\" href=\"#root/_help_Gzjqa934BdH4\">Configuration (config.ini or environment variables)</a> via the <code>.ini</code> file:</p><pre><code class=\"language-text-x-trilium-auto\">[Logging]\nretentionDays=7</code></pre><p>Or via the environment variable <code>TRILIUM_LOGGING_RETENTION_DAYS</code>.</p><p>Special cases:</p><ul><li>Positive values indicate the number of days worth of logs to keep</li><li>A value of 0 results with the default value (90 days) to be used</li><li>Negative values (e.g. <code>-1</code>) result with all logs to be kept, irrespective how ancient and numerous (and</li></ul><aside class=\"admonition note\"><p>If you set the retention days to a low number, you might notice that not all the log files are being deleted. This is because a minimum number of logs (7 at the time of writing) is maintained at all times.</p></aside>",+ "_help_bnyigUA2UK7s": "<h2>Accessing via the backend log</h2><p>In the <a class=\"reference-link\" href=\"#root/_help_x3i7MxGccDuM\">Global menu</a>, go to <em>Advanced</em> → <em>Show backend log</em>. This will display the current backend log (i.e. today's), with the historical information available only on disk (see below).</p><p>The log is colored to make it easier to read: request status codes by class, slow SQL queries, and errors in red. The statement of a slow SQL query is highlighted the same way as in the <a class=\"reference-link\" href=\"#root/_help_YKWqdJhzi2VY\">SQL Console</a>, using the color scheme of code notes (see <em>Color schemes</em> in <a class=\"reference-link\" href=\"#root/_help_6f9hih2hXXZk\">Code</a>).</p><p>Interaction since v0.104.0:</p><ul><li>The file can be downloaded as a text file using the <span class=\"tn-icon bx bx-download\"></span> button in the <a class=\"reference-link\" href=\"#root/_help_8YBEPzcpUgxw\">Note menu</a> area.</li><li>The file can also be searched just like a normal <a class=\"reference-link\" href=\"#root/_help_6f9hih2hXXZk\">Code</a> note.</li></ul><h2>Location on the disk</h2><p>Backend logs are stored on the file system. To find them, open the <a class=\"reference-link\" href=\"#root/_help_tAassRL4RSQL\">Data directory</a>, go to the <code>log</code> subdirectory and find the latest log file, e.g. <code>trilium-2022-12-14.log</code>. </p><h2>In the terminal</h2><p>The server also prints every log entry to its standard output, which is what <code>docker logs</code> or <code>journalctl</code> show. When that output is a terminal, each entry starts with its time and is colored like the backend log in the app: request status codes by class, slow SQL queries, and errors in red. The statement of a slow SQL query is shown in a single color there. The startup banner with the Trilium logo and version information is colored too. Output to a file, a pipe or a container log stays plain text without the time, since those tools add their own.</p><p>To turn the colors off in a terminal, set the <code>NO_COLOR</code> environment variable; to force them on elsewhere, set <code>FORCE_COLOR=1</code>.</p><h2>Reporting backend bugs</h2><p>You can attach the whole file to the bug report (preferable) or open it and copy-paste only the last lines / lines you believe are relevant.</p><h2>Customizing the retention of logs</h2><p>The backend logs are fully managed by the Trilium server. By default the last 90 days worth of logs are kept; the logs older than that are deleted in order to reduce the space consumption.</p><p>It's possible to change the retention period by modifying the <a class=\"reference-link\" href=\"#root/_help_Gzjqa934BdH4\">Configuration (config.ini or environment variables)</a> via the <code>.ini</code> file:</p><pre><code class=\"language-text-x-trilium-auto\">[Logging]\nretentionDays=7</code></pre><p>Or via the environment variable <code>TRILIUM_LOGGING_RETENTION_DAYS</code>.</p><p>Special cases:</p><ul><li>Positive values indicate the number of days worth of logs to keep</li><li>A value of 0 results with the default value (90 days) to be used</li><li>Negative values (e.g. <code>-1</code>) result with all logs to be kept, irrespective how ancient and numerous (and</li></ul><aside class=\"admonition note\"><p>If you set the retention days to a low number, you might notice that not all the log files are being deleted. This is because a minimum number of logs (7 at the time of writing) is maintained at all times.</p></aside>","_help_9yEHzMyFirZR": "<p>To provide frontend logs, we need to open the Developer Console. Often the easiest way is to press <kbd>Ctrl</kbd>-<kbd>Shift</kbd>-<kbd>I</kbd> which should work in most browsers (and desktop app). Make sure that the error producing action happened right before you copy&paste the errors, the console is cleared on app restart.</p><p>If that doesn't work, then:</p><ul><li>in Trilium desktop app, go to top-left menu button -> Advanced -> Open dev tools</li><li>In Firefox/Chrome right-click anywhere in the page and click Inspect:</li></ul><p><img src=\"{{helpAssets}}/User%20Guide/Troubleshooting/Error%20logs/Frontend%20logs_error-logs-firefox-context-men.png\"></p><p>Once you have Dev Tools open, click on \"Console\" tab:</p><p><img src=\"{{helpAssets}}/User%20Guide/Troubleshooting/Error%20logs/Frontend%20logs_image.jpg\"></p><p>Copy-paste (or screenshot) the logs. It's better to provide not just errors, but the whole log, which might provide context while analyzing the bug.</p>","_help_vdlYGAcpXAgc": "<p>Synchronization can sometimes take a long amount of time in order to compute the items that require update. When running behind a reverse proxy, the request can time out.</p><p>The solution is to increase the timeout at proxy level.</p><h2>Nginx</h2><p>Add the following to the configuration file:</p><pre><code class=\"language-text-x-nginx-conf\">proxy_connect_timeout 300;\nproxy_send_timeout 300;\nproxy_read_timeout 300;\nsend_timeout 300;</code></pre><p>And restart the server.</p><p>See <a href=\"#root/_help_ud6MShXL4WpO\">Nginx Proxy Setup</a> for more information about the Nginx setup.</p><p>If it still doesn't work, try increasing the timeout.</p>","_help_s8alTXmpFR61": "<p>Some changes to the application will not take effect immediately and as such it might require to manually reload or refresh the application.</p><p>To do so, simply press <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>R</kbd>.</p>",diff --git a/docs/User Guide/User Guide/Troubleshooting/Error logs/Backend (server) logs.md b/docs/User Guide/User Guide/Troubleshooting/Error logs/Backend (server) logs.mdindex ef8cd11f71..44837849c6 100644--- a/docs/User Guide/User Guide/Troubleshooting/Error logs/Backend (server) logs.md+++ b/docs/User Guide/User Guide/Troubleshooting/Error logs/Backend (server) logs.md@@ -3,6 +3,8 @@In the <a class="reference-link" href="../../Basic%20Concepts%20and%20Features/UI%20Elements/Global%20menu.md">Global menu</a>, go to _Advanced_ → _Show backend log_. This will display the current backend log (i.e. today's), with the historical information available only on disk (see below).+The log is colored to make it easier to read: request status codes by class, slow SQL queries, and errors in red. The statement of a slow SQL query is highlighted the same way as in the <a class="reference-link" href="../../Advanced%20Usage/Database/Manually%20altering%20the%20database/SQL%20Console.md">SQL Console</a>, using the color scheme of code notes (see _Color schemes_ in <a class="reference-link" href="../../Note%20Types/Code.md">Code</a>).+Interaction since v0.104.0:* The file can be downloaded as a text file using the <span class="tn-icon bx bx-download"></span> button in the <a class="reference-link" href="../../Basic%20Concepts%20and%20Features/UI%20Elements/Note%20menu.md">Note menu</a> area.@@ -14,7 +16,7 @@ Backend logs are stored on the file system. To find them, open the <a class="re## In the terminal-The server also prints every log entry to its standard output, which is what `docker logs` or `journalctl` show. When that output is a terminal, each entry starts with its time and is colored the same way as the backend log in the app: request status codes by class, slow SQL queries, and errors in red. The startup banner with the Trilium logo and version information is colored too. Output to a file, a pipe or a container log stays plain text without the time, since those tools add their own.+The server also prints every log entry to its standard output, which is what `docker logs` or `journalctl` show. When that output is a terminal, each entry starts with its time and is colored like the backend log in the app: request status codes by class, slow SQL queries, and errors in red. The statement of a slow SQL query is shown in a single color there. The startup banner with the Trilium logo and version information is colored too. Output to a file, a pipe or a container log stays plain text without the time, since those tools add their own.To turn the colors off in a terminal, set the `NO_COLOR` environment variable; to force them on elsewhere, set `FORCE_COLOR=1`.diff --git a/packages/codemirror/src/code_preview.ts b/packages/codemirror/src/code_preview.tsindex 7391e12d6d..4c1ea1a8cf 100644--- a/packages/codemirror/src/code_preview.ts+++ b/packages/codemirror/src/code_preview.ts@@ -1,13 +1,10 @@-import {- ensureSyntaxTree, type Language, LanguageSupport, StreamLanguage-} from "@codemirror/language";+import { ensureSyntaxTree, type Language } from "@codemirror/language";import { EditorState } from "@codemirror/state";import { highlightCode, type Highlighter, tagHighlighter } from "@lezer/highlight";-import type { SupportedMimeTypes } from "@triliumnext/commons";import { getThemeById } from "./color_themes.js";import { type PreviewTheme, type PreviewThemeSlot, readPreviewTheme } from "./preview_theme.js";-import byMimeType, { MIME_ALIASES } from "./syntax_highlighting.js";+import { loadLanguage } from "./syntax_highlighting.js";export {buildCodePreviewCss, type PreviewTheme, type PreviewThemeColors, type PreviewThemeSlot,@@ -82,7 +79,7 @@ export function loadPreviewTheme(id: string) {export function loadPreviewLanguage(mime: string) {let language = languages.get(mime);if (!language) {- language = readLanguage(mime).then((result) => {+ language = loadLanguage(mime).then((result) => {loadedLanguages.set(mime, result);return result;});@@ -123,19 +120,6 @@ function getHighlighter({ theme, key }: PreviewThemeSlot): Highlighter {})));}-async function readLanguage(mime: string): Promise<Language | null> {- const load = byMimeType[(MIME_ALIASES[mime] ?? mime) as SupportedMimeTypes];- if (!load) return null;-- const syntax = await load();- if ("token" in syntax) return StreamLanguage.define(syntax);-- const extensions = Array.isArray(syntax) ? syntax : [ syntax ];- return extensions- .find((extension): extension is LanguageSupport => extension instanceof LanguageSupport)- ?.language ?? null;-}-function escapeHtml(text: string) {return text.replace(/[&<>"]/g, (c) => HTML_ESCAPES[c]);}diff --git a/packages/codemirror/src/extensions/trilium_log_highlighter.spec.ts b/packages/codemirror/src/extensions/trilium_log_highlighter.spec.tsindex c52787cfd6..166770284e 100644--- a/packages/codemirror/src/extensions/trilium_log_highlighter.spec.ts+++ b/packages/codemirror/src/extensions/trilium_log_highlighter.spec.ts@@ -1,9 +1,14 @@-import { Compartment, EditorState } from "@codemirror/state";+import { HighlightStyle, StreamLanguage, syntaxHighlighting } from "@codemirror/language";+import { sqlite } from "@codemirror/legacy-modes/mode/sql";+import { Compartment, EditorState, type Extension } from "@codemirror/state";import { EditorView } from "@codemirror/view";+import { classHighlighter, tags } from "@lezer/highlight";import { describe, expect, it } from "vitest";import { MAX_HIGHLIGHTED_LINES, triliumLogHighlighter } from "./trilium_log_highlighter.js";+const sqlLanguage = StreamLanguage.define(sqlite);+interface DecorationSpan {from: number;to: number;@@ -21,7 +26,8 @@ describe("triliumLogHighlighter", () => {const before = EditorState.create({ doc, extensions: [ compartment.of([]) ] });expect(decorationsOf(before)).toHaveLength(0);- const after = before.update({ effects: compartment.reconfigure(triliumLogHighlighter) }).state;+ const effects = compartment.reconfigure(triliumLogHighlighter());+ const after = before.update({ effects }).state;expect(decorationsOf(after).length).toBeGreaterThan(0);});@@ -188,10 +194,102 @@ describe("triliumLogHighlighter", () => {expect(timestamps.length).toBeGreaterThan(0);expect(timestamps.length).toBeLessThanOrEqual(MAX_HIGHLIGHTED_LINES);});++ describe("with a SQL language", () => {+ it("highlights the tokens of a slow query's statement", () => {+ const doc = "00:10:01.540 Slow query took 78ms: SELECT noteId FROM notes"+ + " WHERE title LIKE '%a%' AND isDeleted = 0 AND parentNoteId = ?";+ const spans = decorationsOf(sqlState(doc));++ expect(slicesOf(doc, spans, "tok-keyword"))+ .toEqual([ "SELECT", "FROM", "WHERE", "LIKE", "AND", "AND" ]);+ expect(slicesOf(doc, spans, "tok-string")).toEqual([ "'%a%'" ]);+ // The `78` of `took 78ms` is outside the statement.+ expect(slicesOf(doc, spans, "tok-number")).toEqual([ "0" ]);+ expect(sliceOf(doc, spans, "cm-log-query-verb")).toBe("Slow query");++ // Identifiers and placeholders keep the main text color.+ expect(classesOfState(sqlState(doc))).toContain("cm-log-query");+ expect(classesOfState(sqlState(doc))).not.toContain("cm-log-sql");+ const placeholder = doc.indexOf("?");+ expect(spans.filter((d) => d.from <= placeholder && d.to > placeholder)).toEqual([]);+ });++ it("tokenizes nothing on recursive queries, which have no statement", () => {+ const doc = [+ "00:10:01.540 Slow query took 30ms: SELECT 1",+ "00:10:02.113 Slow recursive query took 45ms."+ ].join("\n");+ const tokens = decorationsOf(sqlState(doc)).filter((d) => d.class.startsWith("tok-"));++ expect(tokens.map((d) => doc.slice(d.from, d.to))).toEqual([ "SELECT", "1" ]);+ });++ it("reclassifies the tokens when the theme changes, without rescanning the log", () => {+ const doc = "00:10:01.540 Slow query took 78ms: SELECT 1";+ const theme = new Compartment();+ let s = sqlState(doc, theme.of(syntaxHighlighting(classHighlighter)));+ const logDecorations = logDecorationsOf(s);+ expect(sliceOf(doc, decorationsOf(s), "tok-keyword")).toBe("SELECT");++ const style = HighlightStyle.define([ { tag: tags.keyword, class: "keyword" } ]);+ s = s.update({ effects: theme.reconfigure(syntaxHighlighting(style)) }).state;++ expect(sliceOf(doc, decorationsOf(s), "keyword")).toBe("SELECT");+ expect(classesOfState(s)).not.toContain("tok-keyword");+ expect(logDecorationsOf(s)).toBe(logDecorations);+ });++ it("keeps every decoration set unless the document or the configuration changes", () => {+ const doc = "00:10:01.540 Slow query took 78ms: SELECT 1";+ for (const s of [ state(doc), sqlState(doc) ]) {+ const sets = s.facet(EditorView.decorations);+ const next = s.update({ selection: { anchor: 1 } }).state+ .facet(EditorView.decorations);+ expect(next).toHaveLength(sets.length);+ expect(next.every((set, index) => set === sets[index])).toBe(true);+ }++ // Without a language there are no SQL marks to reclassify.+ const compartment = new Compartment();+ const plain = EditorState.create({+ doc, extensions: [ triliumLogHighlighter(), compartment.of([]) ]+ });+ const reconfigured = plain.update({+ effects: compartment.reconfigure(syntaxHighlighting(classHighlighter))+ }).state;+ expect(logDecorationsOf(reconfigured)).toBe(logDecorationsOf(plain));+ expect(classesOfState(reconfigured)).toContain("cm-log-sql");+ });++ it("only tokenizes statements within the last MAX_HIGHLIGHTED_LINES lines", () => {+ const lines = [ "17:00:00.000 Slow query took 30ms: SELECT 1" ];+ for (let i = 0; i < MAX_HIGHLIGHTED_LINES; i++) {+ lines.push("18:00:00.000 Backend script execution is DISABLED.");+ }+ lines.push("19:00:00.000 Slow query took 30ms: SELECT 2");+ const doc = lines.join("\n");+ const spans = decorationsOf(sqlState(doc));++ expect(slicesOf(doc, spans, "tok-keyword")).toEqual([ "SELECT" ]);+ expect(slicesOf(doc, spans, "tok-number")).toEqual([ "2" ]);+ });+ });});function state(doc: string): EditorState {- return EditorState.create({ doc, extensions: [ triliumLogHighlighter ] });+ return EditorState.create({ doc, extensions: [ triliumLogHighlighter() ] });+}++function sqlState(doc: string, highlighting: Extension = syntaxHighlighting(classHighlighter)) {+ return EditorState.create({+ doc, extensions: [ triliumLogHighlighter(sqlLanguage), highlighting ]+ });+}++/** The first decoration set, which holds the log's own decorations. */+function logDecorationsOf(editorState: EditorState) {+ return editorState.facet(EditorView.decorations).find((set) => typeof set !== "function");}/** All decoration ranges contributed to the view's decoration facet, in document order. */@@ -222,6 +320,11 @@ function sliceOf(doc: string, spans: DecorationSpan[], cls: string): string | unreturn span && doc.slice(span.from, span.to);}+/** The document text covered by each span carrying `cls`, in document order. */+function slicesOf(doc: string, spans: DecorationSpan[], cls: string): string[] {+ return spans.filter((d) => d.class === cls).map((d) => doc.slice(d.from, d.to));+}+/** Classes of the line decorations (zero-length, at the line start) on line `lineNo` (1-based). */function lineClassesAt(editorState: EditorState, lineNo: number): string[] {const from = editorState.doc.line(lineNo).from;diff --git a/packages/codemirror/src/extensions/trilium_log_highlighter.ts b/packages/codemirror/src/extensions/trilium_log_highlighter.tsindex 9d2c67a422..45cca27db2 100644--- a/packages/codemirror/src/extensions/trilium_log_highlighter.ts+++ b/packages/codemirror/src/extensions/trilium_log_highlighter.ts@@ -1,5 +1,7 @@+import { highlightingFor, type Language } from "@codemirror/language";import { type EditorState, type Extension, RangeSetBuilder, StateField } from "@codemirror/state";import { Decoration, type DecorationSet, EditorView } from "@codemirror/view";+import { type Highlighter, highlightTree } from "@lezer/highlight";/*** Lightweight regex highlighter for the Trilium backend log. Each log entry is a single line@@ -20,7 +22,8 @@ import { Decoration, type DecorationSet, EditorView } from "@codemirror/view";* `Slow` marker (prepended by the server for requests taking >= 10ms) is flagged as a warning.* - slow SQL query lines `<ts> Slow [recursive ]query took <n>ms[: <sql>]` — the whole line is* tinted, the entire `Slow [recursive ]query` phrase bolded as the entry's verb (its own colour,- * not the HTTP method's), and the statement (absent for recursive queries) coloured. Unlike HTTP,+ * not the HTTP method's), and the statement (absent for recursive queries) highlighted with the+ * SQL language in the active code theme, or in one colour without a language. Unlike HTTP,* `Slow` is not flagged as a warning here: queries are only logged when slow, so it is uninformative.* - `<ts> JS Error: …` and `<ts> ERROR: …` lines — coloured red, together with their following* continuation lines (e.g. wrapped stack traces), which carry no timestamp of their own@@ -72,6 +75,8 @@ const statusMarks: Record<string, Decoration> = {"4": Decoration.mark({ class: "cm-log-status cm-log-status-4xx" }),"5": Decoration.mark({ class: "cm-log-status cm-log-status-5xx" })};+/** Marks of the SQL tokens, one per class string of the code theme. */+const sqlTokenMarks = new Map<string, Decoration>();/*** The Next themes own the palette (`--log-*` in theme-next-light.css / theme-next-dark.css). The@@ -101,21 +106,49 @@ const logHighlightTheme = EditorView.baseTheme({".cm-log-info": { color: "var(--log-info-color, inherit)" }});-const logHighlightField = StateField.define<DecorationSet>({- create(state) {- return buildLogDecorations(state);- },- update(decorations, tr) {- return tr.docChanged ? buildLogDecorations(tr.state) : decorations;- },- provide: (field) => EditorView.decorations.from(field)-});+/** The statement of a slow query and its start offset in the document. */+interface SqlStatement {+ from: number;+ text: string;+}-/** The extension to register on the backend-log editor (e.g. via a `text/x-trilium-log` MIME type). */-export const triliumLogHighlighter: Extension = [ logHighlightField, logHighlightTheme ];+interface LogHighlights {+ log: DecorationSet;+ /** The statements in the highlighted lines; empty without a SQL language. */+ statements: SqlStatement[];+ /** The tokens of `statements`, with the classes of the active code theme. */+ sql: DecorationSet;+}++/**+ * The extension to register on the backend-log editor (e.g. via a `text/x-trilium-log` MIME type).+ * `sqlLanguage` highlights the statements of slow queries; without it, a statement gets one color.+ */+export function triliumLogHighlighter(sqlLanguage: Language | null = null): Extension {+ const field = StateField.define<LogHighlights>({+ create: (state) => buildHighlights(state, sqlLanguage),+ update(highlights, tr) {+ if (tr.docChanged) return buildHighlights(tr.state, sqlLanguage);+ // A new code theme changes the token classes. `performFind()` also reconfigures, on+ // every search, so a reconfigure rebuilds only the SQL marks.+ if (tr.reconfigured && sqlLanguage) {+ const sql = buildSqlDecorations(tr.state, highlights.statements, sqlLanguage);+ return { ...highlights, sql };+ }+ return highlights;+ },+ provide: (f) => [+ EditorView.decorations.from(f, (highlights) => highlights.log),+ EditorView.decorations.from(f, (highlights) => highlights.sql)+ ]+ });+ return [ field, logHighlightTheme ];+}-function buildLogDecorations(state: EditorState): DecorationSet {+function buildHighlights(state: EditorState, sqlLanguage: Language | null): LogHighlights {const builder = new RangeSetBuilder<Decoration>();+ const statements: SqlStatement[] = [];+ const collectedStatements = sqlLanguage ? statements : null;const doc = state.doc;// Scan at most the last MAX_HIGHLIGHTED_LINES lines. A single top-to-bottom pass over that window@@ -125,12 +158,39 @@ function buildLogDecorations(state: EditorState): DecorationSet {const firstLine = Math.max(1, doc.lines - MAX_HIGHLIGHTED_LINES + 1);let blockLine: Decoration | null = null;for (let n = firstLine; n <= doc.lines; n++) {- blockLine = decorateLine(builder, doc.line(n), blockLine);+ blockLine = decorateLine(builder, doc.line(n), blockLine, collectedStatements);}+ return {+ log: builder.finish(),+ statements,+ sql: sqlLanguage ? buildSqlDecorations(state, statements, sqlLanguage) : Decoration.none+ };+}++/** Marks the tokens of each statement with the classes that the active code theme gives them. */+function buildSqlDecorations(+ state: EditorState, statements: SqlStatement[], language: Language+): DecorationSet {+ const builder = new RangeSetBuilder<Decoration>();+ const highlighter: Highlighter = { style: (tags) => highlightingFor(state, tags) };+ for (const { from, text } of statements) {+ highlightTree(language.parser.parse(text), highlighter, (start, end, classes) => {+ builder.add(from + start, from + end, sqlTokenMark(classes));+ });+ }return builder.finish();}+function sqlTokenMark(classes: string) {+ let mark = sqlTokenMarks.get(classes);+ if (!mark) {+ mark = Decoration.mark({ class: classes });+ sqlTokenMarks.set(classes, mark);+ }+ return mark;+}+/*** Returns the line decoration a multi-line block entry applies to itself and its continuation lines* (e.g. `errorLine` for `JS Error:`/`ERROR:`, `infoLine` for `JS Info:`), or null for entries that@@ -145,9 +205,15 @@ function blockLineFor(text: string): Decoration | null {/*** Decorates a single line and returns the block decoration in effect for the following* (continuation) lines. A timestamped line opens a block (or closes any open one); a line without a- * timestamp inherits the incoming `blockLine`.+ * timestamp inherits the incoming `blockLine`. The statement of a slow query goes into `statements`+ * for SQL highlighting, or gets `sqlMark` if `statements` is null.*/-function decorateLine(builder: RangeSetBuilder<Decoration>, line: { from: number; text: string }, blockLine: Decoration | null): Decoration | null {+function decorateLine(+ builder: RangeSetBuilder<Decoration>,+ line: { from: number; text: string },+ blockLine: Decoration | null,+ statements: SqlStatement[] | null+): Decoration | null {const { from: lineStart, text } = line;if (!TIMESTAMP.test(text)) {@@ -202,7 +268,11 @@ function decorateLine(builder: RangeSetBuilder<Decoration>, line: { from: numberbuilder.add(verbStart, verbEnd, queryVerbMark);if (sql) {const sqlStart = verbEnd + tookWord.length + duration.length + 2; // ": "- builder.add(sqlStart, sqlStart + sql.length, sqlMark);+ if (statements) {+ statements.push({ from: sqlStart, text: sql });+ } else {+ builder.add(sqlStart, sqlStart + sql.length, sqlMark);+ }}}diff --git a/packages/codemirror/src/syntax_highlighting.spec.ts b/packages/codemirror/src/syntax_highlighting.spec.tsindex dffdbaba34..4f2f91aa60 100644--- a/packages/codemirror/src/syntax_highlighting.spec.ts+++ b/packages/codemirror/src/syntax_highlighting.spec.ts@@ -1,7 +1,9 @@import {- ensureSyntaxTree, LanguageSupport, StreamLanguage, type StreamParser+ ensureSyntaxTree, LanguageSupport, StreamLanguage, type StreamParser, syntaxHighlighting} from "@codemirror/language";import { EditorState, type Extension } from "@codemirror/state";+import { EditorView } from "@codemirror/view";+import { classHighlighter } from "@lezer/highlight";import { SupportedMimeTypes } from "@triliumnext/commons";import { describe, expect, it, vi } from "vitest";@@ -51,6 +53,25 @@ describe("byMimeType", () => {expect(byMimeType["application/x-ejs"]).toBeNull();});+ it("highlights the statements of slow queries in the Trilium log as SQLite", async () => {+ const syntax = await byMimeType["text/x-trilium-log"]?.();+ expect(Array.isArray(syntax)).toBe(true);+ if (!Array.isArray(syntax)) return;++ const doc = "00:10:01.540 Slow query took 78ms: SELECT 1";+ const state = EditorState.create({+ doc, extensions: [ syntax, syntaxHighlighting(classHighlighter) ]+ });+ const keywords: string[] = [];+ for (const set of state.facet(EditorView.decorations)) {+ if (typeof set === "function") continue;+ set.between(0, doc.length, (from, to, decoration) => {+ if (decoration.spec.class === "tok-keyword") keywords.push(doc.slice(from, to));+ });+ }+ expect(keywords).toEqual([ "SELECT" ]);+ });+it.each(MARKDOWN_MIMES)("highlights mermaid code blocks in %s", async (mime) => {const syntax = await byMimeType[mime]?.();expect(syntax).toBeTruthy();diff --git a/packages/codemirror/src/syntax_highlighting.ts b/packages/codemirror/src/syntax_highlighting.tsindex 05382329ae..b562fdecef 100644--- a/packages/codemirror/src/syntax_highlighting.ts+++ b/packages/codemirror/src/syntax_highlighting.ts@@ -1,4 +1,6 @@-import { LanguageDescription, LanguageSupport, type StreamParser } from "@codemirror/language";+import {+ type Language, LanguageDescription, LanguageSupport, StreamLanguage, type StreamParser+} from "@codemirror/language";import type { Extension } from "@codemirror/state";import { SupportedMimeTypes } from "@triliumnext/commons";@@ -44,6 +46,15 @@ async function buildJson() {return [ json(), linter(jsonParseLinter()), lintGutter() ];}+/** The backend log, with the statements of slow queries highlighted as SQLite (Trilium) notes. */+async function buildTriliumLog() {+ const [ { triliumLogHighlighter }, sqlLanguage ] = await Promise.all([+ import("./extensions/trilium_log_highlighter.js"),+ loadLanguage("text/x-sqlite;schema=trilium")+ ]);+ return [ triliumLogHighlighter(sqlLanguage) ];+}+const byMimeType: Record<SupportedMimeTypes, (() => Promise<StreamParser<unknown> | LanguageSupport | Extension[]>) | null> = {"text/plain": null,@@ -201,7 +212,7 @@ const byMimeType: Record<SupportedMimeTypes, (() => Promise<StreamParser<unknown"text/x-tiddlywiki": async () => (await import('@codemirror/legacy-modes/mode/tiddlywiki')).tiddlyWiki,"text/x-toml": async () => (await import('@codemirror/legacy-modes/mode/toml')).toml,"text/x-tornado": null,- "text/x-trilium-log": async () => [ (await import('./extensions/trilium_log_highlighter.js')).triliumLogHighlighter ],+ "text/x-trilium-log": buildTriliumLog,"text/x-trilium-search": async () => [ (await import('./extensions/trilium_search_highlighter.js')).triliumSearchHighlighter ],"text/x-ttcn-asn": async () => (await import('@codemirror/legacy-modes/mode/ttcn')).ttcn,"text/x-ttcn-cfg": async () => (await import('@codemirror/legacy-modes/mode/ttcn-cfg')).ttcnCfg,@@ -228,4 +239,18 @@ export const MIME_ALIASES: Record<string, SupportedMimeTypes> = {"image/svg+xml": "text/xml"};+/** Loads the language that highlights `mime` in a code note, or `null` if it has none. */+export async function loadLanguage(mime: string): Promise<Language | null> {+ const load = byMimeType[(MIME_ALIASES[mime] ?? mime) as SupportedMimeTypes];+ if (!load) return null;++ const syntax = await load();+ if ("token" in syntax) return StreamLanguage.define(syntax);++ const extensions = Array.isArray(syntax) ? syntax : [ syntax ];+ return extensions+ .find((extension): extension is LanguageSupport => extension instanceof LanguageSupport)+ ?.language ?? null;+}+export default byMimeType;