Skip to content

Build your personal knowledge base with Trilium Notes

client/backend log: add support for highlighting SQL queries

Adorian Doran5ca811abb76f
diff --git a/apps/server/src/assets/help/help_content.json b/apps/server/src/assets/help/help_content.json
index 458b834c67..f80e5443a1 100644
--- a/apps/server/src/assets/help/help_content.json
+++ b/apps/server/src/assets/help/help_content.json
@@ -188,7 +188,7 @@
"_help_wy8So3yZZlH9": "<p>Go to <a href=\"https://github.com/TriliumNext/Trilium/issues\">Issues · TriliumNext/Trilium</a> and press “New issue”.</p><p>If you are reporting a bug, select “Bug Report” and fill in the details.</p>",
"_help_x59R8J8KV5Bp": "<p><img src=\"{{helpAssets}}/User%20Guide/Troubleshooting/Anonymized%20Database_image.jpg\"></p><p>In certain scenarios, understanding the structure of a database is crucial for troubleshooting issues. However, sharing your actual <a href=\"#root/_help_wX4HbRucYSDD\">database</a> file with personal notes is not advisable. To address this, Trilium offers a feature to anonymize the database. This feature can be accessed via Menu -&gt; Options -&gt; Advanced tab.</p><p>This feature creates a copy of your database with all sensitive data removed. Specifically, it strips out note titles, contents, revisions, history, and some non-system attributes while retaining the overall structure and metadata, such as modification dates. After anonymization, the database undergoes a <a href=\"https://sqlite.org/lang_vacuum.html\">vacuuming process</a> to ensure no sensitive data remnants remain in the file. The anonymized database is saved in the <code>anonymized</code> directory within the <a href=\"#root/_help_tAassRL4RSQL\">data directory</a>, making it safe to share with bug reports.</p><p>This will create a copy of your document and remove all sensitive data (currently note titles, contents, revisions, history and some of the options, and non-system attributes) while leaving all structure and metadata (e.g. date of last change). After this is done, the database is <a href=\"https://sqlite.org/lang_vacuum.html\">VACUUMed</a> to make sure there's no stale sensitive data in the document file. The resulting file is stored in <code>anonymized</code> directory (placed in the <a href=\"#root/_help_tAassRL4RSQL\">data directory</a>). You can safely attach it to a bug report.</p><h2>Command Line Anonymization</h2><p>If your <a href=\"#root/_help_wX4HbRucYSDD\">database</a> is corrupted to the point where Trilium cannot start, the anonymization process can still be executed via the command line:</p><pre><code class=\"language-text-x-trilium-auto\">node src/anonymize.js</code></pre><p>Run this command from the directory containing Trilium's source files, typically found in the <code>resources/app</code> directory for desktop builds.</p>",
"_help_qzNzp9LYQyPT": "<p>It's important to provide all available error logs together with bug reports.</p><p>There are two types of error logs, both of which are useful when reporting bugs.</p><ul><li><a class=\"reference-link\" href=\"#root/_help_bnyigUA2UK7s\">Backend (server) logs</a></li><li><a class=\"reference-link\" href=\"#root/_help_9yEHzMyFirZR\">Frontend logs</a></li></ul><h2>Providing sensitive data</h2><p>If you don't feel comfortable attaching the logs or anything sensitive to the public GitHub issues, feel free to contact the devs in our Matrix <a href=\"https://github.com/TriliumNext/Trilium#-discuss-with-us\">support channel</a>.</p><p>Use this email to also provide anything which could assist in analysing the bug - e.g. files/images/ZIPs being imported or <a href=\"#root/_help_x59R8J8KV5Bp\">anonymized database</a>.</p><h3>Exporting note subtree for reproduction</h3><p>Often times, bugs manifest themselves in specific notes and having them would greatly ease reproduction and fixing.</p><p>In such case, please export the relevant note subtree by right-clicking it on the left tree, choosing Export - HTML as ZIP:</p><p><img src=\"{{helpAssets}}/User%20Guide/Troubleshooting/Error%20logs_error-logs-export-subtree.png\"></p>",
- "_help_bnyigUA2UK7s": "<h2>Accessing via the backend log</h2><p>In the&nbsp;<a class=\"reference-link\" href=\"#root/_help_x3i7MxGccDuM\">Global menu</a>, go to <em>Advanced</em> → <em>Show backend log</em>. This will display the current backend log (i.e. today's), with the historical information available only on disk (see below).</p><p>Interaction since v0.104.0:</p><ul><li>The file can be downloaded as a text file using the&nbsp;<span class=\"tn-icon bx bx-download\"></span> button in the&nbsp;<a class=\"reference-link\" href=\"#root/_help_8YBEPzcpUgxw\">Note menu</a>&nbsp;area.</li><li>The file can also be searched just like a normal&nbsp;<a class=\"reference-link\" href=\"#root/_help_6f9hih2hXXZk\">Code</a>&nbsp;note.</li></ul><h2>Location on the disk</h2><p>Backend logs are stored on the file system. To find them, open the&nbsp;<a class=\"reference-link\" href=\"#root/_help_tAassRL4RSQL\">Data directory</a>, go to the <code>log</code> subdirectory and find the latest log file, e.g. <code>trilium-2022-12-14.log</code>.&nbsp;</p><h2>In the terminal</h2><p>The server also prints every log entry to its standard output, which is what <code>docker logs</code> or <code>journalctl</code> show. When that output is a terminal, each entry starts with its time and is colored the same way as the backend log in the app: request status codes by class, slow SQL queries, and errors in red. The startup banner with the Trilium logo and version information is colored too. Output to a file, a pipe or a container log stays plain text without the time, since those tools add their own.</p><p>To turn the colors off in a terminal, set the <code>NO_COLOR</code> environment variable; to force them on elsewhere, set <code>FORCE_COLOR=1</code>.</p><h2>Reporting backend bugs</h2><p>You can attach the whole file to the bug report (preferable) or open it and copy-paste only the last lines / lines you believe are relevant.</p><h2>Customizing the retention of logs</h2><p>The backend logs are fully managed by the Trilium server. By default the last 90 days worth of logs are kept; the logs older than that are deleted in order to reduce the space consumption.</p><p>It's possible to change the retention period by modifying the&nbsp;<a class=\"reference-link\" href=\"#root/_help_Gzjqa934BdH4\">Configuration (config.ini or environment variables)</a>&nbsp;via the <code>.ini</code> file:</p><pre><code class=\"language-text-x-trilium-auto\">[Logging]\nretentionDays=7</code></pre><p>Or via the environment variable <code>TRILIUM_LOGGING_RETENTION_DAYS</code>.</p><p>Special cases:</p><ul><li>Positive values indicate the number of days worth of logs to keep</li><li>A value of 0 results with the default value (90 days) to be used</li><li>Negative values (e.g. <code>-1</code>) result with all logs to be kept, irrespective how ancient and numerous (and</li></ul><aside class=\"admonition note\"><p>If you set the retention days to a low number, you might notice that not all the log files are being deleted. This is because a minimum number of logs (7 at the time of writing) is maintained at all times.</p></aside>",
+ "_help_bnyigUA2UK7s": "<h2>Accessing via the backend log</h2><p>In the&nbsp;<a class=\"reference-link\" href=\"#root/_help_x3i7MxGccDuM\">Global menu</a>, go to <em>Advanced</em> → <em>Show backend log</em>. This will display the current backend log (i.e. today's), with the historical information available only on disk (see below).</p><p>The log is colored to make it easier to read: request status codes by class, slow SQL queries, and errors in red. The statement of a slow SQL query is highlighted the same way as in the&nbsp;<a class=\"reference-link\" href=\"#root/_help_YKWqdJhzi2VY\">SQL Console</a>, using the color scheme of code notes (see <em>Color schemes</em> in&nbsp;<a class=\"reference-link\" href=\"#root/_help_6f9hih2hXXZk\">Code</a>).</p><p>Interaction since v0.104.0:</p><ul><li>The file can be downloaded as a text file using the&nbsp;<span class=\"tn-icon bx bx-download\"></span> button in the&nbsp;<a class=\"reference-link\" href=\"#root/_help_8YBEPzcpUgxw\">Note menu</a>&nbsp;area.</li><li>The file can also be searched just like a normal&nbsp;<a class=\"reference-link\" href=\"#root/_help_6f9hih2hXXZk\">Code</a>&nbsp;note.</li></ul><h2>Location on the disk</h2><p>Backend logs are stored on the file system. To find them, open the&nbsp;<a class=\"reference-link\" href=\"#root/_help_tAassRL4RSQL\">Data directory</a>, go to the <code>log</code> subdirectory and find the latest log file, e.g. <code>trilium-2022-12-14.log</code>.&nbsp;</p><h2>In the terminal</h2><p>The server also prints every log entry to its standard output, which is what <code>docker logs</code> or <code>journalctl</code> show. When that output is a terminal, each entry starts with its time and is colored like the backend log in the app: request status codes by class, slow SQL queries, and errors in red. The statement of a slow SQL query is shown in a single color there. The startup banner with the Trilium logo and version information is colored too. Output to a file, a pipe or a container log stays plain text without the time, since those tools add their own.</p><p>To turn the colors off in a terminal, set the <code>NO_COLOR</code> environment variable; to force them on elsewhere, set <code>FORCE_COLOR=1</code>.</p><h2>Reporting backend bugs</h2><p>You can attach the whole file to the bug report (preferable) or open it and copy-paste only the last lines / lines you believe are relevant.</p><h2>Customizing the retention of logs</h2><p>The backend logs are fully managed by the Trilium server. By default the last 90 days worth of logs are kept; the logs older than that are deleted in order to reduce the space consumption.</p><p>It's possible to change the retention period by modifying the&nbsp;<a class=\"reference-link\" href=\"#root/_help_Gzjqa934BdH4\">Configuration (config.ini or environment variables)</a>&nbsp;via the <code>.ini</code> file:</p><pre><code class=\"language-text-x-trilium-auto\">[Logging]\nretentionDays=7</code></pre><p>Or via the environment variable <code>TRILIUM_LOGGING_RETENTION_DAYS</code>.</p><p>Special cases:</p><ul><li>Positive values indicate the number of days worth of logs to keep</li><li>A value of 0 results with the default value (90 days) to be used</li><li>Negative values (e.g. <code>-1</code>) result with all logs to be kept, irrespective how ancient and numerous (and</li></ul><aside class=\"admonition note\"><p>If you set the retention days to a low number, you might notice that not all the log files are being deleted. This is because a minimum number of logs (7 at the time of writing) is maintained at all times.</p></aside>",
"_help_9yEHzMyFirZR": "<p>To provide frontend logs, we need to open the Developer Console. Often the easiest way is to press <kbd>Ctrl</kbd>-<kbd>Shift</kbd>-<kbd>I</kbd> which should work in most browsers (and desktop app). Make sure that the error producing action happened right before you copy&amp;paste the errors, the console is cleared on app restart.</p><p>If that doesn't work, then:</p><ul><li>in Trilium desktop app, go to top-left menu button -&gt; Advanced -&gt; Open dev tools</li><li>In Firefox/Chrome right-click anywhere in the page and click Inspect:</li></ul><p><img src=\"{{helpAssets}}/User%20Guide/Troubleshooting/Error%20logs/Frontend%20logs_error-logs-firefox-context-men.png\"></p><p>Once you have Dev Tools open, click on \"Console\" tab:</p><p><img src=\"{{helpAssets}}/User%20Guide/Troubleshooting/Error%20logs/Frontend%20logs_image.jpg\"></p><p>Copy-paste (or screenshot) the logs. It's better to provide not just errors, but the whole log, which might provide context while analyzing the bug.</p>",
"_help_vdlYGAcpXAgc": "<p>Synchronization can sometimes take a long amount of time in order to compute the items that require update. When running behind a reverse proxy, the request can time out.</p><p>The solution is to increase the timeout at proxy level.</p><h2>Nginx</h2><p>Add the following to the configuration file:</p><pre><code class=\"language-text-x-nginx-conf\">proxy_connect_timeout 300;\nproxy_send_timeout 300;\nproxy_read_timeout 300;\nsend_timeout 300;</code></pre><p>And restart the server.</p><p>See&nbsp;<a href=\"#root/_help_ud6MShXL4WpO\">Nginx Proxy Setup</a>&nbsp;for more information about the Nginx setup.</p><p>If it still doesn't work, try increasing the timeout.</p>",
"_help_s8alTXmpFR61": "<p>Some changes to the application will not take effect immediately and as such it might require to manually reload or refresh the application.</p><p>To do so, simply press <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>R</kbd>.</p>",
diff --git a/docs/User Guide/User Guide/Troubleshooting/Error logs/Backend (server) logs.md b/docs/User Guide/User Guide/Troubleshooting/Error logs/Backend (server) logs.md
index ef8cd11f71..44837849c6 100644
--- a/docs/User Guide/User Guide/Troubleshooting/Error logs/Backend (server) logs.md
+++ b/docs/User Guide/User Guide/Troubleshooting/Error logs/Backend (server) logs.md
@@ -3,6 +3,8 @@
In the <a class="reference-link" href="../../Basic%20Concepts%20and%20Features/UI%20Elements/Global%20menu.md">Global menu</a>, go to _Advanced_ → _Show backend log_. This will display the current backend log (i.e. today's), with the historical information available only on disk (see below).
+The log is colored to make it easier to read: request status codes by class, slow SQL queries, and errors in red. The statement of a slow SQL query is highlighted the same way as in the <a class="reference-link" href="../../Advanced%20Usage/Database/Manually%20altering%20the%20database/SQL%20Console.md">SQL Console</a>, using the color scheme of code notes (see _Color schemes_ in <a class="reference-link" href="../../Note%20Types/Code.md">Code</a>).
+
Interaction since v0.104.0:
* The file can be downloaded as a text file using the <span class="tn-icon bx bx-download"></span> button in the <a class="reference-link" href="../../Basic%20Concepts%20and%20Features/UI%20Elements/Note%20menu.md">Note menu</a> area.
@@ -14,7 +16,7 @@ Backend logs are stored on the file system. To find them, open the <a class="re
## In the terminal
-The server also prints every log entry to its standard output, which is what `docker logs` or `journalctl` show. When that output is a terminal, each entry starts with its time and is colored the same way as the backend log in the app: request status codes by class, slow SQL queries, and errors in red. The startup banner with the Trilium logo and version information is colored too. Output to a file, a pipe or a container log stays plain text without the time, since those tools add their own.
+The server also prints every log entry to its standard output, which is what `docker logs` or `journalctl` show. When that output is a terminal, each entry starts with its time and is colored like the backend log in the app: request status codes by class, slow SQL queries, and errors in red. The statement of a slow SQL query is shown in a single color there. The startup banner with the Trilium logo and version information is colored too. Output to a file, a pipe or a container log stays plain text without the time, since those tools add their own.
To turn the colors off in a terminal, set the `NO_COLOR` environment variable; to force them on elsewhere, set `FORCE_COLOR=1`.
diff --git a/packages/codemirror/src/code_preview.ts b/packages/codemirror/src/code_preview.ts
index 7391e12d6d..4c1ea1a8cf 100644
--- a/packages/codemirror/src/code_preview.ts
+++ b/packages/codemirror/src/code_preview.ts
@@ -1,13 +1,10 @@
-import {
- ensureSyntaxTree, type Language, LanguageSupport, StreamLanguage
-} from "@codemirror/language";
+import { ensureSyntaxTree, type Language } from "@codemirror/language";
import { EditorState } from "@codemirror/state";
import { highlightCode, type Highlighter, tagHighlighter } from "@lezer/highlight";
-import type { SupportedMimeTypes } from "@triliumnext/commons";
import { getThemeById } from "./color_themes.js";
import { type PreviewTheme, type PreviewThemeSlot, readPreviewTheme } from "./preview_theme.js";
-import byMimeType, { MIME_ALIASES } from "./syntax_highlighting.js";
+import { loadLanguage } from "./syntax_highlighting.js";
export {
buildCodePreviewCss, type PreviewTheme, type PreviewThemeColors, type PreviewThemeSlot,
@@ -82,7 +79,7 @@ export function loadPreviewTheme(id: string) {
export function loadPreviewLanguage(mime: string) {
let language = languages.get(mime);
if (!language) {
- language = readLanguage(mime).then((result) => {
+ language = loadLanguage(mime).then((result) => {
loadedLanguages.set(mime, result);
return result;
});
@@ -123,19 +120,6 @@ function getHighlighter({ theme, key }: PreviewThemeSlot): Highlighter {
})));
}
-async function readLanguage(mime: string): Promise<Language | null> {
- const load = byMimeType[(MIME_ALIASES[mime] ?? mime) as SupportedMimeTypes];
- if (!load) return null;
-
- const syntax = await load();
- if ("token" in syntax) return StreamLanguage.define(syntax);
-
- const extensions = Array.isArray(syntax) ? syntax : [ syntax ];
- return extensions
- .find((extension): extension is LanguageSupport => extension instanceof LanguageSupport)
- ?.language ?? null;
-}
-
function escapeHtml(text: string) {
return text.replace(/[&<>"]/g, (c) => HTML_ESCAPES[c]);
}
diff --git a/packages/codemirror/src/extensions/trilium_log_highlighter.spec.ts b/packages/codemirror/src/extensions/trilium_log_highlighter.spec.ts
index c52787cfd6..166770284e 100644
--- a/packages/codemirror/src/extensions/trilium_log_highlighter.spec.ts
+++ b/packages/codemirror/src/extensions/trilium_log_highlighter.spec.ts
@@ -1,9 +1,14 @@
-import { Compartment, EditorState } from "@codemirror/state";
+import { HighlightStyle, StreamLanguage, syntaxHighlighting } from "@codemirror/language";
+import { sqlite } from "@codemirror/legacy-modes/mode/sql";
+import { Compartment, EditorState, type Extension } from "@codemirror/state";
import { EditorView } from "@codemirror/view";
+import { classHighlighter, tags } from "@lezer/highlight";
import { describe, expect, it } from "vitest";
import { MAX_HIGHLIGHTED_LINES, triliumLogHighlighter } from "./trilium_log_highlighter.js";
+const sqlLanguage = StreamLanguage.define(sqlite);
+
interface DecorationSpan {
from: number;
to: number;
@@ -21,7 +26,8 @@ describe("triliumLogHighlighter", () => {
const before = EditorState.create({ doc, extensions: [ compartment.of([]) ] });
expect(decorationsOf(before)).toHaveLength(0);
- const after = before.update({ effects: compartment.reconfigure(triliumLogHighlighter) }).state;
+ const effects = compartment.reconfigure(triliumLogHighlighter());
+ const after = before.update({ effects }).state;
expect(decorationsOf(after).length).toBeGreaterThan(0);
});
@@ -188,10 +194,102 @@ describe("triliumLogHighlighter", () => {
expect(timestamps.length).toBeGreaterThan(0);
expect(timestamps.length).toBeLessThanOrEqual(MAX_HIGHLIGHTED_LINES);
});
+
+ describe("with a SQL language", () => {
+ it("highlights the tokens of a slow query's statement", () => {
+ const doc = "00:10:01.540 Slow query took 78ms: SELECT noteId FROM notes"
+ + " WHERE title LIKE '%a%' AND isDeleted = 0 AND parentNoteId = ?";
+ const spans = decorationsOf(sqlState(doc));
+
+ expect(slicesOf(doc, spans, "tok-keyword"))
+ .toEqual([ "SELECT", "FROM", "WHERE", "LIKE", "AND", "AND" ]);
+ expect(slicesOf(doc, spans, "tok-string")).toEqual([ "'%a%'" ]);
+ // The `78` of `took 78ms` is outside the statement.
+ expect(slicesOf(doc, spans, "tok-number")).toEqual([ "0" ]);
+ expect(sliceOf(doc, spans, "cm-log-query-verb")).toBe("Slow query");
+
+ // Identifiers and placeholders keep the main text color.
+ expect(classesOfState(sqlState(doc))).toContain("cm-log-query");
+ expect(classesOfState(sqlState(doc))).not.toContain("cm-log-sql");
+ const placeholder = doc.indexOf("?");
+ expect(spans.filter((d) => d.from <= placeholder && d.to > placeholder)).toEqual([]);
+ });
+
+ it("tokenizes nothing on recursive queries, which have no statement", () => {
+ const doc = [
+ "00:10:01.540 Slow query took 30ms: SELECT 1",
+ "00:10:02.113 Slow recursive query took 45ms."
+ ].join("\n");
+ const tokens = decorationsOf(sqlState(doc)).filter((d) => d.class.startsWith("tok-"));
+
+ expect(tokens.map((d) => doc.slice(d.from, d.to))).toEqual([ "SELECT", "1" ]);
+ });
+
+ it("reclassifies the tokens when the theme changes, without rescanning the log", () => {
+ const doc = "00:10:01.540 Slow query took 78ms: SELECT 1";
+ const theme = new Compartment();
+ let s = sqlState(doc, theme.of(syntaxHighlighting(classHighlighter)));
+ const logDecorations = logDecorationsOf(s);
+ expect(sliceOf(doc, decorationsOf(s), "tok-keyword")).toBe("SELECT");
+
+ const style = HighlightStyle.define([ { tag: tags.keyword, class: "keyword" } ]);
+ s = s.update({ effects: theme.reconfigure(syntaxHighlighting(style)) }).state;
+
+ expect(sliceOf(doc, decorationsOf(s), "keyword")).toBe("SELECT");
+ expect(classesOfState(s)).not.toContain("tok-keyword");
+ expect(logDecorationsOf(s)).toBe(logDecorations);
+ });
+
+ it("keeps every decoration set unless the document or the configuration changes", () => {
+ const doc = "00:10:01.540 Slow query took 78ms: SELECT 1";
+ for (const s of [ state(doc), sqlState(doc) ]) {
+ const sets = s.facet(EditorView.decorations);
+ const next = s.update({ selection: { anchor: 1 } }).state
+ .facet(EditorView.decorations);
+ expect(next).toHaveLength(sets.length);
+ expect(next.every((set, index) => set === sets[index])).toBe(true);
+ }
+
+ // Without a language there are no SQL marks to reclassify.
+ const compartment = new Compartment();
+ const plain = EditorState.create({
+ doc, extensions: [ triliumLogHighlighter(), compartment.of([]) ]
+ });
+ const reconfigured = plain.update({
+ effects: compartment.reconfigure(syntaxHighlighting(classHighlighter))
+ }).state;
+ expect(logDecorationsOf(reconfigured)).toBe(logDecorationsOf(plain));
+ expect(classesOfState(reconfigured)).toContain("cm-log-sql");
+ });
+
+ it("only tokenizes statements within the last MAX_HIGHLIGHTED_LINES lines", () => {
+ const lines = [ "17:00:00.000 Slow query took 30ms: SELECT 1" ];
+ for (let i = 0; i < MAX_HIGHLIGHTED_LINES; i++) {
+ lines.push("18:00:00.000 Backend script execution is DISABLED.");
+ }
+ lines.push("19:00:00.000 Slow query took 30ms: SELECT 2");
+ const doc = lines.join("\n");
+ const spans = decorationsOf(sqlState(doc));
+
+ expect(slicesOf(doc, spans, "tok-keyword")).toEqual([ "SELECT" ]);
+ expect(slicesOf(doc, spans, "tok-number")).toEqual([ "2" ]);
+ });
+ });
});
function state(doc: string): EditorState {
- return EditorState.create({ doc, extensions: [ triliumLogHighlighter ] });
+ return EditorState.create({ doc, extensions: [ triliumLogHighlighter() ] });
+}
+
+function sqlState(doc: string, highlighting: Extension = syntaxHighlighting(classHighlighter)) {
+ return EditorState.create({
+ doc, extensions: [ triliumLogHighlighter(sqlLanguage), highlighting ]
+ });
+}
+
+/** The first decoration set, which holds the log's own decorations. */
+function logDecorationsOf(editorState: EditorState) {
+ return editorState.facet(EditorView.decorations).find((set) => typeof set !== "function");
}
/** All decoration ranges contributed to the view's decoration facet, in document order. */
@@ -222,6 +320,11 @@ function sliceOf(doc: string, spans: DecorationSpan[], cls: string): string | un
return span && doc.slice(span.from, span.to);
}
+/** The document text covered by each span carrying `cls`, in document order. */
+function slicesOf(doc: string, spans: DecorationSpan[], cls: string): string[] {
+ return spans.filter((d) => d.class === cls).map((d) => doc.slice(d.from, d.to));
+}
+
/** Classes of the line decorations (zero-length, at the line start) on line `lineNo` (1-based). */
function lineClassesAt(editorState: EditorState, lineNo: number): string[] {
const from = editorState.doc.line(lineNo).from;
diff --git a/packages/codemirror/src/extensions/trilium_log_highlighter.ts b/packages/codemirror/src/extensions/trilium_log_highlighter.ts
index 9d2c67a422..45cca27db2 100644
--- a/packages/codemirror/src/extensions/trilium_log_highlighter.ts
+++ b/packages/codemirror/src/extensions/trilium_log_highlighter.ts
@@ -1,5 +1,7 @@
+import { highlightingFor, type Language } from "@codemirror/language";
import { type EditorState, type Extension, RangeSetBuilder, StateField } from "@codemirror/state";
import { Decoration, type DecorationSet, EditorView } from "@codemirror/view";
+import { type Highlighter, highlightTree } from "@lezer/highlight";
/**
* Lightweight regex highlighter for the Trilium backend log. Each log entry is a single line
@@ -20,7 +22,8 @@ import { Decoration, type DecorationSet, EditorView } from "@codemirror/view";
* `Slow` marker (prepended by the server for requests taking >= 10ms) is flagged as a warning.
* - slow SQL query lines `<ts> Slow [recursive ]query took <n>ms[: <sql>]` — the whole line is
* tinted, the entire `Slow [recursive ]query` phrase bolded as the entry's verb (its own colour,
- * not the HTTP method's), and the statement (absent for recursive queries) coloured. Unlike HTTP,
+ * not the HTTP method's), and the statement (absent for recursive queries) highlighted with the
+ * SQL language in the active code theme, or in one colour without a language. Unlike HTTP,
* `Slow` is not flagged as a warning here: queries are only logged when slow, so it is uninformative.
* - `<ts> JS Error: …` and `<ts> ERROR: …` lines — coloured red, together with their following
* continuation lines (e.g. wrapped stack traces), which carry no timestamp of their own
@@ -72,6 +75,8 @@ const statusMarks: Record<string, Decoration> = {
"4": Decoration.mark({ class: "cm-log-status cm-log-status-4xx" }),
"5": Decoration.mark({ class: "cm-log-status cm-log-status-5xx" })
};
+/** Marks of the SQL tokens, one per class string of the code theme. */
+const sqlTokenMarks = new Map<string, Decoration>();
/**
* The Next themes own the palette (`--log-*` in theme-next-light.css / theme-next-dark.css). The
@@ -101,21 +106,49 @@ const logHighlightTheme = EditorView.baseTheme({
".cm-log-info": { color: "var(--log-info-color, inherit)" }
});
-const logHighlightField = StateField.define<DecorationSet>({
- create(state) {
- return buildLogDecorations(state);
- },
- update(decorations, tr) {
- return tr.docChanged ? buildLogDecorations(tr.state) : decorations;
- },
- provide: (field) => EditorView.decorations.from(field)
-});
+/** The statement of a slow query and its start offset in the document. */
+interface SqlStatement {
+ from: number;
+ text: string;
+}
-/** The extension to register on the backend-log editor (e.g. via a `text/x-trilium-log` MIME type). */
-export const triliumLogHighlighter: Extension = [ logHighlightField, logHighlightTheme ];
+interface LogHighlights {
+ log: DecorationSet;
+ /** The statements in the highlighted lines; empty without a SQL language. */
+ statements: SqlStatement[];
+ /** The tokens of `statements`, with the classes of the active code theme. */
+ sql: DecorationSet;
+}
+
+/**
+ * The extension to register on the backend-log editor (e.g. via a `text/x-trilium-log` MIME type).
+ * `sqlLanguage` highlights the statements of slow queries; without it, a statement gets one color.
+ */
+export function triliumLogHighlighter(sqlLanguage: Language | null = null): Extension {
+ const field = StateField.define<LogHighlights>({
+ create: (state) => buildHighlights(state, sqlLanguage),
+ update(highlights, tr) {
+ if (tr.docChanged) return buildHighlights(tr.state, sqlLanguage);
+ // A new code theme changes the token classes. `performFind()` also reconfigures, on
+ // every search, so a reconfigure rebuilds only the SQL marks.
+ if (tr.reconfigured && sqlLanguage) {
+ const sql = buildSqlDecorations(tr.state, highlights.statements, sqlLanguage);
+ return { ...highlights, sql };
+ }
+ return highlights;
+ },
+ provide: (f) => [
+ EditorView.decorations.from(f, (highlights) => highlights.log),
+ EditorView.decorations.from(f, (highlights) => highlights.sql)
+ ]
+ });
+ return [ field, logHighlightTheme ];
+}
-function buildLogDecorations(state: EditorState): DecorationSet {
+function buildHighlights(state: EditorState, sqlLanguage: Language | null): LogHighlights {
const builder = new RangeSetBuilder<Decoration>();
+ const statements: SqlStatement[] = [];
+ const collectedStatements = sqlLanguage ? statements : null;
const doc = state.doc;
// Scan at most the last MAX_HIGHLIGHTED_LINES lines. A single top-to-bottom pass over that window
@@ -125,12 +158,39 @@ function buildLogDecorations(state: EditorState): DecorationSet {
const firstLine = Math.max(1, doc.lines - MAX_HIGHLIGHTED_LINES + 1);
let blockLine: Decoration | null = null;
for (let n = firstLine; n <= doc.lines; n++) {
- blockLine = decorateLine(builder, doc.line(n), blockLine);
+ blockLine = decorateLine(builder, doc.line(n), blockLine, collectedStatements);
}
+ return {
+ log: builder.finish(),
+ statements,
+ sql: sqlLanguage ? buildSqlDecorations(state, statements, sqlLanguage) : Decoration.none
+ };
+}
+
+/** Marks the tokens of each statement with the classes that the active code theme gives them. */
+function buildSqlDecorations(
+ state: EditorState, statements: SqlStatement[], language: Language
+): DecorationSet {
+ const builder = new RangeSetBuilder<Decoration>();
+ const highlighter: Highlighter = { style: (tags) => highlightingFor(state, tags) };
+ for (const { from, text } of statements) {
+ highlightTree(language.parser.parse(text), highlighter, (start, end, classes) => {
+ builder.add(from + start, from + end, sqlTokenMark(classes));
+ });
+ }
return builder.finish();
}
+function sqlTokenMark(classes: string) {
+ let mark = sqlTokenMarks.get(classes);
+ if (!mark) {
+ mark = Decoration.mark({ class: classes });
+ sqlTokenMarks.set(classes, mark);
+ }
+ return mark;
+}
+
/**
* Returns the line decoration a multi-line block entry applies to itself and its continuation lines
* (e.g. `errorLine` for `JS Error:`/`ERROR:`, `infoLine` for `JS Info:`), or null for entries that
@@ -145,9 +205,15 @@ function blockLineFor(text: string): Decoration | null {
/**
* Decorates a single line and returns the block decoration in effect for the following
* (continuation) lines. A timestamped line opens a block (or closes any open one); a line without a
- * timestamp inherits the incoming `blockLine`.
+ * timestamp inherits the incoming `blockLine`. The statement of a slow query goes into `statements`
+ * for SQL highlighting, or gets `sqlMark` if `statements` is null.
*/
-function decorateLine(builder: RangeSetBuilder<Decoration>, line: { from: number; text: string }, blockLine: Decoration | null): Decoration | null {
+function decorateLine(
+ builder: RangeSetBuilder<Decoration>,
+ line: { from: number; text: string },
+ blockLine: Decoration | null,
+ statements: SqlStatement[] | null
+): Decoration | null {
const { from: lineStart, text } = line;
if (!TIMESTAMP.test(text)) {
@@ -202,7 +268,11 @@ function decorateLine(builder: RangeSetBuilder<Decoration>, line: { from: number
builder.add(verbStart, verbEnd, queryVerbMark);
if (sql) {
const sqlStart = verbEnd + tookWord.length + duration.length + 2; // ": "
- builder.add(sqlStart, sqlStart + sql.length, sqlMark);
+ if (statements) {
+ statements.push({ from: sqlStart, text: sql });
+ } else {
+ builder.add(sqlStart, sqlStart + sql.length, sqlMark);
+ }
}
}
diff --git a/packages/codemirror/src/syntax_highlighting.spec.ts b/packages/codemirror/src/syntax_highlighting.spec.ts
index dffdbaba34..4f2f91aa60 100644
--- a/packages/codemirror/src/syntax_highlighting.spec.ts
+++ b/packages/codemirror/src/syntax_highlighting.spec.ts
@@ -1,7 +1,9 @@
import {
- ensureSyntaxTree, LanguageSupport, StreamLanguage, type StreamParser
+ ensureSyntaxTree, LanguageSupport, StreamLanguage, type StreamParser, syntaxHighlighting
} from "@codemirror/language";
import { EditorState, type Extension } from "@codemirror/state";
+import { EditorView } from "@codemirror/view";
+import { classHighlighter } from "@lezer/highlight";
import { SupportedMimeTypes } from "@triliumnext/commons";
import { describe, expect, it, vi } from "vitest";
@@ -51,6 +53,25 @@ describe("byMimeType", () => {
expect(byMimeType["application/x-ejs"]).toBeNull();
});
+ it("highlights the statements of slow queries in the Trilium log as SQLite", async () => {
+ const syntax = await byMimeType["text/x-trilium-log"]?.();
+ expect(Array.isArray(syntax)).toBe(true);
+ if (!Array.isArray(syntax)) return;
+
+ const doc = "00:10:01.540 Slow query took 78ms: SELECT 1";
+ const state = EditorState.create({
+ doc, extensions: [ syntax, syntaxHighlighting(classHighlighter) ]
+ });
+ const keywords: string[] = [];
+ for (const set of state.facet(EditorView.decorations)) {
+ if (typeof set === "function") continue;
+ set.between(0, doc.length, (from, to, decoration) => {
+ if (decoration.spec.class === "tok-keyword") keywords.push(doc.slice(from, to));
+ });
+ }
+ expect(keywords).toEqual([ "SELECT" ]);
+ });
+
it.each(MARKDOWN_MIMES)("highlights mermaid code blocks in %s", async (mime) => {
const syntax = await byMimeType[mime]?.();
expect(syntax).toBeTruthy();
diff --git a/packages/codemirror/src/syntax_highlighting.ts b/packages/codemirror/src/syntax_highlighting.ts
index 05382329ae..b562fdecef 100644
--- a/packages/codemirror/src/syntax_highlighting.ts
+++ b/packages/codemirror/src/syntax_highlighting.ts
@@ -1,4 +1,6 @@
-import { LanguageDescription, LanguageSupport, type StreamParser } from "@codemirror/language";
+import {
+ type Language, LanguageDescription, LanguageSupport, StreamLanguage, type StreamParser
+} from "@codemirror/language";
import type { Extension } from "@codemirror/state";
import { SupportedMimeTypes } from "@triliumnext/commons";
@@ -44,6 +46,15 @@ async function buildJson() {
return [ json(), linter(jsonParseLinter()), lintGutter() ];
}
+/** The backend log, with the statements of slow queries highlighted as SQLite (Trilium) notes. */
+async function buildTriliumLog() {
+ const [ { triliumLogHighlighter }, sqlLanguage ] = await Promise.all([
+ import("./extensions/trilium_log_highlighter.js"),
+ loadLanguage("text/x-sqlite;schema=trilium")
+ ]);
+ return [ triliumLogHighlighter(sqlLanguage) ];
+}
+
const byMimeType: Record<SupportedMimeTypes, (() => Promise<StreamParser<unknown> | LanguageSupport | Extension[]>) | null> = {
"text/plain": null,
@@ -201,7 +212,7 @@ const byMimeType: Record<SupportedMimeTypes, (() => Promise<StreamParser<unknown
"text/x-tiddlywiki": async () => (await import('@codemirror/legacy-modes/mode/tiddlywiki')).tiddlyWiki,
"text/x-toml": async () => (await import('@codemirror/legacy-modes/mode/toml')).toml,
"text/x-tornado": null,
- "text/x-trilium-log": async () => [ (await import('./extensions/trilium_log_highlighter.js')).triliumLogHighlighter ],
+ "text/x-trilium-log": buildTriliumLog,
"text/x-trilium-search": async () => [ (await import('./extensions/trilium_search_highlighter.js')).triliumSearchHighlighter ],
"text/x-ttcn-asn": async () => (await import('@codemirror/legacy-modes/mode/ttcn')).ttcn,
"text/x-ttcn-cfg": async () => (await import('@codemirror/legacy-modes/mode/ttcn-cfg')).ttcnCfg,
@@ -228,4 +239,18 @@ export const MIME_ALIASES: Record<string, SupportedMimeTypes> = {
"image/svg+xml": "text/xml"
};
+/** Loads the language that highlights `mime` in a code note, or `null` if it has none. */
+export async function loadLanguage(mime: string): Promise<Language | null> {
+ const load = byMimeType[(MIME_ALIASES[mime] ?? mime) as SupportedMimeTypes];
+ if (!load) return null;
+
+ const syntax = await load();
+ if ("token" in syntax) return StreamLanguage.define(syntax);
+
+ const extensions = Array.isArray(syntax) ? syntax : [ syntax ];
+ return extensions
+ .find((extension): extension is LanguageSupport => extension instanceof LanguageSupport)
+ ?.language ?? null;
+}
+
export default byMimeType;