Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

MCP semantic search via OpenAI-compatible embeddings (#1668)

mike-inkeep730285009808
diff --git a/.changeset/mcp-semantic-search-openai-embeddings.md b/.changeset/mcp-semantic-search-openai-embeddings.md
new file mode 100644
index 000000000..a624b5043
--- /dev/null
+++ b/.changeset/mcp-semantic-search-openai-embeddings.md
@@ -0,0 +1,9 @@
+---
+"@inkeep/open-knowledge": minor
+---
+
+Add opt-in semantic search to the MCP `search` tool, powered by a remote OpenAI-compatible embeddings API. When enabled, an embeddings signal is fused into `full_text` ranking so an agent's query surfaces conceptually-related pages even when they share no keywords (e.g. "auth retries" → a page titled "Session Token Refresh"). It is additive and **off by default**: with the flag off, search is byte-for-byte today's lexical ranking, and the cmd-K omnibar always stays purely lexical and instant.
+
+**Privacy / egress.** When the feature is enabled **and** a key is set, the search query and matching page content are sent to the configured embeddings provider. Egress is explicit and scoped: only corpus content is embedded — anything excluded by `.okignore` / `.gitignore` is never sent — embedding is lazy (nothing leaves the machine until an agent actually runs a semantic search), and the key lives only in a 0600 `~/.ok/secrets.yml` file (never in config, logs, or telemetry). If the key is missing, the provider errors, or you're offline, search silently degrades to lexical and never blocks.
+
+Setup (per machine): `ok embeddings set-key` stores the provider key in `~/.ok/secrets.yml` (0600); set `search.semantic.enabled: true` in project-local config. `ok embeddings status` shows capability. Provider base URL / model / dimensions are configurable (Azure / self-hosted / other). Vectors cache incrementally under `.ok/local/` (content-addressed, keyed by provider + model + dimensions). See the "Semantic search" section of the configuration reference for full setup.
diff --git a/docs/content/reference/configuration.mdx b/docs/content/reference/configuration.mdx
index c8d081f52..c7aafd638 100644
--- a/docs/content/reference/configuration.mdx
+++ b/docs/content/reference/configuration.mdx
@@ -36,6 +36,10 @@ The **Scope** column says which file a setting belongs in: **project** → `.ok/
| `telemetry.localSink.spans.maxBytes` | number | `52428800` (~50 MB) | project | Max size of the local diagnostic spans file before it rotates. |
| `telemetry.localSink.logs.maxBytes` | number | `26214400` (~25 MB) | project | Max size of the local diagnostic logs file before it rotates. |
| `telemetry.localSink.attributeDenylist` | `string[]` | 8 credential keys | project | Attribute keys whose values are redacted (`[REDACTED]`) before any local span/log is written. Extends the built-in denylist (`authorization`, `password`, `cookie`, etc.). |
+| `search.semantic.enabled` | boolean | `false` | project-local | Add embeddings-based semantic ranking to the MCP `search` tool. Default off. **When on and a key is set, the search query and matching page content are sent to the configured embeddings provider** (content egress). See [Semantic search](#semantic-search) below. |
+| `search.semantic.baseUrl` | string | `"https://api.openai.com/v1"` | project-local | Base URL of the OpenAI-compatible embeddings API. Override for Azure / self-hosted / other providers. The API key is **not** stored here — set it with `ok embeddings set-key`. |
+| `search.semantic.model` | string | `"text-embedding-3-small"` | project-local | Embeddings model id. Must be served by the provider at `baseUrl`. Changing it re-embeds the corpus. |
+| `search.semantic.dimensions` | number | (native) | project-local | Optional output vector size. Omit for the model's native size (1536 for `text-embedding-3-small`). Set a smaller value to shrink the on-disk cache, trading a little quality. |
## Folder properties
@@ -69,6 +73,23 @@ Most users never set these — the schema settings above cover the common cases.
| `OPEN_KNOWLEDGE_GITHUB_CLIENT_ID` | Override the GitHub OAuth App client ID |
| `OK_SHOWALL_MAX_ENTRIES` | Max entries the **Show all files** disk walk returns before it stops and marks the list truncated (default `50000`) |
| `OTEL_SDK_DISABLED` | OpenTelemetry OTLP push gate. Inverted sense: set to `false` to **enable** push (it is off by default). |
+| `OK_EMBEDDINGS_API_KEY` | Fallback embeddings provider API key for [semantic search](#semantic-search), used when none is stored on disk. Prefer `ok embeddings set-key` (writes `~/.ok/secrets.yml`) for normal use; the env var is convenient for CI / scripted runs. |
+
+## Semantic search
+
+The MCP `search` tool can fuse an **embeddings-based semantic signal** into its ranking, so an agent's query surfaces conceptually-related pages even when they share no keywords (a query about "auth retries" can surface a page titled "Session Token Refresh"). It is **off by default** and additive — with it off, search is byte-for-byte today's lexical ranking, and the cmd-K omnibar stays purely lexical and instant either way (semantic applies only to the agent-facing MCP tool).
+
+<Callout type="warn">
+ **Content egress.** When semantic search is enabled **and** a key is set, the search query and the matching page content are sent to the configured embeddings provider (OpenAI by default). Only content that is already in your corpus is embedded — anything excluded by `.okignore` / `.gitignore` is never sent — and embedding is lazy: nothing leaves the machine until an agent actually runs a semantic search. The key lives only in a 0600 `~/.ok/secrets.yml` file (never in `config.yml`, logs, or telemetry).
+</Callout>
+
+To turn it on (per machine):
+
+1. Store the provider key in `~/.ok/secrets.yml` (0600, this machine only): `ok embeddings set-key` (reads the key from a hidden prompt or stdin). Check it with `ok embeddings status`.
+2. Set `search.semantic.enabled: true` in project-local config (`<project>/.ok/local/config.yml`).
+3. (Optional) point at a non-OpenAI provider via `search.semantic.baseUrl` / `model` / `dimensions`.
+
+The first semantic search kicks off a background embed of the corpus (cents for a whole vault with `text-embedding-3-small`); vectors are cached incrementally under `.ok/local/` and only changed docs re-embed. If the key is missing, the provider errors, or you're offline, search silently degrades to lexical — it never blocks or errors. Each MCP `search` response reports embedding coverage so an agent knows when vectors are still filling in.
## Verifying changes
diff --git a/docs/content/reference/mcp.mdx b/docs/content/reference/mcp.mdx
index 62ed0866f..acf257ea3 100644
--- a/docs/content/reference/mcp.mdx
+++ b/docs/content/reference/mcp.mdx
@@ -12,7 +12,7 @@ The surface is **17 tools**. Three of the four write verbs — `write`, `edit`,
| Tool | Purpose |
| --- | --- |
| `exec` | Read-only shell (`cat`, `ls`, `grep`, `find`, `head`, `tail`, `wc`, `sort`, `uniq`, `cut`); reads return frontmatter, backlinks, and recent history. Fs-direct; works without the server. |
-| `search` | Ranked workspace search (title boost + body BM25 + recency; same engine the cmd-K palette uses) |
+| `search` | Ranked workspace search (title boost + body BM25 + recency; same engine the cmd-K palette uses). When [semantic search](/docs/reference/configuration#semantic-search) is enabled, an embeddings signal is additionally fused into `full_text` ranking (opt-in, content egress) — pass `semantic: false` to force pure-lexical. |
| `links` | Wiki-link graph. `kind` selects: `backlinks`, `forward`, `dead`, `orphans`, `hubs`, or `suggest`. Accepts an array (e.g. `["dead", "orphans", "hubs"]`) to fetch several views in one call; each view nests under its own key in the response |
| `history` | Version timeline for a doc. Each entry carries a `version` (commit SHA) you pass to `restore_version` |
| `config` | Read the effective merged config (`config({ key: "appearance.theme" })` for a sub-tree; omit `key` for the whole config) |
diff --git a/packages/app/src/components/settings/EmbeddingsKeySection.dom.test.tsx b/packages/app/src/components/settings/EmbeddingsKeySection.dom.test.tsx
new file mode 100644
index 000000000..15f1b0910
--- /dev/null
+++ b/packages/app/src/components/settings/EmbeddingsKeySection.dom.test.tsx
@@ -0,0 +1,151 @@
+import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
+import type { SemanticIndexStatus } from '@inkeep/open-knowledge-core';
+import { cleanup, render, screen, waitFor } from '@testing-library/react';
+import userEvent from '@testing-library/user-event';
+import type { EmbeddingsKeyTransport } from '@/lib/transports/embeddings-key-transport';
+import { EmbeddingsKeySection } from './EmbeddingsKeySection';
+
+type WindowGlobals = { NodeFilter?: typeof NodeFilter };
+type GlobalWithDomShims = typeof globalThis &
+ WindowGlobals & { window?: WindowGlobals; ResizeObserver?: unknown };
+const g = globalThis as GlobalWithDomShims;
+if (g.NodeFilter === undefined && g.window?.NodeFilter !== undefined)
+ g.NodeFilter = g.window.NodeFilter;
+if (g.ResizeObserver === undefined) {
+ class NoopResizeObserver {
+ observe() {}
+ unobserve() {}
+ disconnect() {}
+ }
+ g.ResizeObserver = NoopResizeObserver;
+}
+
+function statusOf(over: Partial<SemanticIndexStatus>): SemanticIndexStatus {
+ return {
+ enabled: false,
+ keyPresent: false,
+ keySource: null,
+ ready: false,
+ capable: false,
+ embedded: 0,
+ total: 0,
+ ...over,
+ };
+}
+
+let mockStatus: SemanticIndexStatus;
+const originalFetch = global.fetch;
+
+function makeTransport(parts?: Partial<EmbeddingsKeyTransport>): {
+ transport: EmbeddingsKeyTransport;
+ setCalls: string[];
+ clearCalls: number;
+} {
+ const setCalls: string[] = [];
+ let clearCalls = 0;
+ const transport: EmbeddingsKeyTransport = {
+ setKey: async (key) => {
+ setCalls.push(key);
+ return parts?.setKey ? parts.setKey(key) : { ok: true };
+ },
+ clearKey: async () => {
+ clearCalls += 1;
+ return parts?.clearKey ? parts.clearKey() : { ok: true };
+ },
+ };
+ return {
+ transport,
+ setCalls,
+ get clearCalls() {
+ return clearCalls;
+ },
+ } as { transport: EmbeddingsKeyTransport; setCalls: string[]; clearCalls: number };
+}
+
+beforeEach(() => {
+ mockStatus = statusOf({});
+ global.fetch = (async () => ({
+ ok: true,
+ json: async () => mockStatus,
+ })) as unknown as typeof fetch;
+});
+afterEach(() => {
+ cleanup();
+ global.fetch = originalFetch;
+});
+
+describe('EmbeddingsKeySection', () => {
+ test('no key: shows an input + Save (disabled until typed), never renders a key', async () => {
+ const { transport } = makeTransport();
+ mockStatus = statusOf({ keyPresent: false });
+ render(<EmbeddingsKeySection transport={transport} />);
+
+ const input = screen.getByTestId('settings-embeddings-key-input');
+ expect(input.getAttribute('type')).toBe('password');
+ expect(
+ screen.getByTestId('settings-embeddings-key-save').getAttribute('disabled'),
+ ).not.toBeNull();
+ expect(screen.queryByTestId('settings-embeddings-key-set')).toBeNull();
+ });
+
+ test('save sends the key to the transport and clears the input', async () => {
+ const user = userEvent.setup();
+ const rec = makeTransport();
+ mockStatus = statusOf({ keyPresent: false });
+ render(<EmbeddingsKeySection transport={rec.transport} />);
+
+ const input = screen.getByTestId('settings-embeddings-key-input');
+ await user.type(input, 'sk-secret-123');
+ await user.click(screen.getByTestId('settings-embeddings-key-save'));
+
+ expect(rec.setCalls).toEqual(['sk-secret-123']);
+ await waitFor(() => expect((input as HTMLInputElement).value).toBe(''));
+ });
+
+ test('save failure surfaces the error and keeps the typed key', async () => {
+ const user = userEvent.setup();
+ const rec = makeTransport({ setKey: async () => ({ ok: false, error: 'Loopback required.' }) });
+ mockStatus = statusOf({ keyPresent: false });
+ render(<EmbeddingsKeySection transport={rec.transport} />);
+
+ const input = screen.getByTestId('settings-embeddings-key-input');
+ await user.type(input, 'sk-bad');
+ await user.click(screen.getByTestId('settings-embeddings-key-save'));
+
+ expect(await screen.findByTestId('settings-embeddings-key-error')).toBeDefined();
+ expect((input as HTMLInputElement).value).toBe('sk-bad');
+ });
+
+ test('key present (file): shows "API key set" + Clear, which calls the transport', async () => {
+ const user = userEvent.setup();
+ const rec = makeTransport();
+ mockStatus = statusOf({ keyPresent: true, keySource: 'file' });
+ render(<EmbeddingsKeySection transport={rec.transport} />);
+
+ expect(await screen.findByTestId('settings-embeddings-key-set')).toBeDefined();
+ await user.click(screen.getByTestId('settings-embeddings-key-clear'));
+ expect(rec.clearCalls).toBe(1);
+ });
+
+ test('clear failure surfaces the error', async () => {
+ const user = userEvent.setup();
+ const rec = makeTransport({
+ clearKey: async () => ({ ok: false, error: 'Loopback required.' }),
+ });
+ mockStatus = statusOf({ keyPresent: true, keySource: 'file' });
+ render(<EmbeddingsKeySection transport={rec.transport} />);
+
+ await user.click(await screen.findByTestId('settings-embeddings-key-clear'));
+ expect(await screen.findByTestId('settings-embeddings-key-error')).toBeDefined();
+ });
+
+ test('env-sourced key: read-only note, no input or clear', async () => {
+ const { transport } = makeTransport();
+ mockStatus = statusOf({ keyPresent: true, keySource: 'env' });
+ render(<EmbeddingsKeySection transport={transport} />);
+
+ expect(await screen.findByTestId('settings-embeddings-key-env')).toBeDefined();
+ expect(screen.queryByTestId('settings-embeddings-key-input')).toBeNull();
+ expect(screen.queryByTestId('settings-embeddings-key-clear')).toBeNull();
+ });
+});
diff --git a/packages/app/src/components/settings/EmbeddingsKeySection.tsx b/packages/app/src/components/settings/EmbeddingsKeySection.tsx
new file mode 100644
index 000000000..9b5c39348
--- /dev/null
+++ b/packages/app/src/components/settings/EmbeddingsKeySection.tsx
@@ -0,0 +1,143 @@
+import { Trans, useLingui } from '@lingui/react/macro';
+import { useState } from 'react';
+import { Button } from '@/components/ui/button';
+import { Input } from '@/components/ui/input';
+import { useSemanticSearchStatus } from '@/hooks/use-semantic-search-status';
+import {
+ type EmbeddingsKeyTransport,
+ httpEmbeddingsKeyTransport,
+} from '@/lib/transports/embeddings-key-transport';
+
+export function EmbeddingsKeySection({ transport }: { transport?: EmbeddingsKeyTransport }) {
+ const { t } = useLingui();
+ const resolved = transport ?? httpEmbeddingsKeyTransport();
+ const { status, refresh } = useSemanticSearchStatus();
+ const [keyInput, setKeyInput] = useState('');
+ const [busy, setBusy] = useState(false);
+ const [error, setError] = useState<string | null>(null);
+
+ const keyPresent = status?.keyPresent ?? false;
+ const keySource = status?.keySource ?? null;
+
+ async function onSave() {
+ const key = keyInput.trim();
+ if (!key || busy) return;
+ setBusy(true);
+ setError(null);
+ const result = await resolved.setKey(key);
+ setBusy(false);
+ if (result.ok) {
+ setKeyInput(''); // Don't keep the secret in component state after it lands.
+ refresh();
+ } else {
+ setError(result.error ?? t`Couldn't save the key — please try again.`);
+ }
+ }
+
+ async function onClear() {
+ if (busy) return;
+ setBusy(true);
+ setError(null);
+ const result = await resolved.clearKey();
+ setBusy(false);
+ if (result.ok) refresh();
+ else setError(result.error ?? t`Couldn't clear the key — please try again.`);
+ }
+
+ return (
+ <section
+ aria-labelledby="settings-embeddings-key-title"
+ className="space-y-3"
+ data-testid="settings-embeddings-key"
+ >
+ <div className="space-y-1">
+ <h3 id="settings-embeddings-key-title" className="text-base font-semibold">
+ <Trans>Embeddings provider key</Trans>
+ </h3>
+ <p className="text-sm text-muted-foreground">
+ <Trans>
+ The API key semantic search uses to embed your content. Stored once for this machine and
+ shared across all projects. Turn the feature on per project in This project → Search.
+ </Trans>
+ </p>
+ </div>
+
+ <div className="space-y-3 rounded-md border p-3">
+ {keySource === 'env' ? (
+ <p className="text-muted-foreground text-1sm" data-testid="settings-embeddings-key-env">
+ <Trans>
+ Using the{' '}
+ <code className="rounded bg-muted px-1 py-0.5 font-mono text-xs">
+ OK_EMBEDDINGS_API_KEY
+ </code>{' '}
+ environment variable (managed outside Open Knowledge).
+ </Trans>
+ </p>
+ ) : (
+ <>
+ {keyPresent ? (
+ <div
+ className="flex items-center justify-between gap-3"
+ data-testid="settings-embeddings-key-set"
+ >
+ <div className="min-w-0">
+ <div className="text-sm font-medium">
+ <Trans>API key set</Trans>
+ </div>
+ <p className="text-muted-foreground text-1sm">
+ <Trans>Stored on this machine.</Trans>
+ </p>
+ </div>
+ <Button
+ variant="outline"
+ onClick={() => void onClear()}
+ disabled={busy}
+ data-testid="settings-embeddings-key-clear"
+ >
+ <Trans>Clear</Trans>
+ </Button>
+ </div>
+ ) : null}
+
+ <div className="space-y-2">
+ <label htmlFor="settings-embeddings-key-input" className="text-sm font-medium">
+ {keyPresent ? <Trans>Replace key</Trans> : <Trans>Add a key</Trans>}
+ </label>
+ <div className="flex items-center gap-2">
+ <Input
+ id="settings-embeddings-key-input"
+ type="password"
+ value={keyInput}
+ onChange={(e) => setKeyInput(e.target.value)}
+ placeholder={t`Paste your API key`}
+ autoComplete="off"
+ spellCheck={false}
+ disabled={busy}
+ data-testid="settings-embeddings-key-input"
+ className="h-8 font-mono text-sm"
+ />
+ <Button
+ onClick={() => void onSave()}
+ disabled={busy || keyInput.trim().length === 0}
+ data-testid="settings-embeddings-key-save"
+ >
+ {busy ? <Trans>Saving</Trans> : <Trans>Save</Trans>}
+ </Button>
+ </div>
+ </div>
+ </>
+ )}
+
+ {error ? (
+ <p
+ role="alert"
+ className="text-sm text-destructive"
+ data-testid="settings-embeddings-key-error"
+ >
+ {error}
+ </p>
+ ) : null}
+ </div>
+ </section>
+ );
+}
diff --git a/packages/app/src/components/settings/SearchSection.dom.test.tsx b/packages/app/src/components/settings/SearchSection.dom.test.tsx
new file mode 100644
index 000000000..29c7385c7
--- /dev/null
+++ b/packages/app/src/components/settings/SearchSection.dom.test.tsx
@@ -0,0 +1,317 @@
+import { afterEach, beforeEach, describe, expect, mock, test } from 'bun:test';
+import type { Config, ConfigBinding, SemanticIndexStatus } from '@inkeep/open-knowledge-core';
+import { cleanup, render, screen, waitFor } from '@testing-library/react';
+import userEvent from '@testing-library/user-event';
+
+type WindowGlobals = { NodeFilter?: typeof NodeFilter };
+type GlobalWithDomShims = typeof globalThis &
+ WindowGlobals & { window?: WindowGlobals; ResizeObserver?: unknown };
+const globalWithDomShims = globalThis as GlobalWithDomShims;
+if (
+ globalWithDomShims.NodeFilter === undefined &&
+ globalWithDomShims.window?.NodeFilter !== undefined
+) {
+ globalWithDomShims.NodeFilter = globalWithDomShims.window.NodeFilter;
+}
+if (globalWithDomShims.ResizeObserver === undefined) {
+ class NoopResizeObserver {
+ observe() {}
+ unobserve() {}
+ disconnect() {}
+ }
+ globalWithDomShims.ResizeObserver = NoopResizeObserver;
+}
+
+let mockProjectLocalConfig: Config | null = null;
+let mockProjectLocalSynced = true;
+let mockProjectLocalBinding: ConfigBinding | null = null;
+
+mock.module('@/lib/config-provider', () => ({
+ useConfigContext: () => ({
+ userBinding: null,
+ userSynced: false,
+ projectBinding: null,
+ projectLocalBinding: mockProjectLocalBinding,
+ okignoreBinding: null,
+ okignoreSynced: false,
+ userConfig: null,
+ projectConfig: null,
+ projectLocalConfig: mockProjectLocalConfig,
+ projectLocalSynced: mockProjectLocalSynced,
+ merged: null,
+ }),
+}));
+
+const { SearchSection } = await import('./SearchSection');
+
+function configWithSemanticEnabled(enabled: boolean): Config {
+ return { search: { semantic: { enabled } } } as unknown as Config;
+}
+
+function makeBinding(): { binding: ConfigBinding; calls: unknown[] } {
+ const calls: unknown[] = [];
+ const binding = {
+ current: () => ({}),
+ patch: (patch: unknown) => {
+ calls.push(patch);
+ return { ok: true, value: { applied: [], effective: {} } };
+ },
+ subscribe: () => () => {},
+ hasSynced: () => true,
+ subscribeSynced: () => () => {},
+ dispose: () => {},
+ } as unknown as ConfigBinding;
+ return { binding, calls };
+}
+
+let mockStatus: SemanticIndexStatus | null = null;
+const originalFetch = global.fetch;
+
+beforeEach(() => {
+ mockProjectLocalConfig = null;
+ mockProjectLocalSynced = true;
+ mockProjectLocalBinding = null;
+ mockStatus = null;
+ global.fetch = (async () => ({
+ ok: true,
+ json: async () => mockStatus,
+ })) as unknown as typeof fetch;
+});
+
+afterEach(() => {
+ cleanup();
+ global.fetch = originalFetch;
+});
+
+describe('SearchSection', () => {
+ test('off: switch is unchecked, body says no content leaves, no coverage panel', () => {
+ const { binding } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(false);
+
+ render(<SearchSection />);
+
+ const toggle = screen.getByTestId('settings-search-semantic-toggle');
+ expect(toggle.getAttribute('aria-checked')).toBe('false');
+ expect(screen.getByTestId('settings-search-body').textContent).toContain(
+ 'No content leaves this computer',
+ );
+ expect(screen.queryByTestId('settings-search-coverage')).toBeNull();
+ expect(screen.queryByTestId('settings-search-needs-key')).toBeNull();
+ });
+
+ test('toggle is disabled until the project-local binding has synced', () => {
+ mockProjectLocalBinding = null;
+ mockProjectLocalSynced = false;
+
+ render(<SearchSection />);
+
+ expect(
+ screen.getByTestId('settings-search-semantic-toggle').getAttribute('disabled'),
+ ).not.toBeNull();
+ });
+
+ test('enabling opens the egress confirm dialog and does NOT write until confirmed', async () => {
+ const user = userEvent.setup();
+ const { binding, calls } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(false);
+
+ render(<SearchSection />);
+
+ await user.click(screen.getByTestId('settings-search-semantic-toggle'));
+
+ expect(await screen.findByText('This sends content off your machine')).toBeDefined();
+ expect(calls.length).toBe(0);
+
+ await user.click(screen.getByTestId('settings-search-confirm-enable'));
+
+ expect(calls).toEqual([{ search: { semantic: { enabled: true } } }]);
+ });
+
+ test('disabling commits immediately with no confirmation dialog', async () => {
+ const user = userEvent.setup();
+ const { binding, calls } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(true);
+ mockStatus = {
+ enabled: true,
+ keyPresent: true,
+ keySource: 'file',
+ ready: true,
+ capable: true,
+ embedded: 2,
+ total: 5,
+ };
+
+ render(<SearchSection />);
+
+ await user.click(screen.getByTestId('settings-search-semantic-toggle'));
+
+ expect(screen.queryByText('This sends content off your machine')).toBeNull();
+ expect(calls).toEqual([{ search: { semantic: { enabled: false } } }]);
+ });
+
+ test('on + keyed + warmed + capable: shows read-only coverage', async () => {
+ const { binding } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(true);
+ mockStatus = {
+ enabled: true,
+ keyPresent: true,
+ keySource: 'file',
+ ready: true,
+ capable: true,
+ embedded: 3,
+ total: 5,
+ };
+
+ render(<SearchSection />);
+
+ const coverage = await screen.findByTestId('settings-search-coverage');
+ expect(coverage.textContent).toMatch(/Indexed\s*3\s*of\s*5/);
+ });
+
+ test('on + capable but nothing embedded yet: shows the lazy-warm hint', async () => {
+ const { binding } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(true);
+ mockStatus = {
+ enabled: true,
+ keyPresent: true,
+ keySource: 'file',
+ ready: true,
+ capable: true,
+ embedded: 0,
+ total: 5,
+ };
+
+ render(<SearchSection />);
+
+ const coverage = await screen.findByTestId('settings-search-coverage');
+ expect(coverage.textContent).toContain('first time a search needs them');
+ });
+
+ test('on + NO key: shows the needs-a-key hint pointing at Account (instant, no warm)', async () => {
+ const { binding } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(true);
+ mockStatus = {
+ enabled: true,
+ keyPresent: false,
+ keySource: null,
+ ready: false,
+ capable: false,
+ embedded: 0,
+ total: 5,
+ };
+
+ render(<SearchSection />);
+
+ const hint = await screen.findByTestId('settings-search-needs-key');
+ expect(hint.textContent).toContain('no API key is set');
+ expect(hint.textContent).toContain('Account');
+ expect(screen.queryByTestId('settings-search-coverage')).toBeNull();
+ expect(screen.queryByTestId('settings-search-pending')).toBeNull();
+ });
+
+ test('on + key present but provider rejected it: shows the provider-error hint', async () => {
+ const { binding } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(true);
+ mockStatus = {
+ enabled: true,
+ keyPresent: true,
+ keySource: 'file',
+ ready: true,
+ capable: false,
+ embedded: 0,
+ total: 5,
+ };
+
+ render(<SearchSection />);
+
+ const err = await screen.findByTestId('settings-search-provider-error');
+ expect(err.textContent).toContain('rejected it');
+ expect(screen.queryByTestId('settings-search-needs-key')).toBeNull();
+ });
+
+ test('on + keyed but not warmed: shows the pending state', async () => {
+ const { binding } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(true);
+ mockStatus = {
+ enabled: true,
+ keyPresent: true,
+ keySource: 'file',
+ ready: false,
+ capable: false,
+ embedded: 0,
+ total: 5,
+ };
+
+ render(<SearchSection />);
+
+ const pending = await screen.findByTestId('settings-search-pending');
+ expect(pending.textContent).toContain('activates the first time');
+ expect(screen.queryByTestId('settings-search-needs-key')).toBeNull();
+ expect(screen.queryByTestId('settings-search-coverage')).toBeNull();
+ });
+
+ test('on but server not yet settled: shows the applying state', async () => {
+ const { binding } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(true);
+ mockStatus = {
+ enabled: false,
+ keyPresent: false,
+ keySource: null,
+ ready: false,
+ capable: false,
+ embedded: 0,
+ total: 5,
+ };
+
+ render(<SearchSection />);
+
+ await waitFor(() =>
+ expect(screen.getByTestId('settings-search-settling').textContent).toContain(
+ 'Applying your change',
+ ),
+ );
+ });
+
+ test('cancelling the confirm dialog writes nothing and leaves the toggle off', async () => {
+ const user = userEvent.setup();
+ const { binding, calls } = makeBinding();
+ mockProjectLocalBinding = binding;
+ mockProjectLocalConfig = configWithSemanticEnabled(false);
+
+ render(<SearchSection />);
+
+ await user.click(screen.getByTestId('settings-search-semantic-toggle'));
+ await user.click(await screen.findByRole('button', { name: /cancel/i }));
+
+ expect(calls.length).toBe(0);
+ expect(screen.getByTestId('settings-search-semantic-toggle').getAttribute('aria-checked')).toBe(
+ 'false',
+ );
+ });
+
+ test('write failure keeps the confirm dialog open for retry (egress consent invariant)', async () => {
+ const user = userEvent.setup();
+ const failBinding = {
+ ...makeBinding().binding,
+ patch: () => ({ ok: false, error: { code: 'noop', message: 'fail' } }),
+ } as unknown as ConfigBinding;
+ mockProjectLocalBinding = failBinding;
+ mockProjectLocalConfig = configWithSemanticEnabled(false);
+
+ render(<SearchSection />);
+
+ await user.click(screen.getByTestId('settings-search-semantic-toggle'));
+ await user.click(await screen.findByTestId('settings-search-confirm-enable'));
+
+ expect(await screen.findByTestId('settings-search-confirm')).toBeDefined();
+ });
+});
diff --git a/packages/app/src/components/settings/SearchSection.tsx b/packages/app/src/components/settings/SearchSection.tsx
new file mode 100644
index 000000000..99ef2fe05
--- /dev/null
+++ b/packages/app/src/components/settings/SearchSection.tsx
@@ -0,0 +1,317 @@
+import { humanFormat } from '@inkeep/open-knowledge-core';
+import { Trans, useLingui } from '@lingui/react/macro';
+import { useEffect, useRef, useState } from 'react';
+import { toast } from 'sonner';
+import { Button } from '@/components/ui/button';
+import {
+ DialogBody,
+ DialogClose,
+ DialogContent,
+ DialogDescription,
+ DialogFooter,
+ DialogHeader,
+ Dialog as DialogRoot,
+ DialogTitle,
+} from '@/components/ui/dialog';
+import { Switch } from '@/components/ui/switch';
+import { useSemanticSearchStatus } from '@/hooks/use-semantic-search-status';
+import { useConfigContext } from '@/lib/config-provider';
+
+const SETTLE_REFRESH_DELAYS_MS = [2500, 5000] as const;
+
+export function SearchSection() {
+ const { t } = useLingui();
+ const { projectLocalConfig, projectLocalSynced, projectLocalBinding } = useConfigContext();
+ const { status, refresh } = useSemanticSearchStatus();
+ const [confirmOpen, setConfirmOpen] = useState(false);
+ const settleTimersRef = useRef<Array<ReturnType<typeof setTimeout>>>([]);
+
+ useEffect(
+ () => () => {
+ for (const timer of settleTimersRef.current) clearTimeout(timer);
+ },
+ [],
+ );
+
+ const enabled = projectLocalConfig?.search?.semantic?.enabled ?? false;
+ const bindingReady = projectLocalSynced && projectLocalBinding !== null;
+
+ function scheduleSettleRefresh() {
+ for (const timer of settleTimersRef.current) clearTimeout(timer);
+ settleTimersRef.current = SETTLE_REFRESH_DELAYS_MS.map((delay) => setTimeout(refresh, delay));
+ }
+
+ function write(next: boolean): boolean {
+ if (projectLocalBinding === null) {
+ toast.error(t`Search settings not yet loaded — try again in a moment`);
+ return false;
+ }
+ const result = projectLocalBinding.patch({ search: { semantic: { enabled: next } } });
+ if (!result.ok) {
+ const detail = humanFormat(result.error);
+ toast.error(
+ next
+ ? t`Failed to enable semantic search — ${detail}`
+ : t`Failed to disable semantic search — ${detail}`,
+ );
+ return false;
+ }
+ refresh();
+ scheduleSettleRefresh();
+ return true;
+ }
+
+ function onToggleRequest(next: boolean) {
+ if (next) {
+ setConfirmOpen(true);
+ return;
+ }
+ write(false);
+ }
+
+ function onConfirm() {
+ if (write(true)) setConfirmOpen(false);
+ }
+
+ const serverEnabled = status?.enabled ?? false;
+ const keyPresent = status?.keyPresent ?? false;
+ const ready = status?.ready ?? false;
+ const capable = status?.capable ?? false;
+ const embedded = status?.embedded ?? 0;
+ const total = status?.total ?? 0;
+
+ return (
+ <section
+ aria-labelledby="settings-search-title"
+ className="space-y-3"
+ data-testid="settings-search"
+ >
+ <div className="space-y-1">
+ <h3 id="settings-search-title" className="text-base font-semibold">
+ <Trans>Semantic search</Trans>
+ </h3>
+ <p className="text-sm text-muted-foreground">
+ <Trans>
+ Add meaning-based ranking to search so conceptually-related pages surface even when they
+ share no keywords. This setting applies only to this computer.
+ </Trans>
+ </p>
+ </div>
+
+ <div className="rounded-md border p-3">
+ <div className="flex items-center justify-between gap-3">
+ <div className="min-w-0">
+ <label htmlFor="settings-search-semantic-toggle" className="text-sm font-medium">
+ <Trans>Semantic search</Trans>
+ </label>
+ <p className="text-muted-foreground text-1sm" data-testid="settings-search-body">
+ {enabled ? (
+ <Trans>
+ On — your search queries and the text of matching pages are sent to your
+ embeddings provider (OpenAI by default) to compute embeddings.
+ </Trans>
+ ) : (
+ <Trans>Off — search ranks by keyword only. No content leaves this computer.</Trans>
+ )}
+ </p>
+ </div>
+ <Switch
+ id="settings-search-semantic-toggle"
+ checked={enabled}
+ disabled={!bindingReady}
+ onCheckedChange={onToggleRequest}
+ aria-label={enabled ? t`Disable semantic search` : t`Enable semantic search`}
+ data-testid="settings-search-semantic-toggle"
+ />
+ </div>
+
+ {enabled ? (
+ <SemanticStatusPanel
+ loaded={status !== null}
+ serverEnabled={serverEnabled}
+ keyPresent={keyPresent}
+ ready={ready}
+ capable={capable}
+ embedded={embedded}
+ total={total}
+ />
+ ) : null}
+ </div>
+
+ <EnableSemanticSearchConfirmDialog
+ open={confirmOpen}
+ onOpenChange={setConfirmOpen}
+ onConfirm={onConfirm}
+ />
+ </section>
+ );
+}
+
+interface SemanticStatusPanelProps {
+ loaded: boolean;
+ serverEnabled: boolean;
+ keyPresent: boolean;
+ ready: boolean;
+ capable: boolean;
+ embedded: number;
+ total: number;
+}
+
+function SemanticStatusPanel({
+ loaded,
+ serverEnabled,
+ keyPresent,
+ ready,
+ capable,
+ embedded,
+ total,
+}: SemanticStatusPanelProps) {
+ if (!loaded || !serverEnabled) {
+ return (
+ <p
+ role="status"
+ aria-live="polite"
+ className="text-muted-foreground text-1sm mt-2"
+ data-testid="settings-search-settling"
+ >
+ <Trans>Applying your change</Trans>
+ </p>
+ );
+ }
+
+ if (!keyPresent) {
+ return (
+ <div
+ role="alert"
+ className="mt-2 rounded-md border border-amber-300 bg-amber-50 px-3 py-2 text-1sm text-amber-900 dark:border-amber-700 dark:bg-amber-950 dark:text-amber-200"
+ data-testid="settings-search-needs-key"
+ >
+ <Trans>
+ Semantic search is on, but no API key is set — search falls back to keyword matching. Add
+ one in <span className="font-medium">Settings → Account</span> (it's stored once for your
+ whole machine).
+ </Trans>
+ </div>
+ );
+ }
+
+ if (ready && !capable) {
+ return (
+ <div
+ role="alert"
+ className="mt-2 rounded-md border border-amber-300 bg-amber-50 px-3 py-2 text-1sm text-amber-900 dark:border-amber-700 dark:bg-amber-950 dark:text-amber-200"
+ data-testid="settings-search-provider-error"
+ >
+ <Trans>
+ A key is set, but the embeddings provider rejected it or was unreachable — search fell
+ back to keyword matching. Check the key in{' '}
+ <span className="font-medium">Settings → Account</span> and your provider settings.
+ </Trans>
+ </div>
+ );
+ }
+
+ if (!ready) {
+ return (
+ <p
+ role="status"
+ className="text-muted-foreground text-1sm mt-2"
+ data-testid="settings-search-pending"
+ >
+ <Trans>Semantic ranking activates the first time an agent runs a search.</Trans>
+ </p>
+ );
+ }
+
+ return (
+ <div className="mt-2 space-y-0.5" data-testid="settings-search-coverage">
+ <p className="text-muted-foreground text-1sm">
+ <Trans>
+ Indexed {embedded} of {total} pages.
+ </Trans>
+ </p>
+ {embedded === 0 ? (
+ <p className="text-muted-foreground text-1sm">
+ <Trans>Pages are embedded the first time a search needs them.</Trans>
+ </p>
+ ) : null}
+ </div>
+ );
+}
+
+interface EnableSemanticSearchConfirmDialogProps {
+ open: boolean;
+ onOpenChange: (open: boolean) => void;
+ onConfirm: () => void;
+}
+
+function EnableSemanticSearchConfirmDialog({
+ open,
+ onOpenChange,
+ onConfirm,
+}: EnableSemanticSearchConfirmDialogProps) {
+ return (
+ <DialogRoot open={open} onOpenChange={onOpenChange}>
+ <DialogContent className="sm:max-w-lg" data-testid="settings-search-confirm">
+ <DialogHeader>
+ <DialogTitle>
+ <Trans>Turn on semantic search?</Trans>
+ </DialogTitle>
+ <DialogDescription>
+ <Trans>
+ Semantic search adds meaning-based ranking to the search tool so conceptually-related
+ pages surface even without shared keywords.
+ </Trans>
+ </DialogDescription>
+ </DialogHeader>
+ <DialogBody>
+ <div
+ role="alert"
+ className="rounded-md border border-amber-300 bg-amber-50 px-3 py-2.5 text-sm text-amber-900 dark:border-amber-700 dark:bg-amber-950 dark:text-amber-200"
+ >
+ <p className="mb-2 font-medium">
+ <Trans>This sends content off your machine</Trans>
+ </p>
+ <ul className="list-disc space-y-1.5 pl-5">
+ <li>
+ <Trans>
+ Your search queries and the full text of matching pages are sent to your
+ embeddings provider (OpenAI by default) to compute embeddings.
+ </Trans>
+ </li>
+ <li>
+ <Trans>
+ Embeddings are computed only when a search runs and are cached locally under{' '}
+ <code className="rounded bg-amber-100 px-1 py-0.5 font-mono text-xs dark:bg-amber-900">
+ .ok/local
+ </code>
+ .
+ </Trans>
+ </li>
+ <li>
+ <Trans>
+ This setting is per-machine and isn't shared with collaborators. It needs an API
+ key set with{' '}
+ <code className="rounded bg-amber-100 px-1 py-0.5 font-mono text-xs dark:bg-amber-900">
+ ok embeddings set-key
+ </code>
+ .
+ </Trans>
+ </li>
+ </ul>
+ </div>
+ </DialogBody>
+ <DialogFooter>
+ <DialogClose asChild>
+ <Button variant="outline">
+ <Trans>Cancel</Trans>
+ </Button>
+ </DialogClose>
+ <Button onClick={onConfirm} data-testid="settings-search-confirm-enable">
+ <Trans>Turn on</Trans>
+ </Button>
+ </DialogFooter>
+ </DialogContent>
+ </DialogRoot>
+ );
+}
diff --git a/packages/app/src/components/settings/SettingsDialogBody.tsx b/packages/app/src/components/settings/SettingsDialogBody.tsx
index 338ed07ba..e842b1f90 100644
--- a/packages/app/src/components/settings/SettingsDialogBody.tsx
+++ b/packages/app/src/components/settings/SettingsDialogBody.tsx
@@ -66,8 +66,10 @@ import {
} from '@/lib/keyboard-shortcuts';
import { cn } from '@/lib/utils';
import { AccountSection } from './AccountSection';
+import { EmbeddingsKeySection } from './EmbeddingsKeySection';
import { OkignoreSection } from './OkignoreSection';
import { ProjectTemplatesSection } from './ProjectTemplatesSection';
+import { SearchSection } from './SearchSection';
import {
getEnumOptions,
getFieldDefault,
@@ -136,11 +138,19 @@ export function SettingsDialogBody({
return <HotkeysSection />;
}
if (activeId === 'account') {
- return <AccountSection />;
+ return (
+ <div className="space-y-8">
+ <AccountSection />
+ <EmbeddingsKeySection />
+ </div>
+ );
}
if (activeId === 'sync') {
return <SyncSection />;
}
+ if (activeId === 'search') {
+ return <SearchSection />;
+ }
if (activeId === 'project-templates') {
return <ProjectTemplatesSection />;
}
diff --git a/packages/app/src/components/settings/SettingsDialogShell.tsx b/packages/app/src/components/settings/SettingsDialogShell.tsx
index 3007bf33a..3573c74ea 100644
--- a/packages/app/src/components/settings/SettingsDialogShell.tsx
+++ b/packages/app/src/components/settings/SettingsDialogShell.tsx
@@ -65,6 +65,7 @@ export function SettingsDialogShell({ open, onOpenChange }: SettingsDialogShellP
enabled: hasProject,
items: [
{ id: 'sync', label: t`Sync` },
+ { id: 'search', label: t`Search` },
{ id: 'project-templates', label: t`Templates` },
{ id: 'okignore', label: t`Ignore patterns` },
{ id: 'sharing', label: t`Config sharing` },
diff --git a/packages/app/src/hooks/use-semantic-search-status.ts b/packages/app/src/hooks/use-semantic-search-status.ts
new file mode 100644
index 000000000..54d86935a
--- /dev/null
+++ b/packages/app/src/hooks/use-semantic-search-status.ts
@@ -0,0 +1,46 @@
+import type { SemanticIndexStatus } from '@inkeep/open-knowledge-core';
+import { useEffect, useState } from 'react';
+import { subscribeToDocumentsChanged } from '@/lib/documents-events';
+
+async function fetchSemanticStatus(): Promise<SemanticIndexStatus | null> {
+ try {
+ const res = await fetch('/api/semantic-status');
+ if (!res.ok) {
+ console.debug('[semantic-status] probe returned', res.status);
+ return null;
+ }
+ return (await res.json()) as SemanticIndexStatus;
+ } catch (err) {
+ console.debug('[semantic-status] probe failed', err);
+ return null;
+ }
+}
+
+interface UseSemanticSearchStatusResult {
+ status: SemanticIndexStatus | null;
+ refresh: () => void;
+}
+
+export function useSemanticSearchStatus(): UseSemanticSearchStatusResult {
+ const [status, setStatus] = useState<SemanticIndexStatus | null>(null);
+
+ function refresh() {
+ void fetchSemanticStatus().then((next) => {
+ if (next) setStatus(next);
+ });
+ }
+
+ // biome-ignore lint/correctness/useExhaustiveDependencies: refresh is intentionally stable (defined in component scope)
+ useEffect(() => {
+ refresh();
+ }, []);
+
+ // biome-ignore lint/correctness/useExhaustiveDependencies: refresh is intentionally stable (defined in component scope)
+ useEffect(() => {
+ return subscribeToDocumentsChanged((channels) => {
+ if (channels.includes('files')) refresh();
+ });
+ }, []);
+
+ return { status, refresh };
+}
diff --git a/packages/app/src/lib/transports/embeddings-key-transport.ts b/packages/app/src/lib/transports/embeddings-key-transport.ts
new file mode 100644
index 000000000..56411cc34
--- /dev/null
+++ b/packages/app/src/lib/transports/embeddings-key-transport.ts
@@ -0,0 +1,37 @@
+import { ProblemDetailsSchema } from '@inkeep/open-knowledge-core';
+
+export interface EmbeddingsKeyTransport {
+ setKey(key: string): Promise<{ ok: true } | { ok: false; error?: string }>;
+ clearKey(): Promise<{ ok: true } | { ok: false; error?: string }>;
+}
+
+async function extractProblemTitle(res: Response): Promise<string | undefined> {
+ try {
+ const result = ProblemDetailsSchema.safeParse(await res.json());
+ if (result.success) return result.data.title;
+ } catch {}
+ return undefined;
+}
+
+async function post(
+ url: string,
+ body: unknown,
+): Promise<{ ok: true } | { ok: false; error?: string }> {
+ try {
+ const res = await fetch(url, {
+ method: 'POST',
+ headers: { 'content-type': 'application/json' },
+ body: JSON.stringify(body),
+ });
+ return res.ok ? { ok: true } : { ok: false, error: await extractProblemTitle(res) };
+ } catch {
+ return { ok: false };
+ }
+}
+
+export function httpEmbeddingsKeyTransport(): EmbeddingsKeyTransport {
+ return {
+ setKey: (key) => post('/api/local-op/embeddings/set-key', { key }),
+ clearKey: () => post('/api/local-op/embeddings/clear-key', {}),
+ };
+}
diff --git a/packages/app/src/locales/en/messages.json b/packages/app/src/locales/en/messages.json
index 9e9d5e011..fc4255b5b 100644
--- a/packages/app/src/locales/en/messages.json
+++ b/packages/app/src/locales/en/messages.json
@@ -119,6 +119,7 @@
"3SDfZl": [
"<0>Mirror — pick a source.</0> Set <1>src</1> + <2>anchor</2> via the property panel to point at a <3><MirrorSource></3> elsewhere."
],
+ "3TCafD": ["Stored on this machine."],
"3VbwPp": ["Open ", ["keyName"], " in browser"],
"3VsNTI": ["Loading document"],
"3YvS-c": ["New tab"],
@@ -134,6 +135,7 @@
"3oh0nQ": ["Nested .okignore error in ", ["path"]],
"3pU8e-": ["Pin tab"],
"3wiBAs": ["Where should it live?"],
+ "3y6XTb": ["Semantic search"],
"3zyg_A": ["Ignore patterns"],
"438Fca": ["Cloned successfully, but could not open the project."],
"44DSl0": [
@@ -197,6 +199,7 @@
],
"5kWklo": ["(some may already be hidden by other rules)"],
"5lWFkC": ["Sign in"],
+ "5mksg0": ["Failed to disable semantic search — ", ["detail"]],
"5o4lc1": [
"Open Knowledge will be initialized at <0>",
["gitRoot"],
@@ -259,6 +262,7 @@
"7KeV5F": ["Undo all failed: ", ["message"]],
"7OhaRy": ["Source search"],
"7OwvEL": ["Show HTML preview"],
+ "7PNNNs": ["This sends content off your machine"],
"7PzzBU": ["User"],
"7_bFEL": ["Drag ", ["patternText"], " to reorder"],
"7d1a0d": ["Public"],
@@ -278,6 +282,9 @@
["fileName"],
") in the project root"
],
+ "8BsGpU": [
+ "A key is set, but the embeddings provider rejected it or was unreachable — search fell back to keyword matching. Check the key in <0>Settings → Account</0> and your provider settings."
+ ],
"8F1i42": ["Page not found"],
"8FqTCr": [
[
@@ -355,14 +362,17 @@
],
"AVW-K9": ["Failed to load orphan pages"],
"AWOSPo": ["Zoom in"],
+ "AdcGcG": ["Applying your change"],
"AddR83": ["Could not create project. Try again or pick a different location."],
"AeHuzT": ["# One pattern per line.\n# Examples:\n# drafts/\n# *.draft.md\n# !keep.md"],
"AeXO77": ["Account"],
"AiNMQ7": [
"This project lives only on this computer. Connect it to GitHub to back it up and share it with other people."
],
+ "Aif3E7": ["Enable semantic search"],
"Am78jY": ["Drag ", ["keyName"], " to reorder"],
"AoX5ld": [["folders", "plural", { "one": ["#", " folder"], "other": ["#", " folders"] }]],
+ "AqnqMJ": ["API key set"],
"ArBWJf": ["Undo last edit"],
"B11p-O": ["Edit markdown link"],
"B2DrPb": ["Auto-save"],
@@ -406,6 +416,7 @@
"COr1Y7": [
"Pick <0>openknowledge.skill</0> from your <1>Downloads</1> folder. It enables automatically."
],
+ "CSL0l1": ["Couldn't clear the key — please try again."],
"CTJLpz": ["\"", ["trimmed"], "\" already exists here"],
"CU3AiE": ["Switching branches"],
"CXj9C-": ["New property name"],
@@ -414,6 +425,7 @@
],
"C_cPoW": ["Your GitHub session expired — sign in again to verify push access."],
"ChAtFg": ["You don't have permission to push to this repo"],
+ "Ck9XRt": ["Couldn't save the key — please try again."],
"CvsD6Y": ["via ", ["viaTags"]],
"D4H2N6": ["Remove ", ["tag"]],
"D8OV7J": ["Only collaborators"],
@@ -465,6 +477,9 @@
"EiVgl5": ["You don't have permission to push to this repo."],
"Eji13z": ["Toggle highlight formatting."],
"El7NbA": ["Heading 2"],
+ "Emz5BG": [
+ "Embeddings are computed only when a search runs and are cached locally under <0>.ok/local</0>."
+ ],
"EqrC_R": ["Undo failed: ", ["message"]],
"EqtBXb": ["Visual editor find open"],
"ErRi6L": ["Template \"", ["label"], "\" deleted"],
@@ -600,6 +615,7 @@
"K0N7zS": ["Config sharing status read failed"],
"K1ckLX": ["Failed to create folder"],
"K6jjWB": [["packName"], " was already set up. Nothing to do."],
+ "K6jxcj": ["Add a key"],
"K8-WWn": ["Move between table cells"],
"K9J00P": ["Clone failed — connection error"],
"KDuPiY": ["Uploading ", ["fileName"]],
@@ -643,6 +659,7 @@
"LRgZAt": [["owner"], "/", ["name"], " already exists"],
"LXdWeS": ["Medium section heading."],
"LXfNU-": ["Navigation and suggestions"],
+ "LYufDw": ["Failed to enable semantic search — ", ["detail"]],
"Lcehxn": ["Save resolution"],
"LdABLa": ["Could not open ", ["target"], ". Try opening it manually."],
"LhWYpm": ["Other OK paths are tracked upstream:"],
@@ -858,6 +875,7 @@
"UTNVyS": ["Select the full source document."],
"UTZ97u": ["Selection must be inside the project"],
"UVLlMV": ["Couldn't reach GitHub. Try again?"],
+ "UW93oq": ["Replace key"],
"UYcGIW": ["Connecting — waiting for collab server"],
"Ua78MH": ["no response"],
"UaXXIF": [
@@ -869,6 +887,9 @@
["src"],
"</1> is no longer available."
],
+ "UfW9Kr": [
+ "Your search queries and the full text of matching pages are sent to your embeddings provider (OpenAI by default) to compute embeddings."
+ ],
"UkQGhd": ["Hide replace"],
"Up2_7p": ["Copy full path"],
"Uuj_R4": ["New from template"],
@@ -884,11 +905,15 @@
"VUijjF": ["Embed an image with optional alt text."],
"VYd_YA": ["Open an existing project folder from disk."],
"VZGh8G": ["Fold, unfold, fold all, or unfold all source ranges."],
+ "VZSBXL": ["Semantic ranking activates the first time an agent runs a search."],
"VbqhBI": ["Already at ", ["assetContentPath"], " — reusing."],
"VdXv4D": ["Clone from GitHub"],
"Vlc75q": ["Keep file deleted"],
"VuxVSP": ["Start source completion"],
"VvS_k_": ["Config sharing is now local-only."],
+ "Vz1amp": [
+ "The API key semantic search uses to embed your content. Stored once for this machine and shared across all projects. Turn the feature on per project in This project → Search."
+ ],
"W-ntoW": ["heading-slug"],
"W1qRuM": ["This document is already active in the editor. Use Open to collapse the graph."],
"W5A0Ly": ["An unexpected error occurred."],
@@ -962,6 +987,7 @@
"YPFBfr": [["detail"], " - ", ["kind"], " \"", ["createdPath"], "\" still exists on disk"],
"YSFv5w": ["No git repository — config sharing does not apply here."],
"YTLmTM": ["Remove the parent <0>.git</0> folder"],
+ "YW7Q86": ["Search settings not yet loaded — try again in a moment"],
"Y_kL7X": ["Customize how the editor looks and behaves."],
"YeQufb": ["Share with my team"],
"Yea-K5": ["Restore to this point"],
@@ -1056,6 +1082,9 @@
"a9OzSn": ["No pages found for \"", ["trimmedQuery"], "\""],
"aA51jP": ["Open in claude.ai, opens in browser with prompt pre-filled"],
"aE2_XM": ["One pattern per line — appended to <0>.okignore</0>."],
+ "aENGMB": [
+ "Semantic search adds meaning-based ranking to the search tool so conceptually-related pages surface even without shared keywords."
+ ],
"aIgBKf": [
"Can't nest projects. An Open Knowledge project already exists at <0>",
["rootPath"],
@@ -1129,9 +1158,15 @@
"dD7NPy": ["Outline"],
"dDpCOp": ["Rename succeeded but the sidebar may be out of date — refresh to resync"],
"dEgA5A": ["Cancel"],
+ "dN20LW": [
+ "Semantic search is on, but no API key is set — search falls back to keyword matching. Add one in <0>Settings → Account</0> (it's stored once for your whole machine)."
+ ],
"dOHMA0": ["Failed to move"],
"dOxPd4": ["Footnote"],
"dTaauv": ["Reason: ", ["reason"]],
+ "dUQ7_J": [
+ "Add meaning-based ranking to search so conceptually-related pages surface even when they share no keywords. This setting applies only to this computer."
+ ],
"dURmlQ": ["\"frontmatter\" is a reserved property name"],
"dbhUX1": ["Hide this file"],
"dbtjJh": ["(opens in new tab)"],
@@ -1163,6 +1198,7 @@
"fG0X0e": ["Install ", ["brand"], " →"],
"fIgqfQ": ["Reinstalling"],
"fJgQHX": ["Failed to create file"],
+ "fNr-MX": ["Disable semantic search"],
"fRv-ei": ["Start a new Open Knowledge project."],
"fVcm9t": [["0", "plural", { "one": ["#", " doc"], "other": ["#", " docs"] }]],
"fWi7pn": ["Initialize starter pack"],
@@ -1261,6 +1297,7 @@
"iGCs04": ["No templates available"],
"iH8pgl": ["Back"],
"iKSP5j": ["Word wrap"],
+ "iMkWoF": ["Turn on"],
"iNFKkm": ["No diff."],
"iOHg1t": ["Open embedded URL in new tab"],
"iRbdVc": [
@@ -1330,6 +1367,7 @@
],
"kYL1lH": ["Agent activity"],
"k_pade": ["Config sharing is now shared. Commit the OK files to share with your team."],
+ "kdzzOK": ["Off — search ranks by keyword only. No content leaves this computer."],
"keNFEd": ["Set identity"],
"kj9pAE": ["Creating page"],
"kknhfj": ["No changes"],
@@ -1337,6 +1375,7 @@
"knjOzA": ["Tag value"],
"kolRU1": ["Do you want to permanently delete instead?"],
"ktATCe": ["Could not launch Terminal"],
+ "ku0yLQ": ["Pages are embedded the first time a search needs them."],
"ky9Tuc": ["Edit ", ["0"], " source"],
"l17ab4": ["Could not write project files. Try a different location."],
"l1aQKz": ["Visual editor table"],
@@ -1349,6 +1388,7 @@
"lHwYMr": ["Server error: ", ["status"], " ", ["statusText"]],
"lIn9L9": ["Disable sync"],
"lPzyrU": ["All hunks resolved"],
+ "lVWEIL": ["Indexed ", ["embedded"], " of ", ["total"], " pages."],
"lWNpNB": ["Add cursors or select matching occurrences through CodeMirror."],
"lXKZGw": ["Heading 1"],
"lZAiSP": ["Backlinks response did not match expected shape."],
@@ -1423,6 +1463,7 @@
"nft48x": [
"Stored at <0>.ok/templates/</0> in this project. Available in every folder (folder-scoped templates can override by filename)."
],
+ "nh3wwx": ["Embeddings provider key"],
"njBeMm": ["Both"],
"njU86j": ["Property \"", ["trimmed"], "\" already exists"],
"noH1JX": [
@@ -1437,6 +1478,9 @@
"o2yoFu": ["Reveal in Finder, No workspace"],
"o49mDZ": ["Loading timeline history"],
"o4dcAf": ["Reveal in File Explorer, ", ["hint"]],
+ "o5gfs1": [
+ "This setting is per-machine and isn't shared with collaborators. It needs an API key set with <0>ok embeddings set-key</0>."
+ ],
"oBXkm2": ["Delete code block"],
"oJl7eO": ["Unknown component: ", ["componentName"], " — source editable below"],
"oS1YpJ": ["Open the Project Navigator."],
@@ -1573,6 +1617,9 @@
"skf5Ix": ["Indent or outdent source editor lines."],
"snrVQ7": ["probe failed"],
"ssuwjF": ["Branch"],
+ "stlgky": [
+ "On — your search queries and the text of matching pages are sent to your embeddings provider (OpenAI by default) to compute embeddings."
+ ],
"sxTdZI": ["Loading paths…"],
"sxkWRg": ["Advanced"],
"t-JXbf": ["Notion-style inline row for a downloadable file. Drag-drop also works."],
@@ -1595,6 +1642,7 @@
"tgWuMB": ["Modified"],
"tiWX6q": ["Source editor"],
"tjXez2": ["blog-post"],
+ "tsiTqO": ["Paste your API key"],
"u-9fWA": ["See <0>Architecture</0> for the system overview."],
"u3BOHd": [["fileCount", "plural", { "one": ["#", " file"], "other": ["#", " files"] }]],
"u3YnKw": ["Could not construct share URL."],
@@ -1642,6 +1690,7 @@
"vbK_ZG": ["Files sidebar"],
"ve8vEM": ["Code expired — please try again"],
"vfRlCI": ["Could not copy full path"],
+ "vhAnzJ": ["Turn on semantic search?"],
"viwC-K": ["Failed to remove project."],
"vv7kpg": ["Cancelled"],
"vvWZnu": ["Diff unavailable"],
@@ -1674,6 +1723,7 @@
")"
],
"x57cJf": ["Open link"],
+ "xCJdfg": ["Clear"],
"xCdTnb": ["Close new tab"],
"xDBki3": ["Heading suggestions"],
"xG4tb6": ["Lost connection to \"", ["docName"], "\"."],
@@ -1746,6 +1796,9 @@
"Saves the file to <0>~/Downloads/openknowledge.skill</0> and opens the Claude Desktop App. Requires Node.js or Bun on your PATH."
],
"zAsDLY": ["Graph visualization of document links"],
+ "zEfRFN": [
+ "Using the <0>OK_EMBEDDINGS_API_KEY</0> environment variable (managed outside Open Knowledge)."
+ ],
"zL0LPh": ["Global outside text fields"],
"zLGXna": ["No starter packs available."],
"zLak8b": ["Activate new tab"],
diff --git a/packages/app/src/locales/en/messages.po b/packages/app/src/locales/en/messages.po
index 64a30eb53..045864772 100644
--- a/packages/app/src/locales/en/messages.po
+++ b/packages/app/src/locales/en/messages.po
@@ -400,6 +400,10 @@ msgstr "📝 or assets/icon.png"
msgid "A folder named <0>{sanitized}</0> already has files at this location. Pick a different name."
msgstr "A folder named <0>{sanitized}</0> already has files at this location. Pick a different name."
+#: src/components/settings/SearchSection.tsx
+msgid "A key is set, but the embeddings provider rejected it or was unreachable — search fell back to keyword matching. Check the key in <0>Settings → Account</0> and your provider settings."
+msgstr "A key is set, but the embeddings provider rejected it or was unreachable — search fell back to keyword matching. Check the key in <0>Settings → Account</0> and your provider settings."
+
#: src/editor/slash-command/items.tsx
msgid "A line with a footnote<0><1>[1]</1></0> and a definition shown below."
msgstr "A line with a footnote<0><1>[1]</1></0> and a definition shown below."
@@ -480,6 +484,10 @@ msgstr "Activity"
msgid "Add"
msgstr "Add"
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Add a key"
+msgstr "Add a key"
+
#: src/components/FolderPropertiesCard.tsx
msgid "Add a property"
msgstr "Add a property"
@@ -496,6 +504,10 @@ msgstr "Add another pattern"
msgid "Add cursors or select matching occurrences through CodeMirror."
msgstr "Add cursors or select matching occurrences through CodeMirror."
+#: src/components/settings/SearchSection.tsx
+msgid "Add meaning-based ranking to search so conceptually-related pages surface even when they share no keywords. This setting applies only to this computer."
+msgstr "Add meaning-based ranking to search so conceptually-related pages surface even when they share no keywords. This setting applies only to this computer."
+
#: src/components/McpConsentDialogBody.tsx
msgid "Add Open Knowledge to your AI tools"
msgstr "Add Open Knowledge to your AI tools"
@@ -600,6 +612,10 @@ msgstr "and more"
msgid "Anyone can see"
msgstr "Anyone can see"
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "API key set"
+msgstr "API key set"
+
#: src/components/CreatedItemsList.tsx
msgid "Append-only timeline"
msgstr "Append-only timeline"
@@ -608,6 +624,10 @@ msgstr "Append-only timeline"
msgid "Apply link"
msgstr "Apply link"
+#: src/components/settings/SearchSection.tsx
+msgid "Applying your change"
+msgstr "Applying your change"
+
#: src/components/FileTree.tsx
msgid "Are you sure you want to delete {folderName}/ and all files inside? This action cannot be undone."
msgstr "Are you sure you want to delete {folderName}/ and all files inside? This action cannot be undone."
@@ -835,6 +855,7 @@ msgstr "Can't nest projects. An Open Knowledge project already exists at <0>{roo
#: src/components/NewTemplateDialog.tsx
#: src/components/PublishToGitHubDialog.tsx
#: src/components/SeedDialog.tsx
+#: src/components/settings/SearchSection.tsx
#: src/components/ShareBranchSwitchDialog.tsx
#: src/components/ShareReceiveDialog.tsx
#: src/components/TemplateEditDialog.tsx
@@ -894,6 +915,10 @@ msgstr "Choose whether to commit <0>.ok/</0>, <1>.mcp.json</1> (and per-editor v
msgid "Claude Desktop"
msgstr "Claude Desktop"
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Clear"
+msgstr "Clear"
+
#: src/components/PageHeaderWidgets.tsx
msgid "Clear {keyName}"
msgstr "Clear {keyName}"
@@ -1400,6 +1425,10 @@ msgstr "Could not write project files. Try a different location."
msgid "Couldn't auto-copy"
msgstr "Couldn't auto-copy"
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Couldn't clear the key — please try again."
+msgstr "Couldn't clear the key — please try again."
+
#: src/components/PublishToGitHubDialog.tsx
msgid "Couldn't construct the share URL. Try Done and re-share."
msgstr "Couldn't construct the share URL. Try Done and re-share."
@@ -1484,6 +1513,10 @@ msgstr "Couldn't resolve the conflict for {filePath}."
msgid "Couldn't save template: {error}"
msgstr "Couldn't save template: {error}"
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Couldn't save the key — please try again."
+msgstr "Couldn't save the key — please try again."
+
#: src/components/DiffViewBoundary.tsx
msgid "Couldn't save the resolution for {filePath}."
msgstr "Couldn't save the resolution for {filePath}."
@@ -1700,6 +1733,10 @@ msgstr "Diff unavailable"
msgid "Disable git auto-sync"
msgstr "Disable git auto-sync"
+#: src/components/settings/SearchSection.tsx
+msgid "Disable semantic search"
+msgstr "Disable semantic search"
+
#: src/components/SyncStatusBadge.tsx
msgid "Disable sync"
msgstr "Disable sync"
@@ -1921,6 +1958,14 @@ msgstr "Embed an external page in an inline iframe (docs, demos, Figma, CodeSand
msgid "Embed an image with optional alt text."
msgstr "Embed an image with optional alt text."
+#: src/components/settings/SearchSection.tsx
+msgid "Embeddings are computed only when a search runs and are cached locally under <0>.ok/local</0>."
+msgstr "Embeddings are computed only when a search runs and are cached locally under <0>.ok/local</0>."
+
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Embeddings provider key"
+msgstr "Embeddings provider key"
+
#: src/components/PropertyWidgets.tsx
msgid "Empty"
msgstr "Empty"
@@ -1942,6 +1987,10 @@ msgstr "Enable git auto-sync"
msgid "Enable git auto-sync?"
msgstr "Enable git auto-sync?"
+#: src/components/settings/SearchSection.tsx
+msgid "Enable semantic search"
+msgstr "Enable semantic search"
+
#: src/components/SyncStatusBadge.tsx
msgid "Enable sync"
msgstr "Enable sync"
@@ -2039,6 +2088,10 @@ msgstr "Failed to create page"
msgid "Failed to delete path"
msgstr "Failed to delete path"
+#: src/components/settings/SearchSection.tsx
+msgid "Failed to disable semantic search — {detail}"
+msgstr "Failed to disable semantic search — {detail}"
+
#: src/hooks/use-enable-sync-with-confirm.ts
msgid "Failed to disable sync — {detail}"
msgstr "Failed to disable sync — {detail}"
@@ -2047,6 +2100,10 @@ msgstr "Failed to disable sync — {detail}"
msgid "Failed to duplicate path"
msgstr "Failed to duplicate path"
+#: src/components/settings/SearchSection.tsx
+msgid "Failed to enable semantic search — {detail}"
+msgstr "Failed to enable semantic search — {detail}"
+
#: src/hooks/use-enable-sync-with-confirm.ts
msgid "Failed to enable sync — {detail}"
msgstr "Failed to enable sync — {detail}"
@@ -2597,6 +2654,10 @@ msgstr "Indent or outdent the current list item."
msgid "Indented blockquote for citations."
msgstr "Indented blockquote for citations."
+#: src/components/settings/SearchSection.tsx
+msgid "Indexed {embedded} of {total} pages."
+msgstr "Indexed {embedded} of {total} pages."
+
#: src/components/FrontmatterRow.tsx
msgid "inherited"
msgstr "inherited"
@@ -3486,6 +3547,10 @@ msgstr "Numbered list of items."
msgid "of {totalPages}"
msgstr "of {totalPages}"
+#: src/components/settings/SearchSection.tsx
+msgid "Off — search ranks by keyword only. No content leaves this computer."
+msgstr "Off — search ranks by keyword only. No content leaves this computer."
+
#: src/components/SyncStatusBadge.tsx
msgid "Offline"
msgstr "Offline"
@@ -3513,6 +3578,10 @@ msgstr "OK Desktop"
msgid "ok ui responded but server.lock has no port yet"
msgstr "ok ui responded but server.lock has no port yet"
+#: src/components/settings/SearchSection.tsx
+msgid "On — your search queries and the text of matching pages are sent to your embeddings provider (OpenAI by default) to compute embeddings."
+msgstr "On — your search queries and the text of matching pages are sent to your embeddings provider (OpenAI by default) to compute embeddings."
+
#: src/components/ConsentDialogBody.tsx
msgid "One pattern per line — appended to <0>.okignore</0>."
msgstr "One pattern per line — appended to <0>.okignore</0>."
@@ -3917,6 +3986,10 @@ msgstr "Page thumbnails"
msgid "page-name"
msgstr "page-name"
+#: src/components/settings/SearchSection.tsx
+msgid "Pages are embedded the first time a search needs them."
+msgstr "Pages are embedded the first time a search needs them."
+
#: src/lib/keyboard-shortcuts.ts
msgid "Paragraph"
msgstr "Paragraph"
@@ -3945,6 +4018,10 @@ msgstr "Paste URL, owner/repo, or search your repos"
msgid "Paste was incomplete — {chunksCompleted} of {totalChunks} chunks landed.{restoreSuffix}"
msgstr "Paste was incomplete — {chunksCompleted} of {totalChunks} chunks landed.{restoreSuffix}"
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Paste your API key"
+msgstr "Paste your API key"
+
#: src/components/NewItemDialog.tsx
msgid "Path cannot contain \"..\""
msgstr "Path cannot contain \"..\""
@@ -4339,6 +4416,10 @@ msgstr "Replace current match"
msgid "Replace current match from field"
msgstr "Replace current match from field"
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Replace key"
+msgstr "Replace key"
+
#: src/lib/keyboard-shortcuts.ts
msgid "Replace the active match from the replace field."
msgstr "Replace the active match from the replace field."
@@ -4495,6 +4576,7 @@ msgid "Sandboxed HTML embed — write HTML, see the rendered preview live."
msgstr "Sandboxed HTML embed — write HTML, see the rendered preview live."
#: src/components/AuthModal.tsx
+#: src/components/settings/EmbeddingsKeySection.tsx
#: src/components/TemplateEditDialog.tsx
#: src/editor/components/CodePreviewEditModal.tsx
#: src/editor/extensions/InternalLinkPropPanel.tsx
@@ -4535,6 +4617,7 @@ msgstr "Saved as <0>{slug}.md</0>"
msgid "Saves the file to <0>~/Downloads/openknowledge.skill</0> and opens the Claude Desktop App. Requires Node.js or Bun on your PATH."
msgstr "Saves the file to <0>~/Downloads/openknowledge.skill</0> and opens the Claude Desktop App. Requires Node.js or Bun on your PATH."
+#: src/components/settings/EmbeddingsKeySection.tsx
#: src/components/TemplateEditDialog.tsx
#: src/components/TimelinePanel.tsx
msgid "Saving"
@@ -4550,6 +4633,7 @@ msgstr "Scope: "
#: src/components/CommandPalette.tsx
#: src/components/empty-state/KeyboardHintsFooter.tsx
+#: src/components/settings/SettingsDialogShell.tsx
#: src/components/SidebarSearchBar.tsx
#: src/lib/keyboard-shortcuts.ts
msgid "Search"
@@ -4587,6 +4671,10 @@ msgstr "Search recent projects"
msgid "Search recent projects..."
msgstr "Search recent projects..."
+#: src/components/settings/SearchSection.tsx
+msgid "Search settings not yet loaded — try again in a moment"
+msgstr "Search settings not yet loaded — try again in a moment"
+
#: src/components/NewItemDialog.tsx
msgid "Search templates"
msgstr "Search templates"
@@ -4660,6 +4748,22 @@ msgstr "Selection must be inside the project"
msgid "Selection statistics"
msgstr "Selection statistics"
+#: src/components/settings/SearchSection.tsx
+msgid "Semantic ranking activates the first time an agent runs a search."
+msgstr "Semantic ranking activates the first time an agent runs a search."
+
+#: src/components/settings/SearchSection.tsx
+msgid "Semantic search"
+msgstr "Semantic search"
+
+#: src/components/settings/SearchSection.tsx
+msgid "Semantic search adds meaning-based ranking to the search tool so conceptually-related pages surface even without shared keywords."
+msgstr "Semantic search adds meaning-based ranking to the search tool so conceptually-related pages surface even without shared keywords."
+
+#: src/components/settings/SearchSection.tsx
+msgid "Semantic search is on, but no API key is set — search falls back to keyword matching. Add one in <0>Settings → Account</0> (it's stored once for your whole machine)."
+msgstr "Semantic search is on, but no API key is set — search falls back to keyword matching. Add one in <0>Settings → Account</0> (it's stored once for your whole machine)."
+
#: src/editor/slash-command/items.tsx
msgid "Separator"
msgstr "Separator"
@@ -4977,6 +5081,10 @@ msgstr "Still loading"
msgid "Stored at <0>.ok/templates/</0> in this project. Available in every folder (folder-scoped templates can override by filename)."
msgstr "Stored at <0>.ok/templates/</0> in this project. Available in every folder (folder-scoped templates can override by filename)."
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Stored on this machine."
+msgstr "Stored on this machine."
+
#: src/lib/keyboard-shortcuts.ts
msgid "Strikethrough"
msgstr "Strikethrough"
@@ -5185,6 +5293,10 @@ msgstr "That folder isn't an Open Knowledge project yet. Initialize it and open?
msgid "The {targetNoun} could not be opened — try navigating to it manually."
msgstr "The {targetNoun} could not be opened — try navigating to it manually."
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "The API key semantic search uses to embed your content. Stored once for this machine and shared across all projects. Turn the feature on per project in This project → Search."
+msgstr "The API key semantic search uses to embed your content. Stored once for this machine and shared across all projects. Turn the feature on per project in This project → Search."
+
#: src/components/InstallInClaudeDesktopDialog.tsx
msgid "The Claude Desktop App didn't open the file. Is it installed? ({msg})"
msgstr "The Claude Desktop App didn't open the file. Is it installed? ({msg})"
@@ -5313,6 +5425,14 @@ msgstr "This project lives only on this computer. Connect it to GitHub to back i
msgid "This project's running server doesn't support live editing. Restart Open Knowledge to fix."
msgstr "This project's running server doesn't support live editing. Restart Open Knowledge to fix."
+#: src/components/settings/SearchSection.tsx
+msgid "This sends content off your machine"
+msgstr "This sends content off your machine"
+
+#: src/components/settings/SearchSection.tsx
+msgid "This setting is per-machine and isn't shared with collaborators. It needs an API key set with <0>ok embeddings set-key</0>."
+msgstr "This setting is per-machine and isn't shared with collaborators. It needs an API key set with <0>ok embeddings set-key</0>."
+
#: src/components/TemplateDeleteDialog.tsx
msgid "This template lives in a parent folder — deleting it affects every folder beneath it that doesn't define its own version."
msgstr "This template lives in a parent folder — deleting it affects every folder beneath it that doesn't define its own version."
@@ -5416,6 +5536,14 @@ msgstr "Tree View Options"
msgid "Try again"
msgstr "Try again"
+#: src/components/settings/SearchSection.tsx
+msgid "Turn on"
+msgstr "Turn on"
+
+#: src/components/settings/SearchSection.tsx
+msgid "Turn on semantic search?"
+msgstr "Turn on semantic search?"
+
#: src/editor/components/Pdf.tsx
msgid "Two-page (even)"
msgstr "Two-page (even)"
@@ -5585,6 +5713,10 @@ msgstr "Use letters, digits, <0>-</0> or <1>_</1> only."
msgid "User"
msgstr "User"
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Using the <0>OK_EMBEDDINGS_API_KEY</0> environment variable (managed outside Open Knowledge)."
+msgstr "Using the <0>OK_EMBEDDINGS_API_KEY</0> environment variable (managed outside Open Knowledge)."
+
#: src/components/FolderPropertiesCard.tsx
msgid "Value is required"
msgstr "Value is required"
@@ -5803,6 +5935,10 @@ msgstr "Your knowledge base is now on GitHub at <0>{ownerLogin}/{repoName}</0>."
msgid "Your repositories"
msgstr "Your repositories"
+#: src/components/settings/SearchSection.tsx
+msgid "Your search queries and the full text of matching pages are sent to your embeddings provider (OpenAI by default) to compute embeddings."
+msgstr "Your search queries and the full text of matching pages are sent to your embeddings provider (OpenAI by default) to compute embeddings."
+
#: src/components/PublishToGitHubDialog.tsx
msgid "Your share link is ready below."
msgstr "Your share link is ready below."
diff --git a/packages/app/src/locales/pseudo/messages.json b/packages/app/src/locales/pseudo/messages.json
index 10893b13f..9fb928699 100644
--- a/packages/app/src/locales/pseudo/messages.json
+++ b/packages/app/src/locales/pseudo/messages.json
@@ -119,6 +119,7 @@
"3SDfZl": [
"<0>Ḿĩŕŕōŕ — ƥĩćķ à śōũŕćē.</0> Śēţ <1>śŕć</1> + <2>àńćĥōŕ</2> vĩà ţĥē ƥŕōƥēŕţŷ ƥàńēĺ ţō ƥōĩńţ àţ à <3><MirrorSource></3> ēĺśēŵĥēŕē."
],
+ "3TCafD": ["Śţōŕēď ōń ţĥĩś ḿàćĥĩńē."],
"3VbwPp": ["Ōƥēń ", ["keyName"], " ĩń ƀŕōŵśēŕ"],
"3VsNTI": ["Ĺōàďĩńĝ ďōćũḿēńţ"],
"3YvS-c": ["Ńēŵ ţàƀ"],
@@ -134,6 +135,7 @@
"3oh0nQ": ["Ńēśţēď .ōķĩĝńōŕē ēŕŕōŕ ĩń ", ["path"]],
"3pU8e-": ["Ƥĩń ţàƀ"],
"3wiBAs": ["Ŵĥēŕē śĥōũĺď ĩţ ĺĩvē?"],
+ "3y6XTb": ["Śēḿàńţĩć śēàŕćĥ"],
"3zyg_A": ["Ĩĝńōŕē ƥàţţēŕńś"],
"438Fca": ["Ćĺōńēď śũććēśśƒũĺĺŷ, ƀũţ ćōũĺď ńōţ ōƥēń ţĥē ƥŕōĴēćţ."],
"44DSl0": [
@@ -197,6 +199,7 @@
],
"5kWklo": ["(śōḿē ḿàŷ àĺŕēàďŷ ƀē ĥĩďďēń ƀŷ ōţĥēŕ ŕũĺēś)"],
"5lWFkC": ["Śĩĝń ĩń"],
+ "5mksg0": ["Ƒàĩĺēď ţō ďĩśàƀĺē śēḿàńţĩć śēàŕćĥ — ", ["detail"]],
"5o4lc1": [
"Ōƥēń Ķńōŵĺēďĝē ŵĩĺĺ ƀē ĩńĩţĩàĺĩźēď àţ <0>",
["gitRoot"],
@@ -259,6 +262,7 @@
"7KeV5F": ["Ũńďō àĺĺ ƒàĩĺēď: ", ["message"]],
"7OhaRy": ["Śōũŕćē śēàŕćĥ"],
"7OwvEL": ["Śĥōŵ ĤŢḾĹ ƥŕēvĩēŵ"],
+ "7PNNNs": ["Ţĥĩś śēńďś ćōńţēńţ ōƒƒ ŷōũŕ ḿàćĥĩńē"],
"7PzzBU": ["Ũśēŕ"],
"7_bFEL": ["Ďŕàĝ ", ["patternText"], " ţō ŕēōŕďēŕ"],
"7d1a0d": ["Ƥũƀĺĩć"],
@@ -278,6 +282,9 @@
["fileName"],
") ĩń ţĥē ƥŕōĴēćţ ŕōōţ"
],
+ "8BsGpU": [
+ "À ķēŷ ĩś śēţ, ƀũţ ţĥē ēḿƀēďďĩńĝś ƥŕōvĩďēŕ ŕēĴēćţēď ĩţ ōŕ ŵàś ũńŕēàćĥàƀĺē — śēàŕćĥ ƒēĺĺ ƀàćķ ţō ķēŷŵōŕď ḿàţćĥĩńĝ. Ćĥēćķ ţĥē ķēŷ ĩń <0>Śēţţĩńĝś → Àććōũńţ</0> àńď ŷōũŕ ƥŕōvĩďēŕ śēţţĩńĝś."
+ ],
"8F1i42": ["Ƥàĝē ńōţ ƒōũńď"],
"8FqTCr": [
[
@@ -355,14 +362,17 @@
],
"AVW-K9": ["Ƒàĩĺēď ţō ĺōàď ōŕƥĥàń ƥàĝēś"],
"AWOSPo": ["Źōōḿ ĩń"],
+ "AdcGcG": ["Àƥƥĺŷĩńĝ ŷōũŕ ćĥàńĝē"],
"AddR83": ["Ćōũĺď ńōţ ćŕēàţē ƥŕōĴēćţ. Ţŕŷ àĝàĩń ōŕ ƥĩćķ à ďĩƒƒēŕēńţ ĺōćàţĩōń."],
"AeHuzT": ["# Ōńē ƥàţţēŕń ƥēŕ ĺĩńē.\n# Ēxàḿƥĺēś:\n# ďŕàƒţś/\n# *.ďŕàƒţ.ḿď\n# !ķēēƥ.ḿď"],
"AeXO77": ["Àććōũńţ"],
"AiNMQ7": [
"Ţĥĩś ƥŕōĴēćţ ĺĩvēś ōńĺŷ ōń ţĥĩś ćōḿƥũţēŕ. Ćōńńēćţ ĩţ ţō ĜĩţĤũƀ ţō ƀàćķ ĩţ ũƥ àńď śĥàŕē ĩţ ŵĩţĥ ōţĥēŕ ƥēōƥĺē."
],
+ "Aif3E7": ["Ēńàƀĺē śēḿàńţĩć śēàŕćĥ"],
"Am78jY": ["Ďŕàĝ ", ["keyName"], " ţō ŕēōŕďēŕ"],
"AoX5ld": [["folders", "plural", { "one": ["#", " ƒōĺďēŕ"], "other": ["#", " ƒōĺďēŕś"] }]],
+ "AqnqMJ": ["ÀƤĨ ķēŷ śēţ"],
"ArBWJf": ["Ũńďō ĺàśţ ēďĩţ"],
"B11p-O": ["Ēďĩţ ḿàŕķďōŵń ĺĩńķ"],
"B2DrPb": ["Àũţō-śàvē"],
@@ -406,6 +416,7 @@
"COr1Y7": [
"Ƥĩćķ <0>ōƥēńķńōŵĺēďĝē.śķĩĺĺ</0> ƒŕōḿ ŷōũŕ <1>Ďōŵńĺōàďś</1> ƒōĺďēŕ. Ĩţ ēńàƀĺēś àũţōḿàţĩćàĺĺŷ."
],
+ "CSL0l1": ["Ćōũĺďń'ţ ćĺēàŕ ţĥē ķēŷ — ƥĺēàśē ţŕŷ àĝàĩń."],
"CTJLpz": ["\"", ["trimmed"], "\" àĺŕēàďŷ ēxĩśţś ĥēŕē"],
"CU3AiE": ["Śŵĩţćĥĩńĝ ƀŕàńćĥēś"],
"CXj9C-": ["Ńēŵ ƥŕōƥēŕţŷ ńàḿē"],
@@ -414,6 +425,7 @@
],
"C_cPoW": ["Ŷōũŕ ĜĩţĤũƀ śēśśĩōń ēxƥĩŕēď — śĩĝń ĩń àĝàĩń ţō vēŕĩƒŷ ƥũśĥ àććēśś."],
"ChAtFg": ["Ŷōũ ďōń'ţ ĥàvē ƥēŕḿĩśśĩōń ţō ƥũśĥ ţō ţĥĩś ŕēƥō"],
+ "Ck9XRt": ["Ćōũĺďń'ţ śàvē ţĥē ķēŷ — ƥĺēàśē ţŕŷ àĝàĩń."],
"CvsD6Y": ["vĩà ", ["viaTags"]],
"D4H2N6": ["Ŕēḿōvē ", ["tag"]],
"D8OV7J": ["Ōńĺŷ ćōĺĺàƀōŕàţōŕś"],
@@ -465,6 +477,9 @@
"EiVgl5": ["Ŷōũ ďōń'ţ ĥàvē ƥēŕḿĩśśĩōń ţō ƥũśĥ ţō ţĥĩś ŕēƥō."],
"Eji13z": ["Ţōĝĝĺē ĥĩĝĥĺĩĝĥţ ƒōŕḿàţţĩńĝ."],
"El7NbA": ["Ĥēàďĩńĝ 2"],
+ "Emz5BG": [
+ "Ēḿƀēďďĩńĝś àŕē ćōḿƥũţēď ōńĺŷ ŵĥēń à śēàŕćĥ ŕũńś àńď àŕē ćàćĥēď ĺōćàĺĺŷ ũńďēŕ <0>.ōķ/ĺōćàĺ</0>."
+ ],
"EqrC_R": ["Ũńďō ƒàĩĺēď: ", ["message"]],
"EqtBXb": ["Vĩśũàĺ ēďĩţōŕ ƒĩńď ōƥēń"],
"ErRi6L": ["Ţēḿƥĺàţē \"", ["label"], "\" ďēĺēţēď"],
@@ -600,6 +615,7 @@
"K0N7zS": ["Ćōńƒĩĝ śĥàŕĩńĝ śţàţũś ŕēàď ƒàĩĺēď"],
"K1ckLX": ["Ƒàĩĺēď ţō ćŕēàţē ƒōĺďēŕ"],
"K6jjWB": [["packName"], " ŵàś àĺŕēàďŷ śēţ ũƥ. Ńōţĥĩńĝ ţō ďō."],
+ "K6jxcj": ["Àďď à ķēŷ"],
"K8-WWn": ["Ḿōvē ƀēţŵēēń ţàƀĺē ćēĺĺś"],
"K9J00P": ["Ćĺōńē ƒàĩĺēď — ćōńńēćţĩōń ēŕŕōŕ"],
"KDuPiY": ["Ũƥĺōàďĩńĝ ", ["fileName"]],
@@ -643,6 +659,7 @@
"LRgZAt": [["owner"], "/", ["name"], " àĺŕēàďŷ ēxĩśţś"],
"LXdWeS": ["Ḿēďĩũḿ śēćţĩōń ĥēàďĩńĝ."],
"LXfNU-": ["Ńàvĩĝàţĩōń àńď śũĝĝēśţĩōńś"],
+ "LYufDw": ["Ƒàĩĺēď ţō ēńàƀĺē śēḿàńţĩć śēàŕćĥ — ", ["detail"]],
"Lcehxn": ["Śàvē ŕēśōĺũţĩōń"],
"LdABLa": ["Ćōũĺď ńōţ ōƥēń ", ["target"], ". Ţŕŷ ōƥēńĩńĝ ĩţ ḿàńũàĺĺŷ."],
"LhWYpm": ["Ōţĥēŕ ŌĶ ƥàţĥś àŕē ţŕàćķēď ũƥśţŕēàḿ:"],
@@ -858,6 +875,7 @@
"UTNVyS": ["Śēĺēćţ ţĥē ƒũĺĺ śōũŕćē ďōćũḿēńţ."],
"UTZ97u": ["Śēĺēćţĩōń ḿũśţ ƀē ĩńśĩďē ţĥē ƥŕōĴēćţ"],
"UVLlMV": ["Ćōũĺďń'ţ ŕēàćĥ ĜĩţĤũƀ. Ţŕŷ àĝàĩń?"],
+ "UW93oq": ["Ŕēƥĺàćē ķēŷ"],
"UYcGIW": ["Ćōńńēćţĩńĝ — ŵàĩţĩńĝ ƒōŕ ćōĺĺàƀ śēŕvēŕ"],
"Ua78MH": ["ńō ŕēśƥōńśē"],
"UaXXIF": [
@@ -869,6 +887,9 @@
["src"],
"</1> ĩś ńō ĺōńĝēŕ àvàĩĺàƀĺē."
],
+ "UfW9Kr": [
+ "Ŷōũŕ śēàŕćĥ ǫũēŕĩēś àńď ţĥē ƒũĺĺ ţēxţ ōƒ ḿàţćĥĩńĝ ƥàĝēś àŕē śēńţ ţō ŷōũŕ ēḿƀēďďĩńĝś ƥŕōvĩďēŕ (ŌƥēńÀĨ ƀŷ ďēƒàũĺţ) ţō ćōḿƥũţē ēḿƀēďďĩńĝś."
+ ],
"UkQGhd": ["Ĥĩďē ŕēƥĺàćē"],
"Up2_7p": ["Ćōƥŷ ƒũĺĺ ƥàţĥ"],
"Uuj_R4": ["Ńēŵ ƒŕōḿ ţēḿƥĺàţē"],
@@ -884,11 +905,15 @@
"VUijjF": ["Ēḿƀēď àń ĩḿàĝē ŵĩţĥ ōƥţĩōńàĺ àĺţ ţēxţ."],
"VYd_YA": ["Ōƥēń àń ēxĩśţĩńĝ ƥŕōĴēćţ ƒōĺďēŕ ƒŕōḿ ďĩśķ."],
"VZGh8G": ["Ƒōĺď, ũńƒōĺď, ƒōĺď àĺĺ, ōŕ ũńƒōĺď àĺĺ śōũŕćē ŕàńĝēś."],
+ "VZSBXL": ["Śēḿàńţĩć ŕàńķĩńĝ àćţĩvàţēś ţĥē ƒĩŕśţ ţĩḿē àń àĝēńţ ŕũńś à śēàŕćĥ."],
"VbqhBI": ["Àĺŕēàďŷ àţ ", ["assetContentPath"], " — ŕēũśĩńĝ."],
"VdXv4D": ["Ćĺōńē ƒŕōḿ ĜĩţĤũƀ"],
"Vlc75q": ["Ķēēƥ ƒĩĺē ďēĺēţēď"],
"VuxVSP": ["Śţàŕţ śōũŕćē ćōḿƥĺēţĩōń"],
"VvS_k_": ["Ćōńƒĩĝ śĥàŕĩńĝ ĩś ńōŵ ĺōćàĺ-ōńĺŷ."],
+ "Vz1amp": [
+ "Ţĥē ÀƤĨ ķēŷ śēḿàńţĩć śēàŕćĥ ũśēś ţō ēḿƀēď ŷōũŕ ćōńţēńţ. Śţōŕēď ōńćē ƒōŕ ţĥĩś ḿàćĥĩńē àńď śĥàŕēď àćŕōśś àĺĺ ƥŕōĴēćţś. Ţũŕń ţĥē ƒēàţũŕē ōń ƥēŕ ƥŕōĴēćţ ĩń Ţĥĩś ƥŕōĴēćţ → Śēàŕćĥ."
+ ],
"W-ntoW": ["ĥēàďĩńĝ-śĺũĝ"],
"W1qRuM": ["Ţĥĩś ďōćũḿēńţ ĩś àĺŕēàďŷ àćţĩvē ĩń ţĥē ēďĩţōŕ. Ũśē Ōƥēń ţō ćōĺĺàƥśē ţĥē ĝŕàƥĥ."],
"W5A0Ly": ["Àń ũńēxƥēćţēď ēŕŕōŕ ōććũŕŕēď."],
@@ -962,6 +987,7 @@
"YPFBfr": [["detail"], " - ", ["kind"], " \"", ["createdPath"], "\" śţĩĺĺ ēxĩśţś ōń ďĩśķ"],
"YSFv5w": ["Ńō ĝĩţ ŕēƥōśĩţōŕŷ — ćōńƒĩĝ śĥàŕĩńĝ ďōēś ńōţ àƥƥĺŷ ĥēŕē."],
"YTLmTM": ["Ŕēḿōvē ţĥē ƥàŕēńţ <0>.ĝĩţ</0> ƒōĺďēŕ"],
+ "YW7Q86": ["Śēàŕćĥ śēţţĩńĝś ńōţ ŷēţ ĺōàďēď — ţŕŷ àĝàĩń ĩń à ḿōḿēńţ"],
"Y_kL7X": ["Ćũśţōḿĩźē ĥōŵ ţĥē ēďĩţōŕ ĺōōķś àńď ƀēĥàvēś."],
"YeQufb": ["Śĥàŕē ŵĩţĥ ḿŷ ţēàḿ"],
"Yea-K5": ["Ŕēśţōŕē ţō ţĥĩś ƥōĩńţ"],
@@ -1056,6 +1082,9 @@
"a9OzSn": ["Ńō ƥàĝēś ƒōũńď ƒōŕ \"", ["trimmedQuery"], "\""],
"aA51jP": ["Ōƥēń ĩń ćĺàũďē.àĩ, ōƥēńś ĩń ƀŕōŵśēŕ ŵĩţĥ ƥŕōḿƥţ ƥŕē-ƒĩĺĺēď"],
"aE2_XM": ["Ōńē ƥàţţēŕń ƥēŕ ĺĩńē — àƥƥēńďēď ţō <0>.ōķĩĝńōŕē</0>."],
+ "aENGMB": [
+ "Śēḿàńţĩć śēàŕćĥ àďďś ḿēàńĩńĝ-ƀàśēď ŕàńķĩńĝ ţō ţĥē śēàŕćĥ ţōōĺ śō ćōńćēƥţũàĺĺŷ-ŕēĺàţēď ƥàĝēś śũŕƒàćē ēvēń ŵĩţĥōũţ śĥàŕēď ķēŷŵōŕďś."
+ ],
"aIgBKf": [
"Ćàń'ţ ńēśţ ƥŕōĴēćţś. Àń Ōƥēń Ķńōŵĺēďĝē ƥŕōĴēćţ àĺŕēàďŷ ēxĩśţś àţ <0>",
["rootPath"],
@@ -1129,9 +1158,15 @@
"dD7NPy": ["Ōũţĺĩńē"],
"dDpCOp": ["Ŕēńàḿē śũććēēďēď ƀũţ ţĥē śĩďēƀàŕ ḿàŷ ƀē ōũţ ōƒ ďàţē — ŕēƒŕēśĥ ţō ŕēśŷńć"],
"dEgA5A": ["Ćàńćēĺ"],
+ "dN20LW": [
+ "Śēḿàńţĩć śēàŕćĥ ĩś ōń, ƀũţ ńō ÀƤĨ ķēŷ ĩś śēţ — śēàŕćĥ ƒàĺĺś ƀàćķ ţō ķēŷŵōŕď ḿàţćĥĩńĝ. Àďď ōńē ĩń <0>Śēţţĩńĝś → Àććōũńţ</0> (ĩţ'ś śţōŕēď ōńćē ƒōŕ ŷōũŕ ŵĥōĺē ḿàćĥĩńē)."
+ ],
"dOHMA0": ["Ƒàĩĺēď ţō ḿōvē"],
"dOxPd4": ["Ƒōōţńōţē"],
"dTaauv": ["Ŕēàśōń: ", ["reason"]],
+ "dUQ7_J": [
+ "Àďď ḿēàńĩńĝ-ƀàśēď ŕàńķĩńĝ ţō śēàŕćĥ śō ćōńćēƥţũàĺĺŷ-ŕēĺàţēď ƥàĝēś śũŕƒàćē ēvēń ŵĥēń ţĥēŷ śĥàŕē ńō ķēŷŵōŕďś. Ţĥĩś śēţţĩńĝ àƥƥĺĩēś ōńĺŷ ţō ţĥĩś ćōḿƥũţēŕ."
+ ],
"dURmlQ": ["\"ƒŕōńţḿàţţēŕ\" ĩś à ŕēśēŕvēď ƥŕōƥēŕţŷ ńàḿē"],
"dbhUX1": ["Ĥĩďē ţĥĩś ƒĩĺē"],
"dbtjJh": ["(ōƥēńś ĩń ńēŵ ţàƀ)"],
@@ -1163,6 +1198,7 @@
"fG0X0e": ["Ĩńśţàĺĺ ", ["brand"], " →"],
"fIgqfQ": ["Ŕēĩńśţàĺĺĩńĝ"],
"fJgQHX": ["Ƒàĩĺēď ţō ćŕēàţē ƒĩĺē"],
+ "fNr-MX": ["Ďĩśàƀĺē śēḿàńţĩć śēàŕćĥ"],
"fRv-ei": ["Śţàŕţ à ńēŵ Ōƥēń Ķńōŵĺēďĝē ƥŕōĴēćţ."],
"fVcm9t": [["0", "plural", { "one": ["#", " ďōć"], "other": ["#", " ďōćś"] }]],
"fWi7pn": ["Ĩńĩţĩàĺĩźē śţàŕţēŕ ƥàćķ"],
@@ -1261,6 +1297,7 @@
"iGCs04": ["Ńō ţēḿƥĺàţēś àvàĩĺàƀĺē"],
"iH8pgl": ["ßàćķ"],
"iKSP5j": ["Ŵōŕď ŵŕàƥ"],
+ "iMkWoF": ["Ţũŕń ōń"],
"iNFKkm": ["Ńō ďĩƒƒ."],
"iOHg1t": ["Ōƥēń ēḿƀēďďēď ŨŔĹ ĩń ńēŵ ţàƀ"],
"iRbdVc": [
@@ -1330,6 +1367,7 @@
],
"kYL1lH": ["Àĝēńţ àćţĩvĩţŷ"],
"k_pade": ["Ćōńƒĩĝ śĥàŕĩńĝ ĩś ńōŵ śĥàŕēď. Ćōḿḿĩţ ţĥē ŌĶ ƒĩĺēś ţō śĥàŕē ŵĩţĥ ŷōũŕ ţēàḿ."],
+ "kdzzOK": ["Ōƒƒ — śēàŕćĥ ŕàńķś ƀŷ ķēŷŵōŕď ōńĺŷ. Ńō ćōńţēńţ ĺēàvēś ţĥĩś ćōḿƥũţēŕ."],
"keNFEd": ["Śēţ ĩďēńţĩţŷ"],
"kj9pAE": ["Ćŕēàţĩńĝ ƥàĝē"],
"kknhfj": ["Ńō ćĥàńĝēś"],
@@ -1337,6 +1375,7 @@
"knjOzA": ["Ţàĝ vàĺũē"],
"kolRU1": ["Ďō ŷōũ ŵàńţ ţō ƥēŕḿàńēńţĺŷ ďēĺēţē ĩńśţēàď?"],
"ktATCe": ["Ćōũĺď ńōţ ĺàũńćĥ Ţēŕḿĩńàĺ"],
+ "ku0yLQ": ["Ƥàĝēś àŕē ēḿƀēďďēď ţĥē ƒĩŕśţ ţĩḿē à śēàŕćĥ ńēēďś ţĥēḿ."],
"ky9Tuc": ["Ēďĩţ ", ["0"], " śōũŕćē"],
"l17ab4": ["Ćōũĺď ńōţ ŵŕĩţē ƥŕōĴēćţ ƒĩĺēś. Ţŕŷ à ďĩƒƒēŕēńţ ĺōćàţĩōń."],
"l1aQKz": ["Vĩśũàĺ ēďĩţōŕ ţàƀĺē"],
@@ -1349,6 +1388,7 @@
"lHwYMr": ["Śēŕvēŕ ēŕŕōŕ: ", ["status"], " ", ["statusText"]],
"lIn9L9": ["Ďĩśàƀĺē śŷńć"],
"lPzyrU": ["Àĺĺ ĥũńķś ŕēśōĺvēď"],
+ "lVWEIL": ["Ĩńďēxēď ", ["embedded"], " ōƒ ", ["total"], " ƥàĝēś."],
"lWNpNB": ["Àďď ćũŕśōŕś ōŕ śēĺēćţ ḿàţćĥĩńĝ ōććũŕŕēńćēś ţĥŕōũĝĥ ĆōďēḾĩŕŕōŕ."],
"lXKZGw": ["Ĥēàďĩńĝ 1"],
"lZAiSP": ["ßàćķĺĩńķś ŕēśƥōńśē ďĩď ńōţ ḿàţćĥ ēxƥēćţēď śĥàƥē."],
@@ -1423,6 +1463,7 @@
"nft48x": [
"Śţōŕēď àţ <0>.ōķ/ţēḿƥĺàţēś/</0> ĩń ţĥĩś ƥŕōĴēćţ. Àvàĩĺàƀĺē ĩń ēvēŕŷ ƒōĺďēŕ (ƒōĺďēŕ-śćōƥēď ţēḿƥĺàţēś ćàń ōvēŕŕĩďē ƀŷ ƒĩĺēńàḿē)."
],
+ "nh3wwx": ["Ēḿƀēďďĩńĝś ƥŕōvĩďēŕ ķēŷ"],
"njBeMm": ["ßōţĥ"],
"njU86j": ["Ƥŕōƥēŕţŷ \"", ["trimmed"], "\" àĺŕēàďŷ ēxĩśţś"],
"noH1JX": [
@@ -1437,6 +1478,9 @@
"o2yoFu": ["Ŕēvēàĺ ĩń Ƒĩńďēŕ, Ńō ŵōŕķśƥàćē"],
"o49mDZ": ["Ĺōàďĩńĝ ţĩḿēĺĩńē ĥĩśţōŕŷ"],
"o4dcAf": ["Ŕēvēàĺ ĩń Ƒĩĺē Ēxƥĺōŕēŕ, ", ["hint"]],
+ "o5gfs1": [
+ "Ţĥĩś śēţţĩńĝ ĩś ƥēŕ-ḿàćĥĩńē àńď ĩśń'ţ śĥàŕēď ŵĩţĥ ćōĺĺàƀōŕàţōŕś. Ĩţ ńēēďś àń ÀƤĨ ķēŷ śēţ ŵĩţĥ <0>ōķ ēḿƀēďďĩńĝś śēţ-ķēŷ</0>."
+ ],
"oBXkm2": ["Ďēĺēţē ćōďē ƀĺōćķ"],
"oJl7eO": ["Ũńķńōŵń ćōḿƥōńēńţ: ", ["componentName"], " — śōũŕćē ēďĩţàƀĺē ƀēĺōŵ"],
"oS1YpJ": ["Ōƥēń ţĥē ƤŕōĴēćţ Ńàvĩĝàţōŕ."],
@@ -1573,6 +1617,9 @@
"skf5Ix": ["Ĩńďēńţ ōŕ ōũţďēńţ śōũŕćē ēďĩţōŕ ĺĩńēś."],
"snrVQ7": ["ƥŕōƀē ƒàĩĺēď"],
"ssuwjF": ["ßŕàńćĥ"],
+ "stlgky": [
+ "Ōń — ŷōũŕ śēàŕćĥ ǫũēŕĩēś àńď ţĥē ţēxţ ōƒ ḿàţćĥĩńĝ ƥàĝēś àŕē śēńţ ţō ŷōũŕ ēḿƀēďďĩńĝś ƥŕōvĩďēŕ (ŌƥēńÀĨ ƀŷ ďēƒàũĺţ) ţō ćōḿƥũţē ēḿƀēďďĩńĝś."
+ ],
"sxTdZI": ["Ĺōàďĩńĝ ƥàţĥś…"],
"sxkWRg": ["Àďvàńćēď"],
"t-JXbf": ["Ńōţĩōń-śţŷĺē ĩńĺĩńē ŕōŵ ƒōŕ à ďōŵńĺōàďàƀĺē ƒĩĺē. Ďŕàĝ-ďŕōƥ àĺśō ŵōŕķś."],
@@ -1595,6 +1642,7 @@
"tgWuMB": ["Ḿōďĩƒĩēď"],
"tiWX6q": ["Śōũŕćē ēďĩţōŕ"],
"tjXez2": ["ƀĺōĝ-ƥōśţ"],
+ "tsiTqO": ["Ƥàśţē ŷōũŕ ÀƤĨ ķēŷ"],
"u-9fWA": ["Śēē <0>Àŕćĥĩţēćţũŕē</0> ƒōŕ ţĥē śŷśţēḿ ōvēŕvĩēŵ."],
"u3BOHd": [["fileCount", "plural", { "one": ["#", " ƒĩĺē"], "other": ["#", " ƒĩĺēś"] }]],
"u3YnKw": ["Ćōũĺď ńōţ ćōńśţŕũćţ śĥàŕē ŨŔĹ."],
@@ -1642,6 +1690,7 @@
"vbK_ZG": ["Ƒĩĺēś śĩďēƀàŕ"],
"ve8vEM": ["Ćōďē ēxƥĩŕēď — ƥĺēàśē ţŕŷ àĝàĩń"],
"vfRlCI": ["Ćōũĺď ńōţ ćōƥŷ ƒũĺĺ ƥàţĥ"],
+ "vhAnzJ": ["Ţũŕń ōń śēḿàńţĩć śēàŕćĥ?"],
"viwC-K": ["Ƒàĩĺēď ţō ŕēḿōvē ƥŕōĴēćţ."],
"vv7kpg": ["Ćàńćēĺĺēď"],
"vvWZnu": ["Ďĩƒƒ ũńàvàĩĺàƀĺē"],
@@ -1674,6 +1723,7 @@
")"
],
"x57cJf": ["Ōƥēń ĺĩńķ"],
+ "xCJdfg": ["Ćĺēàŕ"],
"xCdTnb": ["Ćĺōśē ńēŵ ţàƀ"],
"xDBki3": ["Ĥēàďĩńĝ śũĝĝēśţĩōńś"],
"xG4tb6": ["Ĺōśţ ćōńńēćţĩōń ţō \"", ["docName"], "\"."],
@@ -1746,6 +1796,9 @@
"Śàvēś ţĥē ƒĩĺē ţō <0>~/Ďōŵńĺōàďś/ōƥēńķńōŵĺēďĝē.śķĩĺĺ</0> àńď ōƥēńś ţĥē Ćĺàũďē Ďēśķţōƥ Àƥƥ. Ŕēǫũĩŕēś Ńōďē.Ĵś ōŕ ßũń ōń ŷōũŕ ƤÀŢĤ."
],
"zAsDLY": ["Ĝŕàƥĥ vĩśũàĺĩźàţĩōń ōƒ ďōćũḿēńţ ĺĩńķś"],
+ "zEfRFN": [
+ "Ũśĩńĝ ţĥē <0>ŌĶ_ĒḾßĒĎĎĨŃĜŚ_ÀƤĨ_ĶĒŶ</0> ēńvĩŕōńḿēńţ vàŕĩàƀĺē (ḿàńàĝēď ōũţśĩďē Ōƥēń Ķńōŵĺēďĝē)."
+ ],
"zL0LPh": ["Ĝĺōƀàĺ ōũţśĩďē ţēxţ ƒĩēĺďś"],
"zLGXna": ["Ńō śţàŕţēŕ ƥàćķś àvàĩĺàƀĺē."],
"zLak8b": ["Àćţĩvàţē ńēŵ ţàƀ"],
diff --git a/packages/app/src/locales/pseudo/messages.po b/packages/app/src/locales/pseudo/messages.po
index 01a265037..2a7ed325f 100644
--- a/packages/app/src/locales/pseudo/messages.po
+++ b/packages/app/src/locales/pseudo/messages.po
@@ -395,6 +395,10 @@ msgstr ""
msgid "A folder named <0>{sanitized}</0> already has files at this location. Pick a different name."
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "A key is set, but the embeddings provider rejected it or was unreachable — search fell back to keyword matching. Check the key in <0>Settings → Account</0> and your provider settings."
+msgstr ""
+
#: src/editor/slash-command/items.tsx
msgid "A line with a footnote<0><1>[1]</1></0> and a definition shown below."
msgstr ""
@@ -475,6 +479,10 @@ msgstr ""
msgid "Add"
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Add a key"
+msgstr ""
+
#: src/components/FolderPropertiesCard.tsx
msgid "Add a property"
msgstr ""
@@ -491,6 +499,10 @@ msgstr ""
msgid "Add cursors or select matching occurrences through CodeMirror."
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Add meaning-based ranking to search so conceptually-related pages surface even when they share no keywords. This setting applies only to this computer."
+msgstr ""
+
#: src/components/McpConsentDialogBody.tsx
msgid "Add Open Knowledge to your AI tools"
msgstr ""
@@ -595,6 +607,10 @@ msgstr ""
msgid "Anyone can see"
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "API key set"
+msgstr ""
+
#: src/components/CreatedItemsList.tsx
msgid "Append-only timeline"
msgstr ""
@@ -603,6 +619,10 @@ msgstr ""
msgid "Apply link"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Applying your change"
+msgstr ""
+
#: src/components/FileTree.tsx
msgid "Are you sure you want to delete {folderName}/ and all files inside? This action cannot be undone."
msgstr ""
@@ -830,6 +850,7 @@ msgstr ""
#: src/components/NewTemplateDialog.tsx
#: src/components/PublishToGitHubDialog.tsx
#: src/components/SeedDialog.tsx
+#: src/components/settings/SearchSection.tsx
#: src/components/ShareBranchSwitchDialog.tsx
#: src/components/ShareReceiveDialog.tsx
#: src/components/TemplateEditDialog.tsx
@@ -889,6 +910,10 @@ msgstr ""
msgid "Claude Desktop"
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Clear"
+msgstr ""
+
#: src/components/PageHeaderWidgets.tsx
msgid "Clear {keyName}"
msgstr ""
@@ -1395,6 +1420,10 @@ msgstr ""
msgid "Couldn't auto-copy"
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Couldn't clear the key — please try again."
+msgstr ""
+
#: src/components/PublishToGitHubDialog.tsx
msgid "Couldn't construct the share URL. Try Done and re-share."
msgstr ""
@@ -1479,6 +1508,10 @@ msgstr ""
msgid "Couldn't save template: {error}"
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Couldn't save the key — please try again."
+msgstr ""
+
#: src/components/DiffViewBoundary.tsx
msgid "Couldn't save the resolution for {filePath}."
msgstr ""
@@ -1695,6 +1728,10 @@ msgstr ""
msgid "Disable git auto-sync"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Disable semantic search"
+msgstr ""
+
#: src/components/SyncStatusBadge.tsx
msgid "Disable sync"
msgstr ""
@@ -1916,6 +1953,14 @@ msgstr ""
msgid "Embed an image with optional alt text."
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Embeddings are computed only when a search runs and are cached locally under <0>.ok/local</0>."
+msgstr ""
+
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Embeddings provider key"
+msgstr ""
+
#: src/components/PropertyWidgets.tsx
msgid "Empty"
msgstr ""
@@ -1937,6 +1982,10 @@ msgstr ""
msgid "Enable git auto-sync?"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Enable semantic search"
+msgstr ""
+
#: src/components/SyncStatusBadge.tsx
msgid "Enable sync"
msgstr ""
@@ -2034,6 +2083,10 @@ msgstr ""
msgid "Failed to delete path"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Failed to disable semantic search — {detail}"
+msgstr ""
+
#: src/hooks/use-enable-sync-with-confirm.ts
msgid "Failed to disable sync — {detail}"
msgstr ""
@@ -2042,6 +2095,10 @@ msgstr ""
msgid "Failed to duplicate path"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Failed to enable semantic search — {detail}"
+msgstr ""
+
#: src/hooks/use-enable-sync-with-confirm.ts
msgid "Failed to enable sync — {detail}"
msgstr ""
@@ -2592,6 +2649,10 @@ msgstr ""
msgid "Indented blockquote for citations."
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Indexed {embedded} of {total} pages."
+msgstr ""
+
#: src/components/FrontmatterRow.tsx
msgid "inherited"
msgstr ""
@@ -3481,6 +3542,10 @@ msgstr ""
msgid "of {totalPages}"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Off — search ranks by keyword only. No content leaves this computer."
+msgstr ""
+
#: src/components/SyncStatusBadge.tsx
msgid "Offline"
msgstr ""
@@ -3508,6 +3573,10 @@ msgstr ""
msgid "ok ui responded but server.lock has no port yet"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "On — your search queries and the text of matching pages are sent to your embeddings provider (OpenAI by default) to compute embeddings."
+msgstr ""
+
#: src/components/ConsentDialogBody.tsx
msgid "One pattern per line — appended to <0>.okignore</0>."
msgstr ""
@@ -3912,6 +3981,10 @@ msgstr ""
msgid "page-name"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Pages are embedded the first time a search needs them."
+msgstr ""
+
#: src/lib/keyboard-shortcuts.ts
msgid "Paragraph"
msgstr ""
@@ -3940,6 +4013,10 @@ msgstr ""
msgid "Paste was incomplete — {chunksCompleted} of {totalChunks} chunks landed.{restoreSuffix}"
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Paste your API key"
+msgstr ""
+
#: src/components/NewItemDialog.tsx
msgid "Path cannot contain \"..\""
msgstr ""
@@ -4334,6 +4411,10 @@ msgstr ""
msgid "Replace current match from field"
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Replace key"
+msgstr ""
+
#: src/lib/keyboard-shortcuts.ts
msgid "Replace the active match from the replace field."
msgstr ""
@@ -4490,6 +4571,7 @@ msgid "Sandboxed HTML embed — write HTML, see the rendered preview live."
msgstr ""
#: src/components/AuthModal.tsx
+#: src/components/settings/EmbeddingsKeySection.tsx
#: src/components/TemplateEditDialog.tsx
#: src/editor/components/CodePreviewEditModal.tsx
#: src/editor/extensions/InternalLinkPropPanel.tsx
@@ -4530,6 +4612,7 @@ msgstr ""
msgid "Saves the file to <0>~/Downloads/openknowledge.skill</0> and opens the Claude Desktop App. Requires Node.js or Bun on your PATH."
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
#: src/components/TemplateEditDialog.tsx
#: src/components/TimelinePanel.tsx
msgid "Saving"
@@ -4545,6 +4628,7 @@ msgstr ""
#: src/components/CommandPalette.tsx
#: src/components/empty-state/KeyboardHintsFooter.tsx
+#: src/components/settings/SettingsDialogShell.tsx
#: src/components/SidebarSearchBar.tsx
#: src/lib/keyboard-shortcuts.ts
msgid "Search"
@@ -4582,6 +4666,10 @@ msgstr ""
msgid "Search recent projects..."
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Search settings not yet loaded — try again in a moment"
+msgstr ""
+
#: src/components/NewItemDialog.tsx
msgid "Search templates"
msgstr ""
@@ -4655,6 +4743,22 @@ msgstr ""
msgid "Selection statistics"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Semantic ranking activates the first time an agent runs a search."
+msgstr ""
+
+#: src/components/settings/SearchSection.tsx
+msgid "Semantic search"
+msgstr ""
+
+#: src/components/settings/SearchSection.tsx
+msgid "Semantic search adds meaning-based ranking to the search tool so conceptually-related pages surface even without shared keywords."
+msgstr ""
+
+#: src/components/settings/SearchSection.tsx
+msgid "Semantic search is on, but no API key is set — search falls back to keyword matching. Add one in <0>Settings → Account</0> (it's stored once for your whole machine)."
+msgstr ""
+
#: src/editor/slash-command/items.tsx
msgid "Separator"
msgstr ""
@@ -4972,6 +5076,10 @@ msgstr ""
msgid "Stored at <0>.ok/templates/</0> in this project. Available in every folder (folder-scoped templates can override by filename)."
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Stored on this machine."
+msgstr ""
+
#: src/lib/keyboard-shortcuts.ts
msgid "Strikethrough"
msgstr ""
@@ -5180,6 +5288,10 @@ msgstr ""
msgid "The {targetNoun} could not be opened — try navigating to it manually."
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "The API key semantic search uses to embed your content. Stored once for this machine and shared across all projects. Turn the feature on per project in This project → Search."
+msgstr ""
+
#: src/components/InstallInClaudeDesktopDialog.tsx
msgid "The Claude Desktop App didn't open the file. Is it installed? ({msg})"
msgstr ""
@@ -5308,6 +5420,14 @@ msgstr ""
msgid "This project's running server doesn't support live editing. Restart Open Knowledge to fix."
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "This sends content off your machine"
+msgstr ""
+
+#: src/components/settings/SearchSection.tsx
+msgid "This setting is per-machine and isn't shared with collaborators. It needs an API key set with <0>ok embeddings set-key</0>."
+msgstr ""
+
#: src/components/TemplateDeleteDialog.tsx
msgid "This template lives in a parent folder — deleting it affects every folder beneath it that doesn't define its own version."
msgstr ""
@@ -5411,6 +5531,14 @@ msgstr ""
msgid "Try again"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Turn on"
+msgstr ""
+
+#: src/components/settings/SearchSection.tsx
+msgid "Turn on semantic search?"
+msgstr ""
+
#: src/editor/components/Pdf.tsx
msgid "Two-page (even)"
msgstr ""
@@ -5580,6 +5708,10 @@ msgstr ""
msgid "User"
msgstr ""
+#: src/components/settings/EmbeddingsKeySection.tsx
+msgid "Using the <0>OK_EMBEDDINGS_API_KEY</0> environment variable (managed outside Open Knowledge)."
+msgstr ""
+
#: src/components/FolderPropertiesCard.tsx
msgid "Value is required"
msgstr ""
@@ -5798,6 +5930,10 @@ msgstr ""
msgid "Your repositories"
msgstr ""
+#: src/components/settings/SearchSection.tsx
+msgid "Your search queries and the full text of matching pages are sent to your embeddings provider (OpenAI by default) to compute embeddings."
+msgstr ""
+
#: src/components/PublishToGitHubDialog.tsx
msgid "Your share link is ready below."
msgstr ""
diff --git a/packages/app/tests/integration/attribution-sweep-coverage.test.ts b/packages/app/tests/integration/attribution-sweep-coverage.test.ts
index f1098b573..36773bb19 100644
--- a/packages/app/tests/integration/attribution-sweep-coverage.test.ts
+++ b/packages/app/tests/integration/attribution-sweep-coverage.test.ts
@@ -47,6 +47,7 @@ const EXEMPT_HANDLERS = new Set([
'handleForwardLinks',
'handleLinkGraph',
'handleSearch',
+ 'handleSemanticStatus',
'handleDeadLinks',
'handleOrphans',
'handleHubs',
@@ -84,6 +85,8 @@ const EXEMPT_HANDLERS = new Set([
'handleLocalOpAuthPat',
'handleLocalOpAuthIdentity',
'handleLocalOpAuthSetIdentity',
+ 'handleLocalOpEmbeddingsSetKey',
+ 'handleLocalOpEmbeddingsClearKey',
'handleTestReset',
'handlePrincipal',
'handleInstalledAgentsRoute',
diff --git a/packages/app/tests/integration/conflict-gate-coverage.test.ts b/packages/app/tests/integration/conflict-gate-coverage.test.ts
index ca2677fa6..b4d0f057b 100644
--- a/packages/app/tests/integration/conflict-gate-coverage.test.ts
+++ b/packages/app/tests/integration/conflict-gate-coverage.test.ts
@@ -30,6 +30,7 @@ const EXEMPT_HANDLERS = new Set([
'handleForwardLinks',
'handleLinkGraph',
'handleSearch',
+ 'handleSemanticStatus',
'handleDeadLinks',
'handleOrphans',
'handleHubs',
@@ -74,6 +75,8 @@ const EXEMPT_HANDLERS = new Set([
'handleLocalOpAuthPat',
'handleLocalOpAuthIdentity',
'handleLocalOpAuthSetIdentity',
+ 'handleLocalOpEmbeddingsSetKey',
+ 'handleLocalOpEmbeddingsClearKey',
'handleSpawnCursorRoute',
'handleHandoffDispatchRoute',
'handleInstallSkill',
diff --git a/packages/cli/src/auth/embeddings-key-store.ts b/packages/cli/src/auth/embeddings-key-store.ts
new file mode 100644
index 000000000..1f5f631e6
--- /dev/null
+++ b/packages/cli/src/auth/embeddings-key-store.ts
@@ -0,0 +1,6 @@
+export {
+ clearEmbeddingsKeyFromAllBackends,
+ createEmbeddingsSecretStore,
+ describeStoredEmbeddingsKey,
+ makeLazyEmbeddingsKeyStore,
+} from '@inkeep/open-knowledge-server';
diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts
index 320823233..24ff5871b 100644
--- a/packages/cli/src/cli.ts
+++ b/packages/cli/src/cli.ts
@@ -20,6 +20,7 @@ import { cloneCommand } from './commands/clone.ts';
import { configCommand } from './commands/config.ts';
import { createRealDetectDeps, detectDesktop, launchDesktop } from './commands/desktop-dispatch.ts';
import { diagnoseCommand } from './commands/diagnose.ts';
+import { embeddingsCommand } from './commands/embeddings/index.ts';
import { initCommand } from './commands/init.ts';
import { installSkillCommand } from './commands/install-skill.ts';
import { mcpCommand } from './commands/mcp.ts';
@@ -133,6 +134,8 @@ program.addCommand(configCommand());
program.addCommand(authCommand());
+program.addCommand(embeddingsCommand());
+
program.addCommand(cloneCommand(() => resolvedConfig));
program.addCommand(syncCommand(() => resolvedConfig));
diff --git a/packages/cli/src/commands/embeddings/index.ts b/packages/cli/src/commands/embeddings/index.ts
new file mode 100644
index 000000000..68c0ea80a
--- /dev/null
+++ b/packages/cli/src/commands/embeddings/index.ts
@@ -0,0 +1,199 @@
+import { resolve } from 'node:path';
+import { humanFormat } from '@inkeep/open-knowledge-core';
+import { writeConfigPatch } from '@inkeep/open-knowledge-core/server';
+import {
+ DEFAULT_EMBEDDINGS_DIMENSIONS,
+ EMBEDDINGS_API_KEY_ENV,
+ isProcessAlive,
+ readProjectLocalSemanticConfig,
+ readServerLock,
+ resolveLockDir,
+} from '@inkeep/open-knowledge-server';
+import password from '@inquirer/password';
+import { Command } from 'commander';
+import {
+ clearEmbeddingsKeyFromAllBackends,
+ createEmbeddingsSecretStore,
+ describeStoredEmbeddingsKey,
+} from '../../auth/embeddings-key-store.ts';
+
+async function readKey(): Promise<string> {
+ if (!process.stdin.isTTY) {
+ const chunks: Buffer[] = [];
+ for await (const chunk of process.stdin) chunks.push(chunk as Buffer);
+ return Buffer.concat(chunks).toString('utf-8').trim();
+ }
+ return (await password({ message: 'Enter embeddings API key:' })).trim();
+}
+
+function readSemanticConfig(projectDir: string) {
+ return readProjectLocalSemanticConfig(projectDir);
+}
+
+async function resolveKeyPresence(): Promise<{ present: boolean; source: 'file' | 'env' | null }> {
+ const stored = await describeStoredEmbeddingsKey();
+ if (stored.file) return { present: true, source: 'file' };
+ if (process.env[EMBEDDINGS_API_KEY_ENV]) return { present: true, source: 'env' };
+ return { present: false, source: null };
+}
+
+async function fetchLiveCoverage(
+ projectDir: string,
+): Promise<{ embedded: number; total: number } | null> {
+ try {
+ const lock = readServerLock(resolveLockDir(projectDir));
+ if (!lock || lock.port <= 0 || !isProcessAlive(lock.pid)) return null;
+ const res = await fetch(`http://127.0.0.1:${lock.port}/api/semantic-status`, {
+ signal: AbortSignal.timeout(1500),
+ });
+ if (!res.ok) return null;
+ const body = (await res.json()) as { embedded?: unknown; total?: unknown };
+ if (typeof body.embedded !== 'number' || typeof body.total !== 'number') return null;
+ return { embedded: body.embedded, total: body.total };
+ } catch {
+ return null;
+ }
+}
+
+function setKeyCommand(): Command {
+ return new Command('set-key')
+ .description('Store the embeddings provider API key in ~/.ok/secrets.yml')
+ .action(async () => {
+ const key = await readKey();
+ if (!key) {
+ process.stderr.write('No key provided.\n');
+ process.exitCode = 1;
+ return;
+ }
+ await createEmbeddingsSecretStore().set(key);
+ process.stderr.write(
+ '✓ Embeddings API key stored in ~/.ok/secrets.yml (0600, this machine only).\n' +
+ 'Now enable it per project — the easiest path is OK Desktop → Settings → This\n' +
+ 'project → Search (a toggle with an egress-confirmation prompt), or run\n' +
+ '`ok embeddings enable` in the project folder.\n',
+ );
+ });
+}
+
+function clearKeyCommand(): Command {
+ return new Command('clear-key')
+ .description('Remove the stored embeddings provider API key')
+ .action(async () => {
+ const { touched } = await clearEmbeddingsKeyFromAllBackends();
+ if (touched.length === 0) {
+ process.stderr.write('No stored embeddings key found.\n');
+ return;
+ }
+ process.stderr.write(`✓ Embeddings API key cleared (${touched.join(', ')}).\n`);
+ });
+}
+
+function toggleEnabledCommand(name: 'enable' | 'disable', value: boolean): Command {
+ return new Command(name)
+ .description(`Turn semantic search ${value ? 'on' : 'off'} for this project (project-local)`)
+ .option('--cwd <path>', 'Project directory (defaults to the current directory)')
+ .action(async (opts: { cwd?: string }) => {
+ const projectDir = resolve(opts.cwd ?? process.cwd());
+ const result = await writeConfigPatch({
+ cwd: projectDir,
+ scope: 'project-local',
+ patch: { search: { semantic: { enabled: value } } },
+ });
+ if (!result.ok) {
+ process.stderr.write(`Failed to ${name} semantic search — ${humanFormat(result.error)}\n`);
+ process.exitCode = 1;
+ return;
+ }
+ process.stderr.write(
+ `✓ Semantic search ${value ? 'enabled' : 'disabled'} for ${projectDir}\n`,
+ );
+ if (value) {
+ const { present } = await resolveKeyPresence();
+ if (!present) {
+ process.stderr.write(
+ ' Note: no API key set yet — run `ok embeddings set-key`. Until then, search stays lexical.\n',
+ );
+ }
+ }
+ });
+}
+
+function statusCommand(): Command {
+ return new Command('status')
+ .description('Show semantic-search capability: key presence, enabled, coverage, provider')
+ .option('--cwd <path>', 'Project directory (defaults to the current directory)')
+ .option('--json', 'Output JSON', false)
+ .action(async (opts: { cwd?: string; json?: boolean }) => {
+ const projectDir = resolve(opts.cwd ?? process.cwd());
+ const cfg = readSemanticConfig(projectDir);
+ const { present: hasKey, source: keySource } = await resolveKeyPresence();
+ const capable = cfg.enabled && hasKey;
+ const coverage = capable ? await fetchLiveCoverage(projectDir) : null;
+
+ if (opts.json) {
+ process.stdout.write(
+ `${JSON.stringify({
+ project: projectDir,
+ machine: { keyPresent: hasKey, keySource },
+ project_config: {
+ enabled: cfg.enabled,
+ capable,
+ coverage: coverage ? { embedded: coverage.embedded, total: coverage.total } : null,
+ provider: {
+ baseUrl: cfg.baseUrl,
+ model: cfg.model,
+ dimensions: cfg.dimensions ?? null,
+ },
+ },
+ })}\n`,
+ );
+ return;
+ }
+
+ const keyLabel = hasKey
+ ? `set — ${keySource === 'env' ? `environment (${EMBEDDINGS_API_KEY_ENV})` : '~/.ok/secrets.yml'}`
+ : 'not set';
+ const coverageLabel = !capable
+ ? null
+ : coverage
+ ? `${coverage.embedded} / ${coverage.total} pages embedded`
+ : 'server not running — start it to index (or it has not embedded yet)';
+
+ const lines = [
+ 'Semantic search',
+ ` project: ${projectDir}`,
+ '',
+ ' This machine (all projects):',
+ ` API key: ${keyLabel}`,
+ '',
+ ' This project:',
+ ` enabled: ${cfg.enabled ? 'yes' : 'no'}`,
+ ` capability: ${capable ? 'AVAILABLE' : 'unavailable (search stays lexical)'}`,
+ ...(coverageLabel ? [` coverage: ${coverageLabel}`] : []),
+ ` provider: ${cfg.baseUrl}`,
+ ` model: ${cfg.model}`,
+ ` dimensions: ${cfg.dimensions ?? `native (${DEFAULT_EMBEDDINGS_DIMENSIONS})`}`,
+ ];
+
+ const hints: string[] = [];
+ if (!hasKey) {
+ hints.push(`Set a key: ok embeddings set-key (or export ${EMBEDDINGS_API_KEY_ENV})`);
+ }
+ if (!cfg.enabled) {
+ hints.push('Enable it: ok embeddings enable (in this project folder)');
+ }
+ if (hints.length > 0) lines.push('', ...hints.map((h) => ` ${h}`));
+
+ process.stdout.write(`${lines.join('\n')}\n`);
+ });
+}
+
+export function embeddingsCommand(): Command {
+ return new Command('embeddings')
+ .description('Manage the semantic-search embeddings provider key + status')
+ .addCommand(setKeyCommand())
+ .addCommand(clearKeyCommand())
+ .addCommand(toggleEnabledCommand('enable', true))
+ .addCommand(toggleEnabledCommand('disable', false))
+ .addCommand(statusCommand());
+}
diff --git a/packages/cli/src/commands/start.ts b/packages/cli/src/commands/start.ts
index 610679b8e..fa0804ff0 100644
--- a/packages/cli/src/commands/start.ts
+++ b/packages/cli/src/commands/start.ts
@@ -21,6 +21,7 @@ import {
prepareSingleFileOpen,
} from '@inkeep/open-knowledge-server';
import { Command, InvalidArgumentError } from 'commander';
+import { makeLazyEmbeddingsKeyStore } from '../auth/embeddings-key-store.ts';
import { detectGh } from '../auth/gh-detect.ts';
import { makeLazyProbeTokenStore } from '../auth/token-store.ts';
import { OK_DIR, PACKAGE_VERSION } from '../constants.ts';
@@ -408,6 +409,7 @@ export async function bootStartServer(opts: BootStartServerOptions): Promise<Boo
const attachUiSibling = opts.reactShellDistDir === undefined;
const tokenStore = makeLazyProbeTokenStore();
+ const embeddingsKeyStore = makeLazyEmbeddingsKeyStore();
const booted: BootedServer = await bootServer({
config,
@@ -419,6 +421,7 @@ export async function bootStartServer(opts: BootStartServerOptions): Promise<Boo
quiet: false,
detectGh,
tokenStore,
+ embeddingsKeyStore,
...(ephemeral
? {
ephemeral: true as const,
diff --git a/packages/core/src/config/field-registry.test.ts b/packages/core/src/config/field-registry.test.ts
index 761476018..f0af373ee 100644
--- a/packages/core/src/config/field-registry.test.ts
+++ b/packages/core/src/config/field-registry.test.ts
@@ -152,7 +152,7 @@ describe('ConfigSchema coverage (NR3 — every leaf has fieldRegistry metadata)'
]);
});
- test('project-local-strict fields cover autoSync.enabled + appearance.sidebar.*', () => {
+ test('project-local-strict fields cover autoSync.enabled + appearance.sidebar.* + search.semantic.*', () => {
const leaves: { path: string[]; schema: unknown }[] = [];
walkLeaves(ConfigSchema, [], leaves);
const projectLocalStrict = leaves
@@ -163,6 +163,10 @@ describe('ConfigSchema coverage (NR3 — every leaf has fieldRegistry metadata)'
'appearance.sidebar.showAllFiles',
'appearance.sidebar.showHiddenFiles',
'autoSync.enabled',
+ 'search.semantic.baseUrl',
+ 'search.semantic.dimensions',
+ 'search.semantic.enabled',
+ 'search.semantic.model',
]);
});
});
diff --git a/packages/core/src/config/schema.ts b/packages/core/src/config/schema.ts
index c7b1cd62b..e39ed04ac 100644
--- a/packages/core/src/config/schema.ts
+++ b/packages/core/src/config/schema.ts
@@ -15,6 +15,9 @@ export const DEFAULT_TELEMETRY_ATTRIBUTE_DENYLIST: readonly string[] = Object.fr
export const DEFAULT_SPANS_MAX_BYTES = 52_428_800;
export const DEFAULT_LOGS_MAX_BYTES = 26_214_400;
+export const DEFAULT_EMBEDDINGS_BASE_URL = 'https://api.openai.com/v1';
+export const DEFAULT_EMBEDDINGS_MODEL = 'text-embedding-3-small';
+
export const ConfigSchema = z.looseObject({
content: z
.looseObject({
@@ -181,6 +184,66 @@ export const ConfigSchema = z.looseObject({
attributeDenylist: [...DEFAULT_TELEMETRY_ATTRIBUTE_DENYLIST],
},
}),
+ search: z
+ .looseObject({
+ semantic: z
+ .looseObject({
+ enabled: z
+ .boolean()
+ .register(fieldRegistry, {
+ scope: 'project-local',
+ agentSettable: false,
+ defaultScope: 'project-local',
+ description:
+ 'Add semantic (embeddings) ranking to the MCP search tool, fused with the lexical engine so conceptually-related pages surface even with no shared keywords. When ON and an API key is set (`ok embeddings set-key`), the search query and matching document content are sent to the configured embeddings provider — content egress. Default OFF. Per-machine (project-local) — not shared with collaborators.',
+ })
+ .default(false),
+ baseUrl: z
+ .string()
+ .register(fieldRegistry, {
+ scope: 'project-local',
+ agentSettable: false,
+ defaultScope: 'project-local',
+ description:
+ 'Base URL of the OpenAI-compatible embeddings API (default https://api.openai.com/v1). Override for Azure / self-hosted / other providers. The API key is NOT stored here — set it with `ok embeddings set-key` (OS keyring).',
+ })
+ .default(DEFAULT_EMBEDDINGS_BASE_URL),
+ model: z
+ .string()
+ .register(fieldRegistry, {
+ scope: 'project-local',
+ agentSettable: false,
+ defaultScope: 'project-local',
+ description:
+ 'Embeddings model id (default text-embedding-3-small). Must be served by the provider at baseUrl. Changing it re-embeds the corpus (the cache is keyed by provider + model + dimensions).',
+ })
+ .default(DEFAULT_EMBEDDINGS_MODEL),
+ dimensions: z
+ .number()
+ .int()
+ .positive()
+ .register(fieldRegistry, {
+ scope: 'project-local',
+ agentSettable: false,
+ defaultScope: 'project-local',
+ description:
+ "Optional output vector dimensions. Omit to use the model's native size (1536 for text-embedding-3-small). Set a smaller value (text-embedding-3 supports e.g. 512 / 1024) to shrink the on-disk cache, trading a little retrieval quality. Changing it re-embeds the corpus.",
+ })
+ .optional(),
+ })
+ .default({
+ enabled: false,
+ baseUrl: DEFAULT_EMBEDDINGS_BASE_URL,
+ model: DEFAULT_EMBEDDINGS_MODEL,
+ }),
+ })
+ .default({
+ semantic: {
+ enabled: false,
+ baseUrl: DEFAULT_EMBEDDINGS_BASE_URL,
+ model: DEFAULT_EMBEDDINGS_MODEL,
+ },
+ }),
});
export type Config = z.infer<typeof ConfigSchema>;
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index ad08b485a..dd342f56f 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -119,6 +119,8 @@ export {
type Config,
type ConfigPatch,
ConfigSchema,
+ DEFAULT_EMBEDDINGS_BASE_URL,
+ DEFAULT_EMBEDDINGS_MODEL,
DEFAULT_LOGS_MAX_BYTES,
DEFAULT_SPANS_MAX_BYTES,
DEFAULT_TELEMETRY_ATTRIBUTE_DENYLIST,
@@ -594,6 +596,10 @@ export {
LocalOpAuthStatusSuccessSchema,
type LocalOpCloneRequest,
LocalOpCloneRequestSchema,
+ type LocalOpEmbeddingsMutationSuccess,
+ LocalOpEmbeddingsMutationSuccessSchema,
+ type LocalOpEmbeddingsSetKeyRequest,
+ LocalOpEmbeddingsSetKeyRequestSchema,
type LocalOpOkInitFailureReason,
LocalOpOkInitFailureReasonSchema,
type LocalOpOkInitRequest,
@@ -660,6 +666,8 @@ export {
SearchRequestSchema,
type SearchResultEntry,
SearchResultEntrySchema,
+ type SearchSemanticStatus,
+ SearchSemanticStatusSchema,
type SearchSuccess,
SearchSuccessSchema,
type SeedApplyRequest,
@@ -676,6 +684,8 @@ export {
SeedPackInfoSchema,
type SeedPlanSuccess,
SeedPlanSuccessSchema,
+ type SemanticIndexStatus,
+ SemanticIndexStatusSchema,
type ServerInfoSuccess,
ServerInfoSuccessSchema,
type ShareConstructUrlErrorCode,
@@ -815,6 +825,9 @@ export {
export {
createWorkspaceSearchCorpus,
createWorkspaceSearchDocument,
+ DEFAULT_RRF_K,
+ DEFAULT_VECTOR_CANDIDATE_LIMIT,
+ DEFAULT_VECTOR_SIMILARITY_FLOOR,
DEFAULT_WORKSPACE_SEARCH_LIMIT,
MAX_WORKSPACE_SEARCH_LIMIT,
searchWorkspaceCorpus,
@@ -826,6 +839,7 @@ export {
type WorkspaceSearchOptions,
type WorkspaceSearchResult,
type WorkspaceSearchScope,
+ type WorkspaceSemanticInput,
workspaceSearchBasename,
workspaceSearchPathSegments,
} from './search/workspace-search.ts';
@@ -880,7 +894,7 @@ export type {
ShadowContributor,
TimelineEntry,
} from './types/timeline.ts';
-export { isValidDocName, validateDocName } from './util/doc-name.ts';
+export { isHiddenDocName, isValidDocName, validateDocName } from './util/doc-name.ts';
export { applyByPrefixSuffix } from './utils/apply-by-prefix-suffix.ts';
export { toDesktopAssetHref } from './utils/asset-href.ts';
export { ChunkedInsertError, chunkedYTextInsert } from './utils/chunked-insert.ts';
diff --git a/packages/core/src/schemas/api/local-op.ts b/packages/core/src/schemas/api/local-op.ts
index 5f4d3e37d..d2abc8761 100644
--- a/packages/core/src/schemas/api/local-op.ts
+++ b/packages/core/src/schemas/api/local-op.ts
@@ -61,6 +61,22 @@ export const LocalOpAuthPatRequestSchema = z
.loose() satisfies StandardSchemaV1;
export type LocalOpAuthPatRequest = z.infer<typeof LocalOpAuthPatRequestSchema>;
+export const LocalOpEmbeddingsSetKeyRequestSchema = z
+ .object({
+ key: z.string().min(1),
+ })
+ .loose() satisfies StandardSchemaV1;
+export type LocalOpEmbeddingsSetKeyRequest = z.infer<typeof LocalOpEmbeddingsSetKeyRequestSchema>;
+
+export const LocalOpEmbeddingsMutationSuccessSchema = z
+ .object({
+ keyPresent: z.boolean(),
+ })
+ .loose() satisfies StandardSchemaV1;
+export type LocalOpEmbeddingsMutationSuccess = z.infer<
+ typeof LocalOpEmbeddingsMutationSuccessSchema
+>;
+
export const LocalOpAuthSetIdentityRequestSchema = z
.object({
name: z.string().refine((s) => s.trim().length > 0, { message: 'name must be non-empty' }),
diff --git a/packages/core/src/schemas/api/tags-search.ts b/packages/core/src/schemas/api/tags-search.ts
index 70cad26ea..e9a7b208a 100644
--- a/packages/core/src/schemas/api/tags-search.ts
+++ b/packages/core/src/schemas/api/tags-search.ts
@@ -161,6 +161,7 @@ export const SearchRequestSchema = z
scopes: z.array(z.enum(['page', 'folder', 'content'])).optional(),
scope: z.string().optional(),
limit: z.number().int().nonnegative().optional(),
+ semantic: z.boolean().optional(),
})
.loose() satisfies StandardSchemaV1;
export type SearchRequest = z.infer<typeof SearchRequestSchema>;
@@ -177,12 +178,38 @@ export const SearchResultEntrySchema = z
.loose() satisfies StandardSchemaV1;
export type SearchResultEntry = z.infer<typeof SearchResultEntrySchema>;
+export const SearchSemanticStatusSchema = z
+ .object({
+ capable: z.boolean(),
+ applied: z.boolean(),
+ coverage: z.object({
+ embedded: z.number().int().nonnegative(),
+ total: z.number().int().nonnegative(),
+ }),
+ })
+ .loose() satisfies StandardSchemaV1;
+export type SearchSemanticStatus = z.infer<typeof SearchSemanticStatusSchema>;
+
+export const SemanticIndexStatusSchema = z
+ .object({
+ enabled: z.boolean(),
+ keyPresent: z.boolean(),
+ keySource: z.enum(['file', 'env']).nullable(),
+ ready: z.boolean(),
+ capable: z.boolean(),
+ embedded: z.number().int().nonnegative(),
+ total: z.number().int().nonnegative(),
+ })
+ .loose() satisfies StandardSchemaV1;
+export type SemanticIndexStatus = z.infer<typeof SemanticIndexStatusSchema>;
+
export const SearchSuccessSchema = z
.object({
query: z.string(),
intent: z.enum(['autocomplete', 'full_text', 'omnibar']),
results: z.array(SearchResultEntrySchema),
elapsedMs: z.number().nonnegative(),
+ semantic: SearchSemanticStatusSchema.optional(),
})
.loose() satisfies StandardSchemaV1;
export type SearchSuccess = z.infer<typeof SearchSuccessSchema>;
diff --git a/packages/core/src/search/workspace-search.baseline.fixture.json b/packages/core/src/search/workspace-search.baseline.fixture.json
new file mode 100644
index 000000000..a3f38e098
--- /dev/null
+++ b/packages/core/src/search/workspace-search.baseline.fixture.json
@@ -0,0 +1,632 @@
+{
+ "documents": [
+ {
+ "id": "page:auth/session-token-refresh",
+ "kind": "page",
+ "path": "auth/session-token-refresh",
+ "title": "Session Token Refresh",
+ "name": "session-token-refresh",
+ "pathSegments": "auth session-token-refresh",
+ "content": "How the session token refresh flow re-issues credentials when they expire. Retry with backoff.",
+ "modifiedTs": 5000
+ },
+ {
+ "id": "page:auth/login",
+ "kind": "page",
+ "path": "auth/login",
+ "title": "Login",
+ "name": "login",
+ "pathSegments": "auth login",
+ "content": "The login page authenticates a user and starts a session.",
+ "modifiedTs": 3000
+ },
+ {
+ "id": "page:guides/authentication-overview",
+ "kind": "page",
+ "path": "guides/authentication-overview",
+ "title": "Authentication Overview",
+ "name": "authentication-overview",
+ "pathSegments": "guides authentication-overview",
+ "content": "An overview of authentication, authorization, tokens, and sessions across the platform.",
+ "modifiedTs": 9000
+ },
+ {
+ "id": "page:recipes/sourdough",
+ "kind": "page",
+ "path": "recipes/sourdough",
+ "title": "Sourdough Bread",
+ "name": "sourdough",
+ "pathSegments": "recipes sourdough",
+ "content": "A recipe for sourdough bread with a long cold ferment and a crackly crust.",
+ "modifiedTs": 1000
+ },
+ {
+ "id": "page:recipes/focaccia",
+ "kind": "page",
+ "path": "recipes/focaccia",
+ "title": "Focaccia",
+ "name": "focaccia",
+ "pathSegments": "recipes focaccia",
+ "content": "Olive oil focaccia with rosemary; uses the same starter as the sourdough.",
+ "modifiedTs": 2000
+ },
+ {
+ "id": "page:notes/retry-strategies",
+ "kind": "page",
+ "path": "notes/retry-strategies",
+ "title": "Retry Strategies",
+ "name": "retry-strategies",
+ "pathSegments": "notes retry-strategies",
+ "content": "Exponential backoff, jitter, and idempotency keys for retrying failed requests.",
+ "modifiedTs": 8000
+ },
+ {
+ "id": "page:auth",
+ "kind": "page",
+ "path": "auth",
+ "title": "auth",
+ "name": "auth",
+ "pathSegments": "auth",
+ "content": "Index of authentication topics.",
+ "modifiedTs": 4000
+ },
+ {
+ "id": "page:misc/login-history",
+ "kind": "page",
+ "path": "misc/login-history",
+ "title": "Login History",
+ "name": "login-history",
+ "pathSegments": "misc login-history",
+ "content": "A record of past login events and session activity for the account.",
+ "modifiedTs": 6000
+ },
+ {
+ "id": "folder:auth",
+ "kind": "folder",
+ "path": "auth",
+ "title": "auth",
+ "name": "auth",
+ "pathSegments": "auth",
+ "content": "",
+ "modifiedTs": 7000
+ },
+ {
+ "id": "folder:recipes",
+ "kind": "folder",
+ "path": "recipes",
+ "title": "recipes",
+ "name": "recipes",
+ "pathSegments": "recipes",
+ "content": "",
+ "modifiedTs": 1500
+ }
+ ],
+ "cases": [
+ {
+ "query": "",
+ "options": {},
+ "expected": [
+ {
+ "id": "page:auth",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "folder:auth",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "page:auth/login",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "page:auth/session-token-refresh",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "page:guides/authentication-overview",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "page:misc/login-history",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "page:notes/retry-strategies",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "folder:recipes",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "page:recipes/focaccia",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ },
+ {
+ "id": "page:recipes/sourdough",
+ "score": 0,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0,
+ "recency": 0
+ }
+ }
+ ]
+ },
+ {
+ "query": "login",
+ "options": {
+ "intent": "omnibar"
+ },
+ "expected": [
+ {
+ "id": "page:auth/login",
+ "score": 1707.1890049060949,
+ "signals": {
+ "lexical": 700,
+ "fullText": 49.734450245304735,
+ "recency": 12.5
+ }
+ },
+ {
+ "id": "page:misc/login-history",
+ "score": 1479.3204629742795,
+ "signals": {
+ "lexical": 600,
+ "fullText": 42.40352314871397,
+ "recency": 31.25
+ }
+ }
+ ]
+ },
+ {
+ "query": "login",
+ "options": {
+ "intent": "autocomplete"
+ },
+ "expected": [
+ {
+ "id": "page:auth/login",
+ "score": 1890.912389055217,
+ "signals": {
+ "lexical": 700,
+ "fullText": 58.92061945276085,
+ "recency": 12.5
+ }
+ },
+ {
+ "id": "page:misc/login-history",
+ "score": 1626.389211640448,
+ "signals": {
+ "lexical": 600,
+ "fullText": 49.756960582022394,
+ "recency": 31.25
+ }
+ }
+ ]
+ },
+ {
+ "query": "session token refresh",
+ "options": {
+ "intent": "full_text"
+ },
+ "expected": [
+ {
+ "id": "page:auth/session-token-refresh",
+ "score": 1919.9498027912073,
+ "signals": {
+ "lexical": 700,
+ "fullText": 59.74749013956037,
+ "recency": 25
+ }
+ },
+ {
+ "id": "page:guides/authentication-overview",
+ "score": 85.22683776896082,
+ "signals": {
+ "lexical": 0,
+ "fullText": 1.7613418884480405,
+ "recency": 50
+ }
+ },
+ {
+ "id": "page:misc/login-history",
+ "score": 40.54046425448354,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0.46452321272417674,
+ "recency": 31.25
+ }
+ },
+ {
+ "id": "page:auth/login",
+ "score": 24.792405644684873,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0.6146202822342437,
+ "recency": 12.5
+ }
+ }
+ ]
+ },
+ {
+ "query": "auth",
+ "options": {
+ "intent": "omnibar"
+ },
+ "expected": [
+ {
+ "id": "folder:auth",
+ "score": 1573.1702123532718,
+ "signals": {
+ "lexical": 700,
+ "fullText": 41.783510617663595,
+ "recency": 37.5
+ }
+ },
+ {
+ "id": "page:guides/authentication-overview",
+ "score": 1566.0036368361461,
+ "signals": {
+ "lexical": 600,
+ "fullText": 45.80018184180731,
+ "recency": 50
+ }
+ },
+ {
+ "id": "page:auth",
+ "score": 1554.4202123532718,
+ "signals": {
+ "lexical": 700,
+ "fullText": 41.783510617663595,
+ "recency": 18.75
+ }
+ },
+ {
+ "id": "page:auth/session-token-refresh",
+ "score": 713.485200031778,
+ "signals": {
+ "lexical": 550,
+ "fullText": 6.9242600015889,
+ "recency": 25
+ }
+ },
+ {
+ "id": "page:auth/login",
+ "score": 700.985200031778,
+ "signals": {
+ "lexical": 550,
+ "fullText": 6.9242600015889,
+ "recency": 12.5
+ }
+ }
+ ]
+ },
+ {
+ "query": "auth",
+ "options": {
+ "intent": "full_text"
+ },
+ "expected": [
+ {
+ "id": "page:guides/authentication-overview",
+ "score": 1596.714689887646,
+ "signals": {
+ "lexical": 600,
+ "fullText": 47.3357344943823,
+ "recency": 50
+ }
+ },
+ {
+ "id": "page:auth",
+ "score": 1586.048341132585,
+ "signals": {
+ "lexical": 700,
+ "fullText": 43.364917056629245,
+ "recency": 18.75
+ }
+ },
+ {
+ "id": "page:auth/login",
+ "score": 722.372911924148,
+ "signals": {
+ "lexical": 550,
+ "fullText": 7.993645596207396,
+ "recency": 12.5
+ }
+ },
+ {
+ "id": "page:auth/session-token-refresh",
+ "score": 713.485200031778,
+ "signals": {
+ "lexical": 550,
+ "fullText": 6.9242600015889,
+ "recency": 25
+ }
+ }
+ ]
+ },
+ {
+ "query": "authentication",
+ "options": {
+ "intent": "full_text"
+ },
+ "expected": [
+ {
+ "id": "page:guides/authentication-overview",
+ "score": 1579.1012710031655,
+ "signals": {
+ "lexical": 600,
+ "fullText": 46.45506355015828,
+ "recency": 50
+ }
+ },
+ {
+ "id": "page:auth",
+ "score": 50.378128779313016,
+ "signals": {
+ "lexical": 0,
+ "fullText": 1.581406438965651,
+ "recency": 18.75
+ }
+ }
+ ]
+ },
+ {
+ "query": "retry backoff",
+ "options": {
+ "intent": "full_text"
+ },
+ "expected": [
+ {
+ "id": "page:notes/retry-strategies",
+ "score": 993.4562852139901,
+ "signals": {
+ "lexical": 0,
+ "fullText": 47.485314260699504,
+ "recency": 43.75
+ }
+ },
+ {
+ "id": "page:auth/session-token-refresh",
+ "score": 49.16444617605368,
+ "signals": {
+ "lexical": 0,
+ "fullText": 1.208222308802684,
+ "recency": 25
+ }
+ }
+ ]
+ },
+ {
+ "query": "sourdough",
+ "options": {
+ "intent": "full_text"
+ },
+ "expected": [
+ {
+ "id": "page:recipes/sourdough",
+ "score": 1628.0239020710135,
+ "signals": {
+ "lexical": 700,
+ "fullText": 46.40119510355068,
+ "recency": 0
+ }
+ },
+ {
+ "id": "page:recipes/focaccia",
+ "score": 19.347634167019383,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0.6548817083509691,
+ "recency": 6.25
+ }
+ }
+ ]
+ },
+ {
+ "query": "bread",
+ "options": {
+ "intent": "full_text"
+ },
+ "expected": [
+ {
+ "id": "page:recipes/sourdough",
+ "score": 781.1319323262233,
+ "signals": {
+ "lexical": 500,
+ "fullText": 14.056596616311161,
+ "recency": 0
+ }
+ }
+ ]
+ },
+ {
+ "query": "recipes",
+ "options": {
+ "intent": "omnibar",
+ "scopes": ["folder"]
+ },
+ "expected": [
+ {
+ "id": "folder:recipes",
+ "score": 1808.6874410044754,
+ "signals": {
+ "lexical": 700,
+ "fullText": 55.43437205022377,
+ "recency": 0
+ }
+ }
+ ]
+ },
+ {
+ "query": "login",
+ "options": {
+ "intent": "full_text",
+ "limit": 2
+ },
+ "expected": [
+ {
+ "id": "page:auth/login",
+ "score": 1723.0932667574211,
+ "signals": {
+ "lexical": 700,
+ "fullText": 50.52966333787105,
+ "recency": 12.5
+ }
+ },
+ {
+ "id": "page:misc/login-history",
+ "score": 1491.3407282091468,
+ "signals": {
+ "lexical": 600,
+ "fullText": 43.00453641045734,
+ "recency": 31.25
+ }
+ }
+ ]
+ },
+ {
+ "query": "authentication",
+ "options": {
+ "intent": "full_text",
+ "scopes": ["page", "content"]
+ },
+ "expected": [
+ {
+ "id": "page:guides/authentication-overview",
+ "score": 1579.1012710031655,
+ "signals": {
+ "lexical": 600,
+ "fullText": 46.45506355015828,
+ "recency": 50
+ }
+ },
+ {
+ "id": "page:auth",
+ "score": 50.378128779313016,
+ "signals": {
+ "lexical": 0,
+ "fullText": 1.581406438965651,
+ "recency": 18.75
+ }
+ }
+ ]
+ },
+ {
+ "query": "token",
+ "options": {
+ "intent": "full_text"
+ },
+ "expected": [
+ {
+ "id": "page:auth/session-token-refresh",
+ "score": 708.2690256615901,
+ "signals": {
+ "lexical": 500,
+ "fullText": 9.163451283079505,
+ "recency": 25
+ }
+ },
+ {
+ "id": "page:guides/authentication-overview",
+ "score": 67.61341888448041,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0.8806709442240203,
+ "recency": 50
+ }
+ }
+ ]
+ },
+ {
+ "query": "sessions",
+ "options": {
+ "intent": "full_text"
+ },
+ "expected": [
+ {
+ "id": "page:auth/session-token-refresh",
+ "score": 202.3754508192539,
+ "signals": {
+ "lexical": 0,
+ "fullText": 8.868772540962695,
+ "recency": 25
+ }
+ },
+ {
+ "id": "page:guides/authentication-overview",
+ "score": 67.61341888448041,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0.8806709442240203,
+ "recency": 50
+ }
+ },
+ {
+ "id": "page:misc/login-history",
+ "score": 40.54046425448354,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0.46452321272417674,
+ "recency": 31.25
+ }
+ },
+ {
+ "id": "page:auth/login",
+ "score": 24.792405644684873,
+ "signals": {
+ "lexical": 0,
+ "fullText": 0.6146202822342437,
+ "recency": 12.5
+ }
+ }
+ ]
+ }
+ ]
+}
diff --git a/packages/core/src/search/workspace-search.parity.test.ts b/packages/core/src/search/workspace-search.parity.test.ts
new file mode 100644
index 000000000..c8e21fabe
--- /dev/null
+++ b/packages/core/src/search/workspace-search.parity.test.ts
@@ -0,0 +1,45 @@
+import { describe, expect, test } from 'bun:test';
+import { readFileSync } from 'node:fs';
+import {
+ createWorkspaceSearchCorpus,
+ searchWorkspaceCorpus,
+ type WorkspaceSearchDocument,
+ type WorkspaceSearchOptions,
+} from './workspace-search.ts';
+
+interface FixtureCase {
+ query: string;
+ options: WorkspaceSearchOptions;
+ expected: Array<{ id: string; score: number; signals: Record<string, number> }>;
+}
+interface Fixture {
+ documents: WorkspaceSearchDocument[];
+ cases: FixtureCase[];
+}
+
+const fixture = JSON.parse(
+ readFileSync(new URL('./workspace-search.baseline.fixture.json', import.meta.url), 'utf-8'),
+) as Fixture;
+
+describe('flag-OFF parity with the pre-embeddings baseline', () => {
+ const corpus = createWorkspaceSearchCorpus(fixture.documents);
+
+ test('fixture is non-trivial', () => {
+ expect(fixture.cases.length).toBeGreaterThan(5);
+ expect(fixture.documents.length).toBeGreaterThan(5);
+ });
+
+ for (const [i, c] of fixture.cases.entries()) {
+ test(`case ${i}: "${c.query}" ${JSON.stringify(c.options)} is byte-identical`, () => {
+ const actual = searchWorkspaceCorpus(corpus, c.query, c.options).map((r) => ({
+ id: r.document.id,
+ score: r.score,
+ signals: r.signals,
+ }));
+ expect(actual).toEqual(c.expected);
+ for (const r of actual) {
+ expect('vector' in r.signals).toBe(false);
+ }
+ });
+ }
+});
diff --git a/packages/core/src/search/workspace-search.semantic.test.ts b/packages/core/src/search/workspace-search.semantic.test.ts
new file mode 100644
index 000000000..a1bda667d
--- /dev/null
+++ b/packages/core/src/search/workspace-search.semantic.test.ts
@@ -0,0 +1,146 @@
+import { describe, expect, test } from 'bun:test';
+import {
+ createWorkspaceSearchCorpus,
+ createWorkspaceSearchDocument,
+ searchWorkspaceCorpus,
+ type WorkspaceSearchDocument,
+} from './workspace-search.ts';
+
+function page(
+ path: string,
+ title: string,
+ content: string,
+ modifiedTs = 1000,
+): WorkspaceSearchDocument {
+ return createWorkspaceSearchDocument({ kind: 'page', path, title, content, modifiedTs });
+}
+
+function ids(results: ReturnType<typeof searchWorkspaceCorpus>): string[] {
+ return results.map((r) => r.document.path);
+}
+
+describe('semantic ranking — candidate source + brackets-outer/RRF-body', () => {
+ test('C2a: a strong vector match never displaces an exact-title lexical match', () => {
+ const docs = [
+ page('auth/login', 'Login', 'the login page authenticates a user'),
+ page('guides/credentials', 'Credentials Guide', 'how authorization and tokens work'),
+ ];
+ const corpus = createWorkspaceSearchCorpus(docs);
+ const scores = new Map([
+ ['page:auth/login', 0.25],
+ ['page:guides/credentials', 0.97],
+ ]);
+ const results = searchWorkspaceCorpus(corpus, 'login', {
+ intent: 'full_text',
+ semantic: { scores },
+ });
+ expect(results[0].document.path).toBe('auth/login');
+ expect(ids(results)).toContain('guides/credentials'); // still surfaced, just below
+ });
+
+ test('C2b: a zero-token-overlap vector-only doc is unioned as a candidate and surfaces', () => {
+ const docs = [
+ page(
+ 'guides/credential-rotation',
+ 'Credential Rotation',
+ 'the credential rotation flow re-issues secrets when they expire',
+ ),
+ page('recipes/sourdough', 'Sourdough', 'a recipe for sourdough bread'),
+ ];
+ const corpus = createWorkspaceSearchCorpus(docs);
+ const noSemantic = searchWorkspaceCorpus(corpus, 'auth retries', { intent: 'full_text' });
+ expect(noSemantic.length).toBe(0); // proves it's vector-only retrievable
+
+ const scores = new Map([
+ ['page:guides/credential-rotation', 0.82],
+ ['page:recipes/sourdough', 0.2],
+ ]);
+ const withSemantic = searchWorkspaceCorpus(corpus, 'auth retries', {
+ intent: 'full_text',
+ semantic: { scores },
+ });
+ expect(ids(withSemantic)).toEqual(['guides/credential-rotation']);
+ expect(withSemantic[0].signals.vector).toBeCloseTo(0.82, 5);
+ });
+
+ test('similarityFloor keeps a low-cosine vector-only doc out of the candidate pool', () => {
+ const docs = [page('recipes/sourdough', 'Sourdough', 'a recipe for sourdough bread')];
+ const corpus = createWorkspaceSearchCorpus(docs);
+ const scores = new Map([['page:recipes/sourdough', 0.2]]); // below default floor 0.35
+ const results = searchWorkspaceCorpus(corpus, 'auth retries', {
+ intent: 'full_text',
+ semantic: { scores },
+ });
+ expect(results.length).toBe(0);
+ });
+
+ test('signals.vector is present only for docs with a cosine, and only on the semantic path', () => {
+ const docs = [
+ page('auth/login', 'Login', 'login page'),
+ page('notes/misc', 'Misc', 'login appears here too in body'),
+ ];
+ const corpus = createWorkspaceSearchCorpus(docs);
+ for (const r of searchWorkspaceCorpus(corpus, 'login', { intent: 'full_text' })) {
+ expect('vector' in r.signals).toBe(false);
+ }
+ const scores = new Map([['page:auth/login', 0.6]]);
+ const results = searchWorkspaceCorpus(corpus, 'login', {
+ intent: 'full_text',
+ semantic: { scores },
+ });
+ const byPath = new Map(results.map((r) => [r.document.path, r]));
+ expect(byPath.get('auth/login')?.signals.vector).toBeCloseTo(0.6, 5);
+ const misc = byPath.get('notes/misc');
+ expect(misc).toBeDefined();
+ expect('vector' in (misc?.signals ?? {})).toBe(false);
+ });
+
+ test('a candidate whose cosine is below the floor does not report signals.vector', () => {
+ const docs = [page('auth/login', 'Login', 'the login page authenticates a user')];
+ const corpus = createWorkspaceSearchCorpus(docs);
+ const scores = new Map([['page:auth/login', 0.2]]);
+ const results = searchWorkspaceCorpus(corpus, 'login', {
+ intent: 'full_text',
+ semantic: { scores },
+ });
+ expect(results.length).toBe(1);
+ expect('vector' in results[0].signals).toBe(false);
+ });
+
+ test('RRF body tier: a strong vector signal promotes a doc above a stronger-BM25 rival', () => {
+ const docs = [
+ page('obs/observability', 'Observability', 'telemetry telemetry telemetry spans and metrics'),
+ page('data/pipeline', 'Pipeline', 'telemetry pipeline ingestion'),
+ ];
+ const corpus = createWorkspaceSearchCorpus(docs);
+ const lexicalOnly = searchWorkspaceCorpus(corpus, 'telemetry', { intent: 'full_text' });
+ expect(lexicalOnly.length).toBe(2);
+ const bm25Top = lexicalOnly[0].document.path;
+ const bm25Second = lexicalOnly[1].document.path;
+
+ const scores = new Map<string, number>([
+ [`page:${bm25Top}`, 0.1],
+ [`page:${bm25Second}`, 0.95],
+ ]);
+ const fused = searchWorkspaceCorpus(corpus, 'telemetry', {
+ intent: 'full_text',
+ semantic: { scores },
+ });
+ expect(fused[0].document.path).toBe(bm25Second);
+ expect(fused[1].document.path).toBe(bm25Top);
+ });
+
+ test('candidateLimit bounds how many vector-only docs are unioned', () => {
+ const docs = Array.from({ length: 5 }, (_, i) =>
+ page(`v/doc-${i}`, `Doc ${i}`, `body about widget number ${i}`),
+ );
+ const corpus = createWorkspaceSearchCorpus(docs);
+ const scores = new Map(docs.map((d, i) => [d.id, 0.9 - i * 0.05] as const));
+ const results = searchWorkspaceCorpus(corpus, 'unrelated concept query', {
+ intent: 'full_text',
+ semantic: { scores, candidateLimit: 2 },
+ });
+ expect(results.length).toBe(2);
+ expect(ids(results).sort()).toEqual(['v/doc-0', 'v/doc-1']);
+ });
+});
diff --git a/packages/core/src/search/workspace-search.ts b/packages/core/src/search/workspace-search.ts
index a66f1d579..748d83673 100644
--- a/packages/core/src/search/workspace-search.ts
+++ b/packages/core/src/search/workspace-search.ts
@@ -22,13 +22,22 @@ export interface WorkspaceSearchResult {
lexical: number;
fullText: number;
recency: number;
+ vector?: number;
};
}
+export interface WorkspaceSemanticInput {
+ scores: ReadonlyMap<string, number>;
+ rrfK?: number;
+ candidateLimit?: number;
+ similarityFloor?: number;
+}
+
export interface WorkspaceSearchOptions {
intent?: WorkspaceSearchIntent;
scopes?: readonly WorkspaceSearchScope[];
limit?: number;
+ semantic?: WorkspaceSemanticInput;
}
export interface WorkspaceSearchCorpus {
@@ -38,6 +47,10 @@ export interface WorkspaceSearchCorpus {
export const DEFAULT_WORKSPACE_SEARCH_LIMIT = 20;
export const MAX_WORKSPACE_SEARCH_LIMIT = 100;
+export const DEFAULT_RRF_K = 60;
+export const DEFAULT_VECTOR_CANDIDATE_LIMIT = 64;
+export const DEFAULT_VECTOR_SIMILARITY_FLOOR = 0.35;
+
const WORKSPACE_SEARCH_SCHEMA = {
id: 'string',
kind: 'enum',
@@ -253,20 +266,140 @@ export function searchWorkspaceCorpus(
candidates.set(hit.document.id, hit.document);
}
- return [...candidates.values()]
- .map((document) => {
- const lexical = Math.max(0, lexicalScore(document, normalizedQuery));
- const fullText = fullTextScores.get(document.id) ?? 0;
- const recencyScore = recency.get(document.id) ?? 0;
- return {
- document,
- score: lexical + fullText * 20 + recencyScore,
- signals: { lexical, fullText, recency: recencyScore },
- };
- })
+ if (!options.semantic) {
+ return [...candidates.values()]
+ .map((document) => {
+ const lexical = Math.max(0, lexicalScore(document, normalizedQuery));
+ const fullText = fullTextScores.get(document.id) ?? 0;
+ const recencyScore = recency.get(document.id) ?? 0;
+ return {
+ document,
+ score: lexical + fullText * 20 + recencyScore,
+ signals: { lexical, fullText, recency: recencyScore },
+ };
+ })
+ .sort((a, b) => {
+ if (a.score !== b.score) return b.score - a.score;
+ return a.document.path.localeCompare(b.document.path);
+ })
+ .slice(0, limit);
+ }
+
+ return rankWithVector({
+ scopedDocuments,
+ candidates,
+ fullTextScores,
+ recency,
+ normalizedQuery,
+ semantic: options.semantic,
+ limit,
+ });
+}
+
+interface SemanticRow {
+ document: WorkspaceSearchDocument;
+ score: number;
+ signals: WorkspaceSearchResult['signals'];
+ lexical: number;
+ fullText: number;
+ recency: number;
+ cosine: number | undefined;
+}
+
+function denseRank(orderedIds: readonly string[]): Map<string, number> {
+ const ranks = new Map<string, number>();
+ orderedIds.forEach((id, i) => {
+ ranks.set(id, i + 1);
+ });
+ return ranks;
+}
+
+function rankWithVector(args: {
+ scopedDocuments: readonly WorkspaceSearchDocument[];
+ candidates: Map<string, WorkspaceSearchDocument>;
+ fullTextScores: ReadonlyMap<string, number>;
+ recency: ReadonlyMap<string, number>;
+ normalizedQuery: string;
+ semantic: WorkspaceSemanticInput;
+ limit: number;
+}): WorkspaceSearchResult[] {
+ const { scopedDocuments, candidates, fullTextScores, recency, normalizedQuery, semantic, limit } =
+ args;
+ const rrfK = semantic.rrfK ?? DEFAULT_RRF_K;
+ const candidateLimit = semantic.candidateLimit ?? DEFAULT_VECTOR_CANDIDATE_LIMIT;
+ const floor = semantic.similarityFloor ?? DEFAULT_VECTOR_SIMILARITY_FLOOR;
+ const vectorScores = semantic.scores;
+
+ const scopedById = new Map(scopedDocuments.map((d) => [d.id, d] as const));
+ const topVector = [...vectorScores.entries()]
+ .filter(([id, cos]) => cos >= floor && scopedById.has(id))
+ .sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))
+ .slice(0, candidateLimit);
+ for (const [id] of topVector) {
+ if (!candidates.has(id)) {
+ const doc = scopedById.get(id);
+ if (doc) candidates.set(id, doc);
+ }
+ }
+
+ const rows: SemanticRow[] = [...candidates.values()].map((document) => {
+ const lexical = Math.max(0, lexicalScore(document, normalizedQuery));
+ const fullText = fullTextScores.get(document.id) ?? 0;
+ const recencyScore = recency.get(document.id) ?? 0;
+ const cosine = vectorScores.get(document.id);
+ const qualifies = cosine !== undefined && cosine >= floor;
+ return {
+ document,
+ score: lexical + fullText * 20 + recencyScore,
+ signals: qualifies
+ ? { lexical, fullText, recency: recencyScore, vector: cosine }
+ : { lexical, fullText, recency: recencyScore },
+ lexical,
+ fullText,
+ recency: recencyScore,
+ cosine,
+ };
+ });
+
+ const bm25Rank = denseRank(
+ rows
+ .filter((r) => r.fullText > 0)
+ .sort((a, b) => b.fullText - a.fullText || a.document.path.localeCompare(b.document.path))
+ .map((r) => r.document.id),
+ );
+ const vecRank = denseRank(
+ rows
+ .filter(
+ (r): r is SemanticRow & { cosine: number } => r.cosine !== undefined && r.cosine >= floor,
+ )
+ .sort((a, b) => b.cosine - a.cosine || a.document.path.localeCompare(b.document.path))
+ .map((r) => r.document.id),
+ );
+
+ const rrfScore = (id: string): number => {
+ let s = 0;
+ const br = bm25Rank.get(id);
+ if (br !== undefined) s += 1 / (rrfK + br);
+ const vr = vecRank.get(id);
+ if (vr !== undefined) s += 1 / (rrfK + vr);
+ return s;
+ };
+
+ return rows
.sort((a, b) => {
- if (a.score !== b.score) return b.score - a.score;
+ const aLexical = a.lexical > 0 ? 1 : 0;
+ const bLexical = b.lexical > 0 ? 1 : 0;
+ if (aLexical !== bLexical) return bLexical - aLexical; // lexical tier first
+ if (aLexical === 1) {
+ if (a.score !== b.score) return b.score - a.score; // within lexical tier: legacy order
+ return a.document.path.localeCompare(b.document.path);
+ }
+ const ar = rrfScore(a.document.id);
+ const br = rrfScore(b.document.id);
+ if (ar !== br) return br - ar; // body tier: RRF(BM25, vector)
+ if (a.recency !== b.recency) return b.recency - a.recency;
return a.document.path.localeCompare(b.document.path);
})
+ .map(({ document, score, signals }) => ({ document, score, signals }))
.slice(0, limit);
}
diff --git a/packages/core/src/util/doc-name.test.ts b/packages/core/src/util/doc-name.test.ts
index da252e0c1..f9dddb87d 100644
--- a/packages/core/src/util/doc-name.test.ts
+++ b/packages/core/src/util/doc-name.test.ts
@@ -1,5 +1,12 @@
import { describe, expect, test } from 'bun:test';
-import { isValidDocName, validateDocName } from './doc-name.ts';
+import { isHiddenDocName, isValidDocName, validateDocName } from './doc-name.ts';
+
+describe('isHiddenDocName', () => {
+ for (const name of ['.cursor/skills/x', '.claude/foo', 'a/.hidden/b', '.okignore', 'a/.b'])
+ test(`hidden: ${JSON.stringify(name)}`, () => expect(isHiddenDocName(name)).toBe(true));
+ for (const name of ['Characters/Spike Spiegel', 'Music', 'a/b/c', 'note.with.dots'])
+ test(`visible: ${JSON.stringify(name)}`, () => expect(isHiddenDocName(name)).toBe(false));
+});
describe('validateDocName', () => {
test('accepts ordinary extension-less docNames', () => {
diff --git a/packages/core/src/util/doc-name.ts b/packages/core/src/util/doc-name.ts
index 162801247..b1102db51 100644
--- a/packages/core/src/util/doc-name.ts
+++ b/packages/core/src/util/doc-name.ts
@@ -38,3 +38,7 @@ export function validateDocName(name: string): { ok: true } | { ok: false; reaso
export function isValidDocName(name: string): boolean {
return validateDocName(name).ok;
}
+
+export function isHiddenDocName(name: string): boolean {
+ return name.split('/').some((segment) => segment.startsWith('.'));
+}
diff --git a/packages/desktop/src/utility/server-entry.ts b/packages/desktop/src/utility/server-entry.ts
index 8af38f48b..a25075b26 100644
--- a/packages/desktop/src/utility/server-entry.ts
+++ b/packages/desktop/src/utility/server-entry.ts
@@ -9,6 +9,7 @@ import {
ConfigSchema,
ensureProjectGit,
initContent,
+ makeLazyEmbeddingsKeyStore,
} from '@inkeep/open-knowledge-server';
import { type KeyringSmokeResult, runKeyringSmoke } from './keyring-smoke.ts';
@@ -186,6 +187,7 @@ export function setupUtility(deps: SetupUtilityDeps): UtilityHandle {
}
const tokenStore = makeLazyProbeTokenStore();
+ const embeddingsKeyStore = makeLazyEmbeddingsKeyStore();
booted = await server.bootServer({
...msg.opts,
@@ -198,6 +200,7 @@ export function setupUtility(deps: SetupUtilityDeps): UtilityHandle {
autoInitFn: undefined,
detectGh,
tokenStore,
+ embeddingsKeyStore,
serveContentAssets: true,
...(msg.opts.reactShellDistDir ? { reactShellDistDir: msg.opts.reactShellDistDir } : {}),
});
diff --git a/packages/server/src/api-extension.ts b/packages/server/src/api-extension.ts
index b1afdb666..dcf88e632 100644
--- a/packages/server/src/api-extension.ts
+++ b/packages/server/src/api-extension.ts
@@ -87,6 +87,7 @@ import {
type InlineAssetMediaKind,
InstallSkillRequestSchema,
InstallSkillSuccessSchema,
+ isHiddenDocName,
LINKABLE_ASSET_EXTENSIONS,
type LifecycleStatus,
LinkGraphSuccessSchema,
@@ -100,6 +101,8 @@ import {
LocalOpAuthStatusSuccessSchema,
type LocalOpCloneRequest,
LocalOpCloneRequestSchema,
+ LocalOpEmbeddingsMutationSuccessSchema,
+ LocalOpEmbeddingsSetKeyRequestSchema,
LocalOpOkInitRequestSchema,
LocalOpOkInitResponseSchema,
LocalOpOpenRequestSchema,
@@ -128,11 +131,14 @@ import {
SaveVersionRequestSchema,
SaveVersionSuccessSchema,
SearchRequestSchema,
+ type SearchSemanticStatus,
+ type SearchSuccess,
SearchSuccessSchema,
SeedApplyRequestSchema,
SeedApplySuccessSchema,
SeedListPacksSuccessSchema,
SeedPlanSuccessSchema,
+ SemanticIndexStatusSchema,
ServerInfoSuccessSchema,
ShareConstructUrlRequestSchema,
ShareConstructUrlResponseSchema,
@@ -172,7 +178,9 @@ import {
type WorkspaceSearchCorpus,
type WorkspaceSearchDocument,
type WorkspaceSearchIntent,
+ type WorkspaceSearchResult,
type WorkspaceSearchScope,
+ type WorkspaceSemanticInput,
WorkspaceSuccessSchema,
} from '@inkeep/open-knowledge-core';
import {
@@ -222,6 +230,17 @@ import {
} from './content-divergence-gate.ts';
import { recordContributor } from './contributor-tracker.ts';
import { deriveDetection, embedProbeRing, recordEmbedProbe } from './embed-probe.ts';
+import {
+ recordSemanticQuery,
+ type SemanticQueryOutcome,
+} from './embeddings/embeddings-telemetry.ts';
+import {
+ clearEmbeddingsKeyFromAllBackends,
+ EMBEDDINGS_API_KEY_ENV,
+ FileEmbeddingsBackend,
+ SEMANTIC_MIN_QUERY_LENGTH,
+ type SemanticSearchService,
+} from './embeddings/index.ts';
import {
FrontmatterMalformedError,
respondFrontmatterMalformed,
@@ -1554,6 +1573,8 @@ export interface ApiExtensionOptions {
ready?: Promise<void>;
recentlyRemovedDocs?: RecentlyRemovedDocs;
serializeDoc?: (docName: string) => string | null;
+ semanticSearch?: SemanticSearchService;
+ embeddingsSecretsFile?: string;
}
interface WorkspaceSearchCacheEntry {
@@ -1634,6 +1655,8 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
ready,
recentlyRemovedDocs,
serializeDoc,
+ semanticSearch,
+ embeddingsSecretsFile,
ephemeral = false,
} = options;
@@ -10257,10 +10280,142 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
return scopes.length > 0 ? scopes : undefined;
}
+ function parseSemanticParam(value: unknown): boolean | undefined {
+ if (typeof value === 'boolean') return value;
+ if (value === 'true') return true;
+ if (value === 'false') return false;
+ return undefined;
+ }
+
+ interface SemanticResolution {
+ input?: WorkspaceSemanticInput;
+ status?: SearchSemanticStatus;
+ queryEmbedMs: number | null;
+ pageTotal: number;
+ capable: boolean;
+ }
+
+ async function resolveSemantic(
+ query: string,
+ intent: WorkspaceSearchIntent,
+ semanticParam: boolean | undefined,
+ corpus: WorkspaceSearchCorpus,
+ ): Promise<SemanticResolution> {
+ const pageTotal = corpus.documents.reduce((n, d) => n + (d.kind === 'page' ? 1 : 0), 0);
+ if (!semanticSearch?.isEnabled() || semanticParam !== true) {
+ return { queryEmbedMs: null, pageTotal, capable: false };
+ }
+
+ void semanticSearch.embedCorpus(corpus.documents);
+
+ let input: WorkspaceSemanticInput | undefined;
+ let queryEmbedMs: number | null = null;
+ if (intent === 'full_text' && query.trim().length >= SEMANTIC_MIN_QUERY_LENGTH) {
+ const startedAt = performance.now();
+ const scores = await semanticSearch.queryScores(query, corpus.documents);
+ queryEmbedMs = performance.now() - startedAt;
+ if (scores && scores.size > 0) input = { scores };
+ }
+
+ const status = semanticSearch.getStatus();
+ return {
+ input,
+ status: {
+ capable: status.capable,
+ applied: false, // finalized post-ranking (did any result carry a vector)
+ coverage: { embedded: status.embeddedCount, total: pageTotal },
+ },
+ queryEmbedMs,
+ pageTotal,
+ capable: status.capable,
+ };
+ }
+
+ function toSearchResultEntry(
+ result: ReturnType<typeof searchWorkspaceCorpus>[number],
+ query: string,
+ ): {
+ kind: WorkspaceSearchScope;
+ path: string;
+ title: string;
+ score: number;
+ signals: WorkspaceSearchResult['signals'];
+ snippet?: string;
+ } {
+ return {
+ kind: result.document.kind,
+ path: result.document.path,
+ title: result.document.title,
+ score: result.score,
+ signals: result.signals,
+ snippet:
+ result.document.kind === 'page'
+ ? buildSearchSnippet(result.document.content, query)
+ : undefined,
+ };
+ }
+
+ async function buildSearchResponse(params: {
+ query: string;
+ intent: WorkspaceSearchIntent;
+ scopes: WorkspaceSearchScope[] | undefined;
+ limit: number | undefined;
+ semanticParam: boolean | undefined;
+ }): Promise<SearchSuccess> {
+ const startedAt = performance.now();
+ const corpus = await getWorkspaceSearchCorpus();
+ const semantic = await resolveSemantic(
+ params.query,
+ params.intent,
+ params.semanticParam,
+ corpus,
+ );
+ const results = searchWorkspaceCorpus(corpus, params.query, {
+ intent: params.intent,
+ scopes: params.scopes,
+ limit: params.limit,
+ semantic: semantic.input,
+ });
+ const entries = results.map((r) => toSearchResultEntry(r, params.query));
+
+ let semanticStatus: SearchSemanticStatus | undefined;
+ if (semantic.status) {
+ const vectorContributors = entries.reduce(
+ (n, e) => n + (e.signals.vector !== undefined ? 1 : 0),
+ 0,
+ );
+ const applied = vectorContributors > 0;
+ semanticStatus = { ...semantic.status, applied };
+ const outcome: SemanticQueryOutcome = !semantic.capable
+ ? 'incapable'
+ : applied
+ ? 'applied'
+ : semantic.status.coverage.embedded === 0
+ ? 'warming'
+ : 'no_match';
+ recordSemanticQuery({
+ outcome,
+ capable: semantic.capable,
+ embedded: semantic.status.coverage.embedded,
+ total: semantic.pageTotal,
+ queryEmbedMs: semantic.queryEmbedMs,
+ vectorContributors,
+ });
+ }
+
+ return {
+ query: params.query,
+ intent: params.intent,
+ results: entries,
+ elapsedMs: Math.max(0, performance.now() - startedAt),
+ ...(semanticStatus ? { semantic: semanticStatus } : {}),
+ };
+ }
+
async function buildWorkspaceSearchDocumentsFromIndex(): Promise<WorkspaceSearchDocument[]> {
const pages: WorkspaceSearchDocument[] = [];
for (const [docName, entry] of getFileIndex()) {
- if (isSystemDoc(docName) || isConfigDoc(docName)) continue;
+ if (isSystemDoc(docName) || isConfigDoc(docName) || isHiddenDocName(docName)) continue;
let content = '';
let title = docName;
try {
@@ -10284,7 +10439,9 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
function workspaceSearchFingerprint(): string {
return [...getFileIndex()]
- .filter(([docName]) => !isSystemDoc(docName) && !isConfigDoc(docName))
+ .filter(
+ ([docName]) => !isSystemDoc(docName) && !isConfigDoc(docName) && !isHiddenDocName(docName),
+ )
.sort(([a], [b]) => a.localeCompare(b))
.map(
([docName, entry]) =>
@@ -10358,6 +10515,7 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
const scopes = parseSearchScopes(
url.searchParams.get('scope') ?? url.searchParams.get('scopes'),
);
+ const semanticParam = parseSemanticParam(url.searchParams.get('semantic'));
const limitNum = limit === null ? undefined : Number(limit);
if (query.length > 200) {
@@ -10371,35 +10529,14 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
return;
}
try {
- const startedAt = performance.now();
- const corpus = await getWorkspaceSearchCorpus();
- const results = searchWorkspaceCorpus(corpus, query, {
+ const body = await buildSearchResponse({
+ query,
intent,
scopes,
limit: limitNum,
+ semanticParam,
});
- successResponse(
- res,
- 200,
- SearchSuccessSchema,
- {
- query,
- intent,
- results: results.map((result) => ({
- kind: result.document.kind,
- path: result.document.path,
- title: result.document.title,
- score: result.score,
- signals: result.signals,
- snippet:
- result.document.kind === 'page'
- ? buildSearchSnippet(result.document.content, query)
- : undefined,
- })),
- elapsedMs: Math.max(0, performance.now() - startedAt),
- },
- { handler: 'search-get' },
- );
+ successResponse(res, 200, SearchSuccessSchema, body, { handler: 'search-get' });
} catch (e) {
errorResponse(
res,
@@ -10420,6 +10557,7 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
const intent = parseSearchIntent(body.intent);
const scopes = parseSearchScopes(body.scopes ?? body.scope);
const limit = typeof body.limit === 'number' ? body.limit : undefined;
+ const semanticParam = parseSemanticParam(body.semantic);
if (query.length > 200) {
errorResponse(
@@ -10432,35 +10570,14 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
return;
}
try {
- const startedAt = performance.now();
- const corpus = await getWorkspaceSearchCorpus();
- const results = searchWorkspaceCorpus(corpus, query, {
+ const responseBody = await buildSearchResponse({
+ query,
intent,
scopes,
limit,
+ semanticParam,
});
- successResponse(
- res,
- 200,
- SearchSuccessSchema,
- {
- query,
- intent,
- results: results.map((result) => ({
- kind: result.document.kind,
- path: result.document.path,
- title: result.document.title,
- score: result.score,
- signals: result.signals,
- snippet:
- result.document.kind === 'page'
- ? buildSearchSnippet(result.document.content, query)
- : undefined,
- })),
- elapsedMs: Math.max(0, performance.now() - startedAt),
- },
- { handler: 'search-post' },
- );
+ successResponse(res, 200, SearchSuccessSchema, responseBody, { handler: 'search-post' });
} catch (e) {
errorResponse(
res,
@@ -11099,6 +11216,139 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
});
}
+ const HANDLE_LOCAL_OP_EMBEDDINGS_SET_KEY = 'local-op-embeddings-set-key';
+ const HANDLE_LOCAL_OP_EMBEDDINGS_CLEAR_KEY = 'local-op-embeddings-clear-key';
+ const LOCAL_OP_EMBEDDINGS_GUARD = '/api/local-op/embeddings';
+
+ const handleLocalOpEmbeddingsSetKey = withValidation(
+ LocalOpEmbeddingsSetKeyRequestSchema,
+ async (_req, res, body) => {
+ if (!localOpGuard.tryAcquire(LOCAL_OP_EMBEDDINGS_GUARD)) {
+ errorResponse(
+ res,
+ 429,
+ 'urn:ok:error:concurrent-operation',
+ 'An embeddings key operation is already in progress.',
+ { handler: HANDLE_LOCAL_OP_EMBEDDINGS_SET_KEY, extraHeaders: { 'Retry-After': '5' } },
+ );
+ return;
+ }
+ try {
+ await new FileEmbeddingsBackend(embeddingsSecretsFile).set(body.key);
+ successResponse(
+ res,
+ 200,
+ LocalOpEmbeddingsMutationSuccessSchema,
+ { keyPresent: true },
+ {
+ handler: HANDLE_LOCAL_OP_EMBEDDINGS_SET_KEY,
+ extraHeaders: { 'Cache-Control': 'no-store' },
+ },
+ );
+ } catch (e) {
+ errorResponse(res, 500, 'urn:ok:error:internal-server-error', 'Failed to store the key.', {
+ handler: HANDLE_LOCAL_OP_EMBEDDINGS_SET_KEY,
+ cause: e,
+ });
+ } finally {
+ localOpGuard.release(LOCAL_OP_EMBEDDINGS_GUARD);
+ }
+ },
+ {
+ handler: HANDLE_LOCAL_OP_EMBEDDINGS_SET_KEY,
+ method: 'POST',
+ preBodyGate: (req, res) =>
+ checkLocalOpSecurity(req, res, { handler: HANDLE_LOCAL_OP_EMBEDDINGS_SET_KEY }),
+ },
+ );
+
+ const handleLocalOpEmbeddingsClearKey = withValidation(
+ EmptyRequestSchema,
+ async (_req, res) => {
+ if (!localOpGuard.tryAcquire(LOCAL_OP_EMBEDDINGS_GUARD)) {
+ errorResponse(
+ res,
+ 429,
+ 'urn:ok:error:concurrent-operation',
+ 'An embeddings key operation is already in progress.',
+ { handler: HANDLE_LOCAL_OP_EMBEDDINGS_CLEAR_KEY, extraHeaders: { 'Retry-After': '5' } },
+ );
+ return;
+ }
+ try {
+ await clearEmbeddingsKeyFromAllBackends(embeddingsSecretsFile);
+ successResponse(
+ res,
+ 200,
+ LocalOpEmbeddingsMutationSuccessSchema,
+ { keyPresent: false },
+ {
+ handler: HANDLE_LOCAL_OP_EMBEDDINGS_CLEAR_KEY,
+ extraHeaders: { 'Cache-Control': 'no-store' },
+ },
+ );
+ } catch (e) {
+ errorResponse(res, 500, 'urn:ok:error:internal-server-error', 'Failed to clear the key.', {
+ handler: HANDLE_LOCAL_OP_EMBEDDINGS_CLEAR_KEY,
+ cause: e,
+ });
+ } finally {
+ localOpGuard.release(LOCAL_OP_EMBEDDINGS_GUARD);
+ }
+ },
+ {
+ handler: HANDLE_LOCAL_OP_EMBEDDINGS_CLEAR_KEY,
+ method: 'POST',
+ preBodyGate: (req, res) =>
+ checkLocalOpSecurity(req, res, { handler: HANDLE_LOCAL_OP_EMBEDDINGS_CLEAR_KEY }),
+ },
+ );
+
+ const handleSemanticStatus = withValidation(
+ EmptyRequestSchema,
+ async (_req, res) => {
+ try {
+ let enabled = false;
+ let ready = false;
+ let capable = false;
+ let embedded = 0;
+ if (semanticSearch) {
+ const status = semanticSearch.getStatus();
+ enabled = status.enabled;
+ ready = status.ready;
+ capable = status.capable;
+ embedded = status.embeddedCount;
+ }
+ const storedKey = await new FileEmbeddingsBackend(embeddingsSecretsFile).get();
+ const keySource: 'file' | 'env' | null = storedKey
+ ? 'file'
+ : process.env[EMBEDDINGS_API_KEY_ENV]
+ ? 'env'
+ : null;
+ const keyPresent = keySource !== null;
+ let total = 0;
+ for (const [docName] of getFileIndex()) {
+ if (!isSystemDoc(docName) && !isConfigDoc(docName) && !isHiddenDocName(docName)) {
+ total += 1;
+ }
+ }
+ successResponse(
+ res,
+ 200,
+ SemanticIndexStatusSchema,
+ { enabled, keyPresent, keySource, ready, capable, embedded, total },
+ { handler: 'semantic-status', extraHeaders: { 'Cache-Control': 'no-store' } },
+ );
+ } catch (e) {
+ errorResponse(res, 500, 'urn:ok:error:internal-server-error', 'Internal server error.', {
+ handler: 'semantic-status',
+ cause: e,
+ });
+ }
+ },
+ { handler: 'semantic-status', method: 'GET', skipBodyParse: true },
+ );
+
const routes: Record<string, (req: IncomingMessage, res: ServerResponse) => Promise<void>> = {
'/api/config': handleApiConfig,
'/api/asset': handleAsset,
@@ -11118,6 +11368,7 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
'/api/template': handleTemplate,
'/api/templates': handleTemplatesList,
'/api/search': handleSearch,
+ '/api/semantic-status': handleSemanticStatus,
'/api/suggest-links': handleSuggestLinks,
'/api/page-headings': handlePageHeadings,
'/api/create-page': handleCreatePage,
@@ -11168,6 +11419,8 @@ export function createApiExtension(options: ApiExtensionOptions): Extension {
'/api/local-op/auth/pat': handleLocalOpAuthPat,
'/api/local-op/auth/identity': handleLocalOpAuthIdentity,
'/api/local-op/auth/set-identity': handleLocalOpAuthSetIdentity,
+ '/api/local-op/embeddings/set-key': handleLocalOpEmbeddingsSetKey,
+ '/api/local-op/embeddings/clear-key': handleLocalOpEmbeddingsClearKey,
'/api/installed-agents': handleInstalledAgentsRoute,
'/api/spawn-cursor': handleSpawnCursorRoute,
'/api/handoff': handleHandoffDispatchRoute,
diff --git a/packages/server/src/api-search-semantic-factory.test.ts b/packages/server/src/api-search-semantic-factory.test.ts
new file mode 100644
index 000000000..acce49607
--- /dev/null
+++ b/packages/server/src/api-search-semantic-factory.test.ts
@@ -0,0 +1,340 @@
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import type { IncomingMessage, ServerResponse } from 'node:http';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { Readable } from 'node:stream';
+import type { Extension } from '@hocuspocus/server';
+import { createConceptEmbedder } from './embeddings/index.ts';
+import { createServer, type ServerInstance } from './server-factory.ts';
+import { initShadowRepo } from './shadow-repo.ts';
+
+const CONCEPTS = [
+ { id: 'auth', terms: ['auth', 'credential', 'session token', 'login', 'secret', 'sign-in'] },
+ { id: 'retry', terms: ['retry', 'retries', 'refresh', 're-issue', 'rotation', 'backoff'] },
+ { id: 'bread', terms: ['bread', 'sourdough', 'ferment', 'dough'] },
+];
+
+const SERVED_FILES: Record<string, string> = {
+ 'guides/credential-rotation.md':
+ '# Credential Rotation\n\nThe credential rotation flow re-issues secrets when they expire.\n',
+ 'recipes/sourdough.md': '# Sourdough\n\nA recipe for sourdough bread with a long cold ferment.\n',
+ 'auth/login.md': '# Login\n\nThe login page authenticates a user and starts a session.\n',
+};
+const EXCLUDED_FILES: Record<string, string> = {
+ 'archive/old-secrets.md':
+ '# Old Secrets\n\nLegacy notes on credential rotation: re-issue and refresh expired session secrets and login tokens.\n',
+};
+const HIDDEN_FILES: Record<string, string> = {
+ '.cursor/skills/auth-helper.md':
+ '# Auth Helper\n\nCredential rotation: re-issue and refresh expired session secrets and login tokens.\n',
+};
+const SERVED_PAGE_COUNT = Object.keys(SERVED_FILES).length;
+
+interface SearchRow {
+ kind: string;
+ path: string;
+ signals: { lexical: number; fullText: number; recency: number; vector?: number };
+}
+interface SearchBody {
+ results?: SearchRow[];
+ semantic?: { capable: boolean; applied: boolean; coverage: { embedded: number; total: number } };
+}
+
+function makeReq(method: string, url: string, body = ''): IncomingMessage {
+ const readable = Readable.from(Buffer.from(body)) as unknown as IncomingMessage;
+ readable.method = method;
+ readable.url = url;
+ readable.headers = { host: 'localhost' };
+ readable.socket = { remoteAddress: '127.0.0.1' } as unknown as IncomingMessage['socket'];
+ return readable;
+}
+function makeRes(): { res: ServerResponse; captured: { status: number; body: string } } {
+ const captured = { status: 0, body: '' };
+ const res = {
+ writeHead(status: number) {
+ captured.status = status;
+ },
+ end(body?: string) {
+ captured.body = body ?? '';
+ },
+ } as unknown as ServerResponse;
+ return { res, captured };
+}
+
+let tmpDir: string;
+let server: ServerInstance;
+
+async function callViaServer(
+ srv: ServerInstance,
+ method: string,
+ url: string,
+ bodyObj?: Record<string, unknown>,
+): Promise<unknown> {
+ const onRequestExts = srv.hocuspocus.configuration.extensions.filter(
+ (
+ e,
+ ): e is Extension & {
+ onRequest: (c: { request: IncomingMessage; response: ServerResponse }) => Promise<void>;
+ } => typeof (e as { onRequest?: unknown }).onRequest === 'function',
+ );
+ expect(onRequestExts.length, 'createServer must wire an onRequest api extension').toBeGreaterThan(
+ 0,
+ );
+ const { res, captured } = makeRes();
+ for (const ext of onRequestExts) {
+ const req = makeReq(method, url, bodyObj === undefined ? '' : JSON.stringify(bodyObj));
+ await ext.onRequest({ request: req, response: res });
+ if (captured.status !== 0) break;
+ }
+ expect(captured.status).toBe(200);
+ return JSON.parse(captured.body);
+}
+
+function searchViaServer(
+ srv: ServerInstance,
+ bodyObj: Record<string, unknown>,
+): Promise<SearchBody> {
+ return callViaServer(srv, 'POST', '/api/search', bodyObj) as Promise<SearchBody>;
+}
+
+beforeAll(async () => {
+ tmpDir = mkdtempSync(join(tmpdir(), 'ok-sem-factory-'));
+ for (const [rel, content] of Object.entries({
+ ...SERVED_FILES,
+ ...EXCLUDED_FILES,
+ ...HIDDEN_FILES,
+ })) {
+ const abs = join(tmpDir, rel);
+ mkdirSync(join(abs, '..'), { recursive: true });
+ writeFileSync(abs, content, 'utf-8');
+ }
+ writeFileSync(join(tmpDir, '.okignore'), 'archive/\n', 'utf-8');
+ mkdirSync(join(tmpDir, '.ok', 'local'), { recursive: true });
+ writeFileSync(
+ join(tmpDir, '.ok', 'local', 'config.yml'),
+ 'search:\n semantic:\n enabled: true\n',
+ 'utf-8',
+ );
+ writeFileSync(join(tmpDir, '.ok', 'secrets.yml'), 'embeddings: sk-test-factory-key\n', 'utf-8');
+
+ const shadowRepo = await initShadowRepo(tmpDir);
+ const embedder = createConceptEmbedder({ concepts: CONCEPTS });
+ server = createServer({
+ contentDir: tmpDir,
+ projectDir: tmpDir,
+ quiet: true,
+ debounce: 60_000,
+ gitEnabled: false,
+ shadowRepo,
+ skipStateManifestCheck: true,
+ destroyTimeoutMs: 500,
+ configHomedirOverride: tmpDir,
+ embedderLoader: () => Promise.resolve(embedder),
+ });
+ await server.ready;
+});
+
+afterAll(async () => {
+ await server.destroy();
+ rmSync(tmpDir, { recursive: true, force: true });
+});
+
+describe('createServer boot — flag-ON semantic search (factory glue)', () => {
+ test('config-enabled boot fuses a vector signal and reports coverage; excluded content stays out', async () => {
+ const deadline = Date.now() + 20_000;
+ let result: SearchBody | undefined;
+ do {
+ result = await searchViaServer(server, {
+ query: 'auth retries',
+ intent: 'full_text',
+ semantic: true,
+ });
+ for (const r of result.results ?? []) {
+ expect(r.path.startsWith('archive/')).toBe(false);
+ }
+ if ((result.semantic?.coverage.embedded ?? 0) >= SERVED_PAGE_COUNT) break;
+ await new Promise((r) => setTimeout(r, 25));
+ } while (Date.now() < deadline);
+
+ expect(result?.semantic?.capable).toBe(true);
+ expect(result?.semantic?.coverage.total).toBe(SERVED_PAGE_COUNT);
+ expect(result?.semantic?.coverage.embedded).toBe(SERVED_PAGE_COUNT);
+ expect(result?.semantic?.applied).toBe(true);
+
+ const rotation = result?.results?.find((r) => r.path === 'guides/credential-rotation');
+ expect(rotation, 'zero-overlap doc must surface via the vector candidate source').toBeDefined();
+ expect(typeof rotation?.signals.vector).toBe('number');
+ expect(rotation?.signals.vector ?? 0).toBeGreaterThan(0.3);
+
+ expect(result?.results?.find((r) => r.path === 'archive/old-secrets')).toBeUndefined();
+ expect(result?.results?.find((r) => r.path.startsWith('.cursor/'))).toBeUndefined();
+
+ expect(existsSync(join(tmpDir, '.ok', 'local', 'embeddings'))).toBe(true);
+ }, 30_000);
+
+ test('GET /api/semantic-status reports enabled + ready + capable + coverage', async () => {
+ const status = (await callViaServer(server, 'GET', '/api/semantic-status')) as {
+ enabled: boolean;
+ keyPresent: boolean;
+ keySource: string | null;
+ ready: boolean;
+ capable: boolean;
+ embedded: number;
+ total: number;
+ };
+ expect(status.enabled).toBe(true);
+ expect(status.keyPresent).toBe(true);
+ expect(status.keySource).toBe('file');
+ expect(status.ready).toBe(true);
+ expect(status.capable).toBe(true);
+ expect(status.total).toBe(SERVED_PAGE_COUNT);
+ expect(status.embedded).toBe(SERVED_PAGE_COUNT);
+ });
+
+ test('the omnibar call shape stays lexical through the same booted server', async () => {
+ const { results, semantic } = await searchViaServer(server, {
+ query: 'auth retries',
+ intent: 'full_text',
+ });
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+ expect(results?.find((r) => r.path === 'guides/credential-rotation')).toBeUndefined();
+ expect(results?.find((r) => r.path.startsWith('.cursor/'))).toBeUndefined();
+ expect(semantic).toBeUndefined();
+ });
+
+ test('semantic:false forces lexical through the same booted server', async () => {
+ const { results, semantic } = await searchViaServer(server, {
+ query: 'auth retries',
+ intent: 'full_text',
+ semantic: false,
+ });
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+ expect(results?.find((r) => r.path === 'guides/credential-rotation')).toBeUndefined();
+ expect(semantic).toBeUndefined();
+ });
+});
+
+describe('createServer boot — project-local scope enforcement (egress safety)', () => {
+ test('enabled in the COMMITTED project config is ignored — project-local only', async () => {
+ const dir = mkdtempSync(join(tmpdir(), 'ok-sem-scope-'));
+ try {
+ writeFileSync(
+ join(dir, 'note.md'),
+ '# Note\n\nThe credential rotation flow re-issues secrets when they expire.\n',
+ 'utf-8',
+ );
+ mkdirSync(join(dir, '.ok'), { recursive: true });
+ writeFileSync(
+ join(dir, '.ok', 'config.yml'),
+ 'search:\n semantic:\n enabled: true\n',
+ 'utf-8',
+ );
+
+ const shadowRepo = await initShadowRepo(dir);
+ const srv = createServer({
+ contentDir: dir,
+ projectDir: dir,
+ quiet: true,
+ debounce: 60_000,
+ gitEnabled: false,
+ shadowRepo,
+ skipStateManifestCheck: true,
+ destroyTimeoutMs: 500,
+ configHomedirOverride: dir,
+ embedderLoader: () => Promise.resolve(createConceptEmbedder({ concepts: CONCEPTS })),
+ });
+ await srv.ready;
+ try {
+ const { results, semantic } = await searchViaServer(srv, {
+ query: 'auth retries',
+ intent: 'full_text',
+ semantic: true,
+ });
+ expect(semantic).toBeUndefined();
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+
+ const status = (await callViaServer(srv, 'GET', '/api/semantic-status')) as {
+ enabled: boolean;
+ ready: boolean;
+ capable: boolean;
+ };
+ expect(status.enabled).toBe(false);
+ expect(status.ready).toBe(false);
+ expect(status.capable).toBe(false);
+ } finally {
+ await srv.destroy();
+ }
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+});
+
+describe('createServer boot — embeddings key set/clear handlers (Account control)', () => {
+ test('set-key writes the secrets file, status flips keyPresent, clear-key removes it', async () => {
+ const dir = mkdtempSync(join(tmpdir(), 'ok-sem-setkey-'));
+ try {
+ writeFileSync(join(dir, 'note.md'), '# Note\n', 'utf-8');
+ mkdirSync(join(dir, '.ok', 'local'), { recursive: true });
+ writeFileSync(
+ join(dir, '.ok', 'local', 'config.yml'),
+ 'search:\n semantic:\n enabled: true\n',
+ 'utf-8',
+ );
+ const shadowRepo = await initShadowRepo(dir);
+ const srv = createServer({
+ contentDir: dir,
+ projectDir: dir,
+ quiet: true,
+ debounce: 60_000,
+ gitEnabled: false,
+ shadowRepo,
+ skipStateManifestCheck: true,
+ destroyTimeoutMs: 500,
+ configHomedirOverride: dir,
+ embedderLoader: () => Promise.resolve(createConceptEmbedder({ concepts: CONCEPTS })),
+ });
+ await srv.ready;
+ const secretsPath = join(dir, '.ok', 'secrets.yml');
+ try {
+ const before = (await callViaServer(srv, 'GET', '/api/semantic-status')) as {
+ keyPresent: boolean;
+ };
+ expect(before.keyPresent).toBe(false);
+ expect(existsSync(secretsPath)).toBe(false);
+
+ const setRes = (await callViaServer(srv, 'POST', '/api/local-op/embeddings/set-key', {
+ key: 'sk-account-ui-key',
+ })) as { keyPresent: boolean };
+ expect(setRes.keyPresent).toBe(true);
+ expect(readFileSync(secretsPath, 'utf-8')).toContain('sk-account-ui-key');
+
+ const after = (await callViaServer(srv, 'GET', '/api/semantic-status')) as {
+ keyPresent: boolean;
+ keySource: string | null;
+ };
+ expect(after.keyPresent).toBe(true);
+ expect(after.keySource).toBe('file');
+
+ const clearRes = (await callViaServer(
+ srv,
+ 'POST',
+ '/api/local-op/embeddings/clear-key',
+ {},
+ )) as {
+ keyPresent: boolean;
+ };
+ expect(clearRes.keyPresent).toBe(false);
+ const cleared = (await callViaServer(srv, 'GET', '/api/semantic-status')) as {
+ keyPresent: boolean;
+ };
+ expect(cleared.keyPresent).toBe(false);
+ } finally {
+ await srv.destroy();
+ }
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+});
diff --git a/packages/server/src/api-search-semantic.test.ts b/packages/server/src/api-search-semantic.test.ts
new file mode 100644
index 000000000..f60fcb8cf
--- /dev/null
+++ b/packages/server/src/api-search-semantic.test.ts
@@ -0,0 +1,283 @@
+import { describe, expect, test } from 'bun:test';
+import {
+ mkdirSync,
+ mkdtempSync,
+ readdirSync,
+ readFileSync,
+ rmSync,
+ statSync,
+ writeFileSync,
+} from 'node:fs';
+import type { IncomingMessage, ServerResponse } from 'node:http';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { Readable } from 'node:stream';
+import { createWorkspaceSearchDocument } from '@inkeep/open-knowledge-core';
+import { createApiExtension } from './api-extension.ts';
+import { createConceptEmbedder, type Embedder, SemanticSearchService } from './embeddings/index.ts';
+import type { FileIndexEntry } from './file-watcher.ts';
+
+const CONCEPTS = [
+ { id: 'auth', terms: ['auth', 'credential', 'session token', 'login', 'secret', 'sign-in'] },
+ { id: 'retry', terms: ['retry', 'retries', 'refresh', 're-issue', 'rotation', 'backoff'] },
+ { id: 'bread', terms: ['bread', 'sourdough', 'ferment', 'dough'] },
+];
+
+const FILES: Record<string, string> = {
+ 'guides/credential-rotation.md':
+ '# Credential Rotation\n\nThe credential rotation flow re-issues secrets when they expire.\n',
+ 'recipes/sourdough.md': '# Sourdough\n\nA recipe for sourdough bread with a long cold ferment.\n',
+ 'auth/login.md': '# Login\n\nThe login page authenticates a user and starts a session.\n',
+};
+
+interface CapturedResponse {
+ status: number;
+ body: string;
+}
+interface SearchRow {
+ kind: string;
+ path: string;
+ signals: { lexical: number; fullText: number; recency: number; vector?: number };
+}
+interface SearchBody {
+ results?: SearchRow[];
+ semantic?: { capable: boolean; applied: boolean; coverage: { embedded: number; total: number } };
+}
+
+function makeReq(method: string, url: string, body = ''): IncomingMessage {
+ const readable = Readable.from(Buffer.from(body)) as unknown as IncomingMessage;
+ readable.method = method;
+ readable.url = url;
+ readable.headers = { host: 'localhost' };
+ return readable;
+}
+function makeRes(): { res: ServerResponse; captured: CapturedResponse } {
+ const captured: CapturedResponse = { status: 0, body: '' };
+ const res = {
+ writeHead(status: number) {
+ captured.status = status;
+ },
+ end(body?: string) {
+ captured.body = body ?? '';
+ },
+ } as unknown as ServerResponse;
+ return { res, captured };
+}
+
+function buildFileIndex(dir: string, base = ''): ReadonlyMap<string, FileIndexEntry> {
+ const index = new Map<string, FileIndexEntry>();
+ for (const entry of readdirSync(dir, { withFileTypes: true })) {
+ const rel = base ? `${base}/${entry.name}` : entry.name;
+ if (entry.isDirectory()) {
+ for (const [k, v] of buildFileIndex(join(dir, entry.name), rel)) index.set(k, v);
+ } else if (entry.isFile() && entry.name.endsWith('.md')) {
+ const stat = statSync(join(dir, entry.name));
+ index.set(rel.slice(0, -3), {
+ size: stat.size,
+ modified: stat.mtime.toISOString(),
+ canonicalPath: join(dir, entry.name),
+ inode: stat.ino,
+ aliases: [],
+ });
+ }
+ }
+ return index;
+}
+
+function seed(): string {
+ const dir = mkdtempSync(join(tmpdir(), 'ok-sem-search-'));
+ for (const [rel, content] of Object.entries(FILES)) {
+ const abs = join(dir, rel);
+ mkdirSync(join(abs, '..'), { recursive: true });
+ writeFileSync(abs, content, 'utf-8');
+ }
+ return dir;
+}
+
+/** Build a service and PRE-EMBED the page corpus (the production embed hook is
+ * fire-and-forget; we await here so the query path is deterministic). */
+async function makeService(
+ fileIndex: ReadonlyMap<string, FileIndexEntry>,
+ opts: { enabled: boolean; embedder?: Embedder | null },
+): Promise<SemanticSearchService> {
+ const embedder =
+ opts.embedder === undefined ? createConceptEmbedder({ concepts: CONCEPTS }) : opts.embedder;
+ const service = new SemanticSearchService({
+ loadEmbedder: () => Promise.resolve(embedder),
+ cacheDir: null,
+ enabled: opts.enabled,
+ });
+ const docs = [...fileIndex].map(([docName, entry]) =>
+ createWorkspaceSearchDocument({
+ kind: 'page',
+ path: docName,
+ content: readFileSync(entry.canonicalPath, 'utf-8'),
+ modifiedTs: Date.parse(entry.modified),
+ }),
+ );
+ await service.embedCorpus(docs);
+ return service;
+}
+
+async function searchPost(
+ contentDir: string,
+ fileIndex: ReadonlyMap<string, FileIndexEntry>,
+ bodyObj: Record<string, unknown>,
+ service?: SemanticSearchService,
+): Promise<SearchBody> {
+ const ext = createApiExtension({
+ hocuspocus: {} as unknown as Parameters<typeof createApiExtension>[0]['hocuspocus'],
+ sessionManager: {} as unknown as Parameters<typeof createApiExtension>[0]['sessionManager'],
+ contentDir,
+ serverInstanceId: 'test-server',
+ getFileIndex: () => fileIndex,
+ semanticSearch: service,
+ });
+ const req = makeReq('POST', '/api/search', JSON.stringify(bodyObj));
+ const { res, captured } = makeRes();
+ await (
+ ext as {
+ onRequest: (c: { request: IncomingMessage; response: ServerResponse }) => Promise<void>;
+ }
+ ).onRequest({ request: req, response: res });
+ expect(captured.status).toBe(200);
+ return JSON.parse(captured.body) as SearchBody;
+}
+
+describe('POST /api/search — semantic (opt-in)', () => {
+ test('semantic:true surfaces a zero-token-overlap doc via vector + carries signals.vector', async () => {
+ const dir = seed();
+ try {
+ const fileIndex = buildFileIndex(dir);
+ const service = await makeService(fileIndex, { enabled: true });
+ const { results, semantic } = await searchPost(
+ dir,
+ fileIndex,
+ { query: 'auth retries', intent: 'full_text', semantic: true },
+ service,
+ );
+ const rotation = results?.find((r) => r.path === 'guides/credential-rotation');
+ expect(
+ rotation,
+ 'zero-overlap doc must be retrieved via the vector candidate source',
+ ).toBeDefined();
+ expect(typeof rotation?.signals.vector).toBe('number');
+ expect(rotation?.signals.vector ?? 0).toBeGreaterThan(0.3);
+ expect(semantic?.capable).toBe(true);
+ expect(semantic?.applied).toBe(true);
+ expect(semantic?.coverage.total).toBe(3);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('the omnibar call shape (no semantic field) stays lexical and byte-identical', async () => {
+ const dir = seed();
+ try {
+ const fileIndex = buildFileIndex(dir);
+ const service = await makeService(fileIndex, { enabled: true });
+ const { results, semantic } = await searchPost(
+ dir,
+ fileIndex,
+ { query: 'auth retries', intent: 'full_text' },
+ service,
+ );
+ expect(results?.find((r) => r.path === 'guides/credential-rotation')).toBeUndefined();
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+ expect(semantic).toBeUndefined();
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('semantic:false forces lexical — the zero-overlap doc disappears, no block', async () => {
+ const dir = seed();
+ try {
+ const fileIndex = buildFileIndex(dir);
+ const service = await makeService(fileIndex, { enabled: true });
+ const { results, semantic } = await searchPost(
+ dir,
+ fileIndex,
+ { query: 'auth retries', intent: 'full_text', semantic: false },
+ service,
+ );
+ expect(results?.find((r) => r.path === 'guides/credential-rotation')).toBeUndefined();
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+ expect(semantic).toBeUndefined();
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('with no semantic service the response is pure lexical (no vector, no block)', async () => {
+ const dir = seed();
+ try {
+ const fileIndex = buildFileIndex(dir);
+ const { results, semantic } = await searchPost(dir, fileIndex, {
+ query: 'auth retries',
+ intent: 'full_text',
+ semantic: true,
+ });
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+ expect(results?.find((r) => r.path === 'guides/credential-rotation')).toBeUndefined();
+ expect(semantic).toBeUndefined();
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('a sub-min-length query stays lexical even when opted in (gated, block still present)', async () => {
+ const dir = seed();
+ try {
+ const fileIndex = buildFileIndex(dir);
+ const service = await makeService(fileIndex, { enabled: true });
+ const { results, semantic } = await searchPost(
+ dir,
+ fileIndex,
+ { query: 'au', intent: 'full_text', semantic: true },
+ service,
+ );
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+ expect(semantic?.applied).toBe(false); // opted in, but gated off this call
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('an incapable backend degrades to lexical without error (block reports not capable)', async () => {
+ const dir = seed();
+ try {
+ const fileIndex = buildFileIndex(dir);
+ const service = await makeService(fileIndex, { enabled: true, embedder: null });
+ const { results, semantic } = await searchPost(
+ dir,
+ fileIndex,
+ { query: 'auth retries', intent: 'full_text', semantic: true },
+ service,
+ );
+ expect((results ?? []).length).toBeGreaterThan(0);
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+ expect(semantic?.capable).toBe(false);
+ expect(semantic?.applied).toBe(false);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('semantic does not apply to omnibar intent even when opted in', async () => {
+ const dir = seed();
+ try {
+ const fileIndex = buildFileIndex(dir);
+ const service = await makeService(fileIndex, { enabled: true });
+ const { results } = await searchPost(
+ dir,
+ fileIndex,
+ { query: 'auth retries', intent: 'omnibar', semantic: true },
+ service,
+ );
+ for (const r of results ?? []) expect('vector' in r.signals).toBe(false);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+});
diff --git a/packages/server/src/boot.ts b/packages/server/src/boot.ts
index 7bc141d64..09bdaceda 100644
--- a/packages/server/src/boot.ts
+++ b/packages/server/src/boot.ts
@@ -119,6 +119,7 @@ export interface BootServerOptions
| 'lockKind'
| 'detectGh'
| 'tokenStore'
+ | 'embeddingsKeyStore'
| 'singleDocRelPath'
| 'ephemeral'
> {
@@ -273,6 +274,7 @@ async function bootServerInner(opts: BootServerOptions): Promise<BootedServer> {
skipStateManifestCheck: opts.skipStateManifestCheck,
detectGh: opts.detectGh,
tokenStore: opts.tokenStore,
+ embeddingsKeyStore: opts.embeddingsKeyStore,
singleDocRelPath: opts.singleDocRelPath,
ephemeral: opts.ephemeral,
});
diff --git a/packages/server/src/embeddings/chunking.test.ts b/packages/server/src/embeddings/chunking.test.ts
new file mode 100644
index 000000000..17488acb1
--- /dev/null
+++ b/packages/server/src/embeddings/chunking.test.ts
@@ -0,0 +1,55 @@
+import { describe, expect, test } from 'bun:test';
+import {
+ CHUNK_OVERLAP_CHARS,
+ CHUNK_TARGET_CHARS,
+ chunkDocument,
+ MAX_CHUNKS_PER_DOC,
+} from './chunking.ts';
+
+describe('chunkDocument', () => {
+ test('blank input yields no chunks', () => {
+ expect(chunkDocument('')).toEqual([]);
+ expect(chunkDocument(' \n\t ')).toEqual([]);
+ });
+
+ test('short doc is a single trimmed chunk', () => {
+ expect(chunkDocument(' hello world ')).toEqual(['hello world']);
+ });
+
+ test('doc at the target boundary stays one chunk', () => {
+ const text = 'a'.repeat(CHUNK_TARGET_CHARS);
+ expect(chunkDocument(text)).toEqual([text]);
+ });
+
+ test('long doc splits into overlapping chunks that cover all content', () => {
+ const words = Array.from({ length: 1200 }, (_, i) => `word${i}`);
+ const text = words.join(' ');
+ const chunks = chunkDocument(text);
+ expect(chunks.length).toBeGreaterThan(1);
+ const joined = chunks.join(' ');
+ for (const w of [words[0], words[600], words[words.length - 1]]) {
+ expect(joined.includes(w)).toBe(true);
+ }
+ const firstTail = chunks[0].slice(-CHUNK_OVERLAP_CHARS / 2);
+ expect(chunks[1].includes(firstTail.trim().split(' ')[0])).toBe(true);
+ });
+
+ test('chunks never exceed the target length', () => {
+ const text = 'lorem ipsum dolor sit amet '.repeat(2000);
+ for (const chunk of chunkDocument(text)) {
+ expect(chunk.length).toBeLessThanOrEqual(CHUNK_TARGET_CHARS);
+ }
+ });
+
+ test('respects the max-chunk cap on pathologically large input', () => {
+ const text = 'x'.repeat(CHUNK_TARGET_CHARS * (MAX_CHUNKS_PER_DOC + 50));
+ expect(chunkDocument(text).length).toBeLessThanOrEqual(MAX_CHUNKS_PER_DOC);
+ });
+
+ test('an unbroken run with no whitespace still makes forward progress', () => {
+ const text = 'a'.repeat(CHUNK_TARGET_CHARS * 3);
+ const chunks = chunkDocument(text);
+ expect(chunks.length).toBeGreaterThan(1);
+ expect(chunks.length).toBeLessThanOrEqual(MAX_CHUNKS_PER_DOC);
+ });
+});
diff --git a/packages/server/src/embeddings/chunking.ts b/packages/server/src/embeddings/chunking.ts
new file mode 100644
index 000000000..cdbad11eb
--- /dev/null
+++ b/packages/server/src/embeddings/chunking.ts
@@ -0,0 +1,38 @@
+export const CHUNK_TARGET_CHARS = 8000;
+
+export const CHUNK_OVERLAP_CHARS = 400;
+
+export const MAX_CHUNKS_PER_DOC = 80;
+
+export const CHUNK_CONFIG_ID = `c${CHUNK_TARGET_CHARS}-o${CHUNK_OVERLAP_CHARS}-m${MAX_CHUNKS_PER_DOC}`;
+
+export interface ChunkOptions {
+ targetChars?: number;
+ overlapChars?: number;
+ maxChunks?: number;
+}
+
+export function chunkDocument(text: string, options: ChunkOptions = {}): string[] {
+ const target = Math.max(1, options.targetChars ?? CHUNK_TARGET_CHARS);
+ const overlap = Math.max(0, Math.min(options.overlapChars ?? CHUNK_OVERLAP_CHARS, target - 1));
+ const maxChunks = options.maxChunks ?? MAX_CHUNKS_PER_DOC;
+
+ if (text.trim().length === 0) return [];
+ if (text.length <= target) return [text.trim()];
+
+ const chunks: string[] = [];
+ let start = 0;
+ while (start < text.length && chunks.length < maxChunks) {
+ let end = Math.min(text.length, start + target);
+ if (end < text.length) {
+ const boundary = Math.max(text.lastIndexOf(' ', end), text.lastIndexOf('\n', end));
+ if (boundary > start + Math.floor(target / 2)) end = boundary;
+ }
+ const piece = text.slice(start, end).trim();
+ if (piece) chunks.push(piece);
+ if (end >= text.length) break;
+ const next = end - overlap;
+ start = next > start ? next : end; // guarantee forward progress
+ }
+ return chunks;
+}
diff --git a/packages/server/src/embeddings/concept-embedder.test.ts b/packages/server/src/embeddings/concept-embedder.test.ts
new file mode 100644
index 000000000..56c5b37d4
--- /dev/null
+++ b/packages/server/src/embeddings/concept-embedder.test.ts
@@ -0,0 +1,41 @@
+import { describe, expect, test } from 'bun:test';
+import { createConceptEmbedder } from './concept-embedder.ts';
+import { cosineSimilarity, DEFAULT_EMBEDDINGS_DIMENSIONS } from './embedder.ts';
+
+const concepts = [
+ { id: 'auth', terms: ['auth', 'authentication', 'session token', 'credential', 'login'] },
+ { id: 'retry', terms: ['retry', 'retries', 'backoff', 're-issue', 'refresh'] },
+ { id: 'bread', terms: ['bread', 'sourdough', 'ferment', 'dough'] },
+];
+
+describe('createConceptEmbedder', () => {
+ test('produces normalized vectors of the configured dimensionality', async () => {
+ const embedder = createConceptEmbedder({ concepts });
+ const [v] = await embedder.embed(['anything at all'], { role: 'document' });
+ expect(v.length).toBe(DEFAULT_EMBEDDINGS_DIMENSIONS);
+ expect(cosineSimilarity(v, v)).toBeCloseTo(1, 5);
+ });
+
+ test('zero-token-overlap pair sharing a concept scores high cosine (the G1 case)', async () => {
+ const embedder = createConceptEmbedder({ concepts });
+ const [query] = await embedder.embed(['auth retries'], { role: 'query' });
+ const [doc] = await embedder.embed(['session token refresh re-issues credentials'], {
+ role: 'document',
+ });
+ const [unrelated] = await embedder.embed(['sourdough bread cold ferment recipe'], {
+ role: 'document',
+ });
+ const relevant = cosineSimilarity(query, doc);
+ const noise = cosineSimilarity(query, unrelated);
+ expect(relevant).toBeGreaterThan(0.5);
+ expect(relevant).toBeGreaterThan(noise + 0.3);
+ });
+
+ test('is deterministic across calls', async () => {
+ const a = createConceptEmbedder({ concepts });
+ const b = createConceptEmbedder({ concepts });
+ const [va] = await a.embed(['session token login'], { role: 'document' });
+ const [vb] = await b.embed(['session token login'], { role: 'document' });
+ expect(cosineSimilarity(va, vb)).toBeCloseTo(1, 6);
+ });
+});
diff --git a/packages/server/src/embeddings/concept-embedder.ts b/packages/server/src/embeddings/concept-embedder.ts
new file mode 100644
index 000000000..23664bb54
--- /dev/null
+++ b/packages/server/src/embeddings/concept-embedder.ts
@@ -0,0 +1,78 @@
+import {
+ DEFAULT_EMBEDDINGS_DIMENSIONS,
+ type Embedder,
+ type EmbeddingRole,
+ normalizeInPlace,
+} from './embedder.ts';
+
+interface ConceptDefinition {
+ id: string;
+ terms: string[];
+}
+
+export interface ConceptEmbedderOptions {
+ concepts?: ConceptDefinition[];
+ dims?: number;
+ modelId?: string;
+ providerId?: string;
+ baselineWeight?: number;
+}
+
+function hash32(input: string): number {
+ let h = 2166136261 >>> 0;
+ for (let i = 0; i < input.length; i++) {
+ h ^= input.charCodeAt(i);
+ h = Math.imul(h, 16777619) >>> 0;
+ }
+ return h >>> 0;
+}
+
+function basisDirection(key: string, dims: number): Float32Array {
+ const vec = new Float32Array(dims);
+ let state = hash32(key) || 1;
+ for (let i = 0; i < dims; i++) {
+ state ^= state << 13;
+ state >>>= 0;
+ state ^= state >> 17;
+ state ^= state << 5;
+ state >>>= 0;
+ vec[i] = (state / 0xffffffff) * 2 - 1;
+ }
+ return normalizeInPlace(vec);
+}
+
+export function createConceptEmbedder(options: ConceptEmbedderOptions = {}): Embedder {
+ const dims = options.dims ?? DEFAULT_EMBEDDINGS_DIMENSIONS;
+ const modelId = options.modelId ?? 'concept-test-embedder';
+ const providerId = options.providerId ?? 'concept-test';
+ const baselineWeight = options.baselineWeight ?? 0.15;
+ const concepts = (options.concepts ?? []).map((c) => ({
+ ...c,
+ direction: basisDirection(`concept:${c.id}`, dims),
+ terms: c.terms.map((t) => t.toLowerCase()),
+ }));
+
+ function embedOne(text: string): Float32Array {
+ const lower = text.toLowerCase();
+ const vec = new Float32Array(dims);
+ for (const concept of concepts) {
+ if (concept.terms.some((term) => term.length > 0 && lower.includes(term))) {
+ for (let i = 0; i < dims; i++) vec[i] += concept.direction[i];
+ }
+ }
+ for (const token of lower.split(/[^a-z0-9]+/)) {
+ if (!token) continue;
+ vec[hash32(token) % dims] += baselineWeight;
+ }
+ return normalizeInPlace(vec);
+ }
+
+ return {
+ providerId,
+ modelId,
+ dims,
+ embed(texts: readonly string[], _opts: { role: EmbeddingRole }): Promise<Float32Array[]> {
+ return Promise.resolve(texts.map((t) => embedOne(t)));
+ },
+ };
+}
diff --git a/packages/server/src/embeddings/embedder.test.ts b/packages/server/src/embeddings/embedder.test.ts
new file mode 100644
index 000000000..b50f23990
--- /dev/null
+++ b/packages/server/src/embeddings/embedder.test.ts
@@ -0,0 +1,286 @@
+import { afterEach, describe, expect, test } from 'bun:test';
+import {
+ cosineSimilarity,
+ createOpenAiEmbedder,
+ EMBEDDINGS_API_KEY_ENV,
+ EmbeddingDimsMismatchError,
+ loadOpenAiEmbedder,
+ normalizeInPlace,
+ normalizeProviderId,
+} from './embedder.ts';
+
+const KEY = 'sk-secret-test-key-do-not-log';
+
+interface FetchCall {
+ url: string;
+ body: unknown;
+ authHeader: string | undefined;
+}
+
+function stubFetch(replies: Array<{ status?: number; json?: unknown } | { abortable: true }>): {
+ fetchImpl: typeof fetch;
+ calls: FetchCall[];
+} {
+ const calls: FetchCall[] = [];
+ let i = 0;
+ const fetchImpl = ((url: string, init: RequestInit) => {
+ const headers = init.headers as Record<string, string>;
+ calls.push({ url, body: JSON.parse(init.body as string), authHeader: headers?.Authorization });
+ const reply = replies[Math.min(i, replies.length - 1)];
+ i += 1;
+ if ('abortable' in reply) {
+ return new Promise((_resolve, reject) => {
+ init.signal?.addEventListener('abort', () =>
+ reject(Object.assign(new Error('aborted'), { name: 'AbortError' })),
+ );
+ });
+ }
+ const status = reply.status ?? 200;
+ return Promise.resolve({
+ ok: status >= 200 && status < 300,
+ status,
+ json: () => Promise.resolve(reply.json),
+ text: () => Promise.resolve(JSON.stringify(reply.json ?? { error: 'boom' })),
+ } as Response);
+ }) as unknown as typeof fetch;
+ return { fetchImpl, calls };
+}
+
+function embeddingsResponse(count: number, dims: number, totalTokens = 7): unknown {
+ return {
+ data: Array.from({ length: count }, (_, index) => ({
+ index,
+ embedding: Array.from({ length: dims }, (_, j) => (j === index % dims ? 1 : 0)),
+ })),
+ usage: { total_tokens: totalTokens },
+ };
+}
+
+const noSleep = () => Promise.resolve();
+
+afterEach(() => {
+ delete process.env[EMBEDDINGS_API_KEY_ENV];
+});
+
+describe('createOpenAiEmbedder', () => {
+ test('embeds, normalizes, and preserves input order', async () => {
+ const { fetchImpl, calls } = stubFetch([{ json: embeddingsResponse(2, 1536) }]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://api.openai.com/v1', model: 'text-embedding-3-small', apiKey: KEY },
+ { fetchImpl, sleep: noSleep },
+ );
+ const out = await embedder.embed(['alpha', 'beta'], { role: 'document' });
+ expect(out.length).toBe(2);
+ expect(out[0].length).toBe(1536);
+ expect(cosineSimilarity(out[0], out[0])).toBeCloseTo(1, 5);
+ expect(calls).toHaveLength(1);
+ expect(calls[0].url).toBe('https://api.openai.com/v1/embeddings');
+ });
+
+ test('sends the key ONLY in the Authorization header, never in the body', async () => {
+ const { fetchImpl, calls } = stubFetch([{ json: embeddingsResponse(1, 1536) }]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://api.openai.com/v1', model: 'text-embedding-3-small', apiKey: KEY },
+ { fetchImpl, sleep: noSleep },
+ );
+ await embedder.embed(['hello'], { role: 'query' });
+ expect(calls[0].authHeader).toBe(`Bearer ${KEY}`);
+ expect(JSON.stringify(calls[0].body)).not.toContain(KEY);
+ });
+
+ test('splits large input into multiple provider requests (batching)', async () => {
+ const { fetchImpl, calls } = stubFetch([
+ { json: embeddingsResponse(2, 8) },
+ { json: embeddingsResponse(1, 8) },
+ ]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', dimensions: 8, apiKey: KEY },
+ { fetchImpl, sleep: noSleep, maxBatchSize: 2 },
+ );
+ const out = await embedder.embed(['a', 'b', 'c'], { role: 'document' });
+ expect(out.length).toBe(3);
+ expect(calls).toHaveLength(2);
+ expect((calls[0].body as { input: string[] }).input).toEqual(['a', 'b']);
+ expect((calls[1].body as { input: string[] }).input).toEqual(['c']);
+ });
+
+ test('sends the dimensions param only when configured', async () => {
+ const withDims = stubFetch([{ json: embeddingsResponse(1, 512) }]);
+ await createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', dimensions: 512, apiKey: KEY },
+ { fetchImpl: withDims.fetchImpl, sleep: noSleep },
+ ).embed(['q'], { role: 'query' });
+ expect((withDims.calls[0].body as { dimensions?: number }).dimensions).toBe(512);
+
+ const noDims = stubFetch([{ json: embeddingsResponse(1, 1536) }]);
+ await createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', apiKey: KEY },
+ { fetchImpl: noDims.fetchImpl, sleep: noSleep },
+ ).embed(['q'], { role: 'query' });
+ expect('dimensions' in (noDims.calls[0].body as object)).toBe(false);
+ });
+
+ test('retries on 429 then succeeds (bounded backoff)', async () => {
+ const { fetchImpl, calls } = stubFetch([
+ { status: 429 },
+ { status: 503 },
+ { json: embeddingsResponse(1, 1536) },
+ ]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', apiKey: KEY },
+ { fetchImpl, sleep: noSleep, maxRetries: 4 },
+ );
+ const out = await embedder.embed(['q'], { role: 'document' });
+ expect(out.length).toBe(1);
+ expect(calls).toHaveLength(3);
+ });
+
+ test('a non-retryable 401 throws immediately and leaks no key', async () => {
+ const { fetchImpl, calls } = stubFetch([{ status: 401, json: { error: 'bad key' } }]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', apiKey: KEY },
+ { fetchImpl, sleep: noSleep, maxRetries: 4 },
+ );
+ let caught: Error | null = null;
+ await embedder.embed(['q'], { role: 'document' }).catch((e) => {
+ caught = e as Error;
+ });
+ expect(caught).not.toBeNull();
+ expect((caught as unknown as Error).message).toContain('401');
+ expect((caught as unknown as Error).message).not.toContain(KEY);
+ expect(calls).toHaveLength(1); // no retry on 4xx
+ });
+
+ test('a provider error body echoing the key never leaks into the thrown error (R4)', async () => {
+ const { fetchImpl } = stubFetch([
+ { status: 500, json: { error: `bad auth for Bearer ${KEY}` } },
+ ]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', apiKey: KEY },
+ { fetchImpl, sleep: noSleep, maxRetries: 1 },
+ );
+ let caught: Error | null = null;
+ await embedder.embed(['q'], { role: 'document' }).catch((e) => {
+ caught = e as Error;
+ });
+ expect(caught).not.toBeNull();
+ const err = caught as unknown as Error;
+ expect(`${err.message}\n${err.stack ?? ''}`).not.toContain(KEY);
+ expect(err.message).toContain('500');
+ });
+
+ test('times out a hung request and surfaces an error after retries', async () => {
+ const { fetchImpl } = stubFetch([{ abortable: true }]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', apiKey: KEY },
+ { fetchImpl, sleep: noSleep, maxRetries: 1, queryTimeoutMs: 5, docTimeoutMs: 5 },
+ );
+ let caught: Error | null = null;
+ await embedder.embed(['q'], { role: 'query' }).catch((e) => {
+ caught = e as Error;
+ });
+ expect(caught).not.toBeNull();
+ });
+
+ test('throws EmbeddingDimsMismatchError when the provider returns the wrong size', async () => {
+ const { fetchImpl } = stubFetch([{ json: embeddingsResponse(1, 768) }]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', apiKey: KEY }, // declares default 1536
+ { fetchImpl, sleep: noSleep },
+ );
+ let caught: unknown = null;
+ await embedder.embed(['q'], { role: 'document' }).catch((e) => {
+ caught = e;
+ });
+ expect(caught).toBeInstanceOf(EmbeddingDimsMismatchError);
+ });
+
+ test('empty input does not hit the network', async () => {
+ const { fetchImpl, calls } = stubFetch([{ json: embeddingsResponse(0, 1536) }]);
+ const embedder = createOpenAiEmbedder(
+ { baseUrl: 'https://x/v1', model: 'm', apiKey: KEY },
+ { fetchImpl },
+ );
+ expect(await embedder.embed([], { role: 'document' })).toEqual([]);
+ expect(calls).toHaveLength(0);
+ });
+});
+
+describe('normalizeProviderId', () => {
+ test('normalizes host case + trailing slash, keeps path', () => {
+ expect(normalizeProviderId('https://API.OpenAI.com/v1/')).toBe('https://api.openai.com/v1');
+ expect(normalizeProviderId('https://api.openai.com/v1')).toBe('https://api.openai.com/v1');
+ });
+ test('distinguishes different providers', () => {
+ expect(normalizeProviderId('https://api.openai.com/v1')).not.toBe(
+ normalizeProviderId('https://my.azure.com/v1'),
+ );
+ });
+});
+
+describe('loadOpenAiEmbedder', () => {
+ const config = { baseUrl: 'https://api.openai.com/v1', model: 'text-embedding-3-small' };
+
+ test('returns null with no key store and no env var', async () => {
+ const embedder = await loadOpenAiEmbedder({ keyStore: null, config });
+ expect(embedder).toBeNull();
+ });
+
+ test('uses the key store when present', async () => {
+ const keyStore = { get: () => Promise.resolve('sk-from-keyring') };
+ const embedder = await loadOpenAiEmbedder({ keyStore, config });
+ expect(embedder).not.toBeNull();
+ expect(embedder?.modelId).toBe('text-embedding-3-small');
+ expect(embedder?.dims).toBe(1536);
+ expect(embedder?.providerId).toBe('https://api.openai.com/v1');
+ });
+
+ test('falls back to the env var when the store has no key', async () => {
+ process.env[EMBEDDINGS_API_KEY_ENV] = 'sk-from-env';
+ const keyStore = { get: () => Promise.resolve(null) };
+ const embedder = await loadOpenAiEmbedder({ keyStore, config });
+ expect(embedder).not.toBeNull();
+ });
+
+ test('a throwing key store degrades to the env fallback (never throws)', async () => {
+ const keyStore = { get: () => Promise.reject(new Error('keyring exploded')) };
+ expect(await loadOpenAiEmbedder({ keyStore, config })).toBeNull();
+ process.env[EMBEDDINGS_API_KEY_ENV] = 'sk-env';
+ expect(await loadOpenAiEmbedder({ keyStore, config })).not.toBeNull();
+ });
+});
+
+describe('normalizeInPlace', () => {
+ test('produces a unit vector; leaves a zero vector untouched', () => {
+ const v = normalizeInPlace(Float32Array.from([3, 4]));
+ expect(cosineSimilarity(v, v)).toBeCloseTo(1, 6);
+ expect(Array.from(normalizeInPlace(Float32Array.from([0, 0])))).toEqual([0, 0]);
+ });
+});
+
+describe('createOpenAiEmbedder — plaintext key guard', () => {
+ const base = { model: 'text-embedding-3-small', apiKey: KEY };
+ test('an https endpoint constructs', () => {
+ expect(() =>
+ createOpenAiEmbedder({ ...base, baseUrl: 'https://api.openai.com/v1' }),
+ ).not.toThrow();
+ });
+ test('http:// is allowed only for loopback (local dev gateway / CI)', () => {
+ expect(() =>
+ createOpenAiEmbedder({ ...base, baseUrl: 'http://localhost:11434/v1' }),
+ ).not.toThrow();
+ expect(() =>
+ createOpenAiEmbedder({ ...base, baseUrl: 'http://127.0.0.1:8080/v1' }),
+ ).not.toThrow();
+ });
+ test('plaintext http:// to a non-loopback host throws — the key is never sent', () => {
+ expect(() => createOpenAiEmbedder({ ...base, baseUrl: 'http://evil.example/v1' })).toThrow(
+ /non-HTTPS/,
+ );
+ });
+ test('a malformed baseUrl throws at construction', () => {
+ expect(() => createOpenAiEmbedder({ ...base, baseUrl: 'not a url' })).toThrow(
+ /not a valid URL/,
+ );
+ });
+});
diff --git a/packages/server/src/embeddings/embedder.ts b/packages/server/src/embeddings/embedder.ts
new file mode 100644
index 000000000..0e7afad5e
--- /dev/null
+++ b/packages/server/src/embeddings/embedder.ts
@@ -0,0 +1,287 @@
+import {
+ type EmbeddingErrorReason,
+ recordEmbeddingProviderError,
+ recordEmbeddingRequestDuration,
+ recordEmbeddingTokens,
+} from './embeddings-telemetry.ts';
+
+export const DEFAULT_EMBEDDINGS_DIMENSIONS = 1536;
+
+export const EMBEDDINGS_API_KEY_ENV = 'OK_EMBEDDINGS_API_KEY';
+
+export type EmbeddingRole = 'query' | 'document';
+
+export interface Embedder {
+ readonly providerId: string;
+ readonly modelId: string;
+ readonly dims: number;
+ embed(texts: readonly string[], opts: { role: EmbeddingRole }): Promise<Float32Array[]>;
+}
+
+export interface EmbeddingsKeyStore {
+ get(): Promise<string | null>;
+}
+
+export class EmbeddingDimsMismatchError extends Error {
+ readonly name = 'EmbeddingDimsMismatchError';
+ constructor(
+ readonly expected: number,
+ readonly got: number,
+ ) {
+ super(
+ `embeddings provider returned ${got}-dim vectors, expected ${expected}. ` +
+ `Set search.semantic.dimensions to ${got} (or point at the right model).`,
+ );
+ }
+}
+
+class MalformedEmbeddingResponseError extends Error {
+ readonly name = 'MalformedEmbeddingResponseError';
+ constructor(expected: number, got: number) {
+ super(`embeddings response had ${got} vectors, expected ${expected}`);
+ }
+}
+
+export function cosineSimilarity(a: Float32Array, b: Float32Array): number {
+ let sum = 0;
+ const n = Math.min(a.length, b.length);
+ for (let i = 0; i < n; i++) sum += a[i] * b[i];
+ return sum;
+}
+
+export function normalizeInPlace(vec: Float32Array): Float32Array {
+ let norm = 0;
+ for (let i = 0; i < vec.length; i++) norm += vec[i] * vec[i];
+ norm = Math.sqrt(norm);
+ if (norm > 0) {
+ for (let i = 0; i < vec.length; i++) vec[i] /= norm;
+ }
+ return vec;
+}
+
+export interface OpenAiEmbedderConfig {
+ baseUrl: string;
+ model: string;
+ dimensions?: number;
+ apiKey: string;
+}
+
+export interface OpenAiEmbedderOptions {
+ fetchImpl?: typeof fetch;
+ maxBatchSize?: number;
+ maxBatchChars?: number;
+ docTimeoutMs?: number;
+ queryTimeoutMs?: number;
+ maxRetries?: number;
+ backoffBaseMs?: number;
+ sleep?: (ms: number) => Promise<void>;
+}
+
+const DEFAULTS = {
+ maxBatchSize: 96,
+ maxBatchChars: 96_000,
+ docTimeoutMs: 30_000,
+ queryTimeoutMs: 8_000,
+ maxRetries: 4,
+ backoffBaseMs: 500,
+} as const;
+
+interface OpenAiEmbeddingResponse {
+ data?: Array<{ index?: number; embedding?: number[] }>;
+ usage?: { total_tokens?: number; prompt_tokens?: number };
+}
+
+export function normalizeProviderId(baseUrl: string): string {
+ try {
+ const u = new URL(baseUrl);
+ const path = u.pathname.replace(/\/+$/, '');
+ return `${u.protocol}//${u.host.toLowerCase()}${path}`;
+ } catch {
+ return baseUrl.trim().replace(/\/+$/, '');
+ }
+}
+
+function assertSafeEmbeddingsBaseUrl(baseUrl: string): void {
+ let url: URL;
+ try {
+ url = new URL(baseUrl);
+ } catch {
+ throw new Error(`embeddings baseUrl is not a valid URL: ${baseUrl}`);
+ }
+ if (url.protocol === 'https:') return;
+ const host = url.hostname.toLowerCase();
+ const isLoopback =
+ host === 'localhost' || host === '127.0.0.1' || host === '::1' || host === '[::1]';
+ if (url.protocol === 'http:' && isLoopback) return;
+ throw new Error(
+ `refusing to send the embeddings API key to a non-HTTPS endpoint (${url.protocol}//${url.host}); ` +
+ 'use https:// (http:// is allowed only for localhost)',
+ );
+}
+
+const RETRYABLE_STATUS = new Set([408, 409, 429, 500, 502, 503, 504]);
+
+function defaultSleep(ms: number): Promise<void> {
+ return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+export function createOpenAiEmbedder(
+ config: OpenAiEmbedderConfig,
+ options: OpenAiEmbedderOptions = {},
+): Embedder {
+ const fetchImpl = options.fetchImpl ?? fetch;
+ const sleep = options.sleep ?? defaultSleep;
+ const maxBatchSize = options.maxBatchSize ?? DEFAULTS.maxBatchSize;
+ const maxBatchChars = options.maxBatchChars ?? DEFAULTS.maxBatchChars;
+ const maxRetries = options.maxRetries ?? DEFAULTS.maxRetries;
+ const backoffBaseMs = options.backoffBaseMs ?? DEFAULTS.backoffBaseMs;
+ const docTimeoutMs = options.docTimeoutMs ?? DEFAULTS.docTimeoutMs;
+ const queryTimeoutMs = options.queryTimeoutMs ?? DEFAULTS.queryTimeoutMs;
+
+ assertSafeEmbeddingsBaseUrl(config.baseUrl);
+ const dims = config.dimensions ?? DEFAULT_EMBEDDINGS_DIMENSIONS;
+ const endpoint = `${config.baseUrl.replace(/\/+$/, '')}/embeddings`;
+
+ function batchInputs(texts: readonly string[]): string[][] {
+ const batches: string[][] = [];
+ let current: string[] = [];
+ let chars = 0;
+ for (const t of texts) {
+ if (
+ current.length > 0 &&
+ (current.length >= maxBatchSize || chars + t.length > maxBatchChars)
+ ) {
+ batches.push(current);
+ current = [];
+ chars = 0;
+ }
+ current.push(t);
+ chars += t.length;
+ }
+ if (current.length > 0) batches.push(current);
+ return batches;
+ }
+
+ type AttemptResult =
+ | { kind: 'ok'; vectors: Float32Array[] }
+ | { kind: 'retry'; reason: EmbeddingErrorReason; error: Error }
+ | { kind: 'fatal'; reason: EmbeddingErrorReason; error: Error };
+
+ async function attemptOnce(
+ body: string,
+ expectedCount: number,
+ roleLabel: 'query' | 'document',
+ timeoutMs: number,
+ ): Promise<AttemptResult> {
+ const controller = new AbortController();
+ const timer = setTimeout(() => controller.abort(), timeoutMs);
+ const startedAt = performance.now();
+ try {
+ const res = await fetchImpl(endpoint, {
+ method: 'POST',
+ headers: {
+ 'Content-Type': 'application/json',
+ Authorization: `Bearer ${config.apiKey}`,
+ },
+ body,
+ signal: controller.signal,
+ });
+ recordEmbeddingRequestDuration(roleLabel, performance.now() - startedAt);
+
+ if (!res.ok) {
+ await res.text().catch(() => '');
+ const reason: EmbeddingErrorReason = res.status === 429 ? 'rate_limit' : 'http_error';
+ const error = new Error(`embeddings request failed: HTTP ${res.status}`);
+ return RETRYABLE_STATUS.has(res.status)
+ ? { kind: 'retry', reason, error }
+ : { kind: 'fatal', reason, error };
+ }
+ const json = (await res.json()) as OpenAiEmbeddingResponse;
+ const vectors = parseEmbeddingResponse(json, expectedCount, dims);
+ recordEmbeddingTokens(roleLabel, json.usage?.total_tokens ?? 0);
+ return { kind: 'ok', vectors };
+ } catch (err) {
+ if (err instanceof EmbeddingDimsMismatchError) {
+ return { kind: 'fatal', reason: 'dims_mismatch', error: err };
+ }
+ if (err instanceof MalformedEmbeddingResponseError) {
+ return { kind: 'fatal', reason: 'malformed_response', error: err };
+ }
+ const isAbort = err instanceof Error && err.name === 'AbortError';
+ const error = err instanceof Error ? err : new Error(String(err));
+ return { kind: 'retry', reason: isAbort ? 'timeout' : 'network', error };
+ } finally {
+ clearTimeout(timer);
+ }
+ }
+
+ async function embedOneBatch(batch: string[], role: EmbeddingRole): Promise<Float32Array[]> {
+ const timeoutMs = role === 'query' ? queryTimeoutMs : docTimeoutMs;
+ const roleLabel = role === 'query' ? 'query' : 'document';
+ const body = JSON.stringify({
+ model: config.model,
+ input: batch,
+ encoding_format: 'float',
+ ...(config.dimensions !== undefined ? { dimensions: config.dimensions } : {}),
+ });
+
+ let attempt = 0;
+ for (;;) {
+ const result = await attemptOnce(body, batch.length, roleLabel, timeoutMs);
+ if (result.kind === 'ok') return result.vectors;
+ recordEmbeddingProviderError(result.reason);
+ if (result.kind === 'fatal' || attempt >= maxRetries) throw result.error;
+ attempt += 1;
+ const ceiling = backoffBaseMs * 2 ** (attempt - 1);
+ await sleep(Math.round(ceiling / 2 + Math.random() * (ceiling / 2)));
+ }
+ }
+
+ function parseEmbeddingResponse(
+ json: OpenAiEmbeddingResponse,
+ expectedCount: number,
+ expectedDims: number,
+ ): Float32Array[] {
+ const data = json.data;
+ if (!Array.isArray(data) || data.length !== expectedCount) {
+ throw new MalformedEmbeddingResponseError(expectedCount, data?.length ?? 0);
+ }
+ const ordered = [...data].sort((a, b) => (a.index ?? 0) - (b.index ?? 0));
+ const out: Float32Array[] = [];
+ for (const item of ordered) {
+ const emb = item.embedding;
+ if (!Array.isArray(emb)) throw new MalformedEmbeddingResponseError(expectedCount, 0);
+ if (emb.length !== expectedDims)
+ throw new EmbeddingDimsMismatchError(expectedDims, emb.length);
+ out.push(normalizeInPlace(Float32Array.from(emb)));
+ }
+ return out;
+ }
+
+ return {
+ providerId: normalizeProviderId(config.baseUrl),
+ modelId: config.model,
+ dims,
+ async embed(texts, { role }) {
+ if (texts.length === 0) return [];
+ const out: Float32Array[] = [];
+ for (const batch of batchInputs(texts)) {
+ out.push(...(await embedOneBatch(batch, role)));
+ }
+ return out;
+ },
+ };
+}
+
+export interface LoadOpenAiEmbedderInput {
+ keyStore: EmbeddingsKeyStore | null;
+ config: Pick<OpenAiEmbedderConfig, 'baseUrl' | 'model' | 'dimensions'>;
+ options?: OpenAiEmbedderOptions;
+}
+
+export async function loadOpenAiEmbedder(input: LoadOpenAiEmbedderInput): Promise<Embedder | null> {
+ const stored = input.keyStore ? await input.keyStore.get().catch(() => null) : null;
+ const apiKey = stored ?? process.env[EMBEDDINGS_API_KEY_ENV] ?? null;
+ if (!apiKey) return null;
+ return createOpenAiEmbedder({ ...input.config, apiKey }, input.options);
+}
diff --git a/packages/server/src/embeddings/embeddings-telemetry.test.ts b/packages/server/src/embeddings/embeddings-telemetry.test.ts
new file mode 100644
index 000000000..ca91f3946
--- /dev/null
+++ b/packages/server/src/embeddings/embeddings-telemetry.test.ts
@@ -0,0 +1,37 @@
+import { describe, expect, test } from 'bun:test';
+import {
+ __resetEmbeddingsTelemetryForTesting,
+ recordEmbeddingProviderError,
+ recordEmbeddingRequestDuration,
+ recordEmbeddingTokens,
+ recordSemanticQuery,
+} from './embeddings-telemetry.ts';
+
+describe('embeddings telemetry', () => {
+ test('all record paths are no-throw under the default provider', () => {
+ expect(() => {
+ recordEmbeddingTokens('query', 12);
+ recordEmbeddingTokens('document', 0); // zero tokens → skipped, still no throw
+ recordEmbeddingProviderError('rate_limit');
+ recordEmbeddingProviderError('dims_mismatch');
+ recordEmbeddingRequestDuration('document', 123.4);
+ recordSemanticQuery({
+ outcome: 'applied',
+ capable: true,
+ embedded: 5,
+ total: 40,
+ queryEmbedMs: 87.2,
+ vectorContributors: 3,
+ });
+ recordSemanticQuery({
+ outcome: 'incapable',
+ capable: false,
+ embedded: 0,
+ total: 0,
+ queryEmbedMs: null,
+ vectorContributors: 0,
+ });
+ __resetEmbeddingsTelemetryForTesting();
+ }).not.toThrow();
+ });
+});
diff --git a/packages/server/src/embeddings/embeddings-telemetry.ts b/packages/server/src/embeddings/embeddings-telemetry.ts
new file mode 100644
index 000000000..8b40281a1
--- /dev/null
+++ b/packages/server/src/embeddings/embeddings-telemetry.ts
@@ -0,0 +1,122 @@
+import type { Counter, Histogram } from '@opentelemetry/api';
+import { getLogger } from '../logger.ts';
+import { getMeter } from '../telemetry.ts';
+
+const log = getLogger('embeddings');
+
+export type EmbeddingRoleLabel = 'query' | 'document';
+
+export type EmbeddingErrorReason =
+ | 'rate_limit'
+ | 'timeout'
+ | 'http_error'
+ | 'network'
+ | 'dims_mismatch'
+ | 'malformed_response';
+
+export type SemanticQueryOutcome =
+ | 'applied' // a vector signal contributed to ≥1 result
+ | 'no_match' // capable + warm, but no doc cleared the floor
+ | 'warming' // capable but coverage still filling in (no cached vectors yet)
+ | 'incapable' // flag on but no key / load failed → lexical
+ | 'provider_error'; // query embed failed → lexical fallback
+
+let _tokens: Counter | null = null;
+let _errors: Counter | null = null;
+let _requestDuration: Histogram | null = null;
+let _queryTotal: Counter | null = null;
+let _queryEmbedDuration: Histogram | null = null;
+
+function tokensCounter(): Counter {
+ if (!_tokens) {
+ _tokens = getMeter().createCounter('ok.embeddings.tokens_total', {
+ description:
+ 'Embeddings tokens billed, by role. Bounded label: role ∈ {query, document}. Makes spend legible; never includes content.',
+ });
+ }
+ return _tokens;
+}
+
+function errorsCounter(): Counter {
+ if (!_errors) {
+ _errors = getMeter().createCounter('ok.embeddings.provider_errors_total', {
+ description:
+ 'Embeddings provider failures, by reason. Bounded label: reason ∈ {rate_limit, timeout, http_error, network, dims_mismatch, malformed_response}.',
+ });
+ }
+ return _errors;
+}
+
+function requestDurationHist(): Histogram {
+ if (!_requestDuration) {
+ _requestDuration = getMeter().createHistogram('ok.embeddings.request_duration_ms', {
+ description: 'Wall-clock duration of one embeddings API request. Bounded label: role.',
+ unit: 'ms',
+ });
+ }
+ return _requestDuration;
+}
+
+function queryTotalCounter(): Counter {
+ if (!_queryTotal) {
+ _queryTotal = getMeter().createCounter('ok.search.semantic_query_total', {
+ description:
+ 'Semantic-requested searches, by outcome. Bounded label: outcome ∈ {applied, no_match, warming, incapable, provider_error}.',
+ });
+ }
+ return _queryTotal;
+}
+
+function queryEmbedDurationHist(): Histogram {
+ if (!_queryEmbedDuration) {
+ _queryEmbedDuration = getMeter().createHistogram('ok.search.semantic_query_embed_ms', {
+ description: 'Wall-clock latency of the per-query embed on the semantic search path.',
+ unit: 'ms',
+ });
+ }
+ return _queryEmbedDuration;
+}
+
+export function recordEmbeddingTokens(role: EmbeddingRoleLabel, tokens: number): void {
+ if (tokens > 0) tokensCounter().add(tokens, { role });
+}
+
+export function recordEmbeddingProviderError(reason: EmbeddingErrorReason): void {
+ errorsCounter().add(1, { reason });
+}
+
+export function recordEmbeddingRequestDuration(role: EmbeddingRoleLabel, ms: number): void {
+ requestDurationHist().record(Math.max(0, ms), { role });
+}
+
+export function recordSemanticQuery(event: {
+ outcome: SemanticQueryOutcome;
+ capable: boolean;
+ embedded: number;
+ total: number;
+ queryEmbedMs: number | null;
+ vectorContributors: number;
+}): void {
+ queryTotalCounter().add(1, { outcome: event.outcome });
+ if (event.queryEmbedMs !== null) queryEmbedDurationHist().record(Math.max(0, event.queryEmbedMs));
+ log.debug(
+ {
+ outcome: event.outcome,
+ capable: event.capable,
+ embedded: event.embedded,
+ total: event.total,
+ coverage: event.total > 0 ? event.embedded / event.total : 0,
+ queryEmbedMs: event.queryEmbedMs,
+ vectorContributors: event.vectorContributors,
+ },
+ '[search] semantic query',
+ );
+}
+
+export function __resetEmbeddingsTelemetryForTesting(): void {
+ _tokens = null;
+ _errors = null;
+ _requestDuration = null;
+ _queryTotal = null;
+ _queryEmbedDuration = null;
+}
diff --git a/packages/server/src/embeddings/eval/eval-set.json b/packages/server/src/embeddings/eval/eval-set.json
new file mode 100644
index 000000000..a67199897
--- /dev/null
+++ b/packages/server/src/embeddings/eval/eval-set.json
@@ -0,0 +1,285 @@
+{
+ "note": "Labeled query→doc retrieval-quality set for FR2 (spec Q13). Split into 'tune' (calibrate RRF k + bge query prefix) and 'held' (the pre-registered acceptance gate — measured once). Categories: 'zero-overlap' (target shares ~no tokens with the query while a distractor does — only vector can retrieve it; tests D20/C2b), 'paraphrase' (low lexical overlap), 'long-doc' (target is long, matching passage buried — tests C3 length bias), 'lexical-strong' (lexical already wins — the regression budget guards these). Authored solo, so blinding is partial (see eval-methodology.md); the mechanically-enforceable parts (zero-overlap pairs, long-doc pairs, tune/held split, pre-registered bar) are done rigorously.",
+ "corpus": [
+ {
+ "path": "auth/session-token-refresh",
+ "title": "Session Token Refresh",
+ "content": "When an access token expires, the client silently obtains a new one by presenting its long-lived refresh token to the refresh endpoint, which mints a fresh short-lived token. The old token is revoked. This keeps a signed-in user from being interrupted while bounding the blast radius of a leaked token."
+ },
+ {
+ "path": "auth/oauth-device-flow",
+ "title": "OAuth Device Flow",
+ "content": "The device authorization grant lets input-constrained devices like TVs obtain user consent. The device shows a short code; the user visits a verification page on a second device and approves. The device then polls the token endpoint until consent is granted."
+ },
+ {
+ "path": "auth/login-rate-limiting",
+ "title": "Login Rate Limiting",
+ "content": "Repeated failed sign-in attempts from one source are throttled with exponential backoff and temporary lockout to slow credential-stuffing and brute-force password guessing without locking out legitimate users for long."
+ },
+ {
+ "path": "auth/password-reset",
+ "title": "Password Reset",
+ "content": "A user who forgot their password requests a reset link sent to their verified email. The link carries a single-use, time-limited token; following it lets them choose a new password and invalidates old sessions."
+ },
+ {
+ "path": "search/bm25-ranking",
+ "title": "BM25 Ranking",
+ "content": "BM25 scores how well a document matches query keywords using term frequency with saturation and inverse document frequency, normalized by document length. It is the workhorse of keyword search but matches words, not meaning."
+ },
+ {
+ "path": "search/vector-embeddings",
+ "title": "Vector Embeddings",
+ "content": "A sentence-transformer maps text to a dense vector so that passages with similar meaning land near each other in space. Retrieval by cosine distance then surfaces conceptually related content even when it shares no words with the query."
+ },
+ {
+ "path": "search/reciprocal-rank-fusion",
+ "title": "Reciprocal Rank Fusion",
+ "content": "To merge two ranked lists whose scores live on incompatible scales, sum one over a constant plus rank across the lists. Because only rank position matters, there is no need to normalize an unbounded keyword score against a bounded similarity."
+ },
+ {
+ "path": "search/query-autocomplete",
+ "title": "Query Autocomplete",
+ "content": "As the user types in the search box, a prefix index suggests likely completions ordered by popularity and recency so they can jump to a known destination without finishing the word."
+ },
+ {
+ "path": "ops/blue-green-deploys",
+ "title": "Blue-Green Deploys",
+ "content": "Two identical production environments run side by side. New code goes live in the idle one; once it passes smoke checks the router cuts all traffic over instantly. If something breaks, flip the router back to roll the release back in seconds."
+ },
+ {
+ "path": "ops/database-migrations",
+ "title": "Database Migrations",
+ "content": "Schema changes ship as ordered, versioned scripts applied forward at deploy time. Expand-then-contract steps keep each change backward compatible so the application and the schema can roll out independently with no downtime."
+ },
+ {
+ "path": "ops/incident-runbook",
+ "title": "Incident Runbook",
+ "content": "When an alert fires, page the on-call engineer, open a dedicated channel, declare a severity, and work the triage checklist: confirm impact, mitigate first, find the cause second, and write a blameless postmortem after the service recovers."
+ },
+ {
+ "path": "ops/observability",
+ "title": "Observability",
+ "content": "Traces, metrics, and structured logs together let an operator ask new questions of a running system without shipping new code. Spans stitch a request across services; histograms surface tail latency; logs carry the detail."
+ },
+ {
+ "path": "editor/conflict-resolution",
+ "title": "Conflict Resolution",
+ "content": "Two people editing the same paragraph at once never clobber each other because every keystroke is a commutative operation that merges deterministically. The data structure converges to the same state on every replica regardless of order."
+ },
+ {
+ "path": "editor/markdown-pipeline",
+ "title": "Markdown Pipeline",
+ "content": "A document is parsed into an abstract syntax tree, transformed through a chain of handlers that normalize and enrich nodes, and serialized back to canonical markdown so the round trip is stable and lossless."
+ },
+ {
+ "path": "editor/backlinks",
+ "title": "Backlinks",
+ "content": "Every wiki-style link from one note to another is indexed in both directions, so opening a page shows which other pages point at it. Dense backlinking is how the knowledge base stays navigable as it grows."
+ },
+ {
+ "path": "algorithms/token-bucket",
+ "title": "Token Bucket Algorithm",
+ "content": "A token bucket refills at a fixed rate up to a cap. Each request removes a token; an empty bucket means the caller is rate limited. It smooths bursts while bounding the long-run rate."
+ },
+ {
+ "path": "ui/refresh-button",
+ "title": "Refresh Button",
+ "content": "Pressing refresh re-fetches the current view from the server and repaints it, discarding any stale cached copy the browser was showing."
+ },
+ {
+ "path": "design/vector-graphics",
+ "title": "Vector Graphics",
+ "content": "Vector graphics describe images as scalable shapes — paths, curves, and fills — instead of a grid of pixels, so they stay crisp at any size and export to SVG."
+ },
+ {
+ "path": "data/cold-storage",
+ "title": "Cold Storage",
+ "content": "Infrequently accessed data is moved to cheap, slow archival storage with long retrieval times, trading latency for cost on records that are kept for compliance but rarely read."
+ },
+ {
+ "path": "guides/onboarding",
+ "title": "Team Onboarding",
+ "content": "A new engineer's first week: get accounts provisioned, clone the repo, run the app locally, ship a tiny change behind a flag, and pair with a buddy who answers questions and reviews the first pull request."
+ },
+ {
+ "path": "overview/platform-architecture",
+ "title": "Platform Architecture Overview",
+ "content": "This document surveys the whole platform end to end. The edge layer terminates TLS and routes requests to stateless application servers, which talk to a primary datastore and a read-replica fleet. Background workers drain a durable queue for asynchronous jobs like sending email and rebuilding derived indexes. The collaboration core keeps every document as a conflict-free replicated data type so concurrent editors converge without a central lock; a file watcher mirrors changes to disk and a shadow repository tracks per-writer work in progress. Search is served by an in-memory keyword index, and — when enabled — an on-device embedding model adds a meaning-based signal so a query about a concept can find a passage that never uses the query's exact words. Observability spans, metrics, and logs flow to a local sink for diagnosis. Releases roll out blue-green: a fresh environment goes live behind the router and the previous one stays warm for instant rollback. Secrets live in the OS keychain, never on disk. The whole system is designed local-first: a single user's machine runs the full stack with no cloud dependency, and nothing about a private note ever leaves the device."
+ },
+ {
+ "path": "overview/engineering-handbook",
+ "title": "Engineering Handbook",
+ "content": "Our engineering culture in one page. We write small, reversible changes and ship them behind flags so a risky idea can be turned off without a deploy. Code review is mandatory and kind: reviewers explain the why, authors assume good faith. Tests describe behavior, not implementation, so a refactor that preserves behavior keeps them green. We measure before optimizing and keep a calibrated performance budget. On call rotates weekly; when an incident happens we mitigate before we investigate and we never blame a person for a system that let a mistake through. To undo a bad release we flip production traffic back to the previous environment, which restores service in seconds while we diagnose. Documentation lives next to the code and is treated as part of the change, not an afterthought. Above all we optimize for the long-run health of the codebase over the short-run convenience of any single change."
+ }
+ ],
+ "pairs": [
+ {
+ "query": "avoid re-typing my login every hour as my pass quietly renews",
+ "target": "auth/session-token-refresh",
+ "category": "zero-overlap",
+ "split": "tune"
+ },
+ {
+ "query": "approve an app on my smart TV by typing a code on my phone",
+ "target": "auth/oauth-device-flow",
+ "category": "zero-overlap",
+ "split": "held"
+ },
+ {
+ "query": "slow down an attacker guessing passwords over and over",
+ "target": "auth/login-rate-limiting",
+ "category": "paraphrase",
+ "split": "tune"
+ },
+ {
+ "query": "I forgot how to get into my account and need a new secret",
+ "target": "auth/password-reset",
+ "category": "paraphrase",
+ "split": "held"
+ },
+ {
+ "query": "find notes that mean the same thing but use different words",
+ "target": "search/vector-embeddings",
+ "category": "paraphrase",
+ "split": "tune"
+ },
+ {
+ "query": "combine two result lists whose scores are on different scales",
+ "target": "search/reciprocal-rank-fusion",
+ "category": "paraphrase",
+ "split": "held"
+ },
+ {
+ "query": "score how well keywords match a document",
+ "target": "search/bm25-ranking",
+ "category": "lexical-strong",
+ "split": "tune"
+ },
+ {
+ "query": "ship code live and switch traffic over instantly with fast rollback",
+ "target": "ops/blue-green-deploys",
+ "category": "lexical-strong",
+ "split": "held"
+ },
+ {
+ "query": "change the database shape without taking the site down",
+ "target": "ops/database-migrations",
+ "category": "paraphrase",
+ "split": "tune"
+ },
+ {
+ "query": "what to do the moment a production alert pages me",
+ "target": "ops/incident-runbook",
+ "category": "paraphrase",
+ "split": "held"
+ },
+ {
+ "query": "simultaneous writers without overwriting one another's work",
+ "target": "editor/conflict-resolution",
+ "category": "zero-overlap",
+ "split": "tune"
+ },
+ {
+ "query": "see which other pages link to the one I'm reading",
+ "target": "editor/backlinks",
+ "category": "paraphrase",
+ "split": "held"
+ },
+ {
+ "query": "limit how many requests a caller can make per second",
+ "target": "algorithms/token-bucket",
+ "category": "paraphrase",
+ "split": "tune"
+ },
+ {
+ "query": "move rarely-read records to cheap slow archival storage",
+ "target": "data/cold-storage",
+ "category": "lexical-strong",
+ "split": "held"
+ },
+ {
+ "query": "first week plan for a brand new engineer joining the team",
+ "target": "guides/onboarding",
+ "category": "paraphrase",
+ "split": "tune"
+ },
+ {
+ "query": "how meaning-based search finds a passage with none of my words",
+ "target": "overview/platform-architecture",
+ "category": "long-doc",
+ "split": "tune"
+ },
+ {
+ "query": "undo a bad release by sending users back to the old environment",
+ "target": "overview/engineering-handbook",
+ "category": "long-doc",
+ "split": "held"
+ },
+ {
+ "query": "token bucket rate limiter",
+ "target": "algorithms/token-bucket",
+ "category": "lexical-strong",
+ "split": "tune"
+ },
+ {
+ "query": "oauth device flow",
+ "target": "auth/oauth-device-flow",
+ "category": "lexical-strong",
+ "split": "held"
+ },
+ {
+ "query": "reissue a short-lived token from a refresh token",
+ "target": "auth/session-token-refresh",
+ "category": "lexical-strong",
+ "split": "tune"
+ },
+ {
+ "query": "keep concurrent editors from overwriting each other's work",
+ "target": "editor/conflict-resolution",
+ "category": "paraphrase",
+ "split": "held"
+ },
+ {
+ "query": "let a TV get user consent through a second screen",
+ "target": "auth/oauth-device-flow",
+ "category": "paraphrase",
+ "split": "tune"
+ },
+ {
+ "query": "throttle repeated failed logins",
+ "target": "auth/login-rate-limiting",
+ "category": "lexical-strong",
+ "split": "held"
+ },
+ {
+ "query": "turn text into a dense representation for semantic retrieval",
+ "target": "search/vector-embeddings",
+ "category": "lexical-strong",
+ "split": "tune"
+ },
+ {
+ "query": "send users back to the previous deploy to recover quickly",
+ "target": "ops/blue-green-deploys",
+ "category": "paraphrase",
+ "split": "held"
+ },
+ {
+ "query": "email someone a single-use link to choose a new password",
+ "target": "auth/password-reset",
+ "category": "lexical-strong",
+ "split": "tune"
+ },
+ {
+ "query": "make schema changes that roll out with zero downtime",
+ "target": "ops/database-migrations",
+ "category": "lexical-strong",
+ "split": "held"
+ },
+ {
+ "query": "regain entry to my locked-out account by proving my email",
+ "target": "auth/password-reset",
+ "category": "zero-overlap",
+ "split": "tune"
+ }
+ ]
+}
diff --git a/packages/server/src/embeddings/eval/eval-set.test.ts b/packages/server/src/embeddings/eval/eval-set.test.ts
new file mode 100644
index 000000000..d90c5d871
--- /dev/null
+++ b/packages/server/src/embeddings/eval/eval-set.test.ts
@@ -0,0 +1,78 @@
+import { describe, expect, test } from 'bun:test';
+import { type EvalSet, loadEvalSet } from './semantic-eval.ts';
+
+function tokens(s: string): Set<string> {
+ return new Set(
+ s
+ .toLowerCase()
+ .split(/[^a-z0-9]+/)
+ .filter((t) => t.length > 2),
+ );
+}
+
+const STOP = new Set([
+ 'the',
+ 'and',
+ 'for',
+ 'with',
+ 'that',
+ 'this',
+ 'are',
+ 'how',
+ 'get',
+ 'out',
+ 'can',
+ 'use',
+]);
+
+describe('FR2 eval set structure', () => {
+ const set: EvalSet = loadEvalSet();
+ const byPath = new Map(set.corpus.map((d) => [d.path, d]));
+
+ test('every pair targets a real corpus doc', () => {
+ for (const p of set.pairs) {
+ expect(byPath.has(p.target), `unknown target ${p.target}`).toBe(true);
+ }
+ });
+
+ test('both splits are non-trivial', () => {
+ const tune = set.pairs.filter((p) => p.split === 'tune');
+ const held = set.pairs.filter((p) => p.split === 'held');
+ expect(tune.length).toBeGreaterThanOrEqual(8);
+ expect(held.length).toBeGreaterThanOrEqual(8);
+ });
+
+ test('held-out includes zero-overlap, long-doc, and lexical-strong categories', () => {
+ const heldCats = new Set(set.pairs.filter((p) => p.split === 'held').map((p) => p.category));
+ for (const cat of ['zero-overlap', 'long-doc', 'lexical-strong'] as const) {
+ expect(heldCats.has(cat), `held-out missing category ${cat}`).toBe(true);
+ }
+ });
+
+ test('zero-overlap pairs genuinely share ~no content tokens with their target (C2b)', () => {
+ const zero = set.pairs.filter((p) => p.category === 'zero-overlap');
+ expect(zero.length).toBeGreaterThanOrEqual(3);
+ for (const p of zero) {
+ const doc = byPath.get(p.target);
+ const qt = [...tokens(p.query)].filter((t) => !STOP.has(t));
+ const dt = tokens(`${doc?.title} ${doc?.path} ${doc?.content}`);
+ const shared = qt.filter((t) => dt.has(t));
+ expect(
+ shared.length,
+ `zero-overlap "${p.query}" shares ${JSON.stringify(shared)} with ${p.target}`,
+ ).toBeLessThanOrEqual(1);
+ }
+ });
+
+ test('long-doc pairs target a genuinely long document (C3)', () => {
+ const long = set.pairs.filter((p) => p.category === 'long-doc');
+ expect(long.length).toBeGreaterThanOrEqual(1);
+ for (const p of long) {
+ expect(byPath.get(p.target)?.content.length ?? 0).toBeGreaterThan(900);
+ }
+ });
+
+ test('corpus paths are unique', () => {
+ expect(new Set(set.corpus.map((d) => d.path)).size).toBe(set.corpus.length);
+ });
+});
diff --git a/packages/server/src/embeddings/eval/semantic-eval.smoke.test.ts b/packages/server/src/embeddings/eval/semantic-eval.smoke.test.ts
new file mode 100644
index 000000000..3e41f9457
--- /dev/null
+++ b/packages/server/src/embeddings/eval/semantic-eval.smoke.test.ts
@@ -0,0 +1,43 @@
+import { describe, expect, test } from 'bun:test';
+import { searchWorkspaceCorpus } from '@inkeep/open-knowledge-core';
+import { loadEvalEmbedder, loadEvalSet, prepareEval, runHeldOutEval } from './semantic-eval.ts';
+
+const ENABLED = process.env.OK_EMBED_SMOKE === '1';
+
+describe.skipIf(!ENABLED)('FR2 retrieval-quality eval (real embeddings API)', () => {
+ test('candidate-source recall: every zero-overlap target is retrievable', async () => {
+ const embedder = await loadEvalEmbedder();
+ expect(embedder, 'set OK_EMBEDDINGS_API_KEY for the gated eval').not.toBeNull();
+ if (!embedder) return;
+ const set = loadEvalSet();
+ const prep = await prepareEval(embedder, set);
+ for (const pair of set.pairs.filter((p) => p.category === 'zero-overlap')) {
+ const scores = prep.queryScores.get(pair.query) ?? new Map<string, number>();
+ const results = searchWorkspaceCorpus(prep.corpus, pair.query, {
+ intent: 'full_text',
+ limit: 20,
+ semantic: { scores, rrfK: 60 },
+ });
+ const found = results.some((r) => r.document.path === pair.target);
+ expect(found, `zero-overlap "${pair.query}" did not retrieve ${pair.target}`).toBe(true);
+ }
+ }, 180_000);
+
+ test('held-out FR2 gate (pre-registered; reported, not merge-blocking)', async () => {
+ const embedder = await loadEvalEmbedder();
+ if (!embedder) return;
+ const prep = await prepareEval(embedder, loadEvalSet());
+ const report = runHeldOutEval(prep);
+ console.log('[FR2] best:', JSON.stringify(report.calibration.best));
+ console.log(
+ `[FR2] held-out lexical MRR=${report.lexical.mrr.toFixed(4)} semantic MRR=${report.semantic.mrr.toFixed(4)} gain=${report.mrrGain.toFixed(4)}`,
+ );
+ console.log(
+ `[FR2] recall@5 gain=${report.recall5Gain.toFixed(4)} lexical-strong regression=${report.lexicalStrongRegression.toFixed(4)}`,
+ );
+ console.log(
+ `[FR2] GATE ${report.passes ? 'PASS — eligible for flag flip' : 'BELOW THRESHOLD — keep flag off'}`,
+ );
+ expect(report.semantic.mrr).toBeGreaterThan(0);
+ }, 180_000);
+});
diff --git a/packages/server/src/embeddings/eval/semantic-eval.ts b/packages/server/src/embeddings/eval/semantic-eval.ts
new file mode 100644
index 000000000..da01e20bd
--- /dev/null
+++ b/packages/server/src/embeddings/eval/semantic-eval.ts
@@ -0,0 +1,232 @@
+import { readFileSync } from 'node:fs';
+import {
+ createWorkspaceSearchCorpus,
+ createWorkspaceSearchDocument,
+ DEFAULT_RRF_K,
+ searchWorkspaceCorpus,
+ type WorkspaceSearchCorpus,
+ type WorkspaceSearchDocument,
+} from '@inkeep/open-knowledge-core';
+import { chunkDocument } from '../chunking.ts';
+import { cosineSimilarity, type Embedder, loadOpenAiEmbedder } from '../embedder.ts';
+
+const HELD_OUT_MRR_GAIN_MIN = 0.05;
+const HELD_OUT_RECALL5_GAIN_MIN = 0.08;
+const LEXICAL_STRONG_REGRESSION_MAX = 0.03;
+
+const RRF_K_GRID = [10, 20, 40, 60, 80, 120];
+
+interface EvalPair {
+ query: string;
+ target: string;
+ category: 'zero-overlap' | 'paraphrase' | 'long-doc' | 'lexical-strong';
+ split: 'tune' | 'held';
+}
+export interface EvalSet {
+ corpus: Array<{ path: string; title: string; content: string }>;
+ pairs: EvalPair[];
+}
+
+export function loadEvalSet(): EvalSet {
+ return JSON.parse(readFileSync(new URL('./eval-set.json', import.meta.url), 'utf-8')) as EvalSet;
+}
+
+interface Metrics {
+ n: number;
+ mrr: number;
+ recall1: number;
+ recall5: number;
+}
+
+function rankOfTarget(
+ results: ReturnType<typeof searchWorkspaceCorpus>,
+ targetPath: string,
+): number {
+ const idx = results.findIndex((r) => r.document.path === targetPath);
+ return idx < 0 ? Number.POSITIVE_INFINITY : idx + 1;
+}
+
+function aggregate(ranks: number[]): Metrics {
+ const n = ranks.length;
+ const mrr = ranks.reduce((s, r) => s + (Number.isFinite(r) ? 1 / r : 0), 0) / Math.max(1, n);
+ const recall1 = ranks.filter((r) => r <= 1).length / Math.max(1, n);
+ const recall5 = ranks.filter((r) => r <= 5).length / Math.max(1, n);
+ return { n, mrr, recall1, recall5 };
+}
+
+async function embedCorpusVectors(
+ embedder: Embedder,
+ docs: readonly WorkspaceSearchDocument[],
+): Promise<Map<string, Float32Array[]>> {
+ const byDoc = new Map<string, Float32Array[]>();
+ for (const doc of docs) {
+ const chunks = chunkDocument(doc.content);
+ byDoc.set(doc.id, chunks.length ? await embedder.embed(chunks, { role: 'document' }) : []);
+ }
+ return byDoc;
+}
+
+function scoresForQueryVec(
+ queryVec: Float32Array,
+ docVectors: Map<string, Float32Array[]>,
+): Map<string, number> {
+ const scores = new Map<string, number>();
+ for (const [docId, chunks] of docVectors) {
+ let best = Number.NEGATIVE_INFINITY;
+ for (const c of chunks) best = Math.max(best, cosineSimilarity(queryVec, c));
+ if (best > Number.NEGATIVE_INFINITY) scores.set(docId, best);
+ }
+ return scores;
+}
+
+export interface PreparedEval {
+ corpus: WorkspaceSearchCorpus;
+ docVectors: Map<string, Float32Array[]>;
+ queryScores: Map<string, Map<string, number>>;
+ pairs: EvalPair[];
+}
+
+export async function prepareEval(embedder: Embedder, set: EvalSet): Promise<PreparedEval> {
+ const docs = set.corpus.map((d) =>
+ createWorkspaceSearchDocument({
+ kind: 'page',
+ path: d.path,
+ title: d.title,
+ content: d.content,
+ }),
+ );
+ const corpus = createWorkspaceSearchCorpus(docs);
+ const docVectors = await embedCorpusVectors(embedder, docs);
+
+ const queryScores = new Map<string, Map<string, number>>();
+ for (const pair of set.pairs) {
+ if (queryScores.has(pair.query)) continue;
+ const [queryVec] = await embedder.embed([pair.query], { role: 'query' });
+ queryScores.set(pair.query, scoresForQueryVec(queryVec, docVectors));
+ }
+ return { corpus, docVectors, queryScores, pairs: set.pairs };
+}
+
+interface RunConfig {
+ rrfK: number;
+ semantic: boolean;
+}
+
+function evaluate(prep: PreparedEval, pairs: EvalPair[], cfg: RunConfig): Metrics {
+ const ranks = pairs.map((pair) => {
+ const sem = cfg.semantic
+ ? {
+ scores: prep.queryScores.get(pair.query) ?? new Map<string, number>(),
+ rrfK: cfg.rrfK,
+ }
+ : undefined;
+ const results = searchWorkspaceCorpus(prep.corpus, pair.query, {
+ intent: 'full_text',
+ limit: 20,
+ semantic: sem,
+ });
+ return rankOfTarget(results, pair.target);
+ });
+ return aggregate(ranks);
+}
+
+interface CalibrationResult {
+ best: { rrfK: number; mrr: number };
+ grid: Array<{ rrfK: number; mrr: number }>;
+}
+
+function calibrate(prep: PreparedEval): CalibrationResult {
+ const tune = prep.pairs.filter((p) => p.split === 'tune');
+ const grid = RRF_K_GRID.map((rrfK) => ({
+ rrfK,
+ mrr: evaluate(prep, tune, { rrfK, semantic: true }).mrr,
+ }));
+ const best = grid.reduce((a, b) => (b.mrr > a.mrr ? b : a));
+ return { best, grid };
+}
+
+export interface HeldOutReport {
+ calibration: CalibrationResult;
+ lexical: Metrics;
+ semantic: Metrics;
+ mrrGain: number;
+ recall5Gain: number;
+ lexicalStrongRegression: number;
+ passes: boolean;
+}
+
+const PRODUCTION_RRF_K = DEFAULT_RRF_K;
+
+export function runHeldOutEval(prep: PreparedEval): HeldOutReport {
+ const calibration = calibrate(prep);
+ const rrfK = PRODUCTION_RRF_K;
+ const held = prep.pairs.filter((p) => p.split === 'held');
+ const heldStrong = held.filter((p) => p.category === 'lexical-strong');
+
+ const lexical = evaluate(prep, held, { rrfK, semantic: false });
+ const semantic = evaluate(prep, held, { rrfK, semantic: true });
+ const lexStrongLex = evaluate(prep, heldStrong, { rrfK, semantic: false });
+ const lexStrongSem = evaluate(prep, heldStrong, { rrfK, semantic: true });
+
+ const mrrGain = semantic.mrr - lexical.mrr;
+ const recall5Gain = semantic.recall5 - lexical.recall5;
+ const lexicalStrongRegression = lexStrongLex.mrr - lexStrongSem.mrr;
+ const passes =
+ mrrGain >= HELD_OUT_MRR_GAIN_MIN &&
+ recall5Gain >= HELD_OUT_RECALL5_GAIN_MIN &&
+ lexicalStrongRegression <= LEXICAL_STRONG_REGRESSION_MAX;
+
+ return { calibration, lexical, semantic, mrrGain, recall5Gain, lexicalStrongRegression, passes };
+}
+
+export async function loadEvalEmbedder(): Promise<Embedder | null> {
+ return loadOpenAiEmbedder({
+ keyStore: null, // env (OK_EMBEDDINGS_API_KEY) fallback only — gated runs set it
+ config: {
+ baseUrl: process.env.OK_EMBEDDINGS_BASE_URL ?? 'https://api.openai.com/v1',
+ model: process.env.OK_EMBEDDINGS_MODEL ?? 'text-embedding-3-small',
+ dimensions: process.env.OK_EMBEDDINGS_DIMENSIONS
+ ? Number(process.env.OK_EMBEDDINGS_DIMENSIONS)
+ : undefined,
+ },
+ });
+}
+
+async function main(): Promise<void> {
+ const embedder = await loadEvalEmbedder();
+ if (!embedder) {
+ console.error('No embeddings key — set OK_EMBEDDINGS_API_KEY to run the eval.');
+ process.exit(1);
+ }
+ const prep = await prepareEval(embedder, loadEvalSet());
+ const report = runHeldOutEval(prep);
+
+ const f = (x: number) => x.toFixed(4);
+ console.log('=== Calibration (TUNE split only) ===');
+ for (const g of report.calibration.grid) {
+ console.log(` k=${String(g.rrfK).padStart(3)} tuneMRR=${f(g.mrr)}`);
+ }
+ console.log(
+ ` BEST: k=${report.calibration.best.rrfK} (tuneMRR=${f(report.calibration.best.mrr)})`,
+ );
+ console.log('\n=== HELD-OUT (measured once, frozen params) ===');
+ console.log(
+ ` lexical : MRR=${f(report.lexical.mrr)} recall@1=${f(report.lexical.recall1)} recall@5=${f(report.lexical.recall5)} (n=${report.lexical.n})`,
+ );
+ console.log(
+ ` semantic: MRR=${f(report.semantic.mrr)} recall@1=${f(report.semantic.recall1)} recall@5=${f(report.semantic.recall5)}`,
+ );
+ console.log(
+ ` MRR gain=${f(report.mrrGain)} (≥ ${HELD_OUT_MRR_GAIN_MIN}) recall@5 gain=${f(report.recall5Gain)} (≥ ${HELD_OUT_RECALL5_GAIN_MIN})`,
+ );
+ console.log(
+ ` lexical-strong regression=${f(report.lexicalStrongRegression)} (≤ ${LEXICAL_STRONG_REGRESSION_MAX})`,
+ );
+ console.log(
+ `\n FR2 GATE: ${report.passes ? 'PASS' : 'BELOW THRESHOLD (feature stays flag-off — does not block merge)'}`,
+ );
+}
+
+if (import.meta.main) {
+ await main();
+}
diff --git a/packages/server/src/embeddings/index.ts b/packages/server/src/embeddings/index.ts
new file mode 100644
index 000000000..5af5060d8
--- /dev/null
+++ b/packages/server/src/embeddings/index.ts
@@ -0,0 +1,24 @@
+export { createConceptEmbedder } from './concept-embedder.ts';
+export {
+ DEFAULT_EMBEDDINGS_DIMENSIONS,
+ EMBEDDINGS_API_KEY_ENV,
+ type Embedder,
+ type EmbeddingsKeyStore,
+ loadOpenAiEmbedder,
+ normalizeProviderId,
+} from './embedder.ts';
+export {
+ clearEmbeddingsKeyFromAllBackends,
+ createEmbeddingsSecretStore,
+ describeStoredEmbeddingsKey,
+ type EmbeddingsKeyReader,
+ type EmbeddingsSecretStore,
+ FileEmbeddingsBackend,
+ makeLazyEmbeddingsKeyStore,
+ secretsFilePath,
+} from './secrets-store.ts';
+export {
+ type ResolvedSemanticConfig,
+ readProjectLocalSemanticConfig,
+} from './semantic-config.ts';
+export { SEMANTIC_MIN_QUERY_LENGTH, SemanticSearchService } from './semantic-search-service.ts';
diff --git a/packages/server/src/embeddings/secrets-store.test.ts b/packages/server/src/embeddings/secrets-store.test.ts
new file mode 100644
index 000000000..115b1f484
--- /dev/null
+++ b/packages/server/src/embeddings/secrets-store.test.ts
@@ -0,0 +1,133 @@
+import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
+import {
+ chmodSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ rmSync,
+ statSync,
+ writeFileSync,
+} from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import {
+ clearEmbeddingsKeyFromAllBackends,
+ describeStoredEmbeddingsKey,
+ FileEmbeddingsBackend,
+ makeLazyEmbeddingsKeyStore,
+} from './secrets-store.ts';
+
+const KEY = 'sk-secret-embeddings-key-1234567890';
+
+let dir: string;
+let secretsFile: string;
+beforeEach(() => {
+ dir = mkdtempSync(join(tmpdir(), 'ok-embkey-'));
+ secretsFile = join(dir, '.ok', 'secrets.yml');
+});
+afterEach(() => {
+ rmSync(dir, { recursive: true, force: true });
+});
+
+describe('FileEmbeddingsBackend', () => {
+ test('set → get round-trips the key', async () => {
+ const store = new FileEmbeddingsBackend(secretsFile);
+ expect(await store.get()).toBeNull();
+ await store.set(KEY);
+ expect(await store.get()).toBe(KEY);
+ });
+
+ test('writes the secrets file with 0600 permissions', async () => {
+ const store = new FileEmbeddingsBackend(secretsFile);
+ await store.set(KEY);
+ expect(existsSync(secretsFile)).toBe(true);
+ const mode = statSync(secretsFile).mode & 0o777;
+ expect(mode).toBe(0o600);
+ });
+
+ test('re-asserts 0600 on a pre-existing, looser-permissioned secrets file', async () => {
+ mkdirSync(join(dir, '.ok'), { recursive: true });
+ writeFileSync(secretsFile, 'other: keep-me\n');
+ chmodSync(secretsFile, 0o644);
+ await new FileEmbeddingsBackend(secretsFile).set(KEY);
+ expect(statSync(secretsFile).mode & 0o777).toBe(0o600);
+ });
+
+ test('clear removes the key; get returns null again', async () => {
+ const store = new FileEmbeddingsBackend(secretsFile);
+ await store.set(KEY);
+ await store.clear();
+ expect(await store.get()).toBeNull();
+ });
+
+ test('clear unlinks the file when the key was the only secret', async () => {
+ const store = new FileEmbeddingsBackend(secretsFile);
+ await store.set(KEY);
+ expect(existsSync(secretsFile)).toBe(true);
+ await store.clear();
+ expect(existsSync(secretsFile)).toBe(false);
+ });
+
+ test('clear preserves other secrets in the file', async () => {
+ const store = new FileEmbeddingsBackend(secretsFile);
+ await store.set(KEY);
+ const raw = readFileSync(secretsFile, 'utf-8');
+ writeFileSync(secretsFile, `${raw}other: keep-me\n`);
+ await store.clear();
+ expect(await store.get()).toBeNull();
+ expect(readFileSync(secretsFile, 'utf-8')).toContain('other: keep-me');
+ });
+
+ test('empty / absent file reads as no key', async () => {
+ const store = new FileEmbeddingsBackend(secretsFile);
+ expect(await store.get()).toBeNull();
+ });
+});
+
+describe('makeLazyEmbeddingsKeyStore', () => {
+ test('reads the key from the secrets file (file-only, no keychain)', async () => {
+ await new FileEmbeddingsBackend(secretsFile).set(KEY);
+ const reader = makeLazyEmbeddingsKeyStore(secretsFile);
+ expect(await reader.get()).toBe(KEY);
+ });
+
+ test('picks up a key written AFTER the reader was created (re-reads each get)', async () => {
+ const reader = makeLazyEmbeddingsKeyStore(secretsFile);
+ expect(await reader.get()).toBeNull();
+ await new FileEmbeddingsBackend(secretsFile).set(KEY);
+ expect(await reader.get()).toBe(KEY);
+ });
+
+ test('returns null (never throws) when nothing is stored', async () => {
+ const reader = makeLazyEmbeddingsKeyStore(secretsFile);
+ expect(await reader.get()).toBeNull();
+ });
+});
+
+describe('describeStoredEmbeddingsKey', () => {
+ test('reports the file backend when the key lives there', async () => {
+ await new FileEmbeddingsBackend(secretsFile).set(KEY);
+ const desc = await describeStoredEmbeddingsKey(secretsFile);
+ expect(desc.file).toBe(true);
+ });
+
+ test('reports no file backend when nothing is stored', async () => {
+ const desc = await describeStoredEmbeddingsKey(secretsFile);
+ expect(desc.file).toBe(false);
+ });
+});
+
+describe('clearEmbeddingsKeyFromAllBackends', () => {
+ test('reports the file backend when it held a key', async () => {
+ await new FileEmbeddingsBackend(secretsFile).set(KEY);
+ const { touched } = await clearEmbeddingsKeyFromAllBackends(secretsFile);
+ expect(touched).toContain('file');
+ expect(await new FileEmbeddingsBackend(secretsFile).get()).toBeNull();
+ });
+
+ test('reports nothing when no key was stored', async () => {
+ const { touched } = await clearEmbeddingsKeyFromAllBackends(secretsFile);
+ expect(touched).toEqual([]);
+ });
+});
diff --git a/packages/server/src/embeddings/secrets-store.ts b/packages/server/src/embeddings/secrets-store.ts
new file mode 100644
index 000000000..926cd930e
--- /dev/null
+++ b/packages/server/src/embeddings/secrets-store.ts
@@ -0,0 +1,103 @@
+import { chmodSync, existsSync, readFileSync } from 'node:fs';
+import { homedir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { parse as yamlParse, stringify as yamlStringify } from 'yaml';
+import { tracedMkdirSync, tracedUnlinkSync, tracedWriteFileSync } from '../fs-traced.ts';
+
+const EMBEDDINGS_ACCOUNT = 'embeddings';
+
+export function secretsFilePath(homedirOverride?: string): string {
+ return join(homedirOverride ?? homedir(), '.ok', 'secrets.yml');
+}
+
+export interface EmbeddingsKeyReader {
+ get(): Promise<string | null>;
+}
+
+export interface EmbeddingsSecretStore extends EmbeddingsKeyReader {
+ readonly backend: 'file';
+ set(key: string): Promise<void>;
+ clear(): Promise<void>;
+}
+
+export class FileEmbeddingsBackend implements EmbeddingsSecretStore {
+ readonly backend = 'file' as const;
+ private readonly secretsFile: string;
+
+ constructor(secretsFile?: string) {
+ this.secretsFile = secretsFile ?? secretsFilePath();
+ }
+
+ private read(): Record<string, unknown> {
+ if (!existsSync(this.secretsFile)) return {};
+ try {
+ return (yamlParse(readFileSync(this.secretsFile, 'utf-8')) ?? {}) as Record<string, unknown>;
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : 'unknown error';
+ process.stderr.write(
+ `[embeddings] Failed to parse ${this.secretsFile}: ${msg}. Starting with empty secrets.\n`,
+ );
+ return {};
+ }
+ }
+
+ private write(data: Record<string, unknown>): void {
+ const dir = dirname(this.secretsFile);
+ if (!existsSync(dir)) tracedMkdirSync(dir, { recursive: true, mode: 0o700 });
+ tracedWriteFileSync(this.secretsFile, yamlStringify(data), { mode: 0o600 });
+ chmodSync(this.secretsFile, 0o600);
+ }
+
+ get(): Promise<string | null> {
+ const value = this.read()[EMBEDDINGS_ACCOUNT];
+ return Promise.resolve(typeof value === 'string' && value !== '' ? value : null);
+ }
+
+ set(key: string): Promise<void> {
+ const data = this.read();
+ data[EMBEDDINGS_ACCOUNT] = key;
+ this.write(data);
+ return Promise.resolve();
+ }
+
+ clear(): Promise<void> {
+ const data = this.read();
+ if (EMBEDDINGS_ACCOUNT in data) {
+ delete data[EMBEDDINGS_ACCOUNT];
+ if (Object.keys(data).length === 0) {
+ try {
+ tracedUnlinkSync(this.secretsFile);
+ } catch {}
+ } else {
+ this.write(data);
+ }
+ }
+ return Promise.resolve();
+ }
+}
+
+export function createEmbeddingsSecretStore(secretsFile?: string): EmbeddingsSecretStore {
+ return new FileEmbeddingsBackend(secretsFile);
+}
+
+export function makeLazyEmbeddingsKeyStore(secretsFile?: string): EmbeddingsKeyReader {
+ return new FileEmbeddingsBackend(secretsFile);
+}
+
+export async function describeStoredEmbeddingsKey(
+ secretsFile?: string,
+): Promise<{ file: boolean }> {
+ return { file: (await new FileEmbeddingsBackend(secretsFile).get()) != null };
+}
+
+export async function clearEmbeddingsKeyFromAllBackends(
+ secretsFile?: string,
+): Promise<{ touched: Array<'file'> }> {
+ const touched: Array<'file'> = [];
+ const file = new FileEmbeddingsBackend(secretsFile);
+ if ((await file.get()) != null) {
+ await file.clear();
+ touched.push('file');
+ }
+ return { touched };
+}
diff --git a/packages/server/src/embeddings/semantic-config.test.ts b/packages/server/src/embeddings/semantic-config.test.ts
new file mode 100644
index 000000000..36e317be4
--- /dev/null
+++ b/packages/server/src/embeddings/semantic-config.test.ts
@@ -0,0 +1,51 @@
+import { describe, expect, test } from 'bun:test';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { readProjectLocalSemanticConfig } from './semantic-config.ts';
+
+function makeProject(): string {
+ const dir = mkdtempSync(join(tmpdir(), 'ok-semcfg-'));
+ mkdirSync(join(dir, '.ok', 'local'), { recursive: true });
+ return dir;
+}
+
+describe('readProjectLocalSemanticConfig', () => {
+ test('reads enabled + provider from the project-local layer', () => {
+ const dir = makeProject();
+ try {
+ writeFileSync(
+ join(dir, '.ok', 'local', 'config.yml'),
+ 'search:\n semantic:\n enabled: true\n model: text-embedding-3-large\n',
+ );
+ const cfg = readProjectLocalSemanticConfig(dir);
+ expect(cfg.enabled).toBe(true);
+ expect(cfg.model).toBe('text-embedding-3-large');
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('IGNORES a committed project config — project-local only (egress safety)', () => {
+ const dir = makeProject();
+ try {
+ writeFileSync(join(dir, '.ok', 'config.yml'), 'search:\n semantic:\n enabled: true\n');
+ expect(readProjectLocalSemanticConfig(dir).enabled).toBe(false);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('absent config → disabled with provider defaults', () => {
+ const dir = makeProject();
+ try {
+ const cfg = readProjectLocalSemanticConfig(dir);
+ expect(cfg.enabled).toBe(false);
+ expect(cfg.baseUrl).toContain('openai');
+ expect(typeof cfg.model).toBe('string');
+ expect(cfg.dimensions).toBeUndefined();
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+});
diff --git a/packages/server/src/embeddings/semantic-config.ts b/packages/server/src/embeddings/semantic-config.ts
new file mode 100644
index 000000000..2fd1355d3
--- /dev/null
+++ b/packages/server/src/embeddings/semantic-config.ts
@@ -0,0 +1,26 @@
+import { DEFAULT_EMBEDDINGS_BASE_URL, DEFAULT_EMBEDDINGS_MODEL } from '@inkeep/open-knowledge-core';
+import { readConfigSafely, resolveConfigPath } from '@inkeep/open-knowledge-core/server';
+
+export interface ResolvedSemanticConfig {
+ enabled: boolean;
+ baseUrl: string;
+ model: string;
+ dimensions?: number;
+}
+
+export function readProjectLocalSemanticConfig(
+ projectDir: string,
+ opts?: { configHomedirOverride?: string; onWarn?: (message: string) => void },
+): ResolvedSemanticConfig {
+ const semantic = readConfigSafely({
+ absPath: resolveConfigPath('project-local', projectDir, opts?.configHomedirOverride),
+ sideline: false,
+ warn: opts?.onWarn ?? (() => {}),
+ }).value.search?.semantic;
+ return {
+ enabled: semantic?.enabled === true,
+ baseUrl: semantic?.baseUrl ?? DEFAULT_EMBEDDINGS_BASE_URL,
+ model: semantic?.model ?? DEFAULT_EMBEDDINGS_MODEL,
+ dimensions: semantic?.dimensions ?? undefined,
+ };
+}
diff --git a/packages/server/src/embeddings/semantic-search-service.test.ts b/packages/server/src/embeddings/semantic-search-service.test.ts
new file mode 100644
index 000000000..aa7d74d7f
--- /dev/null
+++ b/packages/server/src/embeddings/semantic-search-service.test.ts
@@ -0,0 +1,281 @@
+import { describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import {
+ createWorkspaceSearchDocument,
+ type WorkspaceSearchDocument,
+} from '@inkeep/open-knowledge-core';
+import { CHUNK_CONFIG_ID } from './chunking.ts';
+import { createConceptEmbedder } from './concept-embedder.ts';
+import type { Embedder } from './embedder.ts';
+import { SemanticSearchService } from './semantic-search-service.ts';
+import { VectorCache } from './vector-cache.ts';
+
+const concepts = [
+ { id: 'auth', terms: ['auth', 'authentication', 'session token', 'credential', 'login'] },
+ { id: 'retry', terms: ['retry', 'retries', 'backoff', 're-issue', 'refresh'] },
+ { id: 'bread', terms: ['bread', 'sourdough', 'ferment'] },
+];
+
+function doc(path: string, content: string, modifiedTs = 1): WorkspaceSearchDocument {
+ return createWorkspaceSearchDocument({ kind: 'page', path, title: path, content, modifiedTs });
+}
+
+function makeService(over: Partial<{ embedder: Embedder | null; enabled: boolean }> = {}) {
+ const embedder =
+ over.embedder === undefined ? createConceptEmbedder({ concepts }) : over.embedder;
+ return new SemanticSearchService({
+ loadEmbedder: () => Promise.resolve(embedder),
+ cacheDir: null,
+ enabled: over.enabled ?? false,
+ });
+}
+
+const corpus = [
+ doc('session-tokens', 'The session token refresh flow re-issues credentials when they expire.'),
+ doc('sourdough', 'A recipe for sourdough bread with a long cold ferment.'),
+];
+
+describe('SemanticSearchService', () => {
+ test('disabled service is inert: queryScores returns null, no warm, no key read', async () => {
+ let loaded = false;
+ const svc = new SemanticSearchService({
+ loadEmbedder: () => {
+ loaded = true;
+ return Promise.resolve(createConceptEmbedder({ concepts }));
+ },
+ cacheDir: null,
+ enabled: false,
+ });
+ await svc.embedCorpus(corpus); // no-op while disabled
+ expect(await svc.queryScores('auth retries', corpus)).toBeNull();
+ expect(svc.getStatus().ready).toBe(false);
+ expect(loaded).toBe(false); // disabled never even resolves the key
+ });
+
+ test('queryScores returns null before any corpus is embedded (cold)', async () => {
+ const svc = makeService({ enabled: true });
+ await svc.ensureWarm();
+ expect(await svc.queryScores('auth retries', corpus)).toBeNull();
+ });
+
+ test('embedded corpus yields a high cosine for a zero-token-overlap concept match', async () => {
+ const svc = makeService({ enabled: true });
+ await svc.embedCorpus(corpus);
+ const scores = await svc.queryScores('auth retries', corpus);
+ expect(scores).not.toBeNull();
+ const tokenDoc = scores?.get('page:session-tokens') ?? -1;
+ const breadDoc = scores?.get('page:sourdough') ?? -1;
+ expect(tokenDoc).toBeGreaterThan(0.4);
+ expect(tokenDoc).toBeGreaterThan(breadDoc + 0.2);
+ });
+
+ test('no key (loadEmbedder → null) degrades: capable=false, queryScores null', async () => {
+ const svc = makeService({ embedder: null, enabled: true });
+ await svc.embedCorpus(corpus);
+ expect(svc.getStatus().capable).toBe(false);
+ expect(svc.getStatus().ready).toBe(true);
+ expect(await svc.queryScores('auth retries', corpus)).toBeNull();
+ });
+
+ test('incremental: unchanged docs are not re-embedded (mtime pre-filter)', async () => {
+ let embedCalls = 0;
+ const inner = createConceptEmbedder({ concepts });
+ const counting: Embedder = {
+ providerId: inner.providerId,
+ modelId: inner.modelId,
+ dims: inner.dims,
+ embed: (texts, opts) => {
+ if (opts.role === 'document') embedCalls += texts.length;
+ return inner.embed(texts, opts);
+ },
+ };
+ const svc = new SemanticSearchService({
+ loadEmbedder: () => Promise.resolve(counting),
+ cacheDir: null,
+ enabled: true,
+ });
+ await svc.embedCorpus(corpus);
+ const firstPass = embedCalls;
+ expect(firstPass).toBeGreaterThan(0);
+ await svc.embedCorpus(corpus);
+ expect(embedCalls).toBe(firstPass);
+ const changed = [
+ doc('session-tokens', 'Completely different text about login flows.', 2),
+ corpus[1],
+ ];
+ await svc.embedCorpus(changed);
+ expect(embedCalls).toBeGreaterThan(firstPass);
+ });
+
+ test('coverage grows as docs embed (status.embeddedCount)', async () => {
+ const svc = makeService({ enabled: true });
+ expect(svc.getStatus().embeddedCount).toBe(0);
+ await svc.embedCorpus(corpus);
+ expect(svc.getStatus().embeddedCount).toBe(2);
+ });
+
+ test('partial failure: a bad doc is isolated; its batch-mates still embed', async () => {
+ const inner = createConceptEmbedder({ concepts });
+ const flaky: Embedder = {
+ providerId: inner.providerId,
+ modelId: inner.modelId,
+ dims: inner.dims,
+ embed: (texts, opts) => {
+ if (texts.some((t) => t.includes('POISON'))) {
+ return Promise.reject(new Error('provider rejected input'));
+ }
+ return inner.embed(texts, opts);
+ },
+ };
+ const svc = new SemanticSearchService({
+ loadEmbedder: () => Promise.resolve(flaky),
+ cacheDir: null,
+ enabled: true,
+ });
+ const mixed = [
+ doc('good-1', 'session token authentication credential'),
+ doc('bad', 'POISON content the provider chokes on'),
+ doc('good-2', 'sourdough bread cold ferment'),
+ ];
+ await svc.embedCorpus(mixed); // must not throw
+ expect(svc.getStatus().embeddedCount).toBe(2); // both good docs embedded
+ const scores = await svc.queryScores('authentication login session', mixed);
+ expect(scores?.has('page:good-1')).toBe(true);
+ expect(scores?.has('page:bad')).toBe(false);
+ });
+
+ test('query-path provider error degrades to lexical (queryScores → null, no throw)', async () => {
+ const inner = createConceptEmbedder({ concepts });
+ let failQueries = false;
+ const flaky: Embedder = {
+ providerId: inner.providerId,
+ modelId: inner.modelId,
+ dims: inner.dims,
+ embed: (texts, opts) => {
+ if (opts.role === 'query' && failQueries) return Promise.reject(new Error('provider down'));
+ return inner.embed(texts, opts);
+ },
+ };
+ const svc = new SemanticSearchService({
+ loadEmbedder: () => Promise.resolve(flaky),
+ cacheDir: null,
+ enabled: true,
+ });
+ await svc.embedCorpus(corpus);
+ expect(await svc.queryScores('auth retries', corpus)).not.toBeNull();
+ failQueries = true;
+ expect(await svc.queryScores('auth retries', corpus)).toBeNull(); // degraded, not thrown
+ });
+
+ test('applyConfig disable frees in-memory vectors; re-enable re-warms', async () => {
+ const svc = makeService({ enabled: true });
+ await svc.embedCorpus(corpus);
+ expect(svc.getStatus().embeddedCount).toBe(2);
+ svc.applyConfig({ enabled: false, providerFingerprint: '' });
+ expect(svc.getStatus().embeddedCount).toBe(0);
+ expect(await svc.queryScores('auth retries', corpus)).toBeNull();
+ svc.applyConfig({ enabled: true, providerFingerprint: '' });
+ await svc.embedCorpus(corpus);
+ expect(svc.getStatus().embeddedCount).toBe(2);
+ });
+
+ test('disable racing an in-flight embed pass does NOT wipe the on-disk cache', async () => {
+ const dir = mkdtempSync(join(tmpdir(), 'ok-vec-race-'));
+ try {
+ const inner = createConceptEmbedder({ concepts });
+ let release: (() => void) | null = null;
+ let signalEntered: (() => void) | null = null;
+ const enteredEmbed = new Promise<void>((r) => {
+ signalEntered = r;
+ });
+ const gate = new Promise<void>((r) => {
+ release = r;
+ });
+ let blockNextDocEmbed = false;
+ const gated: Embedder = {
+ providerId: inner.providerId,
+ modelId: inner.modelId,
+ dims: inner.dims,
+ async embed(texts, opts) {
+ if (opts.role === 'document' && blockNextDocEmbed) {
+ blockNextDocEmbed = false;
+ signalEntered?.();
+ await gate;
+ }
+ return inner.embed(texts, opts);
+ },
+ };
+ const svc = new SemanticSearchService({
+ loadEmbedder: () => Promise.resolve(gated),
+ cacheDir: dir,
+ enabled: true,
+ });
+
+ await svc.embedCorpus(corpus);
+ expect(svc.getStatus().embeddedCount).toBe(2);
+
+ blockNextDocEmbed = true;
+ const pending = svc.embedCorpus([
+ ...corpus,
+ doc('new-topic', 'a fresh note about backoff and retries', 5),
+ ]);
+ await enteredEmbed; // the pass is now parked inside embed()
+ svc.applyConfig({ enabled: false, providerFingerprint: '' }); // clearMemory + null cache
+ release?.(); // let the orphaned pass run its tail
+ await pending; // must not throw — and must NOT persist the emptied cache
+
+ const reopened = new VectorCache({
+ cacheDir: dir,
+ providerId: gated.providerId,
+ modelId: gated.modelId,
+ dims: gated.dims,
+ chunkConfigId: CHUNK_CONFIG_ID,
+ });
+ await reopened.init();
+ expect(reopened.embeddedCount).toBe(2);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ test('applyConfig provider-fingerprint change re-loads the embedder', async () => {
+ let loads = 0;
+ const svc = new SemanticSearchService({
+ loadEmbedder: () => {
+ loads += 1;
+ return Promise.resolve(createConceptEmbedder({ concepts }));
+ },
+ cacheDir: null,
+ enabled: true,
+ providerFingerprint: 'openai|text-embedding-3-small|1536',
+ });
+ await svc.embedCorpus(corpus);
+ expect(loads).toBe(1);
+ svc.applyConfig({ enabled: true, providerFingerprint: 'openai|text-embedding-3-small|1536' });
+ await svc.embedCorpus(corpus);
+ expect(loads).toBe(1);
+ svc.applyConfig({ enabled: true, providerFingerprint: 'openai|text-embedding-3-large|3072' });
+ await svc.embedCorpus(corpus);
+ expect(loads).toBe(2);
+ });
+
+ test('max chunk cosine roll-up: a buried passage still surfaces the doc', async () => {
+ const svc = makeService({ enabled: true });
+ const breadBlock = 'sourdough bread cold ferment dough recipe loaf crust. '.repeat(90);
+ const authBlock =
+ 'session token authentication credential login refresh re-issue access. '.repeat(90);
+ const mixed = doc('mixed', `${breadBlock}\n\n${authBlock}`);
+ expect(mixed.content.length).toBeGreaterThan(8000); // proves it chunks
+ const breadOnly = doc('bread-only', breadBlock);
+ await svc.embedCorpus([mixed, breadOnly]);
+ const scores = await svc.queryScores('authentication credential login session', [
+ mixed,
+ breadOnly,
+ ]);
+ const mixedScore = scores?.get('page:mixed') ?? -1;
+ const breadScore = scores?.get('page:bread-only') ?? -1;
+ expect(mixedScore).toBeGreaterThan(breadScore + 0.15);
+ });
+});
diff --git a/packages/server/src/embeddings/semantic-search-service.ts b/packages/server/src/embeddings/semantic-search-service.ts
new file mode 100644
index 000000000..c1339af01
--- /dev/null
+++ b/packages/server/src/embeddings/semantic-search-service.ts
@@ -0,0 +1,271 @@
+import type { WorkspaceSearchDocument } from '@inkeep/open-knowledge-core';
+import { getLogger } from '../logger.ts';
+import { CHUNK_CONFIG_ID, chunkDocument } from './chunking.ts';
+import { cosineSimilarity, type Embedder } from './embedder.ts';
+import { hashContent, VectorCache } from './vector-cache.ts';
+
+const log = getLogger('embeddings');
+
+function errMsg(err: unknown): string {
+ return err instanceof Error ? err.message : String(err);
+}
+
+export const SEMANTIC_MIN_QUERY_LENGTH = 3;
+
+const EMBED_BATCH_CHUNK_LIMIT = 96;
+
+const MAX_CONSECUTIVE_EMBED_FAILURES = 5;
+
+export interface SemanticSearchStatus {
+ enabled: boolean;
+ capable: boolean;
+ ready: boolean;
+ embeddedCount: number;
+}
+
+export interface SemanticSearchServiceOptions {
+ loadEmbedder: () => Promise<Embedder | null>;
+ cacheDir: string | null;
+ enabled?: boolean;
+ providerFingerprint?: string;
+}
+
+export class SemanticSearchService {
+ private readonly loadEmbedder: () => Promise<Embedder | null>;
+ private readonly cacheDir: string | null;
+
+ private enabled: boolean;
+ private providerFingerprint: string;
+ private capable = false;
+ private ready = false;
+ private embedder: Embedder | null = null;
+ private cache: VectorCache | null = null;
+
+ private warmPromise: Promise<void> | null = null;
+ private embedChain: Promise<void> = Promise.resolve();
+ private queuedDocs: readonly WorkspaceSearchDocument[] | null = null;
+
+ constructor(options: SemanticSearchServiceOptions) {
+ this.loadEmbedder = options.loadEmbedder;
+ this.cacheDir = options.cacheDir;
+ this.enabled = options.enabled ?? false;
+ this.providerFingerprint = options.providerFingerprint ?? '';
+ }
+
+ isEnabled(): boolean {
+ return this.enabled;
+ }
+
+ getStatus(): SemanticSearchStatus {
+ return {
+ enabled: this.enabled,
+ capable: this.capable,
+ ready: this.ready,
+ embeddedCount: this.cache?.embeddedCount ?? 0,
+ };
+ }
+
+ applyConfig(input: { enabled: boolean; providerFingerprint: string }): void {
+ if (input.providerFingerprint !== this.providerFingerprint) {
+ this.providerFingerprint = input.providerFingerprint;
+ this.resetWarm();
+ }
+ if (input.enabled === this.enabled) return;
+ this.enabled = input.enabled;
+ if (!input.enabled) {
+ this.cache?.clearMemory();
+ this.resetWarm();
+ }
+ }
+
+ private resetWarm(): void {
+ this.warmPromise = null;
+ this.ready = false;
+ this.capable = false;
+ this.embedder = null;
+ this.cache = null;
+ }
+
+ ensureWarm(): Promise<void> {
+ if (!this.enabled) return Promise.resolve();
+ if (this.ready) return Promise.resolve();
+ if (!this.warmPromise) this.warmPromise = this.warm();
+ return this.warmPromise;
+ }
+
+ private async warm(): Promise<void> {
+ try {
+ const embedder = await this.loadEmbedder();
+ if (!embedder) {
+ this.capable = false;
+ this.ready = true;
+ log.info(
+ {},
+ '[embeddings] no embeddings key configured — semantic search degrades to lexical',
+ );
+ return;
+ }
+ this.embedder = embedder;
+ this.cache = new VectorCache({
+ cacheDir: this.cacheDir,
+ providerId: embedder.providerId,
+ modelId: embedder.modelId,
+ dims: embedder.dims,
+ chunkConfigId: CHUNK_CONFIG_ID,
+ });
+ await this.cache.init();
+ this.capable = true;
+ this.ready = true;
+ } catch (err) {
+ this.capable = false;
+ this.ready = true;
+ log.warn(
+ { err: err instanceof Error ? err.message : String(err) },
+ '[embeddings] warm failed',
+ );
+ }
+ }
+
+ embedCorpus(documents: readonly WorkspaceSearchDocument[]): Promise<void> {
+ if (!this.enabled) return Promise.resolve();
+ this.queuedDocs = documents;
+ this.embedChain = this.embedChain.then(async () => {
+ const next = this.queuedDocs;
+ if (!next) return; // a later call already coalesced this work
+ this.queuedDocs = null;
+ try {
+ await this.runEmbedPass(next);
+ } catch (err) {
+ log.warn(
+ { err: err instanceof Error ? err.message : String(err) },
+ '[embeddings] embed pass failed',
+ );
+ }
+ });
+ return this.embedChain;
+ }
+
+ private async runEmbedPass(documents: readonly WorkspaceSearchDocument[]): Promise<void> {
+ await this.ensureWarm();
+ if (!this.enabled || !this.embedder || !this.cache) return;
+ const cache = this.cache;
+ const embedder = this.embedder;
+ const pageDocs = documents.filter((d) => d.kind === 'page');
+ const activeIds = new Set(pageDocs.map((d) => d.id));
+
+ interface Pending {
+ doc: WorkspaceSearchDocument;
+ contentHash: string;
+ chunks: string[];
+ }
+ const pending: Pending[] = [];
+ for (const doc of pageDocs) {
+ if (!this.enabled) return; // bail if disabled mid-pass
+ const mtimeMs = doc.modifiedTs;
+ if (cache.isFresh(doc.id, mtimeMs)) continue;
+ const contentHash = hashContent(doc.content);
+ if (cache.link(doc.id, contentHash, mtimeMs)) continue;
+ pending.push({ doc, contentHash, chunks: chunkDocument(doc.content) });
+ }
+
+ let consecutiveFailures = 0;
+
+ const storeDoc = (p: Pending, vectors: Float32Array[]): void => {
+ cache.store(p.doc.id, p.contentHash, p.doc.modifiedTs, vectors);
+ };
+
+ const embedGroup = async (group: Pending[]): Promise<boolean> => {
+ const flat = group.flatMap((p) => p.chunks);
+ try {
+ const vectors = flat.length ? await embedder.embed(flat, { role: 'document' }) : [];
+ let offset = 0;
+ for (const p of group) {
+ storeDoc(p, vectors.slice(offset, offset + p.chunks.length));
+ offset += p.chunks.length;
+ }
+ consecutiveFailures = 0;
+ return true;
+ } catch (batchErr) {
+ if (group.length === 1) {
+ log.warn(
+ { docId: group[0].doc.id, err: errMsg(batchErr) },
+ '[embeddings] failed to embed document',
+ );
+ consecutiveFailures += 1;
+ return consecutiveFailures < MAX_CONSECUTIVE_EMBED_FAILURES;
+ }
+ for (const p of group) {
+ if (!this.enabled) return false;
+ try {
+ const v = p.chunks.length ? await embedder.embed(p.chunks, { role: 'document' }) : [];
+ storeDoc(p, v);
+ consecutiveFailures = 0;
+ } catch (docErr) {
+ log.warn(
+ { docId: p.doc.id, err: errMsg(docErr) },
+ '[embeddings] failed to embed document',
+ );
+ consecutiveFailures += 1;
+ if (consecutiveFailures >= MAX_CONSECUTIVE_EMBED_FAILURES) return false;
+ }
+ }
+ return true;
+ }
+ };
+
+ let batch: Pending[] = [];
+ let batchChunks = 0;
+ for (const p of pending) {
+ if (!this.enabled) break;
+ batch.push(p);
+ batchChunks += Math.max(1, p.chunks.length);
+ if (batchChunks >= EMBED_BATCH_CHUNK_LIMIT) {
+ const carryOn = await embedGroup(batch);
+ batch = [];
+ batchChunks = 0;
+ if (!carryOn) break;
+ }
+ }
+ if (batch.length > 0 && this.enabled) await embedGroup(batch);
+
+ if (!this.enabled || this.cache !== cache) return;
+ cache.retain(activeIds);
+ await cache.persist();
+ }
+
+ async queryScores(
+ query: string,
+ documents: readonly WorkspaceSearchDocument[],
+ ): Promise<Map<string, number> | null> {
+ if (!this.enabled || !this.capable || !this.ready) return null;
+ if (!this.embedder || !this.cache) return null;
+ if (this.cache.embeddedCount === 0) return null;
+ const trimmed = query.trim();
+ if (!trimmed) return null;
+
+ let queryVec: Float32Array | undefined;
+ try {
+ [queryVec] = await this.embedder.embed([trimmed], { role: 'query' });
+ } catch (err) {
+ log.warn(
+ { err: err instanceof Error ? err.message : String(err) },
+ '[embeddings] query embed failed — degrading to lexical',
+ );
+ return null;
+ }
+ if (!queryVec) return null;
+
+ const scores = new Map<string, number>();
+ for (const doc of documents) {
+ const vectors = this.cache.getVectors(doc.id);
+ if (!vectors || vectors.length === 0) continue;
+ let best = Number.NEGATIVE_INFINITY;
+ for (const chunk of vectors) {
+ const cos = cosineSimilarity(queryVec, chunk);
+ if (cos > best) best = cos;
+ }
+ if (best > Number.NEGATIVE_INFINITY) scores.set(doc.id, best);
+ }
+ return scores;
+ }
+}
diff --git a/packages/server/src/embeddings/vector-cache.test.ts b/packages/server/src/embeddings/vector-cache.test.ts
new file mode 100644
index 000000000..926c4aec4
--- /dev/null
+++ b/packages/server/src/embeddings/vector-cache.test.ts
@@ -0,0 +1,140 @@
+import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
+import { existsSync, mkdtempSync, readdirSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { hashContent, VectorCache } from './vector-cache.ts';
+
+const DIMS = 4;
+function vec(...xs: number[]): Float32Array {
+ return Float32Array.from(xs);
+}
+
+let dir: string;
+beforeEach(() => {
+ dir = mkdtempSync(join(tmpdir(), 'ok-veccache-'));
+});
+afterEach(() => {
+ rmSync(dir, { recursive: true, force: true });
+});
+
+function makeCache(
+ over: Partial<{ providerId: string; modelId: string; chunkConfigId: string }> = {},
+): VectorCache {
+ return new VectorCache({
+ cacheDir: dir,
+ providerId: over.providerId ?? 'https://api.test/v1',
+ modelId: over.modelId ?? 'test-model',
+ dims: DIMS,
+ chunkConfigId: over.chunkConfigId ?? 'c1-o0',
+ });
+}
+
+describe('VectorCache', () => {
+ test('store → persist → re-init round-trips vectors', async () => {
+ const a = makeCache();
+ await a.init();
+ const hash = hashContent('hello');
+ a.store('page:doc-a', hash, 100, [vec(1, 0, 0, 0), vec(0, 1, 0, 0)]);
+ await a.persist();
+ expect(existsSync(join(dir, 'manifest.json'))).toBe(true);
+ expect(existsSync(join(dir, 'vec', `${hash}.bin`))).toBe(true);
+
+ const b = makeCache();
+ await b.init();
+ const loaded = b.getVectors('page:doc-a');
+ expect(loaded?.length).toBe(2);
+ expect(Array.from(loaded?.[0])).toEqual([1, 0, 0, 0]);
+ expect(b.isFresh('page:doc-a', 100)).toBe(true);
+ expect(b.isFresh('page:doc-a', 101)).toBe(false); // mtime moved
+ });
+
+ test('link reuses identical content across docs (content-addressed dedup)', async () => {
+ const c = makeCache();
+ await c.init();
+ const hash = hashContent('shared body');
+ c.store('page:a', hash, 1, [vec(1, 1, 1, 1)]);
+ expect(c.link('page:b', hash, 2)).toBe(true);
+ expect(c.getVectors('page:b')).toBeDefined();
+ await c.persist();
+ const blobs = readdirSync(join(dir, 'vec'));
+ expect(blobs).toEqual([`${hash}.bin`]);
+ });
+
+ test('link returns false when no vectors are held for the hash', async () => {
+ const c = makeCache();
+ await c.init();
+ expect(c.link('page:a', hashContent('never embedded'), 1)).toBe(false);
+ });
+
+ test('model-id change invalidates the whole cache on init', async () => {
+ const a = makeCache({ modelId: 'model-1' });
+ await a.init();
+ a.store('page:a', hashContent('x'), 1, [vec(1, 0, 0, 0)]);
+ await a.persist();
+
+ const b = makeCache({ modelId: 'model-2' });
+ await b.init();
+ expect(b.getVectors('page:a')).toBeUndefined();
+ expect(b.embeddedCount).toBe(0);
+ });
+
+ test('provider change invalidates the cache (cross-provider vector guard)', async () => {
+ const a = makeCache({ providerId: 'https://api.openai.com/v1' });
+ await a.init();
+ a.store('page:a', hashContent('x'), 1, [vec(1, 0, 0, 0)]);
+ await a.persist();
+ const b = makeCache({ providerId: 'https://my-azure.openai.azure.com/v1' });
+ await b.init();
+ expect(b.getVectors('page:a')).toBeUndefined();
+ expect(b.embeddedCount).toBe(0);
+ });
+
+ test('chunk-config change invalidates the cache', async () => {
+ const a = makeCache({ chunkConfigId: 'c8000-o400' });
+ await a.init();
+ a.store('page:a', hashContent('x'), 1, [vec(1, 0, 0, 0)]);
+ await a.persist();
+ const b = makeCache({ chunkConfigId: 'c800-o100' });
+ await b.init();
+ expect(b.getVectors('page:a')).toBeUndefined();
+ });
+
+ test('retain + persist GCs entries and orphaned blobs for removed docs', async () => {
+ const c = makeCache();
+ await c.init();
+ const hashA = hashContent('a');
+ const hashB = hashContent('b');
+ c.store('page:a', hashA, 1, [vec(1, 0, 0, 0)]);
+ c.store('page:b', hashB, 1, [vec(0, 1, 0, 0)]);
+ await c.persist();
+ expect(readdirSync(join(dir, 'vec')).length).toBe(2);
+
+ c.retain(new Set(['page:a']));
+ await c.persist();
+ expect(c.getVectors('page:b')).toBeUndefined();
+ expect(readdirSync(join(dir, 'vec'))).toEqual([`${hashA}.bin`]);
+ });
+
+ test('embeddedCount counts docs with non-empty vectors', async () => {
+ const c = makeCache();
+ await c.init();
+ c.store('page:a', hashContent('a'), 1, [vec(1, 0, 0, 0)]);
+ c.store('page:blank', hashContent(''), 1, []); // blank doc, no chunks
+ expect(c.embeddedCount).toBe(1);
+ });
+
+ test('memory-only mode (cacheDir null) works without touching disk', async () => {
+ const c = new VectorCache({
+ cacheDir: null,
+ providerId: 'p',
+ modelId: 'm',
+ dims: DIMS,
+ chunkConfigId: 'c',
+ });
+ await c.init();
+ c.store('page:a', hashContent('a'), 1, [vec(1, 0, 0, 0)]);
+ expect(c.getVectors('page:a')).toBeDefined();
+ await c.persist(); // no-op, no throw
+ expect(readdirSync(dir).length).toBe(0);
+ });
+});
diff --git a/packages/server/src/embeddings/vector-cache.ts b/packages/server/src/embeddings/vector-cache.ts
new file mode 100644
index 000000000..7260bcc3d
--- /dev/null
+++ b/packages/server/src/embeddings/vector-cache.ts
@@ -0,0 +1,264 @@
+import { createHash } from 'node:crypto';
+import { existsSync, readdirSync } from 'node:fs';
+import { readFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import {
+ tracedMkdir,
+ tracedRename,
+ tracedRmSync,
+ tracedUnlinkSync,
+ tracedWriteFile,
+} from '../fs-traced.ts';
+import { getLogger } from '../logger.ts';
+
+const log = getLogger('embeddings');
+
+const MANIFEST_SCHEMA_VERSION = 1;
+const VEC_SUBDIR = 'vec';
+const MANIFEST_NAME = 'manifest.json';
+
+interface ManifestEntry {
+ contentHash: string;
+ mtimeMs: number;
+}
+
+interface ManifestFile {
+ schemaVersion: number;
+ providerId: string;
+ modelId: string;
+ dims: number;
+ chunkConfigId: string;
+ entries: Record<string, ManifestEntry>;
+}
+
+interface VectorCacheOptions {
+ cacheDir: string | null;
+ providerId: string;
+ modelId: string;
+ dims: number;
+ chunkConfigId: string;
+}
+
+export function hashContent(content: string): string {
+ return createHash('sha256').update(content, 'utf-8').digest('hex');
+}
+
+function serializeVectors(vectors: readonly Float32Array[]): Uint8Array {
+ let total = 0;
+ for (const v of vectors) total += v.length;
+ const packed = new Float32Array(total);
+ let offset = 0;
+ for (const v of vectors) {
+ packed.set(v, offset);
+ offset += v.length;
+ }
+ return new Uint8Array(packed.buffer, packed.byteOffset, packed.byteLength);
+}
+
+function deserializeVectors(bytes: Buffer, dims: number): Float32Array[] {
+ const aligned = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength);
+ const floats = new Float32Array(aligned);
+ if (floats.length % dims !== 0) {
+ throw new Error(`vector blob length ${floats.length} is not a multiple of dims ${dims}`);
+ }
+ const chunks: Float32Array[] = [];
+ for (let i = 0; i < floats.length; i += dims) {
+ chunks.push(floats.slice(i, i + dims));
+ }
+ return chunks;
+}
+
+export class VectorCache {
+ private readonly cacheDir: string | null;
+ private readonly vecDir: string | null;
+ private readonly manifestPath: string | null;
+ readonly providerId: string;
+ readonly modelId: string;
+ readonly dims: number;
+ readonly chunkConfigId: string;
+
+ private readonly entries = new Map<string, ManifestEntry>();
+ private readonly vectorsByHash = new Map<string, Float32Array[]>();
+ private readonly persistedHashes = new Set<string>();
+ private dirty = false;
+
+ constructor(options: VectorCacheOptions) {
+ this.cacheDir = options.cacheDir;
+ this.vecDir = options.cacheDir ? join(options.cacheDir, VEC_SUBDIR) : null;
+ this.manifestPath = options.cacheDir ? join(options.cacheDir, MANIFEST_NAME) : null;
+ this.providerId = options.providerId;
+ this.modelId = options.modelId;
+ this.dims = options.dims;
+ this.chunkConfigId = options.chunkConfigId;
+ }
+
+ async init(): Promise<void> {
+ if (!this.cacheDir || !this.manifestPath || !this.vecDir) return;
+ let manifest: ManifestFile | null = null;
+ try {
+ if (existsSync(this.manifestPath)) {
+ manifest = JSON.parse(await readFile(this.manifestPath, 'utf-8')) as ManifestFile;
+ }
+ } catch (err) {
+ log.warn(
+ { err: err instanceof Error ? err.message : String(err) },
+ '[embeddings] unreadable cache manifest — rebuilding',
+ );
+ manifest = null;
+ }
+
+ const identityMatches =
+ manifest !== null &&
+ manifest.schemaVersion === MANIFEST_SCHEMA_VERSION &&
+ manifest.providerId === this.providerId &&
+ manifest.modelId === this.modelId &&
+ manifest.dims === this.dims &&
+ manifest.chunkConfigId === this.chunkConfigId;
+
+ if (!identityMatches) {
+ if (manifest !== null) {
+ log.info(
+ { hadModel: manifest.modelId, wantModel: this.modelId },
+ '[embeddings] cache identity changed (provider/model/dims/chunking) — invalidating',
+ );
+ }
+ this.wipeDisk();
+ return;
+ }
+ if (!manifest) return; // unreachable once identity matched; narrows for TS
+
+ for (const [docId, entry] of Object.entries(manifest.entries)) {
+ if (!entry?.contentHash) continue;
+ this.entries.set(docId, { contentHash: entry.contentHash, mtimeMs: entry.mtimeMs ?? 0 });
+ if (!this.vectorsByHash.has(entry.contentHash)) {
+ try {
+ const blobPath = join(this.vecDir, `${entry.contentHash}.bin`);
+ if (existsSync(blobPath)) {
+ const bytes = await readFile(blobPath);
+ this.vectorsByHash.set(entry.contentHash, deserializeVectors(bytes, this.dims));
+ this.persistedHashes.add(entry.contentHash);
+ }
+ } catch (err) {
+ log.warn(
+ { hash: entry.contentHash, err: err instanceof Error ? err.message : String(err) },
+ '[embeddings] corrupt vector blob — will re-embed',
+ );
+ }
+ }
+ }
+ }
+
+ isFresh(docId: string, mtimeMs: number): boolean {
+ const entry = this.entries.get(docId);
+ return (
+ entry !== undefined && entry.mtimeMs === mtimeMs && this.vectorsByHash.has(entry.contentHash)
+ );
+ }
+
+ link(docId: string, contentHash: string, mtimeMs: number): boolean {
+ if (!this.vectorsByHash.has(contentHash)) return false;
+ const prev = this.entries.get(docId);
+ if (!prev || prev.contentHash !== contentHash || prev.mtimeMs !== mtimeMs) this.dirty = true;
+ this.entries.set(docId, { contentHash, mtimeMs });
+ return true;
+ }
+
+ store(docId: string, contentHash: string, mtimeMs: number, vectors: Float32Array[]): void {
+ this.vectorsByHash.set(contentHash, vectors);
+ this.entries.set(docId, { contentHash, mtimeMs });
+ this.dirty = true;
+ }
+
+ getVectors(docId: string): Float32Array[] | undefined {
+ const entry = this.entries.get(docId);
+ if (!entry) return undefined;
+ return this.vectorsByHash.get(entry.contentHash);
+ }
+
+ get embeddedCount(): number {
+ let n = 0;
+ for (const entry of this.entries.values()) {
+ const v = this.vectorsByHash.get(entry.contentHash);
+ if (v && v.length > 0) n += 1;
+ }
+ return n;
+ }
+
+ retain(activeDocIds: ReadonlySet<string>): void {
+ for (const docId of this.entries.keys()) {
+ if (!activeDocIds.has(docId)) {
+ this.entries.delete(docId);
+ this.dirty = true;
+ }
+ }
+ const referenced = new Set<string>();
+ for (const entry of this.entries.values()) referenced.add(entry.contentHash);
+ for (const hash of this.vectorsByHash.keys()) {
+ if (!referenced.has(hash)) this.vectorsByHash.delete(hash);
+ }
+ }
+
+ clearMemory(): void {
+ this.entries.clear();
+ this.vectorsByHash.clear();
+ this.persistedHashes.clear();
+ this.dirty = false;
+ }
+
+ async persist(): Promise<void> {
+ if (!this.cacheDir || !this.manifestPath || !this.vecDir) return;
+ if (!this.dirty) return; // nothing changed since last persist — skip the write
+ try {
+ await tracedMkdir(this.vecDir, { recursive: true });
+ const referenced = new Set<string>();
+ for (const entry of this.entries.values()) referenced.add(entry.contentHash);
+
+ for (const hash of referenced) {
+ if (this.persistedHashes.has(hash)) continue;
+ const vectors = this.vectorsByHash.get(hash);
+ if (!vectors) continue;
+ await tracedWriteFile(join(this.vecDir, `${hash}.bin`), serializeVectors(vectors));
+ this.persistedHashes.add(hash);
+ }
+
+ const manifest: ManifestFile = {
+ schemaVersion: MANIFEST_SCHEMA_VERSION,
+ providerId: this.providerId,
+ modelId: this.modelId,
+ dims: this.dims,
+ chunkConfigId: this.chunkConfigId,
+ entries: Object.fromEntries(this.entries),
+ };
+ const tmp = `${this.manifestPath}.tmp`;
+ await tracedWriteFile(tmp, JSON.stringify(manifest));
+ await tracedRename(tmp, this.manifestPath);
+
+ for (const file of readdirSync(this.vecDir)) {
+ if (!file.endsWith('.bin')) continue;
+ const hash = file.slice(0, -'.bin'.length);
+ if (!referenced.has(hash)) {
+ tracedUnlinkSync(join(this.vecDir, file));
+ this.persistedHashes.delete(hash);
+ }
+ }
+ this.dirty = false;
+ } catch (err) {
+ log.warn(
+ { err: err instanceof Error ? err.message : String(err) },
+ '[embeddings] failed to persist vector cache',
+ );
+ }
+ }
+
+ private wipeDisk(): void {
+ if (!this.cacheDir) return;
+ try {
+ tracedRmSync(this.cacheDir, { recursive: true, force: true });
+ } catch (err) {
+ log.warn(
+ { err: err instanceof Error ? err.message : String(err) },
+ '[embeddings] failed to wipe stale cache',
+ );
+ }
+ }
+}
diff --git a/packages/server/src/index.ts b/packages/server/src/index.ts
index af3a20ca4..cf975a79f 100644
--- a/packages/server/src/index.ts
+++ b/packages/server/src/index.ts
@@ -99,6 +99,20 @@ export {
type DetectClaudeDesktopOptions,
detectClaudeDesktopPresence,
} from './detect-claude-desktop.ts';
+export {
+ clearEmbeddingsKeyFromAllBackends,
+ createEmbeddingsSecretStore,
+ DEFAULT_EMBEDDINGS_DIMENSIONS,
+ describeStoredEmbeddingsKey,
+ EMBEDDINGS_API_KEY_ENV,
+ type EmbeddingsKeyReader,
+ type EmbeddingsKeyStore,
+ type EmbeddingsSecretStore,
+ FileEmbeddingsBackend,
+ makeLazyEmbeddingsKeyStore,
+ type ResolvedSemanticConfig,
+ readProjectLocalSemanticConfig,
+} from './embeddings/index.ts';
export {
applyExternalChange,
createExternalChangeHandler,
diff --git a/packages/server/src/mcp/tools/search.test.ts b/packages/server/src/mcp/tools/search.test.ts
index 496d138da..8e1b00a0c 100644
--- a/packages/server/src/mcp/tools/search.test.ts
+++ b/packages/server/src/mcp/tools/search.test.ts
@@ -99,7 +99,7 @@ describe('search MCP tool — registration', () => {
expect(DESCRIPTION.slice(0, 200)).toContain('grep');
});
- test('inputSchema exposes query, intent, scopes, limit, cwd', () => {
+ test('inputSchema exposes query, intent, scopes, limit, semantic, cwd', () => {
const { server, registered } = makeFakeServer();
register(server, {
resolveCwd: async () => '/tmp/proj',
@@ -113,6 +113,7 @@ describe('search MCP tool — registration', () => {
'limit',
'query',
'scopes',
+ 'semantic',
]);
});
});
@@ -175,6 +176,7 @@ describe('search MCP tool — happy path', () => {
intent: 'full_text',
limit: 5,
scopes: ['page', 'content'],
+ semantic: true,
});
const structured = result.structuredContent as {
@@ -229,6 +231,72 @@ describe('search MCP tool — happy path', () => {
expect(body.limit).toBe(20);
});
+ test('semantic:false is forwarded as the per-call lexical override', async () => {
+ const captured: { body?: string } = {};
+ globalThis.fetch = mock(async (_url: string, init?: RequestInit) => {
+ captured.body = String(init?.body);
+ return new Response(JSON.stringify({ ok: true, results: [] }), {
+ status: 200,
+ headers: { 'content-type': 'application/json' },
+ });
+ }) as unknown as typeof fetch;
+
+ const { server, registered } = makeFakeServer();
+ register(server, {
+ resolveCwd: async () => '/tmp/proj',
+ config: DEFAULT_CONFIG,
+ serverUrl: 'http://localhost:1234',
+ });
+ const tool = expectOneRegisteredTool(registered);
+ await tool.handler({ query: 'q', semantic: false, cwd: '/tmp/proj' });
+ const body = JSON.parse(String(captured.body)) as Record<string, unknown>;
+ expect(body.semantic).toBe(false);
+ });
+
+ test('passes through signals.vector + the non-content semantic coverage block', async () => {
+ mockFetchOk({
+ ok: true,
+ query: 'auth retries',
+ intent: 'full_text',
+ results: [
+ {
+ kind: 'page',
+ path: 'guides/credential-rotation',
+ title: 'Credential Rotation',
+ score: 12.3,
+ signals: { lexical: 0, fullText: 0, recency: 0, vector: 0.82 },
+ },
+ ],
+ semantic: { capable: true, applied: true, coverage: { embedded: 12, total: 40 } },
+ elapsedMs: 5,
+ });
+ const { server, registered } = makeFakeServer();
+ register(server, {
+ resolveCwd: async () => '/tmp/proj',
+ config: DEFAULT_CONFIG,
+ serverUrl: 'http://localhost:1234',
+ });
+ const tool = expectOneRegisteredTool(registered);
+ const result = await tool.handler({ query: 'auth retries', cwd: '/tmp/proj' });
+ const structured = result.structuredContent as {
+ results: Array<{ signals: { vector?: number } }>;
+ semantic?: {
+ capable: boolean;
+ applied: boolean;
+ coverage: { embedded: number; total: number };
+ };
+ };
+ expect(structured.results[0]?.signals.vector).toBeCloseTo(0.82, 5);
+ expect(structured.semantic).toEqual({
+ capable: true,
+ applied: true,
+ coverage: { embedded: 12, total: 40 },
+ });
+ const text = result.content?.find((c) => c.type === 'text')?.text ?? '';
+ expect(text).toContain('Semantic:');
+ expect(text).toContain('12/40');
+ });
+
test('zero results returns "No matches" text + structured.resultCount = 0', async () => {
mockFetchOk({ ok: true, query: 'nope', intent: 'full_text', results: [], elapsedMs: 0.1 });
const { server, registered } = makeFakeServer();
diff --git a/packages/server/src/mcp/tools/search.ts b/packages/server/src/mcp/tools/search.ts
index ca011ca19..5f63fd1b0 100644
--- a/packages/server/src/mcp/tools/search.ts
+++ b/packages/server/src/mcp/tools/search.ts
@@ -19,13 +19,16 @@ import {
export const DESCRIPTION = [
'[Requires: Hocuspocus server] Ranked page retrieval for a query (title boost + body BM25 + recency — the cmd-K engine). For literal search across every line, use `exec` (`grep`) instead.',
'',
- 'Returns scored page (and folder, with `omnibar` intent) hits, each with a body snippet. `exec`-grep covers every occurrence and needs no server.',
+ 'When semantic search is enabled for the workspace (an opt-in setting with an API key), an embeddings signal is additionally fused into `full_text` ranking, surfacing conceptually-related pages that share no keywords. This tool opts in by default; the `semantic` block in the response reports coverage. Note: with semantic enabled, the query and matching page content are sent to the configured embeddings provider (content egress). Set `semantic: false` to force pure-lexical ranking for a call.',
+ '',
+ 'Returns scored page (and folder, with `omnibar` intent) hits, each with a body snippet and a `signals` breakdown (lexical / fullText / recency / vector). `exec`-grep covers every occurrence and needs no server.',
'',
'**Parameters:**',
'- `query` — Free-form; tokenized across title, name, path segments, and (with `full_text`) body.',
'- `intent` (optional) — `omnibar` searches title/path/folders only (fast); `full_text` includes body. Default `full_text`.',
'- `scopes` (optional) — Result scope: `page` | `folder` | `content`. Defaults derive from `intent`.',
'- `limit` (optional) — Max rows; default 20, max 100.',
+ '- `semantic` (optional) — Set `false` to force pure-lexical ranking even when semantic search is enabled. Omit to use semantic when available.',
'',
'If the server is down, the tool returns a recovery hint — use `exec("grep ...")` as the server-free fallback.',
].join('\n');
@@ -54,6 +57,12 @@ const InputSchema = {
"Override the default scope set. Members: 'page', 'folder', 'content'. Defaults derive from intent.",
),
limit: z.number().int().min(1).max(100).optional().describe('Max rows; default 20, max 100.'),
+ semantic: z
+ .boolean()
+ .optional()
+ .describe(
+ 'Set false to force pure-lexical ranking for this call even when semantic search is enabled. Omit to use semantic when available.',
+ ),
cwd: z.string().optional().describe(ROUTED_CWD_DESCRIPTION),
} as const;
@@ -67,12 +76,19 @@ const SearchResultRowSchema = z.object({
lexical: z.number(),
fullText: z.number(),
recency: z.number(),
+ vector: z.number().optional(),
}),
snippet: z.string().optional(),
previewUrl: z.string().nullable(),
previewUrlSource: z.enum(PREVIEW_URL_SOURCES).optional(),
});
+const SearchSemanticStatusSchema = z.object({
+ capable: z.boolean(),
+ applied: z.boolean(),
+ coverage: z.object({ embedded: z.number().int(), total: z.number().int() }),
+});
+
const OutputSchema = outputSchemaWithText({
cwd: z.string(),
query: z.string(),
@@ -80,6 +96,7 @@ const OutputSchema = outputSchemaWithText({
resultCount: z.number().int(),
results: z.array(SearchResultRowSchema),
elapsedMs: z.number().nullable(),
+ semantic: SearchSemanticStatusSchema.optional(),
});
type SearchKind = 'page' | 'folder';
@@ -89,10 +106,16 @@ interface SearchApiRow {
path?: string;
title?: string | null;
score?: number;
- signals?: { lexical?: number; fullText?: number; recency?: number };
+ signals?: { lexical?: number; fullText?: number; recency?: number; vector?: number };
snippet?: string;
}
+interface SearchApiSemanticStatus {
+ capable?: boolean;
+ applied?: boolean;
+ coverage?: { embedded?: number; total?: number };
+}
+
interface SearchApiResponse {
ok: boolean;
error?: string;
@@ -100,6 +123,7 @@ interface SearchApiResponse {
intent?: string;
results?: SearchApiRow[];
elapsedMs?: number;
+ semantic?: SearchApiSemanticStatus;
[key: string]: unknown;
}
@@ -109,12 +133,18 @@ interface SearchResultRow {
docName: string;
title: string | null;
score: number;
- signals: { lexical: number; fullText: number; recency: number };
+ signals: { lexical: number; fullText: number; recency: number; vector?: number };
snippet?: string;
previewUrl: string | null;
previewUrlSource?: PreviewUrlSource;
}
+interface SearchSemanticStatus {
+ capable: boolean;
+ applied: boolean;
+ coverage: { embedded: number; total: number };
+}
+
interface SearchStructuredResult {
cwd: string;
query: string;
@@ -122,6 +152,7 @@ interface SearchStructuredResult {
resultCount: number;
results: SearchResultRow[];
elapsedMs: number | null;
+ semantic?: SearchSemanticStatus;
}
function isSearchKind(value: unknown): value is SearchKind {
@@ -132,14 +163,45 @@ function normalizeSignals(signals: SearchApiRow['signals']): {
lexical: number;
fullText: number;
recency: number;
+ vector?: number;
} {
return {
lexical: typeof signals?.lexical === 'number' ? signals.lexical : 0,
fullText: typeof signals?.fullText === 'number' ? signals.fullText : 0,
recency: typeof signals?.recency === 'number' ? signals.recency : 0,
+ ...(typeof signals?.vector === 'number' ? { vector: signals.vector } : {}),
};
}
+function normalizeSemanticStatus(
+ semantic: SearchApiSemanticStatus | undefined,
+): SearchSemanticStatus | undefined {
+ if (!semantic || typeof semantic.capable !== 'boolean') return undefined;
+ return {
+ capable: semantic.capable,
+ applied: semantic.applied === true,
+ coverage: {
+ embedded: typeof semantic.coverage?.embedded === 'number' ? semantic.coverage.embedded : 0,
+ total: typeof semantic.coverage?.total === 'number' ? semantic.coverage.total : 0,
+ },
+ };
+}
+
+function formatSemanticNote(semantic: SearchSemanticStatus | undefined): string {
+ if (!semantic) return '';
+ if (!semantic.capable) {
+ return '> Semantic: enabled but not ready (no API key, or warming up) — lexical ranking only.';
+ }
+ const { embedded, total } = semantic.coverage;
+ if (semantic.applied) {
+ return `> Semantic: on — vector signal contributed (${embedded}/${total} pages embedded).`;
+ }
+ if (embedded < total) {
+ return `> Semantic: on — indexing ${embedded}/${total} pages; vectors are still filling in, re-run for fuller coverage.`;
+ }
+ return '> Semantic: on — no page cleared the similarity threshold for this query.';
+}
+
function formatResultsBlock(results: SearchResultRow[]): string {
if (results.length === 0) return '';
const lines: string[] = [];
@@ -171,6 +233,7 @@ export function register(server: ServerInstance, deps: SearchDeps): void {
intent?: (typeof INTENT_VALUES)[number];
scopes?: Array<(typeof SCOPE_VALUES)[number]>;
limit?: number;
+ semantic?: boolean;
cwd?: string;
}) => {
try {
@@ -191,7 +254,12 @@ export function register(server: ServerInstance, deps: SearchDeps): void {
const intent = args.intent ?? 'full_text';
const limit = args.limit ?? 20;
- const body: Record<string, unknown> = { query: args.query, intent, limit };
+ const body: Record<string, unknown> = {
+ query: args.query,
+ intent,
+ limit,
+ semantic: args.semantic ?? true,
+ };
if (args.scopes) body.scopes = args.scopes;
const result = (await httpPost(url, '/api/search', body)) as SearchApiResponse;
@@ -220,6 +288,7 @@ export function register(server: ServerInstance, deps: SearchDeps): void {
];
});
+ const semantic = normalizeSemanticStatus(result.semantic);
const structured: SearchStructuredResult = {
cwd,
query: args.query,
@@ -227,13 +296,16 @@ export function register(server: ServerInstance, deps: SearchDeps): void {
resultCount: rows.length,
results: rows,
elapsedMs: typeof result.elapsedMs === 'number' ? result.elapsedMs : null,
+ ...(semantic ? { semantic } : {}),
};
const header = `## Search results for "${args.query}" (${rows.length} hit${rows.length === 1 ? '' : 's'}, intent: ${intent})`;
- const text =
+ const semanticNote = formatSemanticNote(semantic);
+ const resultsText =
rows.length === 0
? `No matches for "${args.query}".`
: `${header}\n\n${formatResultsBlock(rows)}`;
+ const text = semanticNote ? `${resultsText}\n${semanticNote}` : resultsText;
return textPlusStructured(text, structured);
} catch (err) {
diff --git a/packages/server/src/server-factory.ts b/packages/server/src/server-factory.ts
index f14380899..6a8391825 100644
--- a/packages/server/src/server-factory.ts
+++ b/packages/server/src/server-factory.ts
@@ -46,6 +46,17 @@ import { isDocInConflict } from './conflict-errors.ts';
import { createConflictLifecycleSeedExtension } from './conflict-lifecycle-seed.ts';
import { type ContentFilter, createContentFilter } from './content-filter.ts';
import { getDocExtension, stripDocExtension } from './doc-extensions.ts';
+import {
+ DEFAULT_EMBEDDINGS_DIMENSIONS,
+ type Embedder,
+ type EmbeddingsKeyStore,
+ loadOpenAiEmbedder,
+ normalizeProviderId,
+ type ResolvedSemanticConfig,
+ readProjectLocalSemanticConfig,
+ SemanticSearchService,
+ secretsFilePath,
+} from './embeddings/index.ts';
import {
applyDiskContentToDoc,
applyExternalChange,
@@ -151,6 +162,8 @@ export interface ServerOptions {
detectGh?: DetectGhFn;
tokenStore?: ProbeTokenStore | null;
checkPushPermissionFn?: (opts: CheckPushPermissionOptions) => Promise<PushPermission>;
+ embeddingsKeyStore?: EmbeddingsKeyStore | null;
+ embedderLoader?: () => Promise<Embedder | null>;
singleDocRelPath?: string;
ephemeral?: boolean;
}
@@ -233,6 +246,17 @@ export function createServer(options: ServerOptions): ServerInstance {
return project.value.autoSync?.enabled === true;
}
+ function readSemanticSearchConfig(): ResolvedSemanticConfig {
+ return readProjectLocalSemanticConfig(projectDir, {
+ configHomedirOverride,
+ onWarn: (message) => log.warn({ message }, '[config] could not read project-local config'),
+ });
+ }
+
+ function semanticProviderFingerprint(cfg: ResolvedSemanticConfig): string {
+ return `${normalizeProviderId(cfg.baseUrl)}|${cfg.model}|${cfg.dimensions ?? DEFAULT_EMBEDDINGS_DIMENSIONS}`;
+ }
+
initTelemetry();
const serverInstanceId = randomUUID();
@@ -292,6 +316,23 @@ export function createServer(options: ServerOptions): ServerInstance {
let agentFocusBroadcaster: AgentFocusBroadcaster | null = null;
let agentPresenceBroadcaster: AgentPresenceBroadcaster | null = null;
let invalidateReferencedAssetsCache: (() => void) | null = null;
+
+ const initialSemanticConfig = readSemanticSearchConfig();
+ const semanticSearch = new SemanticSearchService({
+ loadEmbedder:
+ options.embedderLoader ??
+ (() => {
+ const cfg = readSemanticSearchConfig();
+ return loadOpenAiEmbedder({
+ keyStore: options.embeddingsKeyStore ?? null,
+ config: { baseUrl: cfg.baseUrl, model: cfg.model, dimensions: cfg.dimensions },
+ });
+ }),
+ cacheDir: join(getLocalDir(projectDir), 'embeddings'),
+ enabled: initialSemanticConfig.enabled,
+ providerFingerprint: semanticProviderFingerprint(initialSemanticConfig),
+ });
+
let loadedPrincipal: Principal | null = null;
const forceUnloadSet = new Set<Document>();
let shutdownAllowsUnload = false;
@@ -625,6 +666,8 @@ export function createServer(options: ServerOptions): ServerInstance {
ready,
recentlyRemovedDocs,
serializeDoc,
+ semanticSearch,
+ embeddingsSecretsFile: secretsFilePath(configHomedirOverride),
ephemeral,
onReferencedAssetsCacheInvalidator: (invalidate) => {
invalidateReferencedAssetsCache = invalidate;
@@ -1636,6 +1679,11 @@ export function createServer(options: ServerOptions): ServerInstance {
log.warn({ err, enabled }, '[sync] failed to apply autoSync.enabled from config');
});
}
+ const semCfg = readSemanticSearchConfig();
+ semanticSearch.applyConfig({
+ enabled: semCfg.enabled,
+ providerFingerprint: semanticProviderFingerprint(semCfg),
+ });
});
configFileWatcherCleanups.push({ docName: configDocName, cleanup });
log.info({ docName: configDocName, path: absPath }, '[config-file-watcher] started');