Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

Keep MCP saves off the derived-index startup queue (PRD-8839) (#5672)

Andrew Mikofalvy31b04d2d1bf4
diff --git a/.changeset/prd-8839-saves-during-index-startup.md b/.changeset/prd-8839-saves-during-index-startup.md
new file mode 100644
index 000000000..d5203a242
--- /dev/null
+++ b/.changeset/prd-8839-saves-during-index-startup.md
@@ -0,0 +1,5 @@
+---
+"@inkeep/open-knowledge": patch
+---
+
+Edits and deletes made through MCP right after the server starts now reach disk immediately, instead of waiting minutes for the link index to finish building. Until it finishes, write results carry a `link-check-deferred` warning in place of link checks.
diff --git a/docs/content/advanced/content-rules/overview.mdx b/docs/content/advanced/content-rules/overview.mdx
index 41617d0f5..c5f1ad23f 100644
--- a/docs/content/advanced/content-rules/overview.mdx
+++ b/docs/content/advanced/content-rules/overview.mdx
@@ -242,7 +242,7 @@ Every write response carries validation findings for the document it touched, on
**`warnings`** carries up to 10 findings across the whole validation plane — lint violations and broken links alike, honoring the project's `validation.links` setting and, for a reserved log, the **Ignore broken links in log.md** switch. Positions are 1-based (`line`/`column`), ready to echo back to a human, and a `links` finding adds `linkTarget`. The field is present only when the write produced findings.
-**`brokenLinks`** is the dedicated link channel, and unlike `warnings` it is **always present**. An empty array is the positive "every outbound link resolves" confirmation, which saves a separate `links({ kind: "dead" })` round-trip, as long as no `brokenLinkSuppression` sits beside it. Each entry names the href exactly as authored, so an agent can grep for it. `reason` is `no-such-doc` (resolved to a docName that doesn't exist), `no-such-file` (a linked asset or source file missing from disk), or `unresolvable` (an empty href, or a relative path escaping the content root); `resolvedTo` is `null` for `unresolvable`.
+**`brokenLinks`** is the dedicated link channel, and unlike `warnings` it is **always present**. An empty array is the positive "every outbound link resolves" confirmation, which saves a separate `links({ kind: "dead" })` round-trip, as long as no `brokenLinkSuppression` sits beside it and `warnings` carries no `link-check-deferred` entry. That entry means the server was still building its link index after starting, so link checks were skipped. That document's links are checked again only when it is written or edited after startup finishes, or by an `audit`, which waits for startup and can time out until then. Each entry names the href exactly as authored, so an agent can grep for it. `reason` is `no-such-doc` (resolved to a docName that doesn't exist), `no-such-file` (a linked asset or source file missing from disk), or `unresolvable` (an empty href, or a relative path escaping the content root); `resolvedTo` is `null` for `unresolvable`.
**`brokenLinkSuppression`** is present only when the project's reserved-log policy kept findings out of `brokenLinks`, which is what makes an empty array conditional rather than a clean bill of health:
diff --git a/docs/content/reference/agentic-search.mdx b/docs/content/reference/agentic-search.mdx
index 83c3bb751..1785e4995 100644
--- a/docs/content/reference/agentic-search.mdx
+++ b/docs/content/reference/agentic-search.mdx
@@ -240,7 +240,7 @@ set("ok");
</script>
```
-`brokenLinks` comes back on every write and edit (empty when they all resolve, unless a `brokenLinkSuppression` beside it reports that the project's reserved-log policy withheld some), so the agent never needs a follow-up dead-link check. A brand-new doc nothing points to comes back flagged as an orphan with a hub to link it from. Every write and edit is versioned, attributed, and the preview updates as it lands. Append to a file by hand and the edit still lands: the file watcher folds it into the live doc and versions it, but anonymously, as `file-system`, and nothing talks back. [Skills](../features/skills/index.mdx) teach the conventions, and the write path enforces the connective tissue.
+`brokenLinks` comes back on every write and edit (empty when they all resolve, unless a `brokenLinkSuppression` beside it reports that the project's reserved-log policy withheld some, or a `link-check-deferred` warning reports that the server was still starting and checked nothing yet), so the agent never needs a follow-up dead-link check. A brand-new doc nothing points to comes back flagged as an orphan with a hub to link it from. Every write and edit is versioned, attributed, and the preview updates as it lands. Append to a file by hand and the edit still lands: the file watcher folds it into the live doc and versions it, but anonymously, as `file-system`, and nothing talks back. [Skills](../features/skills/index.mdx) teach the conventions, and the write path enforces the connective tissue.
## The index is authored, not extracted
diff --git a/docs/content/reference/mcp.mdx b/docs/content/reference/mcp.mdx
index 8b404635d..a9fe22003 100644
--- a/docs/content/reference/mcp.mdx
+++ b/docs/content/reference/mcp.mdx
@@ -48,7 +48,7 @@ The write verbs nest their result under the same target key you wrote, mirroring
## Advisory warnings
-`write` and `edit` results may carry a `warnings` array on the document entry (per entry for a batch); `restore_version` carries it at the top level. Entries are discriminated by `kind`. `content-divergence` and `disk-edit-reconciled` are write-integrity signals — the stored doc differs from what the call composed, so re-read it before editing further. `mermaid-parse-error` means the write landed but that fence won't render — fix the fence and re-edit.
+`write` and `edit` results may carry a `warnings` array on the document entry (per entry for a batch); `restore_version` carries it at the top level. Entries are discriminated by `kind`. `content-divergence` and `disk-edit-reconciled` are write-integrity signals — the stored doc differs from what the call composed, so re-read it before editing further. `mermaid-parse-error` means the write landed but that fence won't render — fix the fence and re-edit. `lint-violation` reports a content-rule finding at a line and column. `link-check-deferred` means the server was still building its link index after starting, so links were not checked and an empty `brokenLinks` is not an all-clear; that document's links are checked again only when it is written or edited after startup finishes, or by an `audit`, which can time out until then.
`delete`, `move` and `install` on a `skill` target carry a separate `warnings` array of plain strings, not `kind`-discriminated entries. It reports work that did not undo the operation: bookkeeping records the server could not update after the skill itself was already deleted, renamed or moved, and — on a delete — that the directory removed was one an earlier failed move had retained for recovery. The operation succeeded; the warning names what to check or repair.
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index 815681f84..f8c334d6c 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -1377,6 +1377,8 @@ export {
isWriteWarningKind,
type LifecycleStatus,
LifecycleStatusSchema,
+ type LinkCheckDeferredWarning,
+ LinkCheckDeferredWarningSchema,
type LinkGraphDocNode,
LinkGraphDocNodeSchema,
type LinkGraphEdge,
diff --git a/packages/core/src/schemas/api/agent-write.ts b/packages/core/src/schemas/api/agent-write.ts
index 649054b26..6c94dc158 100644
--- a/packages/core/src/schemas/api/agent-write.ts
+++ b/packages/core/src/schemas/api/agent-write.ts
@@ -186,11 +186,20 @@ export const LintViolationWarningSchema = z
.loose() satisfies StandardSchemaV1;
export type LintViolationWarning = z.infer<typeof LintViolationWarningSchema>;
+export const LinkCheckDeferredWarningSchema = z
+ .object({
+ kind: z.literal('link-check-deferred'),
+ message: z.string(),
+ })
+ .loose() satisfies StandardSchemaV1;
+export type LinkCheckDeferredWarning = z.infer<typeof LinkCheckDeferredWarningSchema>;
+
export const AdvisoryWarningSchema = z.discriminatedUnion('kind', [
ContentDivergenceWarningSchema,
DiskEditReconciledWarningSchema,
RenderWarningSchema,
LintViolationWarningSchema,
+ LinkCheckDeferredWarningSchema,
]);
export type AdvisoryWarning = z.infer<typeof AdvisoryWarningSchema>;
diff --git a/packages/server/assets/skills/project/SKILL.md b/packages/server/assets/skills/project/SKILL.md
index ecbfc0bb3..a8d0a8182 100644
--- a/packages/server/assets/skills/project/SKILL.md
+++ b/packages/server/assets/skills/project/SKILL.md
@@ -91,7 +91,7 @@ KB docs are factual artifacts: every claim traceable, and **the source lives ins
- **No evidence?** Search and ingest the result, OR mark `(TODO: needs source)`, OR don't write the claim. Do NOT fabricate — unsourced speculation rots into untraceable tribal lore.
## Linking — standard markdown links (MUST)
-Link every noun-phrase that names another document — `[text](./relative/path.md)` — and link liberally. To point at one section of a long doc, in chat replies too, link `./path.md#heading-slug`. **Every link must resolve to a doc that exists by the time you're done** (a same-pass forward-reference you create later in the pass is fine; for one that genuinely won't exist, leave the mention as plain prose + a tracked task). Never backtick a link (`` `[text](./foo.md)` `` is a bug) and never use HTML `<a>`. **After every `write`/`edit`, read `brokenLinks`: fix reported `href`s; `[]` means all links resolve unless `brokenLinkSuppression` withheld reserved-log findings; those are not yours to repair.** `audit` is authoritative; its same marker carries the same meaning. External web sources are NOT inline body links (see Grounding). Full rule set, slug rule + the `[[Page]]` legacy note: `references/linking.md`.
+Link every noun-phrase that names another document — `[text](./relative/path.md)` — and link liberally. To point at one section of a long doc, in chat replies too, link `./path.md#heading-slug`. **Every link must resolve to a doc that exists by the time you're done** (a same-pass forward-reference you create later in the pass is fine; for one that genuinely won't exist, leave the mention as plain prose + a tracked task). Never backtick a link (`` `[text](./foo.md)` `` is a bug) and never use HTML `<a>`. **After every `write`/`edit`, read `brokenLinks`: fix reported `href`s; `[]` means all links resolve unless `brokenLinkSuppression` withheld reserved-log findings (not yours to repair) or a `link-check-deferred` warning says nothing was checked yet (that doc is re-checked only by a later write/edit of it or an `audit`, which may time out until startup finishes).** `audit` is authoritative; its same marker carries the same meaning. External web sources are NOT inline body links (see Grounding). Full rule set, slug rule + the `[[Page]]` legacy note: `references/linking.md`.
## Folders, frontmatter, templates
diff --git a/packages/server/assets/skills/project/references/linking.md b/packages/server/assets/skills/project/references/linking.md
index c3679e105..0344fe4ac 100644
--- a/packages/server/assets/skills/project/references/linking.md
+++ b/packages/server/assets/skills/project/references/linking.md
@@ -10,7 +10,7 @@
- **Every link must resolve to a doc that exists by the time you're done.** Within a single multi-doc authoring pass, linking a page you'll create later in the same pass is fine — `brokenLinks` reports it as `no-such-doc` until the target lands, an expected transient forward-reference rather than a wrong-path error. What's not acceptable is *leaving* a dead link behind: create the target in the same pass, or record it as a tracked task (`TaskCreate`, or your host's task tool — if the host has none, tell the user) and leave the mention as plain prose.
- **Never wrap a link in backticks.** `` `[text](./foo.md)` `` is a bug — the backticks make it render as literal code rather than a link.
- **Never use HTML anchors** (`<a href="...">`). Markdown link syntax only.
-- **`brokenLinks` on the write/edit response is your primary check — read it before walking away.** Every `write`/`edit` returns `brokenLinks` in the SAME payload: `[]` means every outbound link resolves only when `brokenLinkSuppression` is absent; a populated list names each broken `href` with its `resolvedTo` + `reason` (`no-such-doc` = resolved to a doc that doesn't exist; `no-such-file` = resolved to a linked asset / source file that isn't on disk at that path; `unresolvable` = the path escapes the content root, usually one `../` too many). This validates **every** local link, not just doc links: a wrong-depth `[src](../../foo.py)` to a source file is caught too. It is report-only — the write landed regardless; fix or remove every one in a follow-up `edit`.
+- **`brokenLinks` on the write/edit response is your primary check — read it before walking away.** Every `write`/`edit` returns `brokenLinks` in the SAME payload: `[]` means every outbound link resolves only when `brokenLinkSuppression` is absent and `warnings` has no `link-check-deferred` entry (that entry means the server was still starting and checked nothing — that doc's links are re-checked only when you write or edit it again after startup, or by an `audit`, which can time out until then); a populated list names each broken `href` with its `resolvedTo` + `reason` (`no-such-doc` = resolved to a doc that doesn't exist; `no-such-file` = resolved to a linked asset / source file that isn't on disk at that path; `unresolvable` = the path escapes the content root, usually one `../` too many). This validates **every** local link, not just doc links: a wrong-depth `[src](../../foo.py)` to a source file is caught too. It is report-only — the write landed regardless; fix or remove every one in a follow-up `edit`.
- **An empty `brokenLinks` alongside a `brokenLinkSuppression` does not mean every link resolves.** That field appears only when the project's reserved-log policy withheld findings, and it carries `{ reason: "reserved-log-policy", count: N }` and nothing else. No href comes back because none of them is repair work: the write landed in a lowercase-stemmed `log.md` or `log.mdx`, an append-only history whose entries deliberately reference pages that moved or were never written. **Never rewrite a reserved log's existing entries to clear a link.** Append new entries instead, and treat `links({ kind: "dead" })` on that file as a deliberate audit of raw state, never as a repair queue. The project can turn the policy off (**Settings ▸ This project ▸ Preferences ▸ Content rules**), in which case those findings come back through the normal channels.
- **`audit` is the authoritative end-state link check** — run it once at the end of a multi-doc pass. Read `brokenLinkSuppression` on its result too: when present, the reserved-log policy withheld `count` findings, so a clean result is filtered rather than proof that every raw link resolves. Do not repair those log findings; use `links({ kind: "dead" })` only when deliberately inspecting raw state. Caveat: `audit` reports links only while the project's `validation.links` setting is not `off` (at `off` it returns clean with no warning), so if the project has silenced link validation, fall back to `links({ kind: "dead" })` — the unconditional read of the dead-link graph, which also lists skill-bundle docs excluded from the Problems plane (optionally scoped: `links({ kind: "dead", sourceDocuments: ["your/doc"] })`). Otherwise `links` is the graph-navigation reader, supplemental to `audit`, not a substitute. Companion `links` kinds: `backlinks` (incoming), `forward` (outgoing), `orphans` (no incoming), `hubs` (high-incoming), `suggest` (untextualized mentions worth linking).
- **The tool and the editor agree.** Both resolve a bare name the same way — exact, then slug, then folder index, then basename — so a link that opens when you click it is never reported dead, and a link reported dead is genuinely unresolvable. A target that names an asset, or names nothing at all, stays silent rather than being reported as a broken doc link.
diff --git a/packages/server/src/api-derived-index-warmup-durability.test.ts b/packages/server/src/api-derived-index-warmup-durability.test.ts
new file mode 100644
index 000000000..148d68587
--- /dev/null
+++ b/packages/server/src/api-derived-index-warmup-durability.test.ts
@@ -0,0 +1,208 @@
+import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs';
+import type { IncomingMessage, ServerResponse } from 'node:http';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { Readable } from 'node:stream';
+import simpleGit from 'simple-git';
+import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest';
+import { BacklinkIndex } from './backlink-index.ts';
+import { _resetDocExtensionsForTests } from './doc-extensions.ts';
+import { createServer } from './server-factory.ts';
+
+interface ApiExtensionLike {
+ priority?: number;
+ onRequest?: (ctx: { request: IncomingMessage; response: ServerResponse }) => Promise<void>;
+}
+
+interface CapturedResponse {
+ status: number;
+ body: string;
+}
+
+const DEFERRED = expect.objectContaining({ kind: 'link-check-deferred' });
+
+function settleWithin<T>(promise: Promise<T>, label: string, ms = 5_000): Promise<T> {
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const deadline = new Promise<never>((_, reject) => {
+ timer = setTimeout(() => reject(new Error(`${label} did not settle within ${ms}ms`)), ms);
+ });
+ return Promise.race([promise, deadline]).finally(() => clearTimeout(timer));
+}
+
+async function post(
+ server: ReturnType<typeof createServer>,
+ url: string,
+ body: unknown,
+): Promise<CapturedResponse> {
+ const apiExt = server.hocuspocus.configuration.extensions.find(
+ (e): e is ApiExtensionLike =>
+ (e as ApiExtensionLike).priority === 100 &&
+ typeof (e as ApiExtensionLike).onRequest === 'function',
+ );
+ if (!apiExt?.onRequest) throw new Error('API extension (priority 100) not found on server');
+ const request = Readable.from(Buffer.from(JSON.stringify(body))) as unknown as IncomingMessage;
+ request.method = 'POST';
+ request.url = url;
+ request.headers = { host: 'localhost', 'content-type': 'application/json' };
+ const captured: CapturedResponse = { status: 0, body: '' };
+ const response = {
+ writeHead(status: number) {
+ captured.status = status;
+ },
+ setHeader() {},
+ end(text?: string) {
+ captured.body = text ?? '';
+ },
+ } as unknown as ServerResponse;
+ if (!(await server.nativeApi.dispatch(request, response))) {
+ await apiExt.onRequest({ request, response });
+ }
+ return captured;
+}
+
+describe('agent writes while the derived index is still starting (PRD-8839)', () => {
+ let tmpDir: string;
+ let server: ReturnType<typeof createServer>;
+ let releaseStartup: () => void;
+
+ beforeEach(async () => {
+ _resetDocExtensionsForTests();
+ tmpDir = mkdtempSync(join(tmpdir(), 'ok-warmup-durability-'));
+ const git = simpleGit({ baseDir: tmpDir });
+ await git.init();
+ await git.addConfig('user.name', 'Test User');
+ await git.addConfig('user.email', '[email protected]');
+ const startupStalled = new Promise<void>((resolve) => {
+ releaseStartup = resolve;
+ });
+ const rebuild = vi
+ .spyOn(BacklinkIndex.prototype, 'rebuildFromDisk')
+ .mockReturnValue(startupStalled);
+ server = createServer({
+ contentDir: tmpDir,
+ projectDir: tmpDir,
+ quiet: true,
+ debounce: 100,
+ maxDebounce: 500,
+ gitEnabled: false,
+ });
+ await vi.waitFor(() => expect(rebuild).toHaveBeenCalled(), { timeout: 10_000 });
+ });
+
+ afterEach(async () => {
+ releaseStartup();
+ await server.ready;
+ vi.restoreAllMocks();
+ await server.destroy();
+ _resetDocExtensionsForTests();
+ rmSync(tmpDir, { recursive: true, force: true });
+ });
+
+ test('an edit after a write reaches disk, and both respond without link advisories', async () => {
+ const filePath = join(tmpDir, 'warmup-edit.md');
+
+ const write = await settleWithin(
+ post(server, '/api/agent-write-md', {
+ docName: 'warmup-edit',
+ markdown: '# Warm-up\n\nline one. See [[nowhere]].\n',
+ position: 'replace',
+ }),
+ 'agent-write-md',
+ );
+ expect(write.status).toBe(200);
+ expect(readFileSync(filePath, 'utf-8')).toBe('# Warm-up\n\nline one. See [[nowhere]].\n');
+ expect(JSON.parse(write.body)).toMatchObject({ brokenLinks: [], warnings: [DEFERRED] });
+
+ const edit = await settleWithin(
+ post(server, '/api/agent-patch', {
+ docName: 'warmup-edit',
+ find: 'line one.',
+ replace: 'line one, edited.',
+ }),
+ 'agent-patch',
+ );
+ expect(edit.status).toBe(200);
+ expect(readFileSync(filePath, 'utf-8')).toBe(
+ '# Warm-up\n\nline one, edited. See [[nowhere]].\n',
+ );
+ expect(JSON.parse(edit.body)).toMatchObject({ brokenLinks: [], warnings: [DEFERRED] });
+ });
+
+ test('frontmatter patches and batch writes mark their link check as deferred', async () => {
+ await settleWithin(
+ post(server, '/api/agent-write-md', {
+ docName: 'warmup-frontmatter',
+ markdown: '# Tagged\n\nSee [[nowhere]].\n',
+ position: 'replace',
+ }),
+ 'agent-write-md',
+ );
+
+ const frontmatter = await settleWithin(
+ post(server, '/api/frontmatter-patch', {
+ docName: 'warmup-frontmatter',
+ patch: { status: 'draft' },
+ }),
+ 'frontmatter-patch',
+ );
+ expect(frontmatter.status).toBe(200);
+ expect(readFileSync(join(tmpDir, 'warmup-frontmatter.md'), 'utf-8')).toContain('status: draft');
+ expect(JSON.parse(frontmatter.body)).toMatchObject({ brokenLinks: [], warnings: [DEFERRED] });
+
+ const batch = await settleWithin(
+ post(server, '/api/agent-write-batch', {
+ docs: [{ docName: 'warmup-batch', markdown: '# Batch\n\nSee [[nowhere]].\n' }],
+ }),
+ 'agent-write-batch',
+ );
+ expect(batch.status).toBe(200);
+ expect(readFileSync(join(tmpDir, 'warmup-batch.md'), 'utf-8')).toBe(
+ '# Batch\n\nSee [[nowhere]].\n',
+ );
+ expect(JSON.parse(batch.body).results).toEqual([
+ expect.objectContaining({ status: 'written', brokenLinks: [], warnings: [DEFERRED] }),
+ ]);
+ });
+
+ test('delete-path removes an open document and responds', async () => {
+ const filePath = join(tmpDir, 'warmup-delete.md');
+ await settleWithin(
+ post(server, '/api/agent-write-md', {
+ docName: 'warmup-delete',
+ markdown: '# Doomed\n',
+ position: 'replace',
+ }),
+ 'agent-write-md',
+ );
+ expect(existsSync(filePath)).toBe(true);
+
+ const deleted = await settleWithin(
+ post(server, '/api/delete-path', { kind: 'file', path: 'warmup-delete.md' }),
+ 'delete-path',
+ );
+
+ expect(deleted.status).toBe(200);
+ expect(existsSync(filePath)).toBe(false);
+ });
+
+ test('duplicate-path copies a document and responds', async () => {
+ await settleWithin(
+ post(server, '/api/agent-write-md', {
+ docName: 'warmup-original',
+ markdown: '# Original\n',
+ position: 'replace',
+ }),
+ 'agent-write-md',
+ );
+
+ const duplicated = await settleWithin(
+ post(server, '/api/duplicate-path', { kind: 'file', path: 'warmup-original.md' }),
+ 'duplicate-path',
+ );
+
+ expect(duplicated.status).toBe(200);
+ const { duplicatedDocNames } = JSON.parse(duplicated.body) as { duplicatedDocNames: string[] };
+ expect(duplicatedDocNames).toHaveLength(1);
+ expect(readFileSync(join(tmpDir, `${duplicatedDocNames[0]}.md`), 'utf-8')).toBe('# Original\n');
+ });
+});
diff --git a/packages/server/src/api-extension.test-helper.ts b/packages/server/src/api-extension.test-helper.ts
index 61874d62d..66a3f0303 100644
--- a/packages/server/src/api-extension.test-helper.ts
+++ b/packages/server/src/api-extension.test-helper.ts
@@ -37,6 +37,7 @@ export function createDerivedDocumentIndexApiPortStub(
async recordDirectDelete() {},
async recordDirectRename() {},
async recordLinkRewrite() {},
+ isReady: () => true,
async getBacklinks() {
return [];
},
@@ -87,6 +88,7 @@ function createLegacyDerivedIndexPort(
backlinkIndex: BacklinkIndex | null | undefined,
tagIndex: TagIndex | null | undefined,
signalChannel: LegacySignalChannel | undefined,
+ pendingWrites: Set<Promise<unknown>>,
): DerivedDocumentIndexApiPort | undefined {
if (!backlinkIndex && !tagIndex) return undefined;
@@ -109,7 +111,15 @@ function createLegacyDerivedIndexPort(
backlinkIndex?.deleteDocument(documentName);
tagIndex?.deleteDocument(documentName);
};
- const recordDirectMutations = async (
+ const recordDirectMutations = (
+ mutations: readonly DerivedDocumentIndexMutation[],
+ ): Promise<void> => {
+ const write = applyDirectMutations(mutations);
+ pendingWrites.add(write);
+ void write.finally(() => pendingWrites.delete(write)).catch(() => {});
+ return write;
+ };
+ const applyDirectMutations = async (
mutations: readonly DerivedDocumentIndexMutation[],
): Promise<void> => {
if (mutations.length === 0) return;
@@ -153,6 +163,7 @@ function createLegacyDerivedIndexPort(
return {
recordDirectMutations,
+ isReady: () => true,
testOnly: {
async resetDocumentForTest(documentName) {
deleteAll(documentName);
@@ -261,6 +272,7 @@ export function createApiExtension(
}),
...apiOptions
} = options;
+ const pendingLegacyWrites = new Set<Promise<unknown>>();
const extension = createApiExtensionBase({
...apiOptions,
getProjectConfigEpoch,
@@ -271,15 +283,22 @@ export function createApiExtension(
durabilityState: new DocumentDurabilityState(),
derivedDocumentIndex:
derivedDocumentIndex ??
- createLegacyDerivedIndexPort(backlinkIndex, tagIndex, options.signalChannel),
+ createLegacyDerivedIndexPort(
+ backlinkIndex,
+ tagIndex,
+ options.signalChannel,
+ pendingLegacyWrites,
+ ),
});
const legacyOnRequest = extension.onRequest?.bind(extension);
return {
...extension,
async onRequest(payload: { request: IncomingMessage; response: ServerResponse }) {
- if (await extension.nativeApi.dispatch(payload.request, payload.response)) return;
- // biome-ignore lint/suspicious/noExplicitAny: Hocuspocus `onRequest` has no exported payload type
- await legacyOnRequest?.(payload as any);
+ if (!(await extension.nativeApi.dispatch(payload.request, payload.response))) {
+ // biome-ignore lint/suspicious/noExplicitAny: Hocuspocus `onRequest` has no exported payload type
+ await legacyOnRequest?.(payload as any);
+ }
+ await Promise.allSettled(pendingLegacyWrites);
},
};
}
diff --git a/packages/server/src/api-extension.ts b/packages/server/src/api-extension.ts
index e1f0ef2f1..23d46c853 100644
--- a/packages/server/src/api-extension.ts
+++ b/packages/server/src/api-extension.ts
@@ -303,7 +303,9 @@ import {
isPeerAdmitted,
} from './ingress-policy.ts';
import {
+ deferredWriteLinkAdvisory,
type LinkAdvisoryPolicy,
+ type PrepareWriteLinkAdvisory,
projectWriteAdvisoryLinks,
type WriteLinkAdvisoryProjection,
} from './link-advisory-policy.ts';
@@ -1759,7 +1761,7 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {
async function computeOrphanHints(
docName: string,
): Promise<Array<{ type: 'orphan'; parentCandidates: string[]; message: string }> | undefined> {
- if (!derivedDocumentIndex) return undefined;
+ if (!derivedDocumentIndex?.isReady()) return undefined;
try {
const backlinks = await derivedDocumentIndex.getBacklinks(docName);
if (backlinks.length > 0) return undefined;
@@ -1889,6 +1891,29 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {
return contentFilter.isExcluded(relPath);
}
+ function isDerivedIndexReady(): boolean {
+ return derivedDocumentIndex?.isReady() ?? true;
+ }
+
+ const prepareWriteLinkAdvisory: PrepareWriteLinkAdvisory = async (
+ writtenDocNames,
+ { resolveFolderLinks },
+ ) => {
+ if (!isDerivedIndexReady()) return deferredWriteLinkAdvisory;
+ const admitted = await collectAdmittedDocNames();
+ for (const docName of writtenDocNames) admitted.add(docName);
+ const linkedFileExists = createLinkedFileExists();
+ const linkedFolderExists = resolveFolderLinks ? createLinkedFolderExists() : undefined;
+ return (source, docName, suppressLogLinkAdvisories) => ({
+ links: projectWriteAdvisoryLinks(
+ computeWriteAdvisoryLinks(source, docName, admitted, linkedFileExists, linkedFolderExists),
+ docName,
+ suppressLogLinkAdvisories,
+ ),
+ warnings: [],
+ });
+ };
+
async function collectAdmittedDocNames(): Promise<Set<string>> {
const admitted = new Set<string>();
for (const [docName, entry] of getFileIndex()) {
@@ -3842,12 +3867,10 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {
}
}
- const admittedForLinks = await collectAdmittedDocNames();
- for (const p of pending) {
- if (flushErrors.get(p.docName) === undefined) admittedForLinks.add(p.docName);
- }
- const linkedFileExists = createLinkedFileExists();
- const linkedFolderExists = createLinkedFolderExists();
+ const linkAdvisor = await prepareWriteLinkAdvisory(
+ pending.filter((p) => flushErrors.get(p.docName) === undefined).map((p) => p.docName),
+ { resolveFolderLinks: true },
+ );
let lastWrittenDoc: string | undefined;
for (const p of pending) {
@@ -3858,22 +3881,18 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {
}
const writtenSource = p.session.dc.document.getText('source').toString();
registerWrittenDocInFileIndex(p.docName, writtenSource);
+ const linkAdvisory = linkAdvisor(
+ writtenSource,
+ p.docName,
+ linkPolicy.suppressLogLinkAdvisories,
+ );
+ const warnings = [...p.warnings, ...linkAdvisory.warnings];
results[p.index] = {
status: 'written',
docName: p.docName,
...(p.summaryResponse ? { summary: p.summaryResponse } : {}),
- ...(p.warnings.length > 0 ? { warnings: p.warnings } : {}),
- ...projectWriteAdvisoryLinks(
- computeWriteAdvisoryLinks(
- writtenSource,
- p.docName,
- admittedForLinks,
- linkedFileExists,
- linkedFolderExists,
- ),
- p.docName,
- linkPolicy.suppressLogLinkAdvisories,
- ),
+ ...(warnings.length > 0 ? { warnings } : {}),
+ ...linkAdvisory.links,
};
lastWrittenDoc = p.docName;
}
@@ -4741,7 +4760,11 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {
const lintFindings = await lintDocument(source, effective, docName);
let linkFindings: ValidationDiagnostic[] = [];
- if (derivedDocumentIndex && linkPolicy.links !== 'off' && !isLinkIndexExcludedDoc(docName)) {
+ if (
+ derivedDocumentIndex?.isReady() &&
+ linkPolicy.links !== 'off' &&
+ !isLinkIndexExcludedDoc(docName)
+ ) {
await recordDerivedLinkRewriteBestEffort(docName, source, 'lint-validation');
const linksValidator = createProjectValidators({
projectDir: projectDir ?? contentDir,
@@ -5276,9 +5299,7 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {
onAgentWrite,
computeOrphanHints,
registerWrittenDocInFileIndex,
- collectAdmittedDocNames,
- createLinkedFileExists,
- createLinkedFolderExists,
+ prepareWriteLinkAdvisory,
buildReconcileWarning,
computeLintViolations,
log,
diff --git a/packages/server/src/derived-document-index.ts b/packages/server/src/derived-document-index.ts
index 4919be324..5a310746d 100644
--- a/packages/server/src/derived-document-index.ts
+++ b/packages/server/src/derived-document-index.ts
@@ -24,6 +24,7 @@ import { getLogger } from './logger.ts';
import { TagIndex, type TagSummaryEntry } from './tag-index.ts';
const log = getLogger('derived-document-index');
+const SLOW_STARTUP_LOG_THRESHOLD_MS = 10_000;
const DERIVED_INDEX_SAVE_DEBOUNCE_MS = 2000;
const LOCAL_TARGET_REBUILD_RETRY_BASE_MS = 1000;
@@ -96,6 +97,7 @@ export interface DerivedDocumentIndexApiPort {
markdown: string,
): Promise<void>;
recordLinkRewrite(documentName: string, markdown: string): Promise<void>;
+ isReady(): boolean;
getBacklinks(documentName: string): Promise<BacklinkEntry[]>;
getBacklinkCount(documentName: string): Promise<number>;
getBacklinkCounts(documentNames: readonly string[]): Promise<Record<string, number>>;
@@ -199,7 +201,9 @@ export class DerivedDocumentIndex
private saveTagsOnNextDebounce = false;
private branchTransition: BranchTransition | null = null;
private startupBegun = false;
+ private startupBegunAt = 0;
private startupSettled = false;
+ private ready = false;
private liveUpdateToken = 0;
private lastSignaledLocalTargetGeneration = 0;
private closed = false;
@@ -255,6 +259,7 @@ export class DerivedDocumentIndex
throw new Error('Derived document index startup has already begun');
}
this.startupBegun = true;
+ this.startupBegunAt = performance.now();
this.startupBranch = branch;
this.backlinkIndex.switchBranch(branch);
@@ -298,7 +303,15 @@ export class DerivedDocumentIndex
return { tagIndexDegraded };
});
} finally {
+ this.ready = true;
this.readyBarrier.resolve();
+ const startupMs = Math.round(performance.now() - this.startupBegunAt);
+ const logSettled = startupMs >= SLOW_STARTUP_LOG_THRESHOLD_MS ? log.info : log.debug;
+ logSettled.call(
+ log,
+ { startupMs },
+ '[derived-index] startup settled; deferred link checks resume',
+ );
}
}
@@ -560,6 +573,10 @@ export class DerivedDocumentIndex
this.signalAllRelations();
}
+ isReady(): boolean {
+ return this.ready && !this.closed;
+ }
+
getBacklinks(documentName: string): Promise<BacklinkEntry[]> {
return this.runQuery(() => this.backlinkIndex.getBacklinks(documentName));
}
diff --git a/packages/server/src/http/agent-write-routes.test.ts b/packages/server/src/http/agent-write-routes.test.ts
index 5416dc972..07cecf11d 100644
--- a/packages/server/src/http/agent-write-routes.test.ts
+++ b/packages/server/src/http/agent-write-routes.test.ts
@@ -51,9 +51,7 @@ function buildGroup(overrides: Partial<Deps> = {}) {
onAgentWrite: undefined,
computeOrphanHints: notDispatched,
registerWrittenDocInFileIndex: notDispatched,
- collectAdmittedDocNames: notDispatched,
- createLinkedFileExists: notDispatched,
- createLinkedFolderExists: notDispatched,
+ prepareWriteLinkAdvisory: notDispatched,
buildReconcileWarning: notDispatched,
computeLintViolations: notDispatched,
log: loggerFactory.getLogger('test'),
diff --git a/packages/server/src/http/agent-write-routes.ts b/packages/server/src/http/agent-write-routes.ts
index eac1bb442..72fce3b72 100644
--- a/packages/server/src/http/agent-write-routes.ts
+++ b/packages/server/src/http/agent-write-routes.ts
@@ -73,13 +73,12 @@ import {
reconcileDiskBeforeAgentWrite,
} from '../external-change.ts';
import { extractActorIdentity } from '../extract-actor-identity.ts';
-import type { FileIndexEntry } from '../file-watcher.ts';
import {
FrontmatterMalformedError,
respondFrontmatterMalformed,
} from '../frontmatter-malformed-error.ts';
import { recordFrontmatterEditSurface } from '../frontmatter-telemetry.ts';
-import { type LinkAdvisoryPolicy, projectWriteAdvisoryLinks } from '../link-advisory-policy.ts';
+import type { LinkAdvisoryPolicy, PrepareWriteLinkAdvisory } from '../link-advisory-policy.ts';
import { getLogger } from '../logger.ts';
import { validateMermaidFences } from '../mermaid-validator.ts';
import { incrementAgentPatchFindMismatches, incrementAgentWriteCalls } from '../metrics.ts';
@@ -98,7 +97,6 @@ import {
type WriterIdentity,
} from '../shadow-repo.ts';
import { getMeter, withSpanSync } from '../telemetry.ts';
-import { computeWriteAdvisoryLinks } from '../write-advisory-links.ts';
import { type ApiRouteGroup, createApiRouteGroup } from './api-pipeline.ts';
import { errorResponse } from './error-response.ts';
import { getRequestId } from './request-id.ts';
@@ -201,11 +199,7 @@ export interface AgentWriteRouteDeps {
docName: string,
) => Promise<Array<{ type: 'orphan'; parentCandidates: string[]; message: string }> | undefined>;
registerWrittenDocInFileIndex: (docName: string, content: string) => void;
- collectAdmittedDocNames: () => Promise<Set<string>>;
- createLinkedFileExists: (
- allFiles?: ReadonlyMap<string, FileIndexEntry>,
- ) => (contentRootRelativePath: string) => boolean;
- createLinkedFolderExists: () => (folderPath: string) => boolean;
+ prepareWriteLinkAdvisory: PrepareWriteLinkAdvisory;
buildReconcileWarning: (
reconcile: ReconcileBeforeWriteResult,
) => DiskEditReconciledWarning | undefined;
@@ -255,9 +249,7 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
onAgentWrite,
computeOrphanHints,
registerWrittenDocInFileIndex,
- collectAdmittedDocNames,
- createLinkedFileExists,
- createLinkedFolderExists,
+ prepareWriteLinkAdvisory,
buildReconcileWarning,
computeLintViolations,
log,
@@ -272,6 +264,7 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
stripDefaultPathTruncation,
renameAttributionCounter,
} = deps;
+
const versionOpsService = createVersionOpsService({ getCurrentBranch, contentRoot });
function getSubscriberCount(docName: string): number {
@@ -452,19 +445,9 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
const writtenSource = session.dc.document.getText('source').toString();
registerWrittenDocInFileIndex(resolvedDocName, writtenSource);
const renderWarnings = await validateMermaidFences(writtenSource, resolvedDocName);
- const admittedForLinks = await collectAdmittedDocNames();
- admittedForLinks.add(resolvedDocName);
- const linkAdvisory = projectWriteAdvisoryLinks(
- computeWriteAdvisoryLinks(
- writtenSource,
- resolvedDocName,
- admittedForLinks,
- createLinkedFileExists(),
- createLinkedFolderExists(),
- ),
- resolvedDocName,
- linkPolicy.suppressLogLinkAdvisories,
- );
+ const linkAdvisory = (
+ await prepareWriteLinkAdvisory([resolvedDocName], { resolveFolderLinks: true })
+ )(writtenSource, resolvedDocName, linkPolicy.suppressLogLinkAdvisories);
const subscriberCount = getSubscriberCount(resolvedDocName);
const systemSubscriberCount = getSystemSubscriberCount();
if (systemSubscriberCount === 0) {
@@ -485,6 +468,7 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
resolvedDocName,
linkPolicy,
)),
+ ...linkAdvisory.warnings,
];
successResponse(
res,
@@ -497,7 +481,7 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
...(hints ? { hints } : {}),
...(summaryResponse ? { summary: summaryResponse } : {}),
...(writeMdAdvisories.length > 0 ? { warnings: writeMdAdvisories } : {}),
- ...linkAdvisory,
+ ...linkAdvisory.links,
},
{ handler: 'agent-write-md' },
);
@@ -744,18 +728,14 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
resolvedDocName,
session.dc.document.getText('source').toString(),
);
- const admittedForLinks = await collectAdmittedDocNames();
- admittedForLinks.add(resolvedDocName);
- const linkAdvisory = projectWriteAdvisoryLinks(
- computeWriteAdvisoryLinks(
- session.dc.document.getText('source').toString(),
- resolvedDocName,
- admittedForLinks,
- createLinkedFileExists(),
- ),
+ const linkAdvisory = (
+ await prepareWriteLinkAdvisory([resolvedDocName], { resolveFolderLinks: false })
+ )(
+ session.dc.document.getText('source').toString(),
resolvedDocName,
linkPolicy.suppressLogLinkAdvisories,
);
+ const fmAdvisories = [...(fmWarning ? [fmWarning] : []), ...linkAdvisory.warnings];
successResponse(
res,
200,
@@ -766,8 +746,8 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
systemSubscriberCount,
appliedKeys,
...(summaryResponse ? { summary: summaryResponse } : {}),
- ...(fmWarning ? { warnings: [fmWarning] } : {}),
- ...linkAdvisory,
+ ...(fmAdvisories.length > 0 ? { warnings: fmAdvisories } : {}),
+ ...linkAdvisory.links,
},
{ handler: 'frontmatter-patch' },
);
@@ -1062,18 +1042,9 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
const patchedSource = session.dc.document.getText('source').toString();
registerWrittenDocInFileIndex(docName, patchedSource);
const renderWarnings = await validateMermaidFences(patchedSource, docName);
- const admittedForLinks = await collectAdmittedDocNames();
- admittedForLinks.add(docName);
- const linkAdvisory = projectWriteAdvisoryLinks(
- computeWriteAdvisoryLinks(
- patchedSource,
- docName,
- admittedForLinks,
- createLinkedFileExists(),
- ),
- docName,
- linkPolicy.suppressLogLinkAdvisories,
- );
+ const linkAdvisory = (
+ await prepareWriteLinkAdvisory([docName], { resolveFolderLinks: false })
+ )(patchedSource, docName, linkPolicy.suppressLogLinkAdvisories);
const patchWarning = buildReconcileWarning(patchReconcile);
const patchDivergenceEntry =
patchDivergence !== undefined ? toContentDivergenceWarning(patchDivergence) : undefined;
@@ -1086,6 +1057,7 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
docName,
linkPolicy,
)),
+ ...linkAdvisory.warnings,
];
successResponse(
res,
@@ -1097,7 +1069,7 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroup
systemSubscriberCount,
...(summaryResponse ? { summary: summaryResponse } : {}),
...(patchAdvisories.length > 0 ? { warnings: patchAdvisories } : {}),
- ...linkAdvisory,
+ ...linkAdvisory.links,
},
{ handler: 'agent-patch' },
);
diff --git a/packages/server/src/link-advisory-policy.ts b/packages/server/src/link-advisory-policy.ts
index cc8638bbb..5449998f5 100644
--- a/packages/server/src/link-advisory-policy.ts
+++ b/packages/server/src/link-advisory-policy.ts
@@ -1,6 +1,7 @@
import {
type BrokenLinkSuppression,
isReservedLogDoc,
+ type LinkCheckDeferredWarning,
type LinksValidationSetting,
} from '@inkeep/open-knowledge-core';
import { createReservedLogBrokenLinkSuppression } from './broken-link-suppression.ts';
@@ -18,11 +19,37 @@ export function shouldSuppressLogLinkAdvisories(
return suppressLogLinkAdvisories && isReservedLogDoc(sourceDocName);
}
+const LINK_CHECK_DEFERRED_WARNING: LinkCheckDeferredWarning = {
+ kind: 'link-check-deferred',
+ message:
+ 'Link checks were skipped because the link index is still building after server start, so an empty `brokenLinks` does not mean every link resolves. Links in this document are checked again only when it is written or edited after startup finishes, or by an `audit`, which waits for startup and can time out until then.',
+};
+
export interface WriteLinkAdvisoryProjection {
brokenLinks: WriteAdvisoryLink[];
brokenLinkSuppression?: BrokenLinkSuppression;
}
+export interface WriteLinkAdvisory {
+ links: WriteLinkAdvisoryProjection;
+ warnings: LinkCheckDeferredWarning[];
+}
+
+type WriteLinkAdvisor = (
+ source: string,
+ docName: string,
+ suppressLogLinkAdvisories: boolean,
+) => WriteLinkAdvisory;
+
+export type PrepareWriteLinkAdvisory = (
+ writtenDocNames: Iterable<string>,
+ options: { resolveFolderLinks: boolean },
+) => Promise<WriteLinkAdvisor>;
+
+export function deferredWriteLinkAdvisory(): WriteLinkAdvisory {
+ return { links: { brokenLinks: [] }, warnings: [LINK_CHECK_DEFERRED_WARNING] };
+}
+
export function projectWriteAdvisoryLinks(
detected: WriteAdvisoryLink[],
sourceDocName: string,
diff --git a/packages/server/src/live-derived-index.test.ts b/packages/server/src/live-derived-index.test.ts
index db4b2688d..382d2ea0f 100644
--- a/packages/server/src/live-derived-index.test.ts
+++ b/packages/server/src/live-derived-index.test.ts
@@ -189,6 +189,35 @@ describe('createLiveDerivedIndexExtension', () => {
await conn.disconnect();
});
+ test('beforeUnloadDocument resolves while the flushed update is still queued', async () => {
+ const recordLiveDocument = vi.fn(() => new Promise<void>(() => {}));
+ const extension = createLiveDerivedIndexExtension({
+ derivedDocumentIndex: createDerivedIndexPort(recordLiveDocument),
+ debounceMs: 20,
+ });
+ const conn = await hp.openDirectConnection('unload-queued-doc');
+ const doc = getDoc(conn);
+
+ applyExternalChange(durabilityState, hp, 'unload-queued-doc', '# Queued\n');
+ await extension.onChange?.(
+ makeOnChangePayload(hp, doc, 'unload-queued-doc', {
+ source: 'local',
+ context: { origin: 'agent-write' },
+ }),
+ );
+ const unload = extension.beforeUnloadDocument?.({
+ document: doc,
+ documentName: 'unload-queued-doc',
+ instance: hp,
+ });
+
+ await expect(
+ Promise.race([unload, wait(1_000).then(() => Promise.reject(new Error('unload blocked')))]),
+ ).resolves.toBeUndefined();
+ expect(recordLiveDocument).toHaveBeenCalledTimes(1);
+ await conn.disconnect();
+ });
+
test('onDestroy clears pending timers across documents', async () => {
const recordLiveDocument = vi.fn(async () => {});
const extension = createLiveDerivedIndexExtension({
diff --git a/packages/server/src/live-derived-index.ts b/packages/server/src/live-derived-index.ts
index 66b33d96d..4f2c6f68c 100644
--- a/packages/server/src/live-derived-index.ts
+++ b/packages/server/src/live-derived-index.ts
@@ -1,6 +1,9 @@
import type { Document, Extension } from '@hocuspocus/server';
import { isLinkIndexExcludedDoc } from './cc1-broadcast.ts';
-import type { DerivedDocumentIndexLivePort } from './derived-document-index.ts';
+import {
+ type DerivedDocumentIndexLivePort,
+ isDerivedDocumentIndexClosedError,
+} from './derived-document-index.ts';
import { getLogger } from './logger.ts';
export const LIVE_DERIVED_INDEX_DEBOUNCE_MS = 100;
@@ -63,6 +66,13 @@ export function createLiveDerivedIndexExtension(options: LiveDerivedIndexOptions
onDocumentSettled?.(docName);
})
.catch((err) => {
+ if (isDerivedDocumentIndexClosedError(err)) {
+ getLogger('live-derived-index').debug(
+ { docName, err },
+ `Derived index closed before the live update for ${docName} ran`,
+ );
+ return;
+ }
getLogger('live-derived-index').error(
{ docName, err },
`Failed to update derived views for ${docName}`,
@@ -70,12 +80,12 @@ export function createLiveDerivedIndexExtension(options: LiveDerivedIndexOptions
});
}
- async function flushPending(docName: string): Promise<void> {
+ function flushPending(docName: string): void {
const pending = pendingByDoc.get(docName);
if (!pending) return;
clearTimeout(pending.timer);
pendingByDoc.delete(docName);
- await runUpdate(docName, pending.document, pending.token);
+ void runUpdate(docName, pending.document, pending.token);
}
function schedule(docName: string, document: Document, token: number): void {
@@ -111,7 +121,7 @@ export function createLiveDerivedIndexExtension(options: LiveDerivedIndexOptions
},
async beforeUnloadDocument({ documentName }) {
- await flushPending(documentName);
+ flushPending(documentName);
},
async onDestroy() {
diff --git a/packages/server/src/mcp/tools/advisory-warnings.test.ts b/packages/server/src/mcp/tools/advisory-warnings.test.ts
index 381582d1c..00c910546 100644
--- a/packages/server/src/mcp/tools/advisory-warnings.test.ts
+++ b/packages/server/src/mcp/tools/advisory-warnings.test.ts
@@ -1,6 +1,7 @@
import {
type AdvisoryWarning,
type BrokenLink,
+ type LinkCheckDeferredWarning,
type LintViolationWarning,
type RenderWarning,
WRITE_WARNING_KINDS,
@@ -314,6 +315,24 @@ describe('content-rule (lint) violations', () => {
});
});
+describe('deferred link check', () => {
+ const deferred: LinkCheckDeferredWarning = {
+ kind: 'link-check-deferred',
+ message: 'Links in this document were not checked.',
+ };
+
+ test('parses and relays the server message instead of the unrecognized-kind fallback', () => {
+ expect(parseAdvisoryWarnings([deferred])).toEqual([deferred]);
+ expect(formatAdvisoryLines([deferred])).toEqual(['⚠ Links in this document were not checked.']);
+ });
+
+ test('formatAdvisoryBriefs emits one brief however many entries arrive', () => {
+ expect(formatAdvisoryBriefs([deferred, deferred])).toEqual([
+ '⚠ Links not checked yet: the link index is still building (see warnings).',
+ ]);
+ });
+});
+
describe('unrecognized-kind fallback', () => {
const future = { kind: 'future-advisory-kind', detail: 42 } as unknown as AdvisoryWarning;
diff --git a/packages/server/src/mcp/tools/advisory-warnings.ts b/packages/server/src/mcp/tools/advisory-warnings.ts
index c2e9eb17c..415197953 100644
--- a/packages/server/src/mcp/tools/advisory-warnings.ts
+++ b/packages/server/src/mcp/tools/advisory-warnings.ts
@@ -7,6 +7,7 @@ import {
type BrokenLinkSuppression,
BrokenLinkSuppressionSchema,
isWriteWarningKind,
+ type LinkCheckDeferredWarning,
type LintViolationWarning,
type RenderWarning,
WRITE_WARNING_KINDS,
@@ -71,10 +72,15 @@ function lintEntries(warnings: AdvisoryWarning[]): LintViolationWarning[] {
return warnings.filter((w): w is LintViolationWarning => w.kind === 'lint-violation');
}
+function linkCheckDeferredEntries(warnings: AdvisoryWarning[]): LinkCheckDeferredWarning[] {
+ return warnings.filter((w): w is LinkCheckDeferredWarning => w.kind === 'link-check-deferred');
+}
+
const RELAYED_KINDS: ReadonlySet<string> = new Set<string>([
...WRITE_WARNING_KINDS,
'mermaid-parse-error',
'lint-violation',
+ 'link-check-deferred',
]);
function unrecognizedEntries(warnings: AdvisoryWarning[]): AdvisoryWarning[] {
@@ -103,6 +109,7 @@ export function formatAdvisoryLines(warnings: AdvisoryWarning[]): string[] {
const render = renderEntries(warnings);
if (render.length > 0) lines.push(formatRenderWarningsLine(render));
lines.push(...lintEntries(warnings).map(formatLintLine));
+ lines.push(...linkCheckDeferredEntries(warnings).map((d) => `⚠ ${d.message}`));
lines.push(...unrecognizedEntries(warnings).map(formatUnrecognizedLine));
return lines;
}
@@ -112,6 +119,9 @@ export function formatAdvisoryBriefs(warnings: AdvisoryWarning[]): string[] {
const render = renderEntries(warnings);
if (render.length > 0) briefs.push(formatRenderWarningsBrief(render));
briefs.push(...lintEntries(warnings).map(formatLintBrief));
+ if (linkCheckDeferredEntries(warnings).length > 0) {
+ briefs.push('⚠ Links not checked yet: the link index is still building (see warnings).');
+ }
briefs.push(...unrecognizedEntries(warnings).map(formatUnrecognizedBrief));
return briefs;
}
diff --git a/packages/server/src/mcp/tools/edit.ts b/packages/server/src/mcp/tools/edit.ts
index c544352e6..a5432f606 100644
--- a/packages/server/src/mcp/tools/edit.ts
+++ b/packages/server/src/mcp/tools/edit.ts
@@ -656,7 +656,7 @@ export function register(server: ServerInstance, deps: EditDeps): void {
.object(documentResultBaseShape)
.optional()
.describe(
- 'Document edit result. Always present on a successful document edit (body or frontmatter) — it carries `brokenLinks` (possibly `[]`) plus any `brokenLinkSuppression`/`summary`/`warnings`. Read `brokenLinkSuppression` before concluding anything from an empty `brokenLinks`: when it is present, a project policy withheld findings and none of them is yours to repair. Absent only for folder/template edits.',
+ 'Document edit result. Always present on a successful document edit (body or frontmatter) — it carries `brokenLinks` (possibly `[]`) plus any `brokenLinkSuppression`/`summary`/`warnings`. Read `brokenLinkSuppression` and `warnings` before concluding anything from an empty `brokenLinks`: a suppression means a project policy withheld findings that are not yours to repair; a `link-check-deferred` warning means links were not checked yet. Absent only for folder/template edits.',
),
folder: z
.object({
diff --git a/packages/server/src/mcp/tools/shared.ts b/packages/server/src/mcp/tools/shared.ts
index 38697369e..fbe29e731 100644
--- a/packages/server/src/mcp/tools/shared.ts
+++ b/packages/server/src/mcp/tools/shared.ts
@@ -94,7 +94,7 @@ export const previewAttachWarningField = z
const brokenLinksOutputField = z
.array(BrokenLinkSchema)
.describe(
- 'Outbound internal links in the just-written doc that do not resolve. Always present — `[]` means every link resolves UNLESS `brokenLinkSuppression` is also present, in which case a project policy withheld findings. A withholding that arrives in a shape this build cannot validate is dropped rather than relayed, so `brokenLinkSuppression` stays absent even though findings were withheld. The `audit` tool is the surface that discloses that case, through its `warnings`. Each: `{ href (as written), resolvedTo (the docName or content-root file path it pointed at, or null), reason: "no-such-doc" | "no-such-file" | "unresolvable" }`. Report-only — the write landed regardless; fix in a follow-up edit.',
+ 'Outbound internal links in the just-written doc that do not resolve. Always present — `[]` means every link resolves UNLESS `brokenLinkSuppression` is also present, in which case a project policy withheld findings, or `warnings` carries `link-check-deferred`, in which case the links were not checked because the server is still starting. A withholding that arrives in a shape this build cannot validate is dropped rather than relayed, so `brokenLinkSuppression` stays absent even though findings were withheld. The `audit` tool is the surface that discloses that case, through its `warnings`. Each: `{ href (as written), resolvedTo (the docName or content-root file path it pointed at, or null), reason: "no-such-doc" | "no-such-file" | "unresolvable" }`. Report-only — the write landed regardless; fix in a follow-up edit.',
);
const brokenLinkSuppressionOutputField = BrokenLinkSuppressionSchema.optional().describe(
@@ -119,7 +119,7 @@ export const documentResultBaseShape = {
.min(1)
.optional()
.describe(
- "Advisory entries discriminated by `kind`. Write-integrity kinds — `content-divergence` (converged Y.Text didn't byte-match what you composed) and `disk-edit-reconciled` (an out-of-band disk edit was folded in before your write) — mean re-read the doc. The renderability kind `mermaid-parse-error` means the write landed but that fence will not render — fix it and re-edit.",
+ "Advisory entries discriminated by `kind`. Write-integrity kinds — `content-divergence` (converged Y.Text didn't byte-match what you composed) and `disk-edit-reconciled` (an out-of-band disk edit was folded in before your write) — mean re-read the doc. The renderability kind `mermaid-parse-error` means the write landed but that fence will not render — fix it and re-edit. `link-check-deferred` means links were not checked because the server is still starting — that doc is re-checked only by a later write or edit of it, or an `audit`, which can time out until startup finishes.",
),
brokenLinks: brokenLinksOutputField,
brokenLinkSuppression: brokenLinkSuppressionOutputField,
diff --git a/packages/server/src/mcp/tools/write.ts b/packages/server/src/mcp/tools/write.ts
index c0668e1e4..705725f78 100644
--- a/packages/server/src/mcp/tools/write.ts
+++ b/packages/server/src/mcp/tools/write.ts
@@ -935,7 +935,7 @@ export function register(server: ServerInstance, deps: WriteDeps): void {
})
.optional()
.describe(
- 'Single-document write result. Always present on a successful single-doc write — it carries `brokenLinks` (possibly `[]`) plus any `brokenLinkSuppression`/`summary`/`hints`/`warnings`. Read `brokenLinkSuppression` before concluding anything from an empty `brokenLinks`: when it is present, a project policy withheld findings and none of them is yours to repair.',
+ 'Single-document write result. Always present on a successful single-doc write — it carries `brokenLinks` (possibly `[]`) plus any `brokenLinkSuppression`/`summary`/`hints`/`warnings`. Read `brokenLinkSuppression` and `warnings` before concluding anything from an empty `brokenLinks`: a suppression means a project policy withheld findings that are not yours to repair; a `link-check-deferred` warning means links were not checked yet.',
),
folder: z
.object({
@@ -999,7 +999,7 @@ export function register(server: ServerInstance, deps: WriteDeps): void {
documents: looseObjectArray
.optional()
.describe(
- 'Batch write: per-doc result `{ docName, ok, position?, previewUrl?, warnings?, brokenLinks, brokenLinkSuppression?, error? }`. `brokenLinks` (possibly `[]`) is present on each successful entry, same as a single-doc write — and, same as a single-doc write, an empty list means every link resolves only on entries carrying no `brokenLinkSuppression`.',
+ 'Batch write: per-doc result `{ docName, ok, position?, previewUrl?, warnings?, brokenLinks, brokenLinkSuppression?, error? }`. `brokenLinks` (possibly `[]`) is present on each successful entry, same as a single-doc write — and, same as a single-doc write, an empty list means every link resolves only on entries carrying no `brokenLinkSuppression` and no `link-check-deferred` warning (links not checked yet because the server is still starting).',
),
previewUrl: previewUrlOutputField.optional(),
previewUrlSource: previewUrlSourceField,
diff --git a/packages/server/src/persistence.ts b/packages/server/src/persistence.ts
index 6f325549f..68f33155f 100644
--- a/packages/server/src/persistence.ts
+++ b/packages/server/src/persistence.ts
@@ -54,7 +54,10 @@ import {
restoreContributors,
swapContributors,
} from './contributor-tracker.ts';
-import type { DerivedDocumentIndexPersistencePort } from './derived-document-index.ts';
+import {
+ type DerivedDocumentIndexPersistencePort,
+ isDerivedDocumentIndexClosedError,
+} from './derived-document-index.ts';
import { resolveDirectoryRoot, resolveNativePath } from './directory-root.ts';
import { applyDiskContentToDoc, FILE_WATCHER_ORIGIN } from './disk-content-intake.ts';
import {
@@ -1880,14 +1883,19 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persis
durabilityState.clearStoreRefused(documentName);
persistenceDeferCounts.delete(documentName);
- try {
- await derivedDocumentIndex?.recordDurableStore(documentName, markdown);
- } catch (err) {
+ derivedDocumentIndex?.recordDurableStore(documentName, markdown).catch((err: unknown) => {
+ if (isDerivedDocumentIndexClosedError(err)) {
+ log.debug(
+ { err, documentName },
+ '[derived-index] coordinator closed; skipping durable-store projection',
+ );
+ return;
+ }
log.warn(
{ err, documentName },
'[derived-index] durable-store projection failed; disk write remains authoritative',
);
- }
+ });
setActiveSpanAttributes({ 'persistence.bytes': markdown.length });
scheduleGitCommit();
diff --git a/packages/server/src/project-skill-router-contract.test.ts b/packages/server/src/project-skill-router-contract.test.ts
index e9c246558..9d837c581 100644
--- a/packages/server/src/project-skill-router-contract.test.ts
+++ b/packages/server/src/project-skill-router-contract.test.ts
@@ -143,7 +143,8 @@ const routingRules = [
required: [
'Every link must resolve to a doc that exists',
'After every `write`/`edit`, read `brokenLinks`: fix reported `href`s',
- '`brokenLinkSuppression` withheld reserved-log findings; those are not yours to repair',
+ '`brokenLinkSuppression` withheld reserved-log findings (not yours to repair)',
+ 'a `link-check-deferred` warning says nothing was checked yet',
'`audit` is authoritative',
],
},
diff --git a/packages/server/src/services/file-ops.test.ts b/packages/server/src/services/file-ops.test.ts
new file mode 100644
index 000000000..cf5013113
--- /dev/null
+++ b/packages/server/src/services/file-ops.test.ts
@@ -0,0 +1,136 @@
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest';
+import { _resetDocExtensionsForTests } from '../doc-extensions.ts';
+import { createFileOpsService, type FileOpsDeps } from './file-ops.ts';
+
+function neverSettles(): Promise<void> {
+ return new Promise<void>(() => {});
+}
+
+function settleWithin<T>(promise: Promise<T>, label: string, ms = 1_000): Promise<T> {
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ const deadline = new Promise<never>((_, reject) => {
+ timer = setTimeout(() => reject(new Error(`${label} did not settle within ${ms}ms`)), ms);
+ });
+ return Promise.race([promise, deadline]).finally(() => clearTimeout(timer));
+}
+
+describe('file operations while derived-index projections are queued', () => {
+ let contentDir: string;
+
+ beforeEach(() => {
+ _resetDocExtensionsForTests();
+ contentDir = mkdtempSync(join(tmpdir(), 'ok-file-ops-'));
+ });
+
+ afterEach(() => {
+ _resetDocExtensionsForTests();
+ rmSync(contentDir, { recursive: true, force: true });
+ });
+
+ function createService(overrides: Partial<FileOpsDeps> = {}) {
+ const unexpected = () => {
+ throw new Error('not expected in this test');
+ };
+ return createFileOpsService({
+ contentDir,
+ resolveContentEntryPath: (root, _kind, path) => join(root, path),
+ docNameForPath: (relPath) => relPath.replace(/\.md$/, ''),
+ docNameToRelativePath: (docName) => `${docName}.md`,
+ listManagedDocNamesUnderFolder: unexpected,
+ listAffectedDocNames: unexpected,
+ getFileIndex: () => new Map(),
+ conflictFileForDocName: () => null,
+ captureAndCloseDocuments: async () => {},
+ removeFolderIndexEntries: () => {},
+ upsertFolderIndexPathSegments: () => {},
+ deleteDerivedDocumentsBestEffort: neverSettles,
+ invalidateReferencedAssetsCache: () => {},
+ signalFiles: () => {},
+ nextAvailableDuplicateDocName: (docName) => ({ docName: `${docName} copy` }),
+ nextAvailableDuplicateFolderPath: unexpected,
+ resolveDuplicateDocPath: (docName, extension) => join(contentDir, `${docName}${extension}`),
+ collectMarkdownCopies: unexpected,
+ collectFolderPaths: unexpected,
+ recordDerivedDocumentBestEffort: neverSettles,
+ recordDerivedMutationsBestEffort: neverSettles,
+ ...overrides,
+ });
+ }
+
+ test('trash cleanup returns without waiting for the derived delete', async () => {
+ const deleteDerived = vi.fn(neverSettles);
+ const service = createService({
+ getFileIndex: () => new Map([['trashed', {}]]),
+ deleteDerivedDocumentsBestEffort: deleteDerived,
+ });
+
+ const result = await settleWithin(
+ service.trashCleanup('file', 'trashed.md', 'trashed', 'test'),
+ 'trashCleanup',
+ );
+
+ expect(result).toEqual({ deletedDocNames: ['trashed'] });
+ expect(deleteDerived).toHaveBeenCalledExactlyOnceWith(['trashed'], 'trash-cleanup');
+ });
+
+ test('duplicating a file returns without waiting for the derived projection', async () => {
+ writeFileSync(join(contentDir, 'original.md'), '# Original\n');
+ const recordDerived = vi.fn(neverSettles);
+ const service = createService({ recordDerivedDocumentBestEffort: recordDerived });
+
+ const result = await settleWithin(
+ service.duplicatePath('file', 'original.md', 'original'),
+ 'duplicatePath',
+ );
+
+ expect(result).toEqual({
+ ok: true,
+ duplicatedPath: 'original copy',
+ duplicatedDocNames: ['original copy'],
+ });
+ expect(readFileSync(join(contentDir, 'original copy.md'), 'utf-8')).toBe('# Original\n');
+ expect(recordDerived).toHaveBeenCalledExactlyOnceWith(
+ 'original copy',
+ '# Original\n',
+ 'duplicate-path-file',
+ );
+ });
+
+ test('duplicating a folder returns without waiting for the derived projection', async () => {
+ mkdirSync(join(contentDir, 'archive'));
+ writeFileSync(join(contentDir, 'archive', 'entry.md'), '# Entry\n');
+ const recordMutations = vi.fn(neverSettles);
+ const service = createService({
+ listManagedDocNamesUnderFolder: () => ['archive/entry'],
+ nextAvailableDuplicateFolderPath: () => ({ folderPath: 'archive copy' }),
+ collectFolderPaths: (folderPath) => [folderPath],
+ collectMarkdownCopies: (folderPath) => [
+ {
+ docName: `${folderPath}/entry`,
+ fullPath: join(contentDir, folderPath, 'entry.md'),
+ content: readFileSync(join(contentDir, folderPath, 'entry.md'), 'utf-8'),
+ },
+ ],
+ recordDerivedMutationsBestEffort: recordMutations,
+ });
+
+ const result = await settleWithin(
+ service.duplicatePath('folder', 'archive', 'archive'),
+ 'duplicatePath',
+ );
+
+ expect(result).toEqual({
+ ok: true,
+ duplicatedPath: 'archive copy',
+ duplicatedDocNames: ['archive copy/entry'],
+ });
+ expect(readFileSync(join(contentDir, 'archive copy', 'entry.md'), 'utf-8')).toBe('# Entry\n');
+ expect(recordMutations).toHaveBeenCalledExactlyOnceWith(
+ [{ kind: 'upsert', documentName: 'archive copy/entry', markdown: '# Entry\n' }],
+ 'duplicate-path-folder',
+ );
+ });
+});
diff --git a/packages/server/src/services/file-ops.ts b/packages/server/src/services/file-ops.ts
index b6e956cef..e2254ec45 100644
--- a/packages/server/src/services/file-ops.ts
+++ b/packages/server/src/services/file-ops.ts
@@ -223,7 +223,7 @@ export function createFileOpsService(deps: FileOpsDeps): FileOpsService {
deps.invalidateReferencedAssetsCache();
purgeFileIndex(deletedDocNames);
- await deps.deleteDerivedDocumentsBestEffort(deletedDocNames, 'delete-path');
+ void deps.deleteDerivedDocumentsBestEffort(deletedDocNames, 'delete-path');
deps.signalFiles();
return { ok: true, deletedDocNames };
},
@@ -254,7 +254,7 @@ export function createFileOpsService(deps: FileOpsDeps): FileOpsService {
if (operationKind === 'folder') {
deps.removeFolderIndexEntries(path);
}
- await deps.deleteDerivedDocumentsBestEffort(deletedDocNames, 'trash-cleanup');
+ void deps.deleteDerivedDocumentsBestEffort(deletedDocNames, 'trash-cleanup');
deps.signalFiles();
return { deletedDocNames };
},
@@ -359,7 +359,7 @@ export function createFileOpsService(deps: FileOpsDeps): FileOpsService {
deps.mutateFileIndexDelete?.({ path: destinationPath, docName: duplicatedPath });
throw err;
}
- await deps.recordDerivedDocumentBestEffort(duplicatedPath, content, 'duplicate-path-file');
+ void deps.recordDerivedDocumentBestEffort(duplicatedPath, content, 'duplicate-path-file');
return { ok: true, duplicatedPath, duplicatedDocNames: [duplicatedPath] };
}
@@ -458,7 +458,7 @@ export function createFileOpsService(deps: FileOpsDeps): FileOpsService {
}
throw err;
}
- await deps.recordDerivedMutationsBestEffort(derivedMutations, 'duplicate-path-folder');
+ void deps.recordDerivedMutationsBestEffort(derivedMutations, 'duplicate-path-folder');
return { ok: true, duplicatedPath, duplicatedDocNames };
},
};