Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

PRD-8217 Report validators that ran in audit results (#3877)

serafin-garciac3ff882958be
diff --git a/.changeset/README.md b/.changeset/README.md
index 1dc258425..0da04686a 100644
--- a/.changeset/README.md
+++ b/.changeset/README.md
@@ -32,6 +32,6 @@ holds three bits of information:
When `1.0.0` does eventually ship, this convention will be retired and standard semver resumes.
-> Authoring tip: run `bun changeset` from anywhere under `public/open-knowledge/` — the CLI walks up to find this folder. Don't hand-create files; the tool's interactive flow asks for bump types in order and writes a well-formed frontmatter block. Per the convention above, only pick `minor` or `patch`.
+> Authoring tip: run `pnpm changeset` from anywhere under `public/open-knowledge/` — the CLI walks up to find this folder. Don't hand-create files; the tool's interactive flow asks for bump types in order and writes a well-formed frontmatter block. Per the convention above, only pick `minor` or `patch`.
-> SKILL.md metadata is synced automatically by the VP workflow via [`scripts/sync-skill-version.sh`](../scripts/sync-skill-version.sh). You don't need to bump `packages/server/assets/skills/open-knowledge/SKILL.md`'s `metadata.version:` by hand — `bun changeset version` (in CI) bumps the package.json files, then the sync script propagates the new version into SKILL.md. The same script can be run manually from `public/open-knowledge/` if you ever need to re-sync after a hand edit.
+> SKILL.md metadata is synced automatically by the VP workflow via [`scripts/sync-skill-version.sh`](../scripts/sync-skill-version.sh). You don't need to bump the bundled SKILL.md files' `metadata.version:` by hand — `pnpm exec changeset version` (in CI) bumps the package.json files, then the sync script propagates the new version into SKILL.md. The same script can be run manually from `public/open-knowledge/` if you ever need to re-sync after a hand edit.
diff --git a/.changeset/report-validation-sources.md b/.changeset/report-validation-sources.md
new file mode 100644
index 000000000..50c2b4cab
--- /dev/null
+++ b/.changeset/report-validation-sources.md
@@ -0,0 +1,7 @@
+---
+"@inkeep/open-knowledge": minor
+---
+
+Report which checks actually ran. Every successful `ok lint` / `ok audit` report, HTTP response, and MCP `lint` / `audit` result now carries `ran`, naming the source families selected for that call, so a clean result is no longer indistinguishable from one where the check you cared about was switched off. A family absent from `ran` was not checked, and `[]` means nothing was.
+
+A selected check that fails stays in `ran` and is explained in `warnings`, and the summary line says the run could not fully complete rather than reporting clean. On `POST /api/lint/fix`, the singular `warning` field is deprecated: read `warnings`, which now carries the same post-write re-lint failure alongside plugin failures.
diff --git a/docs/content/advanced/content-rules/frontmatter.mdx b/docs/content/advanced/content-rules/frontmatter.mdx
index f5ddb3a9d..887c6a6a7 100644
--- a/docs/content/advanced/content-rules/frontmatter.mdx
+++ b/docs/content/advanced/content-rules/frontmatter.mdx
@@ -76,6 +76,7 @@ docs/guide.md
2:1 warning Frontmatter property "status" must be one of: draft, review, published (got "shipped") frontmatter/enum
2 problems (0 errors, 2 warnings) across 1 file.
+Checks run: frontmatter.
```
As a diagnostic in `ok lint --json` or the MCP `lint` tool's structured content (0-based, end-exclusive ranges — the text report is 1-based):
diff --git a/docs/content/advanced/content-rules/overview.mdx b/docs/content/advanced/content-rules/overview.mdx
index ff8ec5986..45e961f88 100644
--- a/docs/content/advanced/content-rules/overview.mdx
+++ b/docs/content/advanced/content-rules/overview.mdx
@@ -6,7 +6,7 @@ description: A non-blocking linting layer over your project's markdown — probl
Content rules are a lightweight linting layer over your project's markdown. **Enable a linter for a project** and it checks your markdown as you write. Findings are **non-blocking**: they surface as warnings, never stop a save, and never gate an agent's edit.
-Two linters ship today. [markdownlint](/docs/advanced/content-rules/markdownlint) is the standard engine for markdown style — hard tabs, heading increments, list markers, and the rest. [Frontmatter schemas](/docs/advanced/content-rules/frontmatter) validates each doc's frontmatter against standard JSON Schema files. The system is pluggable and more linters are planned.
+Three linters ship today. [markdownlint](/docs/advanced/content-rules/markdownlint) is the standard engine for markdown style — hard tabs, heading increments, list markers, and the rest. [Frontmatter schemas](/docs/advanced/content-rules/frontmatter) validates each doc's frontmatter against standard JSON Schema files. [OKF](/docs/plugins/okf) checks portability to the Open Knowledge Format. The system is pluggable and more linters are planned.
This page covers what's the same whatever linter runs; each linter's page covers its own rules and configuration.
@@ -57,6 +57,7 @@ docs/guide.md
2:1 warning Frontmatter property "status" must be one of: draft, review, published (got "shipped") frontmatter/enum
3 problems (0 errors, 3 warnings) across 1 file.
+Checks run: markdownlint, frontmatter.
```
Configuration problems (a malformed config file, a broken schema) print as `!` lines after the findings — they describe your setup, not a document. `--json` emits the same data as a machine-readable object:
@@ -100,7 +101,8 @@ Configuration problems (a malformed config file, a broken schema) print as `!` l
"fileCount": 1,
"errorCount": 0,
"warningCount": 3,
- "fixedCount": 0
+ "fixedCount": 0,
+ "ran": ["markdownlint", "frontmatter"]
}
```
@@ -109,7 +111,8 @@ Conventions to know:
- JSON ranges are **0-based and end-exclusive** (LSP-aligned); the text report displays 1-based positions.
- `fixes` appears only on auto-fixable findings — its presence is how tooling knows `--fix` would resolve the problem.
- `fixed` is `true` on each file `--fix` rewrote, and the top-level `fixedCount` counts **files**, not problems.
-- The top-level `warnings` array carries the configuration problems.
+- The top-level `warnings` array carries configuration or runtime degradation problems.
+- The top-level `ran` array names the enabled source families selected for the run. A family absent from `ran` was not checked. An explicit `[]` means no checks were selected at all. It does not mean the document was checked and found clean.
With `--fix`, fixable findings are applied in place, fixed files are marked `(fixed)`, and the report lists what remains:
@@ -120,8 +123,11 @@ docs/guide.md (fixed)
2 problems (0 errors, 2 warnings) across 1 file.
Fixed 1 file.
+Checks run: markdownlint, frontmatter.
```
+Every linter ships off, so a project that has not enabled one closes its report with `No checks ran.` in place of the `Checks run:` line. The `✓ No problems` above it then means only that nothing was checked. Turn a linter on under **Settings ▸ This project ▸ Plugins**.
+
`ok audit` widens the same report to the full validation plane — content-rule problems **and** broken internal links, each finding tagged with its source:
```bash
@@ -155,10 +161,13 @@ Unlike `ok lint`, `ok audit` needs the project's server running (`ok start` or O
"warnings": [],
"fileCount": 1,
"errorCount": 0,
- "warningCount": 1
+ "warningCount": 1,
+ "ran": ["markdownlint", "frontmatter", "links"]
}
```
+The audit's top-level `ran` array reports the selected source families: enabled document linters plus `links` unless link validation is `off`. OKF's document and project-tree checks share the public `okf` family, so `ran` reports selection, not the number of internal validators or a pass verdict.
+
Broken links are warnings by default. The project's `validation.links` setting — **Settings ▸ This project ▸ Content rules** — decides both whether they appear and at what severity: `warning` (the default), `error` to gate CI on them with `--errors-only`, or `off` to drop them from the plane entirely. Content-rule findings keep their own severities, so `--errors-only` covers both planes at once. Skill documents are the one exception: their link findings are always excluded, whatever this setting says, because a skill's links name files it creates only at runtime.
## AI agents
@@ -169,29 +178,33 @@ Agents get the same signal you see, across three surfaces. See the [MCP referenc
Lints a single document, or audits the project when `document` is omitted (`path` scopes it to a folder or file). `fix: true` — which requires `document` — auto-fixes fixable rules in place, attributed and live in the preview, the same result as the editor's **Fix** action.
-A single-document call returns a readable summary. Configuration problems ride along as `⚠` lines, and the closing hint tells the agent whether `fix: true` would help:
+A single-document call returns a readable summary. Configuration problems appear under a `Lint incomplete` warning block, and the closing hint tells the agent whether `fix: true` would help:
```text
docs/guide.md: 2 warnings
⚠ line 1 frontmatter/required: Frontmatter property "owner" is required
⚠ line 2 frontmatter/enum: Frontmatter property "status" must be one of: draft, review, published (got "shipped")
+Lint incomplete — 1 warning (findings may be partial):
⚠ frontmatter schema .ok/schemas/missing.schema.json: cannot read (ENOENT: no such file or directory, …)
None are auto-fixable — these need content edits via `edit`/`write`.
+Checks run: frontmatter.
```
+With no linter enabled the report closes with `No checks ran.` instead, the same string `ok lint` prints.
+
The structured content is close to `ok lint --json`, with three differences:
| | `ok lint --json` | MCP `lint` |
| --- | --- | --- |
| Project path | `contentDir` | `cwd` |
| `fixedCount` counts | files rewritten | **problems** resolved |
-| Cap fields | — | `omittedFileCount`, per-file `omittedDiagnosticCount` |
+| Cap fields | — | `omittedFileCount`, per-file `omittedDiagnosticCount`, `omittedWarningCount` |
-Everything else matches: `files[].diagnostics` with 0-based `range`, `severity`, `source`, `code`, `message`, plus `errorCount`, `warningCount`, `fileCount` on an audit, and the configuration-problem `warnings`. Audit output is capped at 10 files × 10 diagnostics per file — the text channel marks the remainder with "… and N more", the structured channel with the two cap fields above. Counts always reflect the full scan, and re-running with a narrower `path` recovers the detail. With `fix: true` the summary reports what was applied and what remains: `Fixed 1 problem in docs/guide.md.` followed by the unfixable findings.
+Everything else matches: `files[].diagnostics` with 0-based `range`, `severity`, `source`, `code`, `message`, plus `errorCount`, `warningCount`, `fileCount` on an audit, configuration or degradation `warnings`, and `ran` for the selected source families. Audit output is capped project-wide at 10 warnings, plus 10 files × 10 diagnostics per file — the text channel marks the remainder with "… and N more", the structured channel with the three cap fields above. Counts always reflect the full scan, and re-running with a narrower `path` recovers the detail. With `fix: true` the summary reports what was applied and what remains: `Fixed 1 problem in docs/guide.md.` followed by the unfixable findings.
### The `audit` tool
-The agent-side `ok audit` — content-rule problems and broken links in one read-only call, grouped by file, same 10 × 10 cap, no fix shape.
+The agent-side `ok audit` — content-rule problems and broken links in one read-only call, grouped by file, same 10 × 10 + 10-warning cap with `omittedWarningCount` when warnings are dropped, no fix shape. Its text and structured results both report `ran`. If a selected validator degrades, its family remains in `ran` and the `warnings` channel explains why; a partial degradation may still have contributed findings.
### Write responses
diff --git a/docs/content/reference/mcp.mdx b/docs/content/reference/mcp.mdx
index 77059cf77..f32792728 100644
--- a/docs/content/reference/mcp.mdx
+++ b/docs/content/reference/mcp.mdx
@@ -19,8 +19,8 @@ To point an MCP client at this server, locally or over a tunnel from another mac
| `exec` | Read-only shell (`cat`, `ls`, `grep`, `find`, `head`, `tail`, `wc`, `sort`, `uniq`, `cut`); reads return frontmatter, backlinks, recent history, and any unresolved [comments](/docs/features/comments) left on the doc. When the [frontmatter schemas](/docs/advanced/content-rules/frontmatter) plugin is on, doc and folder entries also carry `schemas: …` naming the JSON Schema files that govern them — the contract to read before writing. Fs-direct; works without the server. |
| `search` | Ranked workspace search (title boost + body BM25 + recency; same engine the cmd-K palette uses). `intent: "omnibar"` narrows to fast title/path matching (`full_text`, the default, includes the body); `scopes` filters hit kinds; `limit` caps rows (default 20, max 100). When [semantic search](/docs/reference/configuration#semantic-search) is enabled, an embeddings signal is additionally fused into `full_text` ranking (opt-in, content egress) — pass `semantic: false` to force pure-lexical. Right after boot, an empty result set with `ready: false` means the index is still building — retry after a couple of seconds rather than treating it as no matches. |
| `links` | Wiki-link graph. `kind` selects: `backlinks`, `forward`, `dead`, `orphans`, `hubs`, or `suggest`. Accepts an array (e.g. `["dead", "orphans", "hubs"]`) to fetch several views in one call; each view nests under its own key in the response. `dead` is the raw graph view: unlike `audit`, it still lists unresolved links whose source is a skill document |
-| `lint` | Report content-rule problems (markdownlint style rules + frontmatter schema validation) for one `document` or a whole-project audit (`path` scopes it); each finding carries `source`/`code`/`message`/`range`/`severity`, audit capped at 10 files × 10 diagnostics with accurate totals; configuration problems (e.g. a broken schema file) ride a separate `warnings` channel. Pass `fix: true` with `document` to auto-fix fixable rules in place — the fix lands through the collaborative document (attributed, live preview), not a shell edit; the rest need `edit`/`write`. Example output: [content rules overview](/docs/advanced/content-rules/overview#ai-agents) |
-| `audit` | Unified read-only validation audit: every content problem — content-rule violations (markdownlint + frontmatter schemas) **and** broken internal links — in one call, grouped by the file to fix (`path` scopes to a folder or doc). Each finding is tagged with its validator; broken links report under the source doc that contains them, at the offending line, as warnings by default (the project's `validation.links` setting can raise them to errors or hide them). Skill documents are excluded from the broken-link half: a skill's links routinely name files it creates only at runtime (use `links({ kind: "dead" })` to see them; links from other documents to a skill still validate; documents that merely sit under a dot directory, such as `.github/CI_RUNBOOK.md`, keep their findings). Same 10 × 10 output cap as `lint`'s audit, with accurate totals. No fix shape — lint fixes go through `lint`, link repairs through `edit`/`write` |
+| `lint` | Report content-rule problems (markdownlint style rules, frontmatter schema validation, document-level OKF checks) for one `document` or a whole-project audit (`path` scopes it); each finding carries `source`/`code`/`message`/`range`/`severity`, audit capped at 10 files × 10 diagnostics and project-wide at 10 warnings with accurate totals; configuration or runtime degradation problems ride a separate `warnings` channel, with `omittedWarningCount` when warnings are dropped. Successful responses include `ran`, the enabled document-lint source families selected for that call (`markdownlint`, `frontmatter`, and/or document-level `okf`). A family absent from `ran` was not checked, and `[]` means no checks were selected at all. Pass `fix: true` with `document` to auto-fix fixable rules in place — the fix lands through the collaborative document (attributed, live preview), not a shell edit; the rest need `edit`/`write`. Example output: [content rules overview](/docs/advanced/content-rules/overview#ai-agents) |
+| `audit` | Unified read-only validation audit: every content problem — content-rule violations (markdownlint, frontmatter schemas, OKF) **and** broken internal links — in one call, grouped by the file to fix (`path` scopes to a folder or doc). Each finding is tagged with its validator; broken links report under the source doc that contains them, at the offending line, as warnings by default (the project's `validation.links` setting can raise them to errors or hide them). Skill documents are excluded from the broken-link half: a skill's links routinely name files it creates only at runtime (use `links({ kind: "dead" })` to see them; links from other documents to a skill still validate; documents that merely sit under a dot directory, such as `.github/CI_RUNBOOK.md`, keep their findings). Successful responses include `ran`: enabled document-lint families plus `links` unless link validation is off. Document and project-tree OKF checks share the `okf` family, and a family absent from `ran` was not checked. A selected family remains in `ran` if it degrades, with the reason explained in `warnings`; a partial degradation may still have contributed findings. Same 10 × 10 output cap as `lint`'s audit, plus a warning cap set project-wide at 10 warnings, with accurate totals and `omittedWarningCount` when warnings are dropped. No fix shape — lint fixes go through `lint`, link repairs through `edit`/`write` |
| `history` | Version timeline for a `document`, `folder`, or `skill`. Each entry carries a `version` (commit SHA) you pass to `restore_version`. Filter with `kind` (`checkpoint` / `wip` / `upstream`), `author` / `excludeAuthor`, and `branch`; paginate with `limit` / `offset` (default 50, max 200). `folder: ""` selects the project-root timeline |
| `skills` | Find skills. Pass `query` to search skills.sh and return import-ready rows (`name`, `source`, installs, publisher); omit `name` to list every managed skill across Project + Global (rows carry `installed` + `hosts`); pass `name` to read one (addressed by `name`+`scope`, never by path); pass `name` + `file` to read one bundle file's text — the read path for `references/` and `scripts/` content |
| `config` | Read the effective merged config (`config({ key: "appearance.theme" })` for a sub-tree; omit `key` for the whole config) |
diff --git a/packages/app/tests/integration/audit-http.test.ts b/packages/app/tests/integration/audit-http.test.ts
index fec58c2c6..e330d890c 100644
--- a/packages/app/tests/integration/audit-http.test.ts
+++ b/packages/app/tests/integration/audit-http.test.ts
@@ -44,6 +44,39 @@ function api(pathAndQuery: string): string {
}
describe('GET /api/audit', () => {
+ test('okf enabled reaches ran through the real config lift', async () => {
+ // Seeded through `.ok/config.yml`, not a hand-built LinterConfig: the lift
+ // from persisted config to the runtime slices is where a new plugin key
+ // silently vanishes, and a fixture that skips it cannot catch that.
+ const okfServer = await createTestServer({
+ seedProjectConfigYml: 'contentRules:\n okf:\n enabled: true\nvalidation:\n links: off\n',
+ });
+ try {
+ const res = await fetch(`http://127.0.0.1:${okfServer.port}/api/audit`);
+ expect(res.status).toBe(200);
+ const body = ValidationAuditResponseSchema.parse(await res.json());
+ // Once, not twice: the okf lint plugin and the okf project validator both
+ // select this family and it folds to one public id.
+ expect(body.ran).toEqual(['okf']);
+ } finally {
+ await okfServer.cleanup();
+ }
+ });
+
+ test('all validation sources off returns an explicit empty selection', async () => {
+ const disabledServer = await createTestServer({
+ seedProjectConfigYml: 'validation:\n links: off\n',
+ });
+ try {
+ const res = await fetch(`http://127.0.0.1:${disabledServer.port}/api/audit`);
+ expect(res.status).toBe(200);
+ const body = ValidationAuditResponseSchema.parse(await res.json());
+ expect(body.ran).toEqual([]);
+ } finally {
+ await disabledServer.cleanup();
+ }
+ });
+
test(
'returns lint + dead-link findings in one source-tagged plane',
async () => {
@@ -61,6 +94,7 @@ describe('GET /api/audit', () => {
const res = await fetch(api('/api/audit'));
expect(res.status).toBe(200);
const body: ValidationAuditResponse = ValidationAuditResponseSchema.parse(await res.json());
+ expect(body.ran).toEqual(['markdownlint', 'links']);
const tabbed = body.files.find((f) => f.file === 'audit-http/tabbed.md');
const md010 = tabbed?.diagnostics.find((d) => d.code === 'MD010');
@@ -197,8 +231,10 @@ describe('GET /api/audit', () => {
const enumerated = ValidationAuditResponseSchema.parse(
await enumeratedRes.json(),
) satisfies ValidationAuditResponse;
+ const rawCounts: unknown = await countsRes.json();
+ expect(rawCounts).not.toHaveProperty('ran');
const counts = ValidationAuditCountsResponseSchema.parse(
- await countsRes.json(),
+ rawCounts,
) satisfies ValidationAuditCountsResponse;
// Same files, same rollups — the counts plane is a derivation, not a
diff --git a/packages/app/tests/integration/frontmatter-lint-http.test.ts b/packages/app/tests/integration/frontmatter-lint-http.test.ts
index 41c9bdadc..e6aa18efd 100644
--- a/packages/app/tests/integration/frontmatter-lint-http.test.ts
+++ b/packages/app/tests/integration/frontmatter-lint-http.test.ts
@@ -12,6 +12,7 @@ import {
LintAuditResponseSchema,
LintConfigResponseSchema,
LintDocResultSchema,
+ LintFixResultSchema,
} from '@inkeep/open-knowledge-core';
import { afterAll, beforeAll, describe, expect, test } from 'vitest';
import { HARNESS_BOOT_TIMEOUT_MS } from './harness-boot-timeout';
@@ -232,6 +233,21 @@ describe('GET /api/lint/audit — aggregation + config channel', () => {
});
});
+describe('POST /api/lint/fix — config channel', () => {
+ test('a schemaError reaches warnings when no content fix is available', async () => {
+ const res = await fetch(api('/api/lint/fix'), {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json' },
+ body: JSON.stringify({ docName: 'docs/guide', agentId: 'frontmatter-fix-agent' }),
+ });
+ expect(res.status).toBe(200);
+ const body = LintFixResultSchema.parse(await res.json());
+ expect(body.fixedCount).toBe(0);
+ expect(body.ran).toEqual(['frontmatter']);
+ expect(body.warnings?.some((warning) => warning.includes('missing.schema.json'))).toBe(true);
+ });
+});
+
describe('GET /api/lint/config — effective slice for the browser', () => {
test('the effective config inlines loaded schema content + configProblems', async () => {
const res = await fetch(api('/api/lint/config?doc=docs%2Fguide'));
diff --git a/packages/app/tests/integration/lint-http.test.ts b/packages/app/tests/integration/lint-http.test.ts
index 70407a7db..c38cd7788 100644
--- a/packages/app/tests/integration/lint-http.test.ts
+++ b/packages/app/tests/integration/lint-http.test.ts
@@ -12,6 +12,7 @@
import { existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import {
+ LINT_PLUGINS,
type LintAuditResponse,
LintAuditResponseSchema,
type LintConfigResponse,
@@ -21,7 +22,7 @@ import {
type LintFixResult,
LintFixResultSchema,
} from '@inkeep/open-knowledge-core';
-import { afterAll, beforeAll, describe, expect, test } from 'vitest';
+import { afterAll, beforeAll, describe, expect, test, vi } from 'vitest';
import { HARNESS_BOOT_TIMEOUT_MS } from './harness-boot-timeout';
import { createTestServer, type TestServer } from './test-harness.ts';
@@ -54,6 +55,11 @@ describe('GET /api/lint (single document)', () => {
expect(res.status).toBe(200);
const body: LintDocResult = LintDocResultSchema.parse(await res.json());
expect(body.file).toBe('lint-http/tabbed.md');
+ expect(body.ran).toEqual(['markdownlint']);
+ // `ran` is the field that is always present. `warnings` keeps its
+ // documented additive contract — absent when there is nothing to report —
+ // so a consumer branching on its presence is not fired on every clean lint.
+ expect(body).not.toHaveProperty('warnings');
const md010 = body.diagnostics.find((d) => d.code === 'MD010');
expect(md010).toBeDefined();
expect(md010?.source).toBe('markdownlint');
@@ -71,6 +77,29 @@ describe('GET /api/lint (single document)', () => {
const res = await fetch(api(`/api/lint?doc=no-such-doc-${crypto.randomUUID().slice(0, 8)}`));
expect(res.status).toBe(404);
});
+
+ test('a selected plugin degradation stays in ran and is named in warnings', async () => {
+ const folder = join(server.contentDir, 'lint-http-degraded');
+ mkdirSync(folder, { recursive: true });
+ writeFileSync(join(folder, 'doc.md'), TABBED_BODY, 'utf-8');
+ const markdownlint = LINT_PLUGINS.find((plugin) => plugin.id === 'markdownlint');
+ if (!markdownlint) throw new Error('markdownlint plugin missing');
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const lint = vi.spyOn(markdownlint, 'lint').mockRejectedValue(new Error('unavailable'));
+ try {
+ const res = await fetch(api('/api/lint?doc=lint-http-degraded%2Fdoc'));
+ expect(res.status).toBe(200);
+ const body = LintDocResultSchema.parse(await res.json());
+ expect(body.ran).toEqual(['markdownlint']);
+ expect(body.warnings).toEqual([
+ 'source "markdownlint" lint failed on "lint-http-degraded/doc.md": unavailable',
+ ]);
+ } finally {
+ lint.mockRestore();
+ warn.mockRestore();
+ rmSync(folder, { recursive: true, force: true });
+ }
+ });
});
describe('GET /api/lint/audit', () => {
@@ -92,6 +121,7 @@ describe('GET /api/lint/audit', () => {
expect(f.diagnostics.length).toBeGreaterThan(0);
}
expect(fullBody.fileCount).toBeGreaterThanOrEqual(2);
+ expect(fullBody.ran).toEqual(['markdownlint']);
const scoped = await fetch(api('/api/lint/audit?path=lint-http-audit'));
expect(scoped.status).toBe(200);
@@ -101,6 +131,7 @@ describe('GET /api/lint/audit', () => {
'lint-http-audit/b.md',
]);
expect(scopedBody.fileCount).toBe(2);
+ expect(scopedBody.ran).toEqual(['markdownlint']);
} finally {
rmSync(folder, { recursive: true, force: true });
}
@@ -130,6 +161,7 @@ describe('POST /api/lint/fix', () => {
expect(res.status).toBe(200);
const body: LintFixResult = LintFixResultSchema.parse(await res.json());
expect(body.file).toBe('lint-fix/tabbed.md');
+ expect(body.ran).toEqual(['markdownlint']);
expect(body.fixedCount).toBeGreaterThanOrEqual(1);
// MD010 (hard tabs) is auto-fixable — gone from the remaining set.
expect(body.diagnostics.find((d) => d.code === 'MD010')).toBeUndefined();
@@ -142,6 +174,42 @@ describe('POST /api/lint/fix', () => {
}
});
+ test('a throwing plugin lands in warnings while the fix write stays durable', async () => {
+ const folder = join(server.contentDir, 'lint-fix-degraded');
+ mkdirSync(folder, { recursive: true });
+ const file = join(folder, 'tabbed.md');
+ writeFileSync(
+ file,
+ '---\ntitle: Keep Me\n---\n\n# Doc\n\n\tindented with a hard tab\n',
+ 'utf-8',
+ );
+ const markdownlint = LINT_PLUGINS.find((plugin) => plugin.id === 'markdownlint');
+ if (!markdownlint) throw new Error('markdownlint plugin missing');
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ // `lint` throws; `fix` is a separate plugin entry point and keeps working —
+ // the shape where the write succeeds but the surrounding lint passes degrade.
+ const lint = vi.spyOn(markdownlint, 'lint').mockRejectedValue(new Error('unavailable'));
+ try {
+ const res = await postFix('lint-fix-degraded/tabbed');
+ expect(res.status).toBe(200);
+ const body = LintFixResultSchema.parse(await res.json());
+ expect(body.ran).toEqual(['markdownlint']);
+ // The lint pass fails before AND after the write; the summarizer
+ // collapses the two reports of the one fault into a single line.
+ expect(body.warnings).toEqual([
+ 'source "markdownlint" lint failed on "lint-fix-degraded/tabbed.md": unavailable',
+ ]);
+ // The fix landed and persisted despite the degradation.
+ const onDisk = readFileSync(file, 'utf-8');
+ expect(onDisk).not.toContain('\t');
+ expect(onDisk).toContain('title: Keep Me');
+ } finally {
+ lint.mockRestore();
+ warn.mockRestore();
+ rmSync(folder, { recursive: true, force: true });
+ }
+ });
+
test('a clean doc is a no-op: fixedCount 0, no remaining diagnostics', async () => {
const folder = join(server.contentDir, 'lint-fix-clean');
mkdirSync(folder, { recursive: true });
@@ -199,6 +267,36 @@ describe('POST /api/lint/fix', () => {
});
});
+describe('zero selected lint sources', () => {
+ test('single-doc, project, and fix success responses emit ran: []', async () => {
+ const disabledServer = await createTestServer();
+ const disabledApi = (path: string) => `http://127.0.0.1:${disabledServer.port}${path}`;
+ try {
+ const doc = LintDocResultSchema.parse(
+ await (await fetch(disabledApi('/api/lint?doc=test-doc'))).json(),
+ );
+ const audit = LintAuditResponseSchema.parse(
+ await (await fetch(disabledApi('/api/lint/audit'))).json(),
+ );
+ const fix = LintFixResultSchema.parse(
+ await (
+ await fetch(disabledApi('/api/lint/fix'), {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json' },
+ body: JSON.stringify({ docName: 'test-doc', agentId: 'zero-source-agent' }),
+ })
+ ).json(),
+ );
+
+ expect(doc.ran).toEqual([]);
+ expect(audit.ran).toEqual([]);
+ expect(fix.ran).toEqual([]);
+ } finally {
+ await disabledServer.cleanup();
+ }
+ });
+});
+
describe('POST /api/lint/markdownlint-config', () => {
async function postRule(ruleId: string, value: unknown): Promise<Response> {
return fetch(api('/api/lint/markdownlint-config'), {
diff --git a/packages/cli/src/commands/audit.test.ts b/packages/cli/src/commands/audit.test.ts
index f7f865dd7..1207aede1 100644
--- a/packages/cli/src/commands/audit.test.ts
+++ b/packages/cli/src/commands/audit.test.ts
@@ -135,7 +135,7 @@ describe('runAudit', () => {
}
test('clean audit: GET /api/audit with cli version headers, exit 0', async () => {
- const { urls, inits } = stubFetch(payload({ fileCount: 3 }));
+ const { urls, inits } = stubFetch(payload({ fileCount: 3, ran: ['markdownlint', 'links'] }));
const { io, out } = collectIo();
const code = await runAudit(undefined, {}, minimalConfig, dir, dir, io);
@@ -146,6 +146,7 @@ describe('runAudit', () => {
expect(headers['x-ok-client-kind']).toBe('cli');
expect(headers['x-ok-client-runtime']).toBe(RUNTIME_VERSION);
expect(out.join('\n')).toContain('No problems in 3 files');
+ expect(out.join('\n')).toContain('Checks run: markdownlint, links.');
});
test('scopes the query to the contentDir-relative target', async () => {
@@ -163,6 +164,7 @@ describe('runAudit', () => {
fileCount: 2,
errorCount: 1,
warningCount: 1,
+ ran: ['markdownlint', 'links'],
files: [
{
file: 'a.md',
@@ -189,6 +191,7 @@ describe('runAudit', () => {
expect(text).toContain('markdownlint/MD010');
expect(text).toContain('links/dead-link');
expect(text).toContain('2 problems');
+ expect(text).toContain('Checks run: markdownlint, links.');
});
test('--errors-only ignores warning-severity findings for the exit code', async () => {
@@ -210,6 +213,7 @@ describe('runAudit', () => {
const body = payload({
fileCount: 1,
warningCount: 1,
+ ran: ['markdownlint', 'links'],
files: [{ file: 'a.md', diagnostics: [diagnostic()] }],
});
stubFetch(body);
@@ -260,14 +264,16 @@ describe('runAudit', () => {
test('surfaces engine degradation warnings in the report', async () => {
stubFetch(
- payload({ warnings: ['links validation unavailable: backlink index is not configured'] }),
+ payload({
+ warnings: ['source family "links" validation failed: backlink index is not configured'],
+ }),
);
const { io, out } = collectIo();
const code = await runAudit(undefined, {}, minimalConfig, dir, dir, io);
expect(code).toBe(0);
- expect(out.join('\n')).toContain('links validation unavailable');
+ expect(out.join('\n')).toContain('source family "links" validation failed');
});
test('errors with `ok start` guidance when no server lock is live', async () => {
diff --git a/packages/cli/src/commands/audit.ts b/packages/cli/src/commands/audit.ts
index db6d4019d..69d0d4f14 100644
--- a/packages/cli/src/commands/audit.ts
+++ b/packages/cli/src/commands/audit.ts
@@ -169,5 +169,6 @@ function toReportInput(result: ValidationAuditResponse): LintReportInput {
errorCount: result.errorCount,
warningCount: result.warningCount,
fixedCount: 0,
+ ran: result.ran,
};
}
diff --git a/packages/cli/src/commands/lint.test.ts b/packages/cli/src/commands/lint.test.ts
index 763b9d658..79309e1e0 100644
--- a/packages/cli/src/commands/lint.test.ts
+++ b/packages/cli/src/commands/lint.test.ts
@@ -18,6 +18,7 @@ function result(over: Partial<LintRunResult> = {}): LintRunResult {
errorCount: 0,
warningCount: 0,
fixedCount: 0,
+ ran: ['markdownlint'],
...over,
};
}
@@ -26,6 +27,7 @@ describe('formatLintReport', () => {
test('reports a clean run', async () => {
const out = formatLintReport(result({ fileCount: 3 }));
expect(out).toContain('No problems in 3 files');
+ expect(out).toContain('Checks run: markdownlint.');
});
test('groups diagnostics under their file with a summary', async () => {
@@ -33,6 +35,7 @@ describe('formatLintReport', () => {
result({
fileCount: 1,
warningCount: 1,
+ warnings: ['could not read directory drafts'],
files: [
{
file: 'a.md',
@@ -54,6 +57,10 @@ describe('formatLintReport', () => {
expect(out).toContain('Hard tabs');
expect(out).toContain('markdownlint/MD010');
expect(out).toContain('1 problem');
+ expect(out).toContain('Checks run: markdownlint.');
+ expect(out.indexOf('Checks run: markdownlint.')).toBeLessThan(
+ out.indexOf('could not read directory drafts'),
+ );
});
test('marks fixed files and shows a fixed summary', async () => {
@@ -67,9 +74,26 @@ describe('formatLintReport', () => {
expect(out).toContain('Fixed 1 file');
});
- test('surfaces runner warnings', async () => {
- const out = formatLintReport(result({ warnings: ['could not read directory drafts'] }));
+ test('surfaces runner warnings after the summary, not before it', async () => {
+ const out = formatLintReport(
+ result({ fileCount: 500, warnings: ['could not read directory drafts'] }),
+ );
expect(out).toContain('could not read directory drafts');
+ // On a large KB a warning block rendered first scrolls off, leaving a
+ // terminal whose last screen reads clean.
+ expect(out.indexOf('could not read directory drafts')).toBeGreaterThan(
+ out.indexOf('No problems in 500 files'),
+ );
+ // The coverage line still sits adjacent to the no-problems claim it qualifies.
+ expect(out.indexOf('Checks run: markdownlint.')).toBeLessThan(
+ out.indexOf('could not read directory drafts'),
+ );
+ });
+
+ test('distinguishes an explicit empty selection from an older response', () => {
+ expect(formatLintReport(result({ ran: [] }))).toContain('No checks ran.');
+ expect(formatLintReport(result({ ran: undefined }))).not.toContain('Checks run:');
+ expect(formatLintReport(result({ ran: undefined }))).not.toContain('No checks ran.');
});
});
diff --git a/packages/cli/src/commands/lint.ts b/packages/cli/src/commands/lint.ts
index 408d824d9..4d0cfb68d 100644
--- a/packages/cli/src/commands/lint.ts
+++ b/packages/cli/src/commands/lint.ts
@@ -11,6 +11,7 @@ import {
type LinterConfig,
type PersistedLinterConfig,
toEffectiveBase,
+ validationCoverageLines,
} from '@inkeep/open-knowledge-core';
import { type Config, resolveContentDir } from '@inkeep/open-knowledge-server';
import { Command } from 'commander';
@@ -103,6 +104,7 @@ export interface LintReportInput {
errorCount: number;
warningCount: number;
fixedCount: number;
+ ran?: string[];
}
/** Render one file's heading + its diagnostics as report lines. */
@@ -125,9 +127,6 @@ export function formatLintReport(result: LintReportInput): string {
if (file.diagnostics.length > 0) lines.push(...renderFileBlock(file));
}
- for (const w of result.warnings) lines.push(yellow(`! ${w}`));
- if (result.warnings.length > 0) lines.push('');
-
const problemTotal = result.errorCount + result.warningCount;
if (problemTotal === 0) {
lines.push(
@@ -144,6 +143,14 @@ export function formatLintReport(result: LintReportInput): string {
if (result.fixedCount > 0) {
lines.push(dim(`Fixed ${result.fixedCount} file${result.fixedCount === 1 ? '' : 's'}.`));
}
+ lines.push(...validationCoverageLines(result.ran).map((line) => dim(line)));
+ // Below the summary, not above it: on a large KB a warning block rendered
+ // first scrolls off, leaving a terminal whose last screen reads clean. The
+ // coverage line stays adjacent to the no-problems claim it qualifies.
+ if (result.warnings.length > 0) {
+ lines.push('');
+ for (const w of result.warnings) lines.push(yellow(`! ${w}`));
+ }
return lines.join('\n');
}
diff --git a/packages/cli/src/content/lint-runner.test.ts b/packages/cli/src/content/lint-runner.test.ts
index 8990fcb02..7b6c449d7 100644
--- a/packages/cli/src/content/lint-runner.test.ts
+++ b/packages/cli/src/content/lint-runner.test.ts
@@ -11,6 +11,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { DEFAULT_LINTER_CONFIG } from '@inkeep/open-knowledge-core';
import { afterEach, beforeEach, describe, expect, test } from 'vitest';
+import { lintDoc } from '../../../server/src/lint/audit.ts';
import { runLint } from './lint-runner.ts';
let root: string;
@@ -48,6 +49,25 @@ function run(opts: Partial<Parameters<typeof runLint>[0]> = {}) {
}
describe('runLint — walk + lint', () => {
+ test('reports the same selected families as the server lint path', async () => {
+ write('a.md', '# A\n');
+ const local = await run();
+ const server = await lintDoc({
+ projectDir: root,
+ contentDir: root,
+ baseConfig: {
+ ...DEFAULT_LINTER_CONFIG,
+ plugins: {
+ ...DEFAULT_LINTER_CONFIG.plugins,
+ markdownlint: { ...DEFAULT_LINTER_CONFIG.plugins.markdownlint, enabled: true },
+ },
+ },
+ docRelPath: 'a.md',
+ });
+
+ expect(local.ran).toEqual(server.ran);
+ });
+
test('the walk skips hidden segments; an explicit hidden file target still lints', async () => {
write('visible.md', '# A\n\ntext with a\ttab\n');
write('.ok/skills/pack/SKILL.md', '# S\n\ntext with a\ttab\n');
@@ -62,6 +82,7 @@ describe('runLint — walk + lint', () => {
write('a.md', '# A\n\ntext with a\ttab\n');
write('b.md', '# B\n\n#bad heading\n');
const result = await run();
+ expect(result.ran).toEqual(['markdownlint']);
expect(result.fileCount).toBe(2);
const a = result.files.find((f) => f.file === 'a.md');
expect(a?.diagnostics.some((d) => d.code === 'MD010')).toBe(true);
diff --git a/packages/cli/src/content/lint-runner.ts b/packages/cli/src/content/lint-runner.ts
index 6e6c38fb2..dc2ecb63b 100644
--- a/packages/cli/src/content/lint-runner.ts
+++ b/packages/cli/src/content/lint-runner.ts
@@ -16,11 +16,15 @@ import {
} from 'node:fs';
import { dirname, isAbsolute, join, relative, resolve } from 'node:path';
import {
+ deriveValidationRunSources,
fixDocument,
type LintDiagnostic,
type LinterConfig,
+ type LintPluginFailure,
+ type LintPluginId,
lintDocument,
SUPPORTED_DOC_EXTENSIONS,
+ summarizeLintPluginFailures,
} from '@inkeep/open-knowledge-core';
import {
composeEffectiveLinterConfig,
@@ -46,6 +50,7 @@ export interface LintRunResult {
errorCount: number;
warningCount: number;
fixedCount: number;
+ ran: LintPluginId[];
}
export interface RunLintOptions {
@@ -76,6 +81,15 @@ export async function runLint(opts: RunLintOptions): Promise<LintRunResult> {
// load problems surface as report warnings — the same resolution the server
// uses.
const resolvedBase = composeFrontmatterSchemasConfig(projectDir, baseConfig, pushConfigProblem);
+ const ran = deriveValidationRunSources(resolvedBase, { mode: 'lint' });
+ // Collected structured and summarized once at the end rather than pushed as
+ // formatted lines: a plugin that throws on every document would otherwise add
+ // one warning per file, and the `seenConfigProblems` dedup above cannot
+ // collapse them because each carries its own doc path.
+ const pluginFailures: LintPluginFailure[] = [];
+ const pushPluginFailure = (failure: LintPluginFailure): void => {
+ pluginFailures.push(failure);
+ };
// markdownlint `rules` come from the project's native `.markdownlint.*`
// files, resolved per doc with cli2 cascade semantics (nearest file on the
@@ -150,7 +164,7 @@ export async function runLint(opts: RunLintOptions): Promise<LintRunResult> {
const cfg = configForDoc(rel);
let wasFixed = false;
if (fix && cfg.enabled) {
- const fixedText = fixDocument(text, cfg);
+ const fixedText = fixDocument(text, cfg, rel, pushPluginFailure);
if (fixedText !== text) {
// tmp + rename so an interrupted write can never leave the document
// half-written (mirrors the server's markdownlint-write pattern).
@@ -172,7 +186,7 @@ export async function runLint(opts: RunLintOptions): Promise<LintRunResult> {
}
}
- const diagnostics = await lintDocument(text, cfg, rel);
+ const diagnostics = await lintDocument(text, cfg, rel, pushPluginFailure);
for (const d of diagnostics) {
if (d.severity === 'error') errorCount++;
else warningCount++;
@@ -180,6 +194,8 @@ export async function runLint(opts: RunLintOptions): Promise<LintRunResult> {
files.push({ file: rel, diagnostics, fixed: wasFixed });
}
+ warnings.push(...summarizeLintPluginFailures(pluginFailures));
+
return {
contentDir,
files,
@@ -188,6 +204,7 @@ export async function runLint(opts: RunLintOptions): Promise<LintRunResult> {
errorCount,
warningCount,
fixedCount,
+ ran,
};
}
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index 41f333783..e595e0ae0 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -681,6 +681,7 @@ export {
DEFAULT_LINTER_CONFIG,
DEFAULT_MARKDOWNLINT_CONFIG,
DEFAULT_SCHEMA_DIALECT,
+ deriveValidationRunSources,
displayCategoryForRule,
emptyFrontmatterSchemaText,
type FrontmatterFieldConstraint,
@@ -706,6 +707,7 @@ export {
isSupportedSchemaDialect,
isToolManagedSchemaPath,
LINKS_VALIDATION_SETTINGS,
+ LINT_PLUGIN_IDS,
LINT_PLUGINS,
type LinksValidationSetting,
type LintAuditResponse,
@@ -722,6 +724,8 @@ export {
type LintFixResult,
LintFixResultSchema,
type LintPlugin,
+ type LintPluginFailure,
+ type LintPluginFailureReporter,
type LintPluginId,
type LintPosition,
type LintRange,
@@ -775,7 +779,9 @@ export {
selectGoverningFrontmatterSchemas,
selectOkfFrontmatterSchemas,
summarizeAppliesTo,
+ summarizeLintPluginFailures,
toEffectiveBase,
+ VALIDATION_SOURCES,
type ValidationAuditCountsResponse,
ValidationAuditCountsResponseSchema,
type ValidationAuditResponse,
@@ -786,9 +792,11 @@ export {
ValidationDocCountsSchema,
type ValidationDocResult,
ValidationDocResultSchema,
+ type ValidationRunMode,
type ValidationSource,
type ValidationSourceCounts,
type ValidationSourceKey,
+ validationCoverageLines,
ZERO_SOURCE_COUNTS,
} from './markdown/lint/index.ts';
export { markdownToHtml, mdastToHtml } from './markdown/mdast-to-html.ts';
diff --git a/packages/core/src/markdown/lint/config-schemas.ts b/packages/core/src/markdown/lint/config-schemas.ts
index 1a81aa524..cc667b93e 100644
--- a/packages/core/src/markdown/lint/config-schemas.ts
+++ b/packages/core/src/markdown/lint/config-schemas.ts
@@ -163,9 +163,12 @@ const LintDiagnosticSchema = z.object({
frontmatterProperty: z.string().optional(),
});
+const RunSourcesSchema = z.array(z.string()).optional();
+
export const LintDocResultSchema = z.object({
file: z.string(),
diagnostics: z.array(LintDiagnosticSchema),
+ ran: RunSourcesSchema,
warnings: z.array(z.string()).optional(),
});
@@ -175,6 +178,7 @@ export const LintAuditResponseSchema = z.object({
errorCount: z.number(),
warningCount: z.number(),
warnings: z.array(z.string()),
+ ran: RunSourcesSchema,
});
export type LintDocResult = z.infer<typeof LintDocResultSchema>;
@@ -196,6 +200,7 @@ export const ValidationAuditResponseSchema = z.object({
errorCount: z.number(),
warningCount: z.number(),
warnings: z.array(z.string()),
+ ran: RunSourcesSchema,
});
export type ValidationDocResult = z.infer<typeof ValidationDocResultSchema>;
@@ -238,6 +243,8 @@ export const LintFixResultSchema = z.object({
diagnostics: z.array(LintDiagnosticSchema),
errorCount: z.number(),
warningCount: z.number(),
+ ran: RunSourcesSchema,
+ warnings: z.array(z.string()).optional(),
warning: z.string().optional(),
});
export type LintFixResult = z.infer<typeof LintFixResultSchema>;
diff --git a/packages/core/src/markdown/lint/index.ts b/packages/core/src/markdown/lint/index.ts
index 11b5abdfd..781226a5d 100644
--- a/packages/core/src/markdown/lint/index.ts
+++ b/packages/core/src/markdown/lint/index.ts
@@ -1,40 +1,89 @@
-import { LINT_PLUGINS, type LinterConfig } from './plugins.ts';
+import type { LintPluginFailureReporter } from './plugin-failure.ts';
+import { isLintPluginSelected, LINT_PLUGINS, type LinterConfig } from './plugins.ts';
import type { LintDiagnostic } from './types.ts';
export async function lintDocument(
text: string,
config: LinterConfig,
docName?: string,
+ onPluginFailure?: LintPluginFailureReporter,
): Promise<LintDiagnostic[]> {
if (!config.enabled) return [];
const diagnostics: LintDiagnostic[] = [];
for (const plugin of LINT_PLUGINS) {
+ if (!isLintPluginSelected(config, plugin.id)) continue;
const slice = config.plugins[plugin.id];
- if (!slice.enabled) continue;
try {
diagnostics.push(...(await plugin.lint(text, slice as never, { docName })));
} catch (err) {
console.warn(`[lint] plugin "${plugin.id}" failed${docName ? ` on "${docName}"` : ''}`, err);
+ onPluginFailure?.({
+ source: plugin.id,
+ phase: 'lint',
+ ...(docName === undefined ? {} : { docName }),
+ message: errorMessage(err),
+ });
}
}
return diagnostics;
}
-export function fixDocument(text: string, config: LinterConfig): string {
+export function fixDocument(
+ text: string,
+ config: LinterConfig,
+ docName?: string,
+ onPluginFailure?: LintPluginFailureReporter,
+): string {
if (!config.enabled) return text;
let out = text;
for (const plugin of LINT_PLUGINS) {
+ if (!isLintPluginSelected(config, plugin.id) || !plugin.fix) continue;
const slice = config.plugins[plugin.id];
- if (!slice.enabled || !plugin.fix) continue;
try {
out = plugin.fix(out, slice as never);
} catch (err) {
- console.warn(`[lint] plugin "${plugin.id}" fix failed`, err);
+ console.warn(
+ `[lint] plugin "${plugin.id}" fix failed${docName ? ` on "${docName}"` : ''}`,
+ err,
+ );
+ onPluginFailure?.({
+ source: plugin.id,
+ phase: 'fix',
+ ...(docName === undefined ? {} : { docName }),
+ message: errorMessage(err),
+ });
}
}
return out;
}
+/** Bounded because an object throw's `JSON.stringify` is unbounded by
+ * construction, and this string lands verbatim in the agent-context-bound
+ * `warnings[]` channel. */
+const MAX_FAILURE_MESSAGE_LENGTH = 500;
+
+function errorMessage(error: unknown): string {
+ const raw = renderErrorMessage(error);
+ return raw.length > MAX_FAILURE_MESSAGE_LENGTH
+ ? `${raw.slice(0, MAX_FAILURE_MESSAGE_LENGTH)}… (truncated)`
+ : raw;
+}
+
+function renderErrorMessage(error: unknown, depth = 0): string {
+ if (error instanceof Error) {
+ return error.cause === undefined || depth >= 3
+ ? error.message
+ : `${error.message} (cause: ${renderErrorMessage(error.cause, depth + 1)})`;
+ }
+ if (typeof error === 'object' && error !== null) {
+ try {
+ const json = JSON.stringify(error);
+ if (json !== undefined) return json;
+ } catch {}
+ }
+ return String(error);
+}
+
export {
type AppliesToPatternSummary,
type AppliesToSummary,
@@ -122,6 +171,11 @@ export {
type OkfRuleId,
type OkfRuleToggles,
} from './okf-rule-meta.ts';
+export {
+ type LintPluginFailure,
+ type LintPluginFailureReporter,
+ summarizeLintPluginFailures,
+} from './plugin-failure.ts';
export {
DEFAULT_LINTER_CONFIG,
type GoverningFrontmatterSchema,
@@ -172,6 +226,8 @@ export {
DEFAULT_LINKS_VALIDATION,
isFrontmatterScoped,
LINKS_VALIDATION_SETTINGS,
+ LINT_PLUGIN_IDS,
+ VALIDATION_SOURCES,
} from './types.ts';
export {
countDiagnosticsBySource,
@@ -180,3 +236,8 @@ export {
type ValidationSourceKey,
ZERO_SOURCE_COUNTS,
} from './validation-counts.ts';
+export {
+ deriveValidationRunSources,
+ type ValidationRunMode,
+ validationCoverageLines,
+} from './validation-run-sources.ts';
diff --git a/packages/core/src/markdown/lint/plugin-failure.ts b/packages/core/src/markdown/lint/plugin-failure.ts
new file mode 100644
index 000000000..e7d694890
--- /dev/null
+++ b/packages/core/src/markdown/lint/plugin-failure.ts
@@ -0,0 +1,33 @@
+import type { LintPluginId } from './types.ts';
+
+export interface LintPluginFailure {
+ source: LintPluginId;
+ phase: 'lint' | 'fix';
+ docName?: string;
+ message: string;
+}
+
+export type LintPluginFailureReporter = (failure: LintPluginFailure) => void;
+
+function lintPluginFailurePrefix(failure: LintPluginFailure): string {
+ return `source "${failure.source}" ${failure.phase} failed`;
+}
+
+export function formatLintPluginFailure(failure: LintPluginFailure): string {
+ const location = failure.docName === undefined ? '' : ` on "${failure.docName}"`;
+ return `${lintPluginFailurePrefix(failure)}${location}: ${failure.message}`;
+}
+
+export function summarizeLintPluginFailures(failures: readonly LintPluginFailure[]): string[] {
+ const byFault = new Map<string, { first: LintPluginFailure; docNames: Set<string> }>();
+ for (const failure of failures) {
+ const key = [failure.source, failure.phase, failure.message].join('\u0000');
+ const seen = byFault.get(key) ?? { first: failure, docNames: new Set<string>() };
+ if (failure.docName !== undefined) seen.docNames.add(failure.docName);
+ byFault.set(key, seen);
+ }
+ return [...byFault.values()].map(({ first, docNames }) => {
+ if (docNames.size <= 1) return formatLintPluginFailure(first);
+ return `${lintPluginFailurePrefix(first)} on ${docNames.size} documents (first: "${first.docName}"): ${first.message}`;
+ });
+}
diff --git a/packages/core/src/markdown/lint/plugins.ts b/packages/core/src/markdown/lint/plugins.ts
index 09c8450ee..e7455ccf7 100644
--- a/packages/core/src/markdown/lint/plugins.ts
+++ b/packages/core/src/markdown/lint/plugins.ts
@@ -135,6 +135,10 @@ export const DEFAULT_LINTER_CONFIG: LinterConfig = {
) as LinterConfig['plugins'],
};
+export function isLintPluginSelected(config: LinterConfig, id: LintPluginId): boolean {
+ return config.enabled && config.plugins[id]?.enabled === true;
+}
+
export function selectGoverningFrontmatterSchemas(
config: LinterConfig | null,
docName: string | undefined,
diff --git a/packages/core/src/markdown/lint/types.ts b/packages/core/src/markdown/lint/types.ts
index 6836b962d..2cffecae6 100644
--- a/packages/core/src/markdown/lint/types.ts
+++ b/packages/core/src/markdown/lint/types.ts
@@ -39,7 +39,7 @@ export function isFrontmatterScoped(diagnostic: Pick<LintDiagnostic, 'frontmatte
return diagnostic.frontmatterScope !== undefined;
}
-const VALIDATION_SOURCES = [...LINT_PLUGIN_IDS, 'links'] as const;
+export const VALIDATION_SOURCES = [...LINT_PLUGIN_IDS, 'links'] as const;
export type ValidationSource = (typeof VALIDATION_SOURCES)[number];
export type LocalTargetKind = 'document' | 'file' | 'unknown';
diff --git a/packages/core/src/markdown/lint/validation-run-sources.ts b/packages/core/src/markdown/lint/validation-run-sources.ts
new file mode 100644
index 000000000..0e527eebb
--- /dev/null
+++ b/packages/core/src/markdown/lint/validation-run-sources.ts
@@ -0,0 +1,32 @@
+import { isLintPluginSelected, LINT_PLUGINS, type LinterConfig } from './plugins.ts';
+import type { LinksValidationSetting, LintPluginId, ValidationSource } from './types.ts';
+
+export type ValidationRunMode =
+ | { mode: 'lint' }
+ | { mode: 'audit'; linksValidation: LinksValidationSetting };
+
+export function deriveValidationRunSources(
+ config: LinterConfig,
+ run: { mode: 'lint' },
+): LintPluginId[];
+export function deriveValidationRunSources(
+ config: LinterConfig,
+ run: ValidationRunMode,
+): ValidationSource[];
+export function deriveValidationRunSources(
+ config: LinterConfig,
+ run: ValidationRunMode,
+): ValidationSource[] {
+ const sources: ValidationSource[] = LINT_PLUGINS.filter((plugin) =>
+ isLintPluginSelected(config, plugin.id),
+ ).map((plugin) => plugin.id);
+
+ if (run.mode === 'audit' && run.linksValidation !== 'off') sources.push('links');
+ return sources;
+}
+
+export function validationCoverageLines(ran: readonly string[] | undefined): string[] {
+ if (ran === undefined) return [];
+ if (ran.length === 0) return ['No checks ran.'];
+ return [`Checks run: ${ran.join(', ')}.`];
+}
diff --git a/packages/server/assets/skills/project/SKILL.md b/packages/server/assets/skills/project/SKILL.md
index d54c723b4..697adcd46 100644
--- a/packages/server/assets/skills/project/SKILL.md
+++ b/packages/server/assets/skills/project/SKILL.md
@@ -21,11 +21,11 @@ OpenKnowledge (OK) is a markdown-CRDT collaboration platform exposed via MCP. Th
3. **Preview / open a doc — determine your ONE surface FIRST (once per session).** Stop at first match: **`OK_DESKTOP_TERMINAL` or `OK_HOSTED_AGENT` set** → you're inside OpenKnowledge (desktop terminal / in-app agent panel) → `ok open <name>` (switches the window the user is already looking at); never paste a `localhost` URL into your reply here · in-app browser (Claude Code Desktop's Browser pane, Cursor, Codex) → `preview_url`, then open/navigate it to the doc · else plain CLI → `ok open <name>`. `ok open <name>` opens a doc or folder (auto-detected); `--skill <name>` for a skill. The `previewUrl` field is a route id, **not** your open mechanism. Don't `preview_screenshot` to confirm edits. Full Step-0 procedure + per-surface how-to: `references/preview.md`.
4. **Knowledge layers:** capturing a source (ingest), synthesizing findings (research), promoting a decision (consolidate) — procedures, **not tool calls**; there is no `ingest` tool. Ingest ships here (`references/ingest-and-sources.md`); research + consolidate come with the `knowledge-base` pack. Layer model + packs: `references/starter-packs.md`.
5. **Direct questions:** a plain business question ("which customers…", "what did we decide about…") routes to `search` / `exec` + a cited chat answer — no "research" keyword needed. Persist only when durable + multi-doc + not already covered, and *offer* first. See `references/corpus-qa.md`.
-6. **Authoring or improving a skill** ("write/make a skill", "improve this skill", "turn this into a skill"): STOP and invoke the **`/open-knowledge-write-skill`** skill — it owns scope choice (project vs global), the SKILL.md contract, evaluation, and install; don't improvise here. Author through the `skill` target (`write({ skill })`), never a raw path document. Skills are REAL folders under the editor dirs' `skills/` (`.claude` · `.cursor` · `.codex` · `.github` · `.opencode` · `.pi` · `.agents`): ONE is the source, the rest are managed copies/symlinks. **Read and edit via the tools (`skills`, `edit({ skill })`) — they route to the source.** Never hand-edit a non-source copy: unedited copies refresh from the source, and editing one forks it into its own skill that stops refreshing.
+6. **Authoring or improving a skill** ("write/make/improve a skill", "turn this into a skill"): STOP and invoke **`/open-knowledge-write-skill`** for scope (project/global), contract, evaluation, and install. Author through `write({ skill })`, never a document path. Skills are real folders under editor `skills/` dirs (`.claude` · `.cursor` · `.codex` · `.github` · `.opencode` · `.pi` · `.agents`): one source plus managed copies/symlinks. **Read/edit via `skills` and `edit({ skill })` — they route to the source.** Never hand-edit a non-source copy: managed copies refresh from the source; editing one forks it and stops refresh.
## Tool index — 21 tools (router; the MCP tool descriptions carry each tool's full contract)
-- **Reads** — `exec` (primary; read-only `cat`/`ls`/`grep`/… plus frontmatter/backlink/history enrichment; one command or one pipe, not a shell), `search` (ranked BM25 + recency), `history` (doc versions), `links` (`kind: backlinks|forward|dead|orphans|hubs|suggest`, or an array for one call), `skills` (search + read: `query` → skills.sh; omit `name` to LIST managed (Project + Global); `name` READs one — by `name`+`scope`, never path), `config` (resolved config), `palette` (authoring forms + `html preview` starters + theme tokens; `palette({ components })` for JSX schemas), `preview_url` (browser preview URL on demand), `share_link` (GitHub-substrate share URL; read-only, errors without a GitHub remote), `lint` (markdown-lint violations: `document` for one doc, omit for the project; `fix: true` with `document` auto-fixes fixable rules in place — attributed, live in the preview; the rest need `edit`/`write`), `audit` (every lint violation + broken internal link in one read-only report, by source file with lines; `path` scopes; for link VALIDATION use this, not `links`; caveats in `references/linking.md`).
+- **Reads** — `exec` (primary; read-only `cat`/`ls`/`grep`/… plus frontmatter/backlink/history enrichment; one command or one pipe, not a shell), `search` (ranked BM25 + recency), `history` (doc versions), `links` (`kind: backlinks|forward|dead|orphans|hubs|suggest`, or an array for one call), `skills` (search + read: `query` → skills.sh; omit `name` to LIST managed (Project + Global); `name` READs one — by `name`+`scope`, never path), `config` (resolved config), `palette` (authoring forms + `html preview` starters + theme tokens; `palette({ components })` for JSX schemas), `preview_url` (browser preview URL on demand), `share_link` (GitHub-substrate share URL; read-only, errors without a GitHub remote), `lint` (markdown-lint violations: `document` for one doc, omit for the project; `fix: true` with `document` auto-fixes fixable rules in place — attributed, live in the preview; the rest need `edit`/`write`), `audit` (every lint violation + broken internal link in one read-only report, by source file with lines; `path` scopes; for link VALIDATION use this, not `links`; caveats in `references/linking.md`). **Read `ran` on successful `lint`/`audit` results to see which enabled source families were selected. A family absent from `ran` was not checked, and `[]` means no checks were selected at all.**
- **Writes** — four native CRUD verbs, polymorphic over `document` / `folder` / `template` / `skill` / `asset` (pass EXACTLY ONE target, nested under its address key): `write` (create/overwrite; `write({ skill: {…} })` authors a skill as a REAL folder at the project's default skill home — live immediately for that folder's agent), `edit` (body find/replace/frontmatter merge-patch; no asset), `delete` (remove), `move` (move/rename, rewrites referrers; a skill also takes `scope`/`toScope` for Project↔Global — history resets, re-`install`). Output mirrors the input key; the preview envelope (`previewUrl`, `warning`) stays top-level. Plus `install` (WHERE a `skill` lives: `add`/`remove` locations additively — editor ids, `agents`, or custom roots; `mode` + `convert` re-form ONLY the locations named; `source` moves the real folder. The source folder IS the skill — no "uninstall everywhere"; a skill dies only via `delete`), `import` (acquire a skill-dir into `add`'s locations; scripts never run), `checkpoint` (named version), and `restore_version` (roll back). A folder's frontmatter is open-shape and self-only (does NOT cascade); templates are what new docs start with.
- **Conflicts** — `conflicts` (`kind: list|content`), `resolve_conflict` (write a resolution + commit; destructive). See `references/conflict-resolution.md`.
@@ -83,7 +83,7 @@ Link every noun-phrase that names another document — `[text](./relative/path.m
## Folders, frontmatter, templates
-Every `.md` / `.mdx` needs YAML frontmatter — `title` + `description` required, `tags` recommended. **OKF projects (`okf` pack) are the exception:** the pack's rules win — a non-root `index.md` carries NO frontmatter (the `frontmatter-reserved-index` lint warns on any key), `log.md` needs none, and concept docs need only a non-empty `type`; `title`/`description` are optional there. Two **opt-in, nested** folder mechanisms: folder frontmatter (`<folder>/.ok/frontmatter.yml` — the folder's own open-shape properties; self-only, does NOT cascade into child docs) and templates (`<folder>/.ok/templates/` — what new docs start with). Most folders have NO `.ok/`. A doc's frontmatter is exactly its own on-disk YAML. Structural model + the full pre-write checklist: `references/folder-model.md`. Template authoring + folder editing: `references/template-authoring.md`. Frontmatter-vs-body edit rules: `references/doc-editing.md`.
+Every `.md` / `.mdx` needs YAML frontmatter — `title` + `description` required, `tags` recommended. **OKF projects (`okf` pack) are the exception:** pack rules win — a non-root `index.md` carries NO frontmatter (the `frontmatter-reserved-index` lint warns on any key), `log.md` needs none, and concept docs need only a non-empty `type`; `title`/`description` are optional there. Two **opt-in, nested** folder mechanisms: folder frontmatter (`<folder>/.ok/frontmatter.yml` — the folder's own open-shape properties; self-only, does NOT cascade into child docs) and templates (`<folder>/.ok/templates/` — what new docs start with). Most folders have NO `.ok/`. A doc's frontmatter is exactly its own on-disk YAML. Structural model + the full pre-write checklist: `references/folder-model.md`. Template authoring + folder editing: `references/template-authoring.md`. Frontmatter-vs-body edit rules: `references/doc-editing.md`.
- **Read the folder before writing (MUST).** Before creating/editing docs in a folder, call `exec("ls -A <folder>")` once per folder per session — it returns the folder's `title`/`description`/`tags` + `templates_available`. Skipping it lands docs that violate folder discipline. (If a folder has no frontmatter AND no templates AND the repo has substantial content elsewhere, it isn't onboarded — run `references/onboard-existing-repo.md` first.)
- **Use a template when one fits (MUST).** Instantiate via `write({ document: { path, template } })`; inherited templates count. Skip only when none match or the user asked for free-form (note why in chat). Create templates proactively when a shape recurs.
diff --git a/packages/server/src/api-extension.ts b/packages/server/src/api-extension.ts
index 45088df6c..8add4857d 100644
--- a/packages/server/src/api-extension.ts
+++ b/packages/server/src/api-extension.ts
@@ -120,6 +120,7 @@ import {
type LinterConfig,
LintFixRequestSchema,
LintFixResultSchema,
+ type LintPluginFailure,
type LintViolationWarning,
LOCAL_DIR,
LocalOpAuthCancelRequestSchema,
@@ -249,6 +250,7 @@ import {
scanHeadingLine,
skillLiveDocName,
stripFrontmatter,
+ summarizeLintPluginFailures,
TEMPLATE_NAME_REGEX,
TemplateDeleteSuccessSchema,
TemplateGetSuccessSchema,
@@ -22475,11 +22477,15 @@ export function createApiExtension(
onConfigProblem: (problem) => configWarnings.push(problem),
liveSourceFor: liveLintSourceFor,
});
+ // Absent when there is nothing to report, matching the schema's
+ // documented contract and the three sibling lint emitters. `ran` is the
+ // field that is always present; do not carry that rule across to this one.
+ const lintWarnings = [...configWarnings, ...summarizeLintPluginFailures(result.failures)];
successResponse(
res,
200,
LintDocResultSchema,
- configWarnings.length > 0 ? { ...result, warnings: configWarnings } : result,
+ { ...result, ...(lintWarnings.length > 0 ? { warnings: lintWarnings } : {}) },
{ handler: 'lint' },
);
} catch (e) {
@@ -22741,16 +22747,19 @@ export function createApiExtension(
// against the bytes we actually mutate. `fixDocument` delegates to
// upstream markdownlint's `applyFixes` — we author no fix logic.
const source = session.dc.document.getText('source').toString();
- const { cfg, before, fixed } = await lintAndFixSource({
+ const configWarnings: string[] = [];
+ const { cfg, before, fixed, ran, failures } = await lintAndFixSource({
projectDir: projectDir ?? contentDir,
contentDir,
baseConfig,
docRelPath,
source,
+ onConfigProblem: (problem) => configWarnings.push(problem),
});
let after = before;
let reLintWarning: string | undefined;
+ const reLintFailures: LintPluginFailure[] = [];
if (fixed !== source) {
// Land the fixed bytes through the sanctioned agent-write spine:
// attributed to the calling agent (never the anonymous `file-system`
@@ -22828,6 +22837,7 @@ export function createApiExtension(
session.dc.document.getText('source').toString(),
cfg,
docRelPath,
+ (failure) => reLintFailures.push(failure),
);
} catch (relintErr) {
reLintWarning = `Re-lint after fix failed: ${relintErr instanceof Error ? relintErr.message : String(relintErr)}`;
@@ -22843,6 +22853,14 @@ export function createApiExtension(
const warningCount = after.length - errorCount;
// Net problems the auto-fix resolved (clamped — a fix never nets negative).
const fixedCount = Math.max(0, before.length - after.length);
+ // One channel for everything that made this run less than complete:
+ // config faults, plugin throws from the fix + re-lint passes, and the
+ // re-lint failure the deprecated singular `warning` also carries.
+ const responseWarnings = [
+ ...configWarnings,
+ ...summarizeLintPluginFailures([...failures, ...reLintFailures]),
+ ...(reLintWarning ? [reLintWarning] : []),
+ ];
successResponse(
res,
@@ -22854,6 +22872,11 @@ export function createApiExtension(
diagnostics: after,
errorCount,
warningCount,
+ ran,
+ ...(responseWarnings.length > 0 ? { warnings: responseWarnings } : {}),
+ // Emitted in parallel with its copy in `warnings` for one
+ // deprecation window: the text header still branches on it, and
+ // dropping it now would break every caller that reads it today.
...(reLintWarning ? { warning: reLintWarning } : {}),
},
{ handler: 'lint-fix' },
diff --git a/packages/server/src/lint/audit-cache.ts b/packages/server/src/lint/audit-cache.ts
index 93948fbc5..6728bca32 100644
--- a/packages/server/src/lint/audit-cache.ts
+++ b/packages/server/src/lint/audit-cache.ts
@@ -18,8 +18,11 @@
* worker-thread fan-out or an all-rules superset filtered per active rule set;
* both are larger changes than this cache.
*
- * Docs read from a live CRDT overlay are never cached — their bytes move
- * without touching the disk stamp the key rests on.
+ * Two classes are never cached. Docs read from a live CRDT overlay, because
+ * their bytes move without touching the disk stamp the key rests on. And docs
+ * whose lint degraded — a selected plugin threw — because an entry holds
+ * diagnostics alone, so replaying one would drop the failure that says the set
+ * is incomplete and report the doc clean until its stamp or config moves.
*/
import { createHash } from 'node:crypto';
diff --git a/packages/server/src/lint/audit.test.ts b/packages/server/src/lint/audit.test.ts
index e17651e5c..4ea457737 100644
--- a/packages/server/src/lint/audit.test.ts
+++ b/packages/server/src/lint/audit.test.ts
@@ -7,8 +7,12 @@
import { mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
-import { DEFAULT_LINTER_CONFIG, type LinterConfig } from '@inkeep/open-knowledge-core';
-import { afterEach, beforeEach, describe, expect, test } from 'vitest';
+import {
+ DEFAULT_LINTER_CONFIG,
+ LINT_PLUGINS,
+ type LinterConfig,
+} from '@inkeep/open-knowledge-core';
+import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest';
import { auditProject, lintDoc } from './audit.ts';
import { AuditCache } from './audit-cache.ts';
@@ -36,6 +40,7 @@ beforeEach(() => {
root = realpathSync(mkdtempSync(join(tmpdir(), 'ok-audit-')));
});
afterEach(() => {
+ vi.restoreAllMocks();
rmSync(root, { recursive: true, force: true });
});
@@ -50,6 +55,8 @@ describe('lintDoc', () => {
});
expect(result.file).toBe('a.md');
expect(result.diagnostics.some((d) => d.code === 'MD010')).toBe(true);
+ expect(result.ran).toEqual(['markdownlint']);
+ expect(result.failures).toEqual([]);
});
test('honors the native .markdownlint.json (disables a rule)', async () => {
@@ -65,9 +72,100 @@ describe('lintDoc', () => {
});
expect(result.diagnostics.some((d) => d.code === 'MD010')).toBe(false);
});
+
+ test('keeps a selected plugin in ran when it degrades and reports the failure', async () => {
+ write('a.md', DOC_WITH_TAB);
+ const markdownlint = LINT_PLUGINS.find((plugin) => plugin.id === 'markdownlint');
+ if (!markdownlint) throw new Error('markdownlint plugin missing');
+ vi.spyOn(console, 'warn').mockImplementation(() => {});
+ vi.spyOn(markdownlint, 'lint').mockRejectedValue(new Error('boom'));
+
+ const result = await lintDoc({
+ projectDir: root,
+ contentDir: root,
+ baseConfig: base,
+ docRelPath: 'a.md',
+ });
+
+ expect(result.ran).toEqual(['markdownlint']);
+ expect(result.failures).toEqual([
+ { source: 'markdownlint', phase: 'lint', docName: 'a.md', message: 'boom' },
+ ]);
+ });
+
+ test('a degraded lint is not cached — a hit would replay it as clean', async () => {
+ write('a.md', DOC_WITH_TAB);
+ const markdownlint = LINT_PLUGINS.find((plugin) => plugin.id === 'markdownlint');
+ if (!markdownlint) throw new Error('markdownlint plugin missing');
+ vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const lint = vi.spyOn(markdownlint, 'lint').mockRejectedValue(new Error('boom'));
+ const cache = new AuditCache();
+
+ const opts = {
+ projectDir: root,
+ contentDir: root,
+ baseConfig: base,
+ docRelPath: 'a.md',
+ cache,
+ };
+ await lintDoc(opts);
+ // Same doc, same config, same stamp: a cached entry would be served here and
+ // would carry the diagnostics without the failure that says they are partial.
+ const second = await lintDoc(opts);
+ expect(second.failures).toHaveLength(1);
+ expect(cache.stats().entries).toBe(0);
+
+ // Once the plugin recovers the walk caches again — the refusal self-heals
+ // rather than sticking to a key nothing invalidates.
+ lint.mockRestore();
+ const recovered = await lintDoc(opts);
+ expect(recovered.failures).toEqual([]);
+ expect(cache.stats().entries).toBe(1);
+ });
});
describe('auditProject', () => {
+ test('emits an explicit empty selection when linting is disabled', async () => {
+ write('clean.md', CLEAN_DOC);
+ const audit = await auditProject({
+ projectDir: root,
+ contentDir: root,
+ baseConfig: DEFAULT_LINTER_CONFIG,
+ });
+ expect(audit.ran).toEqual([]);
+ });
+
+ test('a plugin failing across the corpus collapses to one counted warning', async () => {
+ for (const name of ['a.md', 'b.md', 'c.md']) write(name, DOC_WITH_TAB);
+ const markdownlint = LINT_PLUGINS.find((plugin) => plugin.id === 'markdownlint');
+ if (!markdownlint) throw new Error('markdownlint plugin missing');
+ vi.spyOn(console, 'warn').mockImplementation(() => {});
+ vi.spyOn(markdownlint, 'lint').mockRejectedValue(new Error('boom'));
+
+ const audit = await auditProject({ projectDir: root, contentDir: root, baseConfig: base });
+
+ // One line for the fault, not one per document it hit — the failure is
+ // systematic, and the agent channels that render this array are capped.
+ expect(audit.warnings).toEqual([
+ 'source "markdownlint" lint failed on 3 documents (first: "a.md"): boom',
+ ]);
+ expect(audit.ran).toEqual(['markdownlint']);
+ });
+
+ test('plugin failures stay distinct from generic configuration warnings', async () => {
+ write('a.md', DOC_WITH_TAB);
+ write('.markdownlint.json', '{ invalid json');
+ const markdownlint = LINT_PLUGINS.find((plugin) => plugin.id === 'markdownlint');
+ if (!markdownlint) throw new Error('markdownlint plugin missing');
+ vi.spyOn(console, 'warn').mockImplementation(() => {});
+ vi.spyOn(markdownlint, 'lint').mockRejectedValue(new Error('boom'));
+
+ const audit = await auditProject({ projectDir: root, contentDir: root, baseConfig: base });
+
+ expect(audit.warnings).toContain('source "markdownlint" lint failed on "a.md": boom');
+ expect(audit.warnings.some((warning) => warning.includes('.markdownlint.json'))).toBe(true);
+ });
+
test('includes only docs that have diagnostics, counts all files', async () => {
write('dirty.md', DOC_WITH_TAB);
write('clean.md', CLEAN_DOC);
@@ -76,6 +174,7 @@ describe('auditProject', () => {
expect(audit.files.map((f) => f.file)).toEqual(['dirty.md']);
expect(audit.warningCount).toBeGreaterThan(0);
expect(audit.errorCount).toBe(0);
+ expect(audit.ran).toEqual(['markdownlint']);
});
test('respects .okignore exclusions', async () => {
@@ -124,6 +223,7 @@ describe('auditProject', () => {
expect(audit.warnings).toEqual([
expect.stringContaining('refusing audit scope outside the content directory'),
]);
+ expect(audit.ran).toEqual([]);
});
test('refuses a relative targetPath that escapes the content dir', async () => {
@@ -137,6 +237,7 @@ describe('auditProject', () => {
expect(audit.warnings).toEqual([
expect.stringContaining('refusing audit scope outside the content directory'),
]);
+ expect(audit.ran).toEqual([]);
});
test('skips hidden path segments — docs there are not addressable to fix or navigate', async () => {
@@ -166,6 +267,7 @@ describe('auditProject', () => {
expect(audit.warnings).toEqual([
expect.stringContaining('refusing audit scope under a hidden path segment'),
]);
+ expect(audit.ran).toEqual([]);
});
test('liveSourceFor overrides disk for loaded docs; null falls back to disk', async () => {
diff --git a/packages/server/src/lint/audit.ts b/packages/server/src/lint/audit.ts
index 87632500e..117d22ff3 100644
--- a/packages/server/src/lint/audit.ts
+++ b/packages/server/src/lint/audit.ts
@@ -10,11 +10,15 @@ import { readdirSync, readFileSync, realpathSync, statSync } from 'node:fs';
import { isAbsolute, join, relative, resolve } from 'node:path';
import { setImmediate as yieldToEventLoop } from 'node:timers/promises';
import {
+ deriveValidationRunSources,
fixDocument,
type LintDiagnostic,
type LinterConfig,
+ type LintPluginFailure,
+ type LintPluginId,
lintDocument,
SUPPORTED_DOC_EXTENSIONS,
+ summarizeLintPluginFailures,
} from '@inkeep/open-knowledge-core';
import { SymlinkEscapeError } from '../apply-managed-rename.ts';
import { createContentFilter } from '../content-filter.ts';
@@ -23,18 +27,29 @@ import { AuditCache } from './audit-cache.ts';
import { unmatchedAppliesToProblems } from './frontmatter-schemas.ts';
import { composeFrontmatterSchemasConfig, resolveEffectiveLinterConfig } from './resolve-config.ts';
-export interface FileLintResult {
+interface FileLintResult {
/** Path relative to `contentDir`. */
file: string;
diagnostics: LintDiagnostic[];
}
+interface LintDocRunResult extends FileLintResult {
+ ran: LintPluginId[];
+ /**
+ * Structured rather than pre-rendered: a whole-project walk collapses the
+ * per-doc repeats of one systematic fault into a single counted line, which
+ * it cannot do once each failure has been formatted with its own docName.
+ */
+ failures: LintPluginFailure[];
+}
+
export interface AuditResult {
files: FileLintResult[];
fileCount: number;
errorCount: number;
warningCount: number;
warnings: string[];
+ ran: LintPluginId[];
}
/**
@@ -110,7 +125,7 @@ export interface AuditOptions {
* per-doc effective config. */
export async function lintDoc(
opts: AuditOptions & { docRelPath: string; onConfigProblem?: (problem: string) => void },
-): Promise<FileLintResult> {
+): Promise<LintDocRunResult> {
const { projectDir, contentDir, baseConfig, docRelPath, onConfigProblem, liveSourceFor, cache } =
opts;
const live = liveSourceFor?.(docRelPath) ?? null;
@@ -123,8 +138,18 @@ export async function lintDoc(
projectDir,
onProblem: onConfigProblem,
});
+ const ran = deriveValidationRunSources(cfg, { mode: 'lint' });
+ const failures: LintPluginFailure[] = [];
+ const onPluginFailure = (failure: LintPluginFailure): void => {
+ failures.push(failure);
+ };
if (live !== null) {
- return { file: docRelPath, diagnostics: await lintDocument(live, cfg, docRelPath) };
+ return {
+ file: docRelPath,
+ diagnostics: await lintDocument(live, cfg, docRelPath, onPluginFailure),
+ ran,
+ failures,
+ };
}
// Symlinks inside the content dir are supported (realpath-based identity),
// but an escape must be refused before the read: lint diagnostics echo
@@ -148,13 +173,19 @@ export async function lintDoc(
}
if (cacheKey !== null) {
const cached = cache.get(cacheKey);
- if (cached !== null) return { file: docRelPath, diagnostics: cached };
+ if (cached !== null) return { file: docRelPath, diagnostics: cached, ran, failures };
}
}
const text = readFileSync(canonical, 'utf-8');
- const diagnostics = await lintDocument(text, cfg, docRelPath);
- if (cacheKey !== null) cache?.set(cacheKey, diagnostics);
- return { file: docRelPath, diagnostics };
+ const diagnostics = await lintDocument(text, cfg, docRelPath, onPluginFailure);
+ // A degraded run is never cached: the entry stores diagnostics ALONE, so a hit
+ // would replay the incomplete set with no failure beside it and report the doc
+ // clean until its mtime, size, or config fingerprint moves. The cost is that a
+ // plugin failing across the corpus keeps every walk cold until it is fixed;
+ // the payoff is that the failure self-heals on the next audit instead of
+ // sticking to a cache key nothing invalidates.
+ if (cacheKey !== null && failures.length === 0) cache?.set(cacheKey, diagnostics);
+ return { file: docRelPath, diagnostics, ran, failures };
}
/**
@@ -172,16 +203,27 @@ export async function lintAndFixSource(
source: string;
onConfigProblem?: (problem: string) => void;
},
-): Promise<{ cfg: LinterConfig; before: LintDiagnostic[]; fixed: string }> {
+): Promise<{
+ cfg: LinterConfig;
+ before: LintDiagnostic[];
+ fixed: string;
+ ran: LintPluginId[];
+ failures: LintPluginFailure[];
+}> {
const { projectDir, contentDir, baseConfig, docRelPath, source, onConfigProblem } = opts;
const cfg = resolveEffectiveLinterConfig(contentDir, baseConfig, {
docName: docRelPath,
projectDir,
onProblem: onConfigProblem,
});
- const before = await lintDocument(source, cfg, docRelPath);
- const fixed = fixDocument(source, cfg);
- return { cfg, before, fixed };
+ const ran = deriveValidationRunSources(cfg, { mode: 'lint' });
+ const failures: LintPluginFailure[] = [];
+ const onPluginFailure = (failure: LintPluginFailure): void => {
+ failures.push(failure);
+ };
+ const before = await lintDocument(source, cfg, docRelPath, onPluginFailure);
+ const fixed = fixDocument(source, cfg, docRelPath, onPluginFailure);
+ return { cfg, before, fixed, ran, failures };
}
/**
@@ -239,6 +281,7 @@ export async function auditProject(
): Promise<AuditResult> {
const { projectDir, contentDir, baseConfig, targetPath, auditGeneration } = opts;
const warnings: string[] = [];
+ let ran: LintPluginId[] = [];
// Sampled before the first yield, so it is the generation every doc in this
// walk is linted under until proven otherwise.
const startGeneration = auditGeneration?.();
@@ -255,19 +298,19 @@ export async function auditProject(
const scopeRel = relative(contentDir, scope.path);
if (scopeRel.startsWith('..') || isAbsolute(scopeRel)) {
warnings.push(`refusing audit scope outside the content directory: ${targetPath ?? ''}`);
- return { files: [], fileCount: 0, errorCount: 0, warningCount: 0, warnings };
+ return { files: [], fileCount: 0, errorCount: 0, warningCount: 0, warnings, ran: [] };
}
// Same hidden-segment rule the walk applies per entry: a scope like
// `.ok/skills` names docs the write path cannot address, so auditing it
// would produce unfixable, unnavigable rows.
if (scopeRel.split('/').some((segment) => segment.startsWith('.'))) {
warnings.push(`refusing audit scope under a hidden path segment: ${targetPath ?? ''}`);
- return { files: [], fileCount: 0, errorCount: 0, warningCount: 0, warnings };
+ return { files: [], fileCount: 0, errorCount: 0, warningCount: 0, warnings, ran: [] };
}
try {
if (!isWithinContentDir(realpathSync(scope.path), realpathSync(contentDir))) {
warnings.push(`symlink-escape: audit scope resolves outside the content directory`);
- return { files: [], fileCount: 0, errorCount: 0, warningCount: 0, warnings };
+ return { files: [], fileCount: 0, errorCount: 0, warningCount: 0, warnings, ran: [] };
}
} catch {
// Scope path missing or unreadable: fall through — the walk/read reports it.
@@ -288,6 +331,7 @@ export async function auditProject(
docFiles.sort();
const files: FileLintResult[] = [];
+ const pluginFailures: LintPluginFailure[] = [];
let errorCount = 0;
let warningCount = 0;
// Config problems repeat for every doc a governing file covers — dedupe.
@@ -300,6 +344,7 @@ export async function auditProject(
// Frontmatter schema loading is doc-independent — resolve once for the whole
// audit; the per-doc resolution sees resolved entries and skips the reads.
const auditBase = composeFrontmatterSchemasConfig(projectDir, baseConfig, onConfigProblem);
+ ran = deriveValidationRunSources(auditBase, { mode: 'lint' });
// Zero-match globs only make sense against the FULL doc set — a sub-path
// audit would flag every pattern scoped to a folder outside the target.
const fmSlice = auditBase.plugins.frontmatter;
@@ -320,7 +365,7 @@ export async function auditProject(
}
sliceStartedAt = performance.now();
}
- let result: FileLintResult;
+ let result: LintDocRunResult;
try {
result = await lintDoc({
projectDir,
@@ -335,12 +380,15 @@ export async function auditProject(
warnings.push(`could not lint ${rel}: ${errMsg(e)}`);
continue;
}
+ pluginFailures.push(...result.failures);
for (const d of result.diagnostics) {
if (d.severity === 'error') errorCount++;
else warningCount++;
}
// Only include files that actually have diagnostics in the audit payload.
- if (result.diagnostics.length > 0) files.push(result);
+ if (result.diagnostics.length > 0) {
+ files.push({ file: result.file, diagnostics: result.diagnostics });
+ }
}
// Small corpora may finish without crossing the time-slice yield above. The
@@ -349,7 +397,13 @@ export async function auditProject(
throw new AuditSupersededError();
}
- return { files, fileCount: docFiles.length, errorCount, warningCount, warnings };
+ // One line per distinct fault, not per doc it hit: a plugin that throws on
+ // every document would otherwise make this array grow with the corpus and
+ // flood the agent channels that render it.
+ // MCP capAuditWarnings applies cap-safe priority at the carrier boundary.
+ warnings.push(...summarizeLintPluginFailures(pluginFailures));
+
+ return { files, fileCount: docFiles.length, errorCount, warningCount, warnings, ran };
}
/**
diff --git a/packages/server/src/lint/okf-project-validator.ts b/packages/server/src/lint/okf-project-validator.ts
index 9f3c07b26..c8c5a0514 100644
--- a/packages/server/src/lint/okf-project-validator.ts
+++ b/packages/server/src/lint/okf-project-validator.ts
@@ -15,12 +15,16 @@
import { dirname, relative } from 'node:path';
import {
+ deriveValidationRunSources,
type LinterConfig,
runOkfProjectRules,
- type ValidationDiagnostic,
} from '@inkeep/open-knowledge-core';
import { collectDocFiles, resolveScope } from './audit.ts';
-import type { ProjectValidator, ValidationScope } from './validation-audit.ts';
+import type {
+ ProjectValidator,
+ ValidationDiagnosticFor,
+ ValidationScope,
+} from './validation-audit.ts';
/**
* Zero-width anchor at the top of the file. A project finding is about the file's
@@ -40,14 +44,25 @@ export interface OkfProjectValidatorDeps {
baseConfig: LinterConfig;
}
-export function createOkfProjectValidator(deps: OkfProjectValidatorDeps): ProjectValidator {
+export function createOkfProjectValidator(deps: OkfProjectValidatorDeps): ProjectValidator<'okf'> {
+ // Selection comes off the shared derivation rather than a second copy of the
+ // enablement predicate, and the early return below reads it back — so "is OKF
+ // selected" is stated once. Lint mode, not audit mode: this validator has no
+ // opinion on link validation, and naming a links setting here would read as if
+ // it did.
+ const sourceFamilies = deriveValidationRunSources(deps.baseConfig, { mode: 'lint' }).filter(
+ (source): source is 'okf' => source === 'okf',
+ );
return {
id: 'okf-project',
+ sourceFamilies,
+ // Tree-shape checks report under the same public family as the okf lint
+ // plugin, so a total failure of this validator belongs to `okf` too.
+ failureSourceFamily: 'okf',
async run(scope: ValidationScope) {
- const slice = deps.baseConfig.plugins.okf;
// Plugin off is a clean empty contribution, not a degradation warning — the
// project chose not to run OKF at all.
- if (!deps.baseConfig.enabled || !slice?.enabled) {
+ if (sourceFamilies.length === 0) {
return { files: [], fileCount: 0, warnings: [] };
}
@@ -72,8 +87,8 @@ export function createOkfProjectValidator(deps: OkfProjectValidatorDeps): Projec
onWarning: (warning) => warnings.push(warning),
});
- const byFile = new Map<string, ValidationDiagnostic[]>();
- for (const finding of runOkfProjectRules(docFiles, slice.rules)) {
+ const byFile = new Map<string, ValidationDiagnosticFor<'okf'>[]>();
+ for (const finding of runOkfProjectRules(docFiles, deps.baseConfig.plugins.okf?.rules)) {
if (targetFile !== undefined && finding.file !== targetFile) continue;
const diagnostics = byFile.get(finding.file) ?? [];
diagnostics.push({
diff --git a/packages/server/src/lint/validation-audit.test.ts b/packages/server/src/lint/validation-audit.test.ts
index e6c801c3d..83698ae71 100644
--- a/packages/server/src/lint/validation-audit.test.ts
+++ b/packages/server/src/lint/validation-audit.test.ts
@@ -20,6 +20,7 @@ import { BacklinkIndex } from '../backlink-index.ts';
import { LocalTargetIndex } from '../local-target-index.ts';
import {
createProjectValidators,
+ type ProjectValidator,
runValidationAudit,
toValidationCountsPlane,
type ValidationAuditDeps,
@@ -126,6 +127,7 @@ describe('runValidationAudit', () => {
expect(result.errorCount).toBe(0);
expect(result.warningCount).toBeGreaterThan(1);
expect(result.fileCount).toBe(2);
+ expect(result.ran).toEqual(['markdownlint', 'links']);
// The engine plane must survive the wire schema untouched — the audit
// route serializes through it.
expect(ValidationAuditResponseSchema.parse(result)).toEqual(result);
@@ -158,6 +160,7 @@ describe('runValidationAudit', () => {
const off = await runValidationAudit(createProjectValidators(deps({ linksValidation: 'off' })));
expect(off.files.every((f) => f.diagnostics.every((d) => d.source !== 'links'))).toBe(true);
+ expect(off.ran).toEqual(['markdownlint']);
// A deliberate off is not a degradation — no warning in the plane.
expect(off.warnings).toEqual([]);
});
@@ -175,6 +178,7 @@ describe('runValidationAudit', () => {
expect(result.errorCount).toBe(0);
expect(result.warningCount).toBe(1);
expect(result.fileCount).toBe(2);
+ expect(result.ran).toEqual(['links']);
});
test('a doc with lint and link problems yields one file entry sorted by position', async () => {
@@ -247,14 +251,20 @@ describe('runValidationAudit', () => {
expect(result.files.map((f) => f.file)).toEqual(['dirty.md']);
expect(result.errorCount).toBe(0);
expect(result.warnings).toContain(
- 'links validation unavailable: backlink index is not configured',
+ 'source family "links" validation failed: backlink index is not configured',
);
+ // A degraded validator stays in `ran`. Dropping it would report
+ // `['markdownlint']` beside a warning that link checking broke, which an
+ // agent reads as "links are not validated in this project" and stops
+ // repairing them.
+ expect(result.ran).toEqual(['markdownlint', 'links']);
});
test('an additional registered validator merges into the plane', async () => {
seedDoc('dirty', DOC_WITH_TAB);
const extra = {
id: 'extra',
+ sourceFamilies: [] as const,
run: async () => ({
files: [
{
@@ -301,6 +311,8 @@ describe('runValidationAudit', () => {
seedDoc('dirty', DOC_WITH_TAB);
const boom = {
id: 'boom',
+ sourceFamilies: ['okf'] as const,
+ failureSourceFamily: 'okf' as const,
run: async () => {
throw new Error('kaboom');
},
@@ -312,7 +324,60 @@ describe('runValidationAudit', () => {
expect(result.files.map((f) => f.file)).toEqual(['dirty.md']);
expect(result.files[0]?.diagnostics.some((d) => d.code === 'MD010')).toBe(true);
// The throw becomes a validator-tagged warning, not an unhandled rejection.
- expect(result.warnings).toContain('validator "boom" failed: kaboom');
+ expect(result.warnings).toContain('source family "okf" validation failed: kaboom');
+ expect(result.ran).toEqual(['markdownlint', 'links', 'okf']);
+ });
+
+ test('a lint-walk failure keeps its label whatever the enabled plugin count is', async () => {
+ // The label is a static property of the validator, not a function of how
+ // many lint families the project happens to enable. Deriving it from
+ // `sourceFamilies.length === 1` made the same walk failure report as
+ // `markdownlint` on one project and `lint` on the next.
+ const walkFailure = async (baseConfig: LinterConfig) => {
+ const validators = createProjectValidators(deps({ baseConfig, linksValidation: 'off' }));
+ const lint = validators.find((validator) => validator.id === 'lint');
+ if (!lint) throw new Error('lint validator missing');
+ const failing: ProjectValidator = {
+ ...lint,
+ run: async () => {
+ throw new Error('EACCES');
+ },
+ };
+ const result = await runValidationAudit([failing]);
+ return result.warnings;
+ };
+
+ const twoFamilies = {
+ ...lintOn,
+ plugins: { ...lintOn.plugins, okf: { enabled: true } },
+ } as LinterConfig;
+
+ expect(await walkFailure(lintOn)).toEqual(['validator "lint" failed: EACCES']);
+ expect(await walkFailure(twoFamilies)).toEqual(['validator "lint" failed: EACCES']);
+ });
+
+ test('a project-validator failure reports under its public source family', async () => {
+ // `okf-project` is an internal engine id that appears in no `ran` array and
+ // no diagnostic `source`; the family it contributes to is what a caller can
+ // join against.
+ const okfOn = {
+ ...lintOn,
+ plugins: { ...lintOn.plugins, okf: { enabled: true } },
+ } as LinterConfig;
+ const validators = createProjectValidators(deps({ baseConfig: okfOn, linksValidation: 'off' }));
+ const okfProject = validators.find((validator) => validator.id === 'okf-project');
+ if (!okfProject) throw new Error('okf project validator missing');
+ const failing: ProjectValidator = {
+ ...okfProject,
+ run: async () => {
+ throw new Error('kaboom');
+ },
+ };
+
+ const result = await runValidationAudit([failing]);
+
+ expect(result.warnings).toEqual(['source family "okf" validation failed: kaboom']);
+ expect(result.ran).toEqual(['okf']);
});
test('a dead link from an admitted-but-unsaved source still names a file', async () => {
@@ -401,7 +466,7 @@ describe('runValidationAudit', () => {
expect.objectContaining({ code: 'dead-link', linkTarget: 'ghost' }),
]);
expect(result.warnings).toContain(
- 'local-target validation unavailable: Local-target index is not ready',
+ 'source family "links" validation degraded: local-target projection unavailable: Local-target index is not ready',
);
});
});
@@ -664,6 +729,16 @@ describe('the OKF project validator', () => {
createProjectValidators(deps({ baseConfig: okfOnly, linksValidation: 'off', ...overrides })),
);
+ test('reports okf in ran exactly once — the lint and tree validators share the family', async () => {
+ writeFile('index.md', '# Index\n\n* [a](a.md) - a\n');
+ writeFile('a.md');
+ const result = await auditOkf();
+ // Both the okf lint plugin (through the lint walk) and the okf project
+ // validator select this family, and they fold into one public id — so the
+ // roster must not double-count them.
+ expect(result.ran).toEqual(['okf']);
+ });
+
test('a mis-cased reserved file reaches the plane under the okf source', async () => {
writeFile('Index.md', '# Index\n\n* [a](a.md) - a\n');
const result = await auditOkf();
diff --git a/packages/server/src/lint/validation-audit.ts b/packages/server/src/lint/validation-audit.ts
index e06cb2d1a..af08ae730 100644
--- a/packages/server/src/lint/validation-audit.ts
+++ b/packages/server/src/lint/validation-audit.ts
@@ -13,11 +13,14 @@
import {
countDiagnosticsBySource,
DEFAULT_LINKS_VALIDATION,
+ deriveValidationRunSources,
type LinksValidationSetting,
type LinterConfig,
+ type LintPluginId,
SUPPORTED_DOC_EXTENSIONS,
type ValidationDiagnostic,
type ValidationDocCounts,
+ type ValidationSource,
} from '@inkeep/open-knowledge-core';
import { isProblemsPlaneExcludedDoc } from '../cc1-broadcast.ts';
import type { DerivedDocumentIndexApiPort } from '../derived-document-index.ts';
@@ -50,10 +53,15 @@ interface ValidationDerivedIndexReader {
): LocalTargetsResult | Promise<LocalTargetsResult>;
}
-interface FileValidationResult {
+export type ValidationDiagnosticFor<Source extends ValidationSource> = Omit<
+ ValidationDiagnostic,
+ 'source'
+> & { source: Source };
+
+interface FileValidationResult<Source extends ValidationSource = ValidationSource> {
/** Path relative to `contentDir`. */
file: string;
- diagnostics: ValidationDiagnostic[];
+ diagnostics: ValidationDiagnosticFor<Source>[];
}
export interface ValidationAuditResult {
@@ -62,6 +70,7 @@ export interface ValidationAuditResult {
errorCount: number;
warningCount: number;
warnings: string[];
+ ran: ValidationSource[];
}
export interface ValidationAuditCountsResult {
@@ -78,6 +87,10 @@ export interface ValidationAuditCountsResult {
* (`countDiagnosticsBySource`), only the diagnostic bodies dropped. Callers
* that keep file-tree tints fresh want the tallies and would discard the
* bodies immediately; on a large KB those bodies are tens of MB per request.
+ *
+ * `ran` is dropped for the same reason: the only consumer is the tint refresh,
+ * which reads counts and nothing else. Restoring it is additive if a consumer
+ * ever needs coverage off this plane.
*/
export function toValidationCountsPlane(
result: ValidationAuditResult,
@@ -99,8 +112,8 @@ export interface ValidationScope {
targetPath?: string;
}
-interface ValidatorRunResult {
- files: FileValidationResult[];
+interface ValidatorRunResult<Source extends ValidationSource = ValidationSource> {
+ files: FileValidationResult<Source>[];
/**
* In-scope docs this validator scanned as a full-scan authority; 0 when it
* reads an index instead of walking. The engine keeps the max across
@@ -108,11 +121,67 @@ interface ValidatorRunResult {
*/
fileCount: number;
warnings: string[];
+ /**
+ * The families this run actually selected, when they differ from the
+ * validator's declaration — the lint walk narrows to `[]` on its scope
+ * refusals. Omit it and the declaration stands, so a validator cannot vanish
+ * from reported coverage by forgetting to echo what it already declared.
+ */
+ ran?: readonly Source[];
}
-export interface ProjectValidator {
+export interface ProjectValidator<Source extends ValidationSource = ValidationSource> {
readonly id: string;
- run(scope: ValidationScope): Promise<ValidatorRunResult>;
+ /**
+ * The public source families this validator selects under the current config.
+ * Required, not optional: the engine reports coverage from it, so a new
+ * validator that omitted it would contribute findings whose `source` never
+ * appears in `ran` — with no type error and no runtime signal.
+ */
+ readonly sourceFamilies: readonly Source[];
+ /**
+ * The family that owns a TOTAL failure of this validator, when it has one.
+ * A static fact, not a config-derived one: the lint validator spans several
+ * families and its walk can fail for reasons that belong to none of them
+ * (a readdir error, a config-composition throw), so it declares nothing and
+ * a failure reports under its own id.
+ */
+ readonly failureSourceFamily?: Source;
+ run(scope: ValidationScope): Promise<ValidatorRunResult<Source>>;
+}
+
+type ValidatorFailureAttribution =
+ | { kind: 'validator'; id: string }
+ | { kind: 'source-family'; sourceFamily: ValidationSource };
+
+/**
+ * Attribute a validator failure in the vocabulary exposed by `ran` whenever a
+ * single source family owns it. MCP `capAuditWarnings` recognizes these
+ * prefixes, so producer wording and cap priority must change together.
+ * Callers that still return findings use `formatValidatorDegradationWarning`.
+ */
+export function formatValidatorFailureWarning(
+ attribution: ValidatorFailureAttribution,
+ message: string,
+): string {
+ return attribution.kind === 'validator'
+ ? `validator "${attribution.id}" failed: ${message}`
+ : `source family "${attribution.sourceFamily}" validation failed: ${message}`;
+}
+
+/**
+ * Attribute a partial degradation when a family produced findings but could
+ * not complete every projection. MCP `capAuditWarnings` recognizes the source
+ * family prefix, while `AUDIT_WARNINGS_DESCRIPTION` quotes the `validation
+ * degraded` suffix; producer wording, cap priority, and field guidance must
+ * change together. This stays cap-safe without claiming the family's whole
+ * contribution is missing.
+ */
+export function formatValidatorDegradationWarning(
+ sourceFamily: ValidationSource,
+ message: string,
+): string {
+ return `source family "${sourceFamily}" validation degraded: ${message}`;
}
export interface ValidationAuditDeps {
@@ -158,8 +227,10 @@ export async function runValidationAudit(
// future validator.
const results = await Promise.all(
validators.map(async (validator) => {
+ const declared = validator.sourceFamilies;
try {
- return await validator.run(scope);
+ const result = await validator.run(scope);
+ return { ...result, ran: result.ran ?? declared };
} catch (error) {
// Supersession is not a validator failure — it invalidates the WHOLE
// plane, so degrading it to a warning would publish the surviving
@@ -172,10 +243,19 @@ export async function runValidationAudit(
'[audit] validator threw; degrading to a plane warning',
);
const message = error instanceof Error ? error.message : String(error);
+ const failureWarning = formatValidatorFailureWarning(
+ validator.failureSourceFamily === undefined
+ ? { kind: 'validator', id: validator.id }
+ : { kind: 'source-family', sourceFamily: validator.failureSourceFamily },
+ message,
+ );
return {
files: [],
fileCount: 0,
- warnings: [`validator "${validator.id}" failed: ${message}`],
+ warnings: [failureWarning],
+ // Selected-and-attempted stays in `ran`; the warning beside it says
+ // the attempt could not finish.
+ ran: declared,
} satisfies ValidatorRunResult;
}
}),
@@ -183,9 +263,11 @@ export async function runValidationAudit(
const byFile = new Map<string, ValidationDiagnostic[]>();
const warnings: string[] = [];
+ const ran = new Set<ValidationSource>();
let fileCount = 0;
for (const result of results) {
warnings.push(...result.warnings);
+ for (const source of result.ran ?? []) ran.add(source);
fileCount = Math.max(fileCount, result.fileCount);
for (const entry of result.files) {
const merged = byFile.get(entry.file);
@@ -207,7 +289,7 @@ export async function runValidationAudit(
}
}
- return { files, fileCount, errorCount, warningCount, warnings };
+ return { files, fileCount, errorCount, warningCount, warnings, ran: [...ran] };
}
function byPosition(a: ValidationDiagnostic, b: ValidationDiagnostic): number {
@@ -219,9 +301,11 @@ function byPosition(a: ValidationDiagnostic, b: ValidationDiagnostic): number {
);
}
-function createLintValidator(deps: ValidationAuditDeps): ProjectValidator {
+function createLintValidator(deps: ValidationAuditDeps): ProjectValidator<LintPluginId> {
+ const sourceFamilies = deriveValidationRunSources(deps.baseConfig, { mode: 'lint' });
return {
id: 'lint',
+ sourceFamilies,
async run(scope) {
const audit = await auditProject({
projectDir: deps.projectDir,
@@ -232,15 +316,30 @@ function createLintValidator(deps: ValidationAuditDeps): ProjectValidator {
cache: deps.cache,
auditGeneration: deps.auditGeneration,
});
- return { files: audit.files, fileCount: audit.fileCount, warnings: audit.warnings };
+ return {
+ files: audit.files,
+ fileCount: audit.fileCount,
+ warnings: audit.warnings,
+ // The one validator that must echo: the walk narrows to `[]` on its
+ // scope refusals, and no family was selected for execution there.
+ ran: audit.ran,
+ };
},
};
}
-function createLinksValidator(deps: ValidationAuditDeps): ProjectValidator {
+function createLinksValidator(deps: ValidationAuditDeps): ProjectValidator<'links'> {
const setting = deps.linksValidation ?? DEFAULT_LINKS_VALIDATION;
+ // Extract this validator's contribution from the whole-plane roster rather
+ // than independently recomputing whether the family was selected.
+ const sourceFamilies = deriveValidationRunSources(deps.baseConfig, {
+ mode: 'audit',
+ linksValidation: setting,
+ }).filter((source): source is 'links' => source === 'links');
return {
id: 'links',
+ sourceFamilies,
+ failureSourceFamily: 'links',
async run(scope) {
// 'off' is a clean empty contribution, not a degradation warning — the
// project chose to silence link validation.
@@ -252,7 +351,12 @@ function createLinksValidator(deps: ValidationAuditDeps): ProjectValidator {
return {
files: [],
fileCount: 0,
- warnings: ['links validation unavailable: backlink index is not configured'],
+ warnings: [
+ formatValidatorFailureWarning(
+ { kind: 'source-family', sourceFamily: 'links' },
+ 'backlink index is not configured',
+ ),
+ ],
};
}
const startGeneration = deps.auditGeneration?.();
@@ -272,8 +376,8 @@ function createLinksValidator(deps: ValidationAuditDeps): ProjectValidator {
}
const deadLinks = await deps.derivedDocumentIndex.getDeadLinks(admitted, sourceFilter);
- const byFile = new Map<string, ValidationDiagnostic[]>();
- const push = (file: string, diagnostic: ValidationDiagnostic): void => {
+ const byFile = new Map<string, ValidationDiagnosticFor<'links'>[]>();
+ const push = (file: string, diagnostic: ValidationDiagnosticFor<'links'>): void => {
const diagnostics = byFile.get(file) ?? [];
diagnostics.push(diagnostic);
byFile.set(file, diagnostics);
@@ -287,7 +391,10 @@ function createLinksValidator(deps: ValidationAuditDeps): ProjectValidator {
// resolution method, fallback target) and the same create-page
// `linkTarget`. Reconciling by source form, as this once did, is a
// hand-maintained rule that drifts the moment either plane learns a form.
- const localTargetDiagnostics: Array<{ file: string; diagnostic: ValidationDiagnostic }> = [];
+ const localTargetDiagnostics: Array<{
+ file: string;
+ diagnostic: ValidationDiagnosticFor<'links'>;
+ }> = [];
const documentTargetsFromAssessment = new Set<string>();
const resolvedTargetsFromAssessment = new Set<string>();
const warnings: string[] = [];
@@ -323,7 +430,12 @@ function createLinksValidator(deps: ValidationAuditDeps): ProjectValidator {
'[audit] local-target projection unavailable; preserving graph link findings',
);
const message = error instanceof Error ? error.message : String(error);
- warnings.push(`local-target validation unavailable: ${message}`);
+ warnings.push(
+ formatValidatorDegradationWarning(
+ 'links',
+ `local-target projection unavailable: ${message}`,
+ ),
+ );
}
for (const { target, sources } of deadLinks) {
@@ -404,7 +516,7 @@ function localTargetMessage(assessment: LocalTargetAssessment, shown: string): s
function toLocalTargetDiagnostic(
assessment: LocalTargetAssessment,
severity: 'error' | 'warning',
-): ValidationDiagnostic | null {
+): ValidationDiagnosticFor<'links'> | null {
// `reason` is null exactly when the target is `exact`, so this both drops
// resolved targets and narrows `reason` to a concrete failure below.
if (assessment.reason === null) return null;
@@ -421,7 +533,7 @@ function toLocalTargetDiagnostic(
const shown = assessment.resolvedTarget ?? occurrence.href;
const line = occurrence.line;
const character = occurrence.column;
- const diagnostic: ValidationDiagnostic = {
+ const diagnostic: ValidationDiagnosticFor<'links'> = {
range: { start: { line, character }, end: { line, character } },
severity,
source: 'links',
diff --git a/packages/server/src/mcp/tools/audit.test.ts b/packages/server/src/mcp/tools/audit.test.ts
index 908a92e19..1bbe585d9 100644
--- a/packages/server/src/mcp/tools/audit.test.ts
+++ b/packages/server/src/mcp/tools/audit.test.ts
@@ -12,12 +12,13 @@ import {
} from 'vitest';
import { type Config, ConfigSchema } from '../../config/schema.ts';
import type { AuditDeps } from './audit.ts';
-import { AUDIT_WARNING_CAP, DESCRIPTION, register } from './audit.ts';
+import { DESCRIPTION, register } from './audit.ts';
import { type FetchTestServer, startFetchTestServer } from './fetch-test-server.test-helper.ts';
import type { ServerInstance } from './shared.ts';
import {
AUDIT_FILE_CAP,
AUDIT_FILE_DIAGNOSTIC_CAP,
+ AUDIT_WARNING_CAP,
HOCUSPOCUS_NOT_RUNNING_ERROR,
} from './shared.ts';
@@ -173,7 +174,7 @@ beforeAll(async () => {
fileCount: 3,
errorCount: 0,
warningCount: 0,
- warnings: ['links validation unavailable: backlink index is not configured'],
+ warnings: ['source family "links" validation failed: backlink index is not configured'],
});
}
if (path === 'local-targets') {
@@ -208,9 +209,10 @@ beforeAll(async () => {
fileCount: 20,
errorCount: 0,
warningCount: 0,
- warnings: Array.from(
- { length: AUDIT_WARNING_CAP + 5 },
- (_, i) => `could not read dir-${i}: EACCES`,
+ warnings: Array.from({ length: AUDIT_WARNING_CAP + 5 }, (_, i) =>
+ i === AUDIT_WARNING_CAP + 4
+ ? 'source family "okf" validation failed: EACCES'
+ : `could not read dir-${i}: EACCES`,
),
});
}
@@ -364,12 +366,16 @@ describe('audit — degradation visibility', () => {
const result = await getTool().handler({ path: 'degraded' });
const text = result.content[0]?.text ?? '';
// The warning reaches the text channel (the channel an agent reads)...
- expect(text).toContain('links validation unavailable: backlink index is not configured');
+ expect(text).toContain(
+ 'source family "links" validation failed: backlink index is not configured',
+ );
// ...and the summary flags the run as incomplete, not a bare "No problems.".
expect(text).toContain('could not fully complete');
expect(text).toContain('Audit incomplete');
const s = result.structuredContent as { warnings?: string[] };
- expect(s.warnings).toEqual(['links validation unavailable: backlink index is not configured']);
+ expect(s.warnings).toEqual([
+ 'source family "links" validation failed: backlink index is not configured',
+ ]);
});
test('warnings surface alongside problems, not only in the structured channel', async () => {
@@ -388,6 +394,7 @@ describe('audit — degradation visibility', () => {
const result = await getTool().handler({ path: 'many-warnings' });
const s = result.structuredContent as { warnings?: string[]; omittedWarningCount?: number };
expect(s.warnings).toHaveLength(AUDIT_WARNING_CAP);
+ expect(s.warnings?.[0]).toBe('source family "okf" validation failed: EACCES');
expect(s.omittedWarningCount).toBe(5);
expect(result.content[0]?.text ?? '').toContain('… and 5 more warnings');
});
diff --git a/packages/server/src/mcp/tools/audit.ts b/packages/server/src/mcp/tools/audit.ts
index f4ea4b298..a45bedffe 100644
--- a/packages/server/src/mcp/tools/audit.ts
+++ b/packages/server/src/mcp/tools/audit.ts
@@ -10,12 +10,15 @@
* `edit`/`write`.
*/
+import { validationCoverageLines } from '@inkeep/open-knowledge-core';
import { z } from 'zod';
import type { ConfigOrResolver, ServerInstance, ServerUrlOrResolver } from './shared.ts';
import {
AUDIT_FILE_CAP,
AUDIT_FILE_DIAGNOSTIC_CAP,
+ capAuditWarnings,
countSummary,
+ degradationBlock,
formatDiagnosticLine,
HOCUSPOCUS_NOT_RUNNING_ERROR,
httpGet,
@@ -31,20 +34,17 @@ export const DESCRIPTION = [
'[Requires: Hocuspocus server] Unified validation audit: every content problem — markdown-lint violations AND broken internal links — in one read-only call, grouped by the file to fix.',
'',
'- No args → audit every in-scope `.md`/`.mdx` doc; pass `path` to scope to a folder or a single file.',
+ '- The result reports `ran`, the source families selected for this run. Project audit can report `markdownlint`, `frontmatter`, `okf`, and `links`. A family absent from `ran` was not checked. `okf` covers both document and project-tree OKF checks here.',
+ '- A selected family stays in `ran` if it degrades, with the reason in `warnings`. A partial degradation may still have contributed findings.',
'- To CHECK whether links resolve, use this tool: each broken link is reported under the SOURCE doc that contains it, at the offending line. (The `links` tool is the navigation/graph reader — backlinks, forward links, orphans, hubs — not the validation surface.)',
'',
- "Each diagnostic carries a `source` naming the validator ('markdownlint' rule violations; 'links' broken internal links), a `code` (e.g. MD010, dead-link), `message`, a 0-based LSP `range` (for links: line exact, column approximate), and `severity` ('error' | 'warning' — broken links default to warnings; the project's `validation.links` setting can raise them to errors or hide them). Only files with at least one problem are listed, plus `fileCount`/`errorCount`/`warningCount` totals. Output (text and structured) is capped at 10 files × 10 diagnostics per file, with explicit '… and N more' indicators; the counts always reflect the full scan — re-run with `path` scoped to a folder or file to see what was omitted.",
+ "Each diagnostic carries a `source` naming the validator ('markdownlint' rule violations; 'links' broken internal links), a `code` (e.g. MD010, dead-link), `message`, a 0-based LSP `range` (for links: line exact, column approximate), and `severity` ('error' | 'warning' — broken links default to warnings; the project's `validation.links` setting can raise them to errors or hide them). Only files with at least one problem are listed, plus `fileCount`/`errorCount`/`warningCount` totals. Output (text and structured) is capped at 10 files × 10 diagnostics per file and project-wide at 10 warnings, with explicit '… and N more' indicators and `omittedWarningCount` when warnings are dropped; the counts always reflect the full scan — re-run with `path` scoped to a folder or file to see what was omitted.",
'',
'Read-only: nothing is modified. Auto-fix fixable lint findings with `lint({ document, fix: true })`; broken links need content edits via `edit`/`write`.',
].join('\n');
-/**
- * Degradation warnings (unreadable files/dirs, an unavailable validator) are
- * agent-context-bound like the diagnostics, so the same cap applies — a project
- * with many unreadable files must not flood the agent through this channel. The
- * HTTP endpoint stays the uncapped surface for GUI consumers.
- */
-export const AUDIT_WARNING_CAP = 10;
+export const AUDIT_WARNINGS_DESCRIPTION =
+ 'Anything that made this run less than a full answer: unreadable files/dirs, config problems, a validator that could not run, or a family that ran only partially (`… validation degraded: …`). A source family named here is still listed in `ran` — it was selected, and a degraded family may still have contributed findings.';
/** Fix routing is per-validator: lint has an auto-fix shape, links never do. */
const FIX_ROUTING_HINT =
@@ -74,6 +74,7 @@ interface AuditResponsePayload {
errorCount?: number;
warningCount?: number;
warnings?: string[];
+ ran?: string[];
}
export interface AuditDeps {
@@ -108,10 +109,13 @@ export function register(server: ServerInstance, deps: AuditDeps): void {
fileCount: z.number().optional().describe('Total in-scope documents scanned.'),
errorCount: z.number().describe('Total error-severity findings across validators.'),
warningCount: z.number().describe('Total warning-severity findings across validators.'),
- warnings: z
+ warnings: z.array(z.string()).optional().describe(AUDIT_WARNINGS_DESCRIPTION),
+ ran: z
.array(z.string())
.optional()
- .describe('Non-fatal issues (unreadable files/dirs, unavailable validators).'),
+ .describe(
+ 'Validation source families selected for this run. A family absent from `ran` was not checked.',
+ ),
omittedWarningCount: z
.number()
.optional()
@@ -152,6 +156,7 @@ async function runAudit(path: string | undefined, url: string, cwd: string) {
const fileCount = data.fileCount ?? 0;
const errorCount = data.errorCount ?? 0;
const warningCount = data.warningCount ?? 0;
+ const coverageLines = validationCoverageLines(data.ran);
// Both channels are agent-context-bound, so both get the cap; the HTTP
// endpoint stays the uncapped surface for GUI consumers.
@@ -168,14 +173,14 @@ async function runAudit(path: string | undefined, url: string, cwd: string) {
const omittedFileCount = files.length - shownFiles.length;
const warnings = data.warnings ?? [];
- const shownWarnings = warnings.slice(0, AUDIT_WARNING_CAP);
- const omittedWarningCount = warnings.length - shownWarnings.length;
+ const { shownWarnings, omittedWarningCount } = capAuditWarnings(warnings);
const structured = {
files: shownFiles,
fileCount,
errorCount,
warningCount,
+ ...(data.ran === undefined ? {} : { ran: data.ran }),
...(shownWarnings.length > 0 ? { warnings: shownWarnings } : {}),
...(omittedWarningCount > 0 ? { omittedWarningCount } : {}),
...(omittedFileCount > 0 ? { omittedFileCount } : {}),
@@ -185,16 +190,7 @@ async function runAudit(path: string | undefined, url: string, cwd: string) {
// Surface degradation warnings in the text channel too: an agent reads text,
// so a partial run (a skipped/unreadable file, an unavailable validator) must
// not be byte-indistinguishable from a genuinely clean one.
- const warningBlock =
- warnings.length > 0
- ? [
- `Audit incomplete — ${warnings.length} warning${warnings.length === 1 ? '' : 's'} (findings may be partial):`,
- ...shownWarnings.map((w) => ` ⚠ ${w}`),
- ...(omittedWarningCount > 0
- ? [` … and ${omittedWarningCount} more warning${omittedWarningCount === 1 ? '' : 's'}`]
- : []),
- ]
- : [];
+ const warningBlock = degradationBlock('Audit', shownWarnings, omittedWarningCount);
const scope = path ? ` in ${path}` : '';
if (files.length === 0) {
@@ -202,7 +198,7 @@ async function runAudit(path: string | undefined, url: string, cwd: string) {
warnings.length > 0
? `No problems found across ${fileCount} document${fileCount === 1 ? '' : 's'}${scope}, but the audit could not fully complete.`
: `No problems across ${fileCount} document${fileCount === 1 ? '' : 's'}${scope}.`;
- return textPlusStructured([summary, ...warningBlock].join('\n'), structured);
+ return textPlusStructured([summary, ...coverageLines, ...warningBlock].join('\n'), structured);
}
const header = `${files.length} of ${fileCount} document${fileCount === 1 ? '' : 's'}${scope} with problems — ${countSummary(errorCount, warningCount)}:`;
const fileBlocks = shownFiles.map((file) => {
@@ -219,7 +215,9 @@ async function runAudit(path: string | undefined, url: string, cwd: string) {
? [`… and ${omittedFileCount} more file${omittedFileCount === 1 ? '' : 's'} with problems`]
: [];
return textPlusStructured(
- [header, ...fileBlocks, ...footer, ...warningBlock, FIX_ROUTING_HINT].join('\n'),
+ [header, ...fileBlocks, ...footer, ...warningBlock, FIX_ROUTING_HINT, ...coverageLines].join(
+ '\n',
+ ),
structured,
);
}
diff --git a/packages/server/src/mcp/tools/lint.ts b/packages/server/src/mcp/tools/lint.ts
index c4835fe6a..e1236b5e5 100644
--- a/packages/server/src/mcp/tools/lint.ts
+++ b/packages/server/src/mcp/tools/lint.ts
@@ -18,6 +18,7 @@
* structured payload.
*/
+import { validationCoverageLines } from '@inkeep/open-knowledge-core';
import { z } from 'zod';
import type { AgentIdentity } from '../agent-identity.ts';
import type { ConfigOrResolver, ServerInstance, ServerUrlOrResolver } from './shared.ts';
@@ -25,7 +26,9 @@ import {
AUDIT_FILE_CAP,
AUDIT_FILE_DIAGNOSTIC_CAP,
agentIdentityFields,
+ capAuditWarnings,
countSummary,
+ degradationBlock,
formatDiagnosticLine,
HOCUSPOCUS_NOT_RUNNING_ERROR,
httpGet,
@@ -45,12 +48,16 @@ export const DESCRIPTION = [
'- `document` given → lint just that doc (extension-less path); `path` is ignored.',
'- `document` omitted → audit every in-scope `.md`/`.mdx` doc; pass `path` to scope to a folder or single file.',
'- `fix: true` (requires `document`) → apply auto-fixable rules to that doc IN PLACE and report what remains.',
+ '- The result reports `ran`, the enabled document-lint source families selected for the run: `markdownlint`, `frontmatter`, and/or document-level `okf`. A family absent from `ran` was not checked. Project-tree OKF checks and link validation run only through `audit`.',
'',
- "Each violation carries a `source` (the plugin, e.g. markdownlint), a `code` (the engine's native rule id, e.g. MD010), `message`, a 0-based LSP `range`, and `severity` ('error' | 'warning' | 'info' | 'hint'). The audit lists only files that have at least one violation, plus `fileCount`/`errorCount`/`warningCount`. Audit output (text and structured) is capped at 10 files × 10 diagnostics per file, with explicit '… and N more' indicators; the counts always reflect the full scan — re-run with `path` scoped to a folder or file to see what was omitted. Lint rules are configured in Settings → Plugins (the toggle is committed to `config.yml`; the rules live in the project's native `.markdownlint.*` file).",
+ "Each violation carries a `source` (the plugin, e.g. markdownlint), a `code` (the engine's native rule id, e.g. MD010), `message`, a 0-based LSP `range`, and `severity` ('error' | 'warning' | 'info' | 'hint'). The audit lists only files that have at least one violation, plus `fileCount`/`errorCount`/`warningCount`. Audit output (text and structured) is capped at 10 files × 10 diagnostics per file and project-wide at 10 warnings, with explicit '… and N more' indicators and `omittedWarningCount` when warnings are dropped; the counts always reflect the full scan — re-run with `path` scoped to a folder or file to see what was omitted. Lint rules are configured in Settings → Plugins (the toggle is committed to `config.yml`; the rules live in the project's native `.markdownlint.*` file).",
'',
'To auto-fix, pass `fix: true` with `document`: the fix lands through the collaborative document — attributed to you and reflected in the live preview, same as the editor. It fixes auto-fixable rules (e.g. hard tabs, trailing spaces); violations that resist auto-fix need content edits via the `edit`/`write` tools. (`ok lint --fix` from a shell remains the headless/CI path, but it writes on disk unattributed — prefer `fix: true` when the server is running.)',
].join('\n');
+export const LINT_WARNINGS_DESCRIPTION =
+ 'Anything that made this run less than a full answer: unreadable files/dirs (audit), lint-config problems such as a broken frontmatter schema file, and selected lint plugins that threw. A source family named here is still listed in `ran` — it was selected, it just could not finish.';
+
/**
* Trailing hint for a single-doc lint, quantified by the fixability the wire
* payload now carries (`fixes` present ⇒ auto-fixable): tells the agent exactly
@@ -93,6 +100,7 @@ interface LintDocPayload {
file?: string;
diagnostics?: LintDiagnosticPayload[];
warnings?: string[];
+ ran?: string[];
}
interface LintFixPayload {
@@ -100,6 +108,8 @@ interface LintFixPayload {
fixedCount?: number;
diagnostics?: LintDiagnosticPayload[];
warning?: string;
+ warnings?: string[];
+ ran?: string[];
}
interface LintAuditPayload {
@@ -108,6 +118,7 @@ interface LintAuditPayload {
errorCount?: number;
warningCount?: number;
warnings?: string[];
+ ran?: string[];
}
export interface LintDeps {
@@ -156,12 +167,17 @@ export function register(server: ServerInstance, deps: LintDeps): void {
fileCount: z.number().optional().describe('Audit only: total in-scope documents scanned.'),
errorCount: z.number().describe('Total error-severity violations.'),
warningCount: z.number().describe('Total warning-severity violations.'),
- warnings: z
+ warnings: z.array(z.string()).optional().describe(LINT_WARNINGS_DESCRIPTION),
+ ran: z
.array(z.string())
.optional()
.describe(
- 'Non-fatal issues: unreadable files/dirs (audit) and lint-config problems such as broken frontmatter schema files (audit + single doc).',
+ 'Lint source families selected for this run. A family absent from `ran` was not checked.',
),
+ omittedWarningCount: z
+ .number()
+ .optional()
+ .describe('Audit only: warnings omitted from `warnings` by the output cap.'),
omittedFileCount: z
.number()
.optional()
@@ -230,7 +246,17 @@ async function fixLintDoc(
const fixedCount = data.fixedCount ?? 0;
const errorCount = diagnostics.filter((d) => d.severity === 'error').length;
const warningCount = diagnostics.length - errorCount;
- const structured = { files: [{ file, diagnostics }], fixedCount, errorCount, warningCount, cwd };
+ const responseWarnings = data.warnings ?? [];
+ const coverageLines = validationCoverageLines(data.ran);
+ const structured = {
+ files: [{ file, diagnostics }],
+ fixedCount,
+ errorCount,
+ warningCount,
+ ...(responseWarnings.length > 0 ? { warnings: responseWarnings } : {}),
+ ...(data.ran === undefined ? {} : { ran: data.ran }),
+ cwd,
+ };
const header = data.warning
? // Fix wrote successfully but the post-write re-lint failed — the doc is
@@ -239,15 +265,27 @@ async function fixLintDoc(
`Applied auto-fixes to ${file}, but re-lint failed (${data.warning}); the fix landed — problems below are the pre-fix set, re-run \`lint\` to confirm.`
: fixedCount > 0
? `Fixed ${fixedCount} problem${fixedCount === 1 ? '' : 's'} in ${file}.`
- : `No auto-fixable problems in ${file}.`;
+ : responseWarnings.length > 0
+ ? `No auto-fixable problems in ${file}, but the lint could not fully complete.`
+ : `No auto-fixable problems in ${file}.`;
const lines = diagnostics.map(formatDiagnosticLine);
+ // The re-lint failure rides both channels during its deprecation window, and
+ // the header already states it verbatim — render it once here.
+ const warningBlock = degradationBlock(
+ 'Lint',
+ responseWarnings.filter((warning) => warning !== data.warning),
+ );
const footer =
diagnostics.length > 0 && !data.warning
? [
`${diagnostics.length} problem${diagnostics.length === 1 ? '' : 's'} remain (${countSummary(errorCount, warningCount)}) — need content edits via \`edit\`/\`write\`.`,
]
: [];
- return textPlusStructured([header, ...lines, ...footer].join('\n'), structured);
+ const textLines =
+ diagnostics.length === 0
+ ? [header, ...coverageLines, ...warningBlock]
+ : [header, ...lines, ...warningBlock, ...footer, ...coverageLines];
+ return textPlusStructured(textLines.join('\n'), structured);
}
async function lintSingleDoc(document: string, url: string, cwd: string) {
@@ -261,24 +299,36 @@ async function lintSingleDoc(document: string, url: string, cwd: string) {
const configWarnings = data.warnings ?? [];
const errorCount = diagnostics.filter((d) => d.severity === 'error').length;
const warningCount = diagnostics.length - errorCount;
+ const coverageLines = validationCoverageLines(data.ran);
const file = { file: data.file ?? normalized.docName, diagnostics };
const structured = {
files: [file],
errorCount,
warningCount,
...(configWarnings.length > 0 ? { warnings: configWarnings } : {}),
+ ...(data.ran === undefined ? {} : { ran: data.ran }),
cwd,
};
const header =
- diagnostics.length === 0
- ? `No problems in ${file.file}.`
- : `${file.file}: ${countSummary(errorCount, warningCount)}`;
+ diagnostics.length > 0
+ ? `${file.file}: ${countSummary(errorCount, warningCount)}`
+ : configWarnings.length > 0
+ ? // A clean set from a run that degraded is not a clean document. The
+ // coverage line below reports SELECTION, so it still lists the family
+ // that failed — the qualifier here is the only thing separating
+ // "checked and clean" from "selected but silently non-functional".
+ `No problems found in ${file.file}, but the lint could not fully complete.`
+ : `No problems in ${file.file}.`;
const lines = diagnostics.map(formatDiagnosticLine);
- const warningLines = configWarnings.map((w) => ` ⚠ ${w}`);
+ const warningBlock = degradationBlock('Lint', configWarnings);
const fixableCount = diagnostics.filter((d) => (d.fixes?.length ?? 0) > 0).length;
const footer = diagnostics.length > 0 ? [singleDocFixHint(fixableCount, diagnostics.length)] : [];
- return textPlusStructured([header, ...lines, ...warningLines, ...footer].join('\n'), structured);
+ const textLines =
+ diagnostics.length === 0
+ ? [header, ...coverageLines, ...warningBlock]
+ : [header, ...lines, ...warningBlock, ...footer, ...coverageLines];
+ return textPlusStructured(textLines.join('\n'), structured);
}
async function lintAudit(path: string | undefined, url: string, cwd: string) {
@@ -291,6 +341,7 @@ async function lintAudit(path: string | undefined, url: string, cwd: string) {
const fileCount = data.fileCount ?? 0;
const errorCount = data.errorCount ?? 0;
const warningCount = data.warningCount ?? 0;
+ const coverageLines = validationCoverageLines(data.ran);
// Both channels are agent-context-bound, so both get the cap; the HTTP
// endpoint stays the uncapped surface for GUI consumers.
@@ -306,22 +357,29 @@ async function lintAudit(path: string | undefined, url: string, cwd: string) {
});
const omittedFileCount = files.length - shownFiles.length;
+ const warnings = data.warnings ?? [];
+ const { shownWarnings, omittedWarningCount } = capAuditWarnings(warnings);
+
const structured = {
files: shownFiles,
fileCount,
errorCount,
warningCount,
- ...(data.warnings && data.warnings.length > 0 ? { warnings: data.warnings } : {}),
+ ...(shownWarnings.length > 0 ? { warnings: shownWarnings } : {}),
+ ...(omittedWarningCount > 0 ? { omittedWarningCount } : {}),
+ ...(data.ran === undefined ? {} : { ran: data.ran }),
...(omittedFileCount > 0 ? { omittedFileCount } : {}),
cwd,
};
const scope = path ? ` in ${path}` : '';
+ const warningBlock = degradationBlock('Lint', shownWarnings, omittedWarningCount);
if (files.length === 0) {
- return textPlusStructured(
- `No problems across ${fileCount} document${fileCount === 1 ? '' : 's'}${scope}.`,
- structured,
- );
+ const summary =
+ warnings.length > 0
+ ? `No problems found across ${fileCount} document${fileCount === 1 ? '' : 's'}${scope}, but the lint could not fully complete.`
+ : `No problems across ${fileCount} document${fileCount === 1 ? '' : 's'}${scope}.`;
+ return textPlusStructured([summary, ...coverageLines, ...warningBlock].join('\n'), structured);
}
const header = `${files.length} of ${fileCount} document${fileCount === 1 ? '' : 's'}${scope} with problems — ${countSummary(errorCount, warningCount)}:`;
const fileBlocks = shownFiles.map((file) => {
@@ -338,7 +396,9 @@ async function lintAudit(path: string | undefined, url: string, cwd: string) {
? [`… and ${omittedFileCount} more file${omittedFileCount === 1 ? '' : 's'} with problems`]
: [];
return textPlusStructured(
- [header, ...fileBlocks, ...footer, AUDIT_FIX_HINT].join('\n'),
+ [header, ...fileBlocks, ...footer, ...warningBlock, AUDIT_FIX_HINT, ...coverageLines].join(
+ '\n',
+ ),
structured,
);
}
diff --git a/packages/server/src/mcp/tools/output-schema-strictness.test.ts b/packages/server/src/mcp/tools/output-schema-strictness.test.ts
index e330ecce4..ee3d3700f 100644
--- a/packages/server/src/mcp/tools/output-schema-strictness.test.ts
+++ b/packages/server/src/mcp/tools/output-schema-strictness.test.ts
@@ -35,13 +35,15 @@ import { join } from 'node:path';
import { normalizeObjectSchema } from '@modelcontextprotocol/sdk/server/zod-compat.js';
import { toJsonSchemaCompat } from '@modelcontextprotocol/sdk/server/zod-json-schema-compat.js';
import { AjvJsonSchemaValidator } from '@modelcontextprotocol/sdk/validation/ajv';
-import { describe, expect, test } from 'vitest';
+import { afterEach, describe, expect, test, vi } from 'vitest';
import { type Config, ConfigSchema } from '../../config/schema.ts';
+import { register as registerAudit } from './audit.ts';
import { register as registerConfig } from './config.ts';
import { registerAllTools } from './index.ts';
+import { register as registerLint } from './lint.ts';
import { register as registerPalette } from './palette.ts';
import { register as registerSearch } from './search.ts';
-import type { ServerInstance } from './shared.ts';
+import { AUDIT_WARNING_CAP, type ServerInstance } from './shared.ts';
const BASE_CONFIG: Config = ConfigSchema.parse({});
@@ -232,6 +234,7 @@ describe('MCP outputSchema strictness — auto-discovered registerTool sweep (no
'write',
'links',
'audit',
+ 'lint',
]);
test('every registerTool registration declares `text` in its outputSchema', () => {
@@ -282,6 +285,351 @@ describe('MCP outputSchema strictness — auto-discovered registerTool sweep (no
}
});
+describe('audit and lint emitted coverage validates against the client schema', () => {
+ afterEach(() => {
+ vi.unstubAllGlobals();
+ });
+
+ const cases = [
+ {
+ name: 'audit',
+ register: registerAudit,
+ args: {},
+ payload: {
+ files: [],
+ fileCount: 1,
+ errorCount: 0,
+ warningCount: 0,
+ warnings: [],
+ ran: ['markdownlint', 'links'],
+ },
+ },
+ {
+ name: 'lint project',
+ register: registerLint,
+ args: {},
+ payload: {
+ files: [],
+ fileCount: 1,
+ errorCount: 0,
+ warningCount: 0,
+ warnings: [],
+ ran: ['markdownlint'],
+ },
+ },
+ {
+ name: 'lint document',
+ register: registerLint,
+ args: { document: 'notes' },
+ payload: { file: 'notes.md', diagnostics: [], warnings: [], ran: ['frontmatter'] },
+ },
+ {
+ name: 'lint fix',
+ register: registerLint,
+ args: { document: 'notes', fix: true },
+ payload: {
+ file: 'notes.md',
+ fixedCount: 0,
+ diagnostics: [],
+ errorCount: 0,
+ warningCount: 0,
+ warnings: [],
+ ran: [],
+ },
+ },
+ ] as const;
+
+ for (const testCase of cases) {
+ test(`${testCase.name}: actual structuredContent admits ran`, async () => {
+ const cwd = newProject();
+ const captured = captureRegistration(testCase.register, {
+ config: BASE_CONFIG,
+ resolveCwd: async () => cwd,
+ serverUrl: 'http://127.0.0.1:31337',
+ });
+ vi.stubGlobal(
+ 'fetch',
+ vi.fn(async () => Response.json(testCase.payload)),
+ );
+
+ const result = await captured.handler(testCase.args);
+ expect(result.structuredContent?.ran).toEqual(testCase.payload.ran);
+ const validator = new AjvJsonSchemaValidator();
+ const validate = validator.getValidator(
+ compileOutputSchemaForClient(captured.cfg.outputSchema),
+ );
+ const validation = validate(result.structuredContent) as {
+ valid: boolean;
+ errorMessage?: string;
+ };
+ expect(validation.valid, validation.errorMessage).toBe(true);
+ });
+ }
+
+ test('audit text distinguishes clean, findings, empty selection, and degradation', async () => {
+ const captured = captureRegistration(registerAudit, {
+ config: BASE_CONFIG,
+ resolveCwd: async () => newProject(),
+ serverUrl: 'http://127.0.0.1:31337',
+ });
+ const call = async (payload: Record<string, unknown>): Promise<string> => {
+ vi.stubGlobal(
+ 'fetch',
+ vi.fn(async () => Response.json(payload)),
+ );
+ const result = await captured.handler({});
+ return result.content[0]?.text ?? '';
+ };
+ const base = { fileCount: 1, errorCount: 0, warningCount: 0, warnings: [] };
+
+ expect(await call({ ...base, files: [], ran: ['links'] })).toContain(
+ 'No problems across 1 document.\nChecks run: links.',
+ );
+
+ const findings = await call({
+ ...base,
+ files: [{ file: 'notes.md', diagnostics: [warningDiagnostic('links')] }],
+ warningCount: 1,
+ ran: ['links'],
+ });
+ expect(findings.endsWith('Checks run: links.')).toBe(true);
+
+ expect(await call({ ...base, files: [], ran: [] })).toContain(
+ 'No problems across 1 document.\nNo checks ran.',
+ );
+
+ const degraded = await call({
+ ...base,
+ files: [],
+ ran: ['links'],
+ warnings: ['source family "links" validation failed: unavailable'],
+ });
+ expect(degraded).toContain('No problems found across 1 document');
+ expect(degraded).toContain('Checks run: links.');
+ expect(degraded).toContain('source family "links" validation failed: unavailable');
+ });
+
+ test('lint text distinguishes clean, findings, empty selection, fix, and degradation', async () => {
+ const captured = captureRegistration(registerLint, {
+ config: BASE_CONFIG,
+ resolveCwd: async () => newProject(),
+ serverUrl: 'http://127.0.0.1:31337',
+ });
+ const call = async (
+ args: Record<string, unknown>,
+ payload: Record<string, unknown>,
+ ): Promise<string> => {
+ vi.stubGlobal(
+ 'fetch',
+ vi.fn(async () => Response.json(payload)),
+ );
+ const result = await captured.handler(args);
+ return result.content[0]?.text ?? '';
+ };
+
+ expect(
+ await call(
+ { document: 'notes' },
+ { file: 'notes.md', diagnostics: [], warnings: [], ran: ['frontmatter'] },
+ ),
+ ).toContain('No problems in notes.md.\nChecks run: frontmatter.');
+
+ const findings = await call(
+ { document: 'notes' },
+ {
+ file: 'notes.md',
+ diagnostics: [warningDiagnostic('frontmatter')],
+ warnings: [],
+ ran: ['frontmatter'],
+ },
+ );
+ expect(findings.endsWith('Checks run: frontmatter.')).toBe(true);
+
+ expect(
+ await call(
+ {},
+ {
+ files: [],
+ fileCount: 1,
+ errorCount: 0,
+ warningCount: 0,
+ warnings: [],
+ ran: [],
+ },
+ ),
+ ).toContain('No problems across 1 document.\nNo checks ran.');
+
+ expect(
+ await call(
+ { document: 'notes', fix: true },
+ {
+ file: 'notes.md',
+ fixedCount: 0,
+ diagnostics: [],
+ errorCount: 0,
+ warningCount: 0,
+ ran: [],
+ },
+ ),
+ ).toContain('No auto-fixable problems in notes.md.\nNo checks ran.');
+
+ // A selected family that degraded is still SELECTED, so the coverage line
+ // below still names it. Without the qualifier on the summary the two most
+ // salient lines both read as reassurance and the only counter-signal is a
+ // bare ⚠ in a channel that also carries benign config nits.
+ const degraded = await call(
+ { document: 'notes' },
+ {
+ file: 'notes.md',
+ diagnostics: [],
+ ran: ['frontmatter'],
+ warnings: ['source "frontmatter" lint failed on "notes.md": unavailable'],
+ },
+ );
+ expect(degraded).toContain(
+ 'No problems found in notes.md, but the lint could not fully complete.\nChecks run: frontmatter.',
+ );
+ expect(degraded).toContain('Lint incomplete — 1 warning (findings may be partial):');
+ expect(degraded).toContain('source "frontmatter" lint failed');
+
+ const degradedAudit = await call(
+ {},
+ {
+ files: [],
+ fileCount: 3,
+ errorCount: 0,
+ warningCount: 0,
+ ran: ['markdownlint'],
+ warnings: ['source "markdownlint" lint failed on 3 documents (first: "a.md"): boom'],
+ },
+ );
+ expect(degradedAudit).toContain(
+ 'No problems found across 3 documents, but the lint could not fully complete.\nChecks run: markdownlint.',
+ );
+
+ const degradedFix = await call(
+ { document: 'notes', fix: true },
+ {
+ file: 'notes.md',
+ fixedCount: 0,
+ diagnostics: [],
+ errorCount: 0,
+ warningCount: 0,
+ ran: ['markdownlint'],
+ warnings: ['source "markdownlint" fix failed on "notes.md": boom'],
+ },
+ );
+ expect(degradedFix).toContain(
+ 'No auto-fixable problems in notes.md, but the lint could not fully complete.',
+ );
+ });
+
+ test('the deprecated fix `warning` rides `warnings` too, and renders once', async () => {
+ const captured = captureRegistration(registerLint, {
+ config: BASE_CONFIG,
+ resolveCwd: async () => newProject(),
+ serverUrl: 'http://127.0.0.1:31337',
+ });
+ const reLint = 'Re-lint after fix failed: boom';
+ vi.stubGlobal(
+ 'fetch',
+ vi.fn(async () =>
+ Response.json({
+ file: 'notes.md',
+ fixedCount: 1,
+ diagnostics: [],
+ errorCount: 0,
+ warningCount: 0,
+ ran: ['markdownlint'],
+ // The shape the server emits during the deprecation window: the
+ // singular field kept for its existing readers, the same message in
+ // the plural channel so a consumer that reads only `warnings` cannot
+ // mistake a stale-diagnostics response for a clean one. `warning` is
+ // declared in no outputSchema, so `warnings` is the ONLY structured
+ // carrier a schema-driven client can see it through.
+ warning: reLint,
+ warnings: [reLint],
+ }),
+ ),
+ );
+
+ const result = await captured.handler({ document: 'notes', fix: true });
+
+ const structured = result.structuredContent as { warnings?: string[] };
+ expect(structured.warnings).toEqual([reLint]);
+ const text = result.content[0]?.text ?? '';
+ expect(text.split(reLint)).toHaveLength(2);
+
+ const validator = new AjvJsonSchemaValidator();
+ const validate = validator.getValidator(
+ compileOutputSchemaForClient(captured.cfg.outputSchema),
+ );
+ const validation = validate(result.structuredContent) as {
+ valid: boolean;
+ errorMessage?: string;
+ };
+ expect(validation.valid, validation.errorMessage).toBe(true);
+ });
+
+ test('lint audit caps the warning channel the way audit does', async () => {
+ const captured = captureRegistration(registerLint, {
+ config: BASE_CONFIG,
+ resolveCwd: async () => newProject(),
+ serverUrl: 'http://127.0.0.1:31337',
+ });
+ const warnings = Array.from({ length: AUDIT_WARNING_CAP + 5 }, (_, i) =>
+ i === AUDIT_WARNING_CAP + 4 ? 'validator "lint" failed: EACCES' : `warning ${i}`,
+ );
+ vi.stubGlobal(
+ 'fetch',
+ vi.fn(async () =>
+ Response.json({
+ files: [],
+ fileCount: 1,
+ errorCount: 0,
+ warningCount: 0,
+ ran: ['markdownlint'],
+ warnings,
+ }),
+ ),
+ );
+
+ const result = await captured.handler({});
+
+ // Both channels are agent-context-bound, so both get the cap — the same
+ // invariant the sibling `audit` tool states and enforces.
+ const structured = result.structuredContent as {
+ warnings?: string[];
+ omittedWarningCount?: number;
+ };
+ expect(structured.warnings).toHaveLength(AUDIT_WARNING_CAP);
+ expect(structured.warnings?.[0]).toBe('validator "lint" failed: EACCES');
+ expect(structured.omittedWarningCount).toBe(5);
+ const text = result.content[0]?.text ?? '';
+ expect(text).toContain('… and 5 more warnings');
+
+ const validator = new AjvJsonSchemaValidator();
+ const validate = validator.getValidator(
+ compileOutputSchemaForClient(captured.cfg.outputSchema),
+ );
+ const validation = validate(result.structuredContent) as {
+ valid: boolean;
+ errorMessage?: string;
+ };
+ expect(validation.valid, validation.errorMessage).toBe(true);
+ });
+
+ function warningDiagnostic(source: string): Record<string, unknown> {
+ return {
+ range: { start: { line: 0, character: 0 }, end: { line: 0, character: 1 } },
+ severity: 'warning',
+ source,
+ code: 'test',
+ message: 'problem',
+ };
+ }
+});
+
describe('move outputSchema admits the cross-level skill-move payloads (CORR-1)', () => {
// The `{text}`-only sweep above can't catch a payload that carries a field the
// schema omits but the sweep never sends. `moveSkillCrossScope` emits
diff --git a/packages/server/src/mcp/tools/shared.test.ts b/packages/server/src/mcp/tools/shared.test.ts
index 55a662b95..8754c9cc1 100644
--- a/packages/server/src/mcp/tools/shared.test.ts
+++ b/packages/server/src/mcp/tools/shared.test.ts
@@ -12,13 +12,20 @@
* tests in CI would let regressions like the 2xx-with-`type`+`title`
* misclassification slip through.
*/
+import { summarizeLintPluginFailures } from '@inkeep/open-knowledge-core';
import { afterAll, beforeAll, describe, expect, test } from 'vitest';
import { z } from 'zod';
import { type Config, ConfigSchema } from '../../config/schema.ts';
import type { LocalApiDispatch } from '../../http/local-api-dispatch.ts';
+import {
+ formatValidatorDegradationWarning,
+ formatValidatorFailureWarning,
+} from '../../lint/validation-audit.ts';
import { type FetchTestServer, startFetchTestServer } from './fetch-test-server.test-helper.ts';
import {
+ AUDIT_WARNING_CAP,
+ capAuditWarnings,
HOCUSPOCUS_NOT_RUNNING_ERROR,
httpGet,
httpPost,
@@ -33,6 +40,36 @@ import {
textResult,
} from './shared.ts';
+describe('capAuditWarnings', () => {
+ test('retains late validator-attributed failures ahead of generic warnings', () => {
+ const generic = Array.from({ length: AUDIT_WARNING_CAP }, (_, i) => `could not read ${i}`);
+ const failures = [
+ ...summarizeLintPluginFailures([{ source: 'markdownlint', phase: 'lint', message: 'boom' }]),
+ ...summarizeLintPluginFailures([
+ { source: 'frontmatter', phase: 'lint', docName: 'a.md', message: 'schema exploded' },
+ { source: 'frontmatter', phase: 'lint', docName: 'b.md', message: 'schema exploded' },
+ ]),
+ formatValidatorFailureWarning(
+ { kind: 'source-family', sourceFamily: 'links' },
+ 'backlink index is not configured',
+ ),
+ formatValidatorDegradationWarning(
+ 'links',
+ 'local-target projection unavailable: stale index',
+ ),
+ formatValidatorFailureWarning({ kind: 'validator', id: 'lint' }, 'EACCES'),
+ ];
+
+ const capped = capAuditWarnings([...generic, ...failures]);
+
+ expect(capped.shownWarnings).toEqual([
+ ...failures,
+ ...generic.slice(0, AUDIT_WARNING_CAP - failures.length),
+ ]);
+ expect(capped.omittedWarningCount).toBe(failures.length);
+ });
+});
+
const TEST_CONFIG: Config = ConfigSchema.parse({ content: { dir: 'content' } });
describe('textResult', () => {
diff --git a/packages/server/src/mcp/tools/shared.ts b/packages/server/src/mcp/tools/shared.ts
index b1ffc19fa..1fd80ef4c 100644
--- a/packages/server/src/mcp/tools/shared.ts
+++ b/packages/server/src/mcp/tools/shared.ts
@@ -923,6 +923,62 @@ export function parseRenameCollidingPairs(value: unknown): RenameCollisionPair[]
export const AUDIT_FILE_CAP = 10;
export const AUDIT_FILE_DIAGNOSTIC_CAP = 10;
+/**
+ * Degradation warnings (unreadable files/dirs, an unavailable validator, a lint
+ * plugin that threw) are agent-context-bound like the diagnostics, so the same
+ * ceiling applies to both `lint`'s and `audit`'s project shapes — a project with
+ * many unreadable files must not flood the agent through this channel. The HTTP
+ * endpoints stay the uncapped surface for GUI consumers.
+ */
+export const AUDIT_WARNING_CAP = 10;
+
+const ATTRIBUTED_VALIDATION_FAILURE = /^(?:source(?: family)?|validator) "/;
+
+/**
+ * Keep validator-attributed failures visible when the agent-context warning
+ * channel is capped. Those lines are the join between a selected `ran` family
+ * and its degraded execution; generic file-system warnings remain in arrival
+ * order behind them, and the uncapped HTTP response is left untouched. Config
+ * problems deliberately stay generic: they are partial and name the offending
+ * file rather than claiming an entire source family degraded.
+ */
+export function capAuditWarnings(warnings: readonly string[]): {
+ shownWarnings: string[];
+ omittedWarningCount: number;
+} {
+ const attributed: string[] = [];
+ const generic: string[] = [];
+ for (const warning of warnings) {
+ (ATTRIBUTED_VALIDATION_FAILURE.test(warning) ? attributed : generic).push(warning);
+ }
+ const shownWarnings = [...attributed, ...generic].slice(0, AUDIT_WARNING_CAP);
+ return {
+ shownWarnings,
+ omittedWarningCount: warnings.length - shownWarnings.length,
+ };
+}
+
+/**
+ * The degradation warnings rendered for the text channel. A bare `⚠` line
+ * reads as a benign config nit beside the findings; the header states outright
+ * that the findings above may be partial. One renderer for the `lint` and
+ * `audit` tools so the framing can never silently diverge (same contract as
+ * the shared caps above — the caps bound this channel, this shapes it).
+ */
+export function degradationBlock(
+ kind: 'Lint' | 'Audit',
+ shown: readonly string[],
+ omitted = 0,
+): string[] {
+ const total = shown.length + omitted;
+ if (total === 0) return [];
+ return [
+ `${kind} incomplete — ${total} warning${total === 1 ? '' : 's'} (findings may be partial):`,
+ ...shown.map((warning) => ` ⚠ ${warning}`),
+ ...(omitted > 0 ? [` … and ${omitted} more warning${omitted === 1 ? '' : 's'}`] : []),
+ ];
+}
+
/**
* Structural diagnostic shape the `lint` and `audit` text formatters read —
* the intersection of both tools' wire payload types (all optional, since it
diff --git a/packages/server/src/validation-coverage-contract.test.ts b/packages/server/src/validation-coverage-contract.test.ts
new file mode 100644
index 000000000..32c9e41df
--- /dev/null
+++ b/packages/server/src/validation-coverage-contract.test.ts
@@ -0,0 +1,112 @@
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { LINT_PLUGIN_IDS, VALIDATION_SOURCES } from '@inkeep/open-knowledge-core';
+import { describe, expect, test } from 'vitest';
+import { formatValidatorDegradationWarning } from './lint/validation-audit.ts';
+import { DESCRIPTION as AUDIT_DESCRIPTION, AUDIT_WARNINGS_DESCRIPTION } from './mcp/tools/audit.ts';
+import { DESCRIPTION as LINT_DESCRIPTION, LINT_WARNINGS_DESCRIPTION } from './mcp/tools/lint.ts';
+
+const PROJECT_SKILL = readFileSync(
+ join(import.meta.dir, '../assets/skills/project/SKILL.md'),
+ 'utf8',
+);
+const CONTENT_RULES_OVERVIEW = readFileSync(
+ join(import.meta.dir, '../../../docs/content/advanced/content-rules/overview.mdx'),
+ 'utf8',
+);
+const MCP_REFERENCE = readFileSync(
+ join(import.meta.dir, '../../../docs/content/reference/mcp.mdx'),
+ 'utf8',
+);
+
+/** The one row of `mcp.mdx`'s tool table that documents `tool`. */
+function mcpReferenceRow(tool: string): string {
+ const row = MCP_REFERENCE.split('\n').find((line) => line.startsWith(`| \`${tool}\` |`));
+ if (row === undefined) throw new Error(`no mcp.mdx table row for \`${tool}\``);
+ return row;
+}
+
+describe('validation coverage source contract', () => {
+ test('tool descriptions name every source family they can select', () => {
+ for (const source of LINT_PLUGIN_IDS) expect(LINT_DESCRIPTION).toContain(`\`${source}\``);
+ for (const source of VALIDATION_SOURCES) expect(AUDIT_DESCRIPTION).toContain(`\`${source}\``);
+ });
+
+ test('every surface states the absence inference, not just the enumeration', () => {
+ // The measured failure is the PARTIAL case — a roster that lists two
+ // families while the one the task named is silently missing. Contraposing
+ // that from an enumeration is a weaker signal than saying it outright, and
+ // the DESCRIPTIONs are the surface an agent reads every turn.
+ const clause = 'absent from `ran` was not checked';
+ expect(LINT_DESCRIPTION).toContain(clause);
+ expect(AUDIT_DESCRIPTION).toContain(clause);
+ expect(PROJECT_SKILL).toContain(clause);
+ expect(CONTENT_RULES_OVERVIEW).toContain(clause);
+ expect(mcpReferenceRow('lint')).toContain(clause);
+ expect(mcpReferenceRow('audit')).toContain(clause);
+ });
+
+ test('agent guidance and public docs explain ran in lockstep', () => {
+ // Phrase pins, not bare source-id substrings: `markdownlint`, `frontmatter`,
+ // and `links` each occur a dozen-plus times on these pages for unrelated
+ // reasons, so a loop over the vocabulary stays green even if the whole `ran`
+ // sentence is deleted from the row it documents.
+ expect(PROJECT_SKILL).toContain('Read `ran`');
+ expect(PROJECT_SKILL).toContain('`[]` means no checks were selected');
+
+ expect(mcpReferenceRow('lint')).toContain('Successful responses include `ran`');
+ expect(mcpReferenceRow('audit')).toContain('Successful responses include `ran`');
+ expect(mcpReferenceRow('audit')).toContain('remains in `ran` if it degrades');
+
+ expect(CONTENT_RULES_OVERVIEW).toContain('"ran": [');
+ expect(CONTENT_RULES_OVERVIEW).toContain('The top-level `ran` array');
+ expect(CONTENT_RULES_OVERVIEW).toContain("The audit's top-level `ran` array");
+ expect(CONTENT_RULES_OVERVIEW).toContain('selected validator degrades');
+ // The zero state is the DEFAULT first-run output — every linter ships off —
+ // so the string a user searches for has to appear on the page.
+ expect(CONTENT_RULES_OVERVIEW).toContain('No checks ran.');
+ });
+
+ test('the audit warning field explains family degradation and its ran join', () => {
+ const produced = formatValidatorDegradationWarning('links', 'test');
+ expect(produced).toContain('validation degraded:');
+ expect(AUDIT_WARNINGS_DESCRIPTION).toContain('validation degraded:');
+ expect(AUDIT_WARNINGS_DESCRIPTION).toContain('still listed in `ran`');
+ expect(LINT_WARNINGS_DESCRIPTION).toContain('still listed in `ran`');
+ for (const surface of [
+ AUDIT_WARNINGS_DESCRIPTION,
+ AUDIT_DESCRIPTION,
+ mcpReferenceRow('audit'),
+ CONTENT_RULES_OVERVIEW,
+ ]) {
+ expect(surface).toContain('may still have contributed findings');
+ }
+ });
+
+ test('every surface distinguishes document lint from project audit scope', () => {
+ expect(LINT_DESCRIPTION).toContain(
+ 'Project-tree OKF checks and link validation run only through',
+ );
+ expect(AUDIT_DESCRIPTION).toContain('document and project-tree OKF checks here');
+ expect(mcpReferenceRow('audit')).toContain(
+ 'Document and project-tree OKF checks share the `okf` family',
+ );
+ expect(CONTENT_RULES_OVERVIEW).toContain(
+ "OKF's document and project-tree checks share the public `okf` family",
+ );
+ });
+
+ test('every capped agent surface documents the warning ceiling and omission signal', () => {
+ for (const surface of [
+ LINT_DESCRIPTION,
+ AUDIT_DESCRIPTION,
+ mcpReferenceRow('lint'),
+ mcpReferenceRow('audit'),
+ ]) {
+ expect(surface).toContain('project-wide at 10 warnings');
+ expect(surface).toContain('omittedWarningCount');
+ }
+ expect(CONTENT_RULES_OVERVIEW).toContain('project-wide at 10 warnings');
+ expect(CONTENT_RULES_OVERVIEW).toContain('omittedWarningCount');
+ });
+});