Skip to content

armgit / mso

Public Imported from GitHub
0

Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.

fix(mcp): lossless scan continuation and exact-id project resolution

rahmanef63dcae9b8a3bf7
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index adc94451..d4e5debe 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -14,6 +14,7 @@ this is the *what*, and it is what Settings → About shows as “What's new”.
**Fixed**
+- `mcp` lossless scan continuation and exact-id project resolution
- `mcp` one project validator, dirent budgets, resumable caps
- `mcp` contain, bound and uniquely identify project discovery
- `mcp` make ChatGPT file imports region agnostic
diff --git a/docs/CONNECTORS-GATEWAY-INTEGRATION.md b/docs/CONNECTORS-GATEWAY-INTEGRATION.md
index 9a4817c9..04e3b496 100644
--- a/docs/CONNECTORS-GATEWAY-INTEGRATION.md
+++ b/docs/CONNECTORS-GATEWAY-INTEGRATION.md
@@ -4,7 +4,7 @@
## Cross-repo contract
-MSO currently exposes **26 tool names** (server `1.5.2`, toolset `2026.08.20.5`). The gateway manifest still maps **15** of them through literal `x-upstream` strings:
+MSO currently exposes **26 tool names** (server `1.5.3`, toolset `2026.08.20.6`). The gateway manifest still maps **15** of them through literal `x-upstream` strings:
```json
{ "id": "mso.fs.delete", "x-upstream": "fs_delete" }
diff --git a/docs/MCP.md b/docs/MCP.md
index cc69cb85..cf973dfd 100644
--- a/docs/MCP.md
+++ b/docs/MCP.md
@@ -104,8 +104,8 @@ The catalog has a stable server version plus a schema-derived toolset signature.
Settings → MCP shows the current version/hash/count and stores a browser-local acknowledgement when the operator marks ChatGPT refreshed. A later signature change becomes an explicit stale-snapshot warning. This does not mutate ChatGPT remotely; it makes the required refresh visible instead of relying on memory.
-Current catalog: **26 tools** (14 read, 10 write, 2 exec), server `1.5.2` / toolset
-`2026.08.20.5`. `projects_list`, `skills_list` and `skills_read` are the new public
+Current catalog: **26 tools** (14 read, 10 write, 2 exec), server `1.5.3` / toolset
+`2026.08.20.6`. `projects_list`, `skills_list` and `skills_read` are the new public
names in this release; `image_generation_status` and `image_generate` were removed
(see the migration note below).
@@ -138,11 +138,19 @@ never reaches the `package.json` or `.git` readers at all. A path hint is valida
component by component from the container down, because canonicalizing first and checking
afterwards is exactly what let a symlinked intermediate through.
-A caller-supplied `rootHint` gets the same treatment: a symlinked or hidden root is
-refused rather than canonicalized into something the caller never named. It is authorized
-against *every* configured read root rather than the scan-capped subset — naming a root
-never widens the jail, but the jail must not shrink because twelve other roots were
-configured ahead of it.
+A caller-supplied `rootHint` gets the same treatment: a symlinked root is refused rather
+than canonicalized into something the caller never named, and **any** dot-prefixed
+component below the authorized root is refused — measured relative to that root, so a
+checkout legitimately living under `~/.claude/worktrees` still works while nothing may
+hide beneath it. It is authorized against *every* configured read root rather than the
+scan-capped subset: naming a root never widens the jail, but the jail must not shrink
+because twelve other roots were configured ahead of it.
+
+**An exact `<rootId>/<project-name>` is resolved first**, before alias, package or fuzzy
+matching. The `rootId` maps to exactly one container across every configured root, the
+project name is then checked by the shared validator, and an unknown `rootId` is refused
+outright. Falling through to fuzzy matching is what made two same-named projects
+unaddressable: asking for the second one's id returned the first.
### Bounds, and telling the truth about them
@@ -168,13 +176,28 @@ so an oversized `package.json`, `SKILL.md` or `packed-refs` costs one stat, not
and a count of entries rejected by the containment/ownership checks. **Do not conclude a
project or skill is absent from a truncated scan** — the tool descriptions say so too.
-**Every cap is resumable.** A truncated report carries `scan.continuation` with the
-containers still pending, a per-container cursor, and an opaque `cursor` string to pass
-back as the `cursor` argument to `projects_list` / `skills_list`. Cursors are positional
-in readdir order (`cursorSemantics: "readdir-position"`) and are valid while the
-directories are unchanged — making them name-ordered would require visiting every dirent
-to find the next N names, which is the unbounded walk the entry cap exists to prevent.
-Ordinary paging *within* one scan uses `offset`/`limit` with `hasMore`/`nextOffset`.
+**Every cap is losslessly resumable.** Both walks are a single streaming pass: each dirent
+is validated as it arrives and the recorded position advances **only after** that entry is
+fully processed, so a cap or deadline that trips mid-entry re-reads it rather than skipping
+it. A truncated report carries `scan.continuation` with the pending roots and an opaque
+`cursor` to pass back.
+
+The project position is `(rootIndex, containerIndex, entriesConsumed)`, where `rootIndex`
+indexes the **uncapped** configured-root list — that offset is what lets `maxRoots` advance
+at all; without it every call rebuilt the same capped prefix and a 13th configured root was
+unreachable forever. The skill cursor records the roots that finished *cleanly*, the
+projects whose every root finished cleanly, and the exact dirent position inside the one
+root that was interrupted; a partially consumed project is re-listed and resumed, never
+marked done.
+
+Cursors are raw readdir stream positions (`cursorSemantics: "readdir-stream-position"`)
+and are valid while the directories are unchanged — name-ordered cursors would require
+visiting every dirent to find the next N names, which is the unbounded walk the entry cap
+exists to prevent. Rows inside a returned page are sorted for presentation; *which* rows
+land in a truncated page is stream order.
+
+There are deliberately **two** paginations and a client needs both: `hasMore`/`nextOffset`
+walks the rows of one scan, `scan.continuation.cursor` resumes the scan itself past a cap.
### Ids, so nothing shadows anything
@@ -184,7 +207,10 @@ of sha256 over the canonical container path. It was 32 bits, and a review found
collision — `/tmp/mso-root-50323` and `/tmp/mso-root-125549` both hashed to `51e156ef` —
which would have merged two roots' same-named projects back into one row. Belt and
braces, nothing dedupes on the hash: the internal key is the full canonical path, so even
-a collision cannot merge two containers. That makes ids **globally unique**: two configured roots may each hold a
+a collision cannot merge two containers. That exact pair is an end-to-end regression
+(`lib/mcp/collision-e2e.test.ts`) asserting that `projects_list`, `skills_list`,
+`skills_read`, `skills_search` and `workflow_start` each return the **second** project
+when handed the second id. That makes ids **globally unique**: two configured roots may each hold a
`widget` shipping `deploy`, and both stay visible, readable and searchable. The derived
`projects/` container gets its own `rootId` for the same reason, so `~/widget` and
`~/projects/widget` cannot collide either. Within one project, `.mso/skills` outranks the
diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md
index 4fb9016f..c327667f 100644
--- a/docs/PROGRESS.md
+++ b/docs/PROGRESS.md
@@ -8,6 +8,60 @@ Running log of what shipped each phase. Newest at top.
> Read those phases as history. **This file is the source of truth for what exists** —
> `ARCHITECTURE.md` is no longer maintained and carries a stale-warning banner.
+## 2026-08-20 — lossless continuation and exact-id project resolution (DONE)
+
+The final fail-closed review found six remaining items. Five were continuation bugs of the
+same species — a cursor that *described* where a scan stopped instead of *being* a position
+you could resume from.
+
+**Hidden `rootHint` still resolved.** `validateRootHint` checked the final entry for
+symlink, shape, uid and credentials but never for a dot-prefixed component, so supplying
+`<authorized-root>/.hidden` as the root resolved a project underneath it by exact name
+while enumeration refused the same tree. The hidden check is now measured *relative to the
+authorized root*: the root's own path may contain dot components (a checkout under
+`~/.claude/worktrees` does), but nothing below it may.
+
+**`maxRoots` could not advance, ever.** `listProjectDirs()` rebuilt the same capped
+`projectContainers()` on every call, so a 13th configured root stayed pending no matter how
+many times a client followed the cursor. Root identity is now an index into the *uncapped*
+configured list, `authorizedRoots(startIndex)` slides that window forward, and the
+continuation carries the index of the first root the scan could not honour.
+
+**`maxProjects` and `maxProjectSkills` lost entries.** Both derived a readdir position from
+sorted accepted rows and a global result count — arithmetic that cannot be truthful, and
+which emitted `entriesConsumed=400` for a root holding 250 dirents. Both walks are now a
+single streaming pass: each dirent is validated as it arrives, every cap is checked
+*before* the entry is touched, and the recorded position advances *only after* the entry is
+fully handled. The skill cursor separately records roots that finished cleanly, projects
+whose every root finished cleanly, and the exact position inside the one interrupted root —
+a partially consumed project is re-listed and resumed rather than marked done.
+
+**Deadlines skipped unprocessed entries.** Both scanners read every name, sorted, then
+validated; a deadline expiring during validation left the position past names nothing had
+looked at. Same streaming fix — the cursor cannot outrun the work.
+
+**Exact project ids were unusable through `workflow_start`.** `resolveProjectHint` had no
+root-qualified-id branch, so `<rootId>/<name>` fell through to fuzzy matching and, with two
+same-named projects, returned the *wrong* one. An exact `<32-hex>/<name>` is now parsed and
+resolved before alias, package or fuzzy; the rootId maps to exactly one container across
+every configured root; an unknown rootId is refused rather than guessed.
+
+The colliding pair from the previous review (`/tmp/mso-root-50323` and
+`/tmp/mso-root-125549`, both `51e156ef` at 8 hex) is now an end-to-end regression:
+`projects_list`, `skills_list`, `skills_read`, `skills_search` and `workflow_start` each
+return the **second** project when handed the second id. Continuation reproducers drain
+13 roots, 2×250 projects, a 437-entry root and a 150+160-skill pair to completion and
+assert no row is dropped or repeated. Note that there are deliberately two paginations —
+`hasMore`/`nextOffset` within one scan, `scan.continuation.cursor` across scans — and the
+docs now say so, because conflating them looks exactly like data loss.
+
+The positive suites are unchanged and green: all-agents-all-tools, the exec/read/write
+ladder, image generation removed everywhere, `fs_upload_file` with regional ChatGPT import,
+and auth/OAuth untouched. `project-identity.ts`, `project-authorized-roots.ts` and
+`catalog-cursor.ts` split the work so every file stays under 200 lines and the value-cycle
+count stays at zero. The MCP server/toolset advance to `1.5.3` / `2026.08.20.6`; the catalog
+stays at **26 tools** (14 read, 10 write, 2 exec).
+
## 2026-08-20 — one validator, dirent-counted budgets, resumable caps (DONE)
A second fail-closed re-review of the discovery hardening found six things still open.
diff --git a/lib/host/project-authorized-roots.ts b/lib/host/project-authorized-roots.ts
new file mode 100644
index 00000000..aa225c87
--- /dev/null
+++ b/lib/host/project-authorized-roots.ts
@@ -0,0 +1,60 @@
+// WHICH configured roots a scan may touch, and where a continuation picks them back up.
+//
+// Two lists on purpose. Authorization is uncapped — a root the owner configured is inside
+// the jail whether or not this call had budget to walk it. A SCAN is capped, and the cap
+// is a sliding window over that list, which is the only thing that makes `maxRoots`
+// continuable at all.
+import { promises as fs } from "fs";
+import { isCredentialPath, readRootList } from "./paths";
+import { PROJECT_LIMITS, shortId, type AuthorizedRoot } from "./project-identity";
+
+export async function allConfiguredRoots(): Promise<AuthorizedRoot[]> {
+ const out: AuthorizedRoot[] = [];
+ const seen = new Set<string>();
+ for (const configured of readRootList()) {
+ const real = await fs.realpath(configured).catch(() => null);
+ if (!real || real === "/" || seen.has(real)) continue;
+ if (!(await fs.stat(real).catch(() => null))?.isDirectory()) continue;
+ if (isCredentialPath(real)) continue;
+ seen.add(real);
+ out.push({ id: shortId(real), configured, path: real, index: out.length });
+ }
+ return out;
+}
+
+/**
+ * The roots ONE scan will honour: `maxRoots` of them, starting at `startIndex`.
+ *
+ * The start offset is what makes `maxRoots` continuable. Without it the second call
+ * rebuilt the identical capped prefix and the 13th configured root was unreachable
+ * forever, however many times the caller followed the cursor.
+ */
+export async function authorizedRoots(startIndex = 0): Promise<AuthorizedRoot[]> {
+ return (await allConfiguredRoots()).slice(startIndex, startIndex + PROJECT_LIMITS.maxRoots);
+}
+
+/**
+ * EVERY configured root, canonicalized, with NO cap.
+ *
+ * `authorizedRoots()` caps at `maxRoots` because a SCAN has to be bounded. Authorization
+ * is a different question: a root the owner configured is inside the jail whether or not
+ * this call had budget to walk it. Using the capped list as the jail predicate is what
+ * made an explicitly named `rootHint` unresolvable once 12 other roots were configured.
+ */
+export async function configuredRootPaths(): Promise<string[]> {
+ return (await allConfiguredRoots()).map((r) => r.path);
+}
+
+/** Roots configured but NOT honoured, so the caller can say so rather than imply the
+ * configuration was fully covered. */
+export async function overflowRoots(startIndex = 0): Promise<{ roots: Array<{ path: string; reason: string }>; nextIndex: number }> {
+ const all = await allConfiguredRoots();
+ const end = startIndex + PROJECT_LIMITS.maxRoots;
+ const remaining = all.slice(end);
+ return {
+ roots: remaining.map((r) => ({ path: r.path, reason: "maxRoots" })),
+ // Where a continuation must pick the root list back up.
+ nextIndex: remaining.length ? end : all.length,
+ };
+}
+
diff --git a/lib/host/project-candidate.ts b/lib/host/project-candidate.ts
index 4bfcd05b..3d88fe84 100644
--- a/lib/host/project-candidate.ts
+++ b/lib/host/project-candidate.ts
@@ -73,10 +73,21 @@ export async function validateProjectDescendant(container: ProjectContainer, tar
return { ok: true, path: parent };
}
-/** The same rules applied to a caller-supplied ROOT directory, before it is trusted as
- * a container. A symlinked or hidden `rootHint` is refused rather than canonicalized
- * into something the caller never named. */
-export async function validateRootHint(absolute: string): Promise<CandidateResult> {
+/**
+ * The same rules applied to a caller-supplied ROOT directory, before it is trusted as a
+ * container. A symlinked or hidden `rootHint` is refused rather than canonicalized into
+ * something the caller never named.
+ *
+ * `authorizedRoot` is required for the hidden check, and the relative calculation is the
+ * whole point: the authorized root's OWN path may legitimately contain dot components
+ * (a checkout under `~/.claude/worktrees`, say), but nothing BELOW it may. Checking only
+ * the final component let `<authorized-root>/.hidden/widget` resolve by exact name while
+ * enumeration refused the same directory.
+ */
+export async function validateRootHint(absolute: string, authorizedRoot: string): Promise<CandidateResult> {
+ if (!isUnderRoot(absolute, authorizedRoot)) return { ok: false, reason: "escape" };
+ const relative = path.relative(authorizedRoot, absolute);
+ if (relative && relative.split(path.sep).some((segment) => segment.startsWith("."))) return { ok: false, reason: "hidden" };
const stat = await fs.lstat(absolute).catch(() => null);
if (!stat) return { ok: false, reason: "missing" };
if (stat.isSymbolicLink()) return { ok: false, reason: "symlink" };
diff --git a/lib/host/project-containers.ts b/lib/host/project-containers.ts
index 5e5e64cf..e602fdea 100644
--- a/lib/host/project-containers.ts
+++ b/lib/host/project-containers.ts
@@ -20,28 +20,22 @@
// Everything the scan omits is REPORTED. `truncated:false` means "this is all of it";
// hitting a root, entry, project or time cap sets `truncated:true` with a named
// reason. A silent slice that claims completeness is worse than a refusal.
-import { createHash } from "crypto";
import { promises as fs } from "fs";
import path from "path";
-import { isCredentialPath, isUnderRoot, readRootList } from "./paths";
-import { boundedGitMeta, packageMeta } from "./project-meta";
-
-export const PROJECT_LIMITS = {
- /** Authorized OS_FS_READ_ROOTS entries honoured per call. */
- maxRoots: 12,
- /** Directory entries READ from one container before the walk stops. */
- maxEntriesPerRoot: 400,
- /** Projects enumerated in total, across all containers. */
- maxProjects: 400,
- /** Wall-clock ceiling for one whole enumeration. */
- maxScanMs: 4000,
- defaultPageSize: 50,
- maxPageSize: 200,
-} as const;
-
-export type AuthorizedRoot = { id: string; configured: string; path: string };
+import { isCredentialPath, isUnderRoot } from "./paths";
+import { PROJECT_LIMITS, shortId, type AuthorizedRoot } from "./project-identity";
+import { allConfiguredRoots, authorizedRoots } from "./project-authorized-roots";
+
+export { allConfiguredRoots, authorizedRoots };
+export { PROJECT_LIMITS, shortId, containerKey } from "./project-identity";
+export type { AuthorizedRoot } from "./project-identity";
+
+
+
+
export type { ScanCursor, ScanReport } from "./project-scan-types";
+export { configuredRootPaths, overflowRoots } from "./project-authorized-roots";
export type ProjectContainer = {
/** Short sha256 of the canonical container path. The DERIVED `projects/` child gets
@@ -52,6 +46,9 @@ export type ProjectContainer = {
authorizedRootId: string;
authorizedRoot: string;
derived: boolean;
+ /** (rootIndex, containerIndex) is this container's exact address for a cursor. */
+ rootIndex: number;
+ containerIndex: number;
};
export type ProjectRow = {
@@ -68,22 +65,7 @@ export type ProjectRow = {
git?: { branch?: string; head?: string };
};
-/**
- * 128 bits of sha256 over the canonical container path.
- *
- * This was 8 hex characters — 32 bits — and a review found a REAL collision in the
- * fixture space it was tested in: `/tmp/mso-root-50323` and `/tmp/mso-root-125549`
- * both hashed to `51e156ef`. Two colliding roots holding same-named projects would
- * have merged back into one row, re-creating the exact bug root-qualified ids exist to
- * fix. 32 hex characters make that computationally unreachable — and, belt and braces,
- * nothing DEDUPES on this value: the internal key is the full canonical path (see
- * `containerKey`), so even a collision cannot merge two containers.
- */
-export const shortId = (real: string) => createHash("sha256").update(real).digest("hex").slice(0, 32);
-/** The internal identity. Always the full canonical path — never the hash — so
- * dedupe/precedence can never be decided by a truncated digest. */
-export const containerKey = (real: string) => real;
const currentUid = (): number | undefined => (typeof process.getuid === "function" ? process.getuid() : undefined);
@@ -101,48 +83,6 @@ export async function ownedByUs(target: string): Promise<boolean> {
* Following a symlinked CONFIGURED root is intended — its realpath simply becomes the
* authorized real root, and everything else is measured against that.
*/
-export async function authorizedRoots(): Promise<AuthorizedRoot[]> {
- const out: AuthorizedRoot[] = [];
- const seen = new Set<string>();
- for (const configured of readRootList()) {
- if (out.length >= PROJECT_LIMITS.maxRoots) break;
- const real = await fs.realpath(configured).catch(() => null);
- if (!real || real === "/" || seen.has(real)) continue;
- if (!(await fs.stat(real).catch(() => null))?.isDirectory()) continue;
- if (isCredentialPath(real)) continue;
- seen.add(real);
- out.push({ id: shortId(real), configured, path: real });
- }
- return out;
-}
-
-/**
- * EVERY configured root, canonicalized, with NO cap.
- *
- * `authorizedRoots()` caps at `maxRoots` because a SCAN has to be bounded. Authorization
- * is a different question: a root the owner configured is inside the jail whether or not
- * this call had budget to walk it. Using the capped list as the jail predicate is what
- * made an explicitly named `rootHint` unresolvable once 12 other roots were configured.
- */
-export async function configuredRootPaths(): Promise<string[]> {
- const out: string[] = [];
- for (const configured of readRootList()) {
- const real = await fs.realpath(configured).catch(() => null);
- if (!real || real === "/" || out.includes(real)) continue;
- if (!(await fs.stat(real).catch(() => null))?.isDirectory()) continue;
- if (isCredentialPath(real)) continue;
- out.push(real);
- }
- return out;
-}
-
-/** Roots configured but NOT honoured, so the caller can say so rather than imply the
- * configuration was fully covered. */
-export async function overflowRoots(): Promise<Array<{ path: string; reason: string }>> {
- const honoured = new Set((await authorizedRoots()).map((r) => r.configured));
- return readRootList().filter((c) => !honoured.has(c)).map((path) => ({ path, reason: "maxRoots-or-unreadable" }));
-}
-
/** `<root>/projects`, only when it is a real, non-symlink directory contained in that
* same authorized root. */
async function derivedContainerPath(root: AuthorizedRoot): Promise<string | null> {
@@ -156,19 +96,43 @@ async function derivedContainerPath(root: AuthorizedRoot): Promise<string | null
/** Every container, deterministically: configured root order, each root followed by
* its derived `projects/` child, deduped by canonical path. */
-export async function projectContainers(): Promise<ProjectContainer[]> {
+export async function projectContainers(startIndex = 0): Promise<ProjectContainer[]> {
const out: ProjectContainer[] = [];
const seen = new Set<string>();
- for (const root of await authorizedRoots()) {
+ for (const root of await authorizedRoots(startIndex)) {
+ let containerIndex = 0;
for (const [candidate, derived] of [[root.path, false], [await derivedContainerPath(root), true]] as const) {
- if (!candidate || seen.has(candidate)) continue;
+ if (!candidate || seen.has(candidate)) { containerIndex += 1; continue; }
seen.add(candidate);
- out.push({ id: shortId(candidate), path: candidate, authorizedRootId: root.id, authorizedRoot: root.path, derived });
+ out.push({
+ id: shortId(candidate), path: candidate, authorizedRootId: root.id, authorizedRoot: root.path,
+ derived, rootIndex: root.index, containerIndex,
+ });
+ containerIndex += 1;
}
}
return out;
}
+/** Resolve a 32-hex rootId to its container, across EVERY configured root — not just the
+ * scan-capped window. An exact id must address the same project on every call. */
+export async function containerById(rootId: string): Promise<ProjectContainer | null> {
+ if (!/^[a-f0-9]{32}$/.test(rootId)) return null;
+ for (const root of await allConfiguredRoots()) {
+ let containerIndex = 0;
+ for (const [candidate, derived] of [[root.path, false], [await derivedContainerPath(root), true]] as const) {
+ if (candidate && shortId(candidate) === rootId) {
+ return {
+ id: rootId, path: candidate, authorizedRootId: root.id, authorizedRoot: root.path,
+ derived, rootIndex: root.index, containerIndex,
+ };
+ }
+ containerIndex += 1;
+ }
+ }
+ return null;
+}
+
/** Back-compat view: just the container paths. */
export async function projectRoots(): Promise<string[]> {
return (await projectContainers()).map((c) => c.path);
@@ -179,6 +143,9 @@ export async function projectRoots(): Promise<string[]> {
* when the global container list is already at its cap. Same id derivation, same
* containment rules — only the discovery path differs. */
export function containerFor(realDir: string): ProjectContainer {
- return { id: shortId(realDir), path: realDir, authorizedRootId: shortId(realDir), authorizedRoot: realDir, derived: false };
+ return {
+ id: shortId(realDir), path: realDir, authorizedRootId: shortId(realDir), authorizedRoot: realDir,
+ derived: false, rootIndex: 0, containerIndex: 0,
+ };
}
diff --git a/lib/host/project-continuation.test.ts b/lib/host/project-continuation.test.ts
new file mode 100644
index 00000000..c4974883
--- /dev/null
+++ b/lib/host/project-continuation.test.ts
@@ -0,0 +1,126 @@
+import { afterAll, describe, expect, it } from "vitest";
+import { promises as fs } from "fs";
+import os from "os";
+import path from "path";
+
+// CONTINUATION MUST BE LOSSLESS. A truncated scan that cannot be resumed to completion is
+// data loss wearing a label, and every one of these cases previously returned zero rows
+// on the second call while insisting there was more.
+const base = await fs.mkdtemp(path.join(os.tmpdir(), "mso-cont-"));
+const previous = process.env.OS_FS_READ_ROOTS;
+const { listProjects, PROJECT_LIMITS } = await import("./project-roots");
+const setRoots = (...roots: string[]) => { process.env.OS_FS_READ_ROOTS = roots.join(":"); };
+
+afterAll(async () => {
+ if (previous === undefined) delete process.env.OS_FS_READ_ROOTS;
+ else process.env.OS_FS_READ_ROOTS = previous;
+ await fs.rm(base, { recursive: true, force: true });
+});
+
+/**
+ * Drain everything. There are deliberately TWO paginations and a caller has to use both:
+ * `hasMore`/`nextOffset` walks the rows of ONE scan, `scan.continuation.cursor` resumes
+ * the scan itself past a cap. Conflating them is how you conclude data is missing when
+ * it is merely on the next page.
+ */
+async function drain(cursor?: string) {
+ const seen = new Set<string>();
+ for (let scan = 0; scan < 40; scan += 1) {
+ let page = await listProjects({ limit: PROJECT_LIMITS.maxPageSize, cursor });
+ for (const p of page.projects) seen.add(p.path);
+ while (page.hasMore) {
+ page = await listProjects({ limit: PROJECT_LIMITS.maxPageSize, offset: page.nextOffset, cursor });
+ for (const p of page.projects) seen.add(p.path);
+ }
+ if (!page.scan.truncated || !page.scan.continuation?.cursor) break;
+ if (page.scan.continuation.cursor === cursor) break; // no forward progress
+ cursor = page.scan.continuation.cursor;
+ }
+ return seen;
+}
+
+describe("maxRoots continuation advances past the capped prefix", () => {
+ it("reaches projects in the 13th configured root", async () => {
+ const roots: string[] = [];
+ for (let i = 0; i <= PROJECT_LIMITS.maxRoots; i += 1) {
+ const dir = path.join(base, `mr-${String(i).padStart(2, "0")}`);
+ await fs.mkdir(path.join(dir, `p${i}`), { recursive: true });
+ roots.push(dir);
+ }
+ setRoots(...roots);
+
+ const first = await listProjects({ limit: PROJECT_LIMITS.maxPageSize });
+ expect(first.scan.truncated).toBe(true);
+ expect(first.scan.truncationReasons).toContain("maxRoots");
+
+ const all = await drain();
+ // The last root's project must be reachable, not merely reported as pending.
+ expect([...all].some((p) => p.endsWith(`mr-${PROJECT_LIMITS.maxRoots}/p${PROJECT_LIMITS.maxRoots}`))).toBe(true);
+ expect(all.size).toBe(PROJECT_LIMITS.maxRoots + 1);
+ });
+});
+
+describe("maxProjects continuation keeps the exact raw dirent position", () => {
+ it("returns the remaining projects instead of zero", async () => {
+ // Two roots of 250 each: the cap trips mid-way through the SECOND root, which is
+ // exactly where a cursor derived from a global accepted-count went wrong.
+ const a = path.join(base, "mp-a");
+ const b = path.join(base, "mp-b");
+ const per = 250;
+ await Promise.all([a, b].flatMap((root) =>
+ Array.from({ length: per }, (_, i) => fs.mkdir(path.join(root, `p${String(i).padStart(4, "0")}`), { recursive: true }))));
+ setRoots(a, b);
+
+ const first = await listProjects({ limit: PROJECT_LIMITS.maxPageSize });
+ expect(first.scan.truncated).toBe(true);
+ expect(first.scan.truncationReasons).toContain("maxProjects");
+
+ const all = await drain();
+ expect(all.size).toBe(per * 2);
+ });
+});
+
+describe("maxEntriesPerRoot continuation resumes at the exact position", () => {
+ it("returns every project across repeated pages", async () => {
+ const root = path.join(base, "me-root");
+ const count = PROJECT_LIMITS.maxEntriesPerRoot + 37;
+ await Promise.all(Array.from({ length: count }, (_, i) =>
+ fs.mkdir(path.join(root, `p${String(i).padStart(4, "0")}`), { recursive: true })));
+ setRoots(root);
+
+ const first = await listProjects({ limit: PROJECT_LIMITS.maxPageSize });
+ expect(first.scan.truncated).toBe(true);
+ const all = await drain();
+ expect(all.size).toBe(count);
+ });
+
+ it("never re-reports a project already returned by an earlier page", async () => {
+ const root = path.join(base, "me-dup");
+ const count = PROJECT_LIMITS.maxEntriesPerRoot + 5;
+ await Promise.all(Array.from({ length: count }, (_, i) =>
+ fs.mkdir(path.join(root, `p${String(i).padStart(4, "0")}`), { recursive: true })));
+ setRoots(root);
+
+ // One full scan (both its pages), then the resumed scan: no row may appear twice.
+ const firstA = await listProjects({ limit: PROJECT_LIMITS.maxPageSize });
+ const firstB = await listProjects({ limit: PROJECT_LIMITS.maxPageSize, offset: firstA.nextOffset });
+ const firstPaths = new Set([...firstA.projects, ...firstB.projects].map((p) => p.path));
+ expect(firstPaths.size).toBe(PROJECT_LIMITS.maxEntriesPerRoot);
+
+ const second = await listProjects({ limit: PROJECT_LIMITS.maxPageSize, cursor: firstA.scan.continuation!.cursor });
+ expect(second.projects.every((p) => !firstPaths.has(p.path))).toBe(true);
+ expect(firstPaths.size + second.projects.length).toBe(count);
+ });
+});
+
+describe("a complete scan still reports no continuation", () => {
+ it("stays truncated=false with the cursor absent", async () => {
+ const root = path.join(base, "small");
+ await fs.mkdir(path.join(root, "one"), { recursive: true });
+ setRoots(root);
+ const { scan, projects } = await listProjects();
+ expect(scan.truncated).toBe(false);
+ expect(scan.continuation).toBeUndefined();
+ expect(projects.map((p) => p.name)).toEqual(["one"]);
+ });
+});
diff --git a/lib/host/project-cursor.ts b/lib/host/project-cursor.ts
index 7c93a907..f6c8ddbe 100644
--- a/lib/host/project-cursor.ts
+++ b/lib/host/project-cursor.ts
@@ -4,10 +4,11 @@
// way to fetch the rest is data loss with a label on it.
import type { ScanCursor } from "./project-containers";
-/** Cursors are positional in readdir order. That is a deliberate trade: making them
- * name-ordered would require visiting every dirent to find the next N names, which is
- * the unbounded walk the entry cap exists to prevent. Valid while the directory is
- * unchanged; `note` says so in the payload. */
+/** Cursors are raw readdir STREAM positions. That is a deliberate trade: a name-ordered
+ * cursor would require visiting every dirent to find the next N names, which is the
+ * unbounded walk the entry cap exists to prevent. Rows within a returned page are
+ * sorted for presentation; WHICH rows land in a truncated page is stream order. Valid
+ * while the directory is unchanged; `note` says so in the payload. */
export function encodeCursor(cursor: ScanCursor): string {
return Buffer.from(JSON.stringify(cursor), "utf8").toString("base64url");
}
@@ -17,9 +18,12 @@ export function decodeCursor(raw: string | undefined): ScanCursor | undefined {
try {
const parsed = JSON.parse(Buffer.from(raw, "base64url").toString("utf8")) as ScanCursor;
if (!parsed || typeof parsed !== "object") return undefined;
+ const int = (v: unknown) => (Number.isFinite(v) ? Math.max(0, Math.floor(v as number)) : 0);
return {
- roots: Array.isArray(parsed.roots) ? parsed.roots.filter((r) => typeof r?.root === "string") : [],
- skipRoots: Array.isArray(parsed.skipRoots) ? parsed.skipRoots.filter((r) => typeof r === "string") : [],
+ rootIndex: int(parsed.rootIndex),
+ containerIndex: int(parsed.containerIndex),
+ entriesConsumed: int(parsed.entriesConsumed),
+ ...(typeof parsed.containerPath === "string" ? { containerPath: parsed.containerPath } : {}),
};
} catch {
return undefined;
diff --git a/lib/host/project-hint.test.ts b/lib/host/project-hint.test.ts
index 23e99147..f69867ab 100644
--- a/lib/host/project-hint.test.ts
+++ b/lib/host/project-hint.test.ts
@@ -19,6 +19,9 @@ await fs.mkdir(path.join(named, "orchard"), { recursive: true });
await fs.writeFile(path.join(named, "orchard", "package.json"), JSON.stringify({ name: "@farm/orchard" }));
await fs.mkdir(path.join(named, "greenhouse-annex"), { recursive: true });
await fs.mkdir(path.join(named, ".hidden-proj"), { recursive: true });
+// A real project sitting inside a HIDDEN directory. Supplying that hidden directory as
+// the rootHint used to resolve it by exact name, while enumeration refused the same tree.
+await fs.mkdir(path.join(named, ".hidden-root", "widget"), { recursive: true });
await fs.mkdir(outside, { recursive: true });
await fs.mkdir(path.join(outside, "elsewhere"), { recursive: true });
await fs.symlink(path.join(outside, "elsewhere"), path.join(named, "linked-proj"));
@@ -89,6 +92,12 @@ describe("the exact-name probe rejects what enumeration excludes", () => {
await expect(resolveProjectHint(".hidden-proj", named)).resolves.toBeNull();
});
+ it("refuses a rootHint that is itself inside a HIDDEN directory", async () => {
+ // The hidden check is measured relative to the authorized root, so a legitimately
+ // dot-bearing authorized root still works while nothing below it may hide.
+ await expect(resolveProjectHint("widget", path.join(named, ".hidden-root"))).resolves.toBeNull();
+ });
+
it("refuses a name that resolves outside the root it was probed in", async () => {
// `../outside/elsewhere` must not traverse out of the named root.
await expect(resolveProjectHint("../outside/elsewhere", named)).resolves.toBeNull();
diff --git a/lib/host/project-identity.ts b/lib/host/project-identity.ts
new file mode 100644
index 00000000..8e8a092d
--- /dev/null
+++ b/lib/host/project-identity.ts
@@ -0,0 +1,42 @@
+// The leaf: scan budgets and container identity. Nothing here imports anything of ours,
+// so both the root list and the container builder can depend on it without a cycle.
+import { createHash } from "crypto";
+
+export const PROJECT_LIMITS = {
+ /** Authorized OS_FS_READ_ROOTS entries honoured per call. */
+ maxRoots: 12,
+ /** Directory entries READ from one container before the walk stops. */
+ maxEntriesPerRoot: 400,
+ /** Projects enumerated in total, across all containers. */
+ maxProjects: 400,
+ /** Wall-clock ceiling for one whole enumeration. */
+ maxScanMs: 4000,
+ defaultPageSize: 50,
+ maxPageSize: 200,
+} as const;
+
+export type AuthorizedRoot = {
+ id: string;
+ configured: string;
+ path: string;
+ /** Position in the UNCAPPED configured list. A cursor stores this so a later call can
+ * slide the `maxRoots` window forward instead of re-scanning the same prefix. */
+ index: number;
+};
+
+/**
+ * 128 bits of sha256 over the canonical container path.
+ *
+ * This was 8 hex characters — 32 bits — and a review found a REAL collision in the
+ * fixture space it was tested in: `/tmp/mso-root-50323` and `/tmp/mso-root-125549`
+ * both hashed to `51e156ef`. Two colliding roots holding same-named projects would
+ * have merged back into one row, re-creating the exact bug root-qualified ids exist to
+ * fix. 32 hex characters make that computationally unreachable — and, belt and braces,
+ * nothing DEDUPES on this value: the internal key is the full canonical path (see
+ * `containerKey`), so even a collision cannot merge two containers.
+ */
+export const shortId = (real: string) => createHash("sha256").update(real).digest("hex").slice(0, 32);
+
+/** The internal identity. Always the full canonical path — never the hash — so
+ * dedupe/precedence can never be decided by a truncated digest. */
+export const containerKey = (real: string) => real;
diff --git a/lib/host/project-roots.ts b/lib/host/project-roots.ts
index dac65495..60578453 100644
--- a/lib/host/project-roots.ts
+++ b/lib/host/project-roots.ts
@@ -1,77 +1,91 @@
// The project WALK: bounded enumeration of the containers project-containers.ts
-// authorized, plus the truthful, RESUMABLE report of everything it could not cover.
+// authorized, plus a truthful, LOSSLESSLY RESUMABLE report of everything it could not
+// cover.
//
-// Three rules earned the hard way:
-// 1. `truncated:false` means "this is all of it". Any cap sets `truncated:true` with
-// a named reason — a silent slice that claims completeness is how a model ends up
+// Four rules earned the hard way:
+// 1. `truncated:false` means "this is all of it". Any cap sets `truncated:true` with a
+// named reason — a silent slice that claims completeness is how a model ends up
// telling the owner a project does not exist.
// 2. EVERY dirent counts against the entry cap, accepted or not. Counting only the
-// entries we kept meant a container of a million regular files still cost a
-// million iterations before the "400 entry" cap was reached.
-// 3. A cap is only honest if it can be continued. Every cap emits a cursor, and
-// passing it back resumes where the walk stopped.
+// entries we kept meant a container of a million regular files still cost a million
+// iterations before the "400 entry" cap was reached.
+// 3. ONE STREAMING PASS. Entries are validated as they arrive and the position advances
+// only AFTER an entry is fully processed. The previous version read every name,
+// sorted them, then validated — so a cap or deadline that tripped during validation
+// left the cursor pointing past names nothing had looked at, and they were lost.
+// 4. Caps are addressed by (rootIndex, containerIndex, entriesConsumed), where
+// rootIndex indexes the UNCAPPED configured-root list. Without that, `maxRoots`
+// could never advance: every call rebuilt the same capped prefix.
import { promises as fs } from "fs";
import path from "path";
import { validateProjectChild } from "./project-candidate";
import { decodeCursor, encodeCursor } from "./project-cursor";
import {
- overflowRoots, projectContainers,
- PROJECT_LIMITS, type ProjectContainer, type ProjectRow, type ScanReport, type ScanCursor,
+ allConfiguredRoots, overflowRoots, projectContainers,
+ PROJECT_LIMITS, type ProjectContainer, type ScanCursor, type ScanReport,
} from "./project-containers";
export {
- authorizedRoots, configuredRootPaths, containerFor, projectContainers, projectRoots, shortId, PROJECT_LIMITS,
+ allConfiguredRoots, authorizedRoots, configuredRootPaths, containerById, containerFor,
+ projectContainers, projectRoots, shortId, PROJECT_LIMITS,
} from "./project-containers";
export type { AuthorizedRoot, ProjectContainer, ProjectRow, ScanReport, ScanCursor } from "./project-containers";
export { decodeCursor, encodeCursor } from "./project-cursor";
-type Walk = { dirs: string[]; entriesVisited: number; hitEntryCap: boolean; skipped: number };
+type StopReason = "maxEntriesPerRoot" | "maxProjects" | "deadline";
+type Walk = {
+ dirs: string[];
+ /** Raw dirents FULLY PROCESSED. Never includes one a cap stopped us before validating. */
+ consumed: number;
+ stop?: StopReason;
+ skipped: number;
+};
/**
- * Bounded `opendir` iteration. Every dirent yielded increments the budget, so a
- * container full of rejected entries costs the cap and no more. `skipEntries` resumes a
- * previously truncated walk at the same readdir position.
+ * One streaming `opendir` pass. `skipEntries` fast-forwards to a recorded position, and
+ * `budget` is how many more projects the whole scan may still accept.
*/
-async function walkContainer(container: ProjectContainer, containerPaths: Set<string>, skipEntries: number, deadlineAt: number): Promise<Walk> {
- const names: string[] = [];
- let entriesVisited = skipEntries;
+async function walkContainer(
+ container: ProjectContainer,
+ containerPaths: Set<string>,
+ skipEntries: number,
+ deadlineAt: number,
+ budget: number,
+): Promise<Walk> {
+ const dirs: string[] = [];
+ let consumed = skipEntries;
let seen = 0;
- let hitEntryCap = false;
+ let processed = 0;
let skipped = 0;
+ let stop: StopReason | undefined;
const handle = await fs.opendir(container.path).catch(() => null);
- if (!handle) return { dirs: [], entriesVisited, hitEntryCap: false, skipped: 0 };
+ if (!handle) return { dirs, consumed, skipped };
try {
for await (const entry of handle) {
seen += 1;
if (seen <= skipEntries) continue;
- // EVERY dirent costs budget — that is the whole point of counting here rather
- // than after the accept/reject decision.
- if (entriesVisited - skipEntries >= PROJECT_LIMITS.maxEntriesPerRoot || Date.now() > deadlineAt) {
- hitEntryCap = true;
- break;
+ // Every check happens BEFORE the entry is processed, so stopping here leaves
+ // `consumed` on the last fully-handled dirent and this one is re-read next time.
+ if (processed >= PROJECT_LIMITS.maxEntriesPerRoot) { stop = "maxEntriesPerRoot"; break; }
+ if (Date.now() > deadlineAt) { stop = "deadline"; break; }
+ if (dirs.length >= budget) { stop = "maxProjects"; break; }
+
+ processed += 1;
+ const full = path.join(container.path, entry.name);
+ // `~/projects` under `~` is a CONTAINER, not a project inside its parent.
+ if (containerPaths.has(full)) skipped += 1;
+ else if (!entry.isDirectory() || entry.name.startsWith(".")) skipped += 1;
+ else {
+ const candidate = await validateProjectChild(container, entry.name);
+ if (candidate.ok) dirs.push(candidate.path);
+ else skipped += 1;
}
- entriesVisited += 1;
- if (!entry.isDirectory() || entry.name.startsWith(".")) { skipped += 1; continue; }
- names.push(entry.name);
+ consumed = seen; // ONLY now — the entry is fully handled.
}
} catch {
- // A directory that vanishes mid-walk yields what we already have.
- }
- names.sort((a, b) => a.localeCompare(b));
-
- const dirs: string[] = [];
- for (const name of names) {
- // The deadline is enforced THROUGH the metadata work too, not only in the dirent
- // loop: 400 lstat+realpath+stat triples on a cold or networked filesystem is where
- // a slow scan actually spends its time.
- if (Date.now() > deadlineAt) { hitEntryCap = true; break; }
- // `~/projects` under `~` is a CONTAINER, not a project inside its parent.
- if (containerPaths.has(path.join(container.path, name))) { skipped += 1; continue; }
- const candidate = await validateProjectChild(container, name);
- if (!candidate.ok) { skipped += 1; continue; }
- dirs.push(candidate.path);
+ // A directory that vanishes mid-walk yields what we already fully processed.
}
- return { dirs, entriesVisited, hitEntryCap, skipped };
+ return { dirs, consumed, stop, skipped };
}
export type ProjectDirs = {
@@ -80,59 +94,70 @@ export type ProjectDirs = {
scan: ScanReport;
};
-/** The one walk. `listProjectDirs()` is this over every discovered container; an
- * explicit `rootHint` is this over exactly one. */
+const complete = (rootCount: number): ScanCursor => ({ rootIndex: rootCount, containerIndex: 0, entriesConsumed: 0 });
+
+/** The one walk. `listProjectDirs()` is this over every discovered container; an explicit
+ * `rootHint` is this over exactly one. */
export async function listProjectDirsIn(
containers: ProjectContainer[],
- options: { explicit?: boolean; cursor?: ScanCursor } = {},
+ options: { explicit?: boolean; cursor?: ScanCursor; startIndex?: number } = {},
): Promise<ProjectDirs> {
const deadlineAt = Date.now() + PROJECT_LIMITS.maxScanMs;
+ const startIndex = options.startIndex ?? 0;
const dirs: Array<{ container: ProjectContainer; dir: string }> = [];
const reasons: string[] = [];
const scannedRoots: string[] = [];
- const skippedRoots = options.explicit ? [] : await overflowRoots();
+ const overflow = options.explicit ? { roots: [], nextIndex: startIndex } : await overflowRoots(startIndex);
+ const skippedRoots = [...overflow.roots];
const containerPaths = new Set(containers.map((c) => c.path));
- const resumeRoots = new Map((options.cursor?.roots ?? []).map((r) => [r.root, r.entriesConsumed]));
- const skipRoots = new Set(options.cursor?.skipRoots ?? []);
- const cursors: ScanCursor["roots"] = [];
+ const cursor = options.cursor;
const pendingRoots: string[] = [];
+ let position: ScanCursor | undefined;
let skippedProjects = 0;
- if (skippedRoots.length) { reasons.push("maxRoots"); pendingRoots.push(...skippedRoots.map((r) => r.path)); }
for (const container of containers) {
- if (skipRoots.has(container.path)) continue;
+ // A recorded position addresses one exact container; earlier ones are already done.
+ if (cursor && (container.rootIndex < cursor.rootIndex
+ || (container.rootIndex === cursor.rootIndex && container.containerIndex < cursor.containerIndex))) continue;
+ const resumeHere = cursor
+ && container.rootIndex === cursor.rootIndex
+ && container.containerIndex === cursor.containerIndex
+ && (!cursor.containerPath || cursor.containerPath === container.path);
+ const skipEntries = resumeHere ? cursor!.entriesConsumed : 0;
+
if (Date.now() > deadlineAt) {
reasons.push("deadline");
- skippedRoots.push({ path: container.path, reason: "deadline" });
- pendingRoots.push(container.path);
- continue;
+ position = { rootIndex: container.rootIndex, containerIndex: container.containerIndex, entriesConsumed: skipEntries, containerPath: container.path };
+ pendingRoots.push(...containers.slice(containers.indexOf(container)).map((c) => c.path));
+ break;
}
- const walk = await walkContainer(container, containerPaths, resumeRoots.get(container.path) ?? 0, deadlineAt);
+
+ const walk = await walkContainer(container, containerPaths, skipEntries, deadlineAt, PROJECT_LIMITS.maxProjects - dirs.length);
scannedRoots.push(container.path);
skippedProjects += walk.skipped;
- if (walk.hitEntryCap) {
- reasons.push(`maxEntriesPerRoot:${container.path}`);
- cursors.push({ root: container.path, entriesConsumed: walk.entriesVisited });
- }
- let capped = false;
- for (const dir of walk.dirs) {
- if (dirs.length >= PROJECT_LIMITS.maxProjects) { reasons.push("maxProjects"); capped = true; break; }
- dirs.push({ container, dir });
- }
- if (capped) {
- // Resume this container from where its accepted entries stopped, and leave the
- // remaining containers pending rather than pretending they were covered.
- cursors.push({ root: container.path, entriesConsumed: walk.entriesVisited - walk.dirs.length + dirs.length });
- pendingRoots.push(...containers.slice(containers.indexOf(container) + 1).map((c) => c.path));
+ for (const dir of walk.dirs) dirs.push({ container, dir });
+
+ if (walk.stop) {
+ reasons.push(walk.stop === "maxEntriesPerRoot" ? `maxEntriesPerRoot:${container.path}` : walk.stop);
+ position = { rootIndex: container.rootIndex, containerIndex: container.containerIndex, entriesConsumed: walk.consumed, containerPath: container.path };
+ pendingRoots.push(...containers.slice(containers.indexOf(container)).map((c) => c.path));
break;
}
}
+ // Nothing stopped mid-container, so the next position is the first root this scan
+ // could not honour — which is how `maxRoots` finally becomes continuable.
+ if (!position && overflow.roots.length) {
+ reasons.push("maxRoots");
+ pendingRoots.push(...overflow.roots.map((r) => r.path));
+ position = { rootIndex: overflow.nextIndex, containerIndex: 0, entriesConsumed: 0 };
+ }
+
const truncationReasons = [...new Set(reasons)];
- const done = [...new Set(scannedRoots)].filter((r) => !cursors.some((c) => c.root === r) && !pendingRoots.includes(r));
+ const rootCount = (await allConfiguredRoots()).length;
return {
containers,
- dirs,
+ dirs: dirs.sort((a, b) => a.container.path.localeCompare(b.container.path) || a.dir.localeCompare(b.dir)),
scan: {
truncated: truncationReasons.length > 0,
truncationReasons,
@@ -142,10 +167,10 @@ export async function listProjectDirsIn(
...(truncationReasons.length ? {
continuation: {
pendingRoots: [...new Set(pendingRoots)],
- cursors,
- cursorSemantics: "readdir-position" as const,
- note: "Pass `cursor` back to resume. Positions are readdir order and are valid while the directory is unchanged.",
- cursor: encodeCursor({ roots: cursors, skipRoots: done }),
+ position: position ?? complete(rootCount),
+ cursorSemantics: "readdir-stream-position" as const,
+ note: "Pass `cursor` back to resume at the exact dirent the scan stopped on. Positions are readdir stream order and are valid while the directories are unchanged.",
+ cursor: encodeCursor(position ?? complete(rootCount)),
},
} : {}),
},
@@ -154,7 +179,8 @@ export async function listProjectDirsIn(
/** Every project directory across every container, in container-then-name order. */
export async function listProjectDirs(cursor?: ScanCursor): Promise<ProjectDirs> {
- return listProjectDirsIn(await projectContainers(), { cursor });
+ const startIndex = cursor?.rootIndex ?? 0;
+ return listProjectDirsIn(await projectContainers(startIndex), { cursor, startIndex });
}
export { listProjects } from "./project-list";
diff --git a/lib/host/project-scan-types.ts b/lib/host/project-scan-types.ts
index f4414620..cfb416d3 100644
--- a/lib/host/project-scan-types.ts
+++ b/lib/host/project-scan-types.ts
@@ -2,11 +2,28 @@
// separate so both the project walk and the skill catalog can describe incompleteness
// in the same shape without importing each other's scanners.
-/** A resumable position. Positional in readdir order — see encodeCursor's note. */
+/**
+ * An EXACT resumable position, not a description of one.
+ *
+ * The previous shape recorded "entries consumed" derived from sorted accepted rows and a
+ * global result count, which could point past dirents that were never processed — and it
+ * had no way at all to advance past the `maxRoots` prefix, so a 13th configured root was
+ * permanently unreachable. This is a raw stream position instead:
+ *
+ * rootIndex index into the UNCAPPED configured-root list to start from, so the
+ * cap window slides forward across calls;
+ * containerIndex 0 = the root itself, 1 = its derived `projects/` child;
+ * entriesConsumed raw dirents FULLY PROCESSED in that container. A dirent that was
+ * read but not validated before a cap or deadline tripped is NOT
+ * counted, so it is re-processed rather than skipped.
+ */
export type ScanCursor = {
- roots: Array<{ root: string; entriesConsumed: number }>;
- /** Containers already fully covered by the previous page; skipped on resume. */
- skipRoots: string[];
+ rootIndex: number;
+ containerIndex: number;
+ entriesConsumed: number;
+ /** Canonical path the position was recorded against; a mismatch discards the position
+ * rather than resuming into a different directory. */
+ containerPath?: string;
};
export type ScanReport = {
@@ -20,8 +37,9 @@ export type ScanReport = {
* cannot resume is just data loss with a label on it. */
continuation?: {
pendingRoots: string[];
- cursors: Array<{ root: string; entriesConsumed: number }>;
- cursorSemantics: "readdir-position";
+ /** The exact stream position the next call resumes from. */
+ position: ScanCursor;
+ cursorSemantics: "readdir-stream-position";
note: string;
cursor: string;
};
diff --git a/lib/host/projects.ts b/lib/host/projects.ts
index 22b4f296..3ae9e1e6 100644
--- a/lib/host/projects.ts
+++ b/lib/host/projects.ts
@@ -4,7 +4,7 @@ import { normalizeProjectKey, projectAliasesFor, projectAliasTarget } from "./pr
import { boundedGitMeta, fullGitMeta, packageMeta } from "./project-meta";
import { homeDir, isUnderRoot } from "./paths";
import { validateProjectChild, validateProjectDescendant, validateRootHint } from "./project-candidate";
-import { configuredRootPaths, containerFor, listProjectDirsIn, projectContainers, type ProjectContainer } from "./project-roots";
+import { configuredRootPaths, containerById, containerFor, listProjectDirsIn, projectContainers, type ProjectContainer } from "./project-roots";
export type ProjectResolution = {
hint: string;
@@ -16,7 +16,7 @@ export type ProjectResolution = {
root: string;
packageName?: string;
aliases: string[];
- matchedBy: "path" | "name" | "alias" | "package" | "fuzzy";
+ matchedBy: "id" | "path" | "name" | "alias" | "package" | "fuzzy";
};
async function resolutionFor(container: ProjectContainer, dir: string, hint: string, matchedBy: ProjectResolution["matchedBy"]): Promise<ProjectResolution> {
@@ -47,13 +47,14 @@ function expandHome(p: string): string {
async function containersFor(rootHint?: string): Promise<ProjectContainer[]> {
if (!rootHint) return projectContainers();
const absolute = expandHome(rootHint);
- const validated = await validateRootHint(absolute);
- if (!validated.ok) return [];
// Checked against EVERY configured root, not the scan-capped subset: naming a root
// must never widen the jail, but the jail must not shrink just because 12 other roots
- // are configured ahead of it.
- const authorized = await configuredRootPaths();
- if (!authorized.some((root) => isUnderRoot(validated.path, root))) return [];
+ // are configured ahead of it. The owning root is also what the hidden-component check
+ // is measured against.
+ const owner = (await configuredRootPaths()).find((root) => isUnderRoot(absolute, root));
+ if (!owner) return [];
+ const validated = await validateRootHint(absolute, owner);
+ if (!validated.ok) return [];
return [containerFor(validated.path)];
}
@@ -72,6 +73,19 @@ export async function resolveProjectHint(hint: string, rootHint?: string): Promi
const containers = await containersFor(rootHint);
if (!containers.length) return null;
+ // EXACT `<32-hex-rootId>/<project-name>` first — the id projects_list advertises and
+ // workflow_start is told to pass. It must never fall through to fuzzy matching: with
+ // two same-named projects that returned the WRONG one, which is the entire failure the
+ // root-qualified id exists to prevent.
+ const exactId = /^([a-f0-9]{32})\/(.+)$/.exec(raw);
+ if (exactId) {
+ const container = await containerById(exactId[1]);
+ if (!container) return null;
+ if (rootHint && !containers.some((c) => c.path === container.path)) return null;
+ const candidate = await validateProjectChild(container, exactId[2]);
+ return candidate.ok ? resolutionFor(container, candidate.path, raw, "id") : null;
+ }
+
const pathHint = raw.startsWith("projects/") ? `~/${raw}` : raw;
if (/^(?:~\/|\/|\.\.?\/)/.test(pathHint)) {
// A path hint gets the SAME component-by-component validation an enumerated entry
diff --git a/lib/mcp/collision-e2e.test.ts b/lib/mcp/collision-e2e.test.ts
new file mode 100644
index 00000000..dd753066
--- /dev/null
+++ b/lib/mcp/collision-e2e.test.ts
@@ -0,0 +1,144 @@
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+vi.mock("server-only", () => ({}));
+import { promises as fs } from "fs";
+import path from "path";
+
+// THE COLLISION, END TO END.
+//
+// `/tmp/mso-root-50323` and `/tmp/mso-root-125549` both hash to `51e156ef` at 8 hex —
+// a real pair found by a probe, not a hypothetical. Each holds a project called `widget`
+// shipping a skill called `deploy`. Under the old 32-bit id these two collapsed into one
+// row and half the box became invisible.
+//
+// It is not enough that the ids differ: every surface a client actually uses must return
+// the SECOND project when handed the second id. That is what this file pins —
+// projects_list, skills_list, skills_read, skills_search and workflow_start.
+const ROOT_A = "/tmp/mso-root-50323";
+const ROOT_B = "/tmp/mso-root-125549";
+const previous = {
+ read: process.env.OS_FS_READ_ROOTS,
+ write: process.env.OS_FS_WRITE_ROOTS,
+ memory: process.env.OS_SKILL_MEMORY_STORE,
+};
+
+async function seed(root: string, marker: string) {
+ const skills = path.join(root, "widget", ".claude/skills/deploy");
+ await fs.mkdir(skills, { recursive: true });
+ await fs.writeFile(
+ path.join(skills, "SKILL.md"),
+ `---\nname: deploy\ndescription: Deploy the widget service from ${marker}.\n---\n\n# deploy ${marker}\n\nRun the ${marker} pipeline.\n`,
+ );
+ await fs.writeFile(path.join(root, "widget", "package.json"), JSON.stringify({ name: `widget-${marker}` }));
+}
+
+beforeAll(async () => {
+ await seed(ROOT_A, "alpha");
+ await seed(ROOT_B, "bravo");
+ process.env.OS_FS_READ_ROOTS = `${ROOT_A}:${ROOT_B}`;
+ process.env.OS_FS_WRITE_ROOTS = `${ROOT_A}:${ROOT_B}`;
+ process.env.OS_SKILL_MEMORY_STORE = path.join(ROOT_A, "memory.json");
+});
+
+afterAll(async () => {
+ for (const [key, value] of Object.entries(previous)) {
+ const name = key === "read" ? "OS_FS_READ_ROOTS" : key === "write" ? "OS_FS_WRITE_ROOTS" : "OS_SKILL_MEMORY_STORE";
+ if (value === undefined) delete process.env[name];
+ else process.env[name] = value;
+ }
+ await Promise.all([ROOT_A, ROOT_B].map((r) => fs.rm(r, { recursive: true, force: true })));
+});
+
+const { DISCOVERY_TOOLS } = await import("./tools-discovery");
+const { LEARNING_TOOLS } = await import("./tools-learning");
+const { shortId } = await import("@/lib/host/project-roots");
+const { searchSkillMemory } = await import("@/lib/skills/search");
+
+const tool = (name: string) => {
+ const found = [...DISCOVERY_TOOLS, ...LEARNING_TOOLS].find((t) => t.name === name);
+ if (!found) throw new Error(`missing ${name}`);
+ return found;
+};
+const run = (name: string, args: Record<string, unknown>, scope: "read" | "write" = "read") =>
+ tool(name).run(args, { scope, actor: "mcp:collision" });
+
+const idA = () => `${shortId(ROOT_A)}/widget`;
+const idB = () => `${shortId(ROOT_B)}/widget`;
+
+describe("the 8-hex collision pair stays distinct on every surface", () => {
+ it("collides at 8 hex and separates at 32 — the premise of this file", () => {
+ expect(shortId(ROOT_A).slice(0, 8)).toBe(shortId(ROOT_B).slice(0, 8));
+ expect(shortId(ROOT_A)).not.toBe(shortId(ROOT_B));
+ });
+
+ it("projects_list returns BOTH widgets with distinct ids", async () => {
+ const result = await run("projects_list", { limit: 200 }) as {
+ projects: Array<{ id: string; name: string; path: string; packageName?: string }>;
+ };
+ const widgets = result.projects.filter((p) => p.name === "widget");
+ expect(widgets).toHaveLength(2);
+ expect(widgets.map((p) => p.id).sort()).toEqual([idA(), idB()].sort());
+ expect(widgets.find((p) => p.id === idB())?.packageName).toBe("widget-bravo");
+ });
+
+ it("skills_list returns both projects' deploy skills under distinct ids", async () => {
+ const { skills } = await run("skills_list", { limit: 200 }) as {
+ skills: Array<{ id: string; description: string; project?: { id: string } }>;
+ };
+ const deploys = skills.filter((s) => s.id.endsWith("/deploy"));
+ expect(deploys.map((s) => s.id).sort()).toEqual([`${idA()}/deploy`, `${idB()}/deploy`].sort());
+ });
+
+ it("skills_list filtered by the SECOND project id returns only its skill", async () => {
+ const only = await run("skills_list", { project: idB(), limit: 200 }) as {
+ skills: Array<{ id: string; description: string }>;
+ };
+ expect(only.skills.map((s) => s.id)).toEqual([`${idB()}/deploy`]);
+ expect(only.skills[0].description).toContain("bravo");
+ });
+
+ it("skills_read on the SECOND id returns the SECOND project's instructions", async () => {
+ const result = await run("skills_read", { name: `${idB()}/deploy` }) as { content: string; project: { id: string } };
+ expect(result.project.id).toBe(idB());
+ expect(result.content).toContain("bravo");
+ expect(result.content).not.toContain("alpha");
+ });
+
+ it("skills_read refuses the ambiguous bare name and offers both exact ids", async () => {
+ await expect(run("skills_read", { name: "deploy" })).rejects.toThrow(/ambiguous across projects/);
+ await expect(run("skills_read", { name: "widget/deploy" })).rejects.toThrow(new RegExp(idB()));
+ });
+
+ it("skills_search surfaces both, each carrying its own project id", async () => {
+ const { hits } = await searchSkillMemory("deploy the widget service");
+ const deploys = hits.filter((h) => h.kind === "skill" && h.id.endsWith("/deploy"));
+ expect(deploys.map((h) => h.id).sort()).toEqual([`${idA()}/deploy`, `${idB()}/deploy`].sort());
+ expect(deploys.find((h) => h.id === `${idB()}/deploy`)?.project?.id).toBe(idB());
+ });
+
+ it("workflow_start resolves the SECOND project from its exact id, not the first", async () => {
+ const result = await run("workflow_start", {
+ intent: "deploy the widget service", project: idB(),
+ }, "write") as {
+ bootstrap: { project: { id: string; path: string; matchedBy: string } };
+ };
+ expect(result.bootstrap.project.matchedBy).toBe("id");
+ expect(result.bootstrap.project.id).toBe(idB());
+ expect(result.bootstrap.project.path).toBe(path.join(ROOT_B, "widget"));
+ });
+
+ it("workflow_start resolves the FIRST project from its own id", async () => {
+ const result = await run("workflow_start", {
+ intent: "deploy the widget service", project: idA(),
+ }, "write") as { bootstrap: { project: { id: string; path: string } } };
+ expect(result.bootstrap.project.id).toBe(idA());
+ expect(result.bootstrap.project.path).toBe(path.join(ROOT_A, "widget"));
+ });
+
+ it("refuses an id whose rootId is unknown rather than falling through to fuzzy", async () => {
+ const bogus = "f".repeat(32);
+ const result = await run("workflow_start", { intent: "x", project: `${bogus}/widget` }, "write") as {
+ bootstrap: { project?: { matchedBy?: string } };
+ };
+ expect(result.bootstrap.project?.matchedBy).toBe("unresolved");
+ });
+});
diff --git a/lib/mcp/dispatch.test.ts b/lib/mcp/dispatch.test.ts
index 3f1c3d2a..06ff4c16 100644
--- a/lib/mcp/dispatch.test.ts
+++ b/lib/mcp/dispatch.test.ts
@@ -44,7 +44,7 @@ describe("protocol", () => {
it("publishes server and toolset metadata so action drift is visible", async () => {
const r = await dispatch({ id: 1, method: "initialize" }, "exec");
const result = r.result as { serverInfo: { version: string }; _meta: { toolset: { toolCount: number; hash: string } } };
- expect(result.serverInfo.version).toBe("1.5.2");
+ expect(result.serverInfo.version).toBe("1.5.3");
expect(result._meta.toolset.toolCount).toBe(TOOLS.length);
expect(result._meta.toolset.hash).toMatch(/^[a-f0-9]{16}$/);
});
diff --git a/lib/mcp/toolset.ts b/lib/mcp/toolset.ts
index bd2ca363..3d20d576 100644
--- a/lib/mcp/toolset.ts
+++ b/lib/mcp/toolset.ts
@@ -2,9 +2,9 @@ import { createHash } from "crypto";
import type { Scope } from "./scope";
import type { McpTool } from "./tool-kit";
-export const MCP_SERVER_VERSION = "1.5.2";
-export const MCP_TOOLSET_VERSION = "2026.08.20.5";
-export const MCP_TOOLSET_CHANGED_AT = "2026-08-20T16:00:00Z";
+export const MCP_SERVER_VERSION = "1.5.3";
+export const MCP_TOOLSET_VERSION = "2026.08.20.6";
+export const MCP_TOOLSET_CHANGED_AT = "2026-08-20T16:40:00Z";
export type McpToolsetInfo = {
serverVersion: string;
diff --git a/lib/skills/catalog-continuation.test.ts b/lib/skills/catalog-continuation.test.ts
new file mode 100644
index 00000000..7800114c
--- /dev/null
+++ b/lib/skills/catalog-continuation.test.ts
@@ -0,0 +1,114 @@
+import { afterEach, describe, expect, it } from "vitest";
+import { mkdir, mkdtemp, rm, writeFile } from "fs/promises";
+import os from "os";
+import path from "path";
+import { catalogSkillsDetailed } from "./catalog";
+import { SKILL_SCAN_LIMITS } from "./catalog-types";
+import { projectRefFor } from "./project-skills";
+
+const temps: string[] = [];
+async function temp() {
+ const dir = await mkdtemp(path.join(os.tmpdir(), "mso-skill-cont-"));
+ temps.push(dir);
+ return dir;
+}
+afterEach(async () => { await Promise.all(temps.splice(0).map((d) => rm(d, { recursive: true, force: true }))); });
+
+const body = (name: string) => `---\nname: ${name}\ndescription: ${name} desc\n---\n\n# ${name}\n`;
+
+async function skillsIn(root: string, count: number, prefix: string) {
+ await mkdir(root, { recursive: true });
+ await Promise.all(Array.from({ length: count }, async (_, i) => {
+ const dir = path.join(root, `${prefix}${String(i).padStart(4, "0")}`);
+ await mkdir(dir, { recursive: true });
+ await writeFile(path.join(dir, "SKILL.md"), body(`${prefix}${i}`));
+ }));
+}
+
+/** Follow the cursor until the catalog reports itself complete. */
+async function drain(opts: { appDir: string; homeDir: string; projects: ReturnType<typeof projectRefFor>[] }) {
+ const ids = new Set<string>();
+ let cursor: string | undefined;
+ for (let i = 0; i < 40; i += 1) {
+ const page = await catalogSkillsDetailed({ ...opts, cursor });
+ for (const s of page.skills) ids.add(s.id);
+ if (!page.scan.truncated || !page.scan.continuation?.cursor) break;
+ if (page.scan.continuation.cursor === cursor) break; // no forward progress
+ cursor = page.scan.continuation.cursor;
+ }
+ return ids;
+}
+
+describe("maxProjectSkills continuation returns the remainder of a partially consumed root", () => {
+ it("drains 150 + 160 project skills across pages without losing the last 10", async () => {
+ const workspace = await temp();
+ const one = path.join(workspace, "one");
+ const two = path.join(workspace, "two");
+ await skillsIn(path.join(one, ".claude/skills"), 150, "a");
+ await skillsIn(path.join(two, ".claude/skills"), 160, "b");
+ const projects = [projectRefFor(one, workspace), projectRefFor(two, workspace)];
+ const opts = { appDir: await temp(), homeDir: await temp(), projects };
+
+ const first = await catalogSkillsDetailed(opts);
+ expect(first.skills.filter((s) => s.project).length).toBe(SKILL_SCAN_LIMITS.maxProjectSkills);
+ expect(first.scan.truncationReasons).toContain("maxProjectSkills");
+ expect(first.scan.continuation!.pendingProjects).toBeGreaterThan(0);
+
+ const all = await drain(opts);
+ // 310 project skills in total, none dropped.
+ expect([...all].filter((id) => id.includes("/")).length).toBe(310);
+ });
+
+ it("records the exact interrupted root rather than marking it complete", async () => {
+ const workspace = await temp();
+ const one = path.join(workspace, "one");
+ await skillsIn(path.join(one, ".claude/skills"), SKILL_SCAN_LIMITS.maxProjectSkills + 20, "c");
+ const projects = [projectRefFor(one, workspace)];
+ const opts = { appDir: await temp(), homeDir: await temp(), projects };
+
+ const first = await catalogSkillsDetailed(opts);
+ const resume = first.scan.continuation!.cursors[0];
+ expect(resume.root).toBe(path.join(one, ".claude/skills"));
+ expect(resume.entriesConsumed).toBeGreaterThan(0);
+ expect(first.scan.continuation!.pendingRoots).toContain(resume.root);
+
+ const all = await drain(opts);
+ expect([...all].filter((id) => id.includes("/")).length).toBe(SKILL_SCAN_LIMITS.maxProjectSkills + 20);
+ });
+});
+
+describe("a deadline never advances the cursor past an unprocessed entry", () => {
+ it("keeps every skill reachable when the deadline expires immediately", async () => {
+ const app = await temp();
+ await skillsIn(path.join(app, "claude-skills"), 3, "d");
+ const opts = { appDir: app, homeDir: await temp(), projects: [] };
+
+ // maxScanMs is a constant, so the honest way to force the deadline branch is a clock
+ // that is already past it on the first check.
+ const realNow = Date.now;
+ let calls = 0;
+ Date.now = () => (calls++ === 0 ? realNow() : realNow() + SKILL_SCAN_LIMITS.maxScanMs + 1_000);
+ let first;
+ try {
+ first = await catalogSkillsDetailed(opts);
+ } finally {
+ Date.now = realNow;
+ }
+ expect(first.scan.truncated).toBe(true);
+ expect(first.scan.truncationReasons).toContain("deadline");
+ // Whatever it could not process must still be reachable on resume.
+ const resumed = await catalogSkillsDetailed({ ...opts, cursor: first.scan.continuation!.cursor });
+ const ids = new Set([...first.skills, ...resumed.skills].map((s) => s.id));
+ expect([...ids].filter((id) => id.startsWith("d"))).toHaveLength(3);
+ });
+});
+
+describe("a complete catalog build reports no continuation", () => {
+ it("stays truncated=false", async () => {
+ const app = await temp();
+ await skillsIn(path.join(app, "claude-skills"), 2, "e");
+ const { scan } = await catalogSkillsDetailed({ appDir: app, homeDir: await temp(), projects: [] });
+ expect(scan).toMatchObject({ truncated: false, truncationReasons: [] });
+ expect(scan.continuation).toBeUndefined();
+ });
+});
diff --git a/lib/skills/catalog-cursor.ts b/lib/skills/catalog-cursor.ts
new file mode 100644
index 00000000..d08b8c87
--- /dev/null
+++ b/lib/skills/catalog-cursor.ts
@@ -0,0 +1,25 @@
+// Encoding for a resumable skill-catalog position. Same contract as the project walk's
+// cursor: a cap the caller cannot continue past is data loss with a label on it.
+import type { SkillScanCursor } from "./catalog-types";
+
+export function encodeSkillCursor(cursor: SkillScanCursor): string {
+ return Buffer.from(JSON.stringify(cursor), "utf8").toString("base64url");
+}
+
+export function decodeSkillCursor(raw: string | undefined): SkillScanCursor | undefined {
+ if (!raw) return undefined;
+ try {
+ const parsed = JSON.parse(Buffer.from(raw, "base64url").toString("utf8")) as SkillScanCursor;
+ if (!parsed || typeof parsed !== "object") return undefined;
+ return {
+ doneRoots: Array.isArray(parsed.doneRoots) ? parsed.doneRoots.filter((r) => typeof r === "string") : [],
+ projectOffset: Number.isFinite(parsed.projectOffset) ? Math.max(0, Math.floor(parsed.projectOffset)) : 0,
+ ...(parsed.resume && typeof parsed.resume.root === "string"
+ ? { resume: { root: parsed.resume.root, entriesConsumed: Math.max(0, Math.floor(parsed.resume.entriesConsumed ?? 0)) } }
+ : {}),
+ };
+ } catch {
+ return undefined;
+ }
+}
+
diff --git a/lib/skills/catalog-scan.ts b/lib/skills/catalog-scan.ts
index 527504f8..d3145a88 100644
--- a/lib/skills/catalog-scan.ts
+++ b/lib/skills/catalog-scan.ts
@@ -70,72 +70,77 @@ async function verifiedBundledSkill(dir: string, md: string): Promise<Pick<Skill
};
}
-/** Bounded `opendir` walk of ONE skill root — global roots included, which previously
- * had no entry budget at all. Names are capped before anything is stat'd or read. */
-async function skillDirNames(root: string, deadlineAt: number, skipEntries = 0): Promise<{ names: string[]; hitCap: boolean; entriesVisited: number }> {
- const names: string[] = [];
- let hitCap = false;
- let entriesVisited = skipEntries;
+/**
+ * ONE streaming pass over a skill root.
+ *
+ * The previous version read every dirent name, sorted them, then validated — so a
+ * deadline that expired during validation left `entriesVisited` pointing past names
+ * nothing had looked at, and continuation skipped them silently. Now each dirent is
+ * fully processed before the position advances, and every cap is checked BEFORE the
+ * entry is touched, so stopping never consumes it.
+ *
+ * `budget` is how many more skills the whole catalog may still accept, which is how the
+ * overall `maxProjectSkills` ceiling is enforced mid-root rather than only between roots.
+ */
+export async function scanRoot(
+ spec: RootSpec,
+ deadlineAt: number,
+ skipEntries = 0,
+ budget = Number.POSITIVE_INFINITY,
+): Promise<{ found: Array<{ skill: SkillInfo; priority: number }>; stop?: "maxEntriesPerRoot" | "deadline" | "budget"; consumed: number }> {
+ const found: Array<{ skill: SkillInfo; priority: number }> = [];
+ let consumed = skipEntries;
let seen = 0;
- const handle = await fs.opendir(root).catch(() => null);
- if (!handle) return { names, hitCap, entriesVisited };
+ let processed = 0;
+ let stop: "maxEntriesPerRoot" | "deadline" | "budget" | undefined;
+ const handle = await fs.opendir(spec.path).catch(() => null);
+ if (!handle) return { found, consumed };
try {
for await (const entry of handle) {
seen += 1;
if (seen <= skipEntries) continue;
- // EVERY dirent costs budget, accepted or not. Counting only directories meant a
- // root holding a million regular files still required a million iterations before
- // the advertised entry cap was reached.
- if (entriesVisited - skipEntries >= SKILL_SCAN_LIMITS.maxEntriesPerRoot || Date.now() > deadlineAt) { hitCap = true; break; }
- entriesVisited += 1;
- if (!entry.isDirectory() && !entry.isSymbolicLink()) continue;
- names.push(entry.name);
- }
- } catch {
- // A root that vanishes mid-walk yields what we already have.
- }
- return { names: names.sort((a, b) => a.localeCompare(b)), hitCap, entriesVisited };
-}
-
-export async function scanRoot(spec: RootSpec, deadlineAt: number, skipEntries = 0): Promise<{ found: Array<{ skill: SkillInfo; priority: number }>; hitCap: boolean; entriesVisited: number }> {
- const { names, hitCap: dirCap, entriesVisited } = await skillDirNames(spec.path, deadlineAt, skipEntries);
- let hitCap = dirCap;
- const found: Array<{ skill: SkillInfo; priority: number }> = [];
- for (const name of names) {
- // The deadline is enforced THROUGH the per-skill stat/read work too, not only in
- // the dirent loop, which is where a slow filesystem actually spends its time.
- if (Date.now() > deadlineAt) { hitCap = true; break; }
- const dir = path.join(spec.path, name);
- if (!(await fs.stat(dir).catch(() => null))?.isDirectory()) continue;
- const file = path.join(dir, SKILL_FILE);
- const md = await readSkillFile(file);
- if (md === null) continue;
+ // EVERY dirent costs budget, accepted or not — and every stop check happens before
+ // the entry is processed, so `consumed` never runs ahead of the work.
+ if (processed >= SKILL_SCAN_LIMITS.maxEntriesPerRoot) { stop = "maxEntriesPerRoot"; break; }
+ if (Date.now() > deadlineAt) { stop = "deadline"; break; }
+ if (found.length >= budget) { stop = "budget"; break; }
+ processed += 1;
- let trust = spec.trust;
- let provenance: SkillInfo["provenance"];
- if (spec.verifyClawHub) {
- const verified = await verifiedBundledSkill(dir, md);
- trust = verified.trust;
- provenance = verified.provenance;
- } else if (spec.project) {
- // Path alone grants nothing: containment, ownership and SKILL.md shape decide.
- trust = await projectSkillTrust(dir, spec.project.path);
+ if (entry.isDirectory() || entry.isSymbolicLink()) {
+ const dir = path.join(spec.path, entry.name);
+ if ((await fs.stat(dir).catch(() => null))?.isDirectory()) {
+ const md = await readSkillFile(path.join(dir, SKILL_FILE));
+ if (md !== null) {
+ let trust = spec.trust;
+ let provenance: SkillInfo["provenance"];
+ if (spec.verifyClawHub) {
+ const verified = await verifiedBundledSkill(dir, md);
+ trust = verified.trust;
+ provenance = verified.provenance;
+ } else if (spec.project) {
+ // Path alone grants nothing: containment, ownership and SKILL.md shape decide.
+ trust = await projectSkillTrust(dir, spec.project.path);
+ }
+ found.push({
+ priority: spec.priority,
+ skill: {
+ id: spec.project ? `${spec.project.id}/${entry.name}` : entry.name,
+ name: entry.name,
+ path: path.join(dir, SKILL_FILE),
+ description: skillDescription(md),
+ source: spec.source,
+ trust,
+ ...(spec.project ? { project: spec.project } : {}),
+ ...(provenance ? { provenance } : {}),
+ },
+ });
+ }
+ }
+ }
+ consumed = seen; // ONLY now — the entry is fully handled.
}
-
- found.push({
- priority: spec.priority,
- skill: {
- id: spec.project ? `${spec.project.id}/${name}` : name,
- name,
- path: file,
- description: skillDescription(md),
- source: spec.source,
- trust,
- ...(spec.project ? { project: spec.project } : {}),
- ...(provenance ? { provenance } : {}),
- },
- });
+ } catch {
+ // A root that vanishes mid-walk yields what we already fully processed.
}
- return { found, hitCap, entriesVisited };
+ return { found, stop, consumed };
}
-
diff --git a/lib/skills/catalog-types.ts b/lib/skills/catalog-types.ts
index 97ce5610..044f01d7 100644
--- a/lib/skills/catalog-types.ts
+++ b/lib/skills/catalog-types.ts
@@ -50,11 +50,13 @@ export type SkillInfo = {
* walk's cursor: name-ordered resume would require visiting every dirent, which is the
* unbounded walk the entry cap exists to prevent. */
export type SkillScanCursor = {
- roots: Array<{ root: string; entriesConsumed: number }>;
- /** Roots fully covered by the previous page; skipped on resume. */
- skipRoots: string[];
- /** Projects already consumed, so a maxProjects/maxProjectSkills cap can be continued. */
+ /** Roots that finished CLEANLY; skipped on resume. */
+ doneRoots: string[];
+ /** Projects every one of whose roots finished cleanly. A partially consumed project is
+ * deliberately NOT counted here — it is re-listed and resumed at `resume`. */
projectOffset: number;
+ /** The exact dirent position inside the one root that was interrupted. */
+ resume?: { root: string; entriesConsumed: number };
};
/** What a catalog build could NOT cover. Mirrors lib/host's ScanReport so the two
diff --git a/lib/skills/catalog.ts b/lib/skills/catalog.ts
index 76a81fbf..bc37690a 100644
--- a/lib/skills/catalog.ts
+++ b/lib/skills/catalog.ts
@@ -1,6 +1,7 @@
import os from "os";
import path from "path";
import { scanRoot, type RootSpec } from "./catalog-scan";
+import { decodeSkillCursor, encodeSkillCursor } from "./catalog-cursor";
import {
SKILL_FILE, SKILL_SCAN_LIMITS, skillIsExecutableByDefault,
type ProjectRef, type SkillInfo, type SkillScanCursor, type SkillScanReport, type SkillSource, type SkillTrust,
@@ -9,6 +10,7 @@ import { discoveredProjects, projectSkillRoots } from "./project-skills";
export { SKILL_FILE, SKILL_SCAN_LIMITS, skillIsExecutableByDefault };
export { readSkillFile, skillDescription } from "./catalog-scan";
+export { decodeSkillCursor, encodeSkillCursor } from "./catalog-cursor";
export type { ProjectRef, SkillInfo, SkillScanCursor, SkillScanReport, SkillSource, SkillTrust };
type CatalogOptions = {
@@ -21,25 +23,6 @@ type CatalogOptions = {
cursor?: string;
};
-export function encodeSkillCursor(cursor: SkillScanCursor): string {
- return Buffer.from(JSON.stringify(cursor), "utf8").toString("base64url");
-}
-
-export function decodeSkillCursor(raw: string | undefined): SkillScanCursor | undefined {
- if (!raw) return undefined;
- try {
- const parsed = JSON.parse(Buffer.from(raw, "base64url").toString("utf8")) as SkillScanCursor;
- if (!parsed || typeof parsed !== "object") return undefined;
- return {
- roots: Array.isArray(parsed.roots) ? parsed.roots.filter((r) => typeof r?.root === "string") : [],
- skipRoots: Array.isArray(parsed.skipRoots) ? parsed.skipRoots.filter((r) => typeof r === "string") : [],
- projectOffset: Number.isFinite(parsed.projectOffset) ? Math.max(0, Math.floor(parsed.projectOffset)) : 0,
- };
- } catch {
- return undefined;
- }
-}
-
/**
* Root priority is a security boundary, not just display order.
*
@@ -75,9 +58,6 @@ export async function catalogSkillsDetailed(options: CatalogOptions = {}): Promi
const homeDir = options.homeDir ?? os.homedir();
const deadlineAt = Date.now() + SKILL_SCAN_LIMITS.maxScanMs;
const cursor = decodeSkillCursor(options.cursor);
- const resumeRoots = new Map((cursor?.roots ?? []).map((r) => [r.root, r.entriesConsumed]));
- const skipRoots = new Set(cursor?.skipRoots ?? []);
- const projectOffset = cursor?.projectOffset ?? 0;
const reasons: string[] = [];
let projects = options.projects;
@@ -86,78 +66,90 @@ export async function catalogSkillsDetailed(options: CatalogOptions = {}): Promi
projects = discovered.projects;
reasons.push(...discovered.truncationReasons);
}
+ // `projectOffset` counts projects FULLY consumed by earlier pages; the partially
+ // consumed one is re-listed and resumed at its exact dirent position.
+ const projectOffset = Math.min(cursor?.projectOffset ?? 0, projects.length);
const remaining = projects.slice(projectOffset);
const { roots: projectRoots, truncated: projectsCapped } = await projectSkillRoots(remaining);
if (projectsCapped) reasons.push("maxProjects");
+ const globalSpecs = skillRoots(appDir, homeDir);
const specs: RootSpec[] = [
- ...skillRoots(appDir, homeDir),
+ ...globalSpecs,
...projectRoots.map((root): RootSpec => ({
path: root.path, source: "project", trust: "untrusted", priority: root.priority, project: root.project,
})),
];
+ const doneRoots = new Set(cursor?.doneRoots ?? []);
const chosen = new Map<string, { skill: SkillInfo; priority: number }>();
- const cursors: SkillScanCursor["roots"] = [];
+ const completedRoots: string[] = [];
const pendingRoots: string[] = [];
- const doneRoots: string[] = [];
- const consumedProjects = new Set<string>();
+ const consumedProjects: string[] = [];
let projectSkills = 0;
+ let resume: SkillScanCursor["resume"];
let stoppedAt = -1;
for (const [index, spec] of specs.entries()) {
- if (skipRoots.has(spec.path)) { doneRoots.push(spec.path); continue; }
- if (Date.now() > deadlineAt) { reasons.push("deadline"); stoppedAt = index; break; }
- if (spec.project && projectSkills >= SKILL_SCAN_LIMITS.maxProjectSkills) {
- reasons.push("maxProjectSkills");
+ if (doneRoots.has(spec.path)) { completedRoots.push(spec.path); continue; }
+ const skipEntries = cursor?.resume?.root === spec.path ? cursor.resume.entriesConsumed : 0;
+ const budget = spec.project ? SKILL_SCAN_LIMITS.maxProjectSkills - projectSkills : Number.POSITIVE_INFINITY;
+
+ if (Date.now() > deadlineAt) {
+ reasons.push("deadline");
+ resume = { root: spec.path, entriesConsumed: skipEntries };
stoppedAt = index;
break;
}
- const { found, hitCap, entriesVisited } = await scanRoot(spec, deadlineAt, resumeRoots.get(spec.path) ?? 0);
- if (hitCap) {
- reasons.push(`maxEntriesPerRoot:${spec.path}`);
- cursors.push({ root: spec.path, entriesConsumed: entriesVisited });
- } else {
- doneRoots.push(spec.path);
+ if (spec.project && budget <= 0) {
+ reasons.push("maxProjectSkills");
+ resume = { root: spec.path, entriesConsumed: skipEntries };
+ stoppedAt = index;
+ break;
}
- let overflowed = false;
+
+ const { found, stop, consumed } = await scanRoot(spec, deadlineAt, skipEntries, budget);
for (const candidate of found) {
- // The overall cap is checked INSIDE the loop. Checking it only before scanRoot
- // let one root carry the total from just under 300 to nearly 500.
- if (spec.project && projectSkills >= SKILL_SCAN_LIMITS.maxProjectSkills) {
- reasons.push("maxProjectSkills");
- overflowed = true;
- break;
- }
if (spec.project) projectSkills += 1;
const current = chosen.get(candidate.skill.id);
if (!current || candidate.priority > current.priority) chosen.set(candidate.skill.id, candidate);
}
- if (spec.project) consumedProjects.add(spec.project.id);
- if (overflowed) { stoppedAt = index; break; }
+
+ if (stop) {
+ reasons.push(stop === "budget" ? "maxProjectSkills" : stop === "deadline" ? "deadline" : `maxEntriesPerRoot:${spec.path}`);
+ // The exact dirent this root stopped on — never "the whole root is done".
+ resume = { root: spec.path, entriesConsumed: consumed };
+ stoppedAt = index;
+ break;
+ }
+ completedRoots.push(spec.path);
+ // A project counts as consumed only once EVERY one of its roots finished cleanly.
+ if (spec.project && specs.slice(index + 1).every((later) => later.project?.id !== spec.project!.id)) {
+ consumedProjects.push(spec.project.id);
+ }
}
if (stoppedAt >= 0) pendingRoots.push(...specs.slice(stoppedAt).map((r) => r.path));
const truncationReasons = [...new Set(reasons)];
- const pendingProjects = Math.max(0, projects.length - projectOffset - consumedProjects.size);
+ const pendingProjects = Math.max(0, projects.length - projectOffset - consumedProjects.length);
return {
skills: [...chosen.values()].map(({ skill }) => skill).sort((a, b) => a.id.localeCompare(b.id)),
scan: {
truncated: truncationReasons.length > 0,
truncationReasons,
scannedRoots: specs.length,
- scannedProjects: consumedProjects.size,
+ scannedProjects: consumedProjects.length,
...(truncationReasons.length ? {
continuation: {
pendingRoots: [...new Set(pendingRoots)],
- cursors,
+ cursors: resume ? [resume] : [],
pendingProjects,
cursorSemantics: "readdir-position" as const,
- note: "Pass `cursor` back to resume. Positions are readdir order and are valid while the directories are unchanged.",
+ note: "Pass `cursor` back to resume at the exact dirent the scan stopped on. Positions are readdir stream order and are valid while the directories are unchanged.",
cursor: encodeSkillCursor({
- roots: cursors,
- skipRoots: [...new Set(doneRoots)],
- projectOffset: projectOffset + consumedProjects.size,
+ doneRoots: completedRoots,
+ projectOffset: projectOffset + consumedProjects.length,
+ ...(resume ? { resume } : {}),
}),
},
} : {}),
diff --git a/skills/README.md b/skills/README.md
index 23444409..05514f3f 100644
--- a/skills/README.md
+++ b/skills/README.md
@@ -57,8 +57,11 @@ checks the cap against `fstat` before any bytes move.
Every discovery response carries a **scan report**. `truncated:false` means "this is all
of it"; hitting a cap sets `truncated:true`, names the reason, and includes
-`continuation` — pending roots, per-root cursors and an opaque `cursor` to pass back and
-resume. Do not conclude a skill is absent from a truncated scan.
+`continuation` — pending roots, the exact interrupted position and an opaque `cursor` to
+pass back. Resumption is lossless: each dirent is validated as it streams and the position
+advances only after that entry is fully handled, so a cap or deadline re-reads the entry
+it stopped on instead of skipping it. A partially consumed project is resumed, never
+marked done. Do not conclude a skill is absent from a truncated scan.
## Contract