Skip to content

armgit / mso

Public Imported from GitHub
0

Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.

fix(mcp): remove session lookup contention

Rahman1aa4738c3b2f
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index a24fd81a..30f8a8b1 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -21,6 +21,7 @@ this is the *what*, and it is what Settings → About shows as “What's new”.
**Fixed**
+- `mcp` remove session lookup contention
- `agent` wake prompt after child tty sigint
- `cli` clarify version and update labels
- `agent` stabilize terminal banner navigation
diff --git a/docs/COGNITIVE-RUNTIME.md b/docs/COGNITIVE-RUNTIME.md
index 12e35420..cbfddfe8 100644
--- a/docs/COGNITIVE-RUNTIME.md
+++ b/docs/COGNITIVE-RUNTIME.md
@@ -39,6 +39,12 @@ session owner derived from `principal + durable session id`; recipes use the sta
second conversation can learn from a successful route without being able to status/finish/cancel the first
conversation's live run.
+Conversation correlation is indexed durably under the private agent-session root. MSO backfills legacy P0
+conversation hashes once at Node startup, then resolves normal MCP calls through an O(1) owner+conversation
+reference. Creation is locked per conversation and record mutation is locked per durable session; unrelated
+chats no longer queue behind one global session-store lock. The index stores only the already-hashed
+conversation key and internal MSO session id, never the raw ChatGPT conversation id.
+
## Context lifecycle
MSO keeps **durable session context** separate from the **model-active context** sent on one turn.
diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md
index f0883c27..44189493 100644
--- a/docs/PROGRESS.md
+++ b/docs/PROGRESS.md
@@ -2,6 +2,12 @@
Running log of what shipped each phase. Newest at top.
+## 2026-09-02 — MCP session lookup contention hardening — SHIPPED
+
+The first live Cognitive Runtime P0 deployment exposed a scalability problem that unit-size fixtures did not: the host already had **2,927 durable agent session files**, while every conversation-bound `/mcp` tool call still took one global session-store lock and scanned the entire directory to find its `openai/session` hash. Parallel ChatGPT conversations could therefore exhaust the 3-second security-store lock budget and return `security store is busy; retry the operation` even though each conversation was correctly isolated.
+
+Conversation correlation now has a private durable index keyed only by the stable principal hash + hashed conversation id. Startup backfills pre-index P0 sessions once before requests are accepted; normal conversation lookup is then O(1). New-session creation uses a per-conversation lock so duplicate parallel calls collapse to one record, while history/title/event mutations use a per-session lock so unrelated chats never serialize behind a global lock. The raw OpenAI conversation id is still never persisted. Regression coverage runs 32 simultaneous lookups for one conversation, 24 unrelated conversations in parallel, legacy-index backfill, cross-session workflow isolation, compaction/archive policy, and MCP route correlation.
+
## 2026-09-02 — MCP-first Cognitive Runtime P0 — SHIPPED
MSO CLI **1.6.0** moves agent quality into the provider-neutral harness instead of relying on one model vendor. ChatGPT/MCP identity is now layered as **authenticated client → durable conversation session → workflow/job**: `_meta["openai/session"]` is hashed with the stable client principal and the raw opaque conversation id is never persisted, transport-session rotation does not split a ChatGPT conversation, active workflows/jobs are session-scoped, and verified learned recipes remain reusable at the stable client-principal boundary. This lets parallel chats benefit from prior successful procedures without being able to status, finish, cancel, or attach to each other's live work.
diff --git a/instrumentation.ts b/instrumentation.ts
index 4a8a1490..9350da44 100644
--- a/instrumentation.ts
+++ b/instrumentation.ts
@@ -6,12 +6,17 @@ import type { Instrumentation } from "next";
export async function register() {
if (process.env.NEXT_RUNTIME !== "nodejs") return;
- // Enforce private session-archive retention at boot even when no new session
- // reaches the compaction threshold for weeks. Best-effort: retention cleanup
- // must never make the application fail to start.
- void import("./lib/agent/session-archive")
- .then(({ pruneAgentSessionArchives }) => pruneAgentSessionArchives())
- .catch(() => undefined);
+ // Warm durable agent-session maintenance before accepting requests. Backfilling
+ // the conversation index once at boot keeps MCP lookup O(1) even on hosts with
+ // thousands of old CLI sessions; archive retention remains best-effort.
+ await Promise.all([
+ import("./lib/agent/session-files")
+ .then(({ backfillConversationIndex }) => backfillConversationIndex())
+ .catch(() => undefined),
+ import("./lib/agent/session-archive")
+ .then(({ pruneAgentSessionArchives }) => pruneAgentSessionArchives())
+ .catch(() => undefined),
+ ]);
// systemd liveness: when the unit sets WatchdogSec, systemd exposes
// NOTIFY_SOCKET + WATCHDOG_USEC. Node has no native AF_UNIX SOCK_DGRAM
// (dgram is UDP-only), so we ping via the `systemd-notify` binary — which
diff --git a/lib/agent/session-files.ts b/lib/agent/session-files.ts
index 885b7574..45d8343d 100644
--- a/lib/agent/session-files.ts
+++ b/lib/agent/session-files.ts
@@ -6,8 +6,11 @@ import { compactThresholdTokens, inferredTitleSource, MAX_EVENTS, MAX_HISTORY, s
import type { AgentSession } from "./session-types";
const ROOT = path.resolve((process.env.OS_AGENT_SESSIONS_DIR || path.join(os.homedir(), ".mso", "agent-sessions")).replace(/^~(?=$|\/)/, os.homedir()));
-export const SESSION_LOCK_TARGET = path.join(ROOT, ".sessions");
+const LOCK_ROOT = path.join(ROOT, ".locks");
+const CONVERSATION_ROOT = path.join(ROOT, ".conversation-index");
+const CONVERSATION_READY = path.join(CONVERSATION_ROOT, ".ready-v1");
export const SESSION_ID = /^\d{8}_\d{6}_[a-f0-9]{8}$/;
+const HASH64 = /^[a-f0-9]{64}$/;
const MAX_SESSION_BYTES = 16 * 1024 * 1024;
export function principalHash(principal: string): string {
@@ -20,6 +23,76 @@ export function newAgentSessionId(now = new Date()): string {
return `${iso.slice(0, 10).replaceAll("-", "")}_${iso.slice(11, 19).replaceAll(":", "")}_${randomBytes(4).toString("hex")}`;
}
+export function sessionLockTarget(id: string): string {
+ if (!SESSION_ID.test(id)) throw new Error("invalid agent session id");
+ return path.join(LOCK_ROOT, "sessions", id);
+}
+
+function conversationRefFile(ownerHash: string, conversationHash: string): string {
+ if (!HASH64.test(ownerHash) || !HASH64.test(conversationHash)) throw new Error("invalid agent conversation index key");
+ return path.join(CONVERSATION_ROOT, ownerHash.slice(0, 2), ownerHash, `${conversationHash}.ref`);
+}
+
+export function conversationLockTarget(ownerHash: string, conversationHash: string): string {
+ return conversationRefFile(ownerHash, conversationHash);
+}
+
+export async function readConversationSession(ownerHash: string, conversationHash: string): Promise<AgentSession | null> {
+ const file = conversationRefFile(ownerHash, conversationHash);
+ let handle: Awaited<ReturnType<typeof fs.open>> | null = null;
+ try {
+ handle = await fs.open(file, fsConstants.O_RDONLY | fsConstants.O_NOFOLLOW);
+ const stat = await handle.stat();
+ if (!stat.isFile() || stat.size <= 0 || stat.size > 128) throw new Error("agent conversation index has an invalid shape");
+ if ((stat.mode & 0o077) !== 0) throw new Error("agent conversation index permissions are too broad; expected 0600");
+ if (typeof process.getuid === "function" && stat.uid !== process.getuid()) throw new Error("agent conversation index is not owned by the MSO user");
+ const id = (await handle.readFile("utf8")).trim();
+ if (!SESSION_ID.test(id)) throw new Error("agent conversation index contains an invalid session id");
+ const session = await readSessionFile(id);
+ if (!session || session.principalHash !== ownerHash || session.source !== "mcp" || session.conversationHash !== conversationHash) return null;
+ return session;
+ } catch (error) {
+ if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
+ throw error;
+ } finally { await handle?.close().catch(() => undefined); }
+}
+
+export async function writeConversationRef(ownerHash: string, conversationHash: string, id: string): Promise<void> {
+ if (!SESSION_ID.test(id)) throw new Error("invalid agent session id");
+ const file = conversationRefFile(ownerHash, conversationHash);
+ await fs.mkdir(path.dirname(file), { recursive: true, mode: 0o700 });
+ await fs.chmod(CONVERSATION_ROOT, 0o700).catch(() => undefined);
+ await fs.chmod(path.dirname(file), 0o700).catch(() => undefined);
+ const tmp = `${file}.${randomUUID()}.tmp`;
+ await fs.writeFile(tmp, `${id}\n`, { encoding: "utf8", mode: 0o600, flag: "wx" });
+ await fs.chmod(tmp, 0o600);
+ await fs.rename(tmp, file);
+ await fs.chmod(file, 0o600);
+}
+
+
+export async function conversationIndexReady(): Promise<boolean> {
+ let handle: Awaited<ReturnType<typeof fs.open>> | null = null;
+ try {
+ handle = await fs.open(CONVERSATION_READY, fsConstants.O_RDONLY | fsConstants.O_NOFOLLOW);
+ const stat = await handle.stat();
+ return stat.isFile() && (stat.mode & 0o077) === 0 &&
+ (typeof process.getuid !== "function" || stat.uid === process.getuid());
+ } catch {
+ return false;
+ } finally { await handle?.close().catch(() => undefined); }
+}
+
+async function markConversationIndexReady(): Promise<void> {
+ await fs.mkdir(CONVERSATION_ROOT, { recursive: true, mode: 0o700 });
+ await fs.chmod(CONVERSATION_ROOT, 0o700).catch(() => undefined);
+ const tmp = `${CONVERSATION_READY}.${randomUUID()}.tmp`;
+ await fs.writeFile(tmp, `${new Date().toISOString()}\n`, { encoding: "utf8", mode: 0o600, flag: "wx" });
+ await fs.chmod(tmp, 0o600);
+ await fs.rename(tmp, CONVERSATION_READY);
+ await fs.chmod(CONVERSATION_READY, 0o600);
+}
+
function fileFor(id: string): string {
if (!SESSION_ID.test(id)) throw new Error("invalid agent session id");
const candidate = path.resolve(ROOT, `${id}.json`);
@@ -90,3 +163,25 @@ export async function listSessionRecords(): Promise<AgentSession[]> {
}
return out;
}
+
+
+export async function backfillConversationIndex(): Promise<{ indexed: number; conversations: number }> {
+ const rows = (await listSessionRecords())
+ .filter((row) => row.source === "mcp" && HASH64.test(row.conversationHash ?? ""))
+ .sort((a, b) => b.updatedAt.localeCompare(a.updatedAt));
+ const latest = new Map<string, AgentSession>();
+ for (const row of rows) {
+ const key = `${row.principalHash}:${row.conversationHash}`;
+ if (!latest.has(key)) latest.set(key, row);
+ }
+ let indexed = 0;
+ for (const row of latest.values()) {
+ const hash = row.conversationHash!;
+ const current = await readConversationSession(row.principalHash, hash).catch(() => null);
+ if (current?.id === row.id) continue;
+ await writeConversationRef(row.principalHash, hash, row.id);
+ indexed += 1;
+ }
+ await markConversationIndexReady();
+ return { indexed, conversations: latest.size };
+}
diff --git a/lib/agent/session-store.test.ts b/lib/agent/session-store.test.ts
index 7f43b19a..6a3b1873 100644
--- a/lib/agent/session-store.test.ts
+++ b/lib/agent/session-store.test.ts
@@ -67,4 +67,42 @@ describe("durable agent session context policy", () => {
expect(current?.title).toBe("Manual name");
expect(current?.titleSource).toBe("manual");
});
+
+ it("deduplicates parallel calls for one conversation without a global session lock", async () => {
+ const principal = "principal:parallel-same";
+ const hash = "a".repeat(64);
+ const rows = await Promise.all(Array.from({ length: 32 }, () =>
+ store.findOrCreateAgentSessionForConversation(principal, hash, "Parallel conversation"),
+ ));
+ expect(new Set(rows.map((row) => row.id)).size).toBe(1);
+ const listed = await store.listAgentSessions(principal, 100);
+ expect(listed).toHaveLength(1);
+ expect(listed[0].source).toBe("mcp");
+ });
+
+ it("lets unrelated conversations create sessions concurrently", async () => {
+ const principal = "principal:parallel-many";
+ const hashes = Array.from({ length: 24 }, (_, i) => i.toString(16).padStart(64, "0"));
+ const rows = await Promise.all(hashes.map((hash) =>
+ store.findOrCreateAgentSessionForConversation(principal, hash, `Conversation ${hash.slice(-2)}`),
+ ));
+ expect(new Set(rows.map((row) => row.id)).size).toBe(hashes.length);
+ expect(await store.listAgentSessions(principal, 100)).toHaveLength(hashes.length);
+ });
+
+ it("backfills legacy conversation refs once and marks the hot-path index ready", async () => {
+ const principal = "principal:legacy-index";
+ const hash = "f".repeat(64);
+ const legacy = await store.createAgentSession(principal, "mcp", {
+ title: "Legacy indexed conversation", conversationHash: hash,
+ });
+ const files = await import("./session-files");
+ expect(await files.conversationIndexReady()).toBe(false);
+ const backfill = await files.backfillConversationIndex();
+ expect(backfill.conversations).toBeGreaterThan(0);
+ expect(await files.conversationIndexReady()).toBe(true);
+ const resolved = await store.findOrCreateAgentSessionForConversation(principal, hash, "Should not duplicate");
+ expect(resolved.id).toBe(legacy.id);
+ });
+
});
diff --git a/lib/agent/session-store.ts b/lib/agent/session-store.ts
index 27bdc969..f2d803a8 100644
--- a/lib/agent/session-store.ts
+++ b/lib/agent/session-store.ts
@@ -2,7 +2,10 @@ import { withSecurityStoreLock } from "@/lib/security-store-lock";
import { redactText } from "@/lib/security/redact-text";
import { snapshotAgentMemory, type AgentMemorySnapshot } from "./memory-store";
import { archiveAgentSession, pruneAgentSessionArchives } from "./session-archive";
-import { listSessionRecords, newAgentSessionId, principalHash, readSessionFile, SESSION_LOCK_TARGET, writeSessionFile } from "./session-files";
+import {
+ conversationIndexReady, conversationLockTarget, listSessionRecords, newAgentSessionId, principalHash,
+ readConversationSession, readSessionFile, sessionLockTarget, writeConversationRef, writeSessionFile,
+} from "./session-files";
import { autoSessionTitle, compactSessionContext, compactThresholdTokens, estimateTokens, MAX_EVENTS, MAX_HISTORY, safeTitle, sessionContextTokens } from "./session-policy";
import type { AgentSession, AgentSessionEvent, AgentSessionResumePacket, AgentSessionSource, AgentSessionSummary, AgentSessionTitleSource } from "./session-types";
export type { AgentSession, AgentSessionEvent, AgentSessionResumePacket, AgentSessionSource, AgentSessionSummary } from "./session-types";
@@ -65,8 +68,8 @@ async function requireOwned(principal: string, id: string): Promise<AgentSession
}
export async function createAgentSession(principal: string, source: AgentSessionSource, options: CreateOptions = {}): Promise<AgentSession> {
- return withSecurityStoreLock(SESSION_LOCK_TARGET, async () => {
- const record = await buildRecord(principal, source, options);
+ const record = await buildRecord(principal, source, options);
+ return withSecurityStoreLock(sessionLockTarget(record.id), async () => {
if (await readSessionFile(record.id)) throw new Error("agent session id already exists");
await writeSessionFile(record); return record;
});
@@ -74,12 +77,28 @@ export async function createAgentSession(principal: string, source: AgentSession
export async function findOrCreateAgentSessionForConversation(principal: string, hash: string, title = "ChatGPT session"): Promise<AgentSession> {
if (!/^[a-f0-9]{64}$/.test(hash)) throw new Error("invalid conversation hash");
- return withSecurityStoreLock(SESSION_LOCK_TARGET, async () => {
- const owner = principalHash(principal);
- const existing = (await listSessionRecords()).find((row) => row.principalHash === owner && row.source === "mcp" && row.conversationHash === hash);
- if (existing) return existing;
+ const owner = principalHash(principal);
+ const indexed = await readConversationSession(owner, hash);
+ if (indexed) return indexed;
+ return withSecurityStoreLock(conversationLockTarget(owner, hash), async () => {
+ const current = await readConversationSession(owner, hash);
+ if (current) return current;
+ // Normal Next startup backfills the durable index before accepting requests,
+ // keeping later conversation lookups O(1). The scan is only a compatibility
+ // fallback for direct library use that bypassed instrumentation startup.
+ if (!(await conversationIndexReady())) {
+ const legacy = (await listSessionRecords()).find((row) =>
+ row.principalHash === owner && row.source === "mcp" && row.conversationHash === hash,
+ );
+ if (legacy) {
+ await writeConversationRef(owner, hash, legacy.id);
+ return legacy;
+ }
+ }
const record = await buildRecord(principal, "mcp", { title, titleSource: "default", conversationHash: hash });
- await writeSessionFile(record); return record;
+ await writeSessionFile(record);
+ await writeConversationRef(owner, hash, record.id);
+ return record;
});
}
@@ -100,7 +119,7 @@ export async function listAgentSessionsOwner(limit = 100): Promise<AgentSessionS
}
export async function updateAgentSessionHistory(principal: string, id: string, history: unknown[], title?: string, titleSource: AgentSessionTitleSource = "auto"): Promise<AgentSession> {
- return withSecurityStoreLock(SESSION_LOCK_TARGET, async () => {
+ return withSecurityStoreLock(sessionLockTarget(id), async () => {
let record = await requireOwned(principal, id);
const oldTokens = estimateTokens(record.history), newHistory = Array.isArray(history) ? history.slice(-MAX_HISTORY) : [];
record.history = newHistory;
@@ -114,7 +133,7 @@ export async function updateAgentSessionHistory(principal: string, id: string, h
}
export async function maybeAutoTitleAgentSession(principal: string, id: string, hint: string): Promise<AgentSession | null> {
- return withSecurityStoreLock(SESSION_LOCK_TARGET, async () => {
+ return withSecurityStoreLock(sessionLockTarget(id), async () => {
const record = await requireOwned(principal, id);
if (record.titleSource === "manual" || record.titleSource === "auto") return record;
record.title = autoSessionTitle(hint); record.titleSource = "auto"; record.updatedAt = new Date().toISOString();
@@ -123,11 +142,15 @@ export async function maybeAutoTitleAgentSession(principal: string, id: string,
}
export async function renameAgentSession(principal: string, id: string, title: string): Promise<AgentSession> {
- return updateAgentSessionHistory(principal, id, (await requireOwned(principal, id)).history, title, "manual");
+ return withSecurityStoreLock(sessionLockTarget(id), async () => {
+ const record = await requireOwned(principal, id);
+ record.title = safeTitle(title); record.titleSource = "manual"; record.updatedAt = new Date().toISOString();
+ await writeSessionFile(record); return record;
+ });
}
export async function appendAgentSessionEvent(principal: string, id: string, event: Omit<AgentSessionEvent, "at"> & { at?: string }): Promise<void> {
- await withSecurityStoreLock(SESSION_LOCK_TARGET, async () => {
+ await withSecurityStoreLock(sessionLockTarget(id), async () => {
let record = await requireOwned(principal, id); const at = event.at ?? new Date().toISOString();
const row: AgentSessionEvent = { at, kind: event.kind,
...(event.tool ? { tool: event.tool.slice(0, 120) } : {}), ...(event.state ? { state: event.state.slice(0, 40) } : {}),
diff --git a/scripts/mso-gateway.test.ts b/scripts/mso-gateway.test.ts
index b00c3110..cf6276de 100644
--- a/scripts/mso-gateway.test.ts
+++ b/scripts/mso-gateway.test.ts
@@ -192,7 +192,10 @@ printf '%s\n' "$@" > "${capture}"
MSO_GATEWAY_STATE_DIR: f.state, MSO_GATEWAY_CURL: f.curl, MSO_GATEWAY_CLOUDFLARED: f.cloudflared },
});
expect(out).toContain("opened http://127.0.0.1:4555");
- for (let i = 0; i < 50 && !fs.existsSync(capture); i++) await new Promise((r) => setTimeout(r, 10));
+ // The browser launcher is deliberately detached. Under full coverage the child
+ // can be scheduled later than the old 500 ms fixture window, so wait boundedly
+ // for its one small argv file instead of turning host load into a flaky failure.
+ for (let i = 0; i < 200 && !fs.existsSync(capture); i++) await new Promise((r) => setTimeout(r, 10));
const argv = fs.readFileSync(capture, "utf8").trim().split(/\n/);
expect(argv).toContain("Start-Process $args[0]");
expect(argv.at(-1)).toBe("http://127.0.0.1:4555");