Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(mcp-host): address PR #1028 review findings
Alfredo Lopez6cce33282820
diff --git a/docs/operations/gfs-large-file-host-downloads.md b/docs/operations/gfs-large-file-host-downloads.mdindex dec77ab98..fe54e99bd 100644--- a/docs/operations/gfs-large-file-host-downloads.md+++ b/docs/operations/gfs-large-file-host-downloads.md@@ -52,7 +52,7 @@ without discarding the path, checksum, version or retained original.| `MCP_HOST_GFS_MAX_FILE_BYTES` | `16777216` | `209715200` | Largest GFS source admitted by MCP Host. || `MCP_HOST_GFS_DOWNLOAD_STORAGE_BYTES` | `1073741824` | deployment-approved | Aggregate retained download budget. || `MCP_HOST_GFS_CALLER_DOWNLOAD_STORAGE_BYTES` | `268435456` | aggregate budget | Per-caller retained download budget. |-| `MCP_HOST_GFS_CALLER_DOWNLOAD_MAX_FILES` | `8` | `64` | Retained completed files per caller. |+| `MCP_HOST_GFS_CALLER_DOWNLOAD_MAX_FILES` | `8` | `64` | Retained files per caller, in any state. || `MCP_HOST_GFS_CALLER_DOWNLOAD_CONCURRENCY` | `1` | `2` | Simultaneous active transfers per caller. || `MCP_HOST_GFS_DOWNLOAD_TTL_HOURS` | `168` | `8760` | Retention period for a completed copy. |@@ -105,9 +105,10 @@ retained caller, resource version, size, expiry, private file and checksum. Avalid copy keeps its download ID, path and expiry without another contenttransfer or quota reservation. Preparation and downloads pin receipts to atrusted task owner while its model, pending approval or execution can still use-them. Reuse adds an owner without replacing another task's pin. A terminal task-releases its pin only after its executions settle. A positively absent copy may-release its charge; corrupt or ambiguous copies remain charged until recovery.+them. Reuse adds an owner without replacing another task's pin. A task releases+its pin when its executor reaches a terminal state. A positively absent copy may+release its charge; corrupt or ambiguous copies remain charged (see+[Retention, quotas, and recovery](#retention-quotas-and-recovery)).A replacement needs fresh GFS authorization.After a clean v2 writer restart, pins alone do not disable delivery. Fresh remote authorization and source version, size,@@ -128,9 +129,13 @@ to recover access.- `shell_exec` never calls the GFS download store (#1019). It does not acquirea processing lease, does not depend on `store.isAvailable()`, and keepsrunning while the store is recovery-required. When a Host-owned store exists,- the shell stays bound to the verified caller root and keeps forced live- approval; before every command it re-verifies that the caller root is still- canonical. That check is per-user directory scoping, not store state.+ the shell stays bound to the verified caller root; before every command it+ re-verifies that the caller root is still canonical. That check is per-user+ directory scoping, not store state. Live approval of `shell_exec` and+ `clerum__gfs_download` is forced only for channel tasks on a Host with+ approval enabled, unless the approval configuration sets `shell_exec` to+ `false`. Cron, internal and approval-disabled tasks keep their existing+ approval policy.- The store's lifecycle (expiry, pressure eviction, quota, `close()`) countsonly its own transfers and task receipt owners. A running shell does notprotect a copy and does not delay `close()`.@@ -140,9 +145,17 @@ Accepted risks of this decoupling:1. Expiry or eviction can remove a retained copy while a command is using it.A file descriptor that is already open keeps reading; a later open gets`ENOENT`.-2. A command can alter a retained copy. SHA-256 re-verification in- `reusableReceipt` and at startup reconciliation quarantines an altered copy- instead of reusing it.+2. A command can alter a retained copy. Two checks prevent its reuse, and they+ handle it differently:+ - `reusableReceipt` checks the file type, size, `0600` mode and SHA-256+ before reuse. On any mismatch, or when the copy cannot be opened, it sets+ the record to `missing`, deletes its stored checksum and persists the+ ledger. It writes no log and no metric. The record stays charged; the+ hourly expiry sweep removes it after its TTL if the Host is still running.+ - Startup reconciliation re-hashes every `completed` copy and sets an+ altered one to `quarantined`, keeping its checksum. It also sets every+ record that is not `completed`, including a `missing` one, to+ `quarantined`. A quarantined record is never expired or evicted.3. The processing lease was never a security boundary: executors share the HostUID and run without a sandbox. Live approval, `BasicSafety` validation andcredential-slot stripping remain the shell's controls.@@ -181,12 +194,14 @@ shared-UID filesystem races; that remains the Stage 2 boundary.- A new store publishes an atomic schema-1 ledger before accepting transfers. Every existing ledger is parsed, including empty content; invalid record or owner maps, and malformed legacy processing-lease maps, are rejected with `corrupt_store_ledger`.- A pre-existing store directory with a missing ledger is unknown accounting, including an interrupted first initialization before ledger publication. Startup rejects it and preserves retained bytes for operator recovery instead of silently resetting quota. This can require recovery after a bootstrap interruption.- Startup does not reconstruct an accounting directory deleted in its entirety while caller copies remain. Approved shell commands share the Host UID and can destroy this state; whole-store deletion remains outside the recovery guarantee and requires operator inventory of retained copies.-- Admission may evict the oldest verified completed copies that have no active- transfer or task receipt owner. The complete eviction plan- must satisfy both Host and caller quotas before any copy is removed. An- impossible or invalid request cannot evict another caller's files. Active,- pinned, inherited, corrupt and ambiguous entries are never pressure victims.- Quota limits still reject a request when no safe complete plan exists.+- Admission may evict the oldest copies whose record is `completed`, that have+ no active transfer and no task receipt owner, and whose content re-verifies.+ The complete eviction plan must satisfy both Host and caller quotas before any+ copy is removed. A caller-quota denial can only evict that caller's own+ copies. Records in any other state (`transferring`, `missing`,+ `cleanup_failed`, `quarantined`), pinned or active copies, and copies that+ fail verification are never pressure victims. Quota limits still reject a+ request when no safe complete plan exists.- Before pressure effects, current verified filesystem capacity must cover theincoming block-rounded reservation, the 16 MiB safety margin and pendingactive reservations. Apparent file length and allocated blocks do not prove@@ -203,16 +218,28 @@ shared-UID filesystem races; that remains the Stage 2 boundary.initialize, after the ledger is parsed and before reconciliation. No lease inan older ledger can protect an executor of the current boot, and a lease leftby a crashed Host would otherwise fence the store forever. The removal is- persisted first; only after that persist succeeds does the Host log a warning- with the count and increment- `clerum_gfs_legacy_processing_leases_discarded_total`. If the persist fails,- initialize rejects, the ledger keeps the lease, and nothing is logged or- counted; the next initialize retries. Records that protected copies are then- reconciled like any other: an intact completed copy is reusable, an altered- or unfinished one is quarantined.-- Records an earlier boot left quarantined are counted at initialize, logged- and added to `clerum_gfs_inherited_quarantined_records_total`. They stay- charged to quota until operator recovery.+ persisted first. When that persist succeeds, the Host logs a warning with the+ count and increments `clerum_gfs_legacy_processing_leases_discarded_total`.+ When it fails, initialize rejects and the Host logs a `warn` with outcome+ `unknown` and the lease count; the counter is not incremented. The outcome is+ unknown because the persist can fail after the rename, while syncing the+ store directory: the lease-free ledger may already be the one on disk. In+ that case the next initialize finds no leases and reports no discard, so the+ `unknown` warning is the only record of it. The counter therefore counts only+ confirmed discards. Records that protected copies are then reconciled like+ any other: an intact completed copy is reusable, an altered or unfinished one+ is quarantined.+- At the end of initialize, after reconciliation and the final persist, the+ Host counts every `quarantined` record in the ledger, including those this+ boot quarantined, and sets the gauge+ `clerum_gfs_download_store_quarantined_records` to that count. The gauge+ holds the current count and does not accumulate across boots. When the count+ is above zero the Host logs the warning+ `GFS download store holds <n> quarantined record(s) charged to quota`. Quarantined records stay charged to+ quota and are never expired, evicted or reused. Operator recovery can return+ one to `completed` only when its published copy still matches its stored size+ and checksum; it cannot reclassify or remove a quarantined record whose+ content no longer matches its checksum (recovery step 5).- Shutdown stops new admission, drains active transfers where possible, and leaves unproven recovery state protected.- Pending and queued admission rechecks shutdown after asynchronous validation and persistence. Shutdown rechecks active ownership before releasing the writer lease.@@ -234,9 +261,11 @@ owners, groups, symlinks and hard links remain errors.A store initialization failure leaves safe RPC capabilities running and recordsa recovery-required download state. Caller workspace binding remains in force.Managed shell execution keeps the verified caller root and continues; it cannot-fall back to the Host's shared workspace. When initialization succeeds but the-store is not available, the runtime logs an error that operator recovery is-required and managed GFS delivery is disabled.+fall back to the Host's shared workspace. A store whose initialization+succeeded is available. It becomes unavailable only afterwards, when a ledger+persist fails or writer ownership is lost. The periodic cleanup lifecycle then+stops, and the runtime logs the error `GFS download store is no longer+available; periodic cleanup stopped and managed GFS operations are disabled`.Verified contention with an active v2 writer and a valid unchanged ledger is atemporary state. The runtime retries that case up to 60 times at two-second@@ -247,10 +276,12 @@ starts after successful acquisition, including acquisition after retry.Shutdown cancels scheduling, joins outstanding work and closes held writerownership even when delivery is unavailable.-Reuse verifies the checksum of a retained completed copy under store-serialization, and startup reconciliation verifies every retained completed-copy. Their cost grows with retained caller bytes; the default caller budget is-256 MiB. This correctness check has not been benchmarked on every supported PVC.+Reuse verifies the checksum of one retained completed copy under store+serialization; its cost is the size of that one copy.+Startup reconciliation hashes every retained completed copy of every caller, so+its cost is bounded by the Host budget, 1024 MiB by default+(`MCP_HOST_GFS_DOWNLOAD_STORAGE_BYTES`). This correctness check has not been+benchmarked on every supported PVC.A pending task pin can survive a Host restart and protect bytes beyond TTL.If the task never resumes, use the operator inventory and exact terminal-owner@@ -277,8 +308,15 @@ The global `/metrics` endpoint exposes fixed-cardinality instruments:- `clerum_gfs_download_quota_total{scope,reason}`- `clerum_gfs_download_expiry_total{outcome}`- `clerum_gfs_shell_output_limits_total{outcome}`-- `clerum_gfs_legacy_processing_leases_discarded_total` (legacy leases removed at initialize)-- `clerum_gfs_inherited_quarantined_records_total` (records an earlier boot quarantined)+- `clerum_gfs_legacy_processing_leases_discarded_total` (counter of legacy+ processing leases whose discard at initialize was persisted; an unconfirmed+ discard is logged with outcome `unknown` and not counted)+- `clerum_gfs_download_store_quarantined_records` (gauge, no labels: GFS+ download store records currently quarantined, set at each initialize after+ reconcile; they stay charged to quota. Operator recovery reclassifies only+ copies whose content still matches their recorded hash; any other+ quarantined copy stays charged, and the operator recovery tool cannot+ release it.)Labels contain bounded enums only. Caller, resource, download, command, path, correlation IDs, filenames, credentials, and raw output are prohibited as metric labels.@@ -301,6 +339,39 @@ Labels contain bounded enums only. Caller, resource, download, command, path, coIf limits are lowered, existing retained copies remain charged and new admissions are rejected until usage falls below the new policy.+### Records quarantined by builds before the #1022 fix++A build before this fix quarantined every record on any boot whose ledger held+a processing lease from an earlier writer session, including intact completed+copies. The fixed build discards the leases but does not reclassify those+records. A quarantined record is never expired and never evicted, and it stays+charged to quota. The defaults are 8 files (`MCP_HOST_GFS_CALLER_DOWNLOAD_MAX_FILES`)+and 256 MiB (`MCP_HOST_GFS_CALLER_DOWNLOAD_STORAGE_BYTES`) per caller, and 64+files (fixed) and 1024 MiB (`MCP_HOST_GFS_DOWNLOAD_STORAGE_BYTES`) per Host.+Pressure eviction cannot free them. A caller whose quarantined records alone+fill its limit gets `caller_quota_exceeded` on every new download; once+quarantined records alone fill the Host limit, every caller gets+`host_quota_exceeded`.++After the first rollout of the fixed build:++1. Read the startup warning `GFS download store holds <N> quarantined+record(s) charged to quota` and the gauge+ `clerum_gfs_download_store_quarantined_records`. N is the number of+ quarantined records in the ledger after startup reconciliation; it includes+ records the old build quarantined and any copy this boot found altered or+ unfinished.+2. Compare N, and the bytes those records hold, with the quotas above before+ running any GFS download check. A caller already at a limit fails every new+ download until its quarantined records are released.+3. To release them, run the+ [explicit operator recovery protocol](#explicit-operator-recovery-protocol)+ under a physical fence. With no lease left in the ledger, every quarantined+ record whose published copy still matches its stored size and checksum+ returns to `completed`, and partial copies without a checksum can be+ selected for removal. A record whose content no longer matches its checksum+ stays quarantined and charged (step 5).+## Explicit operator recovery protocolThe local operator module exports `inspectGfsStoreRecovery` and@@ -319,22 +390,35 @@ contents, filenames, caller identities, commands or credentials.whole operation and recheck it before filesystem effects.3. Inspect and review the three hashes and exact selection IDs. Unknown orunjournaled caches fail inventory even with an empty ledger. Select only- task owners proved terminal and explicitly approved unused partial copies.- The default preserves copies and charges. Since #1019 the Host discards- legacy processing leases itself at initialize, so the operator no longer- settles them to restore availability; `settledProcessingLeaseIds` remains- accepted only for a ledger written by an older build that is inspected- before any current Host opens it.+ task owners proved terminal and explicitly approved unused partial copies+ (`removablePartialIds`). Select every legacy processing lease listed in+ `processingLeaseIds` as settled. Since #1019 no executor holds a processing+ lease, and a current Host discards any legacy lease at initialize, so a+ legacy lease protects nothing once the fence of step 2 holds. A ledger+ inspected after a current Host has opened it has no leases, and the list is+ empty.+ Leaving any lease unselected sets every record to `quarantined`, including+ intact published copies, and writes the remaining leases back to the ledger.+ The next Host discards those leases but keeps the quarantine, and every+ retained copy stays charged until a second fenced recovery run.4. Call `recoverGfsStoreUnderPhysicalFence` with those expected hashes,- any `settledProcessingLeaseIds` from step 3, optional `terminalReceiptOwnerIds` and+ `settledProcessingLeaseIds` (every ID from step 3, or `[]` when the ledger+ has none), optional `terminalReceiptOwnerIds` and`removeSettledTransferIds`, and the real `withPhysicalFence` provider.A constant successful callback or an environment flag is not a provider.5. Verify the returned before/after receipt and preserved published copies.- Only an exact verified partial directory can be removed; its charge is- released after positive absence. A published `source` remains protected even- when a crash left its ledger entry without a completed state or checksum.- Verified copies with their original checksum can become reusable; corrupt- copies remain charged and quarantined.+ When no lease remains, a record whose published `source` matches its stored+ size and checksum becomes `completed` and reusable, whatever its previous+ state, including `quarantined`. Every other record becomes or stays+ `quarantined`. Only an exact verified partial directory can be removed: a+ record that is not `completed`, has no stored checksum and has no published+ `source`. Its charge is released after positive absence. A published+ `source` remains protected even when a crash left its ledger entry without a+ completed state or checksum.+ Recovery cannot release a quarantined record that keeps a checksum its+ content no longer matches, or whose published `source` is gone: it neither+ reclassifies it nor accepts it for removal (`invalid_selection`). The+ operator module has no path that releases that charge.6. Reopen the Host on the same retained PVC. Prove availability, authenticatedreuse, source/version equality and quotas before restoring ordinary traffic.On interruption, preserve the last receipt and accounting objects, obtain adiff --git a/mcp-host/package-lock.json b/mcp-host/package-lock.jsonindex c06fc7ee3..1d5ffdcd0 100644--- a/mcp-host/package-lock.json+++ b/mcp-host/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-mcp-host",- "version": "0.1.508",+ "version": "0.1.509","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-mcp-host",- "version": "0.1.508",+ "version": "0.1.509","license": "SEE LICENSE IN LICENSE","dependencies": {"@anthropic-ai/sdk": "^0.32.0",diff --git a/mcp-host/package.json b/mcp-host/package.jsonindex bbaf43977..705096994 100644--- a/mcp-host/package.json+++ b/mcp-host/package.json@@ -1,6 +1,6 @@{"name": "clerum-mcp-host",- "version": "0.1.508",+ "version": "0.1.509","description": "MCP Host - reads Host CRD and provides LLM access via OpenAI/Claude","main": "dist/main.js","engines": {diff --git a/mcp-host/src/__tests__/gfsRuntime.test.ts b/mcp-host/src/__tests__/gfsRuntime.test.tsindex f63acea2e..ef94475f3 100644--- a/mcp-host/src/__tests__/gfsRuntime.test.ts+++ b/mcp-host/src/__tests__/gfsRuntime.test.ts@@ -189,27 +189,44 @@ describe('GFS runtime bootstrap', () => {expect(vi.getTimerCount()).toBe(0)})- it('U9: logs that recovery is required when initialize succeeds but the store is unavailable', async () => {+ it('U9: logs an error and stops the periodic lifecycle when the store becomes unavailable after initialize', async () => {+ vi.useFakeTimers()const initialize = vi.spyOn(GfsDownloadStore.prototype, 'initialize')- vi.spyOn(GfsDownloadStore.prototype, 'isAvailable').mockReturnValue(false)const error = vi.spyOn(logger, 'error')+ let sweeps = 0+ // The first sweep runs against the real, available store; a failed persist+ // during it is what makes a real store unavailable (`unsafe`).+ const cleanup = vi+ .spyOn(GfsDownloadStore.prototype, 'cleanupExpired')+ .mockImplementation(async () => {+ sweeps += 1+ })const runtime = await bootstrapGfsRuntime(await root())runtimes.push(runtime)-- expect(initialize).toHaveBeenCalledOnce()await expect(initialize.mock.results[0]!.value).resolves.toBeUndefined()- const recoveryLogs = error.mock.calls.filter(([, message]) =>- message.includes('operator recovery is required')+ // initialize() sweeps once itself; the lifecycle's first sweep is the second.+ for (let turn = 0; turn < 20 && sweeps < 2; turn += 1) await Promise.resolve()+ expect(cleanup).toHaveBeenCalledTimes(2)+ expect(runtime.store.isAvailable()).toBe(true)+ expect(vi.getTimerCount()).toBe(1)++ vi.spyOn(runtime.store, 'isAvailable').mockReturnValue(false)+ await vi.advanceTimersByTimeAsync(60 * 60 * 1000)+ const stopLogs = error.mock.calls.filter(+ ([, message]) => typeof message === 'string' && message.includes('no longer available'))- expect(recoveryLogs).toEqual([+ expect(stopLogs).toEqual([[{ component: 'gfs-runtime', available: false },- 'GFS download store initialized but is not available; operator recovery is required and managed GFS delivery is disabled',+ 'GFS download store is no longer available; periodic cleanup stopped and managed GFS operations are disabled',],])+ // Witness: the lifecycle ran its first sweep; it sweeps and schedules nothing more.+ expect(cleanup).toHaveBeenCalledTimes(2)+ expect(vi.getTimerCount()).toBe(0)})- it('closes initialized ownership even when quarantine makes the store unavailable', async () => {+ it('closes initialized ownership even when a failed persist has made the store unavailable', async () => {const runtime = await bootstrapGfsRuntime(await root())runtimes.push(runtime)const close = vi.spyOn(runtime.store, 'close')diff --git a/mcp-host/src/agent/__tests__/taskExecutor.gfsExecutionSafety.test.ts b/mcp-host/src/agent/__tests__/taskExecutor.gfsExecutionSafety.test.tsindex dbd9891b6..62161a55b 100644--- a/mcp-host/src/agent/__tests__/taskExecutor.gfsExecutionSafety.test.ts+++ b/mcp-host/src/agent/__tests__/taskExecutor.gfsExecutionSafety.test.ts@@ -8,7 +8,7 @@ import { ConversationManager } from '../../core/conversation/conversation'import { SimpleEventEmitter } from '../../core/orchestration/eventEmitter'import { ShellTool } from '../../core/tools/shell'import { type ChatMessage, FinishReason, type ToolCall } from '../../core/types'-import { GfsDownloadStore } from '../../internalTools/gfsDownloadStore'+import { GfsDownloadStore, GfsDownloadStoreError } from '../../internalTools/gfsDownloadStore'import { TaskLifecycle } from '../../lifecycle/taskLifecycle'import type { SingleTurnProvider } from '../../llm/types'import type { Task, TaskSource } from '../../queue/types'@@ -102,8 +102,16 @@ async function unavailableStore() {await fs.mkdir(join(root, '.gfs-download-store'), { mode: 0o700 })const store = new GfsDownloadStore(root)stores.push(store)- await store.initialize().catch(() => undefined)- return { root, store }+ // A store directory without a ledger must refuse to initialize. Capture the+ // rejection instead of swallowing it so tests can assert which failure the+ // store reported; an initialize that resolves breaks the fixture loudly.+ const initError = await store.initialize().then(+ () => {+ throw new Error('unavailableStore fixture: initialize() resolved on a ledger-less store')+ },+ (error: unknown) => error+ )+ return { root, store, initError }}/**@@ -145,6 +153,12 @@ async function healthyStore() {return { root, store }}+/** Hooks called by the scripted provider at the start and end of every model turn. */+type TurnObserver = {+ turnStarted(turn: number): void+ turnReturned(turn: number, toolCall: ToolCall | undefined): void+}+async function shellScenario({source,approvalEnabled = true,@@ -154,6 +168,7 @@ async function shellScenario({channelCaller = false,missingCallerRoot = false,calls = [{ id: 'shell-safety', name: 'shell_exec', arguments: { command: 'pwd' } }],+ turnObserver,}: {source: TaskSourceapprovalEnabled?: boolean@@ -163,13 +178,16 @@ async function shellScenario({channelCaller?: booleanmissingCallerRoot?: booleancalls?: ToolCall[]+ turnObserver?: TurnObserver}) {appConfig.enableApproval = approvalEnabled- const { root, store } = legacyLease+ const fixture = legacyLease? await legacyLeaseStore(): healthy? await healthyStore(): await unavailableStore()+ const { root, store } = fixture+ const storeInitError = 'initError' in fixture ? fixture.initError : undefinedconst sourceMessage = channelCaller? {sender: 'gfs-safety-caller',@@ -201,9 +219,12 @@ async function shellScenario({throw new Error('Unexpected non-tool completion')},completeSingleTurnWithTools: async (messages, tools) => {+ const turn = providerCalls.length+ turnObserver?.turnStarted(turn)providerCalls.push(structuredClone(messages))advertisedTools.push(tools.map(tool => tool.name))- const call = calls[providerCalls.length - 1]+ const call = calls[turn]+ turnObserver?.turnReturned(turn, call)return {content: call ? null : 'Shell finished.',tool_calls: call ? [call] : null,@@ -262,6 +283,7 @@ async function shellScenario({onApprovalNeeded,callerWorkspace,store,+ storeInitError,}}@@ -318,6 +340,10 @@ it('X2: a store at corrupt_store_ledger keeps shell and GFS reads but withdraws{ id: 'shell-safety', name: 'shell_exec', arguments: { command: 'pwd' } },],})+ // Witness: the store refused to initialize with exactly corrupt_store_ledger,+ // not with any other initialize failure.+ expect(scenario.storeInitError).toBeInstanceOf(GfsDownloadStoreError)+ expect(scenario.storeInitError).toMatchObject({ code: 'corrupt_store_ledger' })expect(scenario.store.isAvailable()).toBe(false)await scenario.executor.run()@@ -410,7 +436,7 @@ function storeSpyNet(store: GfsDownloadStore) {* Record, for every ShellTool.execute call, the timeout the executor passed and* the store methods called between entering and leaving execute.*/-function shellExecuteWindows(net: ReturnType<typeof storeSpyNet>) {+function shellExecuteWindows(net: ReturnType<typeof storeSpyNet>, timeline?: string[]) {const windows: Array<{ timeoutMs: number | undefined; storeCalls: string[] }> = []const original = ShellTool.prototype.executeconst spy = vi.spyOn(ShellTool.prototype, 'execute').mockImplementation(async function (@@ -418,10 +444,12 @@ function shellExecuteWindows(net: ReturnType<typeof storeSpyNet>) {params,context) {+ timeline?.push('shell execute entered')const before = net.counts()try {return await original.call(this, params, context)} finally {+ timeline?.push('shell execute returned')const after = net.counts()windows.push({timeoutMs: context?.timeoutMs,@@ -435,6 +463,56 @@ function shellExecuteWindows(net: ReturnType<typeof storeSpyNet>) {return windows}+/**+ * Record the store methods called during the whole TaskExecutor dispatch of+ * every shell_exec call: the window opens when the scripted model returns the+ * call to the executor and closes when the executor sends the next model turn,+ * which carries the tool result. Everything the executor does for that call+ * (tool loop, approval gate, registry lookup, ShellTool.execute, output+ * processing) falls inside it. Registry construction before the first turn and+ * the terminal retention settlement after the last turn fall outside it by+ * construction, because no shell call is in flight then.+ */+function shellDispatchWindows(timeline: string[]) {+ let net: ReturnType<typeof storeSpyNet> | undefined+ let open: { toolCallId: string; before: Map<string, number> } | undefined+ const windows: Array<{ toolCallId: string; storeCalls: string[] }> = []+ const liveNet = () => {+ if (!net) throw new Error('shellDispatchWindows: the store spy net was never attached')+ return net+ }+ const observer: TurnObserver = {+ turnStarted(turn) {+ if (open) {+ const after = liveNet().counts()+ const before = open.before+ windows.push({+ toolCallId: open.toolCallId,+ storeCalls: [...after]+ .filter(([name, calls]) => calls > (before.get(name) ?? 0))+ .map(([name]) => name),+ })+ open = undefined+ }+ timeline.push(`turn ${turn} started`)+ },+ turnReturned(turn, toolCall) {+ timeline.push(`turn ${turn} returned ${toolCall?.name ?? 'final answer'}`)+ if (toolCall?.name === 'shell_exec')+ open = { toolCallId: toolCall.id, before: liveNet().counts() }+ },+ }+ return {+ observer,+ windows,+ attach(spyNet: ReturnType<typeof storeSpyNet>) {+ net = spyNet+ },+ /** A window still open after run() means the tool result never reached the model. */+ unclosed: () => open?.toolCallId,+ }+}+it('X1-shell: an inherited legacy lease does not stop an approved shell (#1019)', async () => {const scenario = await shellScenario({source: 'channel',@@ -490,21 +568,41 @@ describe('U5-TE: TaskExecutor runs a managed shell with a timeout above the form})})-it('X3-TE: a shell run inside TaskExecutor makes zero GFS download store calls (#1019)', async () => {- const scenario = await shellScenario({ source: 'cron', healthy: true })+it('X3-TE: a shell dispatch inside TaskExecutor makes zero GFS download store calls (#1019)', async () => {+ const timeline: string[] = []+ const dispatch = shellDispatchWindows(timeline)+ const scenario = await shellScenario({+ source: 'cron',+ healthy: true,+ turnObserver: dispatch.observer,+ })const net = storeSpyNet(scenario.store)- const windows = shellExecuteWindows(net)+ dispatch.attach(net)+ const executeWindows = shellExecuteWindows(net, timeline)await scenario.executor.run()// Witnesses: the net is attached to the store instance the executor wired- // (isAvailable is read while the registry is built), the shell window opened- // once, and the command's stdout reached the model.+ // (isAvailable is read while the registry is built, outside the dispatch+ // window), exactly one shell dispatch window opened and closed with+ // ShellTool.execute nested inside it, and the command's stdout reached the model.expect(net.names).toContain('isAvailable')expect(net.spies.get('isAvailable')!.mock.calls.length).toBeGreaterThan(0)- expect(windows).toHaveLength(1)+ expect(dispatch.unclosed()).toBeUndefined()+ expect(dispatch.windows).toEqual([{ toolCallId: 'shell-safety', storeCalls: expect.any(Array) }])+ expect(executeWindows).toHaveLength(1)+ expect(timeline).toEqual([+ 'turn 0 started',+ 'turn 0 returned shell_exec',+ 'shell execute entered',+ 'shell execute returned',+ 'turn 1 started',+ 'turn 1 returned final answer',+ ])expect(spawn).toHaveBeenCalledOnce()expect(JSON.stringify(scenario.providerCalls.at(-1))).toContain(scenario.callerWorkspace)- // Claim: no store method ran while the shell executed.- expect(windows[0]!.storeCalls).toEqual([])+ // Claim: no store method ran from the moment the executor received the shell+ // call until it returned the shell result to the model.+ expect(dispatch.windows[0]!.storeCalls).toEqual([])+ expect(executeWindows[0]!.storeCalls).toEqual([])})diff --git a/mcp-host/src/agent/taskExecutor.ts b/mcp-host/src/agent/taskExecutor.tsindex 53e5de74e..b6c426644 100644--- a/mcp-host/src/agent/taskExecutor.ts+++ b/mcp-host/src/agent/taskExecutor.ts@@ -1045,8 +1045,8 @@ export class TaskExecutor {/*** A retention owner spans preparation, an approval suspension, and cold resume.- * Release happens only after this executor reaches a terminal state; inherited- * owners are deliberately left quarantined by the store when release is denied.+ * Release happens only after this executor reaches a terminal state. A failed+ * release leaves the owner recorded, so its retained records stay protected.*/private settleGfsRetentionOwner(): Promise<void> {if (this.state === 'waiting_approval' && !this.abortController.signal.aborted)diff --git a/mcp-host/src/config.executionLimits.test.ts b/mcp-host/src/config.executionLimits.test.tsindex 1e6001f4c..ccef7172e 100644--- a/mcp-host/src/config.executionLimits.test.ts+++ b/mcp-host/src/config.executionLimits.test.ts@@ -85,7 +85,28 @@ describe('tool timeouts bounded to the executable timer range (#1021)', () => {vi.resetModules()vi.stubEnv(name, String(largestTimeout + 1))- await expect(import('./config')).rejects.toThrow(`${name} must be a valid bounded integer`)+ await expect(import('./config')).rejects.toThrow(+ `${name} must be a valid bounded integer from 1 to 2147477647 (inclusive)`+ )+ }+ )++ it.each([+ { name: 'CLERUM_AGENT_MAX_TASK_DURATION', field: 'agentMaxTaskDuration' as const },+ { name: 'CLERUM_AGENT_MAX_TOOL_CALLS', field: 'agentMaxToolCallsPerTask' as const },+ ])(+ 'accepts $name at the full timer range 2^31-1 and rejects one more',+ async ({ name, field }) => {+ vi.resetModules()+ vi.stubEnv(name, String(maxTimerDelayMs))+ const { config } = await import('./config')+ expect(config[field]).toBe(maxTimerDelayMs)++ vi.resetModules()+ vi.stubEnv(name, String(maxTimerDelayMs + 1))+ await expect(import('./config')).rejects.toThrow(+ `${name} must be a valid bounded integer from 1 to 2147483647 (inclusive)`+ )})diff --git a/mcp-host/src/config.ts b/mcp-host/src/config.tsindex 18b876f91..448ff4826 100644--- a/mcp-host/src/config.ts+++ b/mcp-host/src/config.ts@@ -341,9 +341,11 @@ function getEnv(key: string, defaultValue?: string): string | undefined {const MAX_TIMER_DELAY_MS = 2_147_483_647/**- * Upper bound for a tool timeout: executeWithTimeout arms the shell's cleanup- * timer on top of the execution deadline and rejects a sum above the timer- * range, so a larger value would fail every shell call at runtime (#1021).+ * Upper bound for a tool timeout: it reserves the shell's declared cleanup+ * budget below the timer range. executeWithTimeout arms no cleanup timer for+ * the shell (it joins abort settlement), but it rejects any timeout whose sum+ * with that budget exceeds the timer range, so a larger value would fail every+ * shell call at runtime (#1021).*/const MAX_TOOL_TIMEOUT_MS = MAX_TIMER_DELAY_MS - SHELL_TIMEOUT_CLEANUP_MS@@ -356,13 +358,11 @@ function getExecutionLimit(const raw = getEnv(key)if (raw === undefined) return defaultValueconst value = Number(raw)- if (- !/^\d+$/.test(raw) ||- !Number.isSafeInteger(value) ||- value < (allowZero ? 0 : 1) ||- value > maximum- ) {- throw new Error(`${key} must be a valid bounded integer`)+ const minimum = allowZero ? 0 : 1+ if (!/^\d+$/.test(raw) || !Number.isSafeInteger(value) || value < minimum || value > maximum) {+ throw new Error(+ `${key} must be a valid bounded integer from ${minimum} to ${maximum} (inclusive)`+ )}return value}diff --git a/mcp-host/src/core/orchestration/__tests__/toolUseLoop.shellNul.test.ts b/mcp-host/src/core/orchestration/__tests__/toolUseLoop.shellNul.test.tsindex 9f1af9053..ade47037c 100644--- a/mcp-host/src/core/orchestration/__tests__/toolUseLoop.shellNul.test.ts+++ b/mcp-host/src/core/orchestration/__tests__/toolUseLoop.shellNul.test.ts@@ -1,5 +1,4 @@import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'-import { spawn } from 'child_process'import { mkdtemp, rm } from 'fs/promises'import { tmpdir } from 'os'import { join } from 'path'@@ -13,17 +12,10 @@ import { SimpleEventEmitter } from '../eventEmitter'import { buildLoopConfig } from '../loopConfig'import { runToolUseLoop } from '../toolUseLoop'-// Real processes would run; the spy only observes whether one was started.-vi.mock('child_process', async importOriginal => {- const actual = await importOriginal<typeof import('child_process')>()- return { ...actual, spawn: vi.fn(actual.spawn) }-})-let workspacePath: stringbeforeEach(async () => {workspacePath = await mkdtemp(join(tmpdir(), 'clerum-shell-nul-loop-'))- vi.mocked(spawn).mockClear()})afterEach(async () => {@@ -68,33 +60,36 @@ async function runShellCall(command: string) {return { result, blocked, reasoning }}+// The loop stops at approval for every shell_exec call, so no process can be+// spawned in this harness with or without a NUL; a spawn negative here could+// never fail. What this level proves is that BasicSafety rejects the NUL+// command before an approval card exists, while the same command without the+// NUL reaches approval.describe('shell_exec NUL commands through the tool loop (#1020)', () => {- it('X5: BasicSafety rejects a NUL command before approval and no process is spawned', async () => {- const { result, blocked, reasoning } = await runShellCall('printf a\0b')+ it('X5: BasicSafety rejects a NUL command before approval, the same command without NUL reaches approval', async () => {+ const rejected = await runShellCall('printf a\0b')- expect(result.type).not.toBe('need_approval')- expect(blocked).toHaveLength(1)- expect(blocked[0]!.data).toMatchObject({+ expect(rejected.result.type).toBe('response')+ expect(rejected.blocked).toHaveLength(1)+ expect(rejected.blocked[0]!.data).toMatchObject({toolName: 'shell_exec',errors: ['shell_exec.command must not contain NUL characters'],})- // Witness: the loop reached the model again with the validation error as the tool result.- expect(reasoning.continueWithToolResults).toHaveBeenCalledOnce()- expect(JSON.stringify(vi.mocked(reasoning.continueWithToolResults).mock.calls[0])).toContain(- 'Parameter validation failed: shell_exec.command must not contain NUL characters'- )- expect(spawn).not.toHaveBeenCalled()- })-- it('X5 witness: the same command without NUL reaches approval', async () => {- const { result, blocked } = await runShellCall('printf ab')+ // The loop reached the model again with the validation error as the tool result.+ expect(rejected.reasoning.continueWithToolResults).toHaveBeenCalledOnce()+ expect(+ JSON.stringify(vi.mocked(rejected.reasoning.continueWithToolResults).mock.calls[0])+ ).toContain('Parameter validation failed: shell_exec.command must not contain NUL characters')- expect(blocked).toEqual([])- expect(result.type).toBe('need_approval')- if (result.type === 'need_approval') {- expect(result.approval.tool_name).toBe('shell_exec')- expect(result.approval.tool_call_id).toBe('tc_nul')+ // Witness: without the NUL, the identical call produces an approval card,+ // so the absence of one above is caused by the NUL rejection.+ const approved = await runShellCall('printf ab')+ expect(approved.blocked).toEqual([])+ expect(approved.result.type).toBe('need_approval')+ if (approved.result.type === 'need_approval') {+ expect(approved.result.approval.tool_name).toBe('shell_exec')+ expect(approved.result.approval.tool_call_id).toBe('tc_nul')}- expect(spawn).not.toHaveBeenCalled()+ expect(approved.reasoning.continueWithToolResults).not.toHaveBeenCalled()})})diff --git a/mcp-host/src/core/safety/safety.ts b/mcp-host/src/core/safety/safety.tsindex 31ecd9991..0e1c9d7c2 100644--- a/mcp-host/src/core/safety/safety.ts+++ b/mcp-host/src/core/safety/safety.ts@@ -1,3 +1,4 @@+import { logger } from '../../logger'import { Safety } from '../interfaces'import { isPrivateIp } from '../tools/httpRequest'import { SanitizedOutput, ValidationResult } from '../types'@@ -139,7 +140,10 @@ export class BasicSafety implements Safety {}const valid = errors.length === 0- console.log(`[NewCore:Safety] validateInput → length=${input?.length ?? 0}, passed=${valid}`)+ logger.debug(+ { component: 'NewCore:Safety', inputLength: input?.length ?? 0, passed: valid },+ 'validateInput'+ )return { is_valid: valid, errors }}@@ -235,8 +239,14 @@ export class BasicSafety implements Safety {const result = this.sanitizeFreeformContent(output, {secretWarning: `Potential secret detected in ${toolName} output`,})- console.log(- `[NewCore:Safety] sanitizeOutput → tool=${toolName}, sanitized=${result.was_modified}, warnings=${result.warnings.length}`+ logger.debug(+ {+ component: 'NewCore:Safety',+ toolName,+ sanitized: result.was_modified,+ warnings: result.warnings.length,+ },+ 'sanitizeOutput')return result}@@ -246,8 +256,13 @@ export class BasicSafety implements Safety {secretWarning: 'Potential secret detected in assistant response',extraFilters: BasicSafety.ASSISTANT_RESPONSE_FILTER_PATTERNS,})- console.log(- `[NewCore:Safety] sanitizeAssistantResponse → sanitized=${result.was_modified}, warnings=${result.warnings.length}`+ logger.debug(+ {+ component: 'NewCore:Safety',+ sanitized: result.was_modified,+ warnings: result.warnings.length,+ },+ 'sanitizeAssistantResponse')return result}@@ -263,7 +278,10 @@ export class BasicSafety implements Safety {// Risk 4.10: Escape potential closing tags in contentconst escaped = content.replace(/<\/tool_output>/gi, '</tool_output>')const wrapped = `<tool_output name="${toolName}" sanitized="${wasSanitized}">\n${escaped}\n</tool_output>`- console.log(`[NewCore:Safety] wrapForLlm → tool=${toolName}, wrappedLength=${wrapped.length}`)+ logger.debug(+ { component: 'NewCore:Safety', toolName, wrappedLength: wrapped.length },+ 'wrapForLlm'+ )return wrapped}diff --git a/mcp-host/src/core/tools/__tests__/shell.test.ts b/mcp-host/src/core/tools/__tests__/shell.test.tsindex ee7372975..cf99bd0e7 100644--- a/mcp-host/src/core/tools/__tests__/shell.test.ts+++ b/mcp-host/src/core/tools/__tests__/shell.test.ts@@ -1,10 +1,11 @@-import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'+import { type MockInstance, afterEach, beforeEach, describe, expect, it, vi } from 'vitest'import { spawn } from 'child_process'import { existsSync } from 'fs'import { mkdir, mkdtemp, readFile, rename, rm, symlink, writeFile } from 'fs/promises'import { tmpdir } from 'os'import { join } from 'path'import { ALL_PROVIDERS, LlmProvider, PROVIDERS } from '../../../llm/registryCore'+import { logger } from '../../../logger'import { executeWithTimeout } from '../../orchestration/toolExecutionTimeout'import { ShellTool } from '../shell'@@ -244,51 +245,49 @@ describe('ShellTool', () => {expect(r2.content).not.toContain('first-run')})- // Skipped on CI: Ubuntu GH Actions runners (dash as /bin/sh) exhibit unreliable- // process-group kill propagation to `sleep` grandchildren even after SIGKILL grace.- // Production behavior is verified by manual minikube e2e + code review.- it.skipIf(process.env.CI)(- 'kills grandchild processes (e.g. sleep in a semicolon chain) via process-group kill',- async () => {- // Spawn a long-sleeping grandchild and write its PID before waiting on it.- // Checking the PID directly avoids `pgrep` false positives from the probe- // command itself and from platform-specific process-list behavior.- const pidFile = join(workspacePath, 'grandchild.pid')- const tool = new ShellTool(workspacePath, 500, ['PATH'])+ // Runs on CI like U6 below. It used to be skipped there because the tool+ // resolved on leader close while a `sleep` grandchild could still be alive.+ // Since 9b1c29e88 the close handler SIGKILLs the process group and resolves+ // only after the group no longer exists (or reports+ // process_group_termination_failed), so that skip reason no longer applies.+ it('kills grandchild processes (e.g. sleep in a semicolon chain) via process-group kill', async () => {+ // Spawn a long-sleeping grandchild and write its PID before waiting on it.+ // Checking the PID directly avoids `pgrep` false positives from the probe+ // command itself and from platform-specific process-list behavior.+ const pidFile = join(workspacePath, 'grandchild.pid')+ const tool = new ShellTool(workspacePath, 500, ['PATH'])- // Run the tool — timeout fires at 500ms.- const result = await tool.execute({- command: `echo start; sleep 60 & echo $! > ${pidFile}; wait`,- })+ // Run the tool — timeout fires at 500ms.+ const result = await tool.execute({+ command: `echo start; sleep 60 & echo $! > ${pidFile}; wait`,+ })- expect(result.is_error).toBe(true)- expect(result.content).toContain('start')- expect(result.content).toContain('timeout')-- // After the tool resolves, give the kernel a moment to clean up.- // Then assert the captured `sleep` PID no longer exists.- const pid = Number((await readFile(pidFile, 'utf8')).trim())- expect(Number.isInteger(pid)).toBe(true)- expect(pid).toBeGreaterThan(0)-- // Poll up to 8s for grandchild to be reaped. SIGTERM → SIGKILL grace is 5s,- // so the window must exceed that to cover slow runners where SIGTERM may- // not reach the grandchild immediately.- let grandchildAlive = true- for (let i = 0; i < 80; i++) {- await new Promise(r => setTimeout(r, 100))- try {- process.kill(pid, 0)- } catch {- grandchildAlive = false- break- }+ expect(result.is_error).toBe(true)+ expect(result.content).toContain('start')+ expect(result.content).toContain('timeout')++ // After the tool resolves, give the kernel a moment to clean up.+ // Then assert the captured `sleep` PID no longer exists.+ const pid = Number((await readFile(pidFile, 'utf8')).trim())+ expect(Number.isInteger(pid)).toBe(true)+ expect(pid).toBeGreaterThan(0)++ // Poll up to 8s for grandchild to be reaped. SIGTERM → SIGKILL grace is 5s,+ // so the window must exceed that to cover slow runners where SIGTERM may+ // not reach the grandchild immediately.+ let grandchildAlive = true+ for (let i = 0; i < 80; i++) {+ await new Promise(r => setTimeout(r, 100))+ try {+ process.kill(pid, 0)+ } catch {+ grandchildAlive = false+ break}+ }- expect(grandchildAlive).toBe(false)- },- 20_000- ) // generous test timeout — 500ms shell + up to 8s polling+ expect(grandchildAlive).toBe(false)+ }, 20_000) // generous test timeout — 500ms shell + up to 8s pollingdescribe('dynamicEnvProvider — ConfigStore snapshot merge', () => {it('exposes ConfigStore values to the subprocess shell at spawn time', async () => {@@ -676,20 +675,97 @@ describe('ShellTool without the GFS download store (#1019)', () => {})describe('ShellTool start failures (#1020)', () => {+ let loggerError: MockInstance<typeof logger.error>+beforeEach(() => {vi.mocked(spawn).mockClear()+ loggerError = vi.spyOn(logger, 'error').mockImplementation(() => {})+ })++ afterEach(() => {+ loggerError.mockRestore()})- it('U3: resolves a synchronous spawn throw from a NUL env value as a start failure', async () => {- const tool = new ShellTool(workspacePath, 5_000, ['PATH'], () => ({ BROKEN: 'a\0b' }))+ // Synthetic secret-looking value; the text on both sides of the NUL must+ // never reach the result or the log.+ const SECRET_HEAD = 'opaque-integration-secret-7f3a9c'+ const SECRET_TAIL = 'tail-after-nul-41be'++ it('U3: rejects a NUL env value before spawn and names only the variable key', async () => {+ const tool = new ShellTool(workspacePath, 5_000, ['PATH'], () => ({+ INTEGRATION_TOKEN: `${SECRET_HEAD}\0${SECRET_TAIL}`,+ }))const result = await tool.execute({ command: 'printf never-runs' })- expect(result.is_error).toBe(true)- expect(result.content).toMatch(/^Command failed to start: /)- expect(result.content).not.toContain('never-runs')+ expect(result).toMatchObject({+ is_error: true,+ content:+ 'Command failed to start: environment variable INTEGRATION_TOKEN contains a NUL character',+ })expect(result.duration_ms).toBeGreaterThanOrEqual(0)- // Witness: spawn was reached and threw; no process exists to report on.+ for (const fragment of [SECRET_HEAD, SECRET_TAIL, SECRET_HEAD.slice(0, 8), '\\x00', '\0']) {+ expect(result.content).not.toContain(fragment)+ }+ // Witness: the rejection was logged once with a fixed code and no value.+ expect(loggerError).toHaveBeenCalledOnce()+ expect(loggerError.mock.calls[0]![0]).toEqual({+ component: 'ShellTool',+ errorCode: 'ENV_VALUE_CONTAINS_NUL',+ })+ const logged = JSON.stringify(loggerError.mock.calls[0])+ expect(logged).not.toContain(SECRET_HEAD.slice(0, 8))+ expect(logged).not.toContain(SECRET_TAIL)+ expect(spawn).not.toHaveBeenCalled()++ // Witness: the same tool and key without the NUL does reach spawn.+ const valid = new ShellTool(workspacePath, 5_000, ['PATH'], () => ({+ INTEGRATION_TOKEN: SECRET_HEAD,+ }))+ const ok = await valid.execute({ command: 'printf runs' })+ expect(ok).toMatchObject({ is_error: false, content: 'stdout:\nruns' })+ expect(spawn).toHaveBeenCalledOnce()+ })++ it('U3b: reports only the error code of a synchronous spawn throw', async () => {+ vi.mocked(spawn).mockImplementationOnce(() => {+ throw Object.assign(new Error(`spawn rejected argument ${SECRET_HEAD}`), { code: 'E2BIG' })+ })+ const tool = new ShellTool(workspacePath, 5_000, ['PATH'])+ const result = await tool.execute({ command: 'printf never-runs' })+ expect(result).toMatchObject({ is_error: true, content: 'Command failed to start: E2BIG' })+ expect(result.content).not.toContain(SECRET_HEAD.slice(0, 8))+ expect(result.content).not.toContain('spawn rejected')+ // Witness: spawn was reached and threw.expect(spawn).toHaveBeenCalledOnce()expect(vi.mocked(spawn).mock.results[0]!.type).toBe('throw')+ expect(loggerError).toHaveBeenCalledOnce()+ expect(loggerError.mock.calls[0]![0]).toEqual({ component: 'ShellTool', errorCode: 'E2BIG' })+ const logged = JSON.stringify(loggerError.mock.calls[0])+ expect(logged).not.toContain(SECRET_HEAD.slice(0, 8))+ expect(logged).not.toContain('spawn rejected')+ })++ it('U3c: reports a fixed message for a synchronous spawn throw without a well-formed code', async () => {+ vi.mocked(spawn)+ .mockImplementationOnce(() => {+ throw new Error(`spawn rejected argument ${SECRET_HEAD}`)+ })+ .mockImplementationOnce(() => {+ throw Object.assign(new Error(`spawn rejected argument ${SECRET_HEAD}`), {+ code: `bad ${SECRET_TAIL}`,+ })+ })+ const tool = new ShellTool(workspacePath, 5_000, ['PATH'])+ for (let call = 0; call < 2; call += 1) {+ const result = await tool.execute({ command: 'printf never-runs' })+ expect(result).toMatchObject({ is_error: true, content: 'Command failed to start' })+ }+ // Witness: both calls reached spawn, threw, and were logged without text.+ expect(vi.mocked(spawn).mock.results.map(r => r.type)).toEqual(['throw', 'throw'])+ expect(loggerError.mock.calls.map(call => call[0])).toEqual([+ { component: 'ShellTool', errorCode: 'UNKNOWN' },+ { component: 'ShellTool', errorCode: 'UNKNOWN' },+ ])+ expect(JSON.stringify(loggerError.mock.calls)).not.toContain(SECRET_TAIL)})it('U4: reports an asynchronous spawn failure (missing cwd) as a start failure', async () => {diff --git a/mcp-host/src/core/tools/shell.ts b/mcp-host/src/core/tools/shell.tsindex e81be4832..578d00446 100644--- a/mcp-host/src/core/tools/shell.ts+++ b/mcp-host/src/core/tools/shell.ts@@ -2,6 +2,7 @@ import { type ChildProcess, spawn } from 'child_process'import { setTimeout as sleep } from 'node:timers/promises'import { recordGfsShellOutputLimit } from '../../internalTools/gfsDownloadMetrics'import { ALL_PROVIDERS, LlmProvider, PROVIDERS } from '../../llm/registryCore'+import { logger } from '../../logger'import { verifyManagedCallerRootPath } from '../../workspace/callerRootBinding'import { ToolError, ToolErrorCode } from '../errors'import { ExecutionContext, Tool } from '../interfaces'@@ -18,6 +19,10 @@ import { SHELL_SIGKILL_GRACE_MS, SHELL_TIMEOUT_CLEANUP_MS } from './shellTimeoutconst STATE_DB_COMMAND_PATTERN =/(^|[^A-Za-z0-9_.-])state\.db(-wal|-shm)?([^A-Za-z0-9_]|$)|\.clerum-state(\/|[^A-Za-z0-9_.-]|$)/+// Node/libuv error codes (E2BIG, ENOMEM, ERR_INVALID_ARG_VALUE) are fixed+// identifiers; anything else is not echoed to the model.+const SPAWN_ERROR_CODE_PATTERN = /^[A-Z0-9_]+$/+function truncateUtf8Bytes(value: string, maximumBytes: number): string {if (maximumBytes <= 0) return ''const bytes = Buffer.from(value, 'utf8')@@ -212,6 +217,24 @@ export class ShellTool implements Tool {}}+ // A NUL byte in an env value (e.g. a ConfigStore secret, which BasicSafety+ // never sees, #1020) makes spawn() throw ERR_INVALID_ARG_VALUE with up to+ // 128 characters of the raw value in its message. Reject it here and name+ // only the key, so no part of the value reaches the model or the logs.+ for (const [key, value] of Object.entries(safeEnv)) {+ if (value.includes('\0')) {+ logger.error(+ { component: 'ShellTool', errorCode: 'ENV_VALUE_CONTAINS_NUL' },+ 'Shell command failed to start'+ )+ return {+ content: `Command failed to start: environment variable ${key} contains a NUL character`,+ duration_ms: Date.now() - startTime,+ is_error: true,+ }+ }+ }+return new Promise<ToolOutput>(resolve => {// detached: true makes the child a process group leader so we can kill// the whole group (shell + grandchildren like `sleep`) with -pid signal.@@ -224,11 +247,25 @@ export class ShellTool implements Tool {detached: true,})} catch (error) {- // spawn() throws synchronously for invalid arguments, e.g. a NUL byte in- // a dynamic env value that BasicSafety never sees (#1020). No process- // exists, so this is a start failure reported as an error result.+ // spawn() throws synchronously for invalid arguments and for most libuv+ // errors (E2BIG for an oversized command, ENOMEM under memory pressure).+ // No process exists, so this is a start failure reported as an error+ // result. Node error messages can embed argument and env values, so+ // only a well-formed error code is reported, never the message.+ const rawCode = (error as NodeJS.ErrnoException | undefined)?.code+ const errorCode =+ typeof rawCode === 'string' && SPAWN_ERROR_CODE_PATTERN.test(rawCode)+ ? rawCode+ : undefined+ logger.error(+ { component: 'ShellTool', errorCode: errorCode ?? 'UNKNOWN' },+ 'Shell command failed to start'+ )resolve({- content: `Command failed to start: ${error instanceof Error ? error.message : String(error)}`,+ content:+ errorCode === undefined+ ? 'Command failed to start'+ : `Command failed to start: ${errorCode}`,duration_ms: Date.now() - startTime,is_error: true,})diff --git a/mcp-host/src/core/tools/shellTimeouts.ts b/mcp-host/src/core/tools/shellTimeouts.tsindex 128751103..c13c0b985 100644--- a/mcp-host/src/core/tools/shellTimeouts.ts+++ b/mcp-host/src/core/tools/shellTimeouts.ts@@ -2,9 +2,11 @@export const SHELL_SIGKILL_GRACE_MS = 5000/**- * Bounded termination and result collection after a shell deadline: the- * SIGKILL grace plus 1s for close/output delivery. executeWithTimeout adds it to- * the execution deadline, so a configured timeout must leave room for it below- * the largest delay a Node.js timer accepts (#1021).+ * Cleanup budget the shell declares after its deadline: the SIGKILL grace plus+ * 1s for close/output delivery. executeWithTimeout does not arm a cleanup timer+ * for the shell, because the shell joins abort settlement and owns its own+ * process-group termination; the budget is only reserved by executeWithTimeout's+ * check that timeout + cleanup stays within the largest delay a Node.js timer+ * accepts, so a configured timeout must leave room for it (#1021).*/export const SHELL_TIMEOUT_CLEANUP_MS = SHELL_SIGKILL_GRACE_MS + 1000diff --git a/mcp-host/src/gfsRuntime.ts b/mcp-host/src/gfsRuntime.tsindex e5b116d59..15e4dc06c 100644--- a/mcp-host/src/gfsRuntime.ts+++ b/mcp-host/src/gfsRuntime.ts@@ -49,6 +49,17 @@ export async function bootstrapGfsRuntime(if (lifecycleTimer) clearTimeout(lifecycleTimer)lifecycleTimer = undefined}+ // A store whose initialize() resolved is available; it becomes unavailable+ // only afterwards, when a persist fails (`unsafe`) or writer ownership is+ // lost. The periodic lifecycle then ends for good, so that is logged as an+ // error rather than returning silently.+ const lifecycleStopped = (): undefined => {+ logger.error(+ { component: 'gfs-runtime', available: false },+ 'GFS download store is no longer available; periodic cleanup stopped and managed GFS operations are disabled'+ )+ return undefined+ }const cycle = async (kind: LifecycleStep['kind']): Promise<LifecycleStep | undefined> => {if (stopped) returnif (kind === 'retry') {@@ -75,17 +86,19 @@ export async function bootstrapGfsRuntime(}if (stopped) returnlogger.info(- { component: 'gfs-runtime', attempt: attempts, available: store.isAvailable() },+ { component: 'gfs-runtime', attempt: attempts },'GFS writer retry initialization completed')}- if (stopped || !store.isAvailable()) return+ if (stopped) return+ if (!store.isAvailable()) return lifecycleStopped()try {await store.cleanupExpired()} catch (error) {logger.warn({ component: 'gfs-runtime', err: error }, 'GFS download cleanup failed')}- if (stopped || !store.isAvailable()) return+ if (stopped) return+ if (!store.isAvailable()) return lifecycleStopped()return { kind: 'cleanup', delayMs: 60 * 60 * 1000 }}const schedule = (step: LifecycleStep): void => {@@ -117,12 +130,7 @@ export async function bootstrapGfsRuntime(try {await store.initialize()- if (store.isAvailable()) startCycle('cleanup')- else- logger.error(- { component: 'gfs-runtime', available: false },- 'GFS download store initialized but is not available; operator recovery is required and managed GFS delivery is disabled'- )+ startCycle('cleanup')} catch (error) {if (transientWriterContention(error)) {logger.warn(@@ -147,8 +155,9 @@ export async function bootstrapGfsRuntime(clearLifecycleTimer()stopping = (async () => {await lifecycleInFlight- // Initialization can own a writer while quarantine makes isAvailable()- // false. Close unconditionally, including a retry that won during stop.+ // An initialized store can still own its writer after a failed persist+ // has made isAvailable() false. Close unconditionally, including a+ // retry that won during stop.await store.close()})()return stoppingdiff --git a/mcp-host/src/internalTools/gfsDownloadMetrics.test.ts b/mcp-host/src/internalTools/gfsDownloadMetrics.test.tsindex 8d7991a41..faea0f738 100644--- a/mcp-host/src/internalTools/gfsDownloadMetrics.test.ts+++ b/mcp-host/src/internalTools/gfsDownloadMetrics.test.ts@@ -7,9 +7,9 @@ import {recordGfsDownloadExpiry,recordGfsDownloadQuota,recordGfsDownloadTransfer,- recordGfsInheritedQuarantinedRecords,recordGfsLegacyProcessingLeasesDiscarded,recordGfsShellOutputLimit,+ setGfsQuarantinedRecords,} from './gfsDownloadMetrics'async function counterValue(name: string): Promise<number> {@@ -53,21 +53,22 @@ describe('GFS download metrics', () => {expect(scraped).toContain('clerum_gfs_download_expiry_total{outcome="cleanup_failed"}')})- it('U10: registers the legacy-lease discard and inherited-quarantine counters', async () => {+ it('U10: registers the legacy-lease discard counter and the quarantined-records gauge', async () => {const scraped = await register.metrics()expect(scraped).toContain('# TYPE clerum_gfs_legacy_processing_leases_discarded_total counter')- expect(scraped).toContain('# TYPE clerum_gfs_inherited_quarantined_records_total counter')+ expect(scraped).toContain('# TYPE clerum_gfs_download_store_quarantined_records gauge')const discardedBefore = await counterValue('clerum_gfs_legacy_processing_leases_discarded_total')- const quarantinedBefore = await counterValue('clerum_gfs_inherited_quarantined_records_total')recordGfsLegacyProcessingLeasesDiscarded(2)- recordGfsInheritedQuarantinedRecords(3)expect(await counterValue('clerum_gfs_legacy_processing_leases_discarded_total')).toBe(discardedBefore + 2)- expect(await counterValue('clerum_gfs_inherited_quarantined_records_total')).toBe(- quarantinedBefore + 3- )+ // A gauge carries the current count: setting it twice never accumulates.+ setGfsQuarantinedRecords(3)+ setGfsQuarantinedRecords(3)+ expect(await counterValue('clerum_gfs_download_store_quarantined_records')).toBe(3)+ setGfsQuarantinedRecords(0)+ expect(await counterValue('clerum_gfs_download_store_quarantined_records')).toBe(0)})})diff --git a/mcp-host/src/internalTools/gfsDownloadMetrics.ts b/mcp-host/src/internalTools/gfsDownloadMetrics.tsindex 5d3472d54..ba12dce61 100644--- a/mcp-host/src/internalTools/gfsDownloadMetrics.ts+++ b/mcp-host/src/internalTools/gfsDownloadMetrics.ts@@ -95,18 +95,17 @@ const gfsLegacyProcessingLeasesDiscardedTotal = getOrCreateCounter({labelNames: [] as const as Array<never>,})-const gfsInheritedQuarantinedRecordsTotal = getOrCreateCounter({- name: 'clerum_gfs_inherited_quarantined_records_total',- help: 'GFS download store records found quarantined by an earlier boot at initialize; they stay charged to quota until operator recovery.',- labelNames: [] as const as Array<never>,+const gfsDownloadStoreQuarantinedRecords = getOrCreateGauge({+ name: 'clerum_gfs_download_store_quarantined_records',+ help: 'GFS download store records currently quarantined, set at each initialize after reconcile; they stay charged to quota. Operator recovery reclassifies only copies whose content still matches their recorded hash; any other quarantined copy stays charged, and the operator recovery tool cannot release it.',})export function recordGfsLegacyProcessingLeasesDiscarded(count: number): void {gfsLegacyProcessingLeasesDiscardedTotal.inc(count)}-export function recordGfsInheritedQuarantinedRecords(count: number): void {- gfsInheritedQuarantinedRecordsTotal.inc(count)+export function setGfsQuarantinedRecords(count: number): void {+ gfsDownloadStoreQuarantinedRecords.set(count)}export function recordGfsDownloadAdmission(outcome: GfsDownloadAdmissionOutcome): void {diff --git a/mcp-host/src/internalTools/gfsDownloadStore.test.ts b/mcp-host/src/internalTools/gfsDownloadStore.test.tsindex 2df5b137f..31173b731 100644--- a/mcp-host/src/internalTools/gfsDownloadStore.test.ts+++ b/mcp-host/src/internalTools/gfsDownloadStore.test.ts@@ -693,7 +693,10 @@ describe('S6: expiry and admission without processing-lease exemptions (#1019)',)}- it('removes an expired record even after a shell read it', async () => {+ // Store behaviour only: this shell has no store reference, so the test shows+ // that expiry is unaffected by a shell run that read the copy. Decoupling of+ // the shell from the store is proven by the TaskExecutor-level X3-TE test.+ it('removes an expired record even after a shell read it: expiry is unaffected by the shell run', async () => {const receipt = await publishFixture()const shell = new ShellTool(callerRoot, 5_000, ['PATH'], () => ({}), undefined, true)const read = await shell.execute({ command: `cat ${JSON.stringify(receipt.path)}` })@@ -767,12 +770,14 @@ describe('S6: expiry and admission without processing-lease exemptions (#1019)',const expired = await publishFixture()const callerBRoot = path.join(hostRoot, 'users', 'caller-b')await fs.mkdir(callerBRoot, { recursive: true, mode: 0o700 })+ // Caller B's expiry is derived from caller A's, so the mocked clock below+ // is strictly before it however fast the fixtures run.const callerBTransfer = await store.createTransfer({callerIdentity: 'caller-b',callerWorkspacePath: callerBRoot,source,sizeBytes: 7,- expiresAt: new Date(Date.now() + 60_000).toISOString(),+ expiresAt: new Date(Date.parse(expired.expiresAt) + 60_000).toISOString(),})await fs.writeFile(path.join(callerBRoot, callerBTransfer.partialPath), 'fixture')const callerBReceipt = await store.publish(diff --git a/mcp-host/src/internalTools/gfsDownloadStore.ts b/mcp-host/src/internalTools/gfsDownloadStore.tsindex 66fb6a0e7..c153f3132 100644--- a/mcp-host/src/internalTools/gfsDownloadStore.ts+++ b/mcp-host/src/internalTools/gfsDownloadStore.ts@@ -7,8 +7,8 @@ import type { GfsImageSource } from '../visualInput/policy'import {recordGfsDownloadExpiry,recordGfsDownloadQuota,- recordGfsInheritedQuarantinedRecords,recordGfsLegacyProcessingLeasesDiscarded,+ setGfsQuarantinedRecords,} from './gfsDownloadMetrics'import {GFS_FILE_LIMITS,@@ -22,6 +22,8 @@ import {} from './gfsStorePrivateFiles'import { GfsStoreWriterLease, GfsStoreWriterOwnershipError } from './gfsStoreWriterLease'+const GFS_DOWNLOAD_STORE_LOG_COMPONENT = 'GfsDownloadStore'+export type GfsDownloadStoreErrorCode =| 'caller_mismatch'| 'caller_quota_exceeded'@@ -391,7 +393,6 @@ export class GfsDownloadStore {// truncated ledger fails closed.this.ledger = parseLedger(raw)await this.discardLegacyProcessingLeases()- this.reportInheritedQuarantine()} else if (createdStoreRoot) {// Genuinely new store directory: make the durable schema-1 ledger exist// before reconciliation or any transfer exposure, so a ledger missing@@ -407,6 +408,7 @@ export class GfsDownloadStore {this.initialized = trueawait this.cleanupExpired()await this.persist()+ this.reportQuarantine()} catch (error) {this.initialized = falseawait this.releaseWriterLease()@@ -1183,9 +1185,14 @@ export class GfsDownloadStore {* #1019: shell_exec no longer holds processing leases, so no lease in a* ledger written by an earlier build protects an executor of this boot. A* lease left by a crashed Host would otherwise fence the store forever. The- * field is removed durably before reconciliation; the warning and the- * counter are emitted only after that removal is persisted, so a failed- * persist leaves the ledger untouched and initialize rejects.+ * field is removed durably before reconciliation. The confirmed warning and+ * the counter are emitted only after that removal is persisted. A failed+ * persist restores the in-memory map and initialize rejects, but the ledger+ * on disk is not necessarily untouched: `persist` can fail after the rename+ * (the directory sync), when the lease-free ledger is already the visible+ * file and no later boot will see the leases again. That outcome is logged+ * as unknown and never counted, because the counter counts confirmed+ * discards only.*/private async discardLegacyProcessingLeases(): Promise<void> {const legacy = this.ledger.processingLeases@@ -1198,30 +1205,40 @@ export class GfsDownloadStore {await this.persist()} catch (error) {this.ledger.processingLeases = legacy+ logger.warn(+ {+ component: GFS_DOWNLOAD_STORE_LOG_COMPONENT,+ legacyProcessingLeases: discarded,+ outcome: 'unknown',+ },+ `GFS download store could not confirm the discard of ${discarded} legacy processing lease(s) at initialize; the discard may already be visible on disk because the persist can fail after the ledger rename`+ )throw error}logger.warn(- { discarded },+ { component: GFS_DOWNLOAD_STORE_LOG_COMPONENT, discarded },`GFS download store discarded ${discarded} legacy processing lease(s) at initialize`)recordGfsLegacyProcessingLeasesDiscarded(discarded)}/**- * Records an earlier boot quarantined stay charged to quota (`usageFor`).- * They are reported so an operator can recover them; this boot never- * reuses or deletes them.+ * Quarantined records stay charged to quota (`usageFor`); this store never+ * reuses or deletes them. Called once per initialize, after reconcile and+ * the final persist, so the count includes records this boot quarantined.+ * The gauge carries the current count, so repeated boots never add the same+ * backlog twice.*/- private reportInheritedQuarantine(): void {+ private reportQuarantine(): void {const quarantined = Object.values(this.ledger.records).filter(record => record.state === 'quarantined').length+ setGfsQuarantinedRecords(quarantined)if (quarantined === 0) returnlogger.warn(- { quarantined },- `GFS download store found ${quarantined} record(s) quarantined by an earlier boot; they stay charged to quota until operator recovery`+ { component: GFS_DOWNLOAD_STORE_LOG_COMPONENT, quarantined },+ `GFS download store holds ${quarantined} quarantined record(s) charged to quota. Operator recovery reclassifies only copies whose content still matches their recorded hash; any other quarantined copy stays charged, and the operator recovery tool cannot release it`)- recordGfsInheritedQuarantinedRecords(quarantined)}private async reconcile(): Promise<void> {diff --git a/mcp-host/src/internalTools/gfsDownloadStoreAdversarialQuota.test.ts b/mcp-host/src/internalTools/gfsDownloadStoreAdversarialQuota.test.tsindex 91d8b7962..b92e9d35c 100644--- a/mcp-host/src/internalTools/gfsDownloadStoreAdversarialQuota.test.ts+++ b/mcp-host/src/internalTools/gfsDownloadStoreAdversarialQuota.test.ts@@ -411,7 +411,10 @@ describe('GFS download store adversarial quota', () => {await store.releaseReceiptOwner('filler-owner', CALLER)})- it('evicts a copy a shell has just read: a shell run holds no protection (#1019)', async () => {+ // Store behaviour only: this shell has no store reference, so the test shows+ // that eviction is unaffected by a shell run that read the copy. Decoupling of+ // the shell from the store is proven by the TaskExecutor-level X3-TE test.+ it('evicts a copy a shell has just read: eviction is unaffected by the shell run (#1019)', async () => {const size = GFS_FILE_LIMITS.inlineTextBytes + 1const receipts = []for (let index = 1; index <= GFS_FILE_LIMITS.callerRetainedFiles; index += 1)diff --git a/mcp-host/src/internalTools/gfsDownloadStoreRecovery.test.ts b/mcp-host/src/internalTools/gfsDownloadStoreRecovery.test.tsindex 3cb2a8ed1..29d9d9ca7 100644--- a/mcp-host/src/internalTools/gfsDownloadStoreRecovery.test.ts+++ b/mcp-host/src/internalTools/gfsDownloadStoreRecovery.test.ts@@ -138,7 +138,7 @@ async function fixture(}const DISCARDED_COUNTER = 'clerum_gfs_legacy_processing_leases_discarded_total'-const QUARANTINED_COUNTER = 'clerum_gfs_inherited_quarantined_records_total'+const QUARANTINED_GAUGE = 'clerum_gfs_download_store_quarantined_records'const FOREIGN_WRITER_SESSION = '33333333-3333-4333-8333-333333333333'const LEGACY_LEASE_ID = '44444444-4444-4444-8444-444444444444'const otherSource: GfsImageSource = {@@ -211,7 +211,10 @@ describe('legacy processing leases after #1019', () => {expect(onDisk).not.toHaveProperty('processingLeases')expect((await counterValue(DISCARDED_COUNTER)) - before).toBe(1)expect(discardWarnings(warn.mock.calls)).toEqual([- [{ discarded: 1 }, 'GFS download store discarded 1 legacy processing lease(s) at initialize'],+ [+ { component: 'GfsDownloadStore', discarded: 1 },+ 'GFS download store discarded 1 legacy processing lease(s) at initialize',+ ],])expect(reopened.isAvailable()).toBe(true)const receipt = await fixture(reopened)@@ -287,7 +290,10 @@ describe('legacy processing leases after #1019', () => {expect(discardWarnings(warn.mock.calls)).toEqual([])})- it('S5: close() is blocked only by its own transfers, never by a running shell', async () => {+ // Store behaviour only: this shell has no store reference, so the test shows+ // that close() is unaffected by a concurrent shell run. Decoupling of the+ // shell from the store is proven by the TaskExecutor-level X3-TE test.+ it('S5: close() is blocked only by its own transfers and is unaffected by a concurrent shell run', async () => {const store = freshStore()await store.initialize()const callerRoot = path.join(root, 'users', 'caller-a')@@ -325,22 +331,40 @@ describe('legacy processing leases after #1019', () => {expect(result.is_error).toBe(true)})- it('S8: an interrupted discard leaves the ledger byte-identical and is retried by the next initialize', async () => {+ it('S8: a discard whose persist fails before the rename leaves the ledger byte-identical and is retried by the next initialize', async () => {await closedStore()const raw = await seedLegacyLeases([legacyLease()])const warn = vi.spyOn(logger, 'warn')const before = await counterValue(DISCARDED_COUNTER)const interrupted = freshStore()- const persist = vi- .spyOn(interrupted as unknown as { persist: () => Promise<void> }, 'persist')- .mockRejectedValueOnce(new Error('injected persist failure'))+ const privateStore = interrupted as unknown as {+ ledger: { processingLeases?: Record<string, unknown> }+ persist: () => Promise<void>+ reconcile: () => Promise<void>+ }+ let leasesAtFailedPersist: unknown = 'persist not reached'+ const persist = vi.spyOn(privateStore, 'persist').mockImplementationOnce(async () => {+ leasesAtFailedPersist = privateStore.ledger.processingLeases+ throw new Error('injected persist failure')+ })+ const reconcile = vi.spyOn(privateStore, 'reconcile')await expect(interrupted.initialize()).rejects.toThrow('injected persist failure')expect(persist).toHaveBeenCalledTimes(1)+ // The failing persist is the discard's own: the lease map was already+ // removed from memory when it ran, and initialize stopped before reconcile.+ expect(leasesAtFailedPersist).toBeUndefined()+ expect(reconcile).not.toHaveBeenCalled()+ expect(privateStore.ledger.processingLeases?.[LEGACY_LEASE_ID]).toBeDefined()expect(interrupted.isAvailable()).toBe(false)expect(await fs.readFile(ledgerFile(), 'utf8')).toBe(raw)expect(JSON.parse(raw).processingLeases[LEGACY_LEASE_ID]).toBeDefined()expect((await counterValue(DISCARDED_COUNTER)) - before).toBe(0)- expect(discardWarnings(warn.mock.calls)).toEqual([])+ expect(discardWarnings(warn.mock.calls)).toEqual([+ [+ { component: 'GfsDownloadStore', legacyProcessingLeases: 1, outcome: 'unknown' },+ 'GFS download store could not confirm the discard of 1 legacy processing lease(s) at initialize; the discard may already be visible on disk because the persist can fail after the ledger rename',+ ],+ ])const retried = freshStore()await retried.initialize()@@ -349,29 +373,82 @@ describe('legacy processing leases after #1019', () => {'processingLeases')expect((await counterValue(DISCARDED_COUNTER)) - before).toBe(1)- expect(discardWarnings(warn.mock.calls)).toHaveLength(1)+ expect(+ discardWarnings(warn.mock.calls).filter(+ call =>+ JSON.stringify(call[0]) ===+ JSON.stringify({ component: 'GfsDownloadStore', discarded: 1 })+ )+ ).toHaveLength(1)})- it('S9: counts and logs records an earlier boot quarantined', async () => {+ it('S8b: a discard whose directory sync fails after the rename warns with an unknown outcome and does not count the discard', async () => {+ await closedStore()+ await seedLegacyLeases([legacyLease()])+ const storeDirectory = await fs.stat(path.join(root, '.gfs-download-store'))+ const probe = await fs.open(ledgerFile(), 'r')+ const prototype = Object.getPrototypeOf(probe) as FileHandle+ await probe.close()+ const originalSync = prototype.sync+ let injected = 0+ // Only the store-directory fsync that follows the ledger rename fails: the+ // writer lease also syncs this directory, but before the discard the+ // visible ledger still carries the legacy lease.+ vi.spyOn(prototype, 'sync').mockImplementation(async function (this: FileHandle) {+ const info = await this.stat()+ if (injected === 0 && info.isDirectory() && info.ino === storeDirectory.ino) {+ const visible = JSON.parse(await fs.readFile(ledgerFile(), 'utf8'))+ if (!('processingLeases' in visible)) {+ injected += 1+ throw Object.assign(new Error('injected directory sync failure'), { code: 'EIO' })+ }+ }+ return originalSync.call(this)+ })+ const warn = vi.spyOn(logger, 'warn')+ const before = await counterValue(DISCARDED_COUNTER)+ const interrupted = freshStore()+ await expect(interrupted.initialize()).rejects.toThrow('injected directory sync failure')+ expect(injected).toBe(1)+ expect(interrupted.isAvailable()).toBe(false)+ // Witness that the rename happened: the visible ledger is already lease-free.+ expect(JSON.parse(await fs.readFile(ledgerFile(), 'utf8'))).not.toHaveProperty(+ 'processingLeases'+ )+ expect(discardWarnings(warn.mock.calls)).toEqual([+ [+ { component: 'GfsDownloadStore', legacyProcessingLeases: 1, outcome: 'unknown' },+ 'GFS download store could not confirm the discard of 1 legacy processing lease(s) at initialize; the discard may already be visible on disk because the persist can fail after the ledger rename',+ ],+ ])+ expect((await counterValue(DISCARDED_COUNTER)) - before).toBe(0)+ })++ it('S9: reports the current quarantined-record count after reconcile without double counting across boots', async () => {const { child, message } = await childStore('transfer')expect(message.transferId).toMatch(/^[0-9a-f-]{36}$/)await kill(child)const warn = vi.spyOn(logger, 'warn')- const before = await counterValue(QUARANTINED_COUNTER)+ const quarantineWarnings = () =>+ warn.mock.calls.filter(+ call =>+ JSON.stringify(call[0]) ===+ JSON.stringify({ component: 'GfsDownloadStore', quarantined: 1 })+ )const recovering = freshStore()await recovering.initialize()- // This boot quarantines the interrupted transfer; it inherited none.+ // The boot that quarantines the interrupted transfer reports it.expect(recovering.debugRecord(message.transferId)?.state).toBe('quarantined')- expect((await counterValue(QUARANTINED_COUNTER)) - before).toBe(0)+ expect(await counterValue(QUARANTINED_GAUGE)).toBe(1)+ expect(quarantineWarnings()).toHaveLength(1)await recovering.close()const next = freshStore()await next.initialize()- expect((await counterValue(QUARANTINED_COUNTER)) - before).toBe(1)- expect(- warn.mock.calls.filter(call => JSON.stringify(call[0]) === JSON.stringify({ quarantined: 1 }))- ).toHaveLength(1)+ // The same backlog on the next boot keeps the gauge at 1, not 2.expect(next.debugRecord(message.transferId)?.state).toBe('quarantined')+ expect(await counterValue(QUARANTINED_GAUGE)).toBe(1)+ expect(quarantineWarnings()).toHaveLength(2)expect(next.debugUsage().bytes).toBe(7)})})diff --git a/mcp-host/src/internalTools/gfsDownloadStoreReservationRollback.test.ts b/mcp-host/src/internalTools/gfsDownloadStoreReservationRollback.test.tsnew file mode 100644index 000000000..bc6ba6cce--- /dev/null+++ b/mcp-host/src/internalTools/gfsDownloadStoreReservationRollback.test.ts@@ -0,0 +1,147 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'+import { createHash } from 'node:crypto'+import * as fs from 'node:fs/promises'+import { tmpdir } from 'node:os'+import * as path from 'node:path'+import type { GfsImageSource } from '../visualInput/policy'+import { GfsDownloadStore } from './gfsDownloadStore'++/**+ * The reservation persist in createTransfer is the only durable write before a+ * transfer is exposed. When it fails, the in-memory record and the retention+ * owner roll back, so no later persist (close() included) writes a phantom+ * reservation that the next boot would quarantine and keep charged to quota.+ */++const source: GfsImageSource = {+ kind: 'gfs',+ drive: 'main',+ resourceId: 'a'.repeat(32),+ gfsUri: `gfs://main/${'a'.repeat(32)}`,+ name: 'input.csv',+ version: 7,+}+const otherSource: GfsImageSource = {+ ...source,+ resourceId: 'b'.repeat(32),+ gfsUri: `gfs://main/${'b'.repeat(32)}`,+ name: 'other.csv',+}+const OWNER_ID = '55555555-5555-4555-8555-555555555555'++let root: string+let callerRoot: string+let store: GfsDownloadStore++beforeEach(async () => {+ root = await fs.mkdtemp(path.join(tmpdir(), 'gfs-reservation-rollback-'))+ callerRoot = path.join(root, 'users', 'caller-a')+ await fs.mkdir(callerRoot, { recursive: true, mode: 0o700 })+ store = new GfsDownloadStore(root)+ await store.initialize()+})++afterEach(async () => {+ vi.restoreAllMocks()+ await store.close(0).catch(() => undefined)+ await fs.rm(root, { recursive: true, force: true })+})++async function ledgerOnDisk(): Promise<{+ records: Record<string, unknown>+ retentionOwners?: Record<string, { recordIds: string[] }>+}> {+ return JSON.parse(+ await fs.readFile(path.join(root, '.gfs-download-store', 'ledger-v1.json'), 'utf8')+ )+}++function privateLedger(): {+ records: Record<string, unknown>+ retentionOwners?: Record<string, { recordIds: string[] }>+} {+ return (store as unknown as { ledger: ReturnType<typeof privateLedger> }).ledger+}++function failNextPersist() {+ return vi+ .spyOn(store as unknown as { persist: () => Promise<void> }, 'persist')+ .mockRejectedValueOnce(new Error('reservation unavailable'))+}++describe('createTransfer reservation persist failure', () => {+ it('rolls back the reservation and a new retention owner, and close() persists no phantom record', async () => {+ const before = await ledgerOnDisk()+ const persist = failNextPersist()+ await expect(+ store.createTransfer({+ callerIdentity: 'caller-a',+ callerWorkspacePath: callerRoot,+ source,+ sizeBytes: 7,+ expiresAt: new Date(Date.now() + 60_000).toISOString(),+ retentionOwnerId: OWNER_ID,+ })+ ).rejects.toThrow('reservation unavailable')+ // Witness: the rejection came from the reservation persist itself.+ expect(persist).toHaveBeenCalledOnce()+ expect(store.debugUsage()).toMatchObject({ bytes: 0, files: 0 })+ expect(privateLedger().records).toEqual({})+ expect(privateLedger().retentionOwners?.[OWNER_ID]).toBeUndefined()+ expect(await ledgerOnDisk()).toEqual(before)+ await expect(fs.lstat(path.join(callerRoot, '.gfs-downloads'))).rejects.toMatchObject({+ code: 'ENOENT',+ })++ await store.close(0)+ const closed = await ledgerOnDisk()+ expect(closed.records).toEqual({})+ expect(closed.retentionOwners?.[OWNER_ID]).toBeUndefined()+ })++ it('restores an existing retention owner and close() persists only the published record', async () => {+ const transfer = await store.createTransfer({+ callerIdentity: 'caller-a',+ callerWorkspacePath: callerRoot,+ source,+ sizeBytes: 7,+ expiresAt: new Date(Date.now() + 60_000).toISOString(),+ retentionOwnerId: OWNER_ID,+ })+ await fs.writeFile(path.join(callerRoot, transfer.partialPath), 'fixture')+ const published = await store.publish(+ transfer.id,+ 'caller-a',+ createHash('sha256').update('fixture').digest('hex')+ )+ const ownerBefore = structuredClone(privateLedger().retentionOwners?.[OWNER_ID])+ expect(ownerBefore?.recordIds).toEqual([published.id])+ const before = await ledgerOnDisk()++ const persist = failNextPersist()+ await expect(+ store.createTransfer({+ callerIdentity: 'caller-a',+ callerWorkspacePath: callerRoot,+ source: otherSource,+ sizeBytes: 7,+ expiresAt: new Date(Date.now() + 60_000).toISOString(),+ retentionOwnerId: OWNER_ID,+ })+ ).rejects.toThrow('reservation unavailable')+ expect(persist).toHaveBeenCalledOnce()+ expect(store.debugUsage()).toMatchObject({ bytes: 7, files: 1 })+ expect(Object.keys(privateLedger().records)).toEqual([published.id])+ expect(privateLedger().retentionOwners?.[OWNER_ID]).toEqual(ownerBefore)+ expect(await ledgerOnDisk()).toEqual(before)+ expect(await fs.readdir(path.join(callerRoot, '.gfs-downloads'))).toEqual([+ path.basename(path.dirname(published.path)),+ ])++ await store.releaseReceiptOwner(OWNER_ID, 'caller-a')+ await store.close(0)+ const closed = await ledgerOnDisk()+ expect(Object.keys(closed.records)).toEqual([published.id])+ expect(closed.retentionOwners?.[OWNER_ID]).toBeUndefined()+ })+})diff --git a/scripts/ci/pinned-vitest-cases/mcp-host.json b/scripts/ci/pinned-vitest-cases/mcp-host.jsonindex 706f5489e..af3851d54 100644--- a/scripts/ci/pinned-vitest-cases/mcp-host.json+++ b/scripts/ci/pinned-vitest-cases/mcp-host.json@@ -70,5 +70,41 @@{"suite": "codexLlmProxyClient.test.ts","fullName": "CodexLlmProxyClient keeps wire malformed_json separate from trusted local refusal provenance"+ },+ {+ "suite": "taskExecutor.gfsExecutionSafety.test.ts",+ "fullName": "X1-shell: an inherited legacy lease does not stop an approved shell (#1019)"+ },+ {+ "suite": "taskExecutor.gfsExecutionSafety.test.ts",+ "fullName": "X1-delivery: an inherited legacy lease is discarded and GFS delivery stays advertised (#1019)"+ },+ {+ "suite": "taskExecutor.gfsExecutionSafety.test.ts",+ "fullName": "U5-TE: TaskExecutor runs a managed shell with a timeout above the former lease ceiling (#1021) exactly one hour"+ },+ {+ "suite": "taskExecutor.gfsExecutionSafety.test.ts",+ "fullName": "U5-TE: TaskExecutor runs a managed shell with a timeout above the former lease ceiling (#1021) one millisecond over one hour"+ },+ {+ "suite": "taskExecutor.gfsExecutionSafety.test.ts",+ "fullName": "X3-TE: a shell dispatch inside TaskExecutor makes zero GFS download store calls (#1019)"+ },+ {+ "suite": "gfsDownloadStoreRecovery.test.ts",+ "fullName": "legacy processing leases after #1019 S1: discards an expired foreign-session lease left by a crashed Host and admits downloads"+ },+ {+ "suite": "gfsDownloadStoreRecovery.test.ts",+ "fullName": "legacy processing leases after #1019 S8: a discard whose persist fails before the rename leaves the ledger byte-identical and is retried by the next initialize"+ },+ {+ "suite": "gfsDownloadStoreRecovery.test.ts",+ "fullName": "legacy processing leases after #1019 S8b: a discard whose directory sync fails after the rename warns with an unknown outcome and does not count the discard"+ },+ {+ "suite": "gfsDownloadStoreRecovery.test.ts",+ "fullName": "legacy processing leases after #1019 S9: reports the current quarantined-record count after reconcile without double counting across boots"}]