The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
fix(runner): preserve tool outcomes through shutdown and restart (#14734)
Dotta94e8dec56b35
diff --git a/doc/run-log-events.md b/doc/run-log-events.mdindex 402f3e7ef8..7b0abaa8b5 100644--- a/doc/run-log-events.md+++ b/doc/run-log-events.md@@ -34,6 +34,47 @@ credential material are never written to the run log.These records remain run-log events. They do not create an OpenTelemetry orPaperclip Telemetry export, and legacy adapters do not use this writer.+## Semantic Settlement Diagnostics and Retry Logs++Before interrupting a Codex turn, the runner durably records a+`harness.diagnostic` event with code `provider_interrupt_requested`, the stop+reason, provider turn ID, and pending tool call IDs. A harmless identical result+replay records `semantic_tool_result_duplicate` with warning severity, call ID,+operation ID, and result digest. It does not fail the task or create a user+attention request. A true conflict includes the call and operation IDs and both+result digests in its error; full arguments belong to the canonical semantic+input record, not the error message.++Incomplete close emits the bounded `native_session_settlement_incomplete`+runner diagnostic. Its evidence includes runner suspension, provider drain,+final provider state, pending call/operation/source-event IDs and input digests,+and incomplete result-delivery command IDs and statuses. If execution and+cleanup both fail, execution retains its original error identity and cleanup is+attached as `cleanupError`.++Instruction writes also commit an `agent.instruction_write_attempted` activity+row and a run-scoped `instructionToolAttempts` entry before permitting the+filesystem effect. They retain the call ID, operation ID, and input digest, not+instruction text. A later transaction rollback cannot erase this attempt proof.+A missing success or definite pre-write failure receipt means the outcome is+unknown, even if the current file contains the requested text. Known validation+and stale-base failures are saved under the attempt's `failure` entry and replay+their original status, message, and details without a new write. Unknown+outcomes remain blocked until reconciled; a+committed receipt with a lost acknowledgement can replay its exact result.++The NDJSON output log appends across repeated `begin` calls for one run. Each+new handle adds an `attemptId` to its lines. If the local file is absent and a+durable S3 mirror exists, `begin` restores that prefix before appending. A+failed restore must not replace the mirror with an empty or partial attempt.+Publishing a restored prefix uses an atomic create-if-absent operation, so a+concurrent restore cannot overwrite lines another attempt has already appended.+Earlier attempts therefore remain available for incident diagnosis. Existing+records without `attemptId` remain readable.++These records use the instance run log and its configured storage. They add no+Paperclip Telemetry or OpenTelemetry export.+## Omitted Unsafe Workspace Export`workspace_export_omitted` is an informational system event in the local run log.diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.mdindex 857de7b712..8ee0b22c97 100644--- a/packages/paperclip-runner/docs/durable-recovery.md+++ b/packages/paperclip-runner/docs/durable-recovery.md@@ -185,6 +185,105 @@ The trace records `logicalEffectCount`. Every completed command has exactly onelogical effect. A policy rejection, such as a new turn during drain or storagepressure, has zero effects.+## Semantic operations that outlive a turn++An execution session includes the provider process, the runner's durable call+receipts and event outbox, and the controller's durable command queue. Ending a+provider turn closes admission of new work. It does not establish whether a+semantic operation already dispatched to the server succeeded or failed.++The tool bridge retains pending call IDs, operation IDs, and exact arguments+across that boundary. It accepts the authority's late result without delivering+it to an obsolete provider turn. An identical result is an idempotent replay;+a different result for that call remains a conflict. Codex records harmless+result replay as `semantic_tool_result_duplicate` in a diagnostic event, with+no task failure. Conflict errors identify the call, operation, and both result+digests without including the result bodies.++The controller checkpoints a reusable session only after every admitted tool+has a matching completed result-delivery command and the provider has drained.+A callback rejected before business dispatch can report+`semantic_tool_not_dispatched`. A callback rejected after dispatch has an+unknown outcome; the controller must not invent a failure or execute it again.+After controller restart, an already committed input without a result remains+unsettled. It requires authoritative reconciliation, not an automatic retry.+A different case is a runner crash after the controller saved the result but+before its delivery command completed. Codex/OpenCode explicitly opt in to+reconciling that exact `semantic_tool.result` command through their durable tool+receipt. The full command fingerprint must match. This delivers the saved+answer without executing the business operation. Unsupported providers and all+other indeterminate commands remain non-reexecutable. This change does not+repair legacy journals already containing contradictory or failed receipts.++`update_agent_instructions` and `restore_agent_instructions` use a durable,+company/run-scoped mutation receipt keyed by call ID. Before attempting a file+write, the authority commits an `instructionToolAttempts` record and an+`agent.instruction_write_attempted` activity row with the call ID, operation ID,+and input digest. This evidence survives a later filesystem or database failure.+Completed calls and retries after authority restart return the exact original receipt.+Changing the arguments under that ID is an idempotency conflict. Receipt replay+still checks current instruction-write authorization. Concurrent callers in the+same process wait for the first handler before checking the receipt again.++The filesystem and PostgreSQL are not one atomic store. If a write is visible+but its receipt transaction rolls back, the attempt record remains durable and+the result is unknown. An identical call with an attempt but no committed result+raises `paperclip_runner_instruction_outcome_unknown`; it does not write again+or pretend to have succeeded. This also applies to another process reaching a+reserved call before its effect transaction starts. The controller retains an+unsettled operation for authoritative reconciliation. If only the commit+acknowledgement was lost, the committed receipt is replayed normally.+The native tool wrapper propagates `SemanticToolOutcomeUnknownError` instead+of turning it into a completed failed tool response. Ordinary validation,+authorization, and stale-base errors still return normal tool errors. Before+returning a definite pre-write failure, the authority saves its status, message,+and details under the attempt's `failure` receipt. Replays return that original+error without executing again, even if the old CAS base becomes current again.+Failure-receipt storage errors retain an unsettled operation instead of exposing+an unrecorded final answer.++### Lossless arguments and bounded transport++Execution arguments do not use the diagnostic preview formatter. Accepted+input remains exact, including Unicode, line endings, and long document tails.+The credential policy rejects input it would have to alter before dispatch.+Diagnostic copies remain redacted and bounded. The semantic input limit is+480 KiB of encoded JSON, leaving room for the encrypted frame's hex expansion+and envelope inside the 1 MiB wire limit. Oversized input fails before a write;+it is never truncated into an apparently successful write.++A result command retains a SHA-256 `inputDigest` instead of duplicating the+full input alongside the full result. The canonical input event retains the+arguments. Settlement validates the digest and all run, event, call, operation,+and correlation identities. Legacy result commands carrying the full input+remain supported. Large results still have to fit the command transport limit;+exceeding it leaves the outcome unsettled rather than retrying an effect.++### Deterministic fault coverage++These tests use explicit barriers, durable reloads, real runner processes with+scripted providers, and a real PostgreSQL database. They do not require a paid+model or rely on an LLM choosing the desired timing.++| Fault | Required assertion | Regression suite |+| --- | --- | --- |+| Stop before/after server result; restart before late delivery | No invented failure; pending identity survives; late result accepted | Rust `provider_bridge`, `codex_provider` |+| Provider terminal before tool completion | New turn blocked until actual result; safe replay and later reuse | Rust `acpx_provider_turns`, `acpx_provider_state` |+| Crash before/after applying a saved delivery receipt, before command completion | Exact result delivery recovers only for opted-in providers; changed commands and ordinary operations never replay | Rust durable runner tests |+| Lost result acknowledgement; conflicting redelivery | Identical result accepted; changed result rejected with IDs and digests | Rust `codex_provider`, `provider_bridge` |+| Concurrent writes held at filesystem commit; authority restart | One write, one audit record, same receipt; revoked authorization rejected | Server `agent-instruction-tools.integration.test.ts` |+| Failure after file rename or rollback while saving the receipt | Durable attempt and audit evidence survive; no success claim and no second write | Server `agent-instruction-tools.integration.test.ts` |+| Receipt commits but its database acknowledgement is lost | Exact original result replays; one file-update audit and one attempt record | Server `agent-instruction-tools.integration.test.ts` |+| Tool handler throws an unknown-outcome error versus a validation error | Unknown propagates without a completed tool event; validation returns an ordinary failed response | `codex-app-server-driver.test.ts` |+| Invalid arguments, oversized input, or stale base; restart and replay | Original failure replays with zero writes, even after the base becomes valid | Server `agent-instruction-tools.integration.test.ts` |+| Failure receipt cannot be stored | No unrecorded final failure; attempt remains unsettled and never reexecutes | Server `agent-instruction-tools.integration.test.ts` |+| Long Unicode input and result; corrupted input digest | Exact bytes across persistence/wire; no copied input in result; bad proof blocks settlement | Rust durable-state tests; `durable-prp-control-plane.test.ts` |+| Handler finishes during close, after close budget, or persistence fails | Checkpoint marked settled only with completed delivery and drained provider | `runnerd-codex-transport.test.ts` |+| Controller restarts with committed input but no result | No second dispatch and no fabricated outcome | `durable-prp-control-plane.test.ts` |+| Execution and cleanup both fail | Original execution error retained; cleanup error attached | `native-session-runtime.test.ts` |+| Retry in same process or after local log loss | Earlier attempts survive locally and in the durable mirror | Server `run-log-store.test.ts` |+| Two restores overlap and one appends before the other publishes | Complete prefix is published without replacement; both attempts survive | Server `run-log-store.test.ts` |+## Residual local trust and revocation windowThe authenticated session removes trust in whichever process wins the configureddiff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rsindex c92f044940..a50de43ef6 100644--- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs@@ -355,6 +355,11 @@ impl AcpxProviderSession {"ACPX provider session already has an active turn",));}+ if self.state.has_pending_tools() {+ return Err(LocalRunnerError::invalid(+ "ACPX provider session still has unsettled semantic tools",+ ));+ }let rotate_turn_identity_ledger = self.state.settled_turn_identity_capacity_reached();let identity_validation = if rotate_turn_identity_ledger {self.state@@ -618,47 +623,38 @@ impl AcpxProviderSession {}}AcpxProviderStateEvent::TurnTerminal { .. } => {- let settlements = match next_bridge.settle_turn("acpx_turn_settled") {- Ok(settlements) => settlements,- Err(error) => {- return Err(self.fail_closed(LocalRunnerError::invalid(format!(+ next_bridge+ .settle_turn("acpx_turn_settled")+ .map_err(|error| {+ self.fail_closed(LocalRunnerError::invalid(format!("ACPX provider tool settlement failed: {error}"- ))));- }- };- let reserved_settlements = match next_reserved_bridge- .settle_turn("acpx_reserved_terminal_unsettled")- {- Ok(settlements) => settlements,- Err(error) => {- return Err(self.fail_closed(LocalRunnerError::invalid(format!(- "ACPX reserved terminal settlement failed: {error}"- ))));- }- };- if !reserved_settlements.is_empty() {- return Err(self.fail_closed(LocalRunnerError::invalid(- "ACPX turn terminated before its reserved terminal invocation produced a correlated result",- )));- }- // `accept_event` clears the candidate reducer's pending- // tools while the bridge clones settle the corresponding- // calls above. Prove both halves reached the same terminal- // state before committing any of them to the reusable- // session.- if next_state.has_pending_tools()- || next_bridge.pending_calls().next().is_some()- || next_reserved_bridge.pending_calls().next().is_some()+ )))+ })?;+ next_reserved_bridge+ .settle_turn("acpx_turn_settled")+ .map_err(|error| {+ self.fail_closed(LocalRunnerError::invalid(format!(+ "ACPX reserved tool settlement failed: {error}"+ )))+ })?;+ // Both ledgers must retain exactly the same dispatched+ // effects. Ending the provider turn cannot determine whether+ // a server-side operation committed.+ let pending: Vec<_> = next_bridge+ .pending_calls()+ .chain(next_reserved_bridge.pending_calls())+ .collect();+ if pending.len() != next_state.pending_tool_count()+ || pending.iter().any(|call| {+ next_state.pending_tool(&call.call_id).is_none_or(|other| {+ other.operation_id != call.operation_id || other.input != call.input+ })+ }){return Err(self.fail_closed(LocalRunnerError::invalid("ACPX terminal settlement left provider tool state inconsistent",)));}- reconciled_events.extend(- settlements- .into_iter()- .map(AcpxProviderStateEvent::ToolResult),- );}AcpxProviderStateEvent::PermissionRequest { .. } => {if self.config.agent == "codex"@@ -685,9 +681,11 @@ impl AcpxProviderSession {}pub fn deliver_tool_result(&mut self, result: &ToolResult) -> Result<(), LocalRunnerError> {- let turn_id = self.ensure_active_turn()?.to_owned();+ self.ensure_open()?;let mut next_state = self.state.clone();- next_state.complete_tool(&result.call_id, &result.operation_id)?;+ if next_state.pending_tool(&result.call_id).is_some() {+ next_state.complete_tool(&result.call_id, &result.operation_id)?;+ }let mut next_bridge = self.tool_bridge.clone();let mut next_reserved_bridge = self.reserved_tool_bridge.clone();let bridge = if is_reserved_terminal_operation(&result.operation_id) {@@ -695,9 +693,24 @@ impl AcpxProviderSession {} else {&mut next_bridge};+ let duplicate = bridge.has_completed_call(&result.call_id);+ let detached = bridge.turn_closed();bridge.apply_result(result.clone()).map_err(|error| {- LocalRunnerError::invalid(format!("ACPX tool result is invalid: {error}"))+ LocalRunnerError::invalid(format!(+ "ACPX tool result for call {} operation {} is invalid: {error}",+ result.call_id, result.operation_id+ ))})?;+ if duplicate {+ return Ok(());+ }+ if detached {+ self.state = next_state;+ self.tool_bridge = next_bridge;+ self.reserved_tool_bridge = next_reserved_bridge;+ return Ok(());+ }+ let turn_id = self.ensure_active_turn()?.to_owned();let resolution = if result.is_error {// The durable result remains authoritative for correlation and// retry bookkeeping, but provider-facing failures expose only adiff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_state.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_state.rsindex 876cf9d00f..ceef328395 100644--- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_state.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_state.rs@@ -365,7 +365,8 @@ impl AcpxProviderState {.to_owned();self.scope.clear_turn(&turn_id)?;let mut events = self.end_pending_runtime_requests(status);- self.clear_pending_requests();+ // Dispatched semantic effects outlive the provider turn. Their+ // authority results, not this terminal event, retire them.if status == AcpxTurnStatus::Completed && !self.assistant_text.is_empty() {events.push(AcpxProviderStateEvent::AssistantMessage {turn_id: turn_id.clone(),@@ -646,12 +647,8 @@ impl AcpxProviderState {events}- fn clear_pending_requests(&mut self) {- self.pending_tools.clear();- self.pending_tool_input_bytes = 0;- self.pending_permissions.clear();- self.pending_inputs.clear();- self.pending_runtime_request_bytes = 0;+ pub(crate) fn pending_tool_count(&self) -> usize {+ self.pending_tools.len()}}diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs b/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rsindex 0a1d9bdf5c..9bcb817a83 100644--- a/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs@@ -251,6 +251,13 @@ pub enum TerminalDeliveryReconciliation {pub trait CommandExecutor {fn execute(&mut self, command: &Command) -> Result<CommandExecution, DurableRunnerError>;+ /// Opt in only when semantic result delivery validates an exact durable+ /// call receipt and cannot dispatch the underlying business operation.+ /// Ordinary commands and providers without this proof remain indeterminate.+ fn can_reconcile_result_delivery(&mut self) -> Result<bool, DurableRunnerError> {+ Ok(false)+ }+/// Advances provider-side event correlation after a durable `run.attach`/// has moved runnerd to the next run-bound authority. The runner validates/// and persists the new authority before invoking this infallible hook.@@ -1724,6 +1731,15 @@ fn process_command<E: CommandExecutor>(}}match state.begin_command(command)? {+ CommandDisposition::Replay(result)+ if result.status == "indeterminate"+ && command.command_type == "semantic_tool.result"+ && executor.can_reconcile_result_delivery()? =>+ {+ // begin_command already checked the complete immutable command+ // fingerprint. Only replay receipt delivery, never the operation.+ state.resume_result_delivery(command)?;+ }CommandDisposition::Replay(result) => {let lifecycle =if result.status == "pending" || state.pending_provider_cleanup.is_some() {@@ -1742,8 +1758,8 @@ fn process_command<E: CommandExecutor>(CommandDisposition::Execute => {}}// Persist the pending marker before any command effect. If the process dies- // in the effect window, recovery returns an indeterminate result and never- // executes the same logical command twice.+ // in the effect window, recovery remains indeterminate unless the provider+ // explicitly supports exact, idempotent semantic result delivery above.store.save(state)?;let mut execution = match executor.execute(command) {Ok(execution) => execution,@@ -4485,6 +4501,150 @@ mod tests {}}+ #[test]+ fn interrupted_result_delivery_reconciles_exact_receipts_without_reexecuting_operations() {+ use crate::provider_bridge::{+ authorized_tool_catalog_digest, AuthorizedTool, AuthorizedToolSet, ProviderToolBridge,+ ToolResult,+ };+ struct ReceiptExecutor {+ path: PathBuf,+ deliveries: usize,+ supported: bool,+ }+ impl CommandExecutor for ReceiptExecutor {+ fn can_reconcile_result_delivery(&mut self) -> Result<bool, DurableRunnerError> {+ Ok(self.supported)+ }+ fn execute(+ &mut self,+ command: &Command,+ ) -> Result<CommandExecution, DurableRunnerError> {+ assert_eq!(+ command.command_type, "semantic_tool.result",+ "must never redispatch an operation"+ );+ self.deliveries += 1;+ let mut bridge: ProviderToolBridge =+ serde_json::from_slice(&fs::read(&self.path).unwrap()).unwrap();+ let result: ToolResult = serde_json::from_value(command.payload.clone()).unwrap();+ bridge+ .apply_result(result)+ .map_err(|error| DurableRunnerError::invalid(error.to_string()))?;+ fs::write(&self.path, serde_json::to_vec(&bridge).unwrap()).unwrap();+ Ok(CommandExecution::result(json!({"status":"delivered"})))+ }+ }+ for applied in [false, true] {+ for supported in [false, true] {+ let directory = std::env::temp_dir().join(format!(+ "result-delivery-crash-{}-{applied}-{supported}",+ std::process::id()+ ));+ let _ = fs::remove_dir_all(&directory);+ let config = config(directory.clone());+ let store = DurableStateStore::new(&directory).unwrap();+ let (mut state, _) = store.load_or_create(&config).unwrap();+ let operations = vec![AuthorizedTool {+ operation_id: "write_document".into(),+ version: 1,+ description: "Fixture write already committed on the server".into(),+ input_schema: json!({"type":"object"}),+ response_schema: json!({"type":"object"}),+ }];+ let mut bridge = ProviderToolBridge::default();+ bridge+ .prepare(AuthorizedToolSet {+ schema: "paperclip.runner.authorized-tools.v1".into(),+ schema_version: 1,+ catalog_digest: authorized_tool_catalog_digest(&operations).unwrap(),+ operations,+ })+ .unwrap();+ bridge+ .begin_call(+ "call-1".into(),+ "write_document".into(),+ json!({"content":"exact original"}),+ )+ .unwrap();+ bridge.settle_turn("provider_turn_stopped").unwrap();+ let path = directory.join("tool-receipts.json");+ fs::write(&path, serde_json::to_vec(&bridge).unwrap()).unwrap();+ let mut executor = ReceiptExecutor {+ path: path.clone(),+ deliveries: 0,+ supported,+ };+ let mut delivery = command("semantic_tool.result");+ delivery.payload = json!({"callId":"call-1", "operationId":"write_document", "isError":false,+ "result":{"revisionId":"one-committed-write"}});+ assert_eq!(+ state.begin_command(&delivery).unwrap(),+ CommandDisposition::Execute+ );+ store.save(&state).unwrap();+ if applied {+ executor.execute(&delivery).unwrap();+ }+ // Crash before receipt application, or after receipt application+ // but before the command completion journal commits.+ let (mut recovered, _) = store.load_or_create(&config).unwrap();+ let before = executor.deliveries;+ let (result, _) =+ process_command(&mut recovered, &store, &config, &mut executor, &delivery)+ .unwrap();+ assert_eq!(+ result.status,+ if supported {+ "completed"+ } else {+ "indeterminate"+ }+ );+ assert_eq!(executor.deliveries, before + usize::from(supported));+ let saved: ProviderToolBridge =+ serde_json::from_slice(&fs::read(&path).unwrap()).unwrap();+ assert_eq!(+ saved.pending_calls().count(),+ usize::from(!applied && !supported)+ );+ let (mut reopened, _) = store.load_or_create(&config).unwrap();+ assert_eq!(+ process_command(&mut reopened, &store, &config, &mut executor, &delivery)+ .unwrap()+ .0,+ result+ );+ assert_eq!(executor.deliveries, before + usize::from(supported));+ let mut changed = delivery.clone();+ changed.payload["result"]["revisionId"] = json!("conflicting-write");+ assert!(+ process_command(&mut reopened, &store, &config, &mut executor, &changed)+ .unwrap_err()+ .to_string()+ .contains("different command data")+ );+ assert_eq!(executor.deliveries, before + usize::from(supported));+ let mut ordinary = command("session.open");+ ordinary.command_id = "ordinary-effect".into();+ ordinary.controller_seq = 2;+ reopened.begin_command(&ordinary).unwrap();+ store.save(&reopened).unwrap();+ let (mut after_crash, _) = store.load_or_create(&config).unwrap();+ assert_eq!(+ process_command(&mut after_crash, &store, &config, &mut executor, &ordinary)+ .unwrap()+ .0+ .status,+ "indeterminate"+ );+ assert_eq!(executor.deliveries, before + usize::from(supported));+ fs::remove_dir_all(directory).unwrap();+ }+ }+ }+#[test]fn process_death_after_journaling_remains_indeterminate_without_reexecution() {let directory = std::env::temp_dir().join(format!(diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/durable/state.rs b/packages/paperclip-runner/runner/crates/runner-core/src/durable/state.rsindex 9d160c5ebe..2a41066875 100644--- a/packages/paperclip-runner/runner/crates/runner-core/src/durable/state.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/src/durable/state.rs@@ -775,6 +775,28 @@ impl DurableState {Ok(CommandDisposition::Execute)}+ pub(crate) fn resume_result_delivery(+ &mut self,+ command: &Command,+ ) -> Result<(), DurableRunnerError> {+ if command.command_type != "semantic_tool.result"+ || !matches!(self.begin_command(command)?, CommandDisposition::Replay(ref prior) if prior.status == "indeterminate")+ {+ return Err(DurableRunnerError::invalid(+ "only exact indeterminate result delivery can resume",+ ));+ }+ self.processed_commands+ .get_mut(&command.command_id)+ .expect("validated retained delivery receipt")+ .status = "pending".to_owned();+ self.record_diagnostic(format!(+ "reconciling semantic result delivery command {}",+ command.command_id+ ));+ Ok(())+ }+pub fn complete_command(&mut self,command: &Command,@@ -1671,52 +1693,55 @@ pub(crate) fn sanitize_semantic_tool_input(operation_id: &str,input: &Value,) -> Result<Value, DurableRunnerError> {- let mut sanitized = sanitize_value(input);- // Mutation prose is the user's intended work, not a diagnostic. Preserve- // ordinary references to a token in these declared text fields; credential- // syntax and high-confidence secret values are still scrubbed. All other- // fields and operations retain the strict diagnostic policy.+ // This value is executed by the authority. Diagnostic preview limits and+ // redaction must never turn it into a different, apparently valid write.+ // Hex-encoded authenticated frames double JSON bytes. Reserve envelope+ // space below the 1 MiB wire cap before any effect can start.+ // Enforce the transport envelope before dispatch, and reject content that+ // cannot cross its credential policy instead of silently rewriting it.+ if serde_json::to_vec(input)+ .map_err(|_| DurableRunnerError::invalid("semantic tool input is not serializable"))?+ .len()+ > 480 * 1024+ {+ return Err(DurableRunnerError::invalid(+ "semantic tool input exceeds the 480 KiB encrypted transport limit",+ ));+ }+ let mut checked = sanitize_bounded_display_value(input);let prose_fields: &[&str] = match operation_id {"create_task" => &["title", "description", "initialPlan"],"create_project" => &["name", "description"],"write_document" => &["title", "body", "changeSummary"],+ "update_agent_instructions" => &["content"],_ => &[],};- if let Some(sanitized_input) = sanitized.as_object_mut() {+ if let Some(object) = checked.as_object_mut() {for field in prose_fields {if let Some(text) = input.get(*field).and_then(Value::as_str) {- sanitized_input.insert(+ object.insert((*field).to_owned(),- // The tool/API schema bounds business content. A diagnostic- // preview limit must never truncate a plan or document.Value::String(redact_sensitive_text_values_with_context(text, true)),);}}}- if !matches!(operation_id, "paperclip_finish" | "paperclip_block") {- return Ok(sanitized);- }- let Some(summary) = input.get("summary").and_then(Value::as_str) else {- return Ok(sanitized);- };- if summary.chars().count() > MAX_COMPLETION_SUMMARY_CHARS {+ if matches!(operation_id, "paperclip_finish" | "paperclip_block")+ && input+ .get("summary")+ .and_then(Value::as_str)+ .is_some_and(|summary| summary.chars().count() > MAX_COMPLETION_SUMMARY_CHARS)+ {return Err(DurableRunnerError::invalid("semantic completion summary exceeds the 12,000 character limit",));}- let Some(sanitized_input) = sanitized.as_object_mut() else {- return Ok(sanitized);- };- // Completion summary is the schema-bounded user-facing answer, not an- // untrusted diagnostic snippet. Preserve it in full while applying the- // same credential scrubber used by every durable string. All other fields- // retain the generic 4 KiB diagnostic bound.- sanitized_input.insert(- "summary".to_owned(),- Value::String(redact_sensitive_text_values(summary)),- );- Ok(sanitized)+ if checked != *input {+ return Err(DurableRunnerError::invalid(+ "semantic tool input contains credential material; refusing to execute altered arguments",+ ));+ }+ Ok(input.clone())}fn finalize_semantic_tool_input_payload(@@ -2944,7 +2969,7 @@ mod tests {let config = config(PathBuf::from("unused"));let mut state = DurableState::new(&config);let summary = format!(- "token=do-not-persist {} Authorization: Bearer late-provider-secret COMPLETE-DURABLE-SUMMARY",+ "Answer: {} COMPLETE-DURABLE-SUMMARY","A complete paragraph for the user. ".repeat(180));let input = json!({"summary": summary});@@ -2988,11 +3013,11 @@ mod tests {.pointer("/payload/payload/semantic_tool/input").unwrap();let transmitted_summary = transmitted["summary"].as_str().unwrap();- assert!(transmitted_summary.starts_with("token=[REDACTED] "));+ assert!(transmitted_summary.starts_with("Answer: "));assert!(transmitted_summary.ends_with(" COMPLETE-DURABLE-SUMMARY"));assert!(!transmitted_summary.contains("do-not-persist"));assert!(!transmitted_summary.contains("late-provider-secret"));- assert!(transmitted_summary.contains("Authorization: Bearer [REDACTED]"));+ assert_eq!(transmitted, &input);assert!(!transmitted_summary.contains("…[truncated]"));assert_eq!(state.outbox[0]@@ -3013,7 +3038,7 @@ mod tests {let mut changed_tail = payload.clone();changed_tail["semantic_tool"]["input"]["summary"] = Value::String(format!(- "token=[REDACTED] {} CHANGED-DURABLE-SUMMARY",+ "Answer: {} CHANGED-DURABLE-SUMMARY","A complete paragraph for the user. ".repeat(180)));assert!(state@@ -3431,16 +3456,31 @@ mod tests {"{}\nAuthorization: Bearer late-credential\nFINAL-ACCEPTANCE-42","Document content. ".repeat(400));- let safe =- sanitize_semantic_tool_input("write_document", &json!({"body": document})).unwrap();- let body = safe["body"].as_str().unwrap();- assert!(body.len() > 4096);- assert!(body.ends_with("FINAL-ACCEPTANCE-42"));- assert!(!body.contains("late-credential"));+ assert!(+ sanitize_semantic_tool_input("write_document", &json!({"body": document})).is_err()+ );}#[test]- fn semantic_prose_does_not_exempt_credential_syntax_or_shapes() {+ fn semantic_credentials_are_rejected_instead_of_executing_redacted_arguments() {+ for input in [+ json!({"content": "Authorization: Bearer opaque-credential"}),+ json!({"content": "literal token=opaque-credential"}),+ json!({"content": "ghp_secretvalue12345678901234567890"}),+ json!({"token": "opaque-credential"}),+ json!({"nested": {"password": "opaque-credential"}}),+ ] {+ let error =+ sanitize_semantic_tool_input("update_agent_instructions", &input).unwrap_err();+ assert!(error+ .to_string()+ .contains("refusing to execute altered arguments"));+ assert!(!error.to_string().contains("opaque-credential"));+ }+ }++ #[test]+ fn semantic_prose_rejects_all_previously_scrubbed_credential_shapes() {for text in ["auth token opaque-credential","access token opaque-credential",@@ -3453,56 +3493,57 @@ mod tests {"literal access_token opaque-credential","literal \"token\" opaque-credential","literal token \"opaque-credential\"",+ "Include the literal token sk-proj-secretvalue123456 in the document.",+ "Include the literal token ghp_secretvalue12345678901234567890 in the document.",+ "Include the literal token github_pat_secretvalue12345678901234567890 in the document.",+ "Include the literal token eyJhbGciOiJIUzI1NiJ9.c2VjcmV0LWNsYWlt.signaturesecret in the document.",] {- assert!(!redact_text(text).contains("opaque-credential"), "{text}");- let input = json!({"description": text, "initialPlan": text});- assert!(- !sanitize_semantic_tool_input("create_task", &input)- .unwrap()- .to_string()- .contains("opaque-credential"),- "{text}"- );- }- for secret in [- "sk-proj-secretvalue123456",- "ghp_secretvalue12345678901234567890",- "github_pat_secretvalue12345678901234567890",- "eyJhbGciOiJIUzI1NiJ9.c2VjcmV0LWNsYWlt.signaturesecret",- ] {- let text = format!("Include the literal token {secret} in the document.");- assert!(!redact_text(&text).contains(secret), "{text}");- assert!(- !sanitize_semantic_tool_input("write_document", &json!({"body": text}))- .unwrap()- .to_string()- .contains(secret)- );+ for (operation, field) in [+ ("create_task", "description"), ("create_task", "initialPlan"),+ ("write_document", "body"), ("update_agent_instructions", "content"),+ ] {+ let input = json!({field: text});+ assert!(sanitize_semantic_tool_input(operation, &input).is_err(), "{operation}: {text}");+ }}- let input = json!({- "description": "Include the literal token ACCEPTANCE-42. Authorization: Bearer opaque-credential",- "token": "opaque-credential",- });- let safe = sanitize_semantic_tool_input("create_task", &input).unwrap();- assert!(safe["description"]- .as_str()- .unwrap()- .contains("ACCEPTANCE-42"));- assert!(!safe.to_string().contains("opaque-credential"));- let diagnostic = json!({"description": "the token opaque-credential"});- assert!(- !sanitize_semantic_tool_input("get_task_context", &diagnostic)- .unwrap()- .to_string()- .contains("opaque-credential")+ assert!(sanitize_semantic_tool_input(+ "get_task_context",+ &json!({+ "description": "the token opaque-credential",+ })+ )+ .is_err());+ }++ #[test]+ fn instruction_arguments_survive_the_complete_outbox_without_byte_changes() {+ let content = format!(+ "{}\nFINAL-TAIL-🦀",+ "# Instructions — 日本語 🦀\n".repeat(400));- assert!(!sanitize_semantic_tool_input(- "create_task",- &json!({"diagnostic": "token opaque-credential"})+ let input = json!({"entryFile": "AGENTS.md", "baseRevisionId": null, "content": content});+ let config = config(PathBuf::from("unused"));+ let mut state = DurableState::new(&config);+ state.enqueue_executor_event(&config, "instruction-write".to_owned(), "semantic_tool.input".to_owned(), EventPriority::P0,+ json!({"semantic_tool": {"schema":"paperclip.prp.semantic_tool.v1", "schemaVersion":1, "phase":"input", "operationId":"update_agent_instructions", "content":{"digest":semantic_value_digest(&input)}, "input": input}})).unwrap();+ let encoded = serde_json::to_vec(&state).unwrap();+ let recovered: DurableState = serde_json::from_slice(&encoded).unwrap();+ let actual = recovered.outbox[0]+ .envelope+ .pointer("/payload/payload/semantic_tool/input")+ .unwrap();+ assert_eq!(actual, &input);+ assert_eq!(+ actual["content"].as_str().unwrap().as_bytes(),+ content.as_bytes()+ );+ assert_eq!(semantic_value_digest(actual), semantic_value_digest(&input));+ assert!(sanitize_semantic_tool_input(+ "update_agent_instructions",+ &json!({"content":"🦀".repeat(200_000)}))- .unwrap()- .to_string()- .contains("opaque-credential"));+ .is_err());+ assert_eq!(state.outbox.len(), 1);}#[test]diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/native_provider_backend.rsindex 73535dc975..7bc424c5c8 100644--- a/packages/paperclip-runner/runner/crates/runner-core/src/native_provider_backend.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/src/native_provider_backend.rs@@ -19,6 +19,14 @@ enum SelectedExecutor {}impl CommandExecutor for SelectedExecutor {+ fn can_reconcile_result_delivery(&mut self) -> Result<bool, DurableRunnerError> {+ match self {+ Self::LocalFacade(executor) => executor.can_reconcile_result_delivery(),+ Self::Acpx(executor) => executor.can_reconcile_result_delivery(),+ Self::Managed(executor) => executor.can_reconcile_result_delivery(),+ }+ }+fn retained_events(&mut self) -> Result<Vec<PolledEvent>, DurableRunnerError> {match self {Self::LocalFacade(executor) => executor.retained_events(),@@ -172,6 +180,13 @@ impl NativeProviderCommandExecutor {}impl CommandExecutor for NativeProviderCommandExecutor {+ fn can_reconcile_result_delivery(&mut self) -> Result<bool, DurableRunnerError> {+ self.select_recovery()?;+ self.selected+ .as_mut()+ .map_or(Ok(false), CommandExecutor::can_reconcile_result_delivery)+ }+fn retained_events(&mut self) -> Result<Vec<PolledEvent>, DurableRunnerError> {self.selected.as_mut()diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rsindex 90cddbdc28..aea835f724 100644--- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs@@ -3070,30 +3070,31 @@ impl CodexCommandExecutor {"reason": reason,})));}- let has_pending_tools = self+ let mut next_state = self.state- .as_ref()- .is_some_and(|state| state.tool_bridge.pending_calls().next().is_some());- if has_pending_tools {- let identity = self.event_identity()?;- let mut next_state = self- .state- .clone()- .expect("Codex state remains available during interruption");- let cancelled = next_state- .tool_bridge- .cancel_pending_calls("provider_turn_stopped")- .map_err(|error| {- DurableRunnerError::invalid(format!(- "failed to cancel pending semantic tools: {error}"- ))- })?;- for result in cancelled {- next_state.push_terminal_event(semantic_result_event(&identity, &result))?;- }- self.persist_state(&next_state)?;- self.state = Some(next_state);- }+ .clone()+ .expect("Codex state remains available during interruption");+ next_state+ .tool_bridge+ .cancel_pending_calls("provider_turn_stopped")+ .map_err(|error| {+ DurableRunnerError::invalid(format!(+ "failed to stop semantic tool admission: {error}"+ ))+ })?;+ next_state.push_terminal_event(NormalizedProviderEvent {+ event_type: "harness.diagnostic".to_owned(),+ priority: EventPriority::P0,+ payload: json!({+ "code": "provider_interrupt_requested",+ "reason": reason,+ "providerTurnId": provider_turn_id,+ "pendingCallIds": next_state.tool_bridge.pending_calls().map(|call| call.call_id.clone()).collect::<Vec<_>>(),+ }),+ })?;+ // Record cause and pending identities before touching the provider.+ self.persist_state(&next_state)?;+ self.state = Some(next_state);self.ensure_provider()?.interrupt_turn().map_err(|error| {DurableRunnerError::invalid(format!("Codex turn interrupt failed: {error}"))})?;@@ -3711,6 +3712,13 @@ impl CodexCommandExecutor {operation_id: String,input: Value,) -> Result<(), DurableRunnerError> {+ if let Err(error) = sanitize_semantic_tool_input(&operation_id, &input) {+ return self.reject_tool_call(+ call_id,+ operation_id,+ ProviderBridgeError::invalid(error.to_string()),+ );+ }let identity = self.event_identity()?;let admission = self.state@@ -3763,6 +3771,7 @@ impl CodexCommandExecutor {.state.clone().ok_or_else(|| DurableRunnerError::invalid("Codex provider is not prepared"))?;+ let delivery_detached = next_state.tool_bridge.turn_closed();let terminal_tool_input = next_state.tool_bridge.pending_calls()@@ -3772,13 +3781,24 @@ impl CodexCommandExecutor {.tool_bridge.apply_result(result.clone()).map_err(|error| {- DurableRunnerError::invalid(format!("semantic tool result was rejected: {error}"))+ DurableRunnerError::invalid(format!(+ "semantic tool result was rejected for call {} operation {}: {error}",+ result.call_id, result.operation_id+ ))})?;if was_completed {- return Ok(CommandExecution::result(json!({- "status": "duplicate",- "callId": result.call_id,- })));+ return Ok(CommandExecution {+ result: json!({ "status": "duplicate", "callId": result.call_id }),+ events: vec![(+ "harness.diagnostic".to_owned(),+ EventPriority::P0,+ json!({+ "code": "semantic_tool_result_duplicate", "severity": "warning",+ "callId": result.call_id, "operationId": result.operation_id,+ "resultDigest": semantic_value_digest(&result.result),+ }),+ )],+ });}let terminal_tool_authoritative =terminal_tool_input@@ -3796,6 +3816,12 @@ impl CodexCommandExecutor {next_state.push_event(semantic_result_event(&identity, &result))?;self.persist_state(&next_state)?;self.state = Some(next_state);+ if delivery_detached {+ return Ok(CommandExecution::result(json!({+ "status": "settled_after_turn",+ "callId": result.call_id,+ })));+ }let provider = self.ensure_provider()?;if terminal_tool_authoritative {provider@@ -4401,6 +4427,12 @@ impl CodexCommandExecutor {}impl CommandExecutor for CodexCommandExecutor {+ fn can_reconcile_result_delivery(&mut self) -> Result<bool, DurableRunnerError> {+ // ProviderToolBridge persists the exact call/result and rejects changed+ // receipts. Delivery cannot execute a server-side semantic operation.+ Ok(true)+ }+fn execute(&mut self, command: &Command) -> Result<CommandExecution, DurableRunnerError> {self.startup_command = Some(ProviderStartupCommand {command_id: command.command_id.clone(),@@ -4457,7 +4489,13 @@ impl CommandExecutor for CodexCommandExecutor {"session.goal.get" => self.get_goal(),"session.goal.set" => self.set_goal(&command.payload),"session.goal.clear" => self.clear_goal(&command.payload),- "turn.interrupt" | "run.cancel" => self.interrupt_turn(&command.command_type),+ "turn.interrupt" | "run.cancel" => self.interrupt_turn(+ command+ .payload+ .get("reason")+ .and_then(Value::as_str)+ .unwrap_or(&command.command_type),+ ),"turn.stop" => self.stop_turn_for_suspension(&command.command_type),"request.resolve" => self.resolve_request(&command.payload),"semantic_tool.result" => self.deliver_semantic_result(&command.payload),@@ -5474,7 +5512,7 @@ mod tests {}#[test]- fn semantic_input_digest_covers_the_transmitted_redacted_value() {+ fn semantic_input_rejects_credential_material_before_dispatch() {let identity = ProviderEventIdentity {runner_instance_id: "runner-1".to_owned(),run_id: "run-1".to_owned(),@@ -5487,17 +5525,14 @@ mod tests {operation_id: "get_task_context".to_owned(),input: json!({"password": "do-not-persist", "safe": true}),};- let event = semantic_input_event(&identity, &call).unwrap();- let transmitted = &event.payload["semantic_tool"]["input"];- assert_eq!(transmitted["password"], "[REDACTED]");- assert_eq!(- event.payload["semantic_tool"]["content"]["digest"],- semantic_value_digest(transmitted)- );+ assert!(semantic_input_event(&identity, &call)+ .unwrap_err()+ .to_string()+ .contains("refusing to execute altered arguments"));}#[test]- fn semantic_finish_input_preserves_a_complete_long_redacted_summary() {+ fn semantic_finish_input_preserves_a_complete_long_summary() {let identity = ProviderEventIdentity {runner_instance_id: "runner-1".to_owned(),run_id: "run-1".to_owned(),@@ -5506,7 +5541,7 @@ mod tests {item_id: "item-1".to_owned(),};let summary = format!(- "token=do-not-persist {} Authorization: Bearer late-provider-secret COMPLETE-LONG-SUMMARY",+ "Answer: {} COMPLETE-LONG-SUMMARY","A complete paragraph for the user. ".repeat(180));assert!(summary.len() > 4_096);@@ -5519,11 +5554,11 @@ mod tests {let event = semantic_input_event(&identity, &call).unwrap();let transmitted = &event.payload["semantic_tool"]["input"];let transmitted_summary = transmitted["summary"].as_str().unwrap();- assert!(transmitted_summary.starts_with("token=[REDACTED] "));+ assert!(transmitted_summary.starts_with("Answer: "));assert!(transmitted_summary.ends_with(" COMPLETE-LONG-SUMMARY"));assert!(!transmitted_summary.contains("do-not-persist"));assert!(!transmitted_summary.contains("late-provider-secret"));- assert!(transmitted_summary.contains("Authorization: Bearer [REDACTED]"));+ assert_eq!(transmitted, &call.input);assert!(!transmitted_summary.contains("…[truncated]"));assert_eq!(event.payload["semantic_tool"]["content"]["digest"],diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_bridge.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_bridge.rsindex 7730787381..a100798336 100644--- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_bridge.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_bridge.rs@@ -192,6 +192,10 @@ pub struct ProviderToolBridge {catalog_operations: Vec<AuthorizedTool>,catalog_digest: Option<String>,pending: BTreeMap<String, PendingToolCall>,+ // A provider stopping is not evidence that a dispatched server operation+ // failed. Keep its input and identity pending until the authority replies.+ #[serde(default)]+ turn_closed: bool,#[serde(deserialize_with = "deserialize_retained_results")]completed: BTreeMap<String, CompletedToolCall>,// Keep authoritative result bodies through the provider's replay window;@@ -227,7 +231,7 @@ pub struct ProviderBridgeError {}impl ProviderBridgeError {- fn invalid(message: impl Into<String>) -> Self {+ pub(crate) fn invalid(message: impl Into<String>) -> Self {Self {message: message.into(),safe_provider_message: None,@@ -321,6 +325,7 @@ impl ProviderToolBridge {));}self.prepare_internal(tool_set, true)?;+ self.turn_closed = false;self.completed.clear();self.settled_results.clear();self.retained_result_bytes = 0;@@ -577,6 +582,7 @@ impl ProviderToolBridge {// can replay the just-settled turn. Release that bulky data at the// verified turn boundary, but retain every call-ID tombstone and any// recovered legacy filter until the owning provider process is reaped.+ self.turn_closed = false;self.settled_results.clear();self.retained_result_bytes = 0;self.durable_run_receipt_limit_reached = self.replay_history_blocks_admission();@@ -687,6 +693,11 @@ impl ProviderToolBridge {"provider reused a completed tool call id",));}+ if self.turn_closed {+ return Err(ProviderBridgeError::invalid(+ "cannot admit a new tool call after the provider turn stopped",+ ));+ }// A durable receipt ledger may be exactly full or legacy state may// contain only a probabilistic summary of evicted identities. Neither// state can admit more work safely in this provider-process epoch. The@@ -769,18 +780,14 @@ impl ProviderToolBridge {return if existing.result == result {Ok(existing.result.result.clone())} else {- Err(ProviderBridgeError::invalid(- "conflicting duplicate tool result",- ))+ Err(conflicting_tool_result(&existing.result, &result))};}if let Some(existing) = self.settled_results.get(&result.call_id) {return if existing.result == result {Ok(existing.result.result.clone())} else {- Err(ProviderBridgeError::invalid(- "conflicting duplicate settled tool result",- ))+ Err(conflicting_tool_result(&existing.result, &result))};}if self.has_settled_call_id(&result.call_id) {@@ -848,7 +855,16 @@ impl ProviderToolBridge {)?;self.pending.remove(&result.call_id);self.retained_result_bytes = next_retained_bytes;- self.completed.insert(result.call_id.clone(), completed);+ if self.turn_closed {+ self.settled_call_ids.extend_recent(+ std::iter::once(result.call_id.clone()),+ MAX_SETTLED_CALL_IDS,+ );+ self.settled_results+ .insert(result.call_id.clone(), completed);+ } else {+ self.completed.insert(result.call_id.clone(), completed);+ }Ok(result.result)}@@ -856,41 +872,29 @@ impl ProviderToolBridge {self.pending.values()}+ /// Close provider admission/delivery, without inventing an operation outcome.+ /// This marker is durable, including when interruption beats result delivery.pub fn cancel_pending_calls(&mut self,code: &str,) -> Result<Vec<ToolResult>, ProviderBridgeError> {- let mut next = self.clone();- let results = next.cancel_pending_calls_internal(code)?;- *self = next;- Ok(results)+ self.settle_turn(code)+ }++ pub fn turn_closed(&self) -> bool {+ self.turn_closed}pub fn settle_turn(&mut self, code: &str) -> Result<Vec<ToolResult>, ProviderBridgeError> {validate_stable_id(code, "tool cancellation code")?;- let results = self- .pending- .values()- .map(|call| cancelled_tool_result(call, code))- .collect::<Vec<_>>();- let mut settled_entries = self.completed.clone();- for (call_id, call) in &self.pending {- settled_entries.insert(- call_id.clone(),- CompletedToolCall {- call: call.clone(),- result: cancelled_tool_result(call, code),- },- );+ if !self.settled_identity_capacity_allows(0) {+ return Err(ProviderBridgeError::active_turn_receipt_limit());}let next_retained_bytes =- retained_result_bytes(self.settled_results.iter().chain(settled_entries.iter()))?;- ensure_settled_result_capacity(next_retained_bytes, std::iter::empty())?;-- // Byte capacity was reserved at admission. Keep every identity exact- // for the lifetime of this provider-process epoch; prepare_turn- // releases only the bulky result bodies.- let new_identity_count = settled_entries+ retained_result_bytes(self.settled_results.iter().chain(self.completed.iter()))?;+ ensure_settled_result_capacity(next_retained_bytes, self.pending.values())?;+ let new_identity_count = self+ .completed.keys().filter(|call_id| !self.settled_call_ids.contains(call_id)).count();@@ -904,28 +908,14 @@ impl ProviderToolBridge {}let evicted = self.settled_call_ids- .extend_recent(settled_entries.keys().cloned(), MAX_SETTLED_CALL_IDS);+ .extend_recent(self.completed.keys().cloned(), MAX_SETTLED_CALL_IDS);debug_assert!(evicted.is_empty());- self.pending.clear();- self.completed.clear();- self.settled_results.append(&mut settled_entries);+ self.settled_results.append(&mut self.completed);self.retained_result_bytes = next_retained_bytes;- Ok(results)- }-- fn cancel_pending_calls_internal(- &mut self,- code: &str,- ) -> Result<Vec<ToolResult>, ProviderBridgeError> {- validate_stable_id(code, "tool cancellation code")?;- let pending = self.pending.values().cloned().collect::<Vec<_>>();- let mut results = Vec::with_capacity(pending.len());- for call in pending {- let result = cancelled_tool_result(&call, code);- self.apply_result(result.clone())?;- results.push(result);- }- Ok(results)+ self.turn_closed = true;+ // Pending effects survive both normal turn termination and process+ // loss. They block the next turn/checkpoint, but accept a late result.+ Ok(Vec::new())}fn retained_value_bytes(&self) -> Result<usize, ProviderBridgeError> {@@ -1271,19 +1261,18 @@ wheredeserializer.deserialize_map(RetainedResultsVisitor)}-fn cancelled_tool_result(call: &PendingToolCall, code: &str) -> ToolResult {- ToolResult {- call_id: call.call_id.clone(),- operation_id: call.operation_id.clone(),- result: serde_json::json!({- "error": {- "code": code,- "message": "The provider turn stopped before this semantic tool completed",- "retryable": false,- },- }),- is_error: true,- }+fn conflicting_tool_result(existing: &ToolResult, incoming: &ToolResult) -> ProviderBridgeError {+ // Correlate both receipts without leaking arguments or result content into+ // a user-visible error or diagnostic stream.+ let digest = |result: &ToolResult| {+ semantic_value_digest(&serde_json::json!({+ "operationId": result.operation_id, "isError": result.is_error, "result": result.result,+ }))+ };+ ProviderBridgeError::invalid(format!(+ "conflicting duplicate tool result: callId={} operationId={} existingDigest={} incomingDigest={}",+ incoming.call_id, incoming.operation_id, digest(existing), digest(incoming),+ ))}pub fn authorized_tool_catalog_digest(@@ -1886,11 +1875,13 @@ mod tests {let cancelled = pending_at_capacity.settle_turn("semantic_tool_turn_receipt_limit").unwrap();- assert_eq!(cancelled.len(), 1);+ assert!(cancelled.is_empty());+ assert_eq!(pending_at_capacity.pending_calls().count(), 1);assert_eq!(pending_at_capacity.settled_call_ids.len(),- MAX_SETTLED_CALL_IDS+ MAX_SETTLED_CALL_IDS - 1);+ assert!(pending_at_capacity.prepare_turn().is_err());bridge.apply_result(ToolResult {diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_event_payload.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_event_payload.rsindex a82105de9b..0dab9435c0 100644--- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_event_payload.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_event_payload.rs@@ -176,7 +176,7 @@ fn rejects_unclassified_and_malformed_runtime_payloads() {#[test]fn decodes_tool_and_permission_requests_after_scope_validation() {let scope = active_scope();- let tool = decode_acpx_event(+ let rejected = decode_acpx_event(&scope,&event(GeneratedAcpxSidecarEventType::RuntimeToolCalled,@@ -187,14 +187,10 @@ fn decodes_tool_and_permission_requests_after_scope_validation() {}),),)- .unwrap();- assert!(matches!(- tool,- AcpxEventPayload::ToolCalled { call_id, operation_id, input, .. }- if call_id == "call-1"- && operation_id == "get_issue"- && input["apiToken"] == "[REDACTED]"- ));+ .unwrap_err();+ assert!(rejected+ .to_string()+ .contains("refusing to execute altered arguments"));let permission = decode_acpx_event(&scope,diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rsindex 89e4cdee12..5a61230715 100644--- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs@@ -99,9 +99,9 @@ fn commits_each_resolution_only_after_sidecar_acknowledgement() {.unwrap();assert!(session.state().pending_tool("call-1").is_none());assert!(session.state().pending_question_set("input-1").is_none());- assert!(session+ session.deliver_tool_result(&tool_result("issues.read"))- .is_err());+ .expect("an identical result replay must be idempotent");session.shutdown("test complete").unwrap();}@@ -200,7 +200,9 @@ fn redacts_failed_tool_payload_without_losing_correlation_or_retry() {failed.operation_id = "issues.read".to_owned();session.deliver_tool_result(&failed).unwrap();assert!(session.state().pending_tool("call-1").is_none());- assert!(session.deliver_tool_result(&failed).is_err());+ session+ .deliver_tool_result(&failed)+ .expect("the failure receipt also deduplicates");session.shutdown("test complete").unwrap();}diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_state.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_state.rsindex 072a812d56..f4f4d23191 100644--- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_state.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_state.rs@@ -465,7 +465,7 @@ fn accepts_global_process_and_diagnostic_events_without_an_active_turn() {}#[test]-fn terminal_events_clear_pending_requests_and_reject_late_turn_events() {+fn terminal_events_retain_dispatched_tools_and_reject_late_turn_events() {let mut state = AcpxProviderState::new("run-1").unwrap();state.begin_turn("turn-1").unwrap();state@@ -484,8 +484,10 @@ fn terminal_events_clear_pending_requests_and_reject_late_turn_events() {json!({"status":"cancelled","error":{"message":"token=secret"}}),)).unwrap();+ assert!(state.pending_tool("call-1").is_some());+ assert!(state.begin_turn("turn-2").is_err());+ state.complete_tool("call-1", "issues.read").unwrap();assert!(state.pending_tool("call-1").is_none());- assert!(state.complete_tool("call-1", "issues.read").is_err());assert!(state.accept_event(&event(3,@@ -509,10 +511,8 @@ fn mutation_prose_survives_sidecar_decode_pending_state_and_semantic_projection("description": "The document must contain the token CHAT8322bda781b81.","initialPlan": plan,"idempotencyKey": "CHAT8322bda781b81-task",- "apiToken": "actual-credential",});- let mut expected = input.clone();- expected["apiToken"] = json!("[REDACTED]");+ let expected = input.clone();let emitted = state.accept_event(&event(1,@@ -578,34 +578,27 @@ fn mutation_prose_survives_sidecar_decode_pending_state_and_semantic_projection(false,),] {- state- .complete_tool(- "call-1",- state- .pending_tool("call-1")- .unwrap()- .operation_id- .clone()- .as_str(),- )- .unwrap();- let emitted = state- .accept_event(&event(- 2,- GeneratedAcpxSidecarEventType::RuntimeToolCalled,- Some("turn-1"),- json!({"callId": "call-1", "operationId": operation, "input": {field: prose}}),- ))- .unwrap();- let AcpxProviderStateEvent::ToolCall { input, .. } = &emitted[0] else {+ let mut candidate = AcpxProviderState::new("run-1").unwrap();+ candidate.begin_turn("turn-1").unwrap();+ let emitted = candidate.accept_event(&event(+ 1,+ GeneratedAcpxSidecarEventType::RuntimeToolCalled,+ Some("turn-1"),+ json!({"callId":"call-1", "operationId":operation, "input":{field:prose}}),+ ));+ if !preserved {+ assert!(emitted+ .unwrap_err()+ .to_string()+ .contains("refusing to execute altered arguments"));+ assert!(candidate.pending_tool("call-1").is_none());+ continue;+ }+ let events = emitted.unwrap();+ let AcpxProviderStateEvent::ToolCall { input, .. } = &events[0] else {panic!("expected tool call");};- assert_eq!(- input[field] == json!(prose),- preserved,- "{operation}: {prose}"- );- assert!(!input.to_string().contains("actual-credential"));+ assert_eq!(input[field], json!(prose));}}diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rsindex f2897a5415..0f8188c623 100644--- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs@@ -7,7 +7,7 @@ use paperclip_runner_core::acpx_provider_session::{use paperclip_runner_core::acpx_provider_state::AcpxProviderStateEvent;use paperclip_runner_core::acpx_sidecar_transport::AcpxSidecarTransportConfig;use paperclip_runner_core::provider_bridge::{- authorized_tool_catalog_digest, AuthorizedTool, AuthorizedToolSet,+ authorized_tool_catalog_digest, AuthorizedTool, AuthorizedToolSet, ToolResult,};use serde_json::json;@@ -219,45 +219,51 @@ fn admits_only_catalog_authorized_tool_calls() {session.shutdown("test complete").unwrap();}+fn late_result(call_id: &str) -> ToolResult {+ ToolResult {+ call_id: call_id.to_owned(),+ operation_id: "issues.read".to_owned(),+ result: json!({"id":"issue-1"}),+ is_error: false,+ }+}+#[test]-fn returns_pending_tool_cancellations_before_the_terminal_event() {+fn retains_pending_tools_until_authoritative_late_results_and_deduplicates_retries() {let mut session = AcpxProviderSession::start(&config("turns-tool-terminal")).unwrap();- session- .start_turn("turn-1", "Please help", &std::env::temp_dir())- .unwrap();- let tool = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();- assert!(matches!(tool[0], AcpxProviderStateEvent::ToolCall { .. }));-- let terminal = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();- assert!(matches!(- &terminal[0],- AcpxProviderStateEvent::ToolResult(result)- if result.call_id == "call-1"- && result.operation_id == "issues.read"- && result.is_error- && result.result["error"]["code"] == "acpx_turn_settled"- ));- assert!(matches!(- terminal.last().unwrap(),- AcpxProviderStateEvent::TurnTerminal { turn_id, .. } if turn_id == "turn-1"- ));-- assert!(session.state().pending_tool("call-1").is_none());- session- .start_turn("turn-2", "Please continue", &std::env::temp_dir())- .expect("terminal settlement must leave the session reusable");- let next_tool = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();- assert!(matches!(- &next_tool[0],- AcpxProviderStateEvent::ToolCall { call_id, operation_id, .. }- if call_id == "call-2" && operation_id == "issues.read"- ));- let next_terminal = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();- assert!(matches!(- next_terminal.last().unwrap(),- AcpxProviderStateEvent::TurnTerminal { turn_id, .. } if turn_id == "turn-2"- ));- assert!(session.state().pending_tool("call-2").is_none());+ for (turn_id, call_id) in [("turn-1", "call-1"), ("turn-2", "call-2")] {+ session+ .start_turn(turn_id, "Please help", &std::env::temp_dir())+ .unwrap();+ let tool = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();+ assert!(+ matches!(&tool[0], AcpxProviderStateEvent::ToolCall { call_id: actual, .. } if actual == call_id)+ );+ let terminal = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();+ assert_eq!(terminal.len(), 1, "must not invent a cancellation result");+ assert!(+ matches!(&terminal[0], AcpxProviderStateEvent::TurnTerminal { turn_id: actual, .. } if actual == turn_id)+ );+ assert!(session.state().pending_tool(call_id).is_some());+ assert!(session+ .start_turn("blocked-turn", "Wait for settlement", &std::env::temp_dir())+ .is_err());+ session.deliver_tool_result(&late_result(call_id)).unwrap();+ session+ .deliver_tool_result(&late_result(call_id))+ .expect("lost acknowledgement is harmless");+ let mut conflict = late_result(call_id);+ conflict.result = json!({"id":"different"});+ let error = session+ .deliver_tool_result(&conflict)+ .unwrap_err()+ .to_string();+ assert!(+ error.contains(call_id) && error.contains("issues.read"),+ "{error}"+ );+ assert!(session.state().pending_tool(call_id).is_none());+ }session.shutdown("test complete").unwrap();}@@ -276,12 +282,10 @@ fn rotates_settled_tool_receipts_between_reusable_turns() {if call_id == "call-reused" && operation_id == "issues.read"));let terminal = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();- assert!(matches!(- &terminal[0],- AcpxProviderStateEvent::ToolResult(result)- if result.call_id == "call-reused"- && result.result["error"]["code"] == "acpx_turn_settled"- ));+ assert_eq!(terminal.len(), 1);+ session+ .deliver_tool_result(&late_result("call-reused"))+ .unwrap();assert!(matches!(terminal.last().unwrap(),AcpxProviderStateEvent::TurnTerminal { turn_id: settled, .. }@@ -332,6 +336,9 @@ fn reaps_the_old_provider_before_admitting_a_late_tool_callback() {.unwrap();session.poll_event(Duration::from_secs(1)).unwrap().unwrap();session.poll_event(Duration::from_secs(1)).unwrap().unwrap();+ session+ .deliver_tool_result(&late_result("call-reused"))+ .unwrap();session.start_turn("turn-2", "Use a fresh provider", &std::env::temp_dir())@@ -360,6 +367,9 @@ fn fails_closed_before_reusing_a_settled_turn_identity() {.unwrap();session.poll_event(Duration::from_secs(1)).unwrap().unwrap();session.poll_event(Duration::from_secs(1)).unwrap().unwrap();+ session+ .deliver_tool_result(&late_result("call-reused"))+ .unwrap();let reused = session.start_turn("turn-1", "Do not alias old events", &std::env::temp_dir())@@ -578,62 +588,28 @@ fn reserved_completion_waits_for_feedback_and_allows_correction_in_same_turn() {}#[test]-fn correlates_reserved_results_by_raw_digest_without_exposing_sensitive_values() {- let mut session =- AcpxProviderSession::start(&config("turns-sensitive-reserved-result-terminal")).unwrap();- session- .start_turn("turn-1", "Please help", &std::env::temp_dir())- .unwrap();-- assert!(matches!(- &session.poll_event(Duration::from_secs(1)).unwrap().unwrap()[0],- AcpxProviderStateEvent::ToolCall { operation_id, .. }- if operation_id == "paperclip_finish"- ));- let result_events = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();- assert!(matches!(- &result_events[0],- AcpxProviderStateEvent::SemanticResult(result)- if result.result["summary"]- .as_str()- .is_some_and(|summary| summary.contains("REDACTED")- && !summary.contains("matching-sensitive-value"))- ));-- let terminal = session.poll_event(Duration::from_secs(1)).unwrap().unwrap();- assert!(matches!(- terminal.last().unwrap(),- AcpxProviderStateEvent::TurnTerminal { turn_id, .. } if turn_id == "turn-1"- ));- session.shutdown("test complete").unwrap();-}--#[test]-fn rejects_sensitive_reserved_results_that_only_match_after_redaction() {- let mut session = AcpxProviderSession::start(&config(+fn rejects_sensitive_reserved_input_before_dispatch_without_exposing_values() {+ for mode in [+ "turns-sensitive-reserved-result-terminal","turns-mismatched-sensitive-reserved-result-terminal",- ))- .unwrap();- session- .start_turn("turn-1", "Please help", &std::env::temp_dir())- .unwrap();- assert!(matches!(- &session.poll_event(Duration::from_secs(1)).unwrap().unwrap()[0],- AcpxProviderStateEvent::ToolCall { operation_id, .. }- if operation_id == "paperclip_finish"- ));-- let error = session- .poll_event(Duration::from_secs(1))- .unwrap_err()- .to_string();- assert!(- error.contains("does not match its authorized invocation"),- "{error}"- );- assert!(!error.contains("matching-sensitive-value"), "{error}");- assert!(!error.contains("different-sensitive-value"), "{error}");- assert!(session.shutdown("already closed").is_ok());+ ] {+ let mut session = AcpxProviderSession::start(&config(mode)).unwrap();+ session+ .start_turn("turn-1", "Please help", &std::env::temp_dir())+ .unwrap();+ let error = session+ .poll_event(Duration::from_secs(1))+ .unwrap_err()+ .to_string();+ assert!(+ error.contains("refusing to execute altered arguments"),+ "{error}"+ );+ assert!(!error.contains("matching-sensitive-value"), "{error}");+ assert!(!error.contains("different-sensitive-value"), "{error}");+ assert!(session.state().pending_tool("call-finish").is_none());+ session.shutdown("already closed").unwrap();+ }}#[test]diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rsindex 060f5d1959..dac280fdf3 100644--- a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs@@ -3054,6 +3054,103 @@ fn codex_resume_advertises_the_same_authorized_tools() {fs::remove_dir_all(directory).expect("remove Codex integration-test directory");}+#[test]+fn interrupted_tool_accepts_one_authoritative_result_across_restart_and_lost_ack() {+ for restart in [false, true] {+ let directory = temporary_directory(if restart {+ "interrupt-tool-restart"+ } else {+ "interrupt-tool-live"+ });+ let config = provider_config(&directory, &["--require-dynamic-tool", "--emit-tool-call"]);+ let runner_config = durable_config(&directory);+ let mut executor = CodexCommandExecutor::with_runner_config(&directory, &runner_config);+ executor+ .execute(&command(+ "prepare",+ 1,+ "run.prepare",+ json!({"provider":config,"authorizedTools":task_context_tool_set()}),+ ))+ .unwrap();+ executor+ .execute(&command("open", 2, "session.open", json!({})))+ .unwrap();+ executor+ .execute(&command(+ "turn",+ 3,+ "turn.start",+ json!({"text":"Hold the actual operation until after interruption."}),+ ))+ .unwrap();+ wait_for_executor_event(&mut executor, "semantic_tool.input");+ executor+ .execute(&command(+ "interrupt",+ 4,+ "turn.interrupt",+ json!({"reason":"test_result_delivery_barrier"}),+ ))+ .unwrap();+ let saved: Value =+ serde_json::from_slice(&fs::read(directory.join("codex-provider-state.json")).unwrap())+ .unwrap();+ assert_eq!(saved["toolBridge"]["pending"].as_object().unwrap().len(), 1);+ assert!(saved["pendingEvents"]+ .as_array()+ .unwrap()+ .iter()+ .any(|event| event["payload"]["reason"] == "test_result_delivery_barrier"));+ if restart {+ drop(executor);+ executor = CodexCommandExecutor::with_runner_config(&directory, &runner_config);+ }+ let result = json!({"callId":"semantic-call-1", "operationId":"get_task_context", "result":{"ok":true,"savedHash":"exact-write"}, "isError":false});+ let delivered = executor+ .execute(&command(+ "result",+ 5,+ "semantic_tool.result",+ result.clone(),+ ))+ .unwrap();+ assert_eq!(delivered.result["status"], "settled_after_turn");+ // Lose the command ACK, restart, then deliver precisely the same result.+ drop(executor);+ executor = CodexCommandExecutor::with_runner_config(&directory, &runner_config);+ let replay = executor+ .execute(&command(+ "retry-result",+ 6,+ "semantic_tool.result",+ result.clone(),+ ))+ .unwrap();+ assert_eq!(replay.result["status"], "duplicate");+ assert!(replay+ .events+ .iter()+ .any(|(kind, _, value)| kind == "harness.diagnostic"+ && value["code"] == "semantic_tool_result_duplicate"+ && value["callId"] == "semantic-call-1"));+ let mut conflict = result;+ conflict["result"]["savedHash"] = json!("different-write");+ let error = executor+ .execute(&command("conflict", 7, "semantic_tool.result", conflict))+ .unwrap_err();+ assert!(error.to_string().contains("semantic-call-1"));+ assert!(error.to_string().contains("get_task_context"));+ assert!(error.to_string().contains("conflicting duplicate"));+ assert!(error.to_string().contains("existingDigest=sha256:"));+ assert!(error.to_string().contains("incomingDigest=sha256:"));+ assert!(!error.to_string().contains("different-write"));+ assert_eq!(call_count(&directory, "turn/start"), 1);+ executor.shutdown().unwrap();+ fs::remove_dir_all(directory).unwrap();+ }+}+#[test]fn durable_backend_routes_a_semantic_tool_result_back_to_codex() {let directory = temporary_directory("durable-dynamic-tool");@@ -3450,33 +3547,15 @@ fn durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended() {break;}}- let semantic_result = observed- .iter()- .position(|event| event == "semantic_tool.result")- .expect("recovery settles the pending semantic tool");- let reconciled = observed- .iter()- .position(|event| event == "session.reconciled")- .expect("recovery emits a reconciliation event");- let terminal = observed- .iter()- .position(|event| event == "run.terminal")- .expect("offline turn recovery terminates the run");- assert!(semantic_result < reconciled);- assert!(reconciled < terminal);- assert!(recovered- .execute(&command(- "late-result",- 5,- "semantic_tool.result",- json!({- "callId": "semantic-call-1",- "operationId": "get_task_context",- "result": {"ok": true},- "isError": false,- }),- ))- .is_err());+ assert!(+ !observed.iter().any(|event| event == "semantic_tool.result"),+ "recovery must not invent an effect outcome"+ );+ assert!(observed.iter().any(|event| event == "session.reconciled"));+ let late = recovered.execute(&command("late-result", 5, "semantic_tool.result", json!({+ "callId":"semantic-call-1", "operationId":"get_task_context", "result":{"ok":true}, "isError":false,+ }))).unwrap();+ assert_eq!(late.result["status"], "settled_after_turn");recovered.shutdown().expect("stop recovered provider");fs::remove_dir_all(directory).expect("remove Codex integration-test directory");@@ -3976,28 +4055,12 @@ fn durable_backend_settles_tools_before_a_natural_terminal_event() {break;}}- let semantic_result = observed- .iter()- .position(|event| event == "semantic_tool.result")- .expect("terminal settlement emits a failed semantic result");- let terminal = observed- .iter()- .position(|event| event == "turn.completed")- .expect("provider terminal event is emitted");- assert!(semantic_result < terminal);- assert!(executor- .execute(&command(- "late-result",- 4,- "semantic_tool.result",- json!({- "callId": "semantic-call-1",- "operationId": "get_task_context",- "result": {"ok": true},- "isError": false,- }),- ))- .is_err());+ assert!(!observed.iter().any(|event| event == "semantic_tool.result"));+ assert!(observed.iter().any(|event| event == "turn.completed"));+ let late = executor.execute(&command("late-result", 4, "semantic_tool.result", json!({+ "callId":"semantic-call-1", "operationId":"get_task_context", "result":{"ok":true}, "isError":false,+ }))).unwrap();+ assert_eq!(late.result["status"], "settled_after_turn");executor.shutdown().unwrap();fs::remove_dir_all(directory).unwrap();@@ -4167,27 +4230,15 @@ fn durable_stop_settles_pending_semantic_tools_without_a_courtesy_interrupt() {.execute(&command("stop", 4, "turn.stop", json!({}))).unwrap();assert_eq!(stopped.result["providerExitConfirmed"], true);- let result = wait_for_executor_event(&mut executor, "semantic_tool.result");- assert_eq!(result.payload["semantic_tool"]["outcome"], "failed");- assert_eq!(- result.payload["semantic_tool"]["callId"],- input.payload["semantic_tool"]["callId"]- );- assert_eq!(- result.payload["semantic_tool"]["correlation"],- input.payload["semantic_tool"]["correlation"]- );- assert!(executor- .execute(&command(- "late-result",- 5,- "semantic_tool.result",- json!({- "callId": "semantic-call-1", "operationId": "get_task_context",- "result": {"ok": true}, "isError": false,- })- ))- .is_err());+ assert_eq!(input.payload["semantic_tool"]["callId"], "semantic-call-1");+ assert!(!poll_and_ack(&mut executor)+ .unwrap()+ .iter()+ .any(|event| event.event_type == "semantic_tool.result"));+ let late = executor.execute(&command("late-result", 5, "semantic_tool.result", json!({+ "callId":"semantic-call-1", "operationId":"get_task_context", "result":{"ok":true}, "isError":false,+ }))).unwrap();+ assert_eq!(late.result["status"], "settled_after_turn");assert_eq!(call_count(&directory, "turn/interrupt"), 0);assert_eq!(call_count(&directory, "thread/start"), 1);assert_eq!(call_count(&directory, "turn/start"), 1);diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/provider_bridge.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/provider_bridge.rsindex 2504f383ed..61a0a02250 100644--- a/packages/paperclip-runner/runner/crates/runner-core/tests/provider_bridge.rs+++ b/packages/paperclip-runner/runner/crates/runner-core/tests/provider_bridge.rs@@ -288,26 +288,62 @@ fn recovered_bridge_rejects_tampered_authorization_state() {}#[test]-fn cancellation_completes_pending_calls_and_rejects_late_results() {- let mut bridge = ProviderToolBridge::default();- bridge.prepare(tools("computed")).unwrap();- bridge- .begin_call("call-1".into(), "get_task_context".into(), json!({}))- .unwrap();- let cancelled = bridge- .cancel_pending_calls("provider_turn_stopped")- .unwrap();- assert_eq!(cancelled.len(), 1);- assert!(cancelled[0].is_error);- assert_eq!(bridge.pending_calls().count(), 0);- assert!(bridge- .apply_result(ToolResult {- call_id: "call-1".into(),- operation_id: "get_task_context".into(),- result: json!({"ok": true}),- is_error: false,- })- .is_err());+fn stop_and_restart_keep_inflight_effects_pending_until_authoritative_result() {+ for stop_before_result in [false, true] {+ for restart in [false, true] {+ let mut bridge = ProviderToolBridge::default();+ bridge.prepare(tools("computed")).unwrap();+ bridge+ .begin_call("call-1".into(), "get_task_context".into(), json!({}))+ .unwrap();+ let result = ToolResult {+ call_id: "call-1".into(),+ operation_id: "get_task_context".into(),+ result: json!({"written": true}),+ is_error: false,+ };+ if !stop_before_result {+ bridge.apply_result(result.clone()).unwrap();+ }+ assert!(bridge+ .cancel_pending_calls("provider_turn_stopped")+ .unwrap()+ .is_empty());+ if restart {+ bridge = serde_json::from_str(&serde_json::to_string(&bridge).unwrap()).unwrap();+ bridge.attach_existing_run().unwrap();+ }+ assert!(bridge+ .begin_call("new-call".into(), "get_task_context".into(), json!({}))+ .is_err());+ if stop_before_result {+ assert_eq!(bridge.pending_calls().count(), 1);+ assert!(bridge.prepare_turn().is_err());+ assert!(bridge.attach_run(tools("computed")).is_err());+ }+ bridge.apply_result(result.clone()).unwrap();+ bridge+ .apply_result(result.clone())+ .expect("identical delivery is idempotent");+ assert_eq!(bridge.pending_calls().count(), 0);+ assert_eq!(+ bridge+ .replay_result("call-1", "get_task_context", &json!({}))+ .unwrap(),+ Some(result.clone())+ );+ assert!(bridge+ .apply_result(ToolResult {+ result: json!({"written": false}),+ ..result+ })+ .is_err());+ bridge.prepare_turn().unwrap();+ bridge+ .begin_call("new-call".into(), "get_task_context".into(), json!({}))+ .unwrap();+ }+ }}#[test]@@ -333,6 +369,7 @@ fn turn_settlement_releases_value_capacity_without_reusing_call_ids() {assert!(bridge.begin_call("call-1".into(), "get_task_context".into(), json!({})).is_err());+ bridge.prepare_turn().unwrap();bridge.begin_call("call-2".into(), "get_task_context".into(), json!({})).expect("a new turn can use a fresh call id after releasing exact values");@@ -405,8 +442,18 @@ fn turn_settlement_cannot_be_blocked_by_completed_value_pressure() {}let settled = bridge.settle_turn("provider_turn_terminated").unwrap();- assert_eq!(settled.len(), 1);- assert!(settled[0].is_error);+ assert!(settled.is_empty());+ assert_eq!(bridge.pending_calls().count(), 1);+ assert!(bridge.prepare_turn().is_err());+ bridge+ .apply_result(ToolResult {+ call_id: "call-4".into(),+ operation_id: "get_task_context".into(),+ result: json!({"ok": true}),+ is_error: false,+ })+ .unwrap();+ bridge.prepare_turn().unwrap();assert!(bridge.begin_call("call-0".into(), "get_task_context".into(), json!({})).is_err());@@ -662,6 +709,7 @@ fn settles_completed_receipts_before_the_next_turn() {.begin_call("call-next".into(), "get_task_context".into(), json!({})).is_err());bridge.settle_turn("provider_turn_terminated").unwrap();+ bridge.prepare_turn().unwrap();assert!(bridge.begin_call("call-next".into(), "get_task_context".into(), json!({})).is_ok());@@ -704,9 +752,6 @@ fn settlement_preserves_call_ids_for_the_durable_run() {assert!(bridge.begin_call("call-1".into(), "get_task_context".into(), json!({})).is_err());- bridge- .begin_call("call-2".into(), "get_task_context".into(), json!({}))- .unwrap();let encoded = serde_json::to_string(&bridge).unwrap();let mut recovered: ProviderToolBridge = serde_json::from_str(&encoded).unwrap();@@ -759,16 +804,18 @@ fn exact_identity_overflow_saturates_the_durable_run() {.expect("the controlled turn stop retains replay protection");assert!(recovered.durable_run_receipt_limit_reached());assert!(recovered.has_completed_call("settled-0"));+ assert!(!recovered.has_completed_call("last-call"));+ assert_eq!(recovered.pending_calls().count(), 1);+ assert!(recovered.prepare_turn().is_err());+ recovered+ .apply_result(ToolResult {+ call_id: "last-call".into(),+ operation_id: "get_task_context".into(),+ result: json!({"ok": true}),+ is_error: false,+ })+ .unwrap();assert!(recovered.has_completed_call("last-call"));- let stopped_turn_receipt = recovered- .replay_result("last-call", "get_task_context", &json!({}))- .unwrap()- .expect("the call admitted before exhaustion retains an exact terminal receipt");- assert!(stopped_turn_receipt.is_error);- assert_eq!(- stopped_turn_receipt.result["error"]["code"],- "provider_turn_terminated"- );assert!(recovered.begin_call("settled-0".into(), "get_task_context".into(), json!({})).is_err());@@ -831,12 +878,12 @@ fn settled_result_byte_exhaustion_recovers_after_turn_cleanup() {json!({}),).expect_err("settled byte exhaustion must stop the durable run");- assert!(error.is_active_turn_receipt_limit());+ assert!(error.to_string().contains("provider turn stopped"));let encoded = serde_json::to_string(&bridge).unwrap();let mut recovered: ProviderToolBridge = serde_json::from_str(&encoded).unwrap();recovered.attach_existing_run().unwrap();- assert!(recovered.durable_run_receipt_limit_reached());+ assert!(recovered.turn_closed());assert_eq!(recovered.replay_result("large-settled-0", "get_task_context", &json!({}))@@ -868,8 +915,16 @@ fn settled_result_byte_exhaustion_recovers_after_turn_cleanup() {.expect("releasing bulky results clears transient byte pressure");recovered.settle_turn("provider_turn_terminated")- .expect("the admitted next-turn call remains settleable");-+ .expect("closing admission preserves the admitted next-turn call");+ assert!(recovered.attach_run(tools("computed")).is_err());+ recovered+ .apply_result(ToolResult {+ call_id: "after-turn-boundary".into(),+ operation_id: "get_task_context".into(),+ result: json!({"ok":true}),+ is_error: false,+ })+ .unwrap();recovered.attach_run(tools("computed")).unwrap();recovered.begin_call(@@ -1003,18 +1058,19 @@ fn recovery_rejects_state_without_room_for_a_pending_result() {}#[test]-fn settles_pending_receipts_with_explicit_terminal_results() {+fn stopped_provider_does_not_prove_pending_effect_failed() {let mut bridge = ProviderToolBridge::default();bridge.prepare(tools("computed")).unwrap();bridge.begin_call("call-1".into(), "get_task_context".into(), json!({})).unwrap();-- let settled = bridge.settle_turn("provider_turn_terminated").unwrap();- assert_eq!(settled.len(), 1);- assert_eq!(settled[0].call_id, "call-1");- assert!(settled[0].is_error);- assert_eq!(bridge.pending_calls().count(), 0);+ assert!(bridge+ .settle_turn("provider_turn_terminated")+ .unwrap()+ .is_empty());+ assert_eq!(bridge.pending_calls().count(), 1);+ assert!(!bridge.has_completed_call("call-1"));+ assert!(bridge.prepare_turn().is_err());}#[test]diff --git a/packages/paperclip-runner/src/contracts/native-session-backend.ts b/packages/paperclip-runner/src/contracts/native-session-backend.tsindex 3fb4f928e5..7a84e9577d 100644--- a/packages/paperclip-runner/src/contracts/native-session-backend.ts+++ b/packages/paperclip-runner/src/contracts/native-session-backend.ts@@ -93,11 +93,26 @@ export interface NativeSessionSnapshotOptions {signal: AbortSignal;}+/** A dispatched operation may have taken effect, but has no proven result. */+export class SemanticToolOutcomeUnknownError extends Error {+ readonly code = "semantic_tool_outcome_unknown";++ constructor(message: string, options?: ErrorOptions) {+ super(message, options);+ this.name = "SemanticToolOutcomeUnknownError";+ }+}++export function isSemanticToolOutcomeUnknownError(error: unknown): error is SemanticToolOutcomeUnknownError {+ // Match across the server's vendored and source-mode runtime boundaries.+ return error instanceof Error && "code" in error && error.code === "semantic_tool_outcome_unknown";+}+/** The exact close owner has torn down its controller without proving suspension. */export class NativeSessionCloseUnrecoverableError extends Error {readonly code = "native_session_close_unrecoverable";- constructor() {+ constructor(readonly settlement?: Record<string, unknown>) {super("provider_transport_failed: runner did not durably suspend before checkpoint",);diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.tsindex 283429f55d..f193280606 100644--- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts+++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts@@ -26,6 +26,7 @@ import { validatePrpEvent } from "../protocol/replay-contract.js";import { digestPaperclipSemanticContent } from "../semantic-tools/receipts.js";import {DurablePrpControlPlane,+ SemanticToolNotDispatchedError,inspectWarmRunTransition,spawnRunner,type RunnerProcessLaunchSpec,@@ -819,7 +820,9 @@ function sendMaskedJson(socket: Socket, value: unknown): void {} else if (payload.length <= 0xffff) {header.push(0x80 | 126, payload.length >>> 8, payload.length & 0xff);} else {- throw new Error("Test client frame exceeds the supported size.");+ const length = Buffer.alloc(8);+ length.writeBigUInt64BE(BigInt(payload.length));+ header.push(0x80 | 127, ...length);}const masked = Buffer.from(payload);for (let index = 0; index < masked.length; index += 1) {@@ -2216,7 +2219,74 @@ describe.sequential("DurablePrpControlPlane", () => {}});- it("recovers one semantic call from the durable event after a coordinator restart", async () => {+ it("delivers a large lossless result without copying its large input into the command", async () => {+ const root = mkdtempSync(resolve(tmpdir(), "paperclip-large-tool-result-"));+ const content = "日本語🦀".repeat(34_000) + "END";+ const args = { content };+ const handler = vi.fn(async () => ({ result: { content } }));+ const options = { stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest, onSemanticToolInput: handler };+ const core = new DurablePrpControlPlane(options);+ try {+ await core.start();+ const client = (await authenticate(core, core.issueBootstrapTicket()))!;+ const event = semanticInputEvent();+ const semantic = ((event.payload as any).payload as any).semantic_tool;+ semantic.input = args;+ semantic.content.digest = digestPaperclipSemanticContent(args);+ sendSecure(client, event);+ const frames = [await receiveSecure(client), await receiveSecure(client)];+ const wire = frames.find((frame) => frame?.kind === "command")!.payload as any;+ expect(wire.payload.result.content).toBe(content);+ expect(wire.payload.input).toBeUndefined();+ expect(wire.payload.inputDigest).toBe(digestPaperclipSemanticContent(args).slice("sha256:".length));+ expect(Buffer.byteLength(JSON.stringify(wire))).toBeLessThan(1024 * 1024);+ sendSecure(client, { protocol: "paperclip.runner", version: 1, kind: "command_result", payload: {+ commandId: wire.commandId, controllerSeq: wire.controllerSeq, commandType: wire.type,+ status: "completed", result: { status: "delivered", callId: "call-1" },+ } });+ await vi.waitFor(() => expect(core.store.state.commands[0]?.status).toBe("completed"));+ expect(core.semanticToolResultsSettled()).toBe(true);+ const savedDigest = core.store.state.commands[0]!.payload.inputDigest;+ core.store.state.commands[0]!.payload.inputDigest = `sha256:${"0".repeat(64)}`;+ expect(core.semanticToolResultsSettled()).toBe(false);+ core.store.state.commands[0]!.payload.inputDigest = savedDigest;+ const reopened = new DurablePrpControlPlane(options);+ expect(reopened.semanticToolResultsSettled()).toBe(true);+ await reopened.stop();+ expect(handler).toHaveBeenCalledTimes(1);+ client.socket.destroy();+ } finally { await core.stop(); rmSync(root, { recursive: true, force: true }); }+ });++ it.each(["before_dispatch", "during_effect"] as const)("settles only proven pre-dispatch cancellation (%s)", async (stage) => {+ const root = mkdtempSync(resolve(tmpdir(), "paperclip-tool-dispatch-boundary-"));+ const handler = vi.fn(async () => {+ if (stage === "before_dispatch") throw new SemanticToolNotDispatchedError();+ throw new Error("connection lost after possibly committing a write");+ });+ const core = new DurablePrpControlPlane({ stateDirectory: root, identity,+ expectedRunnerVersion, expectedRunnerDigest, onSemanticToolInput: handler });+ try {+ await core.start();+ const client = (await authenticate(core, core.issueBootstrapTicket()))!;+ sendSecure(client, semanticInputEvent());+ await receiveSecure(client);+ await vi.waitFor(() => expect(handler).toHaveBeenCalledTimes(1));+ if (stage === "before_dispatch") {+ await vi.waitFor(() => expect(core.store.state.commands).toHaveLength(1));+ expect(core.store.state.commands[0]).toMatchObject({ type: "semantic_tool.result", payload: {+ callId: "call-1", isError: true, result: { error: { code: "semantic_tool_not_dispatched" } },+ } });+ } else {+ await vi.waitFor(() => expect(core.semanticToolSettlementDiagnostics()).toMatchObject({ persistenceFailed: true }));+ expect(core.store.state.commands).toEqual([]);+ }+ expect(core.semanticToolResultsSettled()).toBe(false);+ client.socket.destroy();+ } finally { await core.stop(); rmSync(root, { recursive: true, force: true }); }+ });++ it("retains an uncertain semantic outcome without redispatching after a coordinator restart", async () => {const root = mkdtempSync(resolve(tmpdir(), "paperclip-prp-recovery-"));let firstCalls = 0;const first = new DurablePrpControlPlane({@@ -2264,22 +2334,13 @@ describe.sequential("DurablePrpControlPlane", () => {await recovered.start();const client = await authenticate(recovered, leaseToken!);sendSecure(client!, semanticInputEvent());- const outcomes = [- await receiveSecure(client!),- await receiveSecure(client!),- ];- const command = outcomes.find((outcome) => outcome?.kind === "command");- expect(outcomes.some((outcome) => outcome?.kind === "ack")).toBe(true);- expect(command?.payload).toMatchObject({- type: "semantic_tool.result",- payload: {- callId: "call-1",- operationId: "get_task_context",- result: { ok: true, operationId: "get_task_context" },- isError: false,- },+ await expect(receiveSecure(client!)).resolves.toMatchObject({ kind: "ack" });+ expect(recoveredCalls).toBe(0);+ expect(recovered.semanticToolResultsSettled()).toBe(false);+ expect(recovered.semanticToolSettlementDiagnostics()).toMatchObject({+ pending: [{ callId: "call-1", operationId: "get_task_context" }], deliveries: [],});- expect(recoveredCalls).toBe(1);+ expect(recovered.store.state.commands.filter((entry) => entry.type === "semantic_tool.result")).toEqual([]);const tampered = semanticInputEvent(2);const tamperedEvent = tampered.payload as Record<string, unknown>;diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.tsindex b3385626ec..62f1d6207b 100644--- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts+++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts@@ -52,7 +52,8 @@ const coreStateSchema = "paperclip.runner.durable.control-plane-state.v1";const transitionCoreStateSchema ="paperclip.runner.durable.control-plane-state.warm-transition.v1";const maxFrameBytes = 1024 * 1024;-const maxCommandBytes = maxFrameBytes - 4 * 1024;+// Secure frames hex-encode ciphertext. Reserve envelope/tag space as well.+const maxCommandBytes = Math.floor((maxFrameBytes - 4 * 1024) / 2);const maxCommands = 500;// A provider can emit several 100-event runner batches before the transport's// polling turn regains the event loop. Match the transport's explicit deferred@@ -224,6 +225,13 @@ interface SecureChannel {sessionId: string;}+/** Only the dispatch boundary may assert that no business operation started. */+export class SemanticToolNotDispatchedError extends Error {+ constructor() {+ super("semantic_tool_not_dispatched");+ }+}+export interface DurablePrpControlPlaneOptions {stateDirectory: string;identity: DurableRecoveryIdentity;@@ -586,7 +594,11 @@ function unsettledSemanticInput(command.payload.sourceEventType !== event.eventType ||canonicalJson(command.payload.correlation) !==canonicalJson(expectedCorrelation) ||- canonicalJson(command.payload.input) !== canonicalJson(semantic.input)+ (command.payload.inputDigest === undefined+ ? canonicalJson(command.payload.input) !== canonicalJson(semantic.input)+ : command.payload.inputDigest !== canonicalDigest(semantic.input)+ || (command.payload.input !== undefined+ && canonicalJson(command.payload.input) !== canonicalJson(semantic.input))));} catch {// Malformed retained evidence cannot establish settled authority, and@@ -1622,6 +1634,35 @@ export class DurablePrpControlPlane {);}+ /** Safe run-log evidence: identities and delivery state, never tool contents. */+ semanticToolSettlementDiagnostics(): Record<string, unknown> {+ const pending = this.#store.state.committedEvents.filter((event) =>+ unsettledSemanticInput(event, this.#store.state),+ );+ return {+ persistenceFailed: this.#semanticResultPersistenceFailed,+ pending: pending.map((entry) => {+ const event = entry.envelope.payload as Record<string, unknown>;+ const payload = event.payload as Record<string, unknown>;+ const tool = payload.semantic_tool as Record<string, unknown>;+ return {+ sourceEventId: entry.sourceEventId,+ callId: tool.callId,+ operationId: tool.operationId,+ inputDigest: canonicalDigest(tool.input),+ };+ }),+ deliveries: this.#store.state.commands+ .filter((command) => command.type === "semantic_tool.result" && command.status !== "completed")+ .map((command) => ({+ commandId: command.commandId,+ callId: command.payload.callId,+ operationId: command.payload.operationId,+ status: command.status,+ })),+ };+ }+/*** Atomically advances a settled reusable runner to a new run authority while* retaining its existing connection lease secret. The runner performs the@@ -3215,13 +3256,21 @@ export class DurablePrpControlPlane {const alreadyQueued = this.#store.state.commands.some((command) => command.commandId === commandId,);- if (!alreadyQueued && !this.#pendingSemanticCalls.has(commandId)) {+ // A committed input without a result after restart may already have+ // changed the outside world. Never redispatch it on event replay. Only+ // the original admitted callback may publish its result; missing outcome+ // evidence stays pending for authoritative reconciliation.+ if (existing === undefined && !alreadyQueued && !this.#pendingSemanticCalls.has(commandId)) {this.#pendingSemanticCalls.add(commandId);const queueResult = (result: unknown, isError: boolean): void => {try {+ // Retain the full input once in its canonical event. Copying a+ // large write into its result command can exceed the wire bound+ // after the write has already committed. Bind it by exact digest.+ const { input, ...metadata } = call;this.queueCommand("semantic_tool.result",- { ...call, result, isError },+ { ...metadata, inputDigest: canonicalDigest(input), result, isError },commandId,true,);@@ -3237,9 +3286,20 @@ export class DurablePrpControlPlane {.then((outcome) =>queueResult(outcome.result, outcome.isError === true),)- .catch(() =>- queueResult({ code: "semantic_tool_bridge_failed" }, true),- )+ .catch((error: unknown) => {+ if (error instanceof SemanticToolNotDispatchedError) {+ queueResult({ error: {+ code: "semantic_tool_not_dispatched",+ retryable: false,+ message: "The turn stopped before this operation was dispatched.",+ } }, true);+ return;+ }+ // A rejected dispatcher promise does not prove that its effect+ // rolled back. Do not fabricate a final failure or retry the write.+ this.#semanticResultPersistenceFailed = true;+ this.disconnectActiveRunner();+ }).finally(() => this.#pendingSemanticCalls.delete(commandId));}}diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.test.tsindex bc000a7925..3f68152e45 100644--- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.test.ts+++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.test.ts@@ -1,6 +1,7 @@import { describe, expect, it, vi } from "vitest";import { resolve as resolvePath } from "node:path";import { fileURLToPath } from "node:url";+import { SemanticToolOutcomeUnknownError } from "../../contracts/native-session-backend.js";import {CODEX_BLOCK_RESULT_OUTPUT_SCHEMA,@@ -2958,6 +2959,29 @@ describe("Codex app-server Codex driver", () => {);});+ it.each(["unknown", "validation"])("preserves the dynamic tool outcome boundary (%s)", async (outcome) => {+ const transport = new FakeCodexTransport();+ const error = outcome === "unknown"+ ? new SemanticToolOutcomeUnknownError("write may have committed")+ : new Error("invalid instruction input");+ const handler = vi.fn(async () => { throw error; });+ const session = await makeDriver([transport], {+ dynamicTools: [{ name: "update_agent_instructions", description: "Write instructions.", inputSchema: { type: "object" } }],+ dynamicToolHandler: handler,+ }).openSession({ runId: "run-uncertain-write", normalizedSessionId: "session-uncertain-write", workingDirectory: TEST_WORKING_DIRECTORY });+ await session.startTurn({ message: { role: "user", text: "Update instructions." } });+ const response = transport.invoke({ id: "rpc-write", method: "item/tool/call", params: {+ threadId: "thread-1", turnId: "turn-1", callId: "uncertain-write", tool: "update_agent_instructions", arguments: {},+ } });+ if (outcome === "unknown") await expect(response).rejects.toBe(error);+ else await expect(response).resolves.toMatchObject({ success: false });+ transport.push("turn/completed", { threadId: "thread-1", turn: { id: "turn-1", status: "completed", items: [] } });+ const events = await collectUntilTerminal(session.events());+ const completions = events.filter((event) => event.eventType === "item.completed" && event.itemId === "uncertain-write");+ expect(completions).toHaveLength(outcome === "unknown" ? 0 : 1);+ expect(handler).toHaveBeenCalledTimes(1);+ });+it("fails closed when an agent message changes a tool-committed result", async () => {const transport = new FakeCodexTransport();const session = await makeDriver([transport]).openSession({diff --git a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.tsindex a7843d516d..2b22e9bbb2 100644--- a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts+++ b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts@@ -404,6 +404,7 @@ export class CodexHarnessSessionawait this.transport.request("turn/interrupt", {threadId: this.opened.threadId,turnId,+ ...(reason ? { reason: boundedText(reason) } : {}),});if (this.activeTurnId !== turnId) {throw new HarnessOperationAlreadyTerminalError("interruption");diff --git a/packages/paperclip-runner/src/drivers/codex/codex-session-server-requests.ts b/packages/paperclip-runner/src/drivers/codex/codex-session-server-requests.tsindex 6e35218072..e23b9f942a 100644--- a/packages/paperclip-runner/src/drivers/codex/codex-session-server-requests.ts+++ b/packages/paperclip-runner/src/drivers/codex/codex-session-server-requests.ts@@ -1,4 +1,5 @@import { isAcpxCanonicalInputMethod } from "../acpx/profile-extensions.js";+import { isSemanticToolOutcomeUnknownError } from "../../contracts/native-session-backend.js";import type { HarnessRuntimeRequest, PaperclipQuestionSet } from "../../contracts/harness-driver.js";import {CODEX_BLOCK_TOOL_NAME,@@ -147,6 +148,10 @@ async function handleServerRequestBody();return dynamicToolResponse(result);} catch (error) {+ // Preserve uncertain effects for the durable controller. A normal+ // failed tool response would falsely settle this call and permit+ // the provider to continue as if the write had not happened.+ if (isSemanticToolOutcomeUnknownError(error)) throw error;const message = boundedText(error instanceof Error ? error.message : error,);diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.tsindex 7ee61f16de..a434a1061e 100644--- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts+++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts@@ -737,9 +737,10 @@ it("refuses a reusable close checkpoint when the local provider snapshot is unre}}, 15_000);-it.each(["after_budget", "within_budget", "persistence_failure"] as const)(+it.each(["after_budget", "within_budget", "interrupted_within_budget", "persistence_failure"] as const)("fences reusable suspension against late semantic completion (%s)",async (mode) => {+ const settles = mode === "within_budget" || mode === "interrupted_within_budget";const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-late-semantic-close-"),);@@ -760,23 +761,26 @@ it.each(["after_budget", "within_budget", "persistence_failure"] as const)(// The default 96/48-frame stress burst spends seconds on unrelated// durable text fsyncs before handler entry, consuming this barrier test's// wall-clock budget under the full suite. Stress cases retain defaults.- codexArgs: fakeCodexArgs(+ codexArgs: mode === "interrupted_within_budget"+ ? fakeCodexArgs(stateDirectory, "--emit-tool-call")+ : fakeCodexArgs(stateDirectory, "--split-event-burst","--split-event-prefix-count", "2", "--split-event-suffix-count", "2",),stateDirectory,- closeGraceMs: 2_000,+ closeGraceMs: 5_000,controlPlaneRegistration: async (authority) => {core = authority;await authority.start();return { checkpoint, release: () => undefined };},});- bundle.transport.setServerRequestHandler(async () => {+ const handler = vi.fn(async () => {entered();await handlerRelease;return { success: true, contentItems: [] };});+ bundle.transport.setServerRequestHandler(handler);try {await bundle.transport.request("thread/start", {cwd: tmpdir(),@@ -815,6 +819,10 @@ it.each(["after_budget", "within_budget", "persistence_failure"] as const)(return queue(type, ...args);});}+ if (mode === "interrupted_within_budget") {+ await bundle.transport.request("turn/interrupt", { reason: "test-stop-during-server-write" });+ expect(core.semanticToolResultsSettled()).toBe(false);+ }const closing = bundle.transport.close().then(() => null,(error: unknown) => error,@@ -825,7 +833,8 @@ it.each(["after_budget", "within_budget", "persistence_failure"] as const)(release();}const closeFailure = await closing;- if (mode !== "within_budget") {+ expect(handler).toHaveBeenCalledTimes(1);+ if (!settles) {const artifact = readRunnerdArtifactBinding(defaultCapabilityRunnerdBinary(),);@@ -863,9 +872,9 @@ it.each(["after_budget", "within_budget", "persistence_failure"] as const)(control.identity.runId,);expect(late[0].status).toBe(- mode === "within_budget" ? "completed" : "pending",+ settles ? "completed" : "pending",);- if (mode === "within_budget") {+ if (settles) {const results = control.committedEvents.filter((event: { eventType: string }) =>event.eventType === "semantic_tool.result",@@ -875,7 +884,7 @@ it.each(["after_budget", "within_budget", "persistence_failure"] as const)(}});}- if (mode === "within_budget") {+ if (settles) {expect(closeFailure).toBeNull();expect(core.semanticToolResultsSettled()).toBe(true);expect(checkpoint).toHaveBeenCalledWith("settled");@@ -883,6 +892,10 @@ it.each(["after_budget", "within_budget", "persistence_failure"] as const)(expect(closeFailure).toBeInstanceOf(NativeSessionCloseUnrecoverableError,);+ expect(closeFailure).toHaveProperty("settlement.semanticTools.pending", expect.arrayContaining([+ expect.objectContaining({ callId: expect.any(String), operationId: expect.any(String),+ sourceEventId: expect.any(String), inputDigest: expect.stringMatching(/^[a-f0-9]{64}$/) }),+ ]));expect(checkpoint).toHaveBeenCalledWith("unsettled");expect(checkpoint).not.toHaveBeenCalledWith("settled");}@@ -892,7 +905,7 @@ it.each(["after_budget", "within_budget", "persistence_failure"] as const)(await rm(stateDirectory, { recursive: true, force: true });}},- 15_000,+ 25_000,);it("infers a remote provider turn until its own terminal event is durable", () => {diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.tsindex 3947426a07..1a49999f21 100644--- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts+++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts@@ -48,6 +48,7 @@ import type {} from "../contracts/harness-driver.js";import {DurablePrpControlPlane,+ SemanticToolNotDispatchedError,durableRecoveryInternals,inspectWarmRunTransition,spawnRunner,@@ -4355,7 +4356,18 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {this.#controlPlaneRelease = null;}if (suspensionRequired && !runnerSettled) {- throw new NativeSessionCloseUnrecoverableError();+ const settlement = {+ runnerSuspended,+ providerDrained,+ semanticTools: this.#core?.semanticToolSettlementDiagnostics(),+ finalProviderState,+ };+ try {+ this.options.onDiagnostic?.(`native_session_settlement_incomplete ${JSON.stringify(settlement)}`);+ } catch {+ // Keep the settlement failure authoritative if its observer fails.+ }+ throw new NativeSessionCloseUnrecoverableError(settlement);}if (this.#ownsRoot && !adoptedRunner) {rmSync(this.#root, { recursive: true, force: true });@@ -5497,7 +5509,11 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {NonNullable<DurablePrpControlPlaneOptions["onSemanticToolInput"]>>[0],) {- this.#throwIfFailed();+ try {+ this.#throwIfFailed();+ } catch {+ throw new SemanticToolNotDispatchedError();+ }const core = this.#core;const threadId = this.#threadId;const epoch = this.#turnStartResponseEpoch;@@ -5508,8 +5524,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {const accepted =admission === null? true- : await Promise.race([admission.settled, this.#failureSignal]);- this.#throwIfFailed();+ : await Promise.race([admission.settled, this.#failureSignal]).catch(() => {+ throw new SemanticToolNotDispatchedError();+ });+ try {+ this.#throwIfFailed();+ } catch {+ throw new SemanticToolNotDispatchedError();+ }if (!accepted ||this.#closed ||@@ -5522,9 +5544,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {core.store.state.identity.normalizedSessionId ||call.correlation.turnId !== core.store.state.identity.turnId) {- throw new Error(- "PRP semantic tool call no longer belongs to an admitted turn",- );+ throw new SemanticToolNotDispatchedError();}const outcome = unwrapToolResponse(await this.#handler({diff --git a/packages/paperclip-runner/src/native-session-runtime.test.ts b/packages/paperclip-runner/src/native-session-runtime.test.tsindex 1c02ca05c5..2be94a1100 100644--- a/packages/paperclip-runner/src/native-session-runtime.test.ts+++ b/packages/paperclip-runner/src/native-session-runtime.test.ts@@ -3901,7 +3901,7 @@ describe("executeNativeSession recovery", () => {{ typed: false, closeFails: true },{ typed: true, closeFails: true, startupRace: true },])(- "preserves a permanent integrity failure through required cleanup (%j)",+ "preserves the primary execution failure through required cleanup (%j)",async ({ typed, closeFails, startupRace = false }) => {const failure = typed? new NativeSessionProtocolIntegrityError(@@ -3996,8 +3996,9 @@ describe("executeNativeSession recovery", () => {timeoutMs: 900_000,};await expect(executeNativeSession(options)).rejects.toBe(- typed ? failure : closeFailure,+ failure,);+ if (closeFails) expect(failure).toHaveProperty("cleanupError", closeFailure);expect(close).toHaveBeenCalledOnce();expect(onSession).toHaveBeenLastCalledWith(null);expect(port.completeRun).not.toHaveBeenCalled();@@ -4065,6 +4066,8 @@ describe("executeNativeSession recovery", () => {async *events() { throw new Error("cancelled remote transport"); },startTurn: async () => ({ turnId: "remote-turn" }),result: async () => null,+ async snapshot() { return { backendKind: "mock", sessionId: "stop-session", identity: this.identity(),+ providerSessionId: "stop-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },close: vi.fn(async () => { throw failure; }),};const backend: NativeSessionBackend = {@@ -4080,7 +4083,7 @@ describe("executeNativeSession recovery", () => {const options = { input: scopedInput, backend, controlPlane, runnerInstanceId: "remote-runner",controlPlaneInstanceId: "control", requireSessionCloseBeforeReturn: true,remoteCleanupScope: binding.remoteCleanupScope };- await expect(executeNativeSession(options)).rejects.toBe(failure);+ await expect(executeNativeSession(options)).rejects.toMatchObject({ message: "cancelled remote transport", cleanupError: failure });expect(completeTerminatedRemoteNativeSessionCleanup({ ...binding, runId: "other-run" })).toBe(true);expect(completeTerminatedRemoteNativeSessionCleanup({ ...binding, companyId: "other-company" })).toBe(true);await expect(executeNativeSession(options)).rejects.toBeInstanceOf(NativeSessionCleanupQuarantinedError);@@ -4091,7 +4094,7 @@ describe("executeNativeSession recovery", () => {const independentBackend = { ...backend, openSession: vi.fn(async () => independentSession) };await expect(executeNativeSession({ ...options, remoteCleanupScope: "other-sandbox",input: { ...scopedInput, binding: { ...scopedInput.binding, runId: independent.runId } },- backend: independentBackend })).rejects.toBe(failure);+ backend: independentBackend })).rejects.toMatchObject({ message: "cancelled remote transport", cleanupError: failure });expect(independentBackend.openSession).toHaveBeenCalledOnce();expect(completeTerminatedRemoteNativeSessionCleanup(binding)).toBe(true);// Same company/run, different sandbox: its quarantine must remain intact.@@ -4099,7 +4102,7 @@ describe("executeNativeSession recovery", () => {backend: independentBackend })).rejects.toBeInstanceOf(NativeSessionCleanupQuarantinedError);expect(independentBackend.openSession).toHaveBeenCalledOnce();// Reopening is now possible; the old failure/result was never rewritten.- await expect(executeNativeSession(options)).rejects.toBe(failure);+ await expect(executeNativeSession(options)).rejects.toMatchObject({ message: "cancelled remote transport", cleanupError: failure });expect(backend.openSession).toHaveBeenCalledTimes(2);expect(controlPlane.completeRun).not.toHaveBeenCalled();completeTerminatedRemoteNativeSessionCleanup(binding);@@ -4116,6 +4119,8 @@ describe("executeNativeSession recovery", () => {identity: () => scopedIdentity, capabilities: async () => capabilities,async *events() { throw new Error("local transport stopped"); },startTurn: async () => ({ turnId: "local-turn" }), result: async () => null,+ async snapshot() { return { backendKind: "mock", sessionId: "stop-session", identity: this.identity(),+ providerSessionId: "stop-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },close: vi.fn(async () => { throw failure; }),};const backend: NativeSessionBackend = {@@ -4129,14 +4134,14 @@ describe("executeNativeSession recovery", () => {};const options = { input: scopedInput, backend, controlPlane, runnerInstanceId: binding.runnerInstanceId,controlPlaneInstanceId: "control", requireSessionCloseBeforeReturn: true };- await expect(executeNativeSession(options)).rejects.toBe(failure);+ await expect(executeNativeSession(options)).rejects.toMatchObject({ message: "local transport stopped", cleanupError: failure });completeTerminatedLocalNativeSessionCleanup({ ...binding, companyId: "other-company" });completeTerminatedLocalNativeSessionCleanup({ ...binding, runId: "other-run" });expect(completeTerminatedLocalNativeSessionCleanup({ ...binding, runnerInstanceId: "other-runner" })).toBe(false);await expect(executeNativeSession(options)).rejects.toBeInstanceOf(NativeSessionCleanupQuarantinedError);expect(backend.openSession).toHaveBeenCalledOnce();expect(completeTerminatedLocalNativeSessionCleanup(binding)).toBe(true);- await expect(executeNativeSession(options)).rejects.toBe(failure);+ await expect(executeNativeSession(options)).rejects.toMatchObject({ message: "local transport stopped", cleanupError: failure });expect(backend.openSession).toHaveBeenCalledTimes(2);expect(controlPlane.completeRun).not.toHaveBeenCalled();completeTerminatedLocalNativeSessionCleanup(binding);diff --git a/packages/paperclip-runner/src/native-session-runtime.ts b/packages/paperclip-runner/src/native-session-runtime.tsindex 27b5c204c3..933352b44d 100644--- a/packages/paperclip-runner/src/native-session-runtime.ts+++ b/packages/paperclip-runner/src/native-session-runtime.ts@@ -2123,7 +2123,7 @@ export async function executeNativeSession(let goalCheckpointRequiresSuspension = Boolean(options.sessionGoalControl || options.resumeSessionGoalHeartbeat || persistedSession?.goal,);- let protocolIntegrityFailure: NativeSessionProtocolIntegrityError | null = null;+ let executionFailure: { error: unknown } | null = null;try {// Ownership publication is part of the execution-owned lifetime. If the// callback fails, the finally block below still quarantines and closes the@@ -2730,9 +2730,7 @@ export async function executeNativeSession(executionSucceeded = true;return { ...durableExecutionResult, ...enrichment };} catch (error) {- if (error instanceof NativeSessionProtocolIntegrityError) {- protocolIntegrityFailure = error;- }+ executionFailure = { error };throw error;} finally {const shouldClose =@@ -2757,7 +2755,20 @@ export async function executeNativeSession(// The exact cleanup owner remains retained/quarantined above. Its// rejection must not turn permanent integrity failure into a// generic retryable transport failure at the control-plane boundary.- throw protocolIntegrityFailure ?? closeError;+ if (executionFailure !== null) {+ // Keep the original identity/classification (including arbitrary+ // thrown values). Cleanup remains inspectable without replacing+ // the initiating failure with a generic shutdown timeout.+ if (executionFailure.error instanceof Error) {+ try {+ Object.defineProperty(executionFailure.error, "cleanupError", {+ value: closeError, configurable: true,+ });+ } catch { /* Frozen errors still retain their original identity. */ }+ }+ throw executionFailure.error;+ }+ throw closeError;}await options.onSessionClosed?.();}diff --git a/server/src/services/native-runtime/agent-instruction-tools.integration.test.ts b/server/src/services/native-runtime/agent-instruction-tools.integration.test.tsindex fe79d00aa8..8bcebc91d3 100644--- a/server/src/services/native-runtime/agent-instruction-tools.integration.test.ts+++ b/server/src/services/native-runtime/agent-instruction-tools.integration.test.ts@@ -1,10 +1,10 @@import fs from "node:fs/promises";import os from "node:os";import path from "node:path";-import { randomUUID } from "node:crypto";-import { eq } from "drizzle-orm";-import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";-import { agents, authUsers, companies, companyMemberships, principalPermissionGrants, heartbeatRuns, issues, agentInstructionRevisions, createDb } from "@paperclipai/db";+import { createHash, randomUUID } from "node:crypto";+import { eq, sql } from "drizzle-orm";+import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";+import { activityLog, agents, authUsers, companies, companyMemberships, principalPermissionGrants, heartbeatRuns, issues, agentInstructionRevisions, createDb } from "@paperclipai/db";import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";import { resolveManagedInstructionsRoot } from "../agent-instructions.js";import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";@@ -64,6 +64,189 @@ describe("canonical instruction tools through native authority", () => {await expect(call("update_agent_instructions", { entryFile, content: "stale", baseRevisionId: first.revision.id })).rejects.toMatchObject({ status: 409 });expect((await call("read_agent_instructions")).content).toBe("changed\r\n");});+ it("serializes duplicate writes at the filesystem commit and replays the exact receipt after restart", async () => {+ const first = await call("read_agent_instructions");+ const content = "\uFEFF# Instructions — 日本語 🦀\r\n".repeat(600) + "\nFINAL TAIL\n";+ const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {+ targetAgentId, entryFile, content, baseRevisionId: first.revision.id,+ } };+ const originalRename = fs.rename.bind(fs);+ let reached!: () => void, release!: () => void;+ const atCommit = new Promise<void>((resolve) => { reached = resolve; });+ const barrier = new Promise<void>((resolve) => { release = resolve; });+ let writes = 0;+ const spy = vi.spyOn(fs, "rename").mockImplementation(async (source, destination) => {+ if (destination === path.join(root, entryFile)) {+ writes++;+ reached();+ await barrier;+ }+ return originalRename(source, destination);+ });+ try {+ const pending = authority.execute(request);+ await atCommit;+ const duplicate = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }).execute(request);+ expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(original);+ release();+ const receipt = await pending as any;+ expect(await duplicate).toEqual(receipt);+ authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });+ expect(await authority.execute(request)).toEqual(receipt);+ const bytes = await fs.readFile(path.join(root, entryFile));+ expect(bytes).toEqual(Buffer.from(content));+ expect(receipt.revision.byteLength).toBe(bytes.length);+ expect(receipt.revision.contentHash).toBe(createHash("sha256").update(bytes).digest("hex"));+ expect(writes).toBe(1);+ expect((await db.select().from(activityLog).where(eq(activityLog.runId, runId))).filter((event) => event.action === "agent.files_updated")).toHaveLength(1);+ await expect(authority.execute({ ...request, arguments: { ...request.arguments, content: "conflicting replay" } })).rejects.toThrow("idempotency_conflict");+ expect(await fs.readFile(path.join(root, entryFile))).toEqual(bytes);+ await expect(call("update_agent_instructions", { entryFile, baseRevisionId: receipt.revision.id, content: "🦀".repeat(300_000) })).rejects.toMatchObject({ status: 422 });+ expect(writes).toBe(1);+ await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId));+ await expect(authority.execute(request)).rejects.toMatchObject({ status: 403 });+ } finally {+ release();+ spy.mockRestore();+ }+ });++ it.each(["after_file_rename", "receipt_persistence"])("retains durable attempt evidence and refuses replay after rollback at %s", async (fault) => {+ const first = await call("read_agent_instructions");+ const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {+ targetAgentId, entryFile, content: "written before database rollback\r\n", baseRevisionId: first.revision.id,+ } };+ const key = `instruction:${request.callId}`;+ if (fault === "receipt_persistence") {+ // This is an actual PostgreSQL transaction rollback, after the real+ // filesystem rename and attempted audit insert, before receipt commit.+ await db.execute(sql.raw(`CREATE FUNCTION reject_instruction_test_receipt() RETURNS trigger LANGUAGE plpgsql AS $$+ BEGIN+ IF NEW.id = '${runId}' AND NEW.result_json->'semanticToolReceipts' ? '${key}' THEN+ RAISE EXCEPTION 'injected receipt persistence failure';+ END IF;+ RETURN NEW;+ END $$`));+ await db.execute(sql.raw(`CREATE TRIGGER reject_instruction_test_receipt BEFORE UPDATE ON heartbeat_runs+ FOR EACH ROW EXECUTE FUNCTION reject_instruction_test_receipt()`));+ }+ const originalRename = fs.rename.bind(fs);+ let writes = 0;+ const spy = vi.spyOn(fs, "rename").mockImplementation(async (source, destination) => {+ await originalRename(source, destination);+ if (destination === path.join(root, entryFile)) {+ writes++;+ if (fault === "after_file_rename") throw new Error("injected failure after rename");+ }+ });+ try {+ await expect(authority.execute(request)).rejects.toMatchObject({ code: "semantic_tool_outcome_unknown" });+ expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(request.arguments.content);+ const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));+ expect((run!.resultJson as any).instructionToolAttempts[key]).toMatchObject({+ operationId: request.tool, targetAgentId, inputDigest: expect.stringMatching(/^[a-f0-9]{64}$/),+ });+ expect((run!.resultJson as any).semanticToolReceipts?.[key]).toBeUndefined();+ const audit = await db.select().from(activityLog).where(eq(activityLog.runId, runId));+ expect(audit.filter((event) => event.action === "agent.instruction_write_attempted")).toHaveLength(1);+ expect(audit.filter((event) => event.action === "agent.files_updated")).toHaveLength(0);+ authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });+ await expect(authority.execute(request)).rejects.toMatchObject({ code: "semantic_tool_outcome_unknown" });+ await expect(authority.execute({ ...request, arguments: { ...request.arguments, content: "different" } }))+ .rejects.toThrow("idempotency_conflict");+ expect(writes).toBe(1);+ await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId));+ await expect(authority.execute(request)).rejects.toMatchObject({ status: 403 });+ } finally {+ spy.mockRestore();+ if (fault === "receipt_persistence") {+ await db.execute(sql.raw("DROP TRIGGER reject_instruction_test_receipt ON heartbeat_runs"));+ await db.execute(sql.raw("DROP FUNCTION reject_instruction_test_receipt()"));+ }+ }+ });++ it("replays a committed receipt after its database commit acknowledgement is lost", async () => {+ const first = await call("read_agent_instructions");+ const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {+ targetAgentId, entryFile, content: "committed but acknowledgement lost\n", baseRevisionId: first.revision.id,+ } };+ const transaction = db.transaction.bind(db);+ let commits = 0;+ const spy = vi.spyOn(db, "transaction").mockImplementation(async (...args) => {+ const result = await transaction(...args);+ if (++commits === 2) throw new Error("injected lost commit acknowledgement");+ return result;+ });+ try {+ await expect(authority.execute(request)).rejects.toThrow("injected lost commit acknowledgement");+ const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));+ const receipt = (run!.resultJson as any).semanticToolReceipts[`instruction:${request.callId}`].result;+ expect(receipt).toMatchObject({ changed: true, content: request.arguments.content });+ authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });+ expect(await authority.execute(request)).toEqual(receipt);+ const audit = await db.select().from(activityLog).where(eq(activityLog.runId, runId));+ expect(audit.filter((event) => event.action === "agent.instruction_write_attempted")).toHaveLength(1);+ expect(audit.filter((event) => event.action === "agent.files_updated")).toHaveLength(1);+ expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(request.arguments.content);+ } finally { spy.mockRestore(); }+ });++ it.each(["validation", "stale_base", "oversized"])("replays a definite %s failure after restart without executing again", async (fault) => {+ const first = await call("read_agent_instructions");+ const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {+ targetAgentId, entryFile, content: fault === "oversized" ? "🦀".repeat(300_000) : "requested update",+ baseRevisionId: first.revision.id, ...(fault === "validation" ? { unexpected: true } : {}),+ } };+ if (fault === "stale_base") await fs.writeFile(path.join(root, entryFile), "newer canonical bytes");+ const originalRename = fs.rename.bind(fs);+ let writes = 0;+ const spy = vi.spyOn(fs, "rename").mockImplementation(async (source, destination) => {+ if (destination === path.join(root, entryFile)) writes++;+ return originalRename(source, destination);+ });+ try {+ const failure = await authority.execute(request).then(() => null, (error: unknown) => error) as any;+ expect(failure).toMatchObject({ status: fault === "validation" ? 400 : fault === "stale_base" ? 409 : 422 });+ const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));+ const saved = (run!.resultJson as any).instructionToolAttempts[`instruction:${request.callId}`].failure;+ expect(saved).toMatchObject({ status: failure.status, message: failure.message });+ // The stale request would now pass CAS if it were executed again.+ await fs.writeFile(path.join(root, entryFile), original);+ authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });+ await expect(authority.execute(request)).rejects.toMatchObject(saved);+ await expect(authority.execute(request)).rejects.toMatchObject(saved);+ await expect(authority.execute({ ...request, arguments: { ...request.arguments, content: "different" } }))+ .rejects.toThrow("idempotency_conflict");+ expect(writes).toBe(0);+ expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(original);+ const audit = await db.select().from(activityLog).where(eq(activityLog.runId, runId));+ expect(audit.filter((event) => event.action === "agent.instruction_write_attempted")).toHaveLength(1);+ expect(audit.filter((event) => event.action === "agent.files_updated")).toHaveLength(0);+ } finally { spy.mockRestore(); }+ });++ it("does not claim a final failure when its failure receipt cannot be persisted", async () => {+ const first = await call("read_agent_instructions");+ const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {+ targetAgentId, entryFile, content: "invalid input", baseRevisionId: first.revision.id, unexpected: true,+ } };+ const transaction = db.transaction.bind(db);+ let attempts = 0;+ const spy = vi.spyOn(db, "transaction").mockImplementation(async (...args) => {+ if (++attempts === 3) throw new Error("injected failure receipt outage");+ return transaction(...args);+ });+ try {+ await expect(authority.execute(request)).rejects.toMatchObject({ code: "semantic_tool_outcome_unknown" });+ const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));+ expect((run!.resultJson as any).instructionToolAttempts[`instruction:${request.callId}`].failure).toBeUndefined();+ authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });+ await expect(authority.execute(request)).rejects.toMatchObject({ code: "semantic_tool_outcome_unknown" });+ expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(original);+ } finally { spy.mockRestore(); }+ });+it("rejects supplied identity and missing CAS bases before changing content", async () => {const first = await call("read_agent_instructions");for (const identity of ["companyId", "agentId", "runId", "responsibleUserId", "onBehalfOfUserId", "actor"]) {diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.tsindex 27fd2f45e6..6155700edf 100644--- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts+++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts@@ -1,3 +1,4 @@+import { authorizeInstructionCommit } from "../agent-instruction-authorization.js";import { executeAgentInstructionTool } from "./agent-instruction-tools.js";import { createReadStream } from "node:fs";import { publicChatTaskUrl } from "../chat-task-url.js";@@ -26,7 +27,7 @@ import { getStorageService } from "../../storage/index.js";import type { StorageService } from "../../storage/types.js";import { assetService } from "../assets.js";import { workspaceFileResourceService } from "../workspace-file-resources.js";-import { badRequest, forbidden } from "../../errors.js";+import { badRequest, forbidden, notFound, HttpError } from "../../errors.js";import { searchRunnerApi } from "./runner-api-catalog.js";import { executeRunnerApi, validateRunnerApiCall, RUNNER_API_MAX_BYTES, type RunnerApiFile } from "./runner-api-client.js";import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES, RunnerApiResponseLimitError } from "./runner-api-response-limits.js";@@ -48,7 +49,7 @@ import {issues,issueThreadInteractions,} from "@paperclipai/db";-import { CAPABILITY_SEMANTIC_TOOL_CATALOG, runnerCodexDynamicToolsFit } from "../../vendor/paperclip-runner/index.js";+import { CAPABILITY_SEMANTIC_TOOL_CATALOG, runnerCodexDynamicToolsFit, SemanticToolOutcomeUnknownError } from "../../vendor/paperclip-runner/index.js";import { agentService } from "../agents.js";import { approvalService } from "../approvals.js";import { documentService } from "../documents.js";@@ -139,6 +140,25 @@ type ToolReceipt = {result: unknown;};+// Coalesce live callers, but never use this process-local lock as crash proof.+// The database attempt record below is the authority after process loss.+const instructionCallLocks = new WeakMap<Db, Map<string, Promise<void>>>();+const instructionPreWriteFailureStatuses = new Set([400, 401, 403, 404, 409, 422]);+async function withInstructionCallLock<T>(db: Db, key: string, work: () => Promise<T>): Promise<T> {+ let locks = instructionCallLocks.get(db);+ if (!locks) instructionCallLocks.set(db, locks = new Map());+ const previous = locks.get(key);+ let release!: () => void;+ const pending = new Promise<void>((resolve) => { release = resolve; });+ locks.set(key, pending);+ await previous;+ try { return await work(); }+ finally {+ release();+ if (locks.get(key) === pending) locks.delete(key);+ }+}+function record(value: unknown): Record<string, unknown> {return typeof value === "object" && value !== null && !Array.isArray(value)? value as Record<string, unknown>@@ -371,12 +391,14 @@ export class PaperclipRunnerToolAuthority {}switch (call.tool) {case "read_agent_instructions":- case "update_agent_instructions":case "get_agent_instruction_history":- case "restore_agent_instructions":return executeAgentInstructionTool({ db: this.db, binding: {companyId: this.binding.companyId, agentId: this.binding.agentId, runId: this.binding.runId,}, tool: call.tool, arguments: call.arguments });+ case "update_agent_instructions":+ case "restore_agent_instructions": {+ return this.#withInstructionMutationReceipt(call.tool, call.callId, input);+ }case "create_skill": {const apiUrl = this.binding.apiUrl ?? process.env.PAPERCLIP_API_URL;const token = createLocalAgentJwt(this.binding.agentId, this.binding.companyId, context.actor.adapterType, this.binding.runId, context.run.responsibleUserId);@@ -1476,6 +1498,109 @@ export class PaperclipRunnerToolAuthority {return null;}+ async #withInstructionMutationReceipt(+ tool: "update_agent_instructions" | "restore_agent_instructions",+ callId: string,+ input: Record<string, unknown>,+ ): Promise<unknown> {+ if (!callId || callId.length > 500) throw badRequest("A bounded runner call id is required");+ const key = `instruction:${callId}`;+ const digest = createHash("sha256").update(canonicalJson(input)).digest("hex");+ const targetAgentId = typeof input.targetAgentId === "string" ? input.targetAgentId : this.binding.agentId;+ const authorize = async (tx: Db) => {+ const [target] = await tx.select({ id: agents.id, companyId: agents.companyId }).from(agents)+ .where(and(eq(agents.id, targetAgentId), eq(agents.companyId, this.binding.companyId)));+ if (!target) throw notFound("Agent not found");+ return authorizeInstructionCommit(tx, {+ type: "agent", source: "agent_jwt", companyId: this.binding.companyId,+ agentId: this.binding.agentId, runId: this.binding.runId,+ }, target);+ };+ return withInstructionCallLock(this.db, `${this.binding.companyId}:${this.binding.runId}:${key}`, async () => {+ // A filesystem rename cannot roll back with PostgreSQL. Commit proof of+ // the attempt *before* permitting that effect. If the effect transaction+ // or its acknowledgement is lost, a missing result is unknown, not a+ // license to run the instruction write a second time.+ const reserved = await this.db.transaction(async (tx) => {+ const context = await this.#lockAuthorizedMutationContext(tx as unknown as Db);+ const bound = await authorize(tx as unknown as Db);+ const resultJson = record(context.run.resultJson);+ if (record(resultJson.semanticToolReceipts)[key] !== undefined) return false;+ const attempts = record(resultJson.instructionToolAttempts);+ const prior = record(attempts[key]);+ if (attempts[key] !== undefined) {+ if (prior.operationId !== tool || prior.inputDigest !== digest) {+ throw new Error("paperclip_runner_tool_idempotency_conflict");+ }+ const failure = record(prior.failure);+ if (typeof failure.status === "number" && instructionPreWriteFailureStatuses.has(failure.status)+ && typeof failure.message === "string") {+ throw new HttpError(failure.status, failure.message, failure.details);+ }+ throw new SemanticToolOutcomeUnknownError(`paperclip_runner_instruction_outcome_unknown call_id=${callId} operation_id=${tool} input_digest=${digest}`);+ }+ if (Object.keys(attempts).length >= 512) throw badRequest("Run instruction mutation limit reached");+ attempts[key] = { operationId: tool, inputDigest: digest, targetAgentId };+ await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, instructionToolAttempts: attempts } })+ .where(eq(heartbeatRuns.id, this.binding.runId));+ await tx.insert(activityLog).values({+ companyId: this.binding.companyId, actorType: "agent", actorId: this.binding.agentId,+ agentId: this.binding.agentId, runId: this.binding.runId,+ responsibleUserId: bound.type === "agent" ? bound.onBehalfOfUserId : null,+ action: "agent.instruction_write_attempted", entityType: "agent", entityId: targetAgentId,+ details: { callId, operationId: tool, inputDigest: digest },+ });+ return true;+ });+ try {+ return await this.#withMutationReceipt(tool, key, input, (tx) =>+ executeAgentInstructionTool({ db: tx, binding: {+ companyId: this.binding.companyId, agentId: this.binding.agentId, runId: this.binding.runId,+ }, tool, arguments: input }), { beforeReceiptReplay: async (tx) => { await authorize(tx); } });+ } catch (error) {+ // Instruction validation, authorization and CAS errors precede the+ // file write. Storage errors and lost commit acknowledgements do not+ // prove that a reserved filesystem effect rolled back.+ if (!reserved) throw error;+ if (error instanceof HttpError && instructionPreWriteFailureStatuses.has(error.status)) {+ const failure = { status: error.status, message: error.message, ...(error.details === undefined ? {} : { details: error.details }) };+ try {+ // Save the definite failure before exposing it as a final answer.+ // This is evidence of the admitted call, not another mutation, so+ // retain it even if the run stopped after the effect rolled back.+ await this.db.transaction(async (tx) => {+ const [run] = await tx.select().from(heartbeatRuns).where(and(+ eq(heartbeatRuns.id, this.binding.runId), eq(heartbeatRuns.companyId, this.binding.companyId),+ eq(heartbeatRuns.agentId, this.binding.agentId),+ )).for("update");+ if (!run) throw new Error("instruction_failure_run_missing");+ const resultJson = record(run.resultJson);+ const attempts = record(resultJson.instructionToolAttempts);+ const prior = record(attempts[key]);+ if (prior.operationId !== tool || prior.inputDigest !== digest+ || record(resultJson.semanticToolReceipts)[key] !== undefined) {+ throw new Error("instruction_failure_receipt_conflict");+ }+ attempts[key] = { ...prior, failure };+ await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, instructionToolAttempts: attempts } })+ .where(eq(heartbeatRuns.id, this.binding.runId));+ });+ } catch (persistenceError) {+ throw new SemanticToolOutcomeUnknownError(+ `paperclip_runner_instruction_failure_receipt_unavailable call_id=${callId}`,+ { cause: new AggregateError([error, persistenceError], "Definite instruction failure could not be saved") },+ );+ }+ throw error;+ }+ throw new SemanticToolOutcomeUnknownError(+ `paperclip_runner_instruction_outcome_unknown call_id=${callId}: ${error instanceof Error ? error.message : String(error)}`,+ { cause: error },+ );+ }+ });+ }+async #withMutationReceipt(operationId: string,idempotencyKey: string,diff --git a/server/src/services/run-log-store.test.ts b/server/src/services/run-log-store.test.tsindex ccadb78c30..75b7efda35 100644--- a/server/src/services/run-log-store.test.ts+++ b/server/src/services/run-log-store.test.ts@@ -85,6 +85,96 @@ describe("createDurableRunLogStore", () => {expect(calls.get).toBe(0); // local file present -> no S3 read});+ it.each([false, true])("retains every attempt across retry (pod restart: %s)", async (restart) => {+ const { provider, objects } = createMemoryProvider();+ let store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });+ const attempts: string[] = [];+ for (let attempt = 1; attempt <= 3; attempt++) {+ if (restart && attempt > 1) {+ await fs.rm(baseDir, { recursive: true, force: true });+ store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });+ }+ const handle = await store.begin(begin);+ attempts.push(handle.attemptId!);+ await store.append(handle, { stream: "stderr", chunk: `attempt-${attempt}`, ts: "same-time", seq: attempt });+ const summary = await store.finalize(handle);+ const saved = objects.get(handle.logRef)!;+ const lines = saved.toString().trim().split("\n").map((line) => JSON.parse(line));+ expect(lines.map((line) => line.chunk)).toEqual(Array.from({length: attempt}, (_, i) => `attempt-${i + 1}`));+ expect(lines.map((line) => line.attemptId)).toEqual(attempts);+ expect(new Set(attempts).size).toBe(attempt);+ expect(summary.bytes).toBe(saved.length);+ expect(summary.sha256).toBe(createHash("sha256").update(saved).digest("hex"));+ }+ });++ it("never replaces concurrent appends when two stores restore the same missing log", async () => {+ const { provider, objects } = createMemoryProvider();+ const firstStore = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });+ const secondStore = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });+ const original = await firstStore.begin(begin);+ await firstStore.append(original, { stream: "stderr", chunk: "original", ts: "t0" });+ await firstStore.finalize(original);+ const prefix = Buffer.from(objects.get(original.logRef)!);+ await fs.rm(baseDir, { recursive: true, force: true });+ let reached!: () => void, release!: () => void;+ const restoring = new Promise<void>((resolve) => { reached = resolve; });+ const barrier = new Promise<void>((resolve) => { release = resolve; });+ const spy = vi.spyOn(provider, "getObject").mockImplementationOnce(async () => ({+ stream: Readable.from((async function* () {+ yield prefix;+ reached();+ await barrier;+ })()),+ contentLength: prefix.length,+ }));+ const slow = firstStore.begin(begin);+ try {+ await restoring;+ const fast = await secondStore.begin(begin);+ await secondStore.append(fast, { stream: "stderr", chunk: "fast attempt", ts: "t1" });+ release();+ const late = await slow;+ await firstStore.append(late, { stream: "stderr", chunk: "slow attempt", ts: "t2" });+ await firstStore.finalize(late);+ const saved = objects.get(original.logRef)!;+ expect(saved).toEqual(await fs.readFile(path.join(baseDir, original.logRef)));+ expect(saved.toString().trim().split("\n").map((line) => JSON.parse(line).chunk))+ .toEqual(["original", "fast attempt", "slow attempt"]);+ } finally {+ release();+ await slow.catch(() => undefined);+ spy.mockRestore();+ }+ });++ it.each(["stream_failure", "short_read"])("preserves the durable prefix when restore fails (%s)", async (fault) => {+ const { provider, objects, calls } = createMemoryProvider();+ const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });+ const handle = await store.begin(begin);+ await store.append(handle, { stream: "stderr", chunk: "original failure evidence", ts: "t1" });+ await store.finalize(handle);+ const prefix = Buffer.from(objects.get(handle.logRef)!);+ await fs.rm(baseDir, { recursive: true, force: true });+ const spy = vi.spyOn(provider, "getObject").mockResolvedValueOnce({+ stream: Readable.from((async function* () {+ yield prefix.subarray(0, 10);+ if (fault === "stream_failure") throw new Error("storage connection lost");+ })()),+ contentLength: prefix.length,+ });+ try {+ await expect(store.begin(begin)).rejects.toThrow(+ fault === "stream_failure" ? "storage connection lost" : "incomplete prefix",+ );+ await expect(fs.stat(path.join(baseDir, handle.logRef))).rejects.toMatchObject({ code: "ENOENT" });+ expect(objects.get(handle.logRef)).toEqual(prefix);+ expect(calls.put).toBe(1);+ const next = await store.begin(begin);+ expect(await fs.readFile(path.join(baseDir, next.logRef))).toEqual(prefix);+ } finally { spy.mockRestore(); }+ });+it("uploads the complete log to S3 on finalize", async () => {const { provider, objects, calls } = createMemoryProvider();const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider, keyPrefix: "run-logs" } });@@ -133,7 +223,7 @@ describe("createDurableRunLogStore", () => {expect(summary.sha256).toBe(createHash("sha256").update(local).digest("hex"));expect(objects.get(handle.logRef)).toEqual(local);expect(local.toString()).toBe(outcome === "fails" ? "" : JSON.stringify({- ts: "t1", stream: "stderr", chunk: "late diagnostic",+ ts: "t1", attemptId: handle.attemptId, stream: "stderr", chunk: "late diagnostic",}) + "\n");} finally {release();diff --git a/server/src/services/run-log-store.ts b/server/src/services/run-log-store.tsindex 7e7b4597ea..85a51ba1e0 100644--- a/server/src/services/run-log-store.ts+++ b/server/src/services/run-log-store.ts@@ -1,6 +1,7 @@-import { createReadStream, promises as fs } from "node:fs";+import { createReadStream, createWriteStream, promises as fs } from "node:fs";import path from "node:path";-import { createHash } from "node:crypto";+import { pipeline } from "node:stream/promises";+import { createHash, randomUUID } from "node:crypto";import { addAbortSignal } from "node:stream";import { notFound } from "../errors.js";import { resolvePaperclipInstanceRoot } from "../home-paths.js";@@ -12,6 +13,7 @@ export type RunLogStoreType = "local_file";export interface RunLogHandle {store: RunLogStoreType;logRef: string;+ attemptId?: string;}export interface RunLogReadOptions {@@ -295,9 +297,37 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ruconst relPath = path.join(relDir, `${runId}.ndjson`);await ensureDir(relDir);const absPath = resolveWithin(basePath, relPath);- await fs.writeFile(absPath, "", "utf8");await retireInflightMirror(relPath);- return { store: "local_file", logRef: relPath };+ // Retries share a run ID. Never truncate their earlier evidence. On a+ // new pod, restore the durable prefix before opening another attempt.+ const exists = await fs.stat(absPath).then(() => true, (error) => {+ if (error.code === "ENOENT") return false;+ throw error;+ });+ if (!exists && s3) {+ const key = s3Key(relPath);+ const head = await s3.provider.headObject({ objectKey: key });+ if (head.exists) {+ const temporary = `${absPath}.${randomUUID()}.restore`;+ try {+ const object = await s3.provider.getObject({ objectKey: key });+ await pipeline(object.stream, createWriteStream(temporary, { flags: "wx", mode: 0o600 }));+ if (typeof head.contentLength === "number" && (await fs.stat(temporary)).size !== head.contentLength) {+ throw new Error("Durable run log restore returned an incomplete prefix");+ }+ // Publish the complete prefix without replacing a file another+ // attempt has already restored and started appending to.+ await fs.link(temporary, absPath).catch((error: NodeJS.ErrnoException) => {+ if (error.code !== "EEXIST") throw error;+ });+ } finally {+ await fs.rm(temporary, { force: true });+ }+ }+ }+ const file = await fs.open(absPath, "a", 0o600);+ await file.close();+ return { store: "local_file", logRef: relPath, attemptId: randomUUID() };},async append(handle, event) {@@ -305,6 +335,7 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ruconst absPath = resolveWithin(basePath, handle.logRef);const line = JSON.stringify({ts: event.ts,+ ...(handle.attemptId ? { attemptId: handle.attemptId } : {}),stream: event.stream,chunk: event.chunk,// Monotonic per-run sequence so readers can dedupe and order recordsdiff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.tsindex 4308fa67ff..4142f8fbf4 100644--- a/server/src/vendor/paperclip-runner/index.ts+++ b/server/src/vendor/paperclip-runner/index.ts@@ -78,6 +78,7 @@ export const NativeSessionCleanupQuarantinedError =runner.NativeSessionCleanupQuarantinedError;export const NativeSessionProtocolIntegrityError =runner.NativeSessionProtocolIntegrityError;+export const SemanticToolOutcomeUnknownError = runner.SemanticToolOutcomeUnknownError;export const PaperclipSemanticDispatcher = runner.PaperclipSemanticDispatcher;export const CAPABILITY_SEMANTIC_TOOL_CATALOG =runner.CAPABILITY_SEMANTIC_TOOL_CATALOG;