graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
Preserve prompt retries after connection failures
Rach Pradhan613117eadfd3
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 12bfed27..65668c71 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,13 @@ The release workflow uses a tag's section here as its release notes (ahand-written `docs/releases/<tag>.md` wins if present), so keeping this filecurrent is part of cutting a release.+## v0.302.51++- Retrying the same prompt after a connection failure works again, including after loading the failed conversation. Earlier work stays in history; reconnecting does not require a new chat.+- Includes independent conversation forks for subagents and workflow fan-out (`fork_from`), with optional retained workers for later continuation.+- Includes bounded workflow synthesis that returns earlier findings when the final report cannot complete.+- Retains all changes from v0.302.50.+## v0.302.48- oMLX is a built-in local provider (`omlx`, `127.0.0.1:8000`). Set its key with `graff key set omlx <key>` or `OMLX_API_KEY`. `graff models` lists the models it serves, `--model omlx` starts on the first one, `/effort` is sent as `reasoning_effort`, and it is billed as local (#1695).diff --git a/build.zig b/build.zigindex d2a44ac2..9ec7da12 100644--- a/build.zig+++ b/build.zig@@ -230,6 +230,9 @@ pub fn build(b: *std.Build) void {const retry_cancellation_test = b.addSystemCommand(&.{ "python3", "scripts/test-retry-cancellation.py" });retry_cancellation_test.addArtifactArg(exe);test_step.dependOn(&retry_cancellation_test.step);+ const offline_recovery_test = b.addSystemCommand(&.{ "python3", "scripts/test-offline-recovery.py" });+ offline_recovery_test.addArtifactArg(exe);+ test_step.dependOn(&offline_recovery_test.step);const run_tool_budget_test = b.addSystemCommand(&.{ "python3", "scripts/test-run-tool-budget.py" });run_tool_budget_test.addArtifactArg(exe);test_step.dependOn(&run_tool_budget_test.step);diff --git a/docs/adr/0301-failed-requests-settle-the-pending-turn.md b/docs/adr/0301-failed-requests-settle-the-pending-turn.mdnew file mode 100644index 00000000..9602a1cd--- /dev/null+++ b/docs/adr/0301-failed-requests-settle-the-pending-turn.md@@ -0,0 +1,36 @@+# 0301. Exhausted request retries settle the pending turn in history++## Context++A request that exhausts its transport or server retries returns an error while+leaving its user prompt as the last message. The duplicate-turn guard correctly+rejects consecutive host replays, but cannot distinguish that unanswered prompt+from a new, intentional retry. After reconnecting, the same prompt can therefore+return without making any model request. Saving and loading preserves the trap.++## Decision++At the shared request loop's terminal retry boundary, append a truthful assistant+marker saying the model request did not complete and earlier work remains in the+conversation. It is a historical failure boundary, not a fabricated answer or a+claim that tools did not run. The next identical prompt is a new user turn, both+in memory and after saving/loading. REPL, JSON and ACP use the same engine path.++Keep bounded retries, authentication handling, cancellation, duplicate-host+protection and provider selection unchanged. Do not stream the historical marker+as an additional answer or restart the failed request automatically. Ordinary+cloud inference still requires connectivity; existing local-model routes remain+the way to perform inference without it.++## Consequences++A saved conversation retains the failed attempt and its preceding tool results,+so reconnecting does not require restarting or replacing the conversation. The+marker also covers exhausted throttling/server retries, which otherwise leave+the same ambiguous trailing prompt. It is not a promise to resume a failed turn+without a new prompt, and does not add connectivity probes or automatic fallback.++`scripts/test-offline-recovery.py` drives actual ACP v1/v2 and JSON processes+against a failing loopback service, restores that service, and retries the exact+same prompt. ACP also reloads the failed checkpoint before retrying. The fixture+uses fake credentials and does not change the host's network configuration.diff --git a/docs/adr/README.md b/docs/adr/README.mdindex f18431c4..967e4d16 100644--- a/docs/adr/README.md+++ b/docs/adr/README.md@@ -324,6 +324,8 @@ record only when you need the evidence or the edge cases.| [0299](0299-tool-commands-launch-apps-in-the-background.md) | On macOS tool commands' `open` launches apps without activating them (a `-g` shim first on the tool PATH); `GRAFF_GUI_FOREGROUND=1` on a command is the explicit foreground opt-in. || [0300](0300-a-session-shuts-down-the-simulators-it-booted.md) | Shell commands that look like simulator work are bracketed by booted-device snapshots; devices a session booted are leased under `~/.codegraff/simulators/` and shut down when the last live leaseholder ends, never devices booted before. `GRAFF_KEEP_SIMULATORS=1` opts out. |+| [0301](0301-failed-requests-settle-the-pending-turn.md) | Exhausted request retries leave a truthful failure boundary in history so an identical prompt can run after reconnecting, including after save/load; duplicate-host protection and bounded retries stay unchanged. |+## When to write oneWrite an ADR when a decision is load-bearing and non-obvious: it was reacheddiff --git a/scripts/test-offline-recovery.py b/scripts/test-offline-recovery.pynew file mode 100644index 00000000..de559cfe--- /dev/null+++ b/scripts/test-offline-recovery.py@@ -0,0 +1,77 @@+#!/usr/bin/env python3+"""A failed connection must not swallow the same prompt after reconnecting.++Uses the retry fixture's loopback server, fake credentials and isolated HOME.+No real network outage, credentials, or model service is needed.+"""+import importlib.util+import json+from pathlib import Path+import sys+import tempfile++REPO = Path(__file__).resolve().parents[1]+spec = importlib.util.spec_from_file_location('retry_fixture', REPO / 'scripts/test-retry-cancellation.py')+fixture = importlib.util.module_from_spec(spec)+spec.loader.exec_module(fixture)+++class Client(fixture.Client):+ def until(self, predicate, timeout=30):+ return super().until(predicate, timeout=timeout)+++def exercise(binary, mode, reload_saved=False):+ with tempfile.TemporaryDirectory(prefix='graff-offline-recovery-') as temp:+ model = fixture.Model('transport')+ client = None+ try:+ root = Path(temp)+ client = Client(binary, root, model, mode)+ identifier = client.begin()+ if mode == 'json':+ failed = client.until(lambda event: event.get('type') == 'error', timeout=30)+ elif mode == 'acp2':+ failed = client.until(lambda event: event.get('params', {}).get('update', {}).get('stopReason') == '_error', timeout=30)+ else:+ failed = client.until(lambda event: event.get('id') == identifier, timeout=30)+ assert 'error' in failed, failed+ assert 'network error:' in json.dumps(failed), failed+ assert client.proc.poll() is None, 'connection failure killed the session'+ before = len(model.requests())+ assert before > 1, 'transient connection retries were bypassed'+ if reload_saved:+ saved = list((root / '.graff/sessions').glob('*.session.json'))+ assert len(saved) == 1, saved+ name = saved[0].name.removesuffix('.session.json')+ loaded = client.call('session/load', {'sessionId': name, 'cwd': str(root), 'mcpServers': []})+ assert 'result' in loaded, loaded+ client.sid = name+ model.recover()+ followup = client.begin() # Deliberately identical to the failed prompt.+ client.finish(followup)+ assert len(model.requests()) == before + 1, 'reconnected prompt was lost or replayed'+ assert client.proc.poll() is None, 'session exited after reconnect'+ print(f'PASS {mode}/' + ('reload' if reload_saved else 'live') + ': same prompt recovers after exhausted connection retries', flush=True)+ except BaseException:+ stderr = Path(temp) / 'stderr'+ if stderr.exists():+ print(stderr.read_text()[-2000:], file=sys.stderr)+ raise+ finally:+ if client:+ client.close()+ model.close()+++def main():+ binary = Path(sys.argv[1] if len(sys.argv) > 1 else REPO / 'zig-out/bin/graff').resolve()+ modes = sys.argv[2:] or ('acp1', 'acp2', 'json')+ for mode in modes:+ exercise(binary, mode)+ if mode == 'acp1': # The v2 draft does not advertise session/load.+ exercise(binary, mode, reload_saved=True)+++if __name__ == '__main__':+ main()diff --git a/src/agent_request.zig b/src/agent_request.zigindex 5a41a098..ede0d7cf 100644--- a/src/agent_request.zig+++ b/src/agent_request.zig@@ -391,6 +391,7 @@ pub fn request(self: *Agent, tools_in: ?[]const u8) !std.json.ObjectMap {const failure_kind = if (err == error.RateLimited) "rate limited (429)" else if (err == error.ServerError) "server error (5xx)" else "network error";self.last_api_error = std.fmt.allocPrint(self.arena, "{s}: {s}{s}{s} (gave up after {d} attempts)", .{ failure_kind, @errorName(err), if (down.len > 0) " at " else "", down, max_attempts }) catch null;self.last_request_write_failed = std.mem.eql(u8, @errorName(err), "WriteFailed");+ try scratch.preserveFailedRequest(self);if (telemetry.g_telem) |t| t.errorEvent("net", @errorName(err));if (self.tracer) |tr| tr.api(self.label, self.sub, self.provider.model, 0, body.len, 0, 0, 0, true);return error.ApiError;diff --git a/src/agent_request_scratch.zig b/src/agent_request_scratch.zigindex 0f0ceb98..68afe2f3 100644--- a/src/agent_request_scratch.zig+++ b/src/agent_request_scratch.zig@@ -23,6 +23,13 @@ pub fn reset(scratch: *std.heap.ArenaAllocator) void {}}+/// A terminal request failure is a completed attempt, not an unanswered host+/// replay. Keep a truthful boundary in history so an identical retry works,+/// including after saving and loading the conversation.+pub fn preserveFailedRequest(self: *Agent) !void {+ try self.messages.append(try @import("messages.zig").textMessage(self.arena, "assistant", "[The model request failed before a complete response was received. Earlier work remains in this conversation. Retry when the connection or service is available.]"));+}+/// Detached recaps are cosmetic: keep recovered transport details in the trace,/// but do not surface them as raw worker chatter in the normal REPL.pub fn showRecoveredTransportRetry(kind: run_budget_mod.CallKind) bool {