Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

Background jobs survive a crash and come back on resume

justrach3f2621ad5c74
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6f8e1e24..6545a5a7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -220,6 +220,9 @@ jobs:
- name: PTY /goal run holds for its own background work (#1278)
run: python3 scripts/test-pty-goal-hold.py zig-out/bin/graff
+ - name: Background jobs survive a killed graff and return on resume (ADR 0218)
+ run: python3 scripts/test-job-crash-resume.py zig-out/bin/graff
+
- name: PTY a follow-up typed mid-reply supersedes it (line REPL)
run: python3 scripts/test-pty-steer-mid-reply.py zig-out/bin/graff
diff --git a/docs/adr/0218-background-jobs-survive-a-crash.md b/docs/adr/0218-background-jobs-survive-a-crash.md
new file mode 100644
index 00000000..35716234
--- /dev/null
+++ b/docs/adr/0218-background-jobs-survive-a-crash.md
@@ -0,0 +1,71 @@
+# 0218. Background jobs survive a crash and come back on resume
+
+Status: accepted 2026-09-29
+
+## Context
+
+A finite command started with `run_in_background` wrote into pipes that
+graff read. When graff died without running its defers (a SIGKILL, a panic,
+or a closed terminal, since the TUI re-raises SIGHUP and SIGTERM with the
+default action), those pipes closed with it. The job died of SIGPIPE on its
+next write, or ran on with its output and exit status lost. The resumed
+session still held the job id, and `bash_output` could only call it
+"interrupted or unknown outcome".
+
+The unreal-agent harness keeps each running command's output on disk and
+re-attaches the command after a restart.
+
+## Decision
+
+- A finite root job started in the background writes its stdout and stderr
+ to files (opened `O_APPEND`) in `.graff/job-output/<id>/`, not to pipes.
+ `meta.json` beside them records the id, session, command, cwd, the
+ leader's pid and start identity, and the owning graff's pid and start
+ identity. The job's own shell writes an `exit` receipt from an EXIT trap.
+ The trap runs in that same shell, not in a wrapper process, so `exec cmd`
+ still makes `cmd` the group leader. The command runs under `eval` with no
+ positional parameters and `$0` of `/bin/sh`, as under `sh -c`.
+- While graff lives, a file pump (`job_durable.zig`) keeps the rules of the
+ pipe pump: the 256 KB unread cap, the #199 idle clock, kill, and detach at
+ session end. It polls every 10 ms, rising to 100 ms, and reaps its child
+ with `waitpid`. Once its reader has caught up past 8 MB, it empties the
+ file.
+- `reported` is written when the exit reaches the model: through
+ `bash_output`, through `bash_kill`, or when the completion notice is
+ delivered. Reaping the job at session end removes the directory.
+ A job kept past session end keeps its directory.
+- Loading a session (`--resume`, `/resume`, ACP `session/load`) scans the
+ current workspace's `.graff/job-output`. Captures from this session that
+ are unreported and whose owning graff is gone go back into the job pool
+ under their old ids (`job_recover.zig`). They are watched through the
+ receipt and the leader's start identity (#413), never `waitpid`.
+ - A job that ended while graff was down reports through the normal
+ completion notice.
+ - A job that is still running reports when it ends.
+ - A job that is gone without a receipt reports "ended" with its output.
+
+ `bash_output` and `bash_kill` reach these jobs again. A kill signals the
+ group only while the leader still carries its recorded start identity.
+- Captures from other sessions are left for their own resume. They are swept
+ once a week old and over. A reported capture whose owner is gone is
+ removed.
+- Some jobs stay on pipes: foreground commands (including ones parked after
+ the foreground wait, whose `cmd &` idioms depend on pipe EOF), servers,
+ subagent jobs, runs with no session, and Windows.
+
+## Consequences
+
+- A crash no longer kills or loses a background job. With ADR 0215, a
+ headless `--resume` waits for the re-attached jobs before it ends.
+- Job output sits on disk under `.graff/`, which git ignores (#1273). While
+ no graff is running, nothing trims it: a runaway job grows its file until
+ a resume catches up.
+- After a restart, the exit status comes only from the receipt. A command
+ that `exec`s, or that replaces the EXIT trap, then reports "ended" without
+ a code. While graff lives, `waitpid` still supplies its status.
+- stdout and stderr interleave per poll tick, not per write.
+- A notice that was delivered but not yet saved when graff dies is not
+ repeated. After resume, `bash_output` still returns that result.
+- Processes that a job leaves running in the background are not waited for.
+ Pipes behaved the same way once such a process redirected its output.
+- Follow-up: make a foreground command durable once it is parked.
diff --git a/docs/adr/README.md b/docs/adr/README.md
index 101e4bb4..9e13bfec 100644
--- a/docs/adr/README.md
+++ b/docs/adr/README.md
@@ -244,6 +244,7 @@ record only when you need the evidence or the edge cases.
| [0215](0215-headless-runs-wait-until-idle.md) | Headless roots (`-p`, `--json`, `graff serve`, piped stdin) wait for their own background work to report before a plain final ends the run. Servers do not count, a 10-minute heartbeat wakes the model, and interactive surfaces keep their idle wake. |
| [0216](0216-loop-is-retired.md) | `/loop` is retired: `/goal` is the one autonomous run, and waiting for background work is how every run works (ADR 0215). Typing `/loop` prints a pointer to `/goal` and makes no model call. |
| [0217](0217-follow-ups-supersede-the-streaming-reply.md) | A plain follow-up typed while the root reply streams cuts that stream, drops the uncommitted partial and rebuilds the request with the follow-up (not for compaction, children, started async tools, or GPT-6 WebSocket steer). TUI Enter hands plain text to the running turn; `/btw`, commands and images still wait; ACP is unchanged. |
+| [0218](0218-background-jobs-survive-a-crash.md) | A finite root job started in the background writes its output to `.graff/job-output/<id>/` with an exit receipt from its own shell, so it outlives a crashed graff. Loading the session re-attaches this session's unreported jobs under their old ids, reports what ended while graff was down, and sweeps other sessions' stale captures. Foreground commands, servers, subagent jobs and Windows keep pipes. |
## When to write one
diff --git a/scripts/test-job-crash-resume.py b/scripts/test-job-crash-resume.py
new file mode 100644
index 00000000..280767f2
--- /dev/null
+++ b/scripts/test-job-crash-resume.py
@@ -0,0 +1,176 @@
+#!/usr/bin/env python3
+"""Offline crash-survival regression (ADR 0218): background jobs outlive a
+SIGKILLed graff, and resuming the session re-attaches them.
+
+Graff A starts two background jobs, then dies mid-request with no shutdown.
+One job finishes while no graff is running; the other is still running when
+graff B resumes the session. B must deliver the first job's exit as a
+completion notice, and its bash_output must wait for the second job and
+report its real exit. Before ADR 0218 both died with A's pipes and B could
+only call the handles "interrupted or unknown".
+"""
+import argparse, json, os, re, signal, subprocess, sys, tempfile, threading, time
+from pathlib import Path
+REPO = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(REPO / 'scripts/eval'))
+from mock_model import ScriptedModel
+
+EARLY = 'sleep 1; echo EARLY-DONE; exit 7'
+LATE = 'sleep 5; echo LATE-DONE'
+
+
+def wait_until(fn, timeout=15):
+ end = time.monotonic() + timeout
+ while time.monotonic() < end:
+ if fn(): return True
+ time.sleep(.05)
+ return False
+
+
+def tool_text(body):
+ return [m.get('content', '') for m in body.get('messages', []) if m.get('role') == 'tool']
+
+
+def user_text(body):
+ out = []
+ for m in body.get('messages', []):
+ if m.get('role') != 'user': continue
+ c = m.get('content', '')
+ out.append(c if isinstance(c, str) else json.dumps(c))
+ return '\n'.join(out)
+
+
+def job_id(text):
+ found = re.search(r'\[job (\d+) started:', text)
+ if not found: raise AssertionError(f'no started handle in {text[:300]!r}')
+ return int(found.group(1))
+
+
+class First(ScriptedModel):
+ """Starts both jobs, then holds its third request open until killed."""
+ def __init__(self):
+ super().__init__([])
+ self.ids, self.holding, self.release = [], threading.Event(), threading.Event()
+
+ def next_reply(self, body):
+ with self._lock:
+ self.requests.append(body)
+ n = len(self.requests)
+ if n == 1:
+ return {'tool': 'shell', 'arguments': {'action': 'run', 'command': EARLY, 'run_in_background': True}}
+ if n == 2:
+ self.ids.append(job_id(tool_text(body)[-1]))
+ return {'tool': 'shell', 'arguments': {'action': 'run', 'command': LATE, 'run_in_background': True}}
+ if n == 3:
+ self.ids.append(job_id(tool_text(body)[-1]))
+ self.holding.set()
+ self.release.wait(30)
+ return {'text': 'A done'}
+
+
+class Second(ScriptedModel):
+ """Reads what the resumed session delivered, then waits on the late job."""
+ def __init__(self, late):
+ super().__init__([])
+ self.late, self.seen, self.done = late, {}, threading.Event()
+
+ def next_reply(self, body):
+ with self._lock:
+ self.requests.append(body)
+ n = len(self.requests)
+ if n == 1:
+ self.seen['first_request_users'] = user_text(body)
+ return {'tool': 'bash_output', 'arguments': {'id': self.late, 'wait_ms': 20000}}
+ if n == 2:
+ self.seen['late_output'] = tool_text(body)[-1]
+ self.done.set()
+ return {'text': 'B done'}
+ return {'text': 'done'}
+
+
+def start(binary, directory, model, name, extra):
+ port = model.start(0)
+ env = {k: v for k, v in os.environ.items() if not k.endswith('_API_KEY') and not k.startswith(('GRAFF_', 'HARNESS_'))}
+ env.update(HOME=str(directory / 'home'), AI_GATEWAY_API_KEY='local', GRAFF_FLEET='off',
+ GRAFF_VERCEL_URL=f'http://127.0.0.1:{port}/v1/chat/completions', GRAFF_NO_TELEMETRY='1',
+ GRAFF_NO_SMOLIFY='1', GRAFF_BEHAVIOR_UPLOAD='off', NO_COLOR='1')
+ out = open(directory / (name + '.stdout'), 'w')
+ err = open(directory / (name + '.stderr'), 'w')
+ proc = subprocess.Popen([str(binary), '--json', '--yolo', '--old', '--no-lean', '--max-model-calls', '6', '--model', 'vercel', *extra],
+ cwd=directory, env=env, stdin=subprocess.PIPE, stdout=out, stderr=err, text=True, start_new_session=True)
+ proc.stdin.write(json.dumps({'type': 'user', 'text': 'Run the scripted jobs and handle their results.'}) + '\n')
+ proc.stdin.flush()
+ return proc, out, err
+
+
+def saved_session(directory, late):
+ for path in (directory / '.graff/sessions').glob('*.session.json'):
+ try:
+ if str(late) in path.read_text(): return path.name[:-len('.session.json')]
+ except OSError: pass
+ return None
+
+
+def kill_group(pid):
+ try: os.killpg(pid, signal.SIGKILL)
+ except (ProcessLookupError, PermissionError): pass
+
+
+def run(binary, expect):
+ directory = Path(tempfile.mkdtemp(prefix='graff-job-crash-'))
+ os.chmod(directory, 0o700)
+ (directory / 'home').mkdir(mode=0o700)
+ running, models, leaders = [], [], []
+ try:
+ a = First(); models.append(a)
+ proc_a, out_a, err_a = start(binary, directory, a, 'A', []); running.append((proc_a, out_a, err_a))
+ assert a.holding.wait(30), 'A did not start both jobs'
+ early, late = a.ids
+ assert wait_until(lambda: saved_session(directory, late) is not None), 'A never saved its session'
+ session = saved_session(directory, late)
+ for job in (early, late):
+ meta = directory / '.graff/job-output' / str(job) / 'meta.json'
+ if wait_until(meta.exists, 5): leaders.append(json.loads(meta.read_text())['pid'])
+ proc_a.kill(); proc_a.wait(timeout=5) # abrupt: no defers, no jobsReap
+ a.release.set()
+ time.sleep(2) # the early job ends while no graff runs
+
+ b = Second(late); models.append(b)
+ proc_b, out_b, err_b = start(binary, directory, b, 'B', ['--resume', session]); running.append((proc_b, out_b, err_b))
+ finished = b.done.wait(40)
+ users = b.seen.get('first_request_users', '')
+ late_out = b.seen.get('late_output', '')
+ checks = dict(
+ early_notice=bool(re.search(rf'\[job {early} exited 7:', users)) and 'EARLY-DONE' in users,
+ late_exit='exited with code 0' in late_out and 'LATE-DONE' in late_out,
+ )
+ proc_b.stdin.close() # a clean end reaps the jobs and their captures
+ proc_b.wait(timeout=20)
+ left = list((directory / '.graff/job-output').glob('*')) if (directory / '.graff/job-output').exists() else []
+ checks['captures_cleaned'] = not left
+ passed = finished and all(checks.values())
+ report = dict(binary=str(binary), session=session, early=early, late=late, passed=passed, **checks,
+ late_output=late_out[:300])
+ (directory / 'receipt.json').write_text(json.dumps(report, indent=2))
+ print(json.dumps(dict(directory=str(directory), **report), indent=2))
+ assert passed == (expect == 'pass'), f'expected {expect}, observed {passed}'
+ finally:
+ for model in models:
+ if hasattr(model, 'release'): model.release.set()
+ for process, out, err in running:
+ if process.poll() is None: process.kill()
+ process.wait(timeout=5)
+ out.close(); err.close()
+ for pid in leaders: kill_group(pid) # each job leads its own group
+ for model in models: model.stop()
+
+
+if __name__ == '__main__':
+ if os.name != 'posix':
+ print('SKIP job crash/resume: POSIX process-group fixture')
+ sys.exit(0)
+ parser = argparse.ArgumentParser()
+ parser.add_argument('binary', type=Path, nargs='?', default=REPO / 'zig-out/bin/graff')
+ parser.add_argument('--expect', choices=['pass', 'fail'], default='pass')
+ args = parser.parse_args()
+ run(args.binary.resolve(), args.expect)
diff --git a/src/commands_jobs.zig b/src/commands_jobs.zig
index 99cc10c9..0a8100f4 100644
--- a/src/commands_jobs.zig
+++ b/src/commands_jobs.zig
@@ -90,7 +90,7 @@ fn list(root: *Agent, out: *Io.Writer) !void {
"abnormal";
rows.append(arena, .{
.id = job.id,
- .pid = if (comptime posix) (job.child.id orelse 0) else 0,
+ .pid = job.group_pid, // also set for a job re-attached after a restart (ADR 0218)
.running = !job.done,
.status = status,
.age_ms = @intCast(@max(now - job.started_ms, 0)),
diff --git a/src/exec_bash.zig b/src/exec_bash.zig
index e8c0a9a1..e8197fab 100644
--- a/src/exec_bash.zig
+++ b/src/exec_bash.zig
@@ -389,6 +389,8 @@ fn execUnchecked(ctx: ToolCtx, call: tools.ToolCall) !ToolOutput {
// #1349: only a server-like command is persistent; a finite one
// backgrounded on purpose keeps the wait-until-exit `action=output`.
.persistent = bg and server_like,
+ // ADR 0218: a finite root job started in the background outlives a crash.
+ .session = if (bg and !server_like and !ctx.from_sub) ctx.session_name else "",
}) catch |err| return .{ .text = try spawnFailText(gpa, err), .is_error = true };
if (bg) {
@import("subagent_interactive.zig").request(ctx);
diff --git a/src/job_capture.zig b/src/job_capture.zig
new file mode 100644
index 00000000..68a248f3
--- /dev/null
+++ b/src/job_capture.zig
@@ -0,0 +1,282 @@
+//! Durable background jobs (crash survival, ADR 0218). A finite job started in
+//! the background writes its stdout and stderr straight to files under
+//! `.graff/job-output/<id>/` instead of pipes into graff, and its shell leaves
+//! an exit receipt beside them. graff dying mid-task (a crash, a SIGKILL, a
+//! closed terminal) then neither kills the job on its next write nor loses
+//! what it printed: resuming the session re-attaches a job that is still
+//! running and reports one that finished while graff was down
+//! (job_recover.zig).
+//!
+//! POSIX only. Servers and foreground commands keep pipes: a server never
+//! finishes, and a foreground call's `cmd &` idioms depend on pipe EOF.
+
+const std = @import("std");
+const builtin = @import("builtin");
+const Io = std.Io;
+const Allocator = std.mem.Allocator;
+
+pub const enabled = builtin.os.tag != .windows and builtin.os.tag != .wasi;
+pub const dir_rel = ".graff/job-output";
+
+/// A reader that has caught up past this empties the file: the job appends,
+/// so it carries on from 0 and a chatty job cannot fill the disk while graff
+/// watches it.
+pub const trunc_at: u64 = 8 * 1024 * 1024;
+
+/// The job's own shell writes its exit status to `$2` on the way out. An
+/// EXIT trap in that one shell, not a parent wrapper, so `exec cmd` still
+/// makes `cmd` the process-group leader. `set --` leaves the command with no
+/// positional parameters, as under `sh -c cmd`. A command that execs or sets
+/// its own EXIT trap leaves no receipt; graff's own waitpid still sees it.
+pub const wrapper = "__graff_cmd=$1 __graff_exit=$2; set --; trap 'printf \"%s\\n\" \"$?\" >\"$__graff_exit\"' EXIT; eval \"$__graff_cmd\"";
+
+/// What survives the process that started the job.
+pub const Meta = struct {
+ id: u64,
+ session: []const u8,
+ cmd: []const u8,
+ cwd: []const u8 = "",
+ pid: i32 = 0,
+ start_id: u64 = 0,
+ owner_pid: i32 = 0,
+ owner_start_id: u64 = 0,
+ started_ms: i64 = 0,
+};
+
+pub const Stream = enum(u1) { stdout, stderr };
+
+/// A job's capture directory and how far each output file has been read.
+pub const Capture = struct {
+ dir: []u8, // absolute, gpa-owned
+ offs: [2]u64 = .{ 0, 0 },
+ readers: [2]?Io.File = .{ null, null },
+ /// Re-attached after a restart: not graff's child, so it is watched
+ /// through its receipt and its leader's start identity, never waitpid.
+ adopted: bool = false,
+ start_id: u64 = 0,
+
+ pub fn path(self: *const Capture, buf: []u8, name: []const u8) []const u8 {
+ return std.fmt.bufPrint(buf, "{s}/{s}", .{ self.dir, name }) catch "";
+ }
+};
+
+/// Test override for the capture root, so unit tests never touch the
+/// workspace's own `.graff`.
+pub var test_root: ?[]const u8 = null;
+
+/// `<cwd>/.graff/job-output`, absolute: a job running in a worktree agent's
+/// cwd, or past a later workspace switch, still writes its receipt here.
+pub fn rootPath(io: Io, buf: []u8) ?[]const u8 {
+ if (builtin.is_test) if (test_root) |r| return std.fmt.bufPrint(buf, "{s}", .{r}) catch null;
+ var cwd: [std.fs.max_path_bytes]u8 = undefined;
+ const n = std.process.currentPath(io, &cwd) catch return null;
+ return std.fmt.bufPrint(buf, "{s}/" ++ dir_rel, .{cwd[0..n]}) catch null;
+}
+
+/// A capture made ready for spawn: the child's argv and its two output files.
+pub const Setup = struct {
+ capture: *Capture,
+ argv: [6][]const u8,
+ exit_path: []u8, // gpa-owned; argv borrows it until spawn returns
+ out: Io.File,
+ err: Io.File,
+
+ /// Spawn copied argv and dup'ed the files: the child holds its own.
+ pub fn spawned(self: *Setup, gpa: Allocator, io: Io) void {
+ self.out.close(io);
+ self.err.close(io);
+ gpa.free(self.exit_path);
+ }
+
+ /// Spawn failed: nothing runs, so nothing is kept.
+ pub fn abandon(self: *Setup, gpa: Allocator, io: Io) void {
+ self.spawned(gpa, io);
+ release(gpa, io, self.capture, true);
+ }
+};
+
+/// Create `<root>/<id>/` with empty output files and the argv that runs `cmd`
+/// with an exit receipt. Null when it cannot: durability is best-effort, and
+/// the caller keeps the job on pipes.
+pub fn prepare(io: Io, gpa: Allocator, id: u64, cmd: []const u8) ?Setup {
+ var rbuf: [std.fs.max_path_bytes]u8 = undefined;
+ const root = rootPath(io, &rbuf) orelse return null;
+ if (!builtin.is_test) @import("graff_dir.zig").ensure(io, Io.Dir.cwd()); // #1273: out of git
+ return create(io, gpa, root, id, cmd) catch null;
+}
+
+pub fn create(io: Io, gpa: Allocator, root: []const u8, id: u64, cmd: []const u8) !Setup {
+ const dir = try std.fmt.allocPrint(gpa, "{s}/{d}", .{ root, id });
+ errdefer gpa.free(dir);
+ try Io.Dir.cwd().createDirPath(io, dir);
+ errdefer Io.Dir.cwd().deleteTree(io, dir) catch {};
+ const capture = try gpa.create(Capture);
+ errdefer gpa.destroy(capture);
+ capture.* = .{ .dir = dir };
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ const out = try openAppend(capture.path(&buf, "stdout"));
+ errdefer out.close(io);
+ const err = try openAppend(capture.path(&buf, "stderr"));
+ errdefer err.close(io);
+ const exit_path = try gpa.dupe(u8, capture.path(&buf, "exit"));
+ return .{
+ .capture = capture,
+ .argv = .{ "/bin/sh", "-c", wrapper, "/bin/sh", cmd, exit_path },
+ .exit_path = exit_path,
+ .out = out,
+ .err = err,
+ };
+}
+
+/// O_APPEND, so a reader may empty the file under a writer (see trunc_at).
+fn openAppend(path: []const u8) !Io.File {
+ const fd = try std.posix.openat(std.posix.AT.FDCWD, path, .{ .ACCMODE = .WRONLY, .CREAT = true, .TRUNC = true, .APPEND = true, .CLOEXEC = true }, 0o600);
+ return .{ .handle = fd, .flags = .{ .nonblocking = false } };
+}
+
+/// Best-effort: a missing record costs recovery, never the job.
+pub fn writeMeta(io: Io, capture: *const Capture, meta: Meta) void {
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ var jbuf: [16 * 1024]u8 = undefined;
+ var w: Io.Writer = .fixed(&jbuf);
+ var s: std.json.Stringify = .{ .writer = &w };
+ s.write(meta) catch return;
+ Io.Dir.cwd().writeFile(io, .{ .sub_path = capture.path(&buf, "meta.json"), .data = w.buffered() }) catch {};
+}
+
+pub fn readMeta(io: Io, arena: Allocator, dir: []const u8) ?Meta {
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ const p = std.fmt.bufPrint(&buf, "{s}/meta.json", .{dir}) catch return null;
+ const data = Io.Dir.cwd().readFileAlloc(io, p, arena, .limited(64 * 1024)) catch return null;
+ return std.json.parseFromSliceLeaky(Meta, arena, data, .{ .ignore_unknown_fields = true, .allocate = .alloc_always }) catch null;
+}
+
+/// The recorded exit status, or null while the job runs (or when it ended
+/// without its shell reaching the EXIT trap).
+pub fn exitStatus(io: Io, dir: []const u8) ?u8 {
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ const p = std.fmt.bufPrint(&buf, "{s}/exit", .{dir}) catch return null;
+ var data: [16]u8 = undefined;
+ const text = Io.Dir.cwd().readFile(io, p, &data) catch return null;
+ return std.fmt.parseInt(u8, std.mem.trim(u8, text, " \t\r\n"), 10) catch null;
+}
+
+/// The exit reached the model: a restart must not report it again.
+pub fn markReported(io: Io, capture: *const Capture) void {
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ Io.Dir.cwd().writeFile(io, .{ .sub_path = capture.path(&buf, "reported"), .data = "" }) catch {};
+}
+
+pub fn reported(io: Io, dir: []const u8) bool {
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ const p = std.fmt.bufPrint(&buf, "{s}/reported", .{dir}) catch return false;
+ Io.Dir.cwd().access(io, p, .{}) catch return false;
+ return true;
+}
+
+/// New bytes of one output file since the last read, appended to `out`. Keeps
+/// the newest `keep` bytes of a larger burst and returns true when it skipped
+/// older ones.
+pub fn readNew(io: Io, gpa: Allocator, capture: *Capture, which: Stream, keep: usize, out: *std.ArrayList(u8)) bool {
+ const i = @intFromEnum(which);
+ const f = capture.readers[i] orelse blk: {
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ const opened = Io.Dir.cwd().openFile(io, capture.path(&buf, @tagName(which)), .{ .mode = .read_write }) catch return false;
+ capture.readers[i] = opened;
+ break :blk opened;
+ };
+ const off = &capture.offs[i];
+ const len = f.length(io) catch return false;
+ if (len < off.*) off.* = 0; // emptied elsewhere: start over
+ var skipped = false;
+ if (len - off.* > keep) {
+ off.* = len - keep;
+ skipped = true;
+ }
+ var chunk: [64 * 1024]u8 = undefined;
+ var bufs = [_][]u8{&chunk};
+ while (off.* < len) {
+ const n = f.readPositional(io, &bufs, off.*) catch break;
+ if (n == 0) break;
+ const take = @min(n, len - off.*);
+ out.appendSlice(gpa, chunk[0..take]) catch break;
+ off.* += take;
+ }
+ if (off.* >= trunc_at and off.* == (f.length(io) catch 0)) {
+ f.setLength(io, 0) catch return skipped;
+ off.* = 0;
+ }
+ return skipped;
+}
+
+/// Close the readers and free the capture. The directory goes too unless the
+/// job still writes to it (kept past session end).
+pub fn release(gpa: Allocator, io: Io, capture: *Capture, remove: bool) void {
+ for (capture.readers) |r| if (r) |f| f.close(io);
+ if (remove) Io.Dir.cwd().deleteTree(io, capture.dir) catch {};
+ gpa.free(capture.dir);
+ gpa.destroy(capture);
+}
+
+test "the receipt records the command's own exit status, and output lands in the files" {
+ if (!enabled) return error.SkipZigTest;
+ const gpa = std.testing.allocator;
+ const io = std.testing.io;
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var rbuf: [std.fs.max_path_bytes]u8 = undefined;
+ const root = rbuf[0..try tmp.dir.realPath(io, &rbuf)];
+ var setup = try create(io, gpa, root, 42, "printf out; printf err >&2; [ $# -eq 0 ] || exit 9; exit 7");
+ var child = try std.process.spawn(io, .{ .argv = &setup.argv, .stdin = .ignore, .stdout = .{ .file = setup.out }, .stderr = .{ .file = setup.err } });
+ const capture = setup.capture;
+ setup.spawned(gpa, io);
+ const term = try child.wait(io);
+ try std.testing.expectEqual(@as(u8, 7), term.exited);
+ try std.testing.expectEqual(@as(?u8, 7), exitStatus(io, capture.dir));
+ var out: std.ArrayList(u8) = .empty;
+ defer out.deinit(gpa);
+ try std.testing.expect(!readNew(io, gpa, capture, .stdout, 1024, &out));
+ try std.testing.expect(!readNew(io, gpa, capture, .stderr, 1024, &out));
+ try std.testing.expectEqualStrings("outerr", out.items);
+ _ = readNew(io, gpa, capture, .stdout, 1024, &out); // nothing new: offsets advanced
+ try std.testing.expectEqualStrings("outerr", out.items);
+
+ writeMeta(io, capture, .{ .id = 42, .session = "s-1", .cmd = "exit 7", .pid = 99 });
+ var arena = std.heap.ArenaAllocator.init(gpa);
+ defer arena.deinit();
+ const meta = readMeta(io, arena.allocator(), capture.dir).?;
+ try std.testing.expectEqual(@as(u64, 42), meta.id);
+ try std.testing.expectEqualStrings("s-1", meta.session);
+ try std.testing.expect(!reported(io, capture.dir));
+ markReported(io, capture);
+ try std.testing.expect(reported(io, capture.dir));
+ release(gpa, io, capture, true);
+ try std.testing.expectError(error.FileNotFound, tmp.dir.statFile(io, "42", .{}));
+}
+
+test "exec keeps the command as the group leader; a burst past keep skips its oldest bytes" {
+ if (!enabled) return error.SkipZigTest;
+ const gpa = std.testing.allocator;
+ const io = std.testing.io;
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var rbuf: [std.fs.max_path_bytes]u8 = undefined;
+ const root = rbuf[0..try tmp.dir.realPath(io, &rbuf)];
+ // `exec` replaces the job's shell, so no receipt: the leader is the command.
+ var setup = try create(io, gpa, root, 43, "printf 0123456789; exec /bin/sh -c 'echo $$ >&2'");
+ var child = try std.process.spawn(io, .{ .argv = &setup.argv, .stdin = .ignore, .stdout = .{ .file = setup.out }, .stderr = .{ .file = setup.err }, .pgid = 0 });
+ const pid = child.id.?;
+ const capture = setup.capture;
+ defer release(gpa, io, capture, true);
+ setup.spawned(gpa, io);
+ _ = try child.wait(io);
+ try std.testing.expectEqual(@as(?u8, null), exitStatus(io, capture.dir));
+ var out: std.ArrayList(u8) = .empty;
+ defer out.deinit(gpa);
+ try std.testing.expect(readNew(io, gpa, capture, .stdout, 4, &out));
+ try std.testing.expectEqualStrings("6789", out.items);
+ out.clearRetainingCapacity();
+ _ = readNew(io, gpa, capture, .stderr, 64, &out);
+ var pbuf: [16]u8 = undefined;
+ try std.testing.expectEqualStrings(try std.fmt.bufPrint(&pbuf, "{d}\n", .{pid}), out.items);
+}
diff --git a/src/job_durable.zig b/src/job_durable.zig
new file mode 100644
index 00000000..e5bec703
--- /dev/null
+++ b/src/job_durable.zig
@@ -0,0 +1,185 @@
+//! The pump for a job whose output goes to files (ADR 0218, job_capture.zig).
+//! Same contract as jobs.jobPump — drain output into the job buffer, keep the
+//! #199 idle clock, honour kill and detach, publish through jobs.finish — but
+//! it polls the files instead of reading pipes, and learns that the job
+//! ended from waitpid (graff's own child) or from the exit receipt and the
+//! leader's start identity (a job re-attached after a restart).
+
+const std = @import("std");
+const Io = std.Io;
+const Allocator = std.mem.Allocator;
+
+const jobs = @import("jobs.zig");
+const Job = jobs.Job;
+const g_jobs = &jobs.g_jobs;
+const job_capture = @import("job_capture.zig");
+const Capture = job_capture.Capture;
+const job_idle = @import("job_idle.zig");
+const job_registry = @import("job_registry.zig");
+const proc_identity = @import("proc_identity.zig");
+
+/// Poll ticks: a fresh job is checked often so a quick exit reports quickly;
+/// a long one settles at the max. An adopted job costs a process probe per
+/// tick, so it polls slower.
+const tick_first_ms: i64 = 10;
+const tick_max_ms: i64 = 100;
+const tick_adopted_ms: i64 = 250;
+
+/// Write the recovery record once the job runs (its pid is known now).
+pub fn record(io: Io, job: *Job, capture: *Capture, session: []const u8) void {
+ capture.start_id = switch (proc_identity.probe(io, job.group_pid)) {
+ .id => |v| v,
+ else => 0,
+ };
+ job_capture.writeMeta(io, capture, .{
+ .id = job.id,
+ .session = session,
+ .cmd = job.cmd,
+ .cwd = job.cwd orelse "",
+ .pid = job.group_pid,
+ .start_id = capture.start_id,
+ .owner_pid = proc_identity.selfPid(),
+ .owner_start_id = proc_identity.selfStartId(io),
+ .started_ms = job.started_ms,
+ });
+}
+
+pub fn pump(job: *Job, gpa: Allocator, io: Io) void {
+ const capture = job.capture.?;
+ const pid = job.group_pid;
+ var out: std.ArrayList(u8) = .empty;
+ defer out.deinit(gpa);
+ var err: std.ArrayList(u8) = .empty;
+ defer err.deinit(gpa);
+ var code: ?u8 = null;
+ var ended = false;
+ var killed = false;
+ var detached = false;
+ var tick = if (capture.adopted) tick_adopted_ms else tick_first_ms;
+ while (true) {
+ // The end first, then the output: whatever the job wrote before it
+ // exited is in the files by the time its exit is visible.
+ ended = if (capture.adopted) adoptedEnded(io, capture, pid, &code) else ownEnded(pid, &code);
+ const warn = drain(job, gpa, io, capture, &out, &err, !ended);
+ g_jobs.mutex.lockUncancelable(io);
+ killed = job.kill_requested;
+ detached = job.detach;
+ g_jobs.mutex.unlock(io);
+ if (warn) |idle_ms| job_idle.warn(io, job.id, idle_ms, job.cmd);
+ if (ended or killed or detached) break;
+ io.sleep(.fromMilliseconds(tick), .awake) catch {};
+ if (!capture.adopted) tick = @min(tick * 2, tick_max_ms);
+ }
+ if (!ended and killed and !detached) {
+ stop(io, capture, pid);
+ _ = drain(job, gpa, io, capture, &out, &err, false); // what it printed before the kill
+ }
+ // Reaped (or never ours): nothing may wait on or signal this child again.
+ job.child.id = null;
+ jobs.finish(job, io, if (ended) code else null, killed and !ended, detached and !ended, pid);
+}
+
+/// Read new output into the job buffer; with `idle`, also run the #199 idle
+/// clock. Returns the idle time when the warning is due.
+fn drain(job: *Job, gpa: Allocator, io: Io, capture: *Capture, out: *std.ArrayList(u8), err: *std.ArrayList(u8), idle: bool) ?u64 {
+ out.clearRetainingCapacity();
+ err.clearRetainingCapacity();
+ var skipped = job_capture.readNew(io, gpa, capture, .stdout, jobs.job_unread_cap, out);
+ if (job_capture.readNew(io, gpa, capture, .stderr, jobs.job_unread_cap, err)) skipped = true;
+ const now = jobs.nowMs(io);
+ g_jobs.mutex.lockUncancelable(io);
+ defer g_jobs.mutex.unlock(io);
+ jobs.appendOutput(job, gpa, 0, out.items, now);
+ jobs.appendOutput(job, gpa, 1, err.items, now);
+ if (skipped) job.dropped = true;
+ jobs.capUnread(job, gpa);
+ return if (idle) jobs.idleTick(job, gpa, io, job.group_pid, now) else null;
+}
+
+/// graff's own child: reap it without blocking once it exits.
+fn ownEnded(pid: i32, code: *?u8) bool {
+ if (pid <= 0) return true; // no leader to wait on (never 0: that waits on the group)
+ var status: c_int = 0;
+ while (true) {
+ const rc = std.c.waitpid(pid, &status, std.c.W.NOHANG);
+ if (rc == 0) return false;
+ if (rc == pid) {
+ const st: u32 = @bitCast(status);
+ code.* = if (std.posix.W.IFEXITED(st)) std.posix.W.EXITSTATUS(st) else null;
+ return true;
+ }
+ if (std.posix.errno(rc) == .INTR) continue;
+ code.* = null; // not ours to reap any more: ended, status unknown
+ return true;
+ }
+}
+
+/// A re-attached job: the receipt says it ended; a leader that is gone (or
+/// whose pid now names another process) without one ended with no status.
+fn adoptedEnded(io: Io, capture: *const Capture, pid: i32, code: *?u8) bool {
+ const state = job_registry.stateOf(capture.start_id, proc_identity.probe(io, pid));
+ code.* = job_capture.exitStatus(io, capture.dir);
+ return code.* != null or state == .gone;
+}
+
+/// bash_kill, the idle stop, or session end. The whole group goes (#198);
+/// a re-attached one only while its leader still carries the recorded start
+/// identity (#413), so a recycled pid never gets an unrelated tree killed.
+fn stop(io: Io, capture: *const Capture, pid: i32) void {
+ if (pid <= 0) return;
+ if (capture.adopted) {
+ if (job_registry.stateOf(capture.start_id, proc_identity.probe(io, pid)) == .running)
+ std.posix.kill(-pid, .KILL) catch {};
+ return;
+ }
+ std.posix.kill(-pid, .KILL) catch {};
+ var status: c_int = 0;
+ while (std.c.waitpid(pid, &status, 0) == -1 and std.posix.errno(@as(c_int, -1)) == .INTR) {}
+}
+
+test "a background job with a session writes to files, reports through bash_output, and dies to bash_kill" {
+ if (!job_capture.enabled) return error.SkipZigTest;
+ const gpa = std.testing.allocator;
+ const io = std.testing.io;
+ jobs.g_jobs = .{};
+ @import("job_notify.zig").resetForTest(io);
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var rbuf: [std.fs.max_path_bytes]u8 = undefined;
+ const root = rbuf[0..try tmp.dir.realPath(io, &rbuf)];
+ job_capture.test_root = root;
+ defer job_capture.test_root = null;
+ defer jobs.jobsReap(gpa, io);
+
+ const done = try jobs.spawnJobOpts(gpa, io, "printf a; printf b >&2; exit 4", .{ .session = "s-live" });
+ try std.testing.expect(done.capture != null);
+ const out = try jobs.jobOutput(gpa, io, done.id, 5000);
+ defer gpa.free(out.text);
+ try std.testing.expect(std.mem.indexOf(u8, out.text, "exited with code 4") != null);
+ try std.testing.expect(std.mem.indexOf(u8, out.text, "ab") != null);
+ var arena = std.heap.ArenaAllocator.init(gpa);
+ defer arena.deinit();
+ const meta = job_capture.readMeta(io, arena.allocator(), done.capture.?.dir) orelse return error.NoMeta;
+ try std.testing.expectEqualStrings("s-live", meta.session);
+ try std.testing.expectEqual(done.group_pid, meta.pid);
+ try std.testing.expect(job_capture.reported(io, done.capture.?.dir));
+
+ const slow = try jobs.spawnJobOpts(gpa, io, "sleep 30", .{ .session = "s-live" });
+ const pid = slow.group_pid;
+ const killed = try jobs.jobKill(gpa, io, slow.id);
+ defer gpa.free(killed.text);
+ try std.testing.expect(std.mem.indexOf(u8, killed.text, "killed") != null);
+ try std.testing.expect(proc_identity.probe(io, pid) == .gone);
+
+ // A server keeps its pipes; so does a job with no session to report to.
+ const server = try jobs.spawnJobOpts(gpa, io, "exit 0", .{ .session = "s-live", .persistent = true });
+ try std.testing.expect(server.capture == null);
+ const plain = try jobs.spawnJobOpts(gpa, io, "exit 0", .{});
+ try std.testing.expect(plain.capture == null);
+
+ // Session end reaps them all and drops their capture directories.
+ var nbuf: [32]u8 = undefined;
+ const name = try std.fmt.bufPrint(&nbuf, "{d}", .{done.id});
+ jobs.jobsReap(gpa, io);
+ try std.testing.expectError(error.FileNotFound, tmp.dir.statFile(io, name, .{}));
+}
diff --git a/src/job_foreground.zig b/src/job_foreground.zig
new file mode 100644
index 00000000..3de8e3d2
--- /dev/null
+++ b/src/job_foreground.zig
@@ -0,0 +1,91 @@
+//! Root foreground bash wait (#620 / grok-build auto-background), split out
+//! of jobs.zig (600-line cap): wait for the job, or promote it to the
+//! background at the bound, or kill it on Esc.
+
+const std = @import("std");
+const Io = std.Io;
+const Allocator = std.mem.Allocator;
+
+const jobs = @import("jobs.zig");
+const Job = jobs.Job;
+const g_jobs = &jobs.g_jobs;
+const nowMs = jobs.nowMs;
+const jobKill = jobs.jobKill;
+const job_wait = @import("job_wait.zig");
+const job_notify = @import("job_notify.zig");
+const browser_guard = @import("job_browser_guard.zig");
+const tool_pulse = @import("tool_pulse.zig");
+const Agent = @import("agent.zig").Agent;
+
+/// Outcome of a root foreground wait (#620 / grok-build auto-background).
+pub const FgDone = struct { exit_code: ?u8, killed: bool, output: []u8, dropped: bool };
+pub const FgPartial = struct { id: u64, output: []u8, dropped: bool };
+pub const FgWait = union(enum) { done: FgDone, running: FgPartial, cancelled: FgPartial };
+
+fn takeUnread(gpa: Allocator, job: *Job) error{OutOfMemory}!struct { []u8, bool } {
+ job.stream = null;
+ job.stream_ctx = null;
+ const out = try gpa.dupe(u8, job.buf.items[job.cursor..]);
+ job.cursor = job.buf.items.len;
+ const dropped = job.dropped;
+ job.dropped = false;
+ return .{ out, dropped };
+}
+
+pub fn waitForeground(gpa: Allocator, io: Io, id: u64, wait_ms: u64) !FgWait {
+ const deadline = if (wait_ms == 0) job_wait.wait_cap_ms else @min(wait_ms, job_wait.wait_cap_ms);
+ var waited: u64 = 0;
+ var still = tool_pulse.Pulse{};
+ while (true) {
+ g_jobs.mutex.lockUncancelable(io);
+ const job = g_jobs.find(id) orelse {
+ g_jobs.mutex.unlock(io);
+ return error.NoSuchJob;
+ };
+ browser_guard.touch(job, nowMs(io)); // the foreground wait is activity (#199)
+ if (job.done) {
+ const pair = takeUnread(gpa, job) catch {
+ g_jobs.mutex.unlock(io);
+ return error.OutOfMemory;
+ };
+ job_notify.dismiss(io, id);
+ const result: FgWait = .{ .done = .{ .exit_code = job.exit_code, .killed = job.killed, .output = pair[0], .dropped = pair[1] } };
+ g_jobs.mutex.unlock(io);
+ return result;
+ }
+ if (job_wait.shouldPromote(waited >= deadline, job_wait.followup_pending.load(.acquire))) {
+ const pair = takeUnread(gpa, job) catch {
+ g_jobs.mutex.unlock(io);
+ return error.OutOfMemory;
+ };
+ job.quiet = false; // completion should now notify (#620)
+ const result: FgWait = .{ .running = .{ .id = id, .output = pair[0], .dropped = pair[1] } };
+ g_jobs.mutex.unlock(io);
+ return result;
+ }
+ g_jobs.mutex.unlock(io);
+ if (Agent.esc_cancel.load(.acquire)) {
+ if (jobKill(gpa, io, id)) |out| gpa.free(out.text) else |_| {}
+ g_jobs.mutex.lockUncancelable(io);
+ if (g_jobs.find(id)) |j| {
+ const pair = takeUnread(gpa, j) catch {
+ g_jobs.mutex.unlock(io);
+ return error.OutOfMemory;
+ };
+ g_jobs.mutex.unlock(io);
+ return .{ .cancelled = .{ .id = id, .output = pair[0], .dropped = pair[1] } };
+ }
+ g_jobs.mutex.unlock(io);
+ return .{ .cancelled = .{ .id = id, .output = try gpa.dupe(u8, ""), .dropped = false } };
+ }
+ io.sleep(.fromMilliseconds(100), .awake) catch {
+ waited = deadline;
+ continue;
+ };
+ waited += 100;
+ if (still.due(waited)) {
+ var ebuf: [16]u8 = undefined;
+ tool_pulse.emitNotice(io, "· bash still running · {s}", .{tool_pulse.formatElapsed(&ebuf, waited)});
+ }
+ }
+}
diff --git a/src/job_notify.zig b/src/job_notify.zig
index 001c962a..9254c908 100644
--- a/src/job_notify.zig
+++ b/src/job_notify.zig
@@ -255,6 +255,15 @@ pub fn takeIdleWake(io: Io, buf: []u8) ?[]const u8 {
}
fn drain(io: Io, buf: []u8, idle_too: bool) ?[]const u8 {
+ var ids: [cap]u64 = undefined;
+ var n_ids: usize = 0;
+ const text = drainLocked(io, buf, idle_too, &ids, &n_ids);
+ // ADR 0218: these exits reached the model; a restart must not repeat them.
+ for (ids[0..n_ids]) |id| @import("jobs.zig").markReported(io, id);
+ return text;
+}
+
+fn drainLocked(io: Io, buf: []u8, idle_too: bool, ids: *[cap]u64, n_ids: *usize) ?[]const u8 {
mu.lockUncancelable(io);
defer mu.unlock(io);
if (count == 0) return null;
@@ -277,6 +286,8 @@ fn drain(io: Io, buf: []u8, idle_too: bool) ?[]const u8 {
const n = @min(w.len, buf.len - used);
@memcpy(buf[used .. used + n], w[0..n]);
used += n;
+ ids[n_ids.*] = ring[i].id;
+ n_ids.* += 1;
}
while (i < count) : (i += 1) { // what did not fit stays for next time
ring[keep] = ring[i];
diff --git a/src/job_recover.zig b/src/job_recover.zig
new file mode 100644
index 00000000..313ee77c
--- /dev/null
+++ b/src/job_recover.zig
@@ -0,0 +1,269 @@
+//! Resume re-attaches durable background jobs (ADR 0218). A graff that died
+//! mid-task left its finite background jobs running with their output in
+//! `.graff/job-output/<id>/` (job_capture.zig). Loading that session puts
+//! each one back in the job pool under its old id: a job still running is
+//! watched to its end, one that ended while graff was down is reported, and
+//! either way `bash_output` and `bash_kill` reach it again. Captures of other
+//! sessions are left for their own resume, and swept once stale and over.
+
+const std = @import("std");
+const builtin = @import("builtin");
+const Io = std.Io;
+const Allocator = std.mem.Allocator;
+
+const jobs = @import("jobs.zig");
+const Job = jobs.Job;
+const g_jobs = &jobs.g_jobs;
+const job_capture = @import("job_capture.zig");
+const Meta = job_capture.Meta;
+const job_registry = @import("job_registry.zig");
+const proc_identity = @import("proc_identity.zig");
+const util = @import("util.zig");
+
+/// Another session's capture is swept once this old and over.
+const stale_ms: i64 = 7 * std.time.ms_per_day;
+
+pub const Seen = struct {
+ meta: ?Meta,
+ reported: bool = false,
+ owner_alive: bool = false,
+ over: bool = false, // an exit receipt, or its leader is gone
+ age_ms: i64 = 0,
+};
+
+pub const Verdict = enum { adopt, remove, skip };
+
+/// What resume does with one capture directory.
+pub fn verdict(seen: Seen, session: []const u8) Verdict {
+ const meta = seen.meta orelse return if (seen.age_ms > stale_ms) .remove else .skip;
+ if (seen.owner_alive) return .skip; // a live graff still watches it
+ if (seen.reported) return .remove; // its exit already reached the model
+ if (std.mem.eql(u8, meta.session, session)) return .adopt;
+ return if (seen.over and seen.age_ms > stale_ms) .remove else .skip;
+}
+
+/// session.loadSession: recover from the workspace's capture root.
+pub fn onLoad(gpa: Allocator, io: Io, session: []const u8) void {
+ if (comptime !job_capture.enabled) return;
+ if (builtin.is_test or session.len == 0) return;
+ var buf: [std.fs.max_path_bytes]u8 = undefined;
+ const root = job_capture.rootPath(io, &buf) orelse return;
+ _ = recoverIn(gpa, io, root, session);
+}
+
+/// Returns how many jobs went back into the pool.
+pub fn recoverIn(gpa: Allocator, io: Io, root: []const u8, session: []const u8) usize {
+ if (comptime !job_capture.enabled) return 0;
+ var arena_state = std.heap.ArenaAllocator.init(gpa);
+ defer arena_state.deinit();
+ const arena = arena_state.allocator();
+ var names: std.ArrayList([]const u8) = .empty;
+ {
+ var dir = Io.Dir.cwd().openDir(io, root, .{ .iterate = true }) catch return 0;
+ defer dir.close(io);
+ var it = dir.iterate();
+ while (it.next(io) catch null) |entry| {
+ if (entry.kind != .directory) continue;
+ _ = std.fmt.parseInt(u64, entry.name, 10) catch continue;
+ names.append(arena, arena.dupe(u8, entry.name) catch continue) catch continue;
+ }
+ }
+ const now = util.unixMs(io);
+ var adopted: usize = 0;
+ for (names.items) |name| { // acted on after the listing: no deletes mid-iteration
+ const path = std.fmt.allocPrint(arena, "{s}/{s}", .{ root, name }) catch continue;
+ const seen = look(io, arena, path, now);
+ switch (verdict(seen, session)) {
+ .skip => {},
+ .remove => Io.Dir.cwd().deleteTree(io, path) catch {},
+ .adopt => if (adopt(gpa, io, path, seen.meta.?)) {
+ adopted += 1;
+ } else |_| {},
+ }
+ }
+ return adopted;
+}
+
+fn look(io: Io, arena: Allocator, path: []const u8, now: i64) Seen {
+ const meta = job_capture.readMeta(io, arena, path) orelse {
+ const stat = Io.Dir.cwd().statFile(io, path, .{}) catch return .{ .meta = null };
+ const mtime_ms: i64 = @intCast(@divTrunc(stat.mtime.nanoseconds, std.time.ns_per_ms));
+ return .{ .meta = null, .age_ms = now - mtime_ms };
+ };
+ const owner: proc_identity.Probe = if (meta.owner_pid > 0) proc_identity.probe(io, meta.owner_pid) else .gone;
+ const leader = proc_identity.probe(io, meta.pid);
+ return .{
+ .meta = meta,
+ .reported = job_capture.reported(io, path),
+ .owner_alive = job_registry.stateOf(meta.owner_start_id, owner) != .gone,
+ .over = job_capture.exitStatus(io, path) != null or job_registry.stateOf(meta.start_id, leader) == .gone,
+ .age_ms = now - meta.started_ms,
+ };
+}
+
+/// Back into the pool under its old id, watched by the durable pump.
+fn adopt(gpa: Allocator, io: Io, path: []const u8, meta: Meta) !void {
+ {
+ g_jobs.mutex.lockUncancelable(io);
+ defer g_jobs.mutex.unlock(io);
+ if (g_jobs.find(meta.id) != null) return error.AlreadyTracked;
+ }
+ const capture = try gpa.create(job_capture.Capture);
+ errdefer gpa.destroy(capture);
+ capture.* = .{ .dir = try gpa.dupe(u8, path), .adopted = true, .start_id = meta.start_id };
+ errdefer gpa.free(capture.dir);
+ const cmd = try gpa.dupe(u8, meta.cmd);
+ errdefer gpa.free(cmd);
+ const cwd: ?[]u8 = if (meta.cwd.len > 0) try gpa.dupe(u8, meta.cwd) else null;
+ errdefer if (cwd) |c| gpa.free(c);
+ const job = try gpa.create(Job);
+ errdefer gpa.destroy(job);
+ job.* = .{
+ .id = meta.id,
+ .cmd = cmd,
+ // Not graff's child: nothing may wait on or signal it through `child`.
+ .child = .{ .id = null, .thread_handle = {}, .stdin = null, .stdout = null, .stderr = null, .request_resource_usage_statistics = false },
+ .cwd = cwd,
+ .started_ms = meta.started_ms,
+ .last_active_ms = jobs.nowMs(io),
+ .group_pid = meta.pid,
+ .capture = capture,
+ };
+ // This graff watches it now: another resume of the session leaves it be.
+ // Both records land before the pump, which forgets the registry one.
+ var owned = meta;
+ owned.owner_pid = proc_identity.selfPid();
+ owned.owner_start_id = proc_identity.selfStartId(io);
+ job_capture.writeMeta(io, capture, owned);
+ if (meta.pid > 0) job_registry.write(io, job_registry.home, @import("jobs_admin.zig").recordOf(io, job));
+ errdefer if (meta.pid > 0) job_registry.forget(io, job_registry.home, meta.pid);
+ {
+ g_jobs.mutex.lockUncancelable(io);
+ defer g_jobs.mutex.unlock(io);
+ try g_jobs.list.append(gpa, job);
+ }
+ job.future = io.concurrent(@import("job_durable.zig").pump, .{ job, gpa, io }) catch |e| {
+ g_jobs.mutex.lockUncancelable(io);
+ defer g_jobs.mutex.unlock(io);
+ for (g_jobs.list.items, 0..) |j, i| {
+ if (j == job) {
+ _ = g_jobs.list.swapRemove(i);
+ break;
+ }
+ }
+ return e;
+ };
+}
+
+test "resume adopts its own session's jobs and leaves live owners and other sessions alone" {
+ const own: Meta = .{ .id = 1, .session = "mine", .cmd = "make" };
+ const other: Meta = .{ .id = 2, .session = "theirs", .cmd = "make" };
+ try std.testing.expectEqual(Verdict.adopt, verdict(.{ .meta = own }, "mine"));
+ try std.testing.expectEqual(Verdict.skip, verdict(.{ .meta = own, .owner_alive = true }, "mine"));
+ try std.testing.expectEqual(Verdict.remove, verdict(.{ .meta = own, .reported = true }, "mine"));
+ try std.testing.expectEqual(Verdict.skip, verdict(.{ .meta = own, .reported = true, .owner_alive = true }, "mine"));
+ try std.testing.expectEqual(Verdict.skip, verdict(.{ .meta = other, .over = true }, "mine"));
+ try std.testing.expectEqual(Verdict.skip, verdict(.{ .meta = other, .age_ms = stale_ms + 1 }, "mine"));
+ try std.testing.expectEqual(Verdict.remove, verdict(.{ .meta = other, .over = true, .age_ms = stale_ms + 1 }, "mine"));
+ try std.testing.expectEqual(Verdict.skip, verdict(.{ .meta = null }, "mine"));
+ try std.testing.expectEqual(Verdict.remove, verdict(.{ .meta = null, .age_ms = stale_ms + 1 }, "mine"));
+}
+
+/// A capture as a graff that then died would leave it: the job spawned with
+/// its receipt, its record naming an owner that no longer exists.
+fn orphan(io: Io, root: []const u8, id: u64, cmd: []const u8, session: []const u8) !std.process.Child {
+ const gpa = std.testing.allocator;
+ var setup = try job_capture.create(io, gpa, root, id, cmd);
+ const child = std.process.spawn(io, .{ .argv = &setup.argv, .stdin = .ignore, .stdout = .{ .file = setup.out }, .stderr = .{ .file = setup.err }, .pgid = 0 }) catch |e| {
+ setup.abandon(gpa, io);
+ return e;
+ };
+ const pid = child.id.?;
+ const capture = setup.capture;
+ setup.spawned(gpa, io);
+ job_capture.writeMeta(io, capture, .{
+ .id = id,
+ .session = session,
+ .cmd = cmd,
+ .pid = pid,
+ .start_id = switch (proc_identity.probe(io, pid)) {
+ .id => |v| v,
+ else => 0,
+ },
+ .owner_pid = 0,
+ .started_ms = util.unixMs(io),
+ });
+ job_capture.release(gpa, io, capture, false);
+ return child;
+}
+
+test "a job that outlived graff is re-attached on resume and reports its real exit once" {
+ if (!job_capture.enabled) return error.SkipZigTest;
+ const gpa = std.testing.allocator;
+ const io = std.testing.io;
+ const job_notify = @import("job_notify.zig");
+ jobs.g_jobs = .{};
+ job_notify.resetForTest(io);
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var rbuf: [std.fs.max_path_bytes]u8 = undefined;
+ const root = rbuf[0..try tmp.dir.realPath(io, &rbuf)];
+
+ // One finished while graff was down, one is still running at resume, and
+ // one belongs to another session.
+ var done = try orphan(io, root, 7001, "printf early; exit 3", "s-resume");
+ _ = try done.wait(io);
+ var late = try orphan(io, root, 7002, "sleep 0.3; printf late", "s-resume");
+ defer if (late.wait(io)) |_| {} else |_| {};
+ var other = try orphan(io, root, 7003, "exit 0", "s-other");
+ _ = try other.wait(io);
+
+ try std.testing.expectEqual(@as(usize, 2), recoverIn(gpa, io, root, "s-resume"));
+ defer jobs.jobsReap(gpa, io);
+ const early = try jobs.jobOutput(gpa, io, 7001, 5000);
+ defer gpa.free(early.text);
+ try std.testing.expect(std.mem.indexOf(u8, early.text, "exited with code 3") != null);
+ try std.testing.expect(std.mem.indexOf(u8, early.text, "early") != null);
+ const running = try jobs.jobOutput(gpa, io, 7002, 5000);
+ defer gpa.free(running.text);
+ try std.testing.expect(std.mem.indexOf(u8, running.text, "exited with code 0") != null);
+ try std.testing.expect(std.mem.indexOf(u8, running.text, "late") != null);
+ // Both exits reached the model; the other session's capture is untouched.
+ var pbuf: [std.fs.max_path_bytes]u8 = undefined;
+ try std.testing.expect(job_capture.reported(io, try std.fmt.bufPrint(&pbuf, "{s}/7001", .{root})));
+ try std.testing.expect(job_capture.reported(io, try std.fmt.bufPrint(&pbuf, "{s}/7002", .{root})));
+ try std.testing.expect(!job_capture.reported(io, try std.fmt.bufPrint(&pbuf, "{s}/7003", .{root})));
+ // A second resume while this process owns them adopts nothing.
+ try std.testing.expectEqual(@as(usize, 0), recoverIn(gpa, io, root, "s-resume"));
+}
+
+test "a resumed job's completion notice marks it reported" {
+ if (!job_capture.enabled) return error.SkipZigTest;
+ const gpa = std.testing.allocator;
+ const io = std.testing.io;
+ const job_notify = @import("job_notify.zig");
+ jobs.g_jobs = .{};
+ job_notify.resetForTest(io);
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var rbuf: [std.fs.max_path_bytes]u8 = undefined;
+ const root = rbuf[0..try tmp.dir.realPath(io, &rbuf)];
+ var done = try orphan(io, root, 7101, "printf gone; exit 5", "s-notice");
+ _ = try done.wait(io);
+
+ try std.testing.expectEqual(@as(usize, 1), recoverIn(gpa, io, root, "s-notice"));
+ defer jobs.jobsReap(gpa, io);
+ var buf: [4096]u8 = undefined;
+ var text: []const u8 = "";
+ for (0..200) |_| {
+ text = job_notify.takeWake(io, &buf) orelse {
+ try io.sleep(.fromMilliseconds(10), .awake);
+ continue;
+ };
+ break;
+ }
+ try std.testing.expect(std.mem.indexOf(u8, text, "[job 7101 exited 5: printf gone; exit 5]") != null);
+ try std.testing.expect(std.mem.indexOf(u8, text, "gone") != null);
+ var pbuf: [std.fs.max_path_bytes]u8 = undefined;
+ try std.testing.expect(job_capture.reported(io, try std.fmt.bufPrint(&pbuf, "{s}/7101", .{root})));
+}
diff --git a/src/jobs.zig b/src/jobs.zig
index dcd93d4a..7915b135 100644
--- a/src/jobs.zig
+++ b/src/jobs.zig
@@ -55,7 +55,7 @@ pub const agentWorktreeFinish = agent_worktree.agentWorktreeFinish;
pub const KeepReason = agent_worktree.KeepReason;
pub const keepReasonText = agent_worktree.keepReasonText;
-const Job = struct { // session-global; pump drains pipes; survives Esc
+pub const Job = struct { // session-global; pump drains pipes or capture files; survives Esc
id: u64,
cmd: []u8,
child: std.process.Child,
@@ -83,6 +83,7 @@ const Job = struct { // session-global; pump drains pipes; survives Esc
future: Io.Future(void) = undefined,
stream: ?process_runner.StreamFn = null,
stream_ctx: ?*anyopaque = null,
+ capture: ?*job_capture.Capture = null, // ADR 0218: output in files, not pipes
};
pub const job_unread_cap = 256 * 1024;
@@ -91,6 +92,7 @@ const job_notify = @import("job_notify.zig");
const browser_guard = @import("job_browser_guard.zig");
const job_idle = @import("job_idle.zig"); // #199
const job_registry = @import("job_registry.zig"); // #199
+const job_capture = @import("job_capture.zig"); // ADR 0218
const proc_identity = @import("proc_identity.zig");
const util = @import("util.zig");
const tool_pulse = @import("tool_pulse.zig"); // silence heartbeat during waitForeground
@@ -104,7 +106,7 @@ const Jobs = struct {
list: std.ArrayList(*Job) = .empty,
/// Caller holds the mutex.
- fn find(self: *Jobs, id: u64) ?*Job {
+ pub fn find(self: *Jobs, id: u64) ?*Job {
for (self.list.items) |j| if (j.id == id) return j;
return null;
}
@@ -121,27 +123,54 @@ pub fn markPersistent(io: Io, id: u64) void {
/// Deterministic test pause after done becomes observable, before UI publish.
pub var completion_test_hook: ?*const fn (Io, u64) void = null;
+/// New output into the job buffer (both pumps). Caller holds the mutex.
+pub fn appendOutput(job: *Job, gpa: Allocator, which: u8, bytes: []const u8, now_ms: i64) void {
+ if (bytes.len == 0) return;
+ if (!job.persistent) browser_guard.touch(job, now_ms);
+ if (job.stream) |emit| emit(job.stream_ctx, which, bytes);
+ job.buf.appendSlice(gpa, bytes) catch {};
+}
+
+/// Drop the oldest unread output past the cap. Caller holds the mutex.
+pub fn capUnread(job: *Job, gpa: Allocator) void {
+ if (job.buf.items.len - job.cursor <= job_unread_cap) return;
+ const drop = job.buf.items.len - job.cursor - job_unread_cap;
+ job.buf.replaceRange(gpa, job.cursor, drop, &.{}) catch return;
+ job.dropped = true;
+}
+
/// Drain MultiReader bytes into the job buffer; drop oldest unread past the cap.
fn jobDrain(job: *Job, gpa: Allocator, readers: []const *Io.Reader, now_ms: i64) void {
for (readers, 0..) |r, i| {
const b = r.buffered();
- if (b.len == 0) continue;
- if (!job.persistent) browser_guard.touch(job, now_ms);
- if (job.stream) |emit| emit(job.stream_ctx, @intCast(i), b);
- job.buf.appendSlice(gpa, b) catch {};
+ appendOutput(job, gpa, @intCast(i), b, now_ms);
r.toss(b.len);
}
- if (job.buf.items.len - job.cursor > job_unread_cap) {
- const drop = job.buf.items.len - job.cursor - job_unread_cap;
- job.buf.replaceRange(gpa, job.cursor, drop, &.{}) catch return;
- job.dropped = true;
+ capUnread(job, gpa);
+}
+
+/// #199: silence is the idle clock — no bytes, no read, no pin. Caller holds
+/// the mutex. Returns the idle time when the warning is due; print it after
+/// unlocking.
+pub fn idleTick(job: *Job, gpa: Allocator, io: Io, pid: i32, now: i64) ?u64 {
+ if (job.kill_requested or job.detach or job.exit_cleanup) return null;
+ const idle_ms: u64 = @intCast(@max(now - job.last_active_ms, 0));
+ switch (job_idle.verdict(idle_ms, job.idle_warned, job.pinned)) {
+ .none => {},
+ .warn => {
+ job.idle_warned = true;
+ return idle_ms;
+ },
+ .stop => browser_guard.checkIdle(&g_jobs, job, gpa, io, pid, now),
}
+ return null;
}
/// Pump task (one per job, on its own unit of concurrency): same MultiReader
/// loop as runCapped, but appending into the job buffer under the jobs mutex
/// and ignoring Esc — background jobs outlive turn cancellation by design.
fn jobPump(job: *Job, gpa: Allocator, io: Io) void {
+ if (comptime job_capture.enabled) if (job.capture != null) return @import("job_durable.zig").pump(job, gpa, io);
var mrb: Io.File.MultiReader.Buffer(2) = undefined;
var mr: Io.File.MultiReader = undefined;
mr.init(gpa, io, mrb.toStreams(), &.{ job.child.stdout.?, job.child.stderr.? });
@@ -160,23 +189,11 @@ fn jobPump(job: *Job, gpa: Allocator, io: Io) void {
const now = nowMs(io);
g_jobs.mutex.lockUncancelable(io);
jobDrain(job, gpa, &readers, now);
- // #199: silence is the idle clock — no bytes, no read, no pin.
- const idle_ms: u64 = @intCast(@max(now - job.last_active_ms, 0));
- var warn = false;
- if (!job.kill_requested and !job.detach and !job.exit_cleanup) {
- switch (job_idle.verdict(idle_ms, job.idle_warned, job.pinned)) {
- .none => {},
- .warn => {
- job.idle_warned = true;
- warn = true;
- },
- .stop => browser_guard.checkIdle(&g_jobs, job, gpa, io, pid, now),
- }
- }
+ const warn = idleTick(job, gpa, io, pid, now);
killed = job.kill_requested;
detached = job.detach;
g_jobs.mutex.unlock(io);
- if (warn) job_idle.warn(io, job.id, idle_ms, job.cmd);
+ if (warn) |idle_ms| job_idle.warn(io, job.id, idle_ms, job.cmd);
if (killed or detached) break :loop;
}
g_jobs.mutex.lockUncancelable(io);
@@ -200,6 +217,13 @@ fn jobPump(job: *Job, gpa: Allocator, io: Io) void {
else => null,
};
} else |_| {}
+ finish(job, io, code, killed, detached, pid);
+}
+
+/// A job ended (both pumps). Done + queue publish atomically against
+/// jobOutput/jobKill consuming it: a bounded dismissed-id cache cannot close
+/// a queue-after-unlock race. A detached job publishes nothing.
+pub fn finish(job: *Job, io: Io, code: ?u8, killed: bool, detached: bool, pid: i32) void {
g_jobs.mutex.lockUncancelable(io);
job.exit_code = code;
job.killed = killed and !detached;
@@ -207,8 +231,6 @@ fn jobPump(job: *Job, gpa: Allocator, io: Io) void {
const id = job.id;
const quiet = job.quiet;
const idle = job.stopped_idle;
- // Publish done + queue atomically against jobOutput/jobKill consuming it.
- // A bounded dismissed-id cache cannot close a queue-after-unlock race.
if (!quiet and !detached) job_notify.queue(io, id, code, killed, job.cmd, idle, job.buf.items[job.cursor..]);
g_jobs.mutex.unlock(io);
if (detached) return;
@@ -217,71 +239,28 @@ fn jobPump(job: *Job, gpa: Allocator, io: Io) void {
if (!quiet) job_notify.publish(io, id, code, killed);
}
-fn nowMs(io: Io) i64 {
- return @intCast(@divTrunc(Io.Timestamp.now(io, .awake).nanoseconds, std.time.ns_per_ms));
+/// The model just learned how the job ended: no wake for it (ADR 0061), and
+/// no second report after a restart (ADR 0218). Caller holds the mutex.
+fn consumed(io: Io, job: *Job) void {
+ job_notify.dismiss(io, job.id);
+ if (job.capture) |c| job_capture.markReported(io, c);
}
-/// The leader's start identity (#413), so a recycled pid is never mistaken
-/// for the job. 0 where the platform has no source.
-fn startIdOf(io: Io, pid: i32) u64 {
- return switch (proc_identity.probe(io, pid)) {
- .id => |v| v,
- else => 0,
- };
-}
-
-/// Caller holds the mutex (or owns the job outright).
-fn recordOf(io: Io, job: *Job) job_registry.Record {
- const pid: i32 = if (comptime posix_groups) (job.child.id orelse 0) else 0;
- return .{
- .pid = pid,
- .start_id = startIdOf(io, pid),
- .owner_pid = proc_identity.selfPid(),
- .owner_start_id = proc_identity.selfStartId(io),
- .cmd = job.cmd,
- .cwd = job.cwd orelse "",
- .started_ms = job.started_ms,
- .pinned = job.pinned,
- };
-}
-
-/// /jobs keep|unkeep (#199): exempt from the idle stop, retained at session
-/// end. Null for an unknown id, false for one that already finished.
-pub fn setPinned(io: Io, id: u64, pinned: bool) ?bool {
+/// A completion notice reached the model (job_notify drain, ADR 0218).
+pub fn markReported(io: Io, id: u64) void {
g_jobs.mutex.lockUncancelable(io);
defer g_jobs.mutex.unlock(io);
- const job = g_jobs.find(id) orelse return null;
- if (job.done) return false;
- job.pinned = pinned;
- browser_guard.touch(job, nowMs(io));
- if (comptime posix_groups) job_registry.write(io, job_registry.home, recordOf(io, job));
- return true;
+ const job = g_jobs.find(id) orelse return;
+ if (job.capture) |c| job_capture.markReported(io, c);
}
-/// /jobs restart (#199): rerun a finished job's command in its cwd, as a new
-/// job. The finished record stays listed until reaped.
-pub fn restartJob(gpa: Allocator, io: Io, id: u64) !*Job {
- var cmd: []const u8 = "";
- var cwd: ?[]const u8 = null;
- {
- g_jobs.mutex.lockUncancelable(io);
- defer g_jobs.mutex.unlock(io);
- const job = g_jobs.find(id) orelse return error.NoSuchJob;
- if (!job.done) return error.StillRunning;
- cmd = job.cmd;
- cwd = job.cwd;
- }
- return spawnJobOpts(gpa, io, cmd, .{ .cwd = cwd });
+pub fn nowMs(io: Io) i64 {
+ return @intCast(@divTrunc(Io.Timestamp.now(io, .awake).nanoseconds, std.time.ns_per_ms));
}
-/// Hand retained pipes to a detached drainer; failure never authorizes a kill.
-fn retainAtExit(io: Io, job: *Job) bool {
- if (comptime !posix_groups) return false;
- const rec = recordOf(io, job);
- if (!job_registry.retain(io, job_registry.home, rec, job.child.stdout, job.child.stderr)) return false;
- std.debug.print("kept alive: job {d} (pid {d}) {s} — `graff servers` lists it; `graff servers stop {d}` ends it\n", .{ job.id, rec.pid, job.cmd, rec.pid });
- return true;
-}
+const admin = @import("jobs_admin.zig");
+pub const setPinned = admin.setPinned;
+pub const restartJob = admin.restartJob;
pub fn shellArgv(cmd: []const u8) [3][]const u8 { // /bin/sh -c, or cmd.exe /c on Windows
return if (builtin.os.tag == .windows)
@@ -296,6 +275,9 @@ pub const SpawnOpts = struct {
stream_ctx: ?*anyopaque = null,
quiet: bool = false,
persistent: bool = false,
+ /// ADR 0218: the session a finite job reports back to after a restart.
+ /// Non-empty puts its output in files that outlive graff.
+ session: []const u8 = "",
};
pub fn spawnJob(gpa: Allocator, io: Io, cmd: []const u8) !*Job {
@@ -304,23 +286,35 @@ pub fn spawnJob(gpa: Allocator, io: Io, cmd: []const u8) !*Job {
pub fn spawnJobOpts(gpa: Allocator, io: Io, cmd: []const u8, opts: SpawnOpts) !*Job {
const id = try @import("shell_identity.zig").forJob(io);
- const argv = shellArgv(cmd);
- var child = try std.process.spawn(io, .{
- .argv = &argv,
+ var setup: ?job_capture.Setup = null; // best-effort: pipes without it
+ if (comptime job_capture.enabled) {
+ if (opts.session.len > 0 and !opts.persistent) setup = job_capture.prepare(io, gpa, id, cmd);
+ }
+ const sh = shellArgv(cmd);
+ const argv: []const []const u8 = if (setup) |*s| &s.argv else &sh;
+ var child = std.process.spawn(io, .{
+ .argv = argv,
.cwd = if (opts.cwd) |path| .{ .path = path } else .inherit,
.environ_map = @import("tool_env.zig").get(), // #1267: no provider keys
.stdin = .ignore,
- .stdout = .pipe,
- .stderr = .pipe,
+ .stdout = if (setup) |s| .{ .file = s.out } else .pipe,
+ .stderr = if (setup) |s| .{ .file = s.err } else .pipe,
.pgid = if (posix_groups) 0 else null, // #198: bash_kill reaches grandchildren
- });
+ }) catch |e| {
+ if (setup) |*s| s.abandon(gpa, io);
+ return e;
+ };
+ const capture: ?*job_capture.Capture = if (setup) |*s| blk: {
+ s.spawned(gpa, io);
+ break :blk s.capture;
+ } else null;
const cmd_copy = gpa.dupe(u8, cmd) catch |e| {
- child.kill(io);
+ abortSpawn(gpa, io, &child, capture);
return e;
};
const job = gpa.create(Job) catch |e| {
gpa.free(cmd_copy);
- child.kill(io);
+ abortSpawn(gpa, io, &child, capture);
return e;
};
const cwd_copy: ?[]u8 = if (opts.cwd) |c| (gpa.dupe(u8, c) catch null) else null;
@@ -336,6 +330,7 @@ pub fn spawnJobOpts(gpa: Allocator, io: Io, cmd: []const u8, opts: SpawnOpts) !*
.cwd = cwd_copy,
.started_ms = util.unixMs(io),
.last_active_ms = nowMs(io),
+ .capture = capture,
};
g_jobs.mutex.lockUncancelable(io);
job.id = id;
@@ -345,7 +340,7 @@ pub fn spawnJobOpts(gpa: Allocator, io: Io, cmd: []const u8, opts: SpawnOpts) !*
};
g_jobs.mutex.unlock(io);
if (!appended) {
- job.child.kill(io);
+ abortSpawn(gpa, io, &job.child, capture);
if (job.cwd) |c| gpa.free(c);
gpa.free(job.cmd);
gpa.destroy(job);
@@ -360,17 +355,26 @@ pub fn spawnJobOpts(gpa: Allocator, io: Io, cmd: []const u8, opts: SpawnOpts) !*
}
}
g_jobs.mutex.unlock(io);
- job.child.kill(io);
+ abortSpawn(gpa, io, &job.child, capture);
if (job.cwd) |c| gpa.free(c);
gpa.free(job.cmd);
gpa.destroy(job);
return e;
};
// #199: the ownership record outlives a graff that dies without its defers.
- if (comptime posix_groups) job_registry.write(io, job_registry.home, recordOf(io, job));
+ if (comptime posix_groups) job_registry.write(io, job_registry.home, admin.recordOf(io, job));
+ if (capture) |c| @import("job_durable.zig").record(io, job, c, opts.session);
return job;
}
+/// A spawned job that never joined the pool: stop its group (#198), drop
+/// its capture.
+fn abortSpawn(gpa: Allocator, io: Io, child: *std.process.Child, capture: ?*job_capture.Capture) void {
+ if (comptime posix_groups) if (child.id) |pid| std.posix.kill(-pid, .KILL) catch {};
+ child.kill(io); // reaps the leader (a zombie by now accepts the TERM)
+ if (capture) |c| job_capture.release(gpa, io, c, true);
+}
+
/// bash_output: unread output + status. wait_ms=0 is a snapshot. wait_ms>0
/// blocks until the job exits (or Esc) for finite jobs (ADR 0010). Persistent
/// servers snapshot immediately; wait_ms is ignored (ADR 0152).
@@ -417,7 +421,7 @@ pub fn jobOutput(gpa: Allocator, io: Io, id: u64, wait_ms: u64) !ToolOutput {
}
const text = try aw.toOwnedSlice();
job.cursor = job.buf.items.len;
- if (job.done) job_notify.dismiss(io, id); // the exit was just read here; no wake for it (ADR 0061)
+ if (job.done) consumed(io, job); // the exit was just read here
const result: ToolOutput = .{ .text = text, .pending = !job.done, .is_error = job.done and (job.stopped_idle or job.killed or (job.exit_code orelse 1) != 0), .cancelled = job.killed };
g_jobs.mutex.unlock(io);
return result;
@@ -445,7 +449,7 @@ pub fn jobKill(gpa: Allocator, io: Io, id: u64) !ToolOutput {
defer g_jobs.mutex.unlock(io);
const job = g_jobs.find(id) orelse return .{ .text = try std.fmt.allocPrint(gpa, "background job {d} has no live owner in this process — interrupted or unknown outcome; no process was stopped; /jobs lists current jobs", .{id}), .is_error = true };
if (job.done) {
- job_notify.dismiss(io, id); // already-finished also reports the exit
+ consumed(io, job); // already-finished also reports the exit
const unread = job.buf.items.len - job.cursor;
return .{ .text = try std.fmt.allocPrint(gpa, "job {d} already finished ({d} unread byte(s) — bash_output reads them)", .{ id, unread }) };
}
@@ -460,8 +464,9 @@ pub fn jobKill(gpa: Allocator, io: Io, id: u64) !ToolOutput {
if (done) {
g_jobs.mutex.lockUncancelable(io);
defer g_jobs.mutex.unlock(io);
- const unread = if (g_jobs.find(id)) |job| job.buf.items.len - job.cursor else 0;
- job_notify.dismiss(io, id); // the model just learned the job is dead (ADR 0061)
+ const found = g_jobs.find(id);
+ const unread = if (found) |job| job.buf.items.len - job.cursor else 0;
+ if (found) |job| consumed(io, job) else job_notify.dismiss(io, id); // the model just learned the job is dead
return .{ .text = try std.fmt.allocPrint(gpa, "job {d} killed ({d} unread byte(s) — bash_output reads them)", .{ id, unread }) };
}
io.sleep(.fromMilliseconds(100), .awake) catch break;
@@ -470,23 +475,16 @@ pub fn jobKill(gpa: Allocator, io: Io, id: u64) !ToolOutput {
return .{ .text = try std.fmt.allocPrint(gpa, "job {d}: kill requested (still shutting down — check bash_output)", .{id}) };
}
-/// Outcome of a root foreground wait (#620 / grok-build auto-background).
-pub const FgDone = struct { exit_code: ?u8, killed: bool, output: []u8, dropped: bool };
-pub const FgPartial = struct { id: u64, output: []u8, dropped: bool };
-pub const FgWait = union(enum) { done: FgDone, running: FgPartial, cancelled: FgPartial };
-
-fn takeUnread(gpa: Allocator, job: *Job) error{OutOfMemory}!struct { []u8, bool } {
- job.stream = null;
- job.stream_ctx = null;
- const out = try gpa.dupe(u8, job.buf.items[job.cursor..]);
- job.cursor = job.buf.items.len;
- const dropped = job.dropped;
- job.dropped = false;
- return .{ out, dropped };
-}
+const job_foreground = @import("job_foreground.zig");
+pub const FgDone = job_foreground.FgDone;
+pub const FgPartial = job_foreground.FgPartial;
+pub const FgWait = job_foreground.FgWait;
+pub const waitForeground = job_foreground.waitForeground;
fn freeJob(gpa: Allocator, io: Io, job: *Job) void {
job.future.await(io);
+ // A kept job still writes to its capture: a later resume re-attaches it.
+ if (job.capture) |c| job_capture.release(gpa, io, c, !job.detach);
job.buf.deinit(gpa);
if (job.cwd) |c| gpa.free(c);
gpa.free(job.cmd);
@@ -510,64 +508,6 @@ pub fn reapFinished(gpa: Allocator, io: Io, id: u64) void {
if (found) |job| freeJob(gpa, io, job);
}
-pub fn waitForeground(gpa: Allocator, io: Io, id: u64, wait_ms: u64) !FgWait {
- const deadline = if (wait_ms == 0) job_wait.wait_cap_ms else @min(wait_ms, job_wait.wait_cap_ms);
- var waited: u64 = 0;
- var still = tool_pulse.Pulse{};
- while (true) {
- g_jobs.mutex.lockUncancelable(io);
- const job = g_jobs.find(id) orelse {
- g_jobs.mutex.unlock(io);
- return error.NoSuchJob;
- };
- browser_guard.touch(job, nowMs(io)); // the foreground wait is activity (#199)
- if (job.done) {
- const pair = takeUnread(gpa, job) catch {
- g_jobs.mutex.unlock(io);
- return error.OutOfMemory;
- };
- job_notify.dismiss(io, id);
- const result: FgWait = .{ .done = .{ .exit_code = job.exit_code, .killed = job.killed, .output = pair[0], .dropped = pair[1] } };
- g_jobs.mutex.unlock(io);
- return result;
- }
- if (job_wait.shouldPromote(waited >= deadline, job_wait.followup_pending.load(.acquire))) {
- const pair = takeUnread(gpa, job) catch {
- g_jobs.mutex.unlock(io);
- return error.OutOfMemory;
- };
- job.quiet = false; // completion should now notify (#620)
- const result: FgWait = .{ .running = .{ .id = id, .output = pair[0], .dropped = pair[1] } };
- g_jobs.mutex.unlock(io);
- return result;
- }
- g_jobs.mutex.unlock(io);
- if (Agent.esc_cancel.load(.acquire)) {
- if (jobKill(gpa, io, id)) |out| gpa.free(out.text) else |_| {}
- g_jobs.mutex.lockUncancelable(io);
- if (g_jobs.find(id)) |j| {
- const pair = takeUnread(gpa, j) catch {
- g_jobs.mutex.unlock(io);
- return error.OutOfMemory;
- };
- g_jobs.mutex.unlock(io);
- return .{ .cancelled = .{ .id = id, .output = pair[0], .dropped = pair[1] } };
- }
- g_jobs.mutex.unlock(io);
- return .{ .cancelled = .{ .id = id, .output = try gpa.dupe(u8, ""), .dropped = false } };
- }
- io.sleep(.fromMilliseconds(100), .awake) catch {
- waited = deadline;
- continue;
- };
- waited += 100;
- if (still.due(waited)) {
- var ebuf: [16]u8 = undefined;
- tool_pulse.emitNotice(io, "· bash still running · {s}", .{tool_pulse.formatElapsed(&ebuf, waited)});
- }
- }
-}
-
pub fn jobsReap(gpa: Allocator, io: Io) void {
g_jobs.mutex.lockUncancelable(io);
const jobs = g_jobs.list.toOwnedSlice(gpa) catch {
@@ -581,7 +521,7 @@ pub fn jobsReap(gpa: Allocator, io: Io) void {
g_jobs.mutex.lockUncancelable(io);
if (!job.done and !job.kill_requested) {
if (browser_guard.exitAction(result, job.pinned, false) != .stop) {
- job.detach = browser_guard.exitAction(result, job.pinned, retainAtExit(io, job)) == .detach;
+ job.detach = browser_guard.exitAction(result, job.pinned, admin.retainAtExit(io, job)) == .detach;
if (!job.detach) std.debug.print("retained job pipe handoff failed; cleanup waits while the pump keeps draining (no kill)\n", .{});
} else job.kill_requested = true;
}
@@ -595,6 +535,8 @@ pub fn jobsReap(gpa: Allocator, io: Io) void {
test { // split-out modules: unreferenced, their tests silently never run
_ = worktree_cmd;
_ = @import("worktree_lease.zig");
- _ = .{ job_wait, job_notify, job_idle, job_registry, browser_guard };
+ _ = .{ job_wait, job_notify, job_idle, job_registry, browser_guard, job_capture };
+ _ = @import("job_durable.zig");
+ _ = @import("job_recover.zig");
_ = @import("jobs_tests.zig");
}
diff --git a/src/jobs_admin.zig b/src/jobs_admin.zig
new file mode 100644
index 00000000..ae286da0
--- /dev/null
+++ b/src/jobs_admin.zig
@@ -0,0 +1,80 @@
+//! Job-pool administration split out of jobs.zig (600-line cap): the
+//! ownership record a job leaves in ~/.codegraff/jobs (#199), /jobs keep and
+//! restart, and the pinned-job handoff at session end.
+
+const std = @import("std");
+const builtin = @import("builtin");
+const Io = std.Io;
+const Allocator = std.mem.Allocator;
+
+const jobs = @import("jobs.zig");
+const Job = jobs.Job;
+const g_jobs = &jobs.g_jobs;
+const nowMs = jobs.nowMs;
+const job_registry = @import("job_registry.zig");
+const browser_guard = @import("job_browser_guard.zig");
+const proc_identity = @import("proc_identity.zig");
+
+const posix_groups = builtin.os.tag != .windows and builtin.os.tag != .wasi;
+
+/// The leader's start identity (#413), so a recycled pid is never mistaken
+/// for the job. 0 where the platform has no source.
+fn startIdOf(io: Io, pid: i32) u64 {
+ return switch (proc_identity.probe(io, pid)) {
+ .id => |v| v,
+ else => 0,
+ };
+}
+
+/// Caller holds the mutex (or owns the job outright).
+pub fn recordOf(io: Io, job: *Job) job_registry.Record {
+ const pid: i32 = job.group_pid; // an adopted job (ADR 0218) has no child id
+ return .{
+ .pid = pid,
+ .start_id = startIdOf(io, pid),
+ .owner_pid = proc_identity.selfPid(),
+ .owner_start_id = proc_identity.selfStartId(io),
+ .cmd = job.cmd,
+ .cwd = job.cwd orelse "",
+ .started_ms = job.started_ms,
+ .pinned = job.pinned,
+ };
+}
+
+/// /jobs keep|unkeep (#199): exempt from the idle stop, retained at session
+/// end. Null for an unknown id, false for one that already finished.
+pub fn setPinned(io: Io, id: u64, pinned: bool) ?bool {
+ g_jobs.mutex.lockUncancelable(io);
+ defer g_jobs.mutex.unlock(io);
+ const job = g_jobs.find(id) orelse return null;
+ if (job.done) return false;
+ job.pinned = pinned;
+ browser_guard.touch(job, nowMs(io));
+ if (comptime posix_groups) job_registry.write(io, job_registry.home, recordOf(io, job));
+ return true;
+}
+
+/// /jobs restart (#199): rerun a finished job's command in its cwd, as a new
+/// job. The finished record stays listed until reaped.
+pub fn restartJob(gpa: Allocator, io: Io, id: u64) !*Job {
+ var cmd: []const u8 = "";
+ var cwd: ?[]const u8 = null;
+ {
+ g_jobs.mutex.lockUncancelable(io);
+ defer g_jobs.mutex.unlock(io);
+ const job = g_jobs.find(id) orelse return error.NoSuchJob;
+ if (!job.done) return error.StillRunning;
+ cmd = job.cmd;
+ cwd = job.cwd;
+ }
+ return jobs.spawnJobOpts(gpa, io, cmd, .{ .cwd = cwd });
+}
+
+/// Hand retained pipes to a detached drainer; failure never authorizes a kill.
+pub fn retainAtExit(io: Io, job: *Job) bool {
+ if (comptime !posix_groups) return false;
+ const rec = recordOf(io, job);
+ if (!job_registry.retain(io, job_registry.home, rec, job.child.stdout, job.child.stderr)) return false;
+ std.debug.print("kept alive: job {d} (pid {d}) {s} — `graff servers` lists it; `graff servers stop {d}` ends it\n", .{ job.id, rec.pid, job.cmd, rec.pid });
+ return true;
+}
diff --git a/src/session.zig b/src/session.zig
index 8dfd04ef..03b52793 100644
--- a/src/session.zig
+++ b/src/session.zig
@@ -543,4 +543,6 @@ pub fn loadSession(root: *Agent, keys: *Keys, arena: Allocator, name: []const u8
const subagent = @import("subagent.zig");
if (subagent_ledger.nextId() > subagent.g_agent_jobs.next_id)
subagent.g_agent_jobs.next_id = subagent_ledger.nextId();
+ // ADR 0218: background jobs that outlived the last graff come back.
+ @import("job_recover.zig").onLoad(root.gpa, root.io, name);
}