Skip to content

The open-source app everyone uses to manage agents at work

fix(recovery): surface exhausted review-path repair (#15764)

Devin Foley3389bea7086e
diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md
index 15ef72d769..0be0461e71 100644
--- a/doc/execution-semantics.md
+++ b/doc/execution-semantics.md
@@ -682,6 +682,8 @@ Agent-assigned `in_review` with no typed participant is only healthy when one of
An `in_review` issue is stalled when it has no typed participant, no pending interaction or approval, no user owner, no active monitor, no active run, no queued wake, and no explicit recovery action. Paperclip should surface that state as recovery work rather than silently completing the issue or leaving blocker chains parked indefinitely.
+When a maintained review path is consumed, Paperclip queues one normal-model repair wake per consumed-path fingerprint. If an external check is still pending, that repair must persist a new one-shot issue monitor and report its scheduled check time; a promise to check later does not schedule a wake. If the repair also finishes without a maintained path, Paperclip atomically rechecks the issue's current owner and review paths, then blocks it with a board-owned recovery action and an inline notice linking the repair run. The original assignee is preserved. A new maintained path or a newer run prevents stale finalization from escalating the issue. The stranded-issue sweep retries this same disposition from the saved terminal repair run if a restart or transaction failure prevented finalization. A due or claimed monitor whose bounds remain valid is preserved for monitor dispatch while its owner permits on-demand wakes; passing its due time does not consume it. Disabling those wakes cannot let an overdue monitor hide an exhausted repair indefinitely. Exhaustion never queues another automatic repair wake.
+
When an execution-policy review stage has a pending agent participant, the participant's run is part of the review path only while it is live or queued. If that participant run reaches a terminal state while `executionState.status` remains `pending`, no decision has been recorded. After a successful run with no review decision, Paperclip should queue one bounded normal-model recovery wake for the same participant when the agent is invokable and no other review path exists. A failed participant instead follows the provider-continuity rules below: local conversational adapters can start a bounded continuation turn, while native sessions use validated resume/replacement. Other adapters retain their action-recovery gates. The original assignee stays unchanged. If that recovery run also finishes while the stage remains pending, or the participant cannot be invoked, Paperclip must move the source issue to an explicit blocked/recovery path instead of leaving `in_review` to drift silently.
### Issue monitors
diff --git a/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts b/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts
index 13f770a6b1..ad30fa4034 100644
--- a/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts
+++ b/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts
@@ -1,5 +1,5 @@
import { randomUUID } from "node:crypto";
-import { and, eq } from "drizzle-orm";
+import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
activityLog,
@@ -70,6 +70,7 @@ vi.mock("../adapters/index.ts", async () => {
import { heartbeatService } from "../services/heartbeat.ts";
import { attentionService } from "../services/attention.ts";
import { issueService } from "../services/issues.ts";
+import { escalateExhaustedIssueReviewPathRecovery } from "../services/recovery/review-path-recovery-escalation.ts";
import { runningProcesses } from "../adapters/index.ts";
import {
buildIssueBlockersResolvedWakeStateKey,
@@ -342,7 +343,7 @@ describeEmbeddedPostgres("heartbeat resolved dependency wake reconciliation", ()
return { companyId, agentId, blockedIssueId, blockerIssueId, executionWorkspaceId };
}
- it("runs exactly one bounded review-path recovery before surfacing a stalled decision", async () => {
+ it.each(["pathless", "monitor"] as const)("runs exactly one bounded review-path recovery with %s disposition", async (disposition) => {
const companyId = randomUUID();
const agentId = randomUUID();
const issueId = randomUUID();
@@ -368,7 +369,7 @@ describeEmbeddedPostgres("heartbeat resolved dependency wake reconciliation", ()
await db.insert(issues).values({
id: issueId,
companyId,
- title: "PAP-14994 fingerprint",
+ title: "External build follow-up",
status: "in_review",
priority: "medium",
assigneeAgentId: agentId,
@@ -377,6 +378,16 @@ describeEmbeddedPostgres("heartbeat resolved dependency wake reconciliation", ()
identifier: `${issuePrefix}-1`,
});
+ if (disposition === "monitor") {
+ const execute = mockAdapterExecute.getMockImplementation()!;
+ mockAdapterExecute.mockImplementationOnce(execute).mockImplementationOnce(async () => {
+ await issueService(db).update(issueId, {
+ monitorNextCheckAt: new Date(Date.now() + 60_000), monitorNotes: "Check external build",
+ });
+ return execute();
+ });
+ }
+
const heartbeat = heartbeatService(db);
const followUpRun = await heartbeat.wakeup(agentId, {
source: "automation",
@@ -420,19 +431,195 @@ describeEmbeddedPostgres("heartbeat resolved dependency wake reconciliation", ()
}),
});
- const attention = await issueService(db)
- .listReviewAttention(companyId, [{ id: issueId, companyId, status: "in_review" }]);
- expect(attention.get(issueId)).toMatchObject({ state: "stalled", paths: [] });
+ const issue = await db.select().from(issues).where(eq(issues.id, issueId)).then((rows) => rows[0]);
+ if (disposition === "monitor") {
+ expect(issue).toMatchObject({ status: "in_review", assigneeAgentId: agentId, monitorNextCheckAt: expect.any(Date) });
+ expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
+ expect(await db.select().from(issueComments)).toHaveLength(0);
+ const attention = await issueService(db).listReviewAttention(companyId, [issue]);
+ expect(attention.get(issueId)).toMatchObject({ state: "covered", paths: expect.arrayContaining([expect.objectContaining({ kind: "monitor" })]) });
+ return;
+ }
+ expect(issue).toMatchObject({ status: "blocked", assigneeAgentId: agentId });
+ const actions = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
+ expect(actions).toHaveLength(1);
+ expect(actions[0]).toMatchObject({
+ ownerType: "board", ownerAgentId: null, ownerUserId: null,
+ returnOwnerAgentId: agentId, cause: "issue_review_path_lost", attemptCount: 1, maxAttempts: 1,
+ });
+ const notices = await db.select().from(issueComments).where(eq(issueComments.issueId, issueId));
+ expect(notices).toHaveLength(1);
+ expect(notices[0]).toMatchObject({ authorType: "system", presentation: expect.objectContaining({ title: "No follow-up scheduled" }) });
+
+ const recoveryRun = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, recoveryWakes[0].runId!)).then((rows) => rows[0]);
+ // Replayed finalization must not duplicate the notice, action, or wake.
+ expect(await escalateExhaustedIssueReviewPathRecovery(db, { run: recoveryRun, issueId })).toBeNull();
+ expect(await db.select().from(issueRecoveryActions)).toHaveLength(1);
+ expect(await db.select().from(issueComments)).toHaveLength(1);
+ expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.reason, "issue_review_path_lost"))).toHaveLength(1);
const feed = await attentionService(db).list(companyId, { userId: "responsible-user" });
- expect(feed.items.find((item) => item.subject.id === issueId)).toMatchObject({
- sourceKind: "review",
+ expect(feed.items.find((item) => item.subject.id === actions[0].id)).toMatchObject({
+ sourceKind: "recovery_action",
+ relatedIssue: expect.objectContaining({ id: issueId }),
decisionVerbs: expect.arrayContaining([
- expect.objectContaining({ id: "choose_review_path", label: "Choose review path" }),
+ expect.objectContaining({ id: "resolve", label: "Resolve" }),
]),
});
});
+ it.each(["monitor", "completed", "reassigned", "newer_run", "other_company"] as const)(
+ "does not escalate a stale review recovery after %s",
+ async (change) => {
+ const companyId = randomUUID();
+ const agentId = randomUUID();
+ const issueId = randomUUID();
+ await db.insert(companies).values({ id: companyId, name: "Review Co", issuePrefix: `R${companyId.slice(0, 6)}` });
+ await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", role: "engineer", adapterType: "codex_local" });
+ await db.insert(issues).values({ id: issueId, companyId, title: "Async review", status: "in_review", assigneeAgentId: agentId });
+ const [run] = await db.insert(heartbeatRuns).values({
+ companyId, agentId, status: "succeeded", issueId,
+ createdAt: new Date(Date.now() - 60_000),
+ contextSnapshot: { issueId, wakeReason: "issue_review_path_lost", reviewPathRecoveryAttempt: 1 },
+ }).returning();
+ if (change === "monitor") {
+ await issueService(db).update(issueId, { monitorNextCheckAt: new Date(Date.now() + 60_000), monitorNotes: "Check build" });
+ } else if (change === "completed") {
+ await issueService(db).update(issueId, { status: "done" });
+ } else if (change === "reassigned") {
+ const nextAgentId = randomUUID();
+ await db.insert(agents).values({ id: nextAgentId, companyId, name: "New reviewer", role: "engineer", adapterType: "codex_local" });
+ await issueService(db).update(issueId, { assigneeAgentId: nextAgentId });
+ } else if (change === "newer_run") {
+ await db.insert(heartbeatRuns).values({ companyId, agentId, issueId, status: "succeeded", contextSnapshot: { issueId } });
+ }
+ expect(await escalateExhaustedIssueReviewPathRecovery(db, {
+ run: change === "other_company" ? { ...run, companyId: randomUUID() } : run,
+ issueId,
+ })).toBeNull();
+ expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
+ expect(await db.select().from(issueComments)).toHaveLength(0);
+ const current = await db.select().from(issues).where(eq(issues.id, issueId)).then((rows) => rows[0]);
+ expect(current.status).toBe(change === "completed" ? "done" : "in_review");
+ if (change === "monitor") expect(current.monitorNextCheckAt).not.toBeNull();
+ },
+ );
+
+ it("serializes concurrent finalization of a failed review repair", async () => {
+ const companyId = randomUUID();
+ const agentId = randomUUID();
+ const issueId = randomUUID();
+ await db.insert(companies).values({ id: companyId, name: "Review Co", issuePrefix: `R${companyId.slice(0, 6)}` });
+ await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", role: "engineer", adapterType: "codex_local" });
+ await db.insert(issues).values({ id: issueId, companyId, title: "Failed review repair", status: "in_review", assigneeAgentId: agentId });
+ const [run] = await db.insert(heartbeatRuns).values({
+ companyId, agentId, issueId, status: "failed",
+ contextSnapshot: { issueId, wakeReason: "issue_review_path_lost", reviewPathRecoveryAttempt: 1 },
+ }).returning();
+ const results = await Promise.all([
+ escalateExhaustedIssueReviewPathRecovery(db, { run, issueId }),
+ escalateExhaustedIssueReviewPathRecovery(db, { run, issueId }),
+ ]);
+ expect(results.filter(Boolean)).toHaveLength(1);
+ expect(await db.select().from(issueRecoveryActions)).toHaveLength(1);
+ expect(await db.select().from(issueComments)).toHaveLength(1);
+ expect(await db.select().from(agentWakeupRequests)).toHaveLength(0);
+ expect(await db.select().from(activityLog).where(eq(activityLog.action, "issue.review_path_recovery_exhausted"))).toHaveLength(1);
+ });
+
+ it.each(["restart", "transaction_failure", "due_monitor", "claimed_monitor", "expired_monitor", "exhausted_monitor", "disabled_monitor"] as const)("reconciles a committed review repair after %s with a one-connection pool", async (gap) => {
+ const companyId = randomUUID();
+ const agentId = randomUUID();
+ const issueId = randomUUID();
+ const wakeId = randomUUID();
+ await db.insert(companies).values({ id: companyId, name: "Review Co", issuePrefix: `R${companyId.slice(0, 6)}`, defaultResponsibleUserId: "responsible-user" });
+ await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer", role: "engineer", adapterType: "codex_local", runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } });
+ await db.insert(issues).values({ id: issueId, companyId, title: "Repair finalized before restart", status: "in_review", assigneeAgentId: agentId, responsibleUserId: "responsible-user" });
+ await db.insert(agentWakeupRequests).values({
+ id: wakeId, companyId, agentId, source: "automation", status: "completed",
+ reason: "issue_review_path_lost", payload: { issueId, reviewPathRecoveryAttempt: 1 },
+ });
+ const [repairRun] = await db.insert(heartbeatRuns).values({
+ companyId, agentId, issueId, status: "succeeded", wakeupRequestId: wakeId, finishedAt: new Date(),
+ contextSnapshot: { issueId, wakeReason: "issue_review_path_lost", reviewPathRecoveryAttempt: 1 },
+ }).returning();
+ const dueAt = new Date(Date.now() - 60_000);
+ if (gap.endsWith("monitor")) {
+ const monitored = await issueService(db).update(issueId, {
+ monitorNextCheckAt: dueAt, monitorNotes: "Check external build",
+ executionPolicy: { monitor: {
+ nextCheckAt: dueAt.toISOString(), notes: "Check external build", scheduledBy: "assignee",
+ kind: "external_service", serviceName: "github_checks",
+ ...(gap === "disabled_monitor" ? {} : { timeoutAt: new Date(Date.now() + 60_000).toISOString(), maxAttempts: 3 }),
+ } },
+ });
+ if (gap === "disabled_monitor") {
+ // A policy change after saving an unbounded check must not hide the
+ // spent repair forever: dispatch cannot accept its on-demand wake.
+ await db.update(agents).set({ runtimeConfig: { heartbeat: { wakeOnDemand: false } } }).where(eq(agents.id, agentId));
+ } else if (gap === "claimed_monitor") {
+ await db.update(issues).set({ monitorWakeRequestedAt: new Date() }).where(eq(issues.id, issueId));
+ } else if (gap === "expired_monitor" || gap === "exhausted_monitor") {
+ const policy = monitored!.executionPolicy as { monitor: Record<string, unknown> };
+ await db.update(issues).set({
+ executionPolicy: { ...policy, monitor: { ...policy.monitor, ...(gap === "expired_monitor" ? { timeoutAt: new Date(Date.now() - 1_000).toISOString() } : { maxAttempts: 1 }) } },
+ ...(gap === "exhausted_monitor" ? { monitorAttemptCount: 1 } : {}),
+ }).where(eq(issues.id, issueId));
+ }
+ }
+ // Only the terminal run and completed wake were persisted before the crash.
+ // A new service must restore the missing action without invoking the agent.
+ const singleConnectionDb = createDb(tempDb!.connectionString, { maxConnections: 1 });
+ try {
+ if (gap === "transaction_failure") {
+ await db.execute(sql`alter table issue_comments add constraint review_repair_notice_test check (false) not valid`);
+ try {
+ await expect(escalateExhaustedIssueReviewPathRecovery(singleConnectionDb, { run: repairRun, issueId })).rejects.toThrow();
+ } finally {
+ await db.execute(sql`alter table issue_comments drop constraint review_repair_notice_test`);
+ }
+ expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
+ expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("in_review");
+ }
+ const recovery = heartbeatService(singleConnectionDb);
+ const result = await recovery.reconcileStrandedAssignedIssues();
+ if (gap === "due_monitor" || gap === "claimed_monitor") {
+ expect(result.escalated).toBe(0);
+ expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
+ expect(await db.select().from(issueComments)).toHaveLength(0);
+ const current = (await db.select().from(issues).where(eq(issues.id, issueId)))[0];
+ expect(current).toMatchObject({ status: "in_review", monitorNextCheckAt: dueAt });
+ // A claim left by a crashed dispatcher can be reclaimed after its lease.
+ if (gap === "claimed_monitor") {
+ await db.update(issues).set({ monitorWakeRequestedAt: new Date(Date.now() - 6 * 60_000) }).where(eq(issues.id, issueId));
+ }
+ const execute = mockAdapterExecute.getMockImplementation()!;
+ mockAdapterExecute.mockImplementationOnce(async () => {
+ await issueService(db).update(issueId, { status: "done" });
+ return execute();
+ });
+ const dispatcher = heartbeatService(db, { runtimeEnv: {} });
+ await dispatcher.tickTimers();
+ await dispatcher.drainActiveRunExecutions();
+ const wakes = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.reason, "issue_monitor_due"));
+ expect(wakes).toHaveLength(1);
+ expect(mockAdapterExecute).toHaveBeenCalledOnce();
+ expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "done", monitorNextCheckAt: null });
+ return;
+ }
+ expect(result).toMatchObject({ escalated: 1, issueIds: [issueId] });
+ expect(await db.select().from(issueRecoveryActions)).toHaveLength(1);
+ expect(await db.select().from(issueComments)).toHaveLength(1);
+ const second = await recovery.reconcileStrandedAssignedIssues();
+ expect(second.escalated).toBe(0);
+ expect(await db.select().from(agentWakeupRequests)).toHaveLength(1);
+ expect(mockAdapterExecute).not.toHaveBeenCalled();
+ expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "blocked", assigneeAgentId: agentId });
+ } finally {
+ await singleConnectionDb.$client.end({ timeout: 1 });
+ }
+ });
+
it("keeps resolved dependency wake reconciliation active", async () => {
const { companyId, agentId, blockedIssueId, blockerIssueId } =
await seedResolvedDependencyBackstopFixture({ workspaceState: "none" });
diff --git a/server/src/services/heartbeat/run-lifecycle.ts b/server/src/services/heartbeat/run-lifecycle.ts
index 11abf33852..135a085292 100644
--- a/server/src/services/heartbeat/run-lifecycle.ts
+++ b/server/src/services/heartbeat/run-lifecycle.ts
@@ -150,6 +150,7 @@ import {
reviewPathConsumedRefFromRun,
} from "../recovery/review-path-recovery.js";
import { resolveRequiredSuccessfulRunHandoffOnValidPath } from "../successful-run-handoff-state.js";
+import { escalateExhaustedIssueReviewPathRecovery } from "../recovery/review-path-recovery-escalation.js";
import {
redactCurrentUserText,
redactCurrentUserValue,
@@ -1987,6 +1988,10 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
reviewAttention,
existingWake: Boolean(existingWake),
});
+ if (decision.kind === "exhausted") {
+ await escalateExhaustedIssueReviewPathRecovery(db, { run, issueId: issue.id });
+ return;
+ }
if (decision.kind !== "enqueue") return;
const recoveryRun = await enqueueWakeup(issue.assigneeAgentId, {
diff --git a/server/src/services/recovery/issue-graph-liveness.ts b/server/src/services/recovery/issue-graph-liveness.ts
index 4e270b8bb7..c04e1c37f7 100644
--- a/server/src/services/recovery/issue-graph-liveness.ts
+++ b/server/src/services/recovery/issue-graph-liveness.ts
@@ -194,10 +194,11 @@ function monitorFromIssue(issue: IssueLivenessIssueInput) {
return { policyMonitor, stateMonitor };
}
-export function hasScheduledIssueMonitorPath(issue: IssueLivenessIssueInput, now: Date | string | number) {
+/** A saved, unconsumed check remains dispatchable when its due time passes. */
+export function hasUnconsumedIssueMonitorPath(issue: IssueLivenessIssueInput, now: Date | string | number) {
const nowMs = typeof now === "number" ? now : readDateMs(now) ?? Date.now();
const nextCheckAtMs = readDateMs(issue.monitorNextCheckAt);
- if (nextCheckAtMs === null || nextCheckAtMs <= nowMs) return false;
+ if (nextCheckAtMs === null) return false;
const { policyMonitor, stateMonitor } = monitorFromIssue(issue);
const timeoutAtMs = readDateMs(policyMonitor?.timeoutAt ?? stateMonitor?.timeoutAt);
@@ -211,6 +212,12 @@ export function hasScheduledIssueMonitorPath(issue: IssueLivenessIssueInput, now
return true;
}
+export function hasScheduledIssueMonitorPath(issue: IssueLivenessIssueInput, now: Date | string | number) {
+ const nowMs = typeof now === "number" ? now : readDateMs(now) ?? Date.now();
+ const nextCheckAtMs = readDateMs(issue.monitorNextCheckAt);
+ return nextCheckAtMs !== null && nextCheckAtMs > nowMs && hasUnconsumedIssueMonitorPath(issue, nowMs);
+}
+
export function classifyIssueReviewPaths(
input: IssueGraphLivenessInput,
issue: IssueLivenessIssueInput,
diff --git a/server/src/services/recovery/review-path-recovery-escalation.ts b/server/src/services/recovery/review-path-recovery-escalation.ts
new file mode 100644
index 0000000000..dff88494cf
--- /dev/null
+++ b/server/src/services/recovery/review-path-recovery-escalation.ts
@@ -0,0 +1,134 @@
+import { and, desc, eq, getTableColumns, or, sql } from "drizzle-orm";
+import { agents, heartbeatRuns, issues, type Db } from "@paperclipai/db";
+import { isWaitingConversation } from "../agent-conversations.js";
+import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
+import { issueRecoveryActionService } from "../issue-recovery-actions.js";
+import { externalConversationStateSql } from "../slack-conversation-state.js";
+import { isHeartbeatWakeOnDemandEnabled } from "../heartbeat-policy.js";
+import {
+ executeIssuePostCommitActions,
+ issueService,
+ TERMINAL_HEARTBEAT_RUN_STATUSES,
+ type IssuePostCommitAction,
+} from "../issues.js";
+import {
+ buildIssueReviewPathLostIdempotencyKey,
+ decideIssueReviewPathRecovery,
+ ISSUE_REVIEW_PATH_LOST_WAKE_REASON,
+ reviewPathConsumedRefFromRun,
+} from "./review-path-recovery.js";
+import { buildStrandedRecoveryEscalationNotice } from "./stranded-notice.js";
+import { hasUnconsumedIssueMonitorPath } from "./issue-graph-liveness.js";
+
+/** A spent repair wake must leave an operator action, never another automatic wake. */
+export async function escalateExhaustedIssueReviewPathRecovery(
+ db: Db,
+ input: { run: typeof heartbeatRuns.$inferSelect; issueId: string },
+) {
+ const { run, issueId } = input;
+ const publications: ActivityPublication[] = [];
+ const postCommitActions: IssuePostCommitAction[] = [];
+ const action = await db.transaction(async (tx) => {
+ // Services must use the transaction connection for implicit settings reads
+ // too; borrowing the outer pool deadlocks when its only connection is held.
+ const issuesSvc = issueService(tx as unknown as Db);
+ // Share the issue fence with ownership, monitor, and interaction updates.
+ // Re-read the disposition under the lock rather than blocking a repaired task.
+ const issue = await tx.select({ ...getTableColumns(issues), externalConversationState: externalConversationStateSql() }).from(issues)
+ .where(and(eq(issues.companyId, run.companyId), eq(issues.id, issueId)))
+ .for("update").then((rows) => rows[0] ?? null);
+ if (!issue || issue.status !== "in_review" || issue.assigneeAgentId !== run.agentId
+ || issue.assigneeUserId || isWaitingConversation(issue)
+ || issue.externalConversationState === "waiting") return null;
+
+ // Startup recovery precedes monitor ticks. Preserve a due or claimed check
+ // only while the owner still allows the on-demand wake used by dispatch.
+ // Otherwise the dispatcher leaves it overdue indefinitely without consuming it.
+ if (hasUnconsumedIssueMonitorPath(issue, new Date())) {
+ const owner = await tx.select({ runtimeConfig: agents.runtimeConfig }).from(agents)
+ .where(and(eq(agents.companyId, run.companyId), eq(agents.id, issue.assigneeAgentId)))
+ .then((rows) => rows[0] ?? null);
+ if (owner && isHeartbeatWakeOnDemandEnabled(owner)) return null;
+ }
+
+ const attention = (await issuesSvc.listReviewAttention(run.companyId, [issue], tx)).get(issueId);
+ if (!attention || decideIssueReviewPathRecovery({
+ issueId,
+ sourceRunId: run.id,
+ assigneeAgentId: issue.assigneeAgentId,
+ contextSnapshot: run.contextSnapshot,
+ reviewAttention: attention,
+ existingWake: false,
+ }).kind !== "exhausted") return null;
+
+ const latestRun = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
+ .where(and(eq(heartbeatRuns.companyId, run.companyId), or(
+ eq(heartbeatRuns.issueId, issueId),
+ eq(heartbeatRuns.nativeIssueId, issueId),
+ sql`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId') = ${issueId}`,
+ )))
+ .orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1)
+ .then((rows) => rows[0] ?? null);
+ if (latestRun?.id !== run.id || !TERMINAL_HEARTBEAT_RUN_STATUSES.has(run.status)) return null;
+
+ const consumedPathRef = reviewPathConsumedRefFromRun({
+ runId: run.id, issueId, contextSnapshot: run.contextSnapshot,
+ });
+ const nextAction = "Inspect the run, then restore a scheduled check or review path, retry the original assignee, or record an intentional resolution.";
+ const recoveryAction = await issueRecoveryActionService(tx as unknown as Db).upsertSourceScoped({
+ companyId: run.companyId,
+ sourceIssueId: issueId,
+ kind: "missing_disposition",
+ ownerType: "board",
+ previousOwnerAgentId: run.agentId,
+ returnOwnerAgentId: run.agentId,
+ cause: ISSUE_REVIEW_PATH_LOST_WAKE_REASON,
+ fingerprint: buildIssueReviewPathLostIdempotencyKey({ issueId, consumedPathRef }),
+ evidence: {
+ correctiveRunId: run.id,
+ consumedPathRef,
+ previousStatus: "in_review",
+ recoveryAttempt: 1,
+ maxRecoveryAttempts: 1,
+ },
+ nextAction,
+ wakePolicy: { type: "board_escalation", preservesSourceAssignee: true },
+ attemptCount: 1,
+ maxAttempts: 1,
+ lastAttemptAt: run.finishedAt ?? new Date(),
+ });
+ await issuesSvc.update(issueId, {
+ status: "blocked", companyGuard: run.companyId,
+ }, tx, publications, postCommitActions);
+ const notice = buildStrandedRecoveryEscalationNotice({
+ seed: {
+ title: "No follow-up scheduled",
+ tone: "danger",
+ body: "The automatic review-path repair finished without saving a scheduled check, reviewer, approval, or other maintained path. Paperclip has blocked this task for a board decision; no further automatic follow-up is scheduled.",
+ nextAction,
+ },
+ recoveryCause: ISSUE_REVIEW_PATH_LOST_WAKE_REASON,
+ recoveryActionId: recoveryAction.id,
+ recoveryOwner: null,
+ sourceRun: { id: run.id, agentId: run.agentId, status: run.status },
+ });
+ const comment = await issuesSvc.addComment(issueId, notice.body, {}, {
+ authorType: "system", presentation: notice.presentation, metadata: notice.metadata,
+ }, tx);
+ await logActivity(tx as unknown as Db, {
+ companyId: run.companyId,
+ actorType: "system",
+ actorId: "heartbeat",
+ agentId: run.agentId,
+ runId: run.id,
+ action: "issue.review_path_recovery_exhausted",
+ entityType: "issue",
+ entityId: issueId,
+ details: { recoveryActionId: recoveryAction.id, commentId: comment.id, consumedPathRef, recoveryAttempt: 1, maxRecoveryAttempts: 1 },
+ }, publications);
+ return recoveryAction;
+ });
+ for (const publication of publications) publishActivity(publication);
+ await executeIssuePostCommitActions(db, postCommitActions);
+ return action;
+}
diff --git a/server/src/services/recovery/review-path-recovery.test.ts b/server/src/services/recovery/review-path-recovery.test.ts
index 5e651d4594..a2a49dafed 100644
--- a/server/src/services/recovery/review-path-recovery.test.ts
+++ b/server/src/services/recovery/review-path-recovery.test.ts
@@ -139,20 +139,20 @@ describe("review-path recovery", () => {
},
);
- it("does not requeue when the bounded recovery run also ends pathless", () => {
+ it.each([
+ { wakeReason: ISSUE_REVIEW_PATH_LOST_WAKE_REASON },
+ { reviewPathRecoveryAttempt: 1 },
+ ])("requires escalation when the bounded recovery run also ends pathless (%j)", (contextSnapshot) => {
const decision = decideIssueReviewPathRecovery({
issueId: "issue-1",
sourceRunId: "run-2",
assigneeAgentId: "agent-1",
- contextSnapshot: {
- wakeReason: ISSUE_REVIEW_PATH_LOST_WAKE_REASON,
- reviewPathRecoveryAttempt: 1,
- },
+ contextSnapshot,
reviewAttention: stalled,
existingWake: false,
});
- expect(decision).toEqual({ kind: "skip", reason: "bounded review-path recovery already ran" });
+ expect(decision).toEqual({ kind: "exhausted" });
});
it("never wakes a healthy review", () => {
diff --git a/server/src/services/recovery/review-path-recovery.ts b/server/src/services/recovery/review-path-recovery.ts
index afcd315647..1ab57f2f8d 100644
--- a/server/src/services/recovery/review-path-recovery.ts
+++ b/server/src/services/recovery/review-path-recovery.ts
@@ -6,13 +6,18 @@ import { withRecoveryContext } from "./status-only-context.js";
export const ISSUE_REVIEW_PATH_LOST_WAKE_REASON = "issue_review_path_lost";
export const REVIEW_PATH_RECOVERY_INSTRUCTION =
- "This issue is still in review but its last maintained review path was consumed. Restore a reviewer, interaction, approval, monitor, or other durable waiting path, or choose an explicit disposition. This is the only automatic review-path recovery wake for this fingerprint.";
+ "This issue is still in review but its last maintained review path was consumed. Restore a reviewer, interaction, approval, monitor, or other durable waiting path, or choose an explicit disposition. If an async check is still pending, persist a one-shot issue monitor before ending this run and report its scheduled check time. A promise to check later or a background watcher is not a maintained review path. This is the only automatic review-path recovery wake for this fingerprint; ending without a maintained path will require a board decision.";
const REVIEW_PATH_RECOVERY_IDEMPOTENCY_INDEX = "agent_wakeup_requests_review_path_recovery_idempotency_uq";
function readNonEmptyString(value: unknown) {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
+export function isIssueReviewPathRecoveryRun(contextSnapshot: Record<string, unknown> | null | undefined) {
+ return readNonEmptyString(contextSnapshot?.wakeReason) === ISSUE_REVIEW_PATH_LOST_WAKE_REASON
+ || contextSnapshot?.reviewPathRecoveryAttempt === 1;
+}
+
export function reviewPathConsumedRefFromRun(input: {
runId: string;
issueId: string;
@@ -68,6 +73,7 @@ export type IssueReviewPathRecoveryDecision =
payload: Record<string, unknown>;
contextSnapshot: Record<string, unknown>;
}
+ | { kind: "exhausted" }
| { kind: "skip"; reason: string };
export function decideIssueReviewPathRecovery(input: {
@@ -84,11 +90,8 @@ export function decideIssueReviewPathRecovery(input: {
}
const context = input.contextSnapshot ?? {};
- if (
- readNonEmptyString(context.wakeReason) === ISSUE_REVIEW_PATH_LOST_WAKE_REASON
- || context.reviewPathRecoveryAttempt === 1
- ) {
- return { kind: "skip", reason: "bounded review-path recovery already ran" };
+ if (isIssueReviewPathRecoveryRun(context)) {
+ return { kind: "exhausted" };
}
const consumedPathRef = reviewPathConsumedRefFromRun({
diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts
index 8a90da0fe4..785753f0ec 100644
--- a/server/src/services/recovery/service.ts
+++ b/server/src/services/recovery/service.ts
@@ -4,6 +4,8 @@ import { hasCommittedNativePlanWait } from "../native-runtime/native-plan-wait.j
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
import { settleSlackConversation } from "../slack-conversation-lifecycle.js";
import { externalConversationStateSql } from "../slack-conversation-state.js";
+import { isIssueReviewPathRecoveryRun } from "./review-path-recovery.js";
+import { escalateExhaustedIssueReviewPathRecovery } from "./review-path-recovery-escalation.js";
import { executionRetryAccounting } from "../execution-recovery-attempt.js";
import { isExplicitContinuationRetryClaim } from "../explicit-continuation-retry-claim.js";
import {
@@ -4472,6 +4474,21 @@ export function recoveryService(
}
let latestRun = await getLatestIssueRun(issue.companyId, issue.id);
+ // Terminal run/wake rows survive a restart even if finalization never
+ // reached the escalation transaction. Retry that idempotent disposition
+ // before the participant-only review branch can skip pathless reviews.
+ if (issue.status === "in_review" && latestRun
+ && isTerminalIssueRun(latestRun)
+ && isIssueReviewPathRecoveryRun(latestRun.contextSnapshot)) {
+ const [repairRun] = await db.select().from(heartbeatRuns).where(and(
+ eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, latestRun.id),
+ ));
+ if (repairRun && await escalateExhaustedIssueReviewPathRecovery(db, { run: repairRun, issueId: issue.id })) {
+ result.escalated += 1;
+ result.issueIds.push(issue.id);
+ continue;
+ }
+ }
// A native chat can finish between the earlier settlement read and this
// fresh run read, before its response is materialized. Its trusted
// finalizer owns that settlement; generic productive-work recovery must