Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Harness Run Reliability
Status (2026-08-06): Every implementation unit through PR4 is merged. PR1, PR2, PR5, PR6, #1139's Activity-output settlement closure, PR3 (#1155), and PR4 (#1173) are complete. Only PR7R remains as the next unit, and it is evidence-only. PR4's conditional transport-attempt delta (PR4B) opens only if a current-master reproducer proves a missing durable fact.
This is the execution plan, not the investigation log. The original detailed
diagnosis and its review history remain available in Git before fe821905.
Resolve code references by symbol against current master; do not port old line
numbers or old ownership assumptions.
1. Current baseline
| Capability | Result |
|---|---|
| Result text for delivered Harness runs | #1063 merged |
| Pinned-session reclaim and reservation hardening | #1064 merged |
| Durable, user-visible failure notices | #1072 merged |
| Delivery / Turn / Message ownership model | #1134 merged |
| Teardown-interrupted Run settlement | #1140 merged; supersedes closed #1131 |
| Activity output batch receipt and local settlement | #1139 merged; supersedes #1121 |
| Idle-eviction interlock for queued work | #1155 merged (PR3); scenarios HFR-130…154 |
| Bounded and supervised shared drains | #1173 merged (PR4); scenarios HFR-155…179; attempt-state delta (PR4B) still requires a current-master reproducer |
| Scheduled/watch terminal-time truth and cron liveness | Open — PR7R, evidence-only; HFR-180…219 reserved and unoccupied as of 2026-08-06 |
The post-plan architecture is load-bearing:
message_deliveriesowns submitted input, FIFO position, acceptance, attempts, receipts, and retirement.session_turnsowns native execution and terminal Turn evidence.messagesis accepted communication history, not a queue.- durable Activity snapshots own pending output payload, ordered batch membership, the linked Run union, and the stable output receipt identity. Accepted Message evidence or a persisted Activity batch marked for local settlement only prevents transport replay; a delivered output that still owes local settlement retries only that settlement.
agent_runsis the Harness projection and callback/notice anchor.- #1140 distinguishes an unstarted claim from an execution that crossed the running boundary. Unstarted work returns to queued; infrastructure-interrupted running work fails and is not replayed; explicit user Stop remains canceled.
Consequently, the old PR2 resolver design, PR7's old “claimed message row” decision, and any pre-#1139 single-Activity output assumptions are obsolete. New work must use the current durable owners instead of recreating side maps or a parallel output ledger.
2. Invariants
Every remaining change must preserve these rules:
- One durable owner per phase. Delivery owns input before native acceptance; Turn owns accepted execution; the Activity batch owns pending terminal output; Run reflects the outcome.
- No silent replay after execution starts. Infrastructure interruption is
failedwith a structured cause and an actionable notice. User Stop iscanceledand does not generate an infrastructure-failure notice. - Unstarted work remains retryable. A bare claim or queued Delivery must not be mislabeled as an interrupted execution.
- Terminal writes are guarded. A natural terminal result, user Stop, and infrastructure teardown may race; the exact compare-and-set winner controls projections and notices.
- Absence of a receipt is not proof of non-delivery. Any timeout around an outbound send must use durable delivery evidence before retrying.
- Waiting is not activity. A real inbound message may establish a session baseline, and every exact Turn start establishes a fresh one even when the work came from a long-idle scheduled/watch queue; after that only observable assistant/tool progress refreshes it. Run inactivity is stricter and re-arms only from its exact owning Turn. A claim, queue wait, gate wait, or unrelated Run in the same session must not keep stuck work alive.
- No turn-duration timeout. A healthy turn may run for hours. Bounds may apply to inactivity and post-turn delivery, never to productive execution.
- Failures remain visible. Reconcile paths must use #1072's durable notice path; a terminal row alone is not the exit criterion.
- One output batch has one receipt. Preserve #1139's stable receipt, persisted ordered Activity membership, complete linked Run union, and transport-free local-settlement retry. Incomplete or conflicting recovered membership fails closed; it never emits a partial batch or invents a second receipt.
3. Runtime supervision design
Consider one concrete Session. It has an old P3 Delivery Q1 and no live Turn.
That durable fact is runnable: the delivery lane must claim Q1, while the
idle backend process remains reclaimable until the Turn actually starts. If an
empty P1 send-now races the recovery wake, its synchronous writer validates and
claims exactly the observed head in one transaction. A subsequent
queue.updated hint is for projection and passive recovery, never the authority
for send-now's stale-head/refusal result. At no point is last_activity
falsified, and neither the hint nor the reclaimer owns Q1.
Now suppose recovered Activity output delivery is hung at the same time. Its
exact Activity batch worker remains single-flight for its partition and overdue,
but another output partition and the independent session_deliveries lane can
still progress. This is the user-visible outcome the design must produce.
3.1 Separate facts, hints, and resources
The implementation has three layers with deliberately different failure semantics:
- Durable facts are the existing Delivery, Turn, Activity, and Run rows. They remain correct across process death and are the only source of truth for work ownership and settlement.
- Wake hints say only "durable state may now be actionable; read it again". They carry no payload or lifecycle authority, may be coalesced, and may be lost without losing work.
- Runtime resources are backend clients, transports, asyncio tasks, and worker threads. They are disposable projections that may be reclaimed only after the durable facts prove that no exact owner still needs them.
This separates two existing loops that must not become one larger loop:
- the session runtime reclaimer releases idle backend resources;
- the Harness work supervisor activates and reconciles durable work.
The reclaimer never executes queued work and the work supervisor never decides that a durable Session row should be deleted. They share one read-only runtime ownership provider and one wake interface, not one coroutine.
They also share one runtime activation boundary per exact backend resource
target and generation. This is an in-process mutation interlock, not a durable
owner: every transition that can change the target from
runnable|waiting|reclaimable to transitioning|active enters it before the
authoritative commit, and reclamation enters it before the final snapshot. The
boundary is defined by the disposition change rather than by an enumerated
caller list, so a new admission path cannot bypass it merely because it creates
a Run instead of a Turn or Activity.
3.2 Derived session disposition
The runtime ownership provider reads Delivery, Turn, active Activity, and
fallback Run facts from one SQLite read transaction and returns one immutable
snapshot. It does not persist a new state or maintain an
in-memory shadow ledger. It is strictly side-effect free: ownership reads must
not release locks, repair projections, settle Runs, or call helpers such as
owned_agent_run_ids() that mutate while answering.
The provider explicitly executes SQLite BEGIN before its first SELECT and
keeps that read transaction open through every Delivery, Turn, Activity, Run,
and Session query. SQLAlchemy engine.begin() alone is not accepted as evidence
because pysqlite may defer the actual BEGIN for read-only statements. An
equivalent single compound SQL statement is also valid. Handoff tests use one
real SQLite file, two engines/connections, and a barrier after the first read to
prove that the result cannot combine two commit generations.
class SessionRuntimeDisposition(str, Enum):
ACTIVE = "active" # an exact live owner needs the runtime
TRANSITIONING = "transitioning" # an unresolved handoff forbids reclamation
RUNNABLE = "runnable" # durable work should be activated now
RECLAIMABLE = "reclaimable" # no durable or in-process owner remains
UNKNOWN = "unknown" # the snapshot could not prove safety
@dataclass(frozen=True)
class SessionRuntimeOwnershipSnapshot:
session_id: str
disposition: SessionRuntimeDisposition
delivery_ids: tuple[str, ...]
turn_ids: tuple[str, ...]
active_activity_ids: tuple[str, ...]
fallback_run_ids: tuple[str, ...]
reasons: tuple[str, ...]
@dataclass(frozen=True)
class RuntimeTargetOwnershipSnapshot:
backend: str
resource_key: str
activity_runtime_keys: tuple[str, ...]
sessions: tuple[SessionRuntimeOwnershipSnapshot, ...]
sessionless_active_activity_ids: tuple[str, ...]
sessionless_fallback_run_ids: tuple[str, ...]
disposition: SessionRuntimeDisposition
Reclamation operates on a runtime resource target, not blindly on one Session.
The disposable resource key and durable Activity runtime key are separate
identities. For example, one Codex transport resource is keyed by cwd, while its
Activity/Turn keys are base_session_id:cwd; one cwd can therefore map to many
Activity runtime keys. Each backend exposes its exact current resource key,
candidate Session bindings, and the set of Activity runtime keys belonging to
that resource. Every binding carries the stable agent_sessions.id as its
ownership identity; session_anchor and workdir remain adapter routing facts,
not ownership keys. Archive may rewrite an anchor to vacate an admission alias,
but it cannot transfer or erase unresolved ownership. The provider never
derives this relation by splitting strings or by scanning same-workdir Sessions.
The shared provider resolves those exact Session IDs to durable
agent_sessions rows and
computes each Session snapshot in one SQLite transaction. The same query
collects active Activity rows for the backend and matches them through the
adapter-supplied resource-to-Activity-key mapping even when session_id is NULL;
those rows are target-level owners and force active. An active Activity whose
runtime key cannot be mapped is unknown, not an orphan. The target disposition
is the safest member disposition, using unknown > active > transitioning > runnable > waiting > reclaimable for the aggregate. Before the cleanup
decision, every runnable member emits its wake even when another member already
blocks the shared target. Any target-level active Activity or active,
transitioning, or unknown Session blocks target cleanup.
runnable wakes activation but, like waiting, does not itself own the current
backend process; an all-runnable|waiting|reclaimable target may release idle
resources after the final locked recheck. A positively orphaned runtime with no
durable binding may be reclaimed, but a mapping error is unknown.
Fallback Runs receive the same route treatment. An IM/CLI Run may have no
session_id while its persisted legacy_session_key, backend/agent identity,
and execution metadata still name the target. Each backend maps that exact route
to its resource key without string parsing. A queued Session-less Run is
runnable; its pre-execution and PID-started forms are transitioning and
active on the mapped target. If a nonterminal Session-less Run cannot be
mapped, that backend's target snapshot is unknown and reclamation fails closed
for the cycle rather than silently omitting the owner.
The exact classification is:
| Durable fact | Derived disposition | Runtime effect |
|---|---|---|
| `starting | active` Turn with its exact owned Delivery, persisted active Activity, or exact in-process backend operation | active |
exact linked waiting Turn plus its initial interrupt_waiting Delivery |
transitioning |
fail closed for this pass and reconcile that exact predecessor/successor owner |
| unresolved Delivery fence or an ownership handoff that cannot yet be classified | transitioning |
fail closed; reconcile the exact owner |
| ownerless FIFO head or queued execution-bearing fallback Run not yet represented by a Delivery | runnable |
wake its delivery/request lane; it does not itself pin backend resources |
claimed fallback Run in normalized running / legacy processing with no persisted execution-start marker (pid IS NULL or absent) |
transitioning |
preserve the pre-execution handoff until exact recovery requeues or starts it |
fallback Run in normalized running / legacy processing with the persisted execution-start marker (pid IS NOT NULL) |
active |
preserve its runtime until #1140's exact terminal/teardown settlement |
| terminal Delivery/Turn history and terminal Runs only | reclaimable |
no pin |
read failure, unknown Delivery state or execution-bearing Run type, or an unpaired/mismatched waiting/interrupt_waiting half |
unknown |
fail closed for this cycle and log the reason |
queued therefore does not mean "keep a backend process alive forever". An
ownerless queue head means "activate it now", while its durable row survives
runtime reclamation until the delivery lane creates the exact Turn owner. This
resolves the apparent contradiction between an idle runtime and durable queued
input without inventing fake activity or a second policy state.
The provider consumes DELIVERY_STATE_MATRIX and the current Turn-state
contract as related facts, not as independent liveness enums. The exact
waiting Turn / interrupt_waiting initial-Delivery pair is the specific
exception to the matrix's general turn_owned ordering role: it has not crossed
native start and is transitioning, not active. A starting|active Turn with
its exact owned Delivery is active; a lone or mismatched half is unknown. The
pair's wake is not permission to call activate_waiting_successor() directly.
Activation must re-read the exact predecessor Turn, prove its
control_successor_turn_id/control_successor_delivery_id linkage, and consume
the predecessor's immutable terminal winner through the existing terminal-
resume path. Before that terminal proof, targeted recovery may reconcile only
the predecessor control attempt and must leave the successor waiting.
The provider may add a derived policy helper beside the matrix, but it may not retype
Delivery state names in cleanup callers. watch_runtime is supervisor
bookkeeping and never an execution owner. An Activity persisted in active
phase pins its exact runtime; awaiting_output and terminal Activity snapshots
are settlement-only and do not pin backend resources. Unknown Run or Activity
types fail closed until the shared classification deliberately names them.
3.3 Activation before reclamation
Every reclaimer pass follows one order:
- Read the exact ownership snapshot.
- Derive wakes from the raw member facts before target aggregation. For a
runnable Delivery, emit a coalescing
session_deliverieswake. A fallback queued Run emitsrequestsinstead. A transitioning Delivery/Turn owner, including every exact waiting-successor pair, wakessession_deliverieseven when its active predecessor makes the aggregate dispositionactive; a pre- execution fallback Run wakesrequests. The row is durable work, not a runtime pin. - For
active,transitioning, orunknown, skip reclamation. - For
runnable,waiting, orreclaimable, apply the existing idle grace rule without mutating Session, hold, queue, Delivery, or Run. - Acquire the exact target/generation runtime activation boundary, then read the snapshot again immediately before cleanup. A new exact owner or replaced generation wins; a still-runnable row may outlive the reclaimed runtime and later recreate it safely.
Every path that crosses the disposition boundary participates. This includes
Turn start, a fallback Run claim from queued to normalized running/legacy
processing, the fallback Run PID execution-start transition, and
SessionActivityRegistry.start() or a progress upsert that can recreate an
active row. The adapter maps the durable route or Activity runtime key to the
disposable resource target before acquiring the boundary. The admission path
revalidates the observed generation and holds the boundary through its
authoritative commit. No call may split claim and generation validation across
an unlocked window.
If admission wins first, the reclaimer's locked snapshot observes the new
transitioning|active owner. If cleanup wins first, it marks or detaches that
exact generation while holding the boundary. A delayed old-generation Activity
event may not create an active row; it follows the existing exact backend/Run
settlement path when it has an owner, otherwise it is ignored as stale
telemetry. A queued fallback Run remains queued or is recovered through its
existing exact Run CAS, then may activate against the current/new generation;
it is never marked execution-started against the retired generation. This adds
no Run or Activity state, durable fence, or terminal writer.
The reclaimer does not wait for the request to finish. The wake is a hint; the
session_deliveries lane later claims the exact FIFO head under the existing guarded
Delivery/Turn transaction. If the hint is lost, startup and periodic
reconciliation rediscover the same row. A losing claim re-reads; invalid input
is retired by the existing exact-Delivery validation; temporary unavailability
leaves the row durable and uses bounded lane backoff. None of these outcomes
refreshes session_last_activity or pins the old runtime.
Activation and transition recovery never depend on a runtime target already
existing or reaching the cleanup loop. The session_deliveries lane has two
bounded indexed eligibility queries: Session ids with a claimable queued head
and no live Turn, and unresolved Delivery/Turn fences, starting owners, or
waiting successors that require exact reconciliation. It invokes a narrow
SessionTurnManager claim/recovery entry for the exact Session plus observed
predecessor Turn/control attempt, successor Turn, Delivery, and native identity.
The manager must re-prove the predecessor's exact successor linkage and terminal
snapshot before its existing terminal-resume writer may activate the successor;
the low-level successor CAS is never a standalone recovery entry. A wake that
arrives while the predecessor is still nonterminal no-ops after any exact control
reconciliation. The requests lane likewise includes claimed pre-execution
fallback Runs. Losing or stale identities no-op after their guarded re-read.
Queue events, startup, and periodic reconciliation are the primary triggers; the
reclaimer's wake closes only the admission-versus-cleanup race.
The full global recover_durable_delivery_state() remains startup-only after
backend identity restoration.
The stuck-active backstop is not an ordinary idle timeout. It applies only when the exact owner has stopped producing observable progress past the existing threshold, and it routes through #1140's guarded teardown settlement. A long tool call without progress may still hit this explicit infrastructure-failure policy; this work does not add another recovery layer or silently replay it.
3.4 Work lanes and wake protocol
Add one controller-owned RuntimeWorkSupervisor and replace the serial executor
roles of both ScheduledTaskService._watch_store and
ManagedWatchService._watch_store with independently registered, partitioned
work lanes:
class RuntimeWorkLane(str, Enum):
SESSION_DELIVERIES = "session_deliveries"
TASK_DEFINITIONS = "task_definitions"
WATCH_DEFINITIONS = "watch_definitions"
REQUESTS = "requests"
RUN_CALLBACKS = "run_callbacks"
VAULT_CALLBACKS = "vault_callbacks"
ACTIVITY_OUTPUTS = "activity_outputs"
FAILURE_NOTICES = "failure_notices"
STALE_RUNS = "stale_runs"
def notify_runtime_work(*lanes: RuntimeWorkLane) -> None:
"""Coalesce wake hints; never carry durable payload or imply acceptance."""
The supervisor owns scheduling only. SessionTurnManager registers the
session_deliveries handler; ScheduledTaskService and ManagedWatchService
register only the Harness handlers they own while their service generation is
valid. Registration returns a generation token; duplicate live registration is
rejected, and unregistering a token suppresses re-arm from that generation
before awaiting its exact worker. An event for an unregistered lane is
discarded because startup reconciliation is mandatory when a new handler
registers.
PR3 also registers the minimal requests reconciliation handler needed for
claimed pre-execution fallback Runs: it can only requeue or preserve the exact
observed Run under the existing PID/owner fence. It does not admit general
queued Runs. PR4 replaces that registration, through the same generation-token
handoff, with the complete bounded requests admission handler. PR3 therefore
introduces no interval in which a newly classified transitioning Run can pin a
runtime until process restart, and PR4 does not create a second consumer.
Durable Harness Run admission has one consumer. A scheduler fire, manual task
run, callback, or watch completion commits the queued Run and wakes requests;
ScheduledTaskService._run_task no longer claims, spawns, or awaits the same
Run on a second path. Before enqueue, a scheduler fire computes the existing
one-successor rule in the enqueue transaction from same-definition
scheduler-sourced Runs and their Delivery/Turn facts. An accepted active
execution does not suppress its first follower. Any later Run or Delivery that
has not yet begun its own accepted Turn is the one queued successor and
suppresses further fires. When that successor starts or terminalizes, the next
fire may create one new follower. The existing Run/Delivery/Turn rows are the
evidence; no scheduler fence row or second owner is added.
Callers that require a synchronous result observe the persisted Run outcome
rather than taking execution ownership from the lane.
Likewise, every completed Activity output batch, whether produced during a live
Turn or found after restart, is persisted and wakes activity_outputs. A wake
does not make a newly unbound completion immediately claimable. Preserve
Claude's existing ACTIVITY_OUTPUT_FLUSH_GRACE_SECONDS coalescing window: the
first unbound completion establishes the eligibility deadline from its
persisted completed_at, and later matching completions inside that window do
not extend it. A restart recomputes the remaining window from durable
timestamps. An already bound batch or local-settlement-only retry is complete
evidence and is eligible immediately. No new durable deadline or batch owner is
added.
Once due, both the context-sensitive live flush and restart recovery enter the
same supervisor partition keyed by exact (backend, runtime_key) before
claiming. The live path may use its existing in-memory Turn/context while it
owns that partition. Its existing expected-Turn selection may scan past
unrelated unbound output to choose the causally matching batch; the supervisor
does not replace that #1139 rule with a global row-order policy. The partition
lease spans claim, emit, and settle/requeue, so no second live or recovery
worker can claim or send another batch for that runtime while the selected
batch owns the boundary. After release, the registry again chooses by its
existing causal/FIFO contract. Neither the wake nor the grace timer carries
payload, membership, or receipt authority.
Each lane owns one coordinator event/task plus a bounded map of item workers by partition key. The coordinator alone queries eligibility and never awaits item execution. It starts work only when that lane has capacity and no worker already owns the same partition. The existing guarded durable claim remains the item authority. A wake while the coordinator or an item is running sets the event for another bounded scan; it never starts a second worker for the same partition. A bounded page that leaves eligible work re-arms itself. A transient failure records bounded backoff for that item/partition only. One lane cannot await or cancel another lane.
Eligibility pagination cannot repeatedly stop on rows whose partition is already owned. A lane query either returns one actionable head per partition or uses stable keyset advancement while excluding the coordinator's occupied and backoff keys. One scan continues past full blocked pages until it fills current capacity or reaches the stable end cursor; only then may a later wake restart from the beginning. Durable FIFO within each partition is unchanged.
Partition keys are not invented tenant labels; they are the existing FIFO or
serialization owner key. They are Session id for Delivery activation; request
execution-lock key (or standalone Run id) for Run admission; callback target
Session for Run/Vault callbacks; and exact (backend, runtime_key) output queue
for recovered Activities. Ordinary suppressible failure notices with a
definition use ("failure_streak", definition_id) as their conservative
partition because the existing policy derives one canonical consecutive-
failure streak from that definition's Run history. Suppression-bypass notices
(interruptions and binding changes) and definitionless failures retain their
existing durable failure_id/Run identity. A destination is routing, never a
notice serialization key. Definition-level serialization may conservatively
span two already-closed streaks; it preserves correctness without adding a
streak field or moving streak derivation out of failure_streak_decision().
One selected Activity batch is indivisible and owns that runtime partition
through settlement or requeue. Selection order remains the existing #1139
registry contract, including expected-Turn causal matching; the supervisor
forbids overlap but does not replace that policy with a second queue.
task_definitions, watch_definitions, and stale_runs each retain one bounded
coordinator because their owner operations mutate shared local store/scheduler
state. In particular, every blocking ManagedWatchStore reload, cycle marker,
and runtime-state storage write stays behind the one watch_definitions
single-flight store boundary: its mutable full-store mirror is not safe for
concurrent watch-id writers. The store phase returns an immutable snapshot or
guarded write result. ManagedWatchService.reconcile_watches() and all
_active_tasks creation, cancellation, done-callback, and runtime projection
assembly remain on the controller event loop. Only the assembled projection's
blocking persistence enters the tracked store worker. Long-running waiter
processes/tasks remain outside that maintenance worker. This preserves one
store writer and one loop-owned lifecycle without creating a second mirror
owner.
That prepare/apply split is the general execution-placement rule. Blocking, side-effect-free reads and serialized store commits run in tracked synchronous workers. Asyncio task/timer mutation and in-memory lifecycle application run on their owning event loop after the worker returns. Existing domain claim and settlement APIs remain the semantic owners under the lane partition. Moving a blocking call off-loop never authorizes moving its caller's event-loop state machine into the executor. Every lane has a small internal maximum greater than one where item workers are supported; the limit is injectable in tests and is not a public product setting. One hung tenant therefore consumes one partition/slot rather than the whole lane, while the global bound prevents unbounded tasks. If all slots are occupied, remaining durable work waits and is re-read when one exits.
InboxEventBus.subscribe_callback runs on the publishing thread, which may not
be the controller event-loop thread. The supervisor captures its owning loop at
startup; callbacks only call loop.call_soon_threadsafe(...). The scheduled
loop callback is the sole mutator of coordinator events, tasks, partitions, and
generation state. A closed or superseded loop drops the hint and relies on
startup/periodic reconciliation.
The supervisor belongs to one controller process generation and is gated by the
same global service-instance ownership check as dispatch. Controller shutdown or
global lease loss stops and joins all lanes, including
session_deliveries, before that process releases shared state. A routine
ScheduledTaskService.stop() unregisters and joins only its Harness lane tokens;
it does not stop Session delivery recovery while the controller still owns the
instance. Every lane rechecks generation and global ownership before querying or
claiming, so a superseded controller cannot continue passive recovery.
Official producers use commit then notify:
- Commit the authoritative Delivery, Run, Vault, Activity, or definition change.
- Notify the affected lane(s).
- Treat notification failure as a logged liveness hint failure, not as rollback of the committed work.
Transport readiness is the one non-commit capability edge. After an IM
transport positively becomes ready, the controller directly wakes requests so
Runs previously skipped for transport unavailability do not wait for periodic
reconciliation. The lane still re-reads durable eligibility and the readiness
hint never selects or claims a Run.
Reuse core.inbox_events.bus as the in-process hint source and the existing
permission-restricted POST /internal/events Unix-socket bridge for
cross-process hints. Do not add a second endpoint or socket. The supervisor
subscribes through subscribe_callback and maps event types to lanes; event
payloads remain UI metadata and are never trusted as work state. runs.updated,
queue.updated, and vaults.updated already cover most producers; add one
allowlisted definitions.updated event for definition mutations. Activity
recovery can notify its lane directly because its producer is controller-owned.
The external-process bridge must not depend on that process having zero local browser/event subscribers. Browser presence is unrelated to whether the controller received the hint, and a successful socket POST does not prove the controller-to-UI event stream was connected at that instant. After commit, a non-controller UI publisher therefore fans out locally and also posts the hint to the controller with one ephemeral event id/origin. The UI bridge keeps a small bounded TTL set of ids it already fanned out locally and drops only the matching echoed event; other UI processes still receive it once. This id is transport deduplication metadata, not durable work identity or acceptance evidence. Socket absence, timeout, or a disconnected return stream leaves the command successful and the local UI responsive because the durable row remains authoritative.
The producer map is load-bearing:
| Observed edge | Wake lane(s) |
|---|---|
| Delivery becomes queued with open hold or terminal Turn exposes an open successor/head | session_deliveries |
| committed task or watch mutation after add/update/pause/resume/remove | task_definitions, watch_definitions via the shared event; each lane re-reads only its own store |
| queued Harness Run or callback-created Run | requests |
| terminal Run that owes a Session callback | run_callbacks |
| resolved Vault request that owes auto-resume | vault_callbacks |
| persisted recoverable Activity output or local-settlement debt | activity_outputs |
| terminal failure that owes a durable notice | failure_notices |
| IM transport becomes ready | direct requests wake after readiness |
| no event: vanished owner or time eligibility | stale_runs periodic eligibility |
The bus-to-lane map ignores event payloads:
| Event | Coalesced lanes |
|---|---|
queue.updated |
session_deliveries |
runs.updated |
requests, run_callbacks, failure_notices |
vaults.updated |
vault_callbacks |
definitions.updated |
task_definitions, watch_definitions |
| controller-local Activity recovery hint | activity_outputs |
| startup / 30-second reconcile | every registered lane, including stale_runs |
The implementation must inventory all official producer entry points and add a table-driven contract test that invokes each one against a real SQLite file. It must observe no hint before commit or after rollback/losing CAS, and the mapped hint after a successful commit. Storage functions themselves do not perform socket I/O and no notification is sent from inside a transaction.
3.5 Reconciliation is a safety net
The controller constructs the supervisor paused, registers available handlers, restores backend identities and durable Delivery/Turn owners, then activates the supervisor and wakes every registered lane before normal dispatch exposure. Hints received while paused may coalesce, but startup reconciliation is the load-bearing recovery. During steady state, one slow reconciliation timer wakes all lanes and lets each lane re-read its own indexed eligibility. After producer coverage is proven, the final default is 30 seconds, expressed as an internal constant with an injectable test clock rather than a public product setting. Normal latency comes from commit-triggered wakes, not this timer. The timer never executes storage or delivery work inline.
The injected legacy file stores are the only compatibility exception. They have
no transactional post-commit publisher, so their existing two-second file or
directory signature probes remain as temporary hint sources. A changed task
definition signature wakes task_definitions; a changed managed-watch
definition signature wakes watch_definitions; a changed request-directory
signature wakes both requests and run_callbacks, because that directory also
stores terminal Runs awaiting callback delivery. Each signature check is
separately tracked off the event loop; it performs no reload, claim, callback,
or settlement inline and cannot block another probe or lane. The compatibility
probes are not enabled for the production SQLite stores. This preserves legacy
test/tool responsiveness without making two-second polling the production
executor. Removing those file stores removes the probes with them.
PRAGMA data_version may avoid an unnecessary reload but cannot be the only
edge that keeps work alive: another consumer can observe the edge first, a
process can die between commit and notification, and time-based eligibility has
no write edge. Every lane must therefore be restart-recoverable from its durable
query alone.
Global lease loss follows the same stop path as controller cleanup. A later controller generation performs its own startup reconciliation; no lane or event subscription is transferred between generations.
3.6 Blocking work, supervision, and shutdown
No synchronous SQLite or filesystem call runs on the controller event loop unless a contention test proves it cannot block. Synchronous work runs through a tracked executor future. A wrapper timeout does not cancel that underlying operation or release that partition's single-flight ownership:
- the future remains the partition owner until it actually exits;
- the partition is marked overdue and does not launch a replacement;
- an independent supervisor logs the exact overdue lane and partition;
- completion re-arms the lane if durable work remains;
- shutdown stops new wakes, cancels only work that has not crossed an external- effect boundary, and joins each exact owner before disposing shared stores.
Every handler declares its cancellation boundary. Eligibility reads and work before a native/transport write may be canceled and recovered from durable state. Once an Activity output, callback, notice, or other non-idempotent send may have reached its destination, that worker is shielded from graceful-shutdown cancellation and retains its lane partition, receipt owner, dependencies, and service-instance lock until it persists the exact outcome or exits with a definitive pre-write result. A shutdown grace expiry taints the process as below; it is not permission to abandon the send and let startup replay it. Forced process death remains outside graceful exactly-once guarantees and is the current-master reproducer gate for any separate PR4B attempt contract.
SQLite operations must retain the repository's bounded busy timeout. A rare compound failure that outlives the controller's shutdown grace is logged as critical, retains its store/executor dependencies, and refuses same-process service restart until the exact future exits. Process shutdown may then own termination; the service must not dispose shared state underneath the thread or launch a new generation beside it. This design does not add worker processes, killable thread pools, or a second recovery ledger solely to mask that case.
The current SQLite busy timeout is five seconds, equal to the controller's generic five-second coroutine-stop wait. PR4 must not reuse that equal deadline for a supervised lane: the lane gets a separate shutdown grace longer than the database busy bound, and a grace expiry marks the controller generation tainted.
The process signal handler is never the cleanup executor. It only records the
shutdown intent and schedules controller.request_shutdown(...) onto the
captured loop with a loop-safe call; it does not call cleanup_sync(), wait on a
future, raise SystemExit, or release the service-instance lock while that loop
is running on the same thread. The loop-owned async shutdown path closes new
dispatch, stops and joins the supervisor and registered services, and records
whether the lock is safe to release before stopping the loop. The
Controller.run() finally path performs only any remaining synchronous
cleanup after the loop has stopped. A signal received before loop activation is
latched and consumed as soon as the loop starts, or handled by the same stopped-
loop cleanup path if startup aborts.
On a tainted process shutdown, cleanup skips disposal of the exact
supervisor/store/executor dependency set, refuses any in-process restart, and
does not explicitly release the service-instance lock. The operating system
releases that lock only when the process, including the old worker, actually
exits; a replacement process therefore cannot overlap it. main.py releases
the lock only after reading the loop-owned safe-to-release result. This is
bounded failure containment, not a retry or permission to overlap work.
3.7 No new durable owner
The wake/supervision layer adds no table and no item lifecycle. It must not:
- copy Delivery content or FIFO order into a work lane;
- copy Activity output payload, membership, receipt, or settlement into a lane;
- let a wake imply native acceptance;
- let a watchdog write Turn terminal state;
- infer retry safety from a missing in-memory task.
The lane always re-reads and invokes the existing guarded owner API. #1139's Activity batch remains the only owner for recovered output and #1140's terminal CAS remains the only owner for teardown settlement.
4. Required pre-code baseline
Before each remaining PR, write or run a current-master reproducer. Classify the
old defect as reproduces, fixed by #1134/#1139/#1140, or superseded by a new owner. Do not preserve an old prescription merely because the symptom still
has the same name.
Minimum baseline cases:
- open and held FIFO heads, every Delivery fence/Turn-owned role, every live Turn state, terminal history, and fallback Run ownership each reach both passes of runtime reclamation and produce the §3.2 disposition;
- each backend-specific idle eviction path is classified as requiring the shared owner interlock or as restart-safe for queued work;
- a complete #1139 Activity output batch with multiple Activities and linked Runs is restored in its persisted order; accepted-Message evidence and the persisted local-settlement-only marker each prevent transport replay while local settlement is retried;
- recovered Activity output delivery hangs while a new Harness request becomes ready;
- scheduled and watch Runs transfer to a durable Delivery/Turn and later produce success, failure, and result-less terminal outcomes;
- definition health is observed before and after the owning Run becomes terminal;
- scheduler shutdown/restart distinguishes unstarted, running, naturally terminal, and user-stopped work.
5. PR3 — Session runtime ownership and activation
Merged as #1155 (2026-08-04). Retained as the accepted contract: PR7R and any later implementation unit must consume the runtime ownership snapshot, activation interlock, and work-supervisor interfaces defined here rather than reintroducing side maps or adapter-local reclamation rules.
Goal
Make runtime reclamation and durable work activation consume one coherent session disposition. Active or transitioning work must not lose its runtime, and ownerless runnable work must be woken without keeping a backend process alive forever.
Required behavior
- Implement the §3.2 ownership provider as one SQLite read transaction over
agent_sessions,message_deliveries,session_turns, and fallbackagent_runs, plus Activity rows in the existingruntime_recordsaggregate. Explicitly execute SQLiteBEGINbefore the first SELECT and hold that transaction through the final ownership query; entering SQLAlchemy'sengine.begin()context without an emittedBEGINis insufficient. Reuse the existing engine and storage tables; add no table, cache, or background resolver. Resolve a backend runtime target to every durable Session it can serve, then classify those Sessions in the same transaction. Also classify active Activity rows with a NULL Session as target-level owners, using the backend-provided resource-key to Activity-key mapping rather than assuming those keys are equal. A returned snapshot is valid only for that transaction and exact runtime resource target. Map Session-less fallback Runs from their persisted legacy Session key plus backend/agent execution identity to the exact resource target; an unresolved nonterminal route fails closed for that backend cycle. - Derive
active,transitioning,runnable,reclaimable, orunknownexactly as §3.2 specifies. In particular:- an open, ownerless FIFO head is
runnable, not a permanent runtime pin; - a fence or Turn owner forbids reclamation;
- terminal history and
watch_runtimedo not pin; - an execution-bearing Run is considered only when its exact Delivery/Turn
representation is absent from the same snapshot: queued is
runnable, normalized running/legacy processing without the persisted PID boundary istransitioning, and the same state with a persisted PID isactiveuntil exact settlement.
- an open, ownerless FIFO head is
- Inventory Claude, Codex, and OpenCode runtime-reclamation paths. Every path
that can invalidate the exact runtime needed by a live owner consumes the
shared provider. A backend transport cache that is proven restart-safe may
reclaim under
waiting, but requires a consuming test; do not clone provider rules into adapters. A shared Codex cwd transport may be reclaimed only when every associated Session isrunnable|waiting|reclaimable, after all runnable members have been woken and the locked recheck still finds no live owner. Claude composite keys must be mapped by the stable Session row ID plus the adapter's exact runtime keys, not by a mutable admission anchor or string-prefix guesses. Inventory every path that can change a target fromrunnable|waiting|reclaimabletotransitioning|active, including backend receivers and standalone fallback Run claim/execution start. It must enter the exact resource-target/generation activation boundary before the owning commit and revalidate the observed generation. Direct registry or Run-store transitions that can race cleanup are forbidden. - Add the controller-owned
RuntimeWorkSupervisorcontract and registerSessionTurnManageras the solesession_deliverieshandler. Consult the provider in both passes ofevict_idle_sessions. On a runnable Delivery, notify that lane but do not turn durable unclaimed work into an unbounded runtime pin; a fallback queued Run not yet represented by Delivery notifiesrequests. Recompute under the exact target/generation runtime activation boundary immediately before cleanup. A newly committed fallback Run claim or execution start, Turn, active Activity, or replaced generation wins; a still-runnable row survives runtime reclamation. The reclaimer keeps the boundary through generation detach/cleanup initiation so no delayed old-generation admission can become a new durable pin after the final snapshot. The lane itself uses separate bounded indexed queries for claimable queued Sessions without live Turns and for unresolved Delivery/Turn fences, starting owners, and waiting successors. It invokes a manager entry guarded by the observed Session, Delivery, Turn, attempt, and native identity; stale observations no-op. The exact linkedwaitingTurn /interrupt_waitinginitial-Delivery pair istransitioningeven though the Delivery matrix calls itturn_owned; the pair emits its recovery wake even while its active predecessor keeps the aggregate runtime active. A mismatched half fails closed. Recovery also revalidates the predecessor Turn's exactcontrol_successor_*linkage and immutable terminal winner; it may activate the successor only through the existing terminal-resume writer. A wake before predecessor terminalization cannot call the low-level successor CAS and leaves the pair waiting. Therequestslane applies the equivalent exact recovery to claimed pre-execution fallback Runs. Neither lane hides a full globalrecover_durable_delivery_state()call inside cleanup; that global pass remains startup-only after backend restoration. PR3 registers this fallback-onlyrequestsrecovery handler together withsession_deliveries; it neither scans nor admits ordinary queued Runs. PR4 atomically replaces the registration with the full admission handler. - Use two failure modes:
- a lookup that positively proves one binding is dangling fails open for only that binding, so a deleted target cannot pin an unrelated session forever;
- any exception while resolving a binding, or any provider-wide failure, fails closed for the eviction cycle, because missing safety data is not evidence that eviction is safe.
- Bound exact active/transitioning ownership with the existing real-progress inactivity clock and stuck-active threshold. Merely observing a newly admitted pin does not restart that clock, but the authoritative exact Turn start establishes a fresh baseline for every source, including scheduled, watch, and agent-initiated work admitted after a long queue wait. Beyond the bound, use the #1140 teardown path to settle exact running ownership and preserve queued/unstarted work.
- Inventory every
session_last_activitywriter. Do not touch it on claim, enqueue, gate wait, output wait, polling/protocol frames, wake, provider lookup, or unrelated Session progress. A real inbound message may establish a baseline; the exact Turn-start boundary must establish a fresh baseline before the new execution can be judged stuck. Subsequently attributable assistant/tool/active-Activity progress is the only refresh signal. Shared Codex transport activity cannot refresh another Session's progress clock. - Keep activation asynchronous. The reclaimer emits only the coalescing wake;
SessionTurnManagerremains the claimant and Turn creator. A wake must never select a different queue head, bypass hold, or mutate Delivery/Run state. Existing synchronous commands such as send-now keep their exact-head validation, hold release, claim, refusal, and response semantics; their events only trigger passive recovery after the guarded result is committed.
Required evidence
HFR-130: one SQLite snapshot classifies every Delivery/Turn/Activity/Run ownership combination without a torn handoff; two engines/connections and a deterministic barrier prove that an explicit read transaction prevents mixed commit generations;HFR-131: an open ownerless FIFO head wakessession_deliveries, remains durable if its idle runtime is reclaimed, and is later claimed through the existing exact-head transaction;HFR-132: a held FIFO backlog survives while its disposable backend runtime is reclaimed, then starts after the existing explicit hold release;HFR-133: unresolved fences and Deliveries owned by exactstarting|activeTurns forbid cleanup;HFR-134: an exact linkedwaitingTurn /interrupt_waitinginitial- Delivery pair istransitioningand emits one exact recovery wake even while its active predecessor makes the aggregateactive; a mismatched half isunknown, and terminal Turn/Delivery history alone is reclaimable. Targeted recovery re-proves the predecessor's exactcontrol_successor_*linkage and immutable terminal winner before the existing terminal-resume writer activates that successor once. A wake before the predecessor terminal CAS is a no-op, while a stale CAS cannot start it twice or claim another head;HFR-135: active Activity rows pin their exact runtime, whileawaiting_outputand terminal Activity snapshots do not;HFR-147: an active Activity with no Session id maps from its durable Activity runtime key to the exact backend resource target and cannot be mistaken for an orphan; a missing map fails closed;HFR-136: awatch_runtimeheartbeat sharing the same definition/session does not pin, while an execution-bearing watch Run does;HFR-137: every transition across the runtime activation boundary wins or loses atomically against reclamation. Deterministic barriers cover both interlock orderings for Turn start, Session-bound and Session-less Activity admission,claim_pending_run(), andmark_run_execution_started(). Admission-before-cleanup is visible in the locked snapshot. Cleanup-before- admission prevents an old-generation Activity or execution-started Run from becoming active, while preserving the queued/pre-execution Run for its existing exact recovery path;HFR-138: bare-Run to reserved-Delivery and Delivery to Turn ownership handoffs cannot disappear across a torn provider read;HFR-148: fallback queued, pre-execution claimed, and execution-started Runs classify asrunnable,transitioning, andactiverespectively from the persisted PID boundary;HFR-149: periodic reconciliation finds a persisted Delivery/Turn fence, starting owner, waiting successor, or claimed pre-execution fallback Run after its process disappears and invokes only its exact guarded recovery entry. The active-predecessor/waiting-successor case still emits one coalesced recovery wake before aggregate disposition is applied. Deterministic barriers prove a wake after successor creation but before predecessor terminal proof cannot activate it. A stale observation neither pins the runtime forever nor recovers another item; the fallback Run recovery is live in PR3 before PR4 broadens the lane;HFR-150: Session-less IM/CLI fallback Runs map from persisted route and backend identity to the exact runtime target; archived Session-bound Runs remain visible through the resource's durable Session membership even after an admission anchor is reused. Queued, pre-execution claimed, and PID-started rows deriverunnable,transitioning, andactive, while an unmappable nonterminal row fails closed instead of disappearing;HFR-139: unrelated sessions are not pinned;HFR-140: one positively missing binding fails open, while a per-binding lookup exception or provider failure aborts the cycle;HFR-141: repeated queued followers do not refresh progress or make a held Session immortal; teardown settles only exact running ownership;HFR-142: a claimed or gate-waiting request does not refresh activity, while the exact Turn start after an arbitrarily long queue or gate wait establishes one fresh baseline before reclamation may judge the execution stuck;HFR-143: scheduled, watch, and agent-initiated Turns admitted into a long-idle Session are not immediately evicted after exact Turn start, and a long Turn with attributable observable progress remains live while unrelated shared-transport activity does not refresh it;HFR-144: the stuck threshold settles only the exact active owner through #1140 and never replays it;HFR-145: every enabled backend eviction path either honors the provider or proves queued work resumes without loss or replay;HFR-146: one active Session prevents reclamation of a shared Codex cwd transport while an unrelated held Session on that transport does not create fake activity; Claude composite-key mapping uses the stable Session row ID plus its exact adapter runtime keys.
Exit criterion: open work is activated, held work remains durable without pinning resources, no productive or transitioning owner is reclaimed, and the provider cannot make a stuck session immortal.
6. PR4 — Event-first supervised work lanes
Merged as #1173 (2026-08-05). Retained as the accepted contract. The conditional transport-attempt delta below remains unimplemented and is the only part of this section still open.
Goal
One hung or unbounded request, Run-callback, vault-callback, or post-turn-output pass must not block another Harness tenant. Normal work begins from a post-commit wake; startup and slow reconciliation repair lost hints. This is the direct fix for the observed 65-minute serial-loop stall. #1139 remains the sole Activity-output owner.
Required behavior
Extend the controller-owned §3.4
RuntimeWorkSupervisor; do not add a second scheduler insideScheduledTaskService. Replace bothScheduledTaskService._watch_storeandManagedWatchService._watch_storewith lane registration plus the shared slow reconciliation timer; neither loop may await a drain or perform storage I/O.Give
task_definitions,watch_definitions,requests,run_callbacks,vault_callbacks,activity_outputs,failure_notices, andstale_runsindependent coordinator state and the §3.4 bounded partition workers where item work can block. Reuse current drain functions behind the lane boundary where their ownership is already correct; split only unbounded passes into bounded pages and item calls. Do not create seven new service classes merely to rename seven methods.Implement one shared coalescing notifier. An event means "query durable eligibility again" and carries no item id. A running lane consumes the event before its next page so a concurrent commit cannot be lost between the final query and task exit.
Subscribe the supervisor to the existing inbox event bus and reuse the
/internal/eventssocket bridge. Add onlydefinitions.updatedto the allowlist and shared event vocabulary. Map events to coalescing lane wakes; ignore payload identity and status when deciding durable work. External UI publishers perform local fanout and bridge the hint with an ephemeral event id; the return bridge deduplicates only the origin process's echoed id. Local subscribers therefore neither suppress the controller wake nor miss an update while the return stream reconnects.Instrument every producer in the §3.4 map after its authoritative commit. Cross-process notification is best effort and short-lived. In-process notification marshals event setting onto the captured controller loop with
call_soon_threadsafe. A notifier exception is logged but never changes the producer's committed result. Preservenotify_transport_ready()as the direct non-commitrequestswake after an IM transport reconnects. The transport registry remains the sole readiness owner and adds one synchronous final-decision guard per configured platform. The current global_ready_lockcontinues to protect only brief reads/mutations of the_ready_platformsset; it is never held across a database call. The keyed guards serialize only one platform's readiness transitions and stale settlement, so a slow Slack settlement cannot delay Discord readiness. They are bounded in-memory capability locks, not work owners.The same readiness owner records an in-memory generation and injected UTC
unready_sincefor the current outage. Registering or hot-adding a configured platform that is not ready starts a fresh generation before its child thread or the stale lane can observe it; a ready transition closes it. Every later ready-to-unready edge starts another generation under the exact platform guard, whether that edge comes from explicit disconnect, the client runner'sfinally, or hot removal. Repeated observations while the platform is already unready do not refresh the generation or timestamp. A new process conservatively starts a new generation for each configured but not-ready platform, granting another bounded reconnect window rather than failing a Run from unverifiable pre-restart history. This metadata describes the current capability edge only. It is not a durable queue, Run fact, or settlement owner, and no transition bulk rewrites queued Runs.Candidate discovery happens outside the platform guard. For a queued row carrying
transport_unavailableevidence, the worker first reserves the SQLite write transaction withBEGIN IMMEDIATEor the repository's equivalent, then enters the exact platform guard, briefly reads readiness plus the current outage generation under_ready_lock, releases that brief set lock, and re-reads the Run evidence. If the platform is ready, the worker uses the same exact queued/reason/timestamp CAS to clear the ended outage's stale evidence and returns without terminalizing. If it is unready, the effective outage start is the later of the persistedlast_skip_atand the current generation'sunready_since; settlement is ineligible until that effective start exceeds the reconnect TTL. The worker then commits only an exactstatus=queuedCAS with the observed reason/timestamp while still holding the platform guard. Both ready and ready-to-unready transitions take that same platform guard, mutate the readiness set and generation briefly under_ready_lock, release both, and only then publish any wake; neither transition performs database work while holding either lock. A reconnect that wins first makes terminal settlement no-op and permits exact evidence cleanup; a later disconnect rebases the effective age even when capacity kept the old row stamp from being cleared; a settlement that wins first commits before readiness publication; a request claim that already changed the row torunningmakes the stale CAS lose. This is a final-decision interlock, not a durable readiness state or a second Run owner. A scheduler fire or manual task run only enqueues and notifies; remove_run_task's direct claim/spawn/await path sorequestsis the single durable Run admission consumer. Preserve the existing one queued follower behind an active scheduled Run with a transactional same-definition query across Run, Delivery, and Turn facts: active accepted work allows one successor, while an unstarted successor suppresses later fires. Route both live and restart-discovered completed Activity batch claims through the sameactivity_outputspartition lease; the live path keeps #1139's context-sensitive expected-Turn selection inside that lease, including scanning past unrelated unbound output. The lease remains owned through emit and settle/requeue. No backend-local flush may race the recovery lane as an independent claimant. Preserve the current live coalescing grace before claiming an unbound batch. Compute its due time from the first matching persistedcompleted_at, do not extend it for later members, and preserve the remaining delay across restart. Bound batches and local-settlement-only retries bypass the grace because their immutable membership already exists.Run every lane as bounded work. Request, Run-callback, and Vault-callback queries use explicit page limits. A page that reaches its limit or reports another eligible item re-arms the same lane. A temporarily skipped item uses existing durable eligibility/backoff when present; do not hot-spin on an in-memory event.
Move synchronous
maybe_reload, eligibility, stale-run, cycle-marker, and runtime-state persistence work off the controller event loop.watch_definitionsis the single store worker for managed-watch stale-worker recovery retry, reload, guarded per-watch writes, and persistence of an already assembled runtime projection. It returns immutable snapshots/results to the controller loop. Keepreconcile_watches(),_active_tasks, task creation/cancellation, done callbacks, timer state, and projection assembly on that loop; never callasyncio.create_task()or mutate those maps from an executor thread. The long-running waiter process/task remains under the existing managed-watch lifecycle and is never subjected to a maintenance- lane timeout. Track each underlying store future as its lane owner through timeout/cancellation as §3.6 requires. Never start a replacement store call until that exact future exits.Keep one independent supervisor task that reports the exact overdue lane, partition, and service generation. It observes worker timestamps only and cannot query the stores or write terminal state. Repeated service start/stop creates one new generation and leaves no task from the previous generation able to re-arm work.
Startup wakes every lane after backend identity restoration and Delivery/Turn recovery. The periodic reconcile timer later wakes every lane regardless of
data_version; time eligibility and lost notifications therefore recover. Preserve the existing two-second cadence initially as a compatibility safety net while the producer-coverage test is red/green. Once every official producer is proven, set the final default to 30 seconds in the same PR and prove post-commit wake latency separately from fallback latency. Retain the two-second signature probes only for explicitly injected legacy file stores; task and managed-watch definition signatures waketask_definitionsandwatch_definitionsrespectively, while a request-directory signature wakesrequestsandrun_callbacks. None may execute work inline.A process signal only requests loop-owned asynchronous shutdown. That path disables new dispatch and wakes, cancels only workers still before their declared external-effect boundary, and awaits every worker that may have written until it persists accepted/refused/pre-write evidence. It also awaits each exact synchronous future before tearing down stores, executors, or scheduler state. If the outer shutdown grace expires, keep those workers and dependencies alive and refuse same-process restart; never convert quarantine into permission to dispose or overlap. A completion callback from an old generation may log but cannot re-arm the service. Global controller lease loss stops every lane; routine scheduled-service or managed-watch-service stop unregisters only the Harness tokens it owns. The signal handler never blocks the running loop or releases the process lock;
main.pydoes so only after loop shutdown reports it safe.Bound maintenance work, not Agent execution. No lane timeout is a Turn timeout, transport retry verdict, or permission to settle a Run.
Preserve #1139's persisted Activity batch as the sole owner of output payload, order, Run union, stable receipt, accepted-Message evidence, and local-settlement-only debt. PR4 changes how that owner is scheduled, not its data model or transition writer.
Conditional transport-attempt delta
The old PR4 also proposed new durable transport-attempt state. It is not part of the event-first implementation by default. First run the current-master crash matrix against #1139. If the accepted Message, stable Activity receipt, and local-settlement-only marker already close the reproduced window, record that evidence and add no state.
Only a remaining concrete ambiguity may open a separate PR4B contract review.
That review must name the missing fact, prove the existing Activity owner cannot
express it, and keep any attempt metadata inside the existing Activity aggregate
under SessionActivityRegistry. It may not add an output ledger, copy payload or
membership, create a second receipt, or infer resend safety from timeout alone.
Required evidence
HFR-155: an in-process post-commit wake from either the controller loop or a worker/receiver thread is marshaled onto that loop and starts the exact eligible lane without waiting for periodic reconciliation;HFR-156: a cross-process commit followed by the Unix-socket hint starts the lane, while socket failure leaves the committed work recoverable;HFR-157: process death or a deliberately dropped hint is recovered by startup or slow reconciliation exactly once at the guarded owner boundary;HFR-158: a hung request admission, callback delivery, vault callback, or recovered-output send for one partition does not delay another partition in the same lane, the other lanes, or stale-run sweeps; more than one full page of followers for the occupied partition cannot hide a later actionable partition. A blocked ordinary failure notice for definition A does not delay definition B, while two ordinary failures of A targeting different Sessions never overlap and the later decision observes the first canonical result;HFR-159: a contended task/watch reload probe or stale-run store operation does not block the event loop, suppress independent drain progress, or serialize unrelated partitions;HFR-176: injected legacy file stores retain separately supervised two-second task, watch, and request signature hints without running work inline; a request-directory change wakes both request and callback lanes, while production SQLite uses post-commit wakes plus the slow reconcile timer;HFR-160: a timed-out synchronous worker remains the sole partition owner until its real future exits; no overlapping retry starts for that partition, and shutdown cannot dispose its store or restart the service before it joins;HFR-161: large request, Run-callback, and vault-callback backlogs drain in bounded pages and reliably re-arm; scheduler, manual, callback, and watch producers only enqueue and notify, onerequestsclaim starts each Run, and synchronous callers observe persisted outcomes without becoming a second execution owner; one active accepted scheduled Run permits exactly one queued same-definition successor, and that successor suppresses later fires until it starts or terminalizes;HFR-162: a wake during the final empty query is not lost; timeout, pre-effect cancellation, and exception completion each re-arm remaining eligible work with backoff, while shutdown cancellation does not re-arm;HFR-163: each lane enforces its configured global worker bound and only one item worker per partition while allowing a different partition to progress. Ordinary notices for one definition share its conservative failure-streak partition; interruptions, binding changes, and definitionless failures retain their existing per-failure identities and do not collapse into that streak;HFR-164: routine scheduled-task or managed-watch service stop/restart unregisters and joins only its owned Harness lane registrations and watchdog state; the controller-owned supervisor andsession_deliverieslane remain active;HFR-165: repeated scheduled-service start/stop cycles leave exactly one current Harness registration generation and no stale overdue-drain logs;HFR-166: complete multi-Activity batches preserve persisted order, one stable receipt per batch, and each complete Run union while delayed, restarted, or locally retried. Two same-Turn unbound completions inside the injected live grace produce one batch, receipt, and user-visible message; the first wake cannot claim early, a second completion does not extend the deadline, and a restart midway waits only the durable remaining grace. Bound and local-only retry batches are immediately eligible. One selected batch for exact(backend, runtime_key)owns the partition through emit and settle/requeue so another batch cannot overlap it; after release, the registry preserves #1139's expected-Turn causal selection, including scanning past unrelated unbound output, and loss of live context falls back to persisted Activity/Turn/Run settlement without a competing claimant;HFR-167: an accepted Message or persisted Activity-batch local-settlement-only marker suppresses transport replay, including after restart; graceful shutdown after remote acceptance but before local receipt persistence cannot cancel the owning send and must await its exact settlement;HFR-168: incomplete or conflicting recovered batch membership fails closed before transport and remains under the same Activity owner;HFR-169: work with no Run row still reaches an Activity-owned terminal outcome visible from its session;HFR-170: every skip re-arms with backoff;HFR-171: the independent supervisor reports which owned lane and partition are overdue;HFR-172: the table-driven producer contract fails when an official entry point lacks the mapped post-commit event; it observes no event before commit, after rollback, or after a losing CAS, and a worker-thread publisher touches asyncio state only through the controller loop;HFR-173: unsupported event types remain rejected by the existing socket endpoint; browser subscribers cannot suppress a controller wake; and local fanout plus bounded event-id deduplication delivers once both while the return stream is connected and while it is reconnecting;HFR-174: task and managed-watch definition changes wake their separate reconcile lanes without waiting for fallback; the genericdefinitions.updatedevent safely coalesces both;HFR-175: global service-instance lock loss stops every lane registration, includingsession_deliveries, before the old process can query or claim, and no old-generation completion can re-arm work;HFR-177: IM transport readiness directly wakesrequestsand a skipped Run resumes without waiting for slow reconciliation. Deterministic barriers cover reconnect winning the final readiness guard (no failure notice; one request claim), stale settlement winning and committing before readiness publication (one terminal failure; zero claim), and a queued-to-running request claim making the stale exact CAS lose. They also keep an expired stamp queued when the platform reconnects and drops again before any worker clears that stamp: the second outage receives its own full TTL from the injected readiness clock, then permits exactly one terminal settlement after that TTL. They prove that exact evidence cleanup may win before the second disconnect, that repeated disconnect observations do not extend one outage, and that every disconnect producer uses the same edge owner. Process startup gives a configured but not-ready platform the same fresh bounded window. A blocked Slack final-decision guard does not block Discord readiness mutation, outage-generation changes, or itsrequestswake;HFR-178: a real signal while the controller loop is running requests nonblocking async shutdown; a synchronous lane outliving shutdown grace retains the service-instance lock and exact dependencies until its worker or process exits, so a replacement process cannot overlap it; the same is true for an async worker past its external-effect boundary, and a signal before loop startup follows the same ownership decision;HFR-179: contended managed-watch store reconciliation and per-watch guarded store writes stay single-flight, keep the SQLite row and full-store mirror consistent, and cannot wrongly restart a completed one-shot watch. Blocking store phases run in the tracked worker, whilereconcile_watches(),asyncio.create_task(),_active_tasks, cancellation, callbacks, and runtime projection assembly are asserted to run on the controller loop. Store contention does not block that loop or another lane, while a legitimately long-running waiter is not canceled by the maintenance timeout and remains owned by the managed-watch lifecycle.
Exit criterion: normal work is event-woken, missed hints are recovered, a hung lane cannot block another lane or the controller event loop, lifecycle teardown leaves no old-generation worker, and Activity/Delivery/Turn/Run ownership is unchanged.
7. PR7 — Evidence gate before any new timeout model
Open. This is the next unit of work. PR3 and PR4 removed the serial-drain and reclamation causes; whether any terminal-truth or scheduler-liveness defect survives them is now an open question that only evidence may answer.
Do not implement the old PR7 prescription. #1134 added
complete_on_return, durable Delivery ownership, immutable Turn terminal
evidence, and restart recovery; #1139 added exact Activity output batch receipts,
Run-union settlement, anti-redelivery evidence, and transport-free local
settlement retry; #1140 closed teardown-interrupted Run settlement. The previous
plan guessed at the remaining timeout model before proving which gaps still
exist. PR7 starts with evidence, not a schema or writer.
PR7R — Current-master evidence
PR7R changes tests and the plan only. It must publish one executable matrix for:
- Claude, Codex, and OpenCode;
- direct IM and durable Workbench execution lanes;
- scheduler cron fires, one-shot
atfires, manual CLI/API task runs, and watch Runs; - success, failure, result-less termination, user Stop, terminal persistence failure, pending output delivery, and post-delivery local settlement failure.
For every cell, trace the exact durable facts from Run enqueue through request / Delivery reservation, Turn start, terminal-result latch, Turn terminal evidence, Activity output batch, accepted Message receipt, Run settlement, and definition health projection. The matrix must answer these questions with a consuming test, not prose inference:
- Does the Run remain nonterminal until its actual terminal Turn/result or Activity output batch settles it? If yes, close the old premature-success claim with regression evidence instead of adding another writer.
- Which observable assistant/tool events can be attributed to the exact Turn and participating Runs? Prove this separately for every backend and both lanes. A backend/lane without an exact signal blocks a generic inactivity timeout; session-wide activity is never an acceptable substitute.
- Can scheduler and manual Runs with different source semantics or effective deadlines enter the same Turn? Record the current merge key and every Run in the batch. Cancellation is Turn-level, so no per-Run policy may be specified until this cardinality is explicit.
- Which evidence exists before the Turn becomes terminal? A terminal-result latch, durable pending-output fact, accepted Message, or Activity local-settlement-only marker proves that natural completion has started and must outrank a later inactivity decision.
- Are
health,consecutive_failures,recent_failures,last_run_at, andlast_erroralready monotonic projections of bounded terminal Run history? Dispatch or Delivery acceptance is never task success.
Exit criterion: the checked-in matrix and tests identify each remaining defect and its current owner. PR7R adds no status, timeout field, terminal writer, health cursor, or cancellation path.
After PR7R — close the claim or review the contract
PR7R does not authorize implementation. If its executable evidence disproves an old claim, close that claim without adding code. For every reproduced defect, amend this plan in a separate documentation-only review before opening an implementation PR. Do not reserve PR7A / PR7B as implementation units until that contract review passes.
The amendment must define one complete model, not another list of local patches:
- Name the durable owner and guarded terminal transition at every phase, including a bare Run before Delivery reservation, each nonterminal Delivery role, a starting or active Turn, terminal-result and pending-output evidence, Activity receipt, and post-delivery local settlement. It must define request/coalescing cleanup and the durable user notice for every terminal cause.
- State whether exact-Turn progress must survive restart. If it must, define the persisted owner, timestamp/update CAS, recovery read, and restart-after-progress evidence. If no backend-independent durable signal exists for every backend and lane, do not implement a generic inactivity timeout.
- Define one policy owner for coalesced Runs. Either prove all participants have the same snapshotted policy or keep incompatible Runs in separate Turns. Specify natural completion, user Stop, and timeout precedence before adding a timeout writer.
- If the model includes a user-configurable task inactivity override, specify its V2 config key, concrete default, normalization, persistence, CLI/API/Web inputs, English and Chinese documentation, and unchanged watch semantics. If those supported surfaces are not part of the implementation, remove the override from the model rather than supporting it only through fixtures.
- Reuse the exact Run, Delivery, Turn, Message, and Activity owners already on
master. Any new state, schema, terminal writer, retry, or replay rule needs evidence that those owners cannot express the reproduced behavior.
Terminal-truth and scheduler-liveness defects remain separate implementation reviews even when PR7R reproduces both. Each implementation PR starts with red tests for the approved contract and must not infer missing policy from this evidence plan.
8. Order and review boundaries
complete: #1063, #1064, #1072, #1134, #1139, #1140
|
v
complete: PR3 #1155 runtime ownership + `session_deliveries` supervisor foundation
|
v
complete: PR4 #1173 event-first supervised work lanes
|
v
NEXT: PR7R current-master evidence matrix (evidence-only)
|
v
contract amendment for each reproduced defect (documentation-only)
|
v
separate terminal-truth / scheduler-liveness implementation PRs
PR7R is a separate test/documentation review unit. It may close either old claim without an implementation PR. A reproduced defect first receives the complete contract amendment above; only then may it become a separate implementation unit. PR3 and PR4 stayed separate: PR3 defined session ownership/reclamation; PR4 replaced serial polling as the normal executor. A PR4B transport-attempt change exists only if the current-master reproducer proves a missing durable fact and its separate contract review passes.
Implementation ownership is intentionally narrow:
- PR3 owned the derived SQLite snapshot, the controller-owned supervisor
foundation,
session_deliveriesregistration, the fallback-only pre-executionrequestsrecovery registration, reclaimer consumers, and the exact activation/reclamation tests. It did not admit general queued Harness Runs or restructure the other drains. - PR4 registered Harness lanes, replaced the scheduled-task and managed-watch
_watch_storeloops, extended the existing Unix-socket event vocabulary and bridge use, instrumented post-commit producers, and added lane lifecycle and isolation tests. It did not change eviction disposition or durable owner schemas. - PR7R remains evidence-only.
Each lane owns its files and tests; the orchestrator alone resolves shared controller/supervisor edits and verifies one consuming test per contract before push.
Scenario range status, verified against
tests/scenarios/harness_failure_recovery/catalog.yaml on 2026-08-06:
- PR3:
HFR-130…154— occupied by #1155 - PR4:
HFR-155…179— occupied by #1173 - PR7:
HFR-180…219— reserved and still unoccupied; usable by PR7R
Check the catalog again immediately before coding. The highest merged ID is
now HFR-435, allocated by unrelated capabilities above this plan's reserved
blocks. If HFR-180…219 has been taken by then, allocate a fresh contiguous
block above the highest merged ID; never reuse a closed branch's overflow
table.
9. Validation and non-goals
For every PR:
- add unit and contract tests first against the real owner boundary;
- add/update
tests/scenarios/harness_failure_recovery/catalog.yaml; - name the assigned HFR scenario ID in every affected automated test and list all affected scenario IDs in the implementation PR description;
- assert both the durable state and the user-visible notice where failure is expected;
- use guarded writers and test the losing race, not only the happy winner;
- keep tests hermetic; do not write real
~/.avibestate; - route new backend and frontend display copy through the matching English and Chinese i18n catalogs;
- run Ruff on changed Python files;
- for any UI change, run
cd ui && npm run buildand verify the packagedui/distcontains the intended frontend change; - use the local Incus regression runner for cross-platform verification;
- never restart the local
vibeservice for routine validation.
Non-goals:
- a new Run status value such as
interrupted; - a second Activity output ledger, receipt identity, or settlement owner beside #1139's persisted Activity batch;
- backend-level exactly-once execution;
- an absolute Agent turn-duration timeout;
- widening this work into unrelated session/process lifecycle cleanup;
- preserving old implementation details that #1134, #1139, or #1140 superseded.