Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(pi): make takeover archive-safe and recoverable (#5321)

t0saki419655094266
diff --git a/examples/pi-coding-agent-extension/DESIGN.md b/examples/pi-coding-agent-extension/DESIGN.md
index a88b9378f..bb00b7518 100644
--- a/examples/pi-coding-agent-extension/DESIGN.md
+++ b/examples/pi-coding-agent-extension/DESIGN.md
@@ -80,15 +80,15 @@ The OV session id is `pi-<pi session id>`, derived locally by the shared `derive
**Capture.** `turn_end` hands the whole branch to `extractBranchCapturePayloads` (`lib/capture-adapter.mjs`), which takes the entries past `syncedEntryCount`, normalizes roles, renders tool parts with bounded input and output, and decides entry by entry whether to capture. There are two decision modes. Normally it is the shared `shouldCaptureText` heuristic. Under takeover it is a faithful mode that drops only empty text, slash commands and OpenViking's own status messages: once the boundary advances, a short acknowledgement may be represented to the model *only* through the archive overview, so discarding it as low-signal would lose it outright. A branch shorter than the watermark means pi navigated to a different branch, so the watermark resets to zero and the branch is re-extracted from the start.
-**Delivery.** Payloads leave in one batched request through the shared `sendSessionMessages`. Retryable failures are written to the shared disk pending queue and replayed later. A non-retryable rejection counts as accepted, so the watermark advances past payloads the server will never take rather than re-sending them every turn. The watermark only moves when the whole extraction was accepted.
+**Delivery.** Payloads leave in one batched request through the shared `sendSessionMessages`. Retryable failures are written to the shared disk pending queue and replayed later. The result distinguishes delivered, queued and permanently lost payloads. A non-retryable rejection or failed enqueue advances the capture watermark to avoid duplicate replay, but also records a persistent capture gap; takeover never trims across that gap. The watermark only moves when the whole extraction has either been delivered, queued or recorded as permanently lost.
-**Backlog drain.** `flushForTakeover` is the barrier takeover waits on, and it needs the queue empty for this session. The shared `replayPending` sends one request per entry and stops after a single replay window, which is what let a large offline backlog hold the barrier closed for many turns, so sync drains this session's own queued `addMessage` entries through the batch endpoint, `BATCH_LIMIT` per request, claiming one batch at a time so a failed batch costs only the retries of the entries it held. The drain is bounded by wall time (`OPENVIKING_PENDING_DRAIN_BUDGET_MS`, 60s by default) and optionally by batch count, so a huge backlog cannot block `turn_end` indefinitely; the remainder drains on later turns.
+**Backlog drain.** `flushForTakeover` is the barrier takeover waits on, and it needs the queue empty for this session. Startup restores takeover state before replay, then routes the current session through the tracked Pi drainer rather than the generic replay path. Sync drains queued `addMessage` entries through the batch endpoint, `BATCH_LIMIT` per request, and also treats active `.processing` files as undelivered. The drain is bounded by wall time (`OPENVIKING_PENDING_DRAIN_BUDGET_MS`, 10s by default, narrowed further to whatever takeover's handler budget leaves) and optionally by batch count, so a huge backlog cannot block `turn_end` indefinitely; the remainder drains on later turns. Once this session's queue is empty, startup hands other sessions' entries to the generic replay; while this session still has a backlog they wait for a later start, because the generic path drops rejected entries without recording whose gap they are. Other sessions never affect this session's barrier. The same generic replay runs in every OpenViking harness plugin on the machine over the same queue, so a gap is guaranteed to be recorded only for losses this session's own drainer sees.
**Commit.** Outside takeover, sync asks the server for `pending_tokens` after each accepted turn and commits when it crosses `commitTokenThreshold` — server-side accounting, not a local estimate. A failed commit is queued for replay unless the caller passes `queueOnFailure: false`, which takeover always does, because a commit that lands later cannot justify a boundary that moved now.
### takeover.ts
-A binding, not a mechanism. The state machine lives in `lib/takeover-core.mjs`, which is pure and unit-tested; `takeover.ts` supplies its I/O: flush and commit go to `SyncManager`, the archive overview comes from the session context endpoint, state is persisted through `pi.appendEntry`, and the watermark is read back from sync.
+A binding, not a mechanism. The state machine lives in `lib/takeover-core.mjs`, which is pure and unit-tested; `takeover.ts` supplies its I/O: branch sync, flush and commit go to `SyncManager`, each archive overview is read directly from `<archive_uri>/.overview.md`, state is persisted through `pi.appendEntry`, and the watermark is read back from sync.
Context takeover makes OpenViking the authoritative long-term store for a pi session. Pi still keeps recent turns locally; committed history is represented to the model by OpenViking's archive overview through pi's `context` hook.
@@ -96,12 +96,15 @@ Context takeover makes OpenViking the authoritative long-term store for a pi ses
| Field | Meaning |
|---|---|
-| `coveredUserTurns` | Real user turns already covered by the archive overview |
-| `overview` | Latest archive overview returned by the session context endpoint |
-| `fingerprint` | Fingerprint of the last covered message, for branch-mismatch detection |
+| `coveredThroughEntryId` | The boundary: id of the pi entry the covered prefix ends at, inclusive |
+| `coveredUserTurns` | User turns the boundary covers in the active context (display only) |
+| `overview` | Overview of the archive the boundary advanced behind, read from `<archive_uri>/.overview.md` |
| `pendingTokens` | Estimated synced token pressure since the last successful advance |
-| `lastSeenUserTurns` | User turns counted in the most recent `context` hook |
+| `lastSeenUserTurns` | User turns counted in the most recent `context` hook (display only) |
| `syncedEntryCount` | Pi branch watermark, restored across `pi -p` / `pi -c` processes |
+| `archiveUri` / `historyUri` | Exact archive and parent history directory used by the recovery hint |
+| `pendingArchive` | Accepted archive whose own overview is not ready yet, including its frozen boundary |
+| `captureGap` | Persistent proof that at least one captured message can no longer reach the server |
State is persisted as a pi custom entry:
@@ -109,23 +112,36 @@ State is persisted as a pi custom entry:
pi.appendEntry("ov-takeover", state)
```
-At startup the extension scans the branch from the end for the newest such entry and restores both the boundary and `SyncManager`'s watermark, so a `pi -c` continuation does not resend branch entries OpenViking already has.
+It is written on state transitions — an advance, a pending archive, a capture gap — and at shutdown, not on every turn, since each entry carries the overview. At startup the extension scans the branch from the end for the newest such entry and restores both the boundary and `SyncManager`'s watermark, so a `pi -c` continuation does not resend branch entries OpenViking already has. An entry written by 0.4.1 or earlier holds its boundary as a user-turn count plus an optional fingerprint instead; the first `context` hook locates it the way those versions did and converts it to the entry id in front of the first kept turn, or drops it when it no longer matches.
+
+#### Boundary identity
+
+The boundary is an entry id because pi shows takeover two views of the same session that do not line up. `getBranch()` holds every entry on the path: the `ov-takeover` state entries takeover itself appends between turns, model and thinking-level changes, and after a pi compaction the whole compacted-away prefix. The `context` hook's messages hold none of those. A user-turn count or a "last covered message" fingerprint taken on one view lands somewhere else on the other — the state entry in front of every user turn alone makes a branch-side fingerprint disagree with the context on the next request. An entry id names the same entry in both.
+
+Takeover therefore works on pi's context projection of the branch (`projectContextEntries`: after the latest compaction only its kept entries and what follows, minus entries a context edit removed — pi's own `buildContextEntries`, which pi < 0.86 does not expose). The boundary is the entry in front of the oldest kept user turn. In the `context` hook, the first user entry after it is matched onto the hook's messages by its timestamp, which content rewrites leave alone, and the cut goes there. Whatever sits between the boundary and that turn — what a run appended after a `takeoverKeepRecentTurns` 0 commit, or a branch summary `/tree` left at the boundary — is covered by no archive and stays: the cut steps back over those messages role by role, and when the roles disagree with the hook's messages nothing is trimmed. When the boundary entry is not in the active context — `/tree` moved to another branch, or pi compacted past it — the full context is sent and the boundary is kept, so returning to the branch applies it again.
+
+#### Handler budget
+
+Every takeover step runs inside a pi event handler, and pi hosts cap those at 30s: omp logs `handler timed out after 30000ms`, discards the result and lets the handler run on (#5275). Each handler therefore gives takeover a deadline 25s after it started (`HANDLER_BUDGET_MS`), sync included. Nothing sleeps waiting for a summary in `turn_end` or `before_agent_start`; the drain gets the time the commit does not need, the commit gets the time one overview read does not need, and with less than 10s left the commit waits for a later turn. Only the compaction handler polls, because pi needs its summary now, and it stops at the deadline.
#### Runtime flow
1. `turn_end` captures new branch entries into the OpenViking session, falling back to the disk pending queue when the server is unreachable.
2. When `pendingTokens` reaches `takeoverTokenThreshold`, and there are more user turns than `takeoverKeepRecentTurns`, takeover tries to advance.
-3. Advancing requires the flush barrier: every `addMessage` entry queued for *this* session must be delivered. Queued `commitSession` entries and entries belonging to other sessions do not hold it closed.
-4. The commit runs with `queueOnFailure: false`.
-5. The session context endpoint is polled until `latest_archive_overview` is available — `takeoverOverviewPollMax` attempts, `takeoverOverviewPollMs` apart. An empty overview is never injected; the boundary stays where it is and the token pressure resets so the next threshold crossing retries instead of re-committing every turn.
-6. On success the boundary advances to `lastSeenUserTurns - takeoverKeepRecentTurns`.
-7. The `context` hook then replaces every covered message with one synthetic user message beginning `[OpenViking Session Context]`, keeps the recent tail verbatim, and recall is injected into the newest kept user turn as usual.
+3. On one branch snapshot, takeover freezes the candidate boundary entry, token pressure and the exact number of capture payloads the branch holds after the boundary. A candidate no later than the current boundary is not worth an archive.
+4. It syncs the latest branch and drains this session's queue. A transient queued failure may proceed after the drain succeeds; a permanent rejection, enqueue failure or retry exhaustion persists `captureGap` and blocks takeover for this session.
+5. The commit runs with `queueOnFailure: false` and the retained payload count as `keep_recent_count`. A skipped commit, `archived: false` or missing `archive_uri` cannot advance the boundary.
+6. The extension reads `<archive_uri>/.overview.md` directly, once. It never substitutes an older session-context overview. A summary that is not there yet — phase 2 usually needs longer than a handler may wait — persists the pending archive and frozen boundary; every later `turn_end` and the next `before_agent_start` read that archive once, without committing again. When the archive carries the server's terminal marker — `.done`, written last once phase 2 completed (with Working Memory disabled it completes without a summary), or `.failed.json` — and a last read still finds no summary, the pending archive is dropped and its frozen token pressure spent, so the next archive waits for fresh pressure. The markers are the server's own archive state and, unlike task records, do not expire. A pending archive whose boundary left the active context is dropped without a read.
+7. Once the overview is non-empty, takeover confirms the frozen boundary entry is still in the active context. Only then does it advance; token pressure accumulated while waiting remains for the next archive.
+8. The `context` hook then replaces the covered *conversation* with one synthetic user message beginning `[OpenViking Session Context]`, keeps every covered `system` message in front of it in original order, keeps the recent tail verbatim, and recall is injected into the newest kept user turn as usual. With `takeoverKeepRecentTurns` 0 the boundary is the tip at commit time and applies from the next user turn; conversation the run added after that commit is never trimmed, because no archive covers it.
The overview message's timestamp is derived from the first kept message, so the provider payload stays byte-stable between commits and can benefit from prompt caching.
+On pi ≥ 0.86 the transcript carries the base prompt and its tool declarations as the leading `system` message, and mid-conversation tool additions/removals, section updates and appended instructions as later `system` messages (`@earendil-works/pi-ai`'s `getCurrentTools` / `getCurrentSystemMessage`). Slicing those off with the covered turns would strip the model's tools and instructions, so they are preserved: the overview stands in for the conversation, never for the system state. A `system` message inside the retained tail is left where it is rather than hoisted. On 0.80.3 there are no `system` messages in the branch, so this preserves nothing and the behaviour is unchanged; on 0.87 the host reconciles the declared tools against the executable set on every request, so keeping the existing declarations introduces no duplicate.
+
#### Compaction
-When pi emits `session_before_compact`, takeover runs the same flush → commit → overview sequence. On success it hands pi the overview as the compaction summary and resets the boundary, because pi's own compaction has absorbed it:
+When pi emits `session_before_compact`, takeover first syncs the latest branch, fully archives the captured history with `keep_recent_count: 0`, and polls that archive's overview — `takeoverOverviewPollMax` reads, `takeoverOverviewPollMs` apart — until the handler deadline. On success it hands Pi the overview while preserving Pi's `firstKeptEntryId`; summary coverage may therefore overlap the retained tail. If sync, commit, overview generation, the deadline or cancellation prevents a proven result, the handler returns `undefined` and Pi performs its native compaction. That path changes no takeover state beyond remembering the archive: pi may still cancel or fail its own compaction, and the boundary has to keep applying to the uncompacted context. If pi does compact, the entry-id boundary either falls outside the new context or still covers only what its overview covers.
```ts
{
@@ -146,14 +162,27 @@ If any step fails the handler returns nothing and pi's default compaction runs.
|---|---|
| Health check fails | Extension stays disconnected; pi runs normally |
| Pending `addMessage` replay incomplete | Barrier stays closed, boundary is not advanced, full local history remains visible |
-| Commit fails | Boundary is not advanced; pending token pressure is retained |
-| Overview not ready | Boundary is not advanced; retried at the next threshold or by `/viking commit` |
-| Branch fingerprint mismatch | Boundary resets to 0 and full history is shown until the next successful advance |
+| Permanent delivery failure | Persist `captureGap`; takeover stays disabled for this session and Pi owns compaction |
+| Commit fails or is skipped | Boundary is not advanced; pending token pressure is retained |
+| Overview not ready | Persist this archive and frozen boundary; later turns and the next prompt read it once each, without another commit |
+| Archive terminal (`.done` / `.failed.json`) without an overview | Drop the pending archive and spend its frozen token pressure; the boundary stays where it was |
+| Handler budget nearly spent | Postpone the commit to a later turn |
+| Frozen boundary left the active context while waiting | Discard the pending boundary and leave local history with Pi |
+| Boundary not in the active context (other branch, pi compaction) | Send the full context; keep the boundary for a return to that branch |
| Compaction takeover fails | Returns nothing; pi's default compaction proceeds |
+Successful ordinary and compaction summaries share a recovery footer when the
+required MCP tools are active. It names the exact `archive_uri`, explains that
+the source contains captured historical messages rather than an unfiltered Pi
+transcript, and directs the model to `openviking_list` plus paginated
+`openviking_read`. The footer is appended after overview truncation so the
+summary budget cannot remove the recovery entry point.
+
#### Live gate
-`scripts/e2e-live.sh` drives a real pi binary, a real OpenViking server and a real LLM endpoint (its required and optional environment variables are documented at the top of `scripts/e2e-live.mjs`). It runs three `pi -p` / `pi -c` turns with a tiny takeover threshold and asserts that the third provider payload carries `[OpenViking Session Context]` while the padding from the first turn is gone from the raw conversation history. Nothing in the unit suites covers that end to end, so it stays a manual gate.
+`scripts/e2e-live.sh` drives a real pi binary, a real OpenViking server and a real LLM endpoint (its required and optional environment variables are documented at the top of `scripts/e2e-live.mjs`). It runs three `pi -p` / `pi -c` turns with a tiny takeover threshold and asserts that the third provider payload carries `[OpenViking Session Context]` while the padding from the first turn is gone from the raw conversation history. The third turn also calls a built-in tool, lists the archive with `openviking_list`, and reads `messages.jsonl` with paginated `openviking_read` arguments before recovering the archived detail. Nothing in the unit suites covers that end to end, so it stays a manual gate.
+
+Between the two, `tests/takeover-session-manager.test.mjs` drives the core against pi's real `SessionManager` — takeover's own state entries between turns, a pi compaction, `/tree` away from and back to the covered branch — and checks which prompts actually go out trimmed. It resolves pi from a local or global install (`PI_SESSION_MANAGER` overrides) and skips when none is found.
### MCP tools
diff --git a/examples/pi-coding-agent-extension/README.md b/examples/pi-coding-agent-extension/README.md
index fca956ac1..e40ac5ccc 100644
--- a/examples/pi-coding-agent-extension/README.md
+++ b/examples/pi-coding-agent-extension/README.md
@@ -185,10 +185,25 @@ integrations should configure category `quotas` when they need exact ceilings.
### Context takeover
-Takeover is enabled by default. OpenViking commits archived history, polls the
-session overview, then the `context` hook replaces covered conversation turns
-with a synthetic `[OpenViking Session Context]` user message while keeping the
-recent live tail.
+Takeover is enabled by default. OpenViking commits archived history, reads the
+overview of that exact archive, then the `context` hook replaces covered
+conversation turns with a synthetic `[OpenViking Session Context]` user message
+while keeping the recent live tail. The boundary advances only after every
+captured message in the cut has reached the server and the new archive has a
+non-empty overview. Permanent capture gaps leave the local history with Pi.
+
+Takeover never waits for the summary inside a turn: pi hosts cap each extension
+event handler at 30s, so a summary that is not ready yet is checked once per
+later turn and before the next prompt. An archive the server marks finished
+or failed without a summary (for example with Working Memory disabled) is
+dropped instead of being checked forever.
+
+When `openviking_list` and `openviking_read` are active, the injected overview
+also includes the archive URI and a short recovery instruction. `openviking_list`
+locates the archive and `openviking_read` reads `messages.jsonl` in `offset` /
+`limit` chunks. This file contains captured historical messages; semantic search
+does not return the archive source verbatim, and capture may have sanitized,
+filtered or truncated the original Pi transcript.
| Field | Default | Description |
|--------------------------|------------|--------------------------------------------------------------------------|
@@ -196,8 +211,8 @@ recent live tail.
| `takeoverTokenThreshold` | `30000` | Synced-token pressure that triggers commit and boundary advance |
| `takeoverKeepRecentTurns`| `3` | Recent user turns retained in full fidelity |
| `takeoverOverviewBudget` | `3000` | Token budget for the injected archive overview |
-| `takeoverOverviewPollMs` | `2000` | Delay between overview polling attempts after commit |
-| `takeoverOverviewPollMax`| `15` | Max overview polling attempts before fail-open |
+| `takeoverOverviewPollMs` | `2000` | Delay between overview reads while answering pi's compaction |
+| `takeoverOverviewPollMax`| `15` | Max overview reads for pi's compaction; capped by the 25s handler budget |
### Injection tuning
@@ -311,6 +326,14 @@ The canonical `/viking` command (type `/viking` in pi's chat) displays connectio
## Upgrading from 0.3.x
+0.4.2 makes context takeover conservative: it keeps Pi's system/tool state,
+anchors the trim boundary on a Pi entry id, waits for confirmed capture
+delivery, reads the overview belonging to the exact new archive without
+blocking a turn on it, and exposes a paginated archive-recovery hint when the
+required MCP tools are active. Sessions with a permanent capture gap stay on
+Pi's native compaction. A boundary saved by 0.4.1 or earlier is converted on
+the first request.
+
0.4.0 replaces the seven hand-written `viking_*` REST tools with the server's own MCP tool surface. There is no alias period: the old names are gone.
- **Tool names.** Every tool is now `openviking_` + the server's own name: `viking_search` → `openviking_search`, `viking_read` → `openviking_read`, `viking_remember` → `openviking_remember`, `viking_forget` → `openviking_forget`, `viking_add_resource` → `openviking_add_resource`. `viking_browse` is covered by `openviking_list`, `openviking_tree` and `openviking_glob`; its stat mode and `viking_archive_expand` have no successor.
@@ -385,7 +408,7 @@ TypeScript is loaded directly by pi's jiti transpiler. The official MCP client i
| Tools not showing after `pi -c` resume | Known pi issue (tools not re-registered on resume) | Workaround built in — tools register in `before_agent_start`|
| Extension crashes on load | Wrong OV server URL or network issue | Check `logLevel` and server accessibility |
| No memories extracted | Wrong embedding/extraction model in OV config | Check OV's `embedding` / `vlm` configuration |
-| Takeover never advances | Pending addMessage replay, commit, or overview polling failed | Set `OPENVIKING_DEBUG_LOG=/tmp/ov-pi.log` and retry `/viking commit` |
+| Takeover never advances | Pending addMessage replay or commit failed, or the archive summary never arrives (Working Memory disabled on the server) | Set `OPENVIKING_DEBUG_LOG=/tmp/ov-pi.log` and retry `/viking commit` |
## License
diff --git a/examples/pi-coding-agent-extension/client.ts b/examples/pi-coding-agent-extension/client.ts
index 442ccd9bc..5c2c7572d 100644
--- a/examples/pi-coding-agent-extension/client.ts
+++ b/examples/pi-coding-agent-extension/client.ts
@@ -39,8 +39,18 @@ export interface OVSessionContext {
}
export interface OVCommitResult {
+ /**
+ * `accepted` = phase 1 wrote `history/archive_NNN/messages.jsonl` and a
+ * background task is generating the Working Memory; `skipped` = nothing to
+ * archive (see `reason`, e.g. `no_messages`). Older builds omit the field.
+ */
+ status?: "accepted" | "skipped" | string;
+ /** Whether phase 1 created an archive. */
+ archived?: boolean;
+ reason?: string;
task_id?: string;
- archive_uri?: string;
+ /** `null` on a `skipped` commit — the server sends the key either way. */
+ archive_uri?: string | null;
trace_id?: string;
}
@@ -112,11 +122,12 @@ export class OVClient {
async commitSessionResponse(
sessionId: string,
keepRecentCount = this.cfg.commitKeepRecentCount,
+ timeoutMs = 30000,
): Promise<OVCommitResponse> {
const res = await this.fetchJSON<OVCommitResult>(
`/api/v1/sessions/${encodeURIComponent(sessionId)}/commit`,
{ method: "POST", body: JSON.stringify({ keep_recent_count: keepRecentCount }) },
- { timeoutMs: 30000 },
+ { timeoutMs },
);
if (res.ok && res.result && !res.result.trace_id && res.traceId) {
res.result.trace_id = res.traceId;
@@ -128,4 +139,70 @@ export class OVClient {
status: res.status,
};
}
+
+ /**
+ * Working Memory of one archive: `GET /content/read?uri=<archive_uri>/.overview.md`.
+ *
+ * This reads the Working Memory that belongs to a *specific* commit, keyed by
+ * the `archive_uri` that commit returned — unlike `getSessionContext`, whose
+ * `latest_archive_overview` reflects whatever the session's newest archive is
+ * and can point at an older `/context` summary that this takeover did not
+ * produce. Returns null until commit phase 2 writes the sidecar (the server
+ * answers 404 until then). Other read failures throw so takeover can log a
+ * read failure separately from an unfinished archive. The sidecar is stored as
+ * OKF Markdown — `---\n<yaml>\n---\n\n<body>\n` — and `content/read` only
+ * strips that framing for memory URIs, not session archives, so the
+ * frontmatter is removed here. A file present but empty counts as not ready
+ * (null), so a poller cannot be fooled by an empty write. Takeover reads it
+ * inside pi event handlers, so one read is capped well below their budget.
+ */
+ async readArchiveOverview(archiveUri: string): Promise<string | null> {
+ const base = String(archiveUri ?? "").trim().replace(/\/+$/, "");
+ if (!base) return null;
+ const res = await this.fetchJSON<string>(
+ `/api/v1/content/read?uri=${encodeURIComponent(`${base}/.overview.md`)}`,
+ undefined, { timeoutMs: 5000 },
+ );
+ if (!res.ok) {
+ if (res.status === 404) return null;
+ const detail = res.error?.message || res.error?.code || `HTTP ${res.status ?? 0}`;
+ throw new Error(`archive overview read failed: ${detail}`);
+ }
+ if (typeof res.result !== "string") return null;
+ const body = stripFrontmatter(res.result);
+ return body.trim() ? body : null;
+ }
+
+ /**
+ * Terminal state of one archive, from the markers the server itself uses
+ * (`Session._archive_terminal_state`): `.done` once commit phase 2 completed
+ * — it is written last, after the Working Memory when that is enabled, and
+ * records `working_memory_enabled: false` when it is not — and `.failed.json`
+ * once phase 2 failed for good. "pending" while neither exists; null when the
+ * server could not be asked. Unlike task records, the markers do not expire.
+ */
+ async getArchiveState(archiveUri: string): Promise<"completed" | "failed" | "pending" | null> {
+ const base = String(archiveUri ?? "").trim().replace(/\/+$/, "");
+ if (!base) return null;
+ for (const [marker, state] of [[".done", "completed"], [".failed.json", "failed"]] as const) {
+ const res = await this.fetchJSON<string>(
+ `/api/v1/content/read?uri=${encodeURIComponent(`${base}/${marker}`)}`,
+ undefined, { timeoutMs: 5000 },
+ );
+ if (res.ok) return state;
+ if (res.status !== 404) return null;
+ }
+ return "pending";
+ }
+}
+
+/**
+ * Drop a leading `---` … `---` YAML frontmatter block, together with the blank
+ * line the server writes after it (its sidecars render as `---\n<yaml>\n---\n\n
+ * <body>\n`). Reused rule from the experimental fork's client.
+ */
+function stripFrontmatter(text: string): string {
+ const m = /^---[ \t]*\r?\n[\s\S]*?\r?\n---[ \t]*(?:\r?\n|$)/.exec(text);
+ if (!m) return text;
+ return text.slice(m[0].length).replace(/^(?:[ \t]*\r?\n)+/, "");
}
diff --git a/examples/pi-coding-agent-extension/index.ts b/examples/pi-coding-agent-extension/index.ts
index 95b305592..e6fb3e17c 100644
--- a/examples/pi-coding-agent-extension/index.ts
+++ b/examples/pi-coding-agent-extension/index.ts
@@ -25,6 +25,7 @@ import { guardVikingUriToolCall, noticeVikingUriToolResult } from "./lib/uri-gua
import { createMcpBridge, DEFAULT_HANDSHAKE_BUDGET_MS } from "./lib/mcp-bridge.mjs";
import { registerMcpTools } from "./tools.js";
import { createTakeoverManager } from "./takeover.js";
+import { HANDLER_BUDGET_MS } from "./lib/takeover-core.mjs";
/** This extension's directory, published for the experimental fork's probe. */
const EXTENSION_DIR = dirname(fileURLToPath(import.meta.url));
@@ -172,18 +173,27 @@ export default async function (pi: ExtensionAPI) {
}
return;
}
- await sync.replayPending();
-
- // Profile injection
- profileBlock = await buildSessionProfileBlock(client, config);
+ // Restore takeover state — sync watermark, pending archive and capture
+ // gap — before replaying any queued messages, so recovery and replay
+ // cannot disagree about what has been delivered (plan §4).
const branch = typeof ctx.sessionManager.getBranch === "function"
? ctx.sessionManager.getBranch()
: [];
if (config.takeoverEnabled) {
takeover.restore(branch);
sync.restoreWatermark(takeover.state.syncedEntryCount);
- } else if (sync.sessionId) {
+ }
+
+ await sync.replayPending();
+ if (config.takeoverEnabled && sync.droppedCount > 0) {
+ takeover.recordCaptureGap();
+ }
+
+ // Profile injection
+ profileBlock = await buildSessionProfileBlock(client, config);
+
+ if (!config.takeoverEnabled && sync.sessionId) {
// Resume rehydration — fetch archive overview if session was previously committed.
archiveOverview = await fetchArchiveOverview(client, sync.sessionId, config);
}
@@ -219,6 +229,7 @@ export default async function (pi: ExtensionAPI) {
// --- before_agent_start ---
pi.on("before_agent_start", async (event, ctx) => {
+ const deadline = Date.now() + HANDLER_BUDGET_MS;
// Read before awaiting, because this is what separates the retry below
// from the attempt start() makes: on the turn that runs the startup chain
// this is false, and from the next turn on it is true. Testing `started`
@@ -245,6 +256,12 @@ export default async function (pi: ExtensionAPI) {
if (!connected || bypassed || closed) return;
+ // A summary that finished since the last turn — or in the last `pi -p`
+ // process — trims this prompt's request already.
+ if (config.takeoverEnabled) {
+ await takeover.resumePending(() => ctx.sessionManager.getBranch(), { deadline });
+ }
+
// Queue recall for the context hook. Pi renders the user message before
// that hook, so recall latency does not delay the message appearing.
recall.queueSearch(event.prompt);
@@ -310,8 +327,13 @@ export default async function (pi: ExtensionAPI) {
}
}
+ // Takeover's boundary is a branch entry id; it maps it onto these messages
+ // through pi's context projection of the branch.
const afterTakeover = config.takeoverEnabled
- ? takeover.transformContext(event.messages as any)
+ ? takeover.transformContext(
+ event.messages as any,
+ typeof sessionManager?.getBranch === "function" ? sessionManager.getBranch() : [],
+ )
: event.messages;
const messages = recall.injectRecall(
afterTakeover,
@@ -338,23 +360,32 @@ export default async function (pi: ExtensionAPI) {
pi.on("turn_end", async (event, ctx) => {
if (!connected || bypassed || !isCaptureEnabled(config)) return;
+ // The host's 30s cap covers this whole handler, sync included.
+ const deadline = Date.now() + HANDLER_BUDGET_MS;
const branch = ctx.sessionManager.getBranch();
const result = await sync.syncBranch(branch);
logger.log("turn_end", { added: result.added, tokens: result.tokens });
- await takeover.onTurnSynced(result.tokens);
+ if (result.permanentFailures > 0) takeover.recordCaptureGap();
+ // The branch is what the boundary is frozen against and re-confirmed on, so
+ // takeover needs it, not just the token delta.
+ await takeover.onTurnSynced(result.tokens, () => ctx.sessionManager.getBranch(), { deadline });
updateStatus(ctx, connected, result.added, sync.sessionId, config, takeover.state, toolsReady);
});
// --- session_before_compact ---
- pi.on("session_before_compact", async (event, _ctx) => {
+ pi.on("session_before_compact", async (event, ctx) => {
if (!connected || bypassed) return;
if (config.takeoverEnabled) {
+ const deadline = Date.now() + HANDLER_BUDGET_MS;
const prep = (event as any)?.preparation ?? {};
+ // Native compaction syncs the latest branch, archives all captured
+ // history and reuses pi's own firstKeptEntryId, so hand it the branch.
return await takeover.handleBeforeCompact({
firstKeptEntryId: prep.firstKeptEntryId,
tokensBefore: prep.tokensBefore ?? 0,
- });
+ signal: (event as any)?.signal,
+ }, () => ctx.sessionManager.getBranch(), { deadline });
}
const archiveId = await sync.commit();
@@ -414,10 +445,17 @@ export default async function (pi: ExtensionAPI) {
if (args?.trim() === "commit") {
await sync.shutdown();
const commitResult = config.takeoverEnabled ? null : await sync.commit();
+ // Manual commit freezes and confirms the boundary against the current
+ // branch, exactly like the automatic path.
const ok = config.takeoverEnabled
- ? await takeover.commitAndAdvance()
+ ? await takeover.commitAndAdvance(() => ctx.sessionManager.getBranch())
: commitResult !== null;
- if (ok) {
+ if (!ok && config.takeoverEnabled && takeover.state.pendingArchive) {
+ ctx.ui.notify(
+ "OpenViking: committed; the context is trimmed once the archive summary is ready",
+ "info",
+ );
+ } else if (ok) {
ctx.ui.notify(
"OpenViking: committed successfully" +
(commitResult?.trace_id ? ` (trace_id=${commitResult.trace_id})` : ""),
diff --git a/examples/pi-coding-agent-extension/lib/takeover-core.d.mts b/examples/pi-coding-agent-extension/lib/takeover-core.d.mts
index 374f9eab3..28fdddc1a 100644
--- a/examples/pi-coding-agent-extension/lib/takeover-core.d.mts
+++ b/examples/pi-coding-agent-extension/lib/takeover-core.d.mts
@@ -8,13 +8,39 @@ export interface TakeoverMessage {
[key: string]: any;
}
+/**
+ * A commit that archived but whose Working Memory was not ready in time. While
+ * one is set, no second takeover commit runs — later turns re-check this same
+ * archive by its uri. Persisted so it survives `pi -c` / `pi -p`.
+ */
+export interface PendingArchive {
+ archiveUri: string;
+ taskId: string;
+ historyUri: string;
+ /** The entry the frozen covered prefix ends at; "" for a native-compaction archive. */
+ coveredThroughEntryId: string;
+ coveredUserTurns: number;
+ frozenTokens: number;
+ nativeCompaction?: boolean;
+}
+
export interface TakeoverPersistedState {
+ /** The pi entry id the covered prefix ends at (inclusive); "" when none. */
+ coveredThroughEntryId?: string;
+ /** Display count; also the whole boundary of a 0.4.1-or-earlier state. */
coveredUserTurns: number;
overview: string;
+ /** 0.4.1 and earlier only: fingerprint of the last covered context message. */
fingerprint?: string | null;
pendingTokens: number;
lastSeenUserTurns?: number;
syncedEntryCount?: number;
+ /** A committed archive still waiting on its Working Memory, or null. */
+ pendingArchive?: PendingArchive | null;
+ /** A permanent delivery gap: the archive is missing messages. Blocks advance. */
+ captureGap?: boolean;
+ archiveUri?: string;
+ historyUri?: string;
}
export interface TakeoverConfig {
@@ -26,41 +52,94 @@ export interface TakeoverConfig {
takeoverOverviewPollMax?: number;
}
+export interface SyncBranchResult {
+ added: number;
+ tokens: number;
+ allDelivered: boolean;
+ queued?: number;
+ permanentFailures?: number;
+}
+
export interface TakeoverIo {
- flush?: () => Promise<boolean> | boolean;
- commit?: (opts?: { queueOnFailure?: boolean; keepRecentCount?: number }) => Promise<unknown> | unknown;
- fetchOverview?: (tokenBudget?: number) => Promise<string | { latest_archive_overview?: string | null } | null> | string | { latest_archive_overview?: string | null } | null;
+ /** Deliver a pi branch to the server before committing. */
+ syncBranch?: (branch: any[]) => Promise<SyncBranchResult> | SyncBranchResult;
+ /** Drain this session's queue within `budgetMs`; true once nothing of it is undelivered. */
+ flush?: (budgetMs?: number) => Promise<boolean> | boolean;
+ commit?: (opts?: { queueOnFailure?: boolean; keepRecentCount?: number; timeoutMs?: number }) => Promise<unknown> | unknown;
+ /** Read one archive's `.overview.md` by its uri; null until it is ready. */
+ readArchiveOverview?: (archiveUri: string) => Promise<string | null> | string | null;
+ /** An archive's terminal state from its `.done` / `.failed.json` markers; null when unknown. */
+ archiveState?: (archiveUri: string) => Promise<"completed" | "failed" | "pending" | null> | string | null;
+ /** Exact server keep_recent_count for a retained tail (message count). */
+ captureCount?: (branchSlice: any[]) => number;
persistEntry?: (customType: string, data: TakeoverPersistedState) => void;
getWatermark?: () => number;
+ /** Messages OpenViking will never receive; a capture gap when > 0. */
+ droppedCount?: () => number;
+ availableTools?: () => string[];
sleep?: (ms: number) => Promise<void>;
+ /** Clock for handler deadlines; defaults to Date.now. */
+ now?: () => number;
log?: (message: string) => void;
}
+/** Budget for takeover work inside one pi event handler (hosts cap them at 30s). */
+export const HANDLER_BUDGET_MS: number;
+
+/** Absolute epoch-ms deadline for the handler a call runs in. */
+export interface HandlerDeadline {
+ deadline?: number;
+}
+
+export interface CommitOutcome {
+ accepted: boolean;
+ reason: string;
+ archiveUri?: string;
+ taskId?: string;
+}
+
export function flattenContent(msg: TakeoverMessage): string;
export function fingerprintMessage(msg: TakeoverMessage): string;
export function isUserTurnStart(msg: TakeoverMessage): boolean;
export function countUserTurns(messages: TakeoverMessage[]): number;
export function findBoundaryIndex(messages: TakeoverMessage[], coveredUserTurns: number): number;
+/** pi's context projection of `getBranch()`: compaction-aware, context edits applied. */
+export function projectContextEntries(branch: any[]): any[];
export function estimateTokens(text: string): number;
export function truncateToTokens(text: string, budget: number): string;
export function estimatePayloadTokens(payload: any): number;
-export function buildOverviewMessage(overview: string, firstKeptTs?: number, budget?: number): TakeoverMessage;
+export function buildOverviewMessage(overview: string, firstKeptTs?: number, budget?: number, recoveryHint?: string): TakeoverMessage;
export function countUndeliveredForSession(pendingEntries: any[], sid: string): number;
+export function deriveHistoryUri(archiveUri: string): string;
+export function commitOutcome(committed: unknown): CommitOutcome;
+
+export type TakeoverState = TakeoverPersistedState & {
+ coveredThroughEntryId: string;
+ lastSeenUserTurns: number;
+ syncedEntryCount: number;
+ committing: boolean;
+ pendingArchive: PendingArchive | null;
+ captureGap: boolean;
+ archiveUri: string;
+ historyUri: string;
+};
export class TakeoverCore {
constructor(opts?: { config?: TakeoverConfig; io?: TakeoverIo });
get enabled(): boolean;
- get state(): TakeoverPersistedState & {
- fingerprint: string | null;
- lastSeenUserTurns: number;
- syncedEntryCount: number;
- committing: boolean;
- };
- restore(entries: any[]): this["state"];
- transformContext(messages: TakeoverMessage[]): TakeoverMessage[];
- onTurnSynced(estTokens: number): Promise<boolean>;
- commitAndAdvance(): Promise<boolean>;
- handleBeforeCompact(preparation?: { firstKeptEntryId?: string; tokensBefore?: number }): Promise<
+ get state(): TakeoverState;
+ restore(entries: any[]): TakeoverState;
+ /** `branch` is pi's `getBranch()`; the boundary is located on its context projection. */
+ transformContext(messages: TakeoverMessage[], branch?: any[] | (() => any[])): TakeoverMessage[];
+ onTurnSynced(estTokens: number, branch?: any[] | (() => any[]), opts?: HandlerDeadline): Promise<boolean>;
+ /** One check of a pending archive, e.g. before a prompt; true when the boundary advanced. */
+ resumePending(branch?: any[] | (() => any[]), opts?: HandlerDeadline): Promise<boolean>;
+ commitAndAdvance(branch?: any[] | (() => any[]), opts?: HandlerDeadline): Promise<boolean>;
+ handleBeforeCompact(
+ preparation?: { firstKeptEntryId?: string; tokensBefore?: number; signal?: AbortSignal },
+ branch?: any[] | (() => any[]),
+ opts?: HandlerDeadline,
+ ): Promise<
| {
compaction: {
summary: string;
@@ -74,7 +153,10 @@ export class TakeoverCore {
shutdown(): Promise<void>;
resetBoundary(reason?: string): void;
truncatedOverview(): string;
+ recoveryHint(archiveUri?: string, historyUri?: string): string;
+ recordCaptureGap(): void;
persistedState(): TakeoverPersistedState;
persist(): void;
- pollOverview(): Promise<string>;
+ readOverviewOnce(archiveUri: string): Promise<string>;
+ pollArchiveOverview(archiveUri: string, signal?: AbortSignal, until?: number): Promise<string>;
}
diff --git a/examples/pi-coding-agent-extension/lib/takeover-core.mjs b/examples/pi-coding-agent-extension/lib/takeover-core.mjs
index accd419fd..4c9440827 100644
--- a/examples/pi-coding-agent-extension/lib/takeover-core.mjs
+++ b/examples/pi-coding-agent-extension/lib/takeover-core.mjs
@@ -10,6 +10,19 @@ const DEFAULT_CONFIG = {
takeoverOverviewPollMax: 15,
};
+// pi hosts cap every extension event handler at 30s; omp logs `handler timed
+// out after 30000ms`, discards the result and lets the handler run on (#5275).
+// Takeover's work in one handler stays inside this budget.
+export const HANDLER_BUDGET_MS = 25_000;
+// What a commit plus one overview read needs; with less left, the commit waits
+// for a later turn instead of running past the host's cap.
+const COMMIT_RESERVE_MS = 10_000;
+// One overview read (the client caps it at 5s) with room to spare.
+const READ_RESERVE_MS = 5_000;
+// Archive states after which its summary will not appear any more: the
+// server's `.done` / `.failed.json` markers (idea from #5320).
+const TERMINAL_ARCHIVE_STATES = new Set(["completed", "failed"]);
+
function numberOr(value, fallback) {
const next = Number(value);
return Number.isFinite(next) ? next : fallback;
@@ -31,6 +44,94 @@ function asEntry(value) {
return value.entry && typeof value.entry === "object" ? value.entry : value;
}
+function isUserEntry(entry) {
+ return entry?.type === "message" && isUserTurnStart(entry.message);
+}
+
+/** The role of the non-system message an entry puts into the context, or "". */
+function contextRole(entry) {
+ if (entry?.type === "message") {
+ const role = entry.message?.role;
+ return typeof role === "string" && role !== "system" ? role : "";
+ }
+ if (entry?.type === "custom_message") return "custom";
+ if (entry?.type === "branch_summary" && entry.summary) return "branchSummary";
+ return "";
+}
+
+/**
+ * The branch entries pi builds the model context from, in context order —
+ * pi's own `buildContextEntries` projection, reimplemented because pi < 0.86
+ * does not expose it. After the latest compaction only the compaction entry,
+ * the entries from its `firstKeptEntryId` (system messages excluded) and
+ * everything after it remain; entries a context edit removed are dropped.
+ * `getBranch()` itself still holds the compacted-away prefix and every custom
+ * entry, so turn positions must never be counted on it directly.
+ */
+export function projectContextEntries(branch) {
+ const path = (Array.isArray(branch) ? branch : []).filter((entry) => entry && typeof entry === "object");
+ let compactionIdx = -1;
+ for (let i = path.length - 1; i >= 0; i--) {
+ if (path[i].type === "compaction") {
+ compactionIdx = i;
+ break;
+ }
+ }
+ let entries = path;
+ if (compactionIdx >= 0) {
+ const compaction = path[compactionIdx];
+ const keptIdx = path.findIndex((entry, i) => i < compactionIdx && entry.id === compaction.firstKeptEntryId);
+ const kept = keptIdx >= 0
+ ? path.slice(keptIdx, compactionIdx).filter((entry) => !(entry.type === "message" && entry.message?.role === "system"))
+ : [];
+ entries = [compaction, ...kept, ...path.slice(compactionIdx + 1)];
+ }
+ const removed = new Set();
+ for (const entry of entries) {
+ if (entry.type === "context_edit" && entry.replacement === null && typeof entry.targetId === "string") {
+ removed.add(entry.targetId);
+ }
+ }
+ return removed.size ? entries.filter((entry) => !removed.has(entry.id)) : entries;
+}
+
+/**
+ * Position of a user entry's message among the `context` hook's messages.
+ * Matched on pi's per-message timestamp, which context edits and other
+ * extensions' content rewrites leave alone; the content fingerprint stands in
+ * only when a timestamp is missing. When several messages match, the one at
+ * the expected user-turn ordinal wins.
+ */
+function findUserMessage(messages, message, ordinal) {
+ const ts = message?.timestamp;
+ const byTime = typeof ts === "number" && Number.isFinite(ts);
+ const fp = byTime ? "" : fingerprintMessage(message);
+ let first = -1;
+ let seen = 0;
+ for (let i = 0; i < messages.length; i++) {
+ if (!isUserTurnStart(messages[i])) continue;
+ const match = byTime ? messages[i].timestamp === ts : fingerprintMessage(messages[i]) === fp;
+ if (match) {
+ if (seen === ordinal) return i;
+ if (first < 0) first = i;
+ }
+ seen++;
+ }
+ return first;
+}
+
+function currentBranch(branchOrGetter) {
+ if (typeof branchOrGetter === "function") {
+ try {
+ const value = branchOrGetter();
+ return Array.isArray(value) ? value : [];
+ } catch {
+ return [];
+ }
+ }
+ return Array.isArray(branchOrGetter) ? branchOrGetter : [];
+}
+
function flattenValue(value) {
if (typeof value === "string") return value;
if (Array.isArray(value)) return value.map(flattenValue).filter(Boolean).join("");
@@ -136,7 +237,12 @@ export function estimatePayloadTokens(payload) {
return 0;
}
-export function buildOverviewMessage(overview, firstKeptTs = 0, budget = DEFAULT_CONFIG.takeoverOverviewBudget) {
+export function buildOverviewMessage(
+ overview,
+ firstKeptTs = 0,
+ budget = DEFAULT_CONFIG.takeoverOverviewBudget,
+ recoveryHint = "",
+) {
const raw = String(overview || "");
const truncated = truncateToTokens(raw, budget);
const body = truncated === raw ? raw : `${truncated}\n...(truncated)`;
@@ -145,7 +251,7 @@ export function buildOverviewMessage(overview, firstKeptTs = 0, budget = DEFAULT
role: "user",
content:
`${OVERVIEW_MARKER} Earlier conversation was archived to OpenViking and summarized below. ` +
- `Use openviking_search for details.\n\n${body}`,
+ `Use OpenViking for related context when its tools are available.\n\n${body}${recoveryHint}`,
timestamp,
};
}
@@ -160,26 +266,110 @@ export function countUndeliveredForSession(pendingEntries, sid) {
return count;
}
+/** Derive the parent history directory only for the server's archive shape. */
+export function deriveHistoryUri(archiveUri) {
+ const cleaned = String(archiveUri || "").trim().replace(/\/+$/, "");
+ const m = /^(.*\/history)\/archive_\d+$/.exec(cleaned);
+ return m ? m[1] : "";
+}
+
+/**
+ * Read a commit response and decide whether it archived what this takeover
+ * meant to trim.
+ *
+ * The boundary may only advance behind a commit that actually wrote an archive:
+ * `status: "skipped"` (nothing to archive), an explicit `archived: false`, or a
+ * missing `archive_uri` all mean there is nothing to point the overview at, and
+ * the old `/context` summary must never be reused to fill the gap. Older server
+ * builds omit `status`/`archived` but still return `archive_uri`; those are
+ * accepted on the URI alone.
+ */
+export function commitOutcome(committed) {
+ if (!committed || typeof committed !== "object") {
+ return { accepted: false, reason: "no_result" };
+ }
+ const status = typeof committed.status === "string" ? committed.status : "";
+ if (status === "skipped") {
+ return { accepted: false, reason: `skipped:${committed.reason || "unknown"}` };
+ }
+ if (status && status !== "accepted") {
+ return { accepted: false, reason: `status:${status}` };
+ }
+ if (committed.archived === false) {
+ return { accepted: false, reason: "not_archived" };
+ }
+ const archiveUri =
+ typeof committed.archive_uri === "string" ? committed.archive_uri.trim() : "";
+ if (!archiveUri) {
+ return { accepted: false, reason: "no_archive_uri" };
+ }
+ return {
+ accepted: true,
+ reason: "accepted",
+ archiveUri,
+ taskId: typeof committed.task_id === "string" ? committed.task_id : "",
+ };
+}
+
export class TakeoverCore {
constructor({ config = {}, io = {} } = {}) {
this.config = takeoverConfig(config);
this.io = {
flush: io.flush || (async () => true),
+ // Sync the current branch before committing so the archive covers exactly
+ // the messages this trim is about to drop; not every caller has synced.
+ syncBranch: io.syncBranch || (async () => ({ added: 0, tokens: 0, allDelivered: true })),
commit: io.commit || (async () => null),
- fetchOverview: io.fetchOverview || (async () => ""),
+ // Read the Working Memory of a SPECIFIC archive by its uri, so the summary
+ // provably belongs to this commit and not to an older /context archive.
+ readArchiveOverview: io.readArchiveOverview || (async () => null),
+ // An archive's terminal state from its server markers: "completed",
+ // "failed", "pending", or null when it cannot be asked.
+ archiveState: io.archiveState || (async () => null),
+ // How many messages the capture path would actually send for a slice of
+ // the branch — the server keep_recent_count is a message count, not a
+ // user-turn count, and it must exclude system/custom/filtered entries.
+ captureCount: io.captureCount || (() => 0),
persistEntry: io.persistEntry || (() => {}),
getWatermark: io.getWatermark || (() => 0),
+ // Messages OpenViking will never receive (4xx / retries exhausted): a
+ // capture gap that must stop takeover cutting across it.
+ droppedCount: io.droppedCount || (() => 0),
+ availableTools: io.availableTools || (() => []),
sleep: io.sleep || ((ms) => new Promise((resolve) => setTimeout(resolve, ms))),
+ now: io.now || (() => Date.now()),
log: io.log || (() => {}),
};
+ // The covered prefix ends at this pi entry, inclusive. Entry ids survive
+ // restarts, compaction and branch navigation and identify one entry in
+ // every view of the session, so the boundary needs no turn count or content
+ // fingerprint — the same reason the recall ledger keys on them.
+ this.coveredThroughEntryId = "";
+ // Display only: user turns the boundary covers in the active context.
this.coveredUserTurns = 0;
this.overview = "";
- this.fingerprint = null;
this.pendingTokens = 0;
this.lastSeenUserTurns = 0;
this.syncedEntryCount = 0;
this.committing = false;
this.lastPersisted = "";
+ this.archiveUri = "";
+ this.historyUri = "";
+ // A commit that archived but whose Working Memory was not ready in time:
+ // { archiveUri, taskId, historyUri, coveredThroughEntryId, coveredUserTurns,
+ // frozenTokens, nativeCompaction? }. While set, no second takeover commit
+ // runs — later turns only re-check this same archive.
+ this.pendingArchive = null;
+ // A permanent delivery gap in this session: the archive is missing messages,
+ // so the boundary must never advance past it. Survives restart.
+ this.captureGap = false;
+ // A count-based boundary (0.4.1 and earlier) — the first N user turns of the context, plus the
+ // fingerprint of the last covered message once it was learned — converted
+ // to an entry id the first time the context hook sees the session.
+ this.legacyBoundary = null;
+ // Whether the last context actually carried the boundary, so leaving and
+ // re-entering the covered branch is logged once, not on every request.
+ this.boundaryApplied = false;
}
get enabled() {
@@ -188,13 +378,17 @@ export class TakeoverCore {
get state() {
return {
+ coveredThroughEntryId: this.coveredThroughEntryId,
coveredUserTurns: this.coveredUserTurns,
overview: this.overview,
- fingerprint: this.fingerprint,
pendingTokens: this.pendingTokens,
lastSeenUserTurns: this.lastSeenUserTurns,
syncedEntryCount: this.syncedEntryCount,
committing: this.committing,
+ pendingArchive: this.pendingArchive,
+ captureGap: this.captureGap,
+ archiveUri: this.archiveUri,
+ historyUri: this.historyUri,
};
}
@@ -208,115 +402,531 @@ export class TakeoverCore {
const data = isTakeoverEntry ? entry.data : null;
if (!data || typeof data !== "object") continue;
+ this.coveredThroughEntryId = typeof data.coveredThroughEntryId === "string" ? data.coveredThroughEntryId : "";
this.coveredUserTurns = Math.max(0, Math.floor(Number(data.coveredUserTurns) || 0));
+ this.legacyBoundary = !this.coveredThroughEntryId && this.coveredUserTurns > 0
+ ? {
+ coveredUserTurns: this.coveredUserTurns,
+ fingerprint: typeof data.fingerprint === "string" ? data.fingerprint : null,
+ }
+ : null;
this.overview = typeof data.overview === "string" ? data.overview : "";
- this.fingerprint = typeof data.fingerprint === "string" ? data.fingerprint : null;
this.pendingTokens = Math.max(0, Math.floor(Number(data.pendingTokens) || 0));
this.lastSeenUserTurns = Math.max(0, Math.floor(Number(data.lastSeenUserTurns) || 0));
this.syncedEntryCount = Math.max(0, Math.floor(Number(data.syncedEntryCount) || 0));
+ // New fields default to safe absences so an old persisted entry restores.
+ this.pendingArchive = restorePendingArchive(data.pendingArchive);
+ this.captureGap = data.captureGap === true;
+ this.archiveUri = typeof data.archiveUri === "string" ? data.archiveUri : "";
+ this.historyUri = typeof data.historyUri === "string" ? data.historyUri : "";
this.lastPersisted = JSON.stringify(this.persistedState());
- this.log(`takeover: restored boundary at ${this.coveredUserTurns} user turns, ${this.pendingTokens} pending tokens`);
+ this.log(
+ `takeover: restored boundary ${this.coveredThroughEntryId || (this.legacyBoundary ? `at ${this.coveredUserTurns} user turns (count-based)` : "none")}` +
+ `, ${this.pendingTokens} pending tokens` +
+ (this.pendingArchive ? `, pending archive ${this.pendingArchive.archiveUri}` : "") +
+ (this.captureGap ? ", capture gap set" : ""),
+ );
return this.state;
}
return this.state;
}
- transformContext(messages) {
+ /**
+ * Replace the covered prefix of the `context` hook's messages with the
+ * archive overview. `branch` is pi's `getBranch()`: the boundary is an entry
+ * id, found in pi's context projection of the branch and mapped onto the
+ * messages through the first user turn it keeps.
+ */
+ transformContext(messages, branch = []) {
const list = Array.isArray(messages) ? messages : [];
this.lastSeenUserTurns = countUserTurns(list);
- if (!this.enabled) return list;
- if (this.coveredUserTurns <= 0 || !this.overview) return list;
+ if (!this.enabled || !this.overview) return list;
+ const entries = projectContextEntries(currentBranch(branch));
+ if (this.legacyBoundary) this.adoptLegacyBoundary(list, entries);
+ if (!this.coveredThroughEntryId) return list;
- const boundaryIdx = findBoundaryIndex(list, this.coveredUserTurns);
- if (boundaryIdx <= 0) {
- this.resetBoundary("history shorter than boundary");
- return list;
- }
-
- const lastCovered = list[boundaryIdx - 1];
- const fp = fingerprintMessage(lastCovered);
- if (this.fingerprint === null) {
- this.fingerprint = fp;
- } else if (this.fingerprint !== fp) {
- this.resetBoundary("fingerprint mismatch");
+ const cut = this.locateCut(list, entries);
+ if (cut <= 0) {
+ if (this.boundaryApplied) {
+ this.log("takeover: covered prefix is not in the active context; sending it in full");
+ }
+ this.boundaryApplied = false;
return list;
}
+ this.boundaryApplied = true;
- const kept = list.slice(boundaryIdx);
+ const kept = list.slice(cut);
const firstKeptTs = typeof kept[0]?.timestamp === "number" ? kept[0].timestamp : 1;
+ // Preserve every system message in the covered region, in its original
+ // order. On pi >= 0.86 the transcript carries the base prompt and its tool
+ // declarations as the leading `system` message, and mid-conversation tool
+ // additions/removals, section updates and appended instructions as later
+ // ones (see @earendil-works/pi-ai `getCurrentTools`/`getCurrentSystemMessage`).
+ // Slicing them off with the covered turns would strip the model's tools and
+ // instructions — the overview only stands in for the conversation, never for
+ // the system state. On 0.80.3 the transcript has no system messages here, so
+ // this preserves nothing and the behaviour is unchanged; on 0.87 the host
+ // reconciles tools against the executable set on every request, so keeping
+ // the existing declarations introduces no duplicate. System messages inside
+ // the retained tail are left where they are, not hoisted in front.
+ const coveredSystem = [];
+ for (let i = 0; i < cut; i++) {
+ if (list[i]?.role === "system") coveredSystem.push(list[i]);
+ }
return [
- buildOverviewMessage(this.overview, firstKeptTs, this.config.takeoverOverviewBudget),
+ ...coveredSystem,
+ buildOverviewMessage(
+ this.overview, firstKeptTs, this.config.takeoverOverviewBudget, this.recoveryHint(),
+ ),
...kept,
];
}
- async onTurnSynced(estTokens) {
+ /**
+ * Index in `messages` of the first message kept after the covered prefix,
+ * or -1 when the prefix is not in the active context (another branch, or pi
+ * compacted past it) or no user turn follows it yet. It never clears the
+ * boundary, so returning to the covered branch applies it again.
+ */
+ locateCut(messages, entries) {
+ const through = entries.findIndex((entry) => entry?.id === this.coveredThroughEntryId);
+ if (through < 0) return -1;
+ let covered = 0;
+ for (let i = 0; i <= through; i++) {
+ if (isUserEntry(entries[i])) covered++;
+ }
+ // Messages can sit between the prefix and the next user turn: what a run
+ // added after a keepRecentTurns-0 commit, or a branch summary `/tree` left
+ // at the boundary. No archive covers them, so they are kept.
+ const between = [];
+ for (let i = through + 1; i < entries.length; i++) {
+ const entry = entries[i];
+ if (!isUserEntry(entry)) {
+ const role = contextRole(entry);
+ if (role) between.push(role);
+ continue;
+ }
+ let cut = findUserMessage(messages, entry.message, covered);
+ if (cut <= 0) return -1;
+ // Step back over them role by role; the hook shows no system messages on
+ // pi >= 0.87 and hoisted ones stay put anyway. A mismatch means the two
+ // views disagree, and nothing is trimmed.
+ for (let j = between.length - 1; j >= 0; j--) {
+ do cut--; while (cut >= 0 && messages[cut]?.role === "system");
+ if (cut < 0 || messages[cut]?.role !== between[j]) return -1;
+ }
+ if (cut > 0) this.coveredUserTurns = covered;
+ return cut;
+ }
+ return -1;
+ }
+
+ /**
+ * Convert a count-based boundary to an entry id: locate it the way 0.4.1 did — the
+ * (N+1)-th user turn of the context, checked against the learned fingerprint
+ * of the message before it — then take the entry in front of that turn. A
+ * boundary that no longer matches is dropped, as 0.4.1 itself would have.
+ */
+ adoptLegacyBoundary(messages, entries) {
+ const { coveredUserTurns, fingerprint } = this.legacyBoundary;
+ this.legacyBoundary = null;
+ const cut = findBoundaryIndex(messages, coveredUserTurns);
+ if (cut <= 0 || (fingerprint !== null && fingerprintMessage(messages[cut - 1]) !== fingerprint)) {
+ this.resetBoundary("count-based boundary no longer matches the context");
+ return;
+ }
+ const users = entries.filter(isUserEntry);
+ const kept = findUserMessage(users.map((entry) => entry.message), messages[cut], coveredUserTurns);
+ const keptIdx = kept >= 0 ? entries.indexOf(users[kept]) : -1;
+ const throughId = keptIdx > 0 ? entries[keptIdx - 1]?.id : undefined;
+ if (typeof throughId !== "string" || !throughId) {
+ this.resetBoundary("count-based boundary has no entry id");
+ return;
+ }
+ this.coveredThroughEntryId = throughId;
+ this.coveredUserTurns = coveredUserTurns;
+ this.log(`takeover: count-based boundary adopted through entry ${throughId}`);
+ }
+
+ async onTurnSynced(estTokens, branch = [], { deadline } = {}) {
if (!this.enabled) return false;
+ // A commit already archived and is only waiting for its Working Memory:
+ // account for this turn, then check that one instead of opening another.
+ // Its resolution subtracts only the frozen token snapshot, leaving this new
+ // pressure for the next boundary.
this.pendingTokens += Math.max(0, Math.floor(Number(estTokens) || 0));
+ if (this.pendingArchive) return this.commitAndAdvance(branch, { deadline });
if (this.pendingTokens < this.config.takeoverTokenThreshold) return false;
- if (this.lastSeenUserTurns <= this.config.takeoverKeepRecentTurns) return false;
- return this.commitAndAdvance();
+ // Whether the branch has enough user turns to leave a keep-recent tail is
+ // decided by freezeBoundary on pi's context projection of the branch.
+ return this.commitAndAdvance(branch, { deadline });
+ }
+
+ /**
+ * Check a pending archive once, outside `turn_end` — `before_agent_start`
+ * calls this so a summary that finished between prompts (or between `pi -p`
+ * / `pi -c` processes) trims the very next request.
+ */
+ async resumePending(branch = [], { deadline } = {}) {
+ if (!this.enabled || this.committing || !this.pendingArchive) return false;
+ const until = this.deadlineFrom(deadline);
+ if (this.remaining(until) < 2 * READ_RESERVE_MS) return false;
+ this.committing = true;
+ try {
+ return await this.resolvePendingArchive(branch);
+ } catch (error) {
+ this.log(`takeover: pending archive check failed (${errorMessage(error)}); boundary held`);
+ return false;
+ } finally {
+ this.committing = false;
+ }
}
- async commitAndAdvance() {
+ /**
+ * Advance the boundary behind a confirmed archive, or resume the archive a
+ * previous attempt left pending. Never advances across a capture gap and
+ * never reuses an old /context summary.
+ *
+ * Everything here runs inside a pi event handler, and hosts cap those at
+ * 30s: omp logs `handler timed out`, drops the result and lets the handler
+ * run on (#5275). So nothing waits for a summary — the archive is committed,
+ * its overview read once, and later turns check it again — and the drain and
+ * the commit only get what is left of `deadline`.
+ */
+ async commitAndAdvance(branch = [], { deadline } = {}) {
if (!this.enabled || this.committing) return false;
+ const until = this.deadlineFrom(deadline);
this.committing = true;
try {
- const flushed = await this.io.flush();
- if (!flushed) {
- this.log("takeover: flush failed; commit postponed");
- return false;
- }
+ if (this.pendingArchive) return await this.resolvePendingArchive(branch);
+ return await this.beginArchive(branch, until);
+ } catch (error) {
+ this.log(`takeover: archive preparation failed (${errorMessage(error)}); boundary held`);
+ return false;
+ } finally {
+ this.committing = false;
+ }
+ }
- const committed = await this.io.commit({ queueOnFailure: false, keepRecentCount: this.config.takeoverKeepRecentTurns });
- if (!committed) {
- this.log("takeover: commit failed; retaining pending tokens");
- return false;
+ /** Freeze a boundary, confirm delivery, commit, and read this archive's summary once. */
+ async beginArchive(branch, until) {
+ if (this.captureGap) {
+ this.log("takeover: capture gap present; boundary held, native compaction stays with pi");
+ return false;
+ }
+
+ // One snapshot is frozen, synced and counted, so the archive and
+ // keep_recent_count describe the same entries even if pi appends more
+ // while this commit is in flight.
+ const snapshot = currentBranch(branch);
+ const frozen = this.freezeBoundary(snapshot);
+ if (!frozen) {
+ this.log("takeover: no advanceable boundary; commit skipped");
+ return false;
+ }
+
+ const delivered = await this.confirmDelivery(snapshot, until);
+ if (!delivered) return false;
+
+ const left = this.remaining(until);
+ if (left < COMMIT_RESERVE_MS) {
+ this.log("takeover: handler budget spent before commit; commit postponed");
+ return false;
+ }
+ const committed = await this.io.commit({
+ queueOnFailure: false,
+ keepRecentCount: frozen.keepRecentCount,
+ timeoutMs: left - READ_RESERVE_MS,
+ });
+ const outcome = commitOutcome(committed);
+ if (!outcome.accepted) {
+ // No archive was written: hold the boundary, keep the token pressure so
+ // the next threshold crossing retries, and never fall back to /context.
+ if (outcome.reason === "no_result") {
+ this.log("takeover: commit failed; boundary held");
+ } else if (outcome.reason.startsWith("skipped:")) {
+ this.log(`takeover: archive skipped (${outcome.reason.slice(8)}); boundary held`);
+ } else {
+ this.log(`takeover: commit returned no usable archive (${outcome.reason}); boundary held`);
}
+ return false;
+ }
+
+ const archive = {
+ ...frozen,
+ archiveUri: outcome.archiveUri,
+ taskId: outcome.taskId,
+ historyUri: deriveHistoryUri(outcome.archiveUri),
+ };
+ // Phase 2 writes the summary in the background and usually needs longer
+ // than a handler may wait, so this is one read, not a poll.
+ const overview = this.remaining(until) >= READ_RESERVE_MS
+ ? await this.readOverviewOnce(archive.archiveUri)
+ : "";
+ if (overview) return this.advanceTo(branch, archive, overview);
+
+ // Accepted but not summarized yet: remember exactly this archive and its
+ // frozen boundary, keep local history, and let later turns and prompts
+ // re-check the SAME archive rather than committing again.
+ this.pendingArchive = {
+ archiveUri: archive.archiveUri,
+ taskId: archive.taskId || "",
+ historyUri: archive.historyUri,
+ coveredThroughEntryId: archive.coveredThroughEntryId,
+ coveredUserTurns: archive.coveredUserTurns,
+ frozenTokens: archive.frozenTokens,
+ };
+ this.persist();
+ this.log(`takeover: ${archive.archiveUri} committed, Working Memory not ready; checking on later turns`);
+ return false;
+ }
+
+ /**
+ * Sync the snapshot, then drain the queue: an empty queue alone does not
+ * prove delivery — the just-extracted turn may not have been sent yet. A
+ * permanent loss on either step records the capture gap.
+ */
+ async confirmDelivery(snapshot, until) {
+ const synced = await this.io.syncBranch(snapshot);
+ if ((synced?.permanentFailures || 0) > 0 || this.io.droppedCount() > 0) {
+ this.markCaptureGap();
+ return false;
+ }
+ // A queued transient failure is allowed to proceed to the bounded drainer,
+ // which gets the handler time the commit does not need; only the barrier's
+ // final state proves that every current-session message reached the server.
+ const flushed = await this.io.flush(Math.max(0, this.remaining(until) - COMMIT_RESERVE_MS));
+ // The drain may have exhausted retries or failed to rewrite a queue entry.
+ // Record that permanent gap even when an orphaned processing file keeps the
+ // barrier closed.
+ if (this.io.droppedCount() > 0) {
+ this.markCaptureGap();
+ return false;
+ }
+ if (!flushed) {
+ this.log("takeover: flush barrier closed; commit postponed");
+ return false;
+ }
+ return true;
+ }
- const overview = await this.pollOverview();
+ /**
+ * One check of the archive a prior attempt committed: advance once its
+ * summary is there, and drop it once the archive is terminal without one — a
+ * failed summary, or a server with Working Memory disabled, would otherwise
+ * hold takeover on this archive for the rest of the session.
+ */
+ async resolvePendingArchive(branch) {
+ const pending = this.pendingArchive;
+ if (!pending) return false;
+ // A branch switch or a pi compaction during the wait can take the frozen
+ // prefix out of the active context; the advance is then abandoned, never
+ // moved somewhere else.
+ if (!pending.nativeCompaction && !this.inActiveContext(branch, pending.coveredThroughEntryId)) {
+ this.pendingArchive = null;
+ this.persist();
+ this.log("takeover: frozen boundary no longer in history; advance abandoned");
+ return false;
+ }
+
+ let overview = await this.readOverviewOnce(pending.archiveUri);
+ if (!overview) {
+ const state = await this.io.archiveState(pending.archiveUri);
+ if (!TERMINAL_ARCHIVE_STATES.has(state)) return false;
+ // The summary may have landed between the two reads.
+ overview = await this.readOverviewOnce(pending.archiveUri);
if (!overview) {
- // Commit accepted but overview not ready: don't advance the boundary —
- // never inject an empty overview. Reset the token pressure so the next
- // threshold crossing retries, instead of re-committing on every turn.
- this.log("takeover: overview not ready; boundary unchanged");
- this.pendingTokens = 0;
+ this.pendingArchive = null;
+ // Wait for fresh pressure before the next archive, so a server that
+ // never writes Working Memory does not get a commit on every turn.
+ this.pendingTokens = Math.max(0, this.pendingTokens - (Number(pending.frozenTokens) || 0));
+ this.persist();
+ this.log(`takeover: ${pending.archiveUri} ${state} without Working Memory; boundary held`);
return false;
}
+ }
- const newCovered = Math.max(0, this.lastSeenUserTurns - this.config.takeoverKeepRecentTurns);
- if (newCovered > this.coveredUserTurns) {
- this.coveredUserTurns = newCovered;
- this.fingerprint = null;
- }
- this.overview = overview;
- this.pendingTokens = 0;
- this.syncedEntryCount = Math.max(0, Math.floor(Number(this.io.getWatermark()) || 0));
+ if (pending.nativeCompaction) {
+ // Pi already ran its own compaction after the earlier hook returned
+ // undefined. This archive must not install a second takeover boundary
+ // over Pi's new context, and the recovery hint keeps naming the archive
+ // the current overview came from.
+ this.pendingArchive = null;
+ this.pendingTokens = Math.max(0, this.pendingTokens - (Number(pending.frozenTokens) || 0));
this.persist();
- this.log(`takeover: boundary advanced to ${this.coveredUserTurns} user turns`);
- return true;
- } finally {
- this.committing = false;
+ this.log(`takeover: native-compaction archive completed at ${pending.archiveUri}`);
+ return false;
+ }
+ return this.advanceTo(branch, pending, overview);
+ }
+
+ /** Move the boundary to a summarized archive, re-checking the frozen prefix first. */
+ advanceTo(branch, frozen, overview) {
+ if (!this.inActiveContext(branch, frozen.coveredThroughEntryId)) {
+ this.pendingArchive = null;
+ this.persist();
+ this.log("takeover: frozen boundary no longer in history; advance abandoned");
+ return false;
+ }
+
+ this.coveredThroughEntryId = frozen.coveredThroughEntryId;
+ this.coveredUserTurns = frozen.coveredUserTurns;
+ this.legacyBoundary = null;
+ this.overview = overview;
+ this.archiveUri = frozen.archiveUri;
+ this.historyUri = frozen.historyUri || "";
+ this.pendingArchive = null;
+ // Subtract only the pressure this trim froze; tokens accrued while waiting
+ // for the summary belong to the next boundary and are not cleared.
+ this.pendingTokens = Math.max(0, this.pendingTokens - (Number(frozen.frozenTokens) || 0));
+ this.syncedEntryCount = Math.max(0, Math.floor(Number(this.io.getWatermark()) || 0));
+ this.persist();
+ this.log(
+ `takeover: boundary advanced to ${this.coveredUserTurns} user turns ` +
+ `(through ${this.coveredThroughEntryId}) via ${frozen.archiveUri}`,
+ );
+ return true;
+ }
+
+ inActiveContext(branch, entryId) {
+ if (!entryId) return false;
+ return projectContextEntries(currentBranch(branch)).some((entry) => entry?.id === entryId);
+ }
+
+ deadlineFrom(deadline) {
+ const value = Number(deadline);
+ return Number.isFinite(value) ? value : this.io.now() + HANDLER_BUDGET_MS;
+ }
+
+ remaining(until) {
+ return until - this.io.now();
+ }
+
+ /**
+ * Compute the candidate boundary on pi's context projection of the branch:
+ * keep the last `takeoverKeepRecentTurns` user turns, archive the rest.
+ * Returns the entry the covered prefix ends at, the exact keep_recent_count
+ * and the token snapshot, or null when there is nothing new to archive.
+ */
+ freezeBoundary(branch) {
+ const raw = currentBranch(branch);
+ const entries = projectContextEntries(raw);
+ const users = [];
+ entries.forEach((entry, i) => {
+ if (isUserEntry(entry)) users.push(i);
+ });
+ const keep = this.config.takeoverKeepRecentTurns;
+ if (users.length <= keep) return null;
+
+ // The prefix ends right in front of the oldest kept user turn, or at the
+ // tip when nothing is kept; it then applies from the next user turn on.
+ const through = keep > 0 ? users[users.length - keep] - 1 : entries.length - 1;
+ const id = entries[through]?.id;
+ if (typeof id !== "string" || !id) return null;
+ // Only a prefix longer than the current one is worth an archive. A current
+ // boundary outside the active context (another branch, or compacted away)
+ // does not hold the new one back.
+ if (this.coveredThroughEntryId &&
+ entries.findIndex((entry) => entry?.id === this.coveredThroughEntryId) >= through) {
+ return null;
}
+
+ // keep_recent_count is a server message count: the capture payloads the
+ // branch holds after the covered prefix, in the order sync sent them —
+ // system, custom and filtered entries excluded.
+ const rawThrough = raw.findIndex((entry) => entry?.id === id);
+ return {
+ coveredThroughEntryId: id,
+ coveredUserTurns: users.length - keep,
+ keepRecentCount: Math.max(0, Math.floor(Number(this.io.captureCount(raw.slice(rawThrough + 1))) || 0)),
+ frozenTokens: this.pendingTokens,
+ };
+ }
+
+ markCaptureGap() {
+ if (this.captureGap) return;
+ this.captureGap = true;
+ this.pendingArchive = null;
+ this.persist();
+ this.log("takeover: capture gap recorded; takeover disabled for this session, native compaction stays with pi");
+ }
+
+ recordCaptureGap() {
+ this.markCaptureGap();
}
- async handleBeforeCompact(preparation = {}) {
+ /**
+ * Answer pi's compaction with this session's own archive summary. Pi needs
+ * the summary now, so this is the one place that polls — but only until the
+ * handler deadline. Unless it returns a compaction, it changes no takeover
+ * state: pi may still cancel or fail its own compaction, and the boundary
+ * must then keep applying to the uncompacted context.
+ */
+ async handleBeforeCompact(preparation = {}, branch = [], { deadline } = {}) {
if (!this.enabled || this.committing) return undefined;
if (!preparation.firstKeptEntryId) return undefined;
+ // A gap means the archive is missing messages, so an OpenViking summary
+ // cannot faithfully cover pi's cut — let pi compact natively.
+ if (this.captureGap) return undefined;
+ // A pending takeover archive covers only its frozen head; it cannot stand
+ // in for the broader range pi selected. Never create a second archive while
+ // the first one is unresolved.
+ if (this.pendingArchive) return undefined;
+ if (preparation.signal?.aborted) return undefined;
+ const until = this.deadlineFrom(deadline);
this.committing = true;
try {
- const flushed = await this.io.flush();
- if (!flushed) return undefined;
+ // Sync the latest branch first; native compaction archives all captured
+ // history (keepRecentCount 0) and hands pi its own firstKeptEntryId, so
+ // the summary and the retained tail may overlap.
+ if (!(await this.confirmDelivery(currentBranch(branch), until))) return undefined;
+ if (preparation.signal?.aborted) return undefined;
- const committed = await this.io.commit({ queueOnFailure: false, keepRecentCount: this.config.takeoverKeepRecentTurns });
- if (!committed) return undefined;
+ const left = this.remaining(until);
+ if (left < COMMIT_RESERVE_MS) {
+ this.log("takeover: handler budget spent before native compaction commit; using pi compaction");
+ return undefined;
+ }
+ const committed = await this.io.commit({
+ queueOnFailure: false, keepRecentCount: 0, timeoutMs: left - READ_RESERVE_MS,
+ });
+ const outcome = commitOutcome(committed);
+ if (!outcome.accepted) {
+ if (outcome.reason === "no_result") {
+ this.log("takeover: native compaction commit failed; using pi compaction");
+ } else if (outcome.reason.startsWith("skipped:")) {
+ this.log(`takeover: native compaction archive skipped (${outcome.reason.slice(8)}); using pi compaction`);
+ } else {
+ this.log(`takeover: native compaction returned no usable archive (${outcome.reason}); using pi compaction`);
+ }
+ return undefined;
+ }
- const overview = await this.pollOverview();
- if (!overview) return undefined;
+ const historyUri = deriveHistoryUri(outcome.archiveUri);
+ const overview = await this.pollArchiveOverview(outcome.archiveUri, preparation.signal, until);
+ if (!overview || this.remaining(until) <= 0) {
+ // The server accepted the archive, but this compaction attempt cannot
+ // wait any longer. Remember it so ordinary takeover will not create a
+ // second archive, and hand this compaction back to pi.
+ this.pendingArchive = {
+ archiveUri: outcome.archiveUri,
+ taskId: outcome.taskId || "",
+ historyUri,
+ coveredThroughEntryId: "",
+ coveredUserTurns: 0,
+ frozenTokens: this.pendingTokens,
+ nativeCompaction: true,
+ };
+ this.persist();
+ this.log(`takeover: ${outcome.archiveUri} not summarized in time; using pi compaction`);
+ return undefined;
+ }
this.overview = overview;
+ this.archiveUri = outcome.archiveUri;
+ this.historyUri = historyUri;
this.resetBoundary("pi compaction absorbed boundary");
this.pendingTokens = 0;
this.syncedEntryCount = Math.max(0, Math.floor(Number(this.io.getWatermark()) || 0));
@@ -324,12 +934,17 @@ export class TakeoverCore {
return {
compaction: {
- summary: `${OVERVIEW_MARKER}\n${this.truncatedOverview()}`,
+ summary:
+ `${OVERVIEW_MARKER}\n${this.truncatedOverview()}` +
+ this.recoveryHint(outcome.archiveUri, historyUri),
firstKeptEntryId: preparation.firstKeptEntryId,
tokensBefore: Number(preparation.tokensBefore) || 0,
details: { source: "openviking" },
},
};
+ } catch (error) {
+ this.log(`takeover: native compaction archive failed (${errorMessage(error)}); using pi compaction`);
+ return undefined;
} finally {
this.committing = false;
}
@@ -342,11 +957,13 @@ export class TakeoverCore {
}
resetBoundary(reason = "reset") {
- if (this.coveredUserTurns !== 0 || this.fingerprint !== null) {
+ if (this.coveredThroughEntryId || this.legacyBoundary) {
this.log(`takeover: boundary reset (${reason})`);
}
+ this.coveredThroughEntryId = "";
this.coveredUserTurns = 0;
- this.fingerprint = null;
+ this.legacyBoundary = null;
+ this.boundaryApplied = false;
}
truncatedOverview() {
@@ -355,18 +972,42 @@ export class TakeoverCore {
return truncated === raw ? raw : `${truncated}\n...(truncated)`;
}
+ /** Append a non-budgeted source recovery hint only when both required tools exist. */
+ recoveryHint(archiveUri = this.archiveUri, historyUri = this.historyUri) {
+ const tools = new Set(this.io.availableTools());
+ if (!tools.has("openviking_list") || !tools.has("openviking_read")) return "";
+ const archive = String(archiveUri || "").trim();
+ const history = String(historyUri || "").trim();
+ if (!archive || !history) return "";
+ return (
+ `\n\nArchived capture: ${archive}\n` +
+ `This contains captured historical messages, not the unfiltered Pi transcript. ` +
+ `Semantic search does not retrieve archive source verbatim. Use openviking_list on ${history} ` +
+ `to locate archives, then openviking_read with uris=["${archive}/messages.jsonl"], ` +
+ `offset and limit to read it in chunks.` +
+ (tools.has("openviking_grep") ? ` Use openviking_grep only as an optional locator.` : "")
+ );
+ }
+
persistedState() {
const rawWatermark = Number(this.io.getWatermark());
const watermark = Number.isFinite(rawWatermark)
? Math.max(0, Math.floor(rawWatermark))
: this.syncedEntryCount;
return {
+ coveredThroughEntryId: this.coveredThroughEntryId,
+ // Also what a 0.4.1 reader restores its boundary from after a downgrade.
coveredUserTurns: this.coveredUserTurns,
+ // A count-based boundary not yet adopted keeps its fingerprint across restarts.
+ ...(this.legacyBoundary ? { fingerprint: this.legacyBoundary.fingerprint } : {}),
overview: truncateToTokens(this.overview, this.config.takeoverOverviewBudget),
- fingerprint: this.fingerprint,
pendingTokens: this.pendingTokens,
lastSeenUserTurns: this.lastSeenUserTurns,
syncedEntryCount: watermark,
+ pendingArchive: this.pendingArchive,
+ captureGap: this.captureGap,
+ archiveUri: this.archiveUri,
+ historyUri: this.historyUri,
};
}
@@ -382,16 +1023,32 @@ export class TakeoverCore {
}
}
- async pollOverview() {
+ /** One read of an archive's `.overview.md`; "" until its non-empty body lands. */
+ async readOverviewOnce(archiveUri) {
+ const uri = String(archiveUri || "").trim();
+ if (!uri) return "";
+ try {
+ const value = await this.io.readArchiveOverview(uri);
+ return typeof value === "string" ? value.trim() : "";
+ } catch (error) {
+ this.log(`takeover: archive overview read failed for ${uri} (${errorMessage(error)})`);
+ return "";
+ }
+ }
+
+ /**
+ * Poll one archive's `.overview.md` — `takeoverOverviewPollMax` reads,
+ * `takeoverOverviewPollMs` apart — and stop early at `until`.
+ */
+ async pollArchiveOverview(archiveUri, signal, until = Infinity) {
for (let i = 0; i < this.config.takeoverOverviewPollMax; i++) {
- const value = await this.io.fetchOverview(this.config.takeoverOverviewBudget * 4);
- const overview = typeof value === "string"
- ? value.trim()
- : String(value?.latest_archive_overview || "").trim();
+ if (signal?.aborted) return "";
+ const overview = await this.readOverviewOnce(archiveUri);
+ if (signal?.aborted) return "";
if (overview) return overview;
- if (i < this.config.takeoverOverviewPollMax - 1 && this.config.takeoverOverviewPollMs > 0) {
- await this.io.sleep(this.config.takeoverOverviewPollMs);
- }
+ if (i === this.config.takeoverOverviewPollMax - 1 || this.config.takeoverOverviewPollMs <= 0) break;
+ if (this.remaining(until) < this.config.takeoverOverviewPollMs + READ_RESERVE_MS) break;
+ await this.io.sleep(this.config.takeoverOverviewPollMs);
}
return "";
}
@@ -404,3 +1061,23 @@ export class TakeoverCore {
}
}
}
+
+/** Rehydrate a persisted pending-archive record, dropping malformed shapes. */
+function restorePendingArchive(value) {
+ if (!value || typeof value !== "object" || Array.isArray(value)) return null;
+ const archiveUri = typeof value.archiveUri === "string" ? value.archiveUri.trim() : "";
+ if (!archiveUri) return null;
+ return {
+ archiveUri,
+ taskId: typeof value.taskId === "string" ? value.taskId : "",
+ historyUri: typeof value.historyUri === "string" ? value.historyUri : "",
+ coveredThroughEntryId: typeof value.coveredThroughEntryId === "string" ? value.coveredThroughEntryId : "",
+ coveredUserTurns: Math.max(0, Math.floor(Number(value.coveredUserTurns) || 0)),
+ frozenTokens: Math.max(0, Math.floor(Number(value.frozenTokens) || 0)),
+ ...(value.nativeCompaction === true ? { nativeCompaction: true } : {}),
+ };
+}
+
+function errorMessage(error) {
+ return error instanceof Error ? error.message : String(error || "unknown");
+}
diff --git a/examples/pi-coding-agent-extension/package-lock.json b/examples/pi-coding-agent-extension/package-lock.json
index 7677b715f..8ecdab663 100644
--- a/examples/pi-coding-agent-extension/package-lock.json
+++ b/examples/pi-coding-agent-extension/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "openviking-pi-extension",
- "version": "0.4.1",
+ "version": "0.4.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "openviking-pi-extension",
- "version": "0.4.1",
+ "version": "0.4.2",
"dependencies": {
"@modelcontextprotocol/client": "2.0.0"
},
diff --git a/examples/pi-coding-agent-extension/package.json b/examples/pi-coding-agent-extension/package.json
index 60bcea404..519dbf02b 100644
--- a/examples/pi-coding-agent-extension/package.json
+++ b/examples/pi-coding-agent-extension/package.json
@@ -1,6 +1,6 @@
{
"name": "openviking-pi-extension",
- "version": "0.4.1",
+ "version": "0.4.2",
"description": "OpenViking memory and context extension for the pi coding agent",
"private": true,
"type": "module",
diff --git a/examples/pi-coding-agent-extension/scripts/e2e-live.mjs b/examples/pi-coding-agent-extension/scripts/e2e-live.mjs
index 7f83a383f..8299d1059 100755
--- a/examples/pi-coding-agent-extension/scripts/e2e-live.mjs
+++ b/examples/pi-coding-agent-extension/scripts/e2e-live.mjs
@@ -109,7 +109,7 @@ function copyExtension() {
for (const name of readdirSync(EXT_SRC)) {
const src = join(EXT_SRC, name);
const dst = join(extDir, basename(name));
- if (name.endsWith(".ts") || name === "README.md" || name === "DESIGN.md") {
+ if (name.endsWith(".ts") || ["README.md", "DESIGN.md", "package.json", "package-lock.json"].includes(name)) {
copyFileSync(src, dst);
} else if (["lib", "shared", "scripts"].includes(name)) {
cpSync(src, dst, { recursive: true });
@@ -117,6 +117,15 @@ function copyExtension() {
}
}
copyExtension();
+const install = spawnSync("npm", ["ci", "--omit=dev", "--ignore-scripts"], {
+ cwd: extDir,
+ timeout: 120_000,
+ encoding: "utf8",
+});
+if (install.status !== 0) {
+ console.error(`e2e: dependency install failed\n${install.stderr || install.stdout}`);
+ process.exit(2);
+}
// The extension has no config file of its own; its knobs live in ovcli.conf's
// plugin section like every other harness's.
@@ -199,9 +208,32 @@ function payloadsFor(turn) {
.map((f) => readFileSync(join(outDir, f), "utf8"));
}
+function toolUses(payloads) {
+ const uses = [];
+ for (const raw of payloads) {
+ try {
+ const payload = JSON.parse(raw);
+ for (const message of payload.messages ?? []) {
+ for (const part of Array.isArray(message.content) ? message.content : []) {
+ if (part?.type === "tool_use" && typeof part.name === "string") uses.push(part);
+ }
+ }
+ } catch {
+ // A malformed probe payload is caught by the missing-tool assertions.
+ }
+ }
+ return uses;
+}
+
const t1 = runTurn(1, `Please remember: the release codename is ZEPHYR-9942. Just acknowledge briefly. ${PAD1}`);
const t2 = runTurn(2, `Second note: the deploy window is Friday 03:00 UTC. Acknowledge briefly. ${PAD2}`, ["-c"]);
-const t3 = runTurn(3, "What is the release codename I told you earlier? Answer with just the codename.", ["-c"]);
+const t3 = runTurn(3, [
+ "Recover the release codename from the archived capture.",
+ "You must first call the built-in bash tool with pwd.",
+ "Then follow the recovery hint in [OpenViking Session Context]: call openviking_list on the history URI,",
+ "then call openviking_read on the archive messages.jsonl with explicit offset and limit.",
+ "After those tool calls, answer with just the codename.",
+].join(" "), ["-c"]);
console.log("\ne2e: --- assertions ---");
check(t1.status === 0 && t2.status === 0 && t3.status === 0, "all three pi runs exited 0");
@@ -210,11 +242,21 @@ const p3 = payloadsFor(3);
check(p3.length > 0, `probe captured T3 provider payload (${p3.length} request(s))`);
if (p3.length > 0) {
const last = p3[p3.length - 1];
+ const uses = toolUses(p3);
check(last.includes("[OpenViking Session Context]"), "T3 request contains the OV overview block");
// Match the full padded turn body, not the PADDING-T1 marker: the archive
// overview and recalled memories may legitimately quote the marker.
check(!last.includes(PAD1), "T3 request no longer contains the raw T1 turn body");
check(last.includes("PADDING-T2") || last.includes("codename"), "T3 request keeps recent live context");
+ check(uses.some((part) => part.name === "bash"), "T3 executed a built-in tool");
+ check(uses.some((part) => part.name === "openviking_list"), "T3 listed the archive history");
+ const reads = uses.filter((part) => part.name === "openviking_read");
+ check(reads.length > 0, "T3 read the archived capture");
+ check(reads.some((part) =>
+ Array.isArray(part.input?.uris) &&
+ Number.isFinite(part.input?.offset) &&
+ Number.isFinite(part.input?.limit)
+ ), "T3 used paginated archive read arguments");
}
if (t3.out.includes("ZEPHYR-9942")) pass("model recovered the archived fact from the OV overview");
@@ -222,23 +264,31 @@ else warn("model answer did not contain ZEPHYR-9942; inspect overview quality");
const sessionIdFile = join(outDir, "session-id.txt");
let ovSessionId = null;
+let archiveUri = null;
+const logPath = join(outDir, "takeover.log");
+let takeoverLog = "";
+if (existsSync(logPath)) {
+ takeoverLog = readFileSync(logPath, "utf8");
+ archiveUri = /boundary advanced[^\n]* via (viking:\/\/[^"\s]+)/.exec(takeoverLog)?.[1] ?? null;
+}
if (existsSync(sessionIdFile)) {
ovSessionId = `pi-${readFileSync(sessionIdFile, "utf8").trim()}`;
const ctx = await ovFetch(`/api/v1/sessions/${encodeURIComponent(ovSessionId)}/context?token_budget=4000`);
check(ctx.ok, `OV session ${ovSessionId} readable`);
const result = ctx.body?.result ?? {};
- const overview = (result.latest_archive_overview ?? "").trim();
- check(overview.length > 0, "OV session has a non-empty archive overview");
check((result.stats?.totalArchives ?? 0) >= 1, `OV session has >=1 archive (got ${result.stats?.totalArchives})`);
+ check(Boolean(archiveUri), "takeover log identifies the exact archive URI");
+ if (archiveUri) {
+ const overview = await ovFetch(`/api/v1/content/read?uri=${encodeURIComponent(`${archiveUri}/.overview.md`)}`);
+ check(overview.ok && String(overview.body?.result ?? "").trim().length > 0, "exact archive has a non-empty overview");
+ }
} else {
fail("probe did not record a pi session id");
}
-const logPath = join(outDir, "takeover.log");
-if (existsSync(logPath)) {
- const log = readFileSync(logPath, "utf8");
- check(log.includes("boundary advanced"), "takeover log shows a boundary advance");
- console.log(`\ne2e: takeover.log:\n${log.trim()}`);
+if (takeoverLog) {
+ check(takeoverLog.includes("boundary advanced"), "takeover log shows a boundary advance");
+ console.log(`\ne2e: takeover.log:\n${takeoverLog.trim()}`);
} else {
warn("no takeover debug log written");
}
diff --git a/examples/pi-coding-agent-extension/sync.ts b/examples/pi-coding-agent-extension/sync.ts
index 6a19ad2ee..892f3c04d 100644
--- a/examples/pi-coding-agent-extension/sync.ts
+++ b/examples/pi-coding-agent-extension/sync.ts
@@ -1,4 +1,7 @@
import type { OVClient } from "./client.js";
+import { readdir, readFile } from "node:fs/promises";
+import { homedir } from "node:os";
+import { join } from "node:path";
import { createLogger } from "./shared/debug-log.mjs";
import type { OVConfig } from "./config.js";
import { deriveHarnessSessionId } from "./shared/session-model.mjs";
@@ -18,6 +21,8 @@ export interface SyncBranchResult {
added: number;
tokens: number;
allDelivered: boolean;
+ queued: number;
+ permanentFailures: number;
}
export class SyncManager {
@@ -26,6 +31,14 @@ export class SyncManager {
private logger: ReturnType<typeof createLogger>;
private ovSessionId: string | null = null;
private syncedEntryCount = 0;
+ /**
+ * Messages this session lost for good: non-retryable (4xx) rejections and
+ * queued entries whose retry budget ran out. They are counted as accepted so
+ * the watermark advances and the flush barrier can still clear — but the
+ * archive is then missing them, so this count is takeover's capture-gap
+ * signal. Takeover persists the resulting boolean with its own session state.
+ */
+ private droppedForever = 0;
constructor(client: OVClient, config: OVConfig) {
this.client = client;
@@ -38,6 +51,20 @@ export class SyncManager {
get sessionId(): string | null { return this.ovSessionId; }
get syncedCount(): number { return this.syncedEntryCount; }
+ /** Messages OpenViking will never receive; they are missing from the archive. */
+ get droppedCount(): number { return this.droppedForever; }
+
+ /**
+ * How many payloads the capture path would actually send for a slice of the
+ * branch — the exact `keep_recent_count` the server expects, which is a
+ * message count with system/custom/filtered entries excluded, not a user-turn
+ * count. Runs the same extraction takeover trims to, from a zero watermark so
+ * it measures the slice itself.
+ */
+ captureCount(branchSlice: any[]): number {
+ const extracted = extractBranchCapturePayloads(branchSlice, 0, this.config);
+ return extracted.payloads.length;
+ }
restoreWatermark(n: number): void {
const next = Math.max(0, Math.floor(Number(n) || 0));
@@ -54,20 +81,38 @@ export class SyncManager {
async replayPending(): Promise<void> {
if (!this.client.connected) return;
+ if (this.config.takeoverEnabled) {
+ // The generic replay path intentionally drops 4xx and exhausted entries
+ // without reporting which session lost them. Takeover needs that fact to
+ // block trimming, so its current session always uses the tracked drainer.
+ await this.drainSessionBacklog();
+ // Other sessions' backlog still goes through the generic replay, but only
+ // once none of this session's entries are left for it to drop untracked;
+ // otherwise it waits for a later start.
+ const sid = this.ovSessionId;
+ if (sid && countUndeliveredForSession(await listPending(), sid) > 0) return;
+ if (sid && await hasProcessingMessage(sid)) return;
+ }
await replayPending(
(path: string, init?: any) => this.client.fetchJSON(path, init),
(stage: string, data: unknown) => this.logger.log(stage, data),
);
}
- async flushForTakeover(): Promise<boolean> {
+ async flushForTakeover(budgetMs?: number): Promise<boolean> {
if (!this.ovSessionId) return false;
- // Drain is bounded (time / max batches). Remaining undelivered entries —
- // including any still claimed as `.processing` — keep the barrier closed
- // until a later turn finishes draining them.
- if (this.client.connected) await this.drainSessionBacklog();
+ // Drain is bounded (time / max batches), and takeover passes what is left
+ // of its handler budget. Remaining undelivered entries — including any
+ // still claimed as `.processing` — keep the barrier closed until a later
+ // turn finishes draining them.
+ if (this.client.connected) await this.drainSessionBacklog(budgetMs);
const pending = await listPending();
- return countUndeliveredForSession(pending, this.ovSessionId) === 0;
+ if (countUndeliveredForSession(pending, this.ovSessionId) !== 0) return false;
+ // listPending intentionally exposes only `.json` entries. A fresh
+ // `.processing` file may be owned by another pi process, so absence from
+ // that list is not proof of delivery. Conservatively wait until it is
+ // dequeued or recovered on a later attempt.
+ return !(await hasProcessingMessage(this.ovSessionId));
}
/**
@@ -78,15 +123,18 @@ export class SyncManager {
* Entries are claimed one batch at a time so a failed batch only costs a
* retry for the entries it contained.
*
- * Bounded per call via OPENVIKING_PENDING_DRAIN_BUDGET_MS (default 60s) and
+ * Bounded per call via OPENVIKING_PENDING_DRAIN_BUDGET_MS (default 10s) and
* optional OPENVIKING_PENDING_DRAIN_MAX_BATCHES so a huge backlog cannot
* block turn_end for an unbounded wall time; remainder drains on later turns.
+ * Every caller runs inside a pi event handler, which hosts cap at 30s, so a
+ * caller can narrow the budget further with `budgetMs`.
*/
- private async drainSessionBacklog(): Promise<void> {
+ private async drainSessionBacklog(budgetMs?: number): Promise<void> {
const sid = this.ovSessionId;
if (!sid) return;
const budgetRaw = Number(process.env.OPENVIKING_PENDING_DRAIN_BUDGET_MS);
- const timeBudgetMs = Number.isFinite(budgetRaw) && budgetRaw >= 0 ? budgetRaw : 60_000;
+ const configured = Number.isFinite(budgetRaw) && budgetRaw >= 0 ? budgetRaw : 10_000;
+ const timeBudgetMs = Number.isFinite(budgetMs) ? Math.max(0, Math.min(configured, Number(budgetMs))) : configured;
const maxRaw = Number(process.env.OPENVIKING_PENDING_DRAIN_MAX_BATCHES);
const maxBatches =
Number.isFinite(maxRaw) && maxRaw > 0 ? Math.floor(maxRaw) : Number.POSITIVE_INFINITY;
@@ -117,7 +165,7 @@ export class SyncManager {
batches += 1;
let delivered = 0;
- await sendSessionMessages(
+ const result = await sendSessionMessages(
this.fetchJSON,
sid,
claimed.map(({ entry }) => entry.payload),
@@ -129,10 +177,16 @@ export class SyncManager {
);
if (delivered === claimed.length) continue;
- // Undelivered entries stay queued with one more retry; incrementRetry
- // drops them once the retry budget is exhausted.
+ // A permanent rejection is removed immediately. Retryable failures stay
+ // queued with one more retry; incrementRetry returning false means either
+ // retry exhaustion or a queue write failure. Both are permanent gaps.
for (const { filename, entry } of claimed.slice(delivered)) {
- await incrementRetry(filename, entry);
+ if (!result.retryable) {
+ await dequeue(filename);
+ this.droppedForever += 1;
+ } else if ((await incrementRetry(filename, entry)) === false) {
+ this.droppedForever += 1;
+ }
}
this.logger.log("drain", {
session: sid,
@@ -146,7 +200,9 @@ export class SyncManager {
private fetchJSON = (path: string, init?: any) => this.client.fetchJSON(path, init);
async syncBranch(branch: any[]): Promise<SyncBranchResult> {
- if (!this.ovSessionId) return { added: 0, tokens: 0, allDelivered: true };
+ if (!this.ovSessionId) {
+ return { added: 0, tokens: 0, allDelivered: true, queued: 0, permanentFailures: 0 };
+ }
const extracted = extractBranchCapturePayloads(branch, this.syncedEntryCount, this.config);
if (extracted.resetWatermark) this.syncedEntryCount = 0;
@@ -156,14 +212,20 @@ export class SyncManager {
for (const payload of extracted.payloads.slice(0, added)) {
tokens += estimatePayloadTokens(payload);
}
- const allDelivered = sent.delivered === added;
+ const allDelivered = sent.delivered === added && sent.permanentFailures === 0;
if (added === extracted.payloads.length) {
this.syncedEntryCount = extracted.nextEntryCount;
}
if (added > 0 && !this.config.takeoverEnabled) {
await this.commitIfNeeded();
}
- return { added, tokens, allDelivered };
+ return {
+ added,
+ tokens,
+ allDelivered,
+ queued: sent.queued,
+ permanentFailures: sent.permanentFailures,
+ };
}
async addPayload(payload: any): Promise<AddPayloadResult> {
@@ -176,8 +238,15 @@ export class SyncManager {
* Returns how many payloads were accepted (sent or queued, always a prefix)
* and how many of those were delivered to the server.
*/
- private async sendPayloads(payloads: any[]): Promise<{ accepted: number; delivered: number }> {
- if (!this.ovSessionId || payloads.length === 0) return { accepted: 0, delivered: 0 };
+ private async sendPayloads(payloads: any[]): Promise<{
+ accepted: number;
+ delivered: number;
+ queued: number;
+ permanentFailures: number;
+ }> {
+ if (!this.ovSessionId || payloads.length === 0) {
+ return { accepted: 0, delivered: 0, queued: 0, permanentFailures: 0 };
+ }
const res = await sendSessionMessages(this.fetchJSON, this.ovSessionId, payloads, {
enqueueOnRetryable: true,
});
@@ -194,9 +263,16 @@ export class SyncManager {
// A non-retryable rejection (4xx) drops the remaining payloads, the same
// outcome replayPending() applies to such entries. Count them as accepted
// so the watermark still advances past them; otherwise the next turn would
- // re-extract and re-send the payloads that were already delivered.
- const dropped = res.retryable ? 0 : res.failed;
- return { accepted: res.sent + res.queued + dropped, delivered: res.sent };
+ // re-extract and re-send the payloads that were already delivered. They are
+ // gone from the archive for good, so they also count as a capture gap.
+ const dropped = (res.retryable ? 0 : res.failed) + res.enqueueFailed;
+ if (dropped > 0) this.droppedForever += dropped;
+ return {
+ accepted: res.sent + res.queued + dropped,
+ delivered: res.sent,
+ queued: res.queued,
+ permanentFailures: dropped,
+ };
}
async commitIfNeeded(): Promise<void> {
@@ -208,11 +284,14 @@ export class SyncManager {
}
}
- async commit(opts: { queueOnFailure?: boolean; keepRecentCount?: number } = {}): Promise<any | null> {
+ async commit(
+ opts: { queueOnFailure?: boolean; keepRecentCount?: number; timeoutMs?: number } = {},
+ ): Promise<any | null> {
if (!this.ovSessionId) return null;
const response = await this.client.commitSessionResponse(
this.ovSessionId,
opts.keepRecentCount,
+ opts.timeoutMs,
);
const result = response.result;
if (!result) {
@@ -242,3 +321,24 @@ export class SyncManager {
return;
}
}
+
+async function hasProcessingMessage(sessionId: string): Promise<boolean> {
+ const dir = process.env.OPENVIKING_PENDING_DIR || join(homedir(), ".openviking", "pending");
+ let names: string[];
+ try {
+ names = await readdir(dir);
+ } catch {
+ return false;
+ }
+ for (const name of names) {
+ if (!name.endsWith(".processing")) continue;
+ try {
+ const entry = JSON.parse(await readFile(join(dir, name), "utf8"));
+ if (entry?.type === "addMessage" && entry.sessionId === sessionId) return true;
+ } catch {
+ // An unreadable processing entry cannot be attributed reliably. It stays
+ // on disk and the regular stale recovery path will handle it later.
+ }
+ }
+ return false;
+}
diff --git a/examples/pi-coding-agent-extension/takeover.ts b/examples/pi-coding-agent-extension/takeover.ts
index 99a2d3a4a..c8f53208e 100644
--- a/examples/pi-coding-agent-extension/takeover.ts
+++ b/examples/pi-coding-agent-extension/takeover.ts
@@ -14,22 +14,28 @@ export function createTakeoverManager(opts: {
return new TakeoverCore({
config,
io: {
- flush: () => sync.flushForTakeover(),
- commit: (commitOpts?: { queueOnFailure?: boolean; keepRecentCount?: number }) => sync.commit(commitOpts),
- fetchOverview: async (tokenBudget?: number) => {
- if (!sync.sessionId) return "";
- const ctx = await client.getSessionContext(
- sync.sessionId,
- tokenBudget ?? config.takeoverOverviewBudget * 4,
- );
- return ctx?.latest_archive_overview ?? "";
- },
+ // Deliver the branch, then confirm the queue is empty for this session.
+ syncBranch: (branch: any[]) => sync.syncBranch(branch),
+ flush: (budgetMs?: number) => sync.flushForTakeover(budgetMs),
+ commit: (commitOpts?: { queueOnFailure?: boolean; keepRecentCount?: number; timeoutMs?: number }) =>
+ sync.commit(commitOpts),
+ // Read the Working Memory of the exact archive this commit produced, not
+ // the session's newest `/context` overview — the latter can be an older
+ // archive this takeover did not create.
+ readArchiveOverview: (archiveUri: string) => client.readArchiveOverview(archiveUri),
+ // Whether a still-unsummarized archive can get its summary at all.
+ archiveState: (archiveUri: string) => client.getArchiveState(archiveUri),
+ // The exact server keep_recent_count for the retained tail (message count,
+ // not user-turn count): system, custom and filtered entries excluded.
+ captureCount: (branchSlice: any[]) => sync.captureCount(branchSlice),
persistEntry: (customType: string, data: any) => {
if (typeof pi?.appendEntry === "function") {
pi.appendEntry(customType, data);
}
},
getWatermark: () => sync.syncedCount,
+ droppedCount: () => sync.droppedCount,
+ availableTools: () => typeof pi?.getActiveTools === "function" ? pi.getActiveTools() : [],
log: opts.log,
},
});
diff --git a/examples/pi-coding-agent-extension/tests/client-archive-overview.test.mjs b/examples/pi-coding-agent-extension/tests/client-archive-overview.test.mjs
new file mode 100644
index 000000000..f3012ef8c
--- /dev/null
+++ b/examples/pi-coding-agent-extension/tests/client-archive-overview.test.mjs
@@ -0,0 +1,87 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+import { OVClient } from "../client.ts";
+
+function makeClient() {
+ return new OVClient({
+ endpoint: "http://127.0.0.1:1933",
+ apiKey: "",
+ account: "",
+ user: "",
+ authMode: "trusted",
+ sendIdentityHeaders: false,
+ peerId: "",
+ userAgent: "test",
+ });
+}
+
+async function withFetch(handler, fn) {
+ const original = globalThis.fetch;
+ const calls = [];
+ globalThis.fetch = async (url, init) => {
+ calls.push({ url: String(url), init });
+ const value = await handler(url, init);
+ return new Response(JSON.stringify(value.body), {
+ status: value.status ?? 200,
+ headers: { "content-type": "application/json" },
+ });
+ };
+ try {
+ return await fn(calls);
+ } finally {
+ globalThis.fetch = original;
+ }
+}
+
+test("readArchiveOverview reads only the requested archive and strips OKF frontmatter", async () => {
+ await withFetch(async () => ({
+ body: { status: "ok", result: "---\ntitle: archive\n---\n\n# Working Memory\nbody\n" },
+ }), async (calls) => {
+ const uri = "viking://user/u/sessions/s/history/archive_007";
+ assert.equal(await makeClient().readArchiveOverview(uri), "# Working Memory\nbody\n");
+ assert.equal(calls.length, 1);
+ assert.ok(calls[0].url.includes(encodeURIComponent(`${uri}/.overview.md`)));
+ });
+});
+
+test("readArchiveOverview treats 404 and empty bodies as not ready", async () => {
+ await withFetch(async () => ({ status: 404, body: { status: "error", error: { message: "missing" } } }), async () => {
+ assert.equal(await makeClient().readArchiveOverview("viking://archive/1"), null);
+ });
+ await withFetch(async () => ({ body: { status: "ok", result: "---\ntitle: empty\n---\n\n" } }), async () => {
+ assert.equal(await makeClient().readArchiveOverview("viking://archive/1"), null);
+ });
+});
+
+test("readArchiveOverview surfaces non-404 read failures", async () => {
+ await withFetch(async () => ({ status: 500, body: { status: "error", error: { message: "storage failed" } } }), async () => {
+ await assert.rejects(
+ () => makeClient().readArchiveOverview("viking://archive/1"),
+ /archive overview read failed: storage failed/,
+ );
+ });
+});
+
+test("getArchiveState reads the server's terminal markers for one archive", async () => {
+ const archive = "viking://user/u/sessions/s/history/archive_003";
+ const files = new Map();
+ await withFetch(async (url) => {
+ const uri = new URL(String(url)).searchParams.get("uri");
+ if (uri === "viking://user/u/sessions/broken/history/archive_001/.done") {
+ return { status: 500, body: { status: "error", error: { code: "INTERNAL", message: "boom" } } };
+ }
+ return files.has(uri)
+ ? { body: { status: "ok", result: files.get(uri) } }
+ : { status: 404, body: { status: "error", error: { code: "NOT_FOUND", message: "missing" } } };
+ }, async (calls) => {
+ const client = makeClient();
+ assert.equal(await client.getArchiveState(archive), "pending");
+ files.set(`${archive}/.failed.json`, "{\"error\":\"llm\"}");
+ assert.equal(await client.getArchiveState(archive), "failed");
+ files.set(`${archive}/.done`, "{\"working_memory_enabled\":false}");
+ assert.equal(await client.getArchiveState(`${archive}/`), "completed");
+ assert.equal(await client.getArchiveState("viking://user/u/sessions/broken/history/archive_001"), null);
+ assert.equal(await client.getArchiveState(""), null);
+ assert.match(calls[0].url, /content\/read\?uri=.*archive_003%2F\.done$/);
+ });
+});
diff --git a/examples/pi-coding-agent-extension/tests/sync-barrier.test.mjs b/examples/pi-coding-agent-extension/tests/sync-barrier.test.mjs
index e0d01ce69..f514e7df3 100644
--- a/examples/pi-coding-agent-extension/tests/sync-barrier.test.mjs
+++ b/examples/pi-coding-agent-extension/tests/sync-barrier.test.mjs
@@ -1,6 +1,6 @@
import test from "node:test";
import assert from "node:assert/strict";
-import { mkdtemp, readFile, rm } from "node:fs/promises";
+import { mkdtemp, readFile, rename, rm } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { SyncManager } from "../sync.ts";
@@ -172,6 +172,18 @@ test("other-session addMessage and commit queue entries do not block takeover ba
});
});
+test("a current-session processing entry keeps the takeover barrier closed", async () => {
+ await withPendingDir(async (dir) => {
+ const sync = new SyncManager(client({ connected: false }), config());
+ await sync.ensureSession("pi-session");
+ await enqueue("addMessage", sync.sessionId, { role: "user", content: "in flight" });
+ const [{ filename }] = await listPending();
+ await rename(join(dir, filename), join(dir, filename.replace(/\.json$/, ".processing")));
+
+ assert.equal(await sync.flushForTakeover(), false);
+ });
+});
+
test("restoreWatermark prevents pi -c from re-syncing already captured entries", async () => {
await withPendingDir(async () => {
const calls = [];
@@ -282,11 +294,112 @@ test("non-retryable batch failure drops the payloads but still advances the sync
assert.equal(result.added, 2);
assert.equal(result.allDelivered, false);
assert.equal(sync.syncedCount, 2);
+ assert.equal(sync.droppedCount, 2);
+ assert.equal(result.permanentFailures, 2);
assert.equal(calls.length, 1);
assert.equal((await listPending()).length, 0);
});
});
+test("retry queue write failure records a permanent capture gap", async () => {
+ const previous = process.env.OPENVIKING_PENDING_DIR;
+ process.env.OPENVIKING_PENDING_DIR = "/dev/null/unwritable";
+ try {
+ const { c } = batchClient({ respond: () => ({ ok: false, status: 500, error: { message: "offline" } }) });
+ const sync = new SyncManager(c, config());
+ await sync.ensureSession("pi-session");
+ const result = await sync.syncBranch([
+ { type: "message", message: { role: "user", content: "Cannot reach disk queue." } },
+ ]);
+ assert.equal(result.added, 1);
+ assert.equal(result.permanentFailures, 1);
+ assert.equal(sync.droppedCount, 1);
+ assert.equal(sync.syncedCount, 1);
+ } finally {
+ if (previous === undefined) delete process.env.OPENVIKING_PENDING_DIR;
+ else process.env.OPENVIKING_PENDING_DIR = previous;
+ }
+});
+
+test("takeover startup replay records a non-retryable current-session gap", async () => {
+ await withPendingDir(async () => {
+ const c = client({ fetchJSON: async () => ({ ok: false, status: 400, error: { message: "bad" } }) });
+ const sync = new SyncManager(c, config());
+ await sync.ensureSession("pi-session");
+ await enqueue("addMessage", sync.sessionId, { role: "user", content: "poison" });
+
+ await sync.replayPending();
+ assert.equal(sync.droppedCount, 1);
+ assert.equal((await listPending()).length, 0);
+ });
+});
+
+test("takeover startup replay records retry exhaustion", async () => {
+ await withPendingDir(async () => {
+ const previous = process.env.OPENVIKING_PENDING_MAX_RETRIES;
+ process.env.OPENVIKING_PENDING_MAX_RETRIES = "0";
+ try {
+ const c = client({ fetchJSON: async () => ({ ok: false, status: 500 }) });
+ const sync = new SyncManager(c, config());
+ await sync.ensureSession("pi-session");
+ await enqueue("addMessage", sync.sessionId, { role: "user", content: "exhaust now" });
+
+ await sync.replayPending();
+ assert.equal(sync.droppedCount, 1);
+ assert.equal((await listPending()).length, 0);
+ } finally {
+ if (previous === undefined) delete process.env.OPENVIKING_PENDING_MAX_RETRIES;
+ else process.env.OPENVIKING_PENDING_MAX_RETRIES = previous;
+ }
+ });
+});
+
+test("takeover startup replays other sessions after its own, outside its gap accounting", async () => {
+ await withPendingDir(async () => {
+ const seen = [];
+ const c = client({
+ fetchJSON: async (path) => {
+ seen.push(String(path));
+ // Another session's entry is rejected for good: its loss, not this session's gap.
+ if (String(path).includes("different-session")) {
+ return { ok: false, status: 400, error: { message: "bad" } };
+ }
+ return { ok: true, status: 200, result: {} };
+ },
+ });
+ const sync = new SyncManager(c, config());
+ await sync.ensureSession("pi-session");
+ await enqueue("addMessage", sync.sessionId, { role: "user", content: "mine" });
+ await enqueue("addMessage", "different-session", { role: "user", content: "other" });
+ await enqueue("commitSession", "different-session", {});
+
+ await sync.replayPending();
+ assert.ok(seen.some((path) => path.includes("different-session/messages")));
+ assert.ok(seen.some((path) => path.includes("different-session/commit")));
+ assert.equal((await listPending()).length, 0);
+ assert.equal(sync.droppedCount, 0);
+ });
+});
+
+test("takeover startup leaves other sessions queued while its own backlog remains", async () => {
+ await withPendingDir(async () => {
+ const seen = [];
+ const c = client({
+ fetchJSON: async (path) => { seen.push(String(path)); return { ok: false, status: 500 }; },
+ });
+ const sync = new SyncManager(c, config());
+ await sync.ensureSession("pi-session");
+ await enqueue("addMessage", sync.sessionId, { role: "user", content: "mine" });
+ await enqueue("addMessage", "different-session", { role: "user", content: "other" });
+
+ await sync.replayPending();
+ // The generic replay could drop this session's entry untracked, so it waits.
+ assert.equal(seen.some((path) => path.includes("different-session")), false);
+ assert.equal((await listPending()).length, 2);
+ assert.equal(sync.droppedCount, 0);
+ });
+});
+
test("drainSessionBacklog stops after maxBatches and leaves remainder for later turns", async () => {
await withPendingDir(async () => {
const previous = process.env.OPENVIKING_PENDING_DRAIN_MAX_BATCHES;
@@ -310,3 +423,20 @@ test("drainSessionBacklog stops after maxBatches and leaves remainder for later
}
});
});
+
+test("the takeover barrier drains only within the budget it is given", async () => {
+ await withPendingDir(async () => {
+ const { c, calls } = batchClient();
+ const sync = new SyncManager(c, config());
+ await sync.ensureSession("pi-session");
+ await enqueue("addMessage", sync.sessionId, { role: "user", content: "queued while offline" });
+
+ // No handler time left: nothing is sent and the barrier stays closed.
+ assert.equal(await sync.flushForTakeover(0), false);
+ assert.equal(calls.length, 0);
+ assert.equal((await listPending()).length, 1);
+
+ assert.equal(await sync.flushForTakeover(), true);
+ assert.equal(calls.length, 1);
+ });
+});
diff --git a/examples/pi-coding-agent-extension/tests/takeover-core.test.mjs b/examples/pi-coding-agent-extension/tests/takeover-core.test.mjs
index 9eadeebd7..21b5353ef 100644
--- a/examples/pi-coding-agent-extension/tests/takeover-core.test.mjs
+++ b/examples/pi-coding-agent-extension/tests/takeover-core.test.mjs
@@ -5,6 +5,7 @@ import {
TAKEOVER_ENTRY_TYPE,
TakeoverCore,
buildOverviewMessage,
+ commitOutcome,
countUndeliveredForSession,
countUserTurns,
estimatePayloadTokens,
@@ -13,31 +14,79 @@ import {
fingerprintMessage,
flattenContent,
isUserTurnStart,
+ deriveHistoryUri,
+ projectContextEntries,
truncateToTokens,
} from "../lib/takeover-core.mjs";
-function msg(role, content, timestamp = 0) {
- return { role, content, timestamp };
+// Every message gets its own timestamp, as pi's do: the core maps a user entry
+// onto the context's messages by it.
+let clock = 0;
+let nextId = 0;
+
+function msg(role, content, extra = {}) {
+ clock += 1;
+ return { role, content, timestamp: clock, ...extra };
+}
+
+function user(text) {
+ return msg("user", text);
}
-function user(text, timestamp = 0) {
- return msg("user", text, timestamp);
+function assistant(text) {
+ return msg("assistant", text);
}
-function assistant(text, timestamp = 0) {
- return msg("assistant", text, timestamp);
+function system(text, extra = {}) {
+ return msg("system", text, extra);
+}
+
+/**
+ * A pi branch: messages become `message` entries, `STATE` becomes the
+ * `ov-takeover` custom entry takeover itself appends between turns. That entry
+ * sits in front of every following user turn in a real session, which is what
+ * the plain-message branches of earlier tests never showed.
+ */
+const STATE = Symbol("state");
+function branchOf(...items) {
+ return items.map((item) => {
+ nextId += 1;
+ if (item === STATE) return { id: `c${nextId}`, type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: {} };
+ if (item?.type) return { id: `x${nextId}`, ...item };
+ return { id: `m${nextId}`, type: "message", message: item };
+ });
+}
+
+/** The messages pi's `context` hook sees for a branch. */
+function contextOf(branch) {
+ return projectContextEntries(branch).flatMap((entry) => {
+ if (entry.type === "message") return [entry.message];
+ if (entry.type === "compaction") return [{ role: "compactionSummary", summary: entry.summary, timestamp: 0 }];
+ if (entry.type === "branch_summary") return [{ role: "branchSummary", summary: entry.summary, timestamp: 0 }];
+ if (entry.type === "custom_message") return [{ role: "custom", content: entry.content, timestamp: 0 }];
+ return [];
+ });
}
function makeCore(overrides = {}) {
const calls = {
+ synced: 0,
flushed: 0,
committed: 0,
persisted: [],
slept: [],
logs: [],
+ overviewUris: [],
+ tools: overrides.tools ?? [],
};
let watermark = overrides.watermark ?? 0;
+ let dropped = overrides.dropped ?? 0;
const io = {
+ syncBranch: async (branch) => {
+ calls.synced++;
+ calls.lastSyncBranch = branch;
+ return overrides.syncResult ?? { added: 0, tokens: 0, allDelivered: true };
+ },
flush: async () => {
calls.flushed++;
return overrides.flushResult ?? true;
@@ -45,19 +94,30 @@ function makeCore(overrides = {}) {
commit: async (opts) => {
calls.committed++;
calls.lastCommitOpts = opts;
+ // A concrete archive URI lets the poller bind the summary to this commit.
return overrides.commitResult === undefined
- ? { task_id: "t-1", archive_uri: "viking://archive/1" }
+ ? {
+ status: "accepted",
+ archived: true,
+ task_id: "t-1",
+ archive_uri: "viking://user/x/sessions/s/history/archive_001",
+ }
: overrides.commitResult;
},
- fetchOverview: async (budget) => {
- calls.lastOverviewBudget = budget;
+ readArchiveOverview: async (uri) => {
+ calls.overviewUris.push(uri);
const values = overrides.overviews ?? ["overview ready"];
const value = values[Math.min(calls.overviewCalls || 0, values.length - 1)];
calls.overviewCalls = (calls.overviewCalls || 0) + 1;
return value;
},
+ // What sync's capture path sends: conversational messages only.
+ captureCount: (slice) => slice.filter((entry) =>
+ entry?.type === "message" && entry.message?.role !== "system").length,
persistEntry: (type, data) => calls.persisted.push({ type, data }),
getWatermark: () => watermark,
+ droppedCount: () => dropped,
+ availableTools: () => calls.tools,
sleep: async (ms) => calls.slept.push(ms),
log: (message) => calls.logs.push(message),
};
@@ -73,17 +133,30 @@ function makeCore(overrides = {}) {
},
io: { ...io, ...overrides.io },
});
- return { core, calls, setWatermark: (n) => { watermark = n; } };
+ return {
+ core,
+ calls,
+ setWatermark: (n) => { watermark = n; },
+ setDropped: (n) => { dropped = n; },
+ };
+}
+
+function restoreBoundary(core, coveredThroughEntryId, data = {}) {
+ core.restore([{
+ type: "custom",
+ customType: TAKEOVER_ENTRY_TYPE,
+ data: { coveredThroughEntryId, coveredUserTurns: 1, overview: "archived first turn", pendingTokens: 0, ...data },
+ }]);
}
test("flattenContent handles strings and text arrays", () => {
- assert.equal(flattenContent(user("hello")), "hello");
- assert.equal(flattenContent(user([{ type: "text", text: "a" }, { type: "image" }, { type: "text", text: "b" }])), "ab");
- assert.equal(flattenContent(user(null)), "");
+ assert.equal(flattenContent({ role: "user", content: "hello" }), "hello");
+ assert.equal(flattenContent({ role: "user", content: [{ type: "text", text: "a" }, { type: "image" }, { type: "text", text: "b" }] }), "ab");
+ assert.equal(flattenContent({ role: "user", content: null }), "");
});
test("fingerprintMessage includes role length and 200-char prefix", () => {
- const fp = fingerprintMessage(user("x".repeat(250)));
+ const fp = fingerprintMessage({ role: "user", content: "x".repeat(250) });
assert.equal(fp, `user:250:${"x".repeat(200)}`);
assert.notEqual(fingerprintMessage(user("same")), fingerprintMessage(assistant("same")));
});
@@ -134,9 +207,9 @@ test("buildOverviewMessage is byte-stable for the same inputs", () => {
assert.match(a.content, /\[OpenViking Session Context\]/);
});
-test("buildOverviewMessage points the model at the openviking_search tool", () => {
+test("buildOverviewMessage does not promise an unavailable tool", () => {
const message = buildOverviewMessage("summary", 1, 1000);
- assert.match(message.content, /Use openviking_search for details\./);
+ assert.doesNotMatch(message.content, /openviking_search/);
assert.doesNotMatch(message.content, /viking_archive_expand/);
});
@@ -150,74 +223,210 @@ test("countUndeliveredForSession only counts addMessage for the same session", (
assert.equal(countUndeliveredForSession(pending, "a"), 2);
});
+test("commitOutcome requires this commit's archive URI", () => {
+ assert.deepEqual(commitOutcome(null), { accepted: false, reason: "no_result" });
+ assert.deepEqual(
+ commitOutcome({ status: "skipped", archived: false, archive_uri: null, reason: "no_messages" }),
+ { accepted: false, reason: "skipped:no_messages" },
+ );
+ assert.deepEqual(commitOutcome({ status: "accepted", archived: false, archive_uri: "viking://x" }), {
+ accepted: false, reason: "not_archived",
+ });
+ assert.deepEqual(commitOutcome({ status: "accepted", archived: true }), {
+ accepted: false, reason: "no_archive_uri",
+ });
+ assert.deepEqual(commitOutcome({ status: "failed", archived: true, archive_uri: "viking://bad" }), {
+ accepted: false, reason: "status:failed",
+ });
+ assert.deepEqual(commitOutcome({ archive_uri: "viking://user/u/sessions/s/history/archive_002", task_id: "t" }), {
+ accepted: true, reason: "accepted", archiveUri: "viking://user/u/sessions/s/history/archive_002", taskId: "t",
+ });
+ assert.equal(deriveHistoryUri("viking://user/u/sessions/s/history/archive_002"), "viking://user/u/sessions/s/history");
+ assert.equal(deriveHistoryUri("viking://unexpected/archive"), "");
+});
+
+test("projectContextEntries follows pi's compaction and context-edit projection", () => {
+ const head = branchOf(system("BASE"), user("one"), assistant("a1"), STATE, user("two"), assistant("a2"));
+ assert.deepEqual(projectContextEntries(head), head);
+
+ const compaction = { type: "compaction", summary: "pi summary", firstKeptEntryId: head[4].id };
+ const [compactionEntry, ...after] = branchOf(compaction, user("three"));
+ const compacted = [...head, compactionEntry, ...after];
+ assert.deepEqual(
+ projectContextEntries(compacted).map((entry) => entry.id),
+ [compactionEntry.id, head[4].id, head[5].id, after[0].id],
+ );
+
+ const [removal, replacement] = branchOf(
+ { type: "context_edit", targetId: head[2].id, replacement: null },
+ { type: "context_edit", targetId: head[5].id, replacement: { content: "edited" } },
+ );
+ const edited = projectContextEntries([...head, removal, replacement]).map((entry) => entry.id);
+ assert.equal(edited.includes(head[2].id), false);
+ assert.equal(edited.includes(head[5].id), true);
+});
+
test("transformContext drops covered turns and injects overview before recall", () => {
const { core } = makeCore();
- core.restore([
- {
- type: "custom",
- customType: TAKEOVER_ENTRY_TYPE,
- data: {
- coveredUserTurns: 1,
- overview: "archived first turn",
- pendingTokens: 0,
- },
- },
- ]);
- const messages = [
- user("first", 10),
- assistant("answer", 11),
- user("second", 20),
- assistant("answer 2", 21),
- ];
- const out = core.transformContext(messages);
+ const branch = branchOf(user("first"), assistant("answer"), STATE, user("second"), assistant("answer 2"));
+ restoreBoundary(core, branch[2].id);
+ const out = core.transformContext(contextOf(branch), branch);
assert.equal(out.length, 3);
assert.equal(out[0].role, "user");
assert.match(out[0].content, /archived first turn/);
- assert.equal(out[0].timestamp, 19);
+ assert.equal(out[0].timestamp, branch[3].message.timestamp - 1);
assert.equal(out[1].content, "second");
+ assert.equal(core.state.coveredUserTurns, 1);
});
test("transformContext is stable between commits", () => {
const { core } = makeCore();
- core.restore([
- {
- type: "custom",
- customType: TAKEOVER_ENTRY_TYPE,
- data: { coveredUserTurns: 1, overview: "same overview", pendingTokens: 0 },
- },
- ]);
- const messages = [user("first", 1), assistant("answer", 2), user("second", 3)];
- const a = JSON.stringify(core.transformContext(messages));
- const b = JSON.stringify(core.transformContext(messages));
+ const branch = branchOf(user("first"), assistant("answer"), user("second"));
+ restoreBoundary(core, branch[1].id);
+ const a = JSON.stringify(core.transformContext(contextOf(branch), branch));
+ const b = JSON.stringify(core.transformContext(contextOf(branch), () => branch));
assert.equal(a, b);
});
-test("transformContext resets boundary on fingerprint mismatch", () => {
+test("a covered prefix outside the active context is sent in full and applies again on return", () => {
+ const { core, calls } = makeCore();
+ const covered = branchOf(user("first"), assistant("answer"), STATE, user("second"));
+ restoreBoundary(core, covered[2].id);
+ assert.equal(core.transformContext(contextOf(covered), covered).length, 2);
+
+ // /tree to a sibling of the covered answer: the prefix is not in this context.
+ const other = [covered[0], ...branchOf(assistant("other answer"), user("second"))];
+ const messages = contextOf(other);
+ assert.equal(core.transformContext(messages, other), messages);
+ assert.equal(core.state.coveredThroughEntryId, covered[2].id);
+ assert.ok(calls.logs.some((line) => /not in the active context/.test(line)));
+
+ assert.equal(core.transformContext(contextOf(covered), covered).length, 2);
+});
+
+test("transformContext waits for a user turn after the covered prefix", () => {
const { core } = makeCore();
- core.restore([
- {
- type: "custom",
- customType: TAKEOVER_ENTRY_TYPE,
- data: {
- coveredUserTurns: 1,
- overview: "overview",
- fingerprint: fingerprintMessage(assistant("old answer")),
- pendingTokens: 0,
- },
- },
- ]);
- const messages = [user("first"), assistant("new answer"), user("second")];
- const out = core.transformContext(messages);
- assert.equal(out, messages);
+ const branch = branchOf(user("first"), assistant("answer"));
+ restoreBoundary(core, branch[1].id);
+ const messages = contextOf(branch);
+ assert.equal(core.transformContext(messages, branch), messages);
+});
+
+test("transformContext keeps what a run appended after a keepRecentTurns 0 cut", () => {
+ const { core } = makeCore();
+ // Frozen at the tip after a tool-use turn; the run went on before the next
+ // user turn, and no archive covers those messages.
+ const branch = branchOf(
+ user("first"), assistant("tool call"), msg("toolResult", "output"),
+ STATE, assistant("final answer"), user("second"),
+ );
+ restoreBoundary(core, branch[2].id);
+ const out = core.transformContext(contextOf(branch), branch);
+ assert.match(out[0].content, /archived first turn/);
+ assert.deepEqual(out.slice(1).map((message) => message.content), ["final answer", "second"]);
+});
+
+test("a branch summary /tree left at the boundary is kept", () => {
+ const { core } = makeCore();
+ // /tree back to the first kept turn: pi hangs the summary of the abandoned
+ // turns off the boundary entry, and the retried prompt follows it.
+ const branch = branchOf(
+ user("first"), assistant("answer"),
+ { type: "branch_summary", summary: "abandoned second turn", fromId: "m-old" },
+ user("second, retried"),
+ );
+ restoreBoundary(core, branch[1].id);
+ const out = core.transformContext(contextOf(branch), branch);
+ assert.deepEqual(out.map((message) => message.role), ["user", "branchSummary", "user"]);
+ assert.equal(out[1].summary, "abandoned second turn");
+});
+
+test("messages that disagree with the branch between the prefix and the kept turn trim nothing", () => {
+ const { core } = makeCore();
+ const branch = branchOf(user("first"), assistant("answer"), assistant("more"), user("second"));
+ restoreBoundary(core, branch[1].id);
+ const messages = contextOf(branch);
+ // Another extension's context handler replaced a message in that stretch.
+ const rewritten = [...messages.slice(0, 2), { role: "custom", content: "injected", timestamp: 0 }, messages[3]];
+ assert.equal(core.transformContext(rewritten, branch), rewritten);
+});
+
+test("transformContext keeps covered-region system messages in original order", () => {
+ const { core } = makeCore();
+ const baseSystem = system("BASE PROMPT", { toolsAdded: [{ name: "read" }] });
+ const midSystem = system("added a tool", { toolsAdded: [{ name: "openviking_search" }] });
+ const branch = branchOf(baseSystem, user("first"), assistant("answer"), midSystem, user("second"), assistant("answer 2"));
+ restoreBoundary(core, branch[3].id);
+ const out = core.transformContext(contextOf(branch), branch);
+ // Both covered system messages survive, in order, ahead of the overview.
+ assert.equal(out[0], baseSystem);
+ assert.equal(out[1], midSystem);
+ assert.equal(out[2].role, "user");
+ assert.match(out[2].content, /archived first turn/);
+ assert.equal(out[3].content, "second");
+ // Nothing is dropped from the retained tail, and no system is duplicated.
+ assert.equal(out.filter((m) => m.role === "system").length, 2);
+});
+
+test("transformContext leaves a system message inside the retained tail in place", () => {
+ const { core } = makeCore();
+ const keptSystem = system("tool removed mid-tail", { toolsRemoved: [{ name: "read" }] });
+ const branch = branchOf(system("BASE"), user("first"), assistant("answer"), user("second"), keptSystem, assistant("answer 2"));
+ restoreBoundary(core, branch[2].id);
+ const out = core.transformContext(contextOf(branch), branch);
+ // The covered BASE is hoisted before the overview; the tail system stays put.
+ assert.equal(out[0].content, "BASE");
+ assert.match(out[1].content, /archived/);
+ assert.equal(out[2].content, "second");
+ assert.equal(out[3], keptSystem);
+});
+
+test("transformContext finds the kept turn when pi 0.87 hides system messages from the hook", () => {
+ const { core } = makeCore();
+ const branch = branchOf(system("BASE"), user("first"), assistant("answer"), system("mid"), user("second"));
+ restoreBoundary(core, branch[3].id);
+ const withoutSystem = contextOf(branch).filter((message) => message.role !== "system");
+ const out = core.transformContext(withoutSystem, branch);
+ assert.equal(out.length, 2);
+ assert.equal(out[1].content, "second");
+});
+
+test("a count-based boundary is adopted as the entry in front of its first kept turn", () => {
+ const { core } = makeCore();
+ const branch = branchOf(user("first"), assistant("answer"), STATE, user("second"), assistant("answer 2"));
+ const messages = contextOf(branch);
+ core.restore([{ type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: {
+ coveredUserTurns: 1, overview: "legacy", fingerprint: fingerprintMessage(messages[1]), pendingTokens: 0,
+ } }]);
+ const out = core.transformContext(messages, branch);
+ assert.equal(out.length, 3);
+ assert.match(out[0].content, /legacy/);
+ assert.equal(core.state.coveredThroughEntryId, branch[2].id);
+ assert.equal(core.persistedState().coveredThroughEntryId, branch[2].id);
+ assert.equal("fingerprint" in core.persistedState(), false);
+});
+
+test("a count-based boundary whose fingerprint no longer matches is dropped", () => {
+ const { core } = makeCore();
+ const branch = branchOf(user("first"), assistant("new answer"), user("second"));
+ core.restore([{ type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: {
+ coveredUserTurns: 1, overview: "legacy", fingerprint: fingerprintMessage(assistant("old answer")), pendingTokens: 0,
+ } }]);
+ // Until the context hook sees the session, the fingerprint is carried along.
+ assert.equal(typeof core.persistedState().fingerprint, "string");
+ const messages = contextOf(branch);
+ assert.equal(core.transformContext(messages, branch), messages);
+ assert.equal(core.state.coveredThroughEntryId, "");
assert.equal(core.state.coveredUserTurns, 0);
});
test("restore uses the last ov-takeover entry and restores syncedEntryCount", () => {
const { core } = makeCore();
core.restore([
- { type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: { coveredUserTurns: 1, overview: "old", pendingTokens: 3, syncedEntryCount: 10 } },
- { type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: { coveredUserTurns: 2, overview: "new", pendingTokens: 7, syncedEntryCount: 12 } },
+ { type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: { coveredThroughEntryId: "old", coveredUserTurns: 1, overview: "old", pendingTokens: 3, syncedEntryCount: 10 } },
+ { type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: { coveredThroughEntryId: "new", coveredUserTurns: 2, overview: "new", pendingTokens: 7, syncedEntryCount: 12 } },
]);
+ assert.equal(core.state.coveredThroughEntryId, "new");
assert.equal(core.state.coveredUserTurns, 2);
assert.equal(core.state.overview, "new");
assert.equal(core.state.pendingTokens, 7);
@@ -226,57 +435,323 @@ test("restore uses the last ov-takeover entry and restores syncedEntryCount", ()
test("onTurnSynced waits for threshold and enough user turns", async () => {
const { core, calls } = makeCore({ config: { takeoverTokenThreshold: 50, takeoverKeepRecentTurns: 3 } });
- core.transformContext([user("one"), user("two")]);
- assert.equal(await core.onTurnSynced(60), false);
+ const short = branchOf(user("one"), STATE, user("two"));
+ assert.equal(await core.onTurnSynced(60, short), false);
assert.equal(calls.committed, 0);
- core.transformContext([user("one"), user("two"), user("three"), user("four")]);
- assert.equal(await core.onTurnSynced(0), true);
+ const enough = branchOf(user("one"), STATE, user("two"), STATE, user("three"), STATE, user("four"));
+ assert.equal(await core.onTurnSynced(0, enough), true);
assert.equal(calls.committed, 1);
});
-test("commitAndAdvance advances boundary and persists after overview is ready", async () => {
+test("commitAndAdvance advances to an entry id and the next context is trimmed", async () => {
const { core, calls, setWatermark } = makeCore({
watermark: 4,
- overviews: ["", { latest_archive_overview: "fresh overview" }],
+ overviews: ["", "fresh overview"],
});
- core.transformContext([user("one"), assistant("a"), user("two"), assistant("b"), user("three")]);
- setWatermark(5);
- assert.equal(await core.onTurnSynced(120), true);
+ const branch = branchOf(user("one"), assistant("a"), STATE, user("two"), assistant("b"), STATE, user("three"));
+ setWatermark(7);
+ // The summary is not there right after the commit; turn_end does not wait.
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.ok(core.state.pendingArchive);
+ assert.deepEqual(calls.slept, []);
+ // The next turn reads it once and advances.
+ assert.equal(await core.onTurnSynced(0, branch), true);
+ assert.equal(core.state.coveredThroughEntryId, branch[5].id);
assert.equal(core.state.coveredUserTurns, 2);
assert.equal(core.state.overview, "fresh overview");
assert.equal(core.state.pendingTokens, 0);
- assert.equal(core.state.syncedEntryCount, 5);
+ assert.equal(core.state.syncedEntryCount, 7);
assert.equal(calls.flushed, 1);
assert.equal(calls.committed, 1);
assert.equal(calls.lastCommitOpts.queueOnFailure, false);
- assert.deepEqual(calls.slept, [1]);
- assert.equal(calls.persisted.length, 1);
- assert.equal(calls.persisted[0].type, TAKEOVER_ENTRY_TYPE);
+ assert.equal(calls.lastCommitOpts.keepRecentCount, 1);
+ assert.deepEqual(calls.slept, []);
+ // Once for the pending archive, once for the advance.
+ assert.equal(calls.persisted.length, 2);
+ assert.equal(calls.persisted[1].type, TAKEOVER_ENTRY_TYPE);
+ assert.equal(calls.persisted[1].data.coveredThroughEntryId, branch[5].id);
+ assert.deepEqual(calls.overviewUris, [
+ "viking://user/x/sessions/s/history/archive_001",
+ "viking://user/x/sessions/s/history/archive_001",
+ ]);
+
+ // The state entry takeover just persisted lands in front of the next turn.
+ const next = [...branch, ...branchOf(assistant("c"), STATE, user("four"))];
+ const out = core.transformContext(contextOf(next), next);
+ assert.match(out[0].content, /fresh overview/);
+ assert.deepEqual(out.slice(1).map((message) => message.content), ["three", "c", "four"]);
+});
+
+test("keep_recent_count counts captured messages after the covered prefix", async () => {
+ const { core, calls } = makeCore({ config: { takeoverKeepRecentTurns: 1 } });
+ const branch = branchOf(
+ system("base"), user("one"), assistant("one answer"),
+ STATE, user("two"), assistant("tool one"), system("tool added"), assistant("tool two"),
+ );
+ assert.equal(await core.onTurnSynced(120, branch), true);
+ assert.equal(calls.lastCommitOpts.keepRecentCount, 3);
+ assert.equal(core.state.coveredThroughEntryId, branch[3].id);
+});
+
+test("keepRecentTurns zero archives every captured message and cuts at the tip", async () => {
+ const { core, calls } = makeCore({ config: { takeoverKeepRecentTurns: 0 } });
+ const branch = branchOf(user("one"), assistant("answer"));
+ assert.equal(await core.onTurnSynced(120, branch), true);
+ assert.equal(calls.lastCommitOpts.keepRecentCount, 0);
+ assert.equal(core.state.coveredThroughEntryId, branch[1].id);
+ assert.equal(core.state.coveredUserTurns, 1);
+
+ const next = [...branch, ...branchOf(STATE, user("two"))];
+ const out = core.transformContext(contextOf(next), next);
+ assert.deepEqual(out.slice(1).map((message) => message.content), ["two"]);
+});
+
+test("the boundary is counted on the context pi rebuilt after its own compaction", async () => {
+ const { core, calls } = makeCore({ config: { takeoverKeepRecentTurns: 1 } });
+ const head = branchOf(user("one"), assistant("a1"), STATE, user("two"), assistant("a2"));
+ const tail = branchOf(
+ { type: "compaction", summary: "pi summary", firstKeptEntryId: head[3].id },
+ STATE, user("three"), assistant("a3"), STATE, user("four"),
+ );
+ const branch = [...head, ...tail];
+ assert.equal(await core.onTurnSynced(120, branch), true);
+ // Pre-compaction turn one is not part of the active context and not counted.
+ assert.equal(core.state.coveredUserTurns, 2);
+ assert.equal(core.state.coveredThroughEntryId, tail[4].id);
+ assert.equal(calls.lastCommitOpts.keepRecentCount, 1);
+
+ const out = core.transformContext(contextOf(branch), branch);
+ assert.deepEqual(out.slice(1).map((message) => message.content), ["four"]);
});
test("commitAndAdvance keeps pending tokens when flush fails", async () => {
const { core, calls } = makeCore({ flushResult: false });
- core.transformContext([user("one"), user("two")]);
- assert.equal(await core.onTurnSynced(120), false);
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
assert.equal(core.state.pendingTokens, 120);
assert.equal(calls.committed, 0);
+ assert.equal(core.state.coveredThroughEntryId, "");
+ // A postponed attempt is not a state transition worth a session entry.
assert.equal(calls.persisted.length, 0);
});
-test("commitAndAdvance keeps boundary unchanged when overview is not ready", async () => {
+test("below-threshold turns do not append takeover state entries", async () => {
+ const { core, calls } = makeCore();
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(10, branch), false);
+ assert.equal(await core.onTurnSynced(10, branch), false);
+ assert.equal(calls.persisted.length, 0);
+ assert.equal(core.state.pendingTokens, 20);
+});
+
+test("commitAndAdvance persists the same pending archive until its overview is ready", async () => {
const { core, calls } = makeCore({ overviews: ["", "", ""] });
- core.transformContext([user("one"), user("two")]);
- assert.equal(await core.onTurnSynced(120), false);
- assert.equal(core.state.coveredUserTurns, 0);
- // Token pressure resets so the retry waits for the next threshold crossing
- // instead of re-committing (and spawning a new archive) on every turn.
- assert.equal(core.state.pendingTokens, 0);
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.equal(core.state.coveredThroughEntryId, "");
+ assert.equal(core.state.pendingTokens, 120);
+ assert.equal(core.state.pendingArchive.archiveUri, "viking://user/x/sessions/s/history/archive_001");
+ assert.equal(core.state.pendingArchive.coveredThroughEntryId, branch[0].id);
assert.equal(calls.committed, 1);
- assert.equal(calls.persisted.length, 0);
- assert.equal(await core.onTurnSynced(10), false);
+ assert.equal(calls.persisted.length, 1);
+ assert.equal(await core.onTurnSynced(10, branch), false);
+ assert.equal(calls.committed, 1);
+ assert.equal(core.state.pendingTokens, 130);
+});
+
+test("pending archive survives restore and later advances without another commit", async () => {
+ const first = makeCore({ overviews: ["", "", ""] });
+ const branch = branchOf(user("one"), assistant("a"), user("two"));
+ assert.equal(await first.core.onTurnSynced(120, branch), false);
+ const saved = first.core.persistedState();
+
+ const resumed = makeCore({ overviews: ["restored overview"] });
+ resumed.core.restore([{ type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: saved }]);
+ assert.equal(await resumed.core.onTurnSynced(7, branch), true);
+ assert.equal(resumed.calls.committed, 0);
+ assert.equal(resumed.core.state.pendingArchive, null);
+ assert.equal(resumed.core.state.coveredThroughEntryId, branch[1].id);
+ assert.equal(resumed.core.state.pendingTokens, 7);
+});
+
+test("a pending archive without a boundary entry is dropped instead of advanced", async () => {
+ const { core, calls } = makeCore();
+ core.restore([{ type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: {
+ coveredUserTurns: 0, overview: "", pendingTokens: 120,
+ pendingArchive: { archiveUri: "viking://user/x/sessions/s/history/archive_001", coveredUserTurns: 1, fingerprint: "x" },
+ } }]);
+ assert.equal(await core.onTurnSynced(0, branchOf(user("one"), user("two"))), false);
+ assert.equal(core.state.pendingArchive, null);
+ assert.equal(core.state.coveredThroughEntryId, "");
+ assert.equal(calls.committed, 0);
+});
+
+test("old state without archive fields remains compatible", () => {
+ const { core } = makeCore();
+ core.restore([{ type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: {
+ coveredUserTurns: 1, overview: "legacy", pendingTokens: 4, syncedEntryCount: 2,
+ } }]);
+ assert.equal(core.state.pendingArchive, null);
+ assert.equal(core.state.captureGap, false);
+ assert.equal(core.state.archiveUri, "");
+ assert.equal(core.state.historyUri, "");
+ const branch = branchOf(user("one"), assistant("a"), user("two"));
+ const out = core.transformContext(contextOf(branch), branch);
+ assert.match(out[0].content, /legacy/);
+ assert.doesNotMatch(out[0].content, /openviking_list/);
+});
+
+test("capture gap survives restore and blocks takeover plus native compaction", async () => {
+ const { core, calls } = makeCore();
+ const branch = branchOf(user("one"), user("two"));
+ core.restore([{ type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: {
+ coveredUserTurns: 0, overview: "", pendingTokens: 120, captureGap: true,
+ } }]);
+ assert.equal(await core.commitAndAdvance(branch), false);
+ assert.equal(await core.handleBeforeCompact({ firstKeptEntryId: "pi-kept" }, branch), undefined);
+ assert.equal(calls.committed, 0);
+});
+
+test("skipped, not archived and missing URI results never advance", async () => {
+ const branch = branchOf(user("one"), user("two"));
+ for (const commitResult of [
+ { status: "skipped", archived: false, archive_uri: null, reason: "no_messages" },
+ { status: "accepted", archived: false, archive_uri: "viking://bad" },
+ { status: "accepted", archived: true },
+ ]) {
+ const { core, calls } = makeCore({ commitResult });
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.equal(core.state.coveredThroughEntryId, "");
+ assert.equal(calls.overviewUris.length, 0);
+ }
+});
+
+test("archive overview read failures are logged and keep the boundary pending", async () => {
+ const { core, calls } = makeCore({
+ io: { readArchiveOverview: async () => { throw new Error("storage failed"); } },
+ });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.ok(core.state.pendingArchive);
+ assert.ok(calls.logs.some((line) => /overview read failed.*storage failed/.test(line)));
+});
+
+test("a delayed summary preserves token pressure from messages arriving while waiting", async () => {
+ const { core, calls } = makeCore({ overviews: ["", "ready"] });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.equal(await core.onTurnSynced(17, branch), true);
+ assert.equal(calls.committed, 1);
+ assert.equal(core.state.pendingTokens, 17);
+});
+
+test("messages appended while an archive is pending are not included in its frozen boundary", async () => {
+ const { core, calls } = makeCore({ overviews: ["", "ready"] });
+ const initial = branchOf(user("one"), assistant("a"), STATE, user("two"));
+ assert.equal(await core.onTurnSynced(120, initial), false);
+ const extended = [...initial, ...branchOf(assistant("b"), STATE, user("three"))];
+ assert.equal(await core.onTurnSynced(9, extended), true);
+ assert.equal(core.state.coveredThroughEntryId, initial[2].id);
+ assert.equal(core.state.coveredUserTurns, 1);
+ assert.equal(core.state.pendingTokens, 9);
+ assert.equal(calls.committed, 1);
+});
+
+test("queued transient capture is drained before commit", async () => {
+ const { core, calls } = makeCore({
+ syncResult: { added: 1, tokens: 5, allDelivered: false, queued: 1, permanentFailures: 0 },
+ });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), true);
+ assert.equal(calls.flushed, 1);
assert.equal(calls.committed, 1);
- assert.equal(core.state.pendingTokens, 10);
+});
+
+test("the commit syncs the same branch snapshot it froze", async () => {
+ const branch = branchOf(user("one"), user("two"));
+ let reads = 0;
+ const { core, calls } = makeCore();
+ // A getter that would return a longer branch on a second read.
+ assert.equal(await core.onTurnSynced(120, () => (reads++ === 0 ? branch : [...branch, ...branchOf(user("x"))])), true);
+ assert.equal(calls.lastSyncBranch, branch);
+});
+
+test("permanent capture failure persists a gap and blocks later takeover", async () => {
+ const { core, calls } = makeCore({
+ syncResult: { added: 1, tokens: 5, allDelivered: false, queued: 0, permanentFailures: 1 },
+ });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.equal(core.state.captureGap, true);
+ assert.equal(calls.committed, 0);
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.equal(calls.synced, 1);
+});
+
+test("the advance is abandoned when a fork replaces the covered prefix, even with equal content", async () => {
+ const initial = branchOf(user("one"), assistant("same"), user("two"));
+ let active = initial;
+ const { core, calls } = makeCore({
+ io: {
+ readArchiveOverview: async (uri) => {
+ calls.overviewUris.push(uri);
+ active = [initial[0], ...branchOf(assistant("same"), user("two"))];
+ return "fresh overview";
+ },
+ },
+ });
+ assert.equal(await core.onTurnSynced(120, () => active), false);
+ assert.equal(core.state.coveredThroughEntryId, "");
+ assert.equal(core.state.captureGap, false);
+ assert.match(calls.logs.at(-1), /boundary no longer/);
+});
+
+test("the advance holds for append-only growth and a fork after the covered prefix", async () => {
+ const initial = branchOf(user("one"), assistant("answer one"), user("two"), assistant("answer two"));
+ for (const change of [
+ () => [...initial, ...branchOf(user("three"))],
+ () => [...initial.slice(0, 2), ...branchOf(user("forked"))],
+ ]) {
+ let active = initial;
+ const run = makeCore({
+ io: {
+ readArchiveOverview: async (uri) => {
+ run.calls.overviewUris.push(uri);
+ active = change();
+ return "fresh overview";
+ },
+ },
+ });
+ assert.equal(await run.core.onTurnSynced(120, () => active), true);
+ assert.equal(run.core.state.coveredThroughEntryId, initial[1].id);
+ }
+});
+
+test("two consecutive archives advance one complete user turn at a time", async () => {
+ let nextArchive = 0;
+ const { core, calls } = makeCore({
+ io: {
+ commit: async (opts) => {
+ calls.committed++;
+ calls.lastCommitOpts = opts;
+ nextArchive++;
+ return { status: "accepted", archived: true, archive_uri: `viking://user/x/sessions/s/history/archive_00${nextArchive}` };
+ },
+ },
+ });
+ const first = branchOf(user("one"), assistant("a"), STATE, user("two"));
+ assert.equal(await core.onTurnSynced(120, first), true);
+ assert.equal(core.state.coveredUserTurns, 1);
+
+ const second = [...first, ...branchOf(assistant("b"), STATE, user("three"))];
+ // Nothing new to cover before the next user turn arrives.
+ assert.equal(await core.onTurnSynced(120, first), false);
+ assert.equal(await core.onTurnSynced(0, second), true);
+ assert.equal(core.state.coveredUserTurns, 2);
+ assert.equal(core.state.coveredThroughEntryId, second[5].id);
+ assert.equal(calls.overviewUris.at(-1), "viking://user/x/sessions/s/history/archive_002");
+ assert.equal(calls.committed, 2);
});
test("concurrent commitAndAdvance calls are serialized", async () => {
@@ -291,9 +766,9 @@ test("concurrent commitAndAdvance calls are serialized", async () => {
},
},
});
- core.transformContext([user("one"), user("two")]);
- const first = core.commitAndAdvance();
- const second = core.commitAndAdvance();
+ const branch = branchOf(user("one"), user("two"));
+ const first = core.commitAndAdvance(branch);
+ const second = core.commitAndAdvance(branch);
assert.equal(await second, false);
release();
assert.equal(await first, true);
@@ -303,13 +778,15 @@ test("concurrent commitAndAdvance calls are serialized", async () => {
test("handleBeforeCompact returns OV summary and resets boundary", async () => {
const { core } = makeCore({ overviews: ["compact overview"] });
core.restore([
- { type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: { coveredUserTurns: 2, overview: "old", pendingTokens: 50 } },
+ { type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: { coveredThroughEntryId: "m1", coveredUserTurns: 2, overview: "old", pendingTokens: 50 } },
]);
- const result = await core.handleBeforeCompact({ firstKeptEntryId: "entry-3", tokensBefore: 1234 });
+ const branch = branchOf(user("one"), assistant("a"));
+ const result = await core.handleBeforeCompact({ firstKeptEntryId: "entry-3", tokensBefore: 1234 }, branch);
assert.equal(result.compaction.firstKeptEntryId, "entry-3");
assert.equal(result.compaction.tokensBefore, 1234);
assert.equal(result.compaction.details.source, "openviking");
assert.match(result.compaction.summary, /compact overview/);
+ assert.equal(core.state.coveredThroughEntryId, "");
assert.equal(core.state.coveredUserTurns, 0);
assert.equal(core.state.pendingTokens, 0);
});
@@ -320,19 +797,247 @@ test("handleBeforeCompact fail-opens without firstKeptEntryId or overview", asyn
assert.equal(await core.handleBeforeCompact({ firstKeptEntryId: "x", tokensBefore: 1 }), undefined);
});
+test("handleBeforeCompact uses pi's keep position, archives all messages, and honours cancellation", async () => {
+ const branch = branchOf(user("one"), assistant("a"), user("two"));
+ const normal = makeCore({ overviews: ["native overview"] });
+ const result = await normal.core.handleBeforeCompact(
+ { firstKeptEntryId: "pi-kept", tokensBefore: 44, signal: new AbortController().signal },
+ branch,
+ );
+ assert.equal(result.compaction.firstKeptEntryId, "pi-kept");
+ assert.equal(normal.calls.lastCommitOpts.keepRecentCount, 0);
+ assert.equal(normal.calls.lastSyncBranch, branch);
+
+ const cancelled = makeCore();
+ const controller = new AbortController();
+ controller.abort();
+ assert.equal(await cancelled.core.handleBeforeCompact(
+ { firstKeptEntryId: "pi-kept", signal: controller.signal }, branch,
+ ), undefined);
+ assert.equal(cancelled.calls.committed, 0);
+});
+
+test("handleBeforeCompact cancels after an in-flight overview read", async () => {
+ const controller = new AbortController();
+ const { core } = makeCore({
+ io: { readArchiveOverview: async () => { controller.abort(); return "too late"; } },
+ });
+ assert.equal(await core.handleBeforeCompact(
+ { firstKeptEntryId: "pi-kept", signal: controller.signal }, branchOf(user("one")),
+ ), undefined);
+ assert.equal(core.state.overview, "");
+ assert.ok(core.state.pendingArchive);
+});
+
+test("handleBeforeCompact catches transport failures and returns control to pi", async () => {
+ const { core, calls } = makeCore({ io: { commit: async () => { throw new Error("offline"); } } });
+ assert.equal(await core.handleBeforeCompact(
+ { firstKeptEntryId: "pi-kept" }, branchOf(user("one")),
+ ), undefined);
+ assert.ok(calls.logs.some((line) => /native compaction archive failed.*offline/.test(line)));
+});
+
+test("native compaction does not start another archive while a takeover summary is pending", async () => {
+ const branch = branchOf(user("one"), user("two"));
+ const { core, calls } = makeCore({ overviews: ["", "", ""] });
+ await core.onTurnSynced(120, branch);
+ assert.ok(core.state.pendingArchive);
+ assert.equal(await core.handleBeforeCompact({ firstKeptEntryId: "pi-kept" }, branch), undefined);
+ assert.equal(calls.committed, 1);
+});
+
+test("a native archive not summarized in time leaves the takeover boundary alone", async () => {
+ const branch = branchOf(user("one"), assistant("a"));
+ const first = makeCore({ overviews: ["", "", ""] });
+ first.core.restore([{ type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: {
+ coveredThroughEntryId: branch[0].id, coveredUserTurns: 1, overview: "old overview", pendingTokens: 20,
+ archiveUri: "viking://user/x/sessions/s/history/archive_000", historyUri: "viking://user/x/sessions/s/history",
+ } }]);
+ assert.equal(await first.core.handleBeforeCompact({ firstKeptEntryId: "pi-kept" }, branch), undefined);
+ assert.equal(first.core.state.pendingArchive.nativeCompaction, true);
+ // Pi may still cancel or fail its own compaction, so the boundary stays.
+ assert.equal(first.core.state.coveredThroughEntryId, branch[0].id);
+ assert.equal(first.core.state.overview, "old overview");
+
+ const resumed = makeCore({ overviews: ["late native overview"] });
+ resumed.core.restore([{ type: "custom", customType: TAKEOVER_ENTRY_TYPE, data: first.core.persistedState() }]);
+ assert.equal(await resumed.core.onTurnSynced(5, branch), false);
+ assert.equal(resumed.calls.committed, 0);
+ assert.equal(resumed.core.state.pendingArchive, null);
+ assert.equal(resumed.core.state.coveredThroughEntryId, branch[0].id);
+ // The recovery hint keeps naming the archive the current overview came from.
+ assert.equal(resumed.core.state.archiveUri, "viking://user/x/sessions/s/history/archive_000");
+});
+
+test("turn_end never sleeps waiting for a summary", async () => {
+ const { core, calls } = makeCore({ overviews: [""] });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.equal(await core.onTurnSynced(5, branch), false);
+ assert.equal(await core.onTurnSynced(5, branch), false);
+ assert.deepEqual(calls.slept, []);
+ assert.equal(calls.overviewUris.length, 3);
+ assert.equal(calls.committed, 1);
+});
+
+test("a pending archive that is terminal without a summary is dropped once", async () => {
+ const asked = [];
+ const { core, calls } = makeCore({
+ overviews: [""],
+ // `.done` without Working Memory: the server has it disabled.
+ io: { archiveState: async (uri) => { asked.push(uri); return "completed"; } },
+ });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.ok(core.state.pendingArchive);
+ assert.equal(await core.onTurnSynced(10, branch), false);
+ assert.deepEqual(asked, ["viking://user/x/sessions/s/history/archive_001"]);
+ assert.equal(core.state.pendingArchive, null);
+ assert.equal(core.state.coveredThroughEntryId, "");
+ // Its frozen pressure is spent, so the next archive waits for fresh pressure.
+ assert.equal(core.state.pendingTokens, 10);
+ assert.equal(await core.onTurnSynced(10, branch), false);
+ assert.equal(calls.committed, 1);
+ assert.ok(calls.logs.some((line) => /completed without Working Memory/.test(line)));
+});
+
+test("a summary that lands just as its archive completes still advances", async () => {
+ const { core } = makeCore({
+ overviews: ["", "", "late but ready"],
+ io: { archiveState: async () => "completed" },
+ });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ assert.equal(await core.onTurnSynced(0, branch), true);
+ assert.equal(core.state.overview, "late but ready");
+});
+
+test("a pending archive is kept while it is pending or cannot be asked", async () => {
+ for (const status of ["pending", null]) {
+ const { core } = makeCore({ overviews: [""], io: { archiveState: async () => status } });
+ const branch = branchOf(user("one"), user("two"));
+ await core.onTurnSynced(120, branch);
+ await core.onTurnSynced(0, branch);
+ assert.ok(core.state.pendingArchive, String(status));
+ }
+});
+
+test("a pending archive whose boundary left the context is dropped without a read", async () => {
+ const initial = branchOf(user("one"), assistant("a"), user("two"));
+ const { core, calls } = makeCore({ overviews: [""] });
+ assert.equal(await core.onTurnSynced(120, initial), false);
+ const reads = calls.overviewUris.length;
+ const forked = [initial[0], ...branchOf(assistant("other"), user("two"))];
+ assert.equal(await core.onTurnSynced(0, forked), false);
+ assert.equal(calls.overviewUris.length, reads);
+ assert.equal(core.state.pendingArchive, null);
+});
+
+test("resumePending advances before the next prompt", async () => {
+ const { core, calls } = makeCore({ overviews: ["", "ready"] });
+ const branch = branchOf(user("one"), assistant("a"), user("two"), assistant("b"));
+ assert.equal(await core.onTurnSynced(120, branch), false);
+ const next = [...branch, ...branchOf(user("three"))];
+ assert.equal(await core.resumePending(next), true);
+ assert.equal(calls.committed, 1);
+ const out = core.transformContext(contextOf(next), next);
+ assert.deepEqual(out.slice(1).map((message) => message.content), ["two", "b", "three"]);
+ assert.equal(await core.resumePending(next), false);
+});
+
+test("the drain and the commit only get what is left of the handler budget", async () => {
+ let clock = 1_000_000;
+ const flushBudgets = [];
+ const { core, calls } = makeCore({
+ io: {
+ now: () => clock,
+ flush: async (budgetMs) => { flushBudgets.push(budgetMs); clock += 4_000; return true; },
+ },
+ });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch, { deadline: clock + 25_000 }), true);
+ assert.deepEqual(flushBudgets, [15_000]);
+ assert.equal(calls.lastCommitOpts.timeoutMs, 16_000);
+});
+
+test("a commit that would run past the handler deadline is postponed", async () => {
+ let clock = 1_000_000;
+ const { core, calls } = makeCore({
+ io: { now: () => clock, flush: async () => { clock += 18_000; return true; } },
+ });
+ const branch = branchOf(user("one"), user("two"));
+ assert.equal(await core.onTurnSynced(120, branch, { deadline: clock + 25_000 }), false);
+ assert.equal(calls.committed, 0);
+ assert.equal(core.state.pendingTokens, 120);
+ assert.ok(calls.logs.some((line) => /budget spent before commit/.test(line)));
+});
+
+test("native compaction polls only until the handler deadline", async () => {
+ let clock = 1_000_000;
+ const deadline = clock + 25_000;
+ const branch = branchOf(user("one"), assistant("a"), user("two"));
+ const { core, calls } = makeCore({
+ overviews: [""],
+ config: { takeoverOverviewPollMs: 2_000, takeoverOverviewPollMax: 15 },
+ io: { now: () => clock, sleep: async (ms) => { calls.slept.push(ms); clock += ms; } },
+ });
+ restoreBoundary(core, branch[1].id);
+ assert.equal(await core.handleBeforeCompact({ firstKeptEntryId: "pi-kept" }, branch, { deadline }), undefined);
+ assert.ok(clock <= deadline);
+ assert.ok(calls.slept.length < 14);
+ assert.equal(core.state.coveredThroughEntryId, branch[1].id);
+ assert.equal(core.state.pendingArchive.nativeCompaction, true);
+});
+
+test("recovery hint is outside the overview budget and requires list plus read", () => {
+ const archiveUri = "viking://user/x/sessions/s/history/archive_001";
+ const branch = branchOf(user("one"), assistant("a"), user("two"));
+ const both = makeCore({ tools: ["openviking_list", "openviking_read", "openviking_grep"] });
+ restoreBoundary(both.core, branch[1].id, {
+ overview: "x".repeat(1000), archiveUri, historyUri: "viking://user/x/sessions/s/history",
+ });
+ const out = both.core.transformContext(contextOf(branch), branch);
+ assert.match(out[0].content, /openviking_list/);
+ assert.match(out[0].content, /openviking_read/);
+ assert.match(out[0].content, /offset and limit/);
+ assert.match(out[0].content, /captured historical messages/);
+ assert.ok(out[0].content.length > 1000);
+
+ const missingRead = makeCore({ tools: ["openviking_list"] });
+ restoreBoundary(missingRead.core, branch[1].id, {
+ overview: "summary", archiveUri, historyUri: "viking://user/x/sessions/s/history",
+ });
+ const noHint = missingRead.core.transformContext(contextOf(branch), branch);
+ assert.doesNotMatch(noHint[0].content, /openviking_list/);
+});
+
+test("native compaction summary uses the same recovery hint", async () => {
+ const { core } = makeCore({
+ tools: ["openviking_list", "openviking_read"],
+ overviews: ["compact overview"],
+ });
+ const result = await core.handleBeforeCompact(
+ { firstKeptEntryId: "pi-kept", tokensBefore: 100 }, branchOf(user("one")),
+ );
+ assert.match(result.compaction.summary, /captured historical messages/);
+ assert.match(result.compaction.summary, /openviking_list/);
+ assert.match(result.compaction.summary, /openviking_read/);
+});
+
test("disabled takeover is a passthrough", async () => {
const { core, calls } = makeCore({ config: { takeoverEnabled: false } });
- const messages = [user("one"), user("two")];
- assert.equal(core.transformContext(messages), messages);
- assert.equal(await core.onTurnSynced(999), false);
- assert.equal(await core.commitAndAdvance(), false);
+ const branch = branchOf(user("one"), user("two"));
+ const messages = contextOf(branch);
+ assert.equal(core.transformContext(messages, branch), messages);
+ assert.equal(await core.onTurnSynced(999, branch), false);
+ assert.equal(await core.commitAndAdvance(branch), false);
assert.equal(await core.handleBeforeCompact({ firstKeptEntryId: "x" }), undefined);
assert.equal(calls.committed, 0);
});
test("shutdown persists deduped state once", async () => {
const { core, calls } = makeCore({ watermark: 9 });
- core.transformContext([user("one")]);
+ core.transformContext([user("one")], []);
await core.shutdown();
await core.shutdown();
assert.equal(calls.persisted.length, 1);
diff --git a/examples/pi-coding-agent-extension/tests/takeover-session-manager.test.mjs b/examples/pi-coding-agent-extension/tests/takeover-session-manager.test.mjs
new file mode 100644
index 000000000..33c17e421
--- /dev/null
+++ b/examples/pi-coding-agent-extension/tests/takeover-session-manager.test.mjs
@@ -0,0 +1,158 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+import { existsSync } from "node:fs";
+import { createRequire } from "node:module";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { TakeoverCore } from "../lib/takeover-core.mjs";
+
+const EXTENSION_DIR = dirname(fileURLToPath(import.meta.url));
+
+/**
+ * pi's real SessionManager, wherever pi is installed; the version-matrix gate
+ * points PI_SESSION_MANAGER at a specific pi. The unit suite's hand-built
+ * branches cannot prove that the boundary survives what a live session puts
+ * between turns — takeover's own state entries, pi's compaction entries, tree
+ * navigation — so these checks drive the real session projection and skip
+ * only when no pi is resolvable.
+ */
+function findSessionManager() {
+ if (process.env.PI_SESSION_MANAGER !== undefined) {
+ const explicit = process.env.PI_SESSION_MANAGER;
+ return explicit && existsSync(explicit) ? explicit : "";
+ }
+ const candidates = [];
+ try {
+ candidates.push(createRequire(join(EXTENSION_DIR, "package.json"))
+ .resolve("@earendil-works/pi-coding-agent/dist/core/session-manager.js"));
+ } catch {
+ // not installed locally
+ }
+ candidates.push(
+ "/opt/homebrew/lib/node_modules/@earendil-works/pi-coding-agent/dist/core/session-manager.js",
+ "/usr/local/lib/node_modules/@earendil-works/pi-coding-agent/dist/core/session-manager.js",
+ );
+ return candidates.find((path) => path && existsSync(path)) ?? "";
+}
+
+const SESSION_MANAGER_PATH = findSessionManager();
+
+async function openSession(t, config = {}) {
+ if (!SESSION_MANAGER_PATH) {
+ t.skip("pi SessionManager not resolvable");
+ return null;
+ }
+ const { SessionManager } = await import(SESSION_MANAGER_PATH);
+ const sm = SessionManager.inMemory("/tmp");
+ const logs = [];
+ let archives = 0;
+ const core = new TakeoverCore({
+ config: {
+ takeoverEnabled: true,
+ takeoverTokenThreshold: 100,
+ takeoverKeepRecentTurns: 1,
+ takeoverOverviewPollMs: 0,
+ takeoverOverviewPollMax: 1,
+ ...config,
+ },
+ io: {
+ syncBranch: async () => ({ added: 0, tokens: 0, allDelivered: true, permanentFailures: 0 }),
+ flush: async () => true,
+ commit: async () => ({
+ status: "accepted",
+ archived: true,
+ archive_uri: `viking://user/u/sessions/pi-s/history/archive_${String(++archives).padStart(3, "0")}`,
+ }),
+ readArchiveOverview: async () => "OV SUMMARY",
+ captureCount: (slice) => slice.filter((entry) =>
+ entry?.type === "message" && entry.message?.role !== "system").length,
+ // Exactly how takeover.ts persists: a custom entry at the session leaf.
+ persistEntry: (type, data) => sm.appendCustomEntry(type, data),
+ getWatermark: () => sm.getEntries().length,
+ log: (message) => logs.push(message),
+ },
+ });
+ let clock = 1_000;
+ const text = (value) => [{ type: "text", text: value }];
+ /** One prompt: the user message, the context hook, the reply, then turn_end. */
+ async function turn(label) {
+ const userId = sm.appendMessage({ role: "user", content: text(label), timestamp: clock++ });
+ // pi >= 0.87 hides system messages from the `context` hook.
+ const context = sm.buildSessionContext().messages.filter((message) => message.role !== "system");
+ const out = core.transformContext(context, sm.getBranch());
+ sm.appendMessage({
+ role: "assistant", content: text(`ack ${label}`), timestamp: clock++,
+ api: "test", provider: "test", model: "test", usage: {}, stopReason: "stop",
+ });
+ await core.onTurnSynced(150, () => sm.getBranch());
+ return { userId, trimmed: out.length < context.length, out };
+ }
+ return { sm, core, logs, turn };
+}
+
+test("takeover state entries between turns do not reset the boundary", async (t) => {
+ const session = await openSession(t);
+ if (!session) return;
+ const trimmed = [];
+ for (let i = 1; i <= 5; i++) {
+ if ((await session.turn(`turn ${i}`)).trimmed) trimmed.push(i);
+ }
+ assert.deepEqual(trimmed, [3, 4, 5]);
+ assert.ok(session.sm.getBranch().some((entry) => entry.type === "custom"));
+ assert.equal(session.logs.some((line) => /boundary reset/.test(line)), false);
+});
+
+test("takeover keeps trimming after pi's own compaction", async (t) => {
+ const session = await openSession(t);
+ if (!session) return;
+ const trimmed = [];
+ for (let i = 1; i <= 8; i++) {
+ const { userId, trimmed: cut } = await session.turn(`turn ${i}`);
+ if (cut) trimmed.push(i);
+ if (i === 4) {
+ assert.ok(await session.core.handleBeforeCompact({ firstKeptEntryId: userId, tokensBefore: 1 }, () => session.sm.getBranch()));
+ session.sm.appendCompaction("pi summary", userId, 1);
+ }
+ }
+ // Turn 5 is the first after the compaction absorbed the boundary.
+ assert.deepEqual(trimmed, [3, 4, 6, 7, 8]);
+});
+
+test("the boundary follows tree navigation away from and back to the covered branch", async (t) => {
+ const session = await openSession(t);
+ if (!session) return;
+ const { sm, core, turn } = session;
+ const first = await turn("turn 1");
+ await turn("turn 2");
+ assert.equal((await turn("turn 3")).trimmed, true);
+ const coveredLeaf = sm.getLeafId();
+
+ // Fork from the first prompt: the covered prefix is not on this branch.
+ sm.branch(first.userId);
+ const forked = sm.buildSessionContext().messages;
+ assert.equal(core.transformContext(forked, sm.getBranch()), forked);
+
+ sm.branch(coveredLeaf);
+ const back = sm.buildSessionContext().messages;
+ assert.ok(core.transformContext(back, sm.getBranch()).length < back.length);
+});
+
+test("a branch summary /tree leaves at the boundary survives the cut", async (t) => {
+ const session = await openSession(t);
+ if (!session) return;
+ const { sm, turn } = session;
+ await turn("turn 1");
+ await turn("turn 2");
+ const third = await turn("turn 3");
+ assert.equal(third.trimmed, true);
+ const boundary = session.core.state.coveredThroughEntryId;
+ // turn_end of turn 3 advanced the boundary to just in front of it.
+ assert.equal(sm.getBranch().find((entry) => entry.id === third.userId).parentId, boundary);
+
+ // /tree back to "turn 3", the first kept turn, with a summary of what is
+ // abandoned: pi hangs it off the boundary entry, in front of the retry.
+ sm.branchWithSummary(boundary, "abandoned turn 3");
+ const retry = await turn("turn 3, retried");
+ assert.equal(retry.trimmed, true);
+ assert.deepEqual(retry.out.map((message) => message.role), ["user", "branchSummary", "user"]);
+});
diff --git a/examples/pi-coding-agent-extension/tests/takeover-system-tools.test.mjs b/examples/pi-coding-agent-extension/tests/takeover-system-tools.test.mjs
new file mode 100644
index 000000000..12b92dcba
--- /dev/null
+++ b/examples/pi-coding-agent-extension/tests/takeover-system-tools.test.mjs
@@ -0,0 +1,194 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+import { existsSync } from "node:fs";
+import { createRequire } from "node:module";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import {
+ OVERVIEW_MARKER,
+ TAKEOVER_ENTRY_TYPE,
+ TakeoverCore,
+} from "../lib/takeover-core.mjs";
+
+const EXTENSION_DIR = dirname(fileURLToPath(import.meta.url));
+
+/**
+ * pi's real transcript helpers, wherever pi is installed. The stable extension
+ * does not depend on pi (only `@modelcontextprotocol/client`), so this resolves
+ * against a local install, the global homebrew prefix, or an explicit override.
+ * When none is found the tool-declaration checks skip rather than testing a
+ * reimplementation of pi's own merge — the pure ordering checks in
+ * takeover-core.test.mjs still run everywhere.
+ *
+ * pi < 0.86 has no `system` role and ships no `utils/transcript.js`, so this
+ * also skips there, which is correct: there are no covered system messages to
+ * preserve on 0.80.3.
+ */
+function findTranscriptModule() {
+ // An explicit override is also used by the version-matrix gate. Do not fall
+ // back to a different globally installed pi when that version has no helper.
+ if (process.env.PI_PI_AI_TRANSCRIPT !== undefined) {
+ const explicit = process.env.PI_PI_AI_TRANSCRIPT;
+ return explicit && existsSync(explicit) ? explicit : "";
+ }
+ const candidates = [];
+ for (const base of [
+ "@earendil-works/pi-coding-agent/node_modules/@earendil-works/pi-ai/dist/utils/transcript.js",
+ "@earendil-works/pi-ai/dist/utils/transcript.js",
+ ]) {
+ try {
+ candidates.push(createRequire(join(EXTENSION_DIR, "package.json")).resolve(base));
+ } catch {
+ // not installed at this location
+ }
+ }
+ candidates.push(
+ "/opt/homebrew/lib/node_modules/@earendil-works/pi-coding-agent/node_modules/@earendil-works/pi-ai/dist/utils/transcript.js",
+ "/usr/local/lib/node_modules/@earendil-works/pi-coding-agent/node_modules/@earendil-works/pi-ai/dist/utils/transcript.js",
+ );
+ return candidates.find((path) => path && existsSync(path)) ?? "";
+}
+
+const TRANSCRIPT_PATH = findTranscriptModule();
+
+/**
+ * Run the takeover cut over `messages` as a pi >= 0.86 branch (every message,
+ * system ones included, is a `message` entry), with the covered prefix ending
+ * right before the user turn `keptText`. Asserts the cut really happened, so a
+ * passing tool check can never come from an untrimmed context.
+ */
+function cut(messages, keptText) {
+ const branch = messages.map((message, i) => ({ id: `e${i}`, type: "message", message }));
+ const kept = messages.findIndex((message) => message.role === "user" && message.content === keptText);
+ const out = makeCore(branch[kept - 1].id).transformContext(messages, branch);
+ assert.ok(out.some((message) => String(message.content).startsWith(OVERVIEW_MARKER)));
+ assert.ok(out.length < messages.length);
+ return out;
+}
+
+function makeCore(coveredThroughEntryId, overrides = {}) {
+ const core = new TakeoverCore({
+ config: {
+ takeoverEnabled: true,
+ takeoverTokenThreshold: 100,
+ takeoverKeepRecentTurns: 1,
+ takeoverOverviewBudget: 1000,
+ takeoverOverviewPollMs: 1,
+ takeoverOverviewPollMax: 3,
+ ...overrides,
+ },
+ });
+ core.restore([
+ {
+ type: "custom",
+ customType: TAKEOVER_ENTRY_TYPE,
+ data: { coveredThroughEntryId, coveredUserTurns: 1, overview: "archived first turn", pendingTokens: 0 },
+ },
+ ]);
+ return core;
+}
+
+function sys(content, extra = {}) {
+ return { role: "system", content, timestamp: 0, ...extra };
+}
+let clock = 0;
+function user(content) {
+ return { role: "user", content, timestamp: ++clock };
+}
+function assistant(content) {
+ return { role: "assistant", content, timestamp: ++clock };
+}
+
+test("transformContext preserves tool declarations, replacements, sections and appended instructions", async (t) => {
+ if (!TRANSCRIPT_PATH) {
+ t.skip("pi @earendil-works/pi-ai transcript helpers not resolvable (skips on pi < 0.86)");
+ return;
+ }
+ const { getCurrentSystemMessage, getCurrentSystemPrompt, getCurrentTools } = await import(TRANSCRIPT_PATH);
+
+ const readTool = { name: "read", description: "read a file", parameters: { type: "object" } };
+ const searchTool = { name: "openviking_search", description: "search", parameters: { type: "object" } };
+ const updatedReadTool = { name: "read", description: "read safely", parameters: { type: "object" } };
+
+ const messages = [
+ // The leading system message: base prompt + the initial tools, exactly the
+ // shape @earendil-works/pi-ai's normalizeContext folds Context into.
+ sys("BASE PROMPT", { toolsAdded: [readTool], sections: { style: "be terse", obsolete: "drop me" } }),
+ user("first"),
+ assistant("answer"),
+ // A mid-conversation tool addition and a section update, both in the covered
+ // region — this is what a naive slice would strip.
+ sys("APPENDED INSTRUCTION", {
+ toolsRemoved: [{ name: "read" }],
+ toolsAdded: [updatedReadTool, searchTool],
+ sections: { style: "be exact", obsolete: null, extra: "note" },
+ }),
+ user("second"),
+ assistant("answer 2"),
+ ];
+
+ const before = new Set(getCurrentTools(messages).map((tt) => tt.name));
+ assert.deepEqual([...before].sort(), ["openviking_search", "read"]);
+
+ const out = cut(messages, "second");
+ const after = new Set(getCurrentTools(out).map((tt) => tt.name));
+
+ // The model's tools are unchanged by the takeover cut.
+ assert.deepEqual([...after].sort(), [...before].sort());
+ assert.equal(getCurrentTools(out).find((tool) => tool.name === "read").description, "read safely");
+
+ // The base prompt and both section updates survive the merge, too.
+ const prompt = getCurrentSystemPrompt(out);
+ assert.match(prompt, /BASE PROMPT/);
+ assert.match(prompt, /APPENDED INSTRUCTION/);
+ const systemState = getCurrentSystemMessage(out);
+ assert.deepEqual(systemState.sections, { style: "be exact", extra: "note" });
+ assert.equal(out.filter((message) => message.role === "system").length, 2);
+});
+
+test("transformContext preserves a mid-conversation tool removal", async (t) => {
+ if (!TRANSCRIPT_PATH) {
+ t.skip("pi @earendil-works/pi-ai transcript helpers not resolvable (skips on pi < 0.86)");
+ return;
+ }
+ const { getCurrentTools } = await import(TRANSCRIPT_PATH);
+
+ const readTool = { name: "read", description: "read", parameters: { type: "object" } };
+ const writeTool = { name: "write", description: "write", parameters: { type: "object" } };
+
+ const messages = [
+ sys("BASE", { toolsAdded: [readTool, writeTool] }),
+ user("first"),
+ assistant("answer"),
+ // `write` is removed mid-conversation, in the covered region.
+ sys("", { toolsRemoved: [{ name: "write" }] }),
+ user("second"),
+ assistant("answer 2"),
+ ];
+
+ assert.deepEqual(getCurrentTools(messages).map((tt) => tt.name), ["read"]);
+ const out = cut(messages, "second");
+ // Without preserving the removal, `write` would come back to life.
+ assert.deepEqual(getCurrentTools(out).map((tt) => tt.name), ["read"]);
+});
+
+test("dropping covered system messages loses tools (regression witness)", async (t) => {
+ if (!TRANSCRIPT_PATH) {
+ t.skip("pi @earendil-works/pi-ai transcript helpers not resolvable (skips on pi < 0.86)");
+ return;
+ }
+ const { getCurrentTools } = await import(TRANSCRIPT_PATH);
+ const readTool = { name: "read", description: "read", parameters: { type: "object" } };
+ const messages = [
+ sys("BASE", { toolsAdded: [readTool] }),
+ user("first"),
+ assistant("answer"),
+ user("second"),
+ ];
+ // What the pre-fix slice produced: covered system messages gone entirely.
+ const naive = messages.slice(3);
+ assert.deepEqual(getCurrentTools(naive), []);
+ // The fix keeps them, so pi still sees the tool.
+ const out = cut(messages, "second");
+ assert.deepEqual(getCurrentTools(out).map((tt) => tt.name), ["read"]);
+});