Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Python agentagent-osaiai-agentsai-partnerchatopsclaudeclaude-codecodexdevtoolsdiscord-botlark-botllmlocal-firstlong-term-memoryopencodeslack-botvibe-codingvibeywechat
"""C-5 transcript store over the ``messages`` and ``agent_events`` tables.
The Avibe Agent's model context is Avibe's own rows, with no second copy
(``docs/plans/agent-core-contracts/transcript.md``). A row is in a Session's
context exactly when its ``context_seq`` is non-null:
* inputs are existing ``messages`` rows (``user``, ``harness``,
``agent_initiated``, ``annotation``); consuming one sets its ``context_seq``
and ``content_json.model`` once;
* responses are new ``assistant`` / ``result`` / ``error`` rows with
``content_json.model`` (with the C-9 request facts when the loop gives them)
and a rendered ``content_text``;
* tool results, checkpoints, context edits, and hook and guard state are new
``agent_events`` rows with ``visibility='context'``;
* audit rows (C-9: a conversation attempt's usage, a checkpoint turn with its
own attempts) are ``agent_events`` rows with ``visibility='audit'`` and no
``context_seq``: never context.
Every entry carries its row's ``created_at`` as epoch seconds, as written and
as loaded (C-9 reads a response's to tell a cold provider cache after a
restart). ``append_payloads`` commits several payload rows in one transaction.
A checkpoint row rewrites its Session's ``session_digests`` row in the same one:
the mind, memo, and now that other Sessions recall it by.
Allocation. Every write takes SQLite's writer lock before its first read
(``reserve_write_lock``) and runs under a per-Session lock, so
``max(context_seq over both tables) + 1`` cannot be allocated twice, across
tasks or processes; the partial unique indexes back the rule.
Attribution. Inserted rows take ``platform`` and ``scope_id`` from the Turn's
initial input: the latest consumed input names its Turn through its accepted
Delivery, and that Turn's initial Delivery names the input row. That is the
channel the Turn answers, where ``persist_agent_message`` attributes the same
output, and a steer from another surface does not re-home the reply. An input
without a Delivery link stands for itself. ``agent_events.turn_id`` is that
Turn.
Fork (C-10 ``fork.md``). A child inherits its source's context rows up to its
fork point, read from the top-level Session metadata ``fork_source_session_id``
and ``fork_source_context_seq`` and followed through the source's own fork. The
point is resolved once, when the fork is reserved (``resolve_fork_point``), and
is settled, so a child never inherits an open call; rows that receive a
``context_seq`` later can never move into or out of the prefix. A fork without
that key (a released fork, or one from another backend) inherits nothing. The
child's own rows continue from the point plus one.
Display. A response row commits with the display text its renderer gives it.
Its display columns (``content_text``, the display keys of ``content_json``,
type ``result`` or ``error``) are written again when the dispatcher delivers
it, exactly as for the other backends' rows (``persist_agent_message`` with
``existing_row_id``). The transcript payload, ``content_json.model``, never
changes after commit. A final whose ``final_outcome`` failed commits as
``error``.
Generated images. A committed row that names a generated image (a hosted
record's ``images``) registers each file in ``media_objects`` in its own
transaction, as an input snapshot is registered: the media sweep keeps exactly
the files some row registers (``vibey-hosted-tools.md`` §7.3).
Cancellation. A worker thread cannot be interrupted, so a cancelled write keeps
the Session's lock until its transaction settles and only then raises: when a
write call returns or raises, the rows already show whether it committed.
"""
from __future__ import annotations
import asyncio
import json
from dataclasses import dataclass, replace
from datetime import datetime, timezone
from typing import Any, Callable, Literal, Mapping, Optional, Sequence, TypeVar
from sqlalchemy import and_, bindparam, func, or_, select, text
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.engine import Connection, Engine
from core.agent_core.harness.fork import fork_point, latest_cut
from core.agent_core.harness.projection import open_tool_calls
from core.agent_core.harness.store import ContextEntry, EntryKind
from core.agent_core.messages import (
AssistantMessage,
HostedImage,
HostedRecord,
HostedToolBlock,
TextBlock,
ToolResultMessage,
has_reply,
UserMessage,
message_from_dict,
message_to_dict,
require_json_value,
)
from storage import agent_events_service, media_service, messages_service
from storage.agent_session_rows import reserve_write_lock
from storage.message_deliveries import TURN_OWNER_STATES
from storage.models import (
MESSAGE_TURN_ID_SQL,
agent_events,
agent_sessions,
message_deliveries,
messages,
scopes,
session_digests,
session_turns,
)
from storage.vibey_recovery import VIBEY_RECOVERY_INPUT_PREDICATE
INPUT_TYPES = ("user", "harness", "agent_initiated", "annotation")
# A final response is ``result``, or ``error`` when ``final_outcome`` says it failed:
# the row carries its outcome, as the other backends' terminal rows do. A final with
# nothing to show (a silent reply) is the hidden response type ``assistant``: context,
# but no transcript row, inbox reply, or unread result, as the other backends persist
# nothing visible for it. A non-final reply that pending input superseded is a reply
# all the same, ``result`` (docs/plans/reply-and-turn-end.md); released versions
# committed it as ``output``, which stays a response type.
FINAL_TYPES = ("result", "error")
RESPONSE_TYPES = ("assistant", "output", *FINAL_TYPES)
CONTEXT_VISIBILITY = "context"
AUDIT_VISIBILITY = "audit"
AuditKind = Literal["attempt", "fork_turn"]
_AUDIT_EVENT_TYPE: dict[str, str] = {"attempt": "model_attempt", "fork_turn": "fork_turn"}
# Only Vibey writes context rows; the Session's routed backend may change mid-Turn.
CONTEXT_WRITER = "vibey"
PAYLOAD_VERSION = 1
# Two bounded index seeks recover all route clocks; no per-Turn audit-history scan.
CACHE_ATTEMPTS_SQL = """
SELECT CASE WHEN json_valid(content_json) THEN json_extract(content_json, '$.cache_state') END
FROM agent_events
WHERE session_id = :session_id AND event_type = :event_type
ORDER BY created_at DESC, id DESC LIMIT 1
"""
PayloadKind = Literal["compaction", "context_edit", "agent_state", "replay_refusal"]
_EVENT_TYPE_BY_KIND: dict[str, str] = {
"tool_result": "tool_result",
"compaction": "context_compaction",
"context_edit": "context_edit",
"agent_state": "agent_state",
"replay_refusal": "replay_refusal",
}
_KIND_BY_EVENT_TYPE = {event_type: kind for kind, event_type in _EVENT_TYPE_BY_KIND.items()}
# ``render(message, final=..., replied=..., images=...)``: a response row's commit-time display text; ``replied`` is
# the decision ``final_outcome`` takes, and ``images`` those a final delivers with its reply.
DisplayRenderer = Callable[..., str]
_T = TypeVar("_T")
FinalOutcome = Literal["completed", "failed"]
def final_outcome(message: AssistantMessage, *, replied: Optional[bool] = None) -> FinalOutcome:
"""The Turn outcome of a committed final response.
The one rule, shared by the row's type at commit and live settlement: a final
that does not reply, or explicitly stopped at its output limit, failed. Whether
it replies is the loop's decision (``has_reply``), taken as given; without one
(``replied`` None), the response alone decides. Partial text must not turn a
provider's incomplete final into a successful result. Pending-input responses
are not finals. Failures only a live run can observe after the commit are
applied by the live caller.
"""
if replied is None:
replied = has_reply(message)
return "completed" if replied and message.stop_reason != "length" else "failed"
def _response_type(message: AssistantMessage, *, final: bool, display_text: str, replied: bool, superseded: bool) -> str:
if not final:
return "result" if superseded and display_text.strip() else "assistant"
if final_outcome(message, replied=replied) == "failed":
return "error"
return "result" if display_text.strip() else "assistant"
#: What a response's display puts between two text blocks another block (a tool or hosted call, thinking) separates.
PARAGRAPH_BREAK = "\n\n"
def text_parts(message: AssistantMessage) -> list[tuple[str, TextBlock]]:
"""A response's text blocks as its display joins them: each non-empty block with what goes before it.
Adjacent blocks are one span of prose and join as written: a provider may split
its answer into many, as Anthropic does at every cited span. A paragraph break
goes only where another block came between two of them.
"""
parts: list[tuple[str, TextBlock]] = []
separated = False
for block in message.content:
if not isinstance(block, TextBlock):
separated = True
elif block.text:
parts.append((PARAGRAPH_BREAK if parts and separated else "", block))
separated = False
return parts
def render_text(
message: AssistantMessage, *, final: bool = False, replied: bool = True, images: Sequence[HostedImage] = ()
) -> str:
"""Default display copy of a response: its text blocks, verbatim, in order, joined as ``text_parts`` says."""
return "".join(separator + block.text for separator, block in text_parts(message))
class TranscriptError(RuntimeError):
"""A write the context rules refuse, or rows that cannot form a context."""
@dataclass(frozen=True)
class _TurnOrigin:
platform: str
scope_id: Optional[str]
turn_id: Optional[str]
agent_name: Optional[str]
backend: Optional[str]
class SQLiteTranscriptStore:
"""``TranscriptStore`` over Avibe's tables.
``render(message, final=...)`` produces a response row's commit-time
``content_text``; the adapter supplies its display rendering, the default keeps
the text blocks.
"""
def __init__(self, engine: Engine, *, render: DisplayRenderer = render_text) -> None:
self._engine = engine
self._render = render
self._locks: dict[str, asyncio.Lock] = {}
def forget(self, session_id: str) -> None:
"""Drop an idle Session's write lock; the next write creates it again."""
lock = self._locks.get(session_id)
if lock is not None and not lock.locked():
del self._locks[session_id]
# --- TranscriptStore ----------------------------------------------------
async def load(self, session_id: str) -> Sequence[ContextEntry]:
return await asyncio.to_thread(self._load, session_id)
async def load_turn(self, session_id: str, turn_id: str) -> Sequence[ContextEntry]:
"""Load one durable Turn's own context, excluding older history and fork ancestry."""
return await asyncio.to_thread(self._load, session_id, turn_id=turn_id)
async def load_turn_response_metadata(
self, session_id: str, turn_id: str
) -> Sequence[tuple[Mapping[str, Any], Optional[AssistantMessage]]]:
"""Read response recovery facts through the Turn index, never tool outputs.
Newly written rows need only their small metadata. Older rows have no
recovery projection, so decode their original immutable response instead.
"""
def work() -> list[tuple[Mapping[str, Any], Optional[AssistantMessage]]]:
query = text(
"SELECT id, metadata_json, "
"CASE WHEN json_type(metadata_json, '$.native_search_recovery') IS NULL "
"THEN content_json END AS legacy_content "
"FROM messages INDEXED BY ix_messages_turn_id "
f"WHERE session_id = :session_id AND ({MESSAGE_TURN_ID_SQL}) = :turn_id "
"AND context_seq IS NOT NULL AND type IN :response_types ORDER BY context_seq"
).bindparams(bindparam("response_types", expanding=True))
result = []
with self._engine.connect() as conn:
rows = conn.execute(query, {
"session_id": session_id, "turn_id": turn_id, "response_types": RESPONSE_TYPES,
}).mappings()
for row in rows:
metadata = _json_object(row["metadata_json"], row["id"])
message = None
if row["legacy_content"] is not None:
model = _versioned(_json_object(row["legacy_content"], row["id"]).get("model"), row["id"])
message = _message(model.get("message"), AssistantMessage, row["id"])
result.append((metadata, message))
return result
return await asyncio.to_thread(work)
async def consume_input(self, session_id: str, message_id: str, message: UserMessage) -> ContextEntry:
if not isinstance(message, UserMessage):
raise TypeError("consume_input takes a UserMessage")
model = _canonical({"version": PAYLOAD_VERSION, "message": message_to_dict(message)}, "input")
def work(conn: Connection) -> ContextEntry:
row = conn.execute(
select(
messages.c.session_id,
messages.c.type,
messages.c.context_seq,
messages.c.content_json,
messages.c.created_at,
).where(messages.c.id == message_id)
).mappings().first()
if row is None or row["session_id"] != session_id:
raise TranscriptError(f"input {message_id} is not a message of Session {session_id}")
if row["type"] not in INPUT_TYPES:
raise TranscriptError(f"message {message_id} of type {row['type']!r} is not an input")
content = _json_object(row["content_json"], message_id)
seq = row["context_seq"]
if seq is not None:
# A retried commit of the same input is the committed entry; anything
# else would rewrite a context row after commit.
if content.get("model") != model:
raise TranscriptError(f"input {message_id} is already in the context")
else:
seq = _next_context_seq(conn, session_id)
conn.execute(
messages.update()
.where(messages.c.id == message_id, messages.c.context_seq.is_(None))
.values(context_seq=seq, content_json=json.dumps({**content, "model": model}), updated_at=_utc_now())
)
return ContextEntry(
session_id, seq, "input", message_id, message=message, payload=model, created_at=_epoch(row["created_at"])
)
return await self._write(session_id, work)
async def load_cache_attempts(self, session_id: str) -> Sequence[Mapping[str, Any]]:
"""Read only optional diagnostics, by the existing Session index and on a worker."""
from core.agent_core.harness.cache_metrics import CacheTracker
def read():
tracker = CacheTracker()
with self._engine.connect() as conn:
for event_type in ("model_attempt", "fork_turn"):
raw = conn.execute(text(CACHE_ATTEMPTS_SQL), {"session_id": session_id, "event_type": event_type}).scalar()
try:
records = json.loads(raw) if raw else []
if isinstance(records, list):
tracker.restore((), records)
except (ValueError, TypeError):
continue
return list(tracker.latest.values())
return await asyncio.to_thread(read)
async def append_response(
self,
session_id: str,
message: AssistantMessage,
*,
final: bool,
request: Optional[Mapping[str, Any]] = None,
cache: Optional[Mapping[str, Any]] = None,
replied: Optional[bool] = None,
images: Sequence[HostedImage] = (),
superseded: bool = False,
metadata: Optional[Mapping[str, Any]] = None,
agent_name: Optional[str] = None,
turn_id: Optional[str] = None,
) -> ContextEntry:
if not isinstance(message, AssistantMessage):
raise TypeError("append_response takes an AssistantMessage")
row_metadata = dict(metadata or {})
stored_message = _with_render_metadata(message, row_metadata)
response: dict[str, Any] = {"version": PAYLOAD_VERSION, "message": message_to_dict(stored_message)}
if request is not None:
response["request"] = dict(request) # C-9 ``ModelResponse.request``: the anchor's request facts
if cache is not None:
response["cache"] = dict(cache)
model = _canonical(response, "response")
if replied is None:
replied = has_reply(stored_message)
display_text = self._render(stored_message, final=final, replied=replied, images=tuple(images))
def work(conn: Connection) -> ContextEntry:
origin = _turn_origin(conn, session_id, agent_name, turn_id)
seq = _next_context_seq(conn, session_id)
row_metadata_for_write = dict(row_metadata)
if origin.turn_id:
row_metadata_for_write["turn_id"] = origin.turn_id
row = messages_service.append(
conn,
scope_id=origin.scope_id,
session_id=session_id,
platform=origin.platform,
author="agent",
source="agent",
author_name=origin.agent_name,
message_type=_response_type(
message, final=final, display_text=display_text, replied=replied, superseded=superseded
),
text=display_text,
content={"model": model},
metadata=row_metadata_for_write or None,
)
conn.execute(messages.update().where(messages.c.id == row["id"]).values(context_seq=seq))
_register_images(conn, origin.scope_id, session_id, stored_message)
return ContextEntry(
session_id, seq, "response", row["id"], message=replace(stored_message, transcript_row_id=row["id"]),
payload=model, created_at=_epoch(row["created_at"]),
)
return await self._write(session_id, work)
async def append_tool_result(
self,
session_id: str,
message: ToolResultMessage,
*,
details: Mapping[str, Any],
agent_name: Optional[str] = None,
turn_id: Optional[str] = None,
) -> ContextEntry:
if not isinstance(message, ToolResultMessage):
raise TypeError("append_tool_result takes a ToolResultMessage")
payload: dict[str, Any] = {"version": PAYLOAD_VERSION, "message": message_to_dict(message)}
if details:
payload["details"] = dict(details)
payload = _canonical(payload, "tool result")
def work(conn: Connection) -> ContextEntry:
# A call instance is settled once, whoever writes (a run, T2 at resume, a
# recovery pass): checked under the writer lock, so a racing second writer
# gets the first result back instead of a duplicate.
settled = _settled_result(conn, session_id, message.tool_call_id)
if settled is not None:
return settled
return self._append_event(conn, session_id, "tool_result", payload, message, agent_name, turn_id)
return await self._write(session_id, work)
async def append_payload(
self, session_id: str, kind: PayloadKind, payload: Mapping[str, Any], *, agent_name: Optional[str] = None,
turn_id: Optional[str] = None,
) -> ContextEntry:
return (await self.append_payloads(
session_id, [(kind, payload)], agent_name=agent_name, turn_id=turn_id
))[0]
async def append_payloads(
self,
session_id: str,
entries: Sequence[tuple[PayloadKind, Mapping[str, Any]]],
*,
agent_name: Optional[str] = None,
turn_id: Optional[str] = None,
) -> list[ContextEntry]:
"""Several payload rows in order, in one transaction: all of them or none (C-9 section 10)."""
rows = [(kind, _payload(kind, payload)) for kind, payload in entries]
def work(conn: Connection) -> list[ContextEntry]:
return [self._append_event(conn, session_id, kind, data, None, agent_name, turn_id) for kind, data in rows]
return await self._write(session_id, work)
async def append_audit(
self, session_id: str, kind: AuditKind, payload: Mapping[str, Any], *, agent_name: Optional[str] = None,
turn_id: Optional[str] = None,
) -> str:
"""A non-context audit row (C-9 ``ModelAttempt`` or C-10 ``ForkTurn``); its id. Never loaded as context."""
if kind not in _AUDIT_EVENT_TYPE:
raise ValueError(f"not an audit kind: {kind!r}")
data = _canonical(dict(payload), f"{kind} audit")
def work(conn: Connection) -> str:
origin = _turn_origin(conn, session_id, agent_name, turn_id)
row = agent_events_service.append(
conn,
scope_id=origin.scope_id,
session_id=session_id,
platform=origin.platform,
event_type=_AUDIT_EVENT_TYPE[kind],
content=data,
agent_name=origin.agent_name,
backend=origin.backend,
turn_id=origin.turn_id,
visibility=AUDIT_VISIBILITY,
source="agent",
)
return row["id"]
return await self._write(session_id, work)
# --- implementation -------------------------------------------------------
async def _write(self, session_id: str, work: Callable[[Connection], _T]) -> _T:
lock = self._locks.setdefault(session_id, asyncio.Lock())
async with lock:
commit = asyncio.ensure_future(asyncio.to_thread(self._commit, work))
try:
return await asyncio.shield(commit)
except asyncio.CancelledError:
await _settle(commit)
raise
def _commit(self, work: Callable[[Connection], _T]) -> _T:
with self._engine.begin() as conn:
reserve_write_lock(conn)
return work(conn)
def _append_event(
self,
conn: Connection,
session_id: str,
kind: EntryKind,
payload: dict[str, Any],
message: Optional[ToolResultMessage],
agent_name: Optional[str] = None,
turn_id: Optional[str] = None,
) -> ContextEntry:
origin = _turn_origin(conn, session_id, agent_name, turn_id)
seq = _next_context_seq(conn, session_id)
row = agent_events_service.append(
conn,
scope_id=origin.scope_id,
session_id=session_id,
platform=origin.platform,
event_type=_EVENT_TYPE_BY_KIND[kind],
content=payload,
agent_name=origin.agent_name,
backend=origin.backend,
turn_id=origin.turn_id,
visibility=CONTEXT_VISIBILITY,
source="agent",
context_seq=seq,
)
if message is not None:
_register_images(conn, origin.scope_id, session_id, message)
if kind == "compaction":
_record_digest(conn, session_id, row["id"], row["created_at"], payload)
return ContextEntry(
session_id, seq, kind, row["id"], message=message, payload=payload, created_at=_epoch(row["created_at"])
)
def _load(self, session_id: str, *, turn_id: Optional[str] = None) -> list[ContextEntry]:
with self._engine.begin() as conn:
# One read snapshot across both tables and every ancestor (or just the requested Turn).
if not conn.connection.dbapi_connection.in_transaction:
conn.exec_driver_sql("BEGIN")
if turn_id is not None:
return _turn_context_entries(conn, session_id, turn_id)
return _context_entries(conn, session_id)
async def _settle(task: asyncio.Future[Any]) -> None:
"""Wait for ``task`` to finish, whatever cancellations arrive meanwhile."""
while not task.done():
try:
await asyncio.wait({task})
except asyncio.CancelledError:
pass
if not task.cancelled():
# The cancelled caller does not see this outcome; retrieve it so a failed
# commit is not reported as an unhandled task exception.
task.exception()
# --- allocation and attribution ----------------------------------------------
def _register_images(
conn: Connection, scope_id: Optional[str], session_id: str, message: AssistantMessage | ToolResultMessage
) -> None:
"""Register each generated image ``message`` names, in its row's transaction (module docstring)."""
# WebFetchRecord shares the hosted envelope, not search/image fields.
# Narrow before entering media registration so page results can commit.
records: list[HostedRecord]
if isinstance(message, AssistantMessage):
records = [block.record for block in message.content if isinstance(block, HostedToolBlock)]
elif message.hosted is not None and isinstance(message.hosted.record, HostedRecord):
records = [message.hosted.record]
else:
return
for image in (image for record in records for image in record.images or ()):
media_service.register(
conn,
scope_id=scope_id,
session_id=session_id,
kind="image",
source=media_service.AGENT_CONTEXT_SOURCE,
local_path=image.path,
content_type=image.mime,
)
def _own_bound(conn: Connection, session_id: str) -> Optional[int]:
values = [
conn.execute(
select(func.max(table.c.context_seq)).where(
table.c.session_id == session_id, table.c.context_seq.is_not(None)
)
).scalar()
for table in (messages, agent_events)
]
return max((value for value in values if value is not None), default=None)
def _next_context_seq(conn: Connection, session_id: str) -> int:
top = _own_bound(conn, session_id)
if top is None:
link = fork_link(conn, session_id)
top = link[1] if link is not None else 0
return top + 1
def _turn_origin(
conn: Connection, session_id: str, agent_name: Optional[str] = None, turn_id: Optional[str] = None
) -> _TurnOrigin:
"""Where an inserted row belongs, and the Agent that wrote it.
``agent_name`` is the running Turn's Agent (its request snapshot): the Session's
selected Agent may change mid-Turn, so the Session row is only the fallback for a
write outside a Turn (startup recovery). ``turn_id`` is explicit for live writes:
a transcript-only retry owns a new Turn without consuming another input.
"""
session = conn.execute(
select(agent_sessions.c.agent_name).where(agent_sessions.c.id == session_id)
).first()
if session is None:
raise TranscriptError(f"Session {session_id} does not exist")
if turn_id is not None:
initial = conn.execute(
select(messages.c.platform, messages.c.scope_id)
.select_from(
session_turns.join(message_deliveries, message_deliveries.c.id == session_turns.c.initial_delivery_id)
.join(messages, messages.c.id == message_deliveries.c.message_id)
)
.where(session_turns.c.id == turn_id, session_turns.c.session_id == session_id)
).first()
if initial is None:
raise TranscriptError(f"Turn {turn_id} has no accepted input in Session {session_id}")
return _TurnOrigin(initial.platform, initial.scope_id, turn_id, agent_name or session.agent_name, CONTEXT_WRITER)
latest = conn.execute(
select(messages.c.id, messages.c.platform, messages.c.scope_id)
.where(
messages.c.session_id == session_id,
messages.c.context_seq.is_not(None),
messages.c.type.in_(INPUT_TYPES),
)
.order_by(messages.c.context_seq.desc())
.limit(1)
).first()
if latest is None:
# A fork settles the calls it inherited open (T2) before it consumes its first
# input: those rows belong to the Session's own scope, outside any Turn.
home = (
conn.execute(
select(scopes.c.id, scopes.c.platform)
.select_from(agent_sessions.join(scopes, scopes.c.id == agent_sessions.c.scope_id))
.where(agent_sessions.c.id == session_id)
).first()
if fork_link(conn, session_id) is not None
else None
)
if home is None:
raise TranscriptError(f"Session {session_id} has consumed no input; a context row answers a Turn")
return _TurnOrigin(home.platform, home.id, None, agent_name or session.agent_name, CONTEXT_WRITER)
platform, scope_id = latest.platform, latest.scope_id
turn_id = conn.execute(
select(message_deliveries.c.turn_id)
.where(
message_deliveries.c.session_id == session_id,
message_deliveries.c.state == "accepted",
message_deliveries.c.message_id == latest.id,
)
.limit(1)
).scalar()
if turn_id is not None:
initial = conn.execute(
select(messages.c.platform, messages.c.scope_id)
.select_from(
session_turns.join(
message_deliveries, message_deliveries.c.id == session_turns.c.initial_delivery_id
).join(messages, messages.c.id == message_deliveries.c.message_id)
)
.where(session_turns.c.id == turn_id)
).first()
if initial is not None:
platform, scope_id = initial.platform, initial.scope_id
return _TurnOrigin(platform, scope_id, turn_id, agent_name or session.agent_name, CONTEXT_WRITER)
# --- fork ancestry -------------------------------------------------------------
def resolve_fork_point(
conn: Connection, source_session_id: str, *, as_of: Optional[int] = None, self_fork: bool = False
) -> int:
"""The fork point of a new fork of ``source_session_id`` (C-10 ``fork.md`` section 2), in the caller's transaction.
A user's fork: the end of the previous ended Turn, so a live Turn's input, calls, and responses never reach the
child. ``self_fork``, the Agent's own fork from inside its live Turn: the largest settled point, which keeps that
Turn's input and finished steps and stops before the open call running the fork. ``as_of``: a point a system
mechanism names, which must be in range and settled (``ForkPointError`` otherwise). Decided from committed rows
and the live Turn's first input, never a clock. The reservation persists the result as the child's
``fork_source_context_seq``.
"""
entries = _context_entries(conn, source_session_id)
if as_of is not None:
return fork_point(entries, as_of).as_of
unended = None if self_fork else _unended_input_seq(conn, source_session_id, entries)
return latest_cut(entries, unended_input_seq=unended)
def _unended_input_seq(conn: Connection, session_id: str, entries: Sequence[ContextEntry]) -> Optional[int]:
"""The first input of the earliest unfinished Turn (C-10 ``fork.md`` section 2).
A Turn is unfinished while it is live, while it holds a call without a result (T2), or while an input it
accepted is not yet in the context (T3). A Turn's first input is the row its initial Delivery names; Turns run
one at a time and each writes its rows after that input, so the Turn owning an open call is the latest one whose
input precedes the call. A Turn whose first input is not consumed has no row in the context yet.
"""
turns = conn.execute(
select(session_turns.c.id, session_turns.c.state, messages.c.context_seq)
.select_from(session_turns.join(messages, messages.c.id == session_turns.c.initial_delivery_id))
.where(session_turns.c.session_id == session_id, messages.c.session_id == session_id)
).all()
awaiting_input = set(
conn.execute(
text(
"select message_deliveries.turn_id from messages "
"cross join message_deliveries on message_deliveries.message_id = messages.id "
f"where {VIBEY_RECOVERY_INPUT_PREDICATE} "
"and messages.session_id = :session_id and message_deliveries.state = 'accepted'"
),
{"session_id": session_id},
).scalars()
)
inputs = sorted(seq for _, _, seq in turns if seq is not None)
candidates = [
seq
for turn_id, state, seq in turns
if seq is not None and (state in TURN_OWNER_STATES or turn_id in awaiting_input)
]
open_calls = open_tool_calls(entries)
if open_calls:
first_open = min(owner.context_seq for owner, _ in open_calls)
owner_input = max((seq for seq in inputs if seq <= first_open), default=None)
candidates.append(owner_input if owner_input is not None else first_open)
return min(candidates, default=None)
def _settled_result(conn: Connection, session_id: str, tool_call_id: str) -> Optional[ContextEntry]:
"""The result already committed for the latest call instance with this id in the Session's context.
The call instance is the last response in the context (own rows and the inherited
prefix) that carries a tool call with this id; its result is the first
``tool_result`` for the id after that response, in context order.
"""
chain = _ancestry(conn, session_id)
owner_seq: Optional[int] = None
for member, bound in chain:
query = (
"SELECT MAX(m.context_seq) FROM messages AS m, "
"json_each(json_extract(m.content_json, '$.model.message.content')) AS block "
"WHERE m.session_id = :member AND m.context_seq IS NOT NULL "
"AND json_extract(block.value, '$.type') = 'tool_call' "
"AND json_extract(block.value, '$.id') = :call_id"
+ (" AND m.context_seq <= :bound" if bound is not None else "")
)
seq = conn.execute(
text(query), {"member": member, "call_id": tool_call_id, "bound": bound}
).scalar()
if seq is not None and (owner_seq is None or seq > owner_seq):
owner_seq = seq
if owner_seq is None:
return None
first = None
for member, bound in chain:
conditions = [
agent_events.c.session_id == member,
agent_events.c.event_type == _EVENT_TYPE_BY_KIND["tool_result"],
agent_events.c.visibility == CONTEXT_VISIBILITY,
agent_events.c.context_seq > owner_seq,
func.json_extract(agent_events.c.content_json, "$.message.tool_call_id") == tool_call_id,
]
if bound is not None:
conditions.append(agent_events.c.context_seq <= bound)
row = (
conn.execute(
select(
agent_events.c.id,
agent_events.c.session_id,
agent_events.c.context_seq,
agent_events.c.content_json,
agent_events.c.created_at,
agent_events.c.event_type,
agent_events.c.visibility,
)
.where(*conditions)
.order_by(agent_events.c.context_seq)
.limit(1)
)
.mappings()
.first()
)
if row is not None and (first is None or row["context_seq"] < first["context_seq"]):
first = row
return _event_entry(first["session_id"], first) if first is not None else None
def call_instance_result(
conn: Connection, owner_session_id: str, response_seq: int, tool_call_id: str
) -> Optional[ContextEntry]:
"""The result its owner committed for a call instance, if it has one.
A call's identity is its instance: the response that carries it
(``owner_session_id`` at ``response_seq``) plus its id, because providers may
reuse ids. Its result is the first ``tool_result`` for that id after the
response in the owner's own rows, in context order: every call is settled
before the next model call, so a later reuse of the id comes after it. A fork
anchored between a response and its tool results inherits those calls open
and settles them with this result instead of re-deriving it from job state
that J5 may since have pruned; J5 keeps a job's files until it exists.
"""
row = (
conn.execute(
select(
agent_events.c.id,
agent_events.c.session_id,
agent_events.c.context_seq,
agent_events.c.content_json,
agent_events.c.created_at,
agent_events.c.event_type,
agent_events.c.visibility,
)
.where(
agent_events.c.session_id == owner_session_id,
agent_events.c.event_type == _EVENT_TYPE_BY_KIND["tool_result"],
agent_events.c.visibility == CONTEXT_VISIBILITY,
agent_events.c.context_seq > response_seq,
func.json_extract(agent_events.c.content_json, "$.message.tool_call_id") == tool_call_id,
)
.order_by(agent_events.c.context_seq)
.limit(1)
)
.mappings()
.first()
)
return _event_entry(row["session_id"], row) if row is not None else None
def recovered_watch_ids(conn: Connection, session_id: str, turn_id: str) -> list[str]:
"""The Watches recovery handed a Turn's open calls to (``recovery.md`` T2), in context order.
Only results T2 committed count: a call the Turn handed over itself (``watch: true``,
the foreground window) was already reported to the model, so it is not work the
Turn lost.
"""
from core.agent_core.agent.recovery import RECOVERED
rows = conn.execute(
select(agent_events.c.content_json)
.where(
agent_events.c.session_id == session_id,
agent_events.c.turn_id == turn_id,
agent_events.c.event_type == _EVENT_TYPE_BY_KIND["tool_result"],
agent_events.c.visibility == CONTEXT_VISIBILITY,
func.json_extract(agent_events.c.content_json, f"$.details.{RECOVERED}") == 1,
func.json_extract(agent_events.c.content_json, "$.details.watch_id").is_not(None),
)
.order_by(agent_events.c.context_seq)
).scalars()
return [str(json.loads(raw)["details"]["watch_id"]) for raw in rows]
# A response that stopped this way is never final: the loop commits the partial a failed
# stream carried with ``final=False`` (``Agent._commit_model_message``).
_UNFINISHED_STOP_REASONS = frozenset({"error", "aborted"})
@dataclass(frozen=True)
class CommittedResponse:
"""A response row in a Session's own context: the facts restart recovery reads."""
row_id: str
context_seq: int
message_type: str
turn_id: Optional[str]
text: str
has_tool_calls: bool
stop_reason: str
agent_name: Optional[str] = None
def last_response(conn: Connection, session_id: str) -> Optional[CommittedResponse]:
"""The last model response the Session's own rows committed, by one bounded index read."""
row = conn.execute(
select(
messages.c.id,
messages.c.context_seq,
messages.c.type,
messages.c.content_text,
messages.c.content_json,
messages.c.metadata_json,
messages.c.author_name,
)
.where(
messages.c.session_id == session_id,
messages.c.context_seq.is_not(None),
messages.c.type.in_(RESPONSE_TYPES),
)
.order_by(messages.c.context_seq.desc())
.limit(1)
).mappings().first()
if row is None:
return None
try:
model_message = json.loads(row["content_json"] or "{}")["model"]["message"]
blocks, stop_reason = model_message["content"], str(model_message.get("stop_reason") or "")
except (TypeError, ValueError, KeyError, AttributeError):
blocks, stop_reason = [], ""
try:
metadata = json.loads(row["metadata_json"] or "{}")
except (TypeError, ValueError):
metadata = {}
turn_id = metadata.get("turn_id") if isinstance(metadata, dict) else None
return CommittedResponse(
row_id=str(row["id"]),
context_seq=int(row["context_seq"]),
message_type=str(row["type"]),
turn_id=str(turn_id) if turn_id else None,
text=str(row["content_text"] or ""),
has_tool_calls=any(
isinstance(block, dict) and block.get("type") == "tool_call" for block in blocks or ()
),
stop_reason=stop_reason,
agent_name=str(row["author_name"]) if row["author_name"] else None,
)
def committed_final(conn: Connection, session_id: str, turn_id: str) -> Optional[CommittedResponse]:
"""The final response ``turn_id`` committed, when the Turn's work ended with it.
Read after T2 and T3, with the loop's finality rule (``Agent._commit_model_message``):
a Turn has work left unless its last committed response has no tool calls, did not
stop with ``error`` or ``aborted`` (the partial a failed stream commits), and no
input follows it, in the context or still waiting to enter it (an accepted steer no
run consumed). A reply a steer superseded is a ``result`` like a final, and that
steer is what follows it. Every read is bounded by the Session's context tail or by
the Turn's own Deliveries.
"""
response = last_response(conn, session_id)
if (
response is None
or response.turn_id != turn_id
or response.has_tool_calls
or response.stop_reason in _UNFINISHED_STOP_REASONS
# Released versions committed a reply pending input superseded as ``output``.
or response.message_type == "output"
):
return None
later_input = conn.execute(
select(messages.c.id)
.where(
messages.c.session_id == session_id,
messages.c.context_seq > response.context_seq,
messages.c.type.in_(INPUT_TYPES),
)
.limit(1)
).first()
if later_input is not None:
return None
unconsumed = conn.execute(
text(
"select messages.id from message_deliveries "
"cross join messages on messages.id = message_deliveries.message_id "
"where message_deliveries.turn_id = :turn_id and message_deliveries.state = 'accepted' "
f"and {VIBEY_RECOVERY_INPUT_PREDICATE} limit 1"
),
{"turn_id": turn_id},
).first()
return None if unconsumed is not None else response
def fork_link(conn: Connection, session_id: str) -> Optional[tuple[str, int]]:
"""``(source_session_id, anchor_seq)`` when the Session inherits a context."""
raw = conn.execute(
select(agent_sessions.c.metadata_json).where(agent_sessions.c.id == session_id)
).scalar_one_or_none()
try:
metadata = json.loads(raw or "{}")
except (TypeError, ValueError):
return None
if not isinstance(metadata, dict):
return None
source = str(metadata.get("fork_source_session_id") or "").strip()
anchor = metadata.get("fork_source_context_seq")
if not source or anchor is None:
return None
if not isinstance(anchor, int) or isinstance(anchor, bool) or anchor < 0:
raise TranscriptError(f"Session {session_id} has an invalid fork_source_context_seq: {anchor!r}")
return source, anchor
def _ancestry(conn: Connection, session_id: str) -> list[tuple[str, Optional[int]]]:
"""The Session and its fork sources, each with its ``context_seq`` upper bound."""
chain: list[tuple[str, Optional[int]]] = [(session_id, None)]
seen = {session_id}
member, bound = session_id, None
while (link := fork_link(conn, member)) is not None:
source_id, anchor_seq = link
if source_id in seen:
raise TranscriptError(f"fork ancestry of Session {session_id} loops back to {source_id}")
bound = anchor_seq if bound is None else min(bound, anchor_seq)
chain.append((source_id, bound))
seen.add(source_id)
member = source_id
return chain
# --- reading rows --------------------------------------------------------------
def _context_entries(conn: Connection, session_id: str) -> list[ContextEntry]:
"""The Session's context rows and its fork ancestry's, in ``context_seq`` order, read on ``conn``."""
entries = [entry for member, bound in _ancestry(conn, session_id) for entry in _context_rows(conn, member, bound)]
entries.sort(key=lambda entry: entry.context_seq)
for previous, current in zip(entries, entries[1:]):
if previous.context_seq == current.context_seq:
raise TranscriptError(f"rows {previous.row_id} and {current.row_id} share context_seq {current.context_seq}")
return entries
def _context_rows(conn: Connection, session_id: str, bound: Optional[int]) -> list[ContextEntry]:
entries: list[ContextEntry] = []
for table, columns, to_entry in (
(messages, (messages.c.type, messages.c.metadata_json), _message_entry),
(agent_events, (agent_events.c.event_type, agent_events.c.visibility), _event_entry),
):
conditions = [table.c.session_id == session_id, table.c.context_seq.is_not(None)]
if bound is not None:
conditions.append(table.c.context_seq <= bound)
rows = conn.execute(
select(table.c.id, table.c.context_seq, table.c.content_json, table.c.created_at, *columns).where(
*conditions
)
).mappings()
entries.extend(to_entry(session_id, row) for row in rows)
return entries
def _turn_context_entries(conn: Connection, session_id: str, turn_id: str) -> list[ContextEntry]:
"""Read one Turn through its owner indexes, without loading Session history or forks."""
entries: list[ContextEntry] = []
for table, index, owner, columns, to_entry in (
("messages", "ix_messages_turn_id", MESSAGE_TURN_ID_SQL, "type, metadata_json", _message_entry),
("agent_events", "ix_agent_events_turn_sequence_id", "turn_id", "event_type, visibility", _event_entry),
):
query = text(
f"SELECT id, context_seq, content_json, created_at, {columns} FROM {table} INDEXED BY {index} "
f"WHERE session_id = :session_id AND context_seq IS NOT NULL AND ({owner}) = :turn_id"
+ (" AND type IN :response_types" if table == "messages" else "")
)
params: dict[str, Any] = {"session_id": session_id, "turn_id": turn_id}
if table == "messages":
query = query.bindparams(bindparam("response_types", expanding=True))
params["response_types"] = RESPONSE_TYPES
entries.extend(to_entry(session_id, row) for row in conn.execute(query, params).mappings())
# Inputs are owned by Deliveries, not response metadata. Include accepted steers as barriers in this Turn.
query = text(
"SELECT messages.id, messages.context_seq, messages.content_json, messages.created_at, messages.type "
"FROM message_deliveries INDEXED BY ix_message_deliveries_turn "
"CROSS JOIN messages ON messages.id = message_deliveries.message_id "
"WHERE message_deliveries.turn_id = :turn_id AND message_deliveries.state = 'accepted' "
"AND messages.session_id = :session_id AND messages.context_seq IS NOT NULL AND messages.type IN :input_types"
).bindparams(bindparam("input_types", expanding=True))
rows = conn.execute(query, {"session_id": session_id, "turn_id": turn_id, "input_types": INPUT_TYPES}).mappings()
entries.extend(_message_entry(session_id, row) for row in rows)
return sorted(entries, key=lambda row: row.context_seq)
def _with_render_metadata(message: AssistantMessage, metadata: Mapping[str, Any]) -> AssistantMessage:
"""Optional renderer facts never make an otherwise valid historical row unreadable."""
changes: dict[str, Any] = {}
for key in ("hosted_render_epoch", "hosted_render_rule"):
value = metadata.get(key)
if type(value) is int:
changes[key] = value
omitted = metadata.get("hosted_render_omitted")
if isinstance(omitted, list) and all(isinstance(row, str) and row for row in omitted):
changes["hosted_render_omitted"] = tuple(omitted)
return replace(message, **changes)
def _message_entry(session_id: str, row: Mapping[str, Any]) -> ContextEntry:
row_id = row["id"]
if row["type"] in INPUT_TYPES:
kind: EntryKind = "input"
expected: type = UserMessage
elif row["type"] in RESPONSE_TYPES:
kind, expected = "response", AssistantMessage
else:
raise TranscriptError(f"context row {row_id} has message type {row['type']!r}")
payload = _versioned(_json_object(row["content_json"], row_id).get("model"), row_id)
message = _message(payload.get("message"), expected, row_id)
if isinstance(message, AssistantMessage):
try:
metadata = _json_object(row.get("metadata_json"), row_id)
except TranscriptError:
metadata = {}
message = replace(_with_render_metadata(message, metadata), transcript_row_id=row_id)
return ContextEntry(
session_id, row["context_seq"], kind, row_id, message=message, payload=payload, created_at=_epoch(row["created_at"])
)
def _event_entry(session_id: str, row: Mapping[str, Any]) -> ContextEntry:
row_id = row["id"]
kind = _KIND_BY_EVENT_TYPE.get(row["event_type"])
if kind is None or row["visibility"] != CONTEXT_VISIBILITY:
raise TranscriptError(f"context row {row_id} is a {row['visibility']} {row['event_type']!r} event")
payload = _versioned(_json_object(row["content_json"], row_id), row_id)
message = _message(payload.get("message"), ToolResultMessage, row_id) if kind == "tool_result" else None
return ContextEntry(
session_id, row["context_seq"], kind, row_id, message=message, payload=payload, created_at=_epoch(row["created_at"])
)
def _record_digest(
conn: Connection, session_id: str, compaction_id: str, written_at: str, payload: Mapping[str, Any]
) -> None:
"""Keep the checkpoint's mind, memo, and now as the Session's digest, the memory other Sessions recall.
A checkpoint that holds none of them (a drop that left them out, context.md section 8 c) leaves the previous
digest: it is still the latest memory the Session wrote.
"""
# Only a Vibey runtime appends a checkpoint, and it has the harness loaded; this module's other importers
# (core.session_turns, restart resume) never write one and need not load it.
from core.agent_core.harness.context import MemoItem, memo_text
mind = _block(payload.get("mind"))
memo = memo_text([MemoItem(**item) for item in payload["memo"]]) if payload.get("memo") else None
now = _block(payload.get("checkpoint"))
if not (mind or memo or now):
return
values = {"compaction_id": compaction_id, "updated_at": written_at, "mind": mind, "memo": memo, "now": now}
conn.execute(
sqlite_insert(session_digests)
.values(session_id=session_id, **values)
.on_conflict_do_update(index_elements=[session_digests.c.session_id], set_=values)
)
def _block(value: Any) -> Optional[str]:
return value if isinstance(value, str) and value.strip() else None
def _message(value: Any, expected: type, row_id: str) -> Any:
try:
message = message_from_dict(value)
except (TypeError, ValueError) as exc:
raise TranscriptError(f"context row {row_id}: {exc}") from exc
if not isinstance(message, expected):
raise TranscriptError(f"context row {row_id} holds a {message.role} message")
return message
def _versioned(value: Any, row_id: str) -> dict[str, Any]:
if not isinstance(value, dict) or not _is_current_version(value):
raise TranscriptError(f"context row {row_id} has no version {PAYLOAD_VERSION} payload")
return value
def _is_current_version(payload: Mapping[str, Any]) -> bool:
version = payload.get("version")
return isinstance(version, int) and not isinstance(version, bool) and version == PAYLOAD_VERSION
def _json_object(raw: Any, row_id: str) -> dict[str, Any]:
try:
value = json.loads(raw or "{}")
except (TypeError, ValueError) as exc:
raise TranscriptError(f"row {row_id} holds invalid JSON") from exc
if not isinstance(value, dict):
raise TranscriptError(f"row {row_id} holds a JSON {type(value).__name__}, not an object")
return value
def _canonical(value: dict[str, Any], what: str) -> dict[str, Any]:
"""A detached copy of ``value``, refused unless it reads back from a row unchanged.
A tuple, a non-str key, NaN, or bytes would otherwise be coerced on the way
to disk, and the context would no longer be what its writer committed.
"""
require_json_value(value, what)
return json.loads(json.dumps(value))
def _payload(kind: str, payload: Mapping[str, Any]) -> dict[str, Any]:
if kind not in ("compaction", "context_edit", "agent_state", "replay_refusal"):
raise ValueError(f"not a payload entry kind: {kind!r}")
data = _canonical(dict(payload), f"{kind} payload")
if not _is_current_version(data):
raise ValueError(f"a {kind} payload needs version {PAYLOAD_VERSION}")
return data
def _epoch(value: Any) -> Optional[float]:
"""A row's ``created_at`` (ISO 8601, UTC when it names no zone) as epoch seconds; None if it is not a time."""
text_value = str(value or "").strip()
if text_value.endswith("Z"):
text_value = text_value[:-1] + "+00:00"
try:
instant = datetime.fromisoformat(text_value)
except ValueError:
return None
if instant.tzinfo is None:
instant = instant.replace(tzinfo=timezone.utc)
return instant.timestamp()
def _utc_now() -> str:
return datetime.now(timezone.utc).isoformat(timespec="microseconds").replace("+00:00", "Z")