Skip to content

Self-improving AI Employees for everyone

fix(agent): stop Context nodes causing long runs to forget their conversation

trohitg9bfae071f461
diff --git a/CLAUDE.md b/CLAUDE.md
index 5bdbd4f2..571b59c8 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1380,7 +1380,9 @@ Two different things, deliberately separated. Read [RFC-0002](./RFC-0002-AGENT-C
**Context** is the backend-owned execution journal — what the agent actually sent and received. The `context` node (`server/nodes/context/`, `input-context` handle) is a **read-only observation surface onto it**. Load-bearing invariant, learned the hard way: *attaching a Context node must never change what the agent sends.* It once did — `context_ref` in the LLM payload selected a second implementation that rebuilt the request from the journal instead of sending `messages`, and the rebuilt transcript dropped the user's prompt, so connecting a Context node made the agent answer an empty question. There is now exactly one LLM step; `context_ref` only says *where to record*.
-The journal records, it never reconstructs. `agent.execute_llm_step` writes each turn from the exact message list it hands to `ChatUnifier.chat` (`_journal_llm_turn`), immediately after the call returns. `agent.prepare_context` deliberately journals nothing — it runs before the request exists, so anything it wrote would be assembled from configuration. Journal operation ids derive from the per-firing `context_execution_id`, **not** `execution_id`: the latter is generation-scoped, so every message in a generation minted identical ids and the store's idempotency guard silently discarded every turn after the first.
+The journal records, it never reconstructs. `agent.execute_llm_step` writes each turn from the exact message list it hands to `ChatUnifier.chat` (`_journal_llm_turn`), immediately after the call returns. `agent.prepare_context` deliberately journals nothing — it runs before the request exists, so anything it wrote would be assembled from configuration. Journal operation ids derive from the per-firing `context_execution_id`, **not** `execution_id`: the latter is generation-scoped, so every message in a generation minted identical ids and the store's idempotency guard silently discarded every turn after the first. They are **also scoped by `agent_node_id`** — two agents wired to one Context node resolve to the same thread, so a firing-only id made them collide on `(thread, operation_id)` and one agent's turns vanished as replays.
+
+**The one place the journal is read back is the `continue_as_new` rollover.** `AgentWorkflow` rolls over under history pressure *only when `context_ref` is set* (`agent_workflow.py`), and a rollover carries references, not the conversation — so the resumed run rebuilds `messages` from system + memory + prompt alone. Without a replay step that means attaching a Context node is what makes a long run silently forget everything it had said: the same class of bug as the original `context_ref` regression, one boundary later. The resumed run therefore calls `agent.reconstruct_context_messages` (read-only, wraps `reconstruct_message_wire_v2`) before its first LLM step. That activity **degrades instead of raising** — an opaque provider checkpoint or an unreadable thread returns `restored: False` with a reason and the run continues from the rebuilt prompt, because a rehydration miss must never turn an unreadable thread into a dead agent. Locked by `tests/temporal/test_agent_workflow.py::TestContextJournalIdentity`, which also asserts the workflow source carries no comment claiming the LLM activity rebuilds its request from the store — that sentence was false for a long time and is exactly what a future reader would re-implement.
A Context thread exists only under an admitted generation (`generation > 0`), which only Start assigns — manual canvas Runs journal nothing by design. Each generation gets its own thread, so conversation does not carry across a Stop/Start. Workflow Reset rotates the thread to a fresh epoch and leaves prior events as archived history; the panel's journal view is scoped to the live epoch, or a reset looks like it did nothing.
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index d6f57f13..82e9d2f1 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -429,8 +429,9 @@ Durable event listeners retain their deployment graph as a fallback, but each fi
**Canvas-aware tools** opt into receiving the parent workflow's `nodes`/`edges` by declaring `needs_canvas: ClassVar[bool] = True` on their `BaseNode` subclass. The F4.B tool dispatch reads this via `services.node_registry.get_node_class(node_type).needs_canvas` and forwards `context.get("nodes")` / `context.get("edges")` into `tool_payload`; default plugins keep the empty-canvas optimisation. Today only `agentBuilder` opts in (it walks edges to resolve its calling agent and mutates the canvas). Operations inside agentBuilder reload via `database.get_workflow(workflow_id)` so in-run duplicate detection sees mutations from earlier calls in the same workflow run — see [agent_builder section in CLAUDE.md](../CLAUDE.md).
-`collect_agent_activities()` registers **16 agent activities**. Seven are the
-core loop activities:
+`collect_agent_activities()` registers the agent activities — read the live set
+from that function rather than trusting a count here, which drifts on every
+addition. Seven are the core loop activities:
| Activity | Purpose |
|---|---|
@@ -456,6 +457,24 @@ The other nine support skills and durable delegation:
| `agent.finish_delegation` | Idempotently records a delegated child's terminal result and usage. |
| `agent.finalize_team` | Finalizes a team after its required tasks reach accepted or terminal states. |
+The Context journal surface adds `agent.prepare_context` (resolves the thread;
+journals nothing, because it runs before the request exists),
+`agent.reconstruct_context_messages`, `agent.append_context` and
+`agent.compact_context`.
+
+**`agent.reconstruct_context_messages` is the only place the journal is read
+back on the execution path.** `AgentWorkflow` rolls over under history pressure
+*only when `context_ref` is set*, and a rollover carries references rather than
+the conversation — so without a replay step, attaching a Context node is what
+makes a long run silently forget everything it had said. The resumed run calls
+this activity before its first LLM step and seeds `messages` from the result. It
+degrades rather than raising: an opaque provider checkpoint or an unreadable
+thread returns `restored: False` with a reason and the run continues from the
+rebuilt system + memory + prompt, since a rehydration miss must not turn an
+unreadable thread into a dead agent. This does **not** weaken the rule that the
+request is always built from `messages` — nothing reconstructs mid-loop, only at
+the rollover boundary.
+
The F4.B compaction threshold is prepared from the model context length and
the ratio configuration. It currently does not read
`SessionTokenState.custom_threshold` or `compaction_enabled`; those stored
diff --git a/server/services/temporal/agent_activities.py b/server/services/temporal/agent_activities.py
index dca7be7b..514ca9bf 100644
--- a/server/services/temporal/agent_activities.py
+++ b/server/services/temporal/agent_activities.py
@@ -1930,6 +1930,7 @@ def collect_agent_activities() -> List[Any]:
# plain coroutine invoked by each plugin's own activity wrapper in
# services/plugin/base.py, not a Temporal activity of its own.
prepare_context,
+ reconstruct_context_messages,
append_context,
compact_context,
]
@@ -2186,6 +2187,56 @@ async def prepare_context(payload: Dict[str, Any]) -> Dict[str, Any]:
return {"context_ref": _ref_dict(thread)}
[email protected](name="agent.reconstruct_context_messages")
+async def reconstruct_context_messages(payload: Dict[str, Any]) -> Dict[str, Any]:
+ """Replay a Context thread's active transcript. Read-only.
+
+ Used only by ``AgentWorkflow`` after ``continue_as_new``. A rollover
+ carries references, not the conversation, so without this the resumed run
+ rebuilds ``messages`` from system + memory + prompt alone and the agent
+ silently forgets everything it had said — but only when a Context node is
+ attached, since the rollover itself is gated on ``context_ref``.
+
+ Returns ``{"messages": [...], "restored": bool, "reason": str}``. A thread
+ that cannot be replayed reports ``restored: False`` rather than raising, so
+ the workflow falls back to the freshly built list instead of failing a run
+ over a rehydration miss.
+ """
+
+ ref_data = payload.get("context_ref") or {}
+ if not ref_data:
+ return {"messages": [], "restored": False, "reason": "no_context_ref"}
+
+ from services.agent_context import (
+ OpaqueCheckpointError,
+ reconstruct_message_wire_v2,
+ )
+
+ store = _context_store()
+ try:
+ _ref, wires = await reconstruct_message_wire_v2(
+ store, _context_ref(ref_data)
+ )
+ except OpaqueCheckpointError:
+ # A provider-native checkpoint holds opaque continuation state, not a
+ # message list. The provider itself carries that context, so the
+ # rebuilt prompt is the correct fallback.
+ return {"messages": [], "restored": False, "reason": "opaque_checkpoint"}
+ except Exception as exc: # noqa: BLE001 -- never fail a run over replay
+ activity.logger.warning(
+ "Context transcript replay failed; resuming from the rebuilt "
+ f"prompt instead: {exc!r}"
+ )
+ return {"messages": [], "restored": False, "reason": "replay_failed"}
+
+ messages = [dict(wire) for wire in wires]
+ return {
+ "messages": messages,
+ "restored": bool(messages),
+ "reason": "ok" if messages else "empty_thread",
+ }
+
+
def _pending_tool_identities(
calls: Iterable[Mapping[str, Any]],
configured_tools: Iterable[Mapping[str, Any]],
diff --git a/server/services/temporal/agent_workflow.py b/server/services/temporal/agent_workflow.py
index db11169b..cf911f9f 100644
--- a/server/services/temporal/agent_workflow.py
+++ b/server/services/temporal/agent_workflow.py
@@ -720,11 +720,14 @@ class AgentWorkflow:
# Native executions use the SDK-neutral v2 wire codec. Legacy
# histories keep the legacy canonical {type, data} shape exactly.
# ---- Context journal -------------------------------------------
- # A Context node on the agent's input makes the backend journal the
- # source of truth for the transcript: the LLM activity reconstructs
- # from the store rather than from `messages`, so a run resumes the
- # prior conversation instead of starting empty. The workflow holds
- # only references.
+ # A Context node makes the backend journal every turn the agent sends.
+ # It never steers the request: the LLM activity always builds from
+ # ``messages``, and ``context_ref`` only says where to record. The
+ # workflow itself holds references, never transcript content.
+ #
+ # The one place the journal is read back is the continue-as-new
+ # rollover below, which carries references rather than the
+ # conversation.
context_ref: Optional[Dict[str, Any]] = None
runtime_config_ref = ""
if payload.get("context_descriptor"):
@@ -746,8 +749,8 @@ class AgentWorkflow:
prepared_context.get("runtime_config_ref") or ""
)
if resume:
- # A resumed run reconstructs its transcript from the journal, so
- # only references and counters cross the rollover boundary.
+ # Only references and counters cross the rollover boundary; the
+ # transcript is replayed from the journal further down.
context_ref = resume.get("context_ref") or context_ref
runtime_config_ref = (
str(resume.get("runtime_config_ref") or "") or runtime_config_ref
@@ -788,6 +791,35 @@ class AgentWorkflow:
{"type": "human", "data": {"content": user_prompt}}
)
+ # ---- Resume: replay the transcript from the journal --------------
+ # continue_as_new carries references, not the conversation, so the
+ # list built above is just system + memory + prompt. Without this the
+ # agent forgets everything it had said mid-run — and only when a
+ # Context node is attached, since the rollover is gated on
+ # ``context_ref``. The journal is authoritative here because it holds
+ # the exact message list each turn was sent with.
+ if resume and context_ref:
+ await self._wait_until_resumed()
+ replayed = await workflow.execute_activity(
+ "agent.reconstruct_context_messages",
+ args=[{"context_ref": context_ref}],
+ activity_id="reconstruct-context-messages",
+ start_to_close_timeout=PERSIST_TURN_TIMEOUT * 2,
+ retry_policy=AGENT_ACTIVITY_RETRY,
+ )
+ restored = replayed.get("messages") or []
+ if restored:
+ messages = [dict(message) for message in restored]
+ else:
+ # An opaque provider checkpoint or an unreadable thread. The
+ # rebuilt prompt is the honest fallback; log which, so a
+ # forgetful agent is diagnosable rather than mysterious.
+ workflow.logger.warning(
+ "AgentWorkflow resumed without a replayed transcript "
+ f"({replayed.get('reason') or 'unknown'}); continuing from "
+ "the rebuilt prompt"
+ )
+
# Map LLM tool name -> {node_type, version, task_queue, node_id,
# parameters} so the workflow can schedule the right activity
# when the LLM emits a tool_call.
@@ -864,8 +896,15 @@ class AgentWorkflow:
**(
{
"context_ref": context_ref,
+ # Scoped by agent node as well as firing: two
+ # agents wired to one Context node resolve to the
+ # same thread, so a firing-only id made them mint
+ # identical operation ids, collide on
+ # (thread, operation_id), and have their turns
+ # discarded by the store's idempotency guard.
"journal_operation_id": (
- f"{journal_operation_id}:iter:{iteration}"
+ f"{journal_operation_id}:{agent_node_id}"
+ f":iter:{iteration}"
),
}
if context_ref
diff --git a/server/tests/temporal/test_agent_workflow.py b/server/tests/temporal/test_agent_workflow.py
index 5b39a740..9890d4f0 100644
--- a/server/tests/temporal/test_agent_workflow.py
+++ b/server/tests/temporal/test_agent_workflow.py
@@ -18,6 +18,18 @@ from __future__ import annotations
from unittest.mock import AsyncMock, MagicMock
+import pytest
+
+# Bounded reference shape carried across a continue_as_new rollover.
+_CONTEXT_REF = {
+ "workflow_id": "workflow-1",
+ "context_node_id": "workflow-1:context:1",
+ "generation": 2,
+ "thread_id": "session:abc",
+ "epoch": 1,
+ "revision": 7,
+}
+
class TestAgentWorkflowDefinition:
"""``AgentWorkflow`` must be a valid Temporal workflow definition
@@ -116,6 +128,144 @@ class TestContextJournalIdentity:
assert f"{{execution_id}}:iter" not in source
assert f'f"{{execution_id}}{suffix}"' not in source
+ def test_journal_operation_ids_are_scoped_per_agent_node(self):
+ """Two agents on one Context node resolve to the same thread.
+
+ With a firing-scoped id alone they minted identical operation ids,
+ collided on (thread, operation_id) and had their turns discarded as
+ replays -- so one of the two agents was simply absent from the journal.
+ """
+ import inspect
+
+ from services.temporal.agent_workflow import AgentWorkflow
+
+ source = inspect.getsource(AgentWorkflow.run)
+ assert 'f"{journal_operation_id}:{agent_node_id}"' in source, (
+ "journal operation ids must include the agent node id, or "
+ "sibling agents sharing a Context node overwrite each other"
+ )
+
+ def test_resumed_run_replays_the_transcript_from_the_journal(self):
+ """continue_as_new carries references, not the conversation.
+
+ The rollover is gated on ``context_ref``, so without a replay step
+ attaching a Context node is what makes a long run forget everything it
+ had said -- the same class of bug as the original ``context_ref``
+ regression, one boundary later.
+ """
+ import inspect
+
+ from services.temporal.agent_workflow import AgentWorkflow
+
+ source = inspect.getsource(AgentWorkflow.run)
+ assert "agent.reconstruct_context_messages" in source, (
+ "a resumed AgentWorkflow must replay its transcript from the "
+ "journal before continuing"
+ )
+ replay_at = source.index("agent.reconstruct_context_messages")
+ loop_at = source.index("agent.execute_llm_step")
+ assert replay_at < loop_at, (
+ "the transcript must be restored before the first LLM step of "
+ "the resumed run"
+ )
+
+ def test_workflow_never_claims_the_activity_rebuilds_the_request(self):
+ """Guards the comment, not the code.
+
+ Two comments used to state that the LLM activity reconstructs from the
+ store rather than from ``messages``. That was false, and it is exactly
+ the sentence a future reader would implement to reintroduce the
+ original bug.
+ """
+ import inspect
+
+ from services.temporal.agent_workflow import AgentWorkflow
+
+ source = inspect.getsource(AgentWorkflow.run)
+ for claim in (
+ "reconstructs\n",
+ "source of truth for the transcript",
+ "rather than from `messages`",
+ ):
+ assert claim not in source, (
+ f"stale claim {claim!r} in AgentWorkflow.run -- the activity "
+ f"always builds its request from `messages`"
+ )
+
+ @pytest.mark.asyncio
+ async def test_replay_returns_the_journalled_transcript(self, monkeypatch):
+ import services.agent_context as agent_context
+ from services.temporal import agent_activities
+
+ wires = [
+ {"version": 2, "role": "system", "content": "you are helpful"},
+ {"version": 2, "role": "user", "content": "hello"},
+ ]
+
+ async def _replay(_store, ref):
+ return ref, wires
+
+ monkeypatch.setattr(agent_activities, "_context_store", lambda: object())
+ monkeypatch.setattr(agent_context, "reconstruct_message_wire_v2", _replay)
+
+ result = await agent_activities.reconstruct_context_messages(
+ {"context_ref": _CONTEXT_REF}
+ )
+
+ assert result["restored"] is True
+ assert result["messages"] == wires
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize(
+ "failure, reason",
+ [
+ ("opaque", "opaque_checkpoint"),
+ ("boom", "replay_failed"),
+ ],
+ )
+ async def test_replay_degrades_instead_of_failing_the_run(
+ self, monkeypatch, failure, reason
+ ):
+ """A rehydration miss must never kill a resumed run.
+
+ The rebuilt system + memory + prompt is a worse starting point than
+ the journal, but it is a working one; raising here would turn an
+ unreadable thread into a dead agent.
+ """
+ import services.agent_context as agent_context
+ from services.agent_context import OpaqueCheckpointError
+ from services.temporal import agent_activities
+
+ async def _replay(_store, _ref):
+ raise (
+ OpaqueCheckpointError("provider-native checkpoint")
+ if failure == "opaque"
+ else RuntimeError("blob missing")
+ )
+
+ monkeypatch.setattr(agent_activities, "_context_store", lambda: object())
+ monkeypatch.setattr(agent_context, "reconstruct_message_wire_v2", _replay)
+
+ result = await agent_activities.reconstruct_context_messages(
+ {"context_ref": _CONTEXT_REF}
+ )
+
+ assert result["restored"] is False
+ assert result["reason"] == reason
+ assert result["messages"] == []
+
+ @pytest.mark.asyncio
+ async def test_replay_without_a_context_ref_is_a_no_op(self):
+ from services.temporal import agent_activities
+
+ result = await agent_activities.reconstruct_context_messages({})
+
+ assert result == {
+ "messages": [],
+ "restored": False,
+ "reason": "no_context_ref",
+ }
+
def test_llm_step_has_a_single_implementation(self):
"""A Context node observes execution; it must not steer it.
@@ -382,6 +532,7 @@ class TestAgentActivities:
"agent.prepare_context",
"agent.prepare_payload",
"agent.queue_delegation",
+ "agent.reconstruct_context_messages",
"agent.refresh_tools",
"agent.register_task_execution",
"agent.release_subagent_permit",