Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(agent): stop Context nodes causing long runs to forget their conversation
trohitg9bfae071f461
diff --git a/CLAUDE.md b/CLAUDE.mdindex 5bdbd4f2..571b59c8 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1380,7 +1380,9 @@ Two different things, deliberately separated. Read [RFC-0002](./RFC-0002-AGENT-C**Context** is the backend-owned execution journal — what the agent actually sent and received. The `context` node (`server/nodes/context/`, `input-context` handle) is a **read-only observation surface onto it**. Load-bearing invariant, learned the hard way: *attaching a Context node must never change what the agent sends.* It once did — `context_ref` in the LLM payload selected a second implementation that rebuilt the request from the journal instead of sending `messages`, and the rebuilt transcript dropped the user's prompt, so connecting a Context node made the agent answer an empty question. There is now exactly one LLM step; `context_ref` only says *where to record*.-The journal records, it never reconstructs. `agent.execute_llm_step` writes each turn from the exact message list it hands to `ChatUnifier.chat` (`_journal_llm_turn`), immediately after the call returns. `agent.prepare_context` deliberately journals nothing — it runs before the request exists, so anything it wrote would be assembled from configuration. Journal operation ids derive from the per-firing `context_execution_id`, **not** `execution_id`: the latter is generation-scoped, so every message in a generation minted identical ids and the store's idempotency guard silently discarded every turn after the first.+The journal records, it never reconstructs. `agent.execute_llm_step` writes each turn from the exact message list it hands to `ChatUnifier.chat` (`_journal_llm_turn`), immediately after the call returns. `agent.prepare_context` deliberately journals nothing — it runs before the request exists, so anything it wrote would be assembled from configuration. Journal operation ids derive from the per-firing `context_execution_id`, **not** `execution_id`: the latter is generation-scoped, so every message in a generation minted identical ids and the store's idempotency guard silently discarded every turn after the first. They are **also scoped by `agent_node_id`** — two agents wired to one Context node resolve to the same thread, so a firing-only id made them collide on `(thread, operation_id)` and one agent's turns vanished as replays.++**The one place the journal is read back is the `continue_as_new` rollover.** `AgentWorkflow` rolls over under history pressure *only when `context_ref` is set* (`agent_workflow.py`), and a rollover carries references, not the conversation — so the resumed run rebuilds `messages` from system + memory + prompt alone. Without a replay step that means attaching a Context node is what makes a long run silently forget everything it had said: the same class of bug as the original `context_ref` regression, one boundary later. The resumed run therefore calls `agent.reconstruct_context_messages` (read-only, wraps `reconstruct_message_wire_v2`) before its first LLM step. That activity **degrades instead of raising** — an opaque provider checkpoint or an unreadable thread returns `restored: False` with a reason and the run continues from the rebuilt prompt, because a rehydration miss must never turn an unreadable thread into a dead agent. Locked by `tests/temporal/test_agent_workflow.py::TestContextJournalIdentity`, which also asserts the workflow source carries no comment claiming the LLM activity rebuilds its request from the store — that sentence was false for a long time and is exactly what a future reader would re-implement.A Context thread exists only under an admitted generation (`generation > 0`), which only Start assigns — manual canvas Runs journal nothing by design. Each generation gets its own thread, so conversation does not carry across a Stop/Start. Workflow Reset rotates the thread to a fresh epoch and leaves prior events as archived history; the panel's journal view is scoped to the live epoch, or a reset looks like it did nothing.diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex d6f57f13..82e9d2f1 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -429,8 +429,9 @@ Durable event listeners retain their deployment graph as a fallback, but each fi**Canvas-aware tools** opt into receiving the parent workflow's `nodes`/`edges` by declaring `needs_canvas: ClassVar[bool] = True` on their `BaseNode` subclass. The F4.B tool dispatch reads this via `services.node_registry.get_node_class(node_type).needs_canvas` and forwards `context.get("nodes")` / `context.get("edges")` into `tool_payload`; default plugins keep the empty-canvas optimisation. Today only `agentBuilder` opts in (it walks edges to resolve its calling agent and mutates the canvas). Operations inside agentBuilder reload via `database.get_workflow(workflow_id)` so in-run duplicate detection sees mutations from earlier calls in the same workflow run — see [agent_builder section in CLAUDE.md](../CLAUDE.md).-`collect_agent_activities()` registers **16 agent activities**. Seven are the-core loop activities:+`collect_agent_activities()` registers the agent activities — read the live set+from that function rather than trusting a count here, which drifts on every+addition. Seven are the core loop activities:| Activity | Purpose ||---|---|@@ -456,6 +457,24 @@ The other nine support skills and durable delegation:| `agent.finish_delegation` | Idempotently records a delegated child's terminal result and usage. || `agent.finalize_team` | Finalizes a team after its required tasks reach accepted or terminal states. |+The Context journal surface adds `agent.prepare_context` (resolves the thread;+journals nothing, because it runs before the request exists),+`agent.reconstruct_context_messages`, `agent.append_context` and+`agent.compact_context`.++**`agent.reconstruct_context_messages` is the only place the journal is read+back on the execution path.** `AgentWorkflow` rolls over under history pressure+*only when `context_ref` is set*, and a rollover carries references rather than+the conversation — so without a replay step, attaching a Context node is what+makes a long run silently forget everything it had said. The resumed run calls+this activity before its first LLM step and seeds `messages` from the result. It+degrades rather than raising: an opaque provider checkpoint or an unreadable+thread returns `restored: False` with a reason and the run continues from the+rebuilt system + memory + prompt, since a rehydration miss must not turn an+unreadable thread into a dead agent. This does **not** weaken the rule that the+request is always built from `messages` — nothing reconstructs mid-loop, only at+the rollover boundary.+The F4.B compaction threshold is prepared from the model context length andthe ratio configuration. It currently does not read`SessionTokenState.custom_threshold` or `compaction_enabled`; those storeddiff --git a/server/services/temporal/agent_activities.py b/server/services/temporal/agent_activities.pyindex dca7be7b..514ca9bf 100644--- a/server/services/temporal/agent_activities.py+++ b/server/services/temporal/agent_activities.py@@ -1930,6 +1930,7 @@ def collect_agent_activities() -> List[Any]:# plain coroutine invoked by each plugin's own activity wrapper in# services/plugin/base.py, not a Temporal activity of its own.prepare_context,+ reconstruct_context_messages,append_context,compact_context,]@@ -2186,6 +2187,56 @@ async def prepare_context(payload: Dict[str, Any]) -> Dict[str, Any]:return {"context_ref": _ref_dict(thread)}[email protected](name="agent.reconstruct_context_messages")+async def reconstruct_context_messages(payload: Dict[str, Any]) -> Dict[str, Any]:+ """Replay a Context thread's active transcript. Read-only.++ Used only by ``AgentWorkflow`` after ``continue_as_new``. A rollover+ carries references, not the conversation, so without this the resumed run+ rebuilds ``messages`` from system + memory + prompt alone and the agent+ silently forgets everything it had said — but only when a Context node is+ attached, since the rollover itself is gated on ``context_ref``.++ Returns ``{"messages": [...], "restored": bool, "reason": str}``. A thread+ that cannot be replayed reports ``restored: False`` rather than raising, so+ the workflow falls back to the freshly built list instead of failing a run+ over a rehydration miss.+ """++ ref_data = payload.get("context_ref") or {}+ if not ref_data:+ return {"messages": [], "restored": False, "reason": "no_context_ref"}++ from services.agent_context import (+ OpaqueCheckpointError,+ reconstruct_message_wire_v2,+ )++ store = _context_store()+ try:+ _ref, wires = await reconstruct_message_wire_v2(+ store, _context_ref(ref_data)+ )+ except OpaqueCheckpointError:+ # A provider-native checkpoint holds opaque continuation state, not a+ # message list. The provider itself carries that context, so the+ # rebuilt prompt is the correct fallback.+ return {"messages": [], "restored": False, "reason": "opaque_checkpoint"}+ except Exception as exc: # noqa: BLE001 -- never fail a run over replay+ activity.logger.warning(+ "Context transcript replay failed; resuming from the rebuilt "+ f"prompt instead: {exc!r}"+ )+ return {"messages": [], "restored": False, "reason": "replay_failed"}++ messages = [dict(wire) for wire in wires]+ return {+ "messages": messages,+ "restored": bool(messages),+ "reason": "ok" if messages else "empty_thread",+ }++def _pending_tool_identities(calls: Iterable[Mapping[str, Any]],configured_tools: Iterable[Mapping[str, Any]],diff --git a/server/services/temporal/agent_workflow.py b/server/services/temporal/agent_workflow.pyindex db11169b..cf911f9f 100644--- a/server/services/temporal/agent_workflow.py+++ b/server/services/temporal/agent_workflow.py@@ -720,11 +720,14 @@ class AgentWorkflow:# Native executions use the SDK-neutral v2 wire codec. Legacy# histories keep the legacy canonical {type, data} shape exactly.# ---- Context journal -------------------------------------------- # A Context node on the agent's input makes the backend journal the- # source of truth for the transcript: the LLM activity reconstructs- # from the store rather than from `messages`, so a run resumes the- # prior conversation instead of starting empty. The workflow holds- # only references.+ # A Context node makes the backend journal every turn the agent sends.+ # It never steers the request: the LLM activity always builds from+ # ``messages``, and ``context_ref`` only says where to record. The+ # workflow itself holds references, never transcript content.+ #+ # The one place the journal is read back is the continue-as-new+ # rollover below, which carries references rather than the+ # conversation.context_ref: Optional[Dict[str, Any]] = Noneruntime_config_ref = ""if payload.get("context_descriptor"):@@ -746,8 +749,8 @@ class AgentWorkflow:prepared_context.get("runtime_config_ref") or "")if resume:- # A resumed run reconstructs its transcript from the journal, so- # only references and counters cross the rollover boundary.+ # Only references and counters cross the rollover boundary; the+ # transcript is replayed from the journal further down.context_ref = resume.get("context_ref") or context_refruntime_config_ref = (str(resume.get("runtime_config_ref") or "") or runtime_config_ref@@ -788,6 +791,35 @@ class AgentWorkflow:{"type": "human", "data": {"content": user_prompt}})+ # ---- Resume: replay the transcript from the journal --------------+ # continue_as_new carries references, not the conversation, so the+ # list built above is just system + memory + prompt. Without this the+ # agent forgets everything it had said mid-run — and only when a+ # Context node is attached, since the rollover is gated on+ # ``context_ref``. The journal is authoritative here because it holds+ # the exact message list each turn was sent with.+ if resume and context_ref:+ await self._wait_until_resumed()+ replayed = await workflow.execute_activity(+ "agent.reconstruct_context_messages",+ args=[{"context_ref": context_ref}],+ activity_id="reconstruct-context-messages",+ start_to_close_timeout=PERSIST_TURN_TIMEOUT * 2,+ retry_policy=AGENT_ACTIVITY_RETRY,+ )+ restored = replayed.get("messages") or []+ if restored:+ messages = [dict(message) for message in restored]+ else:+ # An opaque provider checkpoint or an unreadable thread. The+ # rebuilt prompt is the honest fallback; log which, so a+ # forgetful agent is diagnosable rather than mysterious.+ workflow.logger.warning(+ "AgentWorkflow resumed without a replayed transcript "+ f"({replayed.get('reason') or 'unknown'}); continuing from "+ "the rebuilt prompt"+ )+# Map LLM tool name -> {node_type, version, task_queue, node_id,# parameters} so the workflow can schedule the right activity# when the LLM emits a tool_call.@@ -864,8 +896,15 @@ class AgentWorkflow:**({"context_ref": context_ref,+ # Scoped by agent node as well as firing: two+ # agents wired to one Context node resolve to the+ # same thread, so a firing-only id made them mint+ # identical operation ids, collide on+ # (thread, operation_id), and have their turns+ # discarded by the store's idempotency guard."journal_operation_id": (- f"{journal_operation_id}:iter:{iteration}"+ f"{journal_operation_id}:{agent_node_id}"+ f":iter:{iteration}"),}if context_refdiff --git a/server/tests/temporal/test_agent_workflow.py b/server/tests/temporal/test_agent_workflow.pyindex 5b39a740..9890d4f0 100644--- a/server/tests/temporal/test_agent_workflow.py+++ b/server/tests/temporal/test_agent_workflow.py@@ -18,6 +18,18 @@ from __future__ import annotationsfrom unittest.mock import AsyncMock, MagicMock+import pytest++# Bounded reference shape carried across a continue_as_new rollover.+_CONTEXT_REF = {+ "workflow_id": "workflow-1",+ "context_node_id": "workflow-1:context:1",+ "generation": 2,+ "thread_id": "session:abc",+ "epoch": 1,+ "revision": 7,+}+class TestAgentWorkflowDefinition:"""``AgentWorkflow`` must be a valid Temporal workflow definition@@ -116,6 +128,144 @@ class TestContextJournalIdentity:assert f"{{execution_id}}:iter" not in sourceassert f'f"{{execution_id}}{suffix}"' not in source+ def test_journal_operation_ids_are_scoped_per_agent_node(self):+ """Two agents on one Context node resolve to the same thread.++ With a firing-scoped id alone they minted identical operation ids,+ collided on (thread, operation_id) and had their turns discarded as+ replays -- so one of the two agents was simply absent from the journal.+ """+ import inspect++ from services.temporal.agent_workflow import AgentWorkflow++ source = inspect.getsource(AgentWorkflow.run)+ assert 'f"{journal_operation_id}:{agent_node_id}"' in source, (+ "journal operation ids must include the agent node id, or "+ "sibling agents sharing a Context node overwrite each other"+ )++ def test_resumed_run_replays_the_transcript_from_the_journal(self):+ """continue_as_new carries references, not the conversation.++ The rollover is gated on ``context_ref``, so without a replay step+ attaching a Context node is what makes a long run forget everything it+ had said -- the same class of bug as the original ``context_ref``+ regression, one boundary later.+ """+ import inspect++ from services.temporal.agent_workflow import AgentWorkflow++ source = inspect.getsource(AgentWorkflow.run)+ assert "agent.reconstruct_context_messages" in source, (+ "a resumed AgentWorkflow must replay its transcript from the "+ "journal before continuing"+ )+ replay_at = source.index("agent.reconstruct_context_messages")+ loop_at = source.index("agent.execute_llm_step")+ assert replay_at < loop_at, (+ "the transcript must be restored before the first LLM step of "+ "the resumed run"+ )++ def test_workflow_never_claims_the_activity_rebuilds_the_request(self):+ """Guards the comment, not the code.++ Two comments used to state that the LLM activity reconstructs from the+ store rather than from ``messages``. That was false, and it is exactly+ the sentence a future reader would implement to reintroduce the+ original bug.+ """+ import inspect++ from services.temporal.agent_workflow import AgentWorkflow++ source = inspect.getsource(AgentWorkflow.run)+ for claim in (+ "reconstructs\n",+ "source of truth for the transcript",+ "rather than from `messages`",+ ):+ assert claim not in source, (+ f"stale claim {claim!r} in AgentWorkflow.run -- the activity "+ f"always builds its request from `messages`"+ )++ @pytest.mark.asyncio+ async def test_replay_returns_the_journalled_transcript(self, monkeypatch):+ import services.agent_context as agent_context+ from services.temporal import agent_activities++ wires = [+ {"version": 2, "role": "system", "content": "you are helpful"},+ {"version": 2, "role": "user", "content": "hello"},+ ]++ async def _replay(_store, ref):+ return ref, wires++ monkeypatch.setattr(agent_activities, "_context_store", lambda: object())+ monkeypatch.setattr(agent_context, "reconstruct_message_wire_v2", _replay)++ result = await agent_activities.reconstruct_context_messages(+ {"context_ref": _CONTEXT_REF}+ )++ assert result["restored"] is True+ assert result["messages"] == wires++ @pytest.mark.asyncio+ @pytest.mark.parametrize(+ "failure, reason",+ [+ ("opaque", "opaque_checkpoint"),+ ("boom", "replay_failed"),+ ],+ )+ async def test_replay_degrades_instead_of_failing_the_run(+ self, monkeypatch, failure, reason+ ):+ """A rehydration miss must never kill a resumed run.++ The rebuilt system + memory + prompt is a worse starting point than+ the journal, but it is a working one; raising here would turn an+ unreadable thread into a dead agent.+ """+ import services.agent_context as agent_context+ from services.agent_context import OpaqueCheckpointError+ from services.temporal import agent_activities++ async def _replay(_store, _ref):+ raise (+ OpaqueCheckpointError("provider-native checkpoint")+ if failure == "opaque"+ else RuntimeError("blob missing")+ )++ monkeypatch.setattr(agent_activities, "_context_store", lambda: object())+ monkeypatch.setattr(agent_context, "reconstruct_message_wire_v2", _replay)++ result = await agent_activities.reconstruct_context_messages(+ {"context_ref": _CONTEXT_REF}+ )++ assert result["restored"] is False+ assert result["reason"] == reason+ assert result["messages"] == []++ @pytest.mark.asyncio+ async def test_replay_without_a_context_ref_is_a_no_op(self):+ from services.temporal import agent_activities++ result = await agent_activities.reconstruct_context_messages({})++ assert result == {+ "messages": [],+ "restored": False,+ "reason": "no_context_ref",+ }+def test_llm_step_has_a_single_implementation(self):"""A Context node observes execution; it must not steer it.@@ -382,6 +532,7 @@ class TestAgentActivities:"agent.prepare_context","agent.prepare_payload","agent.queue_delegation",+ "agent.reconstruct_context_messages","agent.refresh_tools","agent.register_task_execution","agent.release_subagent_permit",