Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

Gate memory extraction on the user's message, not the combined turn length

Rufus469cf25a8e04
diff --git a/packages/core/openexecutive/architecture/architecture-facts.yaml b/packages/core/openexecutive/architecture/architecture-facts.yaml
index e091f441..6a938823 100644
--- a/packages/core/openexecutive/architecture/architecture-facts.yaml
+++ b/packages/core/openexecutive/architecture/architecture-facts.yaml
@@ -915,6 +915,17 @@ memory:
populates `domain` but not `department`, so today the mirror only
fires for explicit dept-scoped writers; falling back to `domain`
is a deliberate follow-up decision, not silent drift.
+ Extraction is gated on MIN_USER_CHARS_FOR_EXTRACTION, measured on the
+ USER's message only. It was a combined user+assistant floor of 1500,
+ which on a live tenant blocked 9 of 16 turns — every instruction the
+ principal gave — while admitting the 7 long analytical exchanges that
+ contained no commitment; the extractor ran 13 times and stored nothing.
+ Decisive instructions are short and the analysis around them is long, so
+ a combined floor discards what it exists to capture, and the user's own
+ words are the only text `_is_valid_user_commitment` accepts a quote from.
+ Every pass now writes a `memory_extraction` audit row (proposed / stored /
+ dropped-with-reason) — before it, an extractor storing nothing looked
+ identical to one with nothing to store.
peer_memory:
what: |
diff --git a/packages/core/openexecutive/architecture/prebuilt/audit.json b/packages/core/openexecutive/architecture/prebuilt/audit.json
index a9226677..3978ed70 100644
--- a/packages/core/openexecutive/architecture/prebuilt/audit.json
+++ b/packages/core/openexecutive/architecture/prebuilt/audit.json
@@ -1,7 +1,7 @@
{
"section_id": "audit",
"title": "Audit Log",
- "markdown": "The audit log is a persistent, searchable, append-only record of what the system did — distinct from episodic memory (which biases future LLM behavior). Every event flows through `audit/logger.log_event`, a synchronous SQLite writer (WAL + 5s busy_timeout) that appends one row to the `audit_log` table in `episodic_memory.db`.\n\n### Event types\nRows are tagged by `event_type`, including `chat_turn`, `specialist_consult`, `tool_invocation`, `scheduled_action`, `alert`, `integration_inbound` (Slack / Discord / email / Telegram / Google Chat), plus `knowledge_retrieval`, `cache_event` (one row per model call from every call site — Executive turns, specialist consults (chat turns and workflow steps), research specialists, the research routing and watchlist passes, triage, memory extraction — carrying tokens, cache hits, cost, server-side search count and, inside a research run, its `run_id`), `memory_snapshot`, `committee_review`, `peer_memory`, and the external-signal/monitor events.\n\n### Session & turn context\nEach row carries `session_id`, `turn_id`, `actor`, `summary`, `details_json`, `full_json`, and `department`. A `turn_id` (`f\"t-{uuid12}\"`) is set per Executive turn and propagated automatically via ContextVars to retriever / specialist / tool / cache emits, grouping one user-assistant exchange. `summary` is the searchable column (LIKE, ~300 char cap); `details_json` is structured and not searched.\n\n### Views\n- `GET /audit` — filterable list with a \"Group by session\" toggle and session pills.\n- `GET /audit/session/{id}` — full ordered timeline plus a server-derived node/edge graph rendered as a three-pane React Flow chart.\n\n### Invariant\nAudit writes never break a chat turn: all writes swallow exceptions and log a warning, and instrumentation observes only — it never touches request building or the cached prompt blocks. Audit is append-only: no UPDATE or DELETE paths exist in product code.",
+ "markdown": "The audit log is a persistent, searchable, append-only record of what the system did — distinct from episodic memory (which biases future LLM behavior). Every event flows through `audit/logger.log_event`, a synchronous SQLite writer (WAL + 5s busy_timeout) that appends one row to the `audit_log` table in `episodic_memory.db`.\n\n### Event types\nRows are tagged by `event_type`, including `chat_turn`, `specialist_consult`, `tool_invocation`, `scheduled_action`, `alert`, `integration_inbound` (Slack / Discord / email / Telegram / Google Chat), plus `knowledge_retrieval`, `memory_extraction` (one row per episodic-extraction pass: how many items the model proposed, how many were stored, and why each drop was rejected — a `proposed>0, stored=0` streak means the quote validator is rejecting everything, which is otherwise invisible), `cache_event` (one row per model call from every call site — Executive turns, specialist consults (chat turns and workflow steps), research specialists, the research routing and watchlist passes, triage, memory extraction — carrying tokens, cache hits, cost, server-side search count and, inside a research run, its `run_id`), `memory_snapshot`, `committee_review`, `peer_memory`, and the external-signal/monitor events.\n\n### Session & turn context\nEach row carries `session_id`, `turn_id`, `actor`, `summary`, `details_json`, `full_json`, and `department`. A `turn_id` (`f\"t-{uuid12}\"`) is set per Executive turn and propagated automatically via ContextVars to retriever / specialist / tool / cache emits, grouping one user-assistant exchange. `summary` is the searchable column (LIKE, ~300 char cap); `details_json` is structured and not searched.\n\n### Views\n- `GET /audit` — filterable list with a \"Group by session\" toggle and session pills.\n- `GET /audit/session/{id}` — full ordered timeline plus a server-derived node/edge graph rendered as a three-pane React Flow chart.\n\n### Invariant\nAudit writes never break a chat turn: all writes swallow exceptions and log a warning, and instrumentation observes only — it never touches request building or the cached prompt blocks. Audit is append-only: no UPDATE or DELETE paths exist in product code.",
"mermaid": "flowchart TD\n Chat[\"Chat turn / consult / tool call\"]\n Sched[\"Scheduled action\"]\n Inbound[\"Inbound integration\"]\n Log[\"audit/logger.log_event\"]\n DB[\"audit_log table (append-only)\"]\n ListView[\"/audit list view\"]\n SessView[\"/audit/session/{id} flow chart\"]\n Chat -->|event + turn_id| Log\n Sched --> Log\n Inbound --> Log\n Log -->|swallow errors, never block| DB\n DB --> ListView\n DB --> SessView\n class Chat entry\n class Sched entry\n class Inbound entry\n class Log compute\n class DB storage\n class ListView hot\n class SessView hot",
"generated_at": "2026-06-01T00:00:00+00:00"
}
diff --git a/packages/core/openexecutive/architecture/prebuilt/lifecycle.json b/packages/core/openexecutive/architecture/prebuilt/lifecycle.json
index 00ed779b..4724aa9c 100644
--- a/packages/core/openexecutive/architecture/prebuilt/lifecycle.json
+++ b/packages/core/openexecutive/architecture/prebuilt/lifecycle.json
@@ -1,7 +1,7 @@
{
"section_id": "lifecycle",
"title": "Request Lifecycle",
- "markdown": "A single chat message makes a full round-trip through the API, the orchestrator's tool-use loop, a parallel specialist fan-out, and synthesis — all streamed back as server-sent events.\n\n### Steps\n\n1. **Arrival** — the client posts to `POST /chat` (`openexecutive.api.routes.chat`). The turn carries a `<current_speaker>` hint resolved from the caller's `person_id`, plus a `<briefing>` digest of current open alerts (built in parallel with RAG/episodic/peer-memory fetches) so the Executive can discuss any item shown on the `/today` briefing page A turn arriving from a chat adapter instead of the web app also carries a `<channel>` block naming the surface and stating which approvals can be completed there; on Slack the `<briefing>` digest is gated to the principal's DMs. All three ride in the **user turn**, never a cached system block.\n2. **Tool-use loop** — the `Executive` calls Claude. For a cross-domain question the assistant turn emits multiple `consult_specialist` tool-use blocks at once.\n3. **Parallel fan-out** — `openexecutive.orchestrator.router.route_parallel` dispatches those calls concurrently, bounded by `MAX_PARALLEL_SPECIALISTS`; calls past the cap get a skip tool_result.\n4. **Per-specialist RAG** — each specialist runs its own domain-filtered ChromaDB query, then reasons over the retrieved chunks and returns analysis as a tool_result.\n5. **Synthesis** — the Executive feeds the tool_results back to Claude and streams one coherent reply; a `thinking` keepalive covers the specialist round.\n6. **Optional committee review** — when enabled, `Executive.stream_chat_with_committee()` runs a silent draft, parallel reviewer critiques, then a streamed revision pass.\n7. **Side effects** — any side-effecting tool call surfaces an `action_taken` chip; the stream closes with a `done` event. One side-effecting tool is gated server-side rather than by its description: on a turn that arrived from a chat channel, `ack_alert` will only accept an `alert_id` the session recorded as having been shown to the model this turn — and it fails closed, so an id it cannot even parse is refused rather than waved through to the handler's own parse. The unattended passes (the nightly reflection, the research synthesis) do not carry the tool at all, because they set no origin channel and so would not be checked.\n8. **Tool failures** — a handler that raises instead of returning its JSON error is isolated: both tool gathers run with `return_exceptions=True`, the model gets an error `tool_result` naming only the exception's TYPE — a message can carry a filesystem path, a validation error echoing the input, or a third-party HTTP body, and anything in model context can be quoted back to the user, so the full repr stays in the log and the audit row — and the failure is audited with `ok=false`. Only `CancelledError` is re-raised, so turn-timeout and client-disconnect handling still work. Before this one crashing tool aborted the whole turn, which reached the user as an unexplained apology with no audit row (#136).\n\nWeb search, when invoked on the Anthropic path, executes inside the Anthropic API itself (`web_search_20250305`) and returns within the same generation; for any model routed via OpenRouter the same tool becomes OpenRouter's `openrouter:web_search` server tool, which also runs inside the generation with the same search cap and domain lists.",
+ "markdown": "A single chat message makes a full round-trip through the API, the orchestrator's tool-use loop, a parallel specialist fan-out, and synthesis — all streamed back as server-sent events.\n\n### Steps\n\n1. **Arrival** — the client posts to `POST /chat` (`openexecutive.api.routes.chat`). The turn carries a `<current_speaker>` hint resolved from the caller's `person_id`, plus a `<briefing>` digest of current open alerts (built in parallel with RAG/episodic/peer-memory fetches) so the Executive can discuss any item shown on the `/today` briefing page A turn arriving from a chat adapter instead of the web app also carries a `<channel>` block naming the surface and stating which approvals can be completed there; on Slack the `<briefing>` digest is gated to the principal's DMs. All three ride in the **user turn**, never a cached system block.\n2. **Tool-use loop** — the `Executive` calls Claude. For a cross-domain question the assistant turn emits multiple `consult_specialist` tool-use blocks at once.\n3. **Parallel fan-out** — `openexecutive.orchestrator.router.route_parallel` dispatches those calls concurrently, bounded by `MAX_PARALLEL_SPECIALISTS`; calls past the cap get a skip tool_result.\n4. **Per-specialist RAG** — each specialist runs its own domain-filtered ChromaDB query, then reasons over the retrieved chunks and returns analysis as a tool_result.\n5. **Synthesis** — the Executive feeds the tool_results back to Claude and streams one coherent reply; a `thinking` keepalive covers the specialist round.\n6. **Optional committee review** — when enabled, `Executive.stream_chat_with_committee()` runs a silent draft, parallel reviewer critiques, then a streamed revision pass.\n7. **Side effects** — any side-effecting tool call surfaces an `action_taken` chip; the stream closes with a `done` event. One side-effecting tool is gated server-side rather than by its description: on a turn that arrived from a chat channel, `ack_alert` will only accept an `alert_id` the session recorded as having been shown to the model this turn — and it fails closed, so an id it cannot even parse is refused rather than waved through to the handler's own parse. The unattended passes (the nightly reflection, the research synthesis) do not carry the tool at all, because they set no origin channel and so would not be checked. After the stream closes, the turn is also handed to the episodic memory extractor (`memory.episodic.schedule_extraction`, fire-and-forget) — but only if the USER's message clears `MIN_USER_CHARS_FOR_EXTRACTION`. Both this path and the committee path share that constant and both measure the user's side only: the floor used to sum in the assistant's reply, which let a verbose answer drag a trivial turn in and discarded the short, decisive instructions that are the whole point of extracting. See **Memory** for what that cost on a live tenant.\n8. **Tool failures** — a handler that raises instead of returning its JSON error is isolated: both tool gathers run with `return_exceptions=True`, the model gets an error `tool_result` naming only the exception's TYPE — a message can carry a filesystem path, a validation error echoing the input, or a third-party HTTP body, and anything in model context can be quoted back to the user, so the full repr stays in the log and the audit row — and the failure is audited with `ok=false`. Only `CancelledError` is re-raised, so turn-timeout and client-disconnect handling still work. Before this one crashing tool aborted the whole turn, which reached the user as an unexplained apology with no audit row (#136).\n\nWeb search, when invoked on the Anthropic path, executes inside the Anthropic API itself (`web_search_20250305`) and returns within the same generation; for any model routed via OpenRouter the same tool becomes OpenRouter's `openrouter:web_search` server tool, which also runs inside the generation with the same search cap and domain lists.",
"mermaid": "sequenceDiagram\n participant C as Client\n participant API as FastAPI /chat\n participant E as Executive\n participant CL as Claude\n participant S as Specialists\n participant KB as ChromaDB\n C->>API: POST /chat message\n API->>E: start turn\n E->>CL: messages (tool-use loop)\n CL-->>E: consult_specialist calls\n E->>S: route_parallel fan-out\n S->>KB: domain-filtered RAG\n KB-->>S: chunks\n S-->>E: analysis tool_results\n E->>CL: synthesize\n CL-->>E: streamed reply\n E-->>API: SSE chunks then done\n API-->>C: streamed response",
"generated_at": "2026-09-18T20:54:42.334412+00:00"
}
diff --git a/packages/core/openexecutive/architecture/prebuilt/memory.json b/packages/core/openexecutive/architecture/prebuilt/memory.json
index 645b4650..df755372 100644
--- a/packages/core/openexecutive/architecture/prebuilt/memory.json
+++ b/packages/core/openexecutive/architecture/prebuilt/memory.json
@@ -1,7 +1,7 @@
{
"section_id": "memory",
"title": "Memory System",
- "markdown": "The Executive's durable memory is an episodic SQLite store at `./episodic_memory.db`, managed by `openexecutive.memory.episodic`. It holds four tables:\n\n- `decisions` — choices made, with `domain`, `summary`, `rationale`, `outcome`, and `tags`.\n- `initiatives` — tracked efforts with a `status` and `summary` (idempotent upserts).\n- `advice_given` — query/advice pairs per domain.\n- `scheduled_actions` — a dispatch queue, treated separately from recall memory.\n\n### Session scoping\n\n`decisions` and `advice_given` rows carry a `session_id`, so per-thread channels (Discord, Telegram) can pull just-this-thread history without leaking cross-conversation context. A null `session_id` falls back to a global view.\n\n### Surfacing\n\n`openexecutive.memory.episodic.format_for_prompt` renders recent decisions and initiatives into a compact `<past_decisions>` block injected into the user turn — alongside `<retrieved_context>` and peer memory — never into a cached system block.\n\nThe chat route also injects a `<briefing>` block into the same user turn: a compact digest of the current open (unread) alerts, built by `openexecutive.briefing.context.format_open_alerts_for_prompt` (company-wide, `status=\"unread\"`, mirroring `/today`). This is what lets the Executive discuss a briefing item the principal clicked or named on `/today` (e.g. \"Gulf Coast Port Cyberattack\"). It is resolved in parallel with the other context fetches, omitted when there are no open alerts, and never placed in a cached system block.\n\n### System of record\n\nSQL is the system of record; the peer-memory service is a recall layer mirrored from it, not a replacement. When `store_decision` / `store_initiative` / `store_advice` insert a row whose `department` is non-empty and not `\"general\"`, the row is mirrored as a kind-tagged note into that department's peer via `append_department_note`. The mirror fires only after a successful INSERT, is fire-and-forget, and never rolls back the SQL row on peer-memory failure. `scheduled_actions` is never mirrored — it is a queue, not memory.",
+ "markdown": "The Executive's durable memory is an episodic SQLite store at `./episodic_memory.db`, managed by `openexecutive.memory.episodic`. It holds four tables:\n\n- `decisions` — choices made, with `domain`, `summary`, `rationale`, `outcome`, and `tags`.\n- `initiatives` — tracked efforts with a `status` and `summary` (idempotent upserts).\n- `advice_given` — query/advice pairs per domain.\n- `scheduled_actions` — a dispatch queue, treated separately from recall memory.\n\n### What gets extracted, and when\n\n`episodic.extract_and_store` runs after a turn and asks a utility-fast model for decisions, initiatives and advice. Every candidate must carry a `user_commitment_quote` that appears verbatim in the principal's own message (`_is_valid_user_commitment`) — the model has repeatedly tried to log the Executive's *recommendations* as the user's commitments, and that quote is the hard gate against it.\n\nA turn reaches the extractor only if the **user's** message clears `MIN_USER_CHARS_FOR_EXTRACTION`. This was a combined user+assistant floor, and on a live tenant it selected almost exactly the wrong turns: 9 of 16 exchanges were blocked, and the 9 held every instruction the principal gave, while the 7 admitted were long analytical exchanges with no commitment in them. The extractor ran 13 times and stored nothing. Decisive instructions are short (\"Correct it there\", \"Don't track this\"); the analysis around them is long, so a floor that scales with the Executive's verbosity discards exactly what it is meant to capture. The floor now measures only the user's side — the one place a commitment, and an acceptable quote, can come from.\n\nEach pass writes a `memory_extraction` audit row: how many items the model proposed, how many were stored, and the reason each drop was rejected. Without it, \"the model found nothing\" and \"the model found things and every one was rejected\" are indistinguishable from outside the database — which is why the failure above survived undetected and took a SQLite session on the tenant to find. A `proposed>0, stored=0` streak is the signal to watch.\n\n### Session scoping\n\n`decisions` and `advice_given` rows carry a `session_id`, so per-thread channels (Discord, Telegram) can pull just-this-thread history without leaking cross-conversation context. A null `session_id` falls back to a global view.\n\n### Surfacing\n\n`openexecutive.memory.episodic.format_for_prompt` renders recent decisions and initiatives into a compact `<past_decisions>` block injected into the user turn — alongside `<retrieved_context>` and peer memory — never into a cached system block.\n\nThe chat route also injects a `<briefing>` block into the same user turn: a compact digest of the current open (unread) alerts, built by `openexecutive.briefing.context.format_open_alerts_for_prompt` (company-wide, `status=\"unread\"`, mirroring `/today`). This is what lets the Executive discuss a briefing item the principal clicked or named on `/today` (e.g. \"Gulf Coast Port Cyberattack\"). It is resolved in parallel with the other context fetches, omitted when there are no open alerts, and never placed in a cached system block.\n\n### System of record\n\nSQL is the system of record; the peer-memory service is a recall layer mirrored from it, not a replacement. When `store_decision` / `store_initiative` / `store_advice` insert a row whose `department` is non-empty and not `\"general\"`, the row is mirrored as a kind-tagged note into that department's peer via `append_department_note`. The mirror fires only after a successful INSERT, is fire-and-forget, and never rolls back the SQL row on peer-memory failure. `scheduled_actions` is never mirrored — it is a queue, not memory.",
"mermaid": "flowchart TD\n Store[\"store_decision / store_initiative / store_advice\"]\n DB[\"episodic_memory.db<br/>decisions / initiatives<br/>advice_given / scheduled_actions\"]\n Fmt[\"format_for_prompt\"]\n Turn[\"User turn<br/>past_decisions block\"]\n Mirror[\"append_department_note\"]\n Peer[\"Department peer<br/>(Honcho)\"]\n Store --> DB\n DB --> Fmt\n Fmt --> Turn\n Store -->|\"dept set & not general\"| Mirror\n Mirror -->|\"fire-and-forget\"| Peer\n class Store entry\n class DB storage\n class Fmt compute\n class Turn hot\n class Peer external\n",
"generated_at": "2026-06-01T00:00:00+00:00"
}
diff --git a/packages/core/openexecutive/memory/episodic.py b/packages/core/openexecutive/memory/episodic.py
index 6147e0b0..16440611 100644
--- a/packages/core/openexecutive/memory/episodic.py
+++ b/packages/core/openexecutive/memory/episodic.py
@@ -1833,12 +1833,79 @@ def _is_valid_user_commitment(quote: str, user_message: str) -> bool:
_MAX_INPUT_CHARS = 20_000 # cap each side to avoid runaway cost
-# Minimum combined chars (user + assistant) required to schedule extraction.
-# Below this floor, the turn is a clarifying question or small-talk exchange
-# and the LLM gate's cost isn't justified. Both stream_chat and the committee
-# path use this constant — keeping them symmetric prevents the committee path
-# from firing 2.5x more often than streaming, which it used to.
-MIN_TURN_CHARS_FOR_EXTRACTION = 1500
+# Minimum chars in the USER's message required to schedule extraction.
+#
+# This was a combined user+assistant floor of 1500, and measured against real
+# traffic it selected almost exactly the wrong turns. On one tenant it blocked
+# 9 of 16 exchanges, and the 9 included every substantive instruction the
+# principal gave — "It's a mistake on the lp purchased properties worksheet.
+# Correct it there. And record this as fixed." (1312 combined), "This is not
+# relevant for us. Don't track this" (910), "The briefing is incorrect. We are
+# not under water 2.9m" (1032) — while the 7 it admitted were long analytical
+# exchanges containing no user commitment at all. The extractor ran 13 times
+# and stored nothing, because it only ever saw the turns with nothing in them.
+#
+# The combined length was the wrong axis. A decision lives in what the
+# PRINCIPAL said, which is also the only text `_is_valid_user_commitment` will
+# accept a quote from; the Executive's analysis around it can run to thousands
+# of characters without containing a single commitment. Decisive instructions
+# are short by nature ("Correct it there", "Don't track this"), so a floor that
+# scales with the assistant's verbosity systematically discards them.
+#
+# The floor now only has to separate a real instruction from a bare
+# acknowledgement — "No", "Done", "Who am I?" — which is what the old comment
+# meant by small talk. Both stream_chat and the committee path use this
+# constant; keeping them symmetric prevents the committee path from firing more
+# often than streaming, which it used to.
+MIN_USER_CHARS_FOR_EXTRACTION = 40
+
+# Back-compat alias. External callers (and older tests) may still import the
+# old name; it no longer reflects what is measured, so nothing new should use
+# it.
+MIN_TURN_CHARS_FOR_EXTRACTION = MIN_USER_CHARS_FOR_EXTRACTION
+
+
+def _audit_extraction(
+ proposed: dict[str, int],
+ stored: dict[str, int],
+ dropped: list[str],
+ *,
+ session_id: str,
+) -> None:
+ """One `memory_extraction` audit row per extraction pass.
+
+ The point is that "the model proposed nothing" and "the model proposed
+ things and every one was rejected" are different failures with different
+ fixes, and until this row existed they were indistinguishable outside a
+ SQLite session on the tenant. A pass that proposes and stores nothing is
+ normal on most turns, so this is deliberately not a warning — the signal
+ is the RATIO over time, which a `proposed>0, stored=0` streak makes
+ obvious.
+
+ Never raises: auditing an extraction must not be able to break the turn
+ that produced it.
+ """
+ total_proposed = sum(proposed.values())
+ total_stored = sum(stored.values())
+ try:
+ from openexecutive.audit import log_event
+
+ log_event(
+ "memory_extraction",
+ f"proposed={total_proposed} stored={total_stored} dropped={len(dropped)}",
+ session_id=session_id or None,
+ actor="memory_extractor",
+ details={
+ "proposed": proposed,
+ "stored": stored,
+ "dropped_count": len(dropped),
+ # Reasons, not content: enough to tell a validator rejection
+ # from an empty pass without copying company data into the log.
+ "dropped": dropped[:10],
+ },
+ )
+ except Exception:
+ logger.debug("memory extraction audit failed", exc_info=True)
async def extract_and_store(
@@ -1936,6 +2003,16 @@ async def _extract_and_store(
log_model_usage(response, model=routing_model, actor="memory_extractor")
+ # Extraction outcome, per turn. Without this a working extractor and a
+ # broken one look identical from the outside: drops were `logger.debug`
+ # and a successful store wrote no row either, so the only symptom of a
+ # total failure was an empty `decisions` table nobody was watching. It
+ # took reading a tenant's SQLite to find that the turn gate had been
+ # discarding every commitment for the whole life of the install.
+ proposed = {"decisions": 0, "initiatives": 0, "advice": 0}
+ stored = {"decisions": 0, "initiatives": 0, "advice": 0}
+ dropped: list[str] = []
+
for block in response.content:
if block.type != "tool_use" or block.name != "store_memories":
continue
@@ -1950,18 +2027,21 @@ async def _extract_and_store(
# hard gate that catches that pattern; the prompt is now just
# the soft instruction layer.
for d in inp.get("decisions", []):
+ proposed["decisions"] += 1
domain = d.get("domain", "general")
summary = d.get("summary", "")
quote = d.get("user_commitment_quote", "")
if not summary:
continue
if not _is_valid_user_commitment(quote, user_message):
+ dropped.append(f"decision:bad_quote:{summary[:60]}")
logger.debug(
"Dropping decision — invalid user_commitment_quote %r (summary=%r)",
quote[:120],
summary[:80],
)
continue
+ stored["decisions"] += 1
store_decision(
domain=domain,
summary=summary,
@@ -1970,6 +2050,7 @@ async def _extract_and_store(
db_path=db_path,
)
for i in inp.get("initiatives", []):
+ proposed["initiatives"] += 1
title = i.get("title", "")
status = i.get("status", "active")
summary = i.get("summary", "")
@@ -1977,14 +2058,17 @@ async def _extract_and_store(
if not (title and summary):
continue
if not _is_valid_user_commitment(quote, user_message):
+ dropped.append(f"initiative:bad_quote:{title[:60]}")
logger.debug(
"Dropping initiative — invalid user_commitment_quote %r (title=%r)",
quote[:120],
title[:80],
)
continue
+ stored["initiatives"] += 1
store_initiative(title=title, status=status, summary=summary, db_path=db_path)
for a in inp.get("advice", []):
+ proposed["advice"] += 1
domain = a.get("domain", "general")
query_summary = a.get("query_summary", "")
advice_summary = a.get("advice_summary", "")
@@ -1992,12 +2076,14 @@ async def _extract_and_store(
if not (query_summary and advice_summary):
continue
if not _is_valid_user_commitment(quote, user_message):
+ dropped.append(f"advice:bad_quote:{query_summary[:60]}")
logger.debug(
"Dropping advice — invalid user_commitment_quote %r (query=%r)",
quote[:120],
query_summary[:80],
)
continue
+ stored["advice"] += 1
store_advice(
domain=domain,
query_summary=query_summary,
@@ -2006,6 +2092,8 @@ async def _extract_and_store(
db_path=db_path,
)
+ _audit_extraction(proposed, stored, dropped, session_id=session_id)
+
except Exception:
logger.exception("Episodic memory extraction failed — skipping silently")
diff --git a/packages/core/openexecutive/orchestrator/executive.py b/packages/core/openexecutive/orchestrator/executive.py
index e52245fe..4c13e540 100644
--- a/packages/core/openexecutive/orchestrator/executive.py
+++ b/packages/core/openexecutive/orchestrator/executive.py
@@ -792,7 +792,7 @@ class Executive:
)
from openexecutive.memory.episodic import (
- MIN_TURN_CHARS_FOR_EXTRACTION,
+ MIN_USER_CHARS_FOR_EXTRACTION,
schedule_extraction,
)
@@ -808,7 +808,7 @@ class Executive:
# call time, so scheduling it out here would snapshot (None,
# None) and the memory_extractor's model call would record
# unattributed however correct the snapshot itself was.
- if len(full_response) + len(user_message) >= MIN_TURN_CHARS_FOR_EXTRACTION:
+ if len(user_message.strip()) >= MIN_USER_CHARS_FOR_EXTRACTION:
schedule_extraction(
user_message, full_response, session_id=session.session_id
)
@@ -1220,10 +1220,10 @@ class Executive:
)
from openexecutive.memory.episodic import (
- MIN_TURN_CHARS_FOR_EXTRACTION,
+ MIN_USER_CHARS_FOR_EXTRACTION,
schedule_extraction,
)
- if len(final_response) + len(user_message) >= MIN_TURN_CHARS_FOR_EXTRACTION:
+ if len(user_message.strip()) >= MIN_USER_CHARS_FOR_EXTRACTION:
schedule_extraction(user_message, final_response, session_id=session.session_id)
# Mirror the completed exchange into Honcho (see stream_chat for
diff --git a/packages/core/tests/unit/test_episodic.py b/packages/core/tests/unit/test_episodic.py
index 308aeb51..6718b4ea 100644
--- a/packages/core/tests/unit/test_episodic.py
+++ b/packages/core/tests/unit/test_episodic.py
@@ -443,17 +443,34 @@ def test_extraction_prompt_includes_source_attribution_rule() -> None:
)
-def test_min_turn_chars_for_extraction_is_exported_and_sane() -> None:
- """The threshold is the single floor that gates extraction in both
- stream_chat and the committee path. If it disappears or drops back into
- the low-hundreds, extraction starts firing on trivial Q&A again.
+def test_extraction_floor_admits_instructions_and_blocks_acknowledgements() -> None:
+ """The floor gates extraction in both stream_chat and the committee path.
+
+ This asserted `>= 1000` on a COMBINED user+assistant length, to stop
+ extraction firing on trivial Q&A. The cost concern was real; the axis was
+ not. Measured on a live tenant, that floor blocked 9 of 16 exchanges and
+ the 9 contained every instruction the principal gave, because decisive
+ instructions are short and the Executive's analysis around them is long.
+ The extractor ran 13 times and stored nothing.
+
+ The floor now measures the user's own message — the only text a commitment
+ can come from, and the only text the quote validator will accept. Pinned to
+ real messages rather than a magic number, so a future change is judged on
+ what it admits and blocks, not on whether a constant looks big enough.
"""
- from openexecutive.memory.episodic import MIN_TURN_CHARS_FOR_EXTRACTION
+ from openexecutive.memory.episodic import MIN_USER_CHARS_FOR_EXTRACTION
- assert isinstance(MIN_TURN_CHARS_FOR_EXTRACTION, int)
- # Anything under 1000 would re-create the bug — clarifying questions
- # are commonly ~500-800 chars combined.
- assert MIN_TURN_CHARS_FOR_EXTRACTION >= 1000
+ assert isinstance(MIN_USER_CHARS_FOR_EXTRACTION, int)
+ # Real instructions from the tenant that the old floor discarded.
+ for instruction in (
+ "This is not relevant for us. Don't track this",
+ "Well remember you fixed it or something. So we don't keep getting. Notified.",
+ ):
+ assert len(instruction.strip()) >= MIN_USER_CHARS_FOR_EXTRACTION
+
+ # Bare acknowledgements still buy nothing — this is what the floor is for.
+ for noise in ("No", "Done", "ok", "yes", "Nope."):
+ assert len(noise.strip()) < MIN_USER_CHARS_FOR_EXTRACTION
def test_executive_call_sites_use_min_turn_chars_constant() -> None:
@@ -469,10 +486,15 @@ def test_executive_call_sites_use_min_turn_chars_constant() -> None:
src = inspect.getsource(executive)
# The constant must be referenced at least twice (once per call site).
- assert src.count("MIN_TURN_CHARS_FOR_EXTRACTION") >= 2, (
- "executive.py must reference MIN_TURN_CHARS_FOR_EXTRACTION in both "
+ assert src.count("MIN_USER_CHARS_FOR_EXTRACTION") >= 2, (
+ "executive.py must reference MIN_USER_CHARS_FOR_EXTRACTION in both "
"the stream_chat and stream_chat_with_committee extraction guards"
)
+ # And they must measure the USER's message. Summing in the assistant's
+ # length is the original defect: it let a verbose reply drag a trivial turn
+ # in, and discarded a one-line instruction answered concisely.
+ assert "len(full_response) + len(user_message)" not in src
+ assert "len(final_response) + len(user_message)" not in src
# And the old hard-coded thresholds must be gone — guard against a
# partial revert that wires the constant in one place but leaves the
# other path on the old number.
diff --git a/packages/core/tests/unit/test_extraction_gate.py b/packages/core/tests/unit/test_extraction_gate.py
new file mode 100644
index 00000000..671b912f
--- /dev/null
+++ b/packages/core/tests/unit/test_extraction_gate.py
@@ -0,0 +1,117 @@
+"""The turn gate in front of the episodic memory extractor.
+
+Regression set for an install where the extractor ran 13 times, cost real
+money, and stored nothing. The gate was a combined user+assistant floor of
+1500 chars, and on real traffic it selected almost exactly the wrong turns:
+of 16 exchanges it blocked 9, and those 9 held every instruction the principal
+gave, while the 7 it admitted were long analytical exchanges with no
+commitment in them at all.
+
+The turns below are the real ones, with their real lengths.
+"""
+from __future__ import annotations
+
+from typing import Any
+from unittest import mock
+
+import pytest
+
+from openexecutive.memory import episodic
+
+# (user message, assistant length, what it is) — measured from the tenant.
+_COMMITMENTS = [
+ ("It's a mistake on the lp purchased properties worksheet. That's it. "
+ "Correct it there. And record this as fixed.", 1200),
+ ("Nope. The briefing is incorrect. We are not under water 2.9m that is "
+ "just how the finicals look as received Q2", 920),
+ ("This is not relevant for us. Don't track this", 864),
+ ("Well remember you fixed it or something. So we don't keep getting. "
+ "Notified.", 612),
+]
+
+# Bare acknowledgements and one-line questions: nothing to extract from the
+# user's own words, which is what the floor is actually for.
+_NOISE = ["No", "Done", "Nope.", "Who am I?", "ok", "yes"]
+
+
[email protected]("message,assistant_len", _COMMITMENTS)
+def test_real_commitments_reach_the_extractor(message: str, assistant_len: int) -> None:
+ """Every one of these was blocked by the old combined-length floor.
+
+ Each is well under 1500 chars combined — `len(user) + assistant_len` — yet
+ each is the principal giving a decision. Asserted against the real lengths
+ so the test states the bug rather than restating the new constant.
+ """
+ assert len(message) + assistant_len < 1500, "should have been blocked before"
+ assert len(message.strip()) >= episodic.MIN_USER_CHARS_FOR_EXTRACTION
+
+
[email protected]("message", _NOISE)
+def test_bare_acknowledgements_are_still_skipped(message: str) -> None:
+ """The floor still has a job: an LLM call on "No" buys nothing."""
+ assert len(message.strip()) < episodic.MIN_USER_CHARS_FOR_EXTRACTION
+
+
+def test_a_long_assistant_reply_no_longer_drags_a_short_turn_in() -> None:
+ """The inverse of the bug: verbosity on the Executive's side is not
+ evidence the principal committed to anything, and the quote validator will
+ only ever accept text from the user's message anyway."""
+ assert len("Done".strip()) < episodic.MIN_USER_CHARS_FOR_EXTRACTION
+
+
+def test_old_constant_still_resolves_for_external_callers() -> None:
+ assert episodic.MIN_TURN_CHARS_FOR_EXTRACTION == episodic.MIN_USER_CHARS_FOR_EXTRACTION
+
+
+async def test_extraction_audits_proposed_stored_and_dropped() -> None:
+ """A pass that proposes items and stores none must be visible.
+
+ That is exactly what happened in production, and it was indistinguishable
+ from "the model found nothing" because drops were logger.debug and stores
+ wrote no row.
+ """
+ class _Block:
+ type = "tool_use"
+ name = "store_memories"
+ input: dict[str, Any] = {
+ "decisions": [
+ # Valid: the quote is the user's own words, terminated.
+ {"domain": "finance", "summary": "Cut burn to 400k",
+ "user_commitment_quote": "Cut burn to 400k"},
+ # Invalid: quote is not in the user message at all.
+ {"domain": "finance", "summary": "Sell the building",
+ "user_commitment_quote": "sell the building"},
+ ],
+ }
+
+ class _Response:
+ content = [_Block()]
+
+ rows: list[dict[str, Any]] = []
+
+ def _capture(event_type: str, summary: str, **kw: Any) -> None:
+ rows.append({"event_type": event_type, "summary": summary, **kw})
+
+ with (
+ mock.patch.object(episodic, "store_decision") as store,
+ mock.patch("openexecutive.audit.log_event", _capture),
+ mock.patch("openexecutive.audit.usage.log_model_usage"),
+ mock.patch("openexecutive.config.get_settings"),
+ mock.patch.object(episodic, "get_active_initiatives", return_value=[]),
+ mock.patch("openexecutive.providers.get_provider") as provider,
+ ):
+ provider.return_value.messages_create = mock.AsyncMock(
+ return_value=_Response()
+ )
+ await episodic.extract_and_store(
+ "Cut burn to 400k. Tell the team.", "Understood.", session_id="s-1"
+ )
+
+ assert store.call_count == 1
+ audit = [r for r in rows if r["event_type"] == "memory_extraction"]
+ assert len(audit) == 1
+ assert audit[0]["details"]["proposed"]["decisions"] == 2
+ assert audit[0]["details"]["stored"]["decisions"] == 1
+ assert audit[0]["details"]["dropped_count"] == 1
+ assert "decision:bad_quote:Sell the building" in audit[0]["details"]["dropped"]