Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Conversation extraction keeps wishes and preferences again (GBRA-574) (#9656)

Garry Tan15f17a311bbf
diff --git a/behavior-changes.d/conversation_extraction_no_slot_gbra574.md b/behavior-changes.d/conversation_extraction_no_slot_gbra574.md
new file mode 100644
index 0000000000..ee8d7a5353
--- /dev/null
+++ b/behavior-changes.d/conversation_extraction_no_slot_gbra574.md
@@ -0,0 +1,4 @@
+---
+kind: fix
+---
+`gbrain extract-conversation-facts` no longer asks the model for a slot (where a person lives, who employs them or their role), because asking made it save fewer facts, wishes and preferences most of all. New conversation facts carry no slot, so skip the paid `--force` re-extraction the 0.60.735.0 notice suggested for adding slots: it now adds none. A fact stored with a slot under 0.60.735.0 keeps it, and its validity window, until the part of its page it came from is extracted again (an edit there, or `--force`, which re-extracts every page); the new rows then have no slot, and those facts read as they did before 0.60.735.0. The fact text is kept either way.
diff --git a/changelog.d/gbra574-extraction-keeps-preferences.md b/changelog.d/gbra574-extraction-keeps-preferences.md
new file mode 100644
index 0000000000..e083d13cd7
--- /dev/null
+++ b/changelog.d/gbra574-extraction-keeps-preferences.md
@@ -0,0 +1,6 @@
+**Conversation extraction keeps wishes and preferences again: the extractor no longer asks the model for a slot on every fact.**
+
+### Fixed
+- Since 0.60.735.0, `gbrain extract-conversation-facts` asked the model for a slot (where a person lives, who employs them or their role) on every fact. That request made it save fewer facts, with wishes, preferences and ideas down about 15%. The main extraction call now sends the prompt and response shape it used before 0.60.735.0, so no new fact carries a slot, and a `--force` re-extraction adds none: skip the paid run the 0.60.735.0 notice suggested for them. A fact stored with a slot under 0.60.735.0 keeps it, and the facts block still works out its validity window, until the part of its page it came from is extracted again (an edit there, or `--force`, which re-extracts every page); the new rows carry no slot and read as they did before 0.60.735.0. Fact text is kept either way.
+
+<!-- behavior-change: conversation_extraction_no_slot_gbra574 -->
diff --git a/docs/architecture/key-files/commands-3.md b/docs/architecture/key-files/commands-3.md
index bb7b69951c..84461d75b1 100644
--- a/docs/architecture/key-files/commands-3.md
+++ b/docs/architecture/key-files/commands-3.md
@@ -29,6 +29,6 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/commands/export.ts` + `src/core/export-snapshot.ts` + `src/core/export-stage.ts` + `src/core/export-publish.ts` — local Markdown export collects source/page/canonical-withdrawal/tag/raw state in one export-scoped repeatable-read read-only transaction, using 256-key ID batches and private disk-backed SQLite staging. Primary-key page reads reuse the canonical withdrawal overlay without source-wide slug scans under stale planner statistics. Each 256-key batch groups by source and retains only one capacity-checked withdrawal ledger at a time; bodies with no literal `gbrain:facts:` marker skip normalization, while malformed/orphan markers still use canonical SQL fingerprinting. No engine defaults or shared migration iterators change. Explicit source selection is validated; restore-only selects one source through `resolveRestoreTarget` (`src/core/restore-target.ts`, shared with `gbrain storage status`) and checks its recorded file path through the existing native path resolver. The manifest rejects unsafe paths, same/cross-source aliases, case/Unicode and file-prefix collisions before destination output, preserving the `<slug>.md` and `.raw/<name>.json` layout. Existing planned paths refuse rather than overwrite. The portable first-party native addon anchors no-follow directories and publishes leaves with atomic no-replace operations, retaining an exclusive persistent `.gbrain-export-status` marker; success requires flushed data and a final COMPLETE line. Unsupported native capabilities fail closed without a pathname fallback. A fresh destination is the retry path; unrelated files are never pruned. Fixed capacity/time limits fail visibly, never truncate. This is a historical snapshot, not a database backup or a sandbox against another same-user process renaming directories. See [safe export](../../storage-tiering.md#safe-export). Engine-free `export --help` documents scope, no-overwrite publication and incomplete-output recovery. Wave B PR 2: a segmented page streams (`segmentedExportMarkdown`: serializeMarkdown's head (an optional `revision` pins the body open to that page revision, as `unreadPageFile` does with its snapshot's, so a page that moved since refuses instead of mixing one revision's frontmatter with the next one's body), the trusted stored body through `readBodyRange` under a body ledger, here `exportBodyLedger`, the same withdrawal overlay; `restore_page` of a page over the single-string ceiling passes its snapshot's ledger; the timeline separator before the first timeline byte) into its own stage file (`ExportStage.addStream`, a full collection per 1 MiB read), published with the native path API; its payload estimate is `body_bytes`, the per-page limit is `exportPayloadLimit(pages.max_bytes)` (32 MiB while segmented storage is off), a streamed page adds 1 ms per KiB to the 10-minute snapshot deadline and stage files must leave 512 MiB free on the staging filesystem (`test/e2e/large-pages-export.test.ts`: inline vs segmented same bytes at 2 and 20 MiB, export(import(F)) == F, a gated 128 MiB memory arm). `exportPagePath` writes a page whose slug is not a portable file name under an escaped name with its `slug:` line (`:` and other Windows-unsafe characters become `~`, a reserved device name or trailing dot/space gets `~` appended, an over-budget segment is cut with a `~<8 hex>` hash and marked not restorable); `runExport` returns an `ExportReport`, prints the escaped pages, and collects every `ExportPathCollision` into one refusal whose lines name each fix (`test/pathological/b5-export-unsafe-slugs.test.ts`). Pinned by both-engine export safety/independent-roundtrip/snapshot/failure-release suites, the 100001-page scale fixture with staging bytes and full export SQL statement accounting, the native publication/compiled-wrapper suite, actual CLI help tests, and the PostgreSQL export E2E wrapper. A page's file is rendered by `export-snapshot.ts` (`inlineExportMarkdown`, `segmentedExportHead` for `segmentedExportMarkdown`; frontmatter with the #5966 slug stamp), which the git-gap equality contract (`persistence/git-gap.ts`) shares.
-- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into bounded windows via `src/core/facts/conversation-windows.ts` (#6267: eligibility per 30-min gap group of ≥2 messages decided before windowing; windows close at 30 messages or when the rendered text with its header would exceed 6,500 chars; oversized turns split at sentence boundaries with the speaker prefix repeated; a trailing single reply of an eligible group is its own window, never glued across a gap; nothing is clipped and an unrepresentable header fails the page before prior-fact cleanup; a page's frontmatter `conversation_segment_gap_minutes`, an integer from 1 to 10080, overrides the gap for that page, and any other value is ignored with a stderr warning naming the accepted range), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES` = `CONVERSATION_BODY_MAX_BYTES` in `conversation-parser/body.ts`, `PAGE_MAX_BYTES` + 1 MiB = 25 MiB, measured on the parser input; `readConversationBodyForParsing` stats a `raw_transcript` sidecar and throws `ConversationBodyTooLargeError` over the cap before any read, the snapshot carries `oversizedSidecarBytes`, and a size skip writes a `skipped_too_large` receipt to `page_extraction_state` (doctor `extraction_input_truncated`) that a later run under the cap clears; test `test/conversation-facts-oversized-sidecar.test.ts`; a segmented page is read in ranges; one over `CONVERSATION_WINDOW_BYTES` with no conversation facts is extracted from its newest window (day-header date, `tail_window` receipt, `window_rule: no_prior_facts`); one with facts is skipped with nothing cleared (`skipped_too_large`, `window_rule: prior_facts_kept`), since the run clears prior facts the window may not see, until PR 4's window anchors (test `test/large-pages/segmented-conversation-facts.test.ts`)); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd` (a DEFAULT cap is dropped with a warning when the extraction or parser chat model is unpriced, via `src/core/facts/conversation-budget.ts` `conversationFactsCostCap` (#5823); explicit CLI/caller/cycle caps stay fail-closed and a `no_pricing` stop reports `budget_reason`/`budget_model`, exits 1 and names `pricing.overrides`); **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); **`--background` via `maybeBackground`**; and **speaker-shaped-fold decline** — when the parse reports `unrecognized_headings` containing a speaker-shaped label (1-2 title-cased words, not in the doc-heading stoplist — the stoplist gates the DECLINE only, so a miss is warn-noise, never data loss) AND the parse produced fewer than two distinct speakers, the page is declined instead of extracted under wrong attribution: counted in `pages_skipped_unrecognized_speaker` (Result/CLI/cycle surfaces), warned to stderr with the folding pattern id, and deliberately NON-terminal (no durable audit row and no orphan cleanup, so a future parser/pattern fix retries the page); multi-speaker pages with folds proceed warn-only. `publishConversationFacts` skips a non-outcome row an active conversation fact of the source already states (`findCaptureDuplicate`, no time bound; `duplicates_skipped`, #7043). The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source except `cycle.conversation_facts_backfill.exclude_sources` (reported as `sources_excluded`), reports managed pages left accepted and still publishing as `pages_pending` in its details and summary (`test/cycle-conversation-facts-backfill-pending-7500.test.ts`), and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts:<source>:<slug>`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). A page's rows are buffered and published once after its last segment (`publishPageFacts` on an unmanaged brain, one `writeDerivedFacts` transaction; the managed batch otherwise), through `publishConversationFacts` (facts/conversation-publication.ts), so a crashed, failed, empty or partial run never removes the prior batch and a terminal audit row commits only with its page's facts (#7013); a run's `scope` is `partial` when fewer segments than in scope ran, `window` when `--since` dropped messages, else `full`, and a page that changed mid-run keeps an unmanaged run's facts as a `partial` batch. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Additive `pages_skipped_unparsed`, `pages_skipped_type_mismatch`, `pages_skipped_insufficient_turns`, and `pages_skipped_since` explain the unchanged `pages_skipped` aggregate across core, CLI, workers, and cycle sources; durable completion remains separate. `--dry-run` stops after segmentation, with no parser-fallback, extractor, or embedding calls and no durable knowledge/checkpoint/receipt/rollup writes. The narrow `python-dict-utterance` registry format and its quoted/nested-example rejection are documented in [conversation parser patterns](../conversation-parser-patterns.md). Checkpoint state is a shared `cpMap: Map<slug, endIso>` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-diagnostics.serial.test.ts` (pool workers, per-page lock skip + exit 3, replay replacing a prior partial run's facts, D15 preflight, `--workers` round-trip). `src/commands/doctor.ts` reads the same outcomes, which survive checkpoint GC. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Explicit `slugs` batches contain page failures like the pool (cancellation and budget stops halt); logs carry only the closed reason and a retry command. Completion requires every in-scope segment (exactly `--segment-limit` counts). Extracted rows take `facts.default_visibility` (resolved once per run); audit rows stay private. Each page's own entities go to the extractor as `entityHints` (slugs, `src/core/facts/conversation-entity-hints.ts`, computed once per page, cap `CONVERSATION_ENTITY_HINTS_CAP` = 40 through `ExtractInput.entityHintsCap`): authored edges only (the page's frontmatter edges with the page as origin, then outgoing markdown, wikilink-resolved and manual links), same source, live fact-entity pages, and a private entity only when the run's facts are private; the `extract_facts` op keeps its 5-hint cap and prompt. Pinned by `test/conversation-entity-hints.test.ts`. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed. `no_match`, errors, cancellation, and dry runs never become durable negatives, bar `facts/conversation-skip.ts` (prose meetings, single emails, undated turns). Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. `--dry-run` also prints the paid-run estimate (`facts/conversation-budget.ts` `conversationRunEstimate`: segments, input and output tokens and dollars on the configured chat model, or a `pricing set` hint for an unpriced model; JSON `estimate`); the conversation extractor asks for slots (GBRA-574 B2), so re-extracting old pages for them is `--force` after that preview. On a managed brain the core runs `managedDerivedFactsPreflight` (local writer, active source) before any provider work, and a page's facts and terminal (or non-extractable) audit row are buffered, then published in its receipted batch after source activity, page id, revision and parser-input snapshot are rechecked under the page lock, so a failed or raced re-extraction keeps the prior batch; the CLI no longer refuses up front and receipt pages are skipped. Tests: `test/managed-facts-writers.test.ts`, `test/managed-unsupported-preflight.serial.test.ts` (refusal before spend for an unaccepted writer).
+- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into bounded windows via `src/core/facts/conversation-windows.ts` (#6267: eligibility per 30-min gap group of ≥2 messages decided before windowing; windows close at 30 messages or when the rendered text with its header would exceed 6,500 chars; oversized turns split at sentence boundaries with the speaker prefix repeated; a trailing single reply of an eligible group is its own window, never glued across a gap; nothing is clipped and an unrepresentable header fails the page before prior-fact cleanup; a page's frontmatter `conversation_segment_gap_minutes`, an integer from 1 to 10080, overrides the gap for that page, and any other value is ignored with a stderr warning naming the accepted range), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES` = `CONVERSATION_BODY_MAX_BYTES` in `conversation-parser/body.ts`, `PAGE_MAX_BYTES` + 1 MiB = 25 MiB, measured on the parser input; `readConversationBodyForParsing` stats a `raw_transcript` sidecar and throws `ConversationBodyTooLargeError` over the cap before any read, the snapshot carries `oversizedSidecarBytes`, and a size skip writes a `skipped_too_large` receipt to `page_extraction_state` (doctor `extraction_input_truncated`) that a later run under the cap clears; test `test/conversation-facts-oversized-sidecar.test.ts`; a segmented page is read in ranges; one over `CONVERSATION_WINDOW_BYTES` with no conversation facts is extracted from its newest window (day-header date, `tail_window` receipt, `window_rule: no_prior_facts`); one with facts is skipped with nothing cleared (`skipped_too_large`, `window_rule: prior_facts_kept`), since the run clears prior facts the window may not see, until PR 4's window anchors (test `test/large-pages/segmented-conversation-facts.test.ts`)); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd` (a DEFAULT cap is dropped with a warning when the extraction or parser chat model is unpriced, via `src/core/facts/conversation-budget.ts` `conversationFactsCostCap` (#5823); explicit CLI/caller/cycle caps stay fail-closed and a `no_pricing` stop reports `budget_reason`/`budget_model`, exits 1 and names `pricing.overrides`); **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); **`--background` via `maybeBackground`**; and **speaker-shaped-fold decline** — when the parse reports `unrecognized_headings` containing a speaker-shaped label (1-2 title-cased words, not in the doc-heading stoplist — the stoplist gates the DECLINE only, so a miss is warn-noise, never data loss) AND the parse produced fewer than two distinct speakers, the page is declined instead of extracted under wrong attribution: counted in `pages_skipped_unrecognized_speaker` (Result/CLI/cycle surfaces), warned to stderr with the folding pattern id, and deliberately NON-terminal (no durable audit row and no orphan cleanup, so a future parser/pattern fix retries the page); multi-speaker pages with folds proceed warn-only. `publishConversationFacts` skips a non-outcome row an active conversation fact of the source already states (`findCaptureDuplicate`, no time bound; `duplicates_skipped`, #7043). The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source except `cycle.conversation_facts_backfill.exclude_sources` (reported as `sources_excluded`), reports managed pages left accepted and still publishing as `pages_pending` in its details and summary (`test/cycle-conversation-facts-backfill-pending-7500.test.ts`), and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts:<source>:<slug>`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). A page's rows are buffered and published once after its last segment (`publishPageFacts` on an unmanaged brain, one `writeDerivedFacts` transaction; the managed batch otherwise), through `publishConversationFacts` (facts/conversation-publication.ts), so a crashed, failed, empty or partial run never removes the prior batch and a terminal audit row commits only with its page's facts (#7013); a run's `scope` is `partial` when fewer segments than in scope ran, `window` when `--since` dropped messages, else `full`, and a page that changed mid-run keeps an unmanaged run's facts as a `partial` batch. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Additive `pages_skipped_unparsed`, `pages_skipped_type_mismatch`, `pages_skipped_insufficient_turns`, and `pages_skipped_since` explain the unchanged `pages_skipped` aggregate across core, CLI, workers, and cycle sources; durable completion remains separate. `--dry-run` stops after segmentation, with no parser-fallback, extractor, or embedding calls and no durable knowledge/checkpoint/receipt/rollup writes. The narrow `python-dict-utterance` registry format and its quoted/nested-example rejection are documented in [conversation parser patterns](../conversation-parser-patterns.md). Checkpoint state is a shared `cpMap: Map<slug, endIso>` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-diagnostics.serial.test.ts` (pool workers, per-page lock skip + exit 3, replay replacing a prior partial run's facts, D15 preflight, `--workers` round-trip). `src/commands/doctor.ts` reads the same outcomes, which survive checkpoint GC. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Explicit `slugs` batches contain page failures like the pool (cancellation and budget stops halt); logs carry only the closed reason and a retry command. Completion requires every in-scope segment (exactly `--segment-limit` counts). Extracted rows take `facts.default_visibility` (resolved once per run); audit rows stay private. Each page's own entities go to the extractor as `entityHints` (slugs, `src/core/facts/conversation-entity-hints.ts`, computed once per page, cap `CONVERSATION_ENTITY_HINTS_CAP` = 40 through `ExtractInput.entityHintsCap`): authored edges only (the page's frontmatter edges with the page as origin, then outgoing markdown, wikilink-resolved and manual links), same source, live fact-entity pages, and a private entity only when the run's facts are private; the `extract_facts` op keeps its 5-hint cap and prompt. Pinned by `test/conversation-entity-hints.test.ts`. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed. `no_match`, errors, cancellation, and dry runs never become durable negatives, bar `facts/conversation-skip.ts` (prose meetings, single emails, undated turns). Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. `--dry-run` also prints the paid-run estimate (`facts/conversation-budget.ts` `conversationRunEstimate`: segments, input and output tokens and dollars on the configured chat model, or a `pricing set` hint for an unpriced model; JSON `estimate`). On a managed brain the core runs `managedDerivedFactsPreflight` (local writer, active source) before any provider work, and a page's facts and terminal (or non-extractable) audit row are buffered, then published in its receipted batch after source activity, page id, revision and parser-input snapshot are rechecked under the page lock, so a failed or raced re-extraction keeps the prior batch; the CLI no longer refuses up front and receipt pages are skipped. Tests: `test/managed-facts-writers.test.ts`, `test/managed-unsupported-preflight.serial.test.ts` (refusal before spend for an unaccepted writer).
[Continue this subsystem](commands-3-continued.md).
diff --git a/docs/evidence-delivery.md b/docs/evidence-delivery.md
index 192b834ef9..cd707c197b 100644
--- a/docs/evidence-delivery.md
+++ b/docs/evidence-delivery.md
@@ -195,9 +195,10 @@ starts at the fact's `valid_from`: `remember` sets it from its `valid_from`
parameter (when the fact was said or became true). A `valid_from` that only
records the write time (within a minute of `created_at`) reads `unknown`, so a
header never presents when a fact was written as when it became true. An open
-validity end reads `unknown`. A conversation fact that states where a person
-lives, who employs them or their role carries a slot (`location`, `employer`
-or `role`), and the facts block and the temporal fact reserve work out its end
+validity end reads `unknown`. When a conversation fact carries a slot
+(`location`, `employer` or `role`; extraction does not ask the model for one
+at the moment, so only facts stored with one under 0.60.735.0 have it, until
+the part of the page they came from is extracted again), the facts block and the temporal fact reserve work out its end
at read time: the start of the next different value for the same person, when
that value came from the same conversation or its fact says the value changed
(moved, joined, left and similar words). Otherwise both values stay open and
diff --git a/src/core/facts/conversation-budget.ts b/src/core/facts/conversation-budget.ts
index 5dbe318624..f563efb7bc 100644
--- a/src/core/facts/conversation-budget.ts
+++ b/src/core/facts/conversation-budget.ts
@@ -31,7 +31,7 @@ export async function conversationFactsCostCap(
return undefined;
}
-/** Output tokens one segment's reply takes at most in practice: the extractor's 10-fact cap at about 200 tokens a fact (measured with slots on). */
+/** Output tokens one segment's reply takes at most in practice: the extractor's 10-fact cap at about 200 tokens a fact. */
export const OUTPUT_TOKENS_PER_SEGMENT = 2000;
export interface ConversationRunEstimate { model: string; segments: number; input_tokens: number; output_tokens: number; usd: number | null }
@@ -44,7 +44,7 @@ export interface ConversationRunEstimate { model: string; segments: number; inpu
*/
export async function conversationRunEstimate(engine: BrainEngine, segments: number): Promise<ConversationRunEstimate> {
const model = await getFactsExtractionModel(engine);
- const system = buildExtractorSystem(true, { ...await getExtractorVariant(engine), slots: true });
+ const system = buildExtractorSystem(true, await getExtractorVariant(engine));
const perSegment = Math.ceil((system.length + SEGMENT_TEXT_CHAR_LIMIT) / CHAT_INPUT_CHARS_PER_TOKEN);
const price = usageCostUsd(model, perSegment, OUTPUT_TOKENS_PER_SEGMENT, 'chat', await loadPricingOverrides(engine));
return { model, segments, input_tokens: segments * perSegment, output_tokens: segments * OUTPUT_TOKENS_PER_SEGMENT, usd: price == null ? null : segments * price };
diff --git a/src/core/facts/conversation-window-extract.ts b/src/core/facts/conversation-window-extract.ts
index 988a714cc2..63dc338300 100644
--- a/src/core/facts/conversation-window-extract.ts
+++ b/src/core/facts/conversation-window-extract.ts
@@ -64,7 +64,7 @@ export async function extractConversationWindow(run: ConversationWindowRun, page
const sessionId = `${CONVERSATION_FACTS_SOURCE_PREFIX}:${page.slug}`;
let extracted: ExtractedFact[];
const input: ExtractInput = { turnText: text, sessionId, source: CONVERSATION_FACTS_SOURCE_PREFIX, engine: run.engine, abortSignal: run.signal,
- entityHints, entityHintsCap: CONVERSATION_ENTITY_HINTS_CAP, slots: true };
+ entityHints, entityHintsCap: CONVERSATION_ENTITY_HINTS_CAP };
if (run.extractor) {
extracted = await run.extractor(input);
} else {
diff --git a/src/core/facts/extract.ts b/src/core/facts/extract.ts
index eab297dec9..77630b6ff6 100644
--- a/src/core/facts/extract.ts
+++ b/src/core/facts/extract.ts
@@ -35,7 +35,6 @@ import type { BrainEngine, NewFact, FactKind, FactAttribution } from '../engine.
import { normalizeMetricLabel } from './extract-from-fence.ts';
import { isNullLikeEntity } from './write-single.ts';
import { isAIInvocationPolicyError } from '../ai/invocation-guard.ts';
-import { normalizeSlot } from './slots.ts';
/**
* v0.31 (D15): kill-switch for fact extraction.
@@ -146,8 +145,6 @@ export interface ExtractorVariant {
dateGrounding?: boolean;
/** Keep assistant-made claims as their own, attributed facts. */
attribution?: boolean;
- /** GBRA-574 B2: tag facts with a slot (facts/slots.ts). Never read from config: set by ExtractInput.slots. */
- slots?: boolean;
}
const flagIs = (v: string | null, values: readonly string[]) => v != null && values.includes(v.trim().toLowerCase());
@@ -301,11 +298,6 @@ export interface ExtractInput {
variant?: ExtractorVariant;
/** #8064: the agent (session-speaker label) that wrote the turn's user side; the owner when absent. */
userSpeaker?: string;
- /**
- * GBRA-574 B2: ask for a per-fact slot (location, employer, role). Only the conversation extractor
- * (conversation-window-extract.ts) sets it, so every other caller's prompt and schema stay as they were.
- */
- slots?: boolean;
/** Optional pre-embedding admission selector for extracted fact tiers. */
notabilityAdmission?: {
allowed: readonly FactNotability[];
@@ -458,21 +450,10 @@ const ATTRIBUTION_BLOCK = [
' did, "other" for a named third party, null when the speaker is unclear.',
].join('\n');
-/** Slot block (GBRA-574 B2): the one single-valued dimension a fact states, for read-time validity windows. */
-const SLOT_BLOCK = [
- '',
- 'Slots: a fact that states where a person lives, which organization employs them, or their role or job title adds',
- '"slot":["location|employer|role","<the place, organization or title alone>"]. Every other fact leaves "slot" out.',
- '- Only with an entity: the person the value belongs to. No entity, no slot.',
- '- A fact saying the value changed (moved, joined, was promoted) carries the new value. A fact that only ends a value',
- ' ("left the company") or states a past one ("used to live in ...") has no slot.',
- '- No slot for plans, wishes, trips, visits or guesses.',
-].join('\n');
-
-/** @internal Exported for the prompt-shape test. Every variant off → the historical prompt, byte-identical. */
+/** @internal Exported for the prompt-shape test. Both variants off → the historical prompt, byte-identical. */
export function buildExtractorSystem(admitsLow: boolean, variant: ExtractorVariant = {}): string {
const base = admitsLow ? EXTRACTOR_SYSTEM_ADMITS_LOW : EXTRACTOR_SYSTEM_SKIPS_LOW;
- return base + (variant.dateGrounding ? DATE_GROUNDING_BLOCK : '') + (variant.attribution ? ATTRIBUTION_BLOCK : '') + (variant.slots ? SLOT_BLOCK : '');
+ return base + (variant.dateGrounding ? DATE_GROUNDING_BLOCK : '') + (variant.attribution ? ATTRIBUTION_BLOCK : '');
}
/** Extractor input ceiling; corpus windows (context/corpus-windows.ts) are cut to fit it. */
@@ -522,15 +503,14 @@ const FACTS_RESPONSE_SCHEMA = { name: 'facts_extraction', schema: FACTS_EXTRACTI
/**
* The variant's response schema: grounding adds a nullable `valid_from`,
- * attribution a nullable `attributed_to`, slots a nullable `slot`; all off → the historical schema.
+ * attribution a nullable `attributed_to`; both off → the historical schema.
*/
function factsResponseSchema(variant: ExtractorVariant): typeof FACTS_RESPONSE_SCHEMA {
- if (!variant.dateGrounding && !variant.attribution && !variant.slots) return FACTS_RESPONSE_SCHEMA;
+ if (!variant.dateGrounding && !variant.attribution) return FACTS_RESPONSE_SCHEMA;
const items = (FACTS_EXTRACTION_SCHEMA.properties as { facts: { items: Record<string, unknown> } }).facts.items;
const extra: Record<string, unknown> = {
...(variant.dateGrounding ? { valid_from: { type: ['string', 'null'] } } : {}),
...(variant.attribution ? { attributed_to: { type: ['string', 'null'], enum: ['user', 'assistant', 'other', null] } } : {}),
- ...(variant.slots ? { slot: { anyOf: [{ type: 'null' }, { type: 'array', items: { type: 'string' } }] } } : {}),
};
const variantItems = {
...items,
@@ -666,13 +646,12 @@ export async function extractFactsFromTurnWithOutcome(
// because those reuse `extractorSystem`. Read AFTER the availability gate —
// a chat_unavailable early return must not pay config round-trips (#4298
// resolved the model/gate ordering; these reads sit behind it).
- const [promptAppendix, junkFilterOn, missingConfidence, configured] = await Promise.all([
+ const [promptAppendix, junkFilterOn, missingConfidence, variant] = await Promise.all([
getFactsExtractionPromptAppendix(input.engine),
isJunkFilterEnabled(input.engine),
getMissingConfidence(input.engine),
input.variant ?? getExtractorVariant(input.engine),
]);
- const variant: ExtractorVariant = input.slots ? { ...configured, slots: true } : configured;
const extractorSystem = promptAppendix
? `${buildExtractorSystem(admitsLow, variant)}\n\n${promptAppendix}`
: buildExtractorSystem(admitsLow, variant);
@@ -863,8 +842,6 @@ export async function extractFactsFromTurnWithOutcome(
const claimValue = candidate.value ?? null;
const claimUnit = candidate.unit ?? null;
const claimPeriod = candidate.period ?? null;
- const entitySlug = isUnknownSpeakerLabel(candidate.entity) || isNullLikeEntity(candidate.entity) ? null : (candidate.entity ?? null);
- const slot = variant.slots ? normalizeSlot(candidate.slot?.dimension, candidate.slot?.value, entitySlug) : null;
facts.push({
fact: factText,
@@ -877,7 +854,9 @@ export async function extractFactsFromTurnWithOutcome(
// where the prompt asked for JSON null — otherwise the resolver's
// fallback adopts the token as the slug and the facts land unreachable
// under entity_slug='null'. Same token set the `remember` verb applies.
- entity_slug: entitySlug,
+ entity_slug: isUnknownSpeakerLabel(candidate.entity) || isNullLikeEntity(candidate.entity)
+ ? null
+ : (candidate.entity ?? null),
source: input.source,
source_session: input.sessionId ?? null,
confidence,
@@ -895,7 +874,6 @@ export async function extractFactsFromTurnWithOutcome(
? { valid_from: parseExtractedEventDate(candidate.valid_from)! }
: {}),
...(variant.attribution && candidate.attributed_to ? { attributed_to: candidate.attributed_to } : {}),
- ...(slot ? { dimension: slot.dimension, value: slot.value } : {}),
});
}
@@ -950,8 +928,6 @@ interface RawExtracted {
/** Date-grounding variant: the event date the extractor stated (raw string; validated on use). */
valid_from?: string | null; /** Attribution variant: who asserted the claim; anything outside the three speakers parses as null. */
attributed_to?: FactAttribution | null;
- /** Slots variant: the raw slot (validated on use by normalizeSlot). */
- slot?: { dimension: string; value: string } | null;
}
/**
@@ -1038,7 +1014,6 @@ function tryArrayShapeDetailed(s: string): ParsedExtractorShape | null {
period: typeof o.period === 'string' ? o.period : null,
valid_from: typeof o.valid_from === 'string' ? o.valid_from : null,
attributed_to: o.attributed_to === 'user' || o.attributed_to === 'assistant' || o.attributed_to === 'other' ? o.attributed_to : null,
- slot: rawSlot(o.slot),
});
}
return { facts: out, invalidCandidates };
@@ -1047,12 +1022,6 @@ function tryArrayShapeDetailed(s: string): ParsedExtractorShape | null {
}
}
-function rawSlot(s: unknown): RawExtracted['slot'] {
- if (typeof s !== 'object' || s === null) return null;
- const [dimension, value] = Array.isArray(s) ? s : [(s as Record<string, unknown>).dimension, (s as Record<string, unknown>).value];
- return typeof dimension === 'string' && typeof value === 'string' ? { dimension, value } : null;
-}
-
function clampConfidence(x: number | undefined, missing: number): number {
if (typeof x !== 'number' || !Number.isFinite(x)) return missing;
if (x < 0) return 0;
diff --git a/test/facts/extract-slots.test.ts b/test/facts/extract-slots.test.ts
index b1091b7460..ad29cf2db9 100644
--- a/test/facts/extract-slots.test.ts
+++ b/test/facts/extract-slots.test.ts
@@ -1,46 +1,21 @@
/**
- * GBRA-574 B2: the conversation extractor's per-fact slot (dimension, value) on the ontology columns.
+ * GBRA-574: the conversation extractor's main call sends the historical prompt and response schema, with no slot.
*
- * Protects: only a caller that asks for slots (the conversation extractor) gets the slot clause and schema field; every
- * other caller's system prompt stays byte-identical to the historical one (pinned by hash) and its schema has no slot;
- * the parser keeps a slot in the closed vocabulary (location, employer, role, after the dimension aliases) and drops an
- * unknown dimension, an empty or over-long value, and a slot on a fact with no entity, always keeping the fact.
- * Seams: gateway chat transport stub (no provider call).
+ * Protects: asking the main extraction call for a per-fact slot made it save fewer facts (about 15% fewer wishes,
+ * preferences and ideas on two public dev worlds), so the call's system prompt stays byte-identical to the historical one
+ * for every variant (pinned by hash), its schema has no slot field, and a slot the model returns anyway is never stored.
+ * Regression: a conversation extraction call that carries slot text or a slot field.
+ * Seams: gateway chat transport stub (no provider call), PGLite with no git repository.
*/
import { createHash } from 'node:crypto';
-import { afterAll, beforeEach, describe, expect, test } from 'bun:test';
-import { configureGateway, resetGateway, __setChatTransportForTests } from '../../src/core/ai/gateway.ts';
-import type { ChatOpts, ChatResult } from '../../src/core/ai/gateway.ts';
-import { buildExtractorSystem, extractFactsFromTurnWithOutcome, type ExtractInput } from '../../src/core/facts/extract.ts';
-
-function chatResult(text: string): ChatResult {
- return {
- text, blocks: [{ type: 'text', text }], stopReason: 'end',
- usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
- model: 'anthropic:claude-sonnet-4-6', providerId: 'anthropic',
- } as ChatResult;
-}
-
-beforeEach(() => {
- resetGateway();
- __setChatTransportForTests(null);
- configureGateway({ chat_model: 'anthropic:claude-sonnet-4-6', env: { ANTHROPIC_API_KEY: 'sk-ant-test' } });
-});
-afterAll(() => {
- __setChatTransportForTests(null);
- configureGateway({ embedding_model: 'openai:text-embedding-3-large', embedding_dimensions: 1536, env: { ...process.env } });
-});
-
-async function run(input: Partial<ExtractInput>, reply: unknown) {
- const seen: ChatOpts[] = [];
- __setChatTransportForTests(async (opts) => { seen.push(opts); return chatResult(JSON.stringify(reply)); });
- const outcome = await extractFactsFromTurnWithOutcome({ turnText: 'Alice Example (2026-03-10): I work at Acme Example now.', source: 'test:slots',
- variant: {}, embedding: null, ...input } as ExtractInput);
- return { seen, outcome };
-}
+import { afterAll, afterEach, beforeAll, describe, expect, spyOn, test } from 'bun:test';
+import { runExtractConversationFactsCore } from '../../src/commands/extract-conversation-facts.ts';
+import { __setChatTransportForTests, resetGateway, type ChatOpts, type ChatResult } from '../../src/core/ai/gateway.ts';
+import { buildExtractorSystem } from '../../src/core/facts/extract.ts';
+import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
const sha = (s: string) => createHash('sha256').update(s).digest('hex');
-/** buildExtractorSystem on master before GBRA-574 B2, by (admitsLow, variant). */
+/** buildExtractorSystem before GBRA-574 B2, by (admitsLow, variant). */
const HISTORICAL: Array<[boolean, { dateGrounding?: boolean; attribution?: boolean }, string]> = [
[true, {}, 'a4b468b86e867f3703a4c834583f5ac9a99f47ff4f0c50484fdd6b22706465c7'],
[true, { dateGrounding: true }, 'd49e4a26fe594b8a4445172c736b1c7407af9fb55c20c7f7f9dce05e068bcd64'],
@@ -51,92 +26,60 @@ const HISTORICAL: Array<[boolean, { dateGrounding?: boolean; attribution?: boole
[false, { attribution: true }, '364637bf592f858bad211525a4dcb9c2fd15e3ea8cba2af90acb109e1cd7875a'],
[false, { dateGrounding: true, attribution: true }, '3b15e8a337a2557ed4b81255cf27a386069490d6209567c753942d310064eb25'],
];
+const HISTORICAL_HASHES = new Set(HISTORICAL.map(([, , hash]) => hash));
-describe('slot prompt', () => {
- test('every prompt variant without slots is byte-identical to the historical one', () => {
- for (const [admitsLow, variant, hash] of HISTORICAL) expect(sha(buildExtractorSystem(admitsLow, variant))).toBe(hash);
- });
+const PAGE = 'conversations/2026-03-10/chat-example';
+const BODY = [
+ '**user** (2026-03-10 9:00 AM): Alice Example joined Acme Example as head of design last week.',
+ '**assistant** (2026-03-10 9:01 AM): Good news for her.',
+ '**user** (2026-03-10 9:02 AM): I want a quieter desk lamp for the study.',
+].join('\n');
- test('a caller that does not ask for slots sends the historical prompt and a schema without a slot', async () => {
- const { seen } = await run({}, { facts: [] });
- expect(sha(seen[0]!.system as string)).toBe(HISTORICAL[0]![2]);
- expect(JSON.stringify(seen[0]!.responseSchema)).not.toContain('slot');
- });
+let engine: PGLiteEngine;
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ await engine.setConfig('facts.extraction_enabled', 'true');
+ await engine.setConfig('facts.default_visibility', 'world');
+ await engine.setConfig('conversation_parser.llm_fallback_enabled', 'false');
+ await engine.putPage('people/alice-example', { type: 'person' as never, title: 'Alice Example', compiled_truth: '# Alice Example', timeline: '', frontmatter: {} });
+ await engine.putPage(PAGE, { type: 'conversation' as never, title: 'Conversation 2026-03-10', compiled_truth: BODY, timeline: '', frontmatter: { date: '2026-03-10' } });
+}, 60_000);
+afterAll(async () => { await engine.disconnect(); });
+afterEach(() => { __setChatTransportForTests(null); resetGateway(); });
- test('a caller that asks for slots gets the historical prompt plus the slot clause, and a nullable slot in the schema', async () => {
- const { seen } = await run({ slots: true } as Partial<ExtractInput>, { facts: [] });
- const system = seen[0]!.system as string;
- expect(system.startsWith(buildExtractorSystem(true, {}))).toBe(true);
- expect(system.length).toBeGreaterThan(buildExtractorSystem(true, {}).length);
- expect(system).toContain('"slot"');
- for (const d of ['location', 'employer', 'role']) expect(system).toContain(d);
- // About 150 tokens (4 characters a token): the clause stays short.
- expect(system.length - buildExtractorSystem(true, {}).length).toBeLessThan(1000);
- expect(JSON.stringify(seen[0]!.responseSchema)).toContain('"slot"');
- });
-});
-
-describe('slot parsing', () => {
- const fact = (text: string, slot: unknown, entity: string | null = 'people/alice-example') =>
- ({ fact: text, kind: 'fact', entity, confidence: 1, notability: 'high', slot });
-
- test('keeps a valid slot (aliases normalized, value trimmed) and drops every invalid one, keeping each fact', async () => {
- const { outcome } = await run({ slots: true } as Partial<ExtractInput>, { facts: [
- fact('Alice Example works at Acme Example.', { dimension: 'Company', value: ' Acme Example ' }),
- fact('Alice Example lives in Springfield.', { dimension: 'location', value: 'Springfield' }),
- fact('Alice Example is the head of design.', { dimension: 'job title', value: 'head of design' }),
- fact('Alice Example plays chess.', { dimension: 'hobby', value: 'chess' }),
- fact('Alice Example moved somewhere.', { dimension: 'location', value: ' ' }),
- fact('Alice Example has a very long title.', { dimension: 'role', value: 'x'.repeat(81) }),
- fact('Someone works at Acme Example.', { dimension: 'employer', value: 'Acme Example' }, null),
- fact('Alice Example likes tea.', null),
- ] });
- expect(outcome.ok).toBe(true);
- if (!outcome.ok) return;
- const slots = outcome.facts.map(f => [f.fact, (f as { dimension?: string | null }).dimension ?? null, (f as { value?: string | null }).value ?? null]);
- expect(slots).toEqual([
- ['Alice Example works at Acme Example.', 'employer', 'Acme Example'],
- ['Alice Example lives in Springfield.', 'location', 'Springfield'],
- ['Alice Example is the head of design.', 'role', 'head of design'],
- ['Alice Example plays chess.', null, null],
- ['Alice Example moved somewhere.', null, null],
- ['Alice Example has a very long title.', null, null],
- ['Someone works at Acme Example.', null, null],
- ['Alice Example likes tea.', null, null],
- ]);
- });
-
- test('reads the compact array form and a fact with no slot key, and still reads the object form', async () => {
- const bare = (text: string) => ({ fact: text, kind: 'fact', entity: 'people/alice-example', confidence: 1, notability: 'high' });
- const { seen, outcome } = await run({ slots: true } as Partial<ExtractInput>, { facts: [
- fact('Alice Example works at Acme Example.', ['employer', 'Acme Example']),
- fact('Alice Example lives in Springfield.', { dimension: 'location', value: 'Springfield' }),
- bare('Alice Example likes tea.'),
- fact('Alice Example plays chess.', ['hobby', 'chess']),
- fact('Alice Example is a role.', ['role']),
- fact('Alice Example is numbered.', ['role', 7]),
- ] });
- expect(seen[0]!.system as string).toContain('"slot":["');
- expect(seen[0]!.system as string).not.toContain('"slot":null');
- expect(outcome.ok).toBe(true);
- if (!outcome.ok) return;
- expect(outcome.facts.map(f => [f.fact, (f as { dimension?: string | null }).dimension ?? null, (f as { value?: string | null }).value ?? null])).toEqual([
- ['Alice Example works at Acme Example.', 'employer', 'Acme Example'],
- ['Alice Example lives in Springfield.', 'location', 'Springfield'],
- ['Alice Example likes tea.', null, null],
- ['Alice Example plays chess.', null, null],
- ['Alice Example is a role.', null, null],
- ['Alice Example is numbered.', null, null],
- ]);
- });
-
- test('an 80-character value is kept', async () => {
- const { outcome } = await run({ slots: true } as Partial<ExtractInput>, { facts: [fact('Alice Example has a long title.', { dimension: 'role', value: 'y'.repeat(80) })] });
- expect(outcome.ok && (outcome.facts[0] as { value?: string }).value).toBe('y'.repeat(80));
+describe('conversation extraction main call', () => {
+ test('every prompt variant is byte-identical to the historical one', () => {
+ for (const [admitsLow, variant, hash] of HISTORICAL) expect(sha(buildExtractorSystem(admitsLow, variant))).toBe(hash);
});
- test('a caller that did not ask for slots stores none, even when the model returns one', async () => {
- const { outcome } = await run({}, { facts: [fact('Alice Example works at Acme Example.', { dimension: 'employer', value: 'Acme Example' })] });
- expect(outcome.ok && (outcome.facts[0] as { dimension?: string | null }).dimension).toBeFalsy();
+ test('sends the historical prompt and a schema with no slot, and stores no slot the model returns anyway', async () => {
+ const seen: ChatOpts[] = [];
+ __setChatTransportForTests(async (opts): Promise<ChatResult> => {
+ seen.push(opts);
+ const text = JSON.stringify({ facts: [
+ { fact: 'Alice Example joined Acme Example as head of design.', kind: 'fact', entity: 'people/alice-example', confidence: 1,
+ notability: 'high', slot: ['role', 'head of design'] },
+ { fact: 'User wants a quieter desk lamp for the study.', kind: 'preference', entity: null, confidence: 1, notability: 'high' },
+ ] });
+ return { text, blocks: [{ type: 'text', text }], stopReason: 'end', model: 'test:stub', providerId: 'test',
+ usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 } } as ChatResult;
+ });
+ const spy = spyOn(process.stderr, 'write').mockImplementation(() => true);
+ try {
+ await runExtractConversationFactsCore(engine, { sourceId: 'default', slugs: [PAGE], sleepMs: 0, overrideDisabled: true });
+ } finally { spy.mockRestore(); }
+ expect(seen.length).toBeGreaterThan(0);
+ for (const opts of seen) {
+ expect(HISTORICAL_HASHES.has(sha(opts.system as string))).toBe(true);
+ expect(opts.system as string).not.toContain('slot');
+ expect(JSON.stringify(opts.responseSchema ?? null)).not.toContain('slot');
+ }
+ const rows = await engine.executeRaw<{ fact: string; dimension: string | null }>(
+ 'SELECT fact, dimension FROM facts WHERE source_markdown_slug = $1 ORDER BY id', [PAGE]);
+ expect(rows.map(row => row.fact)).toEqual([
+ 'Alice Example joined Acme Example as head of design.', 'User wants a quieter desk lamp for the study.', 'EXTRACTION_COMPLETE']);
+ for (const row of rows) expect(row.dimension).toBeNull();
});
});
diff --git a/test/facts/normalize-slot.test.ts b/test/facts/normalize-slot.test.ts
new file mode 100644
index 0000000000..afc2cfbc48
--- /dev/null
+++ b/test/facts/normalize-slot.test.ts
@@ -0,0 +1,29 @@
+/**
+ * GBRA-574: normalizeSlot is the gate every stored conversation slot passes through (conversation-publication.ts
+ * insertableSlots). The extractor no longer asks for slots, so these cases call it directly.
+ */
+import { describe, expect, test } from 'bun:test';
+import { normalizeSlot } from '../../src/core/facts/slots.ts';
+
+const ALICE = 'people/alice-example';
+
+describe('normalizeSlot', () => {
+ test('keeps a valid slot with aliases normalized and the value trimmed', () => {
+ expect(normalizeSlot('Company', ' Acme Example ', ALICE)).toEqual({ dimension: 'employer', value: 'Acme Example' });
+ expect(normalizeSlot('location', 'Springfield', ALICE)).toEqual({ dimension: 'location', value: 'Springfield' });
+ expect(normalizeSlot('job title', 'head of design', ALICE)).toEqual({ dimension: 'role', value: 'head of design' });
+ });
+
+ test('keeps an 80-character value and drops an 81-character one', () => {
+ expect(normalizeSlot('role', 'y'.repeat(80), ALICE)).toEqual({ dimension: 'role', value: 'y'.repeat(80) });
+ expect(normalizeSlot('role', 'x'.repeat(81), ALICE)).toBeNull();
+ });
+
+ test('drops a dimension outside the vocabulary, a blank or non-string value, and a slot with no entity', () => {
+ expect(normalizeSlot('hobby', 'chess', ALICE)).toBeNull();
+ expect(normalizeSlot('location', ' ', ALICE)).toBeNull();
+ expect(normalizeSlot('role', 7, ALICE)).toBeNull();
+ expect(normalizeSlot(undefined, 'Acme Example', ALICE)).toBeNull();
+ expect(normalizeSlot('employer', 'Acme Example', null)).toBeNull();
+ });
+});