Skip to content

World's first open-source enterprise world model.

16.9 KB
Raw Blame

Extraction writes what the document says

Extraction turns a chunk into open statements. This file says what it reads, what it writes, what it refuses, and how it is measured. Records: [0044] (the contract), [0039] and [0040] (the chunk it sees, in sources), [0041] (names), [0001] (drop signals), [0015] (memory documents), and the superseded typed contract of [0006], [0012], [0031], [0037]. Cut 1 landed in #731, memory documents joined in #735, and the typed path is deleted in #736.

What it does today

Input. One call per chunk, at temperature 0, with the document's filename, the things already accepted earlier in the document, the document's opening (its first 1,500 characters, #680) and the chunk text. No ontology, no document date, no request for dates [0044 d2, #731]. The chunk is the whole context: what its boundary cut away cannot be recovered by the prompt, so the chunker's rules are extraction's rules [0039].

Output, the compact contract (utopia_extract::open). e: things, each a name or a description, the document's kind word, and whether it is named or described. s: statements, each opening with its own verbatim quote and then naming its subject and object in words, the relation phrase as written, an object name or a literal value, qualifiers keyed by the document's role words, and when / ended time words. n: other names the document gives a thing. Statements name their sides in words and carry their own quote because a numbered contract mixed the ids up on dense passages (misworded 10 to 14% with ids, 2 to 7% with names) [#731]. Parsing is per item: a malformed item is counted and a truncated reply is repaired to its last complete item.

What is a thing, what is a statement [#743, prior-work items 1 and 6]. A thing is named when its name is a proper name or a fixed term, one that means the same thing in any document (a person, an organization, a product, a place, a document, a law, an event; a disease, a drug, an industry, a product category, an indicator); it is described when it is a role or a generic phrase whose referent the passage decides ("the company", "patients", "各部门"), however particular it is there. Named things resolve across documents by name [0041]; described things exist only in their document and only when a statement points at them. A phrase is the verb with the words that belong to it, so that subject, phrase and object read as a sentence on their own, never a bare verb cut from a longer verb phrase; several verbs sharing one object are one statement, one verb with several objects is one statement per object. The object is what the verb acts on; where, how, why, with what and for whom go in qualifiers under the passage's own role word, a named thing mentioned there is still listed and the qualifier names it, and a generic phrase that appears only there stays words in the qualifier. In a table, a cell is a statement about its row's thing whose phrase is the column heading; when the heading names a time it is when instead, the statement is about the thing the caption names and the phrase is the row label with its section path, and a unit the caption gives is a qualifier [#744]. A statement the passage requires, plans, expects, forecasts or makes conditional carries a qualifier keyed mood whose value is the passage's own words for that ("要", "should", "will", "is expected to"); a verb that reports (announced, said) is not a mood; the phrase stays as written, and alignment never materializes a statement with a mood as a typed fact [#745]. A rule that gave a subjectless sentence the passage's addressee or speaker as its subject was tried and withdrawn in the same change: under it the model returned nothing for a fifth of the chunks of a filing, tables included, and the bullets of a release already take the company as their subject without it. These are contract rules, not server checks: the shape checks below look at structure, never at vocabulary.

What a call to the endpoint survives [#757]. Extraction, both aligners and time resolution share one retry: a failure that fixes itself is waited out (five tries, backing off to a minute, honouring Retry-After), anything else is raised. Four kinds fix themselves: the endpoint rate limits, it answers 408, 502, 503 or 504, the request never reaches it, or its stream stops in the middle of an answer. A 500 does not, because that can be the endpoint's own bug, and neither does a read timeout: the ceiling is 300 seconds of silence and retrying that five times is 25 minutes.

The call is streamed and the answer assembled, so that the timeout measures silence rather than thinking. Not streamed, the first byte waits for the whole answer, so with reasoning on the model's thinking counts as silence: one dense passage of a filing crossed 300 seconds three times and was declared dead, while the same passage streamed has bytes at 2.6 seconds and finishes at 231. It is not faster, it is only honest about what silence means. A stream that ends without [DONE] or a finish_reason is a failure and not a short answer, because half an answer is a valid string and the statements it lost would go unmentioned. A chunk that still cannot be extracted is a drop row (chunk_unextracted), so the gap survives the document being marked done.

What a reasoning cap costs. Thinking is where the time goes: on a dense 1,310-character passage the model spends 33,000 reasoning tokens and 1,500 on the answer. Of the ways to ask for less, this endpoint honours reasoning_effort, whose low setting halves the thinking (15.5k and 16.5k reasoning tokens against 33k and 35k at medium, 32k and 29.5k at high, 33k with the field left out). It ignores thinking.budget_tokens, reasoning.max_tokens and max_tokens: a call capped at 4,096 returned 45,428 completion tokens and finished normally.

Run end to end on the same four filings, the same 14 classes and 28 properties and the same 55 chunks, the low setting is a quarter quicker per chunk (150 seconds to 115) and costs on every other axis: 885 open statements become 729, figures in tables that reach no statement go from 2% to 16% and in prose from 0% to 3%, quotes that do not occur in the chunk go from 46 to 107, and the judge reads 6.9% of statements as misworded against 3.7%, where judging the same graph twice moves the number by 1.4. The lever is measured and left alone. The read timeout it was meant to answer is already answered by streaming.

Where a long answer stops [#760]. The extraction and alignment calls send an explicit max_tokens of 65,536. Without it the ceiling is whatever the endpoint defaults to: with reasoning disabled a dense passage's answer stopped at exactly 4,096 completion tokens with finish_reason: length, and that JSON could not be parsed.

The number is measured against the completion, not against the answer, because max_tokens does not mean the same thing everywhere: Anthropic counts thinking inside it and OpenAI's max_completion_tokens counts reasoning tokens. On an endpoint that counts reasoning, a ceiling chosen to fit the answer would be a budget for the thinking instead, and that is the lever measured and rejected under What a reasoning cap costs above: holding this model to about 16,000 reasoning tokens cost 885 statements against 729, and took table figures reaching no statement from 2% to 16%. The largest completion measured on this path is 45,428 tokens, so 65,536 sits above it and can only ever guard against the endpoint's own default. The answer itself is about 1,500 tokens, nowhere near either line.

Sending it is not a guarantee. The same endpoint ignores the field with reasoning on, where a call capped at 4,096 returned 45,428 completion tokens and finished normally; it is honoured with reasoning off, which is where the truncation was seen. The field costs nothing where it is not read.

finish_reason travels with the answer rather than being dropped once the stream is known to have ended, because a reply cut at the ceiling is a valid string and looks complete to the parser. With the reason carried through, a chunk that hits the ceiling says so — the drop row reads hit the token ceiling instead of only could not be parsed — and the two causes stay apart: an answer that is too long is a number to change, a reply that does not fit the shape is not. Across all bases in the e2e database truncated_reply has fired 10 times, and those rows keep every complete item.

Where a looping answer stops [#1301]. At temperature 0 a small local model sometimes falls into a loop and writes the same row until the ceiling: on qwen3-8b about 3% of extraction calls, each holding a slot for over an hour and together about 60% of the tokens generated. The ceiling stays where it is, for the reasons above. Instead the streamed reply is read until the same span of 30 to 600 bytes fills its tail five times in a row, and only if that run's shortest period is also at least 30 bytes: every statement starts with a verbatim quote, and a quoted dot leader or a row of empty table cells repeats with a short period. A loop of rows shorter than 30 bytes is left to run to the ceiling (4 of 57 cuts in one night of a local deployment) rather than risk cutting a quote. The reply comes back marked repeating, apart from finish_reason, so it is never counted as the ceiling. Its drop row reads started repeating itself under the same truncated_reply code (chunk_unextracted when nothing before the loop parses), and it keeps every complete item before the loop. The guard sits in the streaming call, so the alignment, errata, implication, time and ontology calls that stream get it too.

Server checks, each a drop reason in extraction_drops. Every quote must occur in the chunk (quote_not_in_chunk); every name in its quote (name_not_in_text); a time mention only when its words occur in the statement's own quote (time_not_in_quote), because the model otherwise attaches one time to unrelated sentences; every subject must be a listed thing (unknown_ref); an unlisted object becomes a literal value with a signal (object_undeclared); a string one entity already claims in the document is not a name of another (name_claimed_by_another). Offsets are computed on the server by locating the quote, never taken from the model. The remaining codes are malformed_item, truncated_reply and object_missing; a phrase that is the value itself or the subject's own name is kept and counted (phrase_is_value, phrase_is_subject, the two shapes a model writes for a table row that lost its heading or its caption; the subject and the value are there, so nothing is lost, #744); the 28 codes of the typed path went with it [#736]. A drop is a row, never silence; the table is cleared per document when extraction starts and shown per document in the Library, apart from Review [0001, 0005].

What it writes (see ledger). Open statements as facts rows with layer = 'open', the phrase, no predicate and no valid_*; statement_qualifiers keyed by role word; time_mentions as verbatim words with an offset; evidence with quote_start / quote_end; a described thing as an entity with description and no name fact; name facts on known_as [0041]; attested_at from the document's date only when that date came from the content or a dating source, never the upload time (#714). Dedup is on (base, subject, phrase, object), which closes the NULL-predicate dedup hole of [0010].

What it skips. The ontology lists and per-chunk retrieval, predicate matching, the temporal reconcile, signature checks, type resolution and ontology bootstrap [#731, #736]. Identity still runs per handle (resolve_handle, namesake reviews, the govern and adjudicate enqueues); a chunk of origin described keeps its confidence ceiling [0040]. Until alignment, a base holds open statements and no typed facts, so timelines, conflicts and signature queues stay empty.

Memory documents take the same path; their statements wait in pending_facts with phrase, qualifiers, time words and quote span, and a nod writes an open statement [0015, #735].

Why

  • Statements in the document's words are faithful: 0 of 333 not stated and 2% misworded in the prototype, against 4 to 9% not stated when bound at write time [0044].
  • No ontology in the prompt: binding happens per signature and the ontology grows with use; a 1,010-class ontology cost 108k tokens a chunk and made the model choose worse [0044, 0006].
  • No dates from the model: arithmetic inside the model leaves no trace, and "today" resolved against the upload day; time is words now and an interpretation later [0045, #714].
  • Code checks structure, never vocabulary. A name is kept because it is in the text, not because a list says it is a name; the same rule for quotes and time words [0041, 0044 d8].
  • A described thing is an entity and gets no name fact, so a description never bridges two documents [#731].
  • Named sides and per-statement quotes cost tokens and buy the not-stated rate [#731]; the quote comes first so the statement is written from the copied sentence [Huang 2024, prior-work item 9, #748].
  • Every drop is a row because an append-only ledger with evidence on every fact cannot drop silently [0001].
  • Temperature 0 because the endpoint default swung one paragraph between 8 and 31 statements [#731].

Benches and thresholds

scripts/bench/recall.mjs (SEC filings, pharma, ai-timeline; Re-DocRED fetched by script and never used as prompt examples) scores entity-pair recall; judge_open.mjs has a judge model read the chunk and reports stated, misworded and not stated, then in a second pass marks each misworded statement extrapolated (it goes beyond what the document gives the pair) or contradicted (the document says otherwise: direction, phrase or the value of another cell) [prior-work item 8, #749], and separately whether the statement reads on its own without the document (alone, the uninformative-phrase class of prior-work item 1); identity.mjs, govern.mjs, temporal.mjs and the lease bench cover the other domains. The judge's own variance is measured: the same 909 statements judged twice came out misworded 5.0% and 3.6% with 97% of verdicts identical, and reads-alone 4.8%, 8.6% and 10.7% across three passes, so a difference under two points of misworded between two runs is noise and reads-alone is reported but not compared between single runs [#749]. Extraction has its own spread, measured the same way: the four filings extracted twice under one configuration give 885 and 824 statements, misworded 3.7% and 4.2%, not stated 0.0% and 0.2%, figures in tables reaching no statement 2% and 4%. A run moves the statement count by about seven per cent and misworded by half a point, so the two spreads together are what a claim has to clear. figures.mjs counts every figure a passage writes against every statement drawn from that passage, reads no model, and separates table rows from prose; the judge counts a statement once per piece of evidence, so its denominator is statement-evidence pairs rather than statements (845 against 824 on the second run). Every cut reports at least three domains, two runs per configuration, with the judge's calibration stated; no F1 against Re-DocRED, whose gold omits true facts [0044]. Thresholds: not stated at most 2%, entity-pair recall no lower than before, prompt tokens per document reported. Measured at cut 1: not stated at or under 2% in 16 of 18 corpus-rounds; NVDA full documents 44 to 46 of 52 (one standard deviation about 2.7) against 43 and 45 before [#731, 0039].

Superseded

The typed contract: the ontology laid out under a 24,000-character budget or retrieved per chunk with ancestor completion [0006]; [event] / [eternal] marks and the zoned-time sentence [0031, 0024]; qualifiers listed after a relation and keyed by a declared attribute [0037]; direction corrected by signature at write time [0012]; specific_type and proposed_type for type resolution [0001]; the model's relative flag on a deadline [0022]; the word lists for entity names and clauses [0012]. The ledger rules those records made stand (see ledger).

Proposed and not built

  • Identity profiles [0044 d6]; a layer marker in the interface (an open row renders as an unnamed relation labelled by its phrase). Alignment producing typed facts [0044 cut 2] and time interpretation against the document [0045 cuts 1 to 3] were on this list and are built.
  • A second pass asking which figures and dates are not yet in a statement (dense sentences drop the amount); the period column of a financial-table cell [#729].
  • The errata agent over the typed graph [0044 d7] was on this list and is built: see design/ontology.

Open questions

  • Chinese chunks get less context at 300 tokens than at 1,200 characters; unmeasured [0039].
  • Whether a stated observation should raise the confidence of a fact first seen in a description [0040].
  • Recall of facts that need more than one sentence waits for derivation rules [0044].