Skip to content

World's first open-source enterprise world model.

10.4 KB
Raw Blame

A source hands over its history

Records: [0013] (connectors), [0023] and [0033] (RSS), [0039] (chunks), [0040] (origin and media), [0035] (vector index), [0036] d7 (schema documents), [0005] (sync alerts), [0022] and [0045] d3 (document dates), [0016] E.

What it does today

Ingest. A file or a synced item goes into a content-addressed blob store; documents and document_versions keep every version; ingest_item() owns identity (external_key), sha256 dedup and versions; a document is ready for search once embedded while extraction queues behind it [0001, 0013, pipeline]. documents.doc_time with doc_time_source (content, source, upload_time, none) is the document's own date; upload time is recorded time and never a document date, so an upload-dated document counts as undated for attestation and ordering [0022, 0045, #714].

Connectors. GitHub issues, Jira, Notion, RSS, WebDAV, S3 / Azure / GCS, and custom push with an ingest_token; the kinds come from one SourceKind enum that the allowlist, the dispatcher and the web list derive from; credentials never leave the server (SOURCE_SECRET_KEYS) and a blank field keeps the stored value; a truncated round says so [0013]. A ticket is one document whose body carries its history as dated declarative sentences and a ## History list; GitHub fetches events per ticket; timestamps are written to the second [0013]. A GitHub repository file is one document keyed by its path and dated by the last commit that touched it; a file whose blob SHA has not changed is not fetched again, and identical files at several paths go in once [#1303].

Statements. A statements source takes the open extraction contract itself (e, s, n) on POST /sources/{id}/statements, with the api push's identity, versions, tombstones and run history; the payload is the document and its single chunk, and extraction parses the chunk instead of prompting a model, then runs the unchanged path, so a pushed statement is an open statement with the item as its evidence (null offsets) and reaches the typed graph only through alignment. The contract has no slot for a property or a class and unknown keys are refused; a second push under the same identity marks the earlier statements stale rather than closing them; tables belong on a mount, not here [0054].

RSS. Observations are not documents: one table with baseline, candidate and no_source rows; jobs own attempts, documents own identity; an entry without a GUID or an article link is skipped; purge and reappearance are fenced by database time; one Readability extractor serves linked pages [0023]. The source list aggregates only the listed source's current generation with a lateral join, and the API carries a typed nested rss_full_content summary [0033].

Parse and chunk. The parser's Markdown is read into blocks (heading, paragraph, table with header and rows, rule); the packer fills a token budget (300 by default, cl100k, UTOPIA_CHUNK_TOKENS); a table travels with its caption and header and a continuation repeats both; a caption attaches to every table that follows it; a row is never split; headings become a breadcrumb prefix and chunks.heading; a thematic break is not a boundary and a table it split is joined back; no overlap; chunk text is verbatim slices so a quote is found by offset; a document is re-chunked on its next reprocess [0039]. A Word paragraph outside a table ends a Markdown paragraph, so the caption and packing rules see the same boundaries as Markdown and HTML: a blank line follows it, a w:br inside it stays a single newline, a paragraph inside a table cell stays a space, and empty paragraphs disappear [#1024]. A DOCX paragraph outside a table is a heading when it has an outline level: its own w:outlineLvl, else its style's, else a style named heading N, else the style it is based on (style ids differ by the language of Word, the names do not); so a Word file's sections reach the chunks and the section dates of 0064.

Tables [#744, #750, prior-work item 26]. An HTML table is rendered from the DOM before the Markdown converter sees it, by structure alone: cells hidden by style are skipped, spanning cells are laid on a grid, columns empty in every row go, a cell holding only a symbol ("$", ")") rejoins the number beside it. Rows are classified by what they hold: a lone cell before any header or data is a caption line, a row of words with no label column is a header row, a lone label after the headers is a section, a label followed by figures is data. Header rows are stacked per column into one heading ("Three Months Ended July 26, 2026"); sections fold into the labels of the rows under them ("Current assets › Accounts receivable, net"), with indentation read from leading empty cells or the label's padding, and a section ends at a row of its own depth only once it has had a deeper child; caption lines become a paragraph before the table that ends with a colon. The chunker then does what it already did: the caption and the header travel with every piece, and a run of up to five short paragraphs before a table is its caption, so a statement's chunk names the report, the unit and the period. Nested tables and the tables of other parsers still take the converter's path with the first row promoted to header. Measured on the NVDA earnings release: 52 chunks became 32, none of them a table without its header. DOCX tables and spreadsheets are laid on the same grid by their parsers (#750): a DOCX cell keeps its column span and the indent of its first paragraph, a sheet's first row with two or more cells is its header even when the headings are years, and a table with no figure in it is read as a header row followed by records; a CSV or TSV file is read as records instead: its first record is the header, every field keeps its own column, and an empty field is a missing value, not a section or a label (#1035); a sheet's date cell is written as the date it shows (ISO, with its time of day, or an elapsed time as hours:minutes:seconds), not as Excel's day count; the text layer of a PDF carries no table structure.

Origin and anchor. chunks.origin (stated, ocr, transcribed, described), origin_model, and an anchor whose shape is checked per origin (page and box; start, end and speakers; page and image or part); the packer never mixes origins in one chunk; a described block is its own chunk with the breadcrumb and caption. Images and recordings are recognised by header or extension and a PDF with an empty text layer is a scan; without the reader the document fails once with reader_needed and a document.needs_reader alert, and saving the setting queues it again [0040]. Scans and images go to the workspace's chosen OCR provider (llm_settings.ocr_*), one segment per page. MinerU waits with Deferred on the remote task recorded on the document; Ark reads the whole file in one pass, sending images whole and rendering PDF pages with Poppler, with bounded page retries and no persistent progress [0065]. Ark transcribes only written text, records the model and real page numbers, and adds no guessed boxes; recordings go to a diarizing transcription model (transcribe_*) and a transcript without speaker labels is refused; speakers are written into the text as turns [0040 cuts 2 and 3]. Facts from a described chunk enter below the auto-close threshold [0040 d4].

Embedding and index. chunks.embedding and entities.profile_embedding are vector with no fixed dimension; the first write of a dimension requests a partial HNSW expression index built by a build_vector_index job outside any transaction (512 MB maintenance_work_mem); reads write the dimension as a literal, set hnsw.iterative_scan = relaxed_order with a raised scan memory, and let the planner choose; above 2000 dimensions stays exact [0035]. Superseded chunks keep their vectors so retrieval rewinds [0019].

A schema document is a search corpus, not a source of facts: a mounted database's schema is indexed and never extracted (sources.config.extract) [0036 d7].

Why

  • A source is worth connecting if it has real timestamps, can contradict itself, has a stable identity and holds enterprise knowledge; the valuable part of a ticket is its timeline, and most systems keep it and only need to be asked [0013].
  • Sentences with dates extract to facts with a start; key-value pairs make the model guess [0013].
  • Fixtures come from real responses; hand-written JSON proves only the imagined shape [0013].
  • No provider abstraction until a third source shares a shape [0013].
  • A feed item can vanish before acquisition and a summary is not full content, so observations are retained apart from documents [0023].
  • The chunk boundary is part of the extraction contract: 13 of 35 chunks began mid-table without a header and every miss was a literal; a 1,000-token budget lost (36 of 52) because the extractor's output does not grow with its input [0039].
  • The pipeline eats text and cannot tell a paraphrase of a chart from a sentence; the ledger acts on what it is told, so a description may not close a correct fact on its own [0040].
  • Per-modality settings because a board recording is more sensitive than chat and may stay on a local model [0040].
  • A job, not a migration, for the index: the dimension is known only when vectors arrive and CREATE INDEX CONCURRENTLY cannot run inside a migration; ivfflat loses recall quietly [0035].
  • Truncation and partial rounds are reported, or "sync complete" misleads [0013].

Proposed and not built

  • Descriptions of charts and photos under the ceiling (cut 4) and video (cut 5); the origin and anchor in the interface, opening the page, image or recording [0040].
  • Feishu and Confluence connectors with a block-tree renderer and sampled versions [0013].
  • An external parser behind the block model for PDF layout; evidence naming a table cell or an image region; real Markdown tables from the DOCX, PDF, spreadsheet and CSV parsers, rendered the way HTML tables now are [0039].
  • Backup and restore (#712), a 100k-document benchmark (#713) [0016 E].

Open questions

  • Whether low engine confidence on an OCR or transcript segment should cap facts [0040].
  • The header the parser hands over, and the budget measured on its own [0039].
  • A recording's date as a doc_time_source; what gets embedded when a chunk holds references [0040].
  • Recall on two large tenants of different subject matter; an old dimension's index when a workspace changes model [0035].