Skip to content

Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.

Add git-native team sync with conflict-free merge; async store interface

anirudhkumar-nanonets1ec96cb95873
diff --git a/.gitignore b/.gitignore
index c8bba29a..5498508d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,7 +6,10 @@ dist/
*.db
*.db-shm
*.db-wal
-.context-graph/
+# Ignore the local graph dir's contents (the SQLite replica) but keep the
+# committable, git-native team-sync file (Mode A).
+.context-graph/*
+!.context-graph/graph.jsonl
# Env
.env
diff --git a/README.md b/README.md
index b3ae18aa..0dc896ac 100644
--- a/README.md
+++ b/README.md
@@ -186,9 +186,9 @@ context-graph-web &
tailscale serve localhost:4680 # private HTTPS URL, visible only to your tailnet
```
-The two modes below sync **between separate graphs** instead — no shared server at all. Both rely on the graph's **conflict-free merge**: observations are a grow-only counter and free-text fields are last-writer-wins, so re-importing the same facts never double-counts and merge order never changes the result.
+### Git mode — sync separate graphs, no server at all
-### Mode A — Git-native (zero infra)
+Prefer no shared server? Each teammate keeps their own local graph and git moves the knowledge around. This relies on the graph's **conflict-free merge**: observations are a grow-only counter and free-text fields are last-writer-wins, so re-importing the same facts never double-counts and merge order never changes the result.
Commit the graph to your repo as a human-diffable file and let git move it around.
@@ -204,20 +204,6 @@ context-graph sync
`sync` imports the committed `.context-graph/graph.jsonl`, re-merges it into the local graph, and writes the merged result back. It's idempotent (safe to run repeatedly) and commutative (order-independent), so two teammates' additions always converge. The repo's `.gitignore` keeps the local SQLite replica private while tracking `graph.jsonl`. Best for teams already living in git who want no servers and offline-friendly sharing; the only friction is the occasional merge conflict on the file, resolved by re-running `sync`.
-### Mode B — Shared store (libSQL / Turso embedded replica)
-
-Point the engine at a shared libSQL primary. Each machine keeps a **local replica** (fast local reads); writes go to the primary, so everyone merges against authoritative shared state in near-real-time — no commit/pull step.
-
-```bash
-export CONTEXT_GRAPH_SYNC_URL="libsql://your-db.turso.io" # or http://your-sqld:8080
-export CONTEXT_GRAPH_AUTH_TOKEN="…" # token for the primary
-export CONTEXT_GRAPH_DB="./.context-graph/graph.db" # local replica file
-# optional: CONTEXT_GRAPH_SYNC_INTERVAL=30 (background pull, seconds)
-context-graph query "auth" # reads local; writes sync to the primary
-```
-
-Works with **Turso Cloud** or a **self-hosted `sqld`**. Best for teams that want live shared memory and are OK running (or renting) one small database; the friction is provisioning that primary and a token once.
-
> Both modes assume one embedding model per graph (see the note under *How it works*). Imports across mismatched embedding dimensions keep the facts but drop the incompatible vectors, and warn you to re-ingest sources to re-embed.
---
@@ -248,7 +234,7 @@ This is what makes contributions compound: the tenth agent to confirm a fact str
└────────────────────────┘
```
-- **Storage** — SQLite (via `better-sqlite3`) by default; a libSQL/Turso embedded replica for shared team graphs (set `CONTEXT_GRAPH_SYNC_URL`); or swap in your own `GraphStore`.
+- **Storage** — SQLite (via `better-sqlite3`) by default, or swap in your own `GraphStore`.
- **Extraction** — local **Ollama** (`llama3.2` by default) using structured JSON output; automatically upgrades to **OpenRouter** (`openai/gpt-4o-mini` by default, any tool-calling model) if `OPENROUTER_API_KEY` is set.
- **Embeddings** — local **in-process** model (`Xenova/all-MiniLM-L6-v2`, 384-dim) by default; automatically upgrades to **OpenAI** (`text-embedding-3-small`) if `OPENAI_API_KEY` is set.
- **Retrieval** — semantic match over entities + one-hop graph expansion + supporting source passages.
@@ -264,10 +250,6 @@ Everything is configurable via constructor options, environment variables, or de
```ts
new ContextGraphEngine({
dbPath: "./.context-graph/graph.db", // CONTEXT_GRAPH_DB
- // Team sharing — Mode B (libSQL/Turso embedded replica; opt-in):
- syncUrl: process.env.CONTEXT_GRAPH_SYNC_URL, // libsql://… or http://your-sqld
- authToken: process.env.CONTEXT_GRAPH_AUTH_TOKEN,
- syncIntervalSeconds: undefined, // CONTEXT_GRAPH_SYNC_INTERVAL (background pull)
// Cloud (used automatically when set):
openrouterApiKey: process.env.OPENROUTER_API_KEY,
openrouterModel: "openai/gpt-4o-mini", // CONTEXT_GRAPH_OPENROUTER_MODEL
diff --git a/examples/quickstart.ts b/examples/quickstart.ts
index d2cf2972..dbe3765b 100644
--- a/examples/quickstart.ts
+++ b/examples/quickstart.ts
@@ -47,8 +47,8 @@ async function main() {
const after = await engine.read("how is the team notified about failed charges?");
console.log(after.prompt);
- console.log("\nFinal graph:", engine.stats());
- engine.close();
+ console.log("\nFinal graph:", await engine.stats());
+ await engine.close();
}
main().catch((err) => {
diff --git a/package-lock.json b/package-lock.json
index 9f932034..6cb07793 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -20,7 +20,8 @@
},
"bin": {
"context-graph": "dist/cli.js",
- "context-graph-mcp": "dist/mcp.js"
+ "context-graph-mcp": "dist/mcp.js",
+ "context-graph-web": "dist/web.js"
},
"devDependencies": {
"@types/better-sqlite3": "^7.6.13",
@@ -3345,6 +3346,29 @@
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"license": "ISC"
},
+ "node_modules/ws": {
+ "version": "8.21.0",
+ "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz",
+ "integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==",
+ "license": "MIT",
+ "optional": true,
+ "peer": true,
+ "engines": {
+ "node": ">=10.0.0"
+ },
+ "peerDependencies": {
+ "bufferutil": "^4.0.1",
+ "utf-8-validate": ">=5.0.2"
+ },
+ "peerDependenciesMeta": {
+ "bufferutil": {
+ "optional": true
+ },
+ "utf-8-validate": {
+ "optional": true
+ }
+ }
+ },
"node_modules/yallist": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/yallist/-/yallist-5.0.0.tgz",
diff --git a/package.json b/package.json
index af746269..88e48116 100644
--- a/package.json
+++ b/package.json
@@ -39,6 +39,7 @@
"mcp": "tsx src/mcp.ts",
"web": "tsx src/web.ts",
"example": "tsx examples/quickstart.ts",
+ "test": "node --import tsx --test test/*.test.ts",
"prepare": "npm run build",
"prepublishOnly": "npm run build"
},
diff --git a/src/cli.ts b/src/cli.ts
index a42238b8..a401bd1c 100644
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -12,7 +12,7 @@
*/
import "dotenv/config";
import { Command } from "commander";
-import { readFileSync, writeFileSync } from "node:fs";
+import { readFileSync, writeFileSync, existsSync } from "node:fs";
import { ContextGraphEngine } from "./engine.js";
import { toHtml, toMermaid } from "./graph/export.js";
@@ -47,7 +47,7 @@ program
}
}
} finally {
- engine.close();
+ await engine.close();
}
});
@@ -67,7 +67,7 @@ program
: `✓ ${r.chunks} chunks, +${r.nodesCreated} nodes, +${r.edgesCreated} edges`,
);
} finally {
- engine.close();
+ await engine.close();
}
});
@@ -87,7 +87,7 @@ program
});
console.log(opts.json ? JSON.stringify(bundle, null, 2) : bundle.prompt);
} finally {
- engine.close();
+ await engine.close();
}
});
@@ -104,7 +104,7 @@ program
`✓ contributed — +${r.nodesCreated} nodes (${r.nodesUpdated} reinforced), +${r.edgesCreated} edges (${r.edgesUpdated} reinforced)`,
);
} finally {
- engine.close();
+ await engine.close();
}
});
@@ -128,7 +128,7 @@ program
);
}
} finally {
- engine.close();
+ await engine.close();
}
});
@@ -137,10 +137,10 @@ program
.description("Export the whole graph for viewing (interactive HTML, JSON, or Mermaid)")
.option("-f, --format <fmt>", "html | json | mermaid", "html")
.option("-o, --out <path>", "output file (default: context-graph.<ext>)")
- .action((opts: { format: string; out?: string }) => {
+ .action(async (opts: { format: string; out?: string }) => {
const engine = engineFrom();
try {
- const g = engine.exportGraph();
+ const g = await engine.exportGraph();
let content: string;
let ext: string;
if (opts.format === "json") {
@@ -158,20 +158,97 @@ program
console.log(`✓ wrote ${g.nodes.length} entities / ${g.edges.length} relationships to ${out}`);
if (ext === "html") console.log(` open it: open ${out}`);
} finally {
- engine.close();
+ await engine.close();
}
});
program
.command("stats")
.description("Show graph statistics")
- .action(() => {
+ .action(async () => {
const engine = engineFrom();
try {
- const s = engine.stats();
+ const s = await engine.stats();
console.log(`documents: ${s.documents}\nnodes: ${s.nodes}\nedges: ${s.edges}\nchunks: ${s.chunks}`);
} finally {
- engine.close();
+ await engine.close();
+ }
+ });
+
+/** Resolve the graph-file path for team sync (flag overrides the default). */
+function syncFilePath(engine: ContextGraphEngine, flag?: string): string {
+ const file = flag ?? engine.graphFilePath;
+ if (!file) {
+ throw new Error(
+ "No graph file path — an in-memory graph can't be synced. Pass --file <path> or use a file-backed --db.",
+ );
+ }
+ return file;
+}
+
+program
+ .command("push")
+ .description("Team sync (git mode): write the graph to a committable JSONL file")
+ .option("--file <path>", "graph file (default: graph.jsonl next to the db)")
+ .action(async (opts: { file?: string }) => {
+ const engine = engineFrom();
+ try {
+ const file = syncFilePath(engine, opts.file);
+ writeFileSync(file, await engine.exportJsonl());
+ const s = await engine.stats();
+ console.log(`✓ wrote ${s.nodes} entities / ${s.edges} relationships to ${file}`);
+ console.log(` commit it: git add ${file} && git commit -m "update context graph"`);
+ } finally {
+ await engine.close();
+ }
+ });
+
+program
+ .command("pull")
+ .description("Team sync (git mode): import + re-merge a teammate's JSONL file")
+ .option("--file <path>", "graph file (default: graph.jsonl next to the db)")
+ .action(async (opts: { file?: string }) => {
+ const engine = engineFrom();
+ try {
+ const file = syncFilePath(engine, opts.file);
+ if (!existsSync(file)) {
+ console.log(`No graph file at ${file} yet — nothing to pull.`);
+ return;
+ }
+ const r = await engine.importJsonl(readFileSync(file, "utf8"));
+ for (const w of r.warnings) console.warn(`⚠ ${w}`);
+ console.log(
+ `✓ merged ${file} — +${r.nodesCreated} nodes (${r.nodesUpdated} reinforced), ` +
+ `+${r.edgesCreated} edges (${r.edgesUpdated} reinforced), ` +
+ `+${r.documentsAdded} docs, +${r.chunksAdded} sources`,
+ );
+ } finally {
+ await engine.close();
+ }
+ });
+
+program
+ .command("sync")
+ .description("Team sync (git mode): pull a teammate's JSONL, re-merge, then push the merged graph")
+ .option("--file <path>", "graph file (default: graph.jsonl next to the db)")
+ .action(async (opts: { file?: string }) => {
+ const engine = engineFrom();
+ try {
+ const file = syncFilePath(engine, opts.file);
+ if (existsSync(file)) {
+ const r = await engine.importJsonl(readFileSync(file, "utf8"));
+ for (const w of r.warnings) console.warn(`⚠ ${w}`);
+ console.log(
+ `↓ merged in — +${r.nodesCreated} nodes (${r.nodesUpdated} reinforced), ` +
+ `+${r.edgesCreated} edges (${r.edgesUpdated} reinforced)`,
+ );
+ }
+ writeFileSync(file, await engine.exportJsonl());
+ const s = await engine.stats();
+ console.log(`↑ wrote merged graph (${s.nodes} entities / ${s.edges} relationships) to ${file}`);
+ console.log(` commit it: git add ${file} && git commit -m "sync context graph"`);
+ } finally {
+ await engine.close();
}
});
diff --git a/src/engine.ts b/src/engine.ts
index 6b2e4860..24358a45 100644
--- a/src/engine.ts
+++ b/src/engine.ts
@@ -1,5 +1,5 @@
import { readFileSync, readdirSync } from "node:fs";
-import { basename, join } from "node:path";
+import { basename, dirname, join } from "node:path";
import { extractPdfText, isPdfPath } from "./ingest/pdf.js";
import { buildGraphExport, type GraphExport } from "./graph/export.js";
import type { GraphStore } from "./graph/store.js";
@@ -11,6 +11,7 @@ import { OpenAIEmbedder } from "./ai/openai.js";
import { LocalEmbedder, OllamaExtractor } from "./ai/local.js";
import { chunkText } from "./ingest/chunker.js";
import { mergeExtraction } from "./graph/merge.js";
+import { serializeGraph, importGraph, type ImportResult } from "./graph/serialize.js";
import { retrieve, type RetrieveOptions } from "./retrieval/retriever.js";
import type { ContextBundle, Chunk, Extraction, GraphStats } from "./graph/types.js";
import { contentHash, newId } from "./util/id.js";
@@ -66,7 +67,7 @@ export class ContextGraphEngine {
this.cfg = resolveConfig(config);
}
- /** The underlying graph store (lazily opened). */
+ /** The underlying graph store (lazily opened): an explicit `store` override, or the local SQLite file. */
get store(): GraphStore {
if (!this._store) {
this._store = this.cfg.store ?? new SqliteStore(this.cfg.dbPath);
@@ -126,7 +127,7 @@ export class ContextGraphEngine {
const title = opts.title ?? source;
const hash = contentHash(text);
- const existing = this.store.getDocumentByHash(hash);
+ const existing = await this.store.getDocumentByHash(hash);
if (existing) {
return {
documentId: existing.id,
@@ -141,35 +142,38 @@ export class ContextGraphEngine {
}
const documentId = newId("doc");
- this.store.insertDocument({ id: documentId, title, source, hash, createdAt: now });
+ await this.store.insertDocument({ id: documentId, title, source, hash, createdAt: now });
const pieces = chunkText(text, this.cfg.chunkSize, this.cfg.chunkOverlap);
const embeddings = await this.embedder.embed(pieces);
- pieces.forEach((piece, i) => {
+ for (let i = 0; i < pieces.length; i++) {
const chunk: Chunk = {
id: newId("chunk"),
documentId,
ordinal: i,
- text: piece,
+ text: pieces[i],
embedding: embeddings[i],
createdAt: now,
};
- this.store.insertChunk(chunk);
- });
+ await this.store.insertChunk(chunk);
+ }
// Extract per chunk (parallel, bounded), then merge sequentially so each
- // merge sees nodes created by earlier chunks.
+ // merge sees nodes created by earlier chunks. The document id is a unique
+ // observation key, so re-importing this document later never double-counts.
const extractions = await mapWithConcurrency(pieces, 4, (piece) =>
this.extractor.extract(piece, { hint: title }),
);
+ const provenance = `doc:${documentId}`;
const totals = { nodesCreated: 0, nodesUpdated: 0, edgesCreated: 0, edgesUpdated: 0 };
for (const extraction of extractions) {
const r = await mergeExtraction(
this.store,
this.embedder,
extraction,
- `doc:${documentId}`,
+ provenance,
+ provenance,
this.cfg.mergeThreshold,
now,
);
@@ -213,10 +217,45 @@ export class ContextGraphEngine {
}
/** A serializable snapshot of the whole graph, for export/visualization. */
- exportGraph(): GraphExport {
+ async exportGraph(): Promise<GraphExport> {
return buildGraphExport(this.store);
}
+ /**
+ * Default path for the git-committed graph file (Mode A), alongside the db
+ * file — e.g. `.context-graph/graph.jsonl` next to `.context-graph/graph.db`.
+ * Undefined for an in-memory graph.
+ */
+ get graphFilePath(): string | undefined {
+ if (this.cfg.dbPath === ":memory:") return undefined;
+ return join(dirname(this.cfg.dbPath), "graph.jsonl");
+ }
+
+ /** Human label for the active embedding model, recorded in exported files. */
+ private embeddingModelLabel(): string {
+ if (!this.cfg.forceLocal && this.cfg.openaiApiKey) return this.cfg.embeddingModel;
+ return this.cfg.localEmbeddingModel;
+ }
+
+ /**
+ * Serialize the whole graph to a JSONL string for git-native team sync
+ * (Mode A). Commit the result; teammates {@link importJsonl} it after a pull.
+ */
+ async exportJsonl(): Promise<string> {
+ return serializeGraph(this.store, {
+ embeddingDimensions: this.embedder.dimensions,
+ embeddingModel: this.embeddingModelLabel(),
+ });
+ }
+
+ /**
+ * Import a serialized graph (from a teammate's commit) and CRDT-merge it into
+ * the local graph. Idempotent — re-importing the same content is a no-op.
+ */
+ async importJsonl(jsonl: string): Promise<ImportResult> {
+ return importGraph(this.store, jsonl, this.embedder.dimensions);
+ }
+
/**
* Contribute a learning discovered by an agent. The text is stored as
* retrievable evidence and its entities/relations are merged into the graph,
@@ -230,12 +269,12 @@ export class ContextGraphEngine {
const hash = contentHash(`${provenance}|${learning}`);
let documentId: string;
- const existing = this.store.getDocumentByHash(hash);
+ const existing = await this.store.getDocumentByHash(hash);
if (existing) {
documentId = existing.id;
} else {
documentId = newId("doc");
- this.store.insertDocument({
+ await this.store.insertDocument({
id: documentId,
title,
source: opts.source ?? provenance,
@@ -243,7 +282,7 @@ export class ContextGraphEngine {
createdAt: now,
});
const [embedding] = await this.embedder.embed([learning]);
- this.store.insertChunk({
+ await this.store.insertChunk({
id: newId("chunk"),
documentId,
ordinal: 0,
@@ -256,11 +295,14 @@ export class ContextGraphEngine {
const extraction: Extraction = await this.extractor.extract(learning, {
hint: "An agent's learning to fold into the shared knowledge graph.",
});
+ // `agent:<id>` is the human-facing origin; the (unique) contribution
+ // document id is the observation key, so replays stay idempotent.
const r = await mergeExtraction(
this.store,
this.embedder,
extraction,
provenance,
+ `doc:${documentId}`,
this.cfg.mergeThreshold,
now,
);
@@ -275,13 +317,13 @@ export class ContextGraphEngine {
}
/** Current graph statistics. */
- stats(): GraphStats {
+ async stats(): Promise<GraphStats> {
return this.store.stats();
}
/** Close the underlying store. */
- close(): void {
- this._store?.close();
+ async close(): Promise<void> {
+ await this._store?.close();
}
}
diff --git a/src/graph/crdt.ts b/src/graph/crdt.ts
new file mode 100644
index 00000000..42c01db6
--- /dev/null
+++ b/src/graph/crdt.ts
@@ -0,0 +1,78 @@
+/**
+ * Conflict-free primitives that make the graph safe to merge across replicas.
+ *
+ * The whole point: two teammates can build the graph independently (offline, on
+ * different machines) and later reconcile it — via a git pull of a serialized
+ * graph, or an embedded-replica sync — **without** double-counting reinforcement
+ * or letting merge order change the result.
+ *
+ * Two ideas do the heavy lifting:
+ *
+ * 1. **Observations are a grow-only counter (G-Counter).** Each observation
+ * event has a globally-unique key (the id of the document/contribution that
+ * produced it). Merging two counters is an element-wise `max`, which is
+ * commutative, associative and *idempotent* — replaying the same record is a
+ * no-op. `observations` (the scalar) and `confidence` are then *derived*
+ * from the counter, never mutated in place.
+ *
+ * 2. **Free-text fields (summary/description) are last-writer-wins registers**
+ * keyed by a wall-clock stamp, with deterministic tiebreaks so every replica
+ * picks the same winner.
+ */
+import type { ObservationCounter } from "./types.js";
+
+/**
+ * Confidence as a pure function of the observation total. Reproduces the
+ * original reinforcement curve exactly — a fresh single observation is 0.6, and
+ * each further observation closes 20% of the gap to 1 (0.6 → 0.68 → 0.744 → …),
+ * capped at 0.99 — but as a *derivation* rather than a mutated register, so it
+ * converges no matter how or how often records are merged.
+ */
+export function confidenceFromObservations(total: number): number {
+ if (total <= 0) return 0.5;
+ return Math.min(0.99, 1 - 0.4 * Math.pow(0.8, total - 1));
+}
+
+/** Sum of a G-Counter's per-source counts (the total observation count). */
+export function totalObservations(sources: ObservationCounter): number {
+ let n = 0;
+ for (const key in sources) n += sources[key];
+ return n;
+}
+
+/** Increment (in place) the count for one observation key. */
+export function bumpObservation(sources: ObservationCounter, key: string, by = 1): void {
+ sources[key] = (sources[key] ?? 0) + by;
+}
+
+/**
+ * Merge two G-Counters by element-wise max. Commutative, associative, and
+ * idempotent — the mathematical guarantee behind replay-safe imports.
+ */
+export function mergeObservationCounters(
+ a: ObservationCounter,
+ b: ObservationCounter,
+): ObservationCounter {
+ const out: ObservationCounter = { ...a };
+ for (const key in b) out[key] = Math.max(out[key] ?? 0, b[key]);
+ return out;
+}
+
+/** Set-union of two grow-only string sets (dropping empties), order-stable. */
+export function unionSet(a: string[], b: string[]): string[] {
+ return [...new Set([...a, ...b].filter((v) => v && v.length > 0))];
+}
+
+/**
+ * Last-writer-wins pick between two timestamped text values. The newer stamp
+ * wins; ties break toward the longer (more informative) text, then lexically —
+ * so all replicas converge on the same choice regardless of merge order.
+ */
+export function pickLatestText(
+ a: { text: string; at: string },
+ b: { text: string; at: string },
+): { text: string; at: string } {
+ if (a.at !== b.at) return a.at > b.at ? a : b;
+ if (a.text.length !== b.text.length) return a.text.length > b.text.length ? a : b;
+ return a.text >= b.text ? a : b;
+}
diff --git a/src/graph/export.ts b/src/graph/export.ts
index d3847c11..51f13c69 100644
--- a/src/graph/export.ts
+++ b/src/graph/export.ts
@@ -8,8 +8,8 @@ export interface GraphExport {
}
/** Read the entire graph out of a store into a serializable snapshot. */
-export function buildGraphExport(store: GraphStore): GraphExport {
- const nodes = store.allNodes().map((n) => ({
+export async function buildGraphExport(store: GraphStore): Promise<GraphExport> {
+ const nodes = (await store.allNodes()).map((n) => ({
id: n.id,
name: n.name,
type: n.type,
@@ -17,7 +17,7 @@ export function buildGraphExport(store: GraphStore): GraphExport {
confidence: n.confidence,
observations: n.observations,
}));
- const edges = store.allEdges().map((e) => ({
+ const edges = (await store.allEdges()).map((e) => ({
sourceId: e.sourceId,
targetId: e.targetId,
relation: e.relation,
diff --git a/src/graph/merge.ts b/src/graph/merge.ts
index 8ea2401a..67e3d091 100644
--- a/src/graph/merge.ts
+++ b/src/graph/merge.ts
@@ -7,11 +7,14 @@ import type {
} from "./types.js";
import { newId, normalizeName } from "../util/id.js";
import { cosineSimilarity } from "../util/similarity.js";
-
-/** Reinforce a confidence score toward 1 as evidence accumulates. */
-function reinforce(confidence: number): number {
- return Math.min(0.99, confidence + (1 - confidence) * 0.2);
-}
+import {
+ bumpObservation,
+ confidenceFromObservations,
+ mergeObservationCounters,
+ pickLatestText,
+ totalObservations,
+ unionSet,
+} from "./crdt.js";
/** Pick the more informative of two summaries (prefers the longer, richer one). */
function betterSummary(existing: string, incoming: string): string {
@@ -20,10 +23,6 @@ function betterSummary(existing: string, incoming: string): string {
return incoming.length > existing.length * 1.2 ? incoming : existing;
}
-function uniq(values: string[]): string[] {
- return [...new Set(values.filter((v) => v && v.length > 0))];
-}
-
export interface MergeResult {
nodesCreated: number;
nodesUpdated: number;
@@ -34,19 +33,28 @@ export interface MergeResult {
}
/**
- * Merge an {@link Extraction} into the graph.
+ * Merge an {@link Extraction} into the graph (the *live* write path used by
+ * ingest/contribute).
*
* Entities are matched against existing nodes by (a) exact normalized name /
- * alias and (b) embedding similarity above `mergeThreshold`. Matches are
- * reinforced (confidence up, observation count up, provenance appended);
- * non-matches are created. Relations are then linked between the resolved node
- * ids, deduped by (source, target, relation).
+ * alias and (b) embedding similarity above `mergeThreshold`. A match records a
+ * fresh observation under `observationKey` (a bump of the CRDT counter);
+ * confidence is then *derived* from the new total, never mutated directly.
+ * Non-matches are created. Relations are linked between resolved node ids,
+ * deduped by (source, target, relation).
+ *
+ * @param provenance Human-facing origin label added to the grow-only set
+ * (e.g. `doc:<id>` or `agent:<id>`).
+ * @param observationKey Globally-unique key for *this* observation event
+ * (the document/contribution id), so re-observing the
+ * same event later is idempotent under counter merge.
*/
export async function mergeExtraction(
store: GraphStore,
embedder: Embedder,
extraction: Extraction,
provenance: string,
+ observationKey: string,
mergeThreshold: number,
now: string,
): Promise<MergeResult> {
@@ -66,25 +74,30 @@ export async function mergeExtraction(
// Cache the full embedded-node set once; refresh with nodes we create so
// duplicates *within this same extraction* also collapse.
- const candidatePool = store.allEmbeddedNodes();
+ const candidatePool = await store.allEmbeddedNodes();
for (let i = 0; i < entities.length; i++) {
const ent = entities[i];
const embedding = embeddings[i];
const norm = normalizeName(ent.name);
- const aliases = uniq([norm, ...(ent.aliases ?? []).map(normalizeName)]);
+ const aliases = unionSet([norm], (ent.aliases ?? []).map(normalizeName));
- const match = findMatch(store, candidatePool, norm, aliases, embedding, mergeThreshold);
+ const match = await findMatch(store, candidatePool, norm, aliases, embedding, mergeThreshold);
if (match) {
- match.summary = betterSummary(match.summary, ent.summary);
- match.aliases = uniq([...match.aliases, ...aliases, norm]);
- match.confidence = reinforce(match.confidence);
- match.observations += 1;
- match.provenance = uniq([...match.provenance, provenance]);
+ const summary = betterSummary(match.summary, ent.summary);
+ if (summary !== match.summary) {
+ match.summary = summary;
+ match.summaryUpdatedAt = now;
+ }
+ match.aliases = unionSet(match.aliases, [...aliases, norm]);
+ bumpObservation(match.observationSources, observationKey);
+ match.observations = totalObservations(match.observationSources);
+ match.confidence = confidenceFromObservations(match.observations);
+ match.provenance = unionSet(match.provenance, [provenance]);
match.embedding = embedding;
match.updatedAt = now;
- store.upsertNode(match);
+ await store.upsertNode(match);
result.nameToId.set(norm, match.id);
result.nodesUpdated += 1;
} else {
@@ -93,15 +106,17 @@ export async function mergeExtraction(
name: ent.name,
type: ent.type,
summary: ent.summary,
+ summaryUpdatedAt: now,
aliases,
- confidence: 0.6,
+ confidence: confidenceFromObservations(1),
observations: 1,
+ observationSources: { [observationKey]: 1 },
embedding,
provenance: [provenance],
createdAt: now,
updatedAt: now,
};
- store.upsertNode(node);
+ await store.upsertNode(node);
candidatePool.push(node); // allow intra-batch dedup
result.nameToId.set(norm, node.id);
result.nodesCreated += 1;
@@ -116,14 +131,19 @@ export async function mergeExtraction(
if (!sourceId || !targetId || sourceId === targetId) continue;
const relation = normalizeName(rel.relation).replace(/\s+/g, "_");
- const existing = store.findEdge(sourceId, targetId, relation);
+ const existing = await store.findEdge(sourceId, targetId, relation);
if (existing) {
- existing.description = betterSummary(existing.description, rel.description ?? "");
- existing.confidence = reinforce(existing.confidence);
- existing.observations += 1;
- existing.provenance = uniq([...existing.provenance, provenance]);
+ const description = betterSummary(existing.description, rel.description ?? "");
+ if (description !== existing.description) {
+ existing.description = description;
+ existing.descriptionUpdatedAt = now;
+ }
+ bumpObservation(existing.observationSources, observationKey);
+ existing.observations = totalObservations(existing.observationSources);
+ existing.confidence = confidenceFromObservations(existing.observations);
+ existing.provenance = unionSet(existing.provenance, [provenance]);
existing.updatedAt = now;
- store.upsertEdge(existing);
+ await store.upsertEdge(existing);
result.edgesUpdated += 1;
} else {
const edge: GraphEdge = {
@@ -132,13 +152,15 @@ export async function mergeExtraction(
targetId,
relation,
description: rel.description ?? "",
- confidence: 0.6,
+ descriptionUpdatedAt: now,
+ confidence: confidenceFromObservations(1),
observations: 1,
+ observationSources: { [observationKey]: 1 },
provenance: [provenance],
createdAt: now,
updatedAt: now,
};
- store.upsertEdge(edge);
+ await store.upsertEdge(edge);
result.edgesCreated += 1;
}
}
@@ -147,16 +169,16 @@ export async function mergeExtraction(
}
/** Find the best existing node to merge an incoming entity into, or undefined. */
-function findMatch(
+async function findMatch(
store: GraphStore,
candidatePool: GraphNode[],
norm: string,
aliases: string[],
embedding: number[],
threshold: number,
-): GraphNode | undefined {
+): Promise<GraphNode | undefined> {
// 1) Exact name/alias hit is authoritative.
- const byName = store.findNodesByName(norm);
+ const byName = await store.findNodesByName(norm);
if (byName.length > 0) {
// Prefer the most-observed among exact matches.
return byName.sort((a, b) => b.observations - a.observations)[0];
@@ -177,3 +199,70 @@ function findMatch(
}
return best;
}
+
+/**
+ * Reconcile two *full node records* for the same logical entity — the CRDT merge
+ * used when importing a serialized graph from a teammate (Mode A) or a replica.
+ *
+ * Unlike {@link mergeExtraction}, this does not add a new observation: it takes
+ * the element-wise **max** of the two observation counters (idempotent — a
+ * re-import is a no-op), unions the grow-only sets, and resolves free text by
+ * last-writer-wins. The result is independent of merge order.
+ */
+export function mergeNodeRecords(existing: GraphNode, incoming: GraphNode): GraphNode {
+ const observationSources = mergeObservationCounters(
+ existing.observationSources,
+ incoming.observationSources,
+ );
+ const observations = totalObservations(observationSources);
+ const summary = pickLatestText(
+ { text: existing.summary, at: existing.summaryUpdatedAt },
+ { text: incoming.summary, at: incoming.summaryUpdatedAt },
+ );
+ return {
+ ...existing,
+ // Keep the canonical name/type of the record we already have, but retain
+ // both surface forms as aliases so either resolves to this node.
+ aliases: unionSet(existing.aliases, incoming.aliases),
+ summary: summary.text,
+ summaryUpdatedAt: summary.at,
+ confidence: confidenceFromObservations(observations),
+ observations,
+ observationSources,
+ provenance: unionSet(existing.provenance, incoming.provenance),
+ embedding: existing.embedding ?? incoming.embedding,
+ createdAt: minStr(existing.createdAt, incoming.createdAt),
+ updatedAt: maxStr(existing.updatedAt, incoming.updatedAt),
+ };
+}
+
+/** CRDT reconciliation of two edge records for the same (source, target, relation). */
+export function mergeEdgeRecords(existing: GraphEdge, incoming: GraphEdge): GraphEdge {
+ const observationSources = mergeObservationCounters(
+ existing.observationSources,
+ incoming.observationSources,
+ );
+ const observations = totalObservations(observationSources);
+ const description = pickLatestText(
+ { text: existing.description, at: existing.descriptionUpdatedAt },
+ { text: incoming.description, at: incoming.descriptionUpdatedAt },
+ );
+ return {
+ ...existing,
+ description: description.text,
+ descriptionUpdatedAt: description.at,
+ confidence: confidenceFromObservations(observations),
+ observations,
+ observationSources,
+ provenance: unionSet(existing.provenance, incoming.provenance),
+ createdAt: minStr(existing.createdAt, incoming.createdAt),
+ updatedAt: maxStr(existing.updatedAt, incoming.updatedAt),
+ };
+}
+
+function minStr(a: string, b: string): string {
+ return a <= b ? a : b;
+}
+function maxStr(a: string, b: string): string {
+ return a >= b ? a : b;
+}
diff --git a/src/graph/serialize.ts b/src/graph/serialize.ts
new file mode 100644
index 00000000..f377b58e
--- /dev/null
+++ b/src/graph/serialize.ts
@@ -0,0 +1,231 @@
+/**
+ * Git-native team sync (Mode A): serialize the whole graph to a single
+ * human-diffable JSONL file that a team commits to their repo, and import it
+ * back with a conflict-free re-merge.
+ *
+ * The file is line-delimited JSON — one record per line, in FK-safe order
+ * (meta, documents, nodes, edges, chunks) — so `git diff` shows exactly which
+ * facts changed. Importing runs the CRDT record-merge ({@link mergeNodeRecords}
+ * / {@link mergeEdgeRecords}), which is idempotent: pulling the same file twice
+ * changes nothing, and two teammates' additions converge regardless of order.
+ */
+import type { GraphStore } from "./store.js";
+import type { Chunk, GraphDocument, GraphEdge, GraphNode } from "./types.js";
+import { mergeEdgeRecords, mergeNodeRecords } from "./merge.js";
+import { normalizeName } from "../util/id.js";
+
+/** Current on-disk format version. Bumped if the record shape changes. */
+export const GRAPH_FILE_FORMAT = 1;
+
+interface GraphFileMeta {
+ kind: "meta";
+ format: number;
+ /** Embedding dimensionality of the graph that produced this file. */
+ embeddingDimensions: number;
+ /** Optional embedding-model label, to warn on cross-model drift. */
+ embeddingModel?: string;
+ exportedAt: string;
+ counts: { documents: number; nodes: number; edges: number; chunks: number };
+}
+
+export interface SerializeOptions {
+ embeddingDimensions: number;
+ embeddingModel?: string;
+ /** Decimal places to round embeddings to (smaller = tidier git diffs). Default 6. */
+ precision?: number;
+}
+
+export interface ImportResult {
+ nodesCreated: number;
+ nodesUpdated: number;
+ edgesCreated: number;
+ edgesUpdated: number;
+ documentsAdded: number;
+ chunksAdded: number;
+ /** Non-fatal problems (e.g. embedding-model mismatch), for the caller to surface. */
+ warnings: string[];
+}
+
+function round(vec: number[] | undefined, precision: number): number[] | undefined {
+ if (!vec) return undefined;
+ const f = 10 ** precision;
+ return vec.map((x) => Math.round(x * f) / f);
+}
+
+/** Serialize the entire graph to a JSONL string. */
+export async function serializeGraph(
+ store: GraphStore,
+ opts: SerializeOptions,
+): Promise<string> {
+ const precision = opts.precision ?? 6;
+ const [documents, nodes, edges, chunks] = await Promise.all([
+ store.allDocuments(),
+ store.allNodes(),
+ store.allEdges(),
+ store.allChunks(),
+ ]);
+
+ const meta: GraphFileMeta = {
+ kind: "meta",
+ format: GRAPH_FILE_FORMAT,
+ embeddingDimensions: opts.embeddingDimensions,
+ embeddingModel: opts.embeddingModel,
+ exportedAt: new Date().toISOString(),
+ counts: {
+ documents: documents.length,
+ nodes: nodes.length,
+ edges: edges.length,
+ chunks: chunks.length,
+ },
+ };
+
+ const lines: string[] = [JSON.stringify(meta)];
+ // Deterministic ordering keeps diffs stable across exports.
+ for (const d of [...documents].sort(byId)) lines.push(JSON.stringify({ kind: "document", ...d }));
+ for (const n of [...nodes].sort(byId)) {
+ lines.push(JSON.stringify({ kind: "node", ...n, embedding: round(n.embedding, precision) }));
+ }
+ for (const e of [...edges].sort(byId)) lines.push(JSON.stringify({ kind: "edge", ...e }));
+ for (const c of [...chunks].sort(byId)) {
+ lines.push(JSON.stringify({ kind: "chunk", ...c, embedding: round(c.embedding, precision) }));
+ }
+ return lines.join("\n") + "\n";
+}
+
+function byId(a: { id: string }, b: { id: string }): number {
+ return a.id < b.id ? -1 : a.id > b.id ? 1 : 0;
+}
+
+/**
+ * Import a serialized graph into a store, CRDT-merging every record. Idempotent:
+ * importing the same content twice is a no-op.
+ *
+ * @param embeddingDimensions The importing graph's embedding size; a mismatch
+ * means the incoming vectors aren't comparable, so they're dropped (a warning
+ * is returned) rather than corrupting retrieval.
+ */
+export async function importGraph(
+ store: GraphStore,
+ jsonl: string,
+ embeddingDimensions: number,
+): Promise<ImportResult> {
+ const result: ImportResult = {
+ nodesCreated: 0,
+ nodesUpdated: 0,
+ edgesCreated: 0,
+ edgesUpdated: 0,
+ documentsAdded: 0,
+ chunksAdded: 0,
+ warnings: [],
+ };
+
+ const documents: GraphDocument[] = [];
+ const nodes: GraphNode[] = [];
+ const edges: GraphEdge[] = [];
+ const chunks: Chunk[] = [];
+ let dropEmbeddings = false;
+
+ for (const raw of jsonl.split("\n")) {
+ const line = raw.trim();
+ if (!line) continue;
+ const rec = JSON.parse(line) as { kind: string } & Record<string, unknown>;
+ switch (rec.kind) {
+ case "meta": {
+ const meta = rec as unknown as GraphFileMeta;
+ if (meta.embeddingDimensions && meta.embeddingDimensions !== embeddingDimensions) {
+ dropEmbeddings = true;
+ result.warnings.push(
+ `Embedding dimension mismatch (file ${meta.embeddingDimensions} vs local ${embeddingDimensions}` +
+ (meta.embeddingModel ? `, model "${meta.embeddingModel}"` : "") +
+ `). Imported facts kept; their embeddings were dropped, so re-embed by re-ingesting sources.`,
+ );
+ }
+ break;
+ }
+ case "document":
+ documents.push(stripKind(rec) as unknown as GraphDocument);
+ break;
+ case "node":
+ nodes.push(stripKind(rec) as unknown as GraphNode);
+ break;
+ case "edge":
+ edges.push(stripKind(rec) as unknown as GraphEdge);
+ break;
+ case "chunk":
+ chunks.push(stripKind(rec) as unknown as Chunk);
+ break;
+ }
+ }
+
+ // Documents (dedup by content hash; remember any id remap for chunks).
+ const docIdMap = new Map<string, string>();
+ for (const doc of documents) {
+ const existing = await store.getDocumentByHash(doc.hash);
+ if (existing) {
+ docIdMap.set(doc.id, existing.id);
+ } else {
+ await store.insertDocument(doc);
+ docIdMap.set(doc.id, doc.id);
+ result.documentsAdded += 1;
+ }
+ }
+
+ // Nodes (resolve identity by id, then by name; remember id remap for edges).
+ const nodeIdMap = new Map<string, string>();
+ for (const incoming of nodes) {
+ if (dropEmbeddings) incoming.embedding = undefined;
+ const byId = await store.getNodeById(incoming.id);
+ if (byId) {
+ await store.upsertNode(mergeNodeRecords(byId, incoming));
+ nodeIdMap.set(incoming.id, byId.id);
+ result.nodesUpdated += 1;
+ continue;
+ }
+ const byName = (await store.findNodesByName(normalizeName(incoming.name))).sort(
+ (a, b) => b.observations - a.observations,
+ )[0];
+ if (byName) {
+ await store.upsertNode(mergeNodeRecords(byName, incoming));
+ nodeIdMap.set(incoming.id, byName.id);
+ result.nodesUpdated += 1;
+ } else {
+ await store.upsertNode(incoming);
+ nodeIdMap.set(incoming.id, incoming.id);
+ result.nodesCreated += 1;
+ }
+ }
+
+ // Edges (remap endpoints, then merge on the (source, target, relation) key).
+ for (const incoming of edges) {
+ const sourceId = nodeIdMap.get(incoming.sourceId) ?? incoming.sourceId;
+ const targetId = nodeIdMap.get(incoming.targetId) ?? incoming.targetId;
+ const mapped: GraphEdge = { ...incoming, sourceId, targetId };
+ const existing = await store.findEdge(sourceId, targetId, incoming.relation);
+ if (existing) {
+ await store.upsertEdge(mergeEdgeRecords(existing, mapped));
+ result.edgesUpdated += 1;
+ } else {
+ await store.upsertEdge(mapped);
+ result.edgesCreated += 1;
+ }
+ }
+
+ // Chunks (grow-only; skip ones we already have or whose document is absent).
+ for (const incoming of chunks) {
+ if (dropEmbeddings) incoming.embedding = undefined;
+ const documentId = docIdMap.get(incoming.documentId) ?? incoming.documentId;
+ if (!(await store.getDocumentById(documentId))) continue;
+ const present = await store.getChunksByIds([incoming.id]);
+ if (present.length > 0) continue;
+ await store.insertChunk({ ...incoming, documentId });
+ result.chunksAdded += 1;
+ }
+
+ return result;
+}
+
+function stripKind(rec: Record<string, unknown>): Record<string, unknown> {
+ const { kind, ...rest } = rec;
+ void kind;
+ return rest;
+}
diff --git a/src/graph/sqlite-store.ts b/src/graph/sqlite-store.ts
index a0cd53e1..09d805fc 100644
--- a/src/graph/sqlite-store.ts
+++ b/src/graph/sqlite-store.ts
@@ -8,8 +8,10 @@ import type {
GraphEdge,
GraphNode,
GraphStats,
+ ObservationCounter,
} from "./types.js";
import { normalizeName } from "../util/id.js";
+import { totalObservations } from "./crdt.js";
interface NodeRow {
id: string;
@@ -17,9 +19,11 @@ interface NodeRow {
norm_name: string;
type: string;
summary: string;
+ summary_updated_at: string | null;
aliases: string;
confidence: number;
observations: number;
+ observation_sources: string | null;
embedding: string | null;
provenance: string;
created_at: string;
@@ -32,8 +36,10 @@ interface EdgeRow {
target_id: string;
relation: string;
description: string;
+ description_updated_at: string | null;
confidence: number;
observations: number;
+ observation_sources: string | null;
provenance: string;
created_at: string;
updated_at: string;
@@ -48,7 +54,13 @@ interface ChunkRow {
created_at: string;
}
-/** File-backed graph store using SQLite via better-sqlite3. */
+/**
+ * File-backed graph store using SQLite via better-sqlite3.
+ *
+ * better-sqlite3 is synchronous; the async {@link GraphStore} methods here just
+ * wrap those synchronous calls and resolve immediately, so the local path stays
+ * as fast as before while conforming to the interface a networked store needs.
+ */
export class SqliteStore implements GraphStore {
private db: Database.Database;
@@ -83,78 +95,98 @@ export class SqliteStore implements GraphStore {
);
CREATE TABLE IF NOT EXISTS nodes (
- id TEXT PRIMARY KEY,
- name TEXT NOT NULL,
- norm_name TEXT NOT NULL,
- type TEXT NOT NULL,
- summary TEXT NOT NULL,
- aliases TEXT NOT NULL DEFAULT '[]',
- confidence REAL NOT NULL DEFAULT 0.5,
- observations INTEGER NOT NULL DEFAULT 1,
- embedding TEXT,
- provenance TEXT NOT NULL DEFAULT '[]',
- created_at TEXT NOT NULL,
- updated_at TEXT NOT NULL
+ id TEXT PRIMARY KEY,
+ name TEXT NOT NULL,
+ norm_name TEXT NOT NULL,
+ type TEXT NOT NULL,
+ summary TEXT NOT NULL,
+ summary_updated_at TEXT,
+ aliases TEXT NOT NULL DEFAULT '[]',
+ confidence REAL NOT NULL DEFAULT 0.5,
+ observations INTEGER NOT NULL DEFAULT 1,
+ observation_sources TEXT NOT NULL DEFAULT '{}',
+ embedding TEXT,
+ provenance TEXT NOT NULL DEFAULT '[]',
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_nodes_norm_name ON nodes(norm_name);
CREATE TABLE IF NOT EXISTS edges (
- id TEXT PRIMARY KEY,
- source_id TEXT NOT NULL REFERENCES nodes(id) ON DELETE CASCADE,
- target_id TEXT NOT NULL REFERENCES nodes(id) ON DELETE CASCADE,
- relation TEXT NOT NULL,
- description TEXT NOT NULL DEFAULT '',
- confidence REAL NOT NULL DEFAULT 0.5,
- observations INTEGER NOT NULL DEFAULT 1,
- provenance TEXT NOT NULL DEFAULT '[]',
- created_at TEXT NOT NULL,
- updated_at TEXT NOT NULL,
+ id TEXT PRIMARY KEY,
+ source_id TEXT NOT NULL REFERENCES nodes(id) ON DELETE CASCADE,
+ target_id TEXT NOT NULL REFERENCES nodes(id) ON DELETE CASCADE,
+ relation TEXT NOT NULL,
+ description TEXT NOT NULL DEFAULT '',
+ description_updated_at TEXT,
+ confidence REAL NOT NULL DEFAULT 0.5,
+ observations INTEGER NOT NULL DEFAULT 1,
+ observation_sources TEXT NOT NULL DEFAULT '{}',
+ provenance TEXT NOT NULL DEFAULT '[]',
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
UNIQUE(source_id, target_id, relation)
);
CREATE INDEX IF NOT EXISTS idx_edges_source ON edges(source_id);
CREATE INDEX IF NOT EXISTS idx_edges_target ON edges(target_id);
`);
+ // Additive migrations for graphs created before the CRDT columns existed.
+ this.addColumnIfMissing("nodes", "summary_updated_at", "TEXT");
+ this.addColumnIfMissing("nodes", "observation_sources", "TEXT NOT NULL DEFAULT '{}'");
+ this.addColumnIfMissing("edges", "description_updated_at", "TEXT");
+ this.addColumnIfMissing("edges", "observation_sources", "TEXT NOT NULL DEFAULT '{}'");
+ }
+
+ private addColumnIfMissing(table: string, column: string, ddl: string): void {
+ const cols = this.db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>;
+ if (!cols.some((c) => c.name === column)) {
+ this.db.exec(`ALTER TABLE ${table} ADD COLUMN ${column} ${ddl}`);
+ }
}
// --- documents ---
- getDocumentByHash(hash: string): GraphDocument | undefined {
- const row = this.db
- .prepare(`SELECT * FROM documents WHERE hash = ?`)
- .get(hash) as (GraphDocument & { created_at: string }) | undefined;
- if (!row) return undefined;
- return {
- id: row.id,
- title: row.title,
- source: row.source,
- hash: row.hash,
- createdAt: (row as unknown as { created_at: string }).created_at,
- };
+ async getDocumentByHash(hash: string): Promise<GraphDocument | undefined> {
+ const row = this.db.prepare(`SELECT * FROM documents WHERE hash = ?`).get(hash) as
+ | (GraphDocument & { created_at: string })
+ | undefined;
+ return row ? this.rowToDocument(row as unknown as Record<string, unknown>) : undefined;
+ }
+
+ async getDocumentById(id: string): Promise<GraphDocument | undefined> {
+ const row = this.db.prepare(`SELECT * FROM documents WHERE id = ?`).get(id) as
+ | Record<string, unknown>
+ | undefined;
+ return row ? this.rowToDocument(row) : undefined;
}
- insertDocument(doc: GraphDocument): void {
+ async insertDocument(doc: GraphDocument): Promise<void> {
this.db
.prepare(
- `INSERT INTO documents (id, title, source, hash, created_at)
+ `INSERT OR IGNORE INTO documents (id, title, source, hash, created_at)
VALUES (@id, @title, @source, @hash, @createdAt)`,
)
.run(doc);
}
- allDocuments(): GraphDocument[] {
+ async allDocuments(): Promise<GraphDocument[]> {
const rows = this.db
- .prepare(`SELECT id, title, source, hash, created_at FROM documents ORDER BY created_at DESC`)
- .all() as Array<{ id: string; title: string; source: string; hash: string; created_at: string }>;
- return rows.map((r) => ({
- id: r.id,
- title: r.title,
- source: r.source,
- hash: r.hash,
- createdAt: r.created_at,
- }));
+ .prepare(`SELECT * FROM documents ORDER BY created_at DESC`)
+ .all() as Array<Record<string, unknown>>;
+ return rows.map((r) => this.rowToDocument(r));
}
- documentTitle(documentId: string): string {
+ private rowToDocument(r: Record<string, unknown>): GraphDocument {
+ return {
+ id: r.id as string,
+ title: r.title as string,
+ source: r.source as string,
+ hash: r.hash as string,
+ createdAt: r.created_at as string,
+ };
+ }
+
+ async documentTitle(documentId: string): Promise<string> {
const row = this.db
.prepare(`SELECT title FROM documents WHERE id = ?`)
.get(documentId) as { title: string } | undefined;
@@ -163,10 +195,10 @@ export class SqliteStore implements GraphStore {
// --- chunks ---
- insertChunk(chunk: Chunk): void {
+ async insertChunk(chunk: Chunk): Promise<void> {
this.db
.prepare(
- `INSERT INTO chunks (id, document_id, ordinal, text, embedding, created_at)
+ `INSERT OR IGNORE INTO chunks (id, document_id, ordinal, text, embedding, created_at)
VALUES (@id, @documentId, @ordinal, @text, @embedding, @createdAt)`,
)
.run({
@@ -175,7 +207,7 @@ export class SqliteStore implements GraphStore {
});
}
- getChunksByIds(ids: string[]): Chunk[] {
+ async getChunksByIds(ids: string[]): Promise<Chunk[]> {
if (ids.length === 0) return [];
const placeholders = ids.map(() => "?").join(",");
const rows = this.db
@@ -184,13 +216,18 @@ export class SqliteStore implements GraphStore {
return rows.map(this.rowToChunk);
}
- allEmbeddedChunks(): Chunk[] {
+ async allEmbeddedChunks(): Promise<Chunk[]> {
const rows = this.db
.prepare(`SELECT * FROM chunks WHERE embedding IS NOT NULL`)
.all() as ChunkRow[];
return rows.map(this.rowToChunk);
}
+ async allChunks(): Promise<Chunk[]> {
+ const rows = this.db.prepare(`SELECT * FROM chunks`).all() as ChunkRow[];
+ return rows.map(this.rowToChunk);
+ }
+
private rowToChunk(row: ChunkRow): Chunk {
return {
id: row.id,
@@ -204,14 +241,14 @@ export class SqliteStore implements GraphStore {
// --- nodes ---
- getNodeById(id: string): GraphNode | undefined {
+ async getNodeById(id: string): Promise<GraphNode | undefined> {
const row = this.db.prepare(`SELECT * FROM nodes WHERE id = ?`).get(id) as
| NodeRow
| undefined;
return row ? this.rowToNode(row) : undefined;
}
- getNodesByIds(ids: string[]): GraphNode[] {
+ async getNodesByIds(ids: string[]): Promise<GraphNode[]> {
if (ids.length === 0) return [];
const placeholders = ids.map(() => "?").join(",");
const rows = this.db
@@ -220,7 +257,7 @@ export class SqliteStore implements GraphStore {
return rows.map((r) => this.rowToNode(r));
}
- findNodesByName(normalizedName: string): GraphNode[] {
+ async findNodesByName(normalizedName: string): Promise<GraphNode[]> {
// Match either the canonical normalized name or any alias. Aliases are
// stored normalized in a JSON array, so we match the quoted element form
// (e.g. `"oauth"`) to avoid partial-substring false positives.
@@ -234,35 +271,37 @@ export class SqliteStore implements GraphStore {
return rows.map((r) => this.rowToNode(r));
}
- allEmbeddedNodes(): GraphNode[] {
+ async allEmbeddedNodes(): Promise<GraphNode[]> {
const rows = this.db
.prepare(`SELECT * FROM nodes WHERE embedding IS NOT NULL`)
.all() as NodeRow[];
return rows.map((r) => this.rowToNode(r));
}
- allNodes(): GraphNode[] {
+ async allNodes(): Promise<GraphNode[]> {
const rows = this.db.prepare(`SELECT * FROM nodes`).all() as NodeRow[];
return rows.map((r) => this.rowToNode(r));
}
- upsertNode(node: GraphNode): void {
+ async upsertNode(node: GraphNode): Promise<void> {
this.db
.prepare(
`INSERT INTO nodes
- (id, name, norm_name, type, summary, aliases, confidence,
- observations, embedding, provenance, created_at, updated_at)
+ (id, name, norm_name, type, summary, summary_updated_at, aliases, confidence,
+ observations, observation_sources, embedding, provenance, created_at, updated_at)
VALUES
- (@id, @name, @norm_name, @type, @summary, @aliases, @confidence,
- @observations, @embedding, @provenance, @created_at, @updated_at)
+ (@id, @name, @norm_name, @type, @summary, @summary_updated_at, @aliases, @confidence,
+ @observations, @observation_sources, @embedding, @provenance, @created_at, @updated_at)
ON CONFLICT(id) DO UPDATE SET
name = excluded.name,
norm_name = excluded.norm_name,
type = excluded.type,
summary = excluded.summary,
+ summary_updated_at = excluded.summary_updated_at,
aliases = excluded.aliases,
confidence = excluded.confidence,
observations = excluded.observations,
+ observation_sources = excluded.observation_sources,
embedding = excluded.embedding,
provenance = excluded.provenance,
updated_at = excluded.updated_at`,
@@ -273,9 +312,11 @@ export class SqliteStore implements GraphStore {
norm_name: normalizeName(node.name),
type: node.type,
summary: node.summary,
+ summary_updated_at: node.summaryUpdatedAt,
aliases: JSON.stringify(node.aliases),
confidence: node.confidence,
observations: node.observations,
+ observation_sources: JSON.stringify(node.observationSources),
embedding: node.embedding ? JSON.stringify(node.embedding) : null,
provenance: JSON.stringify(node.provenance),
created_at: node.createdAt,
@@ -284,14 +325,17 @@ export class SqliteStore implements GraphStore {
}
private rowToNode(row: NodeRow): GraphNode {
+ const observationSources = parseCounter(row.observation_sources, row.observations);
return {
id: row.id,
name: row.name,
type: row.type,
summary: row.summary,
+ summaryUpdatedAt: row.summary_updated_at ?? row.updated_at,
aliases: JSON.parse(row.aliases),
confidence: row.confidence,
- observations: row.observations,
+ observations: totalObservations(observationSources),
+ observationSources,
embedding: row.embedding ? JSON.parse(row.embedding) : undefined,
provenance: JSON.parse(row.provenance),
createdAt: row.created_at,
@@ -301,7 +345,11 @@ export class SqliteStore implements GraphStore {
// --- edges ---
- findEdge(sourceId: string, targetId: string, relation: string): GraphEdge | undefined {
+ async findEdge(
+ sourceId: string,
+ targetId: string,
+ relation: string,
+ ): Promise<GraphEdge | undefined> {
const row = this.db
.prepare(
`SELECT * FROM edges WHERE source_id = ? AND target_id = ? AND relation = ?`,
@@ -310,19 +358,21 @@ export class SqliteStore implements GraphStore {
return row ? this.rowToEdge(row) : undefined;
}
- upsertEdge(edge: GraphEdge): void {
+ async upsertEdge(edge: GraphEdge): Promise<void> {
this.db
.prepare(
`INSERT INTO edges
- (id, source_id, target_id, relation, description, confidence,
- observations, provenance, created_at, updated_at)
+ (id, source_id, target_id, relation, description, description_updated_at, confidence,
+ observations, observation_sources, provenance, created_at, updated_at)
VALUES
- (@id, @source_id, @target_id, @relation, @description, @confidence,
- @observations, @provenance, @created_at, @updated_at)
+ (@id, @source_id, @target_id, @relation, @description, @description_updated_at, @confidence,
+ @observations, @observation_sources, @provenance, @created_at, @updated_at)
ON CONFLICT(source_id, target_id, relation) DO UPDATE SET
description = excluded.description,
+ description_updated_at = excluded.description_updated_at,
confidence = excluded.confidence,
observations = excluded.observations,
+ observation_sources = excluded.observation_sources,
provenance = excluded.provenance,
updated_at = excluded.updated_at`,
)
@@ -332,15 +382,17 @@ export class SqliteStore implements GraphStore {
target_id: edge.targetId,
relation: edge.relation,
description: edge.description,
+ description_updated_at: edge.descriptionUpdatedAt,
confidence: edge.confidence,
observations: edge.observations,
+ observation_sources: JSON.stringify(edge.observationSources),
provenance: JSON.stringify(edge.provenance),
created_at: edge.createdAt,
updated_at: edge.updatedAt,
});
}
- edgesForNodes(nodeIds: string[]): GraphEdge[] {
+ async edgesForNodes(nodeIds: string[]): Promise<GraphEdge[]> {
if (nodeIds.length === 0) return [];
const placeholders = nodeIds.map(() => "?").join(",");
const rows = this.db
@@ -352,20 +404,23 @@ export class SqliteStore implements GraphStore {
return rows.map((r) => this.rowToEdge(r));
}
- allEdges(): GraphEdge[] {
+ async allEdges(): Promise<GraphEdge[]> {
const rows = this.db.prepare(`SELECT * FROM edges`).all() as EdgeRow[];
return rows.map((r) => this.rowToEdge(r));
}
private rowToEdge(row: EdgeRow): GraphEdge {
+ const observationSources = parseCounter(row.observation_sources, row.observations);
return {
id: row.id,
sourceId: row.source_id,
targetId: row.target_id,
relation: row.relation,
description: row.description,
+ descriptionUpdatedAt: row.description_updated_at ?? row.updated_at,
confidence: row.confidence,
- observations: row.observations,
+ observations: totalObservations(observationSources),
+ observationSources,
provenance: JSON.parse(row.provenance),
createdAt: row.created_at,
updatedAt: row.updated_at,
@@ -374,7 +429,7 @@ export class SqliteStore implements GraphStore {
// --- misc ---
- stats(): GraphStats {
+ async stats(): Promise<GraphStats> {
const count = (table: string): number =>
(this.db.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get() as { n: number }).n;
return {
@@ -385,7 +440,20 @@ export class SqliteStore implements GraphStore {
};
}
- close(): void {
+ async close(): Promise<void> {
this.db.close();
}
}
+
+/**
+ * Parse a stored observation counter. Graphs created before the CRDT columns
+ * existed have an empty counter but a real scalar `observations`; seed a single
+ * "legacy" bucket so their totals survive the migration.
+ */
+function parseCounter(raw: string | null, legacyObservations: number): ObservationCounter {
+ const parsed = raw ? (JSON.parse(raw) as ObservationCounter) : {};
+ if (Object.keys(parsed).length === 0 && legacyObservations > 0) {
+ return { legacy: legacyObservations };
+ }
+ return parsed;
+}
diff --git a/src/graph/store.ts b/src/graph/store.ts
index a612cdda..72cc0309 100644
--- a/src/graph/store.ts
+++ b/src/graph/store.ts
@@ -10,44 +10,52 @@ import type {
* Storage backend for the context graph.
*
* The engine talks to the graph exclusively through this interface, so the
- * SQLite backend can be swapped for another store (Postgres/pgvector, an
- * in-memory store, a remote service) without touching engine logic.
+ * default local SQLite backend can be swapped for another store — a
+ * Postgres/pgvector service, a remote replica, or an in-memory store — without
+ * touching engine logic.
+ *
+ * All methods are **async**. The local `better-sqlite3` store is synchronous
+ * under the hood and simply resolves immediately; a networked store needs the
+ * async surface to talk to its remote.
*/
export interface GraphStore {
// --- documents ---
- getDocumentByHash(hash: string): GraphDocument | undefined;
- insertDocument(doc: GraphDocument): void;
+ getDocumentByHash(hash: string): Promise<GraphDocument | undefined>;
+ getDocumentById(id: string): Promise<GraphDocument | undefined>;
+ insertDocument(doc: GraphDocument): Promise<void>;
/** Every ingested document, newest first (for UI listings). */
- allDocuments(): GraphDocument[];
+ allDocuments(): Promise<GraphDocument[]>;
// --- chunks ---
- insertChunk(chunk: Chunk): void;
- getChunksByIds(ids: string[]): Chunk[];
+ insertChunk(chunk: Chunk): Promise<void>;
+ getChunksByIds(ids: string[]): Promise<Chunk[]>;
/** All chunks that have an embedding, for in-memory vector search. */
- allEmbeddedChunks(): Chunk[];
+ allEmbeddedChunks(): Promise<Chunk[]>;
+ /** Every chunk in the graph (for serialization / export). */
+ allChunks(): Promise<Chunk[]>;
// --- nodes ---
- getNodeById(id: string): GraphNode | undefined;
- getNodesByIds(ids: string[]): GraphNode[];
+ getNodeById(id: string): Promise<GraphNode | undefined>;
+ getNodesByIds(ids: string[]): Promise<GraphNode[]>;
/** Nodes whose canonical name or an alias matches (normalized, exact). */
- findNodesByName(normalizedName: string): GraphNode[];
+ findNodesByName(normalizedName: string): Promise<GraphNode[]>;
/** All nodes that have an embedding, for in-memory vector search & dedup. */
- allEmbeddedNodes(): GraphNode[];
+ allEmbeddedNodes(): Promise<GraphNode[]>;
/** Every node in the graph (for export/visualization). */
- allNodes(): GraphNode[];
- upsertNode(node: GraphNode): void;
+ allNodes(): Promise<GraphNode[]>;
+ upsertNode(node: GraphNode): Promise<void>;
// --- edges ---
/** Existing edge with the same (source, target, relation), if any. */
- findEdge(sourceId: string, targetId: string, relation: string): GraphEdge | undefined;
- upsertEdge(edge: GraphEdge): void;
+ findEdge(sourceId: string, targetId: string, relation: string): Promise<GraphEdge | undefined>;
+ upsertEdge(edge: GraphEdge): Promise<void>;
/** All edges incident to any of the given node ids. */
- edgesForNodes(nodeIds: string[]): GraphEdge[];
+ edgesForNodes(nodeIds: string[]): Promise<GraphEdge[]>;
/** Every edge in the graph (for export/visualization). */
- allEdges(): GraphEdge[];
+ allEdges(): Promise<GraphEdge[]>;
// --- misc ---
- documentTitle(documentId: string): string;
- stats(): GraphStats;
- close(): void;
+ documentTitle(documentId: string): Promise<string>;
+ stats(): Promise<GraphStats>;
+ close(): Promise<void>;
}
diff --git a/src/graph/types.ts b/src/graph/types.ts
index 993341b2..09014412 100644
--- a/src/graph/types.ts
+++ b/src/graph/types.ts
@@ -23,6 +23,16 @@ export interface GraphDocument {
createdAt: string;
}
+/**
+ * A conflict-free observation counter: maps a *unique observation key* (the id
+ * of the document/contribution that observed the fact) to how many times that
+ * single observation event reinforced it. Because keys are globally unique per
+ * event, merging two counters is an element-wise **max** — commutative,
+ * associative, and idempotent. That is what makes re-importing the same records
+ * safe: replaying an observation never double-counts. See {@link ../graph/crdt}.
+ */
+export type ObservationCounter = Record<string, number>;
+
/** An entity or concept in the graph. */
export interface GraphNode {
id: string;
@@ -32,15 +42,30 @@ export interface GraphNode {
type: string;
/** A short natural-language description of the node, synthesized from evidence. */
summary: string;
+ /**
+ * Wall-clock stamp of when `summary` was last set. Used as a last-writer-wins
+ * tiebreaker when reconciling two replicas that hold different summaries.
+ */
+ summaryUpdatedAt: string;
/** Alternate names / surface forms that map to this node. */
aliases: string[];
- /** 0..1 confidence that this node is real and correctly described. */
+ /**
+ * 0..1 confidence that this node is real and correctly described. **Derived**
+ * (a pure function of the observation total), never mutated in place, so it
+ * converges deterministically no matter how records are merged.
+ */
confidence: number;
- /** How many times this entity has been observed across ingests/contributions. */
+ /**
+ * How many times this entity has been observed. **Derived** as the sum of
+ * {@link observationSources}; stored denormalized so existing readers keep
+ * working without recomputing.
+ */
observations: number;
+ /** The CRDT observation counter that {@link observations} is derived from. */
+ observationSources: ObservationCounter;
/** Vector embedding of `name + summary`, used for semantic matching & dedup. */
embedding?: number[];
- /** Where this knowledge came from (document ids, agent ids). */
+ /** Where this knowledge came from (document ids, agent ids) — a grow-only set. */
provenance: string[];
createdAt: string;
updatedAt: string;
@@ -55,8 +80,14 @@ export interface GraphEdge {
relation: string;
/** Optional natural-language description of the relationship. */
description: string;
+ /** Last-writer-wins stamp for `description` (see {@link GraphNode.summaryUpdatedAt}). */
+ descriptionUpdatedAt: string;
+ /** **Derived** from the observation total (see {@link GraphNode.confidence}). */
confidence: number;
+ /** **Derived** as the sum of {@link observationSources}. */
observations: number;
+ /** The CRDT observation counter that {@link observations} is derived from. */
+ observationSources: ObservationCounter;
provenance: string[];
createdAt: string;
updatedAt: string;
diff --git a/src/index.ts b/src/index.ts
index e101cfda..9dce2c89 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -37,7 +37,14 @@ export { retrieve, renderPrompt } from "./retrieval/retriever.js";
export type { GraphStore } from "./graph/store.js";
export { SqliteStore } from "./graph/sqlite-store.js";
-export { mergeExtraction } from "./graph/merge.js";
+export { mergeExtraction, mergeNodeRecords, mergeEdgeRecords } from "./graph/merge.js";
+export {
+ confidenceFromObservations,
+ mergeObservationCounters,
+ totalObservations,
+} from "./graph/crdt.js";
+export { serializeGraph, importGraph, GRAPH_FILE_FORMAT } from "./graph/serialize.js";
+export type { ImportResult, SerializeOptions } from "./graph/serialize.js";
export type { GraphExport } from "./graph/export.js";
export { buildGraphExport, toHtml, toMermaid } from "./graph/export.js";
@@ -46,6 +53,7 @@ export type {
GraphEdge,
GraphDocument,
Chunk,
+ ObservationCounter,
ContextBundle,
RetrievedNode,
RetrievedChunk,
diff --git a/src/mcp.ts b/src/mcp.ts
index dcd4a97c..9a066857 100644
--- a/src/mcp.ts
+++ b/src/mcp.ts
@@ -10,7 +10,7 @@ import "dotenv/config";
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
-import { writeFileSync } from "node:fs";
+import { writeFileSync, readFileSync, existsSync } from "node:fs";
import { ContextGraphEngine } from "./engine.js";
import { toHtml } from "./graph/export.js";
@@ -172,7 +172,7 @@ server.registerTool(
},
},
async ({ path }) => {
- const g = engine.exportGraph();
+ const g = await engine.exportGraph();
writeFileSync(path, toHtml(g));
return {
content: [
@@ -193,7 +193,7 @@ server.registerTool(
inputSchema: {},
},
async () => {
- const s = engine.stats();
+ const s = await engine.stats();
return {
content: [
{
@@ -205,6 +205,42 @@ server.registerTool(
},
);
+server.registerTool(
+ "context_sync",
+ {
+ title: "Sync the graph (git mode)",
+ description:
+ "Team sync (Mode A): import + re-merge the committed graph file, then write the merged graph back so it can be committed. Idempotent — safe to run repeatedly. Uses graph.jsonl next to the db unless a path is given.",
+ inputSchema: {
+ file: z
+ .string()
+ .optional()
+ .describe("Path to the graph JSONL file (default: graph.jsonl next to the db)."),
+ },
+ },
+ async ({ file }) => {
+ const path = file ?? engine.graphFilePath;
+ if (!path) {
+ return {
+ content: [{ type: "text", text: "This graph is in-memory and can't be synced to a file." }],
+ };
+ }
+ const lines: string[] = [];
+ if (existsSync(path)) {
+ const r = await engine.importJsonl(readFileSync(path, "utf8"));
+ for (const w of r.warnings) lines.push(`⚠ ${w}`);
+ lines.push(
+ `Merged in: +${r.nodesCreated} entities (${r.nodesUpdated} reinforced), ` +
+ `+${r.edgesCreated} relationships (${r.edgesUpdated} reinforced).`,
+ );
+ }
+ writeFileSync(path, await engine.exportJsonl());
+ const s = await engine.stats();
+ lines.push(`Wrote merged graph (${s.nodes} entities, ${s.edges} relationships) to ${path}.`);
+ return { content: [{ type: "text", text: lines.join("\n") }] };
+ },
+);
+
async function main(): Promise<void> {
const transport = new StdioServerTransport();
await server.connect(transport);
diff --git a/src/retrieval/retriever.ts b/src/retrieval/retriever.ts
index 017f69fd..dcb79713 100644
--- a/src/retrieval/retriever.ts
+++ b/src/retrieval/retriever.ts
@@ -39,7 +39,7 @@ export async function retrieve(
const [queryVec] = await embedder.embed([query]);
// 1) Direct semantic matches over entities.
- const nodeHits = topK(queryVec, store.allEmbeddedNodes(), (n) => n.embedding, maxNodes)
+ const nodeHits = topK(queryVec, await store.allEmbeddedNodes(), (n) => n.embedding, maxNodes)
.filter((h) => h.score >= minNodeScore);
const nodesById = new Map<string, RetrievedNode>();
@@ -51,14 +51,14 @@ export async function retrieve(
let edges: GraphEdge[] = [];
if (expand && nodesById.size > 0) {
const seedIds = [...nodesById.keys()];
- edges = store.edgesForNodes(seedIds);
+ edges = await store.edgesForNodes(seedIds);
const neighborIds = new Set<string>();
for (const e of edges) {
if (!nodesById.has(e.sourceId)) neighborIds.add(e.sourceId);
if (!nodesById.has(e.targetId)) neighborIds.add(e.targetId);
}
- const neighbors = store.getNodesByIds([...neighborIds]);
+ const neighbors = await store.getNodesByIds([...neighborIds]);
const seedScore = new Map(nodeHits.map((h) => [h.item.id, h.score]));
for (const n of neighbors) {
// Score an expanded node by the best edge that reached it.
@@ -73,12 +73,14 @@ export async function retrieve(
edges = edges.filter((e) => nodeIdSet.has(e.sourceId) && nodeIdSet.has(e.targetId));
// 3) Supporting source passages.
- const chunkHits = topK(queryVec, store.allEmbeddedChunks(), (c) => c.embedding, maxChunks);
- const chunks: RetrievedChunk[] = chunkHits.map(({ item, score }) => ({
- ...item,
- score,
- documentTitle: store.documentTitle(item.documentId),
- }));
+ const chunkHits = topK(queryVec, await store.allEmbeddedChunks(), (c) => c.embedding, maxChunks);
+ const chunks: RetrievedChunk[] = await Promise.all(
+ chunkHits.map(async ({ item, score }) => ({
+ ...item,
+ score,
+ documentTitle: await store.documentTitle(item.documentId),
+ })),
+ );
const bundle: ContextBundle = {
query,
diff --git a/test/helpers.ts b/test/helpers.ts
new file mode 100644
index 00000000..4058c8b6
--- /dev/null
+++ b/test/helpers.ts
@@ -0,0 +1,53 @@
+/**
+ * Offline, deterministic test doubles so the whole pipeline runs with no keys,
+ * no Ollama, and no network — the same "bring your own components" seam the
+ * engine exposes for production (see README → "Bring your own components").
+ */
+import type { Embedder, Extractor } from "../src/index.js";
+import type { Extraction } from "../src/index.js";
+
+/** Deterministic embedder: hashes tokens into a fixed-width, normalized vector. */
+export class FakeEmbedder implements Embedder {
+ readonly dimensions: number;
+ constructor(dimensions = 16) {
+ this.dimensions = dimensions;
+ }
+ async embed(texts: string[]): Promise<number[][]> {
+ return texts.map((t) => this.vec(t));
+ }
+ private vec(text: string): number[] {
+ const v = new Array(this.dimensions).fill(0);
+ for (const tok of text.toLowerCase().split(/\s+/).filter(Boolean)) {
+ let h = 0;
+ for (let i = 0; i < tok.length; i++) h = (h * 31 + tok.charCodeAt(i)) | 0;
+ v[Math.abs(h) % this.dimensions] += 1;
+ }
+ const norm = Math.sqrt(v.reduce((s, x) => s + x * x, 0)) || 1;
+ return v.map((x) => x / norm);
+ }
+}
+
+/**
+ * Extractor driven by simple markup so tests fully control the graph shape:
+ * - `[[Entity Name]]` → an entity
+ * - `[[A]] ==relation==> [[B]]` → a relationship
+ */
+export class BracketExtractor implements Extractor {
+ async extract(text: string): Promise<Extraction> {
+ const names = new Set<string>();
+ for (const m of text.matchAll(/\[\[([^\]]+)\]\]/g)) names.add(m[1].trim());
+ const entities = [...names].map((name) => ({
+ name,
+ type: "concept",
+ summary: `${name} summary`,
+ }));
+ const relations = [...text.matchAll(/\[\[([^\]]+)\]\]\s*==([a-z_]+)==>\s*\[\[([^\]]+)\]\]/g)].map(
+ (m) => ({ source: m[1].trim(), relation: m[2], target: m[3].trim() }),
+ );
+ return { entities, relations };
+ }
+}
+
+export function fakeProviders() {
+ return { embedder: new FakeEmbedder(), extractor: new BracketExtractor() };
+}
diff --git a/test/merge.test.ts b/test/merge.test.ts
new file mode 100644
index 00000000..43ae865b
--- /dev/null
+++ b/test/merge.test.ts
@@ -0,0 +1,88 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { ContextGraphEngine, confidenceFromObservations } from "../src/index.js";
+import { fakeProviders } from "./helpers.ts";
+
+function engine() {
+ return new ContextGraphEngine({ dbPath: ":memory:", ...fakeProviders() });
+}
+
+async function nodeByName(e: ContextGraphEngine, name: string) {
+ const nodes = await e.store.findNodesByName(name.toLowerCase());
+ return nodes[0];
+}
+
+test("confidence is a pure, saturating function of observations", () => {
+ assert.equal(Math.round(confidenceFromObservations(1) * 1000), 600);
+ assert.equal(Math.round(confidenceFromObservations(2) * 1000), 680);
+ assert.equal(Math.round(confidenceFromObservations(3) * 1000), 744);
+ assert.ok(confidenceFromObservations(50) <= 0.99);
+ // Monotonic non-decreasing.
+ assert.ok(confidenceFromObservations(5) > confidenceFromObservations(4));
+});
+
+test("live re-observation reinforces; confidence tracks the count", async () => {
+ const e = engine();
+ await e.ingest("[[Billing Service]] ==depends_on==> [[Payments Gateway]]", { title: "one" });
+ let n = await nodeByName(e, "billing service");
+ assert.equal(n.observations, 1);
+ assert.equal(Math.round(n.confidence * 1000), 600);
+
+ // A distinct document observing the same entity bumps the counter.
+ await e.ingest("[[Billing Service]] charges customers", { title: "two" });
+ n = await nodeByName(e, "billing service");
+ assert.equal(n.observations, 2);
+ assert.equal(Math.round(n.confidence * 1000), 680);
+ // Two distinct observation sources are tracked.
+ assert.equal(Object.keys(n.observationSources).length, 2);
+ await e.close();
+});
+
+test("re-importing the same graph file is idempotent (no double-counting)", async () => {
+ const source = engine();
+ await source.ingest("[[Alpha]] ==relates_to==> [[Beta]]", { title: "doc-a" });
+ await source.ingest("[[Alpha]] again", { title: "doc-b" }); // Alpha now observed twice
+ const jsonl = await source.exportJsonl();
+
+ const alphaSource = await nodeByName(source, "alpha");
+ assert.equal(alphaSource.observations, 2);
+
+ const target = engine();
+ const first = await target.importJsonl(jsonl);
+ assert.equal(first.nodesCreated, 2); // Alpha, Beta
+ const alpha1 = await nodeByName(target, "alpha");
+ assert.equal(alpha1.observations, 2);
+ const conf1 = alpha1.confidence;
+
+ // Import the exact same file again — must change nothing.
+ const second = await target.importJsonl(jsonl);
+ assert.equal(second.nodesCreated, 0);
+ const alpha2 = await nodeByName(target, "alpha");
+ assert.equal(alpha2.observations, 2, "re-import must not inflate observations");
+ assert.equal(alpha2.confidence, conf1, "re-import must not inflate confidence");
+
+ const stats = await target.stats();
+ assert.equal(stats.nodes, 2);
+ assert.equal(stats.edges, 1);
+ await source.close();
+ await target.close();
+});
+
+test("later summaries win by timestamp on merge", async () => {
+ const e = engine();
+ await e.ingest("[[Widget]] summary", { title: "one" });
+ const jsonl = await e.exportJsonl();
+ const older = JSON.parse(
+ jsonl.split("\n").find((l) => l.includes('"kind":"node"')) as string,
+ );
+ // Craft an incoming record with a newer summary + timestamp.
+ older.summary = "a much longer and newer widget description";
+ older.summaryUpdatedAt = "2999-01-01T00:00:00.000Z";
+ const target = engine();
+ await target.importJsonl(jsonl); // seed original
+ await target.importJsonl([JSON.stringify({ kind: "meta", format: 1, embeddingDimensions: 16 }), JSON.stringify({ kind: "node", ...older })].join("\n"));
+ const n = await nodeByName(target, "widget");
+ assert.equal(n.summary, "a much longer and newer widget description");
+ await e.close();
+ await target.close();
+});
diff --git a/test/sync.test.ts b/test/sync.test.ts
new file mode 100644
index 00000000..abbf81a3
--- /dev/null
+++ b/test/sync.test.ts
@@ -0,0 +1,70 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { ContextGraphEngine } from "../src/index.js";
+import { fakeProviders } from "./helpers.ts";
+
+function engine() {
+ return new ContextGraphEngine({ dbPath: ":memory:", ...fakeProviders() });
+}
+
+async function names(e: ContextGraphEngine): Promise<string[]> {
+ return (await e.store.allNodes()).map((n) => n.name).sort();
+}
+
+test("two machines converge to the same graph after cross-import (Mode A)", async () => {
+ // Alice and Bob each learn different facts about a shared entity.
+ const alice = engine();
+ const bob = engine();
+ await alice.ingest("[[Auth Service]] ==uses==> [[OAuth]]", { title: "alice" });
+ await bob.ingest("[[Auth Service]] ==rotates==> [[Tokens]]", { title: "bob" });
+
+ const aliceFile = await alice.exportJsonl();
+ const bobFile = await bob.exportJsonl();
+
+ // Exchange files.
+ await alice.importJsonl(bobFile);
+ await bob.importJsonl(aliceFile);
+
+ // Both now know all three entities and both edges.
+ const expected = ["Auth Service", "OAuth", "Tokens"].sort();
+ assert.deepEqual(await names(alice), expected);
+ assert.deepEqual(await names(bob), expected);
+
+ const aStats = await alice.stats();
+ const bStats = await bob.stats();
+ assert.equal(aStats.nodes, bStats.nodes);
+ assert.equal(aStats.edges, bStats.edges);
+ assert.equal(aStats.edges, 2);
+
+ // "Auth Service" was observed once on each machine → 2 distinct sources.
+ const aAuth = (await alice.store.findNodesByName("auth service"))[0];
+ assert.equal(aAuth.observations, 2, "shared entity sums both machines' observations");
+ await alice.close();
+ await bob.close();
+});
+
+test("import order does not change the result (commutativity)", async () => {
+ const a = engine();
+ const b = engine();
+ await a.ingest("[[X]] ==r==> [[Y]]", { title: "a" });
+ await b.ingest("[[Y]] ==r==> [[Z]]", { title: "b" });
+ const af = await a.exportJsonl();
+ const bf = await b.exportJsonl();
+
+ const left = engine();
+ await left.importJsonl(af);
+ await left.importJsonl(bf);
+
+ const right = engine();
+ await right.importJsonl(bf);
+ await right.importJsonl(af);
+
+ assert.deepEqual(await names(left), await names(right));
+ const ls = await left.stats();
+ const rs = await right.stats();
+ assert.deepEqual([ls.nodes, ls.edges], [rs.nodes, rs.edges]);
+ await a.close();
+ await b.close();
+ await left.close();
+ await right.close();
+});