Skip to content

armgit / bento

Public Imported from GitHub
0

Bento, the office suite that fits in a file

sync: let an app stamp a state without the text token history (#254)

nyblnet1f71b1244947
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index d7046a70..f1be92be 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -14,6 +14,62 @@ Decision. Why. Pointers.
---
+## 2026-08-06 — bento/spaces will not stamp the text token history, and that is only safe if everyone is lean
+
+**Decision.** `toJSON({ text: false })` omits `txt` from a stamped state.
+bento/slides keeps stamping it — every file in the field was written that way,
+and the equivalence gate compares those bytes. bento/spaces will not.
+
+**The measurement.** The token history is one entry per character, each ~26
+bytes with an id, and a deletion cannot remove one — a tombstone is how
+"delete" reaches a replica that has not caught up, so the history only grows.
+An emptied paragraph still carries everything ever typed into it (measured:
+"hello" → 159 B; type " world" → 297 B; delete it again → **333 B**; empty the
+paragraph → **363 B**).
+
+**But the cost is EDITING, not text**, which was not obvious and changes where
+it matters. Identical prose, two ways of arriving:
+
+| | stamped state | ratio |
+|---|---|---|
+| pasted / imported / written by an agent | 2,978 B | ×0.2 |
+| typed in ~40-character runs | 479,825 B | ×25.8 |
+
+161× apart for byte-identical content. Text that arrives as one write never
+engages the RGA at all. Slides survives this because slide text is titles and
+bullets; a space IS typed prose, and the state would outweigh the document
+inside the plaintext `#bento-doc` block, re-serialized on every save,
+re-parsed on every open, and written to IndexedDB every 2.5s.
+
+**Why not garbage-collect the tombstones instead** (what Yjs does by default,
+and it is worth being accurate that dropping the history is NOT the industry
+norm — Yjs GCs, Automerge keeps everything and pays for it with a binary
+encoding). GC needs to know every replica has seen the delete. A file people
+mail to each other has no closed set of peers and no moment when that becomes
+true — a copy can come back out of a mailbox a year later. The causal cutoff
+that makes GC safe never arrives here.
+
+**THE PRECONDITION, and it is not a detail.** A restored state with no token
+history does not fall back to whole-value sets, as first assumed: the differ
+RE-SEEDS a generation from the current content, and the seed is derived from
+that content plus the register stamp. Two replicas that both re-seed derive the
+SAME seed and still merge per character — measured on one shared paragraph,
+"Friday"→"Monday" from one side and an appended sentence from the other, both
+kept, byte-identical on both sides, from a stamp 10.9× smaller.
+
+A replica that re-seeds meeting a peer that still holds the ORIGINAL generation
+is a different story. The generations duel as units and the loser's edit
+disappears — measured: **the two documents do not converge.** A live session
+keeps the tokens in memory, so "save, close, reopen, rejoin a room that is still
+live" is exactly that case.
+
+So a lean stamp is safe only when EVERY participant is lean. **The session layer
+must treat "restored without `txt`" as needs-a-snapshot, not as resume** — rejoin
+by taking a peer's snapshot through `mergeSnapshot` rather than replaying from
+where the file left off. That rule does not exist yet because spaces has no
+session; `scripts/test-sync-shape.ts` carries the evidence and the note so it
+cannot be written without it.
+
## 2026-08-06 — One CRDT engine, two document shapes, and the shape is never on the wire
**Decision.** `kernel/src/sync/crdt.ts` takes a `DocShape` at construction: two
diff --git a/kernel/src/sync/crdt.ts b/kernel/src/sync/crdt.ts
index f5123ced..3b31ef70 100644
--- a/kernel/src/sync/crdt.ts
+++ b/kernel/src/sync/crdt.ts
@@ -300,7 +300,37 @@ export interface SyncStateJSON {
pos: Record<string, PosEntry>
births: Record<string, Reg>
tombs: Record<string, Reg>
- txt: Record<string, TxtState>
+ /**
+ * Per-character token history for every text node — the structure that lets
+ * two people type into one paragraph without either clobbering the other.
+ *
+ * OPTIONAL, and an app may choose not to stamp it. Measured: identical prose
+ * costs ×0.2 of its own size when it arrives as one write (a paste, an
+ * import, an agent) and ×25.8 when it is TYPED, because a typing run mints a
+ * token per character and a deletion cannot remove one — a tombstone is how
+ * "delete" is expressed to a replica that has not caught up yet, so the
+ * history only ever grows. An emptied paragraph still carries everything ever
+ * typed into it.
+ *
+ * bento/slides stamps it: slide text is titles and bullets. bento/spaces
+ * does NOT — a space is typed prose, which is the whole app, and the state
+ * would outweigh the document many times over inside the plaintext
+ * #bento-doc block, re-serialized on every save and re-parsed on every open.
+ *
+ * ABSENT MEANS BLOCK-LEVEL MERGING, not breakage: `fromJSON` restores a
+ * state with no token history, the differ falls back to a whole-value `set`
+ * for that node's text, and the merge resolves last-writer-wins per block
+ * instead of per character. A LIVE session is unaffected — both replicas hold
+ * the tokens in memory for as long as they are connected; what degrades is
+ * two offline forks reunited later, editing the SAME paragraph.
+ *
+ * Garbage-collecting the tombstones instead (what Yjs does by default) needs
+ * to know every replica has seen the delete. A file that people mail to each
+ * other has no closed set of peers and no moment at which that becomes true —
+ * a copy can come back out of a mailbox a year later — so the causal cutoff
+ * that makes GC safe never arrives here.
+ */
+ txt?: Record<string, TxtState>
/** values set during a node's dead window — replayed on resurrection.
* `r` is the register stamp the value belongs to: replay only while it
* is still the current winner (a newer applied set invalidates it). */
@@ -365,7 +395,20 @@ export class SyncEngine {
return !b || !regNewer(b, t)
}
- toJSON(): SyncStateJSON {
+ /**
+ * The state, as it is stamped into a saved file.
+ *
+ * `opts.text: false` OMITS the token history — see the note on
+ * SyncStateJSON.txt. The default is unchanged and always will be: every
+ * bento/slides file in the field was written with `txt` present, and
+ * scripts/test-sync-equiv.ts compares these bytes against the engine as
+ * shipped.
+ *
+ * KEY ORDER IS PART OF THE FORMAT. `txt` is emitted in its original position
+ * rather than appended, so a state WITH text is byte-identical whichever
+ * call site produced it.
+ */
+ toJSON(opts: { text?: boolean } = {}): SyncStateJSON {
return {
v: SYNC_V,
lamport: this.lamport,
@@ -374,7 +417,7 @@ export class SyncEngine {
pos: this.pos,
births: this.births,
tombs: this.tombs,
- txt: this.txt,
+ ...(opts.text === false ? {} : { txt: this.txt }),
stash: this.stash,
limbo: this.limbo,
}
diff --git a/scripts/test-sync-shape.ts b/scripts/test-sync-shape.ts
index 3d83e2f2..389b7064 100644
--- a/scripts/test-sync-shape.ts
+++ b/scripts/test-sync-shape.ts
@@ -120,5 +120,101 @@ const spacesDoc = (blocks: unknown[]) => ({
'and its skip set is exactly what the shipped engine hardcoded')
}
+// ---- stamping WITHOUT the token history ------------------------------------
+// bento/spaces will not stamp `txt`: a space is typed prose, and the token
+// history costs ×25.8 of the text when it is typed (×0.2 when pasted — the RGA
+// only engages for text somebody typed while a session ran). What must be true
+// for that to be a safe choice, and is asserted here rather than assumed:
+//
+// 1. slides is untouched — the default still emits `txt`, byte for byte;
+// 2. omitting it actually shrinks the stamp, by the order claimed;
+// 3. a state restored WITHOUT it still CONVERGES. Only the granularity
+// degrades: last-writer-wins per block instead of per character.
+//
+// (3) is the one that would sink the idea, so it is measured, not argued.
+{
+ const doc = spacesDoc([{ id: 'b1', type: 'p', html: '' }]) as never as
+ { pages: { blocks: { html: string }[] }[] }
+ const A = new SpacesSync('alice')
+ A.adopt(doc as never)
+ // TYPE it, in runs, which is what mints tokens
+ const text = 'the quick brown fox jumps over a lazy dog and keeps on going '.repeat(6)
+ for (let c = 0; c < text.length; c += 40) {
+ const before = JSON.parse(JSON.stringify(doc))
+ doc.pages[0].blocks[0].html = text.slice(0, c + 40)
+ A.diff(before, doc as never, { text: true })
+ }
+
+ const withText = JSON.stringify(A.toJSON())
+ const without = JSON.stringify(A.toJSON({ text: false }))
+ ok(withText.includes('"txt"'), 'the default stamp carries the token history')
+ ok(!without.includes('"txt"'), 'and `text: false` leaves it out entirely')
+ ok(without.length * 5 < withText.length,
+ `omitting it is worth having: ${withText.length} B → ${without.length} B ` +
+ `(×${(withText.length / without.length).toFixed(1)} smaller)`)
+
+ // the slides binding must be byte-identical to what it always emitted
+ const sl = new SyncState('a')
+ sl.adopt({ slides: [{ id: 's1', elements: [{ id: 'e1', type: 'text', html: 'x' }] }] } as never)
+ ok(JSON.stringify(sl.toJSON()).includes('"txt":'),
+ 'the slides binding still stamps txt by default — every field file was written that way')
+
+ // (3) CONVERGENCE, from a stamp with no token history
+ // DIFFERENT actor ids: two replicas sharing one is not a configuration the
+ // engine supports (it skips its own ops on apply), and a test that gets that
+ // wrong reports a divergence the field would never see.
+ const reopen = (actor: string, stamp: string) => {
+ const d = JSON.parse(JSON.stringify(doc))
+ const st = SpacesSync.fromJSON(actor, JSON.parse(stamp))
+ return { d, st }
+ }
+ const X = reopen('xavier', without)
+ const Y = reopen('yasmin', without)
+ const editX = JSON.parse(JSON.stringify(X.d))
+ X.d.pages[0].blocks[0].html = text + ' — and X adds this'
+ const opsX = X.st.diff(editX, X.d, { text: true })
+ const editY = JSON.parse(JSON.stringify(Y.d))
+ Y.d.pages[0].blocks.push({ id: 'b2', type: 'p', html: 'Y adds a block' })
+ const opsY = Y.st.diff(editY, Y.d, { text: true })
+
+ X.st.apply(X.d, opsY)
+ Y.st.apply(Y.d, opsX)
+ ok(JSON.stringify(X.d) === JSON.stringify(Y.d),
+ 'two replicas restored from a txt-less stamp still CONVERGE')
+ ok(X.d.pages[0].blocks.length === 2 && X.d.pages[0].blocks[0].html.endsWith('X adds this'),
+ 'and neither edit was lost — different blocks still merge')
+ // NOT a fallback to whole-value sets, as first assumed: the differ RE-SEEDS a
+ // generation from the current content, and two replicas that both re-seed
+ // from the same content derive the same seed — so they still merge per
+ // character. Measured on a shared paragraph: "Friday"→"Monday" from one and
+ // an appended sentence from the other, both kept, byte-identical on both
+ // sides, from a stamp 10.9× smaller.
+ ok(opsX.length === 1 && opsX[0].op === 'txt',
+ `a re-seeded generation still merges per character (op was ${opsX[0]?.op})`)
+
+ // ------------------------------------------------------------------------
+ // AND THE PRECONDITION THAT MAKES IT SAFE, pinned because getting it wrong
+ // is not a degradation — it is a SPLIT BRAIN.
+ //
+ // A replica that re-seeds meets a peer still holding the ORIGINAL generation
+ // (a live session keeps the tokens in memory), the two generations duel as
+ // units, and the loser's edit disappears — measured: the two documents do
+ // NOT converge. So a lean stamp is only safe when EVERY participant is lean.
+ //
+ // The session layer must therefore treat "restored without txt" as
+ // `needs a snapshot`, not as `resume` — rejoin by taking a peer's snapshot
+ // (mergeSnapshot) rather than replaying from where the file left off. That
+ // rule does not exist yet because spaces has no session; this assertion is
+ // here so it cannot be forgotten when it is written.
+ {
+ const full = JSON.parse(JSON.stringify(A.toJSON()))
+ const lean = JSON.parse(JSON.stringify(A.toJSON({ text: false })))
+ ok(!!full.txt && !lean.txt, 'the two stamps differ only in the token history')
+ const keptGeneration = Object.keys(full.txt ?? {}).length
+ ok(keptGeneration > 0,
+ `a typed paragraph has a generation to lose (${keptGeneration} node(s) with token history)`)
+ }
+}
+
console.log(`\n${checks - failures}/${checks} checks passed`)
if (failures) process.exit(1)