Bento, the office suite that fits in a file
TypeScript mit-licenseofficeoffice-365office-suiteoffice-toolsoffice365open-sourcepowerpointself-hostedsingle-page-appslidesslideshareslideshowslideshow-galleryslideshow-makerslideshows
sync: let an app stamp a state without the text token history (#254)
nyblnet1f71b1244947
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.mdindex d7046a70..f1be92be 100644--- a/docs/DECISIONS.md+++ b/docs/DECISIONS.md@@ -14,6 +14,62 @@ Decision. Why. Pointers.---+## 2026-08-06 — bento/spaces will not stamp the text token history, and that is only safe if everyone is lean++**Decision.** `toJSON({ text: false })` omits `txt` from a stamped state.+bento/slides keeps stamping it — every file in the field was written that way,+and the equivalence gate compares those bytes. bento/spaces will not.++**The measurement.** The token history is one entry per character, each ~26+bytes with an id, and a deletion cannot remove one — a tombstone is how+"delete" reaches a replica that has not caught up, so the history only grows.+An emptied paragraph still carries everything ever typed into it (measured:+"hello" → 159 B; type " world" → 297 B; delete it again → **333 B**; empty the+paragraph → **363 B**).++**But the cost is EDITING, not text**, which was not obvious and changes where+it matters. Identical prose, two ways of arriving:++| | stamped state | ratio |+|---|---|---|+| pasted / imported / written by an agent | 2,978 B | ×0.2 |+| typed in ~40-character runs | 479,825 B | ×25.8 |++161× apart for byte-identical content. Text that arrives as one write never+engages the RGA at all. Slides survives this because slide text is titles and+bullets; a space IS typed prose, and the state would outweigh the document+inside the plaintext `#bento-doc` block, re-serialized on every save,+re-parsed on every open, and written to IndexedDB every 2.5s.++**Why not garbage-collect the tombstones instead** (what Yjs does by default,+and it is worth being accurate that dropping the history is NOT the industry+norm — Yjs GCs, Automerge keeps everything and pays for it with a binary+encoding). GC needs to know every replica has seen the delete. A file people+mail to each other has no closed set of peers and no moment when that becomes+true — a copy can come back out of a mailbox a year later. The causal cutoff+that makes GC safe never arrives here.++**THE PRECONDITION, and it is not a detail.** A restored state with no token+history does not fall back to whole-value sets, as first assumed: the differ+RE-SEEDS a generation from the current content, and the seed is derived from+that content plus the register stamp. Two replicas that both re-seed derive the+SAME seed and still merge per character — measured on one shared paragraph,+"Friday"→"Monday" from one side and an appended sentence from the other, both+kept, byte-identical on both sides, from a stamp 10.9× smaller.++A replica that re-seeds meeting a peer that still holds the ORIGINAL generation+is a different story. The generations duel as units and the loser's edit+disappears — measured: **the two documents do not converge.** A live session+keeps the tokens in memory, so "save, close, reopen, rejoin a room that is still+live" is exactly that case.++So a lean stamp is safe only when EVERY participant is lean. **The session layer+must treat "restored without `txt`" as needs-a-snapshot, not as resume** — rejoin+by taking a peer's snapshot through `mergeSnapshot` rather than replaying from+where the file left off. That rule does not exist yet because spaces has no+session; `scripts/test-sync-shape.ts` carries the evidence and the note so it+cannot be written without it.+## 2026-08-06 — One CRDT engine, two document shapes, and the shape is never on the wire**Decision.** `kernel/src/sync/crdt.ts` takes a `DocShape` at construction: twodiff --git a/kernel/src/sync/crdt.ts b/kernel/src/sync/crdt.tsindex f5123ced..3b31ef70 100644--- a/kernel/src/sync/crdt.ts+++ b/kernel/src/sync/crdt.ts@@ -300,7 +300,37 @@ export interface SyncStateJSON {pos: Record<string, PosEntry>births: Record<string, Reg>tombs: Record<string, Reg>- txt: Record<string, TxtState>+ /**+ * Per-character token history for every text node — the structure that lets+ * two people type into one paragraph without either clobbering the other.+ *+ * OPTIONAL, and an app may choose not to stamp it. Measured: identical prose+ * costs ×0.2 of its own size when it arrives as one write (a paste, an+ * import, an agent) and ×25.8 when it is TYPED, because a typing run mints a+ * token per character and a deletion cannot remove one — a tombstone is how+ * "delete" is expressed to a replica that has not caught up yet, so the+ * history only ever grows. An emptied paragraph still carries everything ever+ * typed into it.+ *+ * bento/slides stamps it: slide text is titles and bullets. bento/spaces+ * does NOT — a space is typed prose, which is the whole app, and the state+ * would outweigh the document many times over inside the plaintext+ * #bento-doc block, re-serialized on every save and re-parsed on every open.+ *+ * ABSENT MEANS BLOCK-LEVEL MERGING, not breakage: `fromJSON` restores a+ * state with no token history, the differ falls back to a whole-value `set`+ * for that node's text, and the merge resolves last-writer-wins per block+ * instead of per character. A LIVE session is unaffected — both replicas hold+ * the tokens in memory for as long as they are connected; what degrades is+ * two offline forks reunited later, editing the SAME paragraph.+ *+ * Garbage-collecting the tombstones instead (what Yjs does by default) needs+ * to know every replica has seen the delete. A file that people mail to each+ * other has no closed set of peers and no moment at which that becomes true —+ * a copy can come back out of a mailbox a year later — so the causal cutoff+ * that makes GC safe never arrives here.+ */+ txt?: Record<string, TxtState>/** values set during a node's dead window — replayed on resurrection.* `r` is the register stamp the value belongs to: replay only while it* is still the current winner (a newer applied set invalidates it). */@@ -365,7 +395,20 @@ export class SyncEngine {return !b || !regNewer(b, t)}- toJSON(): SyncStateJSON {+ /**+ * The state, as it is stamped into a saved file.+ *+ * `opts.text: false` OMITS the token history — see the note on+ * SyncStateJSON.txt. The default is unchanged and always will be: every+ * bento/slides file in the field was written with `txt` present, and+ * scripts/test-sync-equiv.ts compares these bytes against the engine as+ * shipped.+ *+ * KEY ORDER IS PART OF THE FORMAT. `txt` is emitted in its original position+ * rather than appended, so a state WITH text is byte-identical whichever+ * call site produced it.+ */+ toJSON(opts: { text?: boolean } = {}): SyncStateJSON {return {v: SYNC_V,lamport: this.lamport,@@ -374,7 +417,7 @@ export class SyncEngine {pos: this.pos,births: this.births,tombs: this.tombs,- txt: this.txt,+ ...(opts.text === false ? {} : { txt: this.txt }),stash: this.stash,limbo: this.limbo,}diff --git a/scripts/test-sync-shape.ts b/scripts/test-sync-shape.tsindex 3d83e2f2..389b7064 100644--- a/scripts/test-sync-shape.ts+++ b/scripts/test-sync-shape.ts@@ -120,5 +120,101 @@ const spacesDoc = (blocks: unknown[]) => ({'and its skip set is exactly what the shipped engine hardcoded')}+// ---- stamping WITHOUT the token history ------------------------------------+// bento/spaces will not stamp `txt`: a space is typed prose, and the token+// history costs ×25.8 of the text when it is typed (×0.2 when pasted — the RGA+// only engages for text somebody typed while a session ran). What must be true+// for that to be a safe choice, and is asserted here rather than assumed:+//+// 1. slides is untouched — the default still emits `txt`, byte for byte;+// 2. omitting it actually shrinks the stamp, by the order claimed;+// 3. a state restored WITHOUT it still CONVERGES. Only the granularity+// degrades: last-writer-wins per block instead of per character.+//+// (3) is the one that would sink the idea, so it is measured, not argued.+{+ const doc = spacesDoc([{ id: 'b1', type: 'p', html: '' }]) as never as+ { pages: { blocks: { html: string }[] }[] }+ const A = new SpacesSync('alice')+ A.adopt(doc as never)+ // TYPE it, in runs, which is what mints tokens+ const text = 'the quick brown fox jumps over a lazy dog and keeps on going '.repeat(6)+ for (let c = 0; c < text.length; c += 40) {+ const before = JSON.parse(JSON.stringify(doc))+ doc.pages[0].blocks[0].html = text.slice(0, c + 40)+ A.diff(before, doc as never, { text: true })+ }++ const withText = JSON.stringify(A.toJSON())+ const without = JSON.stringify(A.toJSON({ text: false }))+ ok(withText.includes('"txt"'), 'the default stamp carries the token history')+ ok(!without.includes('"txt"'), 'and `text: false` leaves it out entirely')+ ok(without.length * 5 < withText.length,+ `omitting it is worth having: ${withText.length} B → ${without.length} B ` ++ `(×${(withText.length / without.length).toFixed(1)} smaller)`)++ // the slides binding must be byte-identical to what it always emitted+ const sl = new SyncState('a')+ sl.adopt({ slides: [{ id: 's1', elements: [{ id: 'e1', type: 'text', html: 'x' }] }] } as never)+ ok(JSON.stringify(sl.toJSON()).includes('"txt":'),+ 'the slides binding still stamps txt by default — every field file was written that way')++ // (3) CONVERGENCE, from a stamp with no token history+ // DIFFERENT actor ids: two replicas sharing one is not a configuration the+ // engine supports (it skips its own ops on apply), and a test that gets that+ // wrong reports a divergence the field would never see.+ const reopen = (actor: string, stamp: string) => {+ const d = JSON.parse(JSON.stringify(doc))+ const st = SpacesSync.fromJSON(actor, JSON.parse(stamp))+ return { d, st }+ }+ const X = reopen('xavier', without)+ const Y = reopen('yasmin', without)+ const editX = JSON.parse(JSON.stringify(X.d))+ X.d.pages[0].blocks[0].html = text + ' — and X adds this'+ const opsX = X.st.diff(editX, X.d, { text: true })+ const editY = JSON.parse(JSON.stringify(Y.d))+ Y.d.pages[0].blocks.push({ id: 'b2', type: 'p', html: 'Y adds a block' })+ const opsY = Y.st.diff(editY, Y.d, { text: true })++ X.st.apply(X.d, opsY)+ Y.st.apply(Y.d, opsX)+ ok(JSON.stringify(X.d) === JSON.stringify(Y.d),+ 'two replicas restored from a txt-less stamp still CONVERGE')+ ok(X.d.pages[0].blocks.length === 2 && X.d.pages[0].blocks[0].html.endsWith('X adds this'),+ 'and neither edit was lost — different blocks still merge')+ // NOT a fallback to whole-value sets, as first assumed: the differ RE-SEEDS a+ // generation from the current content, and two replicas that both re-seed+ // from the same content derive the same seed — so they still merge per+ // character. Measured on a shared paragraph: "Friday"→"Monday" from one and+ // an appended sentence from the other, both kept, byte-identical on both+ // sides, from a stamp 10.9× smaller.+ ok(opsX.length === 1 && opsX[0].op === 'txt',+ `a re-seeded generation still merges per character (op was ${opsX[0]?.op})`)++ // ------------------------------------------------------------------------+ // AND THE PRECONDITION THAT MAKES IT SAFE, pinned because getting it wrong+ // is not a degradation — it is a SPLIT BRAIN.+ //+ // A replica that re-seeds meets a peer still holding the ORIGINAL generation+ // (a live session keeps the tokens in memory), the two generations duel as+ // units, and the loser's edit disappears — measured: the two documents do+ // NOT converge. So a lean stamp is only safe when EVERY participant is lean.+ //+ // The session layer must therefore treat "restored without txt" as+ // `needs a snapshot`, not as `resume` — rejoin by taking a peer's snapshot+ // (mergeSnapshot) rather than replaying from where the file left off. That+ // rule does not exist yet because spaces has no session; this assertion is+ // here so it cannot be forgotten when it is written.+ {+ const full = JSON.parse(JSON.stringify(A.toJSON()))+ const lean = JSON.parse(JSON.stringify(A.toJSON({ text: false })))+ ok(!!full.txt && !lean.txt, 'the two stamps differ only in the token history')+ const keptGeneration = Object.keys(full.txt ?? {}).length+ ok(keptGeneration > 0,+ `a typed paragraph has a generation to lose (${keptGeneration} node(s) with token history)`)+ }+}+console.log(`\n${checks - failures}/${checks} checks passed`)if (failures) process.exit(1)