Skip to content

Node-based terminal manager for AI coding agents — tmux-backed terminals and parallel agent sessions as draggable nodes on an infinite pan/zoom canvas. macOS, Linux, and a browser Server Edition.

fix(canvas-order): epoch-gated edge mirrors, FIFO gate release, bounded restart degrade (D4)

reviewf6f42754b572
diff --git a/docs/team-presence.md b/docs/team-presence.md
index 250c2aab3..30ba9ddab 100644
--- a/docs/team-presence.md
+++ b/docs/team-presence.md
@@ -405,8 +405,11 @@ independently on each client from the already-broadcast `agent:status` stream; a
what rule 4's `seen` stamps). A reconnect to the SAME core also resets, and a first cast stamped
`seen: 0` there — ⌘Z of a node deleted before the drop — was a stale frame to every peer holding
the tombstone. After a real restart the kept value is above every new `seq`, so our casts read as
- "never stale": the pre-rule-4 verdict, not a split. (The reflector clamps it to `seq - 1`, which
- changes no verdict here; see the note on the clamp under rule 4.)
+ "never stale": the pre-rule-4 verdict, not a split. That window is bounded: the first stamped `seq`
+ heard after a reset re-bases the position when it is at or below it (the core restarted and its
+ counter began again), and leaves it alone when it is above it (the same core carried on). (The
+ reflector clamps it to `seq - 1`, which changes no verdict here; see the note on the clamp under
+ rule 4.)
- **A cast the reflector would refuse is never made** — the publisher validates with the **same**
predicate the reflector's ingest uses (`isCanvasMutation`, moved to `src/shared/canvas-mutations.ts`
so both ends share one verdict) *before* recording a pending entry and *before* casting. A refusal
@@ -878,8 +881,12 @@ our `seen` only when its echo comes back, so a re-creation of the same id cast b
deleted and redrawn, a node deleted and ⌘Z'd, inside one round trip — carried a `seen` below the
remove: every peer dropped it as a stale frame while we kept showing it, and the next whole-file save
became last-writer-wins on disk. Canvas's send callback now refuses any non-remove op for a key with a
-remove of ours in flight (counted per key, not TTL-bound — a late ack is exactly when it matters, and a
-lost one comes with a reconnect, whose `reset` clears it). The refusal keeps the op owed in the
+remove of ours in flight (counted per key, not TTL-bound — a late ack is exactly when it matters). A
+LOST ack does not come with a reconnect: the ui sink drops a single message and keeps the connection
+(`SINK_FAILURE_LIMIT`). Our echoes come back in the order we cast (FIFO), so the echo of a LATER cast
+of ours proves an earlier one was lost, and releases that remove's gate too; Canvas compares the total
+count of pending removes across each `accept` to see it. A reset still clears everything. The
+refusal keeps the op owed in the
publisher, and an adopt in that window keeps an owed EDGE owed as well as an owed node (a teammate's
op is usually what arrives during that round trip). Nothing else would ever cast it — our echo is an
ack and changes no React state — so the echo that clears the gate re-publishes, after the handler and
diff --git a/src/renderer/canvas/Canvas.tsx b/src/renderer/canvas/Canvas.tsx
index e4c285963..6aa62e364 100644
--- a/src/renderer/canvas/Canvas.tsx
+++ b/src/renderer/canvas/Canvas.tsx
@@ -650,7 +650,7 @@ import { registerWorkspaceDirty } from '../state/workspaceDirty'
import { snapNodeToGrid } from '../lib/nodeSizing'
import { snapResizeChanges } from '../lib/resizeSnap'
import { canClearDirty, canCommitCanvas, canCreateOnCanvas, liveCanvasHolds } from '../state/persistGuards'
-import { rebaseOnLatest, useNodesEpoch } from './nodesEpoch'
+import { mirrorLatest, rebaseOnLatest, useNodesEpoch } from './nodesEpoch'
import { boardLiveNodeIds, createKanbanPublisher } from './kanban-sync'
import { createStoredCanvasPublisher } from './stored-publish'
import { isHidden } from '../lib/ui-visibility'
@@ -1234,12 +1234,9 @@ export function Canvas() {
// SyncLane re-render and the setter a DefaultLane one (see nodesEpoch.ts). So the render-time
// mirror only copies the state when the STATE changed; an unconditional copy put the previous
// project's edges back into the ref in exactly the window the synchronous write closes.
+ // It is gated on the epoch as well, beside the node mirror (`mirrorLatest`, below `useNodesEpoch`).
const linkEdgesRef = useRef<Edge[]>(linkEdges)
const linkEdgesMirroredRef = useRef<Edge[]>(linkEdges)
- if (linkEdgesMirroredRef.current !== linkEdges) {
- linkEdgesMirroredRef.current = linkEdges
- linkEdgesRef.current = linkEdges
- }
// "Spawned by" ropes drawn from a control-capable agent to the nodes it opens via the
// `nodeterm` CLI (see the onAgentControl effect) and from browser popups to their opener.
// Merged only at the <ReactFlow> prop and never turned into context links, but PERSISTED
@@ -1248,10 +1245,6 @@ export function Canvas() {
// Same latest-not-rendered mirror as linkEdgesRef above, for the same three writers.
const controlEdgesRef = useRef<Edge[]>(controlEdges)
const controlEdgesMirroredRef = useRef<Edge[]>(controlEdges)
- if (controlEdgesMirroredRef.current !== controlEdges) {
- controlEdgesMirroredRef.current = controlEdges
- controlEdgesRef.current = controlEdges
- }
const [dirty, setDirty] = useState(false)
// Bumped only when a save finished with `dirty` still set (an edit raced it). It exists purely to
// give the debounced-autosave effect a dependency that CHANGES in that case — `dirty` stays true
@@ -1721,6 +1714,14 @@ export function Canvas() {
* empty `useNodesState([])` can never be committed as some project's canvas.
*/
const { nodesRef, nodesProjectIdRef, renderedProjectId, installEpoch } = useNodesEpoch(nodes)
+ // The two edge refs' render-time mirrors (declared beside their state, above), under the node
+ // mirror's two conditions: the state CHANGED, and this render belongs to the latest epoch. Without
+ // the second, a discrete `setLinkEdges(fn)` in a switch window renders fn(the OUTGOING project's
+ // edges) and the mirror put them in the ref under the incoming project's tag, so the edge publisher
+ // diffed the incoming project against them and cast spurious `edge-remove`s into it (D4).
+ const inLatestEpoch = renderedProjectId === nodesProjectIdRef.current
+ mirrorLatest(linkEdges, linkEdgesMirroredRef, linkEdgesRef, inLatestEpoch)
+ mirrorLatest(controlEdges, controlEdgesMirroredRef, controlEdgesRef, inLatestEpoch)
/**
* The project whose webview nodes the NEXT load must retire into the keep-alive pool. Separate
* from `nodesProjectIdRef` on purpose: the epoch tag is invalidated on the load effect's
@@ -4075,11 +4076,11 @@ export function Canvas() {
if (provesPeer(mutation, canvasSrcRef.current)) hasPeersRef.current = true
const order = orderRef.current
if (!order) return
- const key = mutationKey(mutation, projectId)
- const held = order.hasPendingRemove(key)
+ // Counted over EVERY key, before and after: an echo of ours releases its own key's gate, and
+ // also any EARLIER remove of ours whose echo was lost on the way back (canvas-order, FIFO).
+ const heldBefore = order.pendingRemoveCount()
const apply = order.accept(mutation, projectId)
- const released = held && !order.hasPendingRemove(key)
- if (released) queueMicrotask(releaseHeld)
+ if (order.pendingRemoveCount() < heldBefore) queueMicrotask(releaseHeld)
if (!apply) return
// ---- the board (kanban) ----
// Into the projects STORE, for the active project and a background one alike: the board, the
diff --git a/src/renderer/canvas/canvas-sync-ordering.source.test.ts b/src/renderer/canvas/canvas-sync-ordering.source.test.ts
index cceaa5568..58930b772 100644
--- a/src/renderer/canvas/canvas-sync-ordering.source.test.ts
+++ b/src/renderer/canvas/canvas-sync-ordering.source.test.ts
@@ -34,14 +34,16 @@ describe('the re-creation gate (canvas-order hasPendingRemove)', () => {
// Held = owed, but nothing re-publishes on its own: our echo is an ack, it changes no React state,
// so the [nodes] publish effect never runs. The release has to publish.
- it('our own remove coming back releases what the gate held', () => {
+ // Counted over EVERY key (D4): an echo of ours also releases an EARLIER remove whose echo was lost
+ // (canvas-order, FIFO), which is a different key than the one this echo addresses.
+ it('our own remove coming back — or a later echo proving it lost — releases what the gate held', () => {
const body = receiveHandler()
- const held = body.indexOf('const held = order.hasPendingRemove(key)')
+ const before = body.indexOf('const heldBefore = order.pendingRemoveCount()')
const accept = body.indexOf('order.accept(mutation, projectId)')
- expect(held).toBeGreaterThan(-1)
- expect(held).toBeLessThan(accept) // asked BEFORE the ack draws the count down
- expect(body).toMatch(/const released = held && !order\.hasPendingRemove\(key\)/)
- expect(body).toMatch(/if \(released\) queueMicrotask\(releaseHeld\)/)
+ expect(before).toBeGreaterThan(-1)
+ expect(before).toBeLessThan(accept) // asked BEFORE the ack draws the count down
+ expect(body).toMatch(/if \(order\.pendingRemoveCount\(\) < heldBefore\) queueMicrotask\(releaseHeld\)/)
+ expect(body).not.toContain('const released = held')
})
// RULING R4: one order serves every loaded project, and a board's two order ops are per-project
@@ -53,7 +55,6 @@ describe('the re-creation gate (canvas-order hasPendingRemove)', () => {
expect(send).toContain('order.onLocal(stamped, projectId)')
expect(send).not.toMatch(/order\.onLocal\(stamped\)/)
const recv = receiveHandler()
- expect(recv).toContain('const key = mutationKey(mutation, projectId)')
expect(recv).toContain('order.accept(mutation, projectId)')
expect(recv).not.toMatch(/order\.accept\(mutation\)/)
})
diff --git a/src/renderer/canvas/edge-sync-wiring.test.ts b/src/renderer/canvas/edge-sync-wiring.test.ts
index 60877d010..e9b3efe4e 100644
--- a/src/renderer/canvas/edge-sync-wiring.test.ts
+++ b/src/renderer/canvas/edge-sync-wiring.test.ts
@@ -38,11 +38,18 @@ describe('edge ref render-time mirror', () => {
// setLinkEdges/setControlEdges (see nodesEpoch.ts). An unconditional `ref = state` mirror in that
// render put the previous project's edges back into the ref right after the load effect had
// assigned the new ones — reopening the window the synchronous assignment closes.
- it('mirrors the edge state into the ref only when the state changed', () => {
- expect(src).toMatch(/if \(linkEdgesMirroredRef\.current !== linkEdges\) \{\n\s*linkEdgesMirroredRef\.current = linkEdges\n\s*linkEdgesRef\.current = linkEdges/)
- expect(src).toMatch(/if \(controlEdgesMirroredRef\.current !== controlEdges\) \{\n\s*controlEdgesMirroredRef\.current = controlEdges\n\s*controlEdgesRef\.current = controlEdges/)
+ // …and only in a render of the LATEST epoch, like the node mirror (D4): a discrete edge update in
+ // a switch window renders on the outgoing project's edges. Behaviour in nodesEpoch.test.tsx, which
+ // runs `mirrorLatest` in a replica of this render; these pin that Canvas calls it that way.
+ it('mirrors the edge state into the ref only when the state changed, in the latest epoch', () => {
+ expect(src).toContain('const inLatestEpoch = renderedProjectId === nodesProjectIdRef.current')
+ expect(src).toContain('mirrorLatest(linkEdges, linkEdgesMirroredRef, linkEdgesRef, inLatestEpoch)')
+ expect(src).toContain('mirrorLatest(controlEdges, controlEdgesMirroredRef, controlEdgesRef, inLatestEpoch)')
+ // After the epoch pair exists, never before it.
+ expect(src.indexOf('mirrorLatest(linkEdges')).toBeGreaterThan(src.indexOf('= useNodesEpoch(nodes)'))
expect(src).not.toMatch(/^ {2}linkEdgesRef\.current = linkEdges$/m)
expect(src).not.toMatch(/^ {2}controlEdgesRef\.current = controlEdges$/m)
+ expect(src).not.toMatch(/if \(linkEdgesMirroredRef\.current !== linkEdges\)/)
})
})
diff --git a/src/renderer/canvas/nodesEpoch.test.tsx b/src/renderer/canvas/nodesEpoch.test.tsx
index 9960e0282..091d51e0f 100644
--- a/src/renderer/canvas/nodesEpoch.test.tsx
+++ b/src/renderer/canvas/nodesEpoch.test.tsx
@@ -1,10 +1,10 @@
// @vitest-environment jsdom
-import { useEffect, useState } from 'react'
+import { useEffect, useRef, useState } from 'react'
import { flushSync } from 'react-dom'
import { createRoot } from 'react-dom/client'
import { create } from 'zustand'
import { afterEach, beforeEach, expect, it } from 'vitest'
-import { rebaseOnLatest, useNodesEpoch } from './nodesEpoch'
+import { mirrorLatest, rebaseOnLatest, useNodesEpoch } from './nodesEpoch'
import { canCommitCanvas, liveCanvasHolds } from '../state/persistGuards'
// The field bug (2026-09-26): a project switch loaded project B, but a zustand store the load
@@ -23,6 +23,8 @@ const useOther = create<{ n: number }>(() => ({ n: 0 }))
/** One render: the `nodes` STATE it rendered, and the ref pair as that render left it. */
interface Seen { state: string[]; nodes: string[]; epoch: string | null }
let seen: Seen[] = []
+/** Per render: the edge ref (Canvas's `linkEdgesRef`) and the tag it sat beside. */
+let edgeSeen: Array<{ edges: string[]; epoch: string | null }> = []
let root: ReturnType<typeof createRoot> | null = null
/** The projects store: each project's serialized nodes. */
let store: Record<string, string[]> = {}
@@ -32,15 +34,22 @@ let live: {
nodesRef: { current: string[] }
nodesProjectIdRef: { current: string | null }
setNodes: (next: string[] | ((ns: string[]) => string[])) => void
+ setEdges: (next: string[] | ((es: string[]) => string[])) => void
} | null = null
function Canvas() {
const active = useActive((s) => s.id)
useOther((s) => s.n)
const [nodes, setNodes] = useState<string[]>([])
- const { nodesRef, nodesProjectIdRef, installEpoch } = useNodesEpoch(nodes)
+ const [edges, setEdges] = useState<string[]>([])
+ const { nodesRef, nodesProjectIdRef, renderedProjectId, installEpoch } = useNodesEpoch(nodes)
+ // Canvas's edge lists: a latest ref beside the state, mirrored under the node mirror's rule.
+ const edgesRef = useRef<string[]>(edges)
+ const edgesMirroredRef = useRef<string[]>(edges)
+ mirrorLatest(edges, edgesMirroredRef, edgesRef, renderedProjectId === nodesProjectIdRef.current)
seen.push({ state: nodes, nodes: nodesRef.current, epoch: nodesProjectIdRef.current })
- live = { nodesRef, nodesProjectIdRef, setNodes }
+ edgeSeen.push({ edges: edgesRef.current, epoch: nodesProjectIdRef.current })
+ live = { nodesRef, nodesProjectIdRef, setNodes, setEdges }
useEffect(() => {
loaded = { id: active, at: seen.length }
const saved = active ? store[active] : undefined
@@ -52,6 +61,10 @@ function Canvas() {
}
const flow = [...saved]
setNodes(flow)
+ // The load writes the edge ref synchronously beside its setter, as Canvas's does.
+ const flowEdges = [`${active}-edge`]
+ edgesRef.current = flowEdges
+ setEdges(flowEdges)
installEpoch(active, flow)
// Canvas subscribes to stores its load effect writes; each write is a SyncLane re-render.
useOther.setState((s) => ({ n: s.n + 1 }))
@@ -108,6 +121,7 @@ async function mount() {
root.render(<Canvas />)
await until(() => seen.at(-1)?.epoch === 'A' && seen.at(-1)?.nodes[0] === 'A-node')
seen = []
+ edgeSeen = []
}
beforeEach(() => {
@@ -184,6 +198,24 @@ it('a discrete update in the switch window never puts the outgoing nodes under t
expect(store.B.every((n) => n.startsWith('B-'))).toBe(true)
})
+// D4: the edge refs follow the node mirror's epoch rule. A discrete `setLinkEdges(fn)` in the switch
+// window renders fn(the OUTGOING project's edges) — a CHANGED array — and a mirror that copied it put
+// A's edges under B's tag, so Canvas's edge publisher diffed B against them and cast spurious
+// `edge-remove`s into B.
+it('a discrete edge update in the switch window never puts the outgoing edges under the incoming tag (D4)', async () => {
+ await mount()
+ useActive.setState({ id: 'B' })
+ await untilWindow('B')
+ flushSync(() => live?.setEdges((es) => [...es, es[0].replace('-edge', '-local')]))
+ await until(() => seen.at(-1)?.epoch === 'B' && !!seen.at(-1)?.state.includes('B-node'))
+ await settle()
+ for (const e of edgeSeen) {
+ if (e.epoch === null) continue
+ expect(e.edges.every((x) => x.startsWith(`${e.epoch}-`)), JSON.stringify(e)).toBe(true)
+ }
+ expect(edgeSeen.at(-1)).toEqual({ edges: ['B-edge', 'B-local'], epoch: 'B' })
+})
+
// Route by the TAG, not only by the active id: a peer's op for the active project belongs on React
// Flow only if React Flow holds that project. After a bail-out (an unknown project) the previous
// nodes stay mounted with the tag cleared; the op must not be mixed into them.
@@ -209,3 +241,4 @@ it('a peer op landing while a local edit is still queued keeps both', async () =
await settle()
expect(seen.at(-1)?.state).toEqual(['A-node', 'A-local', 'A-peer'])
})
+
diff --git a/src/renderer/canvas/nodesEpoch.ts b/src/renderer/canvas/nodesEpoch.ts
index fe65e18d6..839512aab 100644
--- a/src/renderer/canvas/nodesEpoch.ts
+++ b/src/renderer/canvas/nodesEpoch.ts
@@ -63,6 +63,25 @@ export function rebaseOnLatest<N>(base: N[], next: N[], apply: (ns: N[]) => N[])
return (ns) => (ns === base ? next : apply(ns))
}
+/**
+ * The render-time mirror of one more piece of canvas state into its LATEST ref, under the two
+ * conditions `useNodesEpoch` applies to `nodesRef`: copy only when the state CHANGED (a render that
+ * skipped a pending update must not undo a synchronous write), and only when this render belongs to
+ * the latest epoch (`inLatestEpoch` = the render's `renderedProjectId` equals the tag ref), so a
+ * discrete update rendered on top of the OUTGOING project's state never lands under the incoming
+ * project's tag. Canvas's two edge lists use it.
+ */
+export function mirrorLatest<T>(
+ state: T,
+ mirrored: MutableRefObject<T>,
+ latest: MutableRefObject<T>,
+ inLatestEpoch: boolean
+): void {
+ if (state === mirrored.current || !inLatestEpoch) return
+ mirrored.current = state
+ latest.current = state
+}
+
export function useNodesEpoch<N>(nodes: N[]): NodesEpoch<N> {
const [renderedProjectId, setRenderedProjectId] = useState<string | null>(null)
const nodesRef = useRef<N[]>(nodes)
diff --git a/src/shared/canvas-order.test.ts b/src/shared/canvas-order.test.ts
index 107b72336..c7328dff1 100644
--- a/src/shared/canvas-order.test.ts
+++ b/src/shared/canvas-order.test.ts
@@ -460,6 +460,45 @@ describe('createCanvasOrder', () => {
// redrawn, a node deleted and ⌘Z'd, inside one round trip) carries a `seen` below the remove and
// every peer drops it as a stale frame, while we keep showing it: a split. The caller holds such a
// re-creation back while our remove is unacked; this is the question it asks.
+ // D4: `reset()` keeps our causal position for a same-core reconnect, but after a REAL restart the
+ // kept value sat above every seq the new core handed out until its counter caught up, so every
+ // cast read as "never stale" for that whole stretch. The first seq heard after a reset says which
+ // one it was.
+ describe('the causal position after a reset (D4)', () => {
+ it('a first seq at or below it means the core restarted: the position drops to that seq', () => {
+ const me = createCanvasOrder('me')
+ me.accept(up('n2', 0, 'x', 40)) // the old core had reached 40
+ me.reset()
+ me.accept(up('n3', 0, 'x', 2)) // the NEW core's first op
+ const stale = me.stamp(up('n1', 1, 'me', 0)) // a drag frame, cast in ignorance of…
+ expect(stale.seen).toBe(2)
+ // …a teammate's delete the new core ordered at 3: every peer drops the frame, as it would have
+ // before the restart. With the kept 40 it read as "never stale" and resurrected the node.
+ const peer = createCanvasOrder('peer')
+ expect(peer.accept(rm('n1', 'x', 3))).toBe(true)
+ expect(peer.accept({ ...stale, seq: 4 })).toBe(false)
+ })
+
+ it('a first seq above it is the same core carrying on: the position is kept, then rises', () => {
+ const me = createCanvasOrder('me')
+ me.accept(up('n2', 0, 'x', 40))
+ me.reset()
+ expect(me.stamp(up('n1', 1, 'me', 0)).seen).toBe(40) // nothing heard yet: kept
+ me.accept(up('n3', 0, 'x', 41))
+ expect(me.stamp(up('n1', 1, 'me', 0)).seen).toBe(41)
+ })
+
+ it('only the FIRST stamped seq after a reset decides; an unstamped op does not', () => {
+ const me = createCanvasOrder('me')
+ me.accept(up('n2', 0, 'x', 40))
+ me.reset()
+ me.accept(up('n3', 0, 'x', 0)) // unstamped: says nothing about the counter
+ me.accept(up('n4', 0, 'x', 5)) // the new core
+ me.accept(up('n5', 0, 'x', 2)) // a straggler after it moves nothing back
+ expect(me.stamp(up('n1', 1, 'me', 0)).seen).toBe(5)
+ })
+ })
+
describe('a pending local remove (the re-creation gate)', () => {
it('reports a pending local remove until its echo returns', () => {
const o = createCanvasOrder('a')
@@ -512,6 +551,45 @@ describe('createCanvasOrder', () => {
expect(o.hasPendingRemove('n:n1')).toBe(true) // a later remove still counts from one
})
+ // D4: a single `canvas:mut` can be dropped on its way back with the connection kept (the ui
+ // sink's SINK_FAILURE_LIMIT), so "a lost echo comes with a reconnect" was false and the gate
+ // stuck for the session. Echoes come back in the order we cast (FIFO), so the echo of a LATER
+ // cast of ours proves every earlier one that has not arrived was lost.
+ it('the echo of a LATER cast of ours releases an earlier remove whose echo was lost (D4)', () => {
+ const o = createCanvasOrder('me')
+ o.onLocal(rm('n1', 'me', 0)) // its echo is dropped on the way back
+ o.onLocal(up('n2', 1, 'me', 0))
+ expect(o.hasPendingRemove('n:n1')).toBe(true)
+ expect(o.pendingRemoveCount()).toBe(1)
+ o.accept(up('n2', 1, 'me', 8)) // the next echo of ours arrives
+ expect(o.hasPendingRemove('n:n1')).toBe(false)
+ expect(o.pendingRemoveCount()).toBe(0)
+ })
+
+ it('an echo of an EARLIER cast never releases a later remove, and a peer’s op releases nothing (D4)', () => {
+ const o = createCanvasOrder('me')
+ o.onLocal(up('n2', 1, 'me', 0))
+ o.onLocal(rm('n1', 'me', 0))
+ o.accept(up('n2', 1, 'me', 8)) // the echo of the cast BEFORE the remove
+ expect(o.hasPendingRemove('n:n1')).toBe(true)
+ o.accept(up('n3', 1, 'peer', 9))
+ o.accept(rm('n4', 'peer', 10))
+ expect(o.hasPendingRemove('n:n1')).toBe(true)
+ o.accept(rm('n1', 'me', 11)) // its own echo
+ expect(o.hasPendingRemove('n:n1')).toBe(false)
+ })
+
+ it('counts per cast: a lost remove releases its own count only (D4)', () => {
+ const o = createCanvasOrder('me')
+ o.onLocal(rm('n1', 'me', 0)) // lost
+ o.onLocal(rm('n1', 'me', 0)) // arrives
+ o.onLocal(up('n9', 1, 'me', 0)) // arrives
+ o.accept(rm('n1', 'me', 5)) // FIFO: the FIRST n1 cast's echo — the second n1 is still in flight
+ expect(o.hasPendingRemove('n:n1')).toBe(true)
+ o.accept(up('n9', 1, 'me', 6)) // …and this one proves the second n1 echo lost
+ expect(o.hasPendingRemove('n:n1')).toBe(false)
+ })
+
it('isRemoveOp is the order’s own remove predicate, nodes and edges alike', () => {
expect(isRemoveOp(rm('n1', 'me', 0))).toBe(true)
expect(isRemoveOp({ op: 'edge-remove', kind: 'rope', id: 'r1' })).toBe(true)
diff --git a/src/shared/canvas-order.ts b/src/shared/canvas-order.ts
index b7b6e6b21..0c97e4d65 100644
--- a/src/shared/canvas-order.ts
+++ b/src/shared/canvas-order.ts
@@ -98,6 +98,14 @@ export const PENDING_TTL_MS = 5000
*/
export const REMOVED_MAX = 512
+/**
+ * How many of our own casts the FIFO record behind the re-creation gate's release keeps (see
+ * `hasPendingRemove`). A cast leaves it the moment its echo, or a later one, comes back, so it only
+ * grows while the core answers nothing at all — and then the oldest entry is simply forgotten (the
+ * behaviour before the release existed: that remove's gate waits for its own echo or a reset).
+ */
+export const LOCAL_CASTS_MAX = 4096
+
/**
* The THING a mutation addresses — the key everything below orders by.
*
@@ -208,16 +216,23 @@ export interface CanvasOrder {
* lands, when `stamp` puts the remove in its `seen`.
*
* Counted per key, not a flag: two removes in flight (delete, undo, delete) release on the second
- * ack. Not TTL-bound, unlike rule 2's suppression: a late ack is exactly when the gate matters, and
- * a lost one comes with a reconnect, whose `reset` clears it.
+ * ack. Not TTL-bound, unlike rule 2's suppression: a late ack is exactly when the gate matters. A
+ * LOST ack does not come with a reconnect: one `canvas:mut` can be dropped on its way back while
+ * the connection stays up (the ui sink's per-message failure limit). So a remove is also released
+ * when the echo of a LATER cast of ours arrives: echoes come back in the order we cast them (FIFO),
+ * so that echo proves the earlier one is not coming. A reset still clears everything.
*
* The PROJECT rides in the key: callers build it with `mutationKey(m, projectId)` — the same
* function `onLocal` keys with — so the gate is project-scoped wherever a key is. (Only a
* remove-class op can open the gate, and none of those has a project-scoped key today.)
*/
hasPendingRemove(key: string): boolean
+ /** How many of our remove-class casts are unacked, over every key: a caller that holds re-creations
+ * compares it across an `accept`, because an echo can release a DIFFERENT key's gate (FIFO). */
+ pendingRemoveCount(): number
/** Forget the per-connection order state on a genuine reconnect (see `createReconnectWatch`) —
- * everything except our causal position (`lastSeq`, what `stamp` puts in `seen`). */
+ * everything except our causal position (`lastSeq`, what `stamp` puts in `seen`), which the first
+ * stamped `seq` heard after it re-bases if the core restarted. */
reset(): void
}
@@ -270,6 +285,13 @@ export function createCanvasOrder(
/** The re-creation gate (`hasPendingRemove`): per key, how many of our remove-class casts are
* still unacked. Kept apart from `pending`, which counts EVERY cast of ours for the key. */
const pendingRemoves = new Map<string, number>()
+ /** The sum of `pendingRemoves`, for `pendingRemoveCount`. */
+ let pendingRemoveTotal = 0
+ /** Our own casts in the order we made them, `{ key, remove }` — what an echo is matched against
+ * (FIFO) to prove an earlier echo was lost. Capped at LOCAL_CASTS_MAX. */
+ const casts: Array<{ key: string; remove: boolean }> = []
+ /** Set by `reset`: the next stamped `seq` says whether the core restarted (see `accept`). */
+ let rebase = false
/**
* The highest `seq` we have applied or deliberately dropped, across ALL nodes — our causal
* position in the total order, and what `stamp` puts on every mutation we cast. Global, not per
@@ -279,6 +301,15 @@ export function createCanvasOrder(
*/
let lastSeq = 0
+ /** Draw down one pending remove of `key` (an echo, or proof its echo was lost). */
+ const releaseRemove = (key: string): void => {
+ const r = pendingRemoves.get(key)
+ if (r === undefined) return
+ pendingRemoveTotal--
+ if (r <= 1) pendingRemoves.delete(key)
+ else pendingRemoves.set(key, r - 1)
+ }
+
const noteRemoved = (id: string, seq: number): void => {
removed.delete(id) // re-insert so the Map's iteration order stays the LRU order
removed.set(id, seq)
@@ -333,7 +364,13 @@ export function createCanvasOrder(
} else {
pending.set(id, { count: 1, since: now() })
}
- if (isRemove(m)) pendingRemoves.set(id, (pendingRemoves.get(id) ?? 0) + 1)
+ const remove = isRemove(m)
+ if (remove) {
+ pendingRemoves.set(id, (pendingRemoves.get(id) ?? 0) + 1)
+ pendingRemoveTotal++
+ }
+ casts.push({ key: id, remove })
+ if (casts.length > LOCAL_CASTS_MAX) casts.shift()
// A fresh local edit IS an optimistic value on our canvas again, so rule 1 is sound for this
// node once more and an older echo of ours must not be replayed over it. (This one's own echo
// will be dropped as the ack it is; it carries what we already show.)
@@ -347,6 +384,14 @@ export function createCanvasOrder(
// `seq` 0 means an unstamped mutation (no reflector in the path) — never treat it as stale.
const current = seq === 0 || seq > highest
if (seq > highest) seen.set(id, seq)
+ // After a reset, the first STAMPED seq says which reconnect it was: at or below our position,
+ // the core restarted and its counter began again, so our position is that seq (kept, it would
+ // make every cast read "never stale" until the new counter caught up); above it, the same core
+ // carried on and the position simply rises below.
+ if (rebase && seq > 0) {
+ rebase = false
+ if (seq <= lastSeq) lastSeq = seq
+ }
// Our causal position for the mutations WE cast next (rule 4). Advanced for EVERYTHING we
// process — our own echo included, and a straggler too: both prove the order has reached at
// least that far, and "what did this client know" says nothing about who sent it.
@@ -374,12 +419,14 @@ export function createCanvasOrder(
superseded.delete(id) // every cast of ours is accounted for; the node is settled
}
// The re-creation gate: this remove is now in our `seen` (above), so a re-creation of the
- // key stamped from here on carries it. Only an entry we are tracking is drawn down — an echo
- // straggling in after a reset must not push a later remove's count below one.
- const r = isRemove(m) ? pendingRemoves.get(id) : undefined
- if (r !== undefined) {
- if (r <= 1) pendingRemoves.delete(id)
- else pendingRemoves.set(id, r - 1)
+ // key stamped from here on carries it. Matched against our casts in the order we made them:
+ // this echo is the one for the FIRST cast of this key still recorded, and every cast recorded
+ // before it has had its echo lost (FIFO) — a remove among them will never be acked, so its
+ // gate is released too. An echo we have no record of (straggling in after a reset) releases
+ // nothing, so it cannot push a later remove's count below one.
+ const at = casts.findIndex((c) => c.key === id)
+ if (at !== -1) {
+ for (const c of casts.splice(0, at + 1)) if (c.remove) releaseRemove(c.key)
}
return repair
}
@@ -407,11 +454,17 @@ export function createCanvasOrder(
return (pendingRemoves.get(key) ?? 0) > 0
},
+ pendingRemoveCount() {
+ return pendingRemoveTotal
+ },
+
reset() {
seen.clear()
pending.clear()
superseded.clear()
pendingRemoves.clear()
+ pendingRemoveTotal = 0
+ casts.length = 0
// The core may have restarted at seq 0 — a `removed` entry stamped with the OLD counter would
// then outrank every new mutation and blackhole that node.
removed.clear()
@@ -419,10 +472,12 @@ export function createCanvasOrder(
// clientId, `seq` carrying on), and zeroing it stamped our first cast after it `seen: 0` — a
// ⌘Z of a node deleted before the drop was then a stale frame to every peer holding the
// tombstone, and our own echo of it is no repair: a persistent split. If the core REALLY
- // restarted, the kept value is above every `seq` it hands out, so our casts read as "never
- // stale" there (`supersededByRemove`: it is ≥ every new tombstone) — the pre-rule-4 verdict. A
- // degrade, never a split. The reflector's clamp to `seq - 1` (canvas-sync `stampMutation`)
- // changes no verdict here; it is hygiene.
+ // restarted, the kept value is above every `seq` it hands out, so until we hear from it our
+ // casts read as "never stale" there (`supersededByRemove`: it is ≥ every new tombstone) — the
+ // pre-rule-4 verdict, a degrade, never a split. It is BOUNDED: the first stamped `seq` we hear
+ // after this re-bases the position (`rebase`, in `accept`). The reflector's clamp to `seq - 1`
+ // (canvas-sync `stampMutation`) changes no verdict here; it is hygiene.
+ rebase = true
}
}
}