Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
test(spec): crash and recovery in the refresh-barrier model
garrytan98bbab202beb
diff --git a/spec/tla/README.md b/spec/tla/README.mdindex 49b35cd53f..dd8bce3bc4 100644--- a/spec/tla/README.md+++ b/spec/tla/README.md@@ -151,7 +151,7 @@ With `persistence.refresh_barrier` on, a page write admitted while a managed refbehind that refresh and published by itself once the refresh is released (docs/designs/REFRESH_WRITE_BARRIER.md). Themodel holds one managed worktree with one page P whose canonical file is F, single-page writes to P (one blindput_page, one with expected_revision), one refresh at a time, another agent's plain-git commit to F, managed sync, the-owner's expiry, and Postgres's four row-lock modes per transaction (`KS`, `S`, `NKU`, `U`) over the brain row,+owner's expiry, the refreshing process killed at any step and recovered by `recoverOne`, and Postgres's four row-lock modes per transaction (`KS`, `S`, `NKU`, `U`) over the brain row,the worktree, the source row, the counters, each refresh row and each request row. A transaction holds its locks untilit commits; a waits-for cycle among them is a deadlock (Postgres aborts one side with 40P01), so `NoDeadlock` is aninvariant. Behind constants it also models the stacked follow-ups: writes held for an import and deferral behind@@ -159,14 +159,18 @@ invariant. Behind constants it also models the stacked follow-ups: writes held f| Config | What it checks || --- | --- |-| `RefreshBarrier.cfg` | #8682 as coded: two writes (one blind), one refresh, one foreign commit: every safety property. |+| `RefreshBarrier.cfg` | #8682, with the fix for the fenced-orphan finding below modelled: two writes (one blind), one refresh, one foreign commit: every safety property. || `RefreshBarrierEnv.cfg` | The same with a shared-skill restore and a credential revocation that may each run once, no foreign commit. || `RefreshBarrierLiveness.cfg` | `FairSpec`, one write: every deferred write leaves deferral and every admitted write ends. `RefreshBarrierLivenessNightly.cfg` checks two writes (nightly). || `RefreshBarrierAdopt.cfg` | #8708 on #8705 with its fixes: an uncommitted local edit adopted by a drift refusal, no foreign commit. `RefreshBarrierAdoptNightly.cfg` adds the foreign commit (nightly). || `RefreshBarrierImport.cfg` | #8705 with its fixes (03b89121): writes held for an import, a later write held beside them, the import expiry, deferral behind `recovery_required`: every property. |+Every config kills the refreshing process once (`MaxCrashes = 1`), at any step, after which `recoverOne` resumes the+refresh row it left. A refresh that finds HEAD moved at the merge aborts and the tick retries, as `fenceAndMerge` does,+so `recovery_required` is reached only through recovery.+`test/spec-refresh-barrier.test.ts` keeps the default config on #8682 alone, every safety config on every invariant,-and the workflow's path filter covering every file a mutant names.+every config killing the refresh at least once, and the workflow's path filter covering every file a mutant names.### Properties@@ -179,6 +183,7 @@ and the workflow's path filter covering every file a mutant names.| `FenceNeverMidPrepare` | the fence never comes up, and the checkout never moves, while a write is claimed || `DeferredWaitsForRelease`, `RefreshChangeNeverOverwritten` | a deferred write commits only after its refresh is released, and never over a file that refresh changed || `NoDeferToReleased`, `ReleasedWhenEnded` | admission never defers to a released refresh; a completed or aborted refresh is released |+| `RecoveryOnlyByPolicy` | `recoverOne` leaves a refresh `recovery_required` (a person must act while every write deferred to it waits) only in one of two policy cases, recorded in `rwhy`: a commit gbrain did not make landed before the merge (`foreign_in_window`), or HEAD moved past the target between the merge and `bookkeeping` (`after_merge`); never over gbrain's own commits alone || `ExpiryOnlyWhileWaiting` | an expiry ends only a write that still waits (its refresh unreleased, or its import not landed) || `RestoreLeavesNoOpenWrite` | a restore leaves no write admitted before it open (finding C) || `NoApplyAfterRevoke` | no write commits after its credential was revoked |@@ -197,6 +202,7 @@ and the workflow's path filter covering every file a mutant names.| `ConsAdopt`, `ConsRelease` | #8708 `adoptLocalEdit`, then the claim's release || `ConsDone` | `completeWrite` of a refusal || `RefBegin` … `CompleteCommit` | worktree-refresh.ts: `fenceDraining`, the drain poll, `fenceAndMerge` (re-counts under `lockTopologyRows`), the ff-only merge under the commit lock, `bookkeeping`, `syncMembers`, `resumeOrAbandon`, and the v292 release trigger on completed or aborted |+| `Crash`, `RecoverBegin`, `RecoverCommit` | the refreshing process killed (its open transaction rolls back), then `recoverOne` from the owner's topology scan or `--resume`: draining aborts; fenced with HEAD at old_head aborts, and so does a fenced row whose HEAD moved only through gbrain's own commits with nothing merged (`movedBeforeMerge`, the ledger naming each commit: `ownAt`); fenced with HEAD at the target runs `bookkeeping`; syncing with HEAD at or past the target stays syncing; anything else is `markRecoveryRequired` || `ExpRefBegin`, `ExpRefCommit` | `expireDeferredWrites`: the protocol, the refresh row re-read `released_at IS NULL FOR NO KEY UPDATE`, `completeWrite` || `ExpImpBegin`, `ExpImpRecheck`, `ExpImpCommit` | #8705's import expiry in `expireDeferredWrites` || `SyncBegin` … `SyncStamp` | a managed sync pass, whose rows go through the claim like writes: no write passes the queued row importing its page or a running sync row, and a sync row waits behind earlier writes and is never claimed while a write runs; the page row, then its checkpoint row's `UPDATE sources SET last_commit` (sync-prepare.ts) |@@ -225,9 +231,23 @@ and the workflow's path filter covering every file a mutant names.| `hold-beside-held-off` | guard | `CommitsInAdmissionOrder` (Import) | #8705's compareFileTarget held-first read; model only until #8705 is on master || `import-expiry-locks-source-first` | guard | `NoDeadlock` (Import) | #8705's import expiry; model only until #8705 is on master || `import-expiry-skips-source-lock` | guard | `ExpiryOnlyWhileWaiting` (Import) | the same |+| `resume-marks-unmerged-recovery` | guard | `RecoveryOnlyByPolicy` | `recoverOne` as #8682 coded it (the finding below); model only until the fix is on master |#8705's and #8708's guards get code edits and tests (#8705's repro tests in test/import-write-barrier.test.ts) once their code is on master; until then they are checked in the model only.+### Found in #8682: a refresh killed at fenced after HEAD moved++A flag-on eval found it first, and the model reproduces it (`resume-marks-unmerged-recovery`, `RecoveryOnlyByPolicy`).+The precheck records `old_head` before the drain. A write already queued drains and commits, and its ledger commit moves+HEAD past `old_head`; the fence comes up, and the process is killed before the merge. `recoverOne` then finds a fenced+row whose HEAD is neither `old_head` nor the target and marks it `recovery_required`. The owner's scan never resumes that+state, `--resume` refuses, and the documented way out resets the checkout to `old_head`, which drops the drained write's+commit; until a person acts, every write deferred to the refresh waits. Another agent's commit in the same window is a policy case+instead: the live tick aborts and retries on either, but recovery cannot tell that commit from a person's, so it+stays `recovery_required` (`foreign_in_window`), as test 7 in test/persistence-worktree-refresh.test.ts pins. The fix+(#8705) aborts an orphaned fenced row whose HEAD moved past `old_head` only through commits the commit ledger names, with+nothing of the target merged (`FencedAbortable`); every config passes with it, and both policy cases stay reachable.+### Races found in #8705This model found two races in #8705's first version; 03b89121 fixed both, with the repro tests indiff --git a/spec/tla/refresh-barrier/RefreshBarrier.cfg b/spec/tla/refresh-barrier/RefreshBarrier.cfgindex d959bf5d7d..1467bfc2b5 100644--- a/spec/tla/refresh-barrier/RefreshBarrier.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrier.cfg@@ -1,4 +1,5 @@-\* #8682 as coded: the barrier, its expiry and restore; two writes (one blind), one refresh, one foreign commit.+\* #8682: the barrier, its expiry and restore; two writes (one blind), one refresh, one foreign commit, and the refreshing+\* process killed once at any step and recovered by recoverOne (with the fix for the fenced-orphan finding modelled).SPECIFICATION SpecCHECK_DEADLOCK FALSECONSTANTS@@ -14,6 +15,7 @@ CONSTANTSAdoptLocalEdits = FALSEMayRestore = FALSEMayRevoke = FALSE+ MaxCrashes = 1INVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOnce@@ -25,6 +27,7 @@ INVARIANT DeferredWaitsForReleaseINVARIANT RefreshChangeNeverOverwrittenINVARIANT NoDeferToReleasedINVARIANT ReleasedWhenEnded+INVARIANT RecoveryOnlyByPolicyINVARIANT ExpiryOnlyWhileWaitingINVARIANT RestoreLeavesNoOpenWriteINVARIANT NoApplyAfterRevokediff --git a/spec/tla/refresh-barrier/RefreshBarrier.tla b/spec/tla/refresh-barrier/RefreshBarrier.tlaindex d463e7e80c..121909e149 100644--- a/spec/tla/refresh-barrier/RefreshBarrier.tla+++ b/spec/tla/refresh-barrier/RefreshBarrier.tla@@ -12,6 +12,9 @@(* coordinator.ts publishMutation *)(* refresh worktree-refresh.ts fenceDraining, fenceAndMerge, *)(* bookkeeping, syncMembers, resumeOrAbandon, transition *)+(* recovery worktree-refresh.ts recoverOne, from the owner's topology *)+(* scan (resumeWorktreeRefreshes) or --resume, after the *)+(* refreshing process is killed at any step (MaxCrashes) *)(* expiry refresh-barrier.ts expireDeferredWrites *)(* restore shared-skills/restore.ts quarantineSharedSkillRestore *)(* and, behind constants, the stacked follow-ups: ImportHolds and *)@@ -46,7 +49,8 @@ CONSTANTSRecoveryDefers, \* #8705: admission defers behind a refresh in recovery_required instead of refusingAdoptLocalEdits, \* #8708: a drift refusal on a one-sided local edit adopts the edit and prepares the write againMayRestore, \* a shared-skill restore may run once- MayRevoke \* the writers' credential may be revoked once+ MayRevoke, \* the writers' credential may be revoked once+ MaxCrashes \* times the refreshing process may be killed; the owner's topology scan or --resume then recovers itWrites == 1..NWRefs == 1..MaxRefreshes@@ -103,10 +107,13 @@ VARIABLESrchg, \* the refresh's fast-forward changes Frhead, \* old_headrmoved, \* the refresh changed F in the checkout (git diff old_head target_head names F)+ rtgt, \* target_head as a commit index once it is in the checkout (0 before the merge)+ rwhy, \* why recoverOne marked it recovery_required (the policy case), "none" until it doescur, \* the active refresh (None when none)nref,\* git and the pagehead, verAt, \* HEAD's commit index; the version of F each commit holds+ ownAt, \* each commit is gbrain's own (the commit ledger names it: a write's or an adoption's git effect)fileV, \* F as committed at HEADdbV, \* the page's current revision (as a version of F)hist, \* every version that was ever a revision of the page (page history)@@ -115,6 +122,7 @@ VARIABLESnver, nforeign, nlocal,\* environmentrestored, revoked, preRestore,+ ncrash, \* times the refreshing process was killed\* ghosts for safety properties that talk about a momenthidden, \* a write replaced a committed version of F the page never hadbadExpiry, \* a write ended *_expired although it no longer waited@@ -128,9 +136,9 @@ VARIABLESsseq \* the admission sequence of the sync pass's queued rows (0 when no pass runs)reqv == <<st, code, seq, nseq, dref, dimp, applied, cpos, ncommit, expRev, overdue, adopted>>-refv == <<rstate, rel, rchg, rhead, rmoved, cur, nref>>-gitv == <<head, verAt, fileV, dbV, hist, lastCommit, local, nver, nforeign, nlocal>>-envv == <<restored, revoked, preRestore>>+refv == <<rstate, rel, rchg, rhead, rmoved, rtgt, rwhy, cur, nref>>+gitv == <<head, verAt, ownAt, fileV, dbV, hist, lastCommit, local, nver, nforeign, nlocal>>+envv == <<restored, revoked, preRestore, ncrash>>ghostv == <<hidden, badExpiry, badRelease, badDefer, badRevoke>>lockv == <<held, want>>procv == <<pc, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>@@ -192,6 +200,12 @@ ReleaseOn(s) == s \in {"completed", "aborted"}RestoreCancels == {"queued", "deferred", "running"}\* adoptionIntent: only a one-sided edit (HEAD still holds the page's copy) is adopted.AdoptOneSided == fileV = dbV+\* Every commit on HEAD since the refresh's old_head is gbrain's own (ownCommits: the commit ledger names it).+OwnSince(i) == \A c \in (rhead[i] + 1)..head : ownAt[c]+\* recoverOne's movedBeforeMerge: it aborts an orphaned fenced refresh whose HEAD moved past old_head only through gbrain's+\* own commits (the drained writes' ledger commits), with nothing of the target merged. memberOutsideHead is FALSE here:+\* the model's history is one line.+FencedAbortable(i) == rtgt[i] = 0 /\ OwnSince(i)\* ---------------------------------------------------------------------------@@ -231,10 +245,13 @@ Init ==/\ rchg = [i \in Refs |-> FALSE]/\ rhead = [i \in Refs |-> 0]/\ rmoved = [i \in Refs |-> FALSE]+ /\ rtgt = [i \in Refs |-> 0]+ /\ rwhy = [i \in Refs |-> "none"]/\ cur = None/\ nref = 0/\ head = 1/\ verAt = [c \in 1..MaxHead |-> IF c = 1 THEN 1 ELSE 0]+ /\ ownAt = [c \in 1..MaxHead |-> FALSE]/\ fileV = 1/\ dbV = 1/\ hist = {1}@@ -246,6 +263,7 @@ Init ==/\ restored = FALSE/\ revoked = FALSE/\ preRestore = {}+ /\ ncrash = 0/\ hidden = FALSE/\ badExpiry = FALSE/\ badRelease = FALSE@@ -280,8 +298,9 @@ ConsIdle == /\ pc' = [pc EXCEPT ![CONS] = "idle"] /\ cw' = 0Commit(p) == held' = [held EXCEPT ![p] = NoLocks]Go(p, s) == pc' = [pc EXCEPT ![p] = s]Need(p, ls) == want' = [want EXCEPT ![p] = ls]-NewCommit(v) == /\ head' = head + 1- /\ verAt' = [verAt EXCEPT ![head + 1] = v]+NewCommit(v, mine) == /\ head' = head + 1+ /\ verAt' = [verAt EXCEPT ![head + 1] = v]+ /\ ownAt' = [ownAt EXCEPT ![head + 1] = mine]/\ fileV' = v\* ------------------------------- admission -------------------------------@@ -331,7 +350,7 @@ AdmCommit(w) ==ELSE UNCHANGED <<rstate, rel, cur>>/\ Commit(ADM(w)) /\ Go(ADM(w), "idle") /\ Need(ADM(w), <<>>)/\ aref' = [aref EXCEPT ![w] = None] /\ aread' = [aread EXCEPT ![w] = "none"]- /\ UNCHANGED <<code, dimp, applied, cpos, ncommit, expRev, overdue, adopted, rchg, rhead, rmoved, nref, gitv, envv,+ /\ UNCHANGED <<code, dimp, applied, cpos, ncommit, expRev, overdue, adopted, rchg, rhead, rmoved, rtgt, rwhy, nref, gitv, envv,hidden, badExpiry, badRelease, badRevoke, cw, cprep, cfin, ew, passHead, sseq>>\* -------------------------------- consumer --------------------------------@@ -407,7 +426,7 @@ ConsPub ==v == nver + 1IN IF mine /\ ~denied /\ ~movedTHEN /\ hidden' = (hidden \/ fileV \notin hist)- /\ NewCommit(v) /\ nver' = v /\ dbV' = v /\ hist' = hist \cup {v} /\ local' = None+ /\ NewCommit(v, TRUE) /\ nver' = v /\ dbV' = v /\ hist' = hist \cup {v} /\ local' = None/\ applied' = [applied EXCEPT ![cw] = @ + 1]/\ st' = [st EXCEPT ![cw] = "committed"] /\ code' = [code EXCEPT ![cw] = "committed"]/\ ncommit' = ncommit + 1 /\ cpos' = [cpos EXCEPT ![cw] = ncommit + 1]@@ -415,7 +434,7 @@ ConsPub ==/\ badRevoke' = (badRevoke \/ revoked)ELSE /\ st' = [st EXCEPT ![cw] = IF ~mine THEN @ ELSE IF denied THEN "failed" ELSE "queued"]/\ code' = [code EXCEPT ![cw] = IF mine /\ denied THEN "permission_denied" ELSE @]- /\ UNCHANGED <<head, verAt, fileV, nver, dbV, hist, local, applied, ncommit, cpos, hidden, badRelease, badRevoke>>+ /\ UNCHANGED <<head, verAt, ownAt, fileV, nver, dbV, hist, local, applied, ncommit, cpos, hidden, badRelease, badRevoke>>/\ Commit(CONS) /\ ConsIdle/\ UNCHANGED <<seq, nseq, dref, dimp, expRev, overdue, adopted, refv, lastCommit, nforeign, nlocal, envv, badExpiry, badDefer,want, ew, aref, aread, passHead, sseq>>@@ -436,7 +455,7 @@ ConsHold ==ConsAdopt ==/\ pc[CONS] = "c_adopt"/\ hidden' = (hidden \/ fileV \notin hist)- /\ NewCommit(local) /\ dbV' = local /\ hist' = hist \cup {local} /\ local' = None+ /\ NewCommit(local, TRUE) /\ dbV' = local /\ hist' = hist \cup {local} /\ local' = None/\ adopted' = [adopted EXCEPT ![cw] = TRUE]/\ Need(CONS, <<L(REQ(cw), "U")>>) /\ Go(CONS, "c_release")/\ UNCHANGED <<st, code, seq, nseq, dref, dimp, applied, cpos, ncommit, expRev, overdue, refv, lastCommit, nver, nforeign,@@ -475,7 +494,7 @@ FdCommit ==/\ rhead' = [rhead EXCEPT ![nref + 1] = head]/\ cur' = nref + 1 /\ nref' = nref + 1/\ Commit(RFR) /\ Go(RFR, "r_drain")- /\ UNCHANGED <<reqv, rel, rmoved, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rel, rmoved, rtgt, rwhy, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* The drain poll (no locks), then fenceAndMerge's transaction: lockTopologyRows, drainCounts, transition to fenced.RefDrained ==@@ -495,7 +514,7 @@ FmCommit ==THEN /\ rstate' = [rstate EXCEPT ![nref] = "fenced"] /\ Go(RFR, "r_merge") /\ Need(RFR, <<>>)ELSE /\ UNCHANGED rstate /\ Go(RFR, "r_abort") /\ Need(RFR, <<L(REF(nref), "NKU")>>) \* refresh_source_changed/\ Commit(RFR)- /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, cur, nref, gitv, envv, ghostv, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, rtgt, rwhy, cur, nref, gitv, envv, ghostv, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* transition(..., 'aborted'): the release trigger stamps released_at.AbortCommit ==@@ -505,32 +524,28 @@ AbortCommit ==ELSE UNCHANGED <<rstate, rel>>/\ cur' = None/\ Commit(RFR) /\ Go(RFR, "idle")- /\ UNCHANGED <<reqv, rchg, rhead, rmoved, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, rwhy, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>-\* Under the commit lock: HEAD must still be old_head (another agent's plain-git commit is not), then the ff-only merge.+\* Under the commit lock: HEAD must still be old_head, then the ff-only merge. A HEAD that moved (a drained write's ledger+\* commit, or another agent's plain-git commit) aborts the refresh, refresh_source_changed, and the tick retries.RefMerge ==/\ Ready(RFR, "r_merge")/\ IF head # rhead[nref]- THEN /\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_mark") /\ UNCHANGED <<head, verAt, fileV, nver, rmoved>>+ THEN /\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_abort") /\ UNCHANGED <<head, verAt, ownAt, fileV, nver, rmoved, rtgt>>ELSE /\ IF rchg[nref]- THEN NewCommit(nver + 1) /\ nver' = nver + 1 /\ rmoved' = [rmoved EXCEPT ![nref] = TRUE]- ELSE NewCommit(fileV) /\ UNCHANGED <<nver, rmoved>>+ THEN NewCommit(nver + 1, FALSE) /\ nver' = nver + 1 /\ rmoved' = [rmoved EXCEPT ![nref] = TRUE]+ ELSE NewCommit(fileV, FALSE) /\ UNCHANGED <<nver, rmoved>>+ /\ rtgt' = [rtgt EXCEPT ![nref] = head + 1]/\ Need(RFR, <<L(REF(nref), "NKU"), L(SRC, "NKU")>>) /\ Go(RFR, "r_bk")- /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, cur, nref, dbV, hist, lastCommit, local, nforeign, nlocal, envv, ghostv,+ /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rwhy, cur, nref, dbV, hist, lastCommit, local, nforeign, nlocal, envv, ghostv,held, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>-MarkCommit ==- /\ Ready(RFR, "r_mark")- /\ rstate' = [rstate EXCEPT ![nref] = "recovery"]- /\ Commit(RFR) /\ Go(RFR, "r_recovery")- /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, cur, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>-\* A person resolves recovery_required: reset to old_head and --abandon, or reset to the target and --resume.PersonAbandon ==/\ pc[RFR] = "r_recovery"/\ head' = rhead[nref] /\ fileV' = verAt[rhead[nref]] /\ local' = None/\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_abort")- /\ UNCHANGED <<reqv, refv, verAt, dbV, hist, lastCommit, nver, nforeign, nlocal, envv, ghostv, held,+ /\ UNCHANGED <<reqv, refv, verAt, ownAt, dbV, hist, lastCommit, nver, nforeign, nlocal, envv, ghostv, held,cw, cprep, cfin, ew, aref, aread, passHead, sseq>>PersonResume ==@@ -538,11 +553,13 @@ PersonResume ==/\ LET base == rhead[nref]v == IF rchg[nref] THEN nver + 1 ELSE verAt[base]IN /\ head' = base + 1 /\ verAt' = [verAt EXCEPT ![base + 1] = v] /\ fileV' = v+ /\ ownAt' = [ownAt EXCEPT ![base + 1] = FALSE]/\ nver' = IF rchg[nref] THEN nver + 1 ELSE nver/\ rmoved' = [rmoved EXCEPT ![nref] = rchg[nref]]+ /\ rtgt' = [rtgt EXCEPT ![nref] = base + 1]/\ local' = None/\ Need(RFR, <<L(REF(nref), "NKU"), L(SRC, "NKU")>>) /\ Go(RFR, "r_bk")- /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, cur, nref, dbV, hist, lastCommit, nforeign, nlocal, envv, ghostv, held,+ /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rwhy, cur, nref, dbV, hist, lastCommit, nforeign, nlocal, envv, ghostv, held,cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* bookkeeping: transition to merged (the refresh row), the sources' upstream observation, then syncing.@@ -550,7 +567,7 @@ BkCommit ==/\ Ready(RFR, "r_bk")/\ rstate' = [rstate EXCEPT ![nref] = IF @ \in {"fenced", "merged", "recovery"} THEN "syncing" ELSE @]/\ Commit(RFR) /\ Go(RFR, "r_syncing")- /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, cur, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, rtgt, rwhy, cur, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* syncMembers ends: completed once members reach the target, or (#8072) when a member's sync is blocked or deferred.RefComplete ==@@ -565,7 +582,42 @@ CompleteCommit ==/\ cur' = NoneELSE UNCHANGED <<rstate, rel, cur>>/\ Commit(RFR) /\ Go(RFR, "idle")- /\ UNCHANGED <<reqv, rchg, rhead, rmoved, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, rwhy, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>++\* The refreshing process is killed: its open transaction rolls back and its native locks go. A row it inserted stays+\* in whatever state the last committed transition left it.+Crash ==+ /\ ncrash < MaxCrashes /\ pc[RFR] \notin {"idle", "r_crashed", "r_recovery"}+ /\ ncrash' = ncrash + 1+ /\ held' = [held EXCEPT ![RFR] = NoLocks] /\ Need(RFR, <<>>)+ /\ Go(RFR, IF cur # None THEN "r_crashed" ELSE "idle")+ /\ UNCHANGED <<reqv, refv, gitv, restored, revoked, preRestore, ghostv, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>++\* recoverOne (the owner's topology scan, or the next refresh's --resume path): read HEAD, then one transition.+RecoverBegin ==+ /\ pc[RFR] = "r_crashed"+ /\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_resume")+ /\ UNCHANGED <<reqv, refv, gitv, envv, ghostv, held, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>++RecoverCommit ==+ /\ Ready(RFR, "r_resume")+ /\ LET s == rstate[nref]+ IN CASE s \notin RefActive -> \* another process finished it (the admission that completes a synced refresh)+ /\ UNCHANGED <<rstate, rel, rwhy, cur>> /\ Go(RFR, "idle") /\ Need(RFR, <<>>)+ [] s = "draining" \/ (s = "fenced" /\ (head = rhead[nref] \/ FencedAbortable(nref))) ->+ /\ rstate' = [rstate EXCEPT ![nref] = "aborted"] /\ rel' = [rel EXCEPT ![nref] = ReleaseOn("aborted")]+ /\ cur' = None /\ Go(RFR, "idle") /\ Need(RFR, <<>>) /\ UNCHANGED rwhy+ [] s = "fenced" /\ rtgt[nref] # 0 /\ head = rtgt[nref] -> \* bookkeeping+ /\ UNCHANGED <<rstate, rel, rwhy, cur>> /\ Go(RFR, "r_bk") /\ Need(RFR, <<L(REF(nref), "NKU"), L(SRC, "NKU")>>)+ [] s = "syncing" /\ rtgt[nref] # 0 /\ head >= rtgt[nref] -> \* the target or a descendant: the member syncs finish it+ /\ UNCHANGED <<rstate, rel, rwhy, cur>> /\ Go(RFR, "r_syncing") /\ Need(RFR, <<>>)+ [] OTHER -> \* markRecoveryRequired+ /\ rstate' = [rstate EXCEPT ![nref] = "recovery"] /\ UNCHANGED <<rel, cur>>+ /\ rwhy' = [rwhy EXCEPT ![nref] = IF rtgt[nref] # 0 THEN "after_merge"+ ELSE IF ~OwnSince(nref) THEN "foreign_in_window" ELSE "own_commits"]+ /\ Go(RFR, "r_recovery") /\ Need(RFR, <<>>)+ /\ Commit(RFR)+ /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, nref, gitv, envv, ghostv, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* --------------------------------- expiry ---------------------------------\* The candidate query reads without locks; then one transaction per row.@@ -637,7 +689,7 @@ SyncImport ==THEN dbV' = fileV /\ hist' = hist \cup {fileV}ELSE UNCHANGED <<dbV, hist>> \* nothing to import, or a dirty file sync holds/\ Commit(SYN) /\ Go(SYN, "s_ckpt")- /\ UNCHANGED <<reqv, refv, head, verAt, fileV, lastCommit, local, nver, nforeign, nlocal, envv, ghostv,+ /\ UNCHANGED <<reqv, refv, head, verAt, ownAt, fileV, lastCommit, local, nver, nforeign, nlocal, envv, ghostv,want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* The checkpoint row is claimed once no earlier write is ahead of it; the pass stamp moves last_commit (UPDATE sources).@@ -650,19 +702,19 @@ SyncStamp ==/\ Ready(SYN, "s_stamp")/\ lastCommit' = passHead/\ Commit(SYN) /\ Go(SYN, "idle") /\ passHead' = 0 /\ sseq' = 0- /\ UNCHANGED <<reqv, refv, head, verAt, fileV, dbV, hist, local, nver, nforeign, nlocal, envv, ghostv,+ /\ UNCHANGED <<reqv, refv, head, verAt, ownAt, fileV, dbV, hist, local, nver, nforeign, nlocal, envv, ghostv,want, cw, cprep, cfin, ew, aref, aread>>\* ------------------------------- environment -------------------------------Foreign ==/\ nforeign < MaxForeign /\ local = None- /\ NewCommit(nver + 1) /\ nver' = nver + 1 /\ nforeign' = nforeign + 1+ /\ NewCommit(nver + 1, FALSE) /\ nver' = nver + 1 /\ nforeign' = nforeign + 1/\ UNCHANGED <<reqv, refv, dbV, hist, lastCommit, local, nlocal, envv, ghostv, lockv, procv>>LocalEdit ==/\ nlocal < MaxLocalEdits /\ local = None /\ cur = None/\ local' = nver + 1 /\ nver' = nver + 1 /\ nlocal' = nlocal + 1- /\ UNCHANGED <<reqv, refv, head, verAt, fileV, dbV, hist, lastCommit, nforeign, envv, ghostv, lockv, procv>>+ /\ UNCHANGED <<reqv, refv, head, verAt, ownAt, fileV, dbV, hist, lastCommit, nforeign, envv, ghostv, lockv, procv>>\* created_at passes the limit; only a deferred row's age is ever read.Age(w) ==@@ -673,7 +725,7 @@ Age(w) ==Revoke ==/\ MayRevoke /\ ~revoked/\ revoked' = TRUE- /\ UNCHANGED <<reqv, refv, gitv, restored, preRestore, ghostv, lockv, procv>>+ /\ UNCHANGED <<reqv, refv, gitv, restored, preRestore, ncrash, ghostv, lockv, procv>>\* A backup restore runs on the restored PGLite copy alone (no worker runs there), in one transaction.Restore ==@@ -683,13 +735,13 @@ Restore ==/\ st' = [w \in Writes |-> IF st[w] \in RestoreCancels THEN "cancelled" ELSE st[w]]/\ code' = [w \in Writes |-> IF st[w] \in RestoreCancels THEN "restore_new_brain" ELSE code[w]]/\ restored' = TRUE- /\ UNCHANGED <<seq, nseq, dref, dimp, applied, cpos, ncommit, expRev, overdue, adopted, refv, gitv, revoked, ghostv, lockv, procv>>+ /\ UNCHANGED <<seq, nseq, dref, dimp, applied, cpos, ncommit, expRev, overdue, adopted, refv, gitv, revoked, ncrash, ghostv, lockv, procv>>\* ---------------------------------------------------------------------------ConsStep == ConsBegin \/ ConsSelect \/ ConsClaim \/ ConsPrep \/ ConsPub \/ ConsHold \/ ConsAdopt \/ ConsRelease \/ ConsDone-RefStep == FdCommit \/ RefDrained \/ RefDrainTimeout \/ FmCommit \/ AbortCommit \/ RefMerge \/ MarkCommit- \/ BkCommit \/ RefComplete \/ CompleteCommit+RefStep == FdCommit \/ RefDrained \/ RefDrainTimeout \/ FmCommit \/ AbortCommit \/ RefMerge+ \/ BkCommit \/ RefComplete \/ CompleteCommit \/ RecoverBegin \/ RecoverCommitExpStep == (\E w \in Writes : ExpRefBegin(w) \/ ExpImpBegin(w)) \/ ExpRefCommit \/ ExpImpRecheck \/ ExpImpCommitSyncStep == SyncBegin \/ SyncImportBegin \/ SyncImport \/ SyncStampBegin \/ SyncStampAdmStep(w) == AdmRef(w) \/ AdmRead(w) \/ AdmCommit(w)@@ -698,7 +750,7 @@ Next ==\/ \E p \in Procs : Acquire(p)\/ \E w \in Writes : AdmBegin(w) \/ AdmStep(w) \/ Age(w)\/ ConsStep \/ RefBegin \/ RefStep \/ PersonAbandon \/ PersonResume \/ ExpStep \/ SyncStep- \/ Foreign \/ LocalEdit \/ Revoke \/ Restore+ \/ Foreign \/ LocalEdit \/ Revoke \/ Restore \/ Crash\* Every process that has started runs; a waiting lock request is eventually granted once it can be (Postgres queues\* lock waiters); a person eventually resolves recovery_required; time passes. Producers, other agents' commits,@@ -734,6 +786,10 @@ DeferredWaitsForRelease == ~badReleaseRefreshChangeNeverOverwritten == \A w \in Writes : st[w] = "committed" /\ dref[w] # None => ~rmoved[dref[w]]NoDeferToReleased == ~badDeferReleasedWhenEnded == \A i \in Refs : rstate[i] \in {"completed", "aborted"} => rel[i]+\* recoverOne leaves a refresh recovery_required (a person must act, and every write deferred to it waits on them) only+\* in a policy case: a commit gbrain did not make landed before the merge (foreign_in_window), or HEAD moved past the+\* target between the merge and bookkeeping (after_merge). Never over gbrain's own commits alone.+RecoveryOnlyByPolicy == \A i \in Refs : rstate[i] = "recovery" => rwhy[i] \in {"foreign_in_window", "after_merge"}\* An expiry ends only a write that still waits (its refresh unreleased, or its import not landed).ExpiryOnlyWhileWaiting == ~badExpiry\* A restore leaves no write admitted before it open.diff --git a/spec/tla/refresh-barrier/RefreshBarrierAdopt.cfg b/spec/tla/refresh-barrier/RefreshBarrierAdopt.cfgindex ce7c5cd1df..be92316f06 100644--- a/spec/tla/refresh-barrier/RefreshBarrierAdopt.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierAdopt.cfg@@ -15,6 +15,7 @@ CONSTANTSAdoptLocalEdits = TRUEMayRestore = FALSEMayRevoke = FALSE+ MaxCrashes = 1INVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOnce@@ -26,6 +27,7 @@ INVARIANT DeferredWaitsForReleaseINVARIANT RefreshChangeNeverOverwrittenINVARIANT NoDeferToReleasedINVARIANT ReleasedWhenEnded+INVARIANT RecoveryOnlyByPolicyINVARIANT ExpiryOnlyWhileWaitingINVARIANT RestoreLeavesNoOpenWriteINVARIANT NoApplyAfterRevokediff --git a/spec/tla/refresh-barrier/RefreshBarrierAdoptNightly.cfg b/spec/tla/refresh-barrier/RefreshBarrierAdoptNightly.cfgindex 74d72ac1cd..e238470b3b 100644--- a/spec/tla/refresh-barrier/RefreshBarrierAdoptNightly.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierAdoptNightly.cfg@@ -14,6 +14,7 @@ CONSTANTSAdoptLocalEdits = TRUEMayRestore = FALSEMayRevoke = FALSE+ MaxCrashes = 1INVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOnce@@ -25,6 +26,7 @@ INVARIANT DeferredWaitsForReleaseINVARIANT RefreshChangeNeverOverwrittenINVARIANT NoDeferToReleasedINVARIANT ReleasedWhenEnded+INVARIANT RecoveryOnlyByPolicyINVARIANT ExpiryOnlyWhileWaitingINVARIANT RestoreLeavesNoOpenWriteINVARIANT NoApplyAfterRevokediff --git a/spec/tla/refresh-barrier/RefreshBarrierEnv.cfg b/spec/tla/refresh-barrier/RefreshBarrierEnv.cfgindex 7de5fe94d5..286e5f2cd8 100644--- a/spec/tla/refresh-barrier/RefreshBarrierEnv.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierEnv.cfg@@ -14,6 +14,7 @@ CONSTANTSAdoptLocalEdits = FALSEMayRestore = TRUEMayRevoke = TRUE+ MaxCrashes = 1INVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOnce@@ -25,6 +26,7 @@ INVARIANT DeferredWaitsForReleaseINVARIANT RefreshChangeNeverOverwrittenINVARIANT NoDeferToReleasedINVARIANT ReleasedWhenEnded+INVARIANT RecoveryOnlyByPolicyINVARIANT ExpiryOnlyWhileWaitingINVARIANT RestoreLeavesNoOpenWriteINVARIANT NoApplyAfterRevokediff --git a/spec/tla/refresh-barrier/RefreshBarrierImport.cfg b/spec/tla/refresh-barrier/RefreshBarrierImport.cfgindex 5277ebc2c0..52cc6675e3 100644--- a/spec/tla/refresh-barrier/RefreshBarrierImport.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierImport.cfg@@ -16,6 +16,7 @@ CONSTANTSAdoptLocalEdits = FALSEMayRestore = FALSEMayRevoke = FALSE+ MaxCrashes = 1INVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOnce@@ -27,6 +28,7 @@ INVARIANT DeferredWaitsForReleaseINVARIANT RefreshChangeNeverOverwrittenINVARIANT NoDeferToReleasedINVARIANT ReleasedWhenEnded+INVARIANT RecoveryOnlyByPolicyINVARIANT ExpiryOnlyWhileWaitingINVARIANT RestoreLeavesNoOpenWriteINVARIANT NoApplyAfterRevokediff --git a/spec/tla/refresh-barrier/RefreshBarrierLiveness.cfg b/spec/tla/refresh-barrier/RefreshBarrierLiveness.cfgindex 178ced9844..e42acf7dba 100644--- a/spec/tla/refresh-barrier/RefreshBarrierLiveness.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierLiveness.cfg@@ -15,5 +15,6 @@ CONSTANTSAdoptLocalEdits = FALSEMayRestore = FALSEMayRevoke = FALSE+ MaxCrashes = 1PROPERTY DeferredLeavesPROPERTY EveryWriteEndsdiff --git a/spec/tla/refresh-barrier/RefreshBarrierLivenessNightly.cfg b/spec/tla/refresh-barrier/RefreshBarrierLivenessNightly.cfgindex 64acca86bc..b2a1a98685 100644--- a/spec/tla/refresh-barrier/RefreshBarrierLivenessNightly.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierLivenessNightly.cfg@@ -14,5 +14,6 @@ CONSTANTSAdoptLocalEdits = FALSEMayRestore = FALSEMayRevoke = FALSE+ MaxCrashes = 1PROPERTY DeferredLeavesPROPERTY EveryWriteEndsdiff --git a/spec/tla/refresh-barrier/mutants.json b/spec/tla/refresh-barrier/mutants.jsonindex 7195af7336..ca580c66c6 100644--- a/spec/tla/refresh-barrier/mutants.json+++ b/spec/tla/refresh-barrier/mutants.json@@ -368,6 +368,17 @@"before": "Need(EXP, IF ImportExpiryLocksSource THEN ImportExpiryLocks ELSE <<>>)","after": "Need(EXP, <<>>)"}+ },+ {+ "id": "resume-marks-unmerged-recovery",+ "kind": "guard",+ "title": "recoverOne marks an orphaned fenced refresh recovery_required when HEAD moved past old_head before any merge (#8682 as coded)",+ "expect": "RecoveryOnlyByPolicy",+ "spec": {+ "file": "RefreshBarrier.tla",+ "before": "FencedAbortable(i) == rtgt[i] = 0 /\\ OwnSince(i)",+ "after": "FencedAbortable(i) == head = rhead[i]"+ }}]}diff --git a/test/spec-refresh-barrier.test.ts b/test/spec-refresh-barrier.test.tsindex 60e110a659..6d43944948 100644--- a/test/spec-refresh-barrier.test.ts+++ b/test/spec-refresh-barrier.test.ts@@ -36,6 +36,9 @@ describe('spec/tla/refresh-barrier configs', () => {if (text.includes('SPECIFICATION FairSpec')) expect(checked).toEqual(['DeferredLeaves', 'EveryWriteEnds']);else expect(checked).toEqual(SAFETY);});+ test(cfg + ': kills the refreshing process at least once', () => {+ expect(Number(constants(cfg).MaxCrashes)).toBeGreaterThanOrEqual(1);+ });}});