Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
test(spec): model #8705's merge_started recovery and the remaining policy case
garrytane550af00a16e
diff --git a/spec/tla/README.md b/spec/tla/README.mdindex dd8bce3bc4..f1942e3d02 100644--- a/spec/tla/README.md+++ b/spec/tla/README.md@@ -167,7 +167,8 @@ invariant. Behind constants it also models the stacked follow-ups: writes held fEvery config kills the refreshing process once (`MaxCrashes = 1`), at any step, after which `recoverOne` resumes therefresh row it left. A refresh that finds HEAD moved at the merge aborts and the tick retries, as `fenceAndMerge` does,-so `recovery_required` is reached only through recovery.+so `recovery_required` is reached only through recovery or a refused merge. `RefreshBarrier.cfg` also lets+someone reset the checkout to old_head after the merge (`MayReset`).`test/spec-refresh-barrier.test.ts` keeps the default config on #8682 alone, every safety config on every invariant,every config killing the refresh at least once, and the workflow's path filter covering every file a mutant names.@@ -183,7 +184,7 @@ every config killing the refresh at least once, and the workflow's path filter c| `FenceNeverMidPrepare` | the fence never comes up, and the checkout never moves, while a write is claimed || `DeferredWaitsForRelease`, `RefreshChangeNeverOverwritten` | a deferred write commits only after its refresh is released, and never over a file that refresh changed || `NoDeferToReleased`, `ReleasedWhenEnded` | admission never defers to a released refresh; a completed or aborted refresh is released |-| `RecoveryOnlyByPolicy` | `recoverOne` leaves a refresh `recovery_required` (a person must act while every write deferred to it waits) only in one of two policy cases, recorded in `rwhy`: a commit gbrain did not make landed before the merge (`foreign_in_window`), or HEAD moved past the target between the merge and `bookkeeping` (`after_merge`); never over gbrain's own commits alone |+| `RecoveryOnlyByPolicy` | a refresh goes `recovery_required` (a person must act while every write deferred to it waits) only in the policy case recorded in `rwhy`: its merge started and HEAD no longer holds the target (`merge_started_off_target`: a reset plus an unrelated commit, or a plain-git commit that made the ff-only merge refuse); never for a merge that never started, nor with HEAD at or past the target || `ExpiryOnlyWhileWaiting` | an expiry ends only a write that still waits (its refresh unreleased, or its import not landed) || `RestoreLeavesNoOpenWrite` | a restore leaves no write admitted before it open (finding C) || `NoApplyAfterRevoke` | no write commits after its credential was revoked |@@ -202,7 +203,7 @@ every config killing the refresh at least once, and the workflow's path filter c| `ConsAdopt`, `ConsRelease` | #8708 `adoptLocalEdit`, then the claim's release || `ConsDone` | `completeWrite` of a refusal || `RefBegin` … `CompleteCommit` | worktree-refresh.ts: `fenceDraining`, the drain poll, `fenceAndMerge` (re-counts under `lockTopologyRows`), the ff-only merge under the commit lock, `bookkeeping`, `syncMembers`, `resumeOrAbandon`, and the v292 release trigger on completed or aborted |-| `Crash`, `RecoverBegin`, `RecoverCommit` | the refreshing process killed (its open transaction rolls back), then `recoverOne` from the owner's topology scan or `--resume`: draining aborts; fenced with HEAD at old_head aborts, and so does a fenced row whose HEAD moved only through gbrain's own commits with nothing merged (`movedBeforeMerge`, the ledger naming each commit: `ownAt`); fenced with HEAD at the target runs `bookkeeping`; syncing with HEAD at or past the target stays syncing; anything else is `markRecoveryRequired` |+| `Crash`, `RecoverBegin`, `RecoverCommit` | the refreshing process killed (its open transaction rolls back), then `recoverOne` from the owner's topology scan or `--resume`: draining aborts; fenced with HEAD at old_head aborts, and so does a fenced row whose merge never started (`outcome.merge_started` false: `rmstart`), whoever moved HEAD; fenced with HEAD holding the target (the target or a strict descendant) runs `bookkeeping`; syncing with HEAD at or past the target stays syncing; a `recovery_required` row whose HEAD holds the target runs `bookkeeping`, and otherwise stays; anything else is `markRecoveryRequired` || `ExpRefBegin`, `ExpRefCommit` | `expireDeferredWrites`: the protocol, the refresh row re-read `released_at IS NULL FOR NO KEY UPDATE`, `completeWrite` || `ExpImpBegin`, `ExpImpRecheck`, `ExpImpCommit` | #8705's import expiry in `expireDeferredWrites` || `SyncBegin` … `SyncStamp` | a managed sync pass, whose rows go through the claim like writes: no write passes the queued row importing its page or a running sync row, and a sync row waits behind earlier writes and is never claimed while a write runs; the page row, then its checkpoint row's `UPDATE sources SET last_commit` (sync-prepare.ts) |@@ -231,7 +232,8 @@ every config killing the refresh at least once, and the workflow's path filter c| `hold-beside-held-off` | guard | `CommitsInAdmissionOrder` (Import) | #8705's compareFileTarget held-first read; model only until #8705 is on master || `import-expiry-locks-source-first` | guard | `NoDeadlock` (Import) | #8705's import expiry; model only until #8705 is on master || `import-expiry-skips-source-lock` | guard | `ExpiryOnlyWhileWaiting` (Import) | the same |-| `resume-marks-unmerged-recovery` | guard | `RecoveryOnlyByPolicy` | `recoverOne` as #8682 coded it (the finding below); model only until the fix is on master |+| `resume-marks-unmerged-recovery` | guard | `RecoveryOnlyByPolicy` | `recoverOne` as #8682 coded it: a fenced orphan aborts only with HEAD at old_head (the finding below); model only until the fix is on master |+| `resume-needs-exact-target` | guard | `RecoveryOnlyByPolicy` | `recoverOne` as #8682 coded it: bookkeeping only with HEAD exactly the target, so a commit between the merge and bookkeeping strands the row; the same |#8705's and #8708's guards get code edits and tests (#8705's repro tests in test/import-write-barrier.test.ts) once their code is on master; until then they are checked in the model only.@@ -239,14 +241,19 @@ every config killing the refresh at least once, and the workflow's path filter cA flag-on eval found it first, and the model reproduces it (`resume-marks-unmerged-recovery`, `RecoveryOnlyByPolicy`).The precheck records `old_head` before the drain. A write already queued drains and commits, and its ledger commit moves-HEAD past `old_head`; the fence comes up, and the process is killed before the merge. `recoverOne` then finds a fenced-row whose HEAD is neither `old_head` nor the target and marks it `recovery_required`. The owner's scan never resumes that-state, `--resume` refuses, and the documented way out resets the checkout to `old_head`, which drops the drained write's-commit; until a person acts, every write deferred to the refresh waits. Another agent's commit in the same window is a policy case-instead: the live tick aborts and retries on either, but recovery cannot tell that commit from a person's, so it-stays `recovery_required` (`foreign_in_window`), as test 7 in test/persistence-worktree-refresh.test.ts pins. The fix-(#8705) aborts an orphaned fenced row whose HEAD moved past `old_head` only through commits the commit ledger names, with-nothing of the target merged (`FencedAbortable`); every config passes with it, and both policy cases stay reachable.+HEAD past `old_head` (another agent's plain-git commit does the same); the fence comes up, and the process is killed+before the merge. `recoverOne` then finds a fenced row whose HEAD is neither `old_head` nor the target and marks it+`recovery_required`. The owner's scan never resumes that state, `--resume` refuses, and the documented way out resets the+checkout to `old_head`, which drops the drained write's commit; until a person acts, every write deferred to the refresh+waits. A live tick in the same state aborts and retries instead. A commit landing between the merge and `bookkeeping`+strands the row the same way (`resume-needs-exact-target`), since recovery wanted HEAD exactly at the target.++The fix (#8705) records `outcome.merge_started`, false at the fence and set in its own transaction just before+`git merge --ff-only`. Recovery aborts a fenced row whose merge never started, and runs `bookkeeping` for one whose HEAD+is the target or a strict descendant of it. `recovery_required` remains only for a merge that started on a HEAD that no+longer holds the target (`merge_started_off_target`): a reset plus an unrelated commit (test 7 in+test/persistence-worktree-refresh.test.ts), or a plain-git commit between the HEAD check and the merge, which makes the+ff-only merge refuse. Every config passes with the fix, and that policy case stays reachable.### Races found in #8705diff --git a/spec/tla/refresh-barrier/RefreshBarrier.cfg b/spec/tla/refresh-barrier/RefreshBarrier.cfgindex 1467bfc2b5..315b59793e 100644--- a/spec/tla/refresh-barrier/RefreshBarrier.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrier.cfg@@ -16,6 +16,7 @@ CONSTANTSMayRestore = FALSEMayRevoke = FALSEMaxCrashes = 1+ MayReset = TRUEINVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOncediff --git a/spec/tla/refresh-barrier/RefreshBarrier.tla b/spec/tla/refresh-barrier/RefreshBarrier.tlaindex 121909e149..44d7bf9a76 100644--- a/spec/tla/refresh-barrier/RefreshBarrier.tla+++ b/spec/tla/refresh-barrier/RefreshBarrier.tla@@ -50,7 +50,8 @@ CONSTANTSAdoptLocalEdits, \* #8708: a drift refusal on a one-sided local edit adopts the edit and prepares the write againMayRestore, \* a shared-skill restore may run onceMayRevoke, \* the writers' credential may be revoked once- MaxCrashes \* times the refreshing process may be killed; the owner's topology scan or --resume then recovers it+ MaxCrashes, \* times the refreshing process may be killed; the owner's topology scan or --resume then recovers it+ MayReset \* someone may reset the checkout to old_head after a merge, before bookkeepingWrites == 1..NWRefs == 1..MaxRefreshes@@ -108,12 +109,13 @@ VARIABLESrhead, \* old_headrmoved, \* the refresh changed F in the checkout (git diff old_head target_head names F)rtgt, \* target_head as a commit index once it is in the checkout (0 before the merge)- rwhy, \* why recoverOne marked it recovery_required (the policy case), "none" until it does+ rmstart, \* outcome.merge_started: set just before git merge --ff-only+ lost, \* the checkout was reset below the target after the merge (a person's or another agent's git reset)+ rwhy, \* why the refresh went recovery_required: "merge_started_off_target" (policy) or "unexplained"cur, \* the active refresh (None when none)nref,\* git and the pagehead, verAt, \* HEAD's commit index; the version of F each commit holds- ownAt, \* each commit is gbrain's own (the commit ledger names it: a write's or an adoption's git effect)fileV, \* F as committed at HEADdbV, \* the page's current revision (as a version of F)hist, \* every version that was ever a revision of the page (page history)@@ -136,8 +138,8 @@ VARIABLESsseq \* the admission sequence of the sync pass's queued rows (0 when no pass runs)reqv == <<st, code, seq, nseq, dref, dimp, applied, cpos, ncommit, expRev, overdue, adopted>>-refv == <<rstate, rel, rchg, rhead, rmoved, rtgt, rwhy, cur, nref>>-gitv == <<head, verAt, ownAt, fileV, dbV, hist, lastCommit, local, nver, nforeign, nlocal>>+refv == <<rstate, rel, rchg, rhead, rmoved, rtgt, rmstart, lost, rwhy, cur, nref>>+gitv == <<head, verAt, fileV, dbV, hist, lastCommit, local, nver, nforeign, nlocal>>envv == <<restored, revoked, preRestore, ncrash>>ghostv == <<hidden, badExpiry, badRelease, badDefer, badRevoke>>lockv == <<held, want>>@@ -148,6 +150,10 @@ NoLocks == [r \in Rows |-> "-"]\* TLC splits a disjunction inside an action into branches, so a guard on cur is written as a conjunction.RefIs(S) == cur # None /\ rstate[cur] \in SWorktree == IF local # None THEN local ELSE fileV+\* HEAD holds refresh i's target: its merge ran and HEAD is the target or a descendant (history is one line until a reset).+HoldsTarget(i) == rtgt[i] # 0 /\ head >= rtgt[i] /\ ~lost[i]+\* markRecoveryRequired's case: a merge that started on a HEAD that no longer holds the target is policy; anything else is not.+Why(i) == IF rmstart[i] /\ ~HoldsTarget(i) THEN "merge_started_off_target" ELSE "unexplained"Free(p, r, m) == \A q \in Procs \ {p} : ~Conflict(m, held[q][r])L(r, m) == <<r, m>>@@ -200,12 +206,12 @@ ReleaseOn(s) == s \in {"completed", "aborted"}RestoreCancels == {"queued", "deferred", "running"}\* adoptionIntent: only a one-sided edit (HEAD still holds the page's copy) is adopted.AdoptOneSided == fileV = dbV-\* Every commit on HEAD since the refresh's old_head is gbrain's own (ownCommits: the commit ledger names it).-OwnSince(i) == \A c \in (rhead[i] + 1)..head : ownAt[c]-\* recoverOne's movedBeforeMerge: it aborts an orphaned fenced refresh whose HEAD moved past old_head only through gbrain's-\* own commits (the drained writes' ledger commits), with nothing of the target merged. memberOutsideHead is FALSE here:-\* the model's history is one line.-FencedAbortable(i) == rtgt[i] = 0 /\ OwnSince(i)+\* recoverOne (and --abandon) abort an orphaned fenced refresh that never started its merge (outcome.merge_started is false):+\* HEAD descends from old_head and the target is not in it, whoever made the commits. memberOutsideHead is FALSE here: no+\* member syncs past HEAD.+FencedAbortable(i) == ~rmstart[i]+\* recoverOne runs bookkeeping for an orphaned fenced refresh whose HEAD holds the target (the target or a strict descendant).+ResumeAtTarget(i) == HoldsTarget(i)\* ---------------------------------------------------------------------------@@ -246,12 +252,13 @@ Init ==/\ rhead = [i \in Refs |-> 0]/\ rmoved = [i \in Refs |-> FALSE]/\ rtgt = [i \in Refs |-> 0]+ /\ rmstart = [i \in Refs |-> FALSE]+ /\ lost = [i \in Refs |-> FALSE]/\ rwhy = [i \in Refs |-> "none"]/\ cur = None/\ nref = 0/\ head = 1/\ verAt = [c \in 1..MaxHead |-> IF c = 1 THEN 1 ELSE 0]- /\ ownAt = [c \in 1..MaxHead |-> FALSE]/\ fileV = 1/\ dbV = 1/\ hist = {1}@@ -298,9 +305,8 @@ ConsIdle == /\ pc' = [pc EXCEPT ![CONS] = "idle"] /\ cw' = 0Commit(p) == held' = [held EXCEPT ![p] = NoLocks]Go(p, s) == pc' = [pc EXCEPT ![p] = s]Need(p, ls) == want' = [want EXCEPT ![p] = ls]-NewCommit(v, mine) == /\ head' = head + 1- /\ verAt' = [verAt EXCEPT ![head + 1] = v]- /\ ownAt' = [ownAt EXCEPT ![head + 1] = mine]+NewCommit(v) == /\ head' = head + 1+ /\ verAt' = [verAt EXCEPT ![head + 1] = v]/\ fileV' = v\* ------------------------------- admission -------------------------------@@ -350,7 +356,7 @@ AdmCommit(w) ==ELSE UNCHANGED <<rstate, rel, cur>>/\ Commit(ADM(w)) /\ Go(ADM(w), "idle") /\ Need(ADM(w), <<>>)/\ aref' = [aref EXCEPT ![w] = None] /\ aread' = [aread EXCEPT ![w] = "none"]- /\ UNCHANGED <<code, dimp, applied, cpos, ncommit, expRev, overdue, adopted, rchg, rhead, rmoved, rtgt, rwhy, nref, gitv, envv,+ /\ UNCHANGED <<code, dimp, applied, cpos, ncommit, expRev, overdue, adopted, rchg, rhead, rmoved, rtgt, rmstart, lost, rwhy, nref, gitv, envv,hidden, badExpiry, badRelease, badRevoke, cw, cprep, cfin, ew, passHead, sseq>>\* -------------------------------- consumer --------------------------------@@ -426,7 +432,7 @@ ConsPub ==v == nver + 1IN IF mine /\ ~denied /\ ~movedTHEN /\ hidden' = (hidden \/ fileV \notin hist)- /\ NewCommit(v, TRUE) /\ nver' = v /\ dbV' = v /\ hist' = hist \cup {v} /\ local' = None+ /\ NewCommit(v) /\ nver' = v /\ dbV' = v /\ hist' = hist \cup {v} /\ local' = None/\ applied' = [applied EXCEPT ![cw] = @ + 1]/\ st' = [st EXCEPT ![cw] = "committed"] /\ code' = [code EXCEPT ![cw] = "committed"]/\ ncommit' = ncommit + 1 /\ cpos' = [cpos EXCEPT ![cw] = ncommit + 1]@@ -434,7 +440,7 @@ ConsPub ==/\ badRevoke' = (badRevoke \/ revoked)ELSE /\ st' = [st EXCEPT ![cw] = IF ~mine THEN @ ELSE IF denied THEN "failed" ELSE "queued"]/\ code' = [code EXCEPT ![cw] = IF mine /\ denied THEN "permission_denied" ELSE @]- /\ UNCHANGED <<head, verAt, ownAt, fileV, nver, dbV, hist, local, applied, ncommit, cpos, hidden, badRelease, badRevoke>>+ /\ UNCHANGED <<head, verAt, fileV, nver, dbV, hist, local, applied, ncommit, cpos, hidden, badRelease, badRevoke>>/\ Commit(CONS) /\ ConsIdle/\ UNCHANGED <<seq, nseq, dref, dimp, expRev, overdue, adopted, refv, lastCommit, nforeign, nlocal, envv, badExpiry, badDefer,want, ew, aref, aread, passHead, sseq>>@@ -455,7 +461,7 @@ ConsHold ==ConsAdopt ==/\ pc[CONS] = "c_adopt"/\ hidden' = (hidden \/ fileV \notin hist)- /\ NewCommit(local, TRUE) /\ dbV' = local /\ hist' = hist \cup {local} /\ local' = None+ /\ NewCommit(local) /\ dbV' = local /\ hist' = hist \cup {local} /\ local' = None/\ adopted' = [adopted EXCEPT ![cw] = TRUE]/\ Need(CONS, <<L(REQ(cw), "U")>>) /\ Go(CONS, "c_release")/\ UNCHANGED <<st, code, seq, nseq, dref, dimp, applied, cpos, ncommit, expRev, overdue, refv, lastCommit, nver, nforeign,@@ -494,7 +500,7 @@ FdCommit ==/\ rhead' = [rhead EXCEPT ![nref + 1] = head]/\ cur' = nref + 1 /\ nref' = nref + 1/\ Commit(RFR) /\ Go(RFR, "r_drain")- /\ UNCHANGED <<reqv, rel, rmoved, rtgt, rwhy, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rel, rmoved, rtgt, rmstart, lost, rwhy, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* The drain poll (no locks), then fenceAndMerge's transaction: lockTopologyRows, drainCounts, transition to fenced.RefDrained ==@@ -514,7 +520,7 @@ FmCommit ==THEN /\ rstate' = [rstate EXCEPT ![nref] = "fenced"] /\ Go(RFR, "r_merge") /\ Need(RFR, <<>>)ELSE /\ UNCHANGED rstate /\ Go(RFR, "r_abort") /\ Need(RFR, <<L(REF(nref), "NKU")>>) \* refresh_source_changed/\ Commit(RFR)- /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, rtgt, rwhy, cur, nref, gitv, envv, ghostv, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, rtgt, rmstart, lost, rwhy, cur, nref, gitv, envv, ghostv, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* transition(..., 'aborted'): the release trigger stamps released_at.AbortCommit ==@@ -524,28 +530,51 @@ AbortCommit ==ELSE UNCHANGED <<rstate, rel>>/\ cur' = None/\ Commit(RFR) /\ Go(RFR, "idle")- /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, rwhy, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, rmstart, lost, rwhy, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>-\* Under the commit lock: HEAD must still be old_head, then the ff-only merge. A HEAD that moved (a drained write's ledger-\* commit, or another agent's plain-git commit) aborts the refresh, refresh_source_changed, and the tick retries.+\* Under the commit lock: HEAD must still be old_head. A HEAD that moved (a drained write's ledger commit, or another+\* agent's plain-git commit) aborts the refresh, refresh_source_changed, and the tick retries. Otherwise the+\* merge_started marker commits (synchronous_commit on), then the ff-only merge.RefMerge ==/\ Ready(RFR, "r_merge")- /\ IF head # rhead[nref]- THEN /\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_abort") /\ UNCHANGED <<head, verAt, ownAt, fileV, nver, rmoved, rtgt>>- ELSE /\ IF rchg[nref]- THEN NewCommit(nver + 1, FALSE) /\ nver' = nver + 1 /\ rmoved' = [rmoved EXCEPT ![nref] = TRUE]- ELSE NewCommit(fileV, FALSE) /\ UNCHANGED <<nver, rmoved>>+ /\ Need(RFR, <<L(REF(nref), "NKU")>>)+ /\ Go(RFR, IF head # rhead[nref] THEN "r_abort" ELSE "r_mstart")+ /\ UNCHANGED <<reqv, refv, gitv, envv, ghostv, held, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>++MergeStartCommit ==+ /\ Ready(RFR, "r_mstart")+ /\ rmstart' = [rmstart EXCEPT ![nref] = TRUE]+ /\ Commit(RFR) /\ Go(RFR, "r_ff")+ /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rmoved, rtgt, lost, rwhy, cur, nref, gitv, envv, ghostv, want,+ cw, cprep, cfin, ew, aref, aread, passHead, sseq>>++\* Plain git can still commit between the HEAD check and the merge (only gbrain's commits take the commit lock): the ff-only+\* merge then refuses with HEAD neither old_head nor the target, and the live tick marks recovery_required.+FfMerge ==+ /\ pc[RFR] = "r_ff"+ /\ IF head = rhead[nref]+ THEN /\ IF rchg[nref]+ THEN NewCommit(nver + 1) /\ nver' = nver + 1 /\ rmoved' = [rmoved EXCEPT ![nref] = TRUE]+ ELSE NewCommit(fileV) /\ UNCHANGED <<nver, rmoved>>/\ rtgt' = [rtgt EXCEPT ![nref] = head + 1]/\ Need(RFR, <<L(REF(nref), "NKU"), L(SRC, "NKU")>>) /\ Go(RFR, "r_bk")- /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rwhy, cur, nref, dbV, hist, lastCommit, local, nforeign, nlocal, envv, ghostv,- held, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ ELSE /\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_mark") /\ UNCHANGED <<head, verAt, fileV, nver, rmoved, rtgt>>+ /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rmstart, lost, rwhy, cur, nref, dbV, hist, lastCommit, local, nforeign, nlocal,+ envv, ghostv, held, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>++MarkCommit ==+ /\ Ready(RFR, "r_mark")+ /\ rstate' = [rstate EXCEPT ![nref] = "recovery"] /\ rwhy' = [rwhy EXCEPT ![nref] = Why(nref)]+ /\ Commit(RFR) /\ Go(RFR, "r_recovery")+ /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, rtgt, rmstart, lost, cur, nref, gitv, envv, ghostv, want,+ cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* A person resolves recovery_required: reset to old_head and --abandon, or reset to the target and --resume.PersonAbandon ==/\ pc[RFR] = "r_recovery"/\ head' = rhead[nref] /\ fileV' = verAt[rhead[nref]] /\ local' = None/\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_abort")- /\ UNCHANGED <<reqv, refv, verAt, ownAt, dbV, hist, lastCommit, nver, nforeign, nlocal, envv, ghostv, held,+ /\ UNCHANGED <<reqv, refv, verAt, dbV, hist, lastCommit, nver, nforeign, nlocal, envv, ghostv, held,cw, cprep, cfin, ew, aref, aread, passHead, sseq>>PersonResume ==@@ -553,13 +582,12 @@ PersonResume ==/\ LET base == rhead[nref]v == IF rchg[nref] THEN nver + 1 ELSE verAt[base]IN /\ head' = base + 1 /\ verAt' = [verAt EXCEPT ![base + 1] = v] /\ fileV' = v- /\ ownAt' = [ownAt EXCEPT ![base + 1] = FALSE]/\ nver' = IF rchg[nref] THEN nver + 1 ELSE nver/\ rmoved' = [rmoved EXCEPT ![nref] = rchg[nref]]- /\ rtgt' = [rtgt EXCEPT ![nref] = base + 1]+ /\ rtgt' = [rtgt EXCEPT ![nref] = base + 1] /\ lost' = [lost EXCEPT ![nref] = FALSE]/\ local' = None/\ Need(RFR, <<L(REF(nref), "NKU"), L(SRC, "NKU")>>) /\ Go(RFR, "r_bk")- /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rwhy, cur, nref, dbV, hist, lastCommit, nforeign, nlocal, envv, ghostv, held,+ /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rmstart, rwhy, cur, nref, dbV, hist, lastCommit, nforeign, nlocal, envv, ghostv, held,cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* bookkeeping: transition to merged (the refresh row), the sources' upstream observation, then syncing.@@ -567,7 +595,7 @@ BkCommit ==/\ Ready(RFR, "r_bk")/\ rstate' = [rstate EXCEPT ![nref] = IF @ \in {"fenced", "merged", "recovery"} THEN "syncing" ELSE @]/\ Commit(RFR) /\ Go(RFR, "r_syncing")- /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, rtgt, rwhy, cur, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rel, rchg, rhead, rmoved, rtgt, rmstart, lost, rwhy, cur, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* syncMembers ends: completed once members reach the target, or (#8072) when a member's sync is blocked or deferred.RefComplete ==@@ -582,7 +610,7 @@ CompleteCommit ==/\ cur' = NoneELSE UNCHANGED <<rstate, rel, cur>>/\ Commit(RFR) /\ Go(RFR, "idle")- /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, rwhy, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, rmstart, lost, rwhy, nref, gitv, envv, ghostv, want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* The refreshing process is killed: its open transaction rolls back and its native locks go. A row it inserted stays\* in whatever state the last committed transition left it.@@ -607,17 +635,20 @@ RecoverCommit ==[] s = "draining" \/ (s = "fenced" /\ (head = rhead[nref] \/ FencedAbortable(nref))) ->/\ rstate' = [rstate EXCEPT ![nref] = "aborted"] /\ rel' = [rel EXCEPT ![nref] = ReleaseOn("aborted")]/\ cur' = None /\ Go(RFR, "idle") /\ Need(RFR, <<>>) /\ UNCHANGED rwhy- [] s = "fenced" /\ rtgt[nref] # 0 /\ head = rtgt[nref] -> \* bookkeeping+ [] s = "fenced" /\ ResumeAtTarget(nref) -> \* bookkeeping+ /\ UNCHANGED <<rstate, rel, rwhy, cur>> /\ Go(RFR, "r_bk") /\ Need(RFR, <<L(REF(nref), "NKU"), L(SRC, "NKU")>>)+ [] s = "recovery" /\ HoldsTarget(nref) -> \* a person reset the checkout to the target: bookkeeping/\ UNCHANGED <<rstate, rel, rwhy, cur>> /\ Go(RFR, "r_bk") /\ Need(RFR, <<L(REF(nref), "NKU"), L(SRC, "NKU")>>)- [] s = "syncing" /\ rtgt[nref] # 0 /\ head >= rtgt[nref] -> \* the target or a descendant: the member syncs finish it+ [] s = "recovery" -> \* stays recovery_required, for the person+ /\ UNCHANGED <<rstate, rel, rwhy, cur>> /\ Go(RFR, "r_recovery") /\ Need(RFR, <<>>)+ [] s = "syncing" /\ HoldsTarget(nref) -> \* the target or a descendant: the member syncs finish it/\ UNCHANGED <<rstate, rel, rwhy, cur>> /\ Go(RFR, "r_syncing") /\ Need(RFR, <<>>)[] OTHER -> \* markRecoveryRequired/\ rstate' = [rstate EXCEPT ![nref] = "recovery"] /\ UNCHANGED <<rel, cur>>- /\ rwhy' = [rwhy EXCEPT ![nref] = IF rtgt[nref] # 0 THEN "after_merge"- ELSE IF ~OwnSince(nref) THEN "foreign_in_window" ELSE "own_commits"]+ /\ rwhy' = [rwhy EXCEPT ![nref] = Why(nref)]/\ Go(RFR, "r_recovery") /\ Need(RFR, <<>>)/\ Commit(RFR)- /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, nref, gitv, envv, ghostv, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>+ /\ UNCHANGED <<reqv, rchg, rhead, rmoved, rtgt, rmstart, lost, nref, gitv, envv, ghostv, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* --------------------------------- expiry ---------------------------------\* The candidate query reads without locks; then one transaction per row.@@ -689,7 +720,7 @@ SyncImport ==THEN dbV' = fileV /\ hist' = hist \cup {fileV}ELSE UNCHANGED <<dbV, hist>> \* nothing to import, or a dirty file sync holds/\ Commit(SYN) /\ Go(SYN, "s_ckpt")- /\ UNCHANGED <<reqv, refv, head, verAt, ownAt, fileV, lastCommit, local, nver, nforeign, nlocal, envv, ghostv,+ /\ UNCHANGED <<reqv, refv, head, verAt, fileV, lastCommit, local, nver, nforeign, nlocal, envv, ghostv,want, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* The checkpoint row is claimed once no earlier write is ahead of it; the pass stamp moves last_commit (UPDATE sources).@@ -702,19 +733,26 @@ SyncStamp ==/\ Ready(SYN, "s_stamp")/\ lastCommit' = passHead/\ Commit(SYN) /\ Go(SYN, "idle") /\ passHead' = 0 /\ sseq' = 0- /\ UNCHANGED <<reqv, refv, head, verAt, ownAt, fileV, dbV, hist, local, nver, nforeign, nlocal, envv, ghostv,+ /\ UNCHANGED <<reqv, refv, head, verAt, fileV, dbV, hist, local, nver, nforeign, nlocal, envv, ghostv,want, cw, cprep, cfin, ew, aref, aread>>\* ------------------------------- environment -------------------------------Foreign ==/\ nforeign < MaxForeign /\ local = None- /\ NewCommit(nver + 1, FALSE) /\ nver' = nver + 1 /\ nforeign' = nforeign + 1+ /\ NewCommit(nver + 1) /\ nver' = nver + 1 /\ nforeign' = nforeign + 1/\ UNCHANGED <<reqv, refv, dbV, hist, lastCommit, local, nlocal, envv, ghostv, lockv, procv>>LocalEdit ==/\ nlocal < MaxLocalEdits /\ local = None /\ cur = None/\ local' = nver + 1 /\ nver' = nver + 1 /\ nlocal' = nlocal + 1- /\ UNCHANGED <<reqv, refv, head, verAt, ownAt, fileV, dbV, hist, lastCommit, nforeign, envv, ghostv, lockv, procv>>+ /\ UNCHANGED <<reqv, refv, head, verAt, fileV, dbV, hist, lastCommit, nforeign, envv, ghostv, lockv, procv>>++\* After the merge, someone resets the checkout to old_head with plain git (git reset --hard), before bookkeeping.+ResetAfterMerge ==+ /\ MayReset /\ cur # None /\ RefIs({"fenced"}) /\ HoldsTarget(cur)+ /\ head' = rhead[cur] /\ fileV' = verAt[rhead[cur]] /\ local' = None /\ lost' = [lost EXCEPT ![cur] = TRUE]+ /\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rmoved, rtgt, rmstart, rwhy, cur, nref, verAt, dbV, hist, lastCommit, nver,+ nforeign, nlocal, envv, ghostv, lockv, procv>>\* created_at passes the limit; only a deferred row's age is ever read.Age(w) ==@@ -740,8 +778,8 @@ Restore ==\* ---------------------------------------------------------------------------ConsStep == ConsBegin \/ ConsSelect \/ ConsClaim \/ ConsPrep \/ ConsPub \/ ConsHold \/ ConsAdopt \/ ConsRelease \/ ConsDone-RefStep == FdCommit \/ RefDrained \/ RefDrainTimeout \/ FmCommit \/ AbortCommit \/ RefMerge- \/ BkCommit \/ RefComplete \/ CompleteCommit \/ RecoverBegin \/ RecoverCommit+RefStep == FdCommit \/ RefDrained \/ RefDrainTimeout \/ FmCommit \/ AbortCommit \/ RefMerge \/ MergeStartCommit \/ FfMerge+ \/ MarkCommit \/ BkCommit \/ RefComplete \/ CompleteCommit \/ RecoverBegin \/ RecoverCommitExpStep == (\E w \in Writes : ExpRefBegin(w) \/ ExpImpBegin(w)) \/ ExpRefCommit \/ ExpImpRecheck \/ ExpImpCommitSyncStep == SyncBegin \/ SyncImportBegin \/ SyncImport \/ SyncStampBegin \/ SyncStampAdmStep(w) == AdmRef(w) \/ AdmRead(w) \/ AdmCommit(w)@@ -750,7 +788,7 @@ Next ==\/ \E p \in Procs : Acquire(p)\/ \E w \in Writes : AdmBegin(w) \/ AdmStep(w) \/ Age(w)\/ ConsStep \/ RefBegin \/ RefStep \/ PersonAbandon \/ PersonResume \/ ExpStep \/ SyncStep- \/ Foreign \/ LocalEdit \/ Revoke \/ Restore \/ Crash+ \/ Foreign \/ LocalEdit \/ Revoke \/ Restore \/ Crash \/ ResetAfterMerge\* Every process that has started runs; a waiting lock request is eventually granted once it can be (Postgres queues\* lock waiters); a person eventually resolves recovery_required; time passes. Producers, other agents' commits,@@ -786,10 +824,10 @@ DeferredWaitsForRelease == ~badReleaseRefreshChangeNeverOverwritten == \A w \in Writes : st[w] = "committed" /\ dref[w] # None => ~rmoved[dref[w]]NoDeferToReleased == ~badDeferReleasedWhenEnded == \A i \in Refs : rstate[i] \in {"completed", "aborted"} => rel[i]-\* recoverOne leaves a refresh recovery_required (a person must act, and every write deferred to it waits on them) only-\* in a policy case: a commit gbrain did not make landed before the merge (foreign_in_window), or HEAD moved past the-\* target between the merge and bookkeeping (after_merge). Never over gbrain's own commits alone.-RecoveryOnlyByPolicy == \A i \in Refs : rstate[i] = "recovery" => rwhy[i] \in {"foreign_in_window", "after_merge"}+\* A refresh goes recovery_required (a person must act, and every write deferred to it waits on them) only in the policy+\* case: its merge started and HEAD no longer holds the target (a reset plus an unrelated commit, or a plain-git commit that+\* made the ff-only merge refuse). Never for a merge that never started, nor with HEAD at or past the target.+RecoveryOnlyByPolicy == \A i \in Refs : rstate[i] = "recovery" => rwhy[i] = "merge_started_off_target"\* An expiry ends only a write that still waits (its refresh unreleased, or its import not landed).ExpiryOnlyWhileWaiting == ~badExpiry\* A restore leaves no write admitted before it open.diff --git a/spec/tla/refresh-barrier/RefreshBarrierAdopt.cfg b/spec/tla/refresh-barrier/RefreshBarrierAdopt.cfgindex be92316f06..c11f4222d5 100644--- a/spec/tla/refresh-barrier/RefreshBarrierAdopt.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierAdopt.cfg@@ -16,6 +16,7 @@ CONSTANTSMayRestore = FALSEMayRevoke = FALSEMaxCrashes = 1+ MayReset = FALSEINVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOncediff --git a/spec/tla/refresh-barrier/RefreshBarrierAdoptNightly.cfg b/spec/tla/refresh-barrier/RefreshBarrierAdoptNightly.cfgindex e238470b3b..7f9fedfb0e 100644--- a/spec/tla/refresh-barrier/RefreshBarrierAdoptNightly.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierAdoptNightly.cfg@@ -15,6 +15,7 @@ CONSTANTSMayRestore = FALSEMayRevoke = FALSEMaxCrashes = 1+ MayReset = FALSEINVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOncediff --git a/spec/tla/refresh-barrier/RefreshBarrierEnv.cfg b/spec/tla/refresh-barrier/RefreshBarrierEnv.cfgindex 286e5f2cd8..ad17aaa18b 100644--- a/spec/tla/refresh-barrier/RefreshBarrierEnv.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierEnv.cfg@@ -15,6 +15,7 @@ CONSTANTSMayRestore = TRUEMayRevoke = TRUEMaxCrashes = 1+ MayReset = FALSEINVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOncediff --git a/spec/tla/refresh-barrier/RefreshBarrierImport.cfg b/spec/tla/refresh-barrier/RefreshBarrierImport.cfgindex 52cc6675e3..3361f3d2d3 100644--- a/spec/tla/refresh-barrier/RefreshBarrierImport.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierImport.cfg@@ -17,6 +17,7 @@ CONSTANTSMayRestore = FALSEMayRevoke = FALSEMaxCrashes = 1+ MayReset = FALSEINVARIANT TypeOKINVARIANT NoDeadlockINVARIANT AppliedAtMostOncediff --git a/spec/tla/refresh-barrier/RefreshBarrierLiveness.cfg b/spec/tla/refresh-barrier/RefreshBarrierLiveness.cfgindex e42acf7dba..41fd7c268c 100644--- a/spec/tla/refresh-barrier/RefreshBarrierLiveness.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierLiveness.cfg@@ -16,5 +16,6 @@ CONSTANTSMayRestore = FALSEMayRevoke = FALSEMaxCrashes = 1+ MayReset = FALSEPROPERTY DeferredLeavesPROPERTY EveryWriteEndsdiff --git a/spec/tla/refresh-barrier/RefreshBarrierLivenessNightly.cfg b/spec/tla/refresh-barrier/RefreshBarrierLivenessNightly.cfgindex b2a1a98685..aef7c5240e 100644--- a/spec/tla/refresh-barrier/RefreshBarrierLivenessNightly.cfg+++ b/spec/tla/refresh-barrier/RefreshBarrierLivenessNightly.cfg@@ -15,5 +15,6 @@ CONSTANTSMayRestore = FALSEMayRevoke = FALSEMaxCrashes = 1+ MayReset = FALSEPROPERTY DeferredLeavesPROPERTY EveryWriteEndsdiff --git a/spec/tla/refresh-barrier/mutants.json b/spec/tla/refresh-barrier/mutants.jsonindex ca580c66c6..fe317b0d4c 100644--- a/spec/tla/refresh-barrier/mutants.json+++ b/spec/tla/refresh-barrier/mutants.json@@ -376,9 +376,20 @@"expect": "RecoveryOnlyByPolicy","spec": {"file": "RefreshBarrier.tla",- "before": "FencedAbortable(i) == rtgt[i] = 0 /\\ OwnSince(i)",+ "before": "FencedAbortable(i) == ~rmstart[i]","after": "FencedAbortable(i) == head = rhead[i]"}+ },+ {+ "id": "resume-needs-exact-target",+ "kind": "guard",+ "title": "recoverOne runs bookkeeping for an orphaned fenced refresh only when HEAD is exactly the target, not a descendant (#8682 as coded)",+ "expect": "RecoveryOnlyByPolicy",+ "spec": {+ "file": "RefreshBarrier.tla",+ "before": "ResumeAtTarget(i) == HoldsTarget(i)",+ "after": "ResumeAtTarget(i) == rtgt[i] # 0 /\\ head = rtgt[i] /\\ ~lost[i]"+ }}]}