Skip to content

Self-hosted workflows for coding-agent delivery. Orchestrate CodeBuddy, Codex and Claude Code with scripts and human approval gates. Includes Local workflows for coding assistants.

fix(assistant): keep a live generation through a slow heartbeat (#810)

crispinhu27dd56585276d8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 88f9bcb9..5513299a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,8 @@ Platform releases use `vX.Y.Z` tags and publish `ghcr.io/tencent/loopforge`. Loo
## Unreleased
+- A conversation reply that is still running keeps running through a slow heartbeat. A poll, the conversation list, a follow-up message or a steer used to record it as interrupted after 45 seconds without a stored heartbeat; the process then cancelled the CLI and the next message failed with 「工作区正在使用或保存,请稍后重试」 for up to 15 minutes. Reads of a turn this process executes no longer reap it, and an unchanged heartbeat on MySQL no longer counts as a lost turn. On MySQL a heartbeat's age is measured on the database clock that stamped it, so skew between an API host and MySQL neither reaps a live turn nor keeps a dead one; whether a lease holder stopped renewing is likewise judged from `run_workspace_checkpoints.lease_renewed_at`, stamped on the database clock, not from another replica's `lease_until`. When a turn whose process died is reaped, the workspace lease it recorded is released if nothing renewed it for 45 seconds, so the next message starts at once; a lease another turn took, one still renewed on another replica, and a background save are kept. `migrations/032_assistant_generation_workspace_lease.sql` adds the three columns by hand; startup AutoMigrate adds them too.
+
- The platform image bundles DeepSeek Harness `@deepseek-ai/[email protected]` next to Codex and CodeBuddy, so a DeepSeek key works in conversations and SOP roles without an install step. 0.3.8 had no `dsh` and answered a DeepSeek Harness chat with “AI 助手生成失败,请稍后重试”. The image download grows by about 180 MB. An installation without the image installs the same pinned version from the conversation: every dsh release is a prerelease requiring Node 24.9, and npm on Node 22 found no version for the bare package name.
- A conversation workspace whose image lacks the chosen CLI installs Cursor CLI and Antigravity again. The sandbox drops every capability, so the installer's `tar` running as root could not restore the archive owner (`Cannot change ownership to uid 2000`), and the turn failed with “准备 cursor 原生CLI失败”. The workspace install of DeepSeek Harness uses the pinned `@deepseek-ai/[email protected]` with relaxed engines, as the platform install already did; the bare package name resolved to no version.
- A conversation workspace on a custom Dockerfile without Node.js, npm or Python 3 runs the CLIs bundled in the platform image (Codex, CodeBuddy, DeepSeek Harness). The platform copies its Node.js, npm and a relocatable Python 3.12 into `${LOOPFORGE_DATA_DIR}/cli/toolchain/<build>/` on first use, and a bundled CLI the first time a sandbox needs it; every sandbox mounts that directory read-only at the end of `PATH`, so tools shipped by the image still win. The platform image gains python-build-standalone (about 30 MB to download, checked against a pinned SHA-256). Other CLIs are still installed inside the sandbox, now with the mounted npm. A musl (Alpine) or foreign-architecture image, which lacks the loader the copied `node` needs, keeps its own `PATH` and installs with its package manager as before. Each platform build gets its own directory of about 200 MB plus the CLIs it copied; directories of earlier builds are not removed, so delete the other `cli/toolchain/<build>/` directories after an upgrade.
diff --git a/internal/engine/conversation_workspace.go b/internal/engine/conversation_workspace.go
index e07e57c5..59a6db63 100644
--- a/internal/engine/conversation_workspace.go
+++ b/internal/engine/conversation_workspace.go
@@ -90,7 +90,7 @@ func (e *Engine) conversationGenerationRunning(run domain.Run) bool {
if !ok {
return false
}
- job, ok := e.repo.AssistantGeneration(session.UserID, session.ID)
+ job, ok := e.repo.PeekAssistantGeneration(session.UserID, session.ID)
return ok && job.Status == "running"
}
func (e *Engine) EndConversationWorkspaceUse(ctx context.Context, id, token string) error {
diff --git a/internal/httpapi/assistant.go b/internal/httpapi/assistant.go
index 32a1143c..8d950d3d 100644
--- a/internal/httpapi/assistant.go
+++ b/internal/httpapi/assistant.go
@@ -207,7 +207,7 @@ func (h *assistantHandlers) chat(c *gin.Context) {
}
sink, _ := c.Request.Context().Value(assistantGenerationSinkKey{}).(func(string, any))
if sink == nil {
- if job, ok := h.repo.AssistantGeneration(ownerID, req.SessionID); ok && job.Status == "running" {
+ if job, ok := h.readGeneration(ownerID, req.SessionID); ok && job.Status == "running" {
c.JSON(409, gin.H{"error": "此会话正在后台生成"})
return
}
@@ -338,6 +338,12 @@ func (h *assistantHandlers) chat(c *gin.Context) {
defer files.Cleanup()
if req.RunID == "" && h.engine != nil {
sess.workspaceTurn = newAssistantWorkspaceTurn(h.engine, cancelModel)
+ if scope := assistantControlScopeFrom(modelCtx); scope.GenerationID != "" {
+ generation := domain.AssistantGeneration{ID: scope.GenerationID, SessionID: req.SessionID, UserID: ownerID}
+ sess.workspaceTurn.recordLeases(func(runID, token string) {
+ h.recordGenerationWorkspaceLease(generation, runID, token)
+ })
+ }
defer func() {
if err := sess.workspaceTurn.close(modelCtx); err != nil {
warning := "工作区保存未完成,暂缓回收:" + err.Error()
diff --git a/internal/httpapi/assistant_direct_generation.go b/internal/httpapi/assistant_direct_generation.go
index ac3ba1b6..5355b69b 100644
--- a/internal/httpapi/assistant_direct_generation.go
+++ b/internal/httpapi/assistant_direct_generation.go
@@ -10,6 +10,10 @@ import (
// Synchronous shared chat must occupy the same durable slot as /generations.
// Otherwise a second process can save a pending turn while the first CLI writes
// the shared transcript. A local file lock additionally fences stale writers.
+//
+// The caller must defer the returned finish right after a successful claim (as
+// chat does): it is the only place the turn leaves assistantDirectGenerations,
+// and a deferred call also runs when the handler panics.
func (h *assistantHandlers) claimDirectSharedGeneration(ctx context.Context, req assistantRequest, ownerID int64, actor domain.User, cancel context.CancelFunc) (domain.AssistantGeneration, *assistantDiagnosticsCollector, func(bool), error) {
g := domain.AssistantGeneration{ID: newAssistantSessionID(), SessionID: req.SessionID, UserID: ownerID, ActorID: actor.ID, ActorName: actor.LoginName, ProjectID: req.ProjectID, Status: "running", StartedAt: time.Now().UTC()}
pending := assistantConversation{ID: req.SessionID, Shared: true, ProjectID: req.ProjectID, Model: req.Model, Effort: req.Effort, RunID: req.RunID, Messages: req.Messages}
@@ -17,6 +21,7 @@ func (h *assistantHandlers) claimDirectSharedGeneration(ctx context.Context, req
return g, nil, nil, err
}
diagnostics := newAssistantDiagnosticsCollector(g)
+ assistantDirectGenerations.Store(g.ID, struct{}{})
untrack := inflightTasks.Add()
stop, done := make(chan struct{}), make(chan struct{})
go func() {
@@ -30,7 +35,7 @@ func (h *assistantHandlers) claimDirectSharedGeneration(ctx context.Context, req
case <-ctx.Done():
return
case <-ticker.C:
- current, ok := h.repo.AssistantGeneration(ownerID, req.SessionID)
+ current, ok := h.repo.PeekAssistantGeneration(ownerID, req.SessionID)
if !ok || current.ID != g.ID {
diagnostics.terminate("GENERATION_SUPERSEDED", "当前生成记录已被替换或移除", actor)
cancel()
@@ -56,13 +61,14 @@ func (h *assistantHandlers) claimDirectSharedGeneration(ctx context.Context, req
}()
return g, diagnostics, func(completed bool) {
defer untrack()
+ defer assistantDirectGenerations.Delete(g.ID)
close(stop)
<-done
g.Status = "failed"
if completed {
g.Status = "completed"
} else {
- current, ok := h.repo.AssistantGeneration(ownerID, req.SessionID)
+ current, ok := h.repo.PeekAssistantGeneration(ownerID, req.SessionID)
if ctx.Err() != nil || ok && current.ID == g.ID && current.CancelRequested {
g.Status = "cancelled"
}
diff --git a/internal/httpapi/assistant_direct_generation_live_test.go b/internal/httpapi/assistant_direct_generation_live_test.go
new file mode 100644
index 00000000..a5fce3a5
--- /dev/null
+++ b/internal/httpapi/assistant_direct_generation_live_test.go
@@ -0,0 +1,60 @@
+package httpapi
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+ "github.com/gin-gonic/gin"
+)
+
+// A synchronous shared turn has no live stream, but it is just as alive while
+// this process runs it: a poll or a session read must not reap it after one
+// slow heartbeat, and the sidebar must keep showing it as running.
+func TestDirectSharedGenerationIsNotReapedByItsOwnProcess(t *testing.T) {
+ h, u := workbenchFixture(t)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ req := assistantRequest{ProjectID: "workbench", SessionID: "direct-shared", Messages: []domain.AssistantMessage{{Role: "user", Content: "hi", CreatedAt: time.Now().UTC()}}}
+ g, _, finish, err := h.claimDirectSharedGeneration(ctx, req, u.ID, u, cancel)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !generationExecutingHere(g.ID) {
+ t.Fatal("direct shared turn not registered as executing here")
+ }
+ if err = h.repo.DB.Exec("UPDATE assistant_generations SET updated_at = ? WHERE session_id = ?", time.Now().Add(-time.Minute).UTC(), "direct-shared").Error; err != nil {
+ t.Fatal(err)
+ }
+ c, rec := workbenchContext(u, "GET", "/api/assistant/sessions/direct-shared/generation", bytes.NewBuffer(nil))
+ c.Params = gin.Params{{Key: "id", Value: "direct-shared"}}
+ h.generation(c)
+ var polled domain.AssistantGeneration
+ if err = json.Unmarshal(rec.Body.Bytes(), &polled); err != nil || rec.Code != 200 || polled.Status != "running" {
+ t.Fatalf("poll reaped a live direct turn: %d %s", rec.Code, rec.Body)
+ }
+ if got, ok := readAssistantGeneration(h.repo, u.ID, "direct-shared"); !ok || got.Status != "running" {
+ t.Fatalf("read reaped a live direct turn: %+v", got)
+ }
+ c, rec = workbenchContext(u, "GET", "/api/assistant/sessions?projectId=workbench", bytes.NewBuffer(nil))
+ h.listSessions(c)
+ var list []domain.AssistantSession
+ if err = json.Unmarshal(rec.Body.Bytes(), &list); err != nil {
+ t.Fatal(err)
+ }
+ for _, s := range list {
+ if s.ID == "direct-shared" && s.GenerationStatus != "running" {
+ t.Fatalf("list shows %q for a live direct turn", s.GenerationStatus)
+ }
+ }
+ finish(true)
+ if generationExecutingHere(g.ID) {
+ t.Fatal("finished direct shared turn still registered")
+ }
+ if stored, _ := h.repo.PeekAssistantGeneration(u.ID, "direct-shared"); stored.Status != "completed" {
+ t.Fatalf("direct shared turn stored %q", stored.Status)
+ }
+}
diff --git a/internal/httpapi/assistant_execution.go b/internal/httpapi/assistant_execution.go
index 005b8124..c53a5b4a 100644
--- a/internal/httpapi/assistant_execution.go
+++ b/internal/httpapi/assistant_execution.go
@@ -65,7 +65,7 @@ func (h *assistantHandlers) sessionMode(c *gin.Context) {
c.JSON(404, gin.H{"error": "会话不存在"})
return
}
- if g, ok := h.repo.AssistantGeneration(owner, saved.ID); ok && g.Status == "running" {
+ if g, ok := h.readGeneration(owner, saved.ID); ok && g.Status == "running" {
c.JSON(409, gin.H{"error": "请先停止当前生成再切换模式"})
return
}
diff --git a/internal/httpapi/assistant_fork.go b/internal/httpapi/assistant_fork.go
index 375d5ad3..aa0198cb 100644
--- a/internal/httpapi/assistant_fork.go
+++ b/internal/httpapi/assistant_fork.go
@@ -48,7 +48,7 @@ func (h *assistantHandlers) forkSession(c *gin.Context) {
return
}
// Recover stale jobs before the transactional active-generation check.
- if job, ok := h.repo.AssistantGeneration(owner, source.ID); ok && job.Status == "running" {
+ if job, ok := h.readGeneration(owner, source.ID); ok && job.Status == "running" {
c.JSON(409, gin.H{"error": "请先停止原会话的生成,再创建分支"})
return
}
diff --git a/internal/httpapi/assistant_generation.go b/internal/httpapi/assistant_generation.go
index e5548c78..e29f1fc9 100644
--- a/internal/httpapi/assistant_generation.go
+++ b/internal/httpapi/assistant_generation.go
@@ -187,6 +187,7 @@ func (h *assistantHandlers) startGeneration(c *gin.Context) {
c.JSON(409, gin.H{"error": err.Error()})
return
}
+ h.recordGenerationWorkspaceLease(g, req.RunID, req.workspaceToken)
// Keep the authenticated identity, but disconnect model execution from the
// lifetime of the browser request. Only explicit cancellation stops a turn.
parent := context.WithValue(context.WithoutCancel(c.Request.Context()), assistantFreshCLIKey{}, freshCLI)
@@ -286,8 +287,15 @@ func (h *assistantHandlers) executeGeneration(parent context.Context, req assist
sink := rec.sink
stopped := make(chan struct{})
done := make(chan struct{})
+ renewed := make(chan struct{})
+ // The lease renews on its own ticker. A heartbeat commit that stalls on the
+ // generation row must not also stop the renewal: a replica that reaps this
+ // turn meanwhile releases its lease only once renewal has stopped for 45s.
go func() {
- defer close(done)
+ defer close(renewed)
+ if req.workspaceToken == "" {
+ return
+ }
ticker := time.NewTicker(500 * time.Millisecond)
defer ticker.Stop()
for {
@@ -295,13 +303,24 @@ func (h *assistantHandlers) executeGeneration(parent context.Context, req assist
case <-stopped:
return
case <-ticker.C:
- if req.workspaceToken != "" {
- if err := h.engine.RenewConversationWorkspaceUse(req.RunID, req.workspaceToken); err != nil {
- rec.diagnostics.terminate("WORKSPACE_LEASE_LOST", "会话工作区续租失败:"+err.Error(), diagUser)
- cancel()
- }
+ if err := h.engine.RenewConversationWorkspaceUse(req.RunID, req.workspaceToken); err != nil {
+ rec.diagnostics.terminate("WORKSPACE_LEASE_LOST", "会话工作区续租失败:"+err.Error(), diagUser)
+ cancel()
+ return
}
- old, ok := h.repo.AssistantGeneration(g.UserID, g.SessionID)
+ }
+ }
+ }()
+ go func() {
+ defer close(done)
+ ticker := time.NewTicker(500 * time.Millisecond)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-stopped:
+ return
+ case <-ticker.C:
+ old, ok := h.repo.PeekAssistantGeneration(g.UserID, g.SessionID)
if ok && old.ID == g.ID && old.Steer != nil {
rec.mu.Lock()
rec.g.Steer = old.Steer
@@ -355,6 +374,7 @@ func (h *assistantHandlers) executeGeneration(parent context.Context, req assist
// The turn is complete once its execution has exited, without archive I/O.
close(stopped)
<-done
+ <-renewed
workspaceWarning := ""
if req.workspaceToken != "" {
if result, err := h.engine.EndConversationWorkspaceUseAsync(ctx, req.RunID, req.workspaceToken); err != nil {
@@ -404,7 +424,7 @@ func (h *assistantHandlers) executeGeneration(parent context.Context, req assist
steer := g.Steer
rec.mu.Unlock()
if steer == nil {
- if latest, ok := h.repo.AssistantGeneration(g.UserID, g.SessionID); ok && latest.ID == g.ID && latest.Steer != nil {
+ if latest, ok := h.repo.PeekAssistantGeneration(g.UserID, g.SessionID); ok && latest.ID == g.ID && latest.Steer != nil {
rec.mu.Lock()
g.Steer = latest.Steer
steer = latest.Steer
@@ -463,7 +483,7 @@ func (h *assistantHandlers) steerGeneration(c *gin.Context) {
if content == "" {
content = "请阅读本轮附件并回答。"
}
- g, ok := h.repo.AssistantGeneration(ownerID, c.Param("id"))
+ g, ok := h.readGeneration(ownerID, c.Param("id"))
if !ok || g.Status != "running" {
c.JSON(409, gin.H{"error": "当前没有进行中的回复"})
return
@@ -566,5 +586,6 @@ func (h *assistantHandlers) followSteer(parent context.Context, prev domain.Assi
}
return
}
+ h.recordGenerationWorkspaceLease(next, req.RunID, req.workspaceToken)
h.executeGeneration(parent, req, next, actor)
}
diff --git a/internal/httpapi/assistant_generation_read_paths_test.go b/internal/httpapi/assistant_generation_read_paths_test.go
new file mode 100644
index 00000000..347e6c5d
--- /dev/null
+++ b/internal/httpapi/assistant_generation_read_paths_test.go
@@ -0,0 +1,215 @@
+package httpapi
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+ "agentflow/internal/engine"
+ "agentflow/internal/runtime"
+ "github.com/gin-gonic/gin"
+)
+
+// readPathCase is one handler that refuses to act while a turn is running.
+// Each one must go through readGeneration: a turn this process still executes
+// keeps blocking it after a late heartbeat, and a turn nobody executes any
+// more is reaped so it no longer blocks the conversation.
+type readPathCase struct {
+ name string
+ // busy is the refusal the handler returns while the turn is running.
+ busy string
+ call func(t *testing.T, h *assistantHandlers, u domain.User, sessionID string) (int, string)
+}
+
+func callSession(h *assistantHandlers, u domain.User, method, sessionID string, body []byte) (int, string) {
+ c, rec := workbenchContext(u, method, "/api/assistant/sessions/"+sessionID, bytes.NewBuffer(body))
+ c.Params = gin.Params{{Key: "id", Value: sessionID}}
+ h.session(c)
+ return rec.Code, rec.Body.String()
+}
+
+func generationReadPaths() []readPathCase {
+ return []readPathCase{
+ {name: "session PUT", busy: "生成期间不能改写会话", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {
+ stored, ok := h.repo.AssistantSession(u.ID, id)
+ if !ok {
+ t.Fatalf("session %s missing", id)
+ }
+ body, _ := json.Marshal(stored)
+ return callSession(h, u, "PUT", id, body)
+ }},
+ {name: "session DELETE", busy: "请先停止生成后删除会话", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {
+ return callSession(h, u, "DELETE", id, nil)
+ }},
+ {name: "fork", busy: "请先停止原会话的生成", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {
+ stored, ok := h.repo.AssistantSession(u.ID, id)
+ if !ok {
+ t.Fatalf("session %s missing", id)
+ }
+ body, _ := json.Marshal(gin.H{"messageIndex": 0, "content": "换个问法", "updatedAt": stored.UpdatedAt})
+ c, rec := workbenchContext(u, "POST", "/api/assistant/sessions/"+id+"/fork", bytes.NewBuffer(body))
+ c.Params = gin.Params{{Key: "id", Value: id}}
+ h.forkSession(c)
+ return rec.Code, rec.Body.String()
+ }},
+ {name: "execution mode switch", busy: "请先停止当前生成再切换模式", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {
+ c, rec := workbenchContext(u, "PUT", "/api/assistant/sessions/"+id+"/mode", bytes.NewBufferString(`{"mode":"platform"}`))
+ c.Params = gin.Params{{Key: "id", Value: id}}
+ h.sessionMode(c)
+ return rec.Code, rec.Body.String()
+ }},
+ {name: "proposal decision", busy: "请等待回复结束后再处理建议", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {
+ c, rec := workbenchContext(u, "POST", "/api/assistant/sessions/"+id+"/proposals", bytes.NewBufferString(`{"messageIndex":0,"draftIndex":0,"decision":"declined"}`))
+ c.Params = gin.Params{{Key: "id", Value: id}}
+ h.decideProposal(c)
+ return rec.Code, rec.Body.String()
+ }},
+ }
+}
+
+// The PR routes every non-owner read through readGeneration. Its own test
+// covers polls, the list, chat and steering; these are the remaining
+// handlers that refuse to act on a running turn.
+func TestLocalTurnStillBlocksEveryRunningGuardAfterASlowHeartbeat(t *testing.T) {
+ for i, tc := range generationReadPaths() {
+ t.Run(tc.name, func(t *testing.T) {
+ h, u := workbenchFixture(t)
+ sessionID := "local-guard-" + string(rune('a'+i))
+ g := staleGeneration(t, h, u, sessionID, sessionID+"-turn")
+ markLocal(t, g.ID)
+
+ code, body := tc.call(t, h, u, sessionID)
+ if code != 409 || !strings.Contains(body, tc.busy) {
+ t.Fatalf("a live local turn did not block %s: %d %s", tc.name, code, body)
+ }
+ if stored, ok := h.repo.PeekAssistantGeneration(u.ID, sessionID); !ok || stored.Status != "running" {
+ t.Fatalf("%s wrote the live local turn as %q", tc.name, stored.Status)
+ }
+ if _, ok := h.repo.AssistantSession(u.ID, sessionID); !ok {
+ t.Fatalf("%s removed the conversation of a live turn", tc.name)
+ }
+ })
+ }
+}
+
+// A late turn nobody in this process executes is still reaped on each of
+// those paths, so a dead replica cannot block the conversation for good.
+func TestRemoteStaleTurnIsReapedOnEveryRunningGuard(t *testing.T) {
+ for i, tc := range generationReadPaths() {
+ t.Run(tc.name, func(t *testing.T) {
+ h, u := workbenchFixture(t)
+ sessionID := "remote-guard-" + string(rune('a'+i))
+ staleGeneration(t, h, u, sessionID, sessionID+"-turn")
+
+ code, body := tc.call(t, h, u, sessionID)
+ if strings.Contains(body, tc.busy) {
+ t.Fatalf("a dead remote turn still blocks %s: %d %s", tc.name, code, body)
+ }
+ if stored, ok := h.repo.PeekAssistantGeneration(u.ID, sessionID); ok && stored.Status != "interrupted" {
+ t.Fatalf("%s left the dead remote turn as %q", tc.name, stored.Status)
+ }
+ })
+ }
+}
+
+// The tool gateway asks whether a native tool call still belongs to a running
+// turn. A slow heartbeat must not make it deny the tools of a live local turn.
+func TestNativeToolTurnActiveKeepsALocalTurnThroughASlowHeartbeat(t *testing.T) {
+ h, u := workbenchFixture(t)
+ gateway := &agentToolGateway{deps: h.deps}
+ for _, local := range []bool{true, false} {
+ sessionID := "tool-remote"
+ if local {
+ sessionID = "tool-local"
+ }
+ g := staleGeneration(t, h, u, sessionID, sessionID+"-turn")
+ if local {
+ markLocal(t, g.ID)
+ }
+ ctx := context.WithValue(context.Background(), assistantControlScopeKey{}, assistantControlScope{SessionID: sessionID, GenerationID: g.ID})
+ s := &AgentToolSession{ChatOwnerID: u.ID, ChatSessionID: sessionID, ctx: ctx}
+ active := gateway.nativeToolTurnActive(context.Background(), s)
+ stored, _ := h.repo.PeekAssistantGeneration(u.ID, sessionID)
+ if local && (!active || stored.Status != "running") {
+ t.Fatalf("a live local turn lost its tools: active=%v status=%q", active, stored.Status)
+ }
+ if !local && (active || stored.Status != "interrupted") {
+ t.Fatalf("a dead remote turn kept its tools: active=%v status=%q", active, stored.Status)
+ }
+ }
+}
+
+// A background turn on a SOP conversation takes the run's foreground
+// workspace lease and records exactly that lease on its generation, so a
+// reader that later reaps the turn can release it. The recorded lease does
+// not outlive the turn: the next turn starts with none.
+func TestStartGenerationRecordsTheWorkspaceLeaseItHolds(t *testing.T) {
+ h, a, _, _, session := sharedWorkbenchFixture(t)
+ h.engine = engine.New(h.repo, runtime.NewManager(filepath.Join(t.TempDir(), "workspaces")))
+ entered, release := make(chan struct{}), make(chan struct{})
+ h.generate = func(ctx context.Context, _ string, _ domain.User) (string, error) {
+ close(entered)
+ select {
+ case <-release:
+ return "完成", nil
+ case <-ctx.Done():
+ return "", ctx.Err()
+ }
+ }
+ messages := append(append([]assistantMessage(nil), session.Messages...), assistantMessage{Role: "user", Content: "后台执行"})
+ body, _ := json.Marshal(assistantRequest{ProjectID: session.ProjectID, RunID: session.RunID, SessionID: session.ID, Messages: messages})
+ c, rec := workbenchContext(a, "POST", "/api/assistant/generations", bytes.NewBuffer(body))
+ h.startGeneration(c)
+ if rec.Code != 202 {
+ t.Fatalf("start: %d %s", rec.Code, rec.Body)
+ }
+ released := false
+ t.Cleanup(func() {
+ if !released {
+ close(release)
+ }
+ })
+ select {
+ case <-entered:
+ case <-time.After(5 * time.Second):
+ t.Fatal("generation did not start")
+ }
+ var recorded struct {
+ GenerationID string
+ UserID int64
+ WorkspaceRunID string
+ WorkspaceLease string
+ }
+ if err := h.repo.DB.Table("assistant_generations").Select("generation_id, user_id, workspace_run_id, workspace_lease").Where("session_id = ?", session.ID).Take(&recorded).Error; err != nil {
+ t.Fatal(err)
+ }
+ checkpoint, _ := h.repo.WorkspaceCheckpoint(session.RunID)
+ if recorded.WorkspaceRunID != session.RunID || recorded.WorkspaceLease == "" || recorded.WorkspaceLease != checkpoint.LeaseToken {
+ t.Fatalf("recorded lease %q on %q, workspace holds %q", recorded.WorkspaceLease, recorded.WorkspaceRunID, checkpoint.LeaseToken)
+ }
+ close(release)
+ released = true
+ deadline := time.Now().Add(10 * time.Second)
+ for {
+ if g, ok := h.repo.PeekAssistantGeneration(recorded.UserID, session.ID); ok && g.Status != "running" {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("generation did not finish")
+ }
+ time.Sleep(20 * time.Millisecond)
+ }
+ next := domain.AssistantGeneration{ID: "after-" + recorded.GenerationID, SessionID: session.ID, UserID: recorded.UserID, ProjectID: session.ProjectID, Status: "running"}
+ if err := h.repo.BeginAssistantGeneration(next); err != nil {
+ t.Fatal(err)
+ }
+ ageGeneration(t, h, session.ID)
+ if _, lease, _ := h.repo.ReapAssistantGeneration(recorded.UserID, session.ID); lease.Token != "" {
+ t.Fatalf("the next turn inherited the finished turn's lease: %+v", lease)
+ }
+}
diff --git a/internal/httpapi/assistant_generation_reap_guard_test.go b/internal/httpapi/assistant_generation_reap_guard_test.go
new file mode 100644
index 00000000..7bbc4ba8
--- /dev/null
+++ b/internal/httpapi/assistant_generation_reap_guard_test.go
@@ -0,0 +1,97 @@
+package httpapi
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+ "agentflow/internal/runtime"
+ "github.com/gin-gonic/gin"
+)
+
+// Opening a conversation whose turn this process still executes, after one
+// slow heartbeat, shows the turn as running and does not reap it. Writes to
+// the conversation are covered in assistant_generation_read_paths_test.go.
+func TestOpeningAConversationKeepsALocalTurnAfterSlowHeartbeat(t *testing.T) {
+ h, u := workbenchFixture(t)
+ g := staleGeneration(t, h, u, "guarded-chat", "guarded-turn")
+ markLocal(t, g.ID)
+ c, rec := workbenchContext(u, "GET", "/api/assistant/sessions/guarded-chat", bytes.NewBuffer(nil))
+ c.Params = gin.Params{{Key: "id", Value: "guarded-chat"}}
+ h.session(c)
+ if rec.Code != 200 {
+ t.Fatalf("open: %d %s", rec.Code, rec.Body)
+ }
+ if stored, _ := h.repo.PeekAssistantGeneration(u.ID, "guarded-chat"); stored.Status != "running" {
+ t.Fatalf("opening the conversation wrote the local turn as %q", stored.Status)
+ }
+}
+
+// An ordinary chat turn (no SOP run) records the workspace lease its first
+// tool takes, before the turn ends, so a reader that reaps the turn after its process died can free
+// exactly that lease.
+func TestGenerationRecordsTheWorkspaceLeaseItsToolTakes(t *testing.T) {
+ h, user, fixtureSession, gateway, ids := workspaceTurnFixture(t)
+ type recorded struct {
+ RunID string `gorm:"column:workspace_run_id"`
+ Lease string `gorm:"column:workspace_lease"`
+ }
+ seen := make(chan recorded, 1)
+ h.generate = func(ctx context.Context, _ string, _ domain.User) (string, error) {
+ var session *AgentToolSession
+ agentToolSessions.mu.Lock()
+ for _, candidate := range agentToolSessions.sessions {
+ if candidate.ChatSessionID == fixtureSession.ChatSessionID {
+ session = candidate
+ }
+ }
+ agentToolSessions.mu.Unlock()
+ if session == nil || session.workspaceTurn == nil {
+ return "", errors.New("generation did not own its workspace lifetime")
+ }
+ if _, err := gateway.callMainBotWorkspace(ctx, session, "workbench_workspace_write", map[string]any{"path": "note.txt", "content": "after", "expectedSha256": runtime.WorkspaceContentSHA("before")}); err != nil {
+ return "", err
+ }
+ var row recorded
+ if err := h.repo.DB.Table("assistant_generations").Select("workspace_run_id, workspace_lease").Where("session_id = ?", fixtureSession.ChatSessionID).Scan(&row).Error; err != nil {
+ return "", err
+ }
+ meta, _ := h.repo.WorkspaceCheckpoint(ids[0])
+ if meta.LeaseToken != row.Lease {
+ row.Lease = "mismatch:" + row.Lease + "!=" + meta.LeaseToken
+ }
+ seen <- row
+ return "done", nil
+ }
+ body, _ := json.Marshal(assistantRequest{SessionID: fixtureSession.ChatSessionID, ProjectID: fixtureSession.ProjectID, Messages: []assistantMessage{{Role: "user", Content: "edit"}}})
+ c, rec := workbenchContext(user, "POST", "/api/assistant/generations", bytes.NewBuffer(body))
+ h.startGeneration(c)
+ if rec.Code != 202 {
+ t.Fatalf("generation start: %d %s", rec.Code, rec.Body)
+ }
+ select {
+ case row := <-seen:
+ if row.RunID != ids[0] || row.Lease == "" || len(row.Lease) > 9 && row.Lease[:9] == "mismatch:" {
+ t.Fatalf("lease recorded on the generation: %+v", row)
+ }
+ case <-time.After(10 * time.Second):
+ t.Fatal("generation did not reach its tool")
+ }
+ deadline := time.Now().Add(10 * time.Second)
+ for {
+ if g, ok := h.repo.PeekAssistantGeneration(user.ID, fixtureSession.ChatSessionID); ok && g.Status != "running" {
+ if g.Status != "completed" {
+ t.Fatalf("generation outcome: %+v", g)
+ }
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("generation did not finish")
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+}
diff --git a/internal/httpapi/assistant_generation_reap_test.go b/internal/httpapi/assistant_generation_reap_test.go
new file mode 100644
index 00000000..c354243c
--- /dev/null
+++ b/internal/httpapi/assistant_generation_reap_test.go
@@ -0,0 +1,204 @@
+package httpapi
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+ "agentflow/internal/repo"
+ "github.com/gin-gonic/gin"
+)
+
+// staleGeneration starts a turn whose last durable heartbeat is a minute old,
+// as after a slow commit or a dead process.
+func staleGeneration(t *testing.T, h *assistantHandlers, u domain.User, sessionID, generationID string) domain.AssistantGeneration {
+ t.Helper()
+ if err := h.repo.PutAssistantSession(u.ID, domain.AssistantSession{ID: sessionID, ProjectID: "workbench", Title: sessionID, Messages: []domain.AssistantMessage{{Role: "user", Content: "继续"}}}); err != nil {
+ t.Fatal(err)
+ }
+ g := domain.AssistantGeneration{ID: generationID, SessionID: sessionID, UserID: u.ID, ProjectID: "workbench", Status: "running", MessageCount: 1}
+ if err := h.repo.BeginAssistantGeneration(g); err != nil {
+ t.Fatal(err)
+ }
+ ageGeneration(t, h, sessionID)
+ return g
+}
+
+func ageGeneration(t *testing.T, h *assistantHandlers, sessionID string) {
+ t.Helper()
+ if err := h.repo.DB.Table("assistant_generations").Where("session_id = ?", sessionID).UpdateColumn("updated_at", time.Now().Add(-time.Minute).UTC()).Error; err != nil {
+ t.Fatal(err)
+ }
+}
+
+func markLocal(t *testing.T, id string) {
+ t.Helper()
+ assistantLiveGenerations.Store(id, &assistantLiveGeneration{changed: make(chan struct{})})
+ t.Cleanup(func() { assistantLiveGenerations.Delete(id) })
+}
+
+func listedStatus(t *testing.T, h *assistantHandlers, u domain.User, sessionID string) string {
+ t.Helper()
+ c, rec := workbenchContext(u, "GET", "/api/assistant/sessions?projectId=workbench", bytes.NewBuffer(nil))
+ h.listSessions(c)
+ if rec.Code != 200 {
+ t.Fatalf("list: %d %s", rec.Code, rec.Body)
+ }
+ var list []struct {
+ ID string `json:"id"`
+ GenerationStatus string `json:"generationStatus"`
+ }
+ if err := json.Unmarshal(rec.Body.Bytes(), &list); err != nil {
+ t.Fatal(err)
+ }
+ for _, item := range list {
+ if item.ID == sessionID {
+ return item.GenerationStatus
+ }
+ }
+ t.Fatalf("session %s not listed: %s", sessionID, rec.Body)
+ return ""
+}
+
+// A turn this process is still executing survives a late heartbeat on every
+// read path: polls, the session list, a follow-up message and steering.
+func TestLocalTurnSurvivesSlowHeartbeatOnEveryRead(t *testing.T) {
+ h, u := workbenchFixture(t)
+ g := staleGeneration(t, h, u, "slow-chat", "slow-turn")
+ markLocal(t, g.ID)
+
+ if got, ok := h.readGeneration(u.ID, "slow-chat"); !ok || got.Status != "running" {
+ t.Fatalf("read reaped a local turn: %+v", got)
+ }
+ if got, ok := h.currentGeneration(u.ID, "slow-chat"); !ok || got.Status != "running" {
+ t.Fatalf("poll reaped a local turn: %+v", got)
+ }
+ if status := listedStatus(t, h, u, "slow-chat"); status != "running" {
+ t.Fatalf("list shows a local turn as %q", status)
+ }
+ c, rec := workbenchContext(u, "POST", "/api/assistant/chat", bytes.NewBufferString(`{"projectId":"workbench","sessionId":"slow-chat","messages":[{"role":"user","content":"继续"},{"role":"user","content":"再来"}]}`))
+ h.chat(c)
+ if rec.Code != 409 {
+ t.Fatalf("a second chat on a live turn was not refused: %d %s", rec.Code, rec.Body)
+ }
+ c, rec = workbenchContext(u, "POST", "/api/assistant/sessions/slow-chat/generation/steer", bytes.NewBufferString(`{"content":"改一下"}`))
+ c.Params = gin.Params{{Key: "id", Value: "slow-chat"}}
+ h.steerGeneration(c)
+ if rec.Code == 409 {
+ t.Fatalf("steering a live turn was refused as finished: %s", rec.Body)
+ }
+ if stored, _ := h.repo.PeekAssistantGeneration(u.ID, "slow-chat"); stored.Status != "running" {
+ t.Fatalf("a read wrote the local turn as %q", stored.Status)
+ }
+}
+
+// Without a local owner the late turn is reaped. Once that is stored, the list
+// reports it as stored instead of hiding it behind a stale in-memory entry.
+func TestListReportsAStoredInterruptionEvenWithALiveEntry(t *testing.T) {
+ h, u := workbenchFixture(t)
+ g := staleGeneration(t, h, u, "dead-chat", "dead-turn")
+ if status := listedStatus(t, h, u, "dead-chat"); status != "interrupted" {
+ t.Fatalf("late remote turn listed as %q", status)
+ }
+ if got, _ := h.readGeneration(u.ID, "dead-chat"); got.Status != "interrupted" {
+ t.Fatalf("late remote turn not reaped: %+v", got)
+ }
+ markLocal(t, g.ID)
+ if status := listedStatus(t, h, u, "dead-chat"); status != "interrupted" {
+ t.Fatalf("a stored interruption listed as %q", status)
+ }
+}
+
+// A reaped turn frees its own lease so the next message can start. A lease its
+// holder still renews, or one a newer turn took, is never freed by the reap.
+func TestReapReleasesOnlyTheAbandonedTurnsWorkspaceLease(t *testing.T) {
+ h, u := workbenchFixture(t)
+ now := time.Now().UTC()
+ taken := now.Add(-2 * time.Minute)
+ for _, id := range []string{"run-dead", "run-renewed", "run-replaced"} {
+ if err := h.repo.PutRun(domain.Run{ID: id, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ begin := func(runID string, at time.Time) string {
+ t.Helper()
+ token, err := h.repo.BeginWorkspaceUse(runID, at)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return token
+ }
+ free := func(runID string) bool {
+ t.Helper()
+ token, err := h.repo.BeginWorkspaceUse(runID, time.Now().UTC())
+ if errors.Is(err, repo.ErrWorkspaceInUse) {
+ return false
+ }
+ if err != nil {
+ t.Fatal(err)
+ }
+ _ = h.repo.FinishWorkspaceUse(runID, token, time.Now().UTC(), nil, "")
+ return true
+ }
+
+ // The process died: nothing renews the lease and nothing heartbeats.
+ dead := staleGeneration(t, h, u, "lease-dead", "turn-dead")
+ h.recordGenerationWorkspaceLease(dead, "run-dead", begin("run-dead", taken))
+ ageGeneration(t, h, "lease-dead")
+ if got, _ := h.readGeneration(u.ID, "lease-dead"); got.Status != "interrupted" {
+ t.Fatalf("dead turn not reaped: %+v", got)
+ }
+ if !free("run-dead") {
+ t.Fatal("the dead turn's lease still blocks the next message")
+ }
+
+ // Another replica is executing it: its heartbeat is late but it still
+ // renews the lease. The turn is reaped as before; its lease stays.
+ renewed := staleGeneration(t, h, u, "lease-renewed", "turn-renewed")
+ token := begin("run-renewed", taken)
+ h.recordGenerationWorkspaceLease(renewed, "run-renewed", token)
+ ageGeneration(t, h, "lease-renewed")
+ if err := h.repo.RenewWorkspaceUse("run-renewed", token, time.Now().UTC()); err != nil {
+ t.Fatal(err)
+ }
+ h.readGeneration(u.ID, "lease-renewed")
+ if free("run-renewed") {
+ t.Fatal("a lease its holder still renews was released")
+ }
+
+ // The reaped turn's lease already ended and a newer use took the
+ // workspace before the reader got there.
+ replaced := staleGeneration(t, h, u, "lease-replaced", "turn-replaced")
+ old := begin("run-replaced", taken)
+ h.recordGenerationWorkspaceLease(replaced, "run-replaced", old)
+ ageGeneration(t, h, "lease-replaced")
+ if err := h.repo.FinishWorkspaceUse("run-replaced", old, taken, nil, ""); err != nil {
+ t.Fatal(err)
+ }
+ newer := begin("run-replaced", taken)
+ h.readGeneration(u.ID, "lease-replaced")
+ if kept, _ := h.repo.WorkspaceCheckpoint("run-replaced"); kept.LeaseToken != newer {
+ t.Fatalf("a newer use lost its lease to the reap: %q", kept.LeaseToken)
+ }
+}
+
+// Ordinary conversations take their lease during the turn; it is recorded on
+// the generation as soon as it is taken.
+func TestWorkspaceTurnRecordsEachLeaseItTakes(t *testing.T) {
+ turn := newAssistantWorkspaceTurn(&workspaceTurnEngine{}, nil)
+ t.Cleanup(func() { _ = turn.close(context.Background()) })
+ var recorded []string
+ turn.recordLeases(func(runID, token string) { recorded = append(recorded, runID+"="+token) })
+ for range 2 {
+ if _, err := turn.lease(context.Background(), "run-a"); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if len(recorded) != 1 || recorded[0] != "run-a=token-run-a" {
+ t.Fatalf("recorded leases: %v", recorded)
+ }
+}
diff --git a/internal/httpapi/assistant_native_approval.go b/internal/httpapi/assistant_native_approval.go
index 4022f211..abe26e91 100644
--- a/internal/httpapi/assistant_native_approval.go
+++ b/internal/httpapi/assistant_native_approval.go
@@ -42,7 +42,7 @@ func (h *assistantHandlers) nativeApprovals(c *gin.Context) {
return
}
if item.GenerationID != "" {
- g, ok := h.repo.AssistantGeneration(owner, c.Param("id"))
+ g, ok := h.readGeneration(owner, c.Param("id"))
if !ok || g.ID != item.GenerationID || g.Status != "running" || g.CancelRequested {
c.JSON(409, gin.H{"error": "本轮已结束,审批已失效"})
return
@@ -154,7 +154,7 @@ func (h *agentToolGateway) nativeToolTurnActive(ctx context.Context, s *AgentToo
if h.repo == nil {
return false
}
- current, ok := h.repo.AssistantGeneration(s.ChatOwnerID, s.ChatSessionID)
+ current, ok := readAssistantGeneration(h.repo, s.ChatOwnerID, s.ChatSessionID)
return ok && current.ID == generation && current.Status == "running" && !current.CancelRequested
}
diff --git a/internal/httpapi/assistant_proposals.go b/internal/httpapi/assistant_proposals.go
index c732b66c..4855c1b1 100644
--- a/internal/httpapi/assistant_proposals.go
+++ b/internal/httpapi/assistant_proposals.go
@@ -374,7 +374,7 @@ func (h *assistantHandlers) decideProposal(c *gin.Context) {
c.JSON(400, gin.H{"error": "未知的处理方式"})
return
}
- if job, ok := h.repo.AssistantGeneration(ownerID, id); ok && job.Status == "running" {
+ if job, ok := h.readGeneration(ownerID, id); ok && job.Status == "running" {
c.JSON(409, gin.H{"error": "请等待回复结束后再处理建议"})
return
}
diff --git a/internal/httpapi/assistant_session.go b/internal/httpapi/assistant_session.go
index bef0bc4b..9660e270 100644
--- a/internal/httpapi/assistant_session.go
+++ b/internal/httpapi/assistant_session.go
@@ -88,6 +88,12 @@ func (h *assistantHandlers) listSessions(c *gin.Context) {
out[i].Unread = false
if state, ok := statuses[out[i].ID]; ok {
out[i].GenerationStatus = state.Status
+ // The list only projects a late heartbeat as interrupted; the row
+ // still says running. A turn this process is executing is running.
+ // A row actually written as interrupted is reported as stored.
+ if state.Stale && generationExecutingHere(state.ID) {
+ out[i].GenerationStatus = "running"
+ }
out[i].GenerationID = state.ID
// A reply that completed while the reader was elsewhere stays
// unread until they open the conversation (Codex's blue dot).
@@ -262,7 +268,7 @@ func (h *assistantHandlers) session(c *gin.Context) {
return
}
if !item.Shared && !readOnly {
- if job, ok := h.repo.AssistantGeneration(ownerID, id); !ok || job.Status != "running" {
+ if job, ok := h.readGeneration(ownerID, id); !ok || job.Status != "running" {
if materializeWorkspaceImages(ownerID, &item) {
if err := saveAssistantConversation(h.repo, ownerID, item); err == nil {
if stored, err := loadAssistantConversation(h.repo, ownerID, id); err == nil {
@@ -279,7 +285,7 @@ func (h *assistantHandlers) session(c *gin.Context) {
}
c.JSON(200, decorateRunSessionView(h.repo, item))
case "PUT":
- if job, ok := h.repo.AssistantGeneration(ownerID, id); ok && job.Status == "running" {
+ if job, ok := h.readGeneration(ownerID, id); ok && job.Status == "running" {
c.JSON(409, gin.H{"error": "生成期间不能改写会话"})
return
}
@@ -428,7 +434,7 @@ func (h *assistantHandlers) session(c *gin.Context) {
return
}
}
- if job, ok := h.repo.AssistantGeneration(ownerID, id); ok && job.Status == "running" {
+ if job, ok := h.readGeneration(ownerID, id); ok && job.Status == "running" {
c.JSON(409, gin.H{"error": "请先停止生成后删除会话"})
return
}
diff --git a/internal/httpapi/assistant_tasks.go b/internal/httpapi/assistant_tasks.go
index 72deb542..c606afa5 100644
--- a/internal/httpapi/assistant_tasks.go
+++ b/internal/httpapi/assistant_tasks.go
@@ -285,7 +285,7 @@ func (h *assistantHandlers) readScheduledTurn(user domain.User, task *assistantT
if reply != nil {
msg = reply.Unfinished
}
- if latest, found := h.repo.AssistantGeneration(user.ID, g.SessionID); found && latest.ID == g.ID {
+ if latest, found := h.readGeneration(user.ID, g.SessionID); found && latest.ID == g.ID {
msg = firstNonEmpty(msg, latest.Error)
}
return fail(firstNonEmpty(strings.TrimSpace(msg), "未返回结果"))
diff --git a/internal/httpapi/assistant_unfinished.go b/internal/httpapi/assistant_unfinished.go
index f94f9be2..aae2f445 100644
--- a/internal/httpapi/assistant_unfinished.go
+++ b/internal/httpapi/assistant_unfinished.go
@@ -5,9 +5,11 @@ import (
"fmt"
"log"
"strings"
+ "sync"
"time"
"agentflow/internal/domain"
+ "agentflow/internal/repo"
)
// commitTurnTranscript keeps a turn that is stopped, fails or times out. A turn
@@ -69,10 +71,75 @@ func (h *assistantHandlers) keepInterruptedReply(g domain.AssistantGeneration) {
}
}
+// readGeneration reaps a stale turn only when this process is not still
+// executing it. A poll or a follow-up message used to mark a live turn
+// interrupted after one slow heartbeat, and the next message then failed
+// immediately because the CLI and workspace lease were still held.
+func (h *assistantHandlers) readGeneration(ownerID int64, sessionID string) (domain.AssistantGeneration, bool) {
+ return readAssistantGeneration(h.repo, ownerID, sessionID)
+}
+
+// assistantDirectGenerations holds the synchronous shared turns this process
+// is executing. They have no live stream, so they are not in
+// assistantLiveGenerations, but their own readers must not reap them either.
+var assistantDirectGenerations sync.Map
+
+// generationExecutingHere reports whether this process is running the turn.
+func generationExecutingHere(id string) bool {
+ if id == "" {
+ return false
+ }
+ if _, ok := assistantLiveGenerations.Load(id); ok {
+ return true
+ }
+ _, ok := assistantDirectGenerations.Load(id)
+ return ok
+}
+
+// readAssistantGeneration is the one non-owner read of a turn. Every handler
+// that may meet a stale turn goes through it, so a turn this process is still
+// executing is never written as interrupted by its own readers.
+//
+// A turn owned by another replica is still reaped after the heartbeat timeout,
+// as before. Its workspace lease is released only when the reaped turn recorded
+// it and the lease itself stopped being renewed (see ReleaseAbandonedWorkspaceUse).
+func readAssistantGeneration(store *repo.Store, ownerID int64, sessionID string) (domain.AssistantGeneration, bool) {
+ peek, ok := store.PeekAssistantGeneration(ownerID, sessionID)
+ if !ok {
+ return domain.AssistantGeneration{}, false
+ }
+ if generationExecutingHere(peek.ID) {
+ return peek, true
+ }
+ g, lease, ok := store.ReapAssistantGeneration(ownerID, sessionID)
+ if lease.Token != "" {
+ released, err := store.ReleaseAbandonedWorkspaceUse(lease.RunID, lease.Token, time.Now().UTC())
+ switch {
+ case err != nil:
+ log.Printf("assistant release workspace lease session=%s run=%s: %v", sessionID, lease.RunID, err)
+ case released:
+ log.Printf("assistant released abandoned workspace lease session=%s generation=%s run=%s", sessionID, g.ID, lease.RunID)
+ }
+ }
+ return g, ok
+}
+
+// recordGenerationWorkspaceLease stores the foreground lease a running turn
+// holds. Without it a reaped turn keeps its workspace locked for the rest of
+// the lease term; with it only that exact lease can be released.
+func (h *assistantHandlers) recordGenerationWorkspaceLease(g domain.AssistantGeneration, runID, token string) {
+ if g.ID == "" || runID == "" || token == "" {
+ return
+ }
+ if err := h.repo.RecordAssistantGenerationWorkspaceLease(g.UserID, g.SessionID, g.ID, runID, token); err != nil {
+ log.Printf("assistant record workspace lease session=%s generation=%s run=%s: %v", g.SessionID, g.ID, runID, err)
+ }
+}
+
// currentGeneration reads a session's generation. A turn whose server died is
// only discovered here (as "interrupted"), so its partial reply is kept now.
func (h *assistantHandlers) currentGeneration(ownerID int64, sessionID string) (domain.AssistantGeneration, bool) {
- g, ok := h.repo.AssistantGeneration(ownerID, sessionID)
+ g, ok := h.readGeneration(ownerID, sessionID)
if ok {
if _, local := assistantLiveGenerations.Load(g.ID); !local {
h.settleLegacyWorkspaceNativeApprovals(g)
diff --git a/internal/httpapi/assistant_workspace_turn.go b/internal/httpapi/assistant_workspace_turn.go
index 5a54cd6b..a561977a 100644
--- a/internal/httpapi/assistant_workspace_turn.go
+++ b/internal/httpapi/assistant_workspace_turn.go
@@ -35,8 +35,11 @@ type assistantWorkspaceTurn struct {
closed bool
renewError error
saveErrors map[string]error
- stop chan struct{}
- done chan struct{}
+ // onLease records each foreground lease on the turn's generation so a
+ // reader that reaps a dead turn can release exactly that lease.
+ onLease func(runID, token string)
+ stop chan struct{}
+ done chan struct{}
}
type assistantWorkspaceLeaseEngine interface {
@@ -68,6 +71,13 @@ func newAssistantWorkspaceTurn(en assistantWorkspaceLeaseEngine, cancel context.
return t
}
+// recordLeases sets where each lease this turn takes is recorded.
+func (t *assistantWorkspaceTurn) recordLeases(fn func(runID, token string)) {
+ t.mu.Lock()
+ t.onLease = fn
+ t.mu.Unlock()
+}
+
// Tools may run in parallel, but a binding switch and the final checkpoint
// must wait for all tools using the old workspace to return.
func (t *assistantWorkspaceTurn) lockOperation() func() {
@@ -114,6 +124,9 @@ func (t *assistantWorkspaceTurn) lease(ctx context.Context, runID string) (conte
return ctx, err
}
t.runID, t.token, t.renewedAt = runID, token, time.Now()
+ if t.onLease != nil {
+ t.onLease(runID, token)
+ }
}
ctx = engine.WithConversationWorkspaceToken(ctx, t.token, runID)
t.mu.Unlock()
diff --git a/internal/httpapi/space_ai.go b/internal/httpapi/space_ai.go
index 64adf3d2..4261b889 100644
--- a/internal/httpapi/space_ai.go
+++ b/internal/httpapi/space_ai.go
@@ -224,7 +224,7 @@ func (h *spacesHandlers) reconcileAI(room domain.ChatRoom, messages []domain.Cha
if m.Kind != domain.ChatKindAI || m.AIStatus != domain.AIStatusRunning {
continue
}
- g, ok := h.repo.AssistantGeneration(m.AIRequesterID, m.AISessionID)
+ g, ok := readAssistantGeneration(h.repo, m.AIRequesterID, m.AISessionID)
switch {
case !ok && time.Since(m.CreatedAt) > time.Minute:
m.AIStatus, m.AIError = domain.AIStatusFailed, "没有找到这次 AI 回复的记录"
diff --git a/internal/repo/assistant_generation_approvals.go b/internal/repo/assistant_generation_approvals.go
index 417c2daf..f48b8a3f 100644
--- a/internal/repo/assistant_generation_approvals.go
+++ b/internal/repo/assistant_generation_approvals.go
@@ -7,7 +7,6 @@ import (
"encoding/json"
"fmt"
"slices"
- "time"
"gorm.io/gorm"
"gorm.io/gorm/clause"
@@ -40,8 +39,10 @@ func generationPayload(row assistantGenerationRow) assistantGenerationPayload {
return payload
}
-func nativeApprovalGenerationActive(row assistantGenerationRow) bool {
- return row.Status == "running" && !row.CancelRequested && time.Since(row.UpdatedAt) <= 45*time.Second
+// nativeApprovalGenerationActive measures the heartbeat on the same clock that
+// stamped it (see heartbeatNow).
+func (r *AssistantRepo) nativeApprovalGenerationActive(db *gorm.DB, row assistantGenerationRow) bool {
+ return row.Status == "running" && !row.CancelRequested && !r.heartbeatLate(db, row.UpdatedAt)
}
func (r *AssistantRepo) updateNativeApprovals(owner int64, session, generation string, requireLive bool, update func(*assistantGenerationPayload) error) error {
@@ -50,7 +51,7 @@ func (r *AssistantRepo) updateNativeApprovals(owner int64, session, generation s
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).Where("session_id = ? AND user_id = ? AND generation_id = ?", session, owner, generation).Take(&row).Error; err != nil {
return fmt.Errorf("本轮已结束,审批已失效")
}
- if requireLive && !nativeApprovalGenerationActive(row) {
+ if requireLive && !r.nativeApprovalGenerationActive(tx, row) {
return fmt.Errorf("本轮已结束,审批已失效")
}
payload := generationPayload(row)
@@ -87,7 +88,7 @@ func (r *AssistantRepo) CreateNativeApproval(owner int64, session, phase string,
func (r *AssistantRepo) NativeApproval(owner int64, session, id string) (domain.AssistantNativeApproval, bool) {
var row assistantGenerationRow
- if err := r.DB.Where("session_id = ? AND user_id = ?", session, owner).Take(&row).Error; err != nil || !nativeApprovalGenerationActive(row) {
+ if err := r.DB.Where("session_id = ? AND user_id = ?", session, owner).Take(&row).Error; err != nil || !r.nativeApprovalGenerationActive(r.DB, row) {
return domain.AssistantNativeApproval{}, false
}
for _, item := range generationPayload(row).NativeApprovals {
@@ -101,7 +102,7 @@ func (r *AssistantRepo) NativeApproval(owner int64, session, id string) (domain.
func (r *AssistantRepo) PendingNativeApprovals(owner int64, session string) []domain.AssistantNativeApproval {
out := []domain.AssistantNativeApproval{}
var row assistantGenerationRow
- if err := r.DB.Where("session_id = ? AND user_id = ?", session, owner).Take(&row).Error; err != nil || !nativeApprovalGenerationActive(row) {
+ if err := r.DB.Where("session_id = ? AND user_id = ?", session, owner).Take(&row).Error; err != nil || !r.nativeApprovalGenerationActive(r.DB, row) {
return out
}
for _, item := range generationPayload(row).NativeApprovals {
diff --git a/internal/repo/assistant_generation_clock_mysql_test.go b/internal/repo/assistant_generation_clock_mysql_test.go
new file mode 100644
index 00000000..3676d0bf
--- /dev/null
+++ b/internal/repo/assistant_generation_clock_mysql_test.go
@@ -0,0 +1,88 @@
+package repo
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+)
+
+// MySQL stamps heartbeats with UTC_TIMESTAMP(3), so their age must be measured
+// on the database clock. With this host's time.Now() instead, a database clock
+// behind the API host reaps a turn that just checked in, and one ahead of it
+// never reaps a dead turn. SET timestamp skews the database clock of the one
+// connection the store is pinned to.
+func TestMySQLHeartbeatAgeIsMeasuredOnTheDatabaseClock(t *testing.T) {
+ dsn := os.Getenv("AGENTFLOW_TEST_MYSQL_DSN")
+ if dsn == "" {
+ t.Skip("local isolated MySQL required")
+ }
+ s, err := Open(dsn)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ sqlDB, err := s.DB.DB()
+ if err != nil {
+ t.Fatal(err)
+ }
+ sqlDB.SetMaxOpenConns(1)
+ sqlDB.SetMaxIdleConns(1)
+ skew := func(d time.Duration) {
+ t.Helper()
+ if err := s.DB.Exec("SET timestamp = ?", float64(time.Now().Add(d).UnixMicro())/1e6).Error; err != nil {
+ t.Fatal(err)
+ }
+ }
+ defer s.DB.Exec("SET timestamp = DEFAULT")
+
+ const owner = 41
+ session := fmt.Sprintf("clock-skew-%d", time.Now().UnixNano())
+ defer s.DB.Where("session_id = ?", session).Delete(&assistantGenerationRow{})
+ approval := func(id string) error {
+ return s.CreateNativeApproval(owner, session, "phase", domain.AssistantNativeApproval{ID: id, GenerationID: session, ActorID: owner, Tool: "Write", Input: json.RawMessage(`{"path":"a.txt"}`)})
+ }
+
+ // Database clock two minutes behind this host: the turn just checked in.
+ skew(-2 * time.Minute)
+ g := domain.AssistantGeneration{ID: session, SessionID: session, UserID: owner, ActorID: owner, Status: "running"}
+ if err := s.BeginAssistantGeneration(g); err != nil {
+ t.Fatal(err)
+ }
+ if states := s.AssistantGenerationStatuses(owner, []string{session}); states[session].Stale || states[session].Status != "running" {
+ t.Fatalf("a fresh turn is projected as stale: %+v", states[session])
+ }
+ if got, lease, _ := s.ReapAssistantGeneration(owner, session); got.Status != "running" || lease.Token != "" {
+ t.Fatalf("a fresh turn was reaped: %+v", got)
+ }
+ if err := s.HeartbeatAssistantGeneration(g); err != nil {
+ t.Fatal(err)
+ }
+ if err := approval("fresh"); err != nil {
+ t.Fatalf("a live turn refused an approval: %v", err)
+ }
+ if items := s.PendingNativeApprovals(owner, session); len(items) != 1 {
+ t.Fatalf("a live turn hid its approval: %+v", items)
+ }
+
+ // Database clock two minutes ahead: silent for a minute on that clock.
+ skew(2 * time.Minute)
+ if err := s.DB.Exec("UPDATE assistant_generations SET updated_at = UTC_TIMESTAMP(3) - INTERVAL 60 SECOND WHERE session_id = ?", session).Error; err != nil {
+ t.Fatal(err)
+ }
+ if items := s.PendingNativeApprovals(owner, session); len(items) != 0 {
+ t.Fatalf("a dead turn kept its approval: %+v", items)
+ }
+ if err := approval("late"); err == nil {
+ t.Fatal("a dead turn accepted an approval")
+ }
+ if states := s.AssistantGenerationStatuses(owner, []string{session}); !states[session].Stale || states[session].Status != "interrupted" {
+ t.Fatalf("a dead turn is not projected as stale: %+v", states[session])
+ }
+ if got, _, _ := s.ReapAssistantGeneration(owner, session); got.Status != "interrupted" {
+ t.Fatalf("a dead turn was not reaped: %+v", got)
+ }
+}
diff --git a/internal/repo/assistant_generation_lease_guard_test.go b/internal/repo/assistant_generation_lease_guard_test.go
new file mode 100644
index 00000000..92a7f5c1
--- /dev/null
+++ b/internal/repo/assistant_generation_lease_guard_test.go
@@ -0,0 +1,62 @@
+package repo
+
+import (
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+)
+
+// A lease is recorded only on the running turn it belongs to. A late callback
+// from a finished turn, or one naming another turn of the same conversation,
+// must not leave a token that a later reap would release.
+func TestRecordAssistantGenerationWorkspaceLeaseOnlyOnItsRunningTurn(t *testing.T) {
+ s, err := OpenTest()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ recorded := func() (string, string) {
+ t.Helper()
+ var row assistantGenerationRow
+ if err := s.DB.Where("session_id = ?", "fenced-chat").Take(&row).Error; err != nil {
+ t.Fatal(err)
+ }
+ return row.WorkspaceRunID, row.WorkspaceLease
+ }
+ first := domain.AssistantGeneration{ID: "first", SessionID: "fenced-chat", UserID: 11, Status: "running"}
+ if err = s.BeginAssistantGeneration(first); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "fenced-chat", "other-turn", "run-1", "stray"); err != nil {
+ t.Fatal(err)
+ }
+ if run, lease := recorded(); run != "" || lease != "" {
+ t.Fatalf("a lease for another turn was recorded: %q %q", run, lease)
+ }
+ first.Status = "completed"
+ if err = s.CheckpointAssistantGeneration(first); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "fenced-chat", "first", "run-1", "late"); err != nil {
+ t.Fatal(err)
+ }
+ if run, lease := recorded(); run != "" || lease != "" {
+ t.Fatalf("a finished turn recorded a lease: %q %q", run, lease)
+ }
+
+ // The next turn starts clean and records only its own lease.
+ next := domain.AssistantGeneration{ID: "next", SessionID: "fenced-chat", UserID: 11, Status: "running"}
+ if err = s.BeginAssistantGeneration(next); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "fenced-chat", "next", "run-2", "own"); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.DB.Model(&assistantGenerationRow{}).Where("session_id = ?", "fenced-chat").Update("updated_at", time.Now().Add(-time.Minute).UTC()).Error; err != nil {
+ t.Fatal(err)
+ }
+ if _, lease, ok := s.ReapAssistantGeneration(11, "fenced-chat"); !ok || lease != (AssistantGenerationLease{RunID: "run-2", Token: "own"}) {
+ t.Fatalf("reap returned %+v", lease)
+ }
+}
diff --git a/internal/repo/assistant_generations.go b/internal/repo/assistant_generations.go
index 24875c83..4a406191 100644
--- a/internal/repo/assistant_generations.go
+++ b/internal/repo/assistant_generations.go
@@ -23,6 +23,21 @@ type assistantGenerationRow struct {
// the next turn starts. Rows that ended before the column existed keep
// NULL, which is how read tracking leaves older history unmarked.
FinishedAt *time.Time
+ // WorkspaceRunID and WorkspaceLease name the foreground workspace lease
+ // this turn holds. Only that exact token is released when the turn is
+ // reaped; a lease another turn took in the meantime never matches.
+ WorkspaceRunID string `gorm:"size:64"`
+ WorkspaceLease string `gorm:"size:64"`
+}
+
+// assistantHeartbeatTimeout is how long a running turn may go without a
+// durable heartbeat before a reader that does not own it declares it dead.
+const assistantHeartbeatTimeout = 45 * time.Second
+
+// AssistantGenerationLease is the workspace lease a reaped turn held.
+type AssistantGenerationLease struct {
+ RunID string
+ Token string
}
func (assistantGenerationRow) TableName() string { return "assistant_generations" }
@@ -35,30 +50,119 @@ func decodeGeneration(row assistantGenerationRow) domain.AssistantGeneration {
g.UpdatedAt = row.UpdatedAt
return g
}
-func (r *AssistantRepo) AssistantGeneration(userID int64, sessionID string) (domain.AssistantGeneration, bool) {
+
+// PeekAssistantGeneration reads the latest turn without declaring it dead.
+// The owner of a live turn must use this. AssistantGeneration also reaps a
+// turn whose heartbeat is older than 45 seconds, which is only safe once the
+// owning process is gone: a slow commit stores an old updated_at and the next
+// read would otherwise cancel a CLI that is still working.
+func (r *AssistantRepo) PeekAssistantGeneration(userID int64, sessionID string) (domain.AssistantGeneration, bool) {
+ row, ok := r.loadAssistantGeneration(userID, sessionID)
+ if !ok {
+ return domain.AssistantGeneration{}, false
+ }
+ return decodeGeneration(row), true
+}
+
+func (r *AssistantRepo) loadAssistantGeneration(userID int64, sessionID string) (assistantGenerationRow, bool) {
var row assistantGenerationRow
if err := r.DB.Where("session_id = ? AND user_id = ?", sessionID, userID).Take(&row).Error; err != nil {
- return domain.AssistantGeneration{}, false
+ return assistantGenerationRow{}, false
}
+ return row, true
+}
+
+func (r *AssistantRepo) AssistantGeneration(userID int64, sessionID string) (domain.AssistantGeneration, bool) {
+ g, _, ok := r.ReapAssistantGeneration(userID, sessionID)
+ return g, ok
+}
+
+// ReapAssistantGeneration reads a turn and, when its heartbeat is late,
+// records it as interrupted. The lease is returned only to the caller whose
+// update actually reaped the turn, so the lease is released at most once.
+func (r *AssistantRepo) ReapAssistantGeneration(userID int64, sessionID string) (domain.AssistantGeneration, AssistantGenerationLease, bool) {
+ row, ok := r.loadAssistantGeneration(userID, sessionID)
+ if !ok {
+ return domain.AssistantGeneration{}, AssistantGenerationLease{}, false
+ }
+ var lease AssistantGenerationLease
// A server that died cannot leave a conversation stuck forever. Never replay
// a model turn automatically, because its tools may already have written.
- if row.Status == "running" && time.Since(row.UpdatedAt) > 45*time.Second {
+ if row.Status == "running" && r.heartbeatLate(r.DB, row.UpdatedAt) {
g := decodeGeneration(row)
g.Status = "interrupted"
g.Error = "服务端生成已中断,请检查已执行操作后重试"
g.Failure = &domain.AssistantFailure{Code: "EXECUTION_INTERRUPTED", Phase: "server", Message: g.Error, Suggestion: "检查已执行操作后再重试;可下载诊断信息交给管理员排查。"}
_ = r.DB.Transaction(func(tx *gorm.DB) error {
- res := tx.Model(&assistantGenerationRow{}).Where("session_id = ? AND generation_id = ? AND status = ? AND updated_at = ?", sessionID, row.GenerationID, "running", row.UpdatedAt).Updates(map[string]any{"status": g.Status, "payload": mustJSON(g)})
+ res := r.lateHeartbeatOnly(tx.Model(&assistantGenerationRow{}).Where("session_id = ? AND generation_id = ? AND status = ? AND updated_at = ?", sessionID, row.GenerationID, "running", row.UpdatedAt)).Updates(map[string]any{"status": g.Status, "payload": mustJSON(g), "updated_at": r.generationClock()})
if res.Error != nil || res.RowsAffected != 1 {
return res.Error
}
- return checkpointAssistantDiagnostics(tx, g, nil)
+ if err := checkpointAssistantDiagnostics(tx, g, nil); err != nil {
+ return err
+ }
+ lease = AssistantGenerationLease{RunID: row.WorkspaceRunID, Token: row.WorkspaceLease}
+ return nil
})
- if err := r.DB.Where("session_id = ? AND user_id = ?", sessionID, userID).Take(&row).Error; err != nil {
- return domain.AssistantGeneration{}, false
+ reloaded, ok := r.loadAssistantGeneration(userID, sessionID)
+ if !ok {
+ return domain.AssistantGeneration{}, AssistantGenerationLease{}, false
}
+ row = reloaded
}
- return decodeGeneration(row), true
+ return decodeGeneration(row), lease, true
+}
+
+// RecordAssistantGenerationWorkspaceLease remembers the foreground workspace
+// lease a running turn took, so a reader that reaps the turn after its process
+// died can release exactly that lease. The heartbeat clock is not touched.
+func (r *AssistantRepo) RecordAssistantGenerationWorkspaceLease(userID int64, sessionID, generationID, runID, token string) error {
+ return r.DB.Model(&assistantGenerationRow{}).
+ Where("session_id = ? AND user_id = ? AND generation_id = ? AND status = ?", sessionID, userID, generationID, "running").
+ UpdateColumns(map[string]any{"workspace_run_id": runID, "workspace_lease": token}).Error
+}
+
+// heartbeatNow is the clock a heartbeat's age is measured on. MySQL stamps
+// heartbeats with its own UTC_TIMESTAMP(3) (see generationClock), so their age
+// is measured on that same clock: comparing them with this machine's time.Now()
+// would turn any skew between the API host and the database into turns reaped
+// while alive, or never reaped. ok is false when the clock cannot be read, and
+// callers then treat the heartbeat as fresh rather than reap on a guess.
+func (r *AssistantRepo) heartbeatNow(db *gorm.DB) (time.Time, bool) {
+ if !r.isMySQL() {
+ return time.Now().UTC(), true
+ }
+ var now time.Time
+ if err := db.Raw("SELECT UTC_TIMESTAMP(3)").Scan(&now).Error; err != nil || now.IsZero() {
+ return time.Time{}, false
+ }
+ return now, true
+}
+
+// heartbeatLate reports a heartbeat older than assistantHeartbeatTimeout on
+// the heartbeat clock.
+func (r *AssistantRepo) heartbeatLate(db *gorm.DB, updatedAt time.Time) bool {
+ now, ok := r.heartbeatNow(db)
+ return ok && now.Sub(updatedAt) > assistantHeartbeatTimeout
+}
+
+// lateHeartbeatOnly lets the database itself confirm, when the reap commits,
+// that the heartbeat is still late on its own clock.
+func (r *AssistantRepo) lateHeartbeatOnly(q *gorm.DB) *gorm.DB {
+ if !r.isMySQL() {
+ return q
+ }
+ return q.Where("updated_at < UTC_TIMESTAMP(3) - INTERVAL ? MICROSECOND", assistantHeartbeatTimeout.Microseconds())
+}
+
+// generationClock is the commit-time clock on MySQL. A Go timestamp captured
+// before a stalled commit is already older than the 45 second reap when it
+// becomes visible, so the next read interrupts a turn that just checked in.
+func (r *AssistantRepo) generationClock() any {
+ if r.isMySQL() {
+ return clause.Expr{SQL: "UTC_TIMESTAMP(3)"}
+ }
+ return time.Now().UTC()
}
// BeginAssistantGeneration may lock a session that does not exist yet; two
@@ -113,7 +217,7 @@ func (r *AssistantRepo) BeginAssistantGeneration(g domain.AssistantGeneration, p
if old.Status == "running" {
return fmt.Errorf("此会话正在生成,请等待完成或停止")
}
- if err := tx.Model(&old).Updates(map[string]any{"generation_id": g.ID, "status": "running", "cancel_requested": false, "payload": mustJSON(g), "updated_at": time.Now().UTC(), "finished_at": nil}).Error; err != nil {
+ if err := tx.Model(&old).Updates(map[string]any{"generation_id": g.ID, "status": "running", "cancel_requested": false, "payload": mustJSON(g), "updated_at": r.generationClock(), "finished_at": nil, "workspace_run_id": "", "workspace_lease": ""}).Error; err != nil {
return err
}
return checkpointAssistantDiagnostics(tx, g, nil)
@@ -124,6 +228,12 @@ func (r *AssistantRepo) BeginAssistantGeneration(g domain.AssistantGeneration, p
if err := tx.Create(&assistantGenerationRow{SessionID: g.SessionID, UserID: g.UserID, GenerationID: g.ID, Status: "running", Payload: mustJSON(g), UpdatedAt: time.Now().UTC()}).Error; err != nil {
return err
}
+ if r.isMySQL() {
+ // The first heartbeat is on the database clock like every later one.
+ if err := tx.Model(&assistantGenerationRow{}).Where("session_id = ?", g.SessionID).UpdateColumn("updated_at", r.generationClock()).Error; err != nil {
+ return err
+ }
+ }
return checkpointAssistantDiagnostics(tx, g, nil)
})
})
@@ -155,7 +265,7 @@ func (r *AssistantRepo) checkpointAssistantGeneration(g domain.AssistantGenerati
current.NativeApprovals = nil
}
now := time.Now().UTC()
- changes := map[string]any{"status": g.Status, "payload": mustJSON(current), "updated_at": now}
+ changes := map[string]any{"status": g.Status, "payload": mustJSON(current), "updated_at": r.generationClock()}
if g.Status != "running" {
changes["finished_at"] = now
}
@@ -163,8 +273,17 @@ func (r *AssistantRepo) checkpointAssistantGeneration(g domain.AssistantGenerati
if res.Error != nil {
return res.Error
}
+ // MySQL counts changed rows. A heartbeat whose payload and clock match
+ // the stored row reports 0 even though the turn is still running.
+ // Treating that as "the turn disappeared" cancels a live CLI.
if res.RowsAffected != 1 {
- return fmt.Errorf("生成状态已变化")
+ if g.Status != "running" {
+ return fmt.Errorf("生成状态已变化")
+ }
+ var again assistantGenerationRow
+ if err := tx.Where("session_id = ? AND user_id = ? AND generation_id = ? AND status = ?", g.SessionID, g.UserID, g.ID, "running").Take(&again).Error; err != nil {
+ return fmt.Errorf("生成状态已变化")
+ }
}
if !withDiagnostics {
return nil
@@ -182,7 +301,7 @@ func (r *AssistantRepo) SteerAssistantGeneration(userID int64, sessionID, id str
}
g := decodeGeneration(row)
g.Steer = &steer
- return tx.Model(&row).Updates(map[string]any{"cancel_requested": true, "payload": mustJSON(g), "updated_at": time.Now().UTC()}).Error
+ return tx.Model(&row).Updates(map[string]any{"cancel_requested": true, "payload": mustJSON(g), "updated_at": r.generationClock()}).Error
})
}
func (r *AssistantRepo) CancelAssistantGeneration(userID int64, sessionID, id string) error {
@@ -207,6 +326,9 @@ type AssistantGenerationState struct {
// Seen reports that the reader has opened the conversation since this
// reply (see MarkAssistantSessionRead).
Seen bool
+ // Stale reports a turn still stored as running whose heartbeat is late.
+ // Status reads "interrupted" for it, but nothing has been written yet.
+ Stale bool
}
// AssistantGenerationStatuses reads the latest reply of each conversation
@@ -232,9 +354,11 @@ func (r *AssistantRepo) AssistantGenerationStatuses(readerID int64, ids []string
if err != nil {
return out
}
+ now, clockOK := r.heartbeatNow(r.DB)
for _, row := range rows {
status := row.Status
- if status == "running" && time.Since(row.UpdatedAt) > 45*time.Second {
+ stale := status == "running" && clockOK && now.Sub(row.UpdatedAt) > assistantHeartbeatTimeout
+ if stale {
status = "interrupted"
}
out[row.SessionID] = AssistantGenerationState{
@@ -242,6 +366,7 @@ func (r *AssistantRepo) AssistantGenerationStatuses(readerID int64, ids []string
Status: status,
FinishedAt: row.FinishedAt,
Seen: row.ReadID != nil && *row.ReadID == row.GenerationID,
+ Stale: stale,
}
}
return out
diff --git a/internal/repo/assistant_generations_test.go b/internal/repo/assistant_generations_test.go
index 8b765f7d..29fd7317 100644
--- a/internal/repo/assistant_generations_test.go
+++ b/internal/repo/assistant_generations_test.go
@@ -59,3 +59,87 @@ func TestAssistantGenerationClaimCheckpointAndPrivacy(t *testing.T) {
t.Fatal("orphan generation did not become interrupted")
}
}
+
+func TestPeekAssistantGenerationLeavesStaleTurnRunning(t *testing.T) {
+ s, err := OpenTest()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ g := domain.AssistantGeneration{ID: "live", SessionID: "live-chat", UserID: 11, Status: "running"}
+ if err = s.BeginAssistantGeneration(g); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.DB.Model(&assistantGenerationRow{}).Where("session_id = ?", "live-chat").Update("updated_at", time.Now().Add(-time.Minute)).Error; err != nil {
+ t.Fatal(err)
+ }
+ peek, ok := s.PeekAssistantGeneration(11, "live-chat")
+ if !ok || peek.Status != "running" {
+ t.Fatalf("peek reaped a live turn: %+v", peek)
+ }
+ if err = s.HeartbeatAssistantGeneration(g); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.HeartbeatAssistantGeneration(g); err != nil {
+ t.Fatal(err)
+ }
+ peek, _ = s.PeekAssistantGeneration(11, "live-chat")
+ if peek.Status != "running" || time.Since(peek.UpdatedAt) > 5*time.Second {
+ t.Fatalf("heartbeat did not refresh the live turn: %+v", peek)
+ }
+}
+
+// Only the reader whose update reaps a turn gets its lease back, and the next
+// turn starts without the previous turn's lease.
+func TestReapAssistantGenerationReturnsRecordedLeaseOnce(t *testing.T) {
+ s, err := OpenTest()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ g := domain.AssistantGeneration{ID: "dead", SessionID: "dead-chat", UserID: 11, Status: "running"}
+ if err = s.BeginAssistantGeneration(g); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "dead-chat", "dead", "run-1", "token-1"); err != nil {
+ t.Fatal(err)
+ }
+ // Recording a lease is not a heartbeat.
+ stale := time.Now().Add(-time.Minute).UTC()
+ if err = s.DB.Model(&assistantGenerationRow{}).Where("session_id = ?", "dead-chat").Update("updated_at", stale).Error; err != nil {
+ t.Fatal(err)
+ }
+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "dead-chat", "dead", "run-1", "token-1"); err != nil {
+ t.Fatal(err)
+ }
+ if peek, _ := s.PeekAssistantGeneration(11, "dead-chat"); time.Since(peek.UpdatedAt) < 50*time.Second {
+ t.Fatalf("recording a lease refreshed the heartbeat: %+v", peek.UpdatedAt)
+ }
+ if states := s.AssistantGenerationStatuses(11, []string{"dead-chat"}); !states["dead-chat"].Stale || states["dead-chat"].Status != "interrupted" {
+ t.Fatalf("late heartbeat not projected as stale: %+v", states["dead-chat"])
+ }
+ reaped, lease, ok := s.ReapAssistantGeneration(11, "dead-chat")
+ if !ok || reaped.Status != "interrupted" || lease != (AssistantGenerationLease{RunID: "run-1", Token: "token-1"}) {
+ t.Fatalf("reap = %+v %+v %v", reaped, lease, ok)
+ }
+ if states := s.AssistantGenerationStatuses(11, []string{"dead-chat"}); states["dead-chat"].Stale {
+ t.Fatalf("a stored interruption is reported as stale: %+v", states["dead-chat"])
+ }
+ if _, lease, _ = s.ReapAssistantGeneration(11, "dead-chat"); lease.Token != "" {
+ t.Fatalf("a second reader got the lease again: %+v", lease)
+ }
+ // Another user cannot record a lease on this turn.
+ next := domain.AssistantGeneration{ID: "next", SessionID: "dead-chat", UserID: 11, Status: "running"}
+ if err = s.BeginAssistantGeneration(next); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.RecordAssistantGenerationWorkspaceLease(12, "dead-chat", "next", "run-1", "intruder"); err != nil {
+ t.Fatal(err)
+ }
+ if err = s.DB.Model(&assistantGenerationRow{}).Where("session_id = ?", "dead-chat").Update("updated_at", stale).Error; err != nil {
+ t.Fatal(err)
+ }
+ if _, lease, _ = s.ReapAssistantGeneration(11, "dead-chat"); lease.Token != "" {
+ t.Fatalf("the next turn inherited a lease: %+v", lease)
+ }
+}
diff --git a/internal/repo/workspace_checkpoints.go b/internal/repo/workspace_checkpoints.go
index ad23ccdb..2e67c0ed 100644
--- a/internal/repo/workspace_checkpoints.go
+++ b/internal/repo/workspace_checkpoints.go
@@ -30,6 +30,11 @@ type WorkspaceCheckpoint struct {
LeaseHandoffID string `gorm:"size:64" json:"-"`
LeaseToken string `gorm:"size:64" json:"-"`
LeaseUntil *time.Time `json:"-"`
+ // LeaseRenewedAt is when the lease was last taken or renewed, on the
+ // database clock on MySQL (see leaseRenewalClock). LeaseUntil is written
+ // by the holder's own clock, so it cannot tell another replica how long
+ // ago the holder renewed.
+ LeaseRenewedAt *time.Time `json:"-"`
RestoreStatus string `gorm:"size:32" json:"-"`
RestoreStarted *time.Time `json:"-"`
RestoreError string `gorm:"type:text" json:"-"`
@@ -50,6 +55,9 @@ var ErrWorkspaceCheckpointBackoff = errors.New("workspace checkpoint save is bac
const workspaceBackgroundSavePrefix = "save:"
+// workspaceUseTerm is how long a workspace lease lasts from its last renewal.
+const workspaceUseTerm = 15 * time.Minute
+
func (r *RunsRepo) WorkspaceCheckpoint(runID string) (WorkspaceCheckpoint, bool) {
var item WorkspaceCheckpoint
err := r.DB.Omit("archive").Where("run_id = ?", runID).Take(&item).Error
@@ -128,7 +136,7 @@ func (r *RunsRepo) beginWorkspaceLease(runID string, now time.Time, touch, resto
if !touch && checkpoint.SaveRetryAt != nil && now.Before(*checkpoint.SaveRetryAt) {
return ErrWorkspaceCheckpointBackoff
}
- until := now.Add(15 * time.Minute)
+ until := now.Add(workspaceUseTerm)
lastUsed := now
if !touch && run.Updated != nil {
lastUsed = *run.Updated
@@ -137,7 +145,7 @@ func (r *RunsRepo) beginWorkspaceLease(runID string, now time.Time, touch, resto
if err := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(&row).Error; err != nil {
return err
}
- values := map[string]any{"lease_token": token, "lease_until": until, "lease_handoff_id": ""}
+ values := map[string]any{"lease_token": token, "lease_until": until, "lease_renewed_at": r.leaseRenewalClock(now), "lease_handoff_id": ""}
if touch {
values["last_used_at"] = now
}
@@ -187,8 +195,8 @@ func (r *RunsRepo) RenewWorkspaceUse(runID, token string, now time.Time) error {
}
// The row is locked through the run and was just read with this token;
// an unchanged lease_until (same-millisecond renewal) is still a renewal.
- until := now.Add(15 * time.Minute)
- return tx.Model(&WorkspaceCheckpoint{}).Where("run_id = ? AND lease_token = ?", runID, token).Update("lease_until", until).Error
+ until := now.Add(workspaceUseTerm)
+ return tx.Model(&WorkspaceCheckpoint{}).Where("run_id = ? AND lease_token = ?", runID, token).Updates(map[string]any{"lease_until": until, "lease_renewed_at": r.leaseRenewalClock(now)}).Error
})
}
@@ -355,6 +363,59 @@ func holdsWorkspaceLease(tx *gorm.DB, runID, token string) error {
return nil
}
+// ReleaseAbandonedWorkspaceUse drops the foreground lease a reaped turn held.
+// A dead API process cannot finish the lease it held in memory, and the next
+// turn would fail for the rest of the 15 minute term.
+//
+// Two fences keep a live owner's lease in place. The token must be the one the
+// reaped turn recorded, so a lease another turn took after the reap never
+// matches. Its last renewal must also be older than the heartbeat timeout:
+// every holder renews well within it (generations every 500ms, workspace turns
+// every 30s), so a lease still being renewed by a process on any replica is
+// kept. A background save token never matches a foreground token.
+//
+// The renewal age is measured on the clock that stamped lease_renewed_at (the
+// database's on MySQL), so a holder whose clock runs behind the reader's does
+// not look silent. A lease taken before that column existed has no stamp and
+// falls back to its lease_until.
+func (r *RunsRepo) ReleaseAbandonedWorkspaceUse(runID, token string, now time.Time) (bool, error) {
+ runID, token = strings.TrimSpace(runID), strings.TrimSpace(token)
+ if runID == "" || token == "" || strings.HasPrefix(token, workspaceBackgroundSavePrefix) {
+ return false, nil
+ }
+ released := false
+ err := r.DB.Transaction(func(tx *gorm.DB) error {
+ if _, err := r.takeRunForUpdate(tx, runID); err != nil {
+ if errors.Is(err, gorm.ErrRecordNotFound) {
+ return nil
+ }
+ return err
+ }
+ silent := tx.Where("lease_renewed_at IS NULL AND lease_until <= ?", now.Add(workspaceUseTerm-assistantHeartbeatTimeout))
+ if r.isMySQL() {
+ silent = silent.Or("lease_renewed_at <= UTC_TIMESTAMP(3) - INTERVAL ? MICROSECOND", assistantHeartbeatTimeout.Microseconds())
+ } else {
+ silent = silent.Or("lease_renewed_at <= ?", now.Add(-assistantHeartbeatTimeout))
+ }
+ result := tx.Model(&WorkspaceCheckpoint{}).
+ Where("run_id = ? AND lease_token = ? AND lease_until > ?", runID, token, now).
+ Where(silent).
+ Updates(map[string]any{"lease_token": "", "lease_until": nil, "lease_handoff_id": ""})
+ released = result.RowsAffected == 1
+ return result.Error
+ })
+ return released, err
+}
+
+// leaseRenewalClock stamps lease_renewed_at: the database's commit-time clock
+// on MySQL, shared by every replica, and the caller's clock elsewhere.
+func (r *RunsRepo) leaseRenewalClock(now time.Time) any {
+ if r.isMySQL() {
+ return clause.Expr{SQL: "UTC_TIMESTAMP(3)"}
+ }
+ return now
+}
+
// ReleaseWorkspace marks a workspace no conversation is bound to any more, so
// the idle sweep saves and reclaims it on its next pass.
func (r *RunsRepo) ReleaseWorkspace(runID string, now time.Time) error {
diff --git a/internal/repo/workspace_checkpoints_test.go b/internal/repo/workspace_checkpoints_test.go
index 00b61e2a..a5c0f764 100644
--- a/internal/repo/workspace_checkpoints_test.go
+++ b/internal/repo/workspace_checkpoints_test.go
@@ -8,6 +8,95 @@ import (
"time"
)
+// A reaped turn releases only the lease it recorded, and only once nothing
+// renews it any more. A live holder on any replica, a newer turn's lease and a
+// background save all keep theirs.
+func TestReleaseAbandonedWorkspaceUseFreesOnlyTheReapedTurnsLease(t *testing.T) {
+ s, err := OpenTest()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ now := time.Now().UTC()
+ taken := now.Add(-2 * time.Minute)
+ for _, id := range []string{"abandoned-use", "renewed-use", "replaced-use", "background-save"} {
+ if err = s.PutRun(domain.Run{ID: id, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ release := func(runID, token string) bool {
+ t.Helper()
+ released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return released
+ }
+ leaseOf := func(runID string) string {
+ t.Helper()
+ item, _ := s.WorkspaceCheckpoint(runID)
+ return item.LeaseToken
+ }
+
+ abandoned, err := s.BeginWorkspaceUse("abandoned-use", taken)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !release("abandoned-use", abandoned) {
+ t.Fatal("an unrenewed lease of a reaped turn was kept")
+ }
+ if release("abandoned-use", abandoned) {
+ t.Fatal("a lease was released twice")
+ }
+ if _, err = s.BeginWorkspaceUse("abandoned-use", now); err != nil {
+ t.Fatal("abandoned foreground lease still blocked the next turn", err)
+ }
+
+ // Another replica still executing the turn renews its lease.
+ renewed, err := s.BeginWorkspaceUse("renewed-use", taken)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err = s.RenewWorkspaceUse("renewed-use", renewed, now.Add(-10*time.Second)); err != nil {
+ t.Fatal(err)
+ }
+ if release("renewed-use", renewed) || leaseOf("renewed-use") != renewed {
+ t.Fatal("a lease its holder still renews was released")
+ }
+
+ // The reaped turn's lease ended and a newer turn took the workspace
+ // before the reader released: the stale token must not free it.
+ old, err := s.BeginWorkspaceUse("replaced-use", taken)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err = s.FinishWorkspaceUse("replaced-use", old, taken, nil, ""); err != nil {
+ t.Fatal(err)
+ }
+ newer, err := s.BeginWorkspaceUse("replaced-use", taken)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if release("replaced-use", old) || leaseOf("replaced-use") != newer {
+ t.Fatal("a newer turn's lease was released with the reaped turn's token")
+ }
+
+ token, err := s.BeginWorkspaceUse("background-save", taken)
+ if err != nil {
+ t.Fatal(err)
+ }
+ save, err := s.HandoffWorkspaceUseToSave("background-save", token, taken)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if release("background-save", token) || release("background-save", save) || leaseOf("background-save") != save {
+ t.Fatal("background save lease dropped")
+ }
+ if release("", abandoned) || release("missing-run", abandoned) {
+ t.Fatal("released a lease without a run")
+ }
+}
+
func TestWorkspaceCheckpointLeaseCASAndDurableIdleDeadline(t *testing.T) {
s, err := OpenTest()
if err != nil {
diff --git a/internal/repo/workspace_lease_clock_mysql_test.go b/internal/repo/workspace_lease_clock_mysql_test.go
new file mode 100644
index 00000000..d0902d5f
--- /dev/null
+++ b/internal/repo/workspace_lease_clock_mysql_test.go
@@ -0,0 +1,79 @@
+package repo
+
+import (
+ "fmt"
+ "os"
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+)
+
+// lease_until is written on the holder's clock. Whether a holder on another
+// replica stopped renewing is read from lease_renewed_at, which MySQL stamps
+// with its own clock, so a holder whose clock runs behind keeps a lease it
+// just renewed, and a lease nobody renewed for 45 seconds on the database
+// clock is released. SET timestamp moves the database clock of the one
+// connection the store is pinned to.
+func TestMySQLAbandonedLeaseAgeIsMeasuredOnTheDatabaseClock(t *testing.T) {
+ dsn := os.Getenv("AGENTFLOW_TEST_MYSQL_DSN")
+ if dsn == "" {
+ t.Skip("local isolated MySQL required")
+ }
+ s, err := Open(dsn)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ sqlDB, err := s.DB.DB()
+ if err != nil {
+ t.Fatal(err)
+ }
+ sqlDB.SetMaxOpenConns(1)
+ sqlDB.SetMaxIdleConns(1)
+ skew := func(d time.Duration) {
+ t.Helper()
+ if err := s.DB.Exec("SET timestamp = ?", float64(time.Now().Add(d).UnixMicro())/1e6).Error; err != nil {
+ t.Fatal(err)
+ }
+ }
+ defer s.DB.Exec("SET timestamp = DEFAULT")
+
+ runID := fmt.Sprintf("lease-clock-%d", time.Now().UnixNano())
+ if err := s.PutRun(domain.Run{ID: runID, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {
+ t.Fatal(err)
+ }
+ defer s.DB.Where("run_id = ?", runID).Delete(&WorkspaceCheckpoint{})
+ defer s.DB.Where("id = ?", runID).Delete(&runRow{})
+ now := time.Now().UTC()
+ token, err := s.BeginWorkspaceUse(runID, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // The holder's clock runs two minutes behind the reader's and it renews
+ // now: its lease_until looks two minutes old to the reader.
+ if err := s.RenewWorkspaceUse(runID, token, now.Add(-2*time.Minute)); err != nil {
+ t.Fatal(err)
+ }
+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now); err != nil || released {
+ t.Fatalf("a lease its holder just renewed was released: %v %v", released, err)
+ }
+ // Two minutes later on the database clock nothing renewed it.
+ skew(2 * time.Minute)
+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now); err != nil || !released {
+ t.Fatalf("a lease nobody renewed for two minutes was kept: %v %v", released, err)
+ }
+ skew(0)
+
+ // A lease taken before lease_renewed_at existed falls back to lease_until.
+ legacy, err := s.BeginWorkspaceUse(runID, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := s.DB.Model(&WorkspaceCheckpoint{}).Where("run_id = ?", runID).Updates(map[string]any{"lease_renewed_at": nil, "lease_until": now.Add(workspaceUseTerm - time.Minute)}).Error; err != nil {
+ t.Fatal(err)
+ }
+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, legacy, now); err != nil || !released {
+ t.Fatalf("an unstamped lease silent for a minute was kept: %v %v", released, err)
+ }
+}
diff --git a/internal/repo/workspace_lease_renewal_stamp_test.go b/internal/repo/workspace_lease_renewal_stamp_test.go
new file mode 100644
index 00000000..d4fc3925
--- /dev/null
+++ b/internal/repo/workspace_lease_renewal_stamp_test.go
@@ -0,0 +1,80 @@
+package repo
+
+import (
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+)
+
+// Taking and renewing a lease stamp lease_renewed_at, and the abandoned-lease
+// release goes by that stamp when it is there. A lease taken before the column
+// existed has no stamp and is still judged by its lease_until.
+func TestReleaseAbandonedWorkspaceUseJudgesByRenewalStampOrLegacyLeaseUntil(t *testing.T) {
+ s, err := OpenTest()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ now := time.Now().UTC()
+ for _, id := range []string{"stamped-take", "legacy-silent", "legacy-renewed", "stamp-decides"} {
+ if err = s.PutRun(domain.Run{ID: id, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ take := func(runID string, at time.Time) string {
+ t.Helper()
+ token, err := s.BeginWorkspaceUse(runID, at)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return token
+ }
+ set := func(runID string, values map[string]any) {
+ t.Helper()
+ if err := s.DB.Model(&WorkspaceCheckpoint{}).Where("run_id = ?", runID).Updates(values).Error; err != nil {
+ t.Fatal(err)
+ }
+ }
+ release := func(runID, token string) bool {
+ t.Helper()
+ released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return released
+ }
+
+ // Taking the lease stamps it on the caller's clock (SQLite).
+ taken := now.Add(-20 * time.Second)
+ take("stamped-take", taken)
+ var row WorkspaceCheckpoint
+ if err := s.DB.Where("run_id = ?", "stamped-take").Take(&row).Error; err != nil {
+ t.Fatal(err)
+ }
+ if row.LeaseRenewedAt == nil || row.LeaseRenewedAt.Sub(taken).Abs() > time.Millisecond {
+ t.Fatalf("taking a lease did not stamp lease_renewed_at with %v: %v", taken, row.LeaseRenewedAt)
+ }
+
+ // Unstamped lease whose lease_until says it was last renewed a minute ago.
+ legacy := take("legacy-silent", now)
+ set("legacy-silent", map[string]any{"lease_renewed_at": nil, "lease_until": now.Add(workspaceUseTerm - time.Minute)})
+ if !release("legacy-silent", legacy) {
+ t.Fatal("an unstamped lease silent for a minute was kept")
+ }
+
+ // Unstamped lease renewed ten seconds ago by its lease_until.
+ renewed := take("legacy-renewed", now)
+ set("legacy-renewed", map[string]any{"lease_renewed_at": nil, "lease_until": now.Add(workspaceUseTerm - 10*time.Second)})
+ if release("legacy-renewed", renewed) {
+ t.Fatal("an unstamped lease renewed ten seconds ago was released")
+ }
+
+ // A stamp from ten seconds ago keeps the lease even though lease_until,
+ // written on the holder's clock, looks a minute old.
+ stamped := take("stamp-decides", now)
+ set("stamp-decides", map[string]any{"lease_renewed_at": now.Add(-10 * time.Second), "lease_until": now.Add(workspaceUseTerm - time.Minute)})
+ if release("stamp-decides", stamped) {
+ t.Fatal("a freshly stamped lease was released by its lease_until")
+ }
+}
diff --git a/internal/repo/workspace_lease_take_clock_mysql_test.go b/internal/repo/workspace_lease_take_clock_mysql_test.go
new file mode 100644
index 00000000..82f8b4fd
--- /dev/null
+++ b/internal/repo/workspace_lease_take_clock_mysql_test.go
@@ -0,0 +1,59 @@
+package repo
+
+import (
+ "fmt"
+ "os"
+ "testing"
+ "time"
+
+ "agentflow/internal/domain"
+)
+
+// Taking a lease stamps lease_renewed_at on the database clock too, not only
+// renewing it. A holder whose clock runs two minutes behind the reader's has
+// just taken the lease and not renewed yet: its lease_until looks two minutes
+// old to the reader, but the lease must be kept until the database clock says
+// it went unrenewed.
+func TestMySQLLeaseJustTakenByAHolderWithASlowClockIsKept(t *testing.T) {
+ dsn := os.Getenv("AGENTFLOW_TEST_MYSQL_DSN")
+ if dsn == "" {
+ t.Skip("local isolated MySQL required")
+ }
+ s, err := Open(dsn)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ sqlDB, err := s.DB.DB()
+ if err != nil {
+ t.Fatal(err)
+ }
+ sqlDB.SetMaxOpenConns(1)
+ sqlDB.SetMaxIdleConns(1)
+ skew := func(d time.Duration) {
+ t.Helper()
+ if err := s.DB.Exec("SET timestamp = ?", float64(time.Now().Add(d).UnixMicro())/1e6).Error; err != nil {
+ t.Fatal(err)
+ }
+ }
+ defer s.DB.Exec("SET timestamp = DEFAULT")
+
+ runID := fmt.Sprintf("lease-take-clock-%d", time.Now().UnixNano())
+ if err := s.PutRun(domain.Run{ID: runID, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {
+ t.Fatal(err)
+ }
+ defer s.DB.Where("run_id = ?", runID).Delete(&WorkspaceCheckpoint{})
+ defer s.DB.Where("id = ?", runID).Delete(&runRow{})
+ now := time.Now().UTC()
+ token, err := s.BeginWorkspaceUse(runID, now.Add(-2*time.Minute))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now); err != nil || released {
+ t.Fatalf("a lease its holder just took was released: %v %v", released, err)
+ }
+ skew(2 * time.Minute)
+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now); err != nil || !released {
+ t.Fatalf("a lease nobody renewed for two minutes since it was taken was kept: %v %v", released, err)
+ }
+}
diff --git a/migrations/032_assistant_generation_workspace_lease.sql b/migrations/032_assistant_generation_workspace_lease.sql
new file mode 100644
index 00000000..b2087cf2
--- /dev/null
+++ b/migrations/032_assistant_generation_workspace_lease.sql
@@ -0,0 +1,35 @@
+-- Optional manual additive DDL. Startup AutoMigrate also adds these columns.
+-- A running turn records the foreground workspace lease it holds, so a reader
+-- that reaps the turn after its process died releases exactly that lease.
+-- No backfill: turns started before the columns existed record nothing and
+-- keep the old behaviour (their lease runs out after 15 minutes).
+-- run_workspace_checkpoints.lease_renewed_at stamps each lease take and renewal
+-- with the database clock, so a replica judges how long ago another replica
+-- renewed without comparing two hosts' clocks. A lease taken before the column
+-- existed has no stamp and is judged by its lease_until as before.
+SET @asst_ws_run_ddl = IF(
+ (SELECT COUNT(*) FROM information_schema.columns
+ WHERE table_schema=DATABASE() AND table_name='assistant_generations' AND column_name='workspace_run_id')=0,
+ 'ALTER TABLE assistant_generations ADD COLUMN workspace_run_id VARCHAR(64) NULL',
+ 'SELECT 1');
+PREPARE asst_ws_run_stmt FROM @asst_ws_run_ddl;
+EXECUTE asst_ws_run_stmt;
+DEALLOCATE PREPARE asst_ws_run_stmt;
+
+SET @asst_ws_lease_ddl = IF(
+ (SELECT COUNT(*) FROM information_schema.columns
+ WHERE table_schema=DATABASE() AND table_name='assistant_generations' AND column_name='workspace_lease')=0,
+ 'ALTER TABLE assistant_generations ADD COLUMN workspace_lease VARCHAR(64) NULL',
+ 'SELECT 1');
+PREPARE asst_ws_lease_stmt FROM @asst_ws_lease_ddl;
+EXECUTE asst_ws_lease_stmt;
+DEALLOCATE PREPARE asst_ws_lease_stmt;
+
+SET @ws_lease_renewed_ddl = IF(
+ (SELECT COUNT(*) FROM information_schema.columns
+ WHERE table_schema=DATABASE() AND table_name='run_workspace_checkpoints' AND column_name='lease_renewed_at')=0,
+ 'ALTER TABLE run_workspace_checkpoints ADD COLUMN lease_renewed_at DATETIME(3) NULL',
+ 'SELECT 1');
+PREPARE ws_lease_renewed_stmt FROM @ws_lease_renewed_ddl;
+EXECUTE ws_lease_renewed_stmt;
+DEALLOCATE PREPARE ws_lease_renewed_stmt;