Self-hosted workflows for coding-agent delivery. Orchestrate CodeBuddy, Codex and Claude Code with scripts and human approval gates. Includes Local workflows for coding assistants.
JavaScript agent-orchestrationagentic-workflowai-agentsai-codingclaude-codecode-reviewcodebuddycodexcoding-agentscursordeveloper-toolshuman-on-the-loopllmmcpmulti-agentsdlcself-hostedsopworkflow-automationworkflow-engine
fix(assistant): keep a live generation through a slow heartbeat (#810)
crispinhu27dd56585276d8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 88f9bcb9..5513299a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,8 @@ Platform releases use `vX.Y.Z` tags and publish `ghcr.io/tencent/loopforge`. Loo## Unreleased+- A conversation reply that is still running keeps running through a slow heartbeat. A poll, the conversation list, a follow-up message or a steer used to record it as interrupted after 45 seconds without a stored heartbeat; the process then cancelled the CLI and the next message failed with 「工作区正在使用或保存,请稍后重试」 for up to 15 minutes. Reads of a turn this process executes no longer reap it, and an unchanged heartbeat on MySQL no longer counts as a lost turn. On MySQL a heartbeat's age is measured on the database clock that stamped it, so skew between an API host and MySQL neither reaps a live turn nor keeps a dead one; whether a lease holder stopped renewing is likewise judged from `run_workspace_checkpoints.lease_renewed_at`, stamped on the database clock, not from another replica's `lease_until`. When a turn whose process died is reaped, the workspace lease it recorded is released if nothing renewed it for 45 seconds, so the next message starts at once; a lease another turn took, one still renewed on another replica, and a background save are kept. `migrations/032_assistant_generation_workspace_lease.sql` adds the three columns by hand; startup AutoMigrate adds them too.+- The platform image bundles DeepSeek Harness `@deepseek-ai/[email protected]` next to Codex and CodeBuddy, so a DeepSeek key works in conversations and SOP roles without an install step. 0.3.8 had no `dsh` and answered a DeepSeek Harness chat with “AI 助手生成失败,请稍后重试”. The image download grows by about 180 MB. An installation without the image installs the same pinned version from the conversation: every dsh release is a prerelease requiring Node 24.9, and npm on Node 22 found no version for the bare package name.- A conversation workspace whose image lacks the chosen CLI installs Cursor CLI and Antigravity again. The sandbox drops every capability, so the installer's `tar` running as root could not restore the archive owner (`Cannot change ownership to uid 2000`), and the turn failed with “准备 cursor 原生CLI失败”. The workspace install of DeepSeek Harness uses the pinned `@deepseek-ai/[email protected]` with relaxed engines, as the platform install already did; the bare package name resolved to no version.- A conversation workspace on a custom Dockerfile without Node.js, npm or Python 3 runs the CLIs bundled in the platform image (Codex, CodeBuddy, DeepSeek Harness). The platform copies its Node.js, npm and a relocatable Python 3.12 into `${LOOPFORGE_DATA_DIR}/cli/toolchain/<build>/` on first use, and a bundled CLI the first time a sandbox needs it; every sandbox mounts that directory read-only at the end of `PATH`, so tools shipped by the image still win. The platform image gains python-build-standalone (about 30 MB to download, checked against a pinned SHA-256). Other CLIs are still installed inside the sandbox, now with the mounted npm. A musl (Alpine) or foreign-architecture image, which lacks the loader the copied `node` needs, keeps its own `PATH` and installs with its package manager as before. Each platform build gets its own directory of about 200 MB plus the CLIs it copied; directories of earlier builds are not removed, so delete the other `cli/toolchain/<build>/` directories after an upgrade.diff --git a/internal/engine/conversation_workspace.go b/internal/engine/conversation_workspace.goindex e07e57c5..59a6db63 100644--- a/internal/engine/conversation_workspace.go+++ b/internal/engine/conversation_workspace.go@@ -90,7 +90,7 @@ func (e *Engine) conversationGenerationRunning(run domain.Run) bool {if !ok {return false}- job, ok := e.repo.AssistantGeneration(session.UserID, session.ID)+ job, ok := e.repo.PeekAssistantGeneration(session.UserID, session.ID)return ok && job.Status == "running"}func (e *Engine) EndConversationWorkspaceUse(ctx context.Context, id, token string) error {diff --git a/internal/httpapi/assistant.go b/internal/httpapi/assistant.goindex 32a1143c..8d950d3d 100644--- a/internal/httpapi/assistant.go+++ b/internal/httpapi/assistant.go@@ -207,7 +207,7 @@ func (h *assistantHandlers) chat(c *gin.Context) {}sink, _ := c.Request.Context().Value(assistantGenerationSinkKey{}).(func(string, any))if sink == nil {- if job, ok := h.repo.AssistantGeneration(ownerID, req.SessionID); ok && job.Status == "running" {+ if job, ok := h.readGeneration(ownerID, req.SessionID); ok && job.Status == "running" {c.JSON(409, gin.H{"error": "此会话正在后台生成"})return}@@ -338,6 +338,12 @@ func (h *assistantHandlers) chat(c *gin.Context) {defer files.Cleanup()if req.RunID == "" && h.engine != nil {sess.workspaceTurn = newAssistantWorkspaceTurn(h.engine, cancelModel)+ if scope := assistantControlScopeFrom(modelCtx); scope.GenerationID != "" {+ generation := domain.AssistantGeneration{ID: scope.GenerationID, SessionID: req.SessionID, UserID: ownerID}+ sess.workspaceTurn.recordLeases(func(runID, token string) {+ h.recordGenerationWorkspaceLease(generation, runID, token)+ })+ }defer func() {if err := sess.workspaceTurn.close(modelCtx); err != nil {warning := "工作区保存未完成,暂缓回收:" + err.Error()diff --git a/internal/httpapi/assistant_direct_generation.go b/internal/httpapi/assistant_direct_generation.goindex ac3ba1b6..5355b69b 100644--- a/internal/httpapi/assistant_direct_generation.go+++ b/internal/httpapi/assistant_direct_generation.go@@ -10,6 +10,10 @@ import (// Synchronous shared chat must occupy the same durable slot as /generations.// Otherwise a second process can save a pending turn while the first CLI writes// the shared transcript. A local file lock additionally fences stale writers.+//+// The caller must defer the returned finish right after a successful claim (as+// chat does): it is the only place the turn leaves assistantDirectGenerations,+// and a deferred call also runs when the handler panics.func (h *assistantHandlers) claimDirectSharedGeneration(ctx context.Context, req assistantRequest, ownerID int64, actor domain.User, cancel context.CancelFunc) (domain.AssistantGeneration, *assistantDiagnosticsCollector, func(bool), error) {g := domain.AssistantGeneration{ID: newAssistantSessionID(), SessionID: req.SessionID, UserID: ownerID, ActorID: actor.ID, ActorName: actor.LoginName, ProjectID: req.ProjectID, Status: "running", StartedAt: time.Now().UTC()}pending := assistantConversation{ID: req.SessionID, Shared: true, ProjectID: req.ProjectID, Model: req.Model, Effort: req.Effort, RunID: req.RunID, Messages: req.Messages}@@ -17,6 +21,7 @@ func (h *assistantHandlers) claimDirectSharedGeneration(ctx context.Context, reqreturn g, nil, nil, err}diagnostics := newAssistantDiagnosticsCollector(g)+ assistantDirectGenerations.Store(g.ID, struct{}{})untrack := inflightTasks.Add()stop, done := make(chan struct{}), make(chan struct{})go func() {@@ -30,7 +35,7 @@ func (h *assistantHandlers) claimDirectSharedGeneration(ctx context.Context, reqcase <-ctx.Done():returncase <-ticker.C:- current, ok := h.repo.AssistantGeneration(ownerID, req.SessionID)+ current, ok := h.repo.PeekAssistantGeneration(ownerID, req.SessionID)if !ok || current.ID != g.ID {diagnostics.terminate("GENERATION_SUPERSEDED", "当前生成记录已被替换或移除", actor)cancel()@@ -56,13 +61,14 @@ func (h *assistantHandlers) claimDirectSharedGeneration(ctx context.Context, req}()return g, diagnostics, func(completed bool) {defer untrack()+ defer assistantDirectGenerations.Delete(g.ID)close(stop)<-doneg.Status = "failed"if completed {g.Status = "completed"} else {- current, ok := h.repo.AssistantGeneration(ownerID, req.SessionID)+ current, ok := h.repo.PeekAssistantGeneration(ownerID, req.SessionID)if ctx.Err() != nil || ok && current.ID == g.ID && current.CancelRequested {g.Status = "cancelled"}diff --git a/internal/httpapi/assistant_direct_generation_live_test.go b/internal/httpapi/assistant_direct_generation_live_test.gonew file mode 100644index 00000000..a5fce3a5--- /dev/null+++ b/internal/httpapi/assistant_direct_generation_live_test.go@@ -0,0 +1,60 @@+package httpapi++import (+ "bytes"+ "context"+ "encoding/json"+ "testing"+ "time"++ "agentflow/internal/domain"+ "github.com/gin-gonic/gin"+)++// A synchronous shared turn has no live stream, but it is just as alive while+// this process runs it: a poll or a session read must not reap it after one+// slow heartbeat, and the sidebar must keep showing it as running.+func TestDirectSharedGenerationIsNotReapedByItsOwnProcess(t *testing.T) {+ h, u := workbenchFixture(t)+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ req := assistantRequest{ProjectID: "workbench", SessionID: "direct-shared", Messages: []domain.AssistantMessage{{Role: "user", Content: "hi", CreatedAt: time.Now().UTC()}}}+ g, _, finish, err := h.claimDirectSharedGeneration(ctx, req, u.ID, u, cancel)+ if err != nil {+ t.Fatal(err)+ }+ if !generationExecutingHere(g.ID) {+ t.Fatal("direct shared turn not registered as executing here")+ }+ if err = h.repo.DB.Exec("UPDATE assistant_generations SET updated_at = ? WHERE session_id = ?", time.Now().Add(-time.Minute).UTC(), "direct-shared").Error; err != nil {+ t.Fatal(err)+ }+ c, rec := workbenchContext(u, "GET", "/api/assistant/sessions/direct-shared/generation", bytes.NewBuffer(nil))+ c.Params = gin.Params{{Key: "id", Value: "direct-shared"}}+ h.generation(c)+ var polled domain.AssistantGeneration+ if err = json.Unmarshal(rec.Body.Bytes(), &polled); err != nil || rec.Code != 200 || polled.Status != "running" {+ t.Fatalf("poll reaped a live direct turn: %d %s", rec.Code, rec.Body)+ }+ if got, ok := readAssistantGeneration(h.repo, u.ID, "direct-shared"); !ok || got.Status != "running" {+ t.Fatalf("read reaped a live direct turn: %+v", got)+ }+ c, rec = workbenchContext(u, "GET", "/api/assistant/sessions?projectId=workbench", bytes.NewBuffer(nil))+ h.listSessions(c)+ var list []domain.AssistantSession+ if err = json.Unmarshal(rec.Body.Bytes(), &list); err != nil {+ t.Fatal(err)+ }+ for _, s := range list {+ if s.ID == "direct-shared" && s.GenerationStatus != "running" {+ t.Fatalf("list shows %q for a live direct turn", s.GenerationStatus)+ }+ }+ finish(true)+ if generationExecutingHere(g.ID) {+ t.Fatal("finished direct shared turn still registered")+ }+ if stored, _ := h.repo.PeekAssistantGeneration(u.ID, "direct-shared"); stored.Status != "completed" {+ t.Fatalf("direct shared turn stored %q", stored.Status)+ }+}diff --git a/internal/httpapi/assistant_execution.go b/internal/httpapi/assistant_execution.goindex 005b8124..c53a5b4a 100644--- a/internal/httpapi/assistant_execution.go+++ b/internal/httpapi/assistant_execution.go@@ -65,7 +65,7 @@ func (h *assistantHandlers) sessionMode(c *gin.Context) {c.JSON(404, gin.H{"error": "会话不存在"})return}- if g, ok := h.repo.AssistantGeneration(owner, saved.ID); ok && g.Status == "running" {+ if g, ok := h.readGeneration(owner, saved.ID); ok && g.Status == "running" {c.JSON(409, gin.H{"error": "请先停止当前生成再切换模式"})return}diff --git a/internal/httpapi/assistant_fork.go b/internal/httpapi/assistant_fork.goindex 375d5ad3..aa0198cb 100644--- a/internal/httpapi/assistant_fork.go+++ b/internal/httpapi/assistant_fork.go@@ -48,7 +48,7 @@ func (h *assistantHandlers) forkSession(c *gin.Context) {return}// Recover stale jobs before the transactional active-generation check.- if job, ok := h.repo.AssistantGeneration(owner, source.ID); ok && job.Status == "running" {+ if job, ok := h.readGeneration(owner, source.ID); ok && job.Status == "running" {c.JSON(409, gin.H{"error": "请先停止原会话的生成,再创建分支"})return}diff --git a/internal/httpapi/assistant_generation.go b/internal/httpapi/assistant_generation.goindex e5548c78..e29f1fc9 100644--- a/internal/httpapi/assistant_generation.go+++ b/internal/httpapi/assistant_generation.go@@ -187,6 +187,7 @@ func (h *assistantHandlers) startGeneration(c *gin.Context) {c.JSON(409, gin.H{"error": err.Error()})return}+ h.recordGenerationWorkspaceLease(g, req.RunID, req.workspaceToken)// Keep the authenticated identity, but disconnect model execution from the// lifetime of the browser request. Only explicit cancellation stops a turn.parent := context.WithValue(context.WithoutCancel(c.Request.Context()), assistantFreshCLIKey{}, freshCLI)@@ -286,8 +287,15 @@ func (h *assistantHandlers) executeGeneration(parent context.Context, req assistsink := rec.sinkstopped := make(chan struct{})done := make(chan struct{})+ renewed := make(chan struct{})+ // The lease renews on its own ticker. A heartbeat commit that stalls on the+ // generation row must not also stop the renewal: a replica that reaps this+ // turn meanwhile releases its lease only once renewal has stopped for 45s.go func() {- defer close(done)+ defer close(renewed)+ if req.workspaceToken == "" {+ return+ }ticker := time.NewTicker(500 * time.Millisecond)defer ticker.Stop()for {@@ -295,13 +303,24 @@ func (h *assistantHandlers) executeGeneration(parent context.Context, req assistcase <-stopped:returncase <-ticker.C:- if req.workspaceToken != "" {- if err := h.engine.RenewConversationWorkspaceUse(req.RunID, req.workspaceToken); err != nil {- rec.diagnostics.terminate("WORKSPACE_LEASE_LOST", "会话工作区续租失败:"+err.Error(), diagUser)- cancel()- }+ if err := h.engine.RenewConversationWorkspaceUse(req.RunID, req.workspaceToken); err != nil {+ rec.diagnostics.terminate("WORKSPACE_LEASE_LOST", "会话工作区续租失败:"+err.Error(), diagUser)+ cancel()+ return}- old, ok := h.repo.AssistantGeneration(g.UserID, g.SessionID)+ }+ }+ }()+ go func() {+ defer close(done)+ ticker := time.NewTicker(500 * time.Millisecond)+ defer ticker.Stop()+ for {+ select {+ case <-stopped:+ return+ case <-ticker.C:+ old, ok := h.repo.PeekAssistantGeneration(g.UserID, g.SessionID)if ok && old.ID == g.ID && old.Steer != nil {rec.mu.Lock()rec.g.Steer = old.Steer@@ -355,6 +374,7 @@ func (h *assistantHandlers) executeGeneration(parent context.Context, req assist// The turn is complete once its execution has exited, without archive I/O.close(stopped)<-done+ <-renewedworkspaceWarning := ""if req.workspaceToken != "" {if result, err := h.engine.EndConversationWorkspaceUseAsync(ctx, req.RunID, req.workspaceToken); err != nil {@@ -404,7 +424,7 @@ func (h *assistantHandlers) executeGeneration(parent context.Context, req assiststeer := g.Steerrec.mu.Unlock()if steer == nil {- if latest, ok := h.repo.AssistantGeneration(g.UserID, g.SessionID); ok && latest.ID == g.ID && latest.Steer != nil {+ if latest, ok := h.repo.PeekAssistantGeneration(g.UserID, g.SessionID); ok && latest.ID == g.ID && latest.Steer != nil {rec.mu.Lock()g.Steer = latest.Steersteer = latest.Steer@@ -463,7 +483,7 @@ func (h *assistantHandlers) steerGeneration(c *gin.Context) {if content == "" {content = "请阅读本轮附件并回答。"}- g, ok := h.repo.AssistantGeneration(ownerID, c.Param("id"))+ g, ok := h.readGeneration(ownerID, c.Param("id"))if !ok || g.Status != "running" {c.JSON(409, gin.H{"error": "当前没有进行中的回复"})return@@ -566,5 +586,6 @@ func (h *assistantHandlers) followSteer(parent context.Context, prev domain.Assi}return}+ h.recordGenerationWorkspaceLease(next, req.RunID, req.workspaceToken)h.executeGeneration(parent, req, next, actor)}diff --git a/internal/httpapi/assistant_generation_read_paths_test.go b/internal/httpapi/assistant_generation_read_paths_test.gonew file mode 100644index 00000000..347e6c5d--- /dev/null+++ b/internal/httpapi/assistant_generation_read_paths_test.go@@ -0,0 +1,215 @@+package httpapi++import (+ "bytes"+ "context"+ "encoding/json"+ "path/filepath"+ "strings"+ "testing"+ "time"++ "agentflow/internal/domain"+ "agentflow/internal/engine"+ "agentflow/internal/runtime"+ "github.com/gin-gonic/gin"+)++// readPathCase is one handler that refuses to act while a turn is running.+// Each one must go through readGeneration: a turn this process still executes+// keeps blocking it after a late heartbeat, and a turn nobody executes any+// more is reaped so it no longer blocks the conversation.+type readPathCase struct {+ name string+ // busy is the refusal the handler returns while the turn is running.+ busy string+ call func(t *testing.T, h *assistantHandlers, u domain.User, sessionID string) (int, string)+}++func callSession(h *assistantHandlers, u domain.User, method, sessionID string, body []byte) (int, string) {+ c, rec := workbenchContext(u, method, "/api/assistant/sessions/"+sessionID, bytes.NewBuffer(body))+ c.Params = gin.Params{{Key: "id", Value: sessionID}}+ h.session(c)+ return rec.Code, rec.Body.String()+}++func generationReadPaths() []readPathCase {+ return []readPathCase{+ {name: "session PUT", busy: "生成期间不能改写会话", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {+ stored, ok := h.repo.AssistantSession(u.ID, id)+ if !ok {+ t.Fatalf("session %s missing", id)+ }+ body, _ := json.Marshal(stored)+ return callSession(h, u, "PUT", id, body)+ }},+ {name: "session DELETE", busy: "请先停止生成后删除会话", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {+ return callSession(h, u, "DELETE", id, nil)+ }},+ {name: "fork", busy: "请先停止原会话的生成", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {+ stored, ok := h.repo.AssistantSession(u.ID, id)+ if !ok {+ t.Fatalf("session %s missing", id)+ }+ body, _ := json.Marshal(gin.H{"messageIndex": 0, "content": "换个问法", "updatedAt": stored.UpdatedAt})+ c, rec := workbenchContext(u, "POST", "/api/assistant/sessions/"+id+"/fork", bytes.NewBuffer(body))+ c.Params = gin.Params{{Key: "id", Value: id}}+ h.forkSession(c)+ return rec.Code, rec.Body.String()+ }},+ {name: "execution mode switch", busy: "请先停止当前生成再切换模式", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {+ c, rec := workbenchContext(u, "PUT", "/api/assistant/sessions/"+id+"/mode", bytes.NewBufferString(`{"mode":"platform"}`))+ c.Params = gin.Params{{Key: "id", Value: id}}+ h.sessionMode(c)+ return rec.Code, rec.Body.String()+ }},+ {name: "proposal decision", busy: "请等待回复结束后再处理建议", call: func(t *testing.T, h *assistantHandlers, u domain.User, id string) (int, string) {+ c, rec := workbenchContext(u, "POST", "/api/assistant/sessions/"+id+"/proposals", bytes.NewBufferString(`{"messageIndex":0,"draftIndex":0,"decision":"declined"}`))+ c.Params = gin.Params{{Key: "id", Value: id}}+ h.decideProposal(c)+ return rec.Code, rec.Body.String()+ }},+ }+}++// The PR routes every non-owner read through readGeneration. Its own test+// covers polls, the list, chat and steering; these are the remaining+// handlers that refuse to act on a running turn.+func TestLocalTurnStillBlocksEveryRunningGuardAfterASlowHeartbeat(t *testing.T) {+ for i, tc := range generationReadPaths() {+ t.Run(tc.name, func(t *testing.T) {+ h, u := workbenchFixture(t)+ sessionID := "local-guard-" + string(rune('a'+i))+ g := staleGeneration(t, h, u, sessionID, sessionID+"-turn")+ markLocal(t, g.ID)++ code, body := tc.call(t, h, u, sessionID)+ if code != 409 || !strings.Contains(body, tc.busy) {+ t.Fatalf("a live local turn did not block %s: %d %s", tc.name, code, body)+ }+ if stored, ok := h.repo.PeekAssistantGeneration(u.ID, sessionID); !ok || stored.Status != "running" {+ t.Fatalf("%s wrote the live local turn as %q", tc.name, stored.Status)+ }+ if _, ok := h.repo.AssistantSession(u.ID, sessionID); !ok {+ t.Fatalf("%s removed the conversation of a live turn", tc.name)+ }+ })+ }+}++// A late turn nobody in this process executes is still reaped on each of+// those paths, so a dead replica cannot block the conversation for good.+func TestRemoteStaleTurnIsReapedOnEveryRunningGuard(t *testing.T) {+ for i, tc := range generationReadPaths() {+ t.Run(tc.name, func(t *testing.T) {+ h, u := workbenchFixture(t)+ sessionID := "remote-guard-" + string(rune('a'+i))+ staleGeneration(t, h, u, sessionID, sessionID+"-turn")++ code, body := tc.call(t, h, u, sessionID)+ if strings.Contains(body, tc.busy) {+ t.Fatalf("a dead remote turn still blocks %s: %d %s", tc.name, code, body)+ }+ if stored, ok := h.repo.PeekAssistantGeneration(u.ID, sessionID); ok && stored.Status != "interrupted" {+ t.Fatalf("%s left the dead remote turn as %q", tc.name, stored.Status)+ }+ })+ }+}++// The tool gateway asks whether a native tool call still belongs to a running+// turn. A slow heartbeat must not make it deny the tools of a live local turn.+func TestNativeToolTurnActiveKeepsALocalTurnThroughASlowHeartbeat(t *testing.T) {+ h, u := workbenchFixture(t)+ gateway := &agentToolGateway{deps: h.deps}+ for _, local := range []bool{true, false} {+ sessionID := "tool-remote"+ if local {+ sessionID = "tool-local"+ }+ g := staleGeneration(t, h, u, sessionID, sessionID+"-turn")+ if local {+ markLocal(t, g.ID)+ }+ ctx := context.WithValue(context.Background(), assistantControlScopeKey{}, assistantControlScope{SessionID: sessionID, GenerationID: g.ID})+ s := &AgentToolSession{ChatOwnerID: u.ID, ChatSessionID: sessionID, ctx: ctx}+ active := gateway.nativeToolTurnActive(context.Background(), s)+ stored, _ := h.repo.PeekAssistantGeneration(u.ID, sessionID)+ if local && (!active || stored.Status != "running") {+ t.Fatalf("a live local turn lost its tools: active=%v status=%q", active, stored.Status)+ }+ if !local && (active || stored.Status != "interrupted") {+ t.Fatalf("a dead remote turn kept its tools: active=%v status=%q", active, stored.Status)+ }+ }+}++// A background turn on a SOP conversation takes the run's foreground+// workspace lease and records exactly that lease on its generation, so a+// reader that later reaps the turn can release it. The recorded lease does+// not outlive the turn: the next turn starts with none.+func TestStartGenerationRecordsTheWorkspaceLeaseItHolds(t *testing.T) {+ h, a, _, _, session := sharedWorkbenchFixture(t)+ h.engine = engine.New(h.repo, runtime.NewManager(filepath.Join(t.TempDir(), "workspaces")))+ entered, release := make(chan struct{}), make(chan struct{})+ h.generate = func(ctx context.Context, _ string, _ domain.User) (string, error) {+ close(entered)+ select {+ case <-release:+ return "完成", nil+ case <-ctx.Done():+ return "", ctx.Err()+ }+ }+ messages := append(append([]assistantMessage(nil), session.Messages...), assistantMessage{Role: "user", Content: "后台执行"})+ body, _ := json.Marshal(assistantRequest{ProjectID: session.ProjectID, RunID: session.RunID, SessionID: session.ID, Messages: messages})+ c, rec := workbenchContext(a, "POST", "/api/assistant/generations", bytes.NewBuffer(body))+ h.startGeneration(c)+ if rec.Code != 202 {+ t.Fatalf("start: %d %s", rec.Code, rec.Body)+ }+ released := false+ t.Cleanup(func() {+ if !released {+ close(release)+ }+ })+ select {+ case <-entered:+ case <-time.After(5 * time.Second):+ t.Fatal("generation did not start")+ }+ var recorded struct {+ GenerationID string+ UserID int64+ WorkspaceRunID string+ WorkspaceLease string+ }+ if err := h.repo.DB.Table("assistant_generations").Select("generation_id, user_id, workspace_run_id, workspace_lease").Where("session_id = ?", session.ID).Take(&recorded).Error; err != nil {+ t.Fatal(err)+ }+ checkpoint, _ := h.repo.WorkspaceCheckpoint(session.RunID)+ if recorded.WorkspaceRunID != session.RunID || recorded.WorkspaceLease == "" || recorded.WorkspaceLease != checkpoint.LeaseToken {+ t.Fatalf("recorded lease %q on %q, workspace holds %q", recorded.WorkspaceLease, recorded.WorkspaceRunID, checkpoint.LeaseToken)+ }+ close(release)+ released = true+ deadline := time.Now().Add(10 * time.Second)+ for {+ if g, ok := h.repo.PeekAssistantGeneration(recorded.UserID, session.ID); ok && g.Status != "running" {+ break+ }+ if time.Now().After(deadline) {+ t.Fatal("generation did not finish")+ }+ time.Sleep(20 * time.Millisecond)+ }+ next := domain.AssistantGeneration{ID: "after-" + recorded.GenerationID, SessionID: session.ID, UserID: recorded.UserID, ProjectID: session.ProjectID, Status: "running"}+ if err := h.repo.BeginAssistantGeneration(next); err != nil {+ t.Fatal(err)+ }+ ageGeneration(t, h, session.ID)+ if _, lease, _ := h.repo.ReapAssistantGeneration(recorded.UserID, session.ID); lease.Token != "" {+ t.Fatalf("the next turn inherited the finished turn's lease: %+v", lease)+ }+}diff --git a/internal/httpapi/assistant_generation_reap_guard_test.go b/internal/httpapi/assistant_generation_reap_guard_test.gonew file mode 100644index 00000000..7bbc4ba8--- /dev/null+++ b/internal/httpapi/assistant_generation_reap_guard_test.go@@ -0,0 +1,97 @@+package httpapi++import (+ "bytes"+ "context"+ "encoding/json"+ "errors"+ "testing"+ "time"++ "agentflow/internal/domain"+ "agentflow/internal/runtime"+ "github.com/gin-gonic/gin"+)++// Opening a conversation whose turn this process still executes, after one+// slow heartbeat, shows the turn as running and does not reap it. Writes to+// the conversation are covered in assistant_generation_read_paths_test.go.+func TestOpeningAConversationKeepsALocalTurnAfterSlowHeartbeat(t *testing.T) {+ h, u := workbenchFixture(t)+ g := staleGeneration(t, h, u, "guarded-chat", "guarded-turn")+ markLocal(t, g.ID)+ c, rec := workbenchContext(u, "GET", "/api/assistant/sessions/guarded-chat", bytes.NewBuffer(nil))+ c.Params = gin.Params{{Key: "id", Value: "guarded-chat"}}+ h.session(c)+ if rec.Code != 200 {+ t.Fatalf("open: %d %s", rec.Code, rec.Body)+ }+ if stored, _ := h.repo.PeekAssistantGeneration(u.ID, "guarded-chat"); stored.Status != "running" {+ t.Fatalf("opening the conversation wrote the local turn as %q", stored.Status)+ }+}++// An ordinary chat turn (no SOP run) records the workspace lease its first+// tool takes, before the turn ends, so a reader that reaps the turn after its process died can free+// exactly that lease.+func TestGenerationRecordsTheWorkspaceLeaseItsToolTakes(t *testing.T) {+ h, user, fixtureSession, gateway, ids := workspaceTurnFixture(t)+ type recorded struct {+ RunID string `gorm:"column:workspace_run_id"`+ Lease string `gorm:"column:workspace_lease"`+ }+ seen := make(chan recorded, 1)+ h.generate = func(ctx context.Context, _ string, _ domain.User) (string, error) {+ var session *AgentToolSession+ agentToolSessions.mu.Lock()+ for _, candidate := range agentToolSessions.sessions {+ if candidate.ChatSessionID == fixtureSession.ChatSessionID {+ session = candidate+ }+ }+ agentToolSessions.mu.Unlock()+ if session == nil || session.workspaceTurn == nil {+ return "", errors.New("generation did not own its workspace lifetime")+ }+ if _, err := gateway.callMainBotWorkspace(ctx, session, "workbench_workspace_write", map[string]any{"path": "note.txt", "content": "after", "expectedSha256": runtime.WorkspaceContentSHA("before")}); err != nil {+ return "", err+ }+ var row recorded+ if err := h.repo.DB.Table("assistant_generations").Select("workspace_run_id, workspace_lease").Where("session_id = ?", fixtureSession.ChatSessionID).Scan(&row).Error; err != nil {+ return "", err+ }+ meta, _ := h.repo.WorkspaceCheckpoint(ids[0])+ if meta.LeaseToken != row.Lease {+ row.Lease = "mismatch:" + row.Lease + "!=" + meta.LeaseToken+ }+ seen <- row+ return "done", nil+ }+ body, _ := json.Marshal(assistantRequest{SessionID: fixtureSession.ChatSessionID, ProjectID: fixtureSession.ProjectID, Messages: []assistantMessage{{Role: "user", Content: "edit"}}})+ c, rec := workbenchContext(user, "POST", "/api/assistant/generations", bytes.NewBuffer(body))+ h.startGeneration(c)+ if rec.Code != 202 {+ t.Fatalf("generation start: %d %s", rec.Code, rec.Body)+ }+ select {+ case row := <-seen:+ if row.RunID != ids[0] || row.Lease == "" || len(row.Lease) > 9 && row.Lease[:9] == "mismatch:" {+ t.Fatalf("lease recorded on the generation: %+v", row)+ }+ case <-time.After(10 * time.Second):+ t.Fatal("generation did not reach its tool")+ }+ deadline := time.Now().Add(10 * time.Second)+ for {+ if g, ok := h.repo.PeekAssistantGeneration(user.ID, fixtureSession.ChatSessionID); ok && g.Status != "running" {+ if g.Status != "completed" {+ t.Fatalf("generation outcome: %+v", g)+ }+ break+ }+ if time.Now().After(deadline) {+ t.Fatal("generation did not finish")+ }+ time.Sleep(10 * time.Millisecond)+ }+}diff --git a/internal/httpapi/assistant_generation_reap_test.go b/internal/httpapi/assistant_generation_reap_test.gonew file mode 100644index 00000000..c354243c--- /dev/null+++ b/internal/httpapi/assistant_generation_reap_test.go@@ -0,0 +1,204 @@+package httpapi++import (+ "bytes"+ "context"+ "encoding/json"+ "errors"+ "testing"+ "time"++ "agentflow/internal/domain"+ "agentflow/internal/repo"+ "github.com/gin-gonic/gin"+)++// staleGeneration starts a turn whose last durable heartbeat is a minute old,+// as after a slow commit or a dead process.+func staleGeneration(t *testing.T, h *assistantHandlers, u domain.User, sessionID, generationID string) domain.AssistantGeneration {+ t.Helper()+ if err := h.repo.PutAssistantSession(u.ID, domain.AssistantSession{ID: sessionID, ProjectID: "workbench", Title: sessionID, Messages: []domain.AssistantMessage{{Role: "user", Content: "继续"}}}); err != nil {+ t.Fatal(err)+ }+ g := domain.AssistantGeneration{ID: generationID, SessionID: sessionID, UserID: u.ID, ProjectID: "workbench", Status: "running", MessageCount: 1}+ if err := h.repo.BeginAssistantGeneration(g); err != nil {+ t.Fatal(err)+ }+ ageGeneration(t, h, sessionID)+ return g+}++func ageGeneration(t *testing.T, h *assistantHandlers, sessionID string) {+ t.Helper()+ if err := h.repo.DB.Table("assistant_generations").Where("session_id = ?", sessionID).UpdateColumn("updated_at", time.Now().Add(-time.Minute).UTC()).Error; err != nil {+ t.Fatal(err)+ }+}++func markLocal(t *testing.T, id string) {+ t.Helper()+ assistantLiveGenerations.Store(id, &assistantLiveGeneration{changed: make(chan struct{})})+ t.Cleanup(func() { assistantLiveGenerations.Delete(id) })+}++func listedStatus(t *testing.T, h *assistantHandlers, u domain.User, sessionID string) string {+ t.Helper()+ c, rec := workbenchContext(u, "GET", "/api/assistant/sessions?projectId=workbench", bytes.NewBuffer(nil))+ h.listSessions(c)+ if rec.Code != 200 {+ t.Fatalf("list: %d %s", rec.Code, rec.Body)+ }+ var list []struct {+ ID string `json:"id"`+ GenerationStatus string `json:"generationStatus"`+ }+ if err := json.Unmarshal(rec.Body.Bytes(), &list); err != nil {+ t.Fatal(err)+ }+ for _, item := range list {+ if item.ID == sessionID {+ return item.GenerationStatus+ }+ }+ t.Fatalf("session %s not listed: %s", sessionID, rec.Body)+ return ""+}++// A turn this process is still executing survives a late heartbeat on every+// read path: polls, the session list, a follow-up message and steering.+func TestLocalTurnSurvivesSlowHeartbeatOnEveryRead(t *testing.T) {+ h, u := workbenchFixture(t)+ g := staleGeneration(t, h, u, "slow-chat", "slow-turn")+ markLocal(t, g.ID)++ if got, ok := h.readGeneration(u.ID, "slow-chat"); !ok || got.Status != "running" {+ t.Fatalf("read reaped a local turn: %+v", got)+ }+ if got, ok := h.currentGeneration(u.ID, "slow-chat"); !ok || got.Status != "running" {+ t.Fatalf("poll reaped a local turn: %+v", got)+ }+ if status := listedStatus(t, h, u, "slow-chat"); status != "running" {+ t.Fatalf("list shows a local turn as %q", status)+ }+ c, rec := workbenchContext(u, "POST", "/api/assistant/chat", bytes.NewBufferString(`{"projectId":"workbench","sessionId":"slow-chat","messages":[{"role":"user","content":"继续"},{"role":"user","content":"再来"}]}`))+ h.chat(c)+ if rec.Code != 409 {+ t.Fatalf("a second chat on a live turn was not refused: %d %s", rec.Code, rec.Body)+ }+ c, rec = workbenchContext(u, "POST", "/api/assistant/sessions/slow-chat/generation/steer", bytes.NewBufferString(`{"content":"改一下"}`))+ c.Params = gin.Params{{Key: "id", Value: "slow-chat"}}+ h.steerGeneration(c)+ if rec.Code == 409 {+ t.Fatalf("steering a live turn was refused as finished: %s", rec.Body)+ }+ if stored, _ := h.repo.PeekAssistantGeneration(u.ID, "slow-chat"); stored.Status != "running" {+ t.Fatalf("a read wrote the local turn as %q", stored.Status)+ }+}++// Without a local owner the late turn is reaped. Once that is stored, the list+// reports it as stored instead of hiding it behind a stale in-memory entry.+func TestListReportsAStoredInterruptionEvenWithALiveEntry(t *testing.T) {+ h, u := workbenchFixture(t)+ g := staleGeneration(t, h, u, "dead-chat", "dead-turn")+ if status := listedStatus(t, h, u, "dead-chat"); status != "interrupted" {+ t.Fatalf("late remote turn listed as %q", status)+ }+ if got, _ := h.readGeneration(u.ID, "dead-chat"); got.Status != "interrupted" {+ t.Fatalf("late remote turn not reaped: %+v", got)+ }+ markLocal(t, g.ID)+ if status := listedStatus(t, h, u, "dead-chat"); status != "interrupted" {+ t.Fatalf("a stored interruption listed as %q", status)+ }+}++// A reaped turn frees its own lease so the next message can start. A lease its+// holder still renews, or one a newer turn took, is never freed by the reap.+func TestReapReleasesOnlyTheAbandonedTurnsWorkspaceLease(t *testing.T) {+ h, u := workbenchFixture(t)+ now := time.Now().UTC()+ taken := now.Add(-2 * time.Minute)+ for _, id := range []string{"run-dead", "run-renewed", "run-replaced"} {+ if err := h.repo.PutRun(domain.Run{ID: id, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {+ t.Fatal(err)+ }+ }+ begin := func(runID string, at time.Time) string {+ t.Helper()+ token, err := h.repo.BeginWorkspaceUse(runID, at)+ if err != nil {+ t.Fatal(err)+ }+ return token+ }+ free := func(runID string) bool {+ t.Helper()+ token, err := h.repo.BeginWorkspaceUse(runID, time.Now().UTC())+ if errors.Is(err, repo.ErrWorkspaceInUse) {+ return false+ }+ if err != nil {+ t.Fatal(err)+ }+ _ = h.repo.FinishWorkspaceUse(runID, token, time.Now().UTC(), nil, "")+ return true+ }++ // The process died: nothing renews the lease and nothing heartbeats.+ dead := staleGeneration(t, h, u, "lease-dead", "turn-dead")+ h.recordGenerationWorkspaceLease(dead, "run-dead", begin("run-dead", taken))+ ageGeneration(t, h, "lease-dead")+ if got, _ := h.readGeneration(u.ID, "lease-dead"); got.Status != "interrupted" {+ t.Fatalf("dead turn not reaped: %+v", got)+ }+ if !free("run-dead") {+ t.Fatal("the dead turn's lease still blocks the next message")+ }++ // Another replica is executing it: its heartbeat is late but it still+ // renews the lease. The turn is reaped as before; its lease stays.+ renewed := staleGeneration(t, h, u, "lease-renewed", "turn-renewed")+ token := begin("run-renewed", taken)+ h.recordGenerationWorkspaceLease(renewed, "run-renewed", token)+ ageGeneration(t, h, "lease-renewed")+ if err := h.repo.RenewWorkspaceUse("run-renewed", token, time.Now().UTC()); err != nil {+ t.Fatal(err)+ }+ h.readGeneration(u.ID, "lease-renewed")+ if free("run-renewed") {+ t.Fatal("a lease its holder still renews was released")+ }++ // The reaped turn's lease already ended and a newer use took the+ // workspace before the reader got there.+ replaced := staleGeneration(t, h, u, "lease-replaced", "turn-replaced")+ old := begin("run-replaced", taken)+ h.recordGenerationWorkspaceLease(replaced, "run-replaced", old)+ ageGeneration(t, h, "lease-replaced")+ if err := h.repo.FinishWorkspaceUse("run-replaced", old, taken, nil, ""); err != nil {+ t.Fatal(err)+ }+ newer := begin("run-replaced", taken)+ h.readGeneration(u.ID, "lease-replaced")+ if kept, _ := h.repo.WorkspaceCheckpoint("run-replaced"); kept.LeaseToken != newer {+ t.Fatalf("a newer use lost its lease to the reap: %q", kept.LeaseToken)+ }+}++// Ordinary conversations take their lease during the turn; it is recorded on+// the generation as soon as it is taken.+func TestWorkspaceTurnRecordsEachLeaseItTakes(t *testing.T) {+ turn := newAssistantWorkspaceTurn(&workspaceTurnEngine{}, nil)+ t.Cleanup(func() { _ = turn.close(context.Background()) })+ var recorded []string+ turn.recordLeases(func(runID, token string) { recorded = append(recorded, runID+"="+token) })+ for range 2 {+ if _, err := turn.lease(context.Background(), "run-a"); err != nil {+ t.Fatal(err)+ }+ }+ if len(recorded) != 1 || recorded[0] != "run-a=token-run-a" {+ t.Fatalf("recorded leases: %v", recorded)+ }+}diff --git a/internal/httpapi/assistant_native_approval.go b/internal/httpapi/assistant_native_approval.goindex 4022f211..abe26e91 100644--- a/internal/httpapi/assistant_native_approval.go+++ b/internal/httpapi/assistant_native_approval.go@@ -42,7 +42,7 @@ func (h *assistantHandlers) nativeApprovals(c *gin.Context) {return}if item.GenerationID != "" {- g, ok := h.repo.AssistantGeneration(owner, c.Param("id"))+ g, ok := h.readGeneration(owner, c.Param("id"))if !ok || g.ID != item.GenerationID || g.Status != "running" || g.CancelRequested {c.JSON(409, gin.H{"error": "本轮已结束,审批已失效"})return@@ -154,7 +154,7 @@ func (h *agentToolGateway) nativeToolTurnActive(ctx context.Context, s *AgentTooif h.repo == nil {return false}- current, ok := h.repo.AssistantGeneration(s.ChatOwnerID, s.ChatSessionID)+ current, ok := readAssistantGeneration(h.repo, s.ChatOwnerID, s.ChatSessionID)return ok && current.ID == generation && current.Status == "running" && !current.CancelRequested}diff --git a/internal/httpapi/assistant_proposals.go b/internal/httpapi/assistant_proposals.goindex c732b66c..4855c1b1 100644--- a/internal/httpapi/assistant_proposals.go+++ b/internal/httpapi/assistant_proposals.go@@ -374,7 +374,7 @@ func (h *assistantHandlers) decideProposal(c *gin.Context) {c.JSON(400, gin.H{"error": "未知的处理方式"})return}- if job, ok := h.repo.AssistantGeneration(ownerID, id); ok && job.Status == "running" {+ if job, ok := h.readGeneration(ownerID, id); ok && job.Status == "running" {c.JSON(409, gin.H{"error": "请等待回复结束后再处理建议"})return}diff --git a/internal/httpapi/assistant_session.go b/internal/httpapi/assistant_session.goindex bef0bc4b..9660e270 100644--- a/internal/httpapi/assistant_session.go+++ b/internal/httpapi/assistant_session.go@@ -88,6 +88,12 @@ func (h *assistantHandlers) listSessions(c *gin.Context) {out[i].Unread = falseif state, ok := statuses[out[i].ID]; ok {out[i].GenerationStatus = state.Status+ // The list only projects a late heartbeat as interrupted; the row+ // still says running. A turn this process is executing is running.+ // A row actually written as interrupted is reported as stored.+ if state.Stale && generationExecutingHere(state.ID) {+ out[i].GenerationStatus = "running"+ }out[i].GenerationID = state.ID// A reply that completed while the reader was elsewhere stays// unread until they open the conversation (Codex's blue dot).@@ -262,7 +268,7 @@ func (h *assistantHandlers) session(c *gin.Context) {return}if !item.Shared && !readOnly {- if job, ok := h.repo.AssistantGeneration(ownerID, id); !ok || job.Status != "running" {+ if job, ok := h.readGeneration(ownerID, id); !ok || job.Status != "running" {if materializeWorkspaceImages(ownerID, &item) {if err := saveAssistantConversation(h.repo, ownerID, item); err == nil {if stored, err := loadAssistantConversation(h.repo, ownerID, id); err == nil {@@ -279,7 +285,7 @@ func (h *assistantHandlers) session(c *gin.Context) {}c.JSON(200, decorateRunSessionView(h.repo, item))case "PUT":- if job, ok := h.repo.AssistantGeneration(ownerID, id); ok && job.Status == "running" {+ if job, ok := h.readGeneration(ownerID, id); ok && job.Status == "running" {c.JSON(409, gin.H{"error": "生成期间不能改写会话"})return}@@ -428,7 +434,7 @@ func (h *assistantHandlers) session(c *gin.Context) {return}}- if job, ok := h.repo.AssistantGeneration(ownerID, id); ok && job.Status == "running" {+ if job, ok := h.readGeneration(ownerID, id); ok && job.Status == "running" {c.JSON(409, gin.H{"error": "请先停止生成后删除会话"})return}diff --git a/internal/httpapi/assistant_tasks.go b/internal/httpapi/assistant_tasks.goindex 72deb542..c606afa5 100644--- a/internal/httpapi/assistant_tasks.go+++ b/internal/httpapi/assistant_tasks.go@@ -285,7 +285,7 @@ func (h *assistantHandlers) readScheduledTurn(user domain.User, task *assistantTif reply != nil {msg = reply.Unfinished}- if latest, found := h.repo.AssistantGeneration(user.ID, g.SessionID); found && latest.ID == g.ID {+ if latest, found := h.readGeneration(user.ID, g.SessionID); found && latest.ID == g.ID {msg = firstNonEmpty(msg, latest.Error)}return fail(firstNonEmpty(strings.TrimSpace(msg), "未返回结果"))diff --git a/internal/httpapi/assistant_unfinished.go b/internal/httpapi/assistant_unfinished.goindex f94f9be2..aae2f445 100644--- a/internal/httpapi/assistant_unfinished.go+++ b/internal/httpapi/assistant_unfinished.go@@ -5,9 +5,11 @@ import ("fmt""log""strings"+ "sync""time""agentflow/internal/domain"+ "agentflow/internal/repo")// commitTurnTranscript keeps a turn that is stopped, fails or times out. A turn@@ -69,10 +71,75 @@ func (h *assistantHandlers) keepInterruptedReply(g domain.AssistantGeneration) {}}+// readGeneration reaps a stale turn only when this process is not still+// executing it. A poll or a follow-up message used to mark a live turn+// interrupted after one slow heartbeat, and the next message then failed+// immediately because the CLI and workspace lease were still held.+func (h *assistantHandlers) readGeneration(ownerID int64, sessionID string) (domain.AssistantGeneration, bool) {+ return readAssistantGeneration(h.repo, ownerID, sessionID)+}++// assistantDirectGenerations holds the synchronous shared turns this process+// is executing. They have no live stream, so they are not in+// assistantLiveGenerations, but their own readers must not reap them either.+var assistantDirectGenerations sync.Map++// generationExecutingHere reports whether this process is running the turn.+func generationExecutingHere(id string) bool {+ if id == "" {+ return false+ }+ if _, ok := assistantLiveGenerations.Load(id); ok {+ return true+ }+ _, ok := assistantDirectGenerations.Load(id)+ return ok+}++// readAssistantGeneration is the one non-owner read of a turn. Every handler+// that may meet a stale turn goes through it, so a turn this process is still+// executing is never written as interrupted by its own readers.+//+// A turn owned by another replica is still reaped after the heartbeat timeout,+// as before. Its workspace lease is released only when the reaped turn recorded+// it and the lease itself stopped being renewed (see ReleaseAbandonedWorkspaceUse).+func readAssistantGeneration(store *repo.Store, ownerID int64, sessionID string) (domain.AssistantGeneration, bool) {+ peek, ok := store.PeekAssistantGeneration(ownerID, sessionID)+ if !ok {+ return domain.AssistantGeneration{}, false+ }+ if generationExecutingHere(peek.ID) {+ return peek, true+ }+ g, lease, ok := store.ReapAssistantGeneration(ownerID, sessionID)+ if lease.Token != "" {+ released, err := store.ReleaseAbandonedWorkspaceUse(lease.RunID, lease.Token, time.Now().UTC())+ switch {+ case err != nil:+ log.Printf("assistant release workspace lease session=%s run=%s: %v", sessionID, lease.RunID, err)+ case released:+ log.Printf("assistant released abandoned workspace lease session=%s generation=%s run=%s", sessionID, g.ID, lease.RunID)+ }+ }+ return g, ok+}++// recordGenerationWorkspaceLease stores the foreground lease a running turn+// holds. Without it a reaped turn keeps its workspace locked for the rest of+// the lease term; with it only that exact lease can be released.+func (h *assistantHandlers) recordGenerationWorkspaceLease(g domain.AssistantGeneration, runID, token string) {+ if g.ID == "" || runID == "" || token == "" {+ return+ }+ if err := h.repo.RecordAssistantGenerationWorkspaceLease(g.UserID, g.SessionID, g.ID, runID, token); err != nil {+ log.Printf("assistant record workspace lease session=%s generation=%s run=%s: %v", g.SessionID, g.ID, runID, err)+ }+}+// currentGeneration reads a session's generation. A turn whose server died is// only discovered here (as "interrupted"), so its partial reply is kept now.func (h *assistantHandlers) currentGeneration(ownerID int64, sessionID string) (domain.AssistantGeneration, bool) {- g, ok := h.repo.AssistantGeneration(ownerID, sessionID)+ g, ok := h.readGeneration(ownerID, sessionID)if ok {if _, local := assistantLiveGenerations.Load(g.ID); !local {h.settleLegacyWorkspaceNativeApprovals(g)diff --git a/internal/httpapi/assistant_workspace_turn.go b/internal/httpapi/assistant_workspace_turn.goindex 5a54cd6b..a561977a 100644--- a/internal/httpapi/assistant_workspace_turn.go+++ b/internal/httpapi/assistant_workspace_turn.go@@ -35,8 +35,11 @@ type assistantWorkspaceTurn struct {closed boolrenewError errorsaveErrors map[string]error- stop chan struct{}- done chan struct{}+ // onLease records each foreground lease on the turn's generation so a+ // reader that reaps a dead turn can release exactly that lease.+ onLease func(runID, token string)+ stop chan struct{}+ done chan struct{}}type assistantWorkspaceLeaseEngine interface {@@ -68,6 +71,13 @@ func newAssistantWorkspaceTurn(en assistantWorkspaceLeaseEngine, cancel context.return t}+// recordLeases sets where each lease this turn takes is recorded.+func (t *assistantWorkspaceTurn) recordLeases(fn func(runID, token string)) {+ t.mu.Lock()+ t.onLease = fn+ t.mu.Unlock()+}+// Tools may run in parallel, but a binding switch and the final checkpoint// must wait for all tools using the old workspace to return.func (t *assistantWorkspaceTurn) lockOperation() func() {@@ -114,6 +124,9 @@ func (t *assistantWorkspaceTurn) lease(ctx context.Context, runID string) (contereturn ctx, err}t.runID, t.token, t.renewedAt = runID, token, time.Now()+ if t.onLease != nil {+ t.onLease(runID, token)+ }}ctx = engine.WithConversationWorkspaceToken(ctx, t.token, runID)t.mu.Unlock()diff --git a/internal/httpapi/space_ai.go b/internal/httpapi/space_ai.goindex 64adf3d2..4261b889 100644--- a/internal/httpapi/space_ai.go+++ b/internal/httpapi/space_ai.go@@ -224,7 +224,7 @@ func (h *spacesHandlers) reconcileAI(room domain.ChatRoom, messages []domain.Chaif m.Kind != domain.ChatKindAI || m.AIStatus != domain.AIStatusRunning {continue}- g, ok := h.repo.AssistantGeneration(m.AIRequesterID, m.AISessionID)+ g, ok := readAssistantGeneration(h.repo, m.AIRequesterID, m.AISessionID)switch {case !ok && time.Since(m.CreatedAt) > time.Minute:m.AIStatus, m.AIError = domain.AIStatusFailed, "没有找到这次 AI 回复的记录"diff --git a/internal/repo/assistant_generation_approvals.go b/internal/repo/assistant_generation_approvals.goindex 417c2daf..f48b8a3f 100644--- a/internal/repo/assistant_generation_approvals.go+++ b/internal/repo/assistant_generation_approvals.go@@ -7,7 +7,6 @@ import ("encoding/json""fmt""slices"- "time""gorm.io/gorm""gorm.io/gorm/clause"@@ -40,8 +39,10 @@ func generationPayload(row assistantGenerationRow) assistantGenerationPayload {return payload}-func nativeApprovalGenerationActive(row assistantGenerationRow) bool {- return row.Status == "running" && !row.CancelRequested && time.Since(row.UpdatedAt) <= 45*time.Second+// nativeApprovalGenerationActive measures the heartbeat on the same clock that+// stamped it (see heartbeatNow).+func (r *AssistantRepo) nativeApprovalGenerationActive(db *gorm.DB, row assistantGenerationRow) bool {+ return row.Status == "running" && !row.CancelRequested && !r.heartbeatLate(db, row.UpdatedAt)}func (r *AssistantRepo) updateNativeApprovals(owner int64, session, generation string, requireLive bool, update func(*assistantGenerationPayload) error) error {@@ -50,7 +51,7 @@ func (r *AssistantRepo) updateNativeApprovals(owner int64, session, generation sif err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).Where("session_id = ? AND user_id = ? AND generation_id = ?", session, owner, generation).Take(&row).Error; err != nil {return fmt.Errorf("本轮已结束,审批已失效")}- if requireLive && !nativeApprovalGenerationActive(row) {+ if requireLive && !r.nativeApprovalGenerationActive(tx, row) {return fmt.Errorf("本轮已结束,审批已失效")}payload := generationPayload(row)@@ -87,7 +88,7 @@ func (r *AssistantRepo) CreateNativeApproval(owner int64, session, phase string,func (r *AssistantRepo) NativeApproval(owner int64, session, id string) (domain.AssistantNativeApproval, bool) {var row assistantGenerationRow- if err := r.DB.Where("session_id = ? AND user_id = ?", session, owner).Take(&row).Error; err != nil || !nativeApprovalGenerationActive(row) {+ if err := r.DB.Where("session_id = ? AND user_id = ?", session, owner).Take(&row).Error; err != nil || !r.nativeApprovalGenerationActive(r.DB, row) {return domain.AssistantNativeApproval{}, false}for _, item := range generationPayload(row).NativeApprovals {@@ -101,7 +102,7 @@ func (r *AssistantRepo) NativeApproval(owner int64, session, id string) (domain.func (r *AssistantRepo) PendingNativeApprovals(owner int64, session string) []domain.AssistantNativeApproval {out := []domain.AssistantNativeApproval{}var row assistantGenerationRow- if err := r.DB.Where("session_id = ? AND user_id = ?", session, owner).Take(&row).Error; err != nil || !nativeApprovalGenerationActive(row) {+ if err := r.DB.Where("session_id = ? AND user_id = ?", session, owner).Take(&row).Error; err != nil || !r.nativeApprovalGenerationActive(r.DB, row) {return out}for _, item := range generationPayload(row).NativeApprovals {diff --git a/internal/repo/assistant_generation_clock_mysql_test.go b/internal/repo/assistant_generation_clock_mysql_test.gonew file mode 100644index 00000000..3676d0bf--- /dev/null+++ b/internal/repo/assistant_generation_clock_mysql_test.go@@ -0,0 +1,88 @@+package repo++import (+ "encoding/json"+ "fmt"+ "os"+ "testing"+ "time"++ "agentflow/internal/domain"+)++// MySQL stamps heartbeats with UTC_TIMESTAMP(3), so their age must be measured+// on the database clock. With this host's time.Now() instead, a database clock+// behind the API host reaps a turn that just checked in, and one ahead of it+// never reaps a dead turn. SET timestamp skews the database clock of the one+// connection the store is pinned to.+func TestMySQLHeartbeatAgeIsMeasuredOnTheDatabaseClock(t *testing.T) {+ dsn := os.Getenv("AGENTFLOW_TEST_MYSQL_DSN")+ if dsn == "" {+ t.Skip("local isolated MySQL required")+ }+ s, err := Open(dsn)+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ sqlDB, err := s.DB.DB()+ if err != nil {+ t.Fatal(err)+ }+ sqlDB.SetMaxOpenConns(1)+ sqlDB.SetMaxIdleConns(1)+ skew := func(d time.Duration) {+ t.Helper()+ if err := s.DB.Exec("SET timestamp = ?", float64(time.Now().Add(d).UnixMicro())/1e6).Error; err != nil {+ t.Fatal(err)+ }+ }+ defer s.DB.Exec("SET timestamp = DEFAULT")++ const owner = 41+ session := fmt.Sprintf("clock-skew-%d", time.Now().UnixNano())+ defer s.DB.Where("session_id = ?", session).Delete(&assistantGenerationRow{})+ approval := func(id string) error {+ return s.CreateNativeApproval(owner, session, "phase", domain.AssistantNativeApproval{ID: id, GenerationID: session, ActorID: owner, Tool: "Write", Input: json.RawMessage(`{"path":"a.txt"}`)})+ }++ // Database clock two minutes behind this host: the turn just checked in.+ skew(-2 * time.Minute)+ g := domain.AssistantGeneration{ID: session, SessionID: session, UserID: owner, ActorID: owner, Status: "running"}+ if err := s.BeginAssistantGeneration(g); err != nil {+ t.Fatal(err)+ }+ if states := s.AssistantGenerationStatuses(owner, []string{session}); states[session].Stale || states[session].Status != "running" {+ t.Fatalf("a fresh turn is projected as stale: %+v", states[session])+ }+ if got, lease, _ := s.ReapAssistantGeneration(owner, session); got.Status != "running" || lease.Token != "" {+ t.Fatalf("a fresh turn was reaped: %+v", got)+ }+ if err := s.HeartbeatAssistantGeneration(g); err != nil {+ t.Fatal(err)+ }+ if err := approval("fresh"); err != nil {+ t.Fatalf("a live turn refused an approval: %v", err)+ }+ if items := s.PendingNativeApprovals(owner, session); len(items) != 1 {+ t.Fatalf("a live turn hid its approval: %+v", items)+ }++ // Database clock two minutes ahead: silent for a minute on that clock.+ skew(2 * time.Minute)+ if err := s.DB.Exec("UPDATE assistant_generations SET updated_at = UTC_TIMESTAMP(3) - INTERVAL 60 SECOND WHERE session_id = ?", session).Error; err != nil {+ t.Fatal(err)+ }+ if items := s.PendingNativeApprovals(owner, session); len(items) != 0 {+ t.Fatalf("a dead turn kept its approval: %+v", items)+ }+ if err := approval("late"); err == nil {+ t.Fatal("a dead turn accepted an approval")+ }+ if states := s.AssistantGenerationStatuses(owner, []string{session}); !states[session].Stale || states[session].Status != "interrupted" {+ t.Fatalf("a dead turn is not projected as stale: %+v", states[session])+ }+ if got, _, _ := s.ReapAssistantGeneration(owner, session); got.Status != "interrupted" {+ t.Fatalf("a dead turn was not reaped: %+v", got)+ }+}diff --git a/internal/repo/assistant_generation_lease_guard_test.go b/internal/repo/assistant_generation_lease_guard_test.gonew file mode 100644index 00000000..92a7f5c1--- /dev/null+++ b/internal/repo/assistant_generation_lease_guard_test.go@@ -0,0 +1,62 @@+package repo++import (+ "testing"+ "time"++ "agentflow/internal/domain"+)++// A lease is recorded only on the running turn it belongs to. A late callback+// from a finished turn, or one naming another turn of the same conversation,+// must not leave a token that a later reap would release.+func TestRecordAssistantGenerationWorkspaceLeaseOnlyOnItsRunningTurn(t *testing.T) {+ s, err := OpenTest()+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ recorded := func() (string, string) {+ t.Helper()+ var row assistantGenerationRow+ if err := s.DB.Where("session_id = ?", "fenced-chat").Take(&row).Error; err != nil {+ t.Fatal(err)+ }+ return row.WorkspaceRunID, row.WorkspaceLease+ }+ first := domain.AssistantGeneration{ID: "first", SessionID: "fenced-chat", UserID: 11, Status: "running"}+ if err = s.BeginAssistantGeneration(first); err != nil {+ t.Fatal(err)+ }+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "fenced-chat", "other-turn", "run-1", "stray"); err != nil {+ t.Fatal(err)+ }+ if run, lease := recorded(); run != "" || lease != "" {+ t.Fatalf("a lease for another turn was recorded: %q %q", run, lease)+ }+ first.Status = "completed"+ if err = s.CheckpointAssistantGeneration(first); err != nil {+ t.Fatal(err)+ }+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "fenced-chat", "first", "run-1", "late"); err != nil {+ t.Fatal(err)+ }+ if run, lease := recorded(); run != "" || lease != "" {+ t.Fatalf("a finished turn recorded a lease: %q %q", run, lease)+ }++ // The next turn starts clean and records only its own lease.+ next := domain.AssistantGeneration{ID: "next", SessionID: "fenced-chat", UserID: 11, Status: "running"}+ if err = s.BeginAssistantGeneration(next); err != nil {+ t.Fatal(err)+ }+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "fenced-chat", "next", "run-2", "own"); err != nil {+ t.Fatal(err)+ }+ if err = s.DB.Model(&assistantGenerationRow{}).Where("session_id = ?", "fenced-chat").Update("updated_at", time.Now().Add(-time.Minute).UTC()).Error; err != nil {+ t.Fatal(err)+ }+ if _, lease, ok := s.ReapAssistantGeneration(11, "fenced-chat"); !ok || lease != (AssistantGenerationLease{RunID: "run-2", Token: "own"}) {+ t.Fatalf("reap returned %+v", lease)+ }+}diff --git a/internal/repo/assistant_generations.go b/internal/repo/assistant_generations.goindex 24875c83..4a406191 100644--- a/internal/repo/assistant_generations.go+++ b/internal/repo/assistant_generations.go@@ -23,6 +23,21 @@ type assistantGenerationRow struct {// the next turn starts. Rows that ended before the column existed keep// NULL, which is how read tracking leaves older history unmarked.FinishedAt *time.Time+ // WorkspaceRunID and WorkspaceLease name the foreground workspace lease+ // this turn holds. Only that exact token is released when the turn is+ // reaped; a lease another turn took in the meantime never matches.+ WorkspaceRunID string `gorm:"size:64"`+ WorkspaceLease string `gorm:"size:64"`+}++// assistantHeartbeatTimeout is how long a running turn may go without a+// durable heartbeat before a reader that does not own it declares it dead.+const assistantHeartbeatTimeout = 45 * time.Second++// AssistantGenerationLease is the workspace lease a reaped turn held.+type AssistantGenerationLease struct {+ RunID string+ Token string}func (assistantGenerationRow) TableName() string { return "assistant_generations" }@@ -35,30 +50,119 @@ func decodeGeneration(row assistantGenerationRow) domain.AssistantGeneration {g.UpdatedAt = row.UpdatedAtreturn g}-func (r *AssistantRepo) AssistantGeneration(userID int64, sessionID string) (domain.AssistantGeneration, bool) {++// PeekAssistantGeneration reads the latest turn without declaring it dead.+// The owner of a live turn must use this. AssistantGeneration also reaps a+// turn whose heartbeat is older than 45 seconds, which is only safe once the+// owning process is gone: a slow commit stores an old updated_at and the next+// read would otherwise cancel a CLI that is still working.+func (r *AssistantRepo) PeekAssistantGeneration(userID int64, sessionID string) (domain.AssistantGeneration, bool) {+ row, ok := r.loadAssistantGeneration(userID, sessionID)+ if !ok {+ return domain.AssistantGeneration{}, false+ }+ return decodeGeneration(row), true+}++func (r *AssistantRepo) loadAssistantGeneration(userID int64, sessionID string) (assistantGenerationRow, bool) {var row assistantGenerationRowif err := r.DB.Where("session_id = ? AND user_id = ?", sessionID, userID).Take(&row).Error; err != nil {- return domain.AssistantGeneration{}, false+ return assistantGenerationRow{}, false}+ return row, true+}++func (r *AssistantRepo) AssistantGeneration(userID int64, sessionID string) (domain.AssistantGeneration, bool) {+ g, _, ok := r.ReapAssistantGeneration(userID, sessionID)+ return g, ok+}++// ReapAssistantGeneration reads a turn and, when its heartbeat is late,+// records it as interrupted. The lease is returned only to the caller whose+// update actually reaped the turn, so the lease is released at most once.+func (r *AssistantRepo) ReapAssistantGeneration(userID int64, sessionID string) (domain.AssistantGeneration, AssistantGenerationLease, bool) {+ row, ok := r.loadAssistantGeneration(userID, sessionID)+ if !ok {+ return domain.AssistantGeneration{}, AssistantGenerationLease{}, false+ }+ var lease AssistantGenerationLease// A server that died cannot leave a conversation stuck forever. Never replay// a model turn automatically, because its tools may already have written.- if row.Status == "running" && time.Since(row.UpdatedAt) > 45*time.Second {+ if row.Status == "running" && r.heartbeatLate(r.DB, row.UpdatedAt) {g := decodeGeneration(row)g.Status = "interrupted"g.Error = "服务端生成已中断,请检查已执行操作后重试"g.Failure = &domain.AssistantFailure{Code: "EXECUTION_INTERRUPTED", Phase: "server", Message: g.Error, Suggestion: "检查已执行操作后再重试;可下载诊断信息交给管理员排查。"}_ = r.DB.Transaction(func(tx *gorm.DB) error {- res := tx.Model(&assistantGenerationRow{}).Where("session_id = ? AND generation_id = ? AND status = ? AND updated_at = ?", sessionID, row.GenerationID, "running", row.UpdatedAt).Updates(map[string]any{"status": g.Status, "payload": mustJSON(g)})+ res := r.lateHeartbeatOnly(tx.Model(&assistantGenerationRow{}).Where("session_id = ? AND generation_id = ? AND status = ? AND updated_at = ?", sessionID, row.GenerationID, "running", row.UpdatedAt)).Updates(map[string]any{"status": g.Status, "payload": mustJSON(g), "updated_at": r.generationClock()})if res.Error != nil || res.RowsAffected != 1 {return res.Error}- return checkpointAssistantDiagnostics(tx, g, nil)+ if err := checkpointAssistantDiagnostics(tx, g, nil); err != nil {+ return err+ }+ lease = AssistantGenerationLease{RunID: row.WorkspaceRunID, Token: row.WorkspaceLease}+ return nil})- if err := r.DB.Where("session_id = ? AND user_id = ?", sessionID, userID).Take(&row).Error; err != nil {- return domain.AssistantGeneration{}, false+ reloaded, ok := r.loadAssistantGeneration(userID, sessionID)+ if !ok {+ return domain.AssistantGeneration{}, AssistantGenerationLease{}, false}+ row = reloaded}- return decodeGeneration(row), true+ return decodeGeneration(row), lease, true+}++// RecordAssistantGenerationWorkspaceLease remembers the foreground workspace+// lease a running turn took, so a reader that reaps the turn after its process+// died can release exactly that lease. The heartbeat clock is not touched.+func (r *AssistantRepo) RecordAssistantGenerationWorkspaceLease(userID int64, sessionID, generationID, runID, token string) error {+ return r.DB.Model(&assistantGenerationRow{}).+ Where("session_id = ? AND user_id = ? AND generation_id = ? AND status = ?", sessionID, userID, generationID, "running").+ UpdateColumns(map[string]any{"workspace_run_id": runID, "workspace_lease": token}).Error+}++// heartbeatNow is the clock a heartbeat's age is measured on. MySQL stamps+// heartbeats with its own UTC_TIMESTAMP(3) (see generationClock), so their age+// is measured on that same clock: comparing them with this machine's time.Now()+// would turn any skew between the API host and the database into turns reaped+// while alive, or never reaped. ok is false when the clock cannot be read, and+// callers then treat the heartbeat as fresh rather than reap on a guess.+func (r *AssistantRepo) heartbeatNow(db *gorm.DB) (time.Time, bool) {+ if !r.isMySQL() {+ return time.Now().UTC(), true+ }+ var now time.Time+ if err := db.Raw("SELECT UTC_TIMESTAMP(3)").Scan(&now).Error; err != nil || now.IsZero() {+ return time.Time{}, false+ }+ return now, true+}++// heartbeatLate reports a heartbeat older than assistantHeartbeatTimeout on+// the heartbeat clock.+func (r *AssistantRepo) heartbeatLate(db *gorm.DB, updatedAt time.Time) bool {+ now, ok := r.heartbeatNow(db)+ return ok && now.Sub(updatedAt) > assistantHeartbeatTimeout+}++// lateHeartbeatOnly lets the database itself confirm, when the reap commits,+// that the heartbeat is still late on its own clock.+func (r *AssistantRepo) lateHeartbeatOnly(q *gorm.DB) *gorm.DB {+ if !r.isMySQL() {+ return q+ }+ return q.Where("updated_at < UTC_TIMESTAMP(3) - INTERVAL ? MICROSECOND", assistantHeartbeatTimeout.Microseconds())+}++// generationClock is the commit-time clock on MySQL. A Go timestamp captured+// before a stalled commit is already older than the 45 second reap when it+// becomes visible, so the next read interrupts a turn that just checked in.+func (r *AssistantRepo) generationClock() any {+ if r.isMySQL() {+ return clause.Expr{SQL: "UTC_TIMESTAMP(3)"}+ }+ return time.Now().UTC()}// BeginAssistantGeneration may lock a session that does not exist yet; two@@ -113,7 +217,7 @@ func (r *AssistantRepo) BeginAssistantGeneration(g domain.AssistantGeneration, pif old.Status == "running" {return fmt.Errorf("此会话正在生成,请等待完成或停止")}- if err := tx.Model(&old).Updates(map[string]any{"generation_id": g.ID, "status": "running", "cancel_requested": false, "payload": mustJSON(g), "updated_at": time.Now().UTC(), "finished_at": nil}).Error; err != nil {+ if err := tx.Model(&old).Updates(map[string]any{"generation_id": g.ID, "status": "running", "cancel_requested": false, "payload": mustJSON(g), "updated_at": r.generationClock(), "finished_at": nil, "workspace_run_id": "", "workspace_lease": ""}).Error; err != nil {return err}return checkpointAssistantDiagnostics(tx, g, nil)@@ -124,6 +228,12 @@ func (r *AssistantRepo) BeginAssistantGeneration(g domain.AssistantGeneration, pif err := tx.Create(&assistantGenerationRow{SessionID: g.SessionID, UserID: g.UserID, GenerationID: g.ID, Status: "running", Payload: mustJSON(g), UpdatedAt: time.Now().UTC()}).Error; err != nil {return err}+ if r.isMySQL() {+ // The first heartbeat is on the database clock like every later one.+ if err := tx.Model(&assistantGenerationRow{}).Where("session_id = ?", g.SessionID).UpdateColumn("updated_at", r.generationClock()).Error; err != nil {+ return err+ }+ }return checkpointAssistantDiagnostics(tx, g, nil)})})@@ -155,7 +265,7 @@ func (r *AssistantRepo) checkpointAssistantGeneration(g domain.AssistantGeneraticurrent.NativeApprovals = nil}now := time.Now().UTC()- changes := map[string]any{"status": g.Status, "payload": mustJSON(current), "updated_at": now}+ changes := map[string]any{"status": g.Status, "payload": mustJSON(current), "updated_at": r.generationClock()}if g.Status != "running" {changes["finished_at"] = now}@@ -163,8 +273,17 @@ func (r *AssistantRepo) checkpointAssistantGeneration(g domain.AssistantGeneratiif res.Error != nil {return res.Error}+ // MySQL counts changed rows. A heartbeat whose payload and clock match+ // the stored row reports 0 even though the turn is still running.+ // Treating that as "the turn disappeared" cancels a live CLI.if res.RowsAffected != 1 {- return fmt.Errorf("生成状态已变化")+ if g.Status != "running" {+ return fmt.Errorf("生成状态已变化")+ }+ var again assistantGenerationRow+ if err := tx.Where("session_id = ? AND user_id = ? AND generation_id = ? AND status = ?", g.SessionID, g.UserID, g.ID, "running").Take(&again).Error; err != nil {+ return fmt.Errorf("生成状态已变化")+ }}if !withDiagnostics {return nil@@ -182,7 +301,7 @@ func (r *AssistantRepo) SteerAssistantGeneration(userID int64, sessionID, id str}g := decodeGeneration(row)g.Steer = &steer- return tx.Model(&row).Updates(map[string]any{"cancel_requested": true, "payload": mustJSON(g), "updated_at": time.Now().UTC()}).Error+ return tx.Model(&row).Updates(map[string]any{"cancel_requested": true, "payload": mustJSON(g), "updated_at": r.generationClock()}).Error})}func (r *AssistantRepo) CancelAssistantGeneration(userID int64, sessionID, id string) error {@@ -207,6 +326,9 @@ type AssistantGenerationState struct {// Seen reports that the reader has opened the conversation since this// reply (see MarkAssistantSessionRead).Seen bool+ // Stale reports a turn still stored as running whose heartbeat is late.+ // Status reads "interrupted" for it, but nothing has been written yet.+ Stale bool}// AssistantGenerationStatuses reads the latest reply of each conversation@@ -232,9 +354,11 @@ func (r *AssistantRepo) AssistantGenerationStatuses(readerID int64, ids []stringif err != nil {return out}+ now, clockOK := r.heartbeatNow(r.DB)for _, row := range rows {status := row.Status- if status == "running" && time.Since(row.UpdatedAt) > 45*time.Second {+ stale := status == "running" && clockOK && now.Sub(row.UpdatedAt) > assistantHeartbeatTimeout+ if stale {status = "interrupted"}out[row.SessionID] = AssistantGenerationState{@@ -242,6 +366,7 @@ func (r *AssistantRepo) AssistantGenerationStatuses(readerID int64, ids []stringStatus: status,FinishedAt: row.FinishedAt,Seen: row.ReadID != nil && *row.ReadID == row.GenerationID,+ Stale: stale,}}return outdiff --git a/internal/repo/assistant_generations_test.go b/internal/repo/assistant_generations_test.goindex 8b765f7d..29fd7317 100644--- a/internal/repo/assistant_generations_test.go+++ b/internal/repo/assistant_generations_test.go@@ -59,3 +59,87 @@ func TestAssistantGenerationClaimCheckpointAndPrivacy(t *testing.T) {t.Fatal("orphan generation did not become interrupted")}}++func TestPeekAssistantGenerationLeavesStaleTurnRunning(t *testing.T) {+ s, err := OpenTest()+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ g := domain.AssistantGeneration{ID: "live", SessionID: "live-chat", UserID: 11, Status: "running"}+ if err = s.BeginAssistantGeneration(g); err != nil {+ t.Fatal(err)+ }+ if err = s.DB.Model(&assistantGenerationRow{}).Where("session_id = ?", "live-chat").Update("updated_at", time.Now().Add(-time.Minute)).Error; err != nil {+ t.Fatal(err)+ }+ peek, ok := s.PeekAssistantGeneration(11, "live-chat")+ if !ok || peek.Status != "running" {+ t.Fatalf("peek reaped a live turn: %+v", peek)+ }+ if err = s.HeartbeatAssistantGeneration(g); err != nil {+ t.Fatal(err)+ }+ if err = s.HeartbeatAssistantGeneration(g); err != nil {+ t.Fatal(err)+ }+ peek, _ = s.PeekAssistantGeneration(11, "live-chat")+ if peek.Status != "running" || time.Since(peek.UpdatedAt) > 5*time.Second {+ t.Fatalf("heartbeat did not refresh the live turn: %+v", peek)+ }+}++// Only the reader whose update reaps a turn gets its lease back, and the next+// turn starts without the previous turn's lease.+func TestReapAssistantGenerationReturnsRecordedLeaseOnce(t *testing.T) {+ s, err := OpenTest()+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ g := domain.AssistantGeneration{ID: "dead", SessionID: "dead-chat", UserID: 11, Status: "running"}+ if err = s.BeginAssistantGeneration(g); err != nil {+ t.Fatal(err)+ }+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "dead-chat", "dead", "run-1", "token-1"); err != nil {+ t.Fatal(err)+ }+ // Recording a lease is not a heartbeat.+ stale := time.Now().Add(-time.Minute).UTC()+ if err = s.DB.Model(&assistantGenerationRow{}).Where("session_id = ?", "dead-chat").Update("updated_at", stale).Error; err != nil {+ t.Fatal(err)+ }+ if err = s.RecordAssistantGenerationWorkspaceLease(11, "dead-chat", "dead", "run-1", "token-1"); err != nil {+ t.Fatal(err)+ }+ if peek, _ := s.PeekAssistantGeneration(11, "dead-chat"); time.Since(peek.UpdatedAt) < 50*time.Second {+ t.Fatalf("recording a lease refreshed the heartbeat: %+v", peek.UpdatedAt)+ }+ if states := s.AssistantGenerationStatuses(11, []string{"dead-chat"}); !states["dead-chat"].Stale || states["dead-chat"].Status != "interrupted" {+ t.Fatalf("late heartbeat not projected as stale: %+v", states["dead-chat"])+ }+ reaped, lease, ok := s.ReapAssistantGeneration(11, "dead-chat")+ if !ok || reaped.Status != "interrupted" || lease != (AssistantGenerationLease{RunID: "run-1", Token: "token-1"}) {+ t.Fatalf("reap = %+v %+v %v", reaped, lease, ok)+ }+ if states := s.AssistantGenerationStatuses(11, []string{"dead-chat"}); states["dead-chat"].Stale {+ t.Fatalf("a stored interruption is reported as stale: %+v", states["dead-chat"])+ }+ if _, lease, _ = s.ReapAssistantGeneration(11, "dead-chat"); lease.Token != "" {+ t.Fatalf("a second reader got the lease again: %+v", lease)+ }+ // Another user cannot record a lease on this turn.+ next := domain.AssistantGeneration{ID: "next", SessionID: "dead-chat", UserID: 11, Status: "running"}+ if err = s.BeginAssistantGeneration(next); err != nil {+ t.Fatal(err)+ }+ if err = s.RecordAssistantGenerationWorkspaceLease(12, "dead-chat", "next", "run-1", "intruder"); err != nil {+ t.Fatal(err)+ }+ if err = s.DB.Model(&assistantGenerationRow{}).Where("session_id = ?", "dead-chat").Update("updated_at", stale).Error; err != nil {+ t.Fatal(err)+ }+ if _, lease, _ = s.ReapAssistantGeneration(11, "dead-chat"); lease.Token != "" {+ t.Fatalf("the next turn inherited a lease: %+v", lease)+ }+}diff --git a/internal/repo/workspace_checkpoints.go b/internal/repo/workspace_checkpoints.goindex ad23ccdb..2e67c0ed 100644--- a/internal/repo/workspace_checkpoints.go+++ b/internal/repo/workspace_checkpoints.go@@ -30,6 +30,11 @@ type WorkspaceCheckpoint struct {LeaseHandoffID string `gorm:"size:64" json:"-"`LeaseToken string `gorm:"size:64" json:"-"`LeaseUntil *time.Time `json:"-"`+ // LeaseRenewedAt is when the lease was last taken or renewed, on the+ // database clock on MySQL (see leaseRenewalClock). LeaseUntil is written+ // by the holder's own clock, so it cannot tell another replica how long+ // ago the holder renewed.+ LeaseRenewedAt *time.Time `json:"-"`RestoreStatus string `gorm:"size:32" json:"-"`RestoreStarted *time.Time `json:"-"`RestoreError string `gorm:"type:text" json:"-"`@@ -50,6 +55,9 @@ var ErrWorkspaceCheckpointBackoff = errors.New("workspace checkpoint save is bacconst workspaceBackgroundSavePrefix = "save:"+// workspaceUseTerm is how long a workspace lease lasts from its last renewal.+const workspaceUseTerm = 15 * time.Minute+func (r *RunsRepo) WorkspaceCheckpoint(runID string) (WorkspaceCheckpoint, bool) {var item WorkspaceCheckpointerr := r.DB.Omit("archive").Where("run_id = ?", runID).Take(&item).Error@@ -128,7 +136,7 @@ func (r *RunsRepo) beginWorkspaceLease(runID string, now time.Time, touch, restoif !touch && checkpoint.SaveRetryAt != nil && now.Before(*checkpoint.SaveRetryAt) {return ErrWorkspaceCheckpointBackoff}- until := now.Add(15 * time.Minute)+ until := now.Add(workspaceUseTerm)lastUsed := nowif !touch && run.Updated != nil {lastUsed = *run.Updated@@ -137,7 +145,7 @@ func (r *RunsRepo) beginWorkspaceLease(runID string, now time.Time, touch, restoif err := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(&row).Error; err != nil {return err}- values := map[string]any{"lease_token": token, "lease_until": until, "lease_handoff_id": ""}+ values := map[string]any{"lease_token": token, "lease_until": until, "lease_renewed_at": r.leaseRenewalClock(now), "lease_handoff_id": ""}if touch {values["last_used_at"] = now}@@ -187,8 +195,8 @@ func (r *RunsRepo) RenewWorkspaceUse(runID, token string, now time.Time) error {}// The row is locked through the run and was just read with this token;// an unchanged lease_until (same-millisecond renewal) is still a renewal.- until := now.Add(15 * time.Minute)- return tx.Model(&WorkspaceCheckpoint{}).Where("run_id = ? AND lease_token = ?", runID, token).Update("lease_until", until).Error+ until := now.Add(workspaceUseTerm)+ return tx.Model(&WorkspaceCheckpoint{}).Where("run_id = ? AND lease_token = ?", runID, token).Updates(map[string]any{"lease_until": until, "lease_renewed_at": r.leaseRenewalClock(now)}).Error})}@@ -355,6 +363,59 @@ func holdsWorkspaceLease(tx *gorm.DB, runID, token string) error {return nil}+// ReleaseAbandonedWorkspaceUse drops the foreground lease a reaped turn held.+// A dead API process cannot finish the lease it held in memory, and the next+// turn would fail for the rest of the 15 minute term.+//+// Two fences keep a live owner's lease in place. The token must be the one the+// reaped turn recorded, so a lease another turn took after the reap never+// matches. Its last renewal must also be older than the heartbeat timeout:+// every holder renews well within it (generations every 500ms, workspace turns+// every 30s), so a lease still being renewed by a process on any replica is+// kept. A background save token never matches a foreground token.+//+// The renewal age is measured on the clock that stamped lease_renewed_at (the+// database's on MySQL), so a holder whose clock runs behind the reader's does+// not look silent. A lease taken before that column existed has no stamp and+// falls back to its lease_until.+func (r *RunsRepo) ReleaseAbandonedWorkspaceUse(runID, token string, now time.Time) (bool, error) {+ runID, token = strings.TrimSpace(runID), strings.TrimSpace(token)+ if runID == "" || token == "" || strings.HasPrefix(token, workspaceBackgroundSavePrefix) {+ return false, nil+ }+ released := false+ err := r.DB.Transaction(func(tx *gorm.DB) error {+ if _, err := r.takeRunForUpdate(tx, runID); err != nil {+ if errors.Is(err, gorm.ErrRecordNotFound) {+ return nil+ }+ return err+ }+ silent := tx.Where("lease_renewed_at IS NULL AND lease_until <= ?", now.Add(workspaceUseTerm-assistantHeartbeatTimeout))+ if r.isMySQL() {+ silent = silent.Or("lease_renewed_at <= UTC_TIMESTAMP(3) - INTERVAL ? MICROSECOND", assistantHeartbeatTimeout.Microseconds())+ } else {+ silent = silent.Or("lease_renewed_at <= ?", now.Add(-assistantHeartbeatTimeout))+ }+ result := tx.Model(&WorkspaceCheckpoint{}).+ Where("run_id = ? AND lease_token = ? AND lease_until > ?", runID, token, now).+ Where(silent).+ Updates(map[string]any{"lease_token": "", "lease_until": nil, "lease_handoff_id": ""})+ released = result.RowsAffected == 1+ return result.Error+ })+ return released, err+}++// leaseRenewalClock stamps lease_renewed_at: the database's commit-time clock+// on MySQL, shared by every replica, and the caller's clock elsewhere.+func (r *RunsRepo) leaseRenewalClock(now time.Time) any {+ if r.isMySQL() {+ return clause.Expr{SQL: "UTC_TIMESTAMP(3)"}+ }+ return now+}+// ReleaseWorkspace marks a workspace no conversation is bound to any more, so// the idle sweep saves and reclaims it on its next pass.func (r *RunsRepo) ReleaseWorkspace(runID string, now time.Time) error {diff --git a/internal/repo/workspace_checkpoints_test.go b/internal/repo/workspace_checkpoints_test.goindex 00b61e2a..a5c0f764 100644--- a/internal/repo/workspace_checkpoints_test.go+++ b/internal/repo/workspace_checkpoints_test.go@@ -8,6 +8,95 @@ import ("time")+// A reaped turn releases only the lease it recorded, and only once nothing+// renews it any more. A live holder on any replica, a newer turn's lease and a+// background save all keep theirs.+func TestReleaseAbandonedWorkspaceUseFreesOnlyTheReapedTurnsLease(t *testing.T) {+ s, err := OpenTest()+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ now := time.Now().UTC()+ taken := now.Add(-2 * time.Minute)+ for _, id := range []string{"abandoned-use", "renewed-use", "replaced-use", "background-save"} {+ if err = s.PutRun(domain.Run{ID: id, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {+ t.Fatal(err)+ }+ }+ release := func(runID, token string) bool {+ t.Helper()+ released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now)+ if err != nil {+ t.Fatal(err)+ }+ return released+ }+ leaseOf := func(runID string) string {+ t.Helper()+ item, _ := s.WorkspaceCheckpoint(runID)+ return item.LeaseToken+ }++ abandoned, err := s.BeginWorkspaceUse("abandoned-use", taken)+ if err != nil {+ t.Fatal(err)+ }+ if !release("abandoned-use", abandoned) {+ t.Fatal("an unrenewed lease of a reaped turn was kept")+ }+ if release("abandoned-use", abandoned) {+ t.Fatal("a lease was released twice")+ }+ if _, err = s.BeginWorkspaceUse("abandoned-use", now); err != nil {+ t.Fatal("abandoned foreground lease still blocked the next turn", err)+ }++ // Another replica still executing the turn renews its lease.+ renewed, err := s.BeginWorkspaceUse("renewed-use", taken)+ if err != nil {+ t.Fatal(err)+ }+ if err = s.RenewWorkspaceUse("renewed-use", renewed, now.Add(-10*time.Second)); err != nil {+ t.Fatal(err)+ }+ if release("renewed-use", renewed) || leaseOf("renewed-use") != renewed {+ t.Fatal("a lease its holder still renews was released")+ }++ // The reaped turn's lease ended and a newer turn took the workspace+ // before the reader released: the stale token must not free it.+ old, err := s.BeginWorkspaceUse("replaced-use", taken)+ if err != nil {+ t.Fatal(err)+ }+ if err = s.FinishWorkspaceUse("replaced-use", old, taken, nil, ""); err != nil {+ t.Fatal(err)+ }+ newer, err := s.BeginWorkspaceUse("replaced-use", taken)+ if err != nil {+ t.Fatal(err)+ }+ if release("replaced-use", old) || leaseOf("replaced-use") != newer {+ t.Fatal("a newer turn's lease was released with the reaped turn's token")+ }++ token, err := s.BeginWorkspaceUse("background-save", taken)+ if err != nil {+ t.Fatal(err)+ }+ save, err := s.HandoffWorkspaceUseToSave("background-save", token, taken)+ if err != nil {+ t.Fatal(err)+ }+ if release("background-save", token) || release("background-save", save) || leaseOf("background-save") != save {+ t.Fatal("background save lease dropped")+ }+ if release("", abandoned) || release("missing-run", abandoned) {+ t.Fatal("released a lease without a run")+ }+}+func TestWorkspaceCheckpointLeaseCASAndDurableIdleDeadline(t *testing.T) {s, err := OpenTest()if err != nil {diff --git a/internal/repo/workspace_lease_clock_mysql_test.go b/internal/repo/workspace_lease_clock_mysql_test.gonew file mode 100644index 00000000..d0902d5f--- /dev/null+++ b/internal/repo/workspace_lease_clock_mysql_test.go@@ -0,0 +1,79 @@+package repo++import (+ "fmt"+ "os"+ "testing"+ "time"++ "agentflow/internal/domain"+)++// lease_until is written on the holder's clock. Whether a holder on another+// replica stopped renewing is read from lease_renewed_at, which MySQL stamps+// with its own clock, so a holder whose clock runs behind keeps a lease it+// just renewed, and a lease nobody renewed for 45 seconds on the database+// clock is released. SET timestamp moves the database clock of the one+// connection the store is pinned to.+func TestMySQLAbandonedLeaseAgeIsMeasuredOnTheDatabaseClock(t *testing.T) {+ dsn := os.Getenv("AGENTFLOW_TEST_MYSQL_DSN")+ if dsn == "" {+ t.Skip("local isolated MySQL required")+ }+ s, err := Open(dsn)+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ sqlDB, err := s.DB.DB()+ if err != nil {+ t.Fatal(err)+ }+ sqlDB.SetMaxOpenConns(1)+ sqlDB.SetMaxIdleConns(1)+ skew := func(d time.Duration) {+ t.Helper()+ if err := s.DB.Exec("SET timestamp = ?", float64(time.Now().Add(d).UnixMicro())/1e6).Error; err != nil {+ t.Fatal(err)+ }+ }+ defer s.DB.Exec("SET timestamp = DEFAULT")++ runID := fmt.Sprintf("lease-clock-%d", time.Now().UnixNano())+ if err := s.PutRun(domain.Run{ID: runID, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {+ t.Fatal(err)+ }+ defer s.DB.Where("run_id = ?", runID).Delete(&WorkspaceCheckpoint{})+ defer s.DB.Where("id = ?", runID).Delete(&runRow{})+ now := time.Now().UTC()+ token, err := s.BeginWorkspaceUse(runID, now)+ if err != nil {+ t.Fatal(err)+ }+ // The holder's clock runs two minutes behind the reader's and it renews+ // now: its lease_until looks two minutes old to the reader.+ if err := s.RenewWorkspaceUse(runID, token, now.Add(-2*time.Minute)); err != nil {+ t.Fatal(err)+ }+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now); err != nil || released {+ t.Fatalf("a lease its holder just renewed was released: %v %v", released, err)+ }+ // Two minutes later on the database clock nothing renewed it.+ skew(2 * time.Minute)+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now); err != nil || !released {+ t.Fatalf("a lease nobody renewed for two minutes was kept: %v %v", released, err)+ }+ skew(0)++ // A lease taken before lease_renewed_at existed falls back to lease_until.+ legacy, err := s.BeginWorkspaceUse(runID, now)+ if err != nil {+ t.Fatal(err)+ }+ if err := s.DB.Model(&WorkspaceCheckpoint{}).Where("run_id = ?", runID).Updates(map[string]any{"lease_renewed_at": nil, "lease_until": now.Add(workspaceUseTerm - time.Minute)}).Error; err != nil {+ t.Fatal(err)+ }+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, legacy, now); err != nil || !released {+ t.Fatalf("an unstamped lease silent for a minute was kept: %v %v", released, err)+ }+}diff --git a/internal/repo/workspace_lease_renewal_stamp_test.go b/internal/repo/workspace_lease_renewal_stamp_test.gonew file mode 100644index 00000000..d4fc3925--- /dev/null+++ b/internal/repo/workspace_lease_renewal_stamp_test.go@@ -0,0 +1,80 @@+package repo++import (+ "testing"+ "time"++ "agentflow/internal/domain"+)++// Taking and renewing a lease stamp lease_renewed_at, and the abandoned-lease+// release goes by that stamp when it is there. A lease taken before the column+// existed has no stamp and is still judged by its lease_until.+func TestReleaseAbandonedWorkspaceUseJudgesByRenewalStampOrLegacyLeaseUntil(t *testing.T) {+ s, err := OpenTest()+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ now := time.Now().UTC()+ for _, id := range []string{"stamped-take", "legacy-silent", "legacy-renewed", "stamp-decides"} {+ if err = s.PutRun(domain.Run{ID: id, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {+ t.Fatal(err)+ }+ }+ take := func(runID string, at time.Time) string {+ t.Helper()+ token, err := s.BeginWorkspaceUse(runID, at)+ if err != nil {+ t.Fatal(err)+ }+ return token+ }+ set := func(runID string, values map[string]any) {+ t.Helper()+ if err := s.DB.Model(&WorkspaceCheckpoint{}).Where("run_id = ?", runID).Updates(values).Error; err != nil {+ t.Fatal(err)+ }+ }+ release := func(runID, token string) bool {+ t.Helper()+ released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now)+ if err != nil {+ t.Fatal(err)+ }+ return released+ }++ // Taking the lease stamps it on the caller's clock (SQLite).+ taken := now.Add(-20 * time.Second)+ take("stamped-take", taken)+ var row WorkspaceCheckpoint+ if err := s.DB.Where("run_id = ?", "stamped-take").Take(&row).Error; err != nil {+ t.Fatal(err)+ }+ if row.LeaseRenewedAt == nil || row.LeaseRenewedAt.Sub(taken).Abs() > time.Millisecond {+ t.Fatalf("taking a lease did not stamp lease_renewed_at with %v: %v", taken, row.LeaseRenewedAt)+ }++ // Unstamped lease whose lease_until says it was last renewed a minute ago.+ legacy := take("legacy-silent", now)+ set("legacy-silent", map[string]any{"lease_renewed_at": nil, "lease_until": now.Add(workspaceUseTerm - time.Minute)})+ if !release("legacy-silent", legacy) {+ t.Fatal("an unstamped lease silent for a minute was kept")+ }++ // Unstamped lease renewed ten seconds ago by its lease_until.+ renewed := take("legacy-renewed", now)+ set("legacy-renewed", map[string]any{"lease_renewed_at": nil, "lease_until": now.Add(workspaceUseTerm - 10*time.Second)})+ if release("legacy-renewed", renewed) {+ t.Fatal("an unstamped lease renewed ten seconds ago was released")+ }++ // A stamp from ten seconds ago keeps the lease even though lease_until,+ // written on the holder's clock, looks a minute old.+ stamped := take("stamp-decides", now)+ set("stamp-decides", map[string]any{"lease_renewed_at": now.Add(-10 * time.Second), "lease_until": now.Add(workspaceUseTerm - time.Minute)})+ if release("stamp-decides", stamped) {+ t.Fatal("a freshly stamped lease was released by its lease_until")+ }+}diff --git a/internal/repo/workspace_lease_take_clock_mysql_test.go b/internal/repo/workspace_lease_take_clock_mysql_test.gonew file mode 100644index 00000000..82f8b4fd--- /dev/null+++ b/internal/repo/workspace_lease_take_clock_mysql_test.go@@ -0,0 +1,59 @@+package repo++import (+ "fmt"+ "os"+ "testing"+ "time"++ "agentflow/internal/domain"+)++// Taking a lease stamps lease_renewed_at on the database clock too, not only+// renewing it. A holder whose clock runs two minutes behind the reader's has+// just taken the lease and not renewed yet: its lease_until looks two minutes+// old to the reader, but the lease must be kept until the database clock says+// it went unrenewed.+func TestMySQLLeaseJustTakenByAHolderWithASlowClockIsKept(t *testing.T) {+ dsn := os.Getenv("AGENTFLOW_TEST_MYSQL_DSN")+ if dsn == "" {+ t.Skip("local isolated MySQL required")+ }+ s, err := Open(dsn)+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ sqlDB, err := s.DB.DB()+ if err != nil {+ t.Fatal(err)+ }+ sqlDB.SetMaxOpenConns(1)+ sqlDB.SetMaxIdleConns(1)+ skew := func(d time.Duration) {+ t.Helper()+ if err := s.DB.Exec("SET timestamp = ?", float64(time.Now().Add(d).UnixMicro())/1e6).Error; err != nil {+ t.Fatal(err)+ }+ }+ defer s.DB.Exec("SET timestamp = DEFAULT")++ runID := fmt.Sprintf("lease-take-clock-%d", time.Now().UnixNano())+ if err := s.PutRun(domain.Run{ID: runID, Status: "succeeded", Nodes: map[string]*domain.NodeRun{}}); err != nil {+ t.Fatal(err)+ }+ defer s.DB.Where("run_id = ?", runID).Delete(&WorkspaceCheckpoint{})+ defer s.DB.Where("id = ?", runID).Delete(&runRow{})+ now := time.Now().UTC()+ token, err := s.BeginWorkspaceUse(runID, now.Add(-2*time.Minute))+ if err != nil {+ t.Fatal(err)+ }+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now); err != nil || released {+ t.Fatalf("a lease its holder just took was released: %v %v", released, err)+ }+ skew(2 * time.Minute)+ if released, err := s.ReleaseAbandonedWorkspaceUse(runID, token, now); err != nil || !released {+ t.Fatalf("a lease nobody renewed for two minutes since it was taken was kept: %v %v", released, err)+ }+}diff --git a/migrations/032_assistant_generation_workspace_lease.sql b/migrations/032_assistant_generation_workspace_lease.sqlnew file mode 100644index 00000000..b2087cf2--- /dev/null+++ b/migrations/032_assistant_generation_workspace_lease.sql@@ -0,0 +1,35 @@+-- Optional manual additive DDL. Startup AutoMigrate also adds these columns.+-- A running turn records the foreground workspace lease it holds, so a reader+-- that reaps the turn after its process died releases exactly that lease.+-- No backfill: turns started before the columns existed record nothing and+-- keep the old behaviour (their lease runs out after 15 minutes).+-- run_workspace_checkpoints.lease_renewed_at stamps each lease take and renewal+-- with the database clock, so a replica judges how long ago another replica+-- renewed without comparing two hosts' clocks. A lease taken before the column+-- existed has no stamp and is judged by its lease_until as before.+SET @asst_ws_run_ddl = IF(+ (SELECT COUNT(*) FROM information_schema.columns+ WHERE table_schema=DATABASE() AND table_name='assistant_generations' AND column_name='workspace_run_id')=0,+ 'ALTER TABLE assistant_generations ADD COLUMN workspace_run_id VARCHAR(64) NULL',+ 'SELECT 1');+PREPARE asst_ws_run_stmt FROM @asst_ws_run_ddl;+EXECUTE asst_ws_run_stmt;+DEALLOCATE PREPARE asst_ws_run_stmt;++SET @asst_ws_lease_ddl = IF(+ (SELECT COUNT(*) FROM information_schema.columns+ WHERE table_schema=DATABASE() AND table_name='assistant_generations' AND column_name='workspace_lease')=0,+ 'ALTER TABLE assistant_generations ADD COLUMN workspace_lease VARCHAR(64) NULL',+ 'SELECT 1');+PREPARE asst_ws_lease_stmt FROM @asst_ws_lease_ddl;+EXECUTE asst_ws_lease_stmt;+DEALLOCATE PREPARE asst_ws_lease_stmt;++SET @ws_lease_renewed_ddl = IF(+ (SELECT COUNT(*) FROM information_schema.columns+ WHERE table_schema=DATABASE() AND table_name='run_workspace_checkpoints' AND column_name='lease_renewed_at')=0,+ 'ALTER TABLE run_workspace_checkpoints ADD COLUMN lease_renewed_at DATETIME(3) NULL',+ 'SELECT 1');+PREPARE ws_lease_renewed_stmt FROM @ws_lease_renewed_ddl;+EXECUTE ws_lease_renewed_stmt;+DEALLOCATE PREPARE ws_lease_renewed_stmt;