Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(employees): Reset closes the generation's open work (1b.21)
trohitga6f9af68a657
diff --git a/RFC-0006-TEMPORAL-DESIGN-PATTERNS.md b/RFC-0006-TEMPORAL-DESIGN-PATTERNS.mdindex 0e8ce61b..96c4a9b2 100644--- a/RFC-0006-TEMPORAL-DESIGN-PATTERNS.md+++ b/RFC-0006-TEMPORAL-DESIGN-PATTERNS.md@@ -6172,7 +6172,7 @@ Phase 1b is new to RFC-0005's rollout (§20).| Id | Fix | § ||---|---|---|| 1b.20 | Compensations retry until they land: delegation cleanup (`DELEGATION_CLEANUP_RETRY`, now unlimited) and job failure (`BOOKKEEPING_RETRY`, moved to `_retry_policies.py`). `cancel_agent_delegation` answers `absent` for a missing row. The run record stays best-effort | 10.2 |-| 1b.21 | Reset closes teams, team tasks and employee jobs (`close_open_work`, shared with Stop work). `dispatch_job` refuses a job of an ended generation, and the delivery claim refuses a cancelled job. It ships in the same release as 1b.20 | 10.2 |+| 1b.21 | Reset closes teams, team tasks and employee jobs (`close_open_work`, shared with Stop work). `dispatch_job` cancels a job of an ended generation instead of starting it, and a delivery keeps a cancelled job cancelled and sends nothing more. It ships in the same release as 1b.20 | 10.2 || 1b.22 | Start's compensation runs in reverse order, each step in its own try and logged, and fails the row last. A Start whose socket drops finishes on its own | 10.2 || 1b.23 | Boot reconcile finishes a Reset that a crash interrupted | 10.2 || 1b.24 | The pause-on-failure breaker counts each run once | 11.8 |diff --git a/docs-internal/employee_teams.md b/docs-internal/employee_teams.mdindex 41b25061..7aeae62c 100644--- a/docs-internal/employee_teams.md+++ b/docs-internal/employee_teams.md@@ -62,7 +62,9 @@ stopped generation or drain reviews while Stop/held Resume closes admission.It replaces future snapshots without Reset, preserving Context,memory, chats, files, approvals and queued events. Paused employees staypaused. “Stop work and apply” is the explicit interruption path. The editor's-Reset continues to clear execution state as before.+Reset continues to clear execution state, and it also cancels the team's open+tasks and requests, queued ones included, which the terminated runs can no+longer close.Conversion requires an owner review of a recognizable template graph andkeeps a full configuration snapshot. Custom graphs require Dev review.diff --git a/docs-internal/node-logic-flows/workflow/employeeJob.md b/docs-internal/node-logic-flows/workflow/employeeJob.mdindex ade35dde..6976be6d 100644--- a/docs-internal/node-logic-flows/workflow/employeeJob.md+++ b/docs-internal/node-logic-flows/workflow/employeeJob.md@@ -21,7 +21,13 @@ Operations:that refuses cleanly ends the job failed, and a failed job is neverdelivered again. In a v1 graph `deliver` runs as the node's own activityunder the plugin's default policy, 3 attempts of up to 24 h each, so it- re-attaches only within those.+ re-attaches only within those. A job that a Reset or "Stop work and apply"+ cancels sends nothing more, even when its approval wait ends afterwards.++A Reset cancels the workflow's open jobs, queued ones included, and a queued+job of a generation that has ended (reset, failed or gone) is cancelled+instead of started. A manual canvas run's job has no generation and is never+refused this way.`operation`, `lead_node_id` and `delivery_node_ids` are server-controlledconfiguration. Models can supply only the mission when this node is a tool.diff --git a/docs-internal/normal_mode.md b/docs-internal/normal_mode.mdindex 8905c196..45f3e2df 100644--- a/docs-internal/normal_mode.md+++ b/docs-internal/normal_mode.md@@ -850,8 +850,10 @@ the workflow's conversations and the chat nodes (`chatTrigger`, `chatReply`)clear its thread through the same `reset_execution_state` hook, so Talk andthe editor's chat pane never show a conversation the agent no longer has.It also drops messages queued while paused and cancels the drafts waiting-for the owner. Deleting the workflow deletes its thread (a workflow-deleted-hook `chatReply` registers).+for the owner. It cancels the team's open tasks and requests too, so a later+Apply has nothing to wait on, and a queued request of the ended generation is+cancelled instead of started. Deleting the workflow deletes its thread (a+workflow-deleted hook `chatReply` registers).`pending_changes` is true while a generation is live (starting, running,pausing, paused, resuming) and the saved graph, put through Start's owndiff --git a/docs-internal/temporal-workflow-control.md b/docs-internal/temporal-workflow-control.mdindex 6cfb4525..d3e82d0d 100644--- a/docs-internal/temporal-workflow-control.md+++ b/docs-internal/temporal-workflow-control.md@@ -29,8 +29,9 @@ nested Coding Agents. Native histories retain their recorded execution path.nodes.- **Reset** revision-guards the old generation, closes controller and localadmission, removes Temporal cron schedules, cancels local compatibility- resources, and then performs a final strict execution sweep and resets the- workflow's [Workspace tasks](#workspace-tasks) before archiving the old+ resources, and then performs a final strict execution sweep, cancels the+ workflow's open team work and employee requests, and resets the workflow's+ [Workspace tasks](#workspace-tasks) before archiving the oldgeneration. It leaves the control state `ready`; the user must press**Start** to create the next generation.@@ -138,6 +139,18 @@ archiving the generation. Browser operations and shared profile startup aresettled before releasing ownership. Failed cleanup leaves `resetting` for retry;claims belonging to another generation or direct Workspace tasks are preserved.+Generation Reset then closes the workflow's open team work in the database+(`services/employees/safe_apply.close_open_work`, which "Stop work and apply"+also uses): it dissolves the active teams, cancels their pending, blocked,+queued and running tasks, and cancels the employee requests that are queued or+in progress. The terminated runs close none of these records themselves, and+Apply would wait on them for ever. Teams carry no generation, so the close is+workflow-wide. A request whose send has an unknown outcome+(`delivery_needs_review`) is kept for the owner, and so is a request's in-flight+send mark. A queued request of a generation that has ended is cancelled instead+of started (`services/employees/jobs.dispatch_job`), and a request closed while+its delivery waits for approval sends nothing more.+### Stop and Resume acknowledgement sequenceThe orchestration lives in@@ -470,7 +483,8 @@ state. Every successful **Start** atomically creates athe generation data scope is archived, and node/runtime state has been reset.If any cleanup step fails, the control remains `resetting`; retrying Resetresumes cleanup instead of falsely completing or skipping it. Reset does not-delete outputs, tasks, traces, or the graph snapshot. The toolbar returns to+delete outputs, tasks, traces, or the graph snapshot; open team tasks and+requests are cancelled, not deleted. The toolbar returns to**Start**. The next Start creates a new generation, execution ID, Temporalcontroller, and empty runtime namespace from the then-current saved canvas.Historical scopes therefore remain queryable without leaking state into thediff --git a/server/services/deployment/control.py b/server/services/deployment/control.pyindex d165b967..d6d6bf59 100644--- a/server/services/deployment/control.py+++ b/server/services/deployment/control.py@@ -18,6 +18,11 @@ logger = get_logger(__name__)# lives beside the model so the terminate-sweep guard shares it.ACTIVE_STATES = WORKFLOW_CONTROL_ACTIVE_STATES+# A generation in one of these states still lives: work admitted under it may+# start. Not `resetting`, which ACTIVE_STATES includes: a generation being reset+# admits nothing more.+LIVE_STATES = frozenset({"starting", "running", "pausing", "paused", "resuming"})+def _graph_hash(nodes: list[dict], edges: list[dict]) -> str:value = json.dumps({"nodes": nodes, "edges": edges}, sort_keys=True, separators=(",", ":"), default=str)diff --git a/server/services/deployment/handlers.py b/server/services/deployment/handlers.pyindex aa8d3df7..0412595a 100644--- a/server/services/deployment/handlers.py+++ b/server/services/deployment/handlers.py@@ -2643,6 +2643,12 @@ async def _reset_workflow(data: Dict[str, Any], websocket: WebSocket) -> Dict[stcleaned_browser_tasks = await cleanup_generation_browser_tasks(service.database, wrapper.client if wrapper is not None else None, current,)+ # A terminated run closes none of its own records: close the generation's+ # team work and open requests here, or the next Apply waits on them. Late,+ # so a delivery activity the sweep stopped has had time to see its cancel.+ from services.employees.safe_apply import OPEN_JOB_STATES, close_open_work+ await close_open_work(service.database, workflow_id, reason="The workflow was reset",+ job_states=(*OPEN_JOB_STATES, "queued"))workspace = await _reset_workspace_tasks(workflow_id, service.database, current, data)archived = await service.database.update_workflow_run_data_scope(diff --git a/server/services/employees/jobs.py b/server/services/employees/jobs.pyindex 722c97b5..5e01b75e 100644--- a/server/services/employees/jobs.py+++ b/server/services/employees/jobs.py@@ -205,8 +205,22 @@ async def bind_job_team(database: Any, *, workflow_id: str, lead_node_id: str, eawait session.commit()+async def _generation_ended(database: Any, job: EmployeeJob) -> bool:+ """Whether the controlled generation a job came from no longer lives (reset,+ resetting, failed, or none). A manual run's job (generation 0) has no+ generation to end."""+ generation = int(job.source.get("generation") or 0)+ if generation <= 0:+ return False+ from services.deployment.control import LIVE_STATES+ control = await database.get_workflow_control_generation(job.workflow_id, generation)+ return control is None or control.status not in LIVE_STATES++async def dispatch_job(database: Any, job_id: str) -> bool:- """Start a stable Temporal identity. Restart/retry reattaches to that run."""+ """Start a stable Temporal identity. Restart/retry reattaches to that run.+ A job whose generation was reset or replaced is cancelled instead: the+ controller it would run under is gone."""from core.container import containerfrom services.employees.team_runtime import team_runtime_errorif team_runtime_error():@@ -221,6 +235,16 @@ async def dispatch_job(database: Any, job_id: str) -> bool:return Falseif await database.get_workflow(job.workflow_id) is None:return False+ if await _generation_ended(database, job):+ # Cancelled, or the recovery loop would retry it for ever.+ async with database.reserved_session() as session:+ row = await session.get(EmployeeJob, job.id)+ if row and row.state == "queued":+ row.state, row.updated_at = "cancelled", _now()+ await session.commit()+ from services.employees.events import employee_changed_now+ employee_changed_now(job.workflow_id)+ return Falsefrom temporalio.exceptions import WorkflowAlreadyStartedErrorfrom temporalio.common import WorkflowIDReusePolicyfrom services.temporal.trigger_listener_workflow import event_workflow_search_attributes@@ -354,11 +378,18 @@ async def deliver_job(database: Any, ctx: Any, *, lead_node_id: str) -> dict:async with database.reserved_session() as session:current = await session.get(EmployeeJob, job.id)current.delivery = {**current.delivery, "completed": completed}- current.state = "delivered" if len(completed) == len(pending) else "delivering"+ cancelled = current.state == "cancelled" # a Reset or an Apply ended it meanwhile+ if not cancelled:+ current.state = "delivered" if len(completed) == len(pending) else "delivering"await session.commit()+ if cancelled:+ return {"delivered": False, "state": "cancelled", "job_id": job.id}continueasync with database.reserved_session() as session:current = await session.get(EmployeeJob, job.id)+ if current.state == "cancelled":+ # A Reset or an Apply ended this job: send nothing more.+ return {"delivered": False, "state": "cancelled", "job_id": job.id}# Only one delivery invocation claims this sink.if current.delivery.get("claim_token") != claim or current.delivery.get("inflight") or node_id in current.delivery.get("completed", []):return {"delivered": False, "state": "delivery_needs_review", "job_id": job.id}@@ -419,11 +450,15 @@ async def deliver_job(database: Any, ctx: Any, *, lead_node_id: str) -> dict:raise RuntimeError("The delivery claim changed while sending the result")current.delivery = {**current.delivery, "inflight": None, "completed": completed,"outputs": {**current.delivery.get("outputs", {}), node_id: restored[node_id]}}- current.state = "delivered" if len(completed) == len(pending) else "delivering"+ cancelled = current.state == "cancelled" # a Reset or an Apply ended it meanwhile+ if not cancelled:+ current.state = "delivered" if len(completed) == len(pending) else "delivering"if current.state == "delivered":current.delivery = {**current.delivery, "claim_token": None, "lease_until": None}current.updated_at = _now()await session.commit()+ if cancelled:+ return {"delivered": False, "state": "cancelled", "job_id": job.id}return {"delivered": True, "state": "delivered", "job_id": job.id}@@ -437,7 +472,8 @@ async def job_progress(database: Any, workflow_id: str) -> dict | None:lease = job.delivery.get("lease_until")expired = not lease or datetime.fromisoformat(lease) <= _now()state = "delivery_needs_review" if inflight and expired and node_types.get(inflight) not in {"approvalGate", "chatReply"} else job.state- if inflight and node_types.get(inflight) == "approvalGate":+ # A terminal job answers its own state even with a gate mark left behind.+ if inflight and node_types.get(inflight) == "approvalGate" and job.state not in {"failed", "cancelled"}:return {"request_id": job.id, "state": "waiting_for_approval", "message": "Their result is ready for you to check before sending."}messages = {"queued": "Your request is saved. The team will start when they are ready.","running": "The team is starting your request.", "working": "The team is working on your request and checking the result.",diff --git a/server/services/employees/safe_apply.py b/server/services/employees/safe_apply.pyindex de4f152a..5060d1ea 100644--- a/server/services/employees/safe_apply.py+++ b/server/services/employees/safe_apply.py@@ -23,10 +23,36 @@ def _send(handle, name: str, argument, update_id: str):return bounded_update(handle.execute_update(name, argument, id=update_id), UPDATE_SECONDS)+#: Jobs in these states are work in flight: Apply waits for them, and Stop work+#: and Reset close them.+OPEN_JOB_STATES = ("running", "working", "delivering", "delivery_queued")+++async def close_open_work(database, workflow_id, *, reason: str, job_states=OPEN_JOB_STATES) -> None:+ """Cancel the workflow's open team tasks and jobs, and dissolve its active teams.++ A run that is stopped or terminated closes none of its own records, and+ Apply waits on them. Rows in a final state are kept, and so is a job's+ in-flight send mark, for the owner to resolve."""+ async with database.get_session() as session:+ teams = (await session.execute(select(AgentTeam).where(AgentTeam.workflow_id == workflow_id, AgentTeam.status == "active"))).scalars().all()+ for team in teams:+ for task in await database.get_team_tasks(team.id):+ if task.get("status") in {"pending", "blocked", "queued", "running"}:+ await database.cancel_team_task(team.id, task["id"], reason)+ await database.update_team_status(team.id, "dissolved")+ async with database.reserved_session() as session:+ jobs = (await session.execute(select(EmployeeJob).where(EmployeeJob.workflow_id == workflow_id, EmployeeJob.state.in_(job_states)))).scalars().all()+ for job in jobs:+ job.state = "cancelled"+ job.updated_at = datetime.now(timezone.utc)+ await session.commit()++async def _active(database, workflow_id):async with database.get_session() as session:team = (await session.execute(select(AgentTeam.id).where(AgentTeam.workflow_id == workflow_id, AgentTeam.status == "active"))).first()- job = (await session.execute(select(EmployeeJob.id).where(EmployeeJob.workflow_id == workflow_id, EmployeeJob.state.in_(["running", "working", "delivering", "delivery_queued"])))).first()+ job = (await session.execute(select(EmployeeJob.id).where(EmployeeJob.workflow_id == workflow_id, EmployeeJob.state.in_(OPEN_JOB_STATES)))).first()from core.container import containerfrom services.chat.ledger import live_runsif team or job or await live_runs(database, workflow_id):@@ -147,19 +173,8 @@ async def _apply_saved_changes(database: Any, workflow_id: str, *, owner_id: strtask.cancel()# Durable work records must reflect the explicitly requested stop;# otherwise recovery would wait forever for a cancelled execution.- async with database.get_session() as session:- teams = (await session.execute(select(AgentTeam).where(AgentTeam.workflow_id == workflow_id, AgentTeam.status == "active"))).scalars().all()- for team in teams:- for task in await database.get_team_tasks(team.id):- if task.get("status") in {"pending", "blocked", "queued", "running"}:- await database.cancel_team_task(team.id, task["id"], "The owner stopped work to apply new abilities")- await database.update_team_status(team.id, "dissolved")- async with database.reserved_session() as session:- jobs = (await session.execute(select(EmployeeJob).where(EmployeeJob.workflow_id == workflow_id, EmployeeJob.state.in_(["running", "working", "delivering", "delivery_queued"])))).scalars().all()- for job in jobs:- job.state = "cancelled"- job.updated_at = datetime.now(timezone.utc)- await session.commit()+ # Queued requests are kept: they start after the apply.+ await close_open_work(database, workflow_id, reason="The owner stopped work to apply new abilities")if await _active(database, workflow_id):return {"success": True, "activation_state": "waiting", "message": "Finishing current work before applying the new abilities."}claim = uuid.uuid4().hexdiff --git a/server/tests/services/employees/test_jobs.py b/server/tests/services/employees/test_jobs.pyindex 55e7607a..b6a2c682 100644--- a/server/tests/services/employees/test_jobs.py+++ b/server/tests/services/employees/test_jobs.py@@ -7,7 +7,7 @@ from datetime import datetime, timedelta, timezoneimport pytestfrom models.employees import EmployeeJobfrom models.database import TeamTask-from services.employees.jobs import deliver_job, bind_job_team+from services.employees.jobs import deliver_job, bind_job_team, dispatch_job, job_progressfrom services.plugin import NodeContext@pytest.fixture@@ -228,3 +228,71 @@ async def test_a_refusal_keeps_a_resets_cancelled(database, runtime):runtime.execute.side_effect = refuse_after_resetassert (await deliver_job(database, context(), lead_node_id="lead"))["state"] == "cancelled"assert (await saved(database)).state == "cancelled"+++# 1b.21: a job that a Reset or an Apply closed stays closed.+async def dispatcher(database, monkeypatch):+ start = AsyncMock()+ monkeypatch.setattr("core.container.container", SimpleNamespace(+ temporal_client=lambda: SimpleNamespace(client=SimpleNamespace(start_workflow=start)),+ settings=lambda: SimpleNamespace(temporal_task_queue="employees")))+ monkeypatch.setattr("services.employees.team_runtime.team_runtime_error", lambda **_kwargs: None)+ monkeypatch.setattr("services.employees.events.employee_changed_now", lambda _workflow_id: None)+ monkeypatch.setattr(database, "get_workflow", AsyncMock(return_value=SimpleNamespace(data={})))+ return start++async def queued(database, *, generation=4, status=None):+ from models.database import WorkflowControlExecution+ source = {"nodes": [{"id": "lead", "type": "ai_employee"}], "edges": [], "parameters": {"lead": {}},+ "session_id": "7", "user_id": "owner", "generation": generation}+ async with database.reserved_session() as session:+ session.add(EmployeeJob(id="job", workflow_id="7", origin_execution_id="origin", lead_node_id="lead", source=source, mission="Mission"))+ if status:+ session.add(WorkflowControlExecution(id=f"workflow-control:7:{generation}", workflow_id="7", generation=generation,+ execution_id="origin", root_execution_id="origin", status=status, idempotency_key="start", graph_hash="admitted"))+ await session.commit()+[email protected]("status", ["reset", "resetting", "failed", None])+async def test_a_job_of_an_ended_generation_is_cancelled_not_started(database, dispatcher, status):+ await queued(database, status=status) # None: the generation has no row+ assert await dispatch_job(database, "job") is False+ assert (await saved(database)).state == "cancelled"+ dispatcher.assert_not_awaited()+[email protected]("generation, status", [(4, "running"), (4, "paused"), (0, None)])+async def test_a_job_of_a_live_generation_or_a_manual_run_starts(database, dispatcher, generation, status):+ await queued(database, generation=generation, status=status)+ assert await dispatch_job(database, "job") is True+ dispatcher.assert_awaited_once()+ assert dispatcher.await_args.args[0] == "EmployeeJobWorkflow"+ assert (await saved(database)).state == "running"+[email protected]("approved", [True, False])+async def test_a_job_closed_during_its_gate_wait_sends_nothing_more(database, runtime, approved):+ await install(database)++ async def close_during_the_wait(**_kwargs):+ async with database.reserved_session() as session:+ (await session.get(EmployeeJob, "job")).state = "cancelled"+ await session.commit()+ return {"success": True, "result": {"approved": approved}}++ runtime.execute.side_effect = close_during_the_wait+ assert await deliver_job(database, context(), lead_node_id="lead") == {"delivered": False, "state": "cancelled", "job_id": "job"}+ assert runtime.execute.await_count == 1+ row = await saved(database)+ assert row.state == "cancelled" and row.delivery["completed"] == ["gate"]+[email protected]("state, inflight, message", [+ ("cancelled", "gate", "This request was stopped."),+ ("delivering", "gate", "Their result is ready for you to check before sending."),+ # An external send's mark stays for the owner to resolve.+ ("cancelled", "send", "The send was interrupted. Check whether it arrived before trying again."),+])+async def test_progress_shows_the_gate_line_only_while_the_job_is_open(database, state, inflight, message):+ await install(database, inflight=inflight) # its lease has expired+ async with database.reserved_session() as session:+ (await session.get(EmployeeJob, "job")).state = state+ await session.commit()+ assert (await job_progress(database, "7"))["message"] == messagediff --git a/server/tests/services/employees/test_safe_apply.py b/server/tests/services/employees/test_safe_apply.pyindex 7a44576f..1e5c8ed6 100644--- a/server/tests/services/employees/test_safe_apply.py+++ b/server/tests/services/employees/test_safe_apply.py@@ -297,6 +297,54 @@ async def test_live_claim_serializes_handoffs_and_renews_lease(harness, monkeypaawait first+async def test_close_open_work_leaves_nothing_for_apply_to_wait_on(harness, monkeypatch):+ # 1b.21: Reset closes the generation's open work as Stop work does, and its+ # queued requests too.+ import services.chat.ledger as ledger+ monkeypatch.setattr(ledger, "live_runs", AsyncMock(return_value=[]))+ harness.container.workflow_service().get_deployment_status = lambda _id: {"active_runs": 0}+ harness.client.list_workflows = lambda **_kwargs: EmptyAsync()+ closed_tasks, kept_tasks = ("pending", "blocked", "queued", "running"), ("submitted", "accepted", "failed", "cancelled")+ open_jobs, kept_jobs = ("running", "working", "delivering", "delivery_queued", "queued"), ("delivered", "failed", "cancelled", "delivery_needs_review")+ async with harness.database.get_session() as session:+ session.add(AgentTeam(id="team", workflow_id="7", team_lead_node_id="7:aiAgent:1"))+ session.add(AgentTeam(id="done-team", workflow_id="7", team_lead_node_id="7:aiAgent:1", status="completed"))+ for status in (*closed_tasks, *kept_tasks):+ session.add(TeamTask(id=f"task-{status}", team_id="team", title="Research", created_by="7:aiAgent:1", workflow_id="7", status=status))+ for state in (*open_jobs, *kept_jobs):+ delivery = {"inflight": "send"} if state == "delivering" else {}+ session.add(EmployeeJob(id=f"job-{state}", workflow_id="7", origin_execution_id=f"origin-{state}", lead_node_id="7:aiAgent:1",+ team_id="team", state=state, delivery=delivery))+ await session.commit()+ assert await REAL_ACTIVE(harness.database, "7")+ for _ in range(2): # idempotent: Reset may run again+ await safe_apply.close_open_work(harness.database, "7", reason="The workflow was reset",+ job_states=(*safe_apply.OPEN_JOB_STATES, "queued"))+ assert not await REAL_ACTIVE(harness.database, "7")+ async with harness.database.get_session() as session:+ assert (await session.get(AgentTeam, "team")).status == "dissolved"+ assert (await session.get(AgentTeam, "done-team")).status == "completed"+ for status in closed_tasks:+ assert (await session.get(TeamTask, f"task-{status}")).status == "cancelled"+ for status in kept_tasks:+ assert (await session.get(TeamTask, f"task-{status}")).status == status+ for state in open_jobs:+ assert (await session.get(EmployeeJob, f"job-{state}")).state == "cancelled"+ for state in kept_jobs:+ assert (await session.get(EmployeeJob, f"job-{state}")).state == state+ # The in-flight send mark is kept for the owner to resolve.+ assert (await session.get(EmployeeJob, "job-delivering")).delivery["inflight"] == "send"+++async def test_stop_work_keeps_queued_requests(harness):+ async with harness.database.get_session() as session:+ session.add(EmployeeJob(id="job-queued", workflow_id="7", origin_execution_id="origin", lead_node_id="7:aiAgent:1", state="queued"))+ await session.commit()+ await safe_apply.close_open_work(harness.database, "7", reason="The owner stopped work to apply new abilities")+ async with harness.database.get_session() as session:+ assert (await session.get(EmployeeJob, "job-queued")).state == "queued"++def _hang_on(harness, monkeypatch, hangs):"""1b.15: the controller never answers the Updates ``hangs`` picks."""import services.deployment.handlers as handlersdiff --git a/server/tests/services/test_pi_task_manager_dispatch.py b/server/tests/services/test_pi_task_manager_dispatch.pyindex 7eea25e5..a3a93d25 100644--- a/server/tests/services/test_pi_task_manager_dispatch.py+++ b/server/tests/services/test_pi_task_manager_dispatch.py@@ -51,7 +51,8 @@ async def runtime(monkeypatch):{"id": "talk", "type": "chatAgent"}, {"id": "chat", "type": "chatReply"}]edges = [{"source": "coding", "target": "parent", "targetHandle": "input-teammates"}]parameters = {node["id"]: {"model": "admitted-model"} for node in nodes}- admitted = SimpleNamespace(graph_snapshot={"nodes": nodes, "edges": edges, "parameters": parameters})+ # A live generation: dispatch_job cancels a job of an ended one (1b.21).+ admitted = SimpleNamespace(graph_snapshot={"nodes": nodes, "edges": edges, "parameters": parameters}, status="running")database = SimpleNamespace(get_workflow_control_generation=AsyncMock(return_value=admitted),get_session=session, reserved_session=session,diff --git a/server/tests/services/test_versioned_workflow_control.py b/server/tests/services/test_versioned_workflow_control.pyindex 92275bdd..3e65bacb 100644--- a/server/tests/services/test_versioned_workflow_control.py+++ b/server/tests/services/test_versioned_workflow_control.py@@ -160,6 +160,7 @@ def lifecycle(monkeypatch, state="running"):monkeypatch.setattr(handlers, "_signal_controller", AsyncMock())monkeypatch.setattr(handlers, "_terminate_generation_workflows", AsyncMock(return_value=3))monkeypatch.setattr("services.approvals.waiting.cancel_waits", AsyncMock(return_value=0))+ monkeypatch.setattr("services.employees.safe_apply.close_open_work", AsyncMock(return_value=None))monkeypatch.setattr(handlers, "_reset_workspace_tasks", AsyncMock(return_value={"cancelled": 0}))monkeypatch.setattr(handlers, "handle_cancel_deployment", AsyncMock(return_value={"success": True}))monkeypatch.setattr(handlers, "_clear_failure_streak", AsyncMock())@@ -324,6 +325,22 @@ async def test_browser_cleanup_failure_keeps_resetting_and_scope_unarchived(monkrecord.service.database.update_workflow_run_data_scope.assert_not_awaited()+async def test_reset_closes_open_work_and_a_failed_close_keeps_resetting(monkeypatch):+ # 1b.21: a terminated run closes none of its own records, so Reset does.+ record = lifecycle(monkeypatch)+ close = AsyncMock(side_effect=RuntimeError("database is locked"))+ monkeypatch.setattr("services.employees.safe_apply.close_open_work", close)+ result = await handlers.handle_reset_workflow({"workflow_id": "canvas", "expected_revision": 2}, None)+ assert result == {"success": False, "error": "database is locked"}+ assert record.row.status == "resetting"+ assert close.await_args.args == (record.service.database, "canvas")+ assert close.await_args.kwargs["reason"] == "The workflow was reset"+ assert "queued" in close.await_args.kwargs["job_states"]+ handlers._reset_workspace_tasks.assert_not_awaited()+ record.service.database.update_workflow_run_data_scope.assert_not_awaited()++@pytest.mark.asyncio@pytest.mark.parametrize("state", ["running", "reset"])async def test_stale_reset_generation_cannot_cleanup_current_generation(monkeypatch, state):diff --git a/server/tests/services/test_workflow_control_handlers.py b/server/tests/services/test_workflow_control_handlers.pyindex 1168bdb4..4f8a8150 100644--- a/server/tests/services/test_workflow_control_handlers.py+++ b/server/tests/services/test_workflow_control_handlers.py@@ -1072,6 +1072,9 @@ async def test_reset_retry_resumes_cleanup_before_marking_ready(monkeypatch):# The calls the reset turns waited on end cancelled (RFC-0007 §4.6).cancel_waits = AsyncMock(return_value=0)monkeypatch.setattr("services.approvals.waiting.cancel_waits", cancel_waits)+ # The generation's open team work and requests are closed (1b.21).+ close_open_work = AsyncMock(return_value=None)+ monkeypatch.setattr("services.employees.safe_apply.close_open_work", close_open_work)monkeypatch.setattr(handlers,"_delete_cron_schedules",@@ -1103,6 +1106,10 @@ async def test_reset_retry_resumes_cleanup_before_marking_ready(monkeypatch):assert result["revision"] == 4cancel_waits.assert_awaited_once()assert cancel_waits.await_args.args[1:] == ("wf", resetting.generation)+ close_open_work.assert_awaited_once()+ assert close_open_work.await_args.args[1:] == ("wf",)+ assert close_open_work.await_args.kwargs["reason"] == "The workflow was reset"+ assert "queued" in close_open_work.await_args.kwargs["job_states"]assert result["terminated_executions"] == 2assert result["deleted_schedules"] == 3assert result["local_cleanup_completed"] is Falsediff --git a/server/tests/services/test_workspace_task_reset.py b/server/tests/services/test_workspace_task_reset.pyindex 0045cf94..299c6b5a 100644--- a/server/tests/services/test_workspace_task_reset.py+++ b/server/tests/services/test_workspace_task_reset.py@@ -122,6 +122,7 @@ async def test_a_pending_workspace_cleanup_keeps_a_generation_resetting(monkeypamonkeypatch.setattr(handlers, "handle_cancel_deployment", AsyncMock(return_value={"success": True}))monkeypatch.setattr(handlers, "_terminate_generation_workflows", AsyncMock(return_value=0))monkeypatch.setattr("services.approvals.waiting.cancel_waits", AsyncMock(return_value=0))+ monkeypatch.setattr("services.employees.safe_apply.close_open_work", AsyncMock(return_value=None))monkeypatch.setattr(handlers, "_reset_workspace_tasks", AsyncMock(side_effect=UpdatePending("no answer in 240 s")))monkeypatch.setattr(handlers, "_control_payload", AsyncMock(side_effect=lambda row, **_: {"state": row.status}))monkeypatch.setattr(runtime_state, "archive_and_reset_node_state", archive)@@ -203,6 +204,11 @@ async def test_generation_reset_waits_for_workspace_before_archive_and_final_resmonkeypatch.setattr(handlers, "handle_cancel_deployment", AsyncMock(return_value={"success": True}))monkeypatch.setattr(handlers, "_terminate_generation_workflows", AsyncMock(side_effect=terminate))monkeypatch.setattr("services.approvals.waiting.cancel_waits", AsyncMock(return_value=0))++ async def close_open_work(*_args, **_kwargs):+ order.append("close:work")++ monkeypatch.setattr("services.employees.safe_apply.close_open_work", close_open_work)monkeypatch.setattr(handlers, "_reset_workspace_tasks", workspace)monkeypatch.setattr(runtime_state, "archive_and_reset_node_state", archive)notices = broadcaster(monkeypatch, order)@@ -227,7 +233,7 @@ async def test_generation_reset_waits_for_workspace_before_archive_and_final_resassert result["success"] is True and result["state"] == "ready"assert order[-7:] == ["workspace:start", "workspace:done", "archive:scope", "archive:nodes","db:reset", "workflow_runtime_reset", "control:reset"]- assert order.index("terminate:graph") < order.index("workspace:start")+ assert order.index("terminate:graph") < order.index("close:work") < order.index("workspace:start")assert notices.broadcast.await_args.args[0]["cancelled_workspace_tasks"] == 2