Enterprise agent platform for building, deploying, and operating AI teammates.
feat(v1): cancel a task that is waiting for an answer or paused (#3795)
Alex Liu4581f3d8917d
diff --git a/docs/deployment.md b/docs/deployment.mdindex 8eb867f0c..3429bae17 100644--- a/docs/deployment.md+++ b/docs/deployment.md@@ -1034,3 +1034,30 @@ A running task created through `/v1` can now be cancelled. The cancelled task re### Rollback- A queued turn answers 409 `task_busy` again. Turns already withdrawn stay withdrawn.++## 2026-11-13 — SDK tasks: cancelling a task that is waiting for an answer or paused++`POST /v1/chat/tasks/{task_id}/cancel` now also cancels a task that is waiting for an answer or paused, instead of answering 409 `task_busy`.++### Client-visible changes++- A task waiting for an answer answers 200 and reads `failed` with `error_code` `task_cancelled`. Its question is closed: `pending_interaction` reads `null`, and a reply answers 409 `no_pending_interaction`.+- A paused task answers 200 and reads `failed` with `error_code` `task_cancelled`; automatic resume no longer picks it up.+- While an answer is queued or being applied, the call still answers 409 `task_busy`; calling again once the answered run is running cancels it as a running task.+- A paused task that reads `failed`, because automatic resume gave up or because the turn appended to it was withdrawn or could not be started, still answers 409 `task_not_cancellable`.+- Calling again after this endpoint withdrew a turn appended to a paused task answers the withdrawal's 200 while the task has not moved since, including while that cancel is still being completed, whether the task reads `paused` or `failed`; it does not cancel the paused task.+- A paused task whose automatic resume gives up while the call waits answers 202 when the cancel has not been applied by the deadline and the task has not moved since, because the cancel still applies to it. Calling again reaches the same cancel, answering 202 until it is applied and 200 once it has.+- A queued turn that started and then paused, and whose automatic resume gives up while the call waits, may answer 409 `task_not_cancellable` at the deadline although the cancel can still be applied to it afterwards.+- When the task has moved since a turn was queued, so that turn can no longer start, a task that is now waiting for an answer or paused is cancelled as such, as a running one is in the previous entry.+- A cancel that reaches a queued turn after it started cancels it the way it is by then: running, waiting for an answer, or paused.+- The two earlier entries listed waiting for an answer and paused among the `task_busy` cases; they are no longer.++### Deployment impact++- No migration and no configuration.+- Mixed versions: a process that predates this change refuses a waiting or paused task with 409 `task_busy` before recording a command. When such a process applies a cancel command recorded by an upgraded one, it rejects it as a changed run without writing anything, and the endpoint answers 409 `task_busy`. A process that predates the cancel endpoint answers 503, as in the first entry. Calling again succeeds once every process that executes tasks has been upgraded.+- This is the change the first entry named before the Python SDK `tasks.cancel()` ships.++### Rollback++- Waiting and paused tasks answer 409 `task_busy` again. Tasks already cancelled stay `failed`, with their question closed and their automatic resume stopped. A cancel command still recorded but not yet applied is rejected by the older process without a write.diff --git a/src/xagent/web/api/v1/tasks.py b/src/xagent/web/api/v1/tasks.pyindex 991f48430..29894f034 100644--- a/src/xagent/web/api/v1/tasks.py+++ b/src/xagent/web/api/v1/tasks.py@@ -1517,7 +1517,7 @@ async def cancel_chat_task(response: Response,principal: ApiKeyPrincipal = Depends(get_principal_from_api_key),) -> CancelTaskResponse:- """Stop a task that is still running.+ """Stop a task that has not ended yet.Reuses ``GET``'s ownership rule (another agent's task, another user'sagent, a missing id, or a non-SDK task all answer 404) before anything@@ -1534,19 +1534,30 @@ async def cancel_chat_task(cancelled one; while a later turn is queued after a completed or failedrun the task reads ``pending`` with a null ``error_code``, as ``GET``reads it. 202 means the cancel is recorded but not applied yet, and- calling again reaches the same request.+ calling again reaches the same request, including after it withdrew a+ queued turn and before it is marked done. A paused task whose+ automatic resume gives up while the call waits is answered 202 while+ its cancel is still recorded and the task has not moved since, since+ the cancel still applies to it; calling again reaches that cancel too,+ and answers 200 once it is applied.A turn still queued to start is cancelled too. For a new task thatleaves the task cancelled with no ``run_id``. For a turn appended to afinished or paused task it only withdraws that turn: the task keepsreading as the turn before it, and ``cancelled_run_id`` names the- withdrawn run. A queued turn that can no longer start, because the+ withdrawn run; calling again answers the same while the task has not+ moved since. A queued turn that can no longer start, because thetask has moved since it was queued (another run, or a newer version of- the same one), is not the target: a running run is cancelled, and any- other state is answered as if no turn were queued.-- A task that is waiting for an answer or paused, with no turn queued to- start, answers 409 ``task_busy`` and is left untouched.+ the same one), is not the target: the task is cancelled, or answered,+ as it is now, as if no turn were queued.++ A task waiting for an answer is cancelled with its question closed,+ unlike ``cancel-question``, which leaves the task ``completed``. A+ paused task is cancelled and no longer resumed automatically; one that+ reads ``failed`` -- automatic resume gave up, or its appended turn was+ withdrawn or could not start -- has already ended. While an answer is+ being applied the call answers 409 ``task_busy``; the resumed run can+ then be cancelled.Raises:V1ApiError 401: missing / invalid / revoked key.diff --git a/src/xagent/web/schemas/v1.py b/src/xagent/web/schemas/v1.pyindex 45daf489b..6f6228019 100644--- a/src/xagent/web/schemas/v1.py+++ b/src/xagent/web/schemas/v1.py@@ -1251,10 +1251,16 @@ class CancelTaskResponse(BaseModel):it reads ``pending`` with a null ``error_code``, as``GET /v1/chat/tasks/{task_id}`` does. A turnwithdrawn before it started leaves the task showing the turn before- it, so ``cancelled_run_id`` differs from ``run_id``.+ it, so ``cancelled_run_id`` differs from ``run_id``; calling again+ answers the same while the task has not moved since.202: the cancel was recorded but not applied yet; ``status`` is still- the task's current one and ``cancelled_run_id`` is null. Calling again- reaches the same request.+ the task's current one, as ``GET`` reads it, and ``cancelled_run_id``+ is null. Calling again reaches the same request, including after a+ queued turn was withdrawn and before the cancel is marked done. A+ paused task whose automatic resume gave up while the cancel waited,+ and has not moved since, reads ``failed`` here and is still cancelled+ by it; calling again reaches the same cancel and answers 200 once it+ is applied."""task_id: int = Field(..., description="Existing task primary key.")diff --git a/src/xagent/web/services/sdk_task_cancel.py b/src/xagent/web/services/sdk_task_cancel.pyindex 8d54dd9b3..a4a93668a 100644--- a/src/xagent/web/services/sdk_task_cancel.py+++ b/src/xagent/web/services/sdk_task_cancel.py@@ -27,9 +27,10 @@ as it stands. A repeat of a cancel that withdrew the newest turn rejoinsthat cancel's command and is answered for that turn(``_cancel_of_newest_turn``), since the row no longer shows it.-Besides a queued turn only a RUNNING task is cancelled here. One waiting-for the caller's answer and a paused one are refused as busy, without a-write.+A task waiting for the caller's answer and a paused one have no run+coroutine to stop: the core writes their terminal row itself, after closing+the pending question or stopping the automatic resume the paused run was+waiting for."""from __future__ import annotations@@ -40,7 +41,7 @@ from dataclasses import dataclassfrom time import monotonicfrom typing import TYPE_CHECKING, Any-from sqlalchemy import func, or_, update+from sqlalchemy import func, or_, select, updatefrom sqlalchemy.orm import Sessionfrom ..models.chat_message import TaskChatMessage@@ -114,6 +115,11 @@ _AIMED_RUN_ENDED_MESSAGE = "The run this cancel was aimed at has already ended."_TERMINAL_STATUSES = frozenset({TaskStatus.COMPLETED, TaskStatus.FAILED})+# The statuses of a run a cancel command can end.+_CANCELLABLE_STATUSES = frozenset(+ {TaskStatus.RUNNING, TaskStatus.WAITING_FOR_USER, TaskStatus.PAUSED}+)+def is_sdk_cancelled(status: object, error_message: object) -> bool:"""Whether a task row is the outcome of an SDK cancellation.@@ -304,7 +310,9 @@ def _cancel_of_newest_turn(db: Session, task: Task) -> TaskExecutionCommand | Noexample the turn before was resumed and runs again -- is answered fromthe row, as any other call would be. A new task is the exception: itscancel wrote the row itself, which then holds no run and reads as- cancelled.+ cancelled. A task paused on request whose appended turn was withdrawn+ therefore keeps answering that withdrawal, and is not itself cancelled,+ until the row moves."""newest = (@@ -354,6 +362,37 @@ def _cancel_of_newest_turn(db: Session, task: Task) -> TaskExecutionCommand | Noreturn cancel+def _unapplied_cancel_of_row(db: Session, task: Task) -> TaskExecutionCommand | None:+ """This endpoint's cancel aimed at the row as it stands, while unapplied.++ Found by the identity ``_enqueue_cancel`` gives a cancel of the run the+ row holds at its current version (``_cancel_command_id`` with no queued+ run), and only while it is pending or processing. A repeat that finds+ it rejoins it instead of judging the row: the row may already read as+ ended -- a paused run whose automatic resume gave up -- while the+ command will still cancel it.+ """++ cancel = (+ db.query(TaskExecutionCommand)+ .filter(+ TaskExecutionCommand.task_id == int(task.id),+ TaskExecutionCommand.command_id+ == _cancel_command_id(int(task.id), int(task.state_version or 0), None),+ TaskExecutionCommand.kind == TaskCommandKind.CANCEL.value,+ TaskExecutionCommand.status.in_((COMMAND_PENDING, COMMAND_PROCESSING)),+ )+ .populate_existing()+ .one_or_none()+ )+ if cancel is None:+ return None+ payload: dict[str, Any] = cancel.payload if isinstance(cancel.payload, dict) else {}+ if payload.get("scope") != SDK_COMMAND_SCOPE:+ return None+ return cancel++def _start_of_run(db: Session, task_id: int, run_id: str) -> TaskExecutionCommand | None:@@ -404,7 +443,9 @@ def _prepare_cancel_sync(repeat of a cancel that already withdrew the newest turn finds no suchSTART and rejoins that cancel's command (``_cancel_of_newest_turn``):the same command to poll while it is still being applied, the same- answer once it completed.+ answer once it completed. A repeat of a cancel of the row that is still+ unapplied at the row's version rejoins it too+ (``_unapplied_cancel_of_row``), before the row is judged as ended."""from .task_start import resolve_sdk_task@@ -450,15 +491,43 @@ def _prepare_cancel_sync()if is_sdk_cancelled(task.status, task.error_message):return _row_result(db, task)+ # A repeat of a cancel still unapplied against the row as it stands+ # rejoins it, even when the row already reads as ended: that command+ # decides, as it did for the first call.+ unapplied = _unapplied_cancel_of_row(db, task)+ if unapplied is not None:+ return _CancelCommand(+ command_db_id=int(unapplied.id),+ aimed_run_id=_task_run_id(task),+ aimed_at_queued_turn=False,+ )if _ended_without_cancel(db, task):raise SdkTaskNotCancellable- if task.status != TaskStatus.RUNNING:+ if task.status not in _CANCELLABLE_STATUSES:raise SdkTaskCancelBusy(f"A task that is {task.status.value} cannot be cancelled yet.")+ if task.status == TaskStatus.WAITING_FOR_USER and _answer_in_flight(db, task):+ # The answer owns the waiting run until it resumes it; the+ # resumed run is then cancellable.+ raise SdkTaskCancelBusy(+ "An answer is being applied to this task; retry once it runs."+ )return _stage_cancel(db, task, queued_run_id=None)+def _answer_in_flight(db: Session, task: Task) -> bool:+ """Whether a reply already claimed the waiting run or is queued to, by+ the reply path's own acceptance rules."""++ from .task_question_cancel import _pending_resume_command_exists+ from .task_resume import _RESUMABLE_CONTROL_STATES++ if task.control_state not in _RESUMABLE_CONTROL_STATES:+ return True+ return bool(db.scalar(select(_pending_resume_command_exists(int(task.id)))))++def _stage_cancel(db: Session, task: Task, *, queued_run_id: str | None) -> _CancelCommand:@@ -548,6 +617,7 @@ def _load_result_sync(aimed_run_id: str | None,aimed_at_queued_turn: bool,command_completed: bool,+ command_db_id: int | None = None,) -> SdkTaskCancelResult:"""Read the outcome once the command is over or overdue.@@ -569,7 +639,13 @@ def _load_result_sync(``SdkTaskNotCancellable`` with ``_AIMED_RUN_ENDED_MESSAGE``.- an aimed run that ended on its own raises ``SdkTaskNotCancellable``exactly as the request would have on arrival, whether or not a later- turn is queued behind it.+ turn is queued behind it. The exception is a paused run that reads+ ``failed`` while the command is still pending or processing (the+ deadline's read, which passes ``command_db_id``) and the row is still+ at the version the command is aimed at: its automatic resume+ gave up during the wait, which does not move that version or stop the+ command from cancelling it, so the call answers that the cancel is+ still being applied.A START waiting for a later turn reads as ``pending`` with no code, anda paused row that has stopped for good reads as ``failed``, as on the@@ -604,7 +680,9 @@ def _load_result_sync(}:return _result(db, task, cancelled=False, queued=queued)raise SdkTaskNotCancellable(_AIMED_RUN_ENDED_MESSAGE)- if _ended_without_cancel(db, task):+ if _ended_without_cancel(db, task) and not _still_cancels_paused_row(+ db, task, command_db_id+ ):raise SdkTaskNotCancellablereturn _result(db,@@ -615,6 +693,24 @@ def _load_result_sync()+def _still_cancels_paused_row(+ db: Session, task: Task, command_db_id: int | None+) -> bool:+ """Whether the unapplied command ``command_db_id`` still cancels this+ paused row: it is the pending or processing cancel aimed at the version+ the row is at now, the same command a repeat call rejoins.++ The command is matched by its identity, which carries the version its+ executor fences on. The command row's own version column is read when+ the command is staged and can already be past the version the request+ aimed at, so it does not say what the command will cancel."""++ if command_db_id is None or task.status != TaskStatus.PAUSED:+ return False+ cancel = _unapplied_cancel_of_row(db, task)+ return cancel is not None and int(cancel.id) == command_db_id++async def request_sdk_task_cancel(*, task_id: int, scope: SdkTaskScope) -> SdkTaskCancelResult:@@ -637,7 +733,11 @@ async def request_sdk_task_cancel(calling again reaches the same command, including after it withdrew aqueued turn and before it is marked completed. Raises``SdkTaskNotCancellable`` when the aimed run ended without being- cancelled, whichever way the wait ends.+ cancelled, whichever way the wait ends, except a paused run whose+ automatic resume gave up while the command is still unapplied at the+ deadline and the row is still at the version the command is aimed at: the command+ still cancels it, so that answer is ``cancelled=False``, and calling+ again rejoins the same command."""prepared = await run_db_io_cancellation_safe(@@ -690,6 +790,7 @@ async def request_sdk_task_cancel(aimed_run_id=aimed_run_id,aimed_at_queued_turn=prepared.aimed_at_queued_turn,command_completed=False,+ command_db_id=prepared.command_db_id,))await asyncio.sleep(_COMMAND_POLL_SECONDS)@@ -723,17 +824,17 @@ def _is_settled_sdk_cancel_target("""Whether this exact target already reached its cancel outcome.True for the command's own finalize replayed, and for the settlement of- the run this command cancelled: both leave the target run FAILED at the- command's state version or one past it, carrying the cancellation- sentence. A run that failed on its own carries another text and is not- this command's to report.+ the run this command cancelled: both leave the target run FAILED at or+ past the command's state version (a paused run's stopped automatic+ resume moves it once more), carrying the cancellation sentence. A run+ that failed on its own carries another text and is not this command's+ to report."""return (is_sdk_cancelled(task.status, task.error_message)and _task_run_id(task) == expected_run_id- and int(task.state_version or 0)- in {expected_state_version, expected_state_version + 1}+ and int(task.state_version or 0) >= expected_state_version)@@ -742,14 +843,22 @@ def _is_exact_cancel_target(*,expected_run_id: str | None,expected_state_version: int,- expected_status: TaskStatus,+ expected_status: TaskStatus | None,) -> bool:- """Whether the row is still the target this command was aimed at."""+ """Whether the row is still the target this command was aimed at.++ ``expected_status`` ``None`` accepts any status a cancel can end: the+ command was persisted against the run and version alone.+ """+ if expected_status is None:+ status_matches = task.status in _CANCELLABLE_STATUSES+ else:+ status_matches = task.status == expected_statusreturn (_task_run_id(task) == expected_run_idand int(task.state_version or 0) == expected_state_version- and task.status == expected_status+ and status_matches)@@ -761,15 +870,17 @@ def _stale(task: Task, expected_run_id: str | None, expected_state_version: int))-def _carries_display_events(task: Task, expected_status: TaskStatus) -> bool:+def _carries_display_events(task: Task, cancelled_from: TaskStatus | None) -> bool:"""Whether the terminal broadcast should carry the turn's display events.Only a version-2 task has them, and only a run that ran has the result- fact they are read from: a new task cancelled before it started has- none, and publishing them without it fails.+ fact they are read from. A running, waiting or paused run gets one from+ whichever writer ended it (``None`` is one of those, read before its+ status was known); a new task cancelled before it started has none, and+ publishing them without it fails."""- if expected_status != TaskStatus.RUNNING:+ if cancelled_from == TaskStatus.PENDING:return Falsereturn bool(task.conversation_storage_version == 2)@@ -780,13 +891,16 @@ def _load_cancelable_sdk_task_sync(agent_id: int,expected_run_id: str | None,expected_state_version: int,- expected_status: TaskStatus,+ expected_status: TaskStatus | None,terminal_event_state: dict[str, Any],-) -> bool:- """Return whether the exact cancel target is already settled.-- A settled target fills ``terminal_event_state`` with the control- snapshot its terminal broadcast carries.+) -> TaskStatus | None:+ """The status to cancel the exact target from, or ``None`` when it is+ already settled.++ ``expected_status`` is given for a queued turn, whose target was just+ resolved; otherwise any status a cancel can end is accepted as read. A+ settled target fills ``terminal_event_state`` with the control snapshot+ its terminal broadcast carries."""SessionLocal = get_session_local()@@ -799,7 +913,7 @@ def _load_cancelable_sdk_task_sync():if _carries_display_events(task, expected_status):terminal_event_state.update(task_control_snapshot(task).as_dict())- return True+ return Noneif not _is_exact_cancel_target(task,expected_run_id=expected_run_id,@@ -810,10 +924,12 @@ def _load_cancelable_sdk_task_sync(_stale(task, expected_run_id, expected_state_version),reason="stale_run",)- return False+ return TaskStatus(task.status)-def _close_cancelled_turn_no_commit(db: Session, task: Task) -> str | None:+def _close_cancelled_turn_no_commit(+ db: Session, task: Task, *, was_running: bool+) -> str | None:"""Stage what a failed run's settlement leaves beside the terminal row.The assistant line mirrors the settlement's own -- the generic failure@@ -823,7 +939,10 @@ def _close_cancelled_turn_no_commit(db: Session, task: Task) -> str | None:not be offered for a resend.The running turn is the newest pending user row, which is sound only- because a RUNNING task accepts no new turn. Returns that turn's id.+ because a RUNNING task accepts no new turn. A waiting or paused run has+ no such guarantee -- a paused task accepts an append -- and its turn's+ delivery was settled when the run stopped, so only the line is written.+ Returns the closed turn's id."""from .assistant_history_safety import TASK_FAILURE_MESSAGE_TYPE@@ -836,17 +955,19 @@ def _close_cancelled_turn_no_commit(db: Session, task: Task) -> str | None:from .client_error_messages import CLIENT_SAFE_TASK_FAILUREtask_id = int(task.id)- row = (- db.query(TaskChatMessage.turn_id)- .filter(- TaskChatMessage.task_id == task_id,- TaskChatMessage.role == "user",- TaskChatMessage.delivery_status == DELIVERY_PENDING,+ turn_id: str | None = None+ if was_running:+ row = (+ db.query(TaskChatMessage.turn_id)+ .filter(+ TaskChatMessage.task_id == task_id,+ TaskChatMessage.role == "user",+ TaskChatMessage.delivery_status == DELIVERY_PENDING,+ )+ .order_by(TaskChatMessage.id.desc())+ .first())- .order_by(TaskChatMessage.id.desc())- .first()- )- turn_id = str(row[0]) if row is not None and row[0] is not None else None+ turn_id = str(row[0]) if row is not None and row[0] is not None else Noneif task.user_id is not None:persist_assistant_message_no_commit(db,@@ -863,6 +984,48 @@ def _close_cancelled_turn_no_commit(db: Session, task: Task) -> str | None:return turn_id+def _stop_auto_resume_no_commit(+ db: Session,+ task: Task,+ *,+ expected_run_id: str | None,+ expected_state_version: int,+) -> int:+ """Stop the automatic resume a paused run waits for; return the version+ the task holds afterwards.++ Without this the sweeper could still dispatch the resume it had+ scheduled. A run with nothing pending -- none was scheduled, or it gave+ up -- is left as it is. Raises ``StaleTaskRunError`` when the row read+ under the lock is no longer the run and version the cancel was aimed+ at, or when the stop lost a race, so the caller reads again.+ """++ from .task_auto_resume import cancel_pending_auto_resume_no_commit++ # The stop requires the task row locked and read under that lock.+ locked = (+ db.query(Task)+ .filter(Task.id == task.id)+ .with_for_update()+ .populate_existing()+ .one()+ )+ if (+ _task_run_id(locked) != expected_run_id+ or int(locked.state_version or 0) != expected_state_version+ ):+ db.rollback()+ raise StaleTaskRunError(_stale(locked, expected_run_id, expected_state_version))+ stopped = cancel_pending_auto_resume_no_commit(db, locked, at="sdk_cancel")+ if stopped.raced:+ db.rollback()+ raise StaleTaskRunError(+ f"task {task.id} automatic resume moved while it was being stopped"+ )+ return int(locked.state_version or 0)++def _finalize_sdk_cancel_sync(*,task_id: int,@@ -874,9 +1037,12 @@ def _finalize_sdk_cancel_sync() -> bool:"""Persist cancellation for one exact target; return whether this wrote it.- ``expected_status`` is RUNNING, or PENDING for a new task whose START is- still queued: that task never ran, so it gets the terminal row alone and- its START's own settlement closes the turn.+ ``expected_status`` is the status the target was read in. PENDING is a+ new task whose START is still queued: it never ran, so it gets the+ terminal row alone and its START's own settlement closes the turn. A+ PAUSED run first has its pending automatic resume stopped, which moves+ the version this write is then fenced on. A WAITING_FOR_USER run must+ still be unanswered, and has its question closed with the row.Raises ``StaleTaskRunError`` before any write, or after a rollback, whenthe target is neither still as read nor already settled. When the@@ -886,17 +1052,28 @@ def _finalize_sdk_cancel_sync(commit."""+ from .chat_history_service import supersede_legacy_question_rowsfrom .task_coordinator_runtime import current_task_coordinatorfrom .task_coordinator_service import (lock_task_lease_no_commit,task_lease_predicate,)from .task_execution_event_writer import stage_result_fact_no_commit+ from .task_interaction_close import (+ terminate_active_interaction_for_question_cancel,+ )+ from .task_interaction_read import get_pending_interaction_questionfrom .task_lease_service import task_settlement_ownership_valuesfrom .task_orchestrator import (invalidate_task_cache_best_effort,sync_trigger_run_status,)+ from .task_question_cancel import (+ _latest_question_row,+ _materialize_question_step,+ _pending_resume_command_exists,+ )+ from .task_resume import _RESUMABLE_CONTROL_STATESfrom .workforce_runtime import sync_workforce_run_statusSessionLocal = get_session_local()@@ -930,19 +1107,27 @@ def _finalize_sdk_cancel_sync(_stale(task, expected_run_id, expected_state_version))+ fence_version = expected_state_version+ if expected_status == TaskStatus.PAUSED:+ fence_version = _stop_auto_resume_no_commit(+ db,+ task,+ expected_run_id=expected_run_id,+ expected_state_version=expected_state_version,+ )statement = (update(Task).where(Task.id == task_id,Task.agent_id == agent_id,Task.source == SDK_TASK_SOURCE,- func.coalesce(Task.state_version, 0) == expected_state_version,+ func.coalesce(Task.state_version, 0) == fence_version,Task.status == expected_status,).values(status=TaskStatus.FAILED,control_state=TaskControlState.FAILED.value,- state_version=expected_state_version + 1,+ state_version=fence_version + 1,**task_settlement_ownership_values(task_id),output=None,error_message=SDK_CANCEL_ERROR_MESSAGE,@@ -955,6 +1140,11 @@ def _finalize_sdk_cancel_sync(if coordinator is not None:assert coordinator.lease is not Nonestatement = statement.where(task_lease_predicate(coordinator.lease))+ if expected_status == TaskStatus.WAITING_FOR_USER:+ statement = statement.where(+ Task.control_state.in_(tuple(_RESUMABLE_CONTROL_STATES)),+ ~_pending_resume_command_exists(task_id),+ )# ``task`` is already in the identity map, so RETURNING hands back# that same object; populate_existing refreshes its attributes.@@ -970,10 +1160,31 @@ def _finalize_sdk_cancel_sync(f"task {task_id} changed while finalizing cancel target "f"{expected_run_id}/{expected_state_version}")+ if expected_status == TaskStatus.WAITING_FOR_USER:+ # After the task row's write, which is the lock order the+ # interaction writers require, and in the order abandoning the+ # question uses: the question is read before the rows that carry+ # it are closed, and a question with no trace row is given one+ # so the task's steps still show it.+ question, interactions = get_pending_interaction_question(db, updated)+ question_row = _latest_question_row(db, task_id)+ terminate_active_interaction_for_question_cancel(+ db, task_id=task_id, run_id=expected_run_id+ )+ supersede_legacy_question_rows(db, task_id=task_id)+ _materialize_question_step(+ db,+ task=updated,+ question=question,+ interactions=interactions,+ question_row=question_row,+ )sync_workforce_run_status(db, updated, TaskStatus.FAILED)sync_trigger_run_status(db, updated, TaskStatus.FAILED)- if expected_status == TaskStatus.RUNNING:- turn_id = _close_cancelled_turn_no_commit(db, updated)+ if expected_status != TaskStatus.PENDING:+ turn_id = _close_cancelled_turn_no_commit(+ db, updated, was_running=expected_status == TaskStatus.RUNNING+ )stage_result_fact_no_commit(db,updated,@@ -1119,12 +1330,16 @@ def _resolve_queued_cancel_target_sync(if _task_run_id(task) == queued_run_id:# The queued turn started. Fenced on the version read now:# the handoff and the run itself moved it since the request.+ # It is cancelled the way its current state is, which may+ # already be waiting for an answer or paused.if is_sdk_cancelled(task.status, task.error_message):return None- if task.status != TaskStatus.RUNNING:+ if task.status not in _CANCELLABLE_STATUSES:raise stalereturn _CancelTarget(- queued_run_id, int(task.state_version or 0), TaskStatus.RUNNING+ queued_run_id,+ int(task.state_version or 0),+ TaskStatus(task.status),)start = _start_of_run(db, task_id, queued_run_id)if _start_was_stopped(start):@@ -1197,7 +1412,7 @@ async def cancel_sdk_task_unserialized(so the command ends at once with its reason instead of being retried."""- expected_status = TaskStatus.RUNNING+ expected_status: TaskStatus | None = Noneif queued_run_id is not None:target = await run_db_io_cancellation_safe(lambda: _resolve_queued_cancel_target_sync(@@ -1211,7 +1426,7 @@ async def cancel_sdk_task_unserialized(expected_status = target.statusterminal_event_state: dict[str, Any] = {}- already_settled = await run_db_io_cancellation_safe(+ cancel_from = await run_db_io_cancellation_safe(lambda: _load_cancelable_sdk_task_sync(task_id=task_id,agent_id=agent_id,@@ -1221,14 +1436,17 @@ async def cancel_sdk_task_unserialized(terminal_event_state=terminal_event_state,))- if not already_settled:+ if cancel_from is not None:from .task_execution import background_task_manager- # Returns once the run coroutine has unwound, which normally means- # its own settlement already wrote the cancelled row.- await background_task_manager.cancel_task(- task_id, timeout_seconds=SDK_CANCEL_WAIT_SECONDS- )+ if cancel_from == TaskStatus.RUNNING:+ # Returns once the run coroutine has unwound, which normally+ # means its own settlement already wrote the cancelled row. A+ # waiting or paused run has no coroutine; this core writes its+ # row.+ await background_task_manager.cancel_task(+ task_id, timeout_seconds=SDK_CANCEL_WAIT_SECONDS+ )def finalize() -> bool:return _finalize_sdk_cancel_sync(@@ -1236,7 +1454,7 @@ async def cancel_sdk_task_unserialized(agent_id=agent_id,expected_run_id=expected_run_id,expected_state_version=expected_state_version,- expected_status=expected_status,+ expected_status=cancel_from,terminal_event_state=terminal_event_state,)@@ -1251,8 +1469,8 @@ async def cancel_sdk_task_unserialized(wrote = await run_db_io_cancellation_safe(finalize)if wrote:# No settlement wrote the row, so no coroutine had unwound here.- # One that registered since the call above is stopped now; its- # settlement finds the row terminal and writes nothing.+ # One that registered since the target was read is stopped now;+ # its settlement finds the row terminal and writes nothing.await background_task_manager.cancel_task(task_id, timeout_seconds=SDK_CANCEL_WAIT_SECONDS)diff --git a/tests/web/api/test_sdk_task_cancel.py b/tests/web/api/test_sdk_task_cancel.pyindex 2e7c248fc..e9f5bfea0 100644--- a/tests/web/api/test_sdk_task_cancel.py+++ b/tests/web/api/test_sdk_task_cancel.py@@ -21,10 +21,11 @@ from tests.shared.async_waits import DB_PROGRESS_TIMEOUTfrom xagent.web.api import websocket as websocket_apifrom xagent.web.models.agent import Agentfrom xagent.web.models.chat_message import TaskChatMessage-from xagent.web.models.task import Task, TaskStatus+from xagent.web.models.task import Task, TaskStatus, TraceEventfrom xagent.web.models.task_admission import TaskAdmissionBucket, TaskAdmissionTicketfrom xagent.web.models.task_command import TaskExecutionCommandfrom xagent.web.models.task_execution_event import TaskExecutionEvent+from xagent.web.models.task_interaction import TaskInteractionRequestfrom xagent.web.services import a2a_task_cancel as a2a_cancel_servicefrom xagent.web.services import sdk_task_cancelfrom xagent.web.services import task_command_execution as command_execution_service@@ -35,6 +36,7 @@ from xagent.web.services.assistant_history_safety import TASK_FAILURE_MESSAGE_TYfrom xagent.web.services.chat_history_service import (DELIVERY_DISPATCHED,DELIVERY_PENDING,+ SUPERSEDED_MESSAGE_TYPE,)from xagent.web.services.client_error_messages import CLIENT_SAFE_TASK_FAILUREfrom xagent.web.services.external_task_cancel import EXTERNAL_TURN_INTERRUPTED_MESSAGE@@ -460,9 +462,9 @@ async def test_a_run_registered_after_the_first_cancel_is_stopped_after_the_writ{"status": TaskStatus.PAUSED, "state_version": 5},{"run_id": "run-next"},{"state_version": 6},- {"status": TaskStatus.WAITING_FOR_USER},+ {"status": TaskStatus.PENDING},],- ids=["failed_itself", "completed", "paused", "new_run", "moved", "waiting"],+ ids=["failed_itself", "completed", "paused", "new_run", "moved", "pending"],)async def test_cancel_never_claims_an_outcome_it_did_not_cause(monkeypatch: pytest.MonkeyPatch, moved: dict[str, Any]@@ -1542,3 +1544,339 @@ def test_the_stopped_before_start_wording_has_one_definition() -> None:sdk_task_cancel.STOPPED_BEFORE_START_REASON== task_execution_admission.STOPPED_BEFORE_START_REASON)+++# ----- a run that is waiting for an answer -----+++def _add_active_interaction(task_id: int, run_id: str, key: str) -> int:+ """A native interaction holding the task's one answer slot for ``run_id``."""+ db = _direct_db_session()+ try:+ anchor = TraceEvent(+ task_id=task_id,+ event_id=f"anchor-{key}",+ event_type="agent_execution_checkpoint",+ timestamp=datetime.now(UTC),+ data={},+ )+ db.add(anchor)+ db.flush()+ row = TaskInteractionRequest(+ task_id=task_id,+ run_id=run_id,+ kind="clarification",+ protocol_version=1,+ status="active",+ active_slot=1,+ origin="sdk",+ request_payload={+ "message": "Continue?",+ "interactions": [+ {"type": "text_input", "field": "response", "label": "Answer"}+ ],+ },+ request_idempotency_key=key,+ resume_trace_event_id=int(anchor.id),+ resume_event_id=f"resume-event-{key}",+ resume_execution_id=f"resume-execution-{key}",+ resume_locator_format="trace_event_pk_v1",+ resume_checkpoint_type="agent_execution_checkpoint",+ resume_run_partition=run_id,+ expires_at=datetime.now(UTC) + timedelta(minutes=15),+ )+ db.add(row)+ db.query(Task).filter(Task.id == task_id).update(+ {"interaction_protocol_version": 1}+ )+ db.commit()+ return int(row.id)+ finally:+ db.close()+++def _interaction(row_id: int) -> tuple[str, int | None]:+ db = _direct_db_session()+ try:+ row = db.get(TaskInteractionRequest, row_id)+ return str(row.status), row.active_slot+ finally:+ db.close()+++def _question_traces(task_id: int) -> list[str]:+ db = _direct_db_session()+ try:+ rows = (+ db.query(TraceEvent)+ .filter(+ TraceEvent.task_id == task_id,+ TraceEvent.event_type == "agent_message",+ )+ .all()+ )+ return [str(row.data.get("message")) for row in rows]+ finally:+ db.close()+++def _waiting_task(**overrides: Any) -> tuple[int, int, int]:+ values = {+ "status": TaskStatus.WAITING_FOR_USER,+ "control_state": TaskControlState.WAITING_FOR_USER.value,+ "output": "stale output",+ }+ values.update(overrides)+ task_id, agent_id, owner_user_id = _create_task(**values)+ db = _direct_db_session()+ try:+ db.add(+ TaskChatMessage(+ task_id=task_id,+ user_id=owner_user_id,+ role="assistant",+ content="Continue?",+ message_type="question",+ turn_id="turn-question",+ )+ )+ db.commit()+ finally:+ db.close()+ _add_active_interaction(task_id, "run-sdk", "question-1")+ return task_id, agent_id, owner_user_id+++def _active_interaction_of(task_id: int) -> int:+ db = _direct_db_session()+ try:+ return int(+ db.query(TaskInteractionRequest.id)+ .filter(TaskInteractionRequest.task_id == task_id)+ .one()[0]+ )+ finally:+ db.close()+++async def _cancel_as_read(task_id: int, agent_id: int) -> None:+ await cancel_sdk_task_unserialized(+ task_id=task_id,+ agent_id=agent_id,+ expected_run_id="run-sdk",+ expected_state_version=4,+ )+++async def test_a_waiting_run_is_not_waited_on_before_its_cancel_is_written(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A run waiting for an answer has no coroutine to wait for, so the core+ writes the row without first stopping one: a resume that started after+ the target was read must not be cancelled as if it were the aimed run.+ Only the stop after the write, for a coroutine registered since, runs."""+ task_id, agent_id, _owner = _waiting_task()+ _broadcast_manager(monkeypatch)+ statuses: list[TaskStatus] = []++ async def record_status(_task_id: int, **_kwargs: object) -> SimpleNamespace:+ statuses.append(_load_task(task_id).status)+ return SimpleNamespace(requested=False)++ monkeypatch.setattr(+ task_execution_service.background_task_manager, "cancel_task", record_status+ )++ await _cancel_as_read(task_id, agent_id)++ assert statuses == [TaskStatus.FAILED]++[email protected]("storage_version", [1, 2])+async def test_a_waiting_run_is_cancelled_and_its_question_closed(+ monkeypatch: pytest.MonkeyPatch, storage_version: int+) -> None:+ task_id, agent_id, _owner = _waiting_task(storage_version=storage_version)+ interaction_id = _active_interaction_of(task_id)+ _broadcast_manager(monkeypatch)++ await _cancel_as_read(task_id, agent_id)++ task = _load_task(task_id)+ assert is_sdk_cancelled(task.status, task.error_message)+ assert (task.run_id, task.state_version, task.output) == ("run-sdk", 5, None)+ # The question no longer reads as pending; the failure line follows it.+ assert _assistant_rows(task_id) == [+ ("Continue?", SUPERSEDED_MESSAGE_TYPE),+ (CLIENT_SAFE_TASK_FAILURE, TASK_FAILURE_MESSAGE_TYPE),+ ]+ # The answer slot is released, so a later run of the task can ask again.+ assert _interaction(interaction_id) == ("terminated", None)+ later = _add_active_interaction(task_id, "run-later", "question-2")+ assert _interaction(later) == ("active", 1)+ # A version-1 question with no trace row of its own is given one, so the+ # task's steps still show what was asked.+ assert _question_traces(task_id) == (["Continue?"] if storage_version == 1 else [])+ assert [fact["result"] for fact in _result_facts(task_id)] == (+ [{"status": "cancelled", "error": SDK_CANCEL_ERROR_MESSAGE}]+ if storage_version == 2+ else []+ )++[email protected]("claimed_by", ["control_state", "queued_command"])+def test_a_waiting_run_an_answer_already_claimed_is_not_cancelled(+ claimed_by: str,+) -> None:+ """The answer owns the waiting run until it resumes it."""+ task_id, agent_id, owner_user_id = _waiting_task()+ db = _direct_db_session()+ try:+ if claimed_by == "control_state":+ db.query(Task).filter(Task.id == task_id).update(+ {"control_state": TaskControlState.RESUME_REQUESTED.value}+ )+ else:+ stage_task_command(+ db,+ task_id=task_id,+ actor_user_id=owner_user_id,+ command_id="reply-1",+ kind=TaskCommandKind.RESUME_INPUT,+ payload={},+ )+ db.commit()+ finally:+ db.close()+ before = _row(task_id)++ with pytest.raises(StaleTaskRunError):+ sdk_task_cancel._finalize_sdk_cancel_sync(+ task_id=task_id,+ agent_id=agent_id,+ expected_run_id="run-sdk",+ expected_state_version=4,+ expected_status=TaskStatus.WAITING_FOR_USER,+ terminal_event_state={},+ )++ assert _row(task_id) == before+ assert _assistant_rows(task_id) == [("Continue?", "question")]+++async def test_a_paused_run_is_cancelled_without_touching_a_later_turn(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A paused task accepts an append, so a pending user message may belong+ to a later turn: the cancelled run's settlement leaves it alone."""+ task_id, agent_id, owner_user_id = _create_task(+ status=TaskStatus.PAUSED, control_state=TaskControlState.PAUSED.value+ )+ _seed_pending_user_message(task_id, owner_user_id, "turn-appended")+ _broadcast_manager(monkeypatch)++ await _cancel_as_read(task_id, agent_id)++ task = _load_task(task_id)+ assert is_sdk_cancelled(task.status, task.error_message)+ assert task.state_version == 5+ assert _delivery_status(task_id, "turn-appended") == DELIVERY_PENDING+ assert _assistant_rows(task_id) == [+ (CLIENT_SAFE_TASK_FAILURE, TASK_FAILURE_MESSAGE_TYPE)+ ]++[email protected]("moved", ["run", "state_version"])+def test_a_paused_run_that_moved_before_its_lock_keeps_its_automatic_resume(+ monkeypatch: pytest.MonkeyPatch, moved: str+) -> None:+ """The row read under the lock is checked against the run and version+ the cancel was aimed at before the automatic resume is stopped: a row+ that moved since is stale, and nothing is stopped or written."""+ from xagent.web.services import task_auto_resume++ task_id, _agent_id, _owner = _create_task(+ status=TaskStatus.PAUSED, control_state=TaskControlState.PAUSED.value+ )+ stop = MagicMock()+ monkeypatch.setattr(task_auto_resume, "cancel_pending_auto_resume_no_commit", stop)+ before = _row(task_id)+ db = _direct_db_session()+ try:+ task = db.query(Task).filter(Task.id == task_id).one()+ with pytest.raises(StaleTaskRunError):+ sdk_task_cancel._stop_auto_resume_no_commit(+ db,+ task,+ expected_run_id="run-other" if moved == "run" else "run-sdk",+ expected_state_version=3 if moved == "state_version" else 4,+ )+ finally:+ db.close()++ stop.assert_not_called()+ assert _row(task_id) == before++[email protected]("storage_version", [1, 2])[email protected]("state", ["waiting", "paused"])+async def test_the_terminal_frame_of_a_waiting_or_paused_run_carries_its_snapshot(+ monkeypatch: pytest.MonkeyPatch, state: str, storage_version: int+) -> None:+ """The core writes a waiting or paused run's result fact itself, so a+ version-2 frame carries the turn's display events, on the write and on+ a replay of the same command alike."""+ if state == "waiting":+ task_id, agent_id, _owner = _waiting_task(storage_version=storage_version)+ else:+ task_id, agent_id, _owner = _create_task(+ status=TaskStatus.PAUSED,+ control_state=TaskControlState.PAUSED.value,+ storage_version=storage_version,+ )+ manager = _broadcast_manager(monkeypatch)++ for _replay in range(2):+ manager.broadcast_to_task.reset_mock()+ await _cancel_as_read(task_id, agent_id)++ frame = _terminal_frame(manager)+ if storage_version == 2:+ assert frame["state_version"] == 5+ assert frame["task"]["status"] == "failed"+ else:+ assert "state_version" not in frame+++ "state",+ [+ {+ "status": TaskStatus.WAITING_FOR_USER,+ "control_state": TaskControlState.WAITING_FOR_USER.value,+ },+ {"status": TaskStatus.PAUSED, "control_state": TaskControlState.PAUSED.value},+ ],+ ids=["waiting", "paused"],+)+async def test_a_queued_turn_that_started_and_now_waits_is_cancelled_as_it_is(+ monkeypatch: pytest.MonkeyPatch, state: dict[str, Any]+) -> None:+ """The queued run started and is waiting for an answer or paused by the+ time the command applies: it is cancelled the way its state is."""+ task_id, agent_id, owner_user_id = _create_task(+ run_id=QUEUED_RUN, state_version=7, **state+ )+ _stage_start(task_id, owner_user_id, status="completed", attempt_count=1)+ _broadcast_manager(monkeypatch)++ await _cancel_queued(task_id, agent_id)++ task = _load_task(task_id)+ assert is_sdk_cancelled(task.status, task.error_message)+ assert (task.run_id, task.state_version) == (QUEUED_RUN, 8)diff --git a/tests/web/api/v1/test_v1_cancel_task.py b/tests/web/api/v1/test_v1_cancel_task.pyindex 50c342f89..fe25408ce 100644--- a/tests/web/api/v1/test_v1_cancel_task.py+++ b/tests/web/api/v1/test_v1_cancel_task.py@@ -267,6 +267,7 @@ def test_cancel_running_task_then_repeat_then_append():("values", "expected_code"),[(+ # No START is queued for it, so there is no turn to aim at.{"status": TaskStatus.COMPLETED,"control_state": TaskControlState.COMPLETED.value,@@ -288,8 +289,8 @@ def test_cancel_running_task_then_repeat_then_append():),({- "status": TaskStatus.PAUSED,- "control_state": TaskControlState.PAUSED.value,+ "status": TaskStatus.WAITING_FOR_USER,+ "control_state": TaskControlState.RESUME_REQUESTED.value,},"task_busy",),@@ -297,15 +298,34 @@ def test_cancel_running_task_then_repeat_then_append():{"status": TaskStatus.WAITING_FOR_USER,"control_state": TaskControlState.WAITING_FOR_USER.value,+ "answer_queued": True,},"task_busy",),],- ids=["completed", "failed_itself", "pending", "paused", "waiting"],+ ids=["completed", "failed_itself", "pending", "answer_in_flight", "answer_queued"],)-def test_states_that_are_not_running_are_refused_without_a_write(values, expected_code):+def test_states_with_nothing_to_cancel_are_refused_without_a_write(+ values, expected_code+):task_id, _agent_id, key = _running_task()+ values = dict(values)+ answer_queued = values.pop("answer_queued", False)_update(task_id, runner_id=None, lease_expires_at=None, **values)+ if answer_queued:+ db = _direct_db_session()+ try:+ stage_task_command(+ db,+ task_id=task_id,+ actor_user_id=None,+ command_id="reply-queued",+ kind=TaskCommandKind.RESUME_INPUT,+ payload={},+ )+ db.commit()+ finally:+ db.close()before = _row(task_id)response = _cancel(task_id, key)@@ -562,6 +582,163 @@ def test_a_task_that_reads_failed_after_automatic_resume_gave_up_is_not_cancellaassert _cancel_commands(task_id) == []+def test_a_paused_cancel_staged_after_its_row_moved_is_not_accepted(+ monkeypatch: pytest.MonkeyPatch,+):+ """An accepted append moves the paused row between the request's read and+ the command's staging, so the command aims at the old version while its+ row column records the new one; the append's START then fails and the+ task reads ``failed``. The executor fences on the aimed version and will+ reject the command, so the deadline does not answer 202, and a repeat+ answers the same 409 without staging another command."""+ task_id, _agent_id, key = _running_task()+ _update(+ task_id,+ status=TaskStatus.PAUSED,+ control_state=TaskControlState.PAUSED.value,+ runner_id=None,+ lease_expires_at=None,+ )+ aimed_version = _row(task_id)["state_version"]+ original_enqueue = sdk_task_cancel._enqueue_cancel++ def staged_after_an_append(+ db: Any, task: Any, target_state_version: int, *, queued_run_id: Any+ ) -> Any:+ _update(task_id, state_version=aimed_version + 1)+ return original_enqueue(+ db, task, target_state_version, queued_run_id=queued_run_id+ )++ monkeypatch.setattr(sdk_task_cancel, "_enqueue_cancel", staged_after_an_append)+ start_failed: list[bool] = []+ monkeypatch.setattr(+ task_auto_resume,+ "status_only_terminal_view",+ lambda _db, _task_id: SimpleNamespace(state="manual") if start_failed else None,+ )++ async def unapplied_while_the_start_fails(*_args: Any, **_kwargs: Any) -> bool:+ start_failed.append(True)+ return False++ monkeypatch.setattr(+ sdk_task_cancel,+ "dispatch_one_task_command",+ AsyncMock(side_effect=unapplied_while_the_start_fails),+ )+ monkeypatch.setattr(sdk_task_cancel, "SDK_CANCEL_REQUEST_WAIT_SECONDS", 0.1)+ response = _cancel(task_id, key)++ assert response.status_code == 409, response.text+ assert _error(response) == "task_not_cancellable"+ (command,) = _cancel_commands(task_id)+ assert command.payload["target_state_version"] == aimed_version+ assert command.target_state_version == aimed_version + 1+ again = _cancel(task_id, key)+ assert again.status_code == 409, again.text+ assert _error(again) == "task_not_cancellable"+ assert len(_cancel_commands(task_id)) == 1+ row = _row(task_id)+ assert (row["status"], row["state_version"]) == (+ TaskStatus.PAUSED,+ aimed_version + 1,+ )+++ "ended", ["automatic_resume_gave_up", "gave_up_after_the_row_moved", "completed"]+)+def test_a_target_that_ends_while_the_cancel_is_still_unapplied_at_the_deadline(+ monkeypatch: pytest.MonkeyPatch, ended: str+):+ """The command is recorded against a paused run and still pending at the+ deadline. A paused run whose automatic resume gave up meanwhile reads+ ``failed`` but stays at the version the command is fenced on, and the+ command still cancels it, so the answer is 202 and calling again rejoins+ the same command. Once the row has moved past that version the command+ no longer applies, and a run that completed on its own is over: both+ answer 409."""+ task_id, _agent_id, key = _running_task()+ _update(+ task_id,+ status=TaskStatus.PAUSED,+ control_state=TaskControlState.PAUSED.value,+ runner_id=None,+ lease_expires_at=None,+ )+ before = _row(task_id)+ gave_up: list[bool] = []+ monkeypatch.setattr(+ task_auto_resume,+ "status_only_terminal_view",+ lambda _db, _task_id: SimpleNamespace(state="exhausted") if gave_up else None,+ )++ async def ends_without_applying(_executor: Any, *, command_db_id: int) -> bool:+ if gave_up:+ return False+ if ended == "completed":+ _update(+ task_id,+ status=TaskStatus.COMPLETED,+ control_state=TaskControlState.COMPLETED.value,+ output="done",+ )+ return False+ gave_up.append(True)+ if ended == "gave_up_after_the_row_moved":+ _update(task_id, state_version=before["state_version"] + 1)+ return False++ dispatched = AsyncMock(side_effect=ends_without_applying)+ monkeypatch.setattr(sdk_task_cancel, "dispatch_one_task_command", dispatched)+ monkeypatch.setattr(sdk_task_cancel, "SDK_CANCEL_REQUEST_WAIT_SECONDS", 0.1)++ response = _cancel(task_id, key)++ (command,) = _cancel_commands(task_id)+ assert command.status == "pending"+ if ended != "automatic_resume_gave_up":+ assert response.status_code == 409, response.text+ assert _error(response) == "task_not_cancellable"+ return+ assert response.status_code == 202, response.text+ assert response.json() == {+ "task_id": task_id,+ "status": "failed",+ "run_id": before["run_id"],+ "state_version": before["state_version"],+ "cancelled_run_id": None,+ "error_code": None,+ }+ assert _row(task_id) == before++ again = _cancel(task_id, key)++ assert again.status_code == 202, again.text+ assert again.json() == response.json()+ assert dispatched.await_args.kwargs["command_db_id"] == command.id+ assert [c.id for c in _cancel_commands(task_id)] == [command.id]+ assert _row(task_id) == before+ # The command is then applied as the core applies it.+ _update(+ task_id,+ status=TaskStatus.FAILED,+ control_state=TaskControlState.FAILED.value,+ state_version=before["state_version"] + 1,+ error_message=SDK_CANCEL_ERROR_MESSAGE,+ )+ _settle_cancel_command(command.id, status="completed")++ applied = _cancel(task_id, key)++ assert applied.status_code == 200, applied.text+ assert applied.json()["cancelled_run_id"] == before["run_id"]+ assert applied.json()["error_code"] == "task_cancelled"+ assert len(_cancel_commands(task_id)) == 1++@pytest.mark.parametrize("case", ["other_agent", "other_user", "missing", "non_sdk"])def test_tasks_outside_the_key_scope_are_404_and_untouched(case):task_id, _agent_id, key = _running_task()@@ -1275,6 +1452,33 @@ def test_a_repeat_while_the_withdrawing_cancel_is_still_processing_rejoins_it(assert len(_cancel_commands(task_id)) == 1+def test_a_repeat_while_the_withdrawal_is_processing_does_not_cancel_the_paused_turn(+ monkeypatch: pytest.MonkeyPatch,+):+ """The turn before the withdrawn one is paused, and a paused task can be+ cancelled. The withdrawing cancel has stopped the queued START but is not+ marked completed yet, so no START is queued and the row is paused: the+ repeat must still rejoin that command rather than aim a new one at the+ paused turn, and the paused turn is left as it is."""+ task_id, key, first, command = _withdrawn_before_completion(+ monkeypatch, "previous_run_paused"+ )+ before = _row(task_id)+ dispatched = AsyncMock()+ monkeypatch.setattr(sdk_task_cancel, "dispatch_one_task_command", dispatched)++ again = _cancel(task_id, key)++ assert first.status_code == 200, first.text+ assert again.status_code == 200, again.text+ assert again.json() == first.json()+ assert again.json()["status"] == "paused"+ assert again.json()["cancelled_run_id"] == QUEUED_RUN+ assert dispatched.await_args.kwargs["command_db_id"] == command.id+ assert [c.id for c in _cancel_commands(task_id)] == [command.id]+ assert _row(task_id) == before++def test_a_repeat_while_processing_after_the_turn_before_resumed_cancels_that_turn(monkeypatch: pytest.MonkeyPatch,):@@ -1497,9 +1701,9 @@ def test_a_queued_start_that_can_no_longer_hand_off_is_not_the_cancel_target(monkeypatch: pytest.MonkeyPatch, moved: str):"""The row was moved to another run after the START was accepted (an- automatic resume won the race), so the START never opens its run. A- running row is cancelled as the running run it is; one waiting for an- answer is refused as busy, as it would be with no START at all."""+ automatic resume won the race), so the START never opens its run. The+ row is cancelled as the run it holds, running or waiting for an answer,+ as it would be with no START at all."""task_id, _agent_id, key = _running_task()_update(task_id,@@ -1529,13 +1733,9 @@ def test_a_queued_start_that_can_no_longer_hand_off_is_not_the_cancel_target(response = _cancel(task_id, key)assert _row(task_id) == before- if moved == "waiting_on_another_run":- assert response.status_code == 409, response.text- assert _error(response) == "task_busy"- assert _cancel_commands(task_id) == []- returnassert response.status_code == 202, response.textassert response.json()["run_id"] == "run-first"+ assert response.json()["status"] == row_values["status"].value(command,) = _cancel_commands(task_id)assert "queued_run_id" not in command.payloadassert command.target_run_id == "run-first"@@ -1587,3 +1787,148 @@ def test_a_cancel_whose_queued_turn_started_before_the_deadline_is_accepted(assert response.json()["status"] == "running"assert response.json()["run_id"] == QUEUED_RUNassert response.json()["cancelled_run_id"] is None+++ "values",+ [+ {+ "status": TaskStatus.WAITING_FOR_USER,+ "control_state": TaskControlState.WAITING_FOR_USER.value,+ },+ {"status": TaskStatus.PAUSED, "control_state": TaskControlState.PAUSED.value},+ ],+ ids=["waiting", "paused"],+)+def test_waiting_and_paused_tasks_are_cancelled(values):+ task_id, agent_id, key = _running_task()+ _update(task_id, runner_id=None, lease_expires_at=None, **values)+ before = _row(task_id)++ response = _cancel(task_id, key)++ assert response.status_code == 200, response.text+ assert response.json() == {+ "task_id": task_id,+ "status": "failed",+ "run_id": before["run_id"],+ "state_version": before["state_version"] + 1,+ "cancelled_run_id": before["run_id"],+ "error_code": "task_cancelled",+ }+ read = _get(task_id, key)+ assert (read["status"], read["error_code"]) == ("failed", "task_cancelled")+ assert read["pending_interaction"] is None+ # An answer to the cancelled question has nothing to resume.+ reply = client.post(+ f"/v1/chat/tasks/{task_id}/reply",+ headers=_bearer(key),+ json={+ "agent_id": agent_id,+ "command_id": "reply-after-cancel",+ "message": {"role": "user", "content": "yes"},+ },+ )+ assert reply.status_code == 409, reply.text+ assert _error(reply) == "no_pending_interaction"+++_WAITING_OR_PAUSED = {+ "waiting": {+ "status": TaskStatus.WAITING_FOR_USER,+ "control_state": TaskControlState.WAITING_FOR_USER.value,+ },+ "paused": {+ "status": TaskStatus.PAUSED,+ "control_state": TaskControlState.PAUSED.value,+ },+}+++def _waiting_or_paused_task(state: str) -> tuple[int, int, str]:+ task_id, agent_id, key = _running_task()+ _update(task_id, runner_id=None, lease_expires_at=None, **_WAITING_OR_PAUSED[state])+ return task_id, agent_id, key++[email protected]("state", list(_WAITING_OR_PAUSED))+ "outcome", ["unapplied", "resumed", "ended_stale_run", "ended_deadline"]+)+def test_how_a_cancel_of_a_waiting_or_paused_run_is_answered(+ monkeypatch: pytest.MonkeyPatch, state: str, outcome: str+):+ """The answer follows the aimed waiting or paused run:++ - unapplied: the command was not applied by the deadline, 202.+ - resumed: an answer or an automatic resume took the run first; the+ resumed run is still the aimed one and can be cancelled by retrying,+ so 409 ``task_busy``.+ - ended_*: the run resumed and completed before the command was+ applied; it ended without a cancel, whichever way the wait ends, so+ 409 ``task_not_cancellable``.+ """+ task_id, _agent_id, key = _waiting_or_paused_task(state)+ before = _row(task_id)+ after: list[dict[str, Any]] = []++ def resume_the_run() -> None:+ _update(+ task_id,+ status=TaskStatus.RUNNING,+ control_state=TaskControlState.RUNNING.value,+ state_version=before["state_version"] + 1,+ )+ after.append(_row(task_id))++ def end_the_run() -> None:+ _update(task_id, **_ENDED_ROWS["completed"])+ after.append(_row(task_id))++ if outcome == "unapplied":+ monkeypatch.setattr(sdk_task_cancel, "dispatch_one_task_command", AsyncMock())+ monkeypatch.setattr(sdk_task_cancel, "SDK_CANCEL_REQUEST_WAIT_SECONDS", 0.1)+ elif outcome == "ended_deadline":++ async def ended_while_the_command_waits(*_args: Any, **_kwargs: Any) -> bool:+ end_the_run()+ return False++ monkeypatch.setattr(+ sdk_task_cancel,+ "dispatch_one_task_command",+ AsyncMock(side_effect=ended_while_the_command_waits),+ )+ monkeypatch.setattr(sdk_task_cancel, "SDK_CANCEL_REQUEST_WAIT_SECONDS", 0.1)+ else:+ moves = resume_the_run if outcome == "resumed" else end_the_run++ async def moved_before_it_was_applied(**_kwargs: Any) -> None:+ moves()+ raise TaskCommandRejected("moved", reason="stale_run")++ _replace_cancel_core(monkeypatch, moved_before_it_was_applied)++ response = _cancel(task_id, key)++ if outcome == "unapplied":+ assert response.status_code == 202, response.text+ assert response.json() == {+ "task_id": task_id,+ "status": before["status"].value,+ "run_id": before["run_id"],+ "state_version": before["state_version"],+ "cancelled_run_id": None,+ "error_code": None,+ }+ assert _row(task_id) == before+ assert len(_cancel_commands(task_id)) == 1+ return+ assert response.status_code == 409, response.text+ if outcome == "resumed":+ assert _error(response) == "task_busy"+ assert _row(task_id) == after[0]+ return+ assert _error(response) == "task_not_cancellable"+ assert _row(task_id) == after[0]+ assert len(_cancel_commands(task_id)) == 1diff --git a/tests/web/services/test_sdk_cancel_paused.py b/tests/web/services/test_sdk_cancel_paused.pynew file mode 100644index 000000000..291348149--- /dev/null+++ b/tests/web/services/test_sdk_cancel_paused.py@@ -0,0 +1,119 @@+"""Cancelling a paused SDK run stops the automatic resume it waits for.++Uses the automatic-resume fixtures: a paused task resting on ``RUN`` with+its recovery row recorded at ``PV``.+"""++# Pytest fixture imports are intentionally shadowed by test parameters.+# ruff: noqa: F401, F811++from __future__ import annotations++import pytest++from tests.web.services.task_database_shared import engine+from tests.web.services.test_channel_reply_hold import PV, RUN, _seed, factory+from xagent.web.models.agent import Agent+from xagent.web.models.task import Task, TaskStatus+from xagent.web.models.task_auto_recovery import TaskAutoRecovery+from xagent.web.services import sdk_task_cancel, task_auto_resume+from xagent.web.services.sdk_task_cancel import SDK_CANCEL_ERROR_MESSAGE+from xagent.web.services.task_auto_resume import AutoResumeCancel+from xagent.web.services.task_execution_controller import StaleTaskRunError++[email protected](autouse=True)+def _own_sessions(factory, monkeypatch):+ monkeypatch.setattr(sdk_task_cancel, "get_session_local", lambda: factory)+ monkeypatch.setattr(+ task_auto_resume, "notify_task_command_dispatcher", lambda: None+ )+++def _paused_sdk_task(factory, **seed) -> tuple[int, int]:+ with factory() as db:+ task_id = _seed(db, **seed)+ task = db.get(Task, task_id)+ agent = Agent(user_id=task.user_id, name="Paused cancel")+ db.add(agent)+ db.flush()+ task.agent_id = agent.id+ task.source = "sdk"+ db.commit()+ return task_id, int(agent.id)+++def _finalize(task_id: int, agent_id: int) -> bool:+ return sdk_task_cancel._finalize_sdk_cancel_sync(+ task_id=task_id,+ agent_id=agent_id,+ expected_run_id=RUN,+ expected_state_version=PV,+ expected_status=TaskStatus.PAUSED,+ terminal_event_state={},+ )+++ ("state", "final_version", "row_after"),+ [+ # The pending resume is stopped first, which moves the version once.+ ("scheduled", PV + 2, ("manual", "user_cancelled")),+ ("dispatched", PV + 2, ("manual", "user_cancelled")),+ # Nothing is pending: the row is left as it is.+ ("exhausted", PV + 1, ("exhausted", None)),+ (None, PV + 1, None),+ ],+ ids=["scheduled", "dispatched", "gave_up", "no_recovery_row"],+)+def test_paused_run_is_cancelled_and_its_automatic_resume_stopped(+ factory, state, final_version, row_after+):+ task_id, agent_id = _paused_sdk_task(factory, state=state)++ assert _finalize(task_id, agent_id)++ with factory() as db:+ task = db.get(Task, task_id)+ row = db.get(TaskAutoRecovery, task_id)+ assert (task.status, task.control_state) == (TaskStatus.FAILED, "failed")+ assert task.error_message == SDK_CANCEL_ERROR_MESSAGE+ assert (task.run_id, int(task.state_version)) == (RUN, final_version)+ assert (None if row is None else (row.state, row.state_detail)) == row_after+ if row is not None and state in ("scheduled", "dispatched"):+ # Nothing schedules the stopped row again.+ assert row.next_attempt_at is None+++def test_a_stop_that_lost_its_race_writes_nothing(factory, monkeypatch):+ task_id, agent_id = _paused_sdk_task(factory, state="scheduled")+ monkeypatch.setattr(+ task_auto_resume,+ "cancel_pending_auto_resume_no_commit",+ lambda *_args, **_kwargs: AutoResumeCancel(cancelled=False, raced=True),+ )++ with pytest.raises(StaleTaskRunError):+ _finalize(task_id, agent_id)++ with factory() as db:+ task = db.get(Task, task_id)+ assert (task.status, int(task.state_version)) == (TaskStatus.PAUSED, PV)+ assert task.error_message is None+++def test_a_replay_after_the_resume_was_stopped_finds_its_own_cancel(factory):+ """Stopping the resume moved the version once more, so the cancelled row+ is two past the command's version; a replay still reports it settled+ instead of rejecting a changed run."""+ task_id, agent_id = _paused_sdk_task(factory, state="scheduled")+ assert _finalize(task_id, agent_id)++ assert not _finalize(task_id, agent_id)++ with factory() as db:+ task = db.get(Task, task_id)+ assert (task.error_message, int(task.state_version)) == (+ SDK_CANCEL_ERROR_MESSAGE,+ PV + 2,+ )