Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
Python agentagentic-aiagentic-frameworkagentic-workflowaiautomationclaudedeepseekgenaigptllmlow-codemcpnextjsno-codeopenaipythonskillsworkflow
fix(dify-agent): make run cancellation route-independent (#39942)
盐粒 Yanlidc5192ba11e4
diff --git a/dify-agent/docs/dify-agent/guide/index.md b/dify-agent/docs/dify-agent/guide/index.mdindex c7a37b5bb4d..129568a5643 100644--- a/dify-agent/docs/dify-agent/guide/index.md+++ b/dify-agent/docs/dify-agent/guide/index.md@@ -248,10 +248,11 @@ or worker handoff.Horizontal scaling is possible by running multiple API processes against the sameRedis prefix, but each process executes only the runs it accepted. Redis providesshared status/event visibility, not load balancing or queued-job recovery. The-cancel endpoint can only signal a task owned by the process that receives the-request; a request routed to a different process returns `409` while the run is-still running. Retrying a cancellation after the run is already `cancelled` is-idempotent.+cancel endpoint can atomically accept a running run on any process. The process+that owns the runner observes the shared `run_cancelled` event, then cancels and+cleans up its local task. The HTTP response confirms that logical cancellation is+durable; local runner cleanup may still be in progress. Retrying a cancellation+after the run is already `cancelled` is idempotent.Atomic terminal finalization currently assumes the configured Redis URL targetsone Redis deployment that can execute both run keys in a Lua script. The existing@@ -259,8 +260,11 @@ record and event key names are unchanged and do not contain a shared RedisCluster hash tag, so Redis Cluster is not supported for this transition. Duringa rolling upgrade, older processes can still use the former split event/statuswrites; treat the single-terminal invariant as active only after those processes-have exited. Operators should then alert on more than one terminal event per run-and on disagreement between the run record status and terminal event type.+have exited. Deploy atomic terminal finalization everywhere first, then ensure+every process that can own a runner has the cancellation observer before relying+on route-independent cancellation. Operators should then alert on more than one+terminal event per run and on disagreement between the run record status and+terminal event type.## Run inputs and session snapshots@@ -301,9 +305,9 @@ progress:- `POST /runs` creates a running run and schedules it locally.- `GET /runs/{run_id}` returns `running`, `succeeded`, `failed`, or `cancelled`.-- `POST /runs/{run_id}/cancel` atomically accepts cancellation for a locally- owned running task and emits `run_cancelled`; it returns `409` for a non-local- running task or a run whose success/failure terminal already won.+- `POST /runs/{run_id}/cancel` atomically accepts cancellation on any API process+ and emits `run_cancelled`; it returns `409` only when a success/failure terminal+ already won. Runner cleanup continues asynchronously on the owner process.- `GET /runs/{run_id}/events` polls the Redis Stream event log with `after` and`next_cursor` cursors.- `GET /runs/{run_id}/events/sse` replays and streams events over SSE. The SSEdiff --git a/dify-agent/src/dify_agent/client/_client.py b/dify-agent/src/dify_agent/client/_client.pyindex 084fb52dbd8..a584555ed34 100644--- a/dify-agent/src/dify_agent/client/_client.py+++ b/dify-agent/src/dify_agent/client/_client.py@@ -383,8 +383,8 @@ class Client:async def cancel_run(self, run_id: str, request: CancelRunRequest | None = None) -> CancelRunResponse:"""Request explicit cancellation for ``run_id``.- The server may accept cancellation only for active runs; unsupported- deployments return an HTTP error rather than overloading ``run_failed``.+ Acceptance atomically persists the cancelled state. The process executing+ the run observes that state and performs runner cleanup asynchronously."""request_model = request or CancelRunRequest()try:diff --git a/dify-agent/src/dify_agent/runtime/run_scheduler.py b/dify-agent/src/dify_agent/runtime/run_scheduler.pyindex bff004219a7..8925edee292 100644--- a/dify-agent/src/dify_agent/runtime/run_scheduler.py+++ b/dify-agent/src/dify_agent/runtime/run_scheduler.py@@ -1,10 +1,11 @@"""In-process scheduling for Dify Agent runs.The scheduler is intentionally process-local: it persists a run record, starts an-``asyncio.Task`` for ``AgentRunRunner.run()``, and keeps only a transient active-task registry. Redis remains the durable source for status and event streams, but-there is no Redis job queue or cross-process handoff. If the process crashes,-currently active runs are lost until an external operator marks or retries them.+``asyncio.Task`` supervisor for the local runner and cancellation observer, and+keeps only a transient active task registry. Redis remains the durable source for+status and event streams, but there is no Redis job queue or cross-process+handoff. If the process crashes, currently active runs are lost until an external+operator marks or retries them.Create-run requests are accepted once the scheduler is not stopping and storagecan persist the run record. Request-shaped execution failures are left to``AgentRunRunner`` so bad compositions, ``on_exit`` policies, prompts,@@ -34,7 +35,7 @@ class SchedulerStoppingError(RuntimeError):class RunCancellationConflictError(RuntimeError):- """Raised when a run exists but can no longer be cancelled by this scheduler."""+ """Raised when a run exists but a different terminal state already won."""class RunStore(RunEventSink, Protocol):@@ -44,8 +45,8 @@ class RunStore(RunEventSink, Protocol):"""Persist a new run record and return it with status ``running``."""...- async def get_run(self, run_id: str) -> RunRecord:- """Return the latest persisted run record."""+ async def wait_for_cancellation(self, run_id: str) -> bool:+ """Wait for a terminal state and report whether cancellation won."""...@@ -63,19 +64,20 @@ type RunRunnerFactory = Callable[[RunRecord, CreateRunRequest], RunnableRun]class RunScheduler:"""Owns process-local run tasks and best-effort graceful shutdown.- ``active_tasks`` is mutated only on the event loop that calls ``create_run``- and ``shutdown``. The task registry is not durable; it exists so the lifespan- hook can wait for in-flight work and mark cancelled runs failed before Redis is- closed. A lock guards the stopping flag, run persistence, and task- registration so shutdown cannot begin after a request is admitted.+ ``active_tasks`` contains local supervisor tasks and is mutated only on the+ event loop that calls ``create_run`` and ``shutdown``. The task registry is not+ durable; it exists so the lifespan hook can wait for in-flight work and mark+ shutdown-cancelled runs failed before Redis is closed. It is not consulted+ when accepting cancellation requests. A lock guards the stopping flag, run+ persistence, and task registration so shutdown cannot begin after a request+ is admitted."""store: RunStoreshutdown_grace_seconds: floatactive_tasks: dict[str, asyncio.Task[None]]- cancelled_run_ids: set[str]stopping: bool- runner_factory: RunRunnerFactory+ runner_factory: RunRunnerFactory | Nonelayer_providers: tuple[LayerProviderInput, ...]plugin_daemon_http_client: httpx.AsyncClientdify_api_http_client: httpx.AsyncClient@@ -94,12 +96,11 @@ class RunScheduler:self.store = storeself.shutdown_grace_seconds = shutdown_grace_secondsself.active_tasks = {}- self.cancelled_run_ids = set()self.stopping = Falseself.plugin_daemon_http_client = plugin_daemon_http_clientself.dify_api_http_client = dify_api_http_clientself.layer_providers = layer_providers if layer_providers is not None else create_default_layer_providers()- self.runner_factory = runner_factory or self._default_runner_factory+ self.runner_factory = runner_factoryself._lifecycle_lock = asyncio.Lock()async def create_run(self, request: CreateRunRequest) -> RunRecord:@@ -120,40 +121,15 @@ class RunScheduler:return recordasync def cancel_run(self, run_id: str, request: CancelRunRequest) -> CancelRunResponse:- """Cancel one active task and persist an idempotent cancelled terminal state."""- async with self._lifecycle_lock:- record = await self.store.get_run(run_id)- if record.status == "cancelled":- return CancelRunResponse(run_id=run_id, status="cancelled")- if record.status != "running":- raise RunCancellationConflictError(f"run already finished with status {record.status!r}")-- task = self.active_tasks.get(run_id)- if task is None:- raise RunCancellationConflictError("run is not active in this scheduler process")- finalization = await emit_run_cancelled(- self.store,- run_id=run_id,- reason=request.reason,- message=request.message,- )- if not finalization.applied:- if finalization.status == "cancelled":- return CancelRunResponse(run_id=run_id, status="cancelled")- raise RunCancellationConflictError(f"run already finished with status {finalization.status!r}")- self.cancelled_run_ids.add(run_id)- _ = task.cancel(request.message or request.reason)-- # Some model/tool stacks can consume one CancelledError. Re-inject it- # after the terminal state is durable without making the HTTP request- # wait for arbitrary third-party cleanup.- for _attempt in range(2):- if task.done():- break- _ = task.cancel(request.message or request.reason)- await asyncio.sleep(0)- if task.done():- self._discard_active_run(run_id)+ """Persist an idempotent cancellation without relying on local task ownership."""+ finalization = await emit_run_cancelled(+ self.store,+ run_id=run_id,+ reason=request.reason,+ message=request.message,+ )+ if finalization.status != "cancelled":+ raise RunCancellationConflictError(f"run already finished with status {finalization.status!r}")return CancelRunResponse(run_id=run_id, status="cancelled")async def shutdown(self) -> None:@@ -168,11 +144,7 @@ class RunScheduler:if not pending:return- pending_run_ids = [- run_id- for run_id, task in tasks_by_run_id.items()- if task in pending and run_id not in self.cancelled_run_ids- ]+ pending_run_ids = [run_id for run_id, task in tasks_by_run_id.items() if task in pending]for task in pending:_ = task.cancel()_ = await asyncio.gather(*pending, return_exceptions=True)@@ -180,15 +152,69 @@ class RunScheduler:await self._mark_cancelled_run_failed(run_id)async def _run_record(self, record: RunRecord, request: CreateRunRequest) -> None:- """Execute a stored run and log failures already reflected in events."""+ """Supervise one local runner and its durable cancellation observer."""+ cancel_requested = asyncio.Event()+ runner = self._create_runner(record, request, is_cancelled=cancel_requested.is_set)+ runner_task = asyncio.create_task(runner.run(), name=f"dify-agent-runner-{record.run_id}")+ observer_task = asyncio.create_task(+ self.store.wait_for_cancellation(record.run_id),+ name=f"dify-agent-cancellation-observer-{record.run_id}",+ )try:- await self.runner_factory(record, request).run()+ _ = await asyncio.wait((runner_task, observer_task), return_when=asyncio.FIRST_COMPLETED)+ if observer_task.done():+ try:+ cancellation_won = observer_task.result()+ except Exception as exc:+ cancel_requested.set()+ await self._cancel_and_wait(runner_task, reinject=True)+ _ = await emit_run_failed(+ self.store,+ run_id=record.run_id,+ error=f"run cancellation observer failed: {exc}",+ reason="cancellation_observer",+ )+ raise++ if cancellation_won:+ cancel_requested.set()+ await self._cancel_and_wait(runner_task, reinject=True)+ else:+ await runner_task+ else:+ await runner_taskexcept asyncio.CancelledError:+ cancel_requested.set()+ await self._cancel_and_wait(observer_task)+ await self._cancel_and_wait(runner_task, reinject=True)raiseexcept Exception:logger.exception("scheduled run failed", extra={"run_id": record.run_id})+ finally:+ await self._cancel_and_wait(observer_task)+ if not runner_task.done():+ cancel_requested.set()+ await self._cancel_and_wait(runner_task, reinject=True)- def _default_runner_factory(self, record: RunRecord, request: CreateRunRequest) -> RunnableRun:+ def _create_runner(+ self,+ record: RunRecord,+ request: CreateRunRequest,+ *,+ is_cancelled: Callable[[], bool],+ ) -> RunnableRun:+ """Create a runner while keeping injected test runners source-compatible."""+ if self.runner_factory is not None:+ return self.runner_factory(record, request)+ return self._default_runner_factory(record, request, is_cancelled=is_cancelled)++ def _default_runner_factory(+ self,+ record: RunRecord,+ request: CreateRunRequest,+ *,+ is_cancelled: Callable[[], bool],+ ) -> RunnableRun:"""Create the production runner for a stored run record."""return AgentRunRunner(sink=self.store,@@ -197,12 +223,24 @@ class RunScheduler:plugin_daemon_http_client=self.plugin_daemon_http_client,dify_api_http_client=self.dify_api_http_client,layer_providers=self.layer_providers,- is_cancelled=lambda: record.run_id in self.cancelled_run_ids,+ is_cancelled=is_cancelled,)def _discard_active_run(self, run_id: str) -> None:_ = self.active_tasks.pop(run_id, None)- self.cancelled_run_ids.discard(run_id)++ @staticmethod+ async def _cancel_and_wait(task: asyncio.Task[object], *, reinject: bool = False) -> None:+ """Cancel and reap a child task, with bounded reinjection for runners."""+ if not task.done():+ _ = task.cancel()+ if reinject:+ for _attempt in range(2):+ await asyncio.sleep(0)+ if task.done():+ break+ _ = task.cancel()+ _ = await asyncio.gather(task, return_exceptions=True)async def _mark_cancelled_run_failed(self, run_id: str) -> None:"""Best-effort failure event/status for shutdown-cancelled runs."""diff --git a/dify-agent/src/dify_agent/server/routes/runs.py b/dify-agent/src/dify_agent/server/routes/runs.pyindex 031678789bc..09e1fe3857b 100644--- a/dify-agent/src/dify_agent/server/routes/runs.py+++ b/dify-agent/src/dify_agent/server/routes/runs.py@@ -76,7 +76,7 @@ def create_runs_router(request: CancelRunRequest,scheduler: Annotated[RunScheduler, Depends(scheduler_dep)],) -> CancelRunResponse:- """Cancel a process-local run and publish its terminal event/status."""+ """Persist cancellation; the owner process observes it and stops its runner."""try:return await scheduler.cancel_run(run_id, request)except RunNotFoundError as exc:diff --git a/dify-agent/src/dify_agent/storage/redis_run_store.py b/dify-agent/src/dify_agent/storage/redis_run_store.pyindex 8e47be1f1ec..9afe26a40ea 100644--- a/dify-agent/src/dify_agent/storage/redis_run_store.py+++ b/dify-agent/src/dify_agent/storage/redis_run_store.py@@ -155,6 +155,32 @@ class RedisRunStore(RunEventSink):event_id=event_id,)+ async def wait_for_cancellation(self, run_id: str) -> bool:+ """Wait until cancellation or another terminal state wins for one run.++ The stream cursor is captured before reading the record so a terminal+ transition cannot fall between the initial status check and blocking+ stream read.+ """+ events_key = run_events_key(self.prefix, run_id)+ latest_events = await self.redis.xrevrange(events_key, count=1)+ cursor = _decode_redis_text(latest_events[0][0]) if latest_events else "0-0"+ record = await self.get_run(run_id)+ if record.status != "running":+ return record.status == "cancelled"++ while True:+ response = await self.redis.xread({events_key: cursor}, block=0, count=100)+ for _stream_name, entries in response:+ for raw_id, fields in entries:+ event = self._decode_event(run_id, raw_id, fields)+ if event.id is not None:+ cursor = event.id+ if event.type == "run_cancelled":+ return True+ if event.type in {"run_succeeded", "run_failed"}:+ return False+async def get_events(self, run_id: str, *, after: str = "0-0", limit: int = 100) -> RunEventsResponse:"""Read a bounded page of events after ``after`` cursor."""await self.get_run(run_id)diff --git a/dify-agent/tests/integration/dify_agent/storage/test_terminal_finalization.py b/dify-agent/tests/integration/dify_agent/storage/test_terminal_finalization.pyindex b500ebac3f8..790ac76f780 100644--- a/dify-agent/tests/integration/dify_agent/storage/test_terminal_finalization.py+++ b/dify-agent/tests/integration/dify_agent/storage/test_terminal_finalization.py@@ -1,4 +1,4 @@-"""Real-Redis contracts for atomic terminal run finalization."""+"""Real-Redis contracts for terminal finalization and cancellation observation."""import asynciofrom collections.abc import Iterator@@ -8,17 +8,22 @@ import subprocessimport timefrom uuid import uuid4+import httpximport pytestfrom redis.asyncio import Redisfrom agenton.compositor import CompositorSessionSnapshotfrom dify_agent.protocol.schemas import (+ CancelRunRequest,+ CreateRunRequest,RunCancelledEvent,RunCancelledEventData,+ RunComposition,RunSucceededEvent,RunSucceededEventData,)from dify_agent.runtime.event_sink import TerminalRunEvent, terminal_event_status_and_error+from dify_agent.runtime.run_scheduler import RunSchedulerfrom dify_agent.storage.redis_keys import run_events_key, run_record_keyfrom dify_agent.storage.redis_run_store import RedisRunStore@@ -31,7 +36,7 @@ def redis_url() -> Iterator[str]:"""Start an isolated Redis when the binary is available locally."""redis_server = shutil.which("redis-server")if redis_server is None:- pytest.skip("redis-server is required for the atomic finalization integration test")+ pytest.skip("redis-server is required for run terminal integration tests")with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe:probe.bind(("127.0.0.1", 0))@@ -131,3 +136,66 @@ def test_two_redis_clients_commit_exactly_one_matching_terminal(redis_url: str)await second_client.aclose()asyncio.run(scenario())+++def test_non_owner_scheduler_cancellation_stops_owner_runner(redis_url: str) -> None:+ class BlockingRunner:+ def __init__(self, *, started: asyncio.Event, stopped: asyncio.Event) -> None:+ self.started = started+ self.stopped = stopped++ async def run(self) -> None:+ self.started.set()+ try:+ await asyncio.Event().wait()+ finally:+ self.stopped.set()++ async def scenario() -> None:+ owner_client = Redis.from_url(redis_url)+ remote_client = Redis.from_url(redis_url)+ prefix = f"route-independent-cancellation-{uuid4().hex}"+ owner_store = RedisRunStore(owner_client, prefix=prefix, run_retention_seconds=60)+ remote_store = RedisRunStore(remote_client, prefix=prefix, run_retention_seconds=60)+ runner_started = asyncio.Event()+ runner_stopped = asyncio.Event()+ async with httpx.AsyncClient() as http_client:+ owner_scheduler = RunScheduler(+ store=owner_store,+ plugin_daemon_http_client=http_client,+ dify_api_http_client=http_client,+ runner_factory=lambda _record, _request: BlockingRunner(+ started=runner_started,+ stopped=runner_stopped,+ ),+ )+ remote_scheduler = RunScheduler(+ store=remote_store,+ plugin_daemon_http_client=http_client,+ dify_api_http_client=http_client,+ )+ try:+ record = await owner_scheduler.create_run(CreateRunRequest(composition=RunComposition(layers=[])))+ owner_task = owner_scheduler.active_tasks[record.run_id]+ await asyncio.wait_for(runner_started.wait(), timeout=1)++ response = await remote_scheduler.cancel_run(+ record.run_id,+ CancelRunRequest(reason="remote_cancel"),+ )++ assert response.status == "cancelled"+ assert remote_scheduler.active_tasks == {}+ await asyncio.wait_for(runner_stopped.wait(), timeout=1)+ await asyncio.wait_for(owner_task, timeout=1)+ persisted = await owner_store.get_run(record.run_id)+ events = await remote_store.get_events(record.run_id)+ assert persisted.status == "cancelled"+ assert [event.type for event in events.events] == ["run_cancelled"]+ finally:+ await owner_scheduler.shutdown()+ await remote_scheduler.shutdown()+ await owner_client.aclose()+ await remote_client.aclose()++ asyncio.run(scenario())diff --git a/dify-agent/tests/local/dify_agent/runtime/test_run_scheduler.py b/dify-agent/tests/local/dify_agent/runtime/test_run_scheduler.pyindex cb3990c4256..b38985b2732 100644--- a/dify-agent/tests/local/dify_agent/runtime/test_run_scheduler.py+++ b/dify-agent/tests/local/dify_agent/runtime/test_run_scheduler.py@@ -24,6 +24,7 @@ from dify_agent.runtime.event_sink import (NonTerminalRunEvent,RunFinalizationResult,TerminalRunEvent,+ emit_run_failed,emit_run_succeeded,terminal_event_status_and_error,)@@ -92,18 +93,21 @@ class FakeStore:events: dict[str, list[RunEvent]]statuses: dict[str, RunStatus]errors: dict[str, str | None]+ terminal_changes: dict[str, asyncio.Event]def __init__(self) -> None:self.records = {}self.events = defaultdict(list)self.statuses = {}self.errors = {}+ self.terminal_changes = {}async def create_run(self) -> RunRecord:run_id = f"run-{len(self.records) + 1}"record = RunRecord(run_id=run_id, status="running")self.records[run_id] = recordself.statuses[run_id] = "running"+ self.terminal_changes[run_id] = asyncio.Event()return recordasync def append_event(self, event: NonTerminalRunEvent) -> str:@@ -126,8 +130,14 @@ class FakeStore:self.events[event.run_id].append(event.model_copy(update={"id": event_id}))self.statuses[event.run_id] = statusself.errors[event.run_id] = error+ self.terminal_changes[event.run_id].set()return RunFinalizationResult(applied=True, status=status, event_id=event_id)+ async def wait_for_cancellation(self, run_id: str) -> bool:+ while self.statuses[run_id] == "running":+ await self.terminal_changes[run_id].wait()+ return self.statuses[run_id] == "cancelled"+class SlowCreateStore(FakeStore):create_started: asyncio.Event@@ -144,17 +154,69 @@ class SlowCreateStore(FakeStore):return await super().create_run()+class TrackingStore(FakeStore):+ observer_started: asyncio.Event+ observer_finished: asyncio.Event+ release_observer: asyncio.Event++ def __init__(self, *, pause_observer: bool = False) -> None:+ super().__init__()+ self.observer_started = asyncio.Event()+ self.observer_finished = asyncio.Event()+ self.release_observer = asyncio.Event()+ if not pause_observer:+ self.release_observer.set()++ async def wait_for_cancellation(self, run_id: str) -> bool:+ self.observer_started.set()+ try:+ await self.release_observer.wait()+ return await super().wait_for_cancellation(run_id)+ finally:+ self.observer_finished.set()+++class FailingObserverStore(FakeStore):+ fail_observer: asyncio.Event+ observer_finished: asyncio.Event++ def __init__(self, *, fail_observer: asyncio.Event) -> None:+ super().__init__()+ self.fail_observer = fail_observer+ self.observer_finished = asyncio.Event()++ async def wait_for_cancellation(self, run_id: str) -> bool:+ del run_id+ try:+ await self.fail_observer.wait()+ raise RuntimeError("redis read failed")+ finally:+ self.observer_finished.set()++class ControlledRunner:started: asyncio.Eventrelease: asyncio.Event+ finished: asyncio.Event | None- def __init__(self, *, started: asyncio.Event, release: asyncio.Event) -> None:+ def __init__(+ self,+ *,+ started: asyncio.Event,+ release: asyncio.Event,+ finished: asyncio.Event | None = None,+ ) -> None:self.started = startedself.release = release+ self.finished = finishedasync def run(self) -> None:_ = self.started.set()- await self.release.wait()+ try:+ await self.release.wait()+ finally:+ if self.finished is not None:+ self.finished.set()class SwallowOneCancellationRunner:@@ -201,27 +263,96 @@ class SuccessThenWaitRunner:class IgnoreCancellationThenSucceedRunner:- def __init__(self, *, store: FakeStore, run_id: str, started: asyncio.Event, release: asyncio.Event) -> None:+ def __init__(+ self,+ *,+ store: FakeStore,+ run_id: str,+ started: asyncio.Event,+ release: asyncio.Event,+ finished: asyncio.Event,+ ) -> None:+ self.store = store+ self.run_id = run_id+ self.started = started+ self.release = release+ self.finished = finished++ async def run(self) -> None:+ try:+ self.started.set()+ while not self.release.is_set():+ try:+ await self.release.wait()+ except asyncio.CancelledError:+ continue+ result = await emit_run_succeeded(+ self.store,+ run_id=self.run_id,+ output="late success",+ session_snapshot=CompositorSessionSnapshot(layers=[]),+ )+ assert result.applied is False+ assert result.status == "cancelled"+ finally:+ self.finished.set()+++class ReleaseThenSucceedRunner:+ def __init__(+ self,+ *,+ store: FakeStore,+ run_id: str,+ started: asyncio.Event,+ release: asyncio.Event,+ finished: asyncio.Event,+ ) -> None:self.store = storeself.run_id = run_idself.started = startedself.release = release+ self.finished = finishedasync def run(self) -> None:self.started.set()- while not self.release.is_set():- try:- await self.release.wait()- except asyncio.CancelledError:- continue- result = await emit_run_succeeded(- self.store,- run_id=self.run_id,- output="late success",- session_snapshot=CompositorSessionSnapshot(layers=[]),- )- assert result.applied is False- assert result.status == "cancelled"+ try:+ await self.release.wait()+ result = await emit_run_succeeded(+ self.store,+ run_id=self.run_id,+ output="done",+ session_snapshot=CompositorSessionSnapshot(layers=[]),+ )+ assert result.applied is True+ finally:+ self.finished.set()+++class CompetingFailureRunner:+ def __init__(+ self,+ *,+ store: FakeStore,+ run_id: str,+ started: asyncio.Event,+ release: asyncio.Event,+ failure_attempted: asyncio.Event,+ ) -> None:+ self.store = store+ self.run_id = run_id+ self.started = started+ self.release = release+ self.failure_attempted = failure_attempted++ async def run(self) -> None:+ self.started.set()+ try:+ await self.release.wait()+ except asyncio.CancelledError:+ pass+ _ = await emit_run_failed(self.store, run_id=self.run_id, error="runner failed", reason="model_error")+ self.failure_attempted.set()class FinalizeSuccessOnCancellationRunner:@@ -296,39 +427,91 @@ def test_shutdown_marks_unfinished_runs_failed_and_appends_event() -> None:asyncio.run(scenario())-def test_cancel_run_stops_task_and_persists_cancelled_terminal() -> None:+def test_cancellation_observer_failure_stops_runner_and_finalizes_failed() -> None:async def scenario() -> None:- store = FakeStore()- started = asyncio.Event()+ fail_observer = asyncio.Event()+ store = FailingObserverStore(fail_observer=fail_observer)+ runner_started = asyncio.Event()+ runner_finished = asyncio.Event()async with httpx.AsyncClient() as client:scheduler = RunScheduler(store=store,plugin_daemon_http_client=client,dify_api_http_client=client,- runner_factory=lambda _record, _request: ControlledRunner(started=started, release=asyncio.Event()),+ runner_factory=lambda _record, _request: ControlledRunner(+ started=runner_started,+ release=asyncio.Event(),+ finished=runner_finished,+ ),)record = await scheduler.create_run(_request())+ supervisor_task = scheduler.active_tasks[record.run_id]+ await asyncio.wait_for(runner_started.wait(), timeout=1)++ fail_observer.set()+ await asyncio.wait_for(supervisor_task, timeout=1)++ assert store.statuses[record.run_id] == "failed"+ assert store.errors[record.run_id] == "run cancellation observer failed: redis read failed"+ assert [event.type for event in store.events[record.run_id]] == ["run_failed"]+ assert runner_finished.is_set()+ assert store.observer_finished.is_set()+ await asyncio.sleep(0)+ assert scheduler.active_tasks == {}++ asyncio.run(scenario())+++def test_non_owner_cancel_run_stops_owner_task_and_persists_cancelled_terminal() -> None:+ async def scenario() -> None:+ store = TrackingStore()+ started = asyncio.Event()+ runner_finished = asyncio.Event()+ async with httpx.AsyncClient() as client:+ owner_scheduler = RunScheduler(+ store=store,+ plugin_daemon_http_client=client,+ dify_api_http_client=client,+ runner_factory=lambda _record, _request: ControlledRunner(+ started=started,+ release=asyncio.Event(),+ finished=runner_finished,+ ),+ )+ remote_scheduler = RunScheduler(+ store=store,+ plugin_daemon_http_client=client,+ dify_api_http_client=client,+ )+ record = await owner_scheduler.create_run(_request())+ owner_task = owner_scheduler.active_tasks[record.run_id]await asyncio.wait_for(started.wait(), timeout=1)+ await asyncio.wait_for(store.observer_started.wait(), timeout=1)- response = await scheduler.cancel_run(+ response = await remote_scheduler.cancel_run(record.run_id,CancelRunRequest(reason="workflow_aborted", message="outer workflow stopped"),)assert response.status == "cancelled"- assert scheduler.active_tasks == {}+ assert remote_scheduler.active_tasks == {}assert store.statuses[record.run_id] == "cancelled"assert store.errors[record.run_id] == "outer workflow stopped"assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]+ await asyncio.wait_for(owner_task, timeout=1)+ assert runner_finished.is_set()+ assert store.observer_finished.is_set()+ await asyncio.sleep(0)+ assert owner_scheduler.active_tasks == {}- repeated = await scheduler.cancel_run(record.run_id, CancelRunRequest(reason="duplicate"))+ repeated = await remote_scheduler.cancel_run(record.run_id, CancelRunRequest(reason="duplicate"))assert repeated.status == "cancelled"assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]asyncio.run(scenario())-def test_cancel_run_reinjects_cancellation_without_waiting_for_runner_cleanup() -> None:+def test_owner_observer_reinjects_cancellation_consumed_by_runner() -> None:async def scenario() -> None:store = FakeStore()started = asyncio.Event()@@ -344,6 +527,7 @@ def test_cancel_run_reinjects_cancellation_without_waiting_for_runner_cleanup()),)record = await scheduler.create_run(_request())+ supervisor_task = scheduler.active_tasks[record.run_id]await asyncio.wait_for(started.wait(), timeout=1)response = await asyncio.wait_for(@@ -352,9 +536,10 @@ def test_cancel_run_reinjects_cancellation_without_waiting_for_runner_cleanup())assert response.status == "cancelled"- assert first_cancellation.is_set()assert store.statuses[record.run_id] == "cancelled"assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]+ await asyncio.wait_for(first_cancellation.wait(), timeout=1)+ await asyncio.wait_for(supervisor_task, timeout=1)await asyncio.sleep(0)assert scheduler.active_tasks == {}@@ -394,40 +579,151 @@ def test_cancel_run_does_not_override_successful_terminal() -> None:asyncio.run(scenario())-def test_cancel_run_wins_before_a_runner_that_consumes_cancellation_finishes() -> None:+def test_cancelled_terminal_survives_shutdown_while_runner_cleanup_is_pending() -> None:async def scenario() -> None:- store = FakeStore()+ store = TrackingStore()started = asyncio.Event()release = asyncio.Event()+ runner_finished = asyncio.Event()async with httpx.AsyncClient() as client:scheduler = RunScheduler(store=store,plugin_daemon_http_client=client,dify_api_http_client=client,+ shutdown_grace_seconds=0,runner_factory=lambda record, _request: IgnoreCancellationThenSucceedRunner(store=store,run_id=record.run_id,started=started,release=release,+ finished=runner_finished,),)record = await scheduler.create_run(_request())+ supervisor_task = scheduler.active_tasks[record.run_id]await asyncio.wait_for(started.wait(), timeout=1)+ await asyncio.wait_for(store.observer_started.wait(), timeout=1)response = await scheduler.cancel_run(record.run_id, CancelRunRequest(reason="workflow_aborted"))assert response.status == "cancelled"assert store.statuses[record.run_id] == "cancelled"assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]- task = scheduler.active_tasks[record.run_id]- release.set()- await asyncio.wait_for(task, timeout=1)+ await asyncio.wait_for(store.observer_finished.wait(), timeout=1)+ assert supervisor_task.done() is False+ shutdown_task = asyncio.create_task(scheduler.shutdown())await asyncio.sleep(0)+ assert shutdown_task.done() is False+ release.set()+ await asyncio.wait_for(shutdown_task, timeout=1)++ assert supervisor_task.done()+ assert runner_finished.is_set()+ assert store.observer_finished.is_set()+ assert scheduler.active_tasks == {}+ assert store.statuses[record.run_id] == "cancelled"assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]asyncio.run(scenario())+ ("winner", "expected_event_type"),+ [+ pytest.param("failed", "run_failed", id="failure-first"),+ pytest.param("cancelled", "run_cancelled", id="cancellation-first"),+ ],+)+def test_failure_and_cancellation_keep_the_first_terminal(+ winner: RunStatus,+ expected_event_type: str,+) -> None:+ async def scenario() -> None:+ store = FakeStore()+ runner_started = asyncio.Event()+ release_runner = asyncio.Event()+ failure_attempted = asyncio.Event()+ async with httpx.AsyncClient() as client:+ scheduler = RunScheduler(+ store=store,+ plugin_daemon_http_client=client,+ dify_api_http_client=client,+ runner_factory=lambda record, _request: CompetingFailureRunner(+ store=store,+ run_id=record.run_id,+ started=runner_started,+ release=release_runner,+ failure_attempted=failure_attempted,+ ),+ )+ record = await scheduler.create_run(_request())+ supervisor_task = scheduler.active_tasks[record.run_id]+ await asyncio.wait_for(runner_started.wait(), timeout=1)++ if winner == "failed":+ release_runner.set()+ await asyncio.wait_for(failure_attempted.wait(), timeout=1)+ with pytest.raises(RunCancellationConflictError, match="already finished with status 'failed'"):+ await scheduler.cancel_run(record.run_id, CancelRunRequest(reason="late_cancel"))+ else:+ response = await scheduler.cancel_run(+ record.run_id,+ CancelRunRequest(reason="cancel_before_failure"),+ )+ assert response.run_id == record.run_id+ assert response.status == "cancelled"+ release_runner.set()++ await asyncio.wait_for(failure_attempted.wait(), timeout=1)+ await asyncio.wait_for(supervisor_task, timeout=1)++ assert store.statuses[record.run_id] == winner+ assert [event.type for event in store.events[record.run_id]] == [expected_event_type]++ asyncio.run(scenario())+++def test_shutdown_grace_allows_runner_first_completion_and_reaps_children() -> None:+ async def scenario() -> None:+ store = TrackingStore(pause_observer=True)+ runner_started = asyncio.Event()+ release_runner = asyncio.Event()+ runner_finished = asyncio.Event()+ async with httpx.AsyncClient() as client:+ scheduler = RunScheduler(+ store=store,+ plugin_daemon_http_client=client,+ dify_api_http_client=client,+ shutdown_grace_seconds=1,+ runner_factory=lambda record, _request: ReleaseThenSucceedRunner(+ store=store,+ run_id=record.run_id,+ started=runner_started,+ release=release_runner,+ finished=runner_finished,+ ),+ )+ record = await scheduler.create_run(_request())+ supervisor_task = scheduler.active_tasks[record.run_id]+ await asyncio.wait_for(runner_started.wait(), timeout=1)+ await asyncio.wait_for(store.observer_started.wait(), timeout=1)++ shutdown_task = asyncio.create_task(scheduler.shutdown())+ await asyncio.sleep(0)+ assert shutdown_task.done() is False+ release_runner.set()+ await asyncio.wait_for(shutdown_task, timeout=1)++ assert supervisor_task.done()+ assert runner_finished.is_set()+ assert store.observer_finished.is_set()+ assert scheduler.active_tasks == {}+ assert store.statuses[record.run_id] == "succeeded"+ assert [event.type for event in store.events[record.run_id]] == ["run_succeeded"]++ asyncio.run(scenario())++def test_shutdown_does_not_append_failed_after_success_wins() -> None:async def scenario() -> None:store = FakeStore()diff --git a/dify-agent/tests/local/dify_agent/server/test_runs_routes.py b/dify-agent/tests/local/dify_agent/server/test_runs_routes.pyindex d90f82a152a..7a8da51eb29 100644--- a/dify-agent/tests/local/dify_agent/server/test_runs_routes.py+++ b/dify-agent/tests/local/dify_agent/server/test_runs_routes.py@@ -4,6 +4,7 @@ from dify_agent.protocol import CancelRunResponse, DIFY_AGENT_MODEL_LAYER_IDfrom dify_agent.runtime.run_scheduler import RunCancellationConflictError, SchedulerStoppingErrorfrom dify_agent.server.routes.runs import create_runs_routerfrom dify_agent.server.schemas import RunRecord+from dify_agent.storage.redis_run_store import RunNotFoundErrorclass FakeScheduler:@@ -106,6 +107,26 @@ def test_cancel_run_endpoint_maps_conflict() -> None:assert "already finished" in response.json()["detail"]+def test_cancel_run_endpoint_maps_missing_run() -> None:+ from fastapi import FastAPI++ class MissingRunScheduler(FakeScheduler):+ async def cancel_run(self, run_id: str, request: object) -> CancelRunResponse:+ del request+ raise RunNotFoundError(run_id)++ app = FastAPI()+ app.include_router(+ create_runs_router(lambda: FakeStore(), lambda: MissingRunScheduler()) # pyright: ignore[reportArgumentType]+ )+ client = TestClient(app)++ response = client.post("/runs/missing/cancel", json={})++ assert response.status_code == 404+ assert response.json() == {"detail": "run not found"}++def test_create_run_accepts_valid_full_plugin_graph() -> None:from fastapi import FastAPIdiff --git a/dify-agent/tests/local/dify_agent/storage/test_redis_run_store.py b/dify-agent/tests/local/dify_agent/storage/test_redis_run_store.pyindex 5f22779da79..eb56e6ffc87 100644--- a/dify-agent/tests/local/dify_agent/storage/test_redis_run_store.py+++ b/dify-agent/tests/local/dify_agent/storage/test_redis_run_store.py@@ -31,6 +31,7 @@ class FakeRedis:self.commands = []self.values = {}self.streams = {}+ self.stream_changed = asyncio.Event()async def set(self, key: str, value: object, *, ex: int | None = None) -> None:self.commands.append(("set", key, value, ex))@@ -52,8 +53,46 @@ class FakeRedis:entries = self.streams.setdefault(key, [])event_id = f"{len(entries) + 1}-0"entries.append((event_id, dict(fields)))+ self.stream_changed.set()return event_id+ async def xrevrange(+ self,+ key: str,+ max: str = "+",+ min: str = "-",+ *,+ count: int | None = None,+ ) -> list[tuple[str, dict[str, object]]]:+ self.commands.append(("xrevrange", key, max, min, count))+ entries = list(reversed(self.streams.get(key, [])))+ return entries[:count] if count is not None else entries++ async def xread(+ self,+ streams: Mapping[str, str],+ *,+ count: int | None = None,+ block: int | None = None,+ ) -> list[tuple[str, list[tuple[str, dict[str, object]]]]]:+ self.commands.append(("xread", dict(streams), count, block))+ while True:+ response: list[tuple[str, list[tuple[str, dict[str, object]]]]] = []+ for key, cursor in streams.items():+ entries = [+ entry+ for entry in self.streams.get(key, [])+ if self._stream_id_value(entry[0]) > self._stream_id_value(cursor)+ ]+ if count is not None:+ entries = entries[:count]+ if entries:+ response.append((key, entries))+ if response:+ return response+ self.stream_changed.clear()+ await self.stream_changed.wait()+async def xrange(self, key: str, *, min: str = "-", count: int | None = None) -> list[tuple[str, dict[str, object]]]:@@ -311,6 +350,101 @@ def test_finalize_run_raises_when_record_is_missing() -> None:)+def test_wait_for_cancellation_observes_terminal_record_before_starting() -> None:+ redis = FakeRedis()+ store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]++ async def scenario() -> bool:+ record = await store.create_run()+ _ = await store.finalize_run(+ RunCancelledEvent(run_id=record.run_id, data=RunCancelledEventData(reason="cancelled"))+ )+ redis.commands.clear()+ return await store.wait_for_cancellation(record.run_id)++ assert asyncio.run(scenario()) is True+ assert [command[0] for command in redis.commands] == ["xrevrange", "get"]+++def test_wait_for_cancellation_covers_terminal_transition_during_initialization() -> None:+ class PausingRecordReadRedis(FakeRedis):+ record_read_started: asyncio.Event+ release_record_read: asyncio.Event+ pause_next_record_read: bool++ def __init__(self) -> None:+ super().__init__()+ self.record_read_started = asyncio.Event()+ self.release_record_read = asyncio.Event()+ self.pause_next_record_read = True++ async def get(self, key: str) -> object | None:+ if self.pause_next_record_read and key.endswith(":record"):+ self.pause_next_record_read = False+ self.record_read_started.set()+ await self.release_record_read.wait()+ return await super().get(key)++ redis = PausingRecordReadRedis()+ observer_store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]+ cancelling_store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]++ async def scenario() -> bool:+ record = await observer_store.create_run()+ observer = asyncio.create_task(observer_store.wait_for_cancellation(record.run_id))+ await asyncio.wait_for(redis.record_read_started.wait(), timeout=1)+ _ = await cancelling_store.finalize_run(+ RunCancelledEvent(run_id=record.run_id, data=RunCancelledEventData(reason="cancelled"))+ )+ redis.release_record_read.set()+ return await asyncio.wait_for(observer, timeout=1)++ assert asyncio.run(scenario()) is True+++def test_wait_for_cancellation_advances_past_non_terminal_events() -> None:+ redis = FakeRedis()+ store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]++ async def scenario() -> bool:+ record = await store.create_run()+ observer = asyncio.create_task(store.wait_for_cancellation(record.run_id))+ await asyncio.sleep(0)+ _ = await store.append_event(RunStartedEvent(run_id=record.run_id))+ await asyncio.sleep(0)+ _ = await store.finalize_run(+ RunCancelledEvent(run_id=record.run_id, data=RunCancelledEventData(reason="cancelled"))+ )+ return await asyncio.wait_for(observer, timeout=1)++ assert asyncio.run(scenario()) is True+ cursors = [command[1] for command in redis.commands if command[0] == "xread"]+ assert any("0-0" in streams.values() for streams in cursors if isinstance(streams, dict))+ assert any("1-0" in streams.values() for streams in cursors if isinstance(streams, dict))+++def test_wait_for_cancellation_returns_false_when_success_wins() -> None:+ redis = FakeRedis()+ store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]++ async def scenario() -> bool:+ record = await store.create_run()+ observer = asyncio.create_task(store.wait_for_cancellation(record.run_id))+ await asyncio.sleep(0)+ _ = await store.finalize_run(+ RunSucceededEvent(+ run_id=record.run_id,+ data=RunSucceededEventData(+ output="done",+ session_snapshot=CompositorSessionSnapshot(layers=[]),+ ),+ )+ )+ return await asyncio.wait_for(observer, timeout=1)++ assert asyncio.run(scenario()) is False++def test_append_event_serializes_typed_event_without_id_and_expires_run_keys() -> None:redis = FakeRedis()store = RedisRunStore(redis, prefix="test", run_retention_seconds=60) # pyright: ignore[reportArgumentType]