Skip to content

armgit / dify

Public Imported from GitHub
0

Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.

fix(dify-agent): make run cancellation route-independent (#39942)

盐粒 Yanlidc5192ba11e4
diff --git a/dify-agent/docs/dify-agent/guide/index.md b/dify-agent/docs/dify-agent/guide/index.md
index c7a37b5bb4d..129568a5643 100644
--- a/dify-agent/docs/dify-agent/guide/index.md
+++ b/dify-agent/docs/dify-agent/guide/index.md
@@ -248,10 +248,11 @@ or worker handoff.
Horizontal scaling is possible by running multiple API processes against the same
Redis prefix, but each process executes only the runs it accepted. Redis provides
shared status/event visibility, not load balancing or queued-job recovery. The
-cancel endpoint can only signal a task owned by the process that receives the
-request; a request routed to a different process returns `409` while the run is
-still running. Retrying a cancellation after the run is already `cancelled` is
-idempotent.
+cancel endpoint can atomically accept a running run on any process. The process
+that owns the runner observes the shared `run_cancelled` event, then cancels and
+cleans up its local task. The HTTP response confirms that logical cancellation is
+durable; local runner cleanup may still be in progress. Retrying a cancellation
+after the run is already `cancelled` is idempotent.
Atomic terminal finalization currently assumes the configured Redis URL targets
one Redis deployment that can execute both run keys in a Lua script. The existing
@@ -259,8 +260,11 @@ record and event key names are unchanged and do not contain a shared Redis
Cluster hash tag, so Redis Cluster is not supported for this transition. During
a rolling upgrade, older processes can still use the former split event/status
writes; treat the single-terminal invariant as active only after those processes
-have exited. Operators should then alert on more than one terminal event per run
-and on disagreement between the run record status and terminal event type.
+have exited. Deploy atomic terminal finalization everywhere first, then ensure
+every process that can own a runner has the cancellation observer before relying
+on route-independent cancellation. Operators should then alert on more than one
+terminal event per run and on disagreement between the run record status and
+terminal event type.
## Run inputs and session snapshots
@@ -301,9 +305,9 @@ progress:
- `POST /runs` creates a running run and schedules it locally.
- `GET /runs/{run_id}` returns `running`, `succeeded`, `failed`, or `cancelled`.
-- `POST /runs/{run_id}/cancel` atomically accepts cancellation for a locally
- owned running task and emits `run_cancelled`; it returns `409` for a non-local
- running task or a run whose success/failure terminal already won.
+- `POST /runs/{run_id}/cancel` atomically accepts cancellation on any API process
+ and emits `run_cancelled`; it returns `409` only when a success/failure terminal
+ already won. Runner cleanup continues asynchronously on the owner process.
- `GET /runs/{run_id}/events` polls the Redis Stream event log with `after` and
`next_cursor` cursors.
- `GET /runs/{run_id}/events/sse` replays and streams events over SSE. The SSE
diff --git a/dify-agent/src/dify_agent/client/_client.py b/dify-agent/src/dify_agent/client/_client.py
index 084fb52dbd8..a584555ed34 100644
--- a/dify-agent/src/dify_agent/client/_client.py
+++ b/dify-agent/src/dify_agent/client/_client.py
@@ -383,8 +383,8 @@ class Client:
async def cancel_run(self, run_id: str, request: CancelRunRequest | None = None) -> CancelRunResponse:
"""Request explicit cancellation for ``run_id``.
- The server may accept cancellation only for active runs; unsupported
- deployments return an HTTP error rather than overloading ``run_failed``.
+ Acceptance atomically persists the cancelled state. The process executing
+ the run observes that state and performs runner cleanup asynchronously.
"""
request_model = request or CancelRunRequest()
try:
diff --git a/dify-agent/src/dify_agent/runtime/run_scheduler.py b/dify-agent/src/dify_agent/runtime/run_scheduler.py
index bff004219a7..8925edee292 100644
--- a/dify-agent/src/dify_agent/runtime/run_scheduler.py
+++ b/dify-agent/src/dify_agent/runtime/run_scheduler.py
@@ -1,10 +1,11 @@
"""In-process scheduling for Dify Agent runs.
The scheduler is intentionally process-local: it persists a run record, starts an
-``asyncio.Task`` for ``AgentRunRunner.run()``, and keeps only a transient active
-task registry. Redis remains the durable source for status and event streams, but
-there is no Redis job queue or cross-process handoff. If the process crashes,
-currently active runs are lost until an external operator marks or retries them.
+``asyncio.Task`` supervisor for the local runner and cancellation observer, and
+keeps only a transient active task registry. Redis remains the durable source for
+status and event streams, but there is no Redis job queue or cross-process
+handoff. If the process crashes, currently active runs are lost until an external
+operator marks or retries them.
Create-run requests are accepted once the scheduler is not stopping and storage
can persist the run record. Request-shaped execution failures are left to
``AgentRunRunner`` so bad compositions, ``on_exit`` policies, prompts,
@@ -34,7 +35,7 @@ class SchedulerStoppingError(RuntimeError):
class RunCancellationConflictError(RuntimeError):
- """Raised when a run exists but can no longer be cancelled by this scheduler."""
+ """Raised when a run exists but a different terminal state already won."""
class RunStore(RunEventSink, Protocol):
@@ -44,8 +45,8 @@ class RunStore(RunEventSink, Protocol):
"""Persist a new run record and return it with status ``running``."""
...
- async def get_run(self, run_id: str) -> RunRecord:
- """Return the latest persisted run record."""
+ async def wait_for_cancellation(self, run_id: str) -> bool:
+ """Wait for a terminal state and report whether cancellation won."""
...
@@ -63,19 +64,20 @@ type RunRunnerFactory = Callable[[RunRecord, CreateRunRequest], RunnableRun]
class RunScheduler:
"""Owns process-local run tasks and best-effort graceful shutdown.
- ``active_tasks`` is mutated only on the event loop that calls ``create_run``
- and ``shutdown``. The task registry is not durable; it exists so the lifespan
- hook can wait for in-flight work and mark cancelled runs failed before Redis is
- closed. A lock guards the stopping flag, run persistence, and task
- registration so shutdown cannot begin after a request is admitted.
+ ``active_tasks`` contains local supervisor tasks and is mutated only on the
+ event loop that calls ``create_run`` and ``shutdown``. The task registry is not
+ durable; it exists so the lifespan hook can wait for in-flight work and mark
+ shutdown-cancelled runs failed before Redis is closed. It is not consulted
+ when accepting cancellation requests. A lock guards the stopping flag, run
+ persistence, and task registration so shutdown cannot begin after a request
+ is admitted.
"""
store: RunStore
shutdown_grace_seconds: float
active_tasks: dict[str, asyncio.Task[None]]
- cancelled_run_ids: set[str]
stopping: bool
- runner_factory: RunRunnerFactory
+ runner_factory: RunRunnerFactory | None
layer_providers: tuple[LayerProviderInput, ...]
plugin_daemon_http_client: httpx.AsyncClient
dify_api_http_client: httpx.AsyncClient
@@ -94,12 +96,11 @@ class RunScheduler:
self.store = store
self.shutdown_grace_seconds = shutdown_grace_seconds
self.active_tasks = {}
- self.cancelled_run_ids = set()
self.stopping = False
self.plugin_daemon_http_client = plugin_daemon_http_client
self.dify_api_http_client = dify_api_http_client
self.layer_providers = layer_providers if layer_providers is not None else create_default_layer_providers()
- self.runner_factory = runner_factory or self._default_runner_factory
+ self.runner_factory = runner_factory
self._lifecycle_lock = asyncio.Lock()
async def create_run(self, request: CreateRunRequest) -> RunRecord:
@@ -120,40 +121,15 @@ class RunScheduler:
return record
async def cancel_run(self, run_id: str, request: CancelRunRequest) -> CancelRunResponse:
- """Cancel one active task and persist an idempotent cancelled terminal state."""
- async with self._lifecycle_lock:
- record = await self.store.get_run(run_id)
- if record.status == "cancelled":
- return CancelRunResponse(run_id=run_id, status="cancelled")
- if record.status != "running":
- raise RunCancellationConflictError(f"run already finished with status {record.status!r}")
-
- task = self.active_tasks.get(run_id)
- if task is None:
- raise RunCancellationConflictError("run is not active in this scheduler process")
- finalization = await emit_run_cancelled(
- self.store,
- run_id=run_id,
- reason=request.reason,
- message=request.message,
- )
- if not finalization.applied:
- if finalization.status == "cancelled":
- return CancelRunResponse(run_id=run_id, status="cancelled")
- raise RunCancellationConflictError(f"run already finished with status {finalization.status!r}")
- self.cancelled_run_ids.add(run_id)
- _ = task.cancel(request.message or request.reason)
-
- # Some model/tool stacks can consume one CancelledError. Re-inject it
- # after the terminal state is durable without making the HTTP request
- # wait for arbitrary third-party cleanup.
- for _attempt in range(2):
- if task.done():
- break
- _ = task.cancel(request.message or request.reason)
- await asyncio.sleep(0)
- if task.done():
- self._discard_active_run(run_id)
+ """Persist an idempotent cancellation without relying on local task ownership."""
+ finalization = await emit_run_cancelled(
+ self.store,
+ run_id=run_id,
+ reason=request.reason,
+ message=request.message,
+ )
+ if finalization.status != "cancelled":
+ raise RunCancellationConflictError(f"run already finished with status {finalization.status!r}")
return CancelRunResponse(run_id=run_id, status="cancelled")
async def shutdown(self) -> None:
@@ -168,11 +144,7 @@ class RunScheduler:
if not pending:
return
- pending_run_ids = [
- run_id
- for run_id, task in tasks_by_run_id.items()
- if task in pending and run_id not in self.cancelled_run_ids
- ]
+ pending_run_ids = [run_id for run_id, task in tasks_by_run_id.items() if task in pending]
for task in pending:
_ = task.cancel()
_ = await asyncio.gather(*pending, return_exceptions=True)
@@ -180,15 +152,69 @@ class RunScheduler:
await self._mark_cancelled_run_failed(run_id)
async def _run_record(self, record: RunRecord, request: CreateRunRequest) -> None:
- """Execute a stored run and log failures already reflected in events."""
+ """Supervise one local runner and its durable cancellation observer."""
+ cancel_requested = asyncio.Event()
+ runner = self._create_runner(record, request, is_cancelled=cancel_requested.is_set)
+ runner_task = asyncio.create_task(runner.run(), name=f"dify-agent-runner-{record.run_id}")
+ observer_task = asyncio.create_task(
+ self.store.wait_for_cancellation(record.run_id),
+ name=f"dify-agent-cancellation-observer-{record.run_id}",
+ )
try:
- await self.runner_factory(record, request).run()
+ _ = await asyncio.wait((runner_task, observer_task), return_when=asyncio.FIRST_COMPLETED)
+ if observer_task.done():
+ try:
+ cancellation_won = observer_task.result()
+ except Exception as exc:
+ cancel_requested.set()
+ await self._cancel_and_wait(runner_task, reinject=True)
+ _ = await emit_run_failed(
+ self.store,
+ run_id=record.run_id,
+ error=f"run cancellation observer failed: {exc}",
+ reason="cancellation_observer",
+ )
+ raise
+
+ if cancellation_won:
+ cancel_requested.set()
+ await self._cancel_and_wait(runner_task, reinject=True)
+ else:
+ await runner_task
+ else:
+ await runner_task
except asyncio.CancelledError:
+ cancel_requested.set()
+ await self._cancel_and_wait(observer_task)
+ await self._cancel_and_wait(runner_task, reinject=True)
raise
except Exception:
logger.exception("scheduled run failed", extra={"run_id": record.run_id})
+ finally:
+ await self._cancel_and_wait(observer_task)
+ if not runner_task.done():
+ cancel_requested.set()
+ await self._cancel_and_wait(runner_task, reinject=True)
- def _default_runner_factory(self, record: RunRecord, request: CreateRunRequest) -> RunnableRun:
+ def _create_runner(
+ self,
+ record: RunRecord,
+ request: CreateRunRequest,
+ *,
+ is_cancelled: Callable[[], bool],
+ ) -> RunnableRun:
+ """Create a runner while keeping injected test runners source-compatible."""
+ if self.runner_factory is not None:
+ return self.runner_factory(record, request)
+ return self._default_runner_factory(record, request, is_cancelled=is_cancelled)
+
+ def _default_runner_factory(
+ self,
+ record: RunRecord,
+ request: CreateRunRequest,
+ *,
+ is_cancelled: Callable[[], bool],
+ ) -> RunnableRun:
"""Create the production runner for a stored run record."""
return AgentRunRunner(
sink=self.store,
@@ -197,12 +223,24 @@ class RunScheduler:
plugin_daemon_http_client=self.plugin_daemon_http_client,
dify_api_http_client=self.dify_api_http_client,
layer_providers=self.layer_providers,
- is_cancelled=lambda: record.run_id in self.cancelled_run_ids,
+ is_cancelled=is_cancelled,
)
def _discard_active_run(self, run_id: str) -> None:
_ = self.active_tasks.pop(run_id, None)
- self.cancelled_run_ids.discard(run_id)
+
+ @staticmethod
+ async def _cancel_and_wait(task: asyncio.Task[object], *, reinject: bool = False) -> None:
+ """Cancel and reap a child task, with bounded reinjection for runners."""
+ if not task.done():
+ _ = task.cancel()
+ if reinject:
+ for _attempt in range(2):
+ await asyncio.sleep(0)
+ if task.done():
+ break
+ _ = task.cancel()
+ _ = await asyncio.gather(task, return_exceptions=True)
async def _mark_cancelled_run_failed(self, run_id: str) -> None:
"""Best-effort failure event/status for shutdown-cancelled runs."""
diff --git a/dify-agent/src/dify_agent/server/routes/runs.py b/dify-agent/src/dify_agent/server/routes/runs.py
index 031678789bc..09e1fe3857b 100644
--- a/dify-agent/src/dify_agent/server/routes/runs.py
+++ b/dify-agent/src/dify_agent/server/routes/runs.py
@@ -76,7 +76,7 @@ def create_runs_router(
request: CancelRunRequest,
scheduler: Annotated[RunScheduler, Depends(scheduler_dep)],
) -> CancelRunResponse:
- """Cancel a process-local run and publish its terminal event/status."""
+ """Persist cancellation; the owner process observes it and stops its runner."""
try:
return await scheduler.cancel_run(run_id, request)
except RunNotFoundError as exc:
diff --git a/dify-agent/src/dify_agent/storage/redis_run_store.py b/dify-agent/src/dify_agent/storage/redis_run_store.py
index 8e47be1f1ec..9afe26a40ea 100644
--- a/dify-agent/src/dify_agent/storage/redis_run_store.py
+++ b/dify-agent/src/dify_agent/storage/redis_run_store.py
@@ -155,6 +155,32 @@ class RedisRunStore(RunEventSink):
event_id=event_id,
)
+ async def wait_for_cancellation(self, run_id: str) -> bool:
+ """Wait until cancellation or another terminal state wins for one run.
+
+ The stream cursor is captured before reading the record so a terminal
+ transition cannot fall between the initial status check and blocking
+ stream read.
+ """
+ events_key = run_events_key(self.prefix, run_id)
+ latest_events = await self.redis.xrevrange(events_key, count=1)
+ cursor = _decode_redis_text(latest_events[0][0]) if latest_events else "0-0"
+ record = await self.get_run(run_id)
+ if record.status != "running":
+ return record.status == "cancelled"
+
+ while True:
+ response = await self.redis.xread({events_key: cursor}, block=0, count=100)
+ for _stream_name, entries in response:
+ for raw_id, fields in entries:
+ event = self._decode_event(run_id, raw_id, fields)
+ if event.id is not None:
+ cursor = event.id
+ if event.type == "run_cancelled":
+ return True
+ if event.type in {"run_succeeded", "run_failed"}:
+ return False
+
async def get_events(self, run_id: str, *, after: str = "0-0", limit: int = 100) -> RunEventsResponse:
"""Read a bounded page of events after ``after`` cursor."""
await self.get_run(run_id)
diff --git a/dify-agent/tests/integration/dify_agent/storage/test_terminal_finalization.py b/dify-agent/tests/integration/dify_agent/storage/test_terminal_finalization.py
index b500ebac3f8..790ac76f780 100644
--- a/dify-agent/tests/integration/dify_agent/storage/test_terminal_finalization.py
+++ b/dify-agent/tests/integration/dify_agent/storage/test_terminal_finalization.py
@@ -1,4 +1,4 @@
-"""Real-Redis contracts for atomic terminal run finalization."""
+"""Real-Redis contracts for terminal finalization and cancellation observation."""
import asyncio
from collections.abc import Iterator
@@ -8,17 +8,22 @@ import subprocess
import time
from uuid import uuid4
+import httpx
import pytest
from redis.asyncio import Redis
from agenton.compositor import CompositorSessionSnapshot
from dify_agent.protocol.schemas import (
+ CancelRunRequest,
+ CreateRunRequest,
RunCancelledEvent,
RunCancelledEventData,
+ RunComposition,
RunSucceededEvent,
RunSucceededEventData,
)
from dify_agent.runtime.event_sink import TerminalRunEvent, terminal_event_status_and_error
+from dify_agent.runtime.run_scheduler import RunScheduler
from dify_agent.storage.redis_keys import run_events_key, run_record_key
from dify_agent.storage.redis_run_store import RedisRunStore
@@ -31,7 +36,7 @@ def redis_url() -> Iterator[str]:
"""Start an isolated Redis when the binary is available locally."""
redis_server = shutil.which("redis-server")
if redis_server is None:
- pytest.skip("redis-server is required for the atomic finalization integration test")
+ pytest.skip("redis-server is required for run terminal integration tests")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe:
probe.bind(("127.0.0.1", 0))
@@ -131,3 +136,66 @@ def test_two_redis_clients_commit_exactly_one_matching_terminal(redis_url: str)
await second_client.aclose()
asyncio.run(scenario())
+
+
+def test_non_owner_scheduler_cancellation_stops_owner_runner(redis_url: str) -> None:
+ class BlockingRunner:
+ def __init__(self, *, started: asyncio.Event, stopped: asyncio.Event) -> None:
+ self.started = started
+ self.stopped = stopped
+
+ async def run(self) -> None:
+ self.started.set()
+ try:
+ await asyncio.Event().wait()
+ finally:
+ self.stopped.set()
+
+ async def scenario() -> None:
+ owner_client = Redis.from_url(redis_url)
+ remote_client = Redis.from_url(redis_url)
+ prefix = f"route-independent-cancellation-{uuid4().hex}"
+ owner_store = RedisRunStore(owner_client, prefix=prefix, run_retention_seconds=60)
+ remote_store = RedisRunStore(remote_client, prefix=prefix, run_retention_seconds=60)
+ runner_started = asyncio.Event()
+ runner_stopped = asyncio.Event()
+ async with httpx.AsyncClient() as http_client:
+ owner_scheduler = RunScheduler(
+ store=owner_store,
+ plugin_daemon_http_client=http_client,
+ dify_api_http_client=http_client,
+ runner_factory=lambda _record, _request: BlockingRunner(
+ started=runner_started,
+ stopped=runner_stopped,
+ ),
+ )
+ remote_scheduler = RunScheduler(
+ store=remote_store,
+ plugin_daemon_http_client=http_client,
+ dify_api_http_client=http_client,
+ )
+ try:
+ record = await owner_scheduler.create_run(CreateRunRequest(composition=RunComposition(layers=[])))
+ owner_task = owner_scheduler.active_tasks[record.run_id]
+ await asyncio.wait_for(runner_started.wait(), timeout=1)
+
+ response = await remote_scheduler.cancel_run(
+ record.run_id,
+ CancelRunRequest(reason="remote_cancel"),
+ )
+
+ assert response.status == "cancelled"
+ assert remote_scheduler.active_tasks == {}
+ await asyncio.wait_for(runner_stopped.wait(), timeout=1)
+ await asyncio.wait_for(owner_task, timeout=1)
+ persisted = await owner_store.get_run(record.run_id)
+ events = await remote_store.get_events(record.run_id)
+ assert persisted.status == "cancelled"
+ assert [event.type for event in events.events] == ["run_cancelled"]
+ finally:
+ await owner_scheduler.shutdown()
+ await remote_scheduler.shutdown()
+ await owner_client.aclose()
+ await remote_client.aclose()
+
+ asyncio.run(scenario())
diff --git a/dify-agent/tests/local/dify_agent/runtime/test_run_scheduler.py b/dify-agent/tests/local/dify_agent/runtime/test_run_scheduler.py
index cb3990c4256..b38985b2732 100644
--- a/dify-agent/tests/local/dify_agent/runtime/test_run_scheduler.py
+++ b/dify-agent/tests/local/dify_agent/runtime/test_run_scheduler.py
@@ -24,6 +24,7 @@ from dify_agent.runtime.event_sink import (
NonTerminalRunEvent,
RunFinalizationResult,
TerminalRunEvent,
+ emit_run_failed,
emit_run_succeeded,
terminal_event_status_and_error,
)
@@ -92,18 +93,21 @@ class FakeStore:
events: dict[str, list[RunEvent]]
statuses: dict[str, RunStatus]
errors: dict[str, str | None]
+ terminal_changes: dict[str, asyncio.Event]
def __init__(self) -> None:
self.records = {}
self.events = defaultdict(list)
self.statuses = {}
self.errors = {}
+ self.terminal_changes = {}
async def create_run(self) -> RunRecord:
run_id = f"run-{len(self.records) + 1}"
record = RunRecord(run_id=run_id, status="running")
self.records[run_id] = record
self.statuses[run_id] = "running"
+ self.terminal_changes[run_id] = asyncio.Event()
return record
async def append_event(self, event: NonTerminalRunEvent) -> str:
@@ -126,8 +130,14 @@ class FakeStore:
self.events[event.run_id].append(event.model_copy(update={"id": event_id}))
self.statuses[event.run_id] = status
self.errors[event.run_id] = error
+ self.terminal_changes[event.run_id].set()
return RunFinalizationResult(applied=True, status=status, event_id=event_id)
+ async def wait_for_cancellation(self, run_id: str) -> bool:
+ while self.statuses[run_id] == "running":
+ await self.terminal_changes[run_id].wait()
+ return self.statuses[run_id] == "cancelled"
+
class SlowCreateStore(FakeStore):
create_started: asyncio.Event
@@ -144,17 +154,69 @@ class SlowCreateStore(FakeStore):
return await super().create_run()
+class TrackingStore(FakeStore):
+ observer_started: asyncio.Event
+ observer_finished: asyncio.Event
+ release_observer: asyncio.Event
+
+ def __init__(self, *, pause_observer: bool = False) -> None:
+ super().__init__()
+ self.observer_started = asyncio.Event()
+ self.observer_finished = asyncio.Event()
+ self.release_observer = asyncio.Event()
+ if not pause_observer:
+ self.release_observer.set()
+
+ async def wait_for_cancellation(self, run_id: str) -> bool:
+ self.observer_started.set()
+ try:
+ await self.release_observer.wait()
+ return await super().wait_for_cancellation(run_id)
+ finally:
+ self.observer_finished.set()
+
+
+class FailingObserverStore(FakeStore):
+ fail_observer: asyncio.Event
+ observer_finished: asyncio.Event
+
+ def __init__(self, *, fail_observer: asyncio.Event) -> None:
+ super().__init__()
+ self.fail_observer = fail_observer
+ self.observer_finished = asyncio.Event()
+
+ async def wait_for_cancellation(self, run_id: str) -> bool:
+ del run_id
+ try:
+ await self.fail_observer.wait()
+ raise RuntimeError("redis read failed")
+ finally:
+ self.observer_finished.set()
+
+
class ControlledRunner:
started: asyncio.Event
release: asyncio.Event
+ finished: asyncio.Event | None
- def __init__(self, *, started: asyncio.Event, release: asyncio.Event) -> None:
+ def __init__(
+ self,
+ *,
+ started: asyncio.Event,
+ release: asyncio.Event,
+ finished: asyncio.Event | None = None,
+ ) -> None:
self.started = started
self.release = release
+ self.finished = finished
async def run(self) -> None:
_ = self.started.set()
- await self.release.wait()
+ try:
+ await self.release.wait()
+ finally:
+ if self.finished is not None:
+ self.finished.set()
class SwallowOneCancellationRunner:
@@ -201,27 +263,96 @@ class SuccessThenWaitRunner:
class IgnoreCancellationThenSucceedRunner:
- def __init__(self, *, store: FakeStore, run_id: str, started: asyncio.Event, release: asyncio.Event) -> None:
+ def __init__(
+ self,
+ *,
+ store: FakeStore,
+ run_id: str,
+ started: asyncio.Event,
+ release: asyncio.Event,
+ finished: asyncio.Event,
+ ) -> None:
+ self.store = store
+ self.run_id = run_id
+ self.started = started
+ self.release = release
+ self.finished = finished
+
+ async def run(self) -> None:
+ try:
+ self.started.set()
+ while not self.release.is_set():
+ try:
+ await self.release.wait()
+ except asyncio.CancelledError:
+ continue
+ result = await emit_run_succeeded(
+ self.store,
+ run_id=self.run_id,
+ output="late success",
+ session_snapshot=CompositorSessionSnapshot(layers=[]),
+ )
+ assert result.applied is False
+ assert result.status == "cancelled"
+ finally:
+ self.finished.set()
+
+
+class ReleaseThenSucceedRunner:
+ def __init__(
+ self,
+ *,
+ store: FakeStore,
+ run_id: str,
+ started: asyncio.Event,
+ release: asyncio.Event,
+ finished: asyncio.Event,
+ ) -> None:
self.store = store
self.run_id = run_id
self.started = started
self.release = release
+ self.finished = finished
async def run(self) -> None:
self.started.set()
- while not self.release.is_set():
- try:
- await self.release.wait()
- except asyncio.CancelledError:
- continue
- result = await emit_run_succeeded(
- self.store,
- run_id=self.run_id,
- output="late success",
- session_snapshot=CompositorSessionSnapshot(layers=[]),
- )
- assert result.applied is False
- assert result.status == "cancelled"
+ try:
+ await self.release.wait()
+ result = await emit_run_succeeded(
+ self.store,
+ run_id=self.run_id,
+ output="done",
+ session_snapshot=CompositorSessionSnapshot(layers=[]),
+ )
+ assert result.applied is True
+ finally:
+ self.finished.set()
+
+
+class CompetingFailureRunner:
+ def __init__(
+ self,
+ *,
+ store: FakeStore,
+ run_id: str,
+ started: asyncio.Event,
+ release: asyncio.Event,
+ failure_attempted: asyncio.Event,
+ ) -> None:
+ self.store = store
+ self.run_id = run_id
+ self.started = started
+ self.release = release
+ self.failure_attempted = failure_attempted
+
+ async def run(self) -> None:
+ self.started.set()
+ try:
+ await self.release.wait()
+ except asyncio.CancelledError:
+ pass
+ _ = await emit_run_failed(self.store, run_id=self.run_id, error="runner failed", reason="model_error")
+ self.failure_attempted.set()
class FinalizeSuccessOnCancellationRunner:
@@ -296,39 +427,91 @@ def test_shutdown_marks_unfinished_runs_failed_and_appends_event() -> None:
asyncio.run(scenario())
-def test_cancel_run_stops_task_and_persists_cancelled_terminal() -> None:
+def test_cancellation_observer_failure_stops_runner_and_finalizes_failed() -> None:
async def scenario() -> None:
- store = FakeStore()
- started = asyncio.Event()
+ fail_observer = asyncio.Event()
+ store = FailingObserverStore(fail_observer=fail_observer)
+ runner_started = asyncio.Event()
+ runner_finished = asyncio.Event()
async with httpx.AsyncClient() as client:
scheduler = RunScheduler(
store=store,
plugin_daemon_http_client=client,
dify_api_http_client=client,
- runner_factory=lambda _record, _request: ControlledRunner(started=started, release=asyncio.Event()),
+ runner_factory=lambda _record, _request: ControlledRunner(
+ started=runner_started,
+ release=asyncio.Event(),
+ finished=runner_finished,
+ ),
)
record = await scheduler.create_run(_request())
+ supervisor_task = scheduler.active_tasks[record.run_id]
+ await asyncio.wait_for(runner_started.wait(), timeout=1)
+
+ fail_observer.set()
+ await asyncio.wait_for(supervisor_task, timeout=1)
+
+ assert store.statuses[record.run_id] == "failed"
+ assert store.errors[record.run_id] == "run cancellation observer failed: redis read failed"
+ assert [event.type for event in store.events[record.run_id]] == ["run_failed"]
+ assert runner_finished.is_set()
+ assert store.observer_finished.is_set()
+ await asyncio.sleep(0)
+ assert scheduler.active_tasks == {}
+
+ asyncio.run(scenario())
+
+
+def test_non_owner_cancel_run_stops_owner_task_and_persists_cancelled_terminal() -> None:
+ async def scenario() -> None:
+ store = TrackingStore()
+ started = asyncio.Event()
+ runner_finished = asyncio.Event()
+ async with httpx.AsyncClient() as client:
+ owner_scheduler = RunScheduler(
+ store=store,
+ plugin_daemon_http_client=client,
+ dify_api_http_client=client,
+ runner_factory=lambda _record, _request: ControlledRunner(
+ started=started,
+ release=asyncio.Event(),
+ finished=runner_finished,
+ ),
+ )
+ remote_scheduler = RunScheduler(
+ store=store,
+ plugin_daemon_http_client=client,
+ dify_api_http_client=client,
+ )
+ record = await owner_scheduler.create_run(_request())
+ owner_task = owner_scheduler.active_tasks[record.run_id]
await asyncio.wait_for(started.wait(), timeout=1)
+ await asyncio.wait_for(store.observer_started.wait(), timeout=1)
- response = await scheduler.cancel_run(
+ response = await remote_scheduler.cancel_run(
record.run_id,
CancelRunRequest(reason="workflow_aborted", message="outer workflow stopped"),
)
assert response.status == "cancelled"
- assert scheduler.active_tasks == {}
+ assert remote_scheduler.active_tasks == {}
assert store.statuses[record.run_id] == "cancelled"
assert store.errors[record.run_id] == "outer workflow stopped"
assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]
+ await asyncio.wait_for(owner_task, timeout=1)
+ assert runner_finished.is_set()
+ assert store.observer_finished.is_set()
+ await asyncio.sleep(0)
+ assert owner_scheduler.active_tasks == {}
- repeated = await scheduler.cancel_run(record.run_id, CancelRunRequest(reason="duplicate"))
+ repeated = await remote_scheduler.cancel_run(record.run_id, CancelRunRequest(reason="duplicate"))
assert repeated.status == "cancelled"
assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]
asyncio.run(scenario())
-def test_cancel_run_reinjects_cancellation_without_waiting_for_runner_cleanup() -> None:
+def test_owner_observer_reinjects_cancellation_consumed_by_runner() -> None:
async def scenario() -> None:
store = FakeStore()
started = asyncio.Event()
@@ -344,6 +527,7 @@ def test_cancel_run_reinjects_cancellation_without_waiting_for_runner_cleanup()
),
)
record = await scheduler.create_run(_request())
+ supervisor_task = scheduler.active_tasks[record.run_id]
await asyncio.wait_for(started.wait(), timeout=1)
response = await asyncio.wait_for(
@@ -352,9 +536,10 @@ def test_cancel_run_reinjects_cancellation_without_waiting_for_runner_cleanup()
)
assert response.status == "cancelled"
- assert first_cancellation.is_set()
assert store.statuses[record.run_id] == "cancelled"
assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]
+ await asyncio.wait_for(first_cancellation.wait(), timeout=1)
+ await asyncio.wait_for(supervisor_task, timeout=1)
await asyncio.sleep(0)
assert scheduler.active_tasks == {}
@@ -394,40 +579,151 @@ def test_cancel_run_does_not_override_successful_terminal() -> None:
asyncio.run(scenario())
-def test_cancel_run_wins_before_a_runner_that_consumes_cancellation_finishes() -> None:
+def test_cancelled_terminal_survives_shutdown_while_runner_cleanup_is_pending() -> None:
async def scenario() -> None:
- store = FakeStore()
+ store = TrackingStore()
started = asyncio.Event()
release = asyncio.Event()
+ runner_finished = asyncio.Event()
async with httpx.AsyncClient() as client:
scheduler = RunScheduler(
store=store,
plugin_daemon_http_client=client,
dify_api_http_client=client,
+ shutdown_grace_seconds=0,
runner_factory=lambda record, _request: IgnoreCancellationThenSucceedRunner(
store=store,
run_id=record.run_id,
started=started,
release=release,
+ finished=runner_finished,
),
)
record = await scheduler.create_run(_request())
+ supervisor_task = scheduler.active_tasks[record.run_id]
await asyncio.wait_for(started.wait(), timeout=1)
+ await asyncio.wait_for(store.observer_started.wait(), timeout=1)
response = await scheduler.cancel_run(record.run_id, CancelRunRequest(reason="workflow_aborted"))
assert response.status == "cancelled"
assert store.statuses[record.run_id] == "cancelled"
assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]
- task = scheduler.active_tasks[record.run_id]
- release.set()
- await asyncio.wait_for(task, timeout=1)
+ await asyncio.wait_for(store.observer_finished.wait(), timeout=1)
+ assert supervisor_task.done() is False
+ shutdown_task = asyncio.create_task(scheduler.shutdown())
await asyncio.sleep(0)
+ assert shutdown_task.done() is False
+ release.set()
+ await asyncio.wait_for(shutdown_task, timeout=1)
+
+ assert supervisor_task.done()
+ assert runner_finished.is_set()
+ assert store.observer_finished.is_set()
+ assert scheduler.active_tasks == {}
+ assert store.statuses[record.run_id] == "cancelled"
assert [event.type for event in store.events[record.run_id]] == ["run_cancelled"]
asyncio.run(scenario())
+ ("winner", "expected_event_type"),
+ [
+ pytest.param("failed", "run_failed", id="failure-first"),
+ pytest.param("cancelled", "run_cancelled", id="cancellation-first"),
+ ],
+)
+def test_failure_and_cancellation_keep_the_first_terminal(
+ winner: RunStatus,
+ expected_event_type: str,
+) -> None:
+ async def scenario() -> None:
+ store = FakeStore()
+ runner_started = asyncio.Event()
+ release_runner = asyncio.Event()
+ failure_attempted = asyncio.Event()
+ async with httpx.AsyncClient() as client:
+ scheduler = RunScheduler(
+ store=store,
+ plugin_daemon_http_client=client,
+ dify_api_http_client=client,
+ runner_factory=lambda record, _request: CompetingFailureRunner(
+ store=store,
+ run_id=record.run_id,
+ started=runner_started,
+ release=release_runner,
+ failure_attempted=failure_attempted,
+ ),
+ )
+ record = await scheduler.create_run(_request())
+ supervisor_task = scheduler.active_tasks[record.run_id]
+ await asyncio.wait_for(runner_started.wait(), timeout=1)
+
+ if winner == "failed":
+ release_runner.set()
+ await asyncio.wait_for(failure_attempted.wait(), timeout=1)
+ with pytest.raises(RunCancellationConflictError, match="already finished with status 'failed'"):
+ await scheduler.cancel_run(record.run_id, CancelRunRequest(reason="late_cancel"))
+ else:
+ response = await scheduler.cancel_run(
+ record.run_id,
+ CancelRunRequest(reason="cancel_before_failure"),
+ )
+ assert response.run_id == record.run_id
+ assert response.status == "cancelled"
+ release_runner.set()
+
+ await asyncio.wait_for(failure_attempted.wait(), timeout=1)
+ await asyncio.wait_for(supervisor_task, timeout=1)
+
+ assert store.statuses[record.run_id] == winner
+ assert [event.type for event in store.events[record.run_id]] == [expected_event_type]
+
+ asyncio.run(scenario())
+
+
+def test_shutdown_grace_allows_runner_first_completion_and_reaps_children() -> None:
+ async def scenario() -> None:
+ store = TrackingStore(pause_observer=True)
+ runner_started = asyncio.Event()
+ release_runner = asyncio.Event()
+ runner_finished = asyncio.Event()
+ async with httpx.AsyncClient() as client:
+ scheduler = RunScheduler(
+ store=store,
+ plugin_daemon_http_client=client,
+ dify_api_http_client=client,
+ shutdown_grace_seconds=1,
+ runner_factory=lambda record, _request: ReleaseThenSucceedRunner(
+ store=store,
+ run_id=record.run_id,
+ started=runner_started,
+ release=release_runner,
+ finished=runner_finished,
+ ),
+ )
+ record = await scheduler.create_run(_request())
+ supervisor_task = scheduler.active_tasks[record.run_id]
+ await asyncio.wait_for(runner_started.wait(), timeout=1)
+ await asyncio.wait_for(store.observer_started.wait(), timeout=1)
+
+ shutdown_task = asyncio.create_task(scheduler.shutdown())
+ await asyncio.sleep(0)
+ assert shutdown_task.done() is False
+ release_runner.set()
+ await asyncio.wait_for(shutdown_task, timeout=1)
+
+ assert supervisor_task.done()
+ assert runner_finished.is_set()
+ assert store.observer_finished.is_set()
+ assert scheduler.active_tasks == {}
+ assert store.statuses[record.run_id] == "succeeded"
+ assert [event.type for event in store.events[record.run_id]] == ["run_succeeded"]
+
+ asyncio.run(scenario())
+
+
def test_shutdown_does_not_append_failed_after_success_wins() -> None:
async def scenario() -> None:
store = FakeStore()
diff --git a/dify-agent/tests/local/dify_agent/server/test_runs_routes.py b/dify-agent/tests/local/dify_agent/server/test_runs_routes.py
index d90f82a152a..7a8da51eb29 100644
--- a/dify-agent/tests/local/dify_agent/server/test_runs_routes.py
+++ b/dify-agent/tests/local/dify_agent/server/test_runs_routes.py
@@ -4,6 +4,7 @@ from dify_agent.protocol import CancelRunResponse, DIFY_AGENT_MODEL_LAYER_ID
from dify_agent.runtime.run_scheduler import RunCancellationConflictError, SchedulerStoppingError
from dify_agent.server.routes.runs import create_runs_router
from dify_agent.server.schemas import RunRecord
+from dify_agent.storage.redis_run_store import RunNotFoundError
class FakeScheduler:
@@ -106,6 +107,26 @@ def test_cancel_run_endpoint_maps_conflict() -> None:
assert "already finished" in response.json()["detail"]
+def test_cancel_run_endpoint_maps_missing_run() -> None:
+ from fastapi import FastAPI
+
+ class MissingRunScheduler(FakeScheduler):
+ async def cancel_run(self, run_id: str, request: object) -> CancelRunResponse:
+ del request
+ raise RunNotFoundError(run_id)
+
+ app = FastAPI()
+ app.include_router(
+ create_runs_router(lambda: FakeStore(), lambda: MissingRunScheduler()) # pyright: ignore[reportArgumentType]
+ )
+ client = TestClient(app)
+
+ response = client.post("/runs/missing/cancel", json={})
+
+ assert response.status_code == 404
+ assert response.json() == {"detail": "run not found"}
+
+
def test_create_run_accepts_valid_full_plugin_graph() -> None:
from fastapi import FastAPI
diff --git a/dify-agent/tests/local/dify_agent/storage/test_redis_run_store.py b/dify-agent/tests/local/dify_agent/storage/test_redis_run_store.py
index 5f22779da79..eb56e6ffc87 100644
--- a/dify-agent/tests/local/dify_agent/storage/test_redis_run_store.py
+++ b/dify-agent/tests/local/dify_agent/storage/test_redis_run_store.py
@@ -31,6 +31,7 @@ class FakeRedis:
self.commands = []
self.values = {}
self.streams = {}
+ self.stream_changed = asyncio.Event()
async def set(self, key: str, value: object, *, ex: int | None = None) -> None:
self.commands.append(("set", key, value, ex))
@@ -52,8 +53,46 @@ class FakeRedis:
entries = self.streams.setdefault(key, [])
event_id = f"{len(entries) + 1}-0"
entries.append((event_id, dict(fields)))
+ self.stream_changed.set()
return event_id
+ async def xrevrange(
+ self,
+ key: str,
+ max: str = "+",
+ min: str = "-",
+ *,
+ count: int | None = None,
+ ) -> list[tuple[str, dict[str, object]]]:
+ self.commands.append(("xrevrange", key, max, min, count))
+ entries = list(reversed(self.streams.get(key, [])))
+ return entries[:count] if count is not None else entries
+
+ async def xread(
+ self,
+ streams: Mapping[str, str],
+ *,
+ count: int | None = None,
+ block: int | None = None,
+ ) -> list[tuple[str, list[tuple[str, dict[str, object]]]]]:
+ self.commands.append(("xread", dict(streams), count, block))
+ while True:
+ response: list[tuple[str, list[tuple[str, dict[str, object]]]]] = []
+ for key, cursor in streams.items():
+ entries = [
+ entry
+ for entry in self.streams.get(key, [])
+ if self._stream_id_value(entry[0]) > self._stream_id_value(cursor)
+ ]
+ if count is not None:
+ entries = entries[:count]
+ if entries:
+ response.append((key, entries))
+ if response:
+ return response
+ self.stream_changed.clear()
+ await self.stream_changed.wait()
+
async def xrange(
self, key: str, *, min: str = "-", count: int | None = None
) -> list[tuple[str, dict[str, object]]]:
@@ -311,6 +350,101 @@ def test_finalize_run_raises_when_record_is_missing() -> None:
)
+def test_wait_for_cancellation_observes_terminal_record_before_starting() -> None:
+ redis = FakeRedis()
+ store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]
+
+ async def scenario() -> bool:
+ record = await store.create_run()
+ _ = await store.finalize_run(
+ RunCancelledEvent(run_id=record.run_id, data=RunCancelledEventData(reason="cancelled"))
+ )
+ redis.commands.clear()
+ return await store.wait_for_cancellation(record.run_id)
+
+ assert asyncio.run(scenario()) is True
+ assert [command[0] for command in redis.commands] == ["xrevrange", "get"]
+
+
+def test_wait_for_cancellation_covers_terminal_transition_during_initialization() -> None:
+ class PausingRecordReadRedis(FakeRedis):
+ record_read_started: asyncio.Event
+ release_record_read: asyncio.Event
+ pause_next_record_read: bool
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.record_read_started = asyncio.Event()
+ self.release_record_read = asyncio.Event()
+ self.pause_next_record_read = True
+
+ async def get(self, key: str) -> object | None:
+ if self.pause_next_record_read and key.endswith(":record"):
+ self.pause_next_record_read = False
+ self.record_read_started.set()
+ await self.release_record_read.wait()
+ return await super().get(key)
+
+ redis = PausingRecordReadRedis()
+ observer_store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]
+ cancelling_store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]
+
+ async def scenario() -> bool:
+ record = await observer_store.create_run()
+ observer = asyncio.create_task(observer_store.wait_for_cancellation(record.run_id))
+ await asyncio.wait_for(redis.record_read_started.wait(), timeout=1)
+ _ = await cancelling_store.finalize_run(
+ RunCancelledEvent(run_id=record.run_id, data=RunCancelledEventData(reason="cancelled"))
+ )
+ redis.release_record_read.set()
+ return await asyncio.wait_for(observer, timeout=1)
+
+ assert asyncio.run(scenario()) is True
+
+
+def test_wait_for_cancellation_advances_past_non_terminal_events() -> None:
+ redis = FakeRedis()
+ store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]
+
+ async def scenario() -> bool:
+ record = await store.create_run()
+ observer = asyncio.create_task(store.wait_for_cancellation(record.run_id))
+ await asyncio.sleep(0)
+ _ = await store.append_event(RunStartedEvent(run_id=record.run_id))
+ await asyncio.sleep(0)
+ _ = await store.finalize_run(
+ RunCancelledEvent(run_id=record.run_id, data=RunCancelledEventData(reason="cancelled"))
+ )
+ return await asyncio.wait_for(observer, timeout=1)
+
+ assert asyncio.run(scenario()) is True
+ cursors = [command[1] for command in redis.commands if command[0] == "xread"]
+ assert any("0-0" in streams.values() for streams in cursors if isinstance(streams, dict))
+ assert any("1-0" in streams.values() for streams in cursors if isinstance(streams, dict))
+
+
+def test_wait_for_cancellation_returns_false_when_success_wins() -> None:
+ redis = FakeRedis()
+ store = RedisRunStore(redis, prefix="test") # pyright: ignore[reportArgumentType]
+
+ async def scenario() -> bool:
+ record = await store.create_run()
+ observer = asyncio.create_task(store.wait_for_cancellation(record.run_id))
+ await asyncio.sleep(0)
+ _ = await store.finalize_run(
+ RunSucceededEvent(
+ run_id=record.run_id,
+ data=RunSucceededEventData(
+ output="done",
+ session_snapshot=CompositorSessionSnapshot(layers=[]),
+ ),
+ )
+ )
+ return await asyncio.wait_for(observer, timeout=1)
+
+ assert asyncio.run(scenario()) is False
+
+
def test_append_event_serializes_typed_event_without_id_and_expires_run_keys() -> None:
redis = FakeRedis()
store = RedisRunStore(redis, prefix="test", run_retention_seconds=60) # pyright: ignore[reportArgumentType]