Skip to content

Self-improving AI Employees for everyone

fix(workflow): every Run path reports its failure and node count

trohitgd8f23e9b0f39
diff --git a/docs-internal/DESIGN.md b/docs-internal/DESIGN.md
index 657f8747..adbfa42c 100644
--- a/docs-internal/DESIGN.md
+++ b/docs-internal/DESIGN.md
@@ -635,6 +635,8 @@ if use_parallel and self.settings.redis_enabled:
return await self._execute_sequential(...)
```
+All three return the same Run fields, which the editor's Run dialog reads: `errors`, `error` (the first failure's message), `total_nodes` and `completed_nodes`. See [Temporal Architecture → The run result](./TEMPORAL_ARCHITECTURE.md#4-the-run-result).
+
---
## Comparison: Before vs After
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index cb494d65..8e04868c 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -252,13 +252,23 @@ In the legacy path the server broadcasts status updates, tool-glow events, and p
### 4. The run result
-`MachinaWorkflow.run` returns `{success, outputs, execution_trace, errors}`. `success` is true only when no node failed and every node in the executable graph completed. The first node that fails stops the run. `errors` then holds that failure as `{node_id, error}`, plus whichever of `hint`, `requires_user_action` and `retryable` the node supplied, and it is `None` when nothing failed. Only an empty graph returns a top-level `error` ("No nodes provided") instead.
-
-`TemporalExecutor.execute_workflow` turns that into the Run response:
+`MachinaWorkflow.run` returns `{success, outputs, execution_trace, errors, total_nodes}`.
+- `total_nodes` is the size of the executable graph.
+- `execution_trace` lists every node the run completed, skipped or auto-completed.
+- `success` is true only when no node failed and every node in the executable graph completed.
+- The first node that fails stops the run. `errors` then holds that failure as `{node_id, error}`, plus whichever of `hint`, `requires_user_action` and `retryable` the node supplied. It is `None` when nothing failed.
+- Only an empty graph returns a top-level `error` ("No nodes provided") instead.
+
+`TemporalExecutor.execute_workflow` turns that into the Run response, and `WorkflowService._execute_temporal` forwards it:
- `errors` is the workflow's list unchanged, or `[{"error": ...}]` for an empty graph or a failed Temporal call;
-- `error` is the first entry's message, or `None`.
+- `error` is the first entry's message, or `None`;
+- `total_nodes` is the workflow's count, and `completed_nodes` is the length of `execution_trace`.
+
+The in-process paths return the same four fields (`_run_errors` and `_node_failure` in `services/workflow.py`):
+- the sequential fallback counts the nodes reachable from the start node, which leaves out tools and other nodes wired into an agent;
+- the Redis-only parallel engine counts its node executions, which leave out config nodes and agent sub-nodes.
-`WorkflowService._execute_temporal` forwards both. The editor's Run dialog shows only `error` ("Workflow failed: …"), so before both kept it, every failed Run said "Unknown error" ([Known Errors #31](./errors.md)). Locked by `server/tests/temporal/test_run_errors.py`.
+The editor's Run dialog shows `error` when a run fails ("Workflow failed: …") and `completed_nodes` / `total_nodes` when it succeeds ("N/M nodes completed"). Before these fields were set, a failed Run said "Unknown error" and a successful one said "0/0 nodes completed" ([Known Errors #31](./errors.md)). Locked by `server/tests/temporal/test_run_errors.py` and `server/tests/services/test_workflow_run_result.py`.
## Connection Pooling
diff --git a/docs-internal/errors.md b/docs-internal/errors.md
index f55933e8..dbf41b1c 100644
--- a/docs-internal/errors.md
+++ b/docs-internal/errors.md
@@ -640,14 +640,19 @@ To reproduce the old failure on 3.12.8, run `kill_orphaned_opencompany_processes
See the [cronScheduler card](./node-logic-flows/workflow_triggers/cronScheduler.md).
-## 31. A failed canvas Run says `Workflow failed: Unknown error`
+## 31. A canvas Run says `Workflow failed: Unknown error` when it fails, and `0/0 nodes completed` when it succeeds
-**Symptom**: Run a whole workflow with Temporal on (the default). When a node fails, the result dialog says `Workflow failed: Unknown error` and the response's `errors` list is empty, whatever the node's actual error was.
+**Symptom**: Run a whole workflow.
+- When a node fails, the result dialog says `Workflow failed: Unknown error`, whatever the node's actual error was. With Temporal on (the default), the response's `errors` list is empty too.
+- When the run succeeds, the dialog says `Workflow executed successfully. 0/0 nodes completed.`
-**Root cause**: `MachinaWorkflow.run` stops at the first failed node and returns that failure in `errors`, a list of `{node_id, error, ...}`. Only an empty graph returns a top-level `error`. `TemporalExecutor.execute_workflow` read only the top-level `error`, so it reported no errors, and `WorkflowService._execute_temporal` passed no `error` on at all. The dialog shows only `error`.
+**Root cause**: the dialog shows only `error` for a failure and only `completed_nodes` / `total_nodes` for a success, and no execution path filled them in.
+- `MachinaWorkflow.run` stops at the first failed node and returns that failure in `errors`, a list of `{node_id, error, ...}`. Only an empty graph returns a top-level `error`. `TemporalExecutor.execute_workflow` read only the top-level `error`, so it reported no errors, and `WorkflowService._execute_temporal` passed no `error` on at all.
+- Without Temporal, the sequential fallback reported failures only in `node_results`, and the Redis-only parallel path only in `errors`. Neither set `error`.
+- No path set `total_nodes` or `completed_nodes`, so the WebSocket handler's defaults of 0 reached the dialog.
-**Fix**: the executor keeps the workflow's `errors` and sets `error` to the first message; an empty graph or a failed Temporal call becomes `[{"error": ...}]`. `_execute_temporal` forwards `error`. Locked by `server/tests/temporal/test_run_errors.py`. The result contract is in [Temporal Architecture → The run result](./TEMPORAL_ARCHITECTURE.md#4-the-run-result).
+**Fix**:
+- `51774a77`: the Temporal executor keeps the workflow's `errors` and sets `error` to the first message; an empty graph or a failed Temporal call becomes `[{"error": ...}]`. `_execute_temporal` forwards `error`.
+- A follow-up gave the sequential and parallel paths the same `errors` and `error`, and made every path report `total_nodes` and `completed_nodes`. `MachinaWorkflow` now returns its `total_nodes`.
-**Still open**:
-- A successful Run's dialog says `0/0 nodes completed` on every path, because no path sets `total_nodes` or `completed_nodes`.
-- Without Temporal, a failed Run still says `Unknown error`. The sequential path reports failures only in `node_results`, and the Redis-only parallel path only in `errors`.
+Locked by `server/tests/temporal/test_run_errors.py` and `server/tests/services/test_workflow_run_result.py`. The result contract is in [Temporal Architecture → The run result](./TEMPORAL_ARCHITECTURE.md#4-the-run-result).
diff --git a/server/services/execution/executor.py b/server/services/execution/executor.py
index 0e3f0d7c..cfd415c8 100644
--- a/server/services/execution/executor.py
+++ b/server/services/execution/executor.py
@@ -171,6 +171,7 @@ class WorkflowExecutor:
"execution_id": ctx.execution_id,
"status": ctx.status.value,
"nodes_executed": ctx.get_completed_nodes(),
+ "total_nodes": len(ctx.node_executions),
"outputs": ctx.outputs,
"errors": ctx.errors,
"execution_time": time.time() - start_time,
diff --git a/server/services/temporal/executor.py b/server/services/temporal/executor.py
index a342d4ed..a6e44b85 100644
--- a/server/services/temporal/executor.py
+++ b/server/services/temporal/executor.py
@@ -188,6 +188,8 @@ class TemporalExecutor:
"outputs": result.get("outputs", {}),
"errors": errors,
"error": errors[0].get("error") if errors else None,
+ "total_nodes": result.get("total_nodes", 0),
+ "completed_nodes": len(result.get("execution_trace", [])),
"execution_time": execution_time,
"temporal_execution": True,
"timestamp": datetime.now().isoformat(),
diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.py
index 8692f837..fbf6bce9 100644
--- a/server/services/temporal/workflow.py
+++ b/server/services/temporal/workflow.py
@@ -389,7 +389,8 @@ class MachinaWorkflow:
- tenant_id: Tenant identifier for multi-tenancy
Returns:
- Dict with success, outputs, execution_trace, and errors
+ Dict with success, outputs, execution_trace, errors, and
+ total_nodes (the size of the executable graph)
"""
nodes = workflow_data.get("nodes", [])
edges = workflow_data.get("edges", [])
@@ -732,6 +733,7 @@ class MachinaWorkflow:
"outputs": outputs,
"execution_trace": execution_trace,
"errors": errors if errors else None,
+ "total_nodes": len(node_map),
}
def _get_node_inputs(
diff --git a/server/services/workflow.py b/server/services/workflow.py
index a7792ecb..09519487 100644
--- a/server/services/workflow.py
+++ b/server/services/workflow.py
@@ -43,6 +43,23 @@ _parallel_user_id: ContextVar[str] = ContextVar(
)
+# A Run's result reports failures as ``errors`` plus their first message as
+# ``error`` on every path, because the editor's Run dialog shows only
+# ``error`` (docs-internal/TEMPORAL_ARCHITECTURE.md, "4. The run result").
+def _run_errors(result: Dict[str, Any]) -> List[Dict[str, Any]]:
+ """A run's failures. A run-level ``error`` alone becomes the one entry."""
+ if result.get("errors"):
+ return list(result["errors"])
+ return [{"error": result["error"]}] if result.get("error") else []
+
+
+def _node_failure(node_id: str, result: Dict[str, Any]) -> Dict[str, Any]:
+ """One node's entry in ``errors``, shaped like MachinaWorkflow's."""
+ failure = {"node_id": node_id, "error": result.get("error") or "Unknown error"}
+ failure.update({key: result[key] for key in ("hint", "requires_user_action", "retryable") if key in result})
+ return failure
+
+
class WorkflowService:
"""Workflow execution and deployment service.
@@ -453,6 +470,8 @@ class WorkflowService:
"outputs": result.get("outputs", {}),
"errors": result.get("errors", []),
"error": result.get("error"),
+ "total_nodes": result.get("total_nodes", 0),
+ "completed_nodes": result.get("completed_nodes", 0),
"execution_time": result.get("execution_time", time.time() - start_time),
"temporal_execution": True,
"timestamp": datetime.now().isoformat(),
@@ -486,12 +505,17 @@ class WorkflowService:
finally:
_parallel_user_id.reset(token)
+ errors = _run_errors(result)
+ nodes_executed = result.get("nodes_executed", [])
return {
"success": result.get("success", False),
"execution_id": result.get("execution_id"),
- "nodes_executed": result.get("nodes_executed", []),
+ "nodes_executed": nodes_executed,
"outputs": result.get("outputs", {}),
- "errors": result.get("errors", []),
+ "errors": errors,
+ "error": errors[0].get("error") if errors else None,
+ "total_nodes": result.get("total_nodes", 0),
+ "completed_nodes": len(nodes_executed),
"execution_time": result.get("execution_time", time.time() - start_time),
"parallel_execution": True,
"timestamp": datetime.now().isoformat(),
@@ -591,11 +615,16 @@ class WorkflowService:
if not result.get("success") and self._settings.get("stop_on_error"):
break
+ errors = [_node_failure(node_id, result) for node_id, result in results.items() if not result.get("success")]
return {
"success": all(r.get("success", False) for r in results.values()),
"execution_id": execution_id,
"nodes_executed": executed,
"node_results": results,
+ "errors": errors,
+ "error": errors[0]["error"] if errors else None,
+ "total_nodes": len(execution_order),
+ "completed_nodes": len(executed) - len(errors),
"execution_time": time.time() - start_time,
"parallel_execution": False,
"timestamp": datetime.now().isoformat(),
diff --git a/server/tests/services/test_workflow_run_result.py b/server/tests/services/test_workflow_run_result.py
new file mode 100644
index 00000000..fa53627f
--- /dev/null
+++ b/server/tests/services/test_workflow_run_result.py
@@ -0,0 +1,89 @@
+"""Every Run path reports a failure and its node count the same way.
+
+The editor's Run dialog shows ``error`` when a run fails and
+``completed_nodes`` / ``total_nodes`` when it succeeds, whichever path ran
+it: Temporal (tests/temporal/test_run_errors.py), the Redis-only parallel
+engine or the sequential fallback (here).
+"""
+
+from __future__ import annotations
+
+import time
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from services.workflow import WorkflowService
+
+
+def _chain():
+ graph_nodes = [
+ {"id": "start-1", "type": "start", "data": {}},
+ {"id": "a", "type": "console", "data": {}},
+ {"id": "b", "type": "console", "data": {}},
+ ]
+ edges = [{"id": "e1", "source": "start-1", "target": "a"}, {"id": "e2", "source": "a", "target": "b"}]
+ return graph_nodes, edges
+
+
+def _sequential_service(results):
+ service = WorkflowService.__new__(WorkflowService)
+ service._settings = {}
+
+ async def execute_node(*, node_id, **_kwargs):
+ return results.get(node_id, {"success": True, "result": {}})
+
+ service.execute_node = execute_node
+ return service
+
+
+def _parallel_service(executor_result):
+ service = WorkflowService.__new__(WorkflowService)
+ executor = SimpleNamespace(execute_workflow=AsyncMock(return_value=executor_result))
+ service._get_workflow_executor = lambda _status_callback: executor
+ return service
+
+
+async def test_the_sequential_path_reports_a_failed_node():
+ service = _sequential_service({"a": {"success": False, "error": "a broke", "hint": "Check a"}})
+
+ result = await service._execute_sequential(*_chain(), "session", None, time.time())
+
+ assert result["success"] is False
+ assert result["errors"] == [{"node_id": "a", "error": "a broke", "hint": "Check a"}]
+ assert result["error"] == "a broke"
+ # stop_on_error is off, so b still ran.
+ assert (result["completed_nodes"], result["total_nodes"]) == (2, 3)
+
+
+async def test_the_sequential_path_counts_a_clean_run():
+ result = await _sequential_service({})._execute_sequential(*_chain(), "session", None, time.time())
+
+ assert result["success"] is True
+ assert (result["errors"], result["error"]) == ([], None)
+ assert (result["completed_nodes"], result["total_nodes"]) == (3, 3)
+
+
+async def test_the_parallel_path_passes_a_failed_node_on():
+ failure = {"node_id": "a", "error": "a broke", "timestamp": 1.0}
+ service = _parallel_service({"success": False, "nodes_executed": ["start-1"], "errors": [failure], "total_nodes": 3})
+
+ result = await service._execute_parallel(*_chain(), "session", None, time.time())
+
+ assert result["errors"] == [failure]
+ assert result["error"] == "a broke"
+ assert (result["completed_nodes"], result["total_nodes"]) == (1, 3)
+
+
+async def test_the_parallel_path_keeps_a_run_level_error():
+ service = _parallel_service({"success": False, "status": "cancelled", "error": "Cancelled by user"})
+
+ result = await service._execute_parallel(*_chain(), "session", None, time.time())
+
+ assert result["errors"] == [{"error": "Cancelled by user"}]
+ assert result["error"] == "Cancelled by user"
diff --git a/server/tests/temporal/test_run_errors.py b/server/tests/temporal/test_run_errors.py
index fcbdb1ad..8e85bbd5 100644
--- a/server/tests/temporal/test_run_errors.py
+++ b/server/tests/temporal/test_run_errors.py
@@ -1,14 +1,16 @@
-"""A failed canvas Run on the Temporal path reports why it failed.
+"""A canvas Run on the Temporal path reports why it failed and what it ran.
MachinaWorkflow returns per-node failures as ``errors`` (a list of
``{node_id, error, ...}``), and only an empty graph returns a top-level
-``error``. The Run response the editor shows reads ``error`` for its message
-and ``errors`` for detail, so both must survive TemporalExecutor and
-WorkflowService._execute_temporal.
+``error``. It also counts the nodes in its executable graph as
+``total_nodes``. The editor's Run dialog shows ``error`` when a run fails and
+``completed_nodes`` / ``total_nodes`` when it succeeds, so all of them must
+survive TemporalExecutor and WorkflowService._execute_temporal.
"""
from __future__ import annotations
+import asyncio
import time
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
@@ -63,7 +65,14 @@ async def test_a_successful_run_reports_no_error():
@pytest.mark.asyncio
-async def test_the_workflow_service_passes_the_message_on():
+async def test_the_run_counts_its_nodes():
+ result = await _run({"success": True, "outputs": {}, "execution_trace": ["start-1", "py-1"], "total_nodes": 2})
+
+ assert (result["completed_nodes"], result["total_nodes"]) == (2, 2)
+
+
+async def test_the_workflow_service_passes_the_message_and_counts_on():
from services.workflow import WorkflowService
service = WorkflowService.__new__(WorkflowService)
@@ -71,7 +80,15 @@ async def test_the_workflow_service_passes_the_message_on():
failure = {"node_id": "n1", "error": "boom"}
service._temporal_executor = SimpleNamespace(
execute_workflow=AsyncMock(
- return_value={"success": False, "nodes_executed": [], "outputs": {}, "errors": [failure], "error": "boom"}
+ return_value={
+ "success": False,
+ "nodes_executed": ["start-1"],
+ "outputs": {},
+ "errors": [failure],
+ "error": "boom",
+ "total_nodes": 3,
+ "completed_nodes": 1,
+ }
)
)
@@ -79,3 +96,50 @@ async def test_the_workflow_service_passes_the_message_on():
assert result["errors"] == [failure]
assert result["error"] == "boom"
+ assert (result["completed_nodes"], result["total_nodes"]) == (1, 3)
+
+
+async def _run_machina_workflow(monkeypatch, failing=()):
+ """Run the real MachinaWorkflow body on start -> py-1 -> py-2 without a
+ Temporal server (the pattern of test_machina_workflow_loop.py)."""
+ from temporalio import workflow as temporal_workflow
+
+ from services.temporal.workflow import MachinaWorkflow
+
+ def start_activity(name, **kwargs):
+ node_id = kwargs["args"][0]["node_id"]
+ future = asyncio.get_event_loop().create_future()
+ future.set_result({"success": False, "error": f"{node_id} broke"} if node_id in failing else {"success": True, "result": {}})
+ return future
+
+ async def execute_activity(*args, **kwargs):
+ return None
+
+ monkeypatch.setattr(temporal_workflow, "logger", MagicMock())
+ monkeypatch.setattr(temporal_workflow, "patched", lambda _patch_id: True)
+ monkeypatch.setattr(temporal_workflow, "start_activity", start_activity)
+ monkeypatch.setattr(temporal_workflow, "execute_activity", execute_activity)
+ monkeypatch.setattr(
+ MachinaWorkflow,
+ "_resolve_dispatch",
+ lambda self, node_type, **_kwargs: {"kind": "activity", "name": f"node.{node_type}.v1", "queue": None},
+ )
+ ids = ["start-1", "py-1", "py-2"]
+ graph_nodes = [{"id": node_id, "type": "start" if node_id == "start-1" else "pythonExecutor", "data": {}} for node_id in ids]
+ edges = [
+ {"id": f"e{index}", "source": source, "target": target, "sourceHandle": "output-main", "targetHandle": "input-main"}
+ for index, (source, target) in enumerate(zip(ids, ids[1:]))
+ ]
+ return await MachinaWorkflow().run(
+ {"nodes": graph_nodes, "edges": edges, "session_id": "test", "workflow_id": "wf-1", "execution_id": "wf-1-run"}
+ )
+
+
+async def test_the_workflow_counts_the_nodes_of_its_run(monkeypatch):
+ finished = await _run_machina_workflow(monkeypatch)
+ failed = await _run_machina_workflow(monkeypatch, failing={"py-1"})
+
+ assert (finished["success"], finished["execution_trace"], finished["total_nodes"]) == (True, ["start-1", "py-1", "py-2"], 3)
+ assert (failed["success"], failed["execution_trace"], failed["total_nodes"]) == (False, ["start-1"], 3)
+ assert failed["errors"] == [{"node_id": "py-1", "error": "py-1 broke"}]