Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(workflow): every Run path reports its failure and node count
trohitgd8f23e9b0f39
diff --git a/docs-internal/DESIGN.md b/docs-internal/DESIGN.mdindex 657f8747..adbfa42c 100644--- a/docs-internal/DESIGN.md+++ b/docs-internal/DESIGN.md@@ -635,6 +635,8 @@ if use_parallel and self.settings.redis_enabled:return await self._execute_sequential(...)```+All three return the same Run fields, which the editor's Run dialog reads: `errors`, `error` (the first failure's message), `total_nodes` and `completed_nodes`. See [Temporal Architecture → The run result](./TEMPORAL_ARCHITECTURE.md#4-the-run-result).+---## Comparison: Before vs Afterdiff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.mdindex cb494d65..8e04868c 100644--- a/docs-internal/TEMPORAL_ARCHITECTURE.md+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md@@ -252,13 +252,23 @@ In the legacy path the server broadcasts status updates, tool-glow events, and p### 4. The run result-`MachinaWorkflow.run` returns `{success, outputs, execution_trace, errors}`. `success` is true only when no node failed and every node in the executable graph completed. The first node that fails stops the run. `errors` then holds that failure as `{node_id, error}`, plus whichever of `hint`, `requires_user_action` and `retryable` the node supplied, and it is `None` when nothing failed. Only an empty graph returns a top-level `error` ("No nodes provided") instead.--`TemporalExecutor.execute_workflow` turns that into the Run response:+`MachinaWorkflow.run` returns `{success, outputs, execution_trace, errors, total_nodes}`.+- `total_nodes` is the size of the executable graph.+- `execution_trace` lists every node the run completed, skipped or auto-completed.+- `success` is true only when no node failed and every node in the executable graph completed.+- The first node that fails stops the run. `errors` then holds that failure as `{node_id, error}`, plus whichever of `hint`, `requires_user_action` and `retryable` the node supplied. It is `None` when nothing failed.+- Only an empty graph returns a top-level `error` ("No nodes provided") instead.++`TemporalExecutor.execute_workflow` turns that into the Run response, and `WorkflowService._execute_temporal` forwards it:- `errors` is the workflow's list unchanged, or `[{"error": ...}]` for an empty graph or a failed Temporal call;-- `error` is the first entry's message, or `None`.+- `error` is the first entry's message, or `None`;+- `total_nodes` is the workflow's count, and `completed_nodes` is the length of `execution_trace`.++The in-process paths return the same four fields (`_run_errors` and `_node_failure` in `services/workflow.py`):+- the sequential fallback counts the nodes reachable from the start node, which leaves out tools and other nodes wired into an agent;+- the Redis-only parallel engine counts its node executions, which leave out config nodes and agent sub-nodes.-`WorkflowService._execute_temporal` forwards both. The editor's Run dialog shows only `error` ("Workflow failed: …"), so before both kept it, every failed Run said "Unknown error" ([Known Errors #31](./errors.md)). Locked by `server/tests/temporal/test_run_errors.py`.+The editor's Run dialog shows `error` when a run fails ("Workflow failed: …") and `completed_nodes` / `total_nodes` when it succeeds ("N/M nodes completed"). Before these fields were set, a failed Run said "Unknown error" and a successful one said "0/0 nodes completed" ([Known Errors #31](./errors.md)). Locked by `server/tests/temporal/test_run_errors.py` and `server/tests/services/test_workflow_run_result.py`.## Connection Poolingdiff --git a/docs-internal/errors.md b/docs-internal/errors.mdindex f55933e8..dbf41b1c 100644--- a/docs-internal/errors.md+++ b/docs-internal/errors.md@@ -640,14 +640,19 @@ To reproduce the old failure on 3.12.8, run `kill_orphaned_opencompany_processesSee the [cronScheduler card](./node-logic-flows/workflow_triggers/cronScheduler.md).-## 31. A failed canvas Run says `Workflow failed: Unknown error`+## 31. A canvas Run says `Workflow failed: Unknown error` when it fails, and `0/0 nodes completed` when it succeeds-**Symptom**: Run a whole workflow with Temporal on (the default). When a node fails, the result dialog says `Workflow failed: Unknown error` and the response's `errors` list is empty, whatever the node's actual error was.+**Symptom**: Run a whole workflow.+- When a node fails, the result dialog says `Workflow failed: Unknown error`, whatever the node's actual error was. With Temporal on (the default), the response's `errors` list is empty too.+- When the run succeeds, the dialog says `Workflow executed successfully. 0/0 nodes completed.`-**Root cause**: `MachinaWorkflow.run` stops at the first failed node and returns that failure in `errors`, a list of `{node_id, error, ...}`. Only an empty graph returns a top-level `error`. `TemporalExecutor.execute_workflow` read only the top-level `error`, so it reported no errors, and `WorkflowService._execute_temporal` passed no `error` on at all. The dialog shows only `error`.+**Root cause**: the dialog shows only `error` for a failure and only `completed_nodes` / `total_nodes` for a success, and no execution path filled them in.+- `MachinaWorkflow.run` stops at the first failed node and returns that failure in `errors`, a list of `{node_id, error, ...}`. Only an empty graph returns a top-level `error`. `TemporalExecutor.execute_workflow` read only the top-level `error`, so it reported no errors, and `WorkflowService._execute_temporal` passed no `error` on at all.+- Without Temporal, the sequential fallback reported failures only in `node_results`, and the Redis-only parallel path only in `errors`. Neither set `error`.+- No path set `total_nodes` or `completed_nodes`, so the WebSocket handler's defaults of 0 reached the dialog.-**Fix**: the executor keeps the workflow's `errors` and sets `error` to the first message; an empty graph or a failed Temporal call becomes `[{"error": ...}]`. `_execute_temporal` forwards `error`. Locked by `server/tests/temporal/test_run_errors.py`. The result contract is in [Temporal Architecture → The run result](./TEMPORAL_ARCHITECTURE.md#4-the-run-result).+**Fix**:+- `51774a77`: the Temporal executor keeps the workflow's `errors` and sets `error` to the first message; an empty graph or a failed Temporal call becomes `[{"error": ...}]`. `_execute_temporal` forwards `error`.+- A follow-up gave the sequential and parallel paths the same `errors` and `error`, and made every path report `total_nodes` and `completed_nodes`. `MachinaWorkflow` now returns its `total_nodes`.-**Still open**:-- A successful Run's dialog says `0/0 nodes completed` on every path, because no path sets `total_nodes` or `completed_nodes`.-- Without Temporal, a failed Run still says `Unknown error`. The sequential path reports failures only in `node_results`, and the Redis-only parallel path only in `errors`.+Locked by `server/tests/temporal/test_run_errors.py` and `server/tests/services/test_workflow_run_result.py`. The result contract is in [Temporal Architecture → The run result](./TEMPORAL_ARCHITECTURE.md#4-the-run-result).diff --git a/server/services/execution/executor.py b/server/services/execution/executor.pyindex 0e3f0d7c..cfd415c8 100644--- a/server/services/execution/executor.py+++ b/server/services/execution/executor.py@@ -171,6 +171,7 @@ class WorkflowExecutor:"execution_id": ctx.execution_id,"status": ctx.status.value,"nodes_executed": ctx.get_completed_nodes(),+ "total_nodes": len(ctx.node_executions),"outputs": ctx.outputs,"errors": ctx.errors,"execution_time": time.time() - start_time,diff --git a/server/services/temporal/executor.py b/server/services/temporal/executor.pyindex a342d4ed..a6e44b85 100644--- a/server/services/temporal/executor.py+++ b/server/services/temporal/executor.py@@ -188,6 +188,8 @@ class TemporalExecutor:"outputs": result.get("outputs", {}),"errors": errors,"error": errors[0].get("error") if errors else None,+ "total_nodes": result.get("total_nodes", 0),+ "completed_nodes": len(result.get("execution_trace", [])),"execution_time": execution_time,"temporal_execution": True,"timestamp": datetime.now().isoformat(),diff --git a/server/services/temporal/workflow.py b/server/services/temporal/workflow.pyindex 8692f837..fbf6bce9 100644--- a/server/services/temporal/workflow.py+++ b/server/services/temporal/workflow.py@@ -389,7 +389,8 @@ class MachinaWorkflow:- tenant_id: Tenant identifier for multi-tenancyReturns:- Dict with success, outputs, execution_trace, and errors+ Dict with success, outputs, execution_trace, errors, and+ total_nodes (the size of the executable graph)"""nodes = workflow_data.get("nodes", [])edges = workflow_data.get("edges", [])@@ -732,6 +733,7 @@ class MachinaWorkflow:"outputs": outputs,"execution_trace": execution_trace,"errors": errors if errors else None,+ "total_nodes": len(node_map),}def _get_node_inputs(diff --git a/server/services/workflow.py b/server/services/workflow.pyindex a7792ecb..09519487 100644--- a/server/services/workflow.py+++ b/server/services/workflow.py@@ -43,6 +43,23 @@ _parallel_user_id: ContextVar[str] = ContextVar()+# A Run's result reports failures as ``errors`` plus their first message as+# ``error`` on every path, because the editor's Run dialog shows only+# ``error`` (docs-internal/TEMPORAL_ARCHITECTURE.md, "4. The run result").+def _run_errors(result: Dict[str, Any]) -> List[Dict[str, Any]]:+ """A run's failures. A run-level ``error`` alone becomes the one entry."""+ if result.get("errors"):+ return list(result["errors"])+ return [{"error": result["error"]}] if result.get("error") else []+++def _node_failure(node_id: str, result: Dict[str, Any]) -> Dict[str, Any]:+ """One node's entry in ``errors``, shaped like MachinaWorkflow's."""+ failure = {"node_id": node_id, "error": result.get("error") or "Unknown error"}+ failure.update({key: result[key] for key in ("hint", "requires_user_action", "retryable") if key in result})+ return failure++class WorkflowService:"""Workflow execution and deployment service.@@ -453,6 +470,8 @@ class WorkflowService:"outputs": result.get("outputs", {}),"errors": result.get("errors", []),"error": result.get("error"),+ "total_nodes": result.get("total_nodes", 0),+ "completed_nodes": result.get("completed_nodes", 0),"execution_time": result.get("execution_time", time.time() - start_time),"temporal_execution": True,"timestamp": datetime.now().isoformat(),@@ -486,12 +505,17 @@ class WorkflowService:finally:_parallel_user_id.reset(token)+ errors = _run_errors(result)+ nodes_executed = result.get("nodes_executed", [])return {"success": result.get("success", False),"execution_id": result.get("execution_id"),- "nodes_executed": result.get("nodes_executed", []),+ "nodes_executed": nodes_executed,"outputs": result.get("outputs", {}),- "errors": result.get("errors", []),+ "errors": errors,+ "error": errors[0].get("error") if errors else None,+ "total_nodes": result.get("total_nodes", 0),+ "completed_nodes": len(nodes_executed),"execution_time": result.get("execution_time", time.time() - start_time),"parallel_execution": True,"timestamp": datetime.now().isoformat(),@@ -591,11 +615,16 @@ class WorkflowService:if not result.get("success") and self._settings.get("stop_on_error"):break+ errors = [_node_failure(node_id, result) for node_id, result in results.items() if not result.get("success")]return {"success": all(r.get("success", False) for r in results.values()),"execution_id": execution_id,"nodes_executed": executed,"node_results": results,+ "errors": errors,+ "error": errors[0]["error"] if errors else None,+ "total_nodes": len(execution_order),+ "completed_nodes": len(executed) - len(errors),"execution_time": time.time() - start_time,"parallel_execution": False,"timestamp": datetime.now().isoformat(),diff --git a/server/tests/services/test_workflow_run_result.py b/server/tests/services/test_workflow_run_result.pynew file mode 100644index 00000000..fa53627f--- /dev/null+++ b/server/tests/services/test_workflow_run_result.py@@ -0,0 +1,89 @@+"""Every Run path reports a failure and its node count the same way.++The editor's Run dialog shows ``error`` when a run fails and+``completed_nodes`` / ``total_nodes`` when it succeeds, whichever path ran+it: Temporal (tests/temporal/test_run_errors.py), the Redis-only parallel+engine or the sequential fallback (here).+"""++from __future__ import annotations++import time+from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest++from services.workflow import WorkflowService+++def _chain():+ graph_nodes = [+ {"id": "start-1", "type": "start", "data": {}},+ {"id": "a", "type": "console", "data": {}},+ {"id": "b", "type": "console", "data": {}},+ ]+ edges = [{"id": "e1", "source": "start-1", "target": "a"}, {"id": "e2", "source": "a", "target": "b"}]+ return graph_nodes, edges+++def _sequential_service(results):+ service = WorkflowService.__new__(WorkflowService)+ service._settings = {}++ async def execute_node(*, node_id, **_kwargs):+ return results.get(node_id, {"success": True, "result": {}})++ service.execute_node = execute_node+ return service+++def _parallel_service(executor_result):+ service = WorkflowService.__new__(WorkflowService)+ executor = SimpleNamespace(execute_workflow=AsyncMock(return_value=executor_result))+ service._get_workflow_executor = lambda _status_callback: executor+ return service+++async def test_the_sequential_path_reports_a_failed_node():+ service = _sequential_service({"a": {"success": False, "error": "a broke", "hint": "Check a"}})++ result = await service._execute_sequential(*_chain(), "session", None, time.time())++ assert result["success"] is False+ assert result["errors"] == [{"node_id": "a", "error": "a broke", "hint": "Check a"}]+ assert result["error"] == "a broke"+ # stop_on_error is off, so b still ran.+ assert (result["completed_nodes"], result["total_nodes"]) == (2, 3)+++async def test_the_sequential_path_counts_a_clean_run():+ result = await _sequential_service({})._execute_sequential(*_chain(), "session", None, time.time())++ assert result["success"] is True+ assert (result["errors"], result["error"]) == ([], None)+ assert (result["completed_nodes"], result["total_nodes"]) == (3, 3)+++async def test_the_parallel_path_passes_a_failed_node_on():+ failure = {"node_id": "a", "error": "a broke", "timestamp": 1.0}+ service = _parallel_service({"success": False, "nodes_executed": ["start-1"], "errors": [failure], "total_nodes": 3})++ result = await service._execute_parallel(*_chain(), "session", None, time.time())++ assert result["errors"] == [failure]+ assert result["error"] == "a broke"+ assert (result["completed_nodes"], result["total_nodes"]) == (1, 3)+++async def test_the_parallel_path_keeps_a_run_level_error():+ service = _parallel_service({"success": False, "status": "cancelled", "error": "Cancelled by user"})++ result = await service._execute_parallel(*_chain(), "session", None, time.time())++ assert result["errors"] == [{"error": "Cancelled by user"}]+ assert result["error"] == "Cancelled by user"diff --git a/server/tests/temporal/test_run_errors.py b/server/tests/temporal/test_run_errors.pyindex fcbdb1ad..8e85bbd5 100644--- a/server/tests/temporal/test_run_errors.py+++ b/server/tests/temporal/test_run_errors.py@@ -1,14 +1,16 @@-"""A failed canvas Run on the Temporal path reports why it failed.+"""A canvas Run on the Temporal path reports why it failed and what it ran.MachinaWorkflow returns per-node failures as ``errors`` (a list of``{node_id, error, ...}``), and only an empty graph returns a top-level-``error``. The Run response the editor shows reads ``error`` for its message-and ``errors`` for detail, so both must survive TemporalExecutor and-WorkflowService._execute_temporal.+``error``. It also counts the nodes in its executable graph as+``total_nodes``. The editor's Run dialog shows ``error`` when a run fails and+``completed_nodes`` / ``total_nodes`` when it succeeds, so all of them must+survive TemporalExecutor and WorkflowService._execute_temporal."""from __future__ import annotations+import asyncioimport timefrom types import SimpleNamespacefrom unittest.mock import AsyncMock, MagicMock@@ -63,7 +65,14 @@ async def test_a_successful_run_reports_no_error():@pytest.mark.asyncio-async def test_the_workflow_service_passes_the_message_on():+async def test_the_run_counts_its_nodes():+ result = await _run({"success": True, "outputs": {}, "execution_trace": ["start-1", "py-1"], "total_nodes": 2})++ assert (result["completed_nodes"], result["total_nodes"]) == (2, 2)+++async def test_the_workflow_service_passes_the_message_and_counts_on():from services.workflow import WorkflowServiceservice = WorkflowService.__new__(WorkflowService)@@ -71,7 +80,15 @@ async def test_the_workflow_service_passes_the_message_on():failure = {"node_id": "n1", "error": "boom"}service._temporal_executor = SimpleNamespace(execute_workflow=AsyncMock(- return_value={"success": False, "nodes_executed": [], "outputs": {}, "errors": [failure], "error": "boom"}+ return_value={+ "success": False,+ "nodes_executed": ["start-1"],+ "outputs": {},+ "errors": [failure],+ "error": "boom",+ "total_nodes": 3,+ "completed_nodes": 1,+ }))@@ -79,3 +96,50 @@ async def test_the_workflow_service_passes_the_message_on():assert result["errors"] == [failure]assert result["error"] == "boom"+ assert (result["completed_nodes"], result["total_nodes"]) == (1, 3)+++async def _run_machina_workflow(monkeypatch, failing=()):+ """Run the real MachinaWorkflow body on start -> py-1 -> py-2 without a+ Temporal server (the pattern of test_machina_workflow_loop.py)."""+ from temporalio import workflow as temporal_workflow++ from services.temporal.workflow import MachinaWorkflow++ def start_activity(name, **kwargs):+ node_id = kwargs["args"][0]["node_id"]+ future = asyncio.get_event_loop().create_future()+ future.set_result({"success": False, "error": f"{node_id} broke"} if node_id in failing else {"success": True, "result": {}})+ return future++ async def execute_activity(*args, **kwargs):+ return None++ monkeypatch.setattr(temporal_workflow, "logger", MagicMock())+ monkeypatch.setattr(temporal_workflow, "patched", lambda _patch_id: True)+ monkeypatch.setattr(temporal_workflow, "start_activity", start_activity)+ monkeypatch.setattr(temporal_workflow, "execute_activity", execute_activity)+ monkeypatch.setattr(+ MachinaWorkflow,+ "_resolve_dispatch",+ lambda self, node_type, **_kwargs: {"kind": "activity", "name": f"node.{node_type}.v1", "queue": None},+ )+ ids = ["start-1", "py-1", "py-2"]+ graph_nodes = [{"id": node_id, "type": "start" if node_id == "start-1" else "pythonExecutor", "data": {}} for node_id in ids]+ edges = [+ {"id": f"e{index}", "source": source, "target": target, "sourceHandle": "output-main", "targetHandle": "input-main"}+ for index, (source, target) in enumerate(zip(ids, ids[1:]))+ ]+ return await MachinaWorkflow().run(+ {"nodes": graph_nodes, "edges": edges, "session_id": "test", "workflow_id": "wf-1", "execution_id": "wf-1-run"}+ )+++async def test_the_workflow_counts_the_nodes_of_its_run(monkeypatch):+ finished = await _run_machina_workflow(monkeypatch)+ failed = await _run_machina_workflow(monkeypatch, failing={"py-1"})++ assert (finished["success"], finished["execution_trace"], finished["total_nodes"]) == (True, ["start-1", "py-1", "py-2"], 3)+ assert (failed["success"], failed["execution_trace"], failed["total_nodes"]) == (False, ["start-1"], 3)+ assert failed["errors"] == [{"node_id": "py-1", "error": "py-1 broke"}]