Skip to content

Self-hosted workflows for coding-agent delivery. Orchestrate CodeBuddy, Codex and Claude Code with scripts and human approval gates. Includes Local workflows for coding assistants.

test(e2e): add deterministic end-to-end suite and required CI check (#41)

Yanami-2Kf112b7d93312
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index 2a5f0846..0c34789a 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -21,6 +21,7 @@ List commands that were actually run and mark anything blocked or not run.
```text
bash scripts/validate.sh
bash scripts/smoke-install.sh
+bash scripts/e2e.sh
bash scripts/scan-secrets.sh
```
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
new file mode 100644
index 00000000..ab15b727
--- /dev/null
+++ b/.github/workflows/e2e.yml
@@ -0,0 +1,48 @@
+name: e2e
+
+on:
+ pull_request:
+ push:
+ branches: [main, master]
+
+permissions:
+ contents: read
+
+concurrency:
+ group: e2e-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ e2e:
+ name: e2e (python ${{ matrix.python-version }})
+ runs-on: ubuntu-latest
+ timeout-minutes: 15
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: ["3.8", "3.12"]
+ steps:
+ - uses: actions/[email protected]
+ - uses: actions/[email protected]
+ with:
+ python-version: ${{ matrix.python-version }}
+ - uses: actions/[email protected]
+ with:
+ node-version: "22"
+ - run: python3 -m pip install pyyaml==6.0.3
+ - name: Run E2E suite (packaged CLI, fake host Agent, no API keys)
+ run: bash scripts/e2e.sh
+
+ e2e-gate:
+ # Single stable check name to mark as required in branch protection.
+ name: e2e-required
+ if: always()
+ needs: [e2e]
+ runs-on: ubuntu-latest
+ steps:
+ - name: Require every E2E job to pass
+ run: |
+ if [[ "${{ needs.e2e.result }}" != "success" ]]; then
+ echo "E2E result: ${{ needs.e2e.result }}" >&2
+ exit 1
+ fi
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 07d77a6d..64ddd17b 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -80,6 +80,7 @@ jobs:
bash scripts/smoke-install.sh
bash scripts/smoke-npm.sh
bash scripts/smoke-curl-install.sh
+ bash scripts/e2e.sh
bash scripts/scan-secrets.sh
- name: Log in to GHCR
diff --git a/AGENTS.md b/AGENTS.md
index d61dc783..e05dedad 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -18,6 +18,7 @@ Run these commands before reporting a workflow change complete:
python3 scripts/build-classic-hosts.py --check
bash scripts/validate.sh
bash scripts/smoke-install.sh
+bash scripts/e2e.sh
bash scripts/scan-secrets.sh
```
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index ce58b6cd..f7abeaf8 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -11,7 +11,7 @@ Thanks for contributing to LoopForge.
3. Do not commit credentials, local permission configurations, or production data.
4. First indicate whether a change belongs to Classic, Portable, or shared behaviour contracts. Portable only modifies `skills/`; Classic `.cursor/.claude` files are not edited directly — run `build-classic-hosts.py --write`.
5. When modifying shared workflow contracts, describe the impact on both editions separately; do not copy runtime files between them.
-6. Run `bash scripts/validate.sh` and `bash scripts/smoke-install.sh` before submitting.
+6. Run `bash scripts/validate.sh`, `bash scripts/smoke-install.sh` and `bash scripts/e2e.sh` before submitting. The E2E suite drives the installed workflow with a fake host Agent and recorded stage outputs, so it needs no API key; see [tests/e2e/README.md](tests/e2e/README.md).
## Pull request requirements
diff --git a/CONTRIBUTING.zh-CN.md b/CONTRIBUTING.zh-CN.md
index 79c2ab58..cf316cc5 100644
--- a/CONTRIBUTING.zh-CN.md
+++ b/CONTRIBUTING.zh-CN.md
@@ -11,7 +11,7 @@
3. 不提交任何凭据、本地权限配置或生产数据。
4. 先标明改动属于 Classic、Portable 或共享行为合同。Portable 只修改 `skills/`;Classic 的 `.cursor/.claude` 不直接修改,运行 `build-classic-hosts.py --write`。
5. 修改共享工作流契约时分别说明两个 edition 的影响;不要在二者之间复制运行文件。
-6. 提交前运行 `bash scripts/validate.sh` 和 `bash scripts/smoke-install.sh`。
+6. 提交前运行 `bash scripts/validate.sh`、`bash scripts/smoke-install.sh` 和 `bash scripts/e2e.sh`。E2E 用假宿主 Agent 和录制的阶段产物驱动已安装的工作流,不需要 API Key,说明见 [tests/e2e/README.md](tests/e2e/README.md)。
## Pull Request 要求
diff --git a/scripts/e2e.sh b/scripts/e2e.sh
new file mode 100755
index 00000000..01220d49
--- /dev/null
+++ b/scripts/e2e.sh
@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+# End-to-end tests: install the packaged CLI into throw-away projects and drive
+# every host through install, the full Portable workflow (with a fake host
+# Agent), resume, failure paths, Classic hooks and the local dashboard.
+# Needs Python 3.8+, Node.js 20+ and npm; no network access or API keys.
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+export PYTHONDONTWRITEBYTECODE=1
+
+python3 -m unittest discover -s "$ROOT_DIR/tests/e2e" -t "$ROOT_DIR/tests/e2e" -p 'test_e2e_*.py' -v "$@"
+echo "E2E tests passed."
diff --git a/tests/e2e/README.md b/tests/e2e/README.md
new file mode 100644
index 00000000..a0656e29
--- /dev/null
+++ b/tests/e2e/README.md
@@ -0,0 +1,20 @@
+# E2E tests
+
+`bash scripts/e2e.sh` runs the end-to-end suite. It needs Python 3.8+, Node.js 20+ and npm, and no network access, API key or real coding Agent.
+
+What it does:
+
+1. Builds the npm tarball once (`npm pack`), installs it into a temporary prefix, and uses that `loopforge` binary, which is the same path users take with `npx loopforge-cli`.
+2. Installs every supported host and edition into throw-away git projects, then checks the entrypoint files, `status`, `doctor`, `update` and a clean `uninstall` (`test_e2e_install.py`).
+3. Drives the installed Portable workflow end to end with [`fake_agent.py`](fake_agent.py), a deterministic stand-in for the host Agent. It follows the Skill's `resume`-driven protocol (`init`, `prepare --emit-prompt`, `assign`, `start`, `finish`, `approve`, `decide-knowledge`), but writes the stage outputs recorded in [`fixtures/scenario-medium.json`](fixtures/scenario-medium.json) instead of calling an LLM. The developer stage really changes [`fixtures/sample_project`](fixtures/sample_project/slugify.py), and the test stage really runs its tests. The suite covers every host, resume after a crash and after an interruption, the requirement confirmation gate, review rejection, unfilled artifacts and repeated failures (`test_e2e_portable_workflow.py`).
+4. Runs the Classic CodeBuddy hooks from an installed project: auto-dispatch routing between roles, the fallbacks to the main Agent, and a recorded session flowing through the observability hooks into the local dashboard, which is served over HTTP (`test_e2e_classic_runtime.py`).
+
+Useful options:
+
+```bash
+LOOPFORGE_E2E_SOURCE=1 bash scripts/e2e.sh # run the CLI from src/ and skip npm pack
+bash scripts/e2e.sh -k resume # run only tests matching a pattern
+python3 tests/e2e/fake_agent.py --help # drive a project by hand
+```
+
+When a test fails, the assertion message includes the fake Agent transcript (`.fake-agent/transcript.jsonl` in the project) that shows each action and the command that was rejected.
diff --git a/tests/e2e/fake_agent.py b/tests/e2e/fake_agent.py
new file mode 100644
index 00000000..29f9d905
--- /dev/null
+++ b/tests/e2e/fake_agent.py
@@ -0,0 +1,280 @@
+#!/usr/bin/env python3
+"""Deterministic stand-in for a host coding Agent (Claude Code, Codex, ...).
+
+A real host loads the installed ``devflow`` Skill and follows its contract:
+``init`` once, then repeatedly ask ``workflow_state.py resume`` for the next
+action and carry it out (prepare a stage prompt, spawn and register a stage
+executor, start, let the executor write artifacts, finish, ask the user to
+approve the requirement, ...).
+
+This script follows exactly that protocol against the *installed* copy of the
+Skill in a project, but replaces the LLM with recorded stage outputs from a
+scenario fixture. No network access, API key or real Agent CLI is needed, so
+the run is deterministic in CI.
+
+Exit codes: 0 completed, 3 waiting for the user, 4 blocked, 75 interrupted,
+2 contract violation (a command the real Agent would run was rejected).
+"""
+
+import argparse
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import uuid
+from pathlib import Path
+
+HOST_SKILL_DIRS = {
+ "codebuddy": ".codebuddy/skills",
+ "codex": ".agents/skills",
+ "cursor": ".cursor/skills",
+ "claude": ".claude/skills",
+ "pi": ".pi/skills",
+}
+FIXTURES = Path(__file__).resolve().parent / "fixtures"
+EXIT_COMPLETED, EXIT_CONTRACT, EXIT_WAITING, EXIT_BLOCKED, EXIT_INTERRUPTED = 0, 2, 3, 4, 75
+
+
+class ContractError(Exception):
+ pass
+
+
+class FakeAgent:
+ def __init__(self, args):
+ self.args = args
+ self.host = args.host
+ self.root = args.project_root.resolve()
+ self.scenario = json.loads(args.scenario.read_text(encoding="utf-8"))
+ self.session = uuid.uuid4().hex[:8]
+ self.skill_root = self.root / HOST_SKILL_DIRS[self.host] / "devflow"
+ self.workflow = self.skill_root / "scripts" / "workflow_state.py"
+ self.transcript = self.root / ".fake-agent" / "transcript.jsonl"
+ self.transcript.parent.mkdir(parents=True, exist_ok=True)
+ self.actions = 0
+ self.review_rejections_left = args.review_reject
+ self.failures_left = dict(args.fail_stage)
+ self.fresh_team = bool(args.resume and args.host == "codebuddy")
+ self.rounds = {}
+
+ # -- helpers ---------------------------------------------------------
+ def log(self, kind, **fields):
+ record = {"session": self.session, "kind": kind}
+ record.update(fields)
+ with self.transcript.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(record, ensure_ascii=False) + "\n")
+
+ def run(self, *argv, check=True):
+ command = [sys.executable, str(self.workflow)] + [str(item) for item in argv]
+ result = subprocess.run(
+ command, cwd=str(self.root), text=True, capture_output=True,
+ env=dict(os.environ, PYTHONDONTWRITEBYTECODE="1"),
+ )
+ self.log("command", argv=[str(item) for item in argv[:3]], rc=result.returncode)
+ if check and result.returncode != 0:
+ raise ContractError(
+ "workflow_state.py {} rejected (rc={}): {}".format(
+ argv[0], result.returncode, (result.stderr or result.stdout).strip()
+ )
+ )
+ return result
+
+ def state_path(self):
+ return self.root / "artifacts" / self.args.slug / "workflow-state.json"
+
+ def state(self):
+ return json.loads(self.state_path().read_text(encoding="utf-8"))
+
+ # -- the host side of the Skill contract ----------------------------
+ def load_skill(self):
+ skill = self.skill_root / "SKILL.md"
+ if not skill.is_file():
+ raise ContractError("host cannot find installed Skill: {}".format(skill))
+ text = skill.read_text(encoding="utf-8")
+ for needle in ("workflow_state.py", "prepare", "approve"):
+ if needle not in text:
+ raise ContractError("SKILL.md no longer documents `{}`".format(needle))
+ # The Agent follows the links in SKILL.md; every one must ship with the install.
+ for target in re.findall(r"\]\(([^)#]+)\)", text):
+ if "://" not in target and not (self.skill_root / target).exists():
+ raise ContractError("SKILL.md links to missing file: " + target)
+ contract = (self.skill_root / "references" / "workflow-contract.md").read_text(encoding="utf-8")
+ if "resume --project-root . --slug" not in contract:
+ raise ContractError("workflow-contract.md no longer documents `resume --slug`")
+ self.log("skill-loaded", path=str(skill.relative_to(self.root)))
+
+ def init(self):
+ output = self.run(
+ "init", "--project-root", self.root, "--slug", self.args.slug,
+ "--size", "medium", "--mode", "auto", "--execution-mode", "isolated",
+ "--host-adapter", self.host, "--coordinator-id", "fake-main",
+ ).stdout.strip()
+ self.log("init", state=output)
+
+ def next_action(self):
+ argv = ["resume", "--project-root", self.root, "--slug", self.args.slug]
+ if self.fresh_team:
+ # Per workflow-contract.md a new CodeBuddy main session resumes with a fresh Team.
+ argv.append("--fresh-team")
+ self.fresh_team = False
+ return json.loads(self.run(*argv).stdout)
+
+ def loop(self):
+ while True:
+ action = self.next_action()
+ kind = action.get("action")
+ self.log("action", action=kind, stage=action.get("stage"))
+ if kind == "completed":
+ return EXIT_COMPLETED
+ if kind == "blocked":
+ return EXIT_BLOCKED
+ if kind == "approve":
+ if action.get("requires_user_confirmation") and not self.args.user_confirms:
+ self.log("waiting-for-user", stage=action["stage"])
+ return EXIT_WAITING
+ extra = ["--user-confirmed"] if action.get("requires_user_confirmation") else []
+ self.run("approve", "--state", self.state_path(), "--stage", action["stage"], *extra)
+ elif kind == "rotate-team":
+ rotated = json.loads(self.run("rotate-team", "--state", self.state_path()).stdout)
+ self.log("team-rotated", old=rotated["old_team_name"], new=rotated["team_name"])
+ elif kind == "assess-knowledge":
+ decision = self.scenario["knowledge_decision"]
+ self.run(
+ "decide-knowledge", "--state", self.state_path(),
+ "--decision", decision["decision"], "--reason", decision["reason"],
+ )
+ elif kind == "prepare":
+ prompt = self.run(
+ "prepare", "--state", self.state_path(), "--stage", action["stage"],
+ "--emit-prompt", "--request", self.scenario["request"],
+ ).stdout
+ if not prompt.strip():
+ raise ContractError("empty stage prompt for " + action["stage"])
+ self.log("prompt", stage=action["stage"], role=action.get("executor_role"),
+ chars=len(prompt), mentions_role=action.get("executor_role", "") in prompt)
+ elif kind == "spawn-and-assign":
+ executor_id = "fake-{}-{}".format(action["stage"].lower(), uuid.uuid4().hex[:6])
+ argv = [
+ "assign", "--state", self.state_path(), "--stage", action["stage"],
+ "--role", action["executor_role"], "--executor-type", action["executor_type"],
+ "--executor-id", executor_id, "--dispatch-tool", action["dispatch_tool"],
+ ]
+ if action.get("team_name"):
+ argv += ["--team-name", action["team_name"]]
+ self.run(*argv)
+ self.log("spawned", stage=action["stage"], executor_id=executor_id,
+ executor_type=action["executor_type"],
+ dispatch_tool=action["dispatch_tool"])
+ elif kind in ("start-assigned", "start-coordinator"):
+ self.run("start", "--state", self.state_path(), "--stage", action["stage"])
+ if self.args.crash_in == action["stage"]:
+ self.log("crash", stage=action["stage"])
+ return EXIT_INTERRUPTED
+ self.execute_stage(action["stage"])
+ elif kind == "continue-executor":
+ self.execute_stage(action["stage"])
+ else:
+ raise ContractError("unexpected resume action: {}".format(action))
+ self.actions += 1
+ if self.args.stop_after and self.actions >= self.args.stop_after:
+ self.log("interrupted", after=self.actions)
+ return EXIT_INTERRUPTED
+
+ # -- the fake stage executor ----------------------------------------
+ def execute_stage(self, stage):
+ result = "completed"
+ evidence = self.scenario["stages"][stage]
+ overrides = dict(self.scenario["overrides"])
+ if stage == "REVIEW" and self.review_rejections_left > 0:
+ self.review_rejections_left -= 1
+ result = "failed"
+ evidence = self.scenario["stages"]["REVIEW_REJECTED"]
+ for key in list(overrides):
+ if "通过" in key and "不通过" in key:
+ overrides[key] = evidence
+ if self.failures_left.get(stage, 0) > 0:
+ self.failures_left[stage] -= 1
+ result = "failed"
+ for relative, source in self.scenario.get("code_changes", {}).get(stage, {}).items():
+ shutil.copyfile(str(FIXTURES / source), str(self.root / relative))
+ if stage == "TEST":
+ verify = subprocess.run(
+ [sys.executable] + self.scenario["verify_command"], cwd=str(self.root),
+ text=True, capture_output=True,
+ )
+ self.log("verify", rc=verify.returncode, tail=verify.stderr.strip().splitlines()[-1:])
+ if verify.returncode != 0:
+ result = "failed"
+ evidence = "测试失败:" + verify.stderr.strip()[-200:]
+ self.rounds[stage] = self.rounds.get(stage, 0) + 1
+ if stage not in self.args.leave_placeholders:
+ note = "(session {},第 {} 轮)".format(self.session, self.rounds[stage])
+ self.fill_artifacts(stage, overrides, evidence + note)
+ finished = self.run(
+ "finish", "--state", self.state_path(), "--stage", stage, "--result", result,
+ check=False,
+ )
+ self.log("finish", stage=stage, result=result, rc=finished.returncode,
+ error=finished.stderr.strip()[-300:] if finished.returncode else None)
+ if finished.returncode != 0:
+ raise ContractError("finish {} rejected: {}".format(stage, finished.stderr.strip()))
+
+ def fill_artifacts(self, stage, overrides, evidence):
+ """Write every required artifact afresh from its template, like a real executor."""
+ artifact_root = self.state_path().parent
+ templates = self.skill_root / "templates"
+ manifest = json.loads((templates / "manifest.json").read_text(encoding="utf-8"))
+ sources = {item["path"]: item["template"] for item in manifest["artifacts"]}
+ for relative in self.required_artifacts(stage):
+ text = (templates / sources[relative]).read_text(encoding="utf-8")
+ for source, target in overrides.items():
+ text = text.replace(source, target)
+ text = re.sub(r"\{\{[^{}]+\}\}", evidence, text)
+ path = artifact_root / relative
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(text, encoding="utf-8")
+
+ def required_artifacts(self, stage):
+ sys.path.insert(0, str(self.skill_root / "scripts"))
+ try:
+ from template_registry import required_artifacts # type: ignore
+ finally:
+ sys.path.pop(0)
+ return required_artifacts(stage, bool(self.state().get("requires_design_artifacts")))
+
+ def main(self):
+ try:
+ self.load_skill()
+ if not self.args.resume:
+ self.init()
+ code = self.loop()
+ except ContractError as error:
+ self.log("contract-error", message=str(error))
+ print("fake-agent: " + str(error), file=sys.stderr)
+ return EXIT_CONTRACT
+ self.log("exit", code=code)
+ return code
+
+
+def parse_args(argv=None):
+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+ parser.add_argument("--host", choices=sorted(HOST_SKILL_DIRS), required=True)
+ parser.add_argument("--project-root", type=Path, required=True)
+ parser.add_argument("--slug", default="e2e-task")
+ parser.add_argument("--scenario", type=Path, default=FIXTURES / "scenario-medium.json")
+ parser.add_argument("--resume", action="store_true", help="new session; do not init")
+ parser.add_argument("--no-user-confirmation", dest="user_confirms", action="store_false")
+ parser.add_argument("--stop-after", type=int, default=0, help="interrupt after N actions")
+ parser.add_argument("--crash-in", help="die right after starting this stage")
+ parser.add_argument("--review-reject", type=int, default=0, help="reject REVIEW N times")
+ parser.add_argument("--leave-placeholders", action="append", default=[],
+ help="executor of this stage leaves the template unfilled")
+ parser.add_argument("--fail-stage", action="append", default=[],
+ type=lambda item: (item.split(":")[0], int(item.split(":")[1])),
+ help="STAGE:N report failure N times")
+ return parser.parse_args(argv)
+
+
+if __name__ == "__main__":
+ sys.exit(FakeAgent(parse_args()).main())
diff --git a/tests/e2e/fixtures/patches/slugify.py b/tests/e2e/fixtures/patches/slugify.py
new file mode 100644
index 00000000..ae204032
--- /dev/null
+++ b/tests/e2e/fixtures/patches/slugify.py
@@ -0,0 +1,8 @@
+"""Tiny fixture module used by the LoopForge E2E suite (fixed version)."""
+
+import re
+
+
+def slugify(value):
+ words = re.findall(r"[a-z0-9]+", value.lower())
+ return "-".join(words)
diff --git a/tests/e2e/fixtures/sample_project/check_slugify.py b/tests/e2e/fixtures/sample_project/check_slugify.py
new file mode 100644
index 00000000..b3dd6f39
--- /dev/null
+++ b/tests/e2e/fixtures/sample_project/check_slugify.py
@@ -0,0 +1,15 @@
+import unittest
+
+from slugify import slugify
+
+
+class SlugifyAcceptance(unittest.TestCase):
+ def test_spaces_become_dashes(self):
+ self.assertEqual("order-status-filter", slugify(" Order Status Filter "))
+
+ def test_symbols_are_removed(self):
+ self.assertEqual("v2-release", slugify("v2 release!"))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/e2e/fixtures/sample_project/slugify.py b/tests/e2e/fixtures/sample_project/slugify.py
new file mode 100644
index 00000000..6a0ce52e
--- /dev/null
+++ b/tests/e2e/fixtures/sample_project/slugify.py
@@ -0,0 +1,9 @@
+"""Tiny fixture module used by the LoopForge E2E suite.
+
+The fake developer agent replaces this file with ``patches/slugify.py``; the
+fake test engineer then runs ``check_slugify.py`` to prove the change works.
+"""
+
+
+def slugify(value):
+ return value.lower()
diff --git a/tests/e2e/fixtures/scenario-medium.json b/tests/e2e/fixtures/scenario-medium.json
new file mode 100644
index 00000000..2364e33c
--- /dev/null
+++ b/tests/e2e/fixtures/scenario-medium.json
@@ -0,0 +1,32 @@
+{
+ "description": "Recorded stage outputs for a medium task: fix slugify() in the sample project. Each stage executor fills the real templates produced by `workflow_state.py prepare` with this evidence.",
+ "request": "修复 slugify:空格转连字符、去掉符号,并补充验收测试",
+ "knowledge_decision": {"decision": "skip", "reason": "只有常规变更和测试结果,没有新增可复用约束"},
+ "overrides": {
+ "{{说明当前实现、关键模块、仓库相对路径、约束和非目标}}": "当前实现位于 `slugify.py`,仅修改该文件;验收用例在 `check_slugify.py`。",
+ "{{动作导向的任务名}}": "修复 slugify 规范化逻辑",
+ "{{无,或任务 ID}}": "无",
+ "{{仓库相对路径}}": "slugify.py",
+ "{{工作目录}}": ".",
+ "{{项目原生验证命令}}": "python3 -m unittest check_slugify",
+ "{{明确写“通过”或“不通过”;只有无阻断问题时才能写“通过”}}": "通过,无阻断问题。",
+ "{{无接口变动时写“无接口变动”;有新增、修改或删除时生成文档并写“已生成:`03-code/api-docs.md`”}}": "无接口变动",
+ "{{无接口变动时写“无接口变动”;有新增、修改或删除时生成文档并写“已生成:`01-solo/api-docs.md`”}}": "无接口变动",
+ "{{列出仓库实际已有的测试层级及证据路径/配置/命令;若已有 E2E,明确其覆盖边界;若没有,不要为流程强行引入新框架}}": "现有 unittest 验收用例 `check_slugify.py`;没有浏览器 E2E 框架。",
+ "{{逐项记录“验收标准或风险 | 项目已有的合适测试层级 | 用例或命令 | 结果”;项目已有 E2E 且本次行为属于其覆盖边界时,必须包含相应 E2E 用例}}": "空格转连字符、去掉符号 | unit | `python3 -m unittest check_slugify` | 通过"
+ },
+ "stages": {
+ "REQUIREMENT": "目标:`slugify()` 把空格转成连字符并去掉符号;验收:`python3 -m unittest check_slugify` 通过。",
+ "DESIGN": "用正则提取 `[a-z0-9]+` 片段后以连字符连接,只改 `slugify.py`,回滚即恢复原文件。",
+ "IMPLEMENT": "已修改 `slugify.py`;运行 `python3 -m unittest check_slugify`,结果 passed。",
+ "REVIEW": "审查 `slugify.py` 的 diff,未发现阻断问题。",
+ "REVIEW_REJECTED": "不通过:`slugify.py` 未处理符号,阻断级别,建议改为按字母数字切分。",
+ "TEST": "运行 `python3 -m unittest check_slugify`,2 个用例全部通过。",
+ "KNOWLEDGE": "无新增经验。",
+ "SUMMARY": "交付完成;`python3 -m unittest check_slugify` 已通过,无剩余风险。"
+ },
+ "code_changes": {
+ "IMPLEMENT": {"slugify.py": "patches/slugify.py"}
+ },
+ "verify_command": ["-m", "unittest", "check_slugify"]
+}
diff --git a/tests/e2e/harness.py b/tests/e2e/harness.py
new file mode 100644
index 00000000..592482cc
--- /dev/null
+++ b/tests/e2e/harness.py
@@ -0,0 +1,143 @@
+"""Shared helpers for the LoopForge E2E suite.
+
+By default the suite tests what users actually install: it builds the npm
+tarball once (``npm pack``), installs it into a throw-away prefix and drives the
+resulting ``loopforge`` binary. Set ``LOOPFORGE_E2E_SOURCE=1`` to run the CLI
+straight from ``src/`` for a faster local loop.
+"""
+
+import atexit
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+REPO = Path(__file__).resolve().parents[2]
+E2E = Path(__file__).resolve().parent
+FIXTURES = E2E / "fixtures"
+FAKE_AGENT = E2E / "fake_agent.py"
+
+sys.path.insert(0, str(REPO / "src"))
+from devflow_cli.editions import EDITIONS, entrypoint_for, hosts_for # noqa: E402
+
+_PACKAGE = {}
+
+
+def _env(**extra):
+ env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1", NO_UPDATE_NOTIFIER="1")
+ env.update(extra)
+ return env
+
+
+def run(argv, cwd=None, check=True, env=None):
+ result = subprocess.run(
+ [str(item) for item in argv], cwd=str(cwd) if cwd else None,
+ text=True, capture_output=True, env=env or _env(),
+ )
+ if check and result.returncode != 0:
+ raise AssertionError(
+ "command failed (rc={}): {}\n--- stdout\n{}\n--- stderr\n{}".format(
+ result.returncode, " ".join(map(str, argv)), result.stdout, result.stderr
+ )
+ )
+ return result
+
+
+def loopforge_command():
+ """Return argv prefix for the CLI under test (installed from the npm tarball)."""
+ if os.environ.get("LOOPFORGE_E2E_SOURCE") == "1":
+ return [sys.executable, "-m", "devflow_cli.cli"]
+ if "argv" not in _PACKAGE:
+ if not shutil.which("npm"):
+ raise unittest.SkipTest("npm is required; set LOOPFORGE_E2E_SOURCE=1 to skip packaging")
+ workdir = Path(tempfile.mkdtemp(prefix="loopforge-e2e-pkg-"))
+ _PACKAGE["dir"] = workdir
+ atexit.register(shutil.rmtree, str(workdir), True) # shared by all test modules
+ packed = json.loads(run(
+ ["npm", "pack", REPO, "--pack-destination", workdir, "--json"]
+ ).stdout)
+ tarball = workdir / packed[0]["filename"]
+ prefix = workdir / "prefix"
+ run(["npm", "install", "--global", "--prefix", prefix, tarball])
+ binary = prefix / "bin" / "loopforge"
+ if not binary.exists():
+ raise AssertionError("npm package did not install a loopforge binary")
+ _PACKAGE["argv"] = [binary]
+ return list(_PACKAGE["argv"])
+
+
+def loopforge(*args, cwd=None, check=True):
+ env = _env()
+ if os.environ.get("LOOPFORGE_E2E_SOURCE") == "1":
+ env["PYTHONPATH"] = str(REPO / "src")
+ return run(loopforge_command() + list(args), cwd=cwd, check=check, env=env)
+
+
+def new_project(test_case, with_sample=True):
+ """Create a throw-away git repo that looks like a user's project."""
+ root = Path(tempfile.mkdtemp(prefix="loopforge-e2e-project-"))
+ test_case.addCleanup(shutil.rmtree, str(root), True)
+ if with_sample:
+ for item in (FIXTURES / "sample_project").glob("*.py"):
+ shutil.copy(str(item), str(root / item.name))
+ (root / "USER_NOTES.md").write_text("user file that must survive\n", encoding="utf-8")
+ run(["git", "init", "-q", root])
+ return root
+
+
+def fake_agent(host, project, *extra, check_rc=None):
+ result = run(
+ [sys.executable, FAKE_AGENT, "--host", host, "--project-root", project] + list(extra),
+ check=False,
+ )
+ if check_rc is not None and result.returncode != check_rc:
+ raise AssertionError(
+ "fake agent rc={} (expected {})\n{}\n{}".format(
+ result.returncode, check_rc, result.stderr, transcript_text(project)
+ )
+ )
+ return result
+
+
+def transcript(project):
+ path = Path(project) / ".fake-agent" / "transcript.jsonl"
+ if not path.is_file():
+ return []
+ return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]
+
+
+def transcript_text(project):
+ return "\n".join(
+ json.dumps(item, ensure_ascii=False) for item in transcript(project)
+ if item["kind"] != "command"
+ )
+
+
+def workflow_state(project, slug="e2e-task"):
+ return json.loads(
+ (Path(project) / "artifacts" / slug / "workflow-state.json").read_text(encoding="utf-8")
+ )
+
+
+def files_under(root):
+ root = Path(root)
+ return sorted(
+ str(path.relative_to(root)) for path in root.rglob("*")
+ if (path.is_file() or path.is_symlink()) and ".git" not in path.relative_to(root).parts
+ )
+
+
+def loopforge_validate(project, slug="e2e-task", host_skills=None):
+ """Run the installed ``workflow_state.py validate`` for a finished task."""
+ from fake_agent import HOST_SKILL_DIRS
+ project = Path(project)
+ for skills in HOST_SKILL_DIRS.values():
+ script = project / skills / "devflow/scripts/workflow_state.py"
+ if script.is_file():
+ state = project / "artifacts" / slug / "workflow-state.json"
+ return run([sys.executable, script, "validate", "--state", state]).stdout
+ raise AssertionError("no installed workflow_state.py found")
diff --git a/tests/e2e/test_e2e_classic_runtime.py b/tests/e2e/test_e2e_classic_runtime.py
new file mode 100644
index 00000000..cd32faff
--- /dev/null
+++ b/tests/e2e/test_e2e_classic_runtime.py
@@ -0,0 +1,132 @@
+"""E2E: Classic CodeBuddy runtime hooks and the local observability dashboard.
+
+Classic stages are orchestrated by the host Agent from Markdown rules, so the
+deterministic parts we can drive end to end are the hooks the host runs:
+the auto-dispatch router between role agents, the observability hooks that
+record a session, and the dashboard built from what they wrote.
+"""
+
+import functools
+import json
+import os
+import shutil
+import subprocess
+import threading
+import unittest
+import urllib.request
+from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
+
+import harness as h
+
+ROUTE = [
+ ("requirement-owner", "TASK-01_completed", "architect"),
+ ("architect", "TASK-02_completed", "developer"),
+ ("developer", "TASK-03_completed", "code-reviewer"),
+ ("code-reviewer", "CODE-REVIEW_passed", "test-engineer"),
+ ("test-engineer", "TASK-04_completed", "knowledge-engineer"),
+ ("knowledge-engineer", "TASK-05_completed", "leader"),
+]
+
+
+class ClassicCodeBuddyE2E(unittest.TestCase):
+ def setUp(self):
+ self.project = h.new_project(self)
+ h.loopforge("install", "codebuddy", "--project-root", self.project)
+ self.env = dict(os.environ, CODEBUDDY_PROJECT_DIR=str(self.project), PYTHONDONTWRITEBYTECODE="1")
+ for key in list(self.env):
+ if key.startswith(("CLS_", "TENCENTCLOUD_", "TC_")):
+ del self.env[key] # the dashboard must work fully offline
+ self.state_path = self.project / "artifacts/e2e-classic/workflow-state.json"
+ self.state_path.parent.mkdir(parents=True)
+ shutil.copy(str(self.project / ".codebuddy/assets/workflow-state-template.json"), str(self.state_path))
+
+ def hook(self, command, payload):
+ result = subprocess.run(
+ command, input=json.dumps(payload), cwd=str(self.project), env=self.env,
+ capture_output=True, text=True, timeout=60,
+ )
+ self.assertEqual(0, result.returncode, result.stderr)
+ return result.stdout
+
+ def dispatch(self, sender, event, target, **extra):
+ payload = {"event": event, "dispatch_prompt": "continue as " + target,
+ "next_target": {"subagent_name": target},
+ "workflow_state_path": str(self.state_path)}
+ payload.update(extra)
+ output = self.hook(
+ ["python3", str(self.project / ".codebuddy/hooks/devflow-auto-dispatch.py")],
+ {"tool_name": "SendMessage", "tool_input": {
+ "type": "message", "from": sender, "recipient": "main",
+ "content": json.dumps(payload)}},
+ )
+ return json.loads(output)
+
+ def test_auto_dispatch_routes_each_completed_stage_to_the_next_role(self):
+ for sender, event, target in ROUTE:
+ with self.subTest(event=event):
+ decision = json.dumps(self.dispatch(sender, event, target))
+ self.assertIn('"recipient": "{}"'.format(target), decision)
+ decisions = json.loads(self.state_path.read_text(encoding="utf-8"))["decisions"]
+ self.assertEqual([event for _, event, _ in ROUTE], [d["event"] for d in decisions])
+
+ def test_auto_dispatch_falls_back_to_main_when_routing_is_unsafe(self):
+ cases = {
+ "mismatched target": ("architect", "TASK-02_completed", "leader"),
+ "unknown event": ("developer", "TASK-03_failed", "code-reviewer"),
+ "workflow end": ("leader", "workflow_completed", "main"),
+ }
+ for name, (sender, event, target) in cases.items():
+ with self.subTest(name):
+ self.assertNotIn("modifiedInput", json.dumps(self.dispatch(sender, event, target)))
+ (self.project / ".codebuddy/devflow.config.yaml").write_text(
+ "multi:\n auto_mode: false\n", encoding="utf-8")
+ with self.subTest("manual gate"):
+ gated = self.dispatch("architect", "TASK-02_completed", "developer")
+ self.assertNotIn("modifiedInput", json.dumps(gated))
+
+ def test_recorded_session_shows_up_in_the_local_dashboard(self):
+ skill = self.project / ".codebuddy/skills/agent-observability"
+ transcript = self.project / "session.jsonl"
+ shutil.copy(str(h.REPO / ".codebuddy/skills/agent-observability/tests/fixtures/transcript.jsonl"),
+ str(transcript))
+ base = {"session_id": "e2e-session", "cwd": str(self.project), "transcript_path": str(transcript)}
+ tool = dict(base, tool_name="Bash", tool_input={"command": "ls"}, tool_use_id="call-002")
+ run_hook = [str(skill / "scripts/run_hook.sh")]
+ self.hook(run_hook + ["session-start"], base)
+ self.hook(run_hook + ["user-prompt-submit"], dict(base, prompt="/start-devflow fix slugify"))
+ self.hook(run_hook + ["pre"], tool)
+ self.hook(run_hook + ["post"], dict(tool, tool_response={"stdout": "ok"}))
+ self.dispatch("requirement-owner", "TASK-01_completed", "architect")
+ self.hook(run_hook + ["stop"], base)
+
+ built = subprocess.run(
+ ["python3", str(skill / "scripts/build_dashboard_data.py"), "--project-root", str(self.project)],
+ cwd=str(self.project), env=self.env, capture_output=True, text=True, check=True,
+ ).stdout
+ self.assertIn("1 sessions, 1 devflow runs", built)
+
+ dashboard = skill / "scripts/dashboard"
+ handler = functools.partial(_QuietHandler, directory=str(dashboard))
+ server = ThreadingHTTPServer(("127.0.0.1", 0), handler)
+ threading.Thread(target=server.serve_forever, daemon=True).start()
+ self.addCleanup(server.server_close)
+ self.addCleanup(server.shutdown)
+ url = "http://127.0.0.1:{}/".format(server.server_address[1])
+ page = urllib.request.urlopen(url, timeout=10).read().decode("utf-8")
+ self.assertIn("./dashboard-data.json", page)
+ data = json.loads(urllib.request.urlopen(url + "dashboard-data.json", timeout=10).read())
+ sessions = data["sessions"]
+ self.assertEqual(["e2e-session"], [item["id"] for item in sessions])
+ self.assertGreaterEqual(sessions[0]["toolCalls"], 1)
+ self.assertGreater(sessions[0]["tokens"], 0, "token usage from the transcript is missing")
+ self.assertEqual(1, data["daily"][0]["sessionCount"])
+ self.assertEqual(["e2e-classic"], [run["slug"] for run in data["devflowRuns"]])
+ self.assertEqual(1, data["autoDispatchStats"]["auto_dispatch"])
+
+class _QuietHandler(SimpleHTTPRequestHandler):
+ def log_message(self, *args):
+ pass
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/e2e/test_e2e_install.py b/tests/e2e/test_e2e_install.py
new file mode 100644
index 00000000..8d6cf09f
--- /dev/null
+++ b/tests/e2e/test_e2e_install.py
@@ -0,0 +1,77 @@
+"""E2E: install the packaged CLI into a user project for every supported host."""
+
+import json
+import unittest
+from pathlib import Path
+
+import harness as h
+from fake_agent import HOST_SKILL_DIRS
+
+
+def entry_files(edition, host):
+ """Files the host Agent must find to expose the documented entrypoint."""
+ if edition == "classic":
+ if host == "codex":
+ return [".codex/skills/devflow-codex/SKILL.md", ".agents/skills/devflow-codex/SKILL.md"]
+ return [".{}/commands/start-devflow.md".format(host), ".{}/commands/resume-devflow.md".format(host)]
+ skills = HOST_SKILL_DIRS[host]
+ files = [skills + "/devflow/SKILL.md", skills + "/devflow-clarify-requirements/SKILL.md",
+ skills + "/devflow/scripts/workflow_state.py"]
+ if host in ("codebuddy", "cursor", "claude"):
+ files.append(".{}/agents/devflow-stage-executor.md".format(host))
+ if host == "codebuddy":
+ files.append(".codebuddy/commands/devflow.md")
+ return files
+
+
+def install_args(edition, host, command="install"):
+ if edition == "portable":
+ return ["skills", command, host]
+ return [command, host]
+
+
+class InstallMatrixE2E(unittest.TestCase):
+ def test_every_host_and_edition_installs_updates_and_uninstalls_cleanly(self):
+ for edition in h.EDITIONS:
+ for host in h.hosts_for(edition):
+ with self.subTest(edition=edition, host=host):
+ project = h.new_project(self, with_sample=False)
+ before = h.files_under(project)
+ out = h.loopforge(*install_args(edition, host), "--project-root", project).stdout
+ self.assertIn(h.entrypoint_for(edition, host), out)
+ for relative in entry_files(edition, host):
+ self.assertTrue((project / relative).is_file(), relative)
+
+ status_cmd = ["skills", "status", host] if edition == "portable" else ["status", host]
+ rows = json.loads(h.loopforge(*status_cmd, "--project-root", project, "--json").stdout)
+ self.assertEqual(1, len(rows), rows)
+ self.assertEqual((edition, 0, 0), (rows[0]["edition"], rows[0]["missing"], rows[0]["changed"]))
+ self.assertIn("healthy", h.loopforge("doctor", "--project-root", project).stdout)
+
+ update = h.loopforge(*install_args(edition, host, "update"), "--project-root", project).stdout
+ self.assertIn("written=0", update)
+
+ h.loopforge(*install_args(edition, host, "uninstall"), "--project-root", project)
+ self.assertEqual(before, h.files_under(project))
+
+ def test_classic_is_rejected_for_portable_only_host(self):
+ project = h.new_project(self, with_sample=False)
+ before = h.files_under(project)
+ result = h.loopforge("install", "pi", "--project-root", project, check=False)
+ self.assertNotEqual(0, result.returncode)
+ self.assertEqual(before, h.files_under(project))
+
+ def test_second_edition_for_same_host_requires_force(self):
+ project = h.new_project(self, with_sample=False)
+ h.loopforge("install", "claude", "--project-root", project)
+ before = h.files_under(project)
+ result = h.loopforge("skills", "install", "claude", "--project-root", project, check=False)
+ self.assertNotEqual(0, result.returncode)
+ self.assertEqual(before, h.files_under(project))
+ h.loopforge("skills", "install", "claude", "--project-root", project, "--force")
+ self.assertTrue((project / ".claude/skills/devflow/SKILL.md").is_file())
+ self.assertFalse((project / ".claude/commands/start-devflow.md").exists())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/e2e/test_e2e_portable_workflow.py b/tests/e2e/test_e2e_portable_workflow.py
new file mode 100644
index 00000000..22d9603f
--- /dev/null
+++ b/tests/e2e/test_e2e_portable_workflow.py
@@ -0,0 +1,154 @@
+"""E2E: drive the installed Portable workflow end to end with a fake host Agent.
+
+Each test installs the packaged CLI into a sample project, then runs
+``fake_agent.py``, which follows the Skill's ``resume``-driven protocol exactly
+as a host Agent would, using recorded stage outputs instead of an LLM.
+"""
+
+import json
+import subprocess
+import sys
+import unittest
+
+import harness as h
+
+AGENT_STAGES = ("DESIGN", "IMPLEMENT", "REVIEW", "TEST")
+STAGE_ORDER = ["REQUIREMENT", "DESIGN", "IMPLEMENT", "REVIEW", "TEST", "SUMMARY"]
+
+
+def installed_adapter(project, host):
+ from fake_agent import HOST_SKILL_DIRS
+ path = project / HOST_SKILL_DIRS[host] / "devflow/adapters" / host / "manifest.json"
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def events(project, kind):
+ return [item for item in h.transcript(project) if item["kind"] == kind]
+
+
+class PortableWorkflowE2E(unittest.TestCase):
+ def install(self, host="claude"):
+ project = h.new_project(self)
+ h.loopforge("skills", "install", host, "--project-root", project)
+ return project
+
+ def assert_delivered(self, project):
+ state = h.workflow_state(project)
+ self.assertEqual("completed", state["status"], h.transcript_text(project))
+ for stage in STAGE_ORDER:
+ self.assertEqual("completed", state["stages"][stage]["status"], stage)
+ self.assertEqual("skipped", state["stages"]["KNOWLEDGE"]["status"])
+ verify = subprocess.run(
+ [sys.executable, "-m", "unittest", "check_slugify"], cwd=str(project),
+ capture_output=True, text=True,
+ )
+ self.assertEqual(0, verify.returncode, verify.stderr)
+ self.assertIn("OK", h.loopforge_validate(project))
+ self.assertEqual("user file that must survive\n",
+ (project / "USER_NOTES.md").read_text(encoding="utf-8"))
+ return state
+
+ def test_medium_task_runs_every_stage_on_every_host(self):
+ for host in h.hosts_for("portable"):
+ with self.subTest(host=host):
+ project = self.install(host)
+ h.fake_agent(host, project, check_rc=0)
+ state = self.assert_delivered(project)
+
+ finished = [e["stage"] for e in events(project, "finish") if e["result"] == "completed"]
+ self.assertEqual(STAGE_ORDER, finished)
+ prompts = events(project, "prompt")
+ self.assertEqual(STAGE_ORDER, [p["stage"] for p in prompts])
+ self.assertTrue(all(p["mentions_role"] for p in prompts), prompts)
+
+ adapter = installed_adapter(project, host)
+ spawned = {e["stage"]: e for e in events(project, "spawned")}
+ self.assertEqual(set(AGENT_STAGES), set(spawned))
+ ids = set()
+ for stage in AGENT_STAGES:
+ recorded = state["stages"][stage]
+ self.assertEqual(adapter["stage_executor"], recorded["executor_type"])
+ self.assertEqual(adapter["dispatch_tool"], recorded["dispatch_tool"])
+ self.assertEqual(spawned[stage]["executor_id"], recorded["executor_id"])
+ ids.add(recorded["executor_id"])
+ # Independent review: every agent stage has its own executor.
+ self.assertEqual(len(AGENT_STAGES), len(ids))
+ for stage in ("REQUIREMENT", "SUMMARY"):
+ self.assertEqual("coordinator", state["stages"][stage]["executor_type"])
+
+ def test_resume_after_crash_inside_a_stage(self):
+ for host in ("claude", "codebuddy"):
+ with self.subTest(host=host):
+ project = self.install(host)
+ h.fake_agent(host, project, "--crash-in", "IMPLEMENT", check_rc=75)
+ state = h.workflow_state(project)
+ self.assertEqual("in_progress", state["stages"]["IMPLEMENT"]["status"])
+ self.assertEqual("in_progress", state["status"])
+
+ h.fake_agent(host, project, "--resume", check_rc=0)
+ self.assert_delivered(project)
+ sessions = []
+ for item in h.transcript(project):
+ if item["session"] not in sessions:
+ sessions.append(item["session"])
+ self.assertEqual(2, len(sessions))
+ second = [e for e in events(project, "action") if e["session"] == sessions[1]]
+ if host == "codebuddy":
+ # A new CodeBuddy session must drop the old Team and re-dispatch.
+ self.assertEqual("rotate-team", second[0]["action"])
+ self.assertEqual(1, len(events(project, "team-rotated")))
+ else:
+ self.assertEqual(("continue-executor", "IMPLEMENT"),
+ (second[0]["action"], second[0]["stage"]))
+ self.assertFalse(any(e["kind"] == "init" and e["session"] == sessions[1]
+ for e in h.transcript(project)))
+
+ def test_resume_after_interruption_between_actions(self):
+ project = self.install("codex")
+ h.fake_agent("codex", project, "--stop-after", "7", check_rc=75)
+ self.assertNotEqual("completed", h.workflow_state(project)["status"])
+ h.fake_agent("codex", project, "--resume", check_rc=0)
+ self.assert_delivered(project)
+ finished = [e["stage"] for e in events(project, "finish") if e["result"] == "completed"]
+ self.assertEqual(STAGE_ORDER, finished, "no stage may run twice after resume")
+
+ def test_requirement_gate_waits_for_explicit_user_confirmation(self):
+ project = self.install("cursor")
+ h.fake_agent("cursor", project, "--no-user-confirmation", check_rc=3)
+ state = h.workflow_state(project)
+ self.assertEqual("awaiting_approval", state["status"])
+ self.assertEqual("REQUIREMENT", state["awaiting_approval"])
+ self.assertIsNone(state["stages"]["DESIGN"]["prepared_at"])
+ self.assertFalse(events(project, "spawned"), "no DESIGN executor before approval")
+
+ h.fake_agent("cursor", project, "--resume", check_rc=0)
+ self.assert_delivered(project)
+
+ def test_rejected_review_sends_work_back_to_implement(self):
+ project = self.install("claude")
+ h.fake_agent("claude", project, "--review-reject", "1", check_rc=0)
+ self.assert_delivered(project)
+ outcomes = [(e["stage"], e["result"]) for e in events(project, "finish")
+ if e["stage"] in ("IMPLEMENT", "REVIEW")]
+ self.assertEqual([("IMPLEMENT", "completed"), ("REVIEW", "failed"),
+ ("IMPLEMENT", "completed"), ("REVIEW", "completed")], outcomes)
+
+ def test_unfilled_artifact_is_rejected_and_workflow_does_not_advance(self):
+ project = self.install("claude")
+ result = h.fake_agent("claude", project, "--leave-placeholders", "DESIGN", check_rc=2)
+ self.assertIn("artifact", result.stderr)
+ state = h.workflow_state(project)
+ self.assertEqual("in_progress", state["stages"]["DESIGN"]["status"])
+ self.assertEqual("pending", state["stages"]["IMPLEMENT"]["status"])
+
+ def test_repeated_stage_failures_block_the_workflow(self):
+ project = self.install("pi")
+ h.fake_agent("pi", project, "--fail-stage", "TEST:2", check_rc=4)
+ state = h.workflow_state(project)
+ self.assertEqual("blocked", state["status"])
+ self.assertEqual("failed", state["stages"]["TEST"]["status"])
+ self.assertEqual(2, state["stages"]["TEST"]["retry_count"])
+
+
+if __name__ == "__main__":
+ unittest.main()