Self-hosted workflows for coding-agent delivery. Orchestrate CodeBuddy, Codex and Claude Code with scripts and human approval gates. Includes Local workflows for coding assistants.
JavaScript agent-orchestrationagentic-workflowai-agentsai-codingclaude-codecode-reviewcodebuddycodexcoding-agentscursordeveloper-toolshuman-on-the-loopllmmcpmulti-agentsdlcself-hostedsopworkflow-automationworkflow-engine
test(e2e): add deterministic end-to-end suite and required CI check (#41)
Yanami-2Kf112b7d93312
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.mdindex 2a5f0846..0c34789a 100644--- a/.github/pull_request_template.md+++ b/.github/pull_request_template.md@@ -21,6 +21,7 @@ List commands that were actually run and mark anything blocked or not run.```textbash scripts/validate.shbash scripts/smoke-install.sh+bash scripts/e2e.shbash scripts/scan-secrets.sh```diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.ymlnew file mode 100644index 00000000..ab15b727--- /dev/null+++ b/.github/workflows/e2e.yml@@ -0,0 +1,48 @@+name: e2e++on:+ pull_request:+ push:+ branches: [main, master]++permissions:+ contents: read++concurrency:+ group: e2e-${{ github.event.pull_request.number || github.ref }}+ cancel-in-progress: true++jobs:+ e2e:+ name: e2e (python ${{ matrix.python-version }})+ runs-on: ubuntu-latest+ timeout-minutes: 15+ strategy:+ fail-fast: false+ matrix:+ python-version: ["3.8", "3.12"]+ steps:+ - uses: actions/[email protected]+ - uses: actions/[email protected]+ with:+ python-version: ${{ matrix.python-version }}+ - uses: actions/[email protected]+ with:+ node-version: "22"+ - run: python3 -m pip install pyyaml==6.0.3+ - name: Run E2E suite (packaged CLI, fake host Agent, no API keys)+ run: bash scripts/e2e.sh++ e2e-gate:+ # Single stable check name to mark as required in branch protection.+ name: e2e-required+ if: always()+ needs: [e2e]+ runs-on: ubuntu-latest+ steps:+ - name: Require every E2E job to pass+ run: |+ if [[ "${{ needs.e2e.result }}" != "success" ]]; then+ echo "E2E result: ${{ needs.e2e.result }}" >&2+ exit 1+ fidiff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex 07d77a6d..64ddd17b 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -80,6 +80,7 @@ jobs:bash scripts/smoke-install.shbash scripts/smoke-npm.shbash scripts/smoke-curl-install.sh+ bash scripts/e2e.shbash scripts/scan-secrets.sh- name: Log in to GHCRdiff --git a/AGENTS.md b/AGENTS.mdindex d61dc783..e05dedad 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -18,6 +18,7 @@ Run these commands before reporting a workflow change complete:python3 scripts/build-classic-hosts.py --checkbash scripts/validate.shbash scripts/smoke-install.sh+bash scripts/e2e.shbash scripts/scan-secrets.sh```diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex ce58b6cd..f7abeaf8 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -11,7 +11,7 @@ Thanks for contributing to LoopForge.3. Do not commit credentials, local permission configurations, or production data.4. First indicate whether a change belongs to Classic, Portable, or shared behaviour contracts. Portable only modifies `skills/`; Classic `.cursor/.claude` files are not edited directly — run `build-classic-hosts.py --write`.5. When modifying shared workflow contracts, describe the impact on both editions separately; do not copy runtime files between them.-6. Run `bash scripts/validate.sh` and `bash scripts/smoke-install.sh` before submitting.+6. Run `bash scripts/validate.sh`, `bash scripts/smoke-install.sh` and `bash scripts/e2e.sh` before submitting. The E2E suite drives the installed workflow with a fake host Agent and recorded stage outputs, so it needs no API key; see [tests/e2e/README.md](tests/e2e/README.md).## Pull request requirementsdiff --git a/CONTRIBUTING.zh-CN.md b/CONTRIBUTING.zh-CN.mdindex 79c2ab58..cf316cc5 100644--- a/CONTRIBUTING.zh-CN.md+++ b/CONTRIBUTING.zh-CN.md@@ -11,7 +11,7 @@3. 不提交任何凭据、本地权限配置或生产数据。4. 先标明改动属于 Classic、Portable 或共享行为合同。Portable 只修改 `skills/`;Classic 的 `.cursor/.claude` 不直接修改,运行 `build-classic-hosts.py --write`。5. 修改共享工作流契约时分别说明两个 edition 的影响;不要在二者之间复制运行文件。-6. 提交前运行 `bash scripts/validate.sh` 和 `bash scripts/smoke-install.sh`。+6. 提交前运行 `bash scripts/validate.sh`、`bash scripts/smoke-install.sh` 和 `bash scripts/e2e.sh`。E2E 用假宿主 Agent 和录制的阶段产物驱动已安装的工作流,不需要 API Key,说明见 [tests/e2e/README.md](tests/e2e/README.md)。## Pull Request 要求diff --git a/scripts/e2e.sh b/scripts/e2e.shnew file mode 100755index 00000000..01220d49--- /dev/null+++ b/scripts/e2e.sh@@ -0,0 +1,12 @@+#!/usr/bin/env bash+# End-to-end tests: install the packaged CLI into throw-away projects and drive+# every host through install, the full Portable workflow (with a fake host+# Agent), resume, failure paths, Classic hooks and the local dashboard.+# Needs Python 3.8+, Node.js 20+ and npm; no network access or API keys.+set -euo pipefail++ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+export PYTHONDONTWRITEBYTECODE=1++python3 -m unittest discover -s "$ROOT_DIR/tests/e2e" -t "$ROOT_DIR/tests/e2e" -p 'test_e2e_*.py' -v "$@"+echo "E2E tests passed."diff --git a/tests/e2e/README.md b/tests/e2e/README.mdnew file mode 100644index 00000000..a0656e29--- /dev/null+++ b/tests/e2e/README.md@@ -0,0 +1,20 @@+# E2E tests++`bash scripts/e2e.sh` runs the end-to-end suite. It needs Python 3.8+, Node.js 20+ and npm, and no network access, API key or real coding Agent.++What it does:++1. Builds the npm tarball once (`npm pack`), installs it into a temporary prefix, and uses that `loopforge` binary, which is the same path users take with `npx loopforge-cli`.+2. Installs every supported host and edition into throw-away git projects, then checks the entrypoint files, `status`, `doctor`, `update` and a clean `uninstall` (`test_e2e_install.py`).+3. Drives the installed Portable workflow end to end with [`fake_agent.py`](fake_agent.py), a deterministic stand-in for the host Agent. It follows the Skill's `resume`-driven protocol (`init`, `prepare --emit-prompt`, `assign`, `start`, `finish`, `approve`, `decide-knowledge`), but writes the stage outputs recorded in [`fixtures/scenario-medium.json`](fixtures/scenario-medium.json) instead of calling an LLM. The developer stage really changes [`fixtures/sample_project`](fixtures/sample_project/slugify.py), and the test stage really runs its tests. The suite covers every host, resume after a crash and after an interruption, the requirement confirmation gate, review rejection, unfilled artifacts and repeated failures (`test_e2e_portable_workflow.py`).+4. Runs the Classic CodeBuddy hooks from an installed project: auto-dispatch routing between roles, the fallbacks to the main Agent, and a recorded session flowing through the observability hooks into the local dashboard, which is served over HTTP (`test_e2e_classic_runtime.py`).++Useful options:++```bash+LOOPFORGE_E2E_SOURCE=1 bash scripts/e2e.sh # run the CLI from src/ and skip npm pack+bash scripts/e2e.sh -k resume # run only tests matching a pattern+python3 tests/e2e/fake_agent.py --help # drive a project by hand+```++When a test fails, the assertion message includes the fake Agent transcript (`.fake-agent/transcript.jsonl` in the project) that shows each action and the command that was rejected.diff --git a/tests/e2e/fake_agent.py b/tests/e2e/fake_agent.pynew file mode 100644index 00000000..29f9d905--- /dev/null+++ b/tests/e2e/fake_agent.py@@ -0,0 +1,280 @@+#!/usr/bin/env python3+"""Deterministic stand-in for a host coding Agent (Claude Code, Codex, ...).++A real host loads the installed ``devflow`` Skill and follows its contract:+``init`` once, then repeatedly ask ``workflow_state.py resume`` for the next+action and carry it out (prepare a stage prompt, spawn and register a stage+executor, start, let the executor write artifacts, finish, ask the user to+approve the requirement, ...).++This script follows exactly that protocol against the *installed* copy of the+Skill in a project, but replaces the LLM with recorded stage outputs from a+scenario fixture. No network access, API key or real Agent CLI is needed, so+the run is deterministic in CI.++Exit codes: 0 completed, 3 waiting for the user, 4 blocked, 75 interrupted,+2 contract violation (a command the real Agent would run was rejected).+"""++import argparse+import json+import os+import re+import shutil+import subprocess+import sys+import uuid+from pathlib import Path++HOST_SKILL_DIRS = {+ "codebuddy": ".codebuddy/skills",+ "codex": ".agents/skills",+ "cursor": ".cursor/skills",+ "claude": ".claude/skills",+ "pi": ".pi/skills",+}+FIXTURES = Path(__file__).resolve().parent / "fixtures"+EXIT_COMPLETED, EXIT_CONTRACT, EXIT_WAITING, EXIT_BLOCKED, EXIT_INTERRUPTED = 0, 2, 3, 4, 75+++class ContractError(Exception):+ pass+++class FakeAgent:+ def __init__(self, args):+ self.args = args+ self.host = args.host+ self.root = args.project_root.resolve()+ self.scenario = json.loads(args.scenario.read_text(encoding="utf-8"))+ self.session = uuid.uuid4().hex[:8]+ self.skill_root = self.root / HOST_SKILL_DIRS[self.host] / "devflow"+ self.workflow = self.skill_root / "scripts" / "workflow_state.py"+ self.transcript = self.root / ".fake-agent" / "transcript.jsonl"+ self.transcript.parent.mkdir(parents=True, exist_ok=True)+ self.actions = 0+ self.review_rejections_left = args.review_reject+ self.failures_left = dict(args.fail_stage)+ self.fresh_team = bool(args.resume and args.host == "codebuddy")+ self.rounds = {}++ # -- helpers ---------------------------------------------------------+ def log(self, kind, **fields):+ record = {"session": self.session, "kind": kind}+ record.update(fields)+ with self.transcript.open("a", encoding="utf-8") as handle:+ handle.write(json.dumps(record, ensure_ascii=False) + "\n")++ def run(self, *argv, check=True):+ command = [sys.executable, str(self.workflow)] + [str(item) for item in argv]+ result = subprocess.run(+ command, cwd=str(self.root), text=True, capture_output=True,+ env=dict(os.environ, PYTHONDONTWRITEBYTECODE="1"),+ )+ self.log("command", argv=[str(item) for item in argv[:3]], rc=result.returncode)+ if check and result.returncode != 0:+ raise ContractError(+ "workflow_state.py {} rejected (rc={}): {}".format(+ argv[0], result.returncode, (result.stderr or result.stdout).strip()+ )+ )+ return result++ def state_path(self):+ return self.root / "artifacts" / self.args.slug / "workflow-state.json"++ def state(self):+ return json.loads(self.state_path().read_text(encoding="utf-8"))++ # -- the host side of the Skill contract ----------------------------+ def load_skill(self):+ skill = self.skill_root / "SKILL.md"+ if not skill.is_file():+ raise ContractError("host cannot find installed Skill: {}".format(skill))+ text = skill.read_text(encoding="utf-8")+ for needle in ("workflow_state.py", "prepare", "approve"):+ if needle not in text:+ raise ContractError("SKILL.md no longer documents `{}`".format(needle))+ # The Agent follows the links in SKILL.md; every one must ship with the install.+ for target in re.findall(r"\]\(([^)#]+)\)", text):+ if "://" not in target and not (self.skill_root / target).exists():+ raise ContractError("SKILL.md links to missing file: " + target)+ contract = (self.skill_root / "references" / "workflow-contract.md").read_text(encoding="utf-8")+ if "resume --project-root . --slug" not in contract:+ raise ContractError("workflow-contract.md no longer documents `resume --slug`")+ self.log("skill-loaded", path=str(skill.relative_to(self.root)))++ def init(self):+ output = self.run(+ "init", "--project-root", self.root, "--slug", self.args.slug,+ "--size", "medium", "--mode", "auto", "--execution-mode", "isolated",+ "--host-adapter", self.host, "--coordinator-id", "fake-main",+ ).stdout.strip()+ self.log("init", state=output)++ def next_action(self):+ argv = ["resume", "--project-root", self.root, "--slug", self.args.slug]+ if self.fresh_team:+ # Per workflow-contract.md a new CodeBuddy main session resumes with a fresh Team.+ argv.append("--fresh-team")+ self.fresh_team = False+ return json.loads(self.run(*argv).stdout)++ def loop(self):+ while True:+ action = self.next_action()+ kind = action.get("action")+ self.log("action", action=kind, stage=action.get("stage"))+ if kind == "completed":+ return EXIT_COMPLETED+ if kind == "blocked":+ return EXIT_BLOCKED+ if kind == "approve":+ if action.get("requires_user_confirmation") and not self.args.user_confirms:+ self.log("waiting-for-user", stage=action["stage"])+ return EXIT_WAITING+ extra = ["--user-confirmed"] if action.get("requires_user_confirmation") else []+ self.run("approve", "--state", self.state_path(), "--stage", action["stage"], *extra)+ elif kind == "rotate-team":+ rotated = json.loads(self.run("rotate-team", "--state", self.state_path()).stdout)+ self.log("team-rotated", old=rotated["old_team_name"], new=rotated["team_name"])+ elif kind == "assess-knowledge":+ decision = self.scenario["knowledge_decision"]+ self.run(+ "decide-knowledge", "--state", self.state_path(),+ "--decision", decision["decision"], "--reason", decision["reason"],+ )+ elif kind == "prepare":+ prompt = self.run(+ "prepare", "--state", self.state_path(), "--stage", action["stage"],+ "--emit-prompt", "--request", self.scenario["request"],+ ).stdout+ if not prompt.strip():+ raise ContractError("empty stage prompt for " + action["stage"])+ self.log("prompt", stage=action["stage"], role=action.get("executor_role"),+ chars=len(prompt), mentions_role=action.get("executor_role", "") in prompt)+ elif kind == "spawn-and-assign":+ executor_id = "fake-{}-{}".format(action["stage"].lower(), uuid.uuid4().hex[:6])+ argv = [+ "assign", "--state", self.state_path(), "--stage", action["stage"],+ "--role", action["executor_role"], "--executor-type", action["executor_type"],+ "--executor-id", executor_id, "--dispatch-tool", action["dispatch_tool"],+ ]+ if action.get("team_name"):+ argv += ["--team-name", action["team_name"]]+ self.run(*argv)+ self.log("spawned", stage=action["stage"], executor_id=executor_id,+ executor_type=action["executor_type"],+ dispatch_tool=action["dispatch_tool"])+ elif kind in ("start-assigned", "start-coordinator"):+ self.run("start", "--state", self.state_path(), "--stage", action["stage"])+ if self.args.crash_in == action["stage"]:+ self.log("crash", stage=action["stage"])+ return EXIT_INTERRUPTED+ self.execute_stage(action["stage"])+ elif kind == "continue-executor":+ self.execute_stage(action["stage"])+ else:+ raise ContractError("unexpected resume action: {}".format(action))+ self.actions += 1+ if self.args.stop_after and self.actions >= self.args.stop_after:+ self.log("interrupted", after=self.actions)+ return EXIT_INTERRUPTED++ # -- the fake stage executor ----------------------------------------+ def execute_stage(self, stage):+ result = "completed"+ evidence = self.scenario["stages"][stage]+ overrides = dict(self.scenario["overrides"])+ if stage == "REVIEW" and self.review_rejections_left > 0:+ self.review_rejections_left -= 1+ result = "failed"+ evidence = self.scenario["stages"]["REVIEW_REJECTED"]+ for key in list(overrides):+ if "通过" in key and "不通过" in key:+ overrides[key] = evidence+ if self.failures_left.get(stage, 0) > 0:+ self.failures_left[stage] -= 1+ result = "failed"+ for relative, source in self.scenario.get("code_changes", {}).get(stage, {}).items():+ shutil.copyfile(str(FIXTURES / source), str(self.root / relative))+ if stage == "TEST":+ verify = subprocess.run(+ [sys.executable] + self.scenario["verify_command"], cwd=str(self.root),+ text=True, capture_output=True,+ )+ self.log("verify", rc=verify.returncode, tail=verify.stderr.strip().splitlines()[-1:])+ if verify.returncode != 0:+ result = "failed"+ evidence = "测试失败:" + verify.stderr.strip()[-200:]+ self.rounds[stage] = self.rounds.get(stage, 0) + 1+ if stage not in self.args.leave_placeholders:+ note = "(session {},第 {} 轮)".format(self.session, self.rounds[stage])+ self.fill_artifacts(stage, overrides, evidence + note)+ finished = self.run(+ "finish", "--state", self.state_path(), "--stage", stage, "--result", result,+ check=False,+ )+ self.log("finish", stage=stage, result=result, rc=finished.returncode,+ error=finished.stderr.strip()[-300:] if finished.returncode else None)+ if finished.returncode != 0:+ raise ContractError("finish {} rejected: {}".format(stage, finished.stderr.strip()))++ def fill_artifacts(self, stage, overrides, evidence):+ """Write every required artifact afresh from its template, like a real executor."""+ artifact_root = self.state_path().parent+ templates = self.skill_root / "templates"+ manifest = json.loads((templates / "manifest.json").read_text(encoding="utf-8"))+ sources = {item["path"]: item["template"] for item in manifest["artifacts"]}+ for relative in self.required_artifacts(stage):+ text = (templates / sources[relative]).read_text(encoding="utf-8")+ for source, target in overrides.items():+ text = text.replace(source, target)+ text = re.sub(r"\{\{[^{}]+\}\}", evidence, text)+ path = artifact_root / relative+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(text, encoding="utf-8")++ def required_artifacts(self, stage):+ sys.path.insert(0, str(self.skill_root / "scripts"))+ try:+ from template_registry import required_artifacts # type: ignore+ finally:+ sys.path.pop(0)+ return required_artifacts(stage, bool(self.state().get("requires_design_artifacts")))++ def main(self):+ try:+ self.load_skill()+ if not self.args.resume:+ self.init()+ code = self.loop()+ except ContractError as error:+ self.log("contract-error", message=str(error))+ print("fake-agent: " + str(error), file=sys.stderr)+ return EXIT_CONTRACT+ self.log("exit", code=code)+ return code+++def parse_args(argv=None):+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])+ parser.add_argument("--host", choices=sorted(HOST_SKILL_DIRS), required=True)+ parser.add_argument("--project-root", type=Path, required=True)+ parser.add_argument("--slug", default="e2e-task")+ parser.add_argument("--scenario", type=Path, default=FIXTURES / "scenario-medium.json")+ parser.add_argument("--resume", action="store_true", help="new session; do not init")+ parser.add_argument("--no-user-confirmation", dest="user_confirms", action="store_false")+ parser.add_argument("--stop-after", type=int, default=0, help="interrupt after N actions")+ parser.add_argument("--crash-in", help="die right after starting this stage")+ parser.add_argument("--review-reject", type=int, default=0, help="reject REVIEW N times")+ parser.add_argument("--leave-placeholders", action="append", default=[],+ help="executor of this stage leaves the template unfilled")+ parser.add_argument("--fail-stage", action="append", default=[],+ type=lambda item: (item.split(":")[0], int(item.split(":")[1])),+ help="STAGE:N report failure N times")+ return parser.parse_args(argv)+++if __name__ == "__main__":+ sys.exit(FakeAgent(parse_args()).main())diff --git a/tests/e2e/fixtures/patches/slugify.py b/tests/e2e/fixtures/patches/slugify.pynew file mode 100644index 00000000..ae204032--- /dev/null+++ b/tests/e2e/fixtures/patches/slugify.py@@ -0,0 +1,8 @@+"""Tiny fixture module used by the LoopForge E2E suite (fixed version)."""++import re+++def slugify(value):+ words = re.findall(r"[a-z0-9]+", value.lower())+ return "-".join(words)diff --git a/tests/e2e/fixtures/sample_project/check_slugify.py b/tests/e2e/fixtures/sample_project/check_slugify.pynew file mode 100644index 00000000..b3dd6f39--- /dev/null+++ b/tests/e2e/fixtures/sample_project/check_slugify.py@@ -0,0 +1,15 @@+import unittest++from slugify import slugify+++class SlugifyAcceptance(unittest.TestCase):+ def test_spaces_become_dashes(self):+ self.assertEqual("order-status-filter", slugify(" Order Status Filter "))++ def test_symbols_are_removed(self):+ self.assertEqual("v2-release", slugify("v2 release!"))+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/e2e/fixtures/sample_project/slugify.py b/tests/e2e/fixtures/sample_project/slugify.pynew file mode 100644index 00000000..6a0ce52e--- /dev/null+++ b/tests/e2e/fixtures/sample_project/slugify.py@@ -0,0 +1,9 @@+"""Tiny fixture module used by the LoopForge E2E suite.++The fake developer agent replaces this file with ``patches/slugify.py``; the+fake test engineer then runs ``check_slugify.py`` to prove the change works.+"""+++def slugify(value):+ return value.lower()diff --git a/tests/e2e/fixtures/scenario-medium.json b/tests/e2e/fixtures/scenario-medium.jsonnew file mode 100644index 00000000..2364e33c--- /dev/null+++ b/tests/e2e/fixtures/scenario-medium.json@@ -0,0 +1,32 @@+{+ "description": "Recorded stage outputs for a medium task: fix slugify() in the sample project. Each stage executor fills the real templates produced by `workflow_state.py prepare` with this evidence.",+ "request": "修复 slugify:空格转连字符、去掉符号,并补充验收测试",+ "knowledge_decision": {"decision": "skip", "reason": "只有常规变更和测试结果,没有新增可复用约束"},+ "overrides": {+ "{{说明当前实现、关键模块、仓库相对路径、约束和非目标}}": "当前实现位于 `slugify.py`,仅修改该文件;验收用例在 `check_slugify.py`。",+ "{{动作导向的任务名}}": "修复 slugify 规范化逻辑",+ "{{无,或任务 ID}}": "无",+ "{{仓库相对路径}}": "slugify.py",+ "{{工作目录}}": ".",+ "{{项目原生验证命令}}": "python3 -m unittest check_slugify",+ "{{明确写“通过”或“不通过”;只有无阻断问题时才能写“通过”}}": "通过,无阻断问题。",+ "{{无接口变动时写“无接口变动”;有新增、修改或删除时生成文档并写“已生成:`03-code/api-docs.md`”}}": "无接口变动",+ "{{无接口变动时写“无接口变动”;有新增、修改或删除时生成文档并写“已生成:`01-solo/api-docs.md`”}}": "无接口变动",+ "{{列出仓库实际已有的测试层级及证据路径/配置/命令;若已有 E2E,明确其覆盖边界;若没有,不要为流程强行引入新框架}}": "现有 unittest 验收用例 `check_slugify.py`;没有浏览器 E2E 框架。",+ "{{逐项记录“验收标准或风险 | 项目已有的合适测试层级 | 用例或命令 | 结果”;项目已有 E2E 且本次行为属于其覆盖边界时,必须包含相应 E2E 用例}}": "空格转连字符、去掉符号 | unit | `python3 -m unittest check_slugify` | 通过"+ },+ "stages": {+ "REQUIREMENT": "目标:`slugify()` 把空格转成连字符并去掉符号;验收:`python3 -m unittest check_slugify` 通过。",+ "DESIGN": "用正则提取 `[a-z0-9]+` 片段后以连字符连接,只改 `slugify.py`,回滚即恢复原文件。",+ "IMPLEMENT": "已修改 `slugify.py`;运行 `python3 -m unittest check_slugify`,结果 passed。",+ "REVIEW": "审查 `slugify.py` 的 diff,未发现阻断问题。",+ "REVIEW_REJECTED": "不通过:`slugify.py` 未处理符号,阻断级别,建议改为按字母数字切分。",+ "TEST": "运行 `python3 -m unittest check_slugify`,2 个用例全部通过。",+ "KNOWLEDGE": "无新增经验。",+ "SUMMARY": "交付完成;`python3 -m unittest check_slugify` 已通过,无剩余风险。"+ },+ "code_changes": {+ "IMPLEMENT": {"slugify.py": "patches/slugify.py"}+ },+ "verify_command": ["-m", "unittest", "check_slugify"]+}diff --git a/tests/e2e/harness.py b/tests/e2e/harness.pynew file mode 100644index 00000000..592482cc--- /dev/null+++ b/tests/e2e/harness.py@@ -0,0 +1,143 @@+"""Shared helpers for the LoopForge E2E suite.++By default the suite tests what users actually install: it builds the npm+tarball once (``npm pack``), installs it into a throw-away prefix and drives the+resulting ``loopforge`` binary. Set ``LOOPFORGE_E2E_SOURCE=1`` to run the CLI+straight from ``src/`` for a faster local loop.+"""++import atexit+import json+import os+import shutil+import subprocess+import sys+import tempfile+import unittest+from pathlib import Path++REPO = Path(__file__).resolve().parents[2]+E2E = Path(__file__).resolve().parent+FIXTURES = E2E / "fixtures"+FAKE_AGENT = E2E / "fake_agent.py"++sys.path.insert(0, str(REPO / "src"))+from devflow_cli.editions import EDITIONS, entrypoint_for, hosts_for # noqa: E402++_PACKAGE = {}+++def _env(**extra):+ env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1", NO_UPDATE_NOTIFIER="1")+ env.update(extra)+ return env+++def run(argv, cwd=None, check=True, env=None):+ result = subprocess.run(+ [str(item) for item in argv], cwd=str(cwd) if cwd else None,+ text=True, capture_output=True, env=env or _env(),+ )+ if check and result.returncode != 0:+ raise AssertionError(+ "command failed (rc={}): {}\n--- stdout\n{}\n--- stderr\n{}".format(+ result.returncode, " ".join(map(str, argv)), result.stdout, result.stderr+ )+ )+ return result+++def loopforge_command():+ """Return argv prefix for the CLI under test (installed from the npm tarball)."""+ if os.environ.get("LOOPFORGE_E2E_SOURCE") == "1":+ return [sys.executable, "-m", "devflow_cli.cli"]+ if "argv" not in _PACKAGE:+ if not shutil.which("npm"):+ raise unittest.SkipTest("npm is required; set LOOPFORGE_E2E_SOURCE=1 to skip packaging")+ workdir = Path(tempfile.mkdtemp(prefix="loopforge-e2e-pkg-"))+ _PACKAGE["dir"] = workdir+ atexit.register(shutil.rmtree, str(workdir), True) # shared by all test modules+ packed = json.loads(run(+ ["npm", "pack", REPO, "--pack-destination", workdir, "--json"]+ ).stdout)+ tarball = workdir / packed[0]["filename"]+ prefix = workdir / "prefix"+ run(["npm", "install", "--global", "--prefix", prefix, tarball])+ binary = prefix / "bin" / "loopforge"+ if not binary.exists():+ raise AssertionError("npm package did not install a loopforge binary")+ _PACKAGE["argv"] = [binary]+ return list(_PACKAGE["argv"])+++def loopforge(*args, cwd=None, check=True):+ env = _env()+ if os.environ.get("LOOPFORGE_E2E_SOURCE") == "1":+ env["PYTHONPATH"] = str(REPO / "src")+ return run(loopforge_command() + list(args), cwd=cwd, check=check, env=env)+++def new_project(test_case, with_sample=True):+ """Create a throw-away git repo that looks like a user's project."""+ root = Path(tempfile.mkdtemp(prefix="loopforge-e2e-project-"))+ test_case.addCleanup(shutil.rmtree, str(root), True)+ if with_sample:+ for item in (FIXTURES / "sample_project").glob("*.py"):+ shutil.copy(str(item), str(root / item.name))+ (root / "USER_NOTES.md").write_text("user file that must survive\n", encoding="utf-8")+ run(["git", "init", "-q", root])+ return root+++def fake_agent(host, project, *extra, check_rc=None):+ result = run(+ [sys.executable, FAKE_AGENT, "--host", host, "--project-root", project] + list(extra),+ check=False,+ )+ if check_rc is not None and result.returncode != check_rc:+ raise AssertionError(+ "fake agent rc={} (expected {})\n{}\n{}".format(+ result.returncode, check_rc, result.stderr, transcript_text(project)+ )+ )+ return result+++def transcript(project):+ path = Path(project) / ".fake-agent" / "transcript.jsonl"+ if not path.is_file():+ return []+ return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]+++def transcript_text(project):+ return "\n".join(+ json.dumps(item, ensure_ascii=False) for item in transcript(project)+ if item["kind"] != "command"+ )+++def workflow_state(project, slug="e2e-task"):+ return json.loads(+ (Path(project) / "artifacts" / slug / "workflow-state.json").read_text(encoding="utf-8")+ )+++def files_under(root):+ root = Path(root)+ return sorted(+ str(path.relative_to(root)) for path in root.rglob("*")+ if (path.is_file() or path.is_symlink()) and ".git" not in path.relative_to(root).parts+ )+++def loopforge_validate(project, slug="e2e-task", host_skills=None):+ """Run the installed ``workflow_state.py validate`` for a finished task."""+ from fake_agent import HOST_SKILL_DIRS+ project = Path(project)+ for skills in HOST_SKILL_DIRS.values():+ script = project / skills / "devflow/scripts/workflow_state.py"+ if script.is_file():+ state = project / "artifacts" / slug / "workflow-state.json"+ return run([sys.executable, script, "validate", "--state", state]).stdout+ raise AssertionError("no installed workflow_state.py found")diff --git a/tests/e2e/test_e2e_classic_runtime.py b/tests/e2e/test_e2e_classic_runtime.pynew file mode 100644index 00000000..cd32faff--- /dev/null+++ b/tests/e2e/test_e2e_classic_runtime.py@@ -0,0 +1,132 @@+"""E2E: Classic CodeBuddy runtime hooks and the local observability dashboard.++Classic stages are orchestrated by the host Agent from Markdown rules, so the+deterministic parts we can drive end to end are the hooks the host runs:+the auto-dispatch router between role agents, the observability hooks that+record a session, and the dashboard built from what they wrote.+"""++import functools+import json+import os+import shutil+import subprocess+import threading+import unittest+import urllib.request+from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer++import harness as h++ROUTE = [+ ("requirement-owner", "TASK-01_completed", "architect"),+ ("architect", "TASK-02_completed", "developer"),+ ("developer", "TASK-03_completed", "code-reviewer"),+ ("code-reviewer", "CODE-REVIEW_passed", "test-engineer"),+ ("test-engineer", "TASK-04_completed", "knowledge-engineer"),+ ("knowledge-engineer", "TASK-05_completed", "leader"),+]+++class ClassicCodeBuddyE2E(unittest.TestCase):+ def setUp(self):+ self.project = h.new_project(self)+ h.loopforge("install", "codebuddy", "--project-root", self.project)+ self.env = dict(os.environ, CODEBUDDY_PROJECT_DIR=str(self.project), PYTHONDONTWRITEBYTECODE="1")+ for key in list(self.env):+ if key.startswith(("CLS_", "TENCENTCLOUD_", "TC_")):+ del self.env[key] # the dashboard must work fully offline+ self.state_path = self.project / "artifacts/e2e-classic/workflow-state.json"+ self.state_path.parent.mkdir(parents=True)+ shutil.copy(str(self.project / ".codebuddy/assets/workflow-state-template.json"), str(self.state_path))++ def hook(self, command, payload):+ result = subprocess.run(+ command, input=json.dumps(payload), cwd=str(self.project), env=self.env,+ capture_output=True, text=True, timeout=60,+ )+ self.assertEqual(0, result.returncode, result.stderr)+ return result.stdout++ def dispatch(self, sender, event, target, **extra):+ payload = {"event": event, "dispatch_prompt": "continue as " + target,+ "next_target": {"subagent_name": target},+ "workflow_state_path": str(self.state_path)}+ payload.update(extra)+ output = self.hook(+ ["python3", str(self.project / ".codebuddy/hooks/devflow-auto-dispatch.py")],+ {"tool_name": "SendMessage", "tool_input": {+ "type": "message", "from": sender, "recipient": "main",+ "content": json.dumps(payload)}},+ )+ return json.loads(output)++ def test_auto_dispatch_routes_each_completed_stage_to_the_next_role(self):+ for sender, event, target in ROUTE:+ with self.subTest(event=event):+ decision = json.dumps(self.dispatch(sender, event, target))+ self.assertIn('"recipient": "{}"'.format(target), decision)+ decisions = json.loads(self.state_path.read_text(encoding="utf-8"))["decisions"]+ self.assertEqual([event for _, event, _ in ROUTE], [d["event"] for d in decisions])++ def test_auto_dispatch_falls_back_to_main_when_routing_is_unsafe(self):+ cases = {+ "mismatched target": ("architect", "TASK-02_completed", "leader"),+ "unknown event": ("developer", "TASK-03_failed", "code-reviewer"),+ "workflow end": ("leader", "workflow_completed", "main"),+ }+ for name, (sender, event, target) in cases.items():+ with self.subTest(name):+ self.assertNotIn("modifiedInput", json.dumps(self.dispatch(sender, event, target)))+ (self.project / ".codebuddy/devflow.config.yaml").write_text(+ "multi:\n auto_mode: false\n", encoding="utf-8")+ with self.subTest("manual gate"):+ gated = self.dispatch("architect", "TASK-02_completed", "developer")+ self.assertNotIn("modifiedInput", json.dumps(gated))++ def test_recorded_session_shows_up_in_the_local_dashboard(self):+ skill = self.project / ".codebuddy/skills/agent-observability"+ transcript = self.project / "session.jsonl"+ shutil.copy(str(h.REPO / ".codebuddy/skills/agent-observability/tests/fixtures/transcript.jsonl"),+ str(transcript))+ base = {"session_id": "e2e-session", "cwd": str(self.project), "transcript_path": str(transcript)}+ tool = dict(base, tool_name="Bash", tool_input={"command": "ls"}, tool_use_id="call-002")+ run_hook = [str(skill / "scripts/run_hook.sh")]+ self.hook(run_hook + ["session-start"], base)+ self.hook(run_hook + ["user-prompt-submit"], dict(base, prompt="/start-devflow fix slugify"))+ self.hook(run_hook + ["pre"], tool)+ self.hook(run_hook + ["post"], dict(tool, tool_response={"stdout": "ok"}))+ self.dispatch("requirement-owner", "TASK-01_completed", "architect")+ self.hook(run_hook + ["stop"], base)++ built = subprocess.run(+ ["python3", str(skill / "scripts/build_dashboard_data.py"), "--project-root", str(self.project)],+ cwd=str(self.project), env=self.env, capture_output=True, text=True, check=True,+ ).stdout+ self.assertIn("1 sessions, 1 devflow runs", built)++ dashboard = skill / "scripts/dashboard"+ handler = functools.partial(_QuietHandler, directory=str(dashboard))+ server = ThreadingHTTPServer(("127.0.0.1", 0), handler)+ threading.Thread(target=server.serve_forever, daemon=True).start()+ self.addCleanup(server.server_close)+ self.addCleanup(server.shutdown)+ url = "http://127.0.0.1:{}/".format(server.server_address[1])+ page = urllib.request.urlopen(url, timeout=10).read().decode("utf-8")+ self.assertIn("./dashboard-data.json", page)+ data = json.loads(urllib.request.urlopen(url + "dashboard-data.json", timeout=10).read())+ sessions = data["sessions"]+ self.assertEqual(["e2e-session"], [item["id"] for item in sessions])+ self.assertGreaterEqual(sessions[0]["toolCalls"], 1)+ self.assertGreater(sessions[0]["tokens"], 0, "token usage from the transcript is missing")+ self.assertEqual(1, data["daily"][0]["sessionCount"])+ self.assertEqual(["e2e-classic"], [run["slug"] for run in data["devflowRuns"]])+ self.assertEqual(1, data["autoDispatchStats"]["auto_dispatch"])++class _QuietHandler(SimpleHTTPRequestHandler):+ def log_message(self, *args):+ pass+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/e2e/test_e2e_install.py b/tests/e2e/test_e2e_install.pynew file mode 100644index 00000000..8d6cf09f--- /dev/null+++ b/tests/e2e/test_e2e_install.py@@ -0,0 +1,77 @@+"""E2E: install the packaged CLI into a user project for every supported host."""++import json+import unittest+from pathlib import Path++import harness as h+from fake_agent import HOST_SKILL_DIRS+++def entry_files(edition, host):+ """Files the host Agent must find to expose the documented entrypoint."""+ if edition == "classic":+ if host == "codex":+ return [".codex/skills/devflow-codex/SKILL.md", ".agents/skills/devflow-codex/SKILL.md"]+ return [".{}/commands/start-devflow.md".format(host), ".{}/commands/resume-devflow.md".format(host)]+ skills = HOST_SKILL_DIRS[host]+ files = [skills + "/devflow/SKILL.md", skills + "/devflow-clarify-requirements/SKILL.md",+ skills + "/devflow/scripts/workflow_state.py"]+ if host in ("codebuddy", "cursor", "claude"):+ files.append(".{}/agents/devflow-stage-executor.md".format(host))+ if host == "codebuddy":+ files.append(".codebuddy/commands/devflow.md")+ return files+++def install_args(edition, host, command="install"):+ if edition == "portable":+ return ["skills", command, host]+ return [command, host]+++class InstallMatrixE2E(unittest.TestCase):+ def test_every_host_and_edition_installs_updates_and_uninstalls_cleanly(self):+ for edition in h.EDITIONS:+ for host in h.hosts_for(edition):+ with self.subTest(edition=edition, host=host):+ project = h.new_project(self, with_sample=False)+ before = h.files_under(project)+ out = h.loopforge(*install_args(edition, host), "--project-root", project).stdout+ self.assertIn(h.entrypoint_for(edition, host), out)+ for relative in entry_files(edition, host):+ self.assertTrue((project / relative).is_file(), relative)++ status_cmd = ["skills", "status", host] if edition == "portable" else ["status", host]+ rows = json.loads(h.loopforge(*status_cmd, "--project-root", project, "--json").stdout)+ self.assertEqual(1, len(rows), rows)+ self.assertEqual((edition, 0, 0), (rows[0]["edition"], rows[0]["missing"], rows[0]["changed"]))+ self.assertIn("healthy", h.loopforge("doctor", "--project-root", project).stdout)++ update = h.loopforge(*install_args(edition, host, "update"), "--project-root", project).stdout+ self.assertIn("written=0", update)++ h.loopforge(*install_args(edition, host, "uninstall"), "--project-root", project)+ self.assertEqual(before, h.files_under(project))++ def test_classic_is_rejected_for_portable_only_host(self):+ project = h.new_project(self, with_sample=False)+ before = h.files_under(project)+ result = h.loopforge("install", "pi", "--project-root", project, check=False)+ self.assertNotEqual(0, result.returncode)+ self.assertEqual(before, h.files_under(project))++ def test_second_edition_for_same_host_requires_force(self):+ project = h.new_project(self, with_sample=False)+ h.loopforge("install", "claude", "--project-root", project)+ before = h.files_under(project)+ result = h.loopforge("skills", "install", "claude", "--project-root", project, check=False)+ self.assertNotEqual(0, result.returncode)+ self.assertEqual(before, h.files_under(project))+ h.loopforge("skills", "install", "claude", "--project-root", project, "--force")+ self.assertTrue((project / ".claude/skills/devflow/SKILL.md").is_file())+ self.assertFalse((project / ".claude/commands/start-devflow.md").exists())+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/e2e/test_e2e_portable_workflow.py b/tests/e2e/test_e2e_portable_workflow.pynew file mode 100644index 00000000..22d9603f--- /dev/null+++ b/tests/e2e/test_e2e_portable_workflow.py@@ -0,0 +1,154 @@+"""E2E: drive the installed Portable workflow end to end with a fake host Agent.++Each test installs the packaged CLI into a sample project, then runs+``fake_agent.py``, which follows the Skill's ``resume``-driven protocol exactly+as a host Agent would, using recorded stage outputs instead of an LLM.+"""++import json+import subprocess+import sys+import unittest++import harness as h++AGENT_STAGES = ("DESIGN", "IMPLEMENT", "REVIEW", "TEST")+STAGE_ORDER = ["REQUIREMENT", "DESIGN", "IMPLEMENT", "REVIEW", "TEST", "SUMMARY"]+++def installed_adapter(project, host):+ from fake_agent import HOST_SKILL_DIRS+ path = project / HOST_SKILL_DIRS[host] / "devflow/adapters" / host / "manifest.json"+ return json.loads(path.read_text(encoding="utf-8"))+++def events(project, kind):+ return [item for item in h.transcript(project) if item["kind"] == kind]+++class PortableWorkflowE2E(unittest.TestCase):+ def install(self, host="claude"):+ project = h.new_project(self)+ h.loopforge("skills", "install", host, "--project-root", project)+ return project++ def assert_delivered(self, project):+ state = h.workflow_state(project)+ self.assertEqual("completed", state["status"], h.transcript_text(project))+ for stage in STAGE_ORDER:+ self.assertEqual("completed", state["stages"][stage]["status"], stage)+ self.assertEqual("skipped", state["stages"]["KNOWLEDGE"]["status"])+ verify = subprocess.run(+ [sys.executable, "-m", "unittest", "check_slugify"], cwd=str(project),+ capture_output=True, text=True,+ )+ self.assertEqual(0, verify.returncode, verify.stderr)+ self.assertIn("OK", h.loopforge_validate(project))+ self.assertEqual("user file that must survive\n",+ (project / "USER_NOTES.md").read_text(encoding="utf-8"))+ return state++ def test_medium_task_runs_every_stage_on_every_host(self):+ for host in h.hosts_for("portable"):+ with self.subTest(host=host):+ project = self.install(host)+ h.fake_agent(host, project, check_rc=0)+ state = self.assert_delivered(project)++ finished = [e["stage"] for e in events(project, "finish") if e["result"] == "completed"]+ self.assertEqual(STAGE_ORDER, finished)+ prompts = events(project, "prompt")+ self.assertEqual(STAGE_ORDER, [p["stage"] for p in prompts])+ self.assertTrue(all(p["mentions_role"] for p in prompts), prompts)++ adapter = installed_adapter(project, host)+ spawned = {e["stage"]: e for e in events(project, "spawned")}+ self.assertEqual(set(AGENT_STAGES), set(spawned))+ ids = set()+ for stage in AGENT_STAGES:+ recorded = state["stages"][stage]+ self.assertEqual(adapter["stage_executor"], recorded["executor_type"])+ self.assertEqual(adapter["dispatch_tool"], recorded["dispatch_tool"])+ self.assertEqual(spawned[stage]["executor_id"], recorded["executor_id"])+ ids.add(recorded["executor_id"])+ # Independent review: every agent stage has its own executor.+ self.assertEqual(len(AGENT_STAGES), len(ids))+ for stage in ("REQUIREMENT", "SUMMARY"):+ self.assertEqual("coordinator", state["stages"][stage]["executor_type"])++ def test_resume_after_crash_inside_a_stage(self):+ for host in ("claude", "codebuddy"):+ with self.subTest(host=host):+ project = self.install(host)+ h.fake_agent(host, project, "--crash-in", "IMPLEMENT", check_rc=75)+ state = h.workflow_state(project)+ self.assertEqual("in_progress", state["stages"]["IMPLEMENT"]["status"])+ self.assertEqual("in_progress", state["status"])++ h.fake_agent(host, project, "--resume", check_rc=0)+ self.assert_delivered(project)+ sessions = []+ for item in h.transcript(project):+ if item["session"] not in sessions:+ sessions.append(item["session"])+ self.assertEqual(2, len(sessions))+ second = [e for e in events(project, "action") if e["session"] == sessions[1]]+ if host == "codebuddy":+ # A new CodeBuddy session must drop the old Team and re-dispatch.+ self.assertEqual("rotate-team", second[0]["action"])+ self.assertEqual(1, len(events(project, "team-rotated")))+ else:+ self.assertEqual(("continue-executor", "IMPLEMENT"),+ (second[0]["action"], second[0]["stage"]))+ self.assertFalse(any(e["kind"] == "init" and e["session"] == sessions[1]+ for e in h.transcript(project)))++ def test_resume_after_interruption_between_actions(self):+ project = self.install("codex")+ h.fake_agent("codex", project, "--stop-after", "7", check_rc=75)+ self.assertNotEqual("completed", h.workflow_state(project)["status"])+ h.fake_agent("codex", project, "--resume", check_rc=0)+ self.assert_delivered(project)+ finished = [e["stage"] for e in events(project, "finish") if e["result"] == "completed"]+ self.assertEqual(STAGE_ORDER, finished, "no stage may run twice after resume")++ def test_requirement_gate_waits_for_explicit_user_confirmation(self):+ project = self.install("cursor")+ h.fake_agent("cursor", project, "--no-user-confirmation", check_rc=3)+ state = h.workflow_state(project)+ self.assertEqual("awaiting_approval", state["status"])+ self.assertEqual("REQUIREMENT", state["awaiting_approval"])+ self.assertIsNone(state["stages"]["DESIGN"]["prepared_at"])+ self.assertFalse(events(project, "spawned"), "no DESIGN executor before approval")++ h.fake_agent("cursor", project, "--resume", check_rc=0)+ self.assert_delivered(project)++ def test_rejected_review_sends_work_back_to_implement(self):+ project = self.install("claude")+ h.fake_agent("claude", project, "--review-reject", "1", check_rc=0)+ self.assert_delivered(project)+ outcomes = [(e["stage"], e["result"]) for e in events(project, "finish")+ if e["stage"] in ("IMPLEMENT", "REVIEW")]+ self.assertEqual([("IMPLEMENT", "completed"), ("REVIEW", "failed"),+ ("IMPLEMENT", "completed"), ("REVIEW", "completed")], outcomes)++ def test_unfilled_artifact_is_rejected_and_workflow_does_not_advance(self):+ project = self.install("claude")+ result = h.fake_agent("claude", project, "--leave-placeholders", "DESIGN", check_rc=2)+ self.assertIn("artifact", result.stderr)+ state = h.workflow_state(project)+ self.assertEqual("in_progress", state["stages"]["DESIGN"]["status"])+ self.assertEqual("pending", state["stages"]["IMPLEMENT"]["status"])++ def test_repeated_stage_failures_block_the_workflow(self):+ project = self.install("pi")+ h.fake_agent("pi", project, "--fail-stage", "TEST:2", check_rc=4)+ state = h.workflow_state(project)+ self.assertEqual("blocked", state["status"])+ self.assertEqual("failed", state["stages"]["TEST"]["status"])+ self.assertEqual(2, state["stages"]["TEST"]["retry_count"])+++if __name__ == "__main__":+ unittest.main()