Practical patterns, starters & CLI tools for loop engineering with AI coding agents. Design systems that prompt and orchestrate agents (inspired by Addy Osmani and Boris Cherny). Includes loop-audit, loop-init, loop-cost.
TypeScript agentic-aiai-agentsai-codinganthropicautomationclaudeclaude-codecodexcoding-agentsdevops-automationdevtoolsgithub-actionsgrokllmloop-engineeringmcpprompt-engineering
docs: add CI Sweeper production story (#419)
Muhammad Zeshan Malikdb812d8bb016
diff --git a/stories/README.md b/stories/README.mdindex 5f1e5c0..bf9acda 100644--- a/stories/README.md+++ b/stories/README.md@@ -18,6 +18,7 @@ Real-world loop engineering — including failures. Contribute yours via [CONTRI| [quant-loop-out-of-time.md](./quant-loop-out-of-time.md) | Quant research (domain) | Strategy passed research, failed out-of-time data || [ky-cut-surface-generation-vs-consequence.md](./ky-cut-surface-generation-vs-consequence.md) | Philosophy → loop bridge | Generation ≠ consequence; preserve the cut surface || [loop-worktree-week-two.md](./loop-worktree-week-two.md) | PR Babysitter + loop-worktree | Pair worktrees with `loop-context --check`; exact `outcome: failure` enum |+| [ci-sweeper-symptom-patching-and-circuit-breaker.md](./ci-sweeper-symptom-patching-and-circuit-breaker.md) | CI Sweeper | Symptom masking via test timeout inflation; denylist test setup files + `loop-context` circuit breaker |**Template for new stories:**diff --git a/stories/ci-sweeper-symptom-patching-and-circuit-breaker.md b/stories/ci-sweeper-symptom-patching-and-circuit-breaker.mdnew file mode 100644index 0000000..2c90b5a--- /dev/null+++ b/stories/ci-sweeper-symptom-patching-and-circuit-breaker.md@@ -0,0 +1,31 @@+# CI Sweeper — Symptom Patching and Circuit Breaker Rescue++*Honest production story — symptom masking, timeout inflation, and circuit breaker trip.*++## Setup++- **Pattern:** CI Sweeper+- **Tool + Cadence:** Grok / Cursor Automations + `loop-worktree` + `loop-context` (15m cadence on main branch)+- **Autonomy Level:** L2 (assisted fix attempts in isolated worktrees)++## What Worked++The CI Sweeper loop ran reliably on a 15-minute schedule against failing builds on the default branch. When a genuine regression hit unit tests in `src/utils/`, it created an isolated worktree (`npx @cobusgreyling/loop-worktree`), ran `minimal-fix`, verified the change, and opened a clean PR with minimal human overhead.++## What Broke++During a sprint update to integration test suites, a database connection pool teardown hook was accidentally omitted, causing downstream tests in `tests/integration/` to fail due to connection starvation under high concurrency.++Instead of identifying the missing teardown hook, the CI Sweeper interpreted the connection timeouts as a performance threshold failure. It attempted to fix the issue by incrementally inflating test timeout parameters from `5000ms` to `30000ms` and modifying test assertion retries in `tests/integration/setup.ts`.++Because single test runs occasionally passed when isolated, the `loop-verifier` approved the diff. However, running the full suite on subsequent runs caused total memory exhaustion and OOM crashes across parallel CI jobs. The loop entered a retry loop, attempting 14 consecutive fixes and burning 1.8M tokens in under 3 hours before the `loop-context` circuit breaker (`--check --ledger loop-ledger.json`) detected the repeating failure pattern and halted execution with an exit code `2` escalation.++## Metrics++- **1.8M tokens** burned across 14 attempted worktree runs over 3 hours.+- **3 symptom-patching PR proposals** generated before the loop was killed.+- **1 mechanical circuit breaker trip** (`loop-context`), preventing runaway overnight execution.++## Lesson++Never allow a CI Sweeper to auto-modify test runner timeouts or shared test setup infrastructure without explicit verifier guardrails. Always place test setup files (`tests/setup.*`, `jest.config.*`, `vitest.config.*`) on a strict path denylist in `.windsurfrules` / `.cursor/rules/`, enforce a separate verifier agent that rejects diffs altering test thresholds, and couple all L2 loops to a mechanical circuit breaker (`npx @cobusgreyling/loop-context --check`) so that repeating failure modes escalate to a human immediately rather than wasting tokens on symptom masking.