Practical patterns, starters & CLI tools for loop engineering with AI coding agents. Design systems that prompt and orchestrate agents (inspired by Addy Osmani and Boris Cherny). Includes loop-audit, loop-init, loop-cost.
TypeScript agentic-aiai-agentsai-codinganthropicautomationclaudeclaude-codecodexcoding-agentsdevops-automationdevtoolsgithub-actionsgrokllmloop-engineeringmcpprompt-engineering
stories: add quant loop failure stories from @50thycal (#137)
Cobus Greyling015ad4a57f21
diff --git a/stories/README.md b/stories/README.mdindex 03af16a..fa4e418 100644--- a/stories/README.md+++ b/stories/README.md@@ -12,6 +12,8 @@ Real-world loop engineering — including failures. Contribute yours via [CONTRI| [l1-to-l2-graduation.md](./l1-to-l2-graduation.md) | Daily Triage | Calibration before auto-fix || [changelog-drafter-week-one.md](./changelog-drafter-week-one.md) | Changelog Drafter | Low-risk, high-ROI L1 win || [post-merge-cleanup-honest-win.md](./post-merge-cleanup-honest-win.md) | Post-Merge Cleanup | Off-peak cadence; verifier caught doc/API drift; bot-merge noise |+| [quant-loop-the-verifier-problem.md](./quant-loop-the-verifier-problem.md) | Quant research (domain) | Numerical checker beats LLM-as-verifier for backtests |+| [quant-loop-out-of-time.md](./quant-loop-out-of-time.md) | Quant research (domain) | Strategy passed research, failed out-of-time data |**Template for new stories:**diff --git a/stories/quant-loop-out-of-time.md b/stories/quant-loop-out-of-time.mdnew file mode 100644index 0000000..89797b8--- /dev/null+++ b/stories/quant-loop-out-of-time.md@@ -0,0 +1,62 @@+# Quant Loop, Part II — The Strategy That Passed Research and Still Failed++*Contributed by [@50thycal](https://github.com/50thycal) — adapted from [PR #136](https://github.com/cobusgreyling/loop-engineering/pull/136).*++A follow-up to [the verifier problem](quant-loop-the-verifier-problem.md). We took+the [quant research loop](https://github.com/50thycal/loop-engineering/tree/claude/code-review-discussion-9w5pnt/starters/quant-research-loop)+from "reject overfit garbage" to "hunt for a real strategy," built the anti-self-deception guards the+job demands, then tried to beat them on real BTC. The punchline is the most useful+thing a research harness can produce: **a strategy that cleared honest research and+still failed on data it had never seen.**++## Setup++- Pattern: quant research loop (paper-only, L1/L2), daily BTC close 2010–2026+- Guards built, in order: enforced trial counting → write-once lockbox →+ walk-forward K-of-N → research-budget auto-halt → forward quarantine+- Hypotheses: donchian breakout, time-series momentum, short-term mean reversion,+ volatility-regime-filtered trend — all with an optional vol-targeting overlay++## What Worked++- **The guards caught every form of self-deception we threw at them.** Overfit+ grid winners died in the lockbox. A strategy with a great *pooled* Sharpe but+ 50%+ drawdowns in 3 of 5 regimes died on the K-of-N consistency gate. A forever+ loop got halted by the trial budget.+- **Volatility targeting was a genuine, non-cheating improvement** — it cut+ drawdowns structurally (lower risk targeting generalizes to any data), not by+ curve-fitting.+- **The forward quarantine did the one thing nothing else could:** judged+ strategies on out-of-time data that no search, tuning, or bake-off had touched.++## What Broke (the good kind)++- **We caught ourselves cheating.** Sweeping `target_vol` by hand until a strategy+ passed is *uncounted multiple testing* — the enforced counter tracks the grid,+ not the researcher. Recognizing that, on the page, was the real deliverable.+- **`regime` passed honest research — then failed reality.** The volatility-regime+ trend filter became the first strategy to clear walk-forward on 2010–2020 (5/5+ folds, 14% drawdown). On genuinely unseen 2020–2026 it fell apart (37% drawdown).+ Research success did not survive out-of-time.+- **"Made money" ≠ "has edge."** Every strategy posted big 2020–2026 returns+ (momentum +531%) — because BTC rose. All carried 37–41% drawdowns and sub-bar+ risk-adjusted returns. That is beta to a bull market wearing an alpha costume.++## Metrics (true out-of-time test: research 2010–2020, forward 2020–2026)++| Strategy | Research | Forward (unseen) | Approved |+|----------|----------|------------------|----------|+| regime | **PASS** (5/5, 14% DD) | REJECT — Sharpe 0.82, +161%, 37% DD | No |+| donchian | REJECT | REJECT — +286%, 41% DD | No |+| tsmom | REJECT | REJECT — +531%, 39% DD | No |+| meanrev | REJECT (0/5) | REJECT — +7%, 38% DD | No |++## Lesson++A research harness that only ever says "yes" is worthless; its value is in the+"no"s you would not have said yourself. Ours said no to a strategy that passed+every in-sample test, because the one judge that cannot be tuned — out-of-time+data — disagreed. The loop did not find alpha. It did something more valuable for+a solo builder: it stopped us deploying beta dressed as alpha, and it made the+cost of fooling ourselves visible at every step. Build the loop to be the engineer+who stays honest, not the one who presses go on a +531% backtest.\ No newline at end of filediff --git a/stories/quant-loop-the-verifier-problem.md b/stories/quant-loop-the-verifier-problem.mdnew file mode 100644index 0000000..ddaff05--- /dev/null+++ b/stories/quant-loop-the-verifier-problem.md@@ -0,0 +1,68 @@+# Quant Trading Loop — The Verifier Problem++*Contributed by [@50thycal](https://github.com/50thycal) — adapted from [PR #136](https://github.com/cobusgreyling/loop-engineering/pull/136).*++A viral article ("loop engineering for quant trading") proposed running an entire+hedge fund as an autonomous loop. The architecture was the same five stages this+repo documents. But it shipped the loop at **L3 from day one, with real money,+and an LLM as the verifier.** We rebuilt it as a paper-only quant research loop+([reference implementation](https://github.com/50thycal/loop-engineering/tree/claude/code-review-discussion-9w5pnt/starters/quant-research-loop))+to show what changes when you keep the repo's discipline.++## Setup++- Domain: crypto strategy research+- Stages: ingest → signal (maker) → verify (checker) → execute → risk+- Posture in the article: live execution, `@auto_mode`, "prints alpha 24/7"+- Posture here: **paper-only, numerical checker, report-first**++## What the article got right++- Quant trading genuinely *is* a loop (pull → signal → backtest → execute → repeat).+- The six primitives transfer cleanly (automation, skill, state, verifier,+ worktrees, connectors).+- Its closing warning is correct and is this repo's gospel: stop conditions must+ be checkable by something other than the agent's own claim.++## What broke (in the article's design)++- **The verifier was an LLM asked to opine on a backtest.** A backtest's failure+ mode is *overfitting*, not faulty reasoning. A second opinion cannot catch a+ curve-fit — the backtest is *already* the overfit artifact. Maker/checker only+ helps when the checker does something the maker structurally cannot fake.+- **"Self-improving" was self-overfitting.** Appending a rule after every loss+ (skip FOMC, cap sector at 30%) is in-sample patching. After 1,000 trades you+ have a ruleset shaped by one path of history, not institutional knowledge.+- **L3 with real money on cycle one.** No paper phase, no human gate, capital as+ the blast radius.++## What we changed++- The checker is **numerical and non-overridable**: out-of-sample split, deflated+ Sharpe vs `n_trials`, Probabilistic Sharpe ≥ 0.95, drawdown cap, IS→OOS+ degradation guard. An LLM may narrate it; it may never overturn it.+- Execution is **paper-only**. Live is a separate, human-gated project.+- Every "lesson" is a hypothesis to **re-test out-of-sample**, never a patch.++## The demonstration++Run `python3 -m engine.loop --once` on the default synthetic (random-walk) data:++| Window | Sharpe | Verdict |+|--------|--------|---------|+| In-sample | **+2.54** | looks like a winner |+| Out-of-sample | **−4.33** | falls apart |+| Checker | — | **REJECT — no trade** |++The in-sample Sharpe of 2.54 is exactly the number that would make a retail quant+(or an LLM verifier) ship it. The numerical checker sees the OOS collapse and+refuses to trade. **That refusal is the product.**++## Lesson++The loop is plumbing; it does not manufacture alpha. The maker/checker split is+worthless unless the checker measures something the maker cannot fake. In code,+that's an independent re-derivation of correctness. In trading, it's+out-of-sample, cost-aware, multiple-testing-penalized statistics — never a second+agent's vibe. Phase to live (L1 paper → L2 walk-forward → L3 gated execution);+do not start where the blast radius is capital.\ No newline at end of file