Skip to content

Practical patterns, starters & CLI tools for loop engineering with AI coding agents. Design systems that prompt and orchestrate agents (inspired by Addy Osmani and Boris Cherny). Includes loop-audit, loop-init, loop-cost.

stories: add quant loop failure stories from @50thycal (#137)

Cobus Greyling015ad4a57f21
diff --git a/stories/README.md b/stories/README.md
index 03af16a..fa4e418 100644
--- a/stories/README.md
+++ b/stories/README.md
@@ -12,6 +12,8 @@ Real-world loop engineering — including failures. Contribute yours via [CONTRI
| [l1-to-l2-graduation.md](./l1-to-l2-graduation.md) | Daily Triage | Calibration before auto-fix |
| [changelog-drafter-week-one.md](./changelog-drafter-week-one.md) | Changelog Drafter | Low-risk, high-ROI L1 win |
| [post-merge-cleanup-honest-win.md](./post-merge-cleanup-honest-win.md) | Post-Merge Cleanup | Off-peak cadence; verifier caught doc/API drift; bot-merge noise |
+| [quant-loop-the-verifier-problem.md](./quant-loop-the-verifier-problem.md) | Quant research (domain) | Numerical checker beats LLM-as-verifier for backtests |
+| [quant-loop-out-of-time.md](./quant-loop-out-of-time.md) | Quant research (domain) | Strategy passed research, failed out-of-time data |
**Template for new stories:**
diff --git a/stories/quant-loop-out-of-time.md b/stories/quant-loop-out-of-time.md
new file mode 100644
index 0000000..89797b8
--- /dev/null
+++ b/stories/quant-loop-out-of-time.md
@@ -0,0 +1,62 @@
+# Quant Loop, Part II — The Strategy That Passed Research and Still Failed
+
+*Contributed by [@50thycal](https://github.com/50thycal) — adapted from [PR #136](https://github.com/cobusgreyling/loop-engineering/pull/136).*
+
+A follow-up to [the verifier problem](quant-loop-the-verifier-problem.md). We took
+the [quant research loop](https://github.com/50thycal/loop-engineering/tree/claude/code-review-discussion-9w5pnt/starters/quant-research-loop)
+from "reject overfit garbage" to "hunt for a real strategy," built the anti-self-deception guards the
+job demands, then tried to beat them on real BTC. The punchline is the most useful
+thing a research harness can produce: **a strategy that cleared honest research and
+still failed on data it had never seen.**
+
+## Setup
+
+- Pattern: quant research loop (paper-only, L1/L2), daily BTC close 2010–2026
+- Guards built, in order: enforced trial counting → write-once lockbox →
+ walk-forward K-of-N → research-budget auto-halt → forward quarantine
+- Hypotheses: donchian breakout, time-series momentum, short-term mean reversion,
+ volatility-regime-filtered trend — all with an optional vol-targeting overlay
+
+## What Worked
+
+- **The guards caught every form of self-deception we threw at them.** Overfit
+ grid winners died in the lockbox. A strategy with a great *pooled* Sharpe but
+ 50%+ drawdowns in 3 of 5 regimes died on the K-of-N consistency gate. A forever
+ loop got halted by the trial budget.
+- **Volatility targeting was a genuine, non-cheating improvement** — it cut
+ drawdowns structurally (lower risk targeting generalizes to any data), not by
+ curve-fitting.
+- **The forward quarantine did the one thing nothing else could:** judged
+ strategies on out-of-time data that no search, tuning, or bake-off had touched.
+
+## What Broke (the good kind)
+
+- **We caught ourselves cheating.** Sweeping `target_vol` by hand until a strategy
+ passed is *uncounted multiple testing* — the enforced counter tracks the grid,
+ not the researcher. Recognizing that, on the page, was the real deliverable.
+- **`regime` passed honest research — then failed reality.** The volatility-regime
+ trend filter became the first strategy to clear walk-forward on 2010–2020 (5/5
+ folds, 14% drawdown). On genuinely unseen 2020–2026 it fell apart (37% drawdown).
+ Research success did not survive out-of-time.
+- **"Made money" ≠ "has edge."** Every strategy posted big 2020–2026 returns
+ (momentum +531%) — because BTC rose. All carried 37–41% drawdowns and sub-bar
+ risk-adjusted returns. That is beta to a bull market wearing an alpha costume.
+
+## Metrics (true out-of-time test: research 2010–2020, forward 2020–2026)
+
+| Strategy | Research | Forward (unseen) | Approved |
+|----------|----------|------------------|----------|
+| regime | **PASS** (5/5, 14% DD) | REJECT — Sharpe 0.82, +161%, 37% DD | No |
+| donchian | REJECT | REJECT — +286%, 41% DD | No |
+| tsmom | REJECT | REJECT — +531%, 39% DD | No |
+| meanrev | REJECT (0/5) | REJECT — +7%, 38% DD | No |
+
+## Lesson
+
+A research harness that only ever says "yes" is worthless; its value is in the
+"no"s you would not have said yourself. Ours said no to a strategy that passed
+every in-sample test, because the one judge that cannot be tuned — out-of-time
+data — disagreed. The loop did not find alpha. It did something more valuable for
+a solo builder: it stopped us deploying beta dressed as alpha, and it made the
+cost of fooling ourselves visible at every step. Build the loop to be the engineer
+who stays honest, not the one who presses go on a +531% backtest.
\ No newline at end of file
diff --git a/stories/quant-loop-the-verifier-problem.md b/stories/quant-loop-the-verifier-problem.md
new file mode 100644
index 0000000..ddaff05
--- /dev/null
+++ b/stories/quant-loop-the-verifier-problem.md
@@ -0,0 +1,68 @@
+# Quant Trading Loop — The Verifier Problem
+
+*Contributed by [@50thycal](https://github.com/50thycal) — adapted from [PR #136](https://github.com/cobusgreyling/loop-engineering/pull/136).*
+
+A viral article ("loop engineering for quant trading") proposed running an entire
+hedge fund as an autonomous loop. The architecture was the same five stages this
+repo documents. But it shipped the loop at **L3 from day one, with real money,
+and an LLM as the verifier.** We rebuilt it as a paper-only quant research loop
+([reference implementation](https://github.com/50thycal/loop-engineering/tree/claude/code-review-discussion-9w5pnt/starters/quant-research-loop))
+to show what changes when you keep the repo's discipline.
+
+## Setup
+
+- Domain: crypto strategy research
+- Stages: ingest → signal (maker) → verify (checker) → execute → risk
+- Posture in the article: live execution, `@auto_mode`, "prints alpha 24/7"
+- Posture here: **paper-only, numerical checker, report-first**
+
+## What the article got right
+
+- Quant trading genuinely *is* a loop (pull → signal → backtest → execute → repeat).
+- The six primitives transfer cleanly (automation, skill, state, verifier,
+ worktrees, connectors).
+- Its closing warning is correct and is this repo's gospel: stop conditions must
+ be checkable by something other than the agent's own claim.
+
+## What broke (in the article's design)
+
+- **The verifier was an LLM asked to opine on a backtest.** A backtest's failure
+ mode is *overfitting*, not faulty reasoning. A second opinion cannot catch a
+ curve-fit — the backtest is *already* the overfit artifact. Maker/checker only
+ helps when the checker does something the maker structurally cannot fake.
+- **"Self-improving" was self-overfitting.** Appending a rule after every loss
+ (skip FOMC, cap sector at 30%) is in-sample patching. After 1,000 trades you
+ have a ruleset shaped by one path of history, not institutional knowledge.
+- **L3 with real money on cycle one.** No paper phase, no human gate, capital as
+ the blast radius.
+
+## What we changed
+
+- The checker is **numerical and non-overridable**: out-of-sample split, deflated
+ Sharpe vs `n_trials`, Probabilistic Sharpe ≥ 0.95, drawdown cap, IS→OOS
+ degradation guard. An LLM may narrate it; it may never overturn it.
+- Execution is **paper-only**. Live is a separate, human-gated project.
+- Every "lesson" is a hypothesis to **re-test out-of-sample**, never a patch.
+
+## The demonstration
+
+Run `python3 -m engine.loop --once` on the default synthetic (random-walk) data:
+
+| Window | Sharpe | Verdict |
+|--------|--------|---------|
+| In-sample | **+2.54** | looks like a winner |
+| Out-of-sample | **−4.33** | falls apart |
+| Checker | — | **REJECT — no trade** |
+
+The in-sample Sharpe of 2.54 is exactly the number that would make a retail quant
+(or an LLM verifier) ship it. The numerical checker sees the OOS collapse and
+refuses to trade. **That refusal is the product.**
+
+## Lesson
+
+The loop is plumbing; it does not manufacture alpha. The maker/checker split is
+worthless unless the checker measures something the maker cannot fake. In code,
+that's an independent re-derivation of correctness. In trading, it's
+out-of-sample, cost-aware, multiple-testing-penalized statistics — never a second
+agent's vibe. Phase to live (L1 paper → L2 walk-forward → L3 gated execution);
+do not start where the blast radius is capital.
\ No newline at end of file