Skip to content

Harness engineering beginner tutorial, from 0 to 1

Expand course guidance and add quality document template

sanbuphyec0a6d2a3e1a
diff --git a/lectures/lecture-05-why-long-running-tasks-lose-continuity/README.md b/lectures/lecture-05-why-long-running-tasks-lose-continuity/README.md
index 406a211..17a9004 100644
--- a/lectures/lecture-05-why-long-running-tasks-lose-continuity/README.md
+++ b/lectures/lecture-05-why-long-running-tasks-lose-continuity/README.md
@@ -39,11 +39,32 @@ artifacts that answer:
2. What is failing or unfinished?
3. What is the next executable step?
-Context summaries can reduce short-term rediscovery, but summaries alone are
-not durable state. Long-running reliability requires artifacts that can be
-versioned, diffed, and validated across sessions. In practice, this means the
-harness must treat handoff quality as a formal requirement, not incidental
-documentation.
+There are two distinct strategies for managing context across sessions:
+
+**Compaction** summarizes the current context window in place. It preserves
+conversational continuity and is cheap to execute, but the summary may lose
+important details. The agent continues in the same window with a compressed
+view of what came before.
+
+**Context reset** clears the window entirely and forces the agent to
+reconstruct state from repository artifacts only. It provides a clean starting
+point with no leftover noise, but it only works if the handoff artifacts are
+complete and well-structured.
+
+These are not interchangeable. Compaction works for short-to-medium sessions
+where the conversation thread is still mostly relevant. Context resets work
+better for long multi-session projects where accumulated context becomes stale
+or contradictory. Anthropic's long-running harness work shows that agents
+sometimes exhibit "context anxiety" — wrapping up prematurely as they approach
+perceived context limits. Full context resets with structured handoff files
+eliminate this problem entirely.
+
+The choice between compaction and reset determines what your handoff artifacts
+need to contain. If you rely on resets, the artifacts must be self-sufficient —
+a new agent session must be able to reconstruct the full project state from
+files alone. This is why feature lists, progress logs, and handoff notes are
+not optional documentation; they are the mechanism that makes context resets
+work.
## Examples and Artifacts
diff --git a/lectures/lecture-10-why-end-to-end-testing-changes-results/README.md b/lectures/lecture-10-why-end-to-end-testing-changes-results/README.md
index 3552f9f..d09d23f 100644
--- a/lectures/lecture-10-why-end-to-end-testing-changes-results/README.md
+++ b/lectures/lecture-10-why-end-to-end-testing-changes-results/README.md
@@ -17,6 +17,7 @@ together as a system.
- Architecture boundaries must be enforced as executable rules.
- Repeated review comments can be promoted into harness checks.
- Remediation-oriented failures improve future agent behavior.
+- Linter and test error messages should be written for agents, not just humans.
## Detailed Explanation
@@ -41,6 +42,16 @@ execution:
End-to-end tests surface these interactions and convert architecture intent into
observable pass/fail criteria.
+One particularly effective pattern from OpenAI's harness practice: **write
+linter error messages for the agent, not just for humans.** When a custom
+linter catches an architecture violation, the error message should include a
+short remediation instruction that tells the agent exactly what to do. For
+example, instead of `"Direct filesystem access in renderer"`, the message might
+read `"Direct filesystem access in renderer. All file operations must go through
+the preload bridge. Move this call to preload/file-ops.ts and invoke it via
+window.api."`. This pattern converts structural rules into self-correcting
+feedback loops — the agent sees the error and immediately knows the fix.
+
## Examples and Artifacts
- [`code/architecture-rules.md`](./code/architecture-rules.md): boundary rules
diff --git a/lectures/lecture-11-why-observability-belongs-inside-the-harness/README.md b/lectures/lecture-11-why-observability-belongs-inside-the-harness/README.md
index 6a578f1..479c35c 100644
--- a/lectures/lecture-11-why-observability-belongs-inside-the-harness/README.md
+++ b/lectures/lecture-11-why-observability-belongs-inside-the-harness/README.md
@@ -40,6 +40,31 @@ Observability therefore has two layers:
The two layers should be designed together. Runtime signals explain behavior,
while process artifacts explain why a change should be accepted or rejected.
+### Sprint Contracts
+
+When multiple roles collaborate — planner, generator, evaluator — the point
+where work begins is a common failure surface. If the generator and evaluator
+have different expectations about what "done" looks like, evaluation becomes
+subjective and revision loops become wasteful.
+
+Anthropic's harness practice introduces **sprint contracts** to solve this.
+Before any code is written, the generator and evaluator negotiate a short
+written agreement that specifies:
+
+1. What will be built in this sprint.
+2. How success will be verified (specific tests or checks).
+3. What is explicitly out of scope.
+
+The contract is a file in the repository. Both roles read it before starting.
+If they disagree on the terms, they iterate on the contract first. This
+front-loads alignment and prevents the common pattern where the generator
+builds something the evaluator immediately rejects for predictable reasons.
+
+Sprint contracts are especially valuable when the evaluator is a separate
+agent. Without a contract, the evaluator's review criteria are implicit and
+may shift between runs. With a contract, the evaluator grades against an
+explicit, pre-agreed standard.
+
## Examples and Artifacts
- [`code/sprint-contract.md`](./code/sprint-contract.md): explicit goal and
diff --git a/lectures/lecture-12-why-every-session-must-leave-a-clean-state/README.md b/lectures/lecture-12-why-every-session-must-leave-a-clean-state/README.md
index e4f6d40..dc51096 100644
--- a/lectures/lecture-12-why-every-session-must-leave-a-clean-state/README.md
+++ b/lectures/lecture-12-why-every-session-must-leave-a-clean-state/README.md
@@ -17,6 +17,77 @@ successful single runs.
- Benchmark slices are needed to detect harness drift over time.
- Cleanup loops reduce entropy in code, docs, and quality signals.
- Maintenance is part of normal harness operation, not emergency work.
+- Quality documents track per-domain and per-layer grades over time.
+- Harness components encode assumptions about what the model cannot do; these
+ assumptions go stale as models improve.
+
+## Detailed Explanation
+
+OpenAI's harness framing highlights repeatability: a workflow should continue to
+produce reliable outcomes as tasks and contexts change. This requires measurable
+checks beyond one successful demonstration. Benchmark comparisons help isolate
+whether a harness change improved completion rate, reduced retries, or caught
+more defects before review.
+
+Anthropic's long-running agent observations show why clean session exits are
+critical. When unfinished work, stale guidance, or weak boundaries accumulate,
+later sessions spend effort on state repair instead of forward progress. A clean
+exit policy therefore includes explicit recording of progress, removal of stale
+artifacts, and confirmation that the standard startup path still works.
+
+Operationally, clean state and measurement reinforce each other:
+
+1. Cleanup reduces noise.
+2. Lower noise improves benchmark signal quality.
+3. Clear benchmarks identify where cleanup policy should be strengthened.
+
+### Quality Documents
+
+OpenAI's harness practice includes a quality document that grades each product
+domain and architectural layer. This is not a one-time assessment — it is a
+living artifact that tracks whether the codebase is getting stronger or weaker
+over time.
+
+A quality document typically grades domains on criteria like:
+
+- Does verification pass for this area?
+- Can an agent read and modify this area without confusion?
+- Are tests stable, or do they flake?
+- Are architecture boundaries enforced?
+
+The grades serve multiple purposes:
+
+1. **Session startup**: an agent reads the quality document to identify where
+ the codebase is weakest and prioritize accordingly.
+2. **Session exit**: the agent updates grades to reflect what changed.
+3. **Benchmark**: comparing quality document snapshots before and after a
+ harness change shows whether the change actually improved codebase health.
+4. **Cleanup**: recurring maintenance sessions target domains with the lowest
+ grades first.
+
+A quality document template is available in the course resource library.
+
+### Harness Simplification
+
+Anthropic's harness design work makes an important meta-point: every harness
+component exists because the model cannot do something reliably on its own. As
+models improve, these assumptions become stale. A component that was load-bearing
+three months ago may now be unnecessary overhead.
+
+The recommended practice is to periodically remove one harness component at a
+time and observe the impact:
+
+1. Remove the component.
+2. Run the benchmark task suite.
+3. Compare results with and without the component.
+
+If outcomes do not degrade, the component can stay removed. If they do, it
+should be restored or replaced with a lighter alternative. This process prevents
+harnesses from accumulating complexity that no longer serves a purpose.
+
+The principle is: **find the simplest solution possible, and only increase
+complexity when needed.** A harness that works today may be over-engineered
+tomorrow.
## Detailed Explanation
diff --git a/projects/project-05-grounded-qa-verification/README.md b/projects/project-05-grounded-qa-verification/README.md
index e07ca35..748666d 100644
--- a/projects/project-05-grounded-qa-verification/README.md
+++ b/projects/project-05-grounded-qa-verification/README.md
@@ -39,15 +39,49 @@ Run 2 and Run 3 use upgraded role separation:
## Procedure
+### Sprint Contract
+
+Before any code is written, the generator and evaluator must negotiate a sprint
+contract — a short written agreement specifying what will be built, how success
+will be verified, and what is explicitly out of scope. The contract lives in the
+repository as a file both roles can read.
+
+This step matters because it front-loads alignment. Without a contract, the
+evaluator's criteria are implicit and may shift during review. With a contract,
+the evaluator grades against an explicit, pre-agreed standard.
+
+Template: [`code/sprint-contract.md`](../../lectures/lecture-11-why-observability-belongs-inside-the-harness/code/sprint-contract.md)
+
+### Evaluator Tuning
+
+An out-of-the-box evaluator agent is a poor QA agent — it identifies issues
+then talks itself into approving. The evaluator rubric requires iterative tuning:
+
+1. Run the evaluator on a completed sprint.
+2. Compare the evaluator's scores against your own human judgment.
+3. Where they diverge, update the rubric to be more specific about what
+ constitutes a pass or fail.
+4. Re-run the evaluator on the same output and check alignment.
+5. Repeat until the evaluator's judgment consistently matches human review.
+
+Plan for 3-5 tuning rounds before the evaluator produces reliable scores. Record
+each tuning iteration in the deliverables.
+
+### Execution Steps
+
1. Select upgrade scope and define acceptance criteria before coding.
-2. Start three branches from the same commit:
+2. Write a sprint contract with the generator and evaluator agreeing on scope,
+ verification, and exclusions.
+3. Start three branches from the same commit:
`p05-single`, `p05-gen-eval`, `p05-plan-gen-eval`.
-3. Run each branch with Codex or Claude Code using comparable model and budget.
-4. Use the same evaluator rubric for all three final outputs.
-5. In improved runs, require at least one revision round after evaluator
+4. Run each branch with Codex or Claude Code using comparable model and budget.
+5. Use the same evaluator rubric for all three final outputs.
+6. In improved runs, require at least one revision round after evaluator
feedback.
-6. Verify behavior with runnable evidence and record unresolved defects.
-7. Compare quality and effort across the three harness variants.
+7. Verify behavior with runnable evidence and record unresolved defects.
+8. Tune the evaluator rubric across at least two iterations, recording what
+ changed and why.
+9. Compare quality and effort across the three harness variants.
## What to Measure
@@ -56,9 +90,14 @@ Run 2 and Run 3 use upgraded role separation:
- Rubric score by category (correctness, reliability, maintainability, UX).
- Rework required after review.
- Final robustness of the upgraded product slice.
+- Evaluator tuning iterations: how many rounds before evaluator judgment aligned
+ with human review.
+- Sprint contract effectiveness: did the contract reduce ambiguity in evaluation?
## Deliverables
+- Sprint contract document.
+- Evaluator rubric tuning log (before/after for each tuning iteration).
- Single-role run record (prompt, transcript/log, runnable evidence).
- Generator-evaluator run record (including rubric and revision notes).
- Planner-generator-evaluator run record (including rubric and revisions).
diff --git a/projects/project-06-runtime-observability-and-debugging/README.md b/projects/project-06-runtime-observability-and-debugging/README.md
index 64741d8..092f0b8 100644
--- a/projects/project-06-runtime-observability-and-debugging/README.md
+++ b/projects/project-06-runtime-observability-and-debugging/README.md
@@ -39,20 +39,33 @@ Run 2 must use the strongest harness assembled in the course:
- Explicit scope and verification gates.
- Runtime signals and structural constraints.
- Evaluator or multi-role review where applicable.
+- Quality document tracking per-domain and per-layer grades.
## Procedure
-1. Define one fixed benchmark task suite and one scoring sheet before running
+1. Create a quality document using the template in
+ `resources/en/templates/quality-document.md` (or `resources/zh/templates/`).
+ Fill in initial grades for all product domains and architectural layers based
+ on the current codebase state.
+2. Define one fixed benchmark task suite and one scoring sheet before running
any agent session.
-2. Start from one common commit and branch into `p06-baseline` and
+3. Start from one common commit and branch into `p06-baseline` and
`p06-improved`.
-3. Run baseline and improved sessions with Codex or Claude Code using the same
+4. Run baseline and improved sessions with Codex or Claude Code using the same
model and comparable budget.
-4. Score both runs immediately after execution.
-5. On the improved branch, perform a cleanup pass targeting entropy, dead code,
+5. Score both runs immediately after execution.
+6. Update the quality document after each run, recording grade changes per
+ domain and layer.
+7. On the improved branch, perform a cleanup pass targeting entropy, dead code,
unclear docs, and unstable run paths.
-6. Rerun the same benchmark task suite after cleanup and rescore.
-7. Compare baseline vs improved vs improved-after-cleanup.
+8. Rerun the same benchmark task suite after cleanup and rescore.
+9. Update the quality document one final time.
+10. Compare three quality document snapshots (baseline, improved, post-cleanup)
+ alongside benchmark scores.
+11. **Harness simplification pass:** Remove one harness component (for example,
+ the sprint contract, or the explicit scope gate). Rerun the benchmark. If
+ outcomes do not degrade, the component was unnecessary overhead. If they do,
+ restore it. Record the result.
## What to Measure
@@ -61,10 +74,14 @@ Run 2 must use the strongest harness assembled in the course:
- Defect count before human intervention.
- Cleanup effort (time and files touched).
- Post-cleanup legibility and restart success.
+- Quality document grade changes across the three snapshots.
+- Harness simplification result: which components were removable, which were load-bearing.
## Deliverables
+- Quality document with three snapshots (baseline, improved, post-cleanup).
- Baseline benchmark record with scores and evidence.
- Improved benchmark record with scores and evidence.
- Cleanup run record with before/after score delta.
+- Harness simplification log: component removed, benchmark result, decision.
- Final capstone comparison summary with key lessons learned.
diff --git a/resources/en/templates/README.md b/resources/en/templates/README.md
index c81c82d..797fa3c 100644
--- a/resources/en/templates/README.md
+++ b/resources/en/templates/README.md
@@ -174,3 +174,53 @@ A scorecard for reviewing agent output quality. Use this after a session or at p
- Accept — meets the bar
- Revise — needs fixes before accepting
- Block — fundamental issues that need to be resolved first
+
+**Important: the evaluator needs tuning.** Out of the box, agents are poor self-judges — they identify issues then talk themselves into approving. You will need to iterate:
+
+1. Run the evaluator on a completed sprint.
+2. Compare its scores against your own human judgment.
+3. Where they diverge, make the rubric more specific about pass/fail criteria.
+4. Re-run and check alignment.
+5. Repeat until the evaluator consistently matches human review.
+
+Plan for 3-5 tuning rounds. Record each change so you can track what improved alignment.
+
+## quality-document.md
+
+A quality snapshot that grades each product domain and architectural layer in your project. Tracks codebase health over time, not just individual session output.
+
+**How to use it:**
+
+- Copy to your project root
+- Before starting a session: read it to understand where the codebase is weakest
+- After a session: update grades based on what changed
+- Over time: compare snapshots to see which harness changes actually improved codebase health
+
+**What it grades:**
+
+- **Product domains** (e.g., document import, Q&A flow, indexing): each domain gets a grade (A-D) across verification status, agent legibility, test stability, and key gaps
+- **Architectural layers** (e.g., main process, preload, renderer, services): each layer gets a grade for boundary enforcement and agent legibility
+
+**Why it matters:**
+
+The evaluator rubric scores individual agent outputs. The quality document scores the codebase itself. They answer different questions:
+
+- Evaluator rubric: "Did the agent do good work this session?"
+- Quality document: "Is the project getting stronger or weaker over time?"
+
+**When to update:**
+
+- After each significant session
+- Before benchmark comparisons
+- After cleanup or simplification passes
+- When onboarding a new agent or model to the project
+
+**Harness simplification tie-in:**
+
+The quality document also supports harness simplification. Every harness component encodes an assumption about what the model cannot do. As models improve, these assumptions go stale. To check whether a component is still needed:
+
+1. Take a quality document snapshot.
+2. Remove one harness component.
+3. Run the benchmark task suite.
+4. Take another snapshot.
+5. Compare — if grades didn't drop, the component was overhead. If they did, restore it.
diff --git a/resources/en/templates/quality-document.md b/resources/en/templates/quality-document.md
new file mode 100644
index 0000000..fb8699b
--- /dev/null
+++ b/resources/en/templates/quality-document.md
@@ -0,0 +1,43 @@
+# Quality Document
+
+A quality snapshot for each product domain and architectural layer. Both agents and humans can use this document to quickly understand where the codebase is strong and where it needs work.
+
+**Update cadence:** After each significant session, or before starting a new phase of work.
+
+**Grading scale:**
+
+- **A**: All verification passing, clean architecture, agent-legible, stable tests
+- **B**: Verification passing, mostly clean, minor gaps in legibility or test coverage
+- **C**: Partially working, known gaps, some code areas hard for agents to understand
+- **D**: Not working, or major structural issues
+
+---
+
+## Product Domains
+
+| Domain | Grade | Verification | Agent Legibility | Test Stability | Key Gaps | Last Updated |
+|--------|-------|-------------|-----------------|---------------|----------|-------------|
+| Document Import | - | - | - | - | - | - |
+| Document Management | - | - | - | - | - | - |
+| Document Indexing | - | - | - | - | - | - |
+| Q&A Flow | - | - | - | - | - | - |
+| Grounded Answers | - | - | - | - | - | - |
+
+## Architectural Layers
+
+| Layer | Grade | Boundary Enforcement | Agent Legibility | Key Gaps | Last Updated |
+|-------|-------|---------------------|-----------------|----------|-------------|
+| Main Process | - | - | - | - | - |
+| Preload | - | - | - | - | - |
+| Renderer | - | - | - | - | - |
+| Services | - | - | - | - | - |
+
+## Change History
+
+### YYYY-MM-DD
+
+- Changes:
+- Domains promoted:
+- Domoted:
+- New gaps identified:
+- Gaps closed:
diff --git a/resources/zh/templates/README.md b/resources/zh/templates/README.md
index 09bd7a6..9395970 100644
--- a/resources/zh/templates/README.md
+++ b/resources/zh/templates/README.md
@@ -174,3 +174,53 @@ agent 任何时候只能有一个功能处于 `in_progress`。
- Accept — 达标
- Revise — 需要修补才能接受
- Block — 有根本性问题,需要先解决
+
+**Evaluator 需要校准。** 开箱即用的 agent 做评审很弱——它会发现问题,然后把自己说服到通过。你需要反复校准:
+
+1. 用 evaluator 给一个已完成的 sprint 打分。
+2. 把它的分数和你自己的人工判断对比。
+3. 有分歧的地方,把 rubric 里的通过/失败标准写得更具体。
+4. 对同一个输出重新跑 evaluator,看对齐了没有。
+5. 重复直到 evaluator 的判断和人工评审基本一致。
+
+预计需要 3-5 轮校准。每轮记录改了什么、为什么改。
+
+## quality-document.md
+
+质量快照。给项目的每个产品领域和架构层打分,跟踪代码库随时间是变强了还是变弱了。
+
+**怎么用:**
+
+- 复制到项目根目录
+- 开始会话前:读它,了解代码库哪里最弱
+- 会话结束后:更新评级
+- 长期:对比不同时间点的快照,看哪些 harness 改动真正改善了代码库健康度
+
+**评什么:**
+
+- **产品领域**(如文档导入、问答流程、索引):每个领域按验证状态、Agent 可读性、测试稳定性、关键缺口打分
+- **架构层**(如 Main Process、Preload、Renderer、Services):每层按边界执行和 Agent 可读性打分
+
+**为什么需要它:**
+
+Evaluator rubric 评的是单次 agent 输出的质量。Quality document 评的是代码库本身的质量。它们回答的是不同的问题:
+
+- Evaluator rubric:"这轮 agent 做得好不好?"
+- Quality document:"这个项目是在变强还是变弱?"
+
+**什么时候更新:**
+
+- 每轮重要会话之后
+- 做基准对比之前
+- 做清理或简化之后
+- 换新 agent 或新模型时
+
+**和 harness 简化的关系:**
+
+Quality document 也可以用来验证 harness 是否可以简化。Harness 里的每个组件都编码了一个假设——"模型做不到这件事"。模型变强后,这些假设就过时了。检查某个组件是否还有必要:
+
+1. 拍一份 quality document 快照。
+2. 移除一个 harness 组件。
+3. 跑基准测试。
+4. 再拍一份快照。
+5. 对比——评级没降,说明那个组件是多余的。降了,就恢复。
diff --git a/resources/zh/templates/quality-document.md b/resources/zh/templates/quality-document.md
new file mode 100644
index 0000000..a21367b
--- /dev/null
+++ b/resources/zh/templates/quality-document.md
@@ -0,0 +1,43 @@
+# 质量文档
+
+每个产品领域和架构层的质量快照。agent 和人都能通过这份文档快速了解项目哪里强、哪里弱。
+
+**更新时机:** 每轮重要会话结束后,或开始新一阶段工作前。
+
+**评级标准:**
+
+- **A**:验证全部通过,架构干净,agent 能读懂,测试稳定
+- **B**:验证通过,基本干净,可读性或测试覆盖有少量缺口
+- **C**:部分可用,有已知缺口,部分代码 agent 不容易理解
+- **D**:不可用,或存在重大结构问题
+
+---
+
+## 产品领域
+
+| 领域 | 评级 | 验证状态 | Agent 可读性 | 测试稳定性 | 关键缺口 | 上次更新 |
+|------|------|---------|-------------|-----------|---------|---------|
+| 文档导入 | - | - | - | - | - | - |
+| 文档管理 | - | - | - | - | - | - |
+| 文档索引 | - | - | - | - | - | - |
+| 问答流程 | - | - | - | - | - | - |
+| 引用式回答 | - | - | - | - | - | - |
+
+## 架构层
+
+| 层级 | 评级 | 边界执行 | Agent 可读性 | 关键缺口 | 上次更新 |
+|------|------|---------|-------------|---------|---------|
+| Main Process | - | - | - | - | - |
+| Preload | - | - | - | - | - |
+| Renderer | - | - | - | - | - |
+| Services | - | - | - | - | - |
+
+## 变更历史
+
+### YYYY-MM-DD
+
+- 变更内容:
+- 提升:
+- 下降:
+- 新发现的缺口:
+- 已关闭的缺口: