Harness engineering beginner tutorial, from 0 to 1
TypeScript agentagenticagentic-aiaiai-agentai-agentsdshdsh-pluginharnessharness-engineeringharness-frameworkllm
Expand course guidance and add quality document template
sanbuphyec0a6d2a3e1a
diff --git a/lectures/lecture-05-why-long-running-tasks-lose-continuity/README.md b/lectures/lecture-05-why-long-running-tasks-lose-continuity/README.mdindex 406a211..17a9004 100644--- a/lectures/lecture-05-why-long-running-tasks-lose-continuity/README.md+++ b/lectures/lecture-05-why-long-running-tasks-lose-continuity/README.md@@ -39,11 +39,32 @@ artifacts that answer:2. What is failing or unfinished?3. What is the next executable step?-Context summaries can reduce short-term rediscovery, but summaries alone are-not durable state. Long-running reliability requires artifacts that can be-versioned, diffed, and validated across sessions. In practice, this means the-harness must treat handoff quality as a formal requirement, not incidental-documentation.+There are two distinct strategies for managing context across sessions:++**Compaction** summarizes the current context window in place. It preserves+conversational continuity and is cheap to execute, but the summary may lose+important details. The agent continues in the same window with a compressed+view of what came before.++**Context reset** clears the window entirely and forces the agent to+reconstruct state from repository artifacts only. It provides a clean starting+point with no leftover noise, but it only works if the handoff artifacts are+complete and well-structured.++These are not interchangeable. Compaction works for short-to-medium sessions+where the conversation thread is still mostly relevant. Context resets work+better for long multi-session projects where accumulated context becomes stale+or contradictory. Anthropic's long-running harness work shows that agents+sometimes exhibit "context anxiety" — wrapping up prematurely as they approach+perceived context limits. Full context resets with structured handoff files+eliminate this problem entirely.++The choice between compaction and reset determines what your handoff artifacts+need to contain. If you rely on resets, the artifacts must be self-sufficient —+a new agent session must be able to reconstruct the full project state from+files alone. This is why feature lists, progress logs, and handoff notes are+not optional documentation; they are the mechanism that makes context resets+work.## Examples and Artifactsdiff --git a/lectures/lecture-10-why-end-to-end-testing-changes-results/README.md b/lectures/lecture-10-why-end-to-end-testing-changes-results/README.mdindex 3552f9f..d09d23f 100644--- a/lectures/lecture-10-why-end-to-end-testing-changes-results/README.md+++ b/lectures/lecture-10-why-end-to-end-testing-changes-results/README.md@@ -17,6 +17,7 @@ together as a system.- Architecture boundaries must be enforced as executable rules.- Repeated review comments can be promoted into harness checks.- Remediation-oriented failures improve future agent behavior.+- Linter and test error messages should be written for agents, not just humans.## Detailed Explanation@@ -41,6 +42,16 @@ execution:End-to-end tests surface these interactions and convert architecture intent intoobservable pass/fail criteria.+One particularly effective pattern from OpenAI's harness practice: **write+linter error messages for the agent, not just for humans.** When a custom+linter catches an architecture violation, the error message should include a+short remediation instruction that tells the agent exactly what to do. For+example, instead of `"Direct filesystem access in renderer"`, the message might+read `"Direct filesystem access in renderer. All file operations must go through+the preload bridge. Move this call to preload/file-ops.ts and invoke it via+window.api."`. This pattern converts structural rules into self-correcting+feedback loops — the agent sees the error and immediately knows the fix.+## Examples and Artifacts- [`code/architecture-rules.md`](./code/architecture-rules.md): boundary rulesdiff --git a/lectures/lecture-11-why-observability-belongs-inside-the-harness/README.md b/lectures/lecture-11-why-observability-belongs-inside-the-harness/README.mdindex 6a578f1..479c35c 100644--- a/lectures/lecture-11-why-observability-belongs-inside-the-harness/README.md+++ b/lectures/lecture-11-why-observability-belongs-inside-the-harness/README.md@@ -40,6 +40,31 @@ Observability therefore has two layers:The two layers should be designed together. Runtime signals explain behavior,while process artifacts explain why a change should be accepted or rejected.+### Sprint Contracts++When multiple roles collaborate — planner, generator, evaluator — the point+where work begins is a common failure surface. If the generator and evaluator+have different expectations about what "done" looks like, evaluation becomes+subjective and revision loops become wasteful.++Anthropic's harness practice introduces **sprint contracts** to solve this.+Before any code is written, the generator and evaluator negotiate a short+written agreement that specifies:++1. What will be built in this sprint.+2. How success will be verified (specific tests or checks).+3. What is explicitly out of scope.++The contract is a file in the repository. Both roles read it before starting.+If they disagree on the terms, they iterate on the contract first. This+front-loads alignment and prevents the common pattern where the generator+builds something the evaluator immediately rejects for predictable reasons.++Sprint contracts are especially valuable when the evaluator is a separate+agent. Without a contract, the evaluator's review criteria are implicit and+may shift between runs. With a contract, the evaluator grades against an+explicit, pre-agreed standard.+## Examples and Artifacts- [`code/sprint-contract.md`](./code/sprint-contract.md): explicit goal anddiff --git a/lectures/lecture-12-why-every-session-must-leave-a-clean-state/README.md b/lectures/lecture-12-why-every-session-must-leave-a-clean-state/README.mdindex e4f6d40..dc51096 100644--- a/lectures/lecture-12-why-every-session-must-leave-a-clean-state/README.md+++ b/lectures/lecture-12-why-every-session-must-leave-a-clean-state/README.md@@ -17,6 +17,77 @@ successful single runs.- Benchmark slices are needed to detect harness drift over time.- Cleanup loops reduce entropy in code, docs, and quality signals.- Maintenance is part of normal harness operation, not emergency work.+- Quality documents track per-domain and per-layer grades over time.+- Harness components encode assumptions about what the model cannot do; these+ assumptions go stale as models improve.++## Detailed Explanation++OpenAI's harness framing highlights repeatability: a workflow should continue to+produce reliable outcomes as tasks and contexts change. This requires measurable+checks beyond one successful demonstration. Benchmark comparisons help isolate+whether a harness change improved completion rate, reduced retries, or caught+more defects before review.++Anthropic's long-running agent observations show why clean session exits are+critical. When unfinished work, stale guidance, or weak boundaries accumulate,+later sessions spend effort on state repair instead of forward progress. A clean+exit policy therefore includes explicit recording of progress, removal of stale+artifacts, and confirmation that the standard startup path still works.++Operationally, clean state and measurement reinforce each other:++1. Cleanup reduces noise.+2. Lower noise improves benchmark signal quality.+3. Clear benchmarks identify where cleanup policy should be strengthened.++### Quality Documents++OpenAI's harness practice includes a quality document that grades each product+domain and architectural layer. This is not a one-time assessment — it is a+living artifact that tracks whether the codebase is getting stronger or weaker+over time.++A quality document typically grades domains on criteria like:++- Does verification pass for this area?+- Can an agent read and modify this area without confusion?+- Are tests stable, or do they flake?+- Are architecture boundaries enforced?++The grades serve multiple purposes:++1. **Session startup**: an agent reads the quality document to identify where+ the codebase is weakest and prioritize accordingly.+2. **Session exit**: the agent updates grades to reflect what changed.+3. **Benchmark**: comparing quality document snapshots before and after a+ harness change shows whether the change actually improved codebase health.+4. **Cleanup**: recurring maintenance sessions target domains with the lowest+ grades first.++A quality document template is available in the course resource library.++### Harness Simplification++Anthropic's harness design work makes an important meta-point: every harness+component exists because the model cannot do something reliably on its own. As+models improve, these assumptions become stale. A component that was load-bearing+three months ago may now be unnecessary overhead.++The recommended practice is to periodically remove one harness component at a+time and observe the impact:++1. Remove the component.+2. Run the benchmark task suite.+3. Compare results with and without the component.++If outcomes do not degrade, the component can stay removed. If they do, it+should be restored or replaced with a lighter alternative. This process prevents+harnesses from accumulating complexity that no longer serves a purpose.++The principle is: **find the simplest solution possible, and only increase+complexity when needed.** A harness that works today may be over-engineered+tomorrow.## Detailed Explanationdiff --git a/projects/project-05-grounded-qa-verification/README.md b/projects/project-05-grounded-qa-verification/README.mdindex e07ca35..748666d 100644--- a/projects/project-05-grounded-qa-verification/README.md+++ b/projects/project-05-grounded-qa-verification/README.md@@ -39,15 +39,49 @@ Run 2 and Run 3 use upgraded role separation:## Procedure+### Sprint Contract++Before any code is written, the generator and evaluator must negotiate a sprint+contract — a short written agreement specifying what will be built, how success+will be verified, and what is explicitly out of scope. The contract lives in the+repository as a file both roles can read.++This step matters because it front-loads alignment. Without a contract, the+evaluator's criteria are implicit and may shift during review. With a contract,+the evaluator grades against an explicit, pre-agreed standard.++Template: [`code/sprint-contract.md`](../../lectures/lecture-11-why-observability-belongs-inside-the-harness/code/sprint-contract.md)++### Evaluator Tuning++An out-of-the-box evaluator agent is a poor QA agent — it identifies issues+then talks itself into approving. The evaluator rubric requires iterative tuning:++1. Run the evaluator on a completed sprint.+2. Compare the evaluator's scores against your own human judgment.+3. Where they diverge, update the rubric to be more specific about what+ constitutes a pass or fail.+4. Re-run the evaluator on the same output and check alignment.+5. Repeat until the evaluator's judgment consistently matches human review.++Plan for 3-5 tuning rounds before the evaluator produces reliable scores. Record+each tuning iteration in the deliverables.++### Execution Steps+1. Select upgrade scope and define acceptance criteria before coding.-2. Start three branches from the same commit:+2. Write a sprint contract with the generator and evaluator agreeing on scope,+ verification, and exclusions.+3. Start three branches from the same commit:`p05-single`, `p05-gen-eval`, `p05-plan-gen-eval`.-3. Run each branch with Codex or Claude Code using comparable model and budget.-4. Use the same evaluator rubric for all three final outputs.-5. In improved runs, require at least one revision round after evaluator+4. Run each branch with Codex or Claude Code using comparable model and budget.+5. Use the same evaluator rubric for all three final outputs.+6. In improved runs, require at least one revision round after evaluatorfeedback.-6. Verify behavior with runnable evidence and record unresolved defects.-7. Compare quality and effort across the three harness variants.+7. Verify behavior with runnable evidence and record unresolved defects.+8. Tune the evaluator rubric across at least two iterations, recording what+ changed and why.+9. Compare quality and effort across the three harness variants.## What to Measure@@ -56,9 +90,14 @@ Run 2 and Run 3 use upgraded role separation:- Rubric score by category (correctness, reliability, maintainability, UX).- Rework required after review.- Final robustness of the upgraded product slice.+- Evaluator tuning iterations: how many rounds before evaluator judgment aligned+ with human review.+- Sprint contract effectiveness: did the contract reduce ambiguity in evaluation?## Deliverables+- Sprint contract document.+- Evaluator rubric tuning log (before/after for each tuning iteration).- Single-role run record (prompt, transcript/log, runnable evidence).- Generator-evaluator run record (including rubric and revision notes).- Planner-generator-evaluator run record (including rubric and revisions).diff --git a/projects/project-06-runtime-observability-and-debugging/README.md b/projects/project-06-runtime-observability-and-debugging/README.mdindex 64741d8..092f0b8 100644--- a/projects/project-06-runtime-observability-and-debugging/README.md+++ b/projects/project-06-runtime-observability-and-debugging/README.md@@ -39,20 +39,33 @@ Run 2 must use the strongest harness assembled in the course:- Explicit scope and verification gates.- Runtime signals and structural constraints.- Evaluator or multi-role review where applicable.+- Quality document tracking per-domain and per-layer grades.## Procedure-1. Define one fixed benchmark task suite and one scoring sheet before running+1. Create a quality document using the template in+ `resources/en/templates/quality-document.md` (or `resources/zh/templates/`).+ Fill in initial grades for all product domains and architectural layers based+ on the current codebase state.+2. Define one fixed benchmark task suite and one scoring sheet before runningany agent session.-2. Start from one common commit and branch into `p06-baseline` and+3. Start from one common commit and branch into `p06-baseline` and`p06-improved`.-3. Run baseline and improved sessions with Codex or Claude Code using the same+4. Run baseline and improved sessions with Codex or Claude Code using the samemodel and comparable budget.-4. Score both runs immediately after execution.-5. On the improved branch, perform a cleanup pass targeting entropy, dead code,+5. Score both runs immediately after execution.+6. Update the quality document after each run, recording grade changes per+ domain and layer.+7. On the improved branch, perform a cleanup pass targeting entropy, dead code,unclear docs, and unstable run paths.-6. Rerun the same benchmark task suite after cleanup and rescore.-7. Compare baseline vs improved vs improved-after-cleanup.+8. Rerun the same benchmark task suite after cleanup and rescore.+9. Update the quality document one final time.+10. Compare three quality document snapshots (baseline, improved, post-cleanup)+ alongside benchmark scores.+11. **Harness simplification pass:** Remove one harness component (for example,+ the sprint contract, or the explicit scope gate). Rerun the benchmark. If+ outcomes do not degrade, the component was unnecessary overhead. If they do,+ restore it. Record the result.## What to Measure@@ -61,10 +74,14 @@ Run 2 must use the strongest harness assembled in the course:- Defect count before human intervention.- Cleanup effort (time and files touched).- Post-cleanup legibility and restart success.+- Quality document grade changes across the three snapshots.+- Harness simplification result: which components were removable, which were load-bearing.## Deliverables+- Quality document with three snapshots (baseline, improved, post-cleanup).- Baseline benchmark record with scores and evidence.- Improved benchmark record with scores and evidence.- Cleanup run record with before/after score delta.+- Harness simplification log: component removed, benchmark result, decision.- Final capstone comparison summary with key lessons learned.diff --git a/resources/en/templates/README.md b/resources/en/templates/README.mdindex c81c82d..797fa3c 100644--- a/resources/en/templates/README.md+++ b/resources/en/templates/README.md@@ -174,3 +174,53 @@ A scorecard for reviewing agent output quality. Use this after a session or at p- Accept — meets the bar- Revise — needs fixes before accepting- Block — fundamental issues that need to be resolved first++**Important: the evaluator needs tuning.** Out of the box, agents are poor self-judges — they identify issues then talk themselves into approving. You will need to iterate:++1. Run the evaluator on a completed sprint.+2. Compare its scores against your own human judgment.+3. Where they diverge, make the rubric more specific about pass/fail criteria.+4. Re-run and check alignment.+5. Repeat until the evaluator consistently matches human review.++Plan for 3-5 tuning rounds. Record each change so you can track what improved alignment.++## quality-document.md++A quality snapshot that grades each product domain and architectural layer in your project. Tracks codebase health over time, not just individual session output.++**How to use it:**++- Copy to your project root+- Before starting a session: read it to understand where the codebase is weakest+- After a session: update grades based on what changed+- Over time: compare snapshots to see which harness changes actually improved codebase health++**What it grades:**++- **Product domains** (e.g., document import, Q&A flow, indexing): each domain gets a grade (A-D) across verification status, agent legibility, test stability, and key gaps+- **Architectural layers** (e.g., main process, preload, renderer, services): each layer gets a grade for boundary enforcement and agent legibility++**Why it matters:**++The evaluator rubric scores individual agent outputs. The quality document scores the codebase itself. They answer different questions:++- Evaluator rubric: "Did the agent do good work this session?"+- Quality document: "Is the project getting stronger or weaker over time?"++**When to update:**++- After each significant session+- Before benchmark comparisons+- After cleanup or simplification passes+- When onboarding a new agent or model to the project++**Harness simplification tie-in:**++The quality document also supports harness simplification. Every harness component encodes an assumption about what the model cannot do. As models improve, these assumptions go stale. To check whether a component is still needed:++1. Take a quality document snapshot.+2. Remove one harness component.+3. Run the benchmark task suite.+4. Take another snapshot.+5. Compare — if grades didn't drop, the component was overhead. If they did, restore it.diff --git a/resources/en/templates/quality-document.md b/resources/en/templates/quality-document.mdnew file mode 100644index 0000000..fb8699b--- /dev/null+++ b/resources/en/templates/quality-document.md@@ -0,0 +1,43 @@+# Quality Document++A quality snapshot for each product domain and architectural layer. Both agents and humans can use this document to quickly understand where the codebase is strong and where it needs work.++**Update cadence:** After each significant session, or before starting a new phase of work.++**Grading scale:**++- **A**: All verification passing, clean architecture, agent-legible, stable tests+- **B**: Verification passing, mostly clean, minor gaps in legibility or test coverage+- **C**: Partially working, known gaps, some code areas hard for agents to understand+- **D**: Not working, or major structural issues++---++## Product Domains++| Domain | Grade | Verification | Agent Legibility | Test Stability | Key Gaps | Last Updated |+|--------|-------|-------------|-----------------|---------------|----------|-------------|+| Document Import | - | - | - | - | - | - |+| Document Management | - | - | - | - | - | - |+| Document Indexing | - | - | - | - | - | - |+| Q&A Flow | - | - | - | - | - | - |+| Grounded Answers | - | - | - | - | - | - |++## Architectural Layers++| Layer | Grade | Boundary Enforcement | Agent Legibility | Key Gaps | Last Updated |+|-------|-------|---------------------|-----------------|----------|-------------|+| Main Process | - | - | - | - | - |+| Preload | - | - | - | - | - |+| Renderer | - | - | - | - | - |+| Services | - | - | - | - | - |++## Change History++### YYYY-MM-DD++- Changes:+- Domains promoted:+- Domoted:+- New gaps identified:+- Gaps closed:diff --git a/resources/zh/templates/README.md b/resources/zh/templates/README.mdindex 09bd7a6..9395970 100644--- a/resources/zh/templates/README.md+++ b/resources/zh/templates/README.md@@ -174,3 +174,53 @@ agent 任何时候只能有一个功能处于 `in_progress`。- Accept — 达标- Revise — 需要修补才能接受- Block — 有根本性问题,需要先解决++**Evaluator 需要校准。** 开箱即用的 agent 做评审很弱——它会发现问题,然后把自己说服到通过。你需要反复校准:++1. 用 evaluator 给一个已完成的 sprint 打分。+2. 把它的分数和你自己的人工判断对比。+3. 有分歧的地方,把 rubric 里的通过/失败标准写得更具体。+4. 对同一个输出重新跑 evaluator,看对齐了没有。+5. 重复直到 evaluator 的判断和人工评审基本一致。++预计需要 3-5 轮校准。每轮记录改了什么、为什么改。++## quality-document.md++质量快照。给项目的每个产品领域和架构层打分,跟踪代码库随时间是变强了还是变弱了。++**怎么用:**++- 复制到项目根目录+- 开始会话前:读它,了解代码库哪里最弱+- 会话结束后:更新评级+- 长期:对比不同时间点的快照,看哪些 harness 改动真正改善了代码库健康度++**评什么:**++- **产品领域**(如文档导入、问答流程、索引):每个领域按验证状态、Agent 可读性、测试稳定性、关键缺口打分+- **架构层**(如 Main Process、Preload、Renderer、Services):每层按边界执行和 Agent 可读性打分++**为什么需要它:**++Evaluator rubric 评的是单次 agent 输出的质量。Quality document 评的是代码库本身的质量。它们回答的是不同的问题:++- Evaluator rubric:"这轮 agent 做得好不好?"+- Quality document:"这个项目是在变强还是变弱?"++**什么时候更新:**++- 每轮重要会话之后+- 做基准对比之前+- 做清理或简化之后+- 换新 agent 或新模型时++**和 harness 简化的关系:**++Quality document 也可以用来验证 harness 是否可以简化。Harness 里的每个组件都编码了一个假设——"模型做不到这件事"。模型变强后,这些假设就过时了。检查某个组件是否还有必要:++1. 拍一份 quality document 快照。+2. 移除一个 harness 组件。+3. 跑基准测试。+4. 再拍一份快照。+5. 对比——评级没降,说明那个组件是多余的。降了,就恢复。diff --git a/resources/zh/templates/quality-document.md b/resources/zh/templates/quality-document.mdnew file mode 100644index 0000000..a21367b--- /dev/null+++ b/resources/zh/templates/quality-document.md@@ -0,0 +1,43 @@+# 质量文档++每个产品领域和架构层的质量快照。agent 和人都能通过这份文档快速了解项目哪里强、哪里弱。++**更新时机:** 每轮重要会话结束后,或开始新一阶段工作前。++**评级标准:**++- **A**:验证全部通过,架构干净,agent 能读懂,测试稳定+- **B**:验证通过,基本干净,可读性或测试覆盖有少量缺口+- **C**:部分可用,有已知缺口,部分代码 agent 不容易理解+- **D**:不可用,或存在重大结构问题++---++## 产品领域++| 领域 | 评级 | 验证状态 | Agent 可读性 | 测试稳定性 | 关键缺口 | 上次更新 |+|------|------|---------|-------------|-----------|---------|---------|+| 文档导入 | - | - | - | - | - | - |+| 文档管理 | - | - | - | - | - | - |+| 文档索引 | - | - | - | - | - | - |+| 问答流程 | - | - | - | - | - | - |+| 引用式回答 | - | - | - | - | - | - |++## 架构层++| 层级 | 评级 | 边界执行 | Agent 可读性 | 关键缺口 | 上次更新 |+|------|------|---------|-------------|---------|---------|+| Main Process | - | - | - | - | - |+| Preload | - | - | - | - | - |+| Renderer | - | - | - | - | - |+| Services | - | - | - | - | - |++## 变更历史++### YYYY-MM-DD++- 变更内容:+- 提升:+- 下降:+- 新发现的缺口:+- 已关闭的缺口: