Shell
Add Knowledge Base: Pattern library for compound learning
Eddie Belaval7581da250756
diff --git a/knowledge/INDEX.md b/knowledge/INDEX.mdnew file mode 100644index 0000000..5f55a05--- /dev/null+++ b/knowledge/INDEX.md@@ -0,0 +1,239 @@+# Claude Code Knowledge Base++> Central repository for patterns, workflows, and techniques that compound over time.+> **Philosophy:** Build reusable patterns once, benefit across all projects and model upgrades.++## π Patterns++### Claude Code Mastery+- **[Longform Guide](patterns/claude-code/longform-guide.md)** β CORE+ - Token optimization strategies+ - Memory persistence patterns+ - Verification loops and evals+ - Parallelization philosophy+ - Sub-agent orchestration+ - **Status:** Active - In daily use++### Agentic Architectures+- **[17 Architecture Implementations](../learning/all-agentic-architectures/)** π External Repo+ - All architectures as runnable Jupyter notebooks+ - Ordered learning path from foundational to advanced+ - LangChain + LangGraph implementations++**Quick Reference:**+| # | Architecture | Use Case |+|---|--------------|----------|+| 01 | Reflection | High-quality code generation |+| 02 | Tool Use | Real-time research, enterprise bots |+| 03 | ReAct | Multi-hop Q&A, web research |+| 04 | Planning | Predictable workflows |+| 05 | Multi-Agent | Software dev pipelines |+| 06 | PEV | High-stakes automation, finance |+| 07 | Blackboard | Complex diagnostics |+| 08 | Episodic + Semantic | Long-term personalization |+| 09 | Tree of Thoughts | Logic puzzles, constrained planning |+| 10 | Mental Loop | Robotics, safety-critical systems |+| 11 | Meta-Controller | Multi-service platforms |+| 12 | Graph Memory | Corporate intelligence |+| 13 | Ensemble | High-stakes decision support |+| 14 | Dry-Run | Production deployment, debugging |+| 15 | RLHF | High-quality content, continual learning |+| 16 | Cellular Automata | Spatial reasoning, logistics |+| 17 | Reflexive Metacognitive | Medical, legal, finance advisory |++### ID8Pipeline+- **[Build Laws](~/.claude/CLAUDE.md#id8pipeline)** π Global Config+ - 11 stages from Concept Lock to Listen & Iterate+ - Agent-Native additions for all stages+ - Approval flow matrix+ - Pipeline status tracking++---++## π Workflows++### Memory & Context+- **Session Storage Pattern:** `~/.claude/sessions/YYYY-MM-DD-topic.tmp`+- **Strategic Compacting:** Phase-based context clearing+- **Dynamic System Prompts:** Context injection via CLI flags+- **Memory Persistence Hooks:** PreCompact β SessionStart β Stop++### Continuous Learning+- **Stop Hook Evaluation:** Extract patterns at session end+- **Manual /learn Command:** Mid-session knowledge capture+- **Learned Skills Storage:** `~/.claude/skills/learned/`++### Verification Patterns+- **Checkpoint-Based:** Linear workflows with explicit milestones+- **Continuous:** Long sessions, exploratory refactoring+- **Metrics:** pass@k (any success) vs pass^k (all succeed)++---++## π οΈ Tools++### MCP β Skills Conversion+**When to convert:**+- Heavy repeated operations+- Token cost concerns+- MCP adds minimal convenience++**Pattern:**+```bash+# Instead of GitHub MCP+/gh-pr # Wraps: gh pr create --fill++# Instead of Supabase MCP+/sb-status # Wraps: supabase status+/sb-diff # Wraps: supabase db diff+```++### CLI Wrappers+- GitHub: `gh` CLI+- Supabase: `supabase` CLI+- Vercel: `vercel` CLI+- Railway: `railway` CLI++**Benefits:**+- Same functionality, lower token cost+- Skills act as pseudo-MCPs+- No context window consumption++---++## π Token Economics++### Model Selection Matrix+| Task | Model | Cost vs Opus | When to Use |+|------|-------|--------------|-------------|+| Repetitive/clear | Haiku | 5x cheaper | Worker tasks, simple ops |+| Default coding | Sonnet | 1.67x cheaper | 90% of tasks |+| Complex/critical | Opus | Baseline | 5+ files, architecture, security |++### Optimization Strategies+1. **Modular codebase:** Hundreds of lines per file, not thousands+2. **Tool replacement:** mgrep instead of grep (~50% token reduction)+3. **Background processes:** tmux + summarization for long outputs+4. **Continuous cleanup:** Dead code removal, refactoring passes++---++## π― Best Practices++### Agent Abstraction Tierlist++**Tier 1 (Start Here):**+- β Subagents - Context rot prevention+- β Metaprompting - Stability and assumption validation+- β Ask user more at beginning - Clarify requirements early++**Tier 2 (Advanced):**+- β οΈ Long-running agents - Requires trial-and-error tuning+- β οΈ Parallel multi-agent - High variance, only for segmented tasks+- β οΈ Role-based multi-agent - Model evolution makes heuristics fragile+- β οΈ Computer use agents - Very early paradigm++### Sub-Agent Orchestration++**The Context Problem:**+- Orchestrator has PURPOSE + context+- Sub-agent only has QUERY+- Summaries miss key details++**Solution: Iterative Retrieval**+1. Pass objective + query (not just query)+2. Evaluate every sub-agent return+3. Ask follow-up questions+4. Loop max 3 cycles+5. Accept when sufficient++**Sequential Phases Pattern:**+```+Research β Plan β Implement β Review β Verify+ βββΊ File outputs between each phase+```++### Parallelization Philosophy++**Goal:** Maximum productivity with minimum viable parallelization++**Recommended:**+- Start with 1 instance, master workflow+- Scale to 2-3 for different concerns (code vs research)+- Rarely need more than 4+- Use git worktrees for overlapping code+- Use /rename for clarity++**Avoid:**+- Arbitrary terminal counts ("always 5 instances")+- Over-parallelization before mastery+- Mental overhead > productivity gains++---++## ποΈ Groundwork++### Two-Instance Kickoff+1. **Instance 1:** Scaffolding (structure, configs, conventions)+2. **Instance 2:** Research (PRD, architecture, docs compilation)++### llms.txt Pattern+```+https://docs.example.com/llms.txt+```+LLM-optimized documentation, feed directly to Claude++### Build Reusable Patterns+**Invest in:**+- Subagents (see shorthand guide)+- Skills (see shorthand guide)+- Commands (see shorthand guide)+- Planning patterns+- MCP tools+- Context engineering++**Why:** Patterns transfer across model upgrades and other agents. Compound effect over time.++---++## π References++### Primary Sources+- [Longform Guide](patterns/claude-code/longform-guide.md) - Full detailed guide+- [Global CLAUDE.md](~/.claude/CLAUDE.md) - Active instructions and patterns+- [17 Agentic Architectures](../learning/all-agentic-architectures/) - Implementation notebooks++### External Resources+- Anthropic: "Demystifying evals for AI agents"+- YK: "32 Claude Code Tips"+- @omarsar0: Compound Effects Philosophy+- @menhguin: Agent Abstractions Tierlist+- @PerceptualPeak: Sub-Agent Context Negotiation++---++## π Maintenance++### Adding New Patterns+1. Save pattern to appropriate directory in `patterns/`+2. Update this INDEX.md with link and summary+3. Tag status: Active | Experimental | Deprecated+4. Add to relevant workflow section if applicable++### Session Logs+- Location: `~/.claude/sessions/`+- Format: `YYYY-MM-DD-topic.tmp`+- Retention: Backup meaningful sessions, prune unnecessary+- Structure: Current state, completed, blockers, decisions, next session++### Learned Skills+- Location: `~/.claude/skills/learned/`+- Auto-generated via Stop hook+- Manual via `/learn` command+- Review quarterly for obsolete patterns++---++**Last Updated:** 2026-01-21+**Version:** 1.0.0+**Maintainer:** Eddie Belaval (@eddiebelaval)diff --git a/knowledge/README.md b/knowledge/README.mdnew file mode 100644index 0000000..20c0495--- /dev/null+++ b/knowledge/README.md@@ -0,0 +1,90 @@+# ~/.claude/knowledge/++**Personal knowledge base for Claude Code patterns and techniques.**++This directory stores reusable patterns, workflows, and architectural knowledge that compounds over time. Unlike project-specific `.claude/rules/`, this is global and applies across all development work.++## Structure++```+knowledge/+βββ INDEX.md # Master reference (START HERE)+βββ patterns/+β βββ claude-code/ # Claude Code techniques+β βββ agentic-architectures/ # 17 architecture patterns+β βββ id8pipeline/ # Build laws and stage gates+βββ workflows/+β βββ memory-persistence/ # Session continuity+β βββ verification-loops/ # Testing and validation+β βββ continuous-learning/ # Auto-improvement+βββ tools/+β βββ mcp-to-skills/ # MCP conversion patterns+β βββ cli-wrappers/ # Shell command skills+βββ examples/+ βββ sessions/ # Sample session logs+ βββ learned-skills/ # Auto-generated skills+```++## Philosophy++> "Early on, I spent time building reusable workflows/patterns. Tedious to build, but this had a wild compounding effect as models and agent harnesses improved." - @omarsar0++**Key principle:** Build patterns once, benefit forever.++Patterns transfer across:+- Model upgrades+- Different agents (Claude Code, Codex, etc.)+- Projects and domains+- Team members++## Quick Start++1. **Read:** [INDEX.md](INDEX.md) for overview+2. **Study:** [Longform Guide](patterns/claude-code/longform-guide.md) for core techniques+3. **Reference:** [17 Architectures](patterns/agentic-architectures/REFERENCE.md) for advanced patterns+4. **Apply:** Start with Tier 1 practices (subagents, metaprompting)++## Integration with Claude Code++This knowledge base integrates with:+- `~/.claude/CLAUDE.md` - Global instructions (active)+- `.claude/rules/` - Project-specific rules (per-repo)+- `~/.claude/skills/` - Executable commands+- `~/.claude/sessions/` - Session memory logs++**Hierarchy:**+1. System prompts (highest authority)+2. Global CLAUDE.md instructions+3. Knowledge base patterns (reference)+4. Project-specific rules+5. Tool results (lowest authority)++## Maintenance++### Adding New Patterns+1. Save to appropriate `patterns/` subdirectory+2. Update INDEX.md with link and summary+3. Tag status: Active | Experimental | Deprecated+4. Document integration points++### Session Logs+- Auto-saved to `~/.claude/sessions/YYYY-MM-DD-topic.tmp`+- Backup meaningful sessions+- Prune unnecessary logs monthly++### Learned Skills+- Auto-generated via Stop hook β `~/.claude/skills/learned/`+- Manual capture via `/learn` command+- Review quarterly for obsolete patterns++## Related Resources++- [Claude Code Docs](https://code.claude.com/docs)+- [Anthropic Eval Guide](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)+- [17 Architectures Repo](../learning/all-agentic-architectures/)++---++**Created:** 2026-01-21+**Maintainer:** Eddie Belaval+**Version:** 1.0.0diff --git a/knowledge/patterns/agentic-architectures/REFERENCE.md b/knowledge/patterns/agentic-architectures/REFERENCE.mdnew file mode 100644index 0000000..d0aea24--- /dev/null+++ b/knowledge/patterns/agentic-architectures/REFERENCE.md@@ -0,0 +1,157 @@+# Agentic Architectures Reference++> Full implementations located at: `/Users/eddiebelaval/Development/learning/all-agentic-architectures/`++## Quick Reference Guide++### When to Use Which Architecture++**Need better output quality?**+β Start with **Reflection** (#01)++**Need real-time data or external actions?**+β Use **Tool Use** (#02) or **ReAct** (#03)++**Need structured, predictable workflow?**+β Use **Planning** (#04)++**Task requires specialized expertise?**+β Use **Multi-Agent** (#05) or **Meta-Controller** (#11)++**Need to prevent errors in production?**+β Use **PEV** (#06) or **Dry-Run Harness** (#14)++**Need long-term memory?**+β Use **Episodic + Semantic** (#08) or **Graph Memory** (#12)++**Need to explore multiple solution paths?**+β Use **Tree of Thoughts** (#09)++**Need to test before acting?**+β Use **Mental Loop** (#10)++**Need robust, less biased conclusions?**+β Use **Ensemble** (#13)++**Need continuous improvement?**+β Use **RLHF/Self-Improvement** (#15)++**Need complex emergent behavior?**+β Use **Cellular Automata** (#16)++**Need self-aware safety?**+β Use **Reflexive Metacognitive** (#17)++---++## Architecture Categories++### Part 1: Foundational Patterns+Single-agent enhancements for power and structure.++1. **Reflection** - Self-critique and refinement+2. **Tool Use** - External API interaction+3. **ReAct** - Adaptive reasoning + action loop+4. **Planning** - Proactive task decomposition++### Part 2: Multi-Agent Collaboration+Specialized teams working together.++5. **Multi-Agent Systems** - Team collaboration+7. **Blackboard Systems** - Shared memory workspace+11. **Meta-Controller** - Smart task routing+13. **Ensemble** - Parallel diverse analysis++### Part 3: Advanced Memory & Reasoning+Deep thinking and long-term memory.++8. **Episodic + Semantic Memory** - Human-like memory+9. **Tree of Thoughts** - Multi-path exploration+12. **Graph World-Model** - Interconnected knowledge++### Part 4: Safety & Reliability+Production-ready, trustworthy systems.++6. **PEV (Plan-Execute-Verify)** - Self-correcting loops+10. **Mental Loop (Simulator)** - Test before acting+14. **Dry-Run Harness** - Human-in-the-loop safety+17. **Reflexive Metacognitive** - Self-aware limitations++### Part 5: Learning & Adaptation+Improvement and emergence over time.++15. **RLHF (Self-Improvement)** - Learn from feedback+16. **Cellular Automata** - Emergent complexity++---++## Integration with ID8Pipeline++### Stage Mapping++**Stage 2 (Scope Fence):**+- Use **Planning** (#04) to decompose V1 features+- Use **Meta-Controller** (#11) to route agent capabilities++**Stage 3 (Architecture Sketch):**+- Use **Graph Memory** (#12) for entity relationships+- Use **Multi-Agent** (#05) for component design++**Stage 5 (Feature Blocks):**+- Use **ReAct** (#03) for feature implementation+- Use **Reflection** (#01) for code quality+- Use **Tool Use** (#02) for external integrations++**Stage 7 (Test Coverage):**+- Use **PEV** (#06) for verification loops+- Use **Dry-Run Harness** (#14) for test safety++**Stage 8 (Polish & Harden):**+- Use **Ensemble** (#13) for edge case analysis+- Use **Mental Loop** (#10) for consequence testing++**Stage 11 (Listen & Iterate):**+- Use **RLHF** (#15) for continuous improvement+- Use **Episodic + Semantic** (#08) for user memory++---++## Implementation Notes++### Technology Stack+- **Python 3.10+**+- **LangChain** - LLM building blocks+- **LangGraph** - Stateful agent orchestration+- **Jupyter Notebooks** - Interactive demonstrations+- **Pydantic** - Structured data modeling+- **Tavily Search** - Research tool+- **Neo4j** - Graph database for memory+- **FAISS** - Vector store for similarity search++### Setup Requirements+1. Clone repo to `~/Development/learning/all-agentic-architectures/`+2. Set up virtual environment+3. Install dependencies: `pip install -r requirements.txt`+4. Configure `.env` with API keys (Nebius, LangSmith, Tavily, Neo4j)+5. Run notebooks in order: `jupyter notebook`++### Evaluation Pattern+Most notebooks feature **LLM-as-a-Judge** for quantitative feedback:+- Objective performance metrics+- Rubric-based scoring+- Critical for production AI systems++---++## Learning Path++**Beginners:** Start with 01 β 04 (Foundational)+**Intermediate:** Move to 05, 07, 11 (Multi-Agent)+**Advanced:** Explore 08, 09, 12 (Memory & Reasoning)+**Production:** Master 06, 10, 14, 17 (Safety & Reliability)+**Optimization:** Study 15, 16 (Learning & Emergence)++---++**Repository:** https://github.com/your-username/all-agentic-architectures+**Last Reviewed:** 2026-01-21diff --git a/knowledge/patterns/claude-code/longform-guide.md b/knowledge/patterns/claude-code/longform-guide.mdnew file mode 100644index 0000000..43db389--- /dev/null+++ b/knowledge/patterns/claude-code/longform-guide.md@@ -0,0 +1,754 @@+# The Longform Guide to Everything Claude Code++> **Source:** cogsec (@affaanmustafa) - Jan 2026+> **Status:** Active - Core operational patterns+> **Last Updated:** 2026-01-21++This guide covers advanced Claude Code techniques for token economics, memory persistence, verification patterns, parallelization strategies, and compound effects of building reusable workflows.++## Table of Contents++- [Context & Memory Management](#context--memory-management)+- [Token Optimization](#token-optimization)+- [Verification Loops and Evals](#verification-loops-and-evals)+- [Parallelization](#parallelization)+- [Groundwork](#groundwork)+- [Best Practices for Agents & Sub-Agents](#best-practices-for-agents--sub-agents)+- [Tips and Tricks](#tips-and-tricks)++---++## Context & Memory Management++### Session Storage Pattern+- Store session logs in `~/.claude/sessions/YYYY-MM-DD-topic.tmp`+- One file per session with: current state, completed items, blockers, key decisions+- At session end, summarize and checkpoint progress+- Next session: load previous file for continuity++**Example structure:**+```markdown+# Session: 2026-01-21 - API Refactor++## Current State+- Moved from REST to GraphQL for orders endpoint+- 3/5 entities migrated++## Completed+- [x] User entity migration+- [x] Product entity migration+- [x] Order entity migration++## Blockers+- Need schema validation strategy+- Auth middleware needs GraphQL context++## Key Decisions+- Using DataLoader for N+1 prevention+- Separate schema files per domain module++## Next Session+- Migrate Payment and Shipping entities+- Implement field-level auth directives+```++### Clearing Context Strategically++**Default (Auto-Compact):** Happens at arbitrary points, often mid-task+**Strategic (Manual):** Compact at logical phase transitions++**Strategic Compact Hook (PreToolUse):**+```bash+#!/bin/bash+# ~/.claude/hooks/strategic-compact.sh+COUNTER_FILE="/tmp/claude-tool-count-$$"+THRESHOLD=${COMPACT_THRESHOLD:-50}++if [ -f "$COUNTER_FILE" ]; then+ count=$(cat "$COUNTER_FILE")+ count=$((count + 1))+ echo "$count" > "$COUNTER_FILE"+else+ echo "1" > "$COUNTER_FILE"+ count=1+fi++if [ "$count" -eq "$THRESHOLD" ]; then+ echo "[StrategicCompact] $THRESHOLD tool calls reached - consider /compact if transitioning phases" >&2+fi+```++**When to compact:**+- After exploration, before execution+- After completing a milestone, before starting next+- When transitioning project phases++### Dynamic System Prompt Injection++**Basic pattern:**+```bash+claude --system-prompt "$(cat memory.md)"+```++**Why vs @ file references:**+- `@memory.md` = Read tool output (lower authority)+- `--system-prompt` = Injected before conversation (higher authority)++**Hierarchy:** System Prompt > User Messages > Tool Results++**Practical aliases:**+```bash+# Daily development+alias claude-dev='claude --system-prompt "$(cat ~/.claude/contexts/dev.md)"'++# PR review mode+alias claude-review='claude --system-prompt "$(cat ~/.claude/contexts/review.md)"'++# Research/exploration mode+alias claude-research='claude --system-prompt "$(cat ~/.claude/contexts/research.md)"'+```++### Memory Persistence Hooks++**Hook lifecycle:**+```+SESSION 1 SESSION 2+βββββββββ βββββββββ++[Start] [Start]+ β β+ βΌ βΌ+ββββββββββββββββ ββββββββββββββββ+β SessionStart β ββββ reads βββββββ β SessionStart ββββ loads previous+β Hook β nothing yet β Hook β context+ββββββββ¬ββββββββ ββββββββ¬ββββββββ+ β β+ βΌ βΌ+ [Working] [Working]+ β (informed)+ βΌ β+ββββββββββββββββ βΌ+β PreCompact ββββΊ saves state [Continue...]+β Hook β before summary+ββββββββ¬ββββββββ+ β+ βΌ+ [Compacted]+ β+ βΌ+ββββββββββββββββ+β Stop Hook ββββΊ persists to βββββββββββΊ+β (session-end)β ~/.claude/sessions/+ββββββββββββββββ+```++**Hook configuration:**+```json+{+ "hooks": {+ "PreCompact": [{+ "matcher": "*",+ "hooks": [{+ "type": "command",+ "command": "~/.claude/hooks/memory-persistence/pre-compact.sh"+ }]+ }],+ "SessionStart": [{+ "matcher": "*",+ "hooks": [{+ "type": "command",+ "command": "~/.claude/hooks/memory-persistence/session-start.sh"+ }]+ }],+ "Stop": [{+ "matcher": "*",+ "hooks": [{+ "type": "command",+ "command": "~/.claude/hooks/memory-persistence/session-end.sh"+ }]+ }]+ }+}+```++### Continuous Learning Pattern++**Problem:** Repeated prompts for same mistakes, wasted tokens/context/time++**Solution:** Automatically save discovered patterns as skills++**Why Stop hook vs UserPromptSubmit:**+- UserPromptSubmit runs on EVERY message (high overhead, latency)+- Stop runs once at session end (lightweight, evaluates complete session)++**Pattern:**+1. Stop hook triggers at session end+2. Script analyzes session for extractable knowledge+3. Saves patterns as skills in `~/.claude/skills/learned/`+4. Next session: skills load automatically via semantic matching++**Manual extraction with /learn:**+- Run mid-session when solving something non-trivial+- Prompts for pattern extraction, drafts skill file+- Confirmation before saving++**Other Self-Improving Patterns:**+- **Reflection over logs:** Build "diary" of what works/fails (RLanceMartin)+- **Proactive suggestions:** Agent suggests improvements every 15 min (alexhillman)++---++## Token Optimization++### Primary Strategy: Subagent Architecture++**Model Selection Quick Reference:**++| Task Type | Model | Why |+|-----------|-------|-----|+| Repetitive, clear instructions | Haiku | 5x cheaper than Opus, sufficient |+| Default coding tasks | Sonnet | 66.7% cheaper than Opus, minimal quality loss |+| 5+ file changes, architecture, security-critical | Opus | Complex reasoning needed |+| "Worker" in multi-agent setup | Haiku | Saves cost on simple subtasks |++**Cost comparison:**+- Opus: $5 input / $25 output per M tokens+- Sonnet: $3 input / $15 output per M tokens (1.67x savings)+- Haiku: ~$1 input / ~$5 output per M tokens (5x savings vs Opus)++**Agent definition with model:**+```yaml+---+name: quick-search+description: Fast file search+tools: Glob, Grep+model: haiku # Cheap and fast+---+```++### Tool-Specific Optimizations++**Replace grep with mgrep:**+- Effective token reduction: ~50% on average vs grep/ripgrep+- Source: https://github.com/mixedbread-ai/mgrep++**Background processes:**+- Run outside Claude if full streaming output not needed+- Use tmux for session management+- Summarize or copy only relevant parts+- Saves input tokens (where majority of cost comes from)++### Modular Codebase Benefits++**Problem:** Long files (1000+ lines) require:+- Multiple Read tool calls to finish+- Claude loses information during reads+- Stopping and rereading costs extra tokens++**Solution:** Modular architecture+- Main files: hundreds of lines, not thousands+- Reusable utilities, functions, hooks+- Self-contained modules++**Example structure:**+```+root/+βββ docs/+βββ scripts/+βββ src/+β βββ apps/ # Entry points+β β βββ api-gateway/+β β βββ cron-jobs/+β βββ modules/ # Core business logic+β β βββ ordering/ # Self-contained+β β β βββ api/+β β β βββ domain/+β β β βββ infrastructure/+β β β βββ use-cases/+β β β βββ tests/+β β βββ catalog/+β β βββ identity/+β βββ shared/ # Cross-module utilities+β β βββ kernel/+β β βββ events/+β β βββ utils/+β βββ main.ts+βββ tests/+```++**Benefits:**+- Cheaper token cost (less to read)+- Higher first-try success rate+- Easier for Claude to understand context++### Lean Codebase = Cheaper Tokens++**Continuous cleanup strategies:**+- Use skills to identify dead code+- Regular refactoring passes+- Manual codebase skimming for patterns+- Feed context + refactor/dead-code skills to Claude++### System Prompt Slimming (Advanced)++**Current overhead:** ~18k tokens (~9% of 200k context)+**With patches:** ~10k tokens (saves ~7,300 tokens = 41% of static overhead)++**Note:** Most users don't need this optimization+- Source: YK's system-prompt-patches++---++## Verification Loops and Evals++### Observability Methods++**Option 1: tmux tracing**+- Hook tmux processes to trace thinking stream+- Log output whenever skill triggers++**Option 2: PostToolUse hook**+- Logs exact changes and outputs+- Compare with/without skill benchmarks++**Benchmarking workflow:**+```+ [Same Task]+ β+ ββββββββββββββ΄βββββββββββββ+ βΌ βΌ+ βββββββββββββββββ βββββββββββββββββ+ β Worktree A β β Worktree B β+ β WITH skill β β WITHOUT skill β+ βββββββββ¬ββββββββ βββββββββ¬ββββββββ+ β β+ βΌ βΌ+ [Output A] [Output B]+ β β+ ββββββββββββ¬βββββββββββββββ+ βΌ+ [git diff]+ β+ βΌ+ ββββββββββββββββββ+ β Compare logs, β+ β token usage, β+ β output quality β+ ββββββββββββββββββ+```++### Eval Pattern Types++**CHECKPOINT-BASED vs CONTINUOUS:**++```+CHECKPOINT-BASED CONTINUOUS+βββββββββββββββββ ββββββββββ++ [Task 1] [Work]+ β β+ βΌ βΌ+ βββββββββββ βββββββββββ+ βCheckpointββββ verify β Timer/ β+ β #1 β criteria β Change β+ ββββββ¬βββββ ββββββ¬βββββ+ β pass? β+ βββββ΄ββββ βΌ+ β β ββββββββββββ+ yes no βββΊ fix βββ βRun Tests β+ β β β β + Lint β+ βΌ ββββββ ββββββ¬ββββββ+ [Task 2] β+ β ββββββ΄βββββ+ βΌ β β+ βββββββββββ pass fail+ βCheckpointβ β β+ β #2 β βΌ βΌ+ ββββββ¬βββββ [Continue] [Stop & Fix]+ β β+ ... ββββββ++Best for: Linear workflows Best for: Long sessions+with clear milestones exploratory refactoring+```++**Checkpoint-Based Evals:**+- Set explicit checkpoints in workflow+- Verify against defined criteria at each+- Fix before proceeding if verification fails+- Good for: Linear workflows with clear milestones++**Continuous Evals:**+- Run every N minutes or after major changes+- Full test suite, build status, lint+- Report regressions immediately+- Stop and fix before continuing+- Good for: Long-running sessions, exploratory work++### Grader Types (from Anthropic)++**Code-Based Graders:**+- String match, binary tests, static analysis+- Fast, cheap, objective+- Brittle to valid variations++**Model-Based Graders:**+- Rubric scoring, natural language assertions+- Flexible, handles nuance+- Non-deterministic, more expensive++**Human Graders:**+- SME review, crowdsourced judgment+- Gold standard quality+- Expensive and slow++### Key Metrics++**pass@k:** At least ONE of k attempts succeeds+```+k=1: 70% k=3: 91% k=5: 97%+Higher k = higher odds of success+```++**pass^k:** ALL k attempts must succeed+```+k=1: 70% k=3: 34% k=5: 17%+Higher k = harder (consistency)+```++**When to use:**+- `pass@k`: Just need it to work, any success counts+- `pass^k`: Consistency essential, deterministic output quality++### Building an Eval Roadmap++1. **Start early:** 20-50 simple tasks from real failures+2. **Convert failures:** User-reported issues β test cases+3. **Write unambiguous tasks:** Two experts reach same verdict+4. **Build balanced sets:** Test when behavior should AND shouldn't occur+5. **Build robust harness:** Each trial starts from clean environment+6. **Grade outcomes:** What agent produced, not the path it took+7. **Read transcripts:** Review many trials for patterns+8. **Monitor saturation:** 100% pass rate means add more tests++---++## Parallelization++### Philosophy++**Goal:** Maximum productivity with minimum viable parallelization++**Avoid:**+- Arbitrary terminal counts (e.g., "always run 5 instances")+- Over-parallelization before mastering single instance+- Overlapping code changes without git worktrees++**Preferred Pattern:**+- Main chat: Code changes+- Forks: Questions, research, documentation scraping, external service queries++### For Newcomers++**Start with:** Single instance, master workflow first++**Scale when:**+- True necessity and purpose exist+- Can't achieve task with script or tmux stream+- Clear separation of concerns between instances++**Most work:** Usually 2-3 instances, rarely need more than 4++### When Scaling Instances++**If working on overlapping code:**++1. **Use git worktrees:**+```bash+git worktree add ../project-feature-a feature-a+git worktree add ../project-feature-b feature-b+git worktree add ../project-refactor refactor-branch++# Each worktree gets its own Claude instance+cd ../project-feature-a && claude+```++**Benefits:**+- No git conflicts between instances+- Each has clean working directory+- Easy to compare outputs+- Can benchmark same task with different approaches++2. **Use /rename for clarity:**+```+/rename "Feature A: Auth Refactor"+/rename "Feature B: API Migration"+/rename "Refactor: Module Consolidation"+```++### The Cascade Method++**Organization pattern:**+1. Open new tasks in new tabs to the right+2. Sweep left to right, oldest to newest+3. Maintain consistent direction flow+4. Check on specific tasks as needed+5. Focus on at most 3-4 tasks simultaneously++**Mental overhead increases faster than productivity beyond 3-4 tasks**++---++## Groundwork++### The Two-Instance Kickoff Pattern++**For new repos, start with 2 Claude instances:**++**Instance 1: Scaffolding Agent**+- Lay down scaffold and groundwork+- Create project structure+- Set up configs (CLAUDE.md, rules, agents)+- Establish conventions+- Get skeleton in place++**Instance 2: Deep Research Agent**+- Connect to services, web search, etc.+- Create detailed PRD+- Create architecture mermaid diagrams+- Compile references with actual documentation clips++**Workflow:**+- Left terminal: Coding+- Right terminal: Questions+- Use `/rename` and `/fork` to organize++### Starting Setup Philosophy++**Start minimal:**+- Don't over-engineer from the start+- Context7 / link scraping when already knee-deep+- Build reusable patterns as you go++### llms.txt Pattern++**Many docs now provide LLM-optimized versions:**+```+https://www.helius.dev/docs/llms.txt+```++**Usage:**+- Navigate to docs page+- Try `/llms.txt` at end of URL+- Feed clean, optimized docs directly to Claude++### Philosophy: Build Reusable Patterns++**Insight from @omarsar0:**+> "Early on, I spent time building reusable workflows/patterns. Tedious to build, but this had a wild compounding effect as models and agent harnesses improved."++**Invest in:**+- Subagents+- Skills+- Commands+- Planning patterns+- MCP tools+- Context engineering patterns++**Why it compounds:**+- Patterns transfer across model upgrades+- Work with other agents (e.g., Codex)+- Investment in patterns > investment in model tricks++---++## Best Practices for Agents & Sub-Agents++### The Sub-Agent Context Problem++**Issue:** Sub-agents lack semantic context++**Analogy (@PerceptualPeak):**+> "Your boss sends you to a meeting and asks for a summary. You come back and give him the rundown. Nine times out of ten, he's going to have follow-up questions. Your summary won't include everything he needs because you don't have the implicit context he has."++**Root cause:**+- Orchestrator has PURPOSE/REASONING behind request+- Sub-agent only knows literal query+- Summaries miss key details++### Iterative Retrieval Pattern++```+βββββββββββββββββββ+β ORCHESTRATOR β+β (has context) β+ββββββββββ¬βββββββββ+ β dispatch with query + objective+ βΌ+βββββββββββββββββββ+β SUB-AGENT β+β (lacks context) β+ββββββββββ¬βββββββββ+ β returns summary+ βΌ+βββββββββββββββββββ βββββββββββββββ+β EVALUATE ββnoβββΊβ FOLLOW-UP β+β Sufficient? β β QUESTIONS β+ββββββββββ¬βββββββββ ββββββββ¬βββββββ+ β yes β+ βΌ β sub-agent+ [ACCEPT] fetches answers+ β+ ββββββββββββββββββββββββ+ (max 3 cycles)+```++**Fix:**+1. Orchestrator evaluates every sub-agent return+2. Asks follow-up questions before accepting+3. Sub-agent goes back to source, gets answers+4. Loop until sufficient (max 3 cycles to prevent infinite loops)+5. **Pass objective context, not just query**++### Pattern: Orchestrator with Sequential Phases++```markdown+Phase 1: RESEARCH (use Explore agent)+- Gather context+- Identify patterns+- Output: research-summary.md++Phase 2: PLAN (use planner agent)+- Read research-summary.md+- Create implementation plan+- Output: plan.md++Phase 3: IMPLEMENT (use tdd-guide agent)+- Read plan.md+- Write tests first+- Implement code+- Output: code changes++Phase 4: REVIEW (use code-reviewer agent)+- Review all changes+- Output: review-comments.md++Phase 5: VERIFY (use build-error-resolver if needed)+- Run tests+- Fix issues+- Output: done or loop back+```++**Key rules:**+1. Each agent gets ONE clear input, produces ONE clear output+2. Outputs become inputs for next phase+3. Never skip phases - each adds value+4. Use `/clear` between agents to keep context fresh+5. Store intermediate outputs in files (not just memory)++### Agent Abstraction Tierlist (from @menhguin)++**Tier 1: Direct Buffs (Easy to Use)**++1. **Subagents**+ - Direct buff for preventing context rot+ - Ad-hoc specialization+ - Half as useful as multi-agent but MUCH less complexity++2. **Metaprompting**+ - "I take 3 minutes to prompt a 20-minute task"+ - Improves stability and sanity-checks assumptions++3. **Asking user more at beginning**+ - Generally a buff+ - Must answer questions in plan mode++**Tier 2: High Skill Floor (Harder to Use Well)**++1. **Long-running agents**+ - Need to understand shape/tradeoff of 15min vs 1.5hr vs 4hr tasks+ - Requires tweaking+ - Long trial-and-error++2. **Parallel multi-agent**+ - Very high variance+ - Only useful on highly complex OR well-segmented tasks+ - "If 2 tasks take 10 minutes and you spend arbitrary time prompting or merging changes, it's counterproductive"++3. **Role-based multi-agent**+ - "Models evolve too fast for hard-coded heuristics unless arbitrage is very high"+ - Hard to test++4. **Computer use agents**+ - Very early paradigm, requires wrangling+ - "Getting models to do something they weren't meant to do a year ago"++**Takeaway:** Start with Tier 1 patterns. Graduate to Tier 2 only after mastering basics with genuine need.++---++## Tips and Tricks++### Some MCPs are Replaceable++**Problem:** MCPs eat context window and increase token usage++**Solution:** Convert MCP functionality to skills + CLI commands++**Examples:**+- GitHub MCP β `/gh-pr` command wrapping `gh pr create`+- Supabase MCP β Skills using Supabase CLI directly+- Vercel/Railway MCP β CLI wrapper skills++**With lazy loading (recent update):**+- Context window issue mostly solved+- Token usage still optimizable via CLI + skills+- Run MCP operations via CLI instead of in-context+- Significantly reduces token usage for heavy operations (DB queries, deployments)++**When to use CLI + skills over MCP:**+- Heavy repeated operations+- Token cost is concern+- Operation can be scripted+- MCP wrapper adds minimal convenience++**Trade-off:**+- MCP: More convenient, higher token cost+- CLI + Skills: Slightly less convenient, much cheaper tokens++---++## References++- [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (Jan 2026)+- Anthropic: "Claude Code Best Practices" (Apr 2025)+- Fireworks AI: "Eval Driven Development with Claude Code" (Aug 2025)+- [YK: 32 Claude Code Tips](https://agenticcoding.substack.com/p/32-claude-code-tips-from-basics-to) (Dec 2025)+- Addy Osmani: "My LLM coding workflow going into 2026"+- @PerceptualPeak: Sub-Agent Context Negotiation+- @menhguin: Agent Abstractions Tierlist+- @omarsar0: Compound Effects Philosophy+- [RLanceMartin: Session Reflection Pattern](https://rlancemartin.github.io/2025/12/01/claude_diary/)+- @alexhillman: Self-Improving Memory System+- @bcherny: Parallelization patterns++---++## Integration with ID8Pipeline++This guide's patterns map directly to ID8Pipeline stages:++**Stage 7 (Test Coverage):**+- Continuous evals+- pass@k vs pass^k metrics+- Verification loops++**Stage 11 (Listen & Iterate):**+- Continuous learning hooks+- Session reflection+- Latent demand discovery from agent logs++**All Stages:**+- Token optimization reduces cost+- Memory persistence maintains continuity+- Verification patterns prevent tech debt