an open-source alternative to the dots, bots and muses of the world, run an office of claude code/codex like agents on your laptop, sandboxes or anywhere, uses your existing subscriptions
HTML agent-harnessagent-orchestrationagentsai-agentsautonomous-agentsclaude-codecodexcoding-agentsdesktop-appdeveloper-toolselectrongemini-cliharnessharness-engineeringlocal-firstmemorymulti-agentopencodeorchestrationtypescript
feat(blog): add 8 overflow technical posts (#8,#13,#14,#17,#19,#23,#25,#26)
Chaitanya Giric56379d041f2
diff --git a/blog/src/posts/append-only-event-log-agents.md b/blog/src/posts/append-only-event-log-agents.mdnew file mode 100644index 00000000..f472e6ff--- /dev/null+++ b/blog/src/posts/append-only-event-log-agents.md@@ -0,0 +1,147 @@+---+title: "An Append-Only Event Log for a Hive of Agents"+description: "How an append-only event log makes a multi-agent system debuggable and replayable — what to record per event, and why one JSON line per event beats a database."+date: 2026-06-01+category: internals+categoryLabel: Internals+type: Technical+primaryKeyword: "ai agent event log"+secondaryKeywords: ["event sourcing", "append-only log", "agent observability"]+tags: ["Internals", "Observability", "Multi-Agent", "Event Log"]+author:+ name: Chaitanya Giri+ initials: CG+faq:+ - q: "What is an append-only event log in a multi-agent system?"+ a: "It's a file that only ever grows: every meaningful thing that happens — an agent spawns, a message routes, an item escalates — is recorded as one timestamped line, never edited or deleted. It becomes the system's ground-truth history for debugging and replay."+ - q: "Why one JSON object per line instead of a database?"+ a: "Line-delimited JSON (JSONL) is append-cheap, crash-safe, human-readable, and trivially tailable. Appending a line is a single, atomic-enough write; readers parse line by line and track their own position. You get observability with no schema migrations and no server."+ - q: "What should each event record?"+ a: "A timestamp plus a small, typed payload: the kind of event and the few fields that matter for it — who spawned, who messaged whom, what was escalated. Keep events small and factual; the value is in the sequence, not in any one fat record."+---++<div class="callout tldr"><span class="ic">TL;DR</span><p>An <strong>append-only event log</strong>+records every meaningful thing a hive does as one timestamped JSON line that's never edited or+deleted. It turns a multi-agent system from a black box into something+<strong>debuggable and replayable</strong>: you read the sequence back to see exactly what happened.+One JSONL file, written by the single committer, beats a database for this job — cheap to append,+crash-safe, and human-readable.</p></div>++When a hive of agents does something surprising, the first question is always "what actually+happened?" Without a record, you're guessing from terminal scrollback and vibes. The fix is the+oldest trick in distributed systems: write down everything, in order, and never change it. This post+covers what an append-only event log is, what to put in it, and why it's the backbone of multi-agent+observability.++## Append-only, in one sentence++An append-only log is a file you only ever add to. Each line is one event — a small, timestamped,+typed record of something that happened. You never go back and edit or delete a line. The log grows+forever (or until you rotate it), and its *order* is the truth: event N happened before event N+1,+always.++That immutability is the whole value. A mutable status table tells you the *current* state; an+append-only log tells you the *story* — how you got to the current state, step by step. For debugging+a system where many agents act concurrently, the story is what you need.++## One JSON object per line++The format that makes this practical is **JSONL** — one JSON object per line:++```jsonl+{"ts":1717286400123,"kind":"spawn","agentId":"researcher","name":"Researcher","isGod":false}+{"ts":1717286400456,"kind":"message","from":"researcher","to":"coder","act":"request","subject":"Need the schema","id":"…-a1b2"}+{"ts":1717286401789,"kind":"drain","agentId":"coder","count":1}+{"ts":1717286402111,"kind":"escalate","from":"coder","to":"human","subject":"Drop the users table?","id":"…-c3d4"}+{"ts":1717286402999,"kind":"approval","id":"…-c3d4","approve":false}+```++Why JSONL specifically:++- **Appending is cheap and safe.** Adding a line is a single write to the end of the file. There's no+ index to update, no row to lock, no migration to run.+- **It's tailable.** `tail -f` the file and you have a live activity feed. The same stream drives a UI+ activity panel in real time.+- **It's human-readable.** When something breaks, you `cat` the log and read it. No query language, no+ admin tool.+- **Readers track their own position.** Each consumer remembers how far it has read (a cursor) and+ picks up new lines from there. Multiple consumers, no coordination.++A database would give you indexes and queries you mostly don't need here, at the cost of a schema and a+server. For an audit/observability log, plain JSONL is the right amount of technology.++## What to record per event++The art is choosing events that are **small, typed, and factual.** Each line carries a `ts`+(timestamp) and a `kind`, plus only the fields that matter for that kind. In a hive, the events worth+logging are the coordination milestones:++- **`spawn`** — an agent was created (id, name, whether it's the orchestrator).+- **`message`** — a message routed between agents (from, to, act, subject, id).+- **`drain`** — an agent's inbox was drained at a lifecycle point (who, how many).+- **`escalate`** — an item was routed to the human-approval queue (from, subject, id).+- **`approval`** — a human resolved an escalation (id, approved or rejected).++Notice what's *not* there: no giant payloads, no full message bodies, no model outputs. The log is an+index of *what happened and when*, with ids that point at the fuller artifacts (the message files,+the memory). Keep each event a few fields wide and the log stays fast to write, cheap to store, and+easy to read. The value is in the sequence, not in any single fat record.++## Who writes it (and why that matters)++In a multi-agent system, the log is shared state — so the same discipline that protects every other+shared file applies: **one writer.** The agents don't append to the log directly. The coordinating+process (the harness's main process) records each event as it routes messages, spawns agents, and+resolves approvals, and it's the only thing that writes the file. That keeps the log free of+interleaved, half-written lines, and it pairs with the+[single-committer git pattern](/blog/single-committer-git-pattern/): each batch of coordination is+both appended to the log *and* committed, so the log's history and git's history line up.++## Replay and debugging++Here's where the log earns its keep. Because it's an ordered, immutable record, you can **replay** the+system's history:++- **Reconstruct any moment.** Walk the log up to event N and you know exactly what the hive knew at+ that point — who existed, what had been said, what was pending.+- **Find the divergence.** When the team did something wrong, you scan the log for the event where it+ went sideways — the misrouted message, the escalation that should have fired and didn't.+- **Audit decisions.** Every escalation and every human approval is in the log, so "who approved+ deleting that?" has a precise, timestamped answer.++This is the foundation of [debugging a multi-agent system](/blog/debugging-multi-agent-systems/): the+event log is the timeline you read first, before you dig into individual agents' terminals or message+trails. It's also what lets the office-floor visualization stay honest — the activity stream the UI+shows is the same event stream, not a separate, drift-prone narrative.++## Practical notes++A few things that keep an event log healthy:++- **Make events idempotent to read.** A consumer that re-reads a line it already processed should do+ nothing. Cursors plus immutable lines make this automatic.+- **Don't log secrets or huge blobs.** The log is plain text you'll read often and commit. Reference+ big artifacts by id; don't inline them.+- **Rotate if it grows unbounded.** For a long-running hive, cap the file (or roll it) so it stays+ readable — but never *edit* old entries; archive them.+- **Tail the end for "now."** Reading the last N lines is the cheap way to answer "what's happening+ right now?" without parsing the whole history.++## FAQ++**Is this event sourcing?** It's the same family of idea — derive state from an ordered log of events+— applied pragmatically. A full event-sourcing system rebuilds all state from the log; here the log is+primarily for observability and audit, with live state also held in files like the registry. You get+the debuggability benefit without committing to rebuild-everything-from-events.++**How is this different from logging to stdout?** Stdout is unstructured and ephemeral. A structured,+append-only JSONL file is queryable, replayable, and durable — you can read it weeks later and+reconstruct exactly what the hive did.++---++Munder Difflin records every coordination step to an append-only event log that drives+[the live activity feed](https://munderdiffl.in/#how) and makes a hive replayable — committed+alongside the rest of its state.+[Download Munder Difflin](https://munderdiffl.in/#install) to watch a hive you can actually audit;+it's free and open source.diff --git a/blog/src/posts/architecture-two-planes-one-renderer.md b/blog/src/posts/architecture-two-planes-one-renderer.mdnew file mode 100644index 00000000..34113e19--- /dev/null+++ b/blog/src/posts/architecture-two-planes-one-renderer.md@@ -0,0 +1,152 @@+---+title: "Munder Difflin's Architecture: Two Data Planes, One Renderer"+description: "A walkthrough of the multi-agent harness architecture: a node-pty terminal plane and a hooks/hive event plane feeding one React + Pixi.js renderer over IPC."+date: 2026-06-04+category: internals+categoryLabel: Internals+type: Technical+primaryKeyword: "multi-agent harness architecture"+secondaryKeywords: ["electron architecture", "ipc", "agent harness design"]+tags: ["Internals", "Architecture", "Electron", "Multi-Agent"]+author:+ name: Chaitanya Giri+ initials: CG+faq:+ - q: "What are the two data planes in Munder Difflin?"+ a: "The terminal plane carries raw pseudo-terminal bytes from each agent's shell (node-pty) to an xterm.js view. The event plane carries structured agent state — Claude Code hook events and routed hive messages — to the office-floor visualization. One is byte-for-byte authentic; the other is event-driven."+ - q: "Why separate the terminal stream from the event stream?"+ a: "Each answers a different question. The raw terminal stream shows exactly what the agent output; the event stream tells you which tool is running without parsing that output. Hooks alone miss the stream the user expects to see; the stream alone can't reliably say what's happening. Together they cover both."+ - q: "How do the planes reach the UI?"+ a: "Through Electron IPC. The main process owns the PTYs, a hook socket, and the hive; it forwards terminal bytes and structured events to the renderer, where React and Pixi.js draw the terminals and the office floor."+---++<div class="callout tldr"><span class="ic">TL;DR</span><p>The+<strong>multi-agent harness architecture</strong> rests on one decision: <strong>two data+planes</strong>. A <strong>terminal plane</strong> streams raw node-pty bytes to xterm.js (authentic+shells), and an <strong>event plane</strong> carries structured Claude Code hook events plus routed+hive messages to a Pixi.js office floor (what's happening, at a glance). Both flow over Electron IPC+into <strong>one React + Pixi renderer</strong>. The terminal plane is the truth; the event plane is+the story.</p></div>++Most of what Munder Difflin does follows from a single architectural choice, so it's worth walking+through. The harness runs many real Claude Code agents and shows you two things about each: the exact+terminal output, and a live picture of what it's doing. Those are different kinds of data, so they+travel on **two separate planes** that meet in one renderer. Here's the whole shape.++## The load-bearing decision: two planes++```+┌──────────────────── Electron Renderer (React + Pixi) ───────────────────┐+│ Office floor (Pixi.js) Terminal view (xterm.js) │+│ ▲ avatar state ▲ pty bytes │+└────────┼──────────────────────────────────┼────────────────────────────┘+ │ IPC │ IPC+ ┌─────┴───────────────┐ ┌──────┴──────────────┐+ │ Event Plane │ │ Terminal Plane │+ │ hooks + hive router │ │ node-pty │+ └─────▲───────────────┘ └──────▲──────────────┘+ │ JSON events │ raw bytes+ ┌─────┴───────────────────────────────────┴──────────────┐+ │ Claude Code processes (one per agent) │+ └────────────────────────────────────────────────────────┘+```++Each agent is one real `claude` process. From it we read two streams:++- a **byte stream** — exactly what the shell printed, and+- an **event stream** — structured "I'm about to use this tool," "a message routed," "I stopped."++Keeping them separate is the whole trick. Let's take each plane, then how they converge.++## The terminal plane (node-pty)++The terminal plane is about *fidelity*. Each agent runs in a real pseudo-terminal via+[node-pty](/blog/node-pty-electron-real-terminals/), so its shell behaves byte-for-byte like a normal+terminal — colors, prompts, full-screen TUIs, all authentic. The main process owns the PTY and+forwards its output to the renderer over IPC on a per-session channel; keystrokes from the UI travel+back the same way. The renderer paints it with a pooled xterm.js terminal per agent (the technique+behind [rendering many live terminals](/blog/rendering-many-live-terminals-performance/)).++What the terminal plane gives you: the unfiltered truth of what each agent did. What it *can't* easily+give you: a reliable "which tool is running right now?" without scraping and guessing at output. That's+the other plane's job.++## The event plane (hooks + hive)++The event plane is about *structure*. It has two sources, both feeding the main process:++**Claude Code hooks.** Each agent is launched with lifecycle hooks wired through a tiny shim that+forwards every event — `PreToolUse`, `PostToolUse`, `Stop`, `Notification` — to a Unix domain socket+the main process listens on. These are clean, typed signals: "about to run Bash," "finished editing,"+"idle and waiting." No output parsing required. (See+[Claude Code hooks, explained](/blog/claude-code-hooks-explained/).)++**The hive.** The coordination layer — per-agent mailboxes, a message router, single-committer git, and+shared memory — emits its own events as it works: a message routed from A to B, an item escalated to+the human. The [GOD orchestrator](/blog/how-the-god-orchestrator-works/) lives here as the+intelligence; the main process is the mechanism that moves messages and commits state.++Both sources land in the main process, which forwards structured events to the renderer. There they+drive the office floor: avatars walk to stations on tool events, envelopes fly on message events.++## Why both planes (the rationale)++The temptation is to pick one. Each alone is insufficient:++- **Hooks alone** don't give you the raw stream the user expects to *see* — you'd know an agent ran a+ command but not what it printed.+- **The terminal stream alone** can't tell you which tool is running without fragile output parsing —+ you'd see bytes but have to guess at meaning.++Together they're complementary: the office floor is **event-driven** (precise, structured, cheap to+render), and the terminal view is **byte-for-byte authentic** (the full detail when you want it). Two+planes, two questions, both answered well.++## One renderer convergence++Both planes terminate in a single Electron renderer running React and Pixi.js:++- **React** owns the app shell, panels, and the xterm.js terminal views (the terminal plane).+- **Pixi.js** owns the office floor — avatars, the Tiled map, flying envelopes (the event plane), in+ one camera-controlled world updated by a single ticker (the design in+ [visualizing AI agents with Pixi.js](/blog/visualizing-ai-agents-pixijs/)).++The bridge between the main process and the renderer is a typed **preload API** exposed over Electron's+context bridge — the renderer calls into it to spawn/write/resize PTYs and read hive state, and+subscribes to it for incoming terminal bytes and hive events. Crucially, the renderer never touches the+filesystem, git, or the agents directly; it only talks to the main process. That keeps a hard line+between *mechanism* (main process: PTYs, sockets, git, routing) and *presentation* (renderer: React ++Pixi).++## Why this architecture holds up++A few properties fall out of the two-plane design:++- **Separation of concerns.** Fidelity and structure are independent problems solved independently. You+ can improve terminal rendering without touching the event pipeline, and vice versa.+- **The main process is the single source of authority.** PTYs, the hook socket, the hive's git repo,+ and the memory layer all live in one place; the renderer is a consumer. That's what makes the+ [single-committer git pattern](/blog/single-committer-git-pattern/) and safe IPC possible.+- **Real, not simulated.** Because the event plane carries actual hook and router events, the+ visualization can't drift from reality — it *is* reality, rendered.++It's the architecture that lets "a hive of Claude Code agents" be both something you can *read* (the+terminals) and something you can *watch* (the floor) — which is the heart of what a+[multi-agent harness](https://munderdiffl.in/#what) is.++## FAQ++**Where does memory fit in the two planes?** Alongside the event plane's hive layer. The shared+semantic memory is driven by the main process (a CLI it points each agent at), and recall happens in+the agents themselves — it's part of the coordination mechanism, not a third UI plane.++**Could you add a third plane?** You could, but the discipline is to keep planes orthogonal. New+signals usually fit the event plane (more structured events) or the terminal plane (more byte+streams) rather than needing a new one.++---++Munder Difflin's two-plane architecture — authentic node-pty terminals plus an event-driven Pixi.js+floor, converging in one renderer — is what makes a hive both legible and real.+[Download Munder Difflin](https://munderdiffl.in/#install) to see the architecture in action; it's+free and open source.diff --git a/blog/src/posts/building-an-ai-office-floor.md b/blog/src/posts/building-an-ai-office-floor.mdnew file mode 100644index 00000000..a1acc36b--- /dev/null+++ b/blog/src/posts/building-an-ai-office-floor.md@@ -0,0 +1,158 @@+---+title: "Building an AI \"Office Floor\": Avatars, Pathfinding, Envelopes"+description: "The game-dev techniques behind a dev tool's office floor: Tiled maps, BFS pathfinding, sprite-sheet recoloring, seat assignment, and speech-act envelopes in Pixi.js."+date: 2026-06-03+category: internals+categoryLabel: Internals+type: Technical+primaryKeyword: "ai agent visualization"+secondaryKeywords: ["tiled maps", "pathfinding", "game-ui for tools"]+tags: ["Internals", "Pixi.js", "Game Dev", "Visualization"]+author:+ name: Chaitanya Giri+ initials: CG+faq:+ - q: "What game-dev techniques does an AI office floor use?"+ a: "Tiled tile maps for the level, a grid pathfinder for movement, sprite-sheet slicing and recoloring for distinct avatars, a seat-reservation pool for desk assignment, and tween-based animation for flying message envelopes — all rendered with a 2D engine like Pixi.js."+ - q: "Do you need a game engine to build an office floor?"+ a: "No — a 2D renderer like Pixi.js plus a few classic techniques is enough. The map comes from the Tiled editor, pathfinding is a small BFS, and avatars are sprite sheets. It's game tech, but lightweight enough to live inside a developer tool."+ - q: "How are the avatars made distinct?"+ a: "They start from a shared character walk sheet and are recolored per character with palette recipes (skin, hair, shirt), so each agent reads as a different person on the floor without drawing every sprite by hand."+---++<div class="callout tldr"><span class="ic">TL;DR</span><p>An <strong>AI agent visualization</strong>+that looks like a tiny office game uses ordinary game-dev tech: a <strong>Tiled</strong> tile map for+the floor, a small <strong>BFS pathfinder</strong> for movement, <strong>sprite-sheet</strong> slicing++ recoloring for distinct avatars, a <strong>seat pool</strong> for desk assignment, and tweened+<strong>envelopes</strong> for messages — all in Pixi.js. None of it is heavy; it's game craft applied+to a developer tool.</p></div>++The office floor in Munder Difflin looks like a little workplace sim, and that's on purpose — a+spatial, game-like view makes a hive of agents legible. Under the hood it's a handful of classic+game-development techniques, kept deliberately lightweight so they fit inside a dev tool. This post is+the techniques tour. (For *why* a visualization matters and the Pixi.js plumbing, start with+[visualizing AI agents with Pixi.js](/blog/visualizing-ai-agents-pixijs/).)++A note up front: several of these pieces are ported from an open-source 2D office game+(shahar061/the-office), with character art from LimeZu's sprite packs. Standing on game-dev shoulders+is exactly the point — these are solved problems.++## The floor is a Tiled map++The office isn't drawn in code; it's authored in **Tiled**, the standard 2D map editor, and exported as+a `.tmj` (JSON) file. The renderer reads its layers:++- **Tile layers** — floor, walls, furniture — drawn from tileset textures. Each tile references a+ global id, resolved back to the right tileset by its `firstgid` offset.+- A **collision layer** — which tiles are solid (walls, desks) versus walkable floor.+- A **spawn-points** object layer — named anchors for desks and seats.++Because the level is data, changing the office is a *design* task: open the map in Tiled, move a desk,+re-export. No code changes to rearrange the furniture. One subtlety worth stealing: desk/seat tiles are+marked solid for collision but force-flagged *walkable* for pathing, so an avatar can walk *onto* its+chair instead of treating it as a wall.++## Movement is grid pathfinding++Avatars walk from their desk to a station and back. The pathfinder is a **breadth-first search** on the+map's walkability grid — four-directional, no diagonals:++```ts+// BFS over walkable tiles → shortest tile path from start to goal+function findPath(map, start, goal) {+ if (!map.isWalkable(goal.x, goal.y)) return null;+ const queue = [start], visited = new Set([key(start)]), parent = new Map();+ while (queue.length) {+ const cur = queue.shift();+ for (const d of [{x:0,y:-1},{x:0,y:1},{x:-1,y:0},{x:1,y:0}]) {+ const next = { x: cur.x + d.x, y: cur.y + d.y };+ if (visited.has(key(next)) || !map.isWalkable(next.x, next.y)) continue;+ visited.add(key(next)); parent.set(key(next), cur);+ if (next.x === goal.x && next.y === goal.y) return reconstruct(parent, start, goal);+ queue.push(next);+ }+ }+ return null;+}+```++On an unweighted grid, BFS returns the shortest path for free — no heuristic, no priority queue, no A*+needed at office scale. It's a dozen lines and it's correct, which is the right trade for a map this+size.++## Avatars: one sheet, many people++Drawing fifteen unique characters by hand would be a slog. Instead, avatars start from a shared+**character walk sheet** and are made distinct by *recoloring*:++- A **sprite adapter** slices the sheet into frames — walk cycles for each facing direction. The sheet+ packs directions in a known order (right, up, left, down); the adapter maps them into the+ three-row grid the renderer wants, and *left* is just *right* drawn flipped, so you store fewer+ frames.+- The **character sprite** plays walk / type / read / idle animations and applies a small **leg-crop+ mask** when an agent sits, so it tucks under the desk instead of standing on it.+- Each character gets a **palette recipe** — skin, hair, shirt — so the cast is visually distinct from+ one base sheet. (If the cast looks suspiciously like a certain paper company's staff, that's not an+ accident.)++The payoff: a recognizable, individuated team without an artist hand-drawing every avatar.++## Seats: a reservation pool++When an agent joins, it needs a desk. A tiny **seat pool** holds the ordered list of spawn points from+the map and hands out the first free one, releasing it when the agent leaves:++```ts+reserveNext() { // first unclaimed seat, in map order+ for (const seat of this.seats)+ if (!this.claimed.has(seat)) { this.claimed.add(seat); return seat; }+ return null; // office is full+}+```++First-come, first-served, one desk each — so a growing team fills the office in order rather than+stacking onto one chair. Thirty-three lines of bookkeeping that make dynamic agents Just Work.++## Envelopes: messages you can see fly++When the hive routes a message between agents, the floor spawns a little pixel **envelope** that arcs+from the sender's desk to the recipient's. The animation is a quadratic arc — lerp the endpoints, lift+the midpoint with a sine — eased in and out, with a small burst ring on arrival. The envelope is+**tinted by the message's speech act**: asks are cool-colored, agreements and confirmations warm,+refusals red, and an escalation to the human a distinct color. So you don't just see *that* a message+moved — you see *who asked whom, and what kind of ask it was*, at a glance.++That envelope is pure presentation riding on real data: it only flies because the+[message router](/blog/atomic-file-mailboxes-for-agents/) actually delivered something.++## Keeping it lightweight++The whole point is that this is *game tech in a dev tool*, not a game. So the implementation stays+lean: pixel-art settings (no antialias, nearest-neighbor scaling, integer positions), a single shared+animation ticker, and a camera that transforms one world container rather than every sprite. That+discipline is what lets the floor run alongside a fleet of live terminals without contention — the same+performance mindset as [rendering many live terminals](/blog/rendering-many-live-terminals-performance/).++## Why borrow from games at all++Developer tools usually present work as text. A spatial, animated floor borrows what games are great at+— making *state* immediately readable through position and motion — and applies it to "what is my team+of agents doing?" The techniques are old and solved; the novelty is pointing them at observability+instead of entertainment. An avatar at the terminal, an envelope in flight, a teammate waving for your+input: that's a status dashboard you read with your eyes instead of your patience.++## FAQ++**Is the office floor just eye candy?** No — it's observability. Every position and motion maps to a+real event (a tool call, a routed message), so the floor is a live, glanceable read of the hive's+state, not decoration.++**Do I need to know game dev to build something like this?** Not deeply. A 2D renderer, the Tiled+editor, a BFS, and sprite sheets get you most of the way. These are approachable, well-documented+techniques — the hard part is wiring them to *real* agent events, not the rendering.++---++Munder Difflin's [office floor](https://munderdiffl.in/#how) turns a hive of Claude Code agents into a little workplace you can watch+— Tiled maps, pathing avatars, and flying envelopes, all driven by real activity.+[Download Munder Difflin](https://munderdiffl.in/#install) to see it run; it's free and open source.diff --git a/blog/src/posts/debugging-multi-agent-systems.md b/blog/src/posts/debugging-multi-agent-systems.mdnew file mode 100644index 00000000..c9de677a--- /dev/null+++ b/blog/src/posts/debugging-multi-agent-systems.md@@ -0,0 +1,129 @@+---+title: "How to Debug a Multi-Agent System"+description: "How to debug a multi-agent system: use the event log, per-agent terminals, message trails, and git history to find why a hive of AI agents went sideways."+date: 2026-06-04+category: guides+categoryLabel: Guides+type: Technical+primaryKeyword: "debug multi-agent system"+secondaryKeywords: ["multi-agent debugging", "agent observability", "event log"]+tags: ["Guides", "Debugging", "Observability", "Multi-Agent"]+author:+ name: Chaitanya Giri+ initials: CG+faq:+ - q: "How do you debug a multi-agent system?"+ a: "Start from the timeline, not a single agent. Read the append-only event log to find when things diverged, follow the message trail between the agents involved, then drop into the specific agent's terminal for the byte-level detail. The order — log, messages, terminal — keeps you from guessing."+ - q: "Why is debugging multiple agents harder than one?"+ a: "Concurrency and interaction. A bug may not live in any one agent but in how two of them coordinated — a misrouted message, a missed handoff, a shared assumption that was wrong. You need the cross-agent view, not just per-agent logs."+ - q: "What's the single most useful artifact for multi-agent debugging?"+ a: "The append-only event log. It's the ordered, immutable timeline of everything that happened — who spawned, who messaged whom, what escalated — so you can replay the run and find the exact event where it went wrong."+---++<div class="callout tldr"><span class="ic">TL;DR</span><p>To+<strong>debug a multi-agent system</strong>, work from the outside in: read the+<strong>event log</strong> to find when the run diverged, follow the <strong>message trail</strong>+between the agents involved, then open the offending agent's <strong>terminal</strong> for byte-level+detail — and use <strong>git history</strong> to replay the coordination state. The bug often lives in+the interaction between agents, not inside any one of them, so start with the timeline.</p></div>++Debugging one agent is familiar: read its output, find the bad step. Debugging a *hive* is different,+because the failure is often relational — two agents coordinated wrong, a message went to the wrong+place, a handoff never happened. Staring at a single agent's terminal won't show you that. This is a+practical method for finding out why a multi-agent system went sideways, using the observability a+well-built hive already gives you.++## Start with the timeline, not an agent++The first instinct — "open the agent that misbehaved" — is usually the slow path. The fast path is the+[append-only event log](/blog/append-only-event-log-agents/): the ordered, immutable record of+everything the hive did. Every spawn, every routed message, every escalation and approval is one+timestamped line. Read it and you get the *sequence*, which is exactly what a per-agent view hides.++Scan the log for the moment things diverged from what you expected:++- a message routed to the wrong agent (or never routed),+- an escalation that should have fired and didn't (or fired when it shouldn't),+- an agent that went idle while it still had work,+- a burst of activity where there should have been calm.++The event where reality and expectation part is your starting point. Now you know *which agents* and+*which moment* to investigate — instead of reading six terminals hoping to spot it.++## Follow the message trail++Multi-agent bugs frequently live in the messaging. Once the log points you at the agents involved,+follow the trail their mailboxes leave. Because messaging is file-based+([atomic file mailboxes](/blog/atomic-file-mailboxes-for-agents/)), the evidence is sitting on disk:++- An agent's **outbox archive** (sent messages) shows exactly what it asked for, and how it phrased the+ request.+- An agent's **inbox** and its **handled folder** show what it received and what it acted on.+- The **speech act** on each message (`request`, `inform`, `done`…) tells you whether a reply was even+ expected — a common bug is one agent replying to a terminal `inform` and starting a loop, or *not*+ replying to a `request` and leaving a peer stuck.++Reading the trail answers the relational question: did A actually ask B what you think it did, and did+B get it? Half of "the hive stalled" bugs resolve right here — a message that was malformed, misrouted,+or never sent.++## Drop into the agent's terminal++Now — and only now — open the specific agent's terminal. Because every agent runs in a real+pseudo-terminal, you have its byte-for-byte output, not a summary. With the log and the message trail+having narrowed *which* agent and *which* moment, the terminal gives you the detail: the exact command+that failed, the error it hit, the reasoning it printed. This is where a multi-agent bug finally+becomes a single-agent bug you can fix.++The office floor helps here too: the [live visualization](/blog/visualizing-ai-agents-pixijs/) shows+you, at a glance, which agent is blocked (waving at the mailbox), which is spinning, and which went+idle — often pointing you at the right terminal before you've even opened the log.++## Replay with git++The hive's coordination state lives in a git repo that the main process commits to as it works (the+[single-committer pattern](/blog/single-committer-git-pattern/)). That makes the whole run+**replayable**: each coordination step is a commit, so you can walk the history and reconstruct exactly+what the hive knew at any point — the roster, the pending messages, the board. When "what state was the+system in when it broke?" matters, the git log answers it precisely instead of from memory.++## Check the usual suspects++Some failure modes recur in multi-agent systems. When you're stuck, check these directly:++- **Livelock.** Two agents ping-ponging. The log shows a tight back-and-forth; the fix is usually a+ message type that shouldn't have obligated a reply, or a hop cap that should have escalated.+- **A stuck agent.** One agent waiting on a reply that never came. The message trail shows the unanswered+ `request`; the peer either didn't receive it or didn't treat it as needing a reply.+- **Reprocessing.** An agent acting on the same message twice. Check its cursor and whether handled+ messages were moved aside — idempotent handling should make re-seeing a message a no-op.+- **A silent escalation.** Something critical that should have reached the human but didn't (or vice+ versa). The approvals queue and the log's escalation events tell you what the policy actually did.++## Build for debuggability up front++The reason this method works is that the observability was designed in, not bolted on. If you're+building a multi-agent system, the lesson is to make it debuggable *before* you need to debug it:++- record an **append-only event log** so you always have a timeline,+- keep messaging **file-based and archived** so trails persist,+- give each agent a **real terminal** so detail is never lost, and+- commit coordination state to **git** so any moment is replayable.++Get those four right and "why did the hive do that?" stops being a mystery and becomes a query.++## FAQ++**Can I debug without stopping the agents?** Yes — the log, message files, and git history are all+read-only artifacts you can inspect while the hive runs. The visualization is live. You rarely need to+halt the team to understand it.++**What if the bug is in my orchestrator's judgment?** That's a prompt problem, not a code problem. The+log shows you the routing decision that went wrong; you correct the orchestrator's instructions (its+escalation and routing policy) rather than patching the mechanism.++---++Munder Difflin makes a hive debuggable by design: an event log, file-based message trails, real+per-agent terminals, a git-committed history, and [a live floor](https://munderdiffl.in/#how). [Download Munder Difflin](https://munderdiffl.in/#install)+to run agents you can actually trace; it's free and open source.diff --git a/blog/src/posts/human-in-the-loop-ai-agents.md b/blog/src/posts/human-in-the-loop-ai-agents.mdnew file mode 100644index 00000000..da3b9281--- /dev/null+++ b/blog/src/posts/human-in-the-loop-ai-agents.md@@ -0,0 +1,138 @@+---+title: "Human-in-the-Loop AI Agents: Approvals That Don't Slow You Down"+description: "Design human-in-the-loop AI agents that stay autonomous: an approvals queue that escalates only spend, destructive ops, and scope — and relays your answer back."+date: 2026-06-02+category: orchestration+categoryLabel: Orchestration+type: Technical+primaryKeyword: "human in the loop ai agents"+secondaryKeywords: ["human in the loop", "ai approvals", "agent guardrails"]+tags: ["Orchestration", "Human-in-the-Loop", "Guardrails", "Multi-Agent"]+author:+ name: Chaitanya Giri+ initials: CG+faq:+ - q: "What does human-in-the-loop mean for AI agents?"+ a: "It means a human stays in control of the decisions that matter, while the agents handle everything routine on their own. The skill is in drawing the line: escalate the few critical calls (spend, destructive ops, scope changes) and let the rest run autonomously."+ - q: "Doesn't requiring approvals make agents slow?"+ a: "Only if you approve everything. A good design escalates a small, explicit set of critical actions and resolves the rest automatically, so you're tapped on the shoulder a few times a day, not a few times a minute."+ - q: "How does the human's answer get back to the agent?"+ a: "The escalated item waits in an approvals queue. When you approve or reject — optionally with a note like 'yes, but cap it at $5' — that decision is relayed back to the agent that asked, as a message from the human, and it continues with your guidance."+---++<div class="callout tldr"><span class="ic">TL;DR</span><p>Good+<strong>human-in-the-loop AI agents</strong> don't ask permission for everything — they escalate only+the <strong>critical few</strong> (spending money, destructive operations, scope changes,+unresolvable conflicts) to an <strong>approvals queue</strong>, and resolve the rest themselves. Your+answer, with an optional note, is <strong>relayed back</strong> to the agent that asked. The result:+autonomy by default, human judgment exactly where it matters.</p></div>++"Human-in-the-loop" gets misread as "human approves every step." That version is miserable — it turns+a team of agents into a clicking exercise and throws away the whole point of autonomy. The version+that works is the opposite: agents run on their own, and a human is pulled in *only* for the decisions+that genuinely require a person. This post is about designing that line well.++## The trap: approval fatigue++If every file write, every command, every message needs your sign-off, two things happen. First, you+become the bottleneck — the agents are only as fast as your reaction time. Second, you stop reading+the approvals; after the hundredth "can I edit this file?", you click yes on autopilot, which means the+*one* approval that mattered slips through. Asking for everything is functionally the same as asking+for nothing.++So the goal isn't "more approvals." It's **fewer, better** approvals — a queue that stays empty except+when something real needs you.++## Draw the line explicitly++The core design decision is a written list of what counts as **critical**. Everything on the list+escalates to a human; everything else the agents handle. A solid default:++- **Destructive operations** — deleting data, force-pushing, anything hard to undo.+- **Spending real money** — actions with a dollar cost.+- **Scope changes** — work drifting from what you actually asked for.+- **Unresolvable conflicts** — two agents at an impasse that can't be settled internally.++The power of making this *explicit* is that it becomes a control surface you can tune. Too many+interruptions? Tighten the list. Agents doing things that scare you? Loosen what they're allowed to+decide alone. In a hive, this policy lives in the [orchestrator's](/blog/how-the-god-orchestrator-works/)+instructions — so you adjust behavior by editing a prompt, not by shipping code.++## How escalation actually works++When an agent (or the orchestrator on its behalf) hits something critical, it doesn't act. It addresses+a message to the human — or flags the item as needing human attention — and the coordinating process+diverts it into an **approvals queue** instead of delivering it normally. Mechanically it's the same+messaging fabric agents use for each other ([atomic file mailboxes](/blog/atomic-file-mailboxes-for-agents/)),+with one rule: anything marked for the human lands in the queue, not in another agent's inbox.++The queue surfaces in the UI as a short list of "I need your call on these." Each item carries enough+context to decide: who's asking, what they want to do, and why it's critical.++## The answer round-trips++Approval isn't a dead end — it's a conversation. When you resolve an item, two things can travel back:++1. **The decision.** Approve, and the held action proceeds. Reject, and it doesn't.+2. **A note.** You can attach guidance — "yes, but cap it at $5," "no, use the staging DB instead" —+ and that note is relayed back to the agent that asked, as a message from the human.++That round-trip is what makes human-in-the-loop feel like *delegation* rather than *gatekeeping*. The+agent asked a real question; you answered it; it continues with your answer in hand. You're not+blocking the work, you're steering it at the one fork that needed a person.++## Guardrails that keep the queue honest++A couple of structural safeguards keep escalation from being gamed or from livelocking:++- **A hop cap.** If two agents bounce a question back and forth past a limit, the system stops the+ ping-pong and escalates to the human instead of letting it loop forever. The cap is a fuse: when+ agents can't resolve something between themselves, a person breaks the tie.+- **Fail toward asking.** When the system is unsure whether something is critical — an edge case, an+ addressing ambiguity — the safe default is to route it to the human rather than to act. "Ask" is a+ better failure mode than "guess."++These mean the queue catches the things that *should* reach you even when the agents don't explicitly+flag them.++## Why this beats "approve everything" and "approve nothing"++- **vs. approve everything:** you keep autonomy and your attention. The agents do the 95% that's+ routine without you, and the queue stays short enough that you actually read it.+- **vs. approve nothing (full autonomy):** you keep control of the dangerous 5%. Money, destructive+ ops, and scope drift don't happen behind your back — they wait for you.++The selective-escalation model is what lets a hive run unattended for long stretches and still be+safe, which is the foundation of [letting agents build while you sleep](/blog/claude-code-automation-while-you-sleep/):+you wake up to progress plus a short approvals list, not a pile of irreversible actions. It's also the+core of [coordinating AI coding agents](/blog/coordinating-ai-coding-agents/) — the orchestrator+adjudicates the routine traffic and escalates only what it should.++## Designing your own escalation policy++If you're building this, a few principles:++- **Start stricter, then loosen.** Begin with a broader critical list and relax it as you learn what+ the agents handle well. Earning trust beats assuming it.+- **Make every escalation legible.** An approval you can't understand at a glance is one you'll+ rubber-stamp. Include the *why*, not just the *what*.+- **Log every decision.** Approvals and rejections belong in the [event log](/blog/append-only-event-log-agents/)+ so "who approved this?" always has an answer.+- **Keep the human's note actionable.** The point of the round-trip is steering; make it easy to send+ guidance, not just a yes/no.++## FAQ++**What if I'm away when something escalates?** It waits. The held item sits in the queue until you+resolve it — the agent that asked is blocked on that one decision but the rest of the team keeps+working. Nothing critical happens without you.++**Can the agents escalate too much?** They can, early on — which is exactly why the critical list is+tunable. If the queue is noisy, tighten the policy so more decisions are handled autonomously.++---++Munder Difflin ships human-in-the-loop the right way: a [GOD orchestrator](https://munderdiffl.in/#how) that resolves routine traffic+itself and an approvals queue that escalates only the critical few — with your notes relayed straight+back. [Download Munder Difflin](https://munderdiffl.in/#install) to run agents that stay autonomous+and ask only when they should; it's free and open source.diff --git a/blog/src/posts/markdown-first-agent-memory.md b/blog/src/posts/markdown-first-agent-memory.mdnew file mode 100644index 00000000..b536cda8--- /dev/null+++ b/blog/src/posts/markdown-first-agent-memory.md@@ -0,0 +1,139 @@+---+title: "Why We Built Agent Memory Markdown-First"+description: "The case for markdown-first AI agent long-term memory: plain notes a human can read and git can diff, with a semantic index on top that degrades gracefully."+date: 2026-05-27+category: memory+categoryLabel: Memory+type: Technical+primaryKeyword: "ai agent long-term memory"+secondaryKeywords: ["markdown memory", "agent knowledge base", "long-term memory"]+tags: ["Memory", "MemPalace", "Markdown", "Multi-Agent"]+author:+ name: Chaitanya Giri+ initials: CG+faq:+ - q: "What is markdown-first agent memory?"+ a: "It's an approach where each agent's durable knowledge lives in plain markdown files it reads on startup and appends to as it learns. A semantic index is built on top of those files for fast recall — but the markdown, not the index, is the source of truth."+ - q: "Why not just use a vector database for agent memory?"+ a: "A vector DB stores opaque embeddings you can't read or diff, and many memory frameworks want to own the agent's runtime. Markdown-first keeps memory human-readable and git-diffable, treats the index as a removable accelerator, and fits agents whose runtime is already Claude Code."+ - q: "What happens if the semantic index breaks?"+ a: "Nothing catastrophic — the agent falls back to reading its markdown directly. Because the files are the source of truth and the index sits on top, a broken or missing index degrades to plain markdown memory instead of losing everything."+---++<div class="callout tldr"><span class="ic">TL;DR</span><p>We built+<strong>AI agent long-term memory</strong> markdown-first: each agent writes durable facts to plain+markdown files, and a semantic index sits <em>on top</em> for fast recall. The reason is durability —+markdown is <strong>human-readable</strong>, <strong>git-diffable</strong>, and <strong>degrades+gracefully</strong> (if the index breaks, the files still work). A vector database can't say the same,+and it's the wrong fit when the agent's runtime is already Claude Code.</p></div>++When you add long-term memory to an agent, the tempting first move is to reach for a vector database or+a full memory framework. We went the other way: plain markdown files first, with a semantic index+layered on top. This post is the argument for that ordering — why the *source of truth* should be+files you can read, and the index should be a removable accelerator.++## What "markdown-first" means++The design is simple. Each agent has a `memory.md`. The rule it follows:++- **Read it at the start of a task** — recall what you already know before starting.+- **Append durable facts as you learn them** — a decision, a convention, a gotcha — in small, atomic+ entries.++```markdown+## build+- The web bundle is built with electron-vite, NOT plain vite. Use `npm run build`.+- node-pty needs an electron-rebuild after install (the postinstall handles it).+- Never hand-edit docs/index.html during a release — it's the REL fallback source.+```++A semantic index is built *from* these files so the agent can recall by meaning instead of re-reading+everything ([semantic memory for AI agents](/blog/semantic-memory-for-ai-agents/) covers that layer).+But the markdown is primary. The index is derived, and disposable.++## Three reasons the ordering matters++### 1. Memory you can read is memory you can trust++Open an agent's `memory.md` and you see exactly what it "knows" — in your language, not as a blob of+floating-point vectors. That's not a nicety; it's how you debug an agent that's acting on a stale or+wrong belief. With markdown, you find the bad note and fix it. With opaque embeddings, you're guessing+at what the agent remembers and re-indexing to change it. The most important property of a memory+system is that a human can audit it, and plain text wins that outright.++### 2. Memory that diffs is memory you can review++Because the notes are files, they live in git. Every change to what an agent knows is a diff you can+read in a commit — memory becomes reviewable like any other artifact. You can watch an agent's+understanding evolve, catch the moment it learned something wrong, and roll it back. A database of+embeddings has no meaningful diff; "the vectors changed" tells you nothing. Text under version control+turns memory from a black box into a reviewable record, which pairs naturally with the+[append-only event log](/blog/append-only-event-log-agents/) for full auditability.++### 3. Memory that degrades gracefully never strands you++This is the durability clincher. If the semantic index breaks, isn't installed, or you move to a+machine without it, the markdown files **still work** — the agent reads them directly. The fancy recall+is an upgrade, not a dependency. Contrast a database that won't open: it takes the memory with it. With+markdown-first, the floor never falls out from under the agent; the worst case is slower recall, not+lost knowledge.++## Why not a vector database or a memory framework?++Vector DBs and dedicated memory frameworks are real tools with real uses. They're the wrong default+*here* for two reasons.++**They store what you can't read.** Their native unit is the embedding — great for similarity search,+useless for human inspection or git review. You end up bolting a "show me the source text" layer back+on, which is just markdown wearing a costume.++**They want to own the runtime.** Many agent-memory frameworks assume *they* are the thing running the+agent loop — they expect to be in charge. But when your agents are real `claude` sessions, the runtime+is already Claude Code. The memory layer should be a lightweight companion beside it, not a framework+that wants to replace it. Markdown files plus a CLI-driven index fit that shape exactly: the agent+writes notes as a normal part of its work, and a separate process mines those notes into a searchable+store without intruding on the loop.++That mining is the bridge: the index reads each agent's markdown and makes it recall-able by the whole+team, on a schedule, only re-processing files that changed. You get vector-quality recall *and*+human-quality source text — because the vectors are derived from the text, not instead of it.++## The single-writer detail++One subtlety that markdown-first gets right almost for free: **one writer per memory file.** Each agent+owns its own `memory.md` and only it appends there. A shared index aggregates everyone's notes+read-only, so the whole team can recall across agents without any file being written by two processes.+It's the same single-writer discipline that keeps+[multi-agent git safe](/blog/single-committer-git-pattern/) — applied to memory. Co-edited memory files+would tear; per-agent files plus a read-only shared index don't.++## When markdown-first is the right call++This approach shines when:++- you want to **audit and trust** what your agents remember,+- you want memory **under version control** so you can review and revert it,+- your agents' runtime is something you don't control (a CLI like Claude Code), and+- you value **graceful degradation** over maximal indexing sophistication.++It's less compelling if you're building a system at a scale where you genuinely need a dedicated vector+store's throughput and you're willing to give up readability for it. For a hive of a handful to a few+dozen agents, markdown-first is the durable, debuggable default. For the plain-English version of why+agents forget and what shared memory changes, see+[how to give Claude Code long-term memory](/blog/give-claude-code-long-term-memory/).++## FAQ++**Doesn't markdown get unwieldy at thousands of notes?** The *files* stay fine; the challenge is recall,+which is exactly what the semantic index on top solves — you recall the few relevant notes by meaning+rather than reading them all. The markdown scales as storage; the index scales as retrieval.++**Can I edit an agent's memory by hand?** Yes — it's a text file. Fix a wrong note, remove a stale one,+and the next mining pass picks up the change. Try that with a vector store.++---++Munder Difflin gives every agent a plain `memory.md` plus+[a shared semantic palace mined from it](https://munderdiffl.in/#how) — readable, diffable, and+graceful when the index is gone. [Download Munder Difflin](https://munderdiffl.in/#install)+to give your agents memory you can actually read; it's free and open source.diff --git a/blog/src/posts/rendering-many-live-terminals-performance.md b/blog/src/posts/rendering-many-live-terminals-performance.mdnew file mode 100644index 00000000..dc2a2701--- /dev/null+++ b/blog/src/posts/rendering-many-live-terminals-performance.md@@ -0,0 +1,147 @@+---+title: "Rendering Many Live Terminals Without Melting the CPU"+description: "xterm.js performance for many live PTYs: a persistent terminal pool, render-only-visible, smart scrollback, and accelerated rendering to stream dozens of terminals."+date: 2026-06-03+category: internals+categoryLabel: Internals+type: Technical+primaryKeyword: "xterm.js performance"+secondaryKeywords: ["terminal rendering", "xterm.js performance", "electron performance"]+tags: ["Internals", "Performance", "xterm.js", "Terminals"]+author:+ name: Chaitanya Giri+ initials: CG+faq:+ - q: "How do you render many xterm.js terminals without high CPU?"+ a: "Keep one persistent terminal per session and only render the visible one(s); re-parent the terminal's element instead of recreating it, tune scrollback, and use an accelerated renderer when throughput is high. The buffers keep filling in the background while you paint just what's on screen."+ - q: "Why is recreating an xterm terminal on tab switch slow and buggy?"+ a: "node-pty keeps no scrollback, so a freshly created terminal is blank until the program repaints — and recreating instances thrashes the CPU. A persistent pool avoids both: the terminal lives for the session and its DOM element is moved in and out of views."+ - q: "Does scrollback size affect performance?"+ a: "Yes. xterm holds scrollback in memory per terminal, so a large scrollback multiplied by many terminals adds up. Pick a generous-but-bounded value and tune it to your workload."+---++<div class="callout tldr"><span class="ic">TL;DR</span><p>Streaming dozens of live PTYs to xterm.js+stays cheap if you do four things: keep <strong>one persistent terminal per session</strong> (don't+recreate on tab switch), <strong>render only what's visible</strong> while buffers fill in the+background, <strong>bound your scrollback</strong>, and reach for an <strong>accelerated renderer</strong>+only when throughput demands it. The big win is the persistent pool — it fixes both the blank-terminal+bug and the CPU thrash at once.</p></div>++A single xterm.js terminal is cheap. Thirty of them, all streaming output from live agents while the+user switches between them, is where naive implementations melt the CPU — or show blank panes. This+post is the performance playbook for rendering many live terminals at once, built on the same techniques+that keep an agent harness's floor of terminals smooth.++## The naive approach and why it hurts++The obvious design: when a terminal view mounts, create an xterm `Terminal`; when it unmounts, dispose+it. For one terminal it's fine. For many, it fails twice:++1. **CPU thrash.** Creating and destroying terminal instances as the user switches tabs is expensive —+ you're rebuilding the emulator, re-opening it, re-measuring the grid, over and over.+2. **Blank terminals.** node-pty keeps **no scrollback** — the buffer lives in xterm, not the PTY. So a+ freshly created terminal starts empty and stays blank until the running program happens to repaint.+ For an idle shell, that might be never.++Both problems have the same root: tying the terminal's *lifetime* to a *view's* lifetime. The fix is to+decouple them.++## Technique 1 — A persistent terminal pool++Create **one** xterm terminal per PTY session and keep it alive for the whole app lifetime, independent+of any view:++- The terminal is opened into a **detached host element** and subscribes to its PTY's output stream+ **once**. Its buffer keeps filling whether or not it's currently on screen.+- A view doesn't create a terminal — it **borrows** one. On mount it re-parents the existing host+ element into itself; on unmount it lets go. The rendered content moves with the element, so the+ terminal is populated and visible *immediately*, with no repaint.++```ts+// borrow, don't build: move the live terminal's element into the view+function attachTerminal(entry, container) {+ container.appendChild(entry.host);+ if (!entry.opened) { entry.term.open(entry.host); entry.opened = true; } // open once, lazily+}+```++This single change fixes both naive problems: no per-switch teardown (no thrash) and no empty buffer+(no blank pane). It's the structural decision everything else builds on — the same pool pattern at the+center of [building a terminal UI with xterm.js and node-pty](/blog/building-a-terminal-ui-xterm-node-pty/).++## Technique 2 — Render only what's visible++A pool keeps every session's *buffer* current, but you don't need to *paint* every terminal every+frame. Only the terminal(s) actually on screen need to be laid out and rendered; the rest keep+accumulating output in memory at near-zero cost. When the user switches to one, it's already+up to date — you just attach its element.++The principle: **decouple "receiving output" from "drawing pixels."** Receiving is cheap and always-on;+drawing is the expensive part, so do it only for what's visible. A floor of thirty agents costs you the+rendering of the one or few panes you're looking at, not all thirty.++## Technique 3 — Bound your scrollback++xterm holds scrollback in memory, per terminal. A 10,000-line scrollback is comfortable for one+terminal and adds up fast across many. So:++- Pick a **generous but bounded** scrollback (enough to scroll back usefully, not infinite).+- Remember it's **per terminal** — multiply by your terminal count for the real memory cost.+- Tune to your workload: high-volume log streams want less retained history than an interactive shell.++It's a small knob with a real effect when you have dozens of buffers alive at once.++## Technique 4 — Stream efficiently from the PTY++The bytes have to cross from the main process (where node-pty lives) to the renderer (where xterm+lives) over IPC. A few habits keep that path cheap:++- **Per-session channels.** Each PTY forwards its output on its own channel so the renderer routes+ chunks to the right terminal without inspecting them.+- **Guard sends during teardown.** Killing a PTY fires its exit asynchronously; if the window is gone+ by then, sending throws. Check the target is alive before each send so quit stays clean (and you+ don't pay for sends into the void).+- **Let xterm batch.** xterm coalesces writes internally; feed it raw chunks and let it manage its own+ render scheduling rather than forcing a refresh per byte.++These come straight from how the [node-pty terminal plane](/blog/node-pty-electron-real-terminals/)+streams to the UI.++## Technique 5 — Reach for an accelerated renderer when needed++xterm's default DOM renderer is fine for a handful of terminals. When you're rendering many+high-throughput streams, an accelerated rendering addon (canvas/WebGL) moves the drawing off the DOM+and reduces CPU. The discipline: **measure first.** Add it when your real workload shows the DOM+renderer is the bottleneck — not preemptively, since for modest terminal counts it's unnecessary+complexity.++## The shape of a fast many-terminal UI++Put it together and the system looks like this:++1. **One persistent terminal per PTY**, in a pool, subscribed once.+2. **Only the visible terminals render**; the rest keep filling buffers cheaply.+3. **Bounded scrollback**, tuned per workload.+4. **Per-session IPC streaming** with teardown-safe sends.+5. **An accelerated renderer** added only when measurement says so.++That's how a single window can show a whole floor of live agents — each in its own real terminal,+switchable instantly — without the fans spinning up. The visual companion to all those terminals, the+office floor, follows the same keep-the-hot-loop-tight philosophy in+[visualizing AI agents with Pixi.js](/blog/visualizing-ai-agents-pixijs/).++## FAQ++**How many terminals can this handle?** Enough that your bottleneck becomes the agents and your RAM,+not the rendering. Because only visible terminals paint and buffers are cheap, the count you can keep+*alive* is far higher than the count you'd ever look at simultaneously.++**Do I lose output for terminals I'm not looking at?** No — that's the point of the pool. Off-screen+terminals still receive and buffer their PTY's output; you just aren't paying to draw them until you+switch over.++---++Munder Difflin streams [a whole floor of live Claude Code terminals](https://munderdiffl.in/#what) with this exact approach — a+persistent pool, visible-only rendering, and teardown-safe IPC. [Download Munder Difflin](https://munderdiffl.in/#install)+to run many real terminals without melting your CPU; it's free and open source.diff --git a/blog/src/posts/visualizing-ai-agents-pixijs.md b/blog/src/posts/visualizing-ai-agents-pixijs.mdnew file mode 100644index 00000000..09ba6f18--- /dev/null+++ b/blog/src/posts/visualizing-ai-agents-pixijs.md@@ -0,0 +1,150 @@+---+title: "Visualizing AI Agents with Pixi.js: An Office You Can Watch"+description: "How we render AI agents as avatars on a Pixi.js office floor — driven by real hook and message events, with seat assignment, pathing, and flying envelopes."+date: 2026-06-02+category: internals+categoryLabel: Internals+type: Technical+primaryKeyword: "ai agent visualization"+secondaryKeywords: ["pixijs", "agent observability", "ai agent visualization"]+tags: ["Internals", "Pixi.js", "Visualization", "Observability"]+author:+ name: Chaitanya Giri+ initials: CG+faq:+ - q: "Why visualize AI agents instead of just reading logs?"+ a: "A spatial view answers 'what is everyone doing right now?' at a glance — who's working, who's idle, who's blocked, and who's messaging whom. Logs answer that too, but slowly; a floor you can watch turns a multi-agent system from a wall of text into something you can supervise."+ - q: "Is the office floor a simulation or real activity?"+ a: "Real. Avatars move in response to actual Claude Code hook events (PreToolUse, PostToolUse, Stop) and real routed messages — not a scripted animation. When an avatar walks to the terminal station, an agent really did run a command."+ - q: "Why Pixi.js for an agent visualization?"+ a: "Pixi.js is a fast 2D WebGL renderer that handles many sprites smoothly, which suits a floor of avatars, tile maps, and animated envelopes. It gives game-quality rendering inside an Electron app without a heavy engine."+---++<div class="callout tldr"><span class="ic">TL;DR</span><p>An <strong>AI agent visualization</strong>+turns "what is the team doing?" into something you can see. Munder Difflin renders agents as avatars+on a <strong>Pixi.js</strong> office floor, driven by <strong>real events</strong>: Claude Code hooks+move an avatar to the station matching its current tool, and routed hive messages fly as envelopes+desk-to-desk. It's not a simulation — every motion reflects something an agent actually did.</p></div>++A multi-agent system is, by default, a wall of text: many terminals, many logs, no spatial sense of+who's doing what. The office floor exists to fix that — to make a hive *legible at a glance*. This post+is how it's built with Pixi.js, and the one principle that keeps it honest: it's driven by real+activity, not animation.++## Why a floor at all++The value of a visualization is answering "what is everyone doing right now?" instantly. On the floor:++- an avatar at the terminal station = an agent running a command,+- an avatar at the file shelf = an agent reading or writing,+- an avatar waving at the mailbox = an agent blocked, needing you,+- an avatar at its desk = idle, awaiting work,+- an envelope flying between desks = a message being routed.++You get the team's state in one look. That's not a gimmick — it's+[an office you can actually see](https://munderdiffl.in/#how), the visible counterpart to reading the+[event log](/blog/append-only-event-log-agents/).++## The Pixi.js setup++The renderer is a Pixi `Application`, tuned for crisp pixel art rather than smooth gradients:++```ts+const app = new Application();+await app.init({+ antialias: false, // hard pixel edges, no blur+ roundPixels: true, // snap sprites to integer positions+ resolution: 1+});+// nearest-neighbour scaling keeps the pixel art sharp when zoomed+texture.source.scaleMode = 'nearest';+```++On top of the stage sits a single `world` container that everything else lives in — the tile map, the+character layer, the envelopes. Putting it all under one container means the **camera** can pan and+zoom the entire scene by transforming just that one node, with map-edge clamping and a smooth lerp so+following a selected agent feels natural rather than jumpy.++One more structural choice: a **single shared ticker** drives all animation. Every frame, the ticker+hands out a delta-time and each live thing (avatars, envelopes, camera) advances by `dt`. One loop,+not dozens of timers — which keeps motion synchronized and the frame budget predictable.++## The floor is a Tiled map++The office isn't hand-coded geometry; it's a **Tiled** map (`.tmj`) — the same format 2D game+designers use. The renderer parses its layers:++- **Tile layers** (floor, walls, furniture) are drawn from tileset textures, resolved by each tile's+ global id.+- A **collision layer** marks which tiles are walls vs. walkable floor.+- A **spawn-points** object layer names the desks and seats.++Editing the office is therefore a design task, not a code change — open the map in Tiled, move a desk,+re-export. The renderer just reads it.++## Seats: giving each agent a desk++When an agent joins the hive, it needs a home. A small **seat pool** reserves the next free desk from+the map's spawn points and hands it to that agent; release it when the agent is gone. First-come,+first-served, one desk each — so a dynamically growing team fills the office in order instead of+piling onto one desk.++## Avatars: sprites driven by state++Each avatar is a sprite sliced from a character walk sheet. The frames are mapped into walk / type /+read / idle animations for three facing directions (left is just the right-facing row flipped), and a+small **leg-crop mask** tucks a seated agent under its desk so it reads as *sitting* rather than+standing on the furniture. The cast is recolored per character, so the team is visually+distinguishable at a glance.++But the sprites are just the *body*. What makes them meaningful is what drives them.++## The honest part: real events, not a script++This is the design principle that matters most. The avatars are **not** playing a canned animation —+they move in response to real signals from two sources:++**Claude Code hooks.** Each agent runs lifecycle hooks that report its activity. A `PreToolUse` event+("about to run a tool") sends the avatar walking toward the station that matches the tool — the+terminal station for a shell command, the file shelf for an edit. `PostToolUse` walks it back.+`Stop` drops it to idle at its desk; a notification sends it to the mailbox to wave for you. Because+these come straight from Claude, the floor reflects *real* tool use, with no fragile log-scraping. (The+hook lifecycle is covered in [Claude Code hooks, explained](/blog/claude-code-hooks-explained/).)++**Hive messages.** When the router delivers a message between agents, it emits an event the floor turns+into a flying **envelope** — a little pixel letter that arcs from the sender's desk to the recipient's,+tinted by the message's speech act (asks are cool-colored, agreements warm, refusals red, escalations+to a human a distinct color), with a small burst on arrival. You don't just know a message was sent;+you see *who asked whom, and what kind of ask it was*.++Getting from a station to a desk uses a tiny **breadth-first pathfinder** on the map's walkability+grid — short, predictable shortest paths around the furniture. Nothing fancier is needed at office+scale, and BFS on an unweighted grid gives the shortest route for free.++## Why this scales++A floor of avatars, a tile map, and animated envelopes is a lot of sprites — which is exactly what+Pixi.js is good at. The single-ticker design keeps the per-frame work bounded, the camera transforms+one container instead of every node, and the pixel-art settings (no antialias, nearest scaling) are+cheap to render. The same performance discipline shows up next door in+[rendering many live terminals](/blog/rendering-many-live-terminals-performance/) — keep the hot loop+tight and let the renderer do what it's fast at.++For the deeper game-dev mechanics — Tiled maps, pathfinding, sprite recoloring — see+[building an AI office floor](/blog/building-an-ai-office-floor/).++## FAQ++**Does the visualization slow the agents down?** No — the agents run as normal Claude Code processes;+the floor is a read-only consumer of events they already emit. Rendering happens in the UI, separate+from the agents' work.++**Can I still see the raw terminals?** Yes. The floor is one view; the actual xterm.js terminals are+another. The avatars tell you *what* at a glance; the terminals show you the byte-for-byte detail when+you want it.++---++Munder Difflin renders your hive of Claude Code agents as a live Pixi.js office floor — real avatars,+driven by real events, that you can actually watch. [Download Munder Difflin](https://munderdiffl.in/#install)+to see your agents at work; it's free and open source.