an open-source alternative to the dots, bots and muses of the world, run an office of claude code/codex like agents on your laptop, sandboxes or anywhere, uses your existing subscriptions
HTML agent-harnessagent-orchestrationagentsai-agentsautonomous-agentsclaude-codecodexcoding-agentsdesktop-appdeveloper-toolselectrongemini-cliharnessharness-engineeringlocal-firstmemorymulti-agentopencodeorchestrationtypescript
docs(research): v0.3.5 feature research — voice control, settings redesign, markdown preview, git visualization + QM survey
Chaitanya Giri5c02f5c89565
diff --git a/docs/research/v0.3.5-feature-research.md b/docs/research/v0.3.5-feature-research.mdnew file mode 100644index 00000000..12e7ac9f--- /dev/null+++ b/docs/research/v0.3.5-feature-research.md@@ -0,0 +1,245 @@+# v0.3.5 Feature Research — Voice Control · Settings · Markdown Preview · Git Visualization (+ QM survey)++> Deep-research pass run 2026-08-06 against the v0.3.4 integration branch, by five parallel+> research agents (code-grounded + web-verified). No code changes were made. File:line+> references are against `integrate/v0.3.4`.++---++## 0. Cross-cutting findings++- **`yc-software/qm` is real and relevant**: YC's open-sourced (MIT, 2026-07-31, ~11.6k★)+ "multiplayer agent harness for work" — TypeScript/Fastify/Postgres server + Lit web UI,+ pluggable harnesses (Claude Code, Codex, Pi, OpenCode), durable sandboxes, crons/watches,+ scoped skills, three-tier security postures. **It has NO voice surface and NO git/diff UI**+ — Munder Difflin is ahead of it on both. Adopt *patterns*, not modules (its stack doesn't+ transplant into Electron/React).+- **QM adoption shortlist** (tiered):+ 1. *Security postures*: one Strict/Auto/Dangerous dial + a non-negotiable command deny-floor+ (no recursive deletes / destructive SQL) enforced in every posture → maps to a Settings+ "Autonomy" preset + a hive-level command filter.+ 2. *Crons + watches*: condition-triggered agent runs (CI red, file changed, queue depth)+ alongside schedules; steal the cron humanizer UX.+ 3. *Skills manager UI*: per-agent/project skills viewable/editable/promotable to all agents.+ 4. *Durability by default*: queues/tasks/audit must survive restart (SQLite, not memory).+ 5. *Streaming-markdown pipeline*: streaming render + dedupe + lazy hljs/KaTeX (feeds feature 3).+ 6. Deep-linkable UI state · conversation forking (pairs with worktrees) · per-scope resource+ sidebar · single-source versioned JSONC config with a doctor validator (fixes our 3-file+ HarnessConfig mirror pain) · untrusted-ingress prompt-injection screening (Browser/Slack).+- **Security finding (fix regardless)**: `config:update` IPC (src/main/index.ts:2475) accepts an+ arbitrary unvalidated `Partial<HarnessConfig>`; config carries secrets (groqApiKey,+ slackBotToken, slackSigningSecret, webhookSecret) and dangerous keys (harnessHome). Any+ voice/settings tooling must go through a main-side per-key allowlist — and the raw handler+ deserves validation on its own.++---++## 1. Realtime Michael — deep context + full control (est. 6.5–8.5 days, 7 PRs)++### Current state+- `gpt-realtime-2` over WebRTC (`@openai/agents-realtime` 0.11.8), ephemeral token minted+ main-side (src/main/realtime.ts:50-106). Persona ~950 tokens (renderer session.ts:78-106).+- Context = ONE connect-time snapshot (~150-250 tokens: agent counts, doing/blocked counts,+ last 6 log kinds) — **never refreshed**; only completion pushes + greeting inject mid-session.+- 11 read tools (fleet/tasks/board/cost/schedules/config-allowlist/memory/activity/messages,+ via IPC; `hive:agentDirectory` at index.ts:2885 is already the consolidated per-agent join).+- 14 action verbs with a solid main-side safety spine (realtimeActions.ts): soft vs destructive+ tiers, single pending slot + 120s TTL, distinct-token confirm, god/mass-target hard denylist,+ `michael-voice` attribution + god cross-notify.++### Gaps+- Snapshot goes stale; no push on status/task/spawn changes; live store state (`Agent.action`,+ queue depth) unused; every factual answer costs a tool round trip.+- No voice path for: **resume** (pause is one-way by voice — sharpest gap), auto-delivery,+ `/clear` context, restart/model change, settings, archive, delete task, create schedule,+ tool gating, queue read/clear.+- `/clear` has NO IPC — it's a renderer-store queued message (store.ts:220,677) delivered by+ the idle-gated flush loop (useHive.ts) which zeroes the gauge on delivery.+- `restartWithModel` is component-local in CommandCenterPanel — not callable elsewhere.++### Realtime API facts (verified 2026-08, developers.openai.com)+- GA: gpt-realtime-2.1 / 2.1-mini (128k context, 60-min sessions, reasoning-capable —+ use `reasoning.effort: low` for voice). App pins gpt-realtime-2 (still fine; consider bump).+- `session.update` CAN swap instructions/tools mid-session (voice locks after first audio;+ instructions+tools ≤ 16,384 tokens) — **but it busts the prompt cache**. Caching is automatic+ exact-prefix (≥1,024 tokens); cached audio input is ~99% off ($0.40 vs $32 /1M). Therefore:+ **stable prefix + append-only conversation items**, never mutate instructions.+- `conversation.item.create` injects messages mid-stream; SDK: `sendMessage(msg, {},+ {triggerResponse:false})` silent injection, `updateAgent()`, out-of-band `response.create+ {conversation:"none"}`. Parallel tool calls: NOT supported (sequential). No hard tool-count+ cap — the token budget is the limit; ~30 crisp tools is fine. Mask tool latency with spoken+ preambles; return compact flagged JSON, not prose.++### Recommended architecture+1. **Snapshot at connect + append-only deltas** (never touch instructions): compact floor table+ (one line per agent: name|status|action|ctx%|breaker|inbox; ~400-600 tokens) injected as the+ FIRST conversation item; a main-side **FloorDeltaWatcher** (sibling of+ RealtimeCompletionWatcher) pushes coalesced deltas (status flips, task moves, spawn/archive,+ breaker trips, ctx 70/90%) via the existing sanitized sendMessage path — debounced ≥10-15s,+ ≤150 tokens each. Add `get_floor_state` reading `useStore.getState()` (zero IPC) returning+ flagged JSON.+2. **Extend the verb spine, keep narrow tools** (no free-text query_hive in v1):+ - Soft: `resume_agent`, `set_auto_delivery`, `gate_tool`, `delete_task`, `get/clear_queue`.+ - Destructive (echo-back + distinct token): `clear_context` (main confirms → renderer+ `enqueueMessage('/clear')` so ALL existing delivery gates apply), `restart_agent` (extract+ restartWithModel into a shared renderer service first), `archive_agent`, `create_schedule`,+ `update_setting`.+ - `update_setting` policy table (main-side): forbidden = harnessHome + all secret keys ++ providerBaseUrls + integrations; soft = notifications/themes/freeflow/idle-disconnect/+ strongKeepalive; confirm = autoMode/defaultModel/god engine/worker caps/breaker/slack/+ webhook/maxTurns. Validate type+range; echo old→new in the confirm.+3. **Latency budget**: persona+30 tools ~5k tokens cached (~free); snapshot once; deltas+ ~1-2k/hr; most "what's happening" answers need ZERO round trips; detail = 1 round trip+ masked by a preamble line added to the persona.++### PR sequence+1 snapshot+get_floor_state (0.5-1d) · 2 FloorDeltaWatcher (1-1.5d) · 3 soft verbs (1-1.5d) ·+4 clear_context + queue tools (0.5-1d) · 5 update_setting policy table (1d) ·+6 restart_agent service extraction (1.5-2d, highest risk) · 7 polish/model bump (0.5d).++---++## 2. Settings redesign (est. 3–4 days, 4 phases)++### Problems found (full inventory in agent report; SettingsModal is 1,428 lines)+- "General" is six unrelated domains; dead read-only rows (Auto mode / Semantic memory /+ Command are displayed but not editable — autoMode has NO editor anywhere post-onboarding).+- Onboarding-trapped settings: `strongKeepalive`, `audience`, open-at-login — set once,+ unreachable forever.+- **`defaultModel` — the most consequential setting — has no UI at all**; ditto maxTurns,+ 4 of 6 circuit-breaker fields (incl. hardStop, a safety decision).+- Duplication: auto-compact toggle in General AND Schedules; semanticMemory shown in General,+ toggled in MemoryPanel; theme forks into localStorage + terminalTheme.+- Secrets scattered across 5 entry UIs; OpenAI key in "AI Engines" silently gates Talk in+ "Integrations".+- **Default-drift bugs in config.ts**: knowledgeGraph `enabled:true` contradicts its own+ "Default OFF" comments; SettingsModal seeds freeflowEnabled `?? false` while default is true+ (UI shows OFF when ON); multiWindow code=true vs comment="OFF by default"; legacy `costCapUsd`+ still enforced — delete it.++### Proposed IA — 6 tabs, simple-by-default + per-tab Advanced disclosure + header search+1. **General** — home folder + reset row (Danger Zone absorbed), appearance (ONE theme control;+ office-theme picker absorbs tvShowOffices), notifications, keep-awake, open-at-login,+ explain-simply (audience), auto-update. Adv: multiWindow.+2. **Agents & Models** — default agent model (NEW UI), Michael's engine (shared component with+ Command Center), BYOK keys with "what each key unlocks" note. Adv: per-CLI base URLs ++ default models, defaultCommand, maxTurns.+3. **Autonomy & Budgets** (QM-style) — Ask-first/Autonomous preset (autoMode), full circuit+ breaker as one unit (give the 4 hidden fields UI), floor budget + velocity. Adv: worker+ fleet knobs. Delete costCapUsd.+4. **Connections** — MCP tiers, Slack, Webhook, REST registry; setup guides become collapsible+ flows; ports under Advanced.+5. **Voice** — Free Flow (toggle + Groq key) + Realtime Michael (key status w/ jump-link,+ devices, idle disconnect, cost HUD).+6. **Memory & Knowledge** — semantic memory + embedding model, Knowledge Graph. Adv: reflection+ enabled+interval (leave 4 threshold fields config-only).++Phases: extract SettingRow/Section primitives + split the modal (0.5d) → new nav + move blocks ++missing editors, all via existing `updateConfig` (1d) → Voice tab + collapsible setup + cleanup+(1d) → search + QA (0.5-1d). Watch the 3-file HarnessConfig mirror on every field change; the+15-state rehydration useEffect is the main regression risk.++---++## 3. Markdown preview — IDE + terminal click (est. 3–5 days core, 3 PRs + polish)++### Current state+- IDE (IdePanel.tsx): tabs `{key, rel, mode: 'edit'|'diff'}`; `languageForPath` already maps+ markdown; Monaco fully self-hosted (CSP precedent). **No way to open the IDE "at a file".**+- **Reusable overlay exists**: `fullscreenFilePath` store field → `FullscreenFileEditor`+ (z 280 — already above fullscreen terminal 250, below IDE 290), resolves ABSOLUTE paths+ incl. outside-workspace, renders CodeMirror (lang-markdown already wired).+- **No terminal link plumbing at all** — no WebLinksAddon, no registerLinkProvider; xterm 5.5's+ `ILinkProvider` API is available; agents in store carry cwd+ptyId for relative resolution.+- CSP blocks remote images (a feature for untrusted md).++### Recommendation: `react-markdown@10 + remark-gfm@4` (~45-55 kB min+gz)+Safe **by construction** for untrusted agent output — renders to React elements, no HTML sink,+no rehype-raw, default urlTransform drops `javascript:`. QM uses marked+DOMPurify with a fused+postprocess-sanitize hook (`plugins/web-ui/src/markdown-sanitize.ts`) — validated fallback+pattern, but react-markdown fits a React app better (link interception, tokens theming,+position-based scroll sync). Phase 2: rehype-highlight; mermaid only as a lazy bundled chunk+with `securityLevel:'strict'`. Reject @uiw/react-markdown-preview (enables rehype-raw, forces+GitHub look, 3.6 MB).++### Workflows+- **Shared**: `markdown/MarkdownPreview.tsx` — CTH-token styled (light+dark), every anchor+ intercepted: http(s)/mailto → `app:openExternal` (add scheme allowlist), relative .md →+ open-in-context, all else inert.+- **A. IDE**: per-markdown-tab `view: 'code'|'split'|'preview'` (default split, persisted);+ three-segment toggle in the EditorBar (only for md); Cmd+Shift+V cycles; preview reads the+ live edit buffer via `useDeferredValue` — save flow untouched; md links open sibling tabs.+- **B. Terminal**: custom `ILinkProvider` registered once in `acquireTerminal()` (covers all+ three terminal surfaces automatically); regex for absolute/~/relative *.md (+ tolerant of+ quotes/backticks/`:line`); resolve vs owning agent's cwd; verify existence via ONE new+ `fs:stat` IPC (mirror in preload + index.d.ts — 3-file gotcha); **Cmd/Ctrl+click** →+ `setFullscreenFile(abs, {view:'preview'})` → FullscreenFileEditor gains an edit|preview+ toggle (Obsidian-style; preview default from terminal); "open in IDE" escalation via a new+ `ideInitialFile` store field consumed on IDE mount. FilesTab's narrow editor gets the same+ toggle.+- **Security**: no HTML sink; anchors never navigate the window; terminal-derived paths flow+ through safeJoin + 2 MB cap + binary sniff, opened only on explicit modifier-click.++PRs: 1 MarkdownPreview + IDE split (1-2d) · 2 overlay + FilesTab toggles (0.5-1d) ·+3 link provider + fs:stat + tests (1-2d) · 4 polish: scroll sync, highlight, mermaid (1-2d).++---++## 4. Git visualization (est. 6–8 days, 4 PRs)++### Striking finding: ~40% is already built and UNMOUNTED+- `GitTab.tsx` — complete branch/status/log panel with commit graph — **orphaned** (nothing+ imports it since the tab slim-down).+- `git/CommitGraph.tsx` — themed wrapper over the **`commit-graph` npm dep already in+ package.json** — display-only today (onCommitClick/getDiff props unused).+- `git/graph.ts` — hand-written DAG lane layout — also unused (documented fallback).+- `git.ts` already returns full DAG topology (`getLog` emits hash+parents+refs) and contains+ the exact dirty/ahead safety logic a safe checkout needs (`worktreeHasUnintegratedWork`).+- Missing entirely: per-commit diffs, branch compare, checkout, graph interactivity.++### Library verdict+`[email protected]` (DoltHub; active, React 18, permissive, 323 kB) is best-in-class and takes+exactly our data shape; keep `graph.ts` as the custom-SVG fallback if its popups fight the+theme. @gitgraph/* is dead; vscode Git Graph's license PROHIBITS reuse (approach only);+isomorphic-git unnecessary (we shell out); no diff library needed — Monaco DiffEditor does it.+QM has no git UI to borrow.++### Architecture+- **Data (PR1)**: `getLogGraph` (+`--topo-order`, `--skip` pagination, run at mainRepoRoot so+ all agent/<id> worktree branches appear) · `getCommitFiles` (diff-tree -z -M --name-status,+ handle R/C renames) · `getFileAtRev` (generalize `show HEAD:` to any rev; keep 2 MB cap ++ binary sniff; per-commit diff = sha^ vs sha into MonacoDiff) · `compareRefs` (three-dot+ default w/ two-dot toggle + left-right counts + merge-base) · `listWorktrees` · guarded+ `checkout`. Strict rev regex + `--` separation (renderer-supplied refs). New IPC beside+ index.ts:2570; update renderer type mirrors.+- **UI (PR2/3)**: IDE left rail becomes **CHANGES · HISTORY · COMPARE** sub-tabs. HISTORY:+ mount CommitGraph with onCommitClick → file list (reuse CHANGES row styling) → Monaco diff+ tabs keyed `commit::sha::path`; infinite scroll past 200. COMPARE: base/head dropdowns ++ swap + ahead/behind counts + dot-mode toggle → file list → diff tabs. Checkout via context+ menu on branch chips/commit rows; detached-HEAD affordance exists in GitTab. Re-mount slim+ GitTab as a 4th sidebar tab (one-line mount) for at-a-glance status.+- **Checkout safety (main-process, non-negotiable)**: clean tree only (no auto-stash v1);+ owning agent must be idle — and gate on ANY agent whose cwd resolves to the same root+ (the god-at-mainRepoRoot sneaky case); branch-held-by-worktree errors mapped to "checked out+ by agent X"; fail-safe on any query error.+- **Risks**: big repos (--all --topo-order walk; 8s runGit timeout is the backstop; --skip+ re-walks — fine to ~50k commits); do NOT add graph queries to the 4s polling loops (refresh+ on tab activation/explicit/status change); commit-graph popup styling may need the+ global.css override block extended.++PRs: 1 data layer (1-1.5d) · 2 HISTORY (2-3d) · 3 COMPARE + checkout UX (2d) · 4 polish ++GitTab remount (1d).++---++## 5. Suggested build order (if all four are greenlit)++1. **Markdown PR1-2** (fast, visible, zero risk) → **Git PR1 + mount HISTORY** (the unmounted+ 40% makes this the best value-per-day in the codebase).+2. **Voice PR1-2** (context wins — biggest UX delta for talk mode) → Voice PR3-5 (control).+3. **Settings phases 1-3** (independent; can interleave; fixes real config bugs).+4. Long tail: Git COMPARE/checkout, Voice restart_agent, markdown polish, Settings search.+5. Cross-cutting from QM (separate track): autonomy postures + command deny-floor,+ watches, skills manager, config single-sourcing.++Grand total if everything ships: ~19–26 dev-days.