Skip to content

an open-source alternative to the dots, bots and muses of the world, run an office of claude code/codex like agents on your laptop, sandboxes or anywhere, uses your existing subscriptions

docs(research): v0.3.5 feature research — voice control, settings redesign, markdown preview, git visualization + QM survey

Chaitanya Giri5c02f5c89565
diff --git a/docs/research/v0.3.5-feature-research.md b/docs/research/v0.3.5-feature-research.md
new file mode 100644
index 00000000..12e7ac9f
--- /dev/null
+++ b/docs/research/v0.3.5-feature-research.md
@@ -0,0 +1,245 @@
+# v0.3.5 Feature Research — Voice Control · Settings · Markdown Preview · Git Visualization (+ QM survey)
+
+> Deep-research pass run 2026-08-06 against the v0.3.4 integration branch, by five parallel
+> research agents (code-grounded + web-verified). No code changes were made. File:line
+> references are against `integrate/v0.3.4`.
+
+---
+
+## 0. Cross-cutting findings
+
+- **`yc-software/qm` is real and relevant**: YC's open-sourced (MIT, 2026-07-31, ~11.6k★)
+ "multiplayer agent harness for work" — TypeScript/Fastify/Postgres server + Lit web UI,
+ pluggable harnesses (Claude Code, Codex, Pi, OpenCode), durable sandboxes, crons/watches,
+ scoped skills, three-tier security postures. **It has NO voice surface and NO git/diff UI**
+ — Munder Difflin is ahead of it on both. Adopt *patterns*, not modules (its stack doesn't
+ transplant into Electron/React).
+- **QM adoption shortlist** (tiered):
+ 1. *Security postures*: one Strict/Auto/Dangerous dial + a non-negotiable command deny-floor
+ (no recursive deletes / destructive SQL) enforced in every posture → maps to a Settings
+ "Autonomy" preset + a hive-level command filter.
+ 2. *Crons + watches*: condition-triggered agent runs (CI red, file changed, queue depth)
+ alongside schedules; steal the cron humanizer UX.
+ 3. *Skills manager UI*: per-agent/project skills viewable/editable/promotable to all agents.
+ 4. *Durability by default*: queues/tasks/audit must survive restart (SQLite, not memory).
+ 5. *Streaming-markdown pipeline*: streaming render + dedupe + lazy hljs/KaTeX (feeds feature 3).
+ 6. Deep-linkable UI state · conversation forking (pairs with worktrees) · per-scope resource
+ sidebar · single-source versioned JSONC config with a doctor validator (fixes our 3-file
+ HarnessConfig mirror pain) · untrusted-ingress prompt-injection screening (Browser/Slack).
+- **Security finding (fix regardless)**: `config:update` IPC (src/main/index.ts:2475) accepts an
+ arbitrary unvalidated `Partial<HarnessConfig>`; config carries secrets (groqApiKey,
+ slackBotToken, slackSigningSecret, webhookSecret) and dangerous keys (harnessHome). Any
+ voice/settings tooling must go through a main-side per-key allowlist — and the raw handler
+ deserves validation on its own.
+
+---
+
+## 1. Realtime Michael — deep context + full control (est. 6.5–8.5 days, 7 PRs)
+
+### Current state
+- `gpt-realtime-2` over WebRTC (`@openai/agents-realtime` 0.11.8), ephemeral token minted
+ main-side (src/main/realtime.ts:50-106). Persona ~950 tokens (renderer session.ts:78-106).
+- Context = ONE connect-time snapshot (~150-250 tokens: agent counts, doing/blocked counts,
+ last 6 log kinds) — **never refreshed**; only completion pushes + greeting inject mid-session.
+- 11 read tools (fleet/tasks/board/cost/schedules/config-allowlist/memory/activity/messages,
+ via IPC; `hive:agentDirectory` at index.ts:2885 is already the consolidated per-agent join).
+- 14 action verbs with a solid main-side safety spine (realtimeActions.ts): soft vs destructive
+ tiers, single pending slot + 120s TTL, distinct-token confirm, god/mass-target hard denylist,
+ `michael-voice` attribution + god cross-notify.
+
+### Gaps
+- Snapshot goes stale; no push on status/task/spawn changes; live store state (`Agent.action`,
+ queue depth) unused; every factual answer costs a tool round trip.
+- No voice path for: **resume** (pause is one-way by voice — sharpest gap), auto-delivery,
+ `/clear` context, restart/model change, settings, archive, delete task, create schedule,
+ tool gating, queue read/clear.
+- `/clear` has NO IPC — it's a renderer-store queued message (store.ts:220,677) delivered by
+ the idle-gated flush loop (useHive.ts) which zeroes the gauge on delivery.
+- `restartWithModel` is component-local in CommandCenterPanel — not callable elsewhere.
+
+### Realtime API facts (verified 2026-08, developers.openai.com)
+- GA: gpt-realtime-2.1 / 2.1-mini (128k context, 60-min sessions, reasoning-capable —
+ use `reasoning.effort: low` for voice). App pins gpt-realtime-2 (still fine; consider bump).
+- `session.update` CAN swap instructions/tools mid-session (voice locks after first audio;
+ instructions+tools ≤ 16,384 tokens) — **but it busts the prompt cache**. Caching is automatic
+ exact-prefix (≥1,024 tokens); cached audio input is ~99% off ($0.40 vs $32 /1M). Therefore:
+ **stable prefix + append-only conversation items**, never mutate instructions.
+- `conversation.item.create` injects messages mid-stream; SDK: `sendMessage(msg, {},
+ {triggerResponse:false})` silent injection, `updateAgent()`, out-of-band `response.create
+ {conversation:"none"}`. Parallel tool calls: NOT supported (sequential). No hard tool-count
+ cap — the token budget is the limit; ~30 crisp tools is fine. Mask tool latency with spoken
+ preambles; return compact flagged JSON, not prose.
+
+### Recommended architecture
+1. **Snapshot at connect + append-only deltas** (never touch instructions): compact floor table
+ (one line per agent: name|status|action|ctx%|breaker|inbox; ~400-600 tokens) injected as the
+ FIRST conversation item; a main-side **FloorDeltaWatcher** (sibling of
+ RealtimeCompletionWatcher) pushes coalesced deltas (status flips, task moves, spawn/archive,
+ breaker trips, ctx 70/90%) via the existing sanitized sendMessage path — debounced ≥10-15s,
+ ≤150 tokens each. Add `get_floor_state` reading `useStore.getState()` (zero IPC) returning
+ flagged JSON.
+2. **Extend the verb spine, keep narrow tools** (no free-text query_hive in v1):
+ - Soft: `resume_agent`, `set_auto_delivery`, `gate_tool`, `delete_task`, `get/clear_queue`.
+ - Destructive (echo-back + distinct token): `clear_context` (main confirms → renderer
+ `enqueueMessage('/clear')` so ALL existing delivery gates apply), `restart_agent` (extract
+ restartWithModel into a shared renderer service first), `archive_agent`, `create_schedule`,
+ `update_setting`.
+ - `update_setting` policy table (main-side): forbidden = harnessHome + all secret keys +
+ providerBaseUrls + integrations; soft = notifications/themes/freeflow/idle-disconnect/
+ strongKeepalive; confirm = autoMode/defaultModel/god engine/worker caps/breaker/slack/
+ webhook/maxTurns. Validate type+range; echo old→new in the confirm.
+3. **Latency budget**: persona+30 tools ~5k tokens cached (~free); snapshot once; deltas
+ ~1-2k/hr; most "what's happening" answers need ZERO round trips; detail = 1 round trip
+ masked by a preamble line added to the persona.
+
+### PR sequence
+1 snapshot+get_floor_state (0.5-1d) · 2 FloorDeltaWatcher (1-1.5d) · 3 soft verbs (1-1.5d) ·
+4 clear_context + queue tools (0.5-1d) · 5 update_setting policy table (1d) ·
+6 restart_agent service extraction (1.5-2d, highest risk) · 7 polish/model bump (0.5d).
+
+---
+
+## 2. Settings redesign (est. 3–4 days, 4 phases)
+
+### Problems found (full inventory in agent report; SettingsModal is 1,428 lines)
+- "General" is six unrelated domains; dead read-only rows (Auto mode / Semantic memory /
+ Command are displayed but not editable — autoMode has NO editor anywhere post-onboarding).
+- Onboarding-trapped settings: `strongKeepalive`, `audience`, open-at-login — set once,
+ unreachable forever.
+- **`defaultModel` — the most consequential setting — has no UI at all**; ditto maxTurns,
+ 4 of 6 circuit-breaker fields (incl. hardStop, a safety decision).
+- Duplication: auto-compact toggle in General AND Schedules; semanticMemory shown in General,
+ toggled in MemoryPanel; theme forks into localStorage + terminalTheme.
+- Secrets scattered across 5 entry UIs; OpenAI key in "AI Engines" silently gates Talk in
+ "Integrations".
+- **Default-drift bugs in config.ts**: knowledgeGraph `enabled:true` contradicts its own
+ "Default OFF" comments; SettingsModal seeds freeflowEnabled `?? false` while default is true
+ (UI shows OFF when ON); multiWindow code=true vs comment="OFF by default"; legacy `costCapUsd`
+ still enforced — delete it.
+
+### Proposed IA — 6 tabs, simple-by-default + per-tab Advanced disclosure + header search
+1. **General** — home folder + reset row (Danger Zone absorbed), appearance (ONE theme control;
+ office-theme picker absorbs tvShowOffices), notifications, keep-awake, open-at-login,
+ explain-simply (audience), auto-update. Adv: multiWindow.
+2. **Agents & Models** — default agent model (NEW UI), Michael's engine (shared component with
+ Command Center), BYOK keys with "what each key unlocks" note. Adv: per-CLI base URLs +
+ default models, defaultCommand, maxTurns.
+3. **Autonomy & Budgets** (QM-style) — Ask-first/Autonomous preset (autoMode), full circuit
+ breaker as one unit (give the 4 hidden fields UI), floor budget + velocity. Adv: worker
+ fleet knobs. Delete costCapUsd.
+4. **Connections** — MCP tiers, Slack, Webhook, REST registry; setup guides become collapsible
+ flows; ports under Advanced.
+5. **Voice** — Free Flow (toggle + Groq key) + Realtime Michael (key status w/ jump-link,
+ devices, idle disconnect, cost HUD).
+6. **Memory & Knowledge** — semantic memory + embedding model, Knowledge Graph. Adv: reflection
+ enabled+interval (leave 4 threshold fields config-only).
+
+Phases: extract SettingRow/Section primitives + split the modal (0.5d) → new nav + move blocks +
+missing editors, all via existing `updateConfig` (1d) → Voice tab + collapsible setup + cleanup
+(1d) → search + QA (0.5-1d). Watch the 3-file HarnessConfig mirror on every field change; the
+15-state rehydration useEffect is the main regression risk.
+
+---
+
+## 3. Markdown preview — IDE + terminal click (est. 3–5 days core, 3 PRs + polish)
+
+### Current state
+- IDE (IdePanel.tsx): tabs `{key, rel, mode: 'edit'|'diff'}`; `languageForPath` already maps
+ markdown; Monaco fully self-hosted (CSP precedent). **No way to open the IDE "at a file".**
+- **Reusable overlay exists**: `fullscreenFilePath` store field → `FullscreenFileEditor`
+ (z 280 — already above fullscreen terminal 250, below IDE 290), resolves ABSOLUTE paths
+ incl. outside-workspace, renders CodeMirror (lang-markdown already wired).
+- **No terminal link plumbing at all** — no WebLinksAddon, no registerLinkProvider; xterm 5.5's
+ `ILinkProvider` API is available; agents in store carry cwd+ptyId for relative resolution.
+- CSP blocks remote images (a feature for untrusted md).
+
+### Recommendation: `react-markdown@10 + remark-gfm@4` (~45-55 kB min+gz)
+Safe **by construction** for untrusted agent output — renders to React elements, no HTML sink,
+no rehype-raw, default urlTransform drops `javascript:`. QM uses marked+DOMPurify with a fused
+postprocess-sanitize hook (`plugins/web-ui/src/markdown-sanitize.ts`) — validated fallback
+pattern, but react-markdown fits a React app better (link interception, tokens theming,
+position-based scroll sync). Phase 2: rehype-highlight; mermaid only as a lazy bundled chunk
+with `securityLevel:'strict'`. Reject @uiw/react-markdown-preview (enables rehype-raw, forces
+GitHub look, 3.6 MB).
+
+### Workflows
+- **Shared**: `markdown/MarkdownPreview.tsx` — CTH-token styled (light+dark), every anchor
+ intercepted: http(s)/mailto → `app:openExternal` (add scheme allowlist), relative .md →
+ open-in-context, all else inert.
+- **A. IDE**: per-markdown-tab `view: 'code'|'split'|'preview'` (default split, persisted);
+ three-segment toggle in the EditorBar (only for md); Cmd+Shift+V cycles; preview reads the
+ live edit buffer via `useDeferredValue` — save flow untouched; md links open sibling tabs.
+- **B. Terminal**: custom `ILinkProvider` registered once in `acquireTerminal()` (covers all
+ three terminal surfaces automatically); regex for absolute/~/relative *.md (+ tolerant of
+ quotes/backticks/`:line`); resolve vs owning agent's cwd; verify existence via ONE new
+ `fs:stat` IPC (mirror in preload + index.d.ts — 3-file gotcha); **Cmd/Ctrl+click** →
+ `setFullscreenFile(abs, {view:'preview'})` → FullscreenFileEditor gains an edit|preview
+ toggle (Obsidian-style; preview default from terminal); "open in IDE" escalation via a new
+ `ideInitialFile` store field consumed on IDE mount. FilesTab's narrow editor gets the same
+ toggle.
+- **Security**: no HTML sink; anchors never navigate the window; terminal-derived paths flow
+ through safeJoin + 2 MB cap + binary sniff, opened only on explicit modifier-click.
+
+PRs: 1 MarkdownPreview + IDE split (1-2d) · 2 overlay + FilesTab toggles (0.5-1d) ·
+3 link provider + fs:stat + tests (1-2d) · 4 polish: scroll sync, highlight, mermaid (1-2d).
+
+---
+
+## 4. Git visualization (est. 6–8 days, 4 PRs)
+
+### Striking finding: ~40% is already built and UNMOUNTED
+- `GitTab.tsx` — complete branch/status/log panel with commit graph — **orphaned** (nothing
+ imports it since the tab slim-down).
+- `git/CommitGraph.tsx` — themed wrapper over the **`commit-graph` npm dep already in
+ package.json** — display-only today (onCommitClick/getDiff props unused).
+- `git/graph.ts` — hand-written DAG lane layout — also unused (documented fallback).
+- `git.ts` already returns full DAG topology (`getLog` emits hash+parents+refs) and contains
+ the exact dirty/ahead safety logic a safe checkout needs (`worktreeHasUnintegratedWork`).
+- Missing entirely: per-commit diffs, branch compare, checkout, graph interactivity.
+
+### Library verdict
+`[email protected]` (DoltHub; active, React 18, permissive, 323 kB) is best-in-class and takes
+exactly our data shape; keep `graph.ts` as the custom-SVG fallback if its popups fight the
+theme. @gitgraph/* is dead; vscode Git Graph's license PROHIBITS reuse (approach only);
+isomorphic-git unnecessary (we shell out); no diff library needed — Monaco DiffEditor does it.
+QM has no git UI to borrow.
+
+### Architecture
+- **Data (PR1)**: `getLogGraph` (+`--topo-order`, `--skip` pagination, run at mainRepoRoot so
+ all agent/<id> worktree branches appear) · `getCommitFiles` (diff-tree -z -M --name-status,
+ handle R/C renames) · `getFileAtRev` (generalize `show HEAD:` to any rev; keep 2 MB cap +
+ binary sniff; per-commit diff = sha^ vs sha into MonacoDiff) · `compareRefs` (three-dot
+ default w/ two-dot toggle + left-right counts + merge-base) · `listWorktrees` · guarded
+ `checkout`. Strict rev regex + `--` separation (renderer-supplied refs). New IPC beside
+ index.ts:2570; update renderer type mirrors.
+- **UI (PR2/3)**: IDE left rail becomes **CHANGES · HISTORY · COMPARE** sub-tabs. HISTORY:
+ mount CommitGraph with onCommitClick → file list (reuse CHANGES row styling) → Monaco diff
+ tabs keyed `commit::sha::path`; infinite scroll past 200. COMPARE: base/head dropdowns +
+ swap + ahead/behind counts + dot-mode toggle → file list → diff tabs. Checkout via context
+ menu on branch chips/commit rows; detached-HEAD affordance exists in GitTab. Re-mount slim
+ GitTab as a 4th sidebar tab (one-line mount) for at-a-glance status.
+- **Checkout safety (main-process, non-negotiable)**: clean tree only (no auto-stash v1);
+ owning agent must be idle — and gate on ANY agent whose cwd resolves to the same root
+ (the god-at-mainRepoRoot sneaky case); branch-held-by-worktree errors mapped to "checked out
+ by agent X"; fail-safe on any query error.
+- **Risks**: big repos (--all --topo-order walk; 8s runGit timeout is the backstop; --skip
+ re-walks — fine to ~50k commits); do NOT add graph queries to the 4s polling loops (refresh
+ on tab activation/explicit/status change); commit-graph popup styling may need the
+ global.css override block extended.
+
+PRs: 1 data layer (1-1.5d) · 2 HISTORY (2-3d) · 3 COMPARE + checkout UX (2d) · 4 polish +
+GitTab remount (1d).
+
+---
+
+## 5. Suggested build order (if all four are greenlit)
+
+1. **Markdown PR1-2** (fast, visible, zero risk) → **Git PR1 + mount HISTORY** (the unmounted
+ 40% makes this the best value-per-day in the codebase).
+2. **Voice PR1-2** (context wins — biggest UX delta for talk mode) → Voice PR3-5 (control).
+3. **Settings phases 1-3** (independent; can interleave; fixes real config bugs).
+4. Long tail: Git COMPARE/checkout, Voice restart_agent, markdown polish, Settings search.
+5. Cross-cutting from QM (separate track): autonomy postures + command deny-floor,
+ watches, skills manager, config single-sourcing.
+
+Grand total if everything ships: ~19–26 dev-days.