Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Chat Agent Activity Panel (intermediate progress in Web Chat)
Status: approved design, implementation dispatched (2026-07-17) Owner decisions locked: global toggle in Settings › Messaging; history turns included; assistant interim text shown in full; tool calls summary-first with expandable detail; no token-level streaming; no IM-side changes.
Background
Research-desk users want to watch what the agent is doing during a turn:
intermediate assistant messages and tool calls. Today the Web Chat shows only a
three-dot ThinkingBubble while working, and the reply arrives atomically.
Key fact (verified 2026-07-17): the data pipeline already exists end-to-end.
All three backends emit intermediate assistant / toolcall via
controller.emit_agent_message → core/message_mirror.py persist_agent_message
persists them (assistant → messages table; toolcall → agent_events table
with turn_id/sequence/content_json) → message.new is published on the bus
→ SSE /api/events reaches the browser. The frontend currently drops these
rows in isTranscriptMessage (ui/src/components/workbench/ChatPage.tsx).
This feature is therefore: group-and-render what is already flowing + one
settings toggle + a small history read endpoint. Default off = exact current UI.
Design source of truth
- design.pen frames (avibe-docs
design.pen):BoX4o"Chat · Agent 执行过程 Activity Panel (P1)" spec board (states A–F + anatomy + interaction rules) ands3SZtfull-page in-context mock. Exported PNGs for implementation reference:~/vibe-remote-project/_tmp/activity-design/BoX4o.pngands3SZt.png. - Match the existing chat visual language: message avatar block,
bg-mint/[0.09]-family tints, muted borders, JetBrains Mono for timings/metadata. Map to existing tokens/classes; no hardcoded one-off colors.
UX specification
One new transcript row kind: ActivityGroup — one per agent turn, positioned where that turn's reply renders (directly above the terminal message).
States:
- Running · compact (default) — replaces ThinkingBubble when the toggle is on and the running turn has ≥1 activity row (falls back to ThinkingBubble while empty). Card = header + fixed-height viewport (~3 rows, ≈110px): header: spinner + "Running" + step count + elapsed + chevron; body: newest rows scroll in from the bottom, older rows fade upward; auto-follows the tail. The card itself never grows: no transcript reflow, existing scroll anchoring untouched.
- Running · expanded — click header: max-height ≈40vh, internal scroll, auto-follow latest; manual scroll-up pauses follow and shows a "jump to latest" pill (reuse Transcript's existing jump logic/pattern). Stays expanded for the rest of the turn until collapsed.
- Done · collapsed (default) — on terminal
result: collapse to a single chip "✓ Activity · N steps · 1m 23s" hugging the final reply from above. Click to expand/collapse anytime. Expansion state is not persisted. - History turns — same chip on past turns (including after refresh). Detail is lazy-loaded on first expand; never loaded for the initial transcript render.
- Interrupted / failed — chip variants: "Interrupted · stopped at step N" (gold icon) when the turn ends without a terminal result (Stop/override); "Failed · N steps" (destructive icon) on error result. Error details keep flowing through the existing error bubble — never into the activity panel.
Event rows:
- Tool row: icon (by tool-name prefix: terminal/file-text/pencil/globe/bot,
fallback wrench) + tool name (mono) + one-line summary + duration when
available. Summary = first line of the already-formatted toolcall text
(
format_toolcalloutput) — never raw tool input/output dumps. Click row → expand/collapse inline detail block showing the full stored toolcall text (mono, fromagent_events.content_json). v1 does not read native transcripts; if only the summary line was stored, that is what shows. - Assistant interim row: sparkles icon + full text (owner decision —
users read the content). Wrapped, rendered with the existing
Markdowncomponent in a compact style (smaller font); no truncation. In the compact running viewport the fixed window simply follows the tail. Narration that the Web transcript already draws as a mutedinterimbubble (owner decision, 2026-10-02) is not repeated here: itsassistantrow carriesmetadata.transcript_copy = "interim", whichactivity_role_formaps to rolenonefor both the durable groups and the live stream. Short narration below the bubble threshold and process-only rows stay in the panel.
Grouping rules:
- Anchor by turn boundaries, not by user messages: a group collects
assistant/tool_callrows between the previous terminal row and the next terminal (result/error) row; live turn = rows arriving whileworking. This also covers agent-initiated turns (no inbound user message) and queued turns (each turn groups independently). - Reconnect/refresh mid-turn: persisted interim rows of the running turn re-hydrate the group; empty group falls back to ThinkingBubble.
Settings:
config.ui.show_agent_activity(bool, defaultfalse), server-persisted via the existingconfig.ui.*path (same aschat_message_font_size), edited in Settings › Messaging withSettingsRow+ToggleSwitch.- Independent from IM
agent_progress_style/ status bubble — no IM changes.
Out of scope (v1): token-level streaming (all three backends' delta handlers are unimplemented), drill-down beyond the stored toolcall text, IM platforms, per-chat override toggle, persistence of expand/collapse state.
Implementation notes (grounded pointers, verify before relying)
Frontend (ui/src):
components/workbench/ChatPage.tsx:isTranscriptMessagecurrently filtersassistant/tool_call;Transcript/MessageRow/ThinkingBubble;workingstate; jump-to-latest + scroll-anchor logic;MAX_RETAINED_MESSAGES = 300(mind the retained-window interplay — activity rows must not silently evict visible conversation rows; group or cap accordingly).- SSE:
context/ApiContext.tsxconnectWorkbenchEvents(message.new,turn.start,turn.end,session.status). - Settings page:
components/settings/SettingsMessagingPage.tsx, primitives incomponents/settings/SettingsPrimitives.tsx. - i18n: every new string in both
ui/src/i18n/en.jsonandzh.json. - Collapse pattern precedent: the harness chip branch in
MessageRow.
Backend:
- Verify early (evidence-first, run it, don't assume): whether
tool_callrows currently arrive at the browser viamessage.newand/or the transcript fetch, and whatcontent_jsoncarries for each backend. Adapt: live path should reusemessage.new; if tool_call rows are absent from any needed path, extend at the chokepoint (message_mirrorpublish / transcript API), not per-backend. - History lazy-load: a small read endpoint on
vibe/ui_server.py(native FastAPI async, no per-requestasyncio.run()) serving a turn's activity rows frommessages+agent_events(agent_events_service). Endpoint shape is the lane's choice; document it in the PR description. - Config default + plumbing for
ui.show_agent_activitythrough bootstrap config likechat_message_font_size.
Acceptance criteria (owner ~10 min)
- Toggle exists in Settings › Messaging, default off; off = pixel-identical current behavior (ThinkingBubble and all).
- On: sending a message shows the compact card in the reply position, live tool summaries + full interim assistant text, fixed height, no layout jump.
- Header click expands (~40vh, internal scroll); scroll-up pauses follow and shows jump-to-latest; tool row click toggles full call text.
- On completion the card collapses to the one-line chip above the final reply; chip toggles open/closed.
- Past turns (after refresh) show chips; first expand lazy-loads detail.
- Stop mid-run → "Interrupted" chip; failing turn → "Failed" chip with error bubble unchanged.
- IM platforms behave exactly as before;
npm run buildgreen; touched backend files' per-file pytest green; ruff clean.
As-built implementation notes (PR #934)
Implemented on branch feat/chat-agent-activity-panel. The three items below
record where the build diverged from or sharpened the plan above.
(a) The message.new live-push gap → gated publish at the chokepoint
The plan's "Key fact" assumed intermediate assistant / tool_call rows already
reach the browser via message.new. Verified false. persist_agent_message
(core/message_mirror.py) persists them but publishes no message.new — the
live stream deliberately carries only transcript types
(TRANSCRIPT_TYPES = user/harness/result/notify/error). Proven by the persist
contract tests test_persist_agent_intermediate_persisted_but_not_streamed and
test_persist_agent_toolcall_avibe_writes_event_without_streaming.
Per the plan's own "verify early / extend at the chokepoint, not per-backend"
guidance, the live path is extended only in core/message_mirror.py, gated on
config.ui.show_agent_activity:
- off (default) → byte-identical no-op. No extra publish;
tool_callstill writes only toagent_events,assistantstill stays out of the stream; the existing mirror contract tests are unchanged. - on → interim
assistantrows publishmessage.new(the persisted row), andtool_callevents (which live inagent_events, nevermessages) publish a synthesizedmessage.newpayload shaped like amessagesrow (type='tool_call'), so the one existing browser consumer (ChatPage) can route it into the activity store.inbox.session.updatedand web-push are not emitted for these rows (still process log, not a reply).
The gate reads settings.load_config_or_default().ui.show_agent_activity behind a
short TTL cache keyed by config path (no per-emit disk parse; the path key keeps a
fresh AVIBE_HOME — e.g. per test — from reading a stale value). No IM /
core/message_dispatcher.py / per-adapter edits.
Config plumbing mirrors chat_message_font_size: UiConfig.show_agent_activity: bool = False in config/v2_config.py (+ a bool(...) coercion in from_payload),
surfaced automatically through to_dict / api.config_to_payload, saved via the
deep-merged POST /api/config (a partial {"ui": {...}} patch preserves siblings),
read by ChatPage from bootstrap.config.ui.show_agent_activity.
(b) History read endpoint
GET /api/sessions/<id>/activity (native async FastAPI in vibe/ui_server.py,
reads messages + agent_events via storage services; 404 on unknown session):
- summary (no params) — one entry per turn with ≥1 activity row, no row text:
{"groups": [{"id", "anchor_message_id", "anchor_position", "open", "status", "steps", "started_at", "ended_at", "duration_ms"}]}. Loaded once on chat open so past turns show a chip. - detail (
?group_id=<id>) — that one group plus"rows": [{"id", "kind" (assistant|tool_call), "text", "created_at"}]. The lazy expand; 404 on an unknown group id.
status ∈ done | failed | interrupted. id is the group's first-activity-row id
(stable key across summary/detail).
Anchoring invariant (corrected — was a P1 rendering bug, fixed post-#934). A group is positioned relative to a transcript message that is at or before the group's own end — never a future message:
- done / failed →
anchor_message_id= the turn's terminal reply (result/error/ backend-failurenotify),anchor_position = "before"(the chip hugs the reply from above). - interrupted →
anchor_message_id= the boundary immediately before the turn's activity (its triggering user/harness message, tracked aslast_boundary_id),anchor_position = "after"(the chip sits just below the trigger). It is NEVER anchored to the next turn's opener (a future message) and NEVER to the transcript tail. open= true only for the last un-terminated turn. The frontend promotes theopengroup into the tail live running card while the turn is running; otherwise it renders as an interrupted chip after its trigger. The transcript tail is reserved exclusively for the live card — a settled/interrupted chip is never placed there.anchor_message_idisnullonly in the degenerate no-prior-message case (rendered at the top, never the tail).
Original bug: interrupted turns anchored forward to the next turn's opening
message; while that message did not yet exist (or was minted much later) the
frontend fell back to the tail slot, so the "Interrupted" chip rendered below newer
messages and even below the next turn's live card. The backward-anchor invariant
fixes it at source. Grouping logic lives in storage/agent_activity_service.py; the
low-level read is agent_events_service.list_session_events.
(c) Timestamp-parsed turn-boundary grouping
messages has no turn_id column (only agent_events does), so a turn cannot
be reconstructed by id. Instead a turn is bounded by transcript markers: it ends at
the agent's terminal reply (result / error / backend-failure notify), or — if
the user opened a new turn without one — is reported interrupted. Activity rows
between boundaries form the group.
Both tables write whole-second ...Z timestamps, so a fast turn's tool call
and the terminal reply that followed it tie on time. The two sources are merged
into one timeline sorted by (parsed_timestamp, phase) where phase orders
turn-start < activity < terminal — this keeps a same-second tool call inside its
completed turn instead of sorting it after the terminal (which would orphan it to
the next turn or a spurious interrupted chip). Timestamps are parsed, not
string-compared, so the ordering stays correct if a writer ever changes precision.
Show-Page assistant marks (metadata.source == 'show_page') are excluded from
activity (they belong to the transcript). The scan is bounded to the recent tail
(most-recent 500 messages / 2000 tool-call events, cap documented not silent), and
tool-call events that predate the oldest scanned message are dropped — otherwise an
event whose turn boundary was cut off would anchor a bogus chip to the first
visible turn.
Frontend (single-source-of-truth model — the durable endpoint owns all SETTLED
groups; the live SSE buffer drives ONLY the in-flight running card). The running
card is a pure function of working AND the current-generation buffer
(shouldShowRunningCard) — so a stale buffer is invisible the moment working
goes false. The live buffer never leaves that card: there is deliberately no
client-side group reconstruction. On every settle signal — a terminal
message.new, turn.end, SSE reconnect, visibilitychange, OR the /turn-state
idle poll recovering a dropped terminal (the fifth signal, same contract) —
refreshActivity rebuilds activityGroups from GET /activity and, when no
in-flight turn remains, clears the live buffer so the finished card swaps to the
storage-derived chip (and its rows can't leak into the next turn). When a turn IS in flight, refresh re-hydrates the running card's rows
from storage only if the live stream hasn't already filled them. Settle bursts are
coalesced to one in-flight + at most one trailing fetch (scheduleActivityRefresh),
a transient fetch failure schedules exactly one bounded retry, and nothing fetches
when the toggle is off. The live buffer is a pure generation state machine
(liveActivityReducer, unit-tested): a monotonic generation bumps on every
turn.start (and on the first agent-initiated row after a settle), so a stale buffer
is invisible by construction and a late settle-refresh only clears/rehydrates its
OWN generation — a newer turn's resolution is a structural no-op (no
promise-cancellation or grace-timer bookkeeping). This structure closes the whole
class of live-state edge cases (lossy/stale/partial buffer, interrupted anchoring,
duration source, settle-vs-next-turn races) that iterative patching kept surfacing:
the chip's steps, status, anchor, and duration always come from the well-tested
backend grouping. Activity rows still live in a separate store (never the
messages array), so they never count against MAX_RETAINED_MESSAGES. Show-Page
transcript rows (user AND assistant, metadata.source='show_page') are excluded
from both the backend grouping (never turn openers / activity) and live ingestion.
A lazy detail-fetch failure surfaces a retry affordance rather than a false "no
activity". The activity-streaming flag cache is reset on config save so the toggle
takes effect immediately. Presentation is ActivityCard / ActivityChip in
ui/src/components/workbench/AgentActivityGroup.tsx; pure helpers + the live
generation reducer + wire mapping in ui/src/lib/agentActivity.ts.
Compact running-card height (refined UX, design.pen State A updated). The
compact viewport height is min(content, 3-row cap ≈ 110px) — NOT a constant
reserved height. Below the cap the body is exactly content-tall and grows downward
as rows arrive (natural, like any new message at the transcript tail), so a 1–2 row
turn shows no blank space above the content. Only once content reaches the cap does
it become the constant-height viewport: clamped height, newest rows pinned to the
bottom (justify-end + overflow-hidden clipping the top), older rows fading up
(the top gradient renders ONLY at the cap). The cap is a CSS max-height; a small
layout measurement (offsetHeight >= cap) gates the fade, so the height behavior
itself is CSS-driven — not unit-testable under jsdom, which does no layout. Expanded
mode is unchanged: max-h-[40vh] + internal scroll already fits content up to its
cap. Tail auto-follow (the Transcript scroller) is untouched.
As-built implementation notes (P2 — items A–E)
Owner-approved iteration package (design signed off 2026-07-20). Frontend-only
except one new display-config field. NO-TOUCH held: IM adapters,
message_dispatcher, mirror publish logic, endpoint grouping.
A — Tool-row summary v2 (3-tier degrade, frontend-only parse)
ActivityRow.text for a tool call is the backend format_toolcall STRING
(modules/im/formatters/base_formatter.py): 🔧 `ToolName` `{compact json}`.
Names/arg-keys differ per backend (Claude Capitalized file_path/command; Codex
bash/file_change with file+type; OpenCode lowercase file_path||path; a
restore path emits `name`: `arg` with no JSON) — hence the degrade. Pure
helpers in ui/src/lib/agentActivity.ts: parseToolCall (name + parsed JSON args
or null), toolRecipe (tier-1 known-tool recipe: command → command body; read → dir
muted + basename; edit/write → basename + op badge, op from name or Codex type;
web/search/grep → quoted query/URL; task → description — case-insensitive prefix,
paths probe file_path→path→file), genericChips (tier-2 ≤3 scalar chips +
overflow), toolSummary (tier-3 raw, unchanged). Any parse failure / non-object /
oversize (>20k) → args: null → tier 3; ActivityToolRow also wraps the parse in
try/catch so nothing can blank a row. Rendered by ToolSummary in
AgentActivityGroup.tsx. Unit-tested per tier + exception fallback in
agentActivity.test.ts.
Command summaries omit the decorative shell prompt and unwrap a single
bash/zsh/sh -c or -lc invocation only when its command argument is a
fully decoded literal word. Extra arguments, outer expansions, operators, and
unrecognized syntax keep the original invocation visible. Nested shells are not
recursively unwrapped. Stored arguments, expanded details, and JSON remain
unchanged; this is a display-only projection shared by live and historical rows.
B — Tool-row visibility eye toggle + config.ui.show_tool_calls
New bool UiConfig.show_tool_calls (default true), coerced in
V2Config.from_payload like show_agent_activity. Both runtime serializers are
__dict__-based so it rides along automatically and the #939 serializer-coverage
guard is satisfied with no serializer edits (it derives fields from
fields(UiConfig)). Display-only → NO message_mirror cache / reset hook. The eye
pill (ToolsEyePill, eye/eye-off + "Tools"/"工具", icon-only < sm) sits in the
running-card header and the expanded-panel header (never the collapsed chip); it
flips the same config the Settings toggle writes (global, cross-device) via
api.saveConfig({ ui: { show_tool_calls } }), threaded through the activity prop
in ChatPage. Filter is the pure filterActivityRows (assistant narration ALWAYS
shows; step counts use the unfiltered length). All-filtered placeholders:
expanded → "已隐藏 N 条工具调用"; compact LIVE → "工具调用已隐藏 · 进行中" (never
falls back to ThinkingBubble). Settings row added in SettingsMessagingPage.tsx
(dashboard.showToolCalls*). i18n en+zh.
C — Done/history expanded height cap
The expanded settled panel (ActivityChip, done/interrupted/failed incl. history)
body is max-h-[60vh] overflow-y-auto → height = min(content, ~60vh), opening at
the TOP (natural scrollTop=0, no tail-follow). The LIVE running card's expanded
scroller is unchanged (max-h-[40vh], follows tail). Page-level scroll anchoring
untouched.
D — Inline detail v2 + full-JSON dialog
Row-click detail (ToolDetail) renders the SAME parse as A: a kv table (key mono
muted; long/multiline values as wrapping code blocks; timeout ms humanized);
parse failure → raw text (unchanged fallback). A "{ } JSON" button opens
ToolJsonDialog — the shared components/ui/dialog + the lazy preview-json
viewer FileViewer uses (zero new deps), scrollable, copy button
(copyTextToClipboard(JSON.stringify(args,null,2))), ESC/backdrop close. Parse
failure → dialog shows raw text, still copyable.
E — Settings copy
dashboard.showAgentActivityHint retrimmed per owner dictation (dropped the
Web-only / default-off tail) in en + zh.
Verification & real-browser residual
ui build green; vitest full suite green (A/D parse tiers + B filter logic +
existing reducer/wire tests); backend pytest green (new
test_show_tool_calls_defaults_on_and_round_trips + the #939 serializer-coverage
guard auto-covering the new field); ruff clean; changed-file eslint clean.
jsdom does no layout, so these need the owner's Incus/real-browser pass: the C
60vh scroll feel + top-open, the compact fade-at-cap, the eye-pill live toggle +
cross-device persistence, the D JSON dialog open/copy/close, and the tier-1/2/3
rendering against real per-backend format_toolcall output.
As-built implementation notes (silent-completion terminal taxonomy)
Bug. P1 defined interrupted as "a turn's activity that ends without a terminal
result". That wrongly captured silent completions (final reply is entirely a
<silent> block → stripped → nothing delivered) and reply-less bookkeeping turns
(common for watch/scheduled orchestration): the turn ran tool steps, finished
normally, but wrote no messages row, so the grouping saw "activity + no terminal"
and rendered a gold "Interrupted · stopped at step N" chip. Nothing was interrupted.
Terminal taxonomy (as-built). A turn's activity group closes as:
| Ending | Grouping status |
|---|---|
visible result reply |
done |
invisible silent marker (silent-stripped / empty final, or reply-less completion) |
done |
error, or backend_failure notify |
failed |
| cancel / Stop, or no terminal at all | interrupted |
A plain notify is NOT terminal — agents emit mid-turn notify rows that keep the
turn going (e.g. Claude's model-refusal fallback), so treating every notify as terminal
would split one turn into two. A genuine notify-only COMPLETION is closed by the
silent marker instead (its turn still emits an empty final result at the chokepoint).
Invisible silent marker. When a turn completes NORMALLY with nothing
user-visible to send, the delivery chokepoint
(MessageDispatcher.emit_agent_message, the mutates_turn_lifecycle branch) persists
ONE messages row of a dedicated type='silent' via
message_mirror.persist_silent_completion_marker → messages_service.append
(bypassing the empty-text guard and the message.new publish — no transcript bubble).
Gate: level != "silent" and not is_error and not suppress_delivery. This is the
load-bearing distinction: the real user-stop paths (codex/claude/opencode) emit a
terminal result with level="silent" and is_error=False, so not is_error ALONE
would wrongly mark a stop as done — a stop must stay interrupted. A genuine
<silent>-block/empty completion is level="normal" with an empty body; backend
failures arrive level="silent" (after a visible notify); a suppress_delivery
private/background run intentionally leaves NO history — all excluded.
The marker DOES recompute + publish inbox.session.updated (avibe): since it now
clears the inbox awaiting/replied flag, an open sidebar must drop "awaiting the agent"
live rather than staying stale until reconnect. But it publishes NO message.new (no
transcript bubble) and NO web-push (a silent completion is not a notifiable reply).
A dedicated type (not the originally-sketched result + content.kind='silent')
was chosen after finding the read layer is allowlist-based: type='silent' is
auto-excluded from the transcript (TRANSCRIPT_TYPES), inbox preview, unread, web-push
and the live-publish gate with ZERO new guards — mirroring the invisible-type precedent
(pending/queued/draft/harness_dedupe). (A result-typed marker would have
leaked into ~8 allowlist reads.) Teaching the rest of the system about the new type:
NON_CONVERSATION_TYPESgainssilent— the marker never bumps the inbox activity clock / last-author.- Inbox awaiting/replied compares
last_input_atagainst a terminal timestamp that INCLUDESsilent(not the visible-onlypreview_*), so a silently-completed turn is no longer stuck showing "awaiting the agent"; the preview TEXT stays the last visible reply. ix_messages_inbox_activitypartial-index predicate addssilent(models.py + migrations.py + alembic20260721_0031) — a stricterNOT INquery predicate does NOT reuse a looser partial index on SQLite, so the predicates must match or inbox refreshes fall back to scans.- Session fork treats
silentas terminal in BOTH the anchor query (_latest_source_message_anchor, else the anchor falls back to the input row and the completed turn is trimmed/rolled back as if running) andTERMINAL_AGENT_OUTPUT_TYPES.
Grouping. agent_activity_service adds silent to _RELEVANT_MESSAGE_TYPES (so
its timeline — separate from the transcript allowlist — sees the marker) and
_is_terminal (result/error/silent + backend_failure notify). Because the marker is
invisible in the transcript, a group closing on it anchors to the visible turn
trigger AFTER it (never the marker — which the frontend can't position against, per
the #935 backward anchor invariant), and the marker never becomes last_boundary_id. A
visible terminal still anchors to itself, BEFORE it. Frontend: no change — the
marker never reaches it; done flows through groupFromWire and renders as the ✓ chip.
Evidence. tests/test_agent_activity_service.py — silent completion → done
(watch-triggered, tool steps, silent finish; anchored to the visible trigger); a
mid-turn notify does NOT split/close a turn; backend_failure notify → failed; Stop (no
terminal) → interrupted. tests/test_message_dispatcher_result_fallback.py — the
chokepoint writes the marker on a clean completion, NOT on a level="silent" stop nor
an is_error result. tests/test_message_mirror.py — the marker persists but is
excluded from the transcript allowlist + NON_CONVERSATION_TYPES.
tests/test_messages_service.py — a silent completion clears the inbox "awaiting" flag
while the preview stays the last visible reply. tests/test_session_fork.py — a
silently-completed codex/opencode source turn is terminal (no trim). IM delivery
untouched (marker is avibe-persistence only).