Skip to content

armgit / mimik

Public Imported from GitHub
0

🎬 A browser extension and desktop app that turns your clicks into step-by-step guides with annotated screenshots πŸ“–

docs: update AGENTS.md with current architecture

Luis Ortiz0f0cbc91855e
diff --git a/AGENTS.md b/AGENTS.md
index 6d9f330..66f8f48 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -22,12 +22,32 @@ You click "Record," perform a workflow in your browser, and Mimik automatically
```
src/
β”œβ”€β”€ core/ # Business logic (no UI dependencies)
-β”‚ β”œβ”€β”€ capture/ # Recording pipeline, event detection, screenshots
+β”‚ β”œβ”€β”€ capture/ # Recording pipeline
+β”‚ β”‚ β”œβ”€β”€ ai/ # AI description + title generation (Vercel AI SDK)
+β”‚ β”‚ β”‚ β”œβ”€β”€ description.ts # getAIDescription (DOM context β†’ AI β†’ step text)
+β”‚ β”‚ β”‚ β”œβ”€β”€ title.ts # generateGuideTitle (steps β†’ AI β†’ guide name)
+β”‚ β”‚ β”‚ β”œβ”€β”€ models.ts # AI_PROVIDERS config (OpenAI/Anthropic model lists)
+β”‚ β”‚ β”‚ β”œβ”€β”€ prompts.ts # Prompt templates
+β”‚ β”‚ β”‚ └── provider.ts # createModel factory (OpenAI/Anthropic)
+β”‚ β”‚ β”œβ”€β”€ dom/ # DOM extraction utilities
+β”‚ β”‚ β”‚ β”œβ”€β”€ context.ts # DOMContext extraction + serialization
+β”‚ β”‚ β”‚ β”œβ”€β”€ element-meta.ts # extractElementMeta (selector, text, aria, rect)
+β”‚ β”‚ β”‚ └── element-utils.ts # findFocusableAncestor, isTextField, etc.
+β”‚ β”‚ β”œβ”€β”€ events/ # Event capture system
+β”‚ β”‚ β”‚ β”œβ”€β”€ handlers.ts # CaptureController class + startCapture
+β”‚ β”‚ β”‚ β”œβ”€β”€ highlight.ts # HighlightManager (dashed border overlay)
+β”‚ β”‚ β”‚ └── input-session.ts # InputSession (typing lifecycle)
+β”‚ β”‚ β”œβ”€β”€ machine.ts # xstate capture state machine
+β”‚ β”‚ β”œβ”€β”€ rrweb-recorder.ts # DOM recording for replay
+β”‚ β”‚ β”œβ”€β”€ session.ts # CaptureSession (lifecycle manager)
+β”‚ β”‚ β”œβ”€β”€ spa-nav.ts # SPA navigation tracking
+β”‚ β”‚ β”œβ”€β”€ start-notification.ts # Recording notification overlay
+β”‚ β”‚ └── step-description.ts # Fallback rule-based descriptions
β”‚ β”œβ”€β”€ export/ # HTML, PDF, Markdown export generators
β”‚ └── guides/ # Data layer: types, Dexie DB, CRUD service
β”œβ”€β”€ entrypoints/ # Chrome extension entry points (WXT)
β”‚ β”œβ”€β”€ background/ # Service worker: state machine, message handlers, tab management
-β”‚ β”œβ”€β”€ content.ts # Content script: event capture, overlay, rrweb recording
+β”‚ β”œβ”€β”€ content.ts # Content script: CaptureSession, event listeners, rrweb
β”‚ β”œβ”€β”€ sidepanel/ # Side panel React mount
β”‚ β”œβ”€β”€ fullview/ # Full-page view React mount
β”‚ └── options/ # Settings page React mount
@@ -59,6 +79,8 @@ src/
β”‚ β”œβ”€β”€ ExportMenu.tsx
β”‚ β”œβ”€β”€ BlurCanvas.tsx
β”‚ └── ZoomScreenshot.tsx
+ β”œβ”€β”€ shared/ # Shared UI components
+ β”‚ └── SettingsView.tsx # AI settings (provider, model, API key)
└── options/ # Settings page
└── App.tsx
```
@@ -72,16 +94,17 @@ src/
| Persistence | Dexie (IndexedDB) | Guides, steps, screenshots, rrweb chunks |
| Service worker recovery | sessionStorage | xstate machine snapshot persistence |
| Background β†’ Sidepanel | Port messaging | Real-time state broadcast |
+| Cross-context sync | BroadcastChannel | Guide mutations (star, delete) across sidepanel/fullview |
## Extension Entry Points
| Entry Point | File | Purpose |
|-------------|------|---------|
-| Background | `entrypoints/background/` | Service worker: xstate actor, message handlers, tab management, navigation tracking |
-| Content Script | `entrypoints/content.ts` | Injected into all tabs: CaptureSession, event listeners, rrweb, highlight overlay |
-| Side Panel | `entrypoints/sidepanel/` | Recording controls, library, guide editor |
+| Background | `entrypoints/background/` | Service worker: xstate actor, message handlers, tab management |
+| Content Script | `entrypoints/content.ts` | Injected into all tabs: CaptureSession, event listeners, rrweb |
+| Side Panel | `entrypoints/sidepanel/` | Recording controls, library, guide editor, settings |
| Full View | `entrypoints/fullview/` | Dashboard: library browse, guide viewer, Ctrl+K search |
-| Options | `entrypoints/options/` | AI API key settings (OpenAI / Anthropic) |
+| Options | `entrypoints/options/` | Settings page (shared SettingsView in centered card) |
## Messaging
@@ -92,13 +115,15 @@ Content Script ←→ Background Service Worker ←→ Sidepanel / Fullview
**Extension messages** (webext-core, `lib/messaging.ts`):
- `getState` β†’ current capture state, step count, guide ID
- `startRecording({url})` β†’ creates guide, returns guideId
-- `stopRecording()` β†’ finalizes guide
-- `userAction({guideId, action, elementMeta})` β†’ processes captured step
+- `stopRecording()` β†’ finalizes guide, generates AI title
+- `captureStep({guideId, action, elementMeta, domContext})` β†’ screenshots + creates step
+- `updateInputStep({stepId, description})` β†’ updates typing step description
+- `finalizeInputStep({stepId, elementMeta, domContext})` β†’ final screenshot + AI description
- `rrwebChunk({guideId, events, timestamp})` β†’ stores DOM recording chunk
**Tab messages** (content script ↔ background, `lib/tab-messages.ts`):
- `PING` / `START_CAPTURE` / `STOP_CAPTURE` β€” lifecycle
-- `HIDE_OVERLAY` / `SHOW_OVERLAY` β€” overlay toggle before/after screenshot
+- `HIDE_OVERLAY` / `SHOW_OVERLAY` β€” overlay toggle (fallback, used for non-click captures)
- `SHOW_NOTIFICATION` β€” "Recording started" overlay
- `URL_CHANGED` / `GET_ROUTE` β€” SPA navigation tracking
@@ -108,21 +133,47 @@ Content Script ←→ Background Service Worker ←→ Sidepanel / Fullview
1. User clicks "Start Capture" in sidepanel
2. Background transitions xstate machine IDLE β†’ RECORDING
3. Creates Guide in IndexedDB, broadcasts `START_CAPTURE` to all tabs
-4. Content scripts initialize CaptureSession (event listeners + rrweb)
+4. Content scripts create CaptureSession β†’ CaptureController (event listeners) + rrweb
5. Shows recording notification overlay on active tab
-**Capture a step:**
-1. Content script detects user action (click, input, drag)
-2. Extracts ElementMeta (selector, text, aria labels, position)
-3. Sends `userAction` to background
-4. Background hides overlay β†’ `captureVisibleTab` β†’ shows overlay
-5. Optionally sends screenshot to AI for description (OpenAI/Anthropic)
-6. Saves Step + Screenshot to IndexedDB
+**Capture a click:**
+1. Content script's CaptureController detects click via DOM event listener
+2. Click handler pushes async work into PQueue (concurrency: 1)
+3. Queue processes: hides overlay instantly β†’ sends `captureStep` to background β†’ waits
+4. Background calls `captureVisibleTab` (overlay is hidden, page hasn't reacted yet)
+5. Saves Screenshot + Step to IndexedDB
+6. Optionally generates AI description from DOM context text (not screenshot)
+7. Returns `{ stepId }` β†’ content script shows overlay β†’ queue processes next event
+
+**Capture text input (typing):**
+1. Click on text field β†’ CaptureController starts InputSession β†’ `captureStep` with initial screenshot
+2. Each keystroke β†’ InputSession.update() β†’ `updateInputStep` (description only, fire-and-forget)
+3. Enter/Escape/focusout β†’ InputSession.finalize() β†’ `finalizeInputStep` β†’ final screenshot replaces initial + AI description
+4. Result: one step for entire typing interaction
**Stop recording:**
1. Background transitions RECORDING β†’ IDLE
-2. Broadcasts `STOP_CAPTURE`, content scripts flush rrweb events
-3. Opens fullview dashboard with the guide
+2. Broadcasts `STOP_CAPTURE`, content scripts flush pending input sessions + rrweb events
+3. Background generates guide title from step descriptions + URLs via AI
+4. Opens fullview dashboard with the guide
+
+## DOM Context (AI Input)
+
+Instead of sending screenshots to the AI for step descriptions, Mimik extracts a lightweight DOM context (~50-100 tokens) from around the target element:
+
+```
+Page: "Public profile - Settings" /settings/profile
+Container: form "Public profile"
+Heading: "Public profile"
+Siblings: input "Name", input "Email", textarea "Bio", button "Update profile"
+β†’ Target: button "Update profile" (click)
+```
+
+Extraction walks up from the target element to find:
+- Page title + URL path
+- Nearest semantic container (form, nav, dialog, section) or 3 levels up
+- Nearest heading
+- Sibling interactive elements in the same container (max 10)
## Export Formats
@@ -147,7 +198,8 @@ Content Script ←→ Background Service Worker ←→ Sidepanel / Fullview
| Messaging | webext-core |
| Session recording | rrweb |
| Export | jsPDF, client-side HTML/Markdown |
-| AI (optional) | OpenAI SDK, Anthropic SDK |
+| AI (optional) | Vercel AI SDK (`ai`, `@ai-sdk/openai`, `@ai-sdk/anthropic`) |
+| Event queue | p-queue (concurrency: 1) |
| Icons | Lucide React |
| Dates | dayjs |
| DOM utils | css-selector-generator |
@@ -172,10 +224,13 @@ Font: Poppins (loaded via `@fontsource/poppins`).
## Key Technical Details
-- **Screenshot capture** uses `chrome.tabs.captureVisibleTab` with overlay hidden to avoid capturing the highlight
-- **Event deduplication** merges fast clicks within 300ms on same element; max 4 clicks per 800ms window
+- **Async event queue** in content script (PQueue, concurrency 1) serializes capture work β€” each action awaits the full background round-trip before the next starts
+- **Overlay hidden from content script side** (instant `display:none`) before sending capture message β€” no round-trip delay for overlay toggle
+- **Input session** aggregates all typing on a field into one step β€” click creates it, keystrokes update description, finalize takes final screenshot
+- **DOM context** sent as text to AI instead of screenshots β€” 15-30x cheaper per step
- **Highlight overlay** uses a custom web component (`<mimik-highlight>`) with closed Shadow DOM at max z-index
- **Content script injection** pings first, falls back to `chrome.scripting.executeScript()` for tabs without the script
- **xstate snapshot** persisted to sessionStorage so the state machine survives service worker restarts
- **Recording notification** uses `animationend` event (not hardcoded delays) for timing
- **Font loading** uses `@fontsource/poppins` (CSP-safe, no CDN dependency)
+- **Cross-context sync** via BroadcastChannel β€” star/delete events update other views without full reload