Skip to content

Desktop app that records your on-screen work session and uses the GitHub Copilot SDK to reconstruct it as an intent + ordered steps, then builds a reusable Skill or Automation for Microsoft Autopilots like Scout, Microsoft Copilot Cowork, or Copilot Studio.

Add optional offline voice narration + transcript tool (#14)

adilei43362a8aecce
diff --git a/common/ipc.ts b/common/ipc.ts
index d351acc..33a5b9a 100644
--- a/common/ipc.ts
+++ b/common/ipc.ts
@@ -144,6 +144,12 @@ export interface StartResult {
error?: string;
}
+/** Per-session capture choices the user makes in the HUD before recording. */
+export interface StartOptions {
+ /** Capture microphone narration for this session (opt-in, off by default). */
+ narration?: boolean;
+}
+
export interface StopResult {
ok: boolean;
sessionId?: string;
@@ -242,7 +248,7 @@ export const IPC = {
/** Shape exposed on `window.skillRecorder` by the preload bridge. */
export interface SkillRecorderApi {
- start(): Promise<StartResult>;
+ start(options?: StartOptions): Promise<StartResult>;
stop(): Promise<StopResult>;
status(): Promise<RecorderStatus>;
marker(note: string): Promise<MarkerResult>;
diff --git a/common/narration.ts b/common/narration.ts
new file mode 100644
index 0000000..e0a20af
--- /dev/null
+++ b/common/narration.ts
@@ -0,0 +1,25 @@
+/**
+ * The offline voice-narration transcript. This is the spoken form of the typed
+ * marker: the user's own stated intent while recording. It is produced after
+ * Stop by the narration stage (Whisper via transformers.js) and written to
+ * `narration.json`. It is NEVER appended to the finalized `events.jsonl`; the
+ * describer reads it through the `get_narration` tool, and it only leaves the
+ * machine on Analyze, exactly like screenshots.
+ */
+export interface NarrationSegment {
+ /** Segment start, in ms since the session started (same clock as step offsets). */
+ atMs: number;
+ /** Segment end, in ms since the session started. */
+ endMs: number;
+ /** The spoken text for this segment. */
+ text: string;
+}
+
+export interface NarrationTranscript {
+ /** The Whisper model id that produced this transcript. */
+ model: string;
+ segments: NarrationSegment[];
+}
+
+/** Filename of the transcript within a session folder. */
+export const NARRATION_FILE = "narration.json";
diff --git a/docs/future-features.md b/docs/future-features.md
new file mode 100644
index 0000000..571da3d
--- /dev/null
+++ b/docs/future-features.md
@@ -0,0 +1,38 @@
+# Future features
+
+Capabilities that are wired up in the codebase but deliberately not surfaced in
+the UI yet. Each entry is dormant, not dead: the backend path stays intact so the
+feature can be revived without re-plumbing it.
+
+## Manual markers ("Add marker")
+
+**Status:** backend retained, UI removed.
+
+A marker is a user-authored note captured mid-recording ("what are you doing
+right now?"). It was originally a HUD button that opened a blocking prompt.
+
+**Why the button was removed:** voice narration supersedes it. Narration captures
+the same "stated intent" signal continuously, hands-free, and timestamped, without
+interrupting the task or letting a prompt dialog leak into the recorded frames. In
+practice the button was never used, and the describer already treated the "flick
+back to add a marker" moment as noise.
+
+**What still exists (the plumbing behind it):**
+
+| Layer | Location |
+| --- | --- |
+| Renderer bridge | `electron/preload.cjs` (`window.skillRecorder.marker`) |
+| IPC channel + result type | `common/ipc.ts`, `electron/ipc.ts` |
+| Recorder handler | `electron/recorder/controller.ts` (`marker()`) |
+| Event type + payload | `common/events.ts` (`EventType.Marker`, `MarkerPayload`) |
+| Correlation / bundling | `common/correlation.ts`, `common/bundle.ts` (`step.markers`) |
+| Description surfacing | `common/describe.ts`, describer + skillbuilder `tools.ts` |
+
+**What was removed:** the `Add marker` button and its `addMarker` handler in
+`src/Recorder.tsx`, plus the `.marker` styles in `src/App.css`.
+
+**How to revive it well:** don't bring back the blocking `window.prompt`. Prefer a
+low-friction trigger that doesn't interrupt the recording, e.g. a global hotkey
+that flags the current moment (optionally with a quick inline note), so a marker
+becomes a precise "this instant matters" signal that complements the continuous
+narration transcript rather than duplicating it.
diff --git a/docs/windows-capture.md b/docs/windows-capture.md
index 68829ff..65cdffd 100644
--- a/docs/windows-capture.md
+++ b/docs/windows-capture.md
@@ -15,6 +15,7 @@ before trusting a Windows build.
| Browser URLs | UI Automation address bar read (`powershell.exe` host) | Functional, not byte exact | None |
| Clipboard | Electron clipboard | Full | None |
| Screen video + frames | `desktopCapturer` + ffmpeg + sharp | Full | Screen capture |
+| Voice narration (opt-in) | hidden-window `getUserMedia` mic + `MediaRecorder`, transcribed offline (Whisper via transformers.js) | Should work, unverified here | Microphone |
Notes:
@@ -62,7 +63,13 @@ Run a real recording on Windows and verify each source lands in the session's
4. **Clipboard.** Copy some text. Expect a `clipboard.change` event with a
preview and hash.
5. **Video.** Confirm a `.webm` is written and `frame.captured` events appear.
-6. **Stop.** The recording should show up in the library as `recorded`, and
+6. **Voice narration.** Turn on the **Narrate** switch before starting, then speak
+ a sentence or two during the recording. After Stop, confirm `audio.webm` +
+ `audio.json` are written and, once processing finishes, `narration.json`
+ contains your words with `atMs` offsets. First run downloads the ~250 MB
+ Whisper model once (needs network); later runs are offline. On Windows the mic
+ grant is requested by the OS on first use.
+7. **Stop.** The recording should show up in the library as `recorded`, and
analysis should produce a coherent intent + ordered steps.
If a source produces nothing, check the doctor row for it first, then the main
@@ -72,9 +79,11 @@ unavailable on this platform", or a reduced-capture notice).
## Packaging
`package.json` `build` configures electron-builder for both `mac` and `win`
-(nsis x64). Native modules (`get-windows`, `sharp`, `@img/*`, `ffmpeg-static`)
-are listed under `asarUnpack` so their binaries load from disk rather than from
-inside the asar archive.
+(nsis x64). Native modules (`get-windows`, `sharp`, `@img/*`, `ffmpeg-static`,
+`@huggingface/transformers`, `onnxruntime-node`) are listed under `asarUnpack` so
+their binaries load from disk rather than from inside the asar archive. The
+Whisper model itself is not bundled; it downloads once to the app's user-data
+`models` folder on first narrated session.
Build the Windows installer on Windows (or a Windows CI runner) so the native
binaries for `win32-x64` are present:
@@ -94,6 +103,11 @@ binaries and is not supported here.
approach is tracked in issue #7.
- Semantic UI events (focus/invoke/value via UI Automation) are not implemented
on either platform yet.
+- Voice narration is verified end to end on macOS (mic capture -> `audio.webm` ->
+ offline Whisper transcription -> `narration.json`). The Windows mic-capture path
+ is written the same way but has not been run on a real Windows machine; verify it
+ with step 6 of the checklist above. `onnxruntime-node` ships prebuilt binaries for
+ `win32-x64`/`win32-arm64`, so no compiler is needed there.
- The Windows paths above are validated by typecheck, a PowerShell parse check of
the UIA script, and a `win32` describer eval (`evals/scenarios/windows-deploy.ts`).
Live capture must still be verified on a real Windows machine using the checklist
diff --git a/electron/audio/capture-preload.cjs b/electron/audio/capture-preload.cjs
new file mode 100644
index 0000000..ab55241
--- /dev/null
+++ b/electron/audio/capture-preload.cjs
@@ -0,0 +1,86 @@
+// Narration capture preload — CommonJS, runs in the hidden capture window's
+// isolated world. It has both Web APIs (navigator.mediaDevices, MediaRecorder)
+// and ipcRenderer, so it does the whole microphone capture here and streams webm
+// chunks back to the main process, which writes them to disk. Channel names
+// mirror electron/audio/recorder.ts.
+const { ipcRenderer } = require("electron");
+
+/** @type {MediaRecorder | null} */
+let recorder = null;
+/** @type {MediaStream | null} */
+let stream = null;
+// Serialises chunk sends so the final blob (dispatched just before `stop`) is
+// fully forwarded before we tell main the recording stopped — otherwise the last
+// cluster is lost and the webm ends prematurely.
+let sendChain = Promise.resolve();
+
+function cleanup() {
+ try {
+ if (stream) for (const track of stream.getTracks()) track.stop();
+ } catch {
+ // ignore
+ }
+ stream = null;
+ recorder = null;
+}
+
+ipcRenderer.on("audio:start", async (_event, opts) => {
+ const { bitsPerSecond } = opts || {};
+ try {
+ // A plain microphone request. Echo cancellation / noise suppression keep the
+ // narration clean for the transcriber without us touching the samples.
+ stream = await navigator.mediaDevices.getUserMedia({
+ audio: {
+ echoCancellation: true,
+ noiseSuppression: true,
+ channelCount: 1,
+ },
+ video: false,
+ });
+
+ const mimeType = MediaRecorder.isTypeSupported("audio/webm;codecs=opus")
+ ? "audio/webm;codecs=opus"
+ : "audio/webm";
+ recorder = new MediaRecorder(stream, { mimeType, audioBitsPerSecond: bitsPerSecond });
+
+ recorder.ondataavailable = (e) => {
+ if (!e.data || e.data.size === 0) return;
+ // Preserve order and completion; Uint8Array survives IPC structured clone.
+ sendChain = sendChain.then(async () => {
+ const buf = await e.data.arrayBuffer();
+ ipcRenderer.send("audio:chunk", new Uint8Array(buf));
+ });
+ };
+ recorder.onstart = () => ipcRenderer.send("audio:started", Date.now());
+ recorder.onstop = () => {
+ // Wait for every queued chunk (including the final one) to be sent.
+ sendChain.then(() => {
+ cleanup();
+ ipcRenderer.send("audio:stopped");
+ });
+ };
+ recorder.onerror = (e) => {
+ ipcRenderer.send("audio:error", String((e && e.error) || e));
+ };
+
+ // Emit a chunk every second so long sessions stream to disk incrementally.
+ recorder.start(1000);
+ } catch (err) {
+ cleanup();
+ ipcRenderer.send("audio:error", err instanceof Error ? err.message : String(err));
+ }
+});
+
+ipcRenderer.on("audio:stop", () => {
+ try {
+ if (recorder && recorder.state !== "inactive") {
+ recorder.requestData();
+ recorder.stop();
+ } else {
+ ipcRenderer.send("audio:stopped");
+ }
+ } catch (err) {
+ ipcRenderer.send("audio:error", err instanceof Error ? err.message : String(err));
+ ipcRenderer.send("audio:stopped");
+ }
+});
diff --git a/electron/audio/capture.html b/electron/audio/capture.html
new file mode 100644
index 0000000..172eecb
--- /dev/null
+++ b/electron/audio/capture.html
@@ -0,0 +1,19 @@
+<!doctype html>
+<html>
+ <head>
+ <meta charset="utf-8" />
+ <meta
+ http-equiv="Content-Security-Policy"
+ content="default-src 'none'; script-src 'none'"
+ />
+ <title>Skill Recorder — narration</title>
+ </head>
+ <body>
+ <!--
+ Hidden utility window. All microphone-capture logic lives in the preload
+ (audio/capture-preload.cjs), which has DOM/Web-API access
+ (navigator.mediaDevices, MediaRecorder) plus ipcRenderer. No page script is
+ needed, so the CSP forbids scripts entirely.
+ -->
+ </body>
+</html>
diff --git a/electron/audio/recorder.ts b/electron/audio/recorder.ts
new file mode 100644
index 0000000..caa3590
--- /dev/null
+++ b/electron/audio/recorder.ts
@@ -0,0 +1,177 @@
+import { createWriteStream, existsSync, type WriteStream } from "node:fs";
+import { unlink, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { BrowserWindow, ipcMain, type IpcMainEvent } from "electron";
+
+import { createLogger } from "../logger";
+
+const log = createLogger("Narration/audio");
+const dirname = path.dirname(fileURLToPath(import.meta.url));
+
+/** Result metadata written alongside the webm as `audio.json`. */
+export interface AudioResult {
+ file: string;
+ /** Wall-clock epoch (ms) when MediaRecorder actually started — the anchor that
+ * maps a transcript segment's offset back onto the session clock. */
+ startEpoch: number;
+ stopEpoch: number;
+ durationMs: number;
+ bytes: number;
+}
+
+// Opus at 24 kbps mono is transparent for speech and keeps narration files tiny.
+const BITS_PER_SECOND = 24_000;
+const STOP_TIMEOUT_MS = 4000;
+
+/**
+ * Captures the user's microphone narration for a session, mirroring the video
+ * sidecar: the heavy lifting (getUserMedia + MediaRecorder) happens in a hidden
+ * renderer window (electron/audio/capture.html + capture-preload.cjs); this class
+ * orchestrates it from the main process and writes the webm + `audio.json` to the
+ * session folder. Capture is strictly opt-in (started only when the user enabled
+ * narration for the session) and best-effort: if there is no microphone or the
+ * permission is denied, it logs and the session proceeds without narration.
+ */
+export class AudioRecorder {
+ private win: BrowserWindow | null = null;
+ private stream: WriteStream | null = null;
+ private file = "";
+ private dir = "";
+ private bytes = 0;
+ private startEpoch: number | null = null;
+ private stoppedResolve: (() => void) | null = null;
+ private failed = false;
+
+ private readonly onChunk = (e: IpcMainEvent, chunk: Uint8Array) => {
+ if (e.sender !== this.win?.webContents || !this.stream) return;
+ const buf = Buffer.from(chunk);
+ this.bytes += buf.byteLength;
+ this.stream.write(buf);
+ };
+
+ private readonly onStarted = (e: IpcMainEvent, epoch: number) => {
+ if (e.sender !== this.win?.webContents) return;
+ this.startEpoch = epoch;
+ log.info("narration started; anchor epoch", epoch);
+ };
+
+ private readonly onStopped = (e: IpcMainEvent) => {
+ if (e.sender !== this.win?.webContents) return;
+ this.stoppedResolve?.();
+ };
+
+ private readonly onError = (e: IpcMainEvent, message: string) => {
+ if (e.sender !== this.win?.webContents) return;
+ this.failed = true;
+ log.warn("microphone unavailable:", message);
+ // Unblock a pending stop() if the recorder never really started.
+ this.stoppedResolve?.();
+ };
+
+ /** Begin capturing into `<sessionDir>/audio.webm`. Never throws. */
+ async start(sessionDir: string): Promise<void> {
+ try {
+ this.dir = sessionDir;
+ this.file = path.join(sessionDir, "audio.webm");
+ this.bytes = 0;
+ this.startEpoch = null;
+ this.failed = false;
+ this.stream = createWriteStream(this.file);
+
+ ipcMain.on("audio:chunk", this.onChunk);
+ ipcMain.on("audio:started", this.onStarted);
+ ipcMain.on("audio:stopped", this.onStopped);
+ ipcMain.on("audio:error", this.onError);
+
+ this.win = new BrowserWindow({
+ show: false,
+ webPreferences: {
+ preload: path.join(dirname, "audio", "capture-preload.cjs"),
+ contextIsolation: true,
+ nodeIntegration: false,
+ sandbox: false,
+ },
+ });
+ await this.win.loadFile(path.join(dirname, "audio", "capture.html"));
+
+ this.win.webContents.send("audio:start", { bitsPerSecond: BITS_PER_SECOND });
+ log.info("narration capture requested ->", this.file);
+ } catch (err) {
+ log.warn("failed to start narration:", err instanceof Error ? err.message : err);
+ await this.teardown();
+ }
+ }
+
+ /** Stop capturing and return the result, or null if no usable audio was produced. */
+ async stop(): Promise<AudioResult | null> {
+ if (!this.win || !this.stream) {
+ await this.teardown();
+ return null;
+ }
+
+ const stopped = new Promise<void>((resolve) => {
+ this.stoppedResolve = resolve;
+ this.win?.webContents.send("audio:stop");
+ });
+ await Promise.race([stopped, delay(STOP_TIMEOUT_MS)]);
+
+ const stopEpoch = Date.now();
+ await this.closeStream();
+
+ const bytes = this.bytes;
+ const startEpoch = this.startEpoch;
+ const file = this.file;
+ const failed = this.failed;
+ await this.teardown();
+
+ if (failed || bytes === 0 || startEpoch == null) {
+ // Remove a useless/empty file so the session folder stays clean.
+ if (existsSync(file)) await unlink(file).catch(() => undefined);
+ log.warn("no usable narration captured");
+ return null;
+ }
+
+ const result: AudioResult = {
+ file: path.basename(file),
+ startEpoch,
+ stopEpoch,
+ durationMs: stopEpoch - startEpoch,
+ bytes,
+ };
+ await writeFile(
+ path.join(this.dirFor(file), "audio.json"),
+ JSON.stringify(result, null, 2),
+ ).catch((err) => log.warn("failed to write audio.json:", err instanceof Error ? err.message : err));
+ log.info(`narration saved: ${result.file} (${(bytes / 1000).toFixed(0)} KB, ${result.durationMs} ms)`);
+ return result;
+ }
+
+ private dirFor(file: string): string {
+ return this.dir || path.dirname(file);
+ }
+
+ private closeStream(): Promise<void> {
+ return new Promise((resolve) => {
+ if (!this.stream) return resolve();
+ this.stream.end(() => resolve());
+ });
+ }
+
+ private async teardown(): Promise<void> {
+ ipcMain.removeListener("audio:chunk", this.onChunk);
+ ipcMain.removeListener("audio:started", this.onStarted);
+ ipcMain.removeListener("audio:stopped", this.onStopped);
+ ipcMain.removeListener("audio:error", this.onError);
+ this.stoppedResolve = null;
+ await this.closeStream();
+ this.stream = null;
+ if (this.win && !this.win.isDestroyed()) this.win.destroy();
+ this.win = null;
+ }
+}
+
+function delay(ms: number): Promise<void> {
+ return new Promise((resolve) => setTimeout(resolve, ms));
+}
diff --git a/electron/describer/instructions.ts b/electron/describer/instructions.ts
index 5965dff..7149d56 100644
--- a/electron/describer/instructions.ts
+++ b/electron/describer/instructions.ts
@@ -29,6 +29,10 @@ A low-frame-rate **screen video** may also exist. It is OPPORTUNISTIC enrichment
— you pull frames only where the events are ambiguous. Do NOT assume you must
look at video; most steps are fully explained by events alone.
+The user may also have recorded **voice narration** — spoken commentary describing
+what they were doing. When present, it is the single most direct statement of their
+intent. Read it early via get_narration.
+
All times are **\`atMs\` = milliseconds since the recording started**.
## Your tools
@@ -36,6 +40,9 @@ All times are **\`atMs\` = milliseconds since the recording started**.
commands / clipboard counts / markers) with their \`atMs\` start + duration. Start here.
- **get_events({ types?, fromMs?, toMs? })** — the raw event stream (with clipboard
text, full titles, full URLs, commands). Use to inspect a specific window closely.
+- **get_narration({ query? })** — the user's spoken narration as timestamped lines,
+ in their own words. Optionally \`query\` to grep it. Absent/empty means the user did
+ not narrate. When it exists, let it lead the intent and step ordering.
- **list_frames** — index of screen frames already available (file + \`atMs\` + why kept).
Empty/absent means no video was recorded.
- **get_frames({ fromMs, toMs, fps?, crop?, reason? })** — sample and **view** screen
@@ -47,15 +54,17 @@ All times are **\`atMs\` = milliseconds since the recording started**.
## Method
1. **Read the timeline** (get_timeline) to get the shape of the session.
-2. **Form a hypothesis** about the overall intent from apps + urls + commands.
-3. **Read events** (get_events) around anything unclear — clipboard text, exact
+2. **Read any narration** (get_narration). If the user narrated, their words state
+ the intent directly — anchor your hypothesis and step names to them.
+3. **Form a hypothesis** about the overall intent from apps + urls + commands.
+4. **Read events** (get_events) around anything unclear — clipboard text, exact
URLs, the sequence of title changes.
-4. **Look at frames ONLY where events are silent or ambiguous** (get_frames): e.g. a
+5. **Look at frames ONLY where events are silent or ambiguous** (get_frames): e.g. a
step with a visual change but no explaining event, a clipboard copy whose purpose
is unclear, or a terminal step with no captured command. Budget ~5 frames for a
~30–60s session. Cost should scale with ambiguity, not video length.
-5. **Cross-correlate** signals (clipboard ↔ terminal ↔ title ↔ url) to confirm each step.
-6. **Call submit_analysis** with the intent and ordered steps.
+6. **Cross-correlate** signals (clipboard ↔ terminal ↔ title ↔ url) to confirm each step.
+7. **Call submit_analysis** with the intent and ordered steps.
## Noise to ignore
- **The Skill Recorder app itself** (this Electron recorder, app name "Skill Recorder").
diff --git a/electron/describer/tools.ts b/electron/describer/tools.ts
index a81993d..c764222 100644
--- a/electron/describer/tools.ts
+++ b/electron/describer/tools.ts
@@ -6,6 +6,7 @@ import type { Tool } from "@github/copilot-sdk";
import { AnalysisSubmissionSchema, type AnalysisSubmission } from "../../common/analysis";
import { MEANINGFUL_EVENT_TYPES } from "../../common/correlation";
import type { SessionBundle } from "../../common/bundle";
+import { NARRATION_FILE, type NarrationTranscript } from "../../common/narration";
import type { RecEvent } from "../../common/types";
import { readEvents } from "../frames/correlate";
import type { FrameExtractor } from "../frames/extractor";
@@ -44,6 +45,16 @@ function readBundle(sessionDir: string): SessionBundle | null {
}
}
+function readNarration(sessionDir: string): NarrationTranscript | null {
+ const p = path.join(sessionDir, NARRATION_FILE);
+ if (!existsSync(p)) return null;
+ try {
+ return JSON.parse(readFileSync(p, "utf8")) as NarrationTranscript;
+ } catch {
+ return null;
+ }
+}
+
/**
* Build the custom in-process tools exposed to one Copilot session. Handlers run
* in the Electron main process (dispatched over JSON-RPC by the SDK), read the
@@ -58,6 +69,10 @@ export function createDescriberTools(ctx: ToolContext): Tool[] {
const rel = (epoch: number) => Math.round(epoch - startedAt);
const abs = (atMs: number) => startedAt + atMs;
const sec = (atMs: number) => (atMs / 1000).toFixed(1);
+ const mmss = (atMs: number) => {
+ const total = Math.max(0, Math.round(atMs / 1000));
+ return `${String(Math.floor(total / 60)).padStart(2, "0")}:${String(total % 60).padStart(2, "0")}`;
+ };
const getTimeline: Tool = {
name: "get_timeline",
@@ -262,6 +277,38 @@ export function createDescriberTools(ctx: ToolContext): Tool[] {
},
};
+ const getNarration: Tool = {
+ name: "get_narration",
+ description:
+ "Return the user's spoken voice narration for this session as timestamped lines ([mm:ss] text). This is the user's own words describing what they are doing and why, so treat it as the most direct statement of intent and use it to name and order the steps. Optionally pass `query` to return only lines containing that text (case-insensitive). Returns a note when the user recorded no narration.",
+ parameters: {
+ type: "object",
+ properties: {
+ query: {
+ type: "string",
+ description: "Only return narration lines containing this text (case-insensitive).",
+ },
+ },
+ additionalProperties: false,
+ },
+ handler: (raw) => {
+ const args = (raw ?? {}) as { query?: string };
+ const transcript = readNarration(sessionDir);
+ if (!transcript || transcript.segments.length === 0) {
+ return "The user did not record any voice narration for this session.";
+ }
+ progress(args.query ? `Searching narration for “${args.query}”…` : "Reading the voice narration…");
+ const q = args.query?.trim().toLowerCase();
+ const lines = transcript.segments
+ .filter((s) => (q ? s.text.toLowerCase().includes(q) : true))
+ .map((s) => `[${mmss(s.atMs)}] ${s.text}`);
+ if (lines.length === 0) {
+ return `No narration lines match “${args.query}”. Call get_narration with no query to read the whole transcript.`;
+ }
+ return lines.join("\n");
+ },
+ };
+
const submitAnalysis: Tool = {
name: "submit_analysis",
description:
@@ -315,5 +362,5 @@ export function createDescriberTools(ctx: ToolContext): Tool[] {
},
};
- return [getTimeline, getEvents, listFrames, getFrames, submitAnalysis];
+ return [getTimeline, getEvents, getNarration, listFrames, getFrames, submitAnalysis];
}
diff --git a/electron/ipc.ts b/electron/ipc.ts
index e4a333e..3df6990 100644
--- a/electron/ipc.ts
+++ b/electron/ipc.ts
@@ -12,6 +12,7 @@ import type {
SkillBuildInput,
SkillCreateResult,
SkillPlanResult,
+ StartOptions,
} from "../common/ipc";
import { IPC } from "../common/ipc";
import { AutomationBuilder, loadPersistedAutomation } from "./automationbuilder/builder";
@@ -32,7 +33,7 @@ export function registerIpc(
builder: SkillBuilder,
automationBuilder: AutomationBuilder,
): void {
- ipcMain.handle(IPC.start, () => recorder.start());
+ ipcMain.handle(IPC.start, (_event, options?: StartOptions) => recorder.start(options));
ipcMain.handle(IPC.stop, () => recorder.stop());
ipcMain.handle(IPC.status, () => recorder.status());
ipcMain.handle(IPC.marker, (_event, note: string) => recorder.marker(note));
diff --git a/electron/main.ts b/electron/main.ts
index dc06391..6b606d6 100644
--- a/electron/main.ts
+++ b/electron/main.ts
@@ -11,6 +11,7 @@ import { RecorderController } from "./recorder/controller";
import { SkillBuilder } from "./skillbuilder/builder";
import { AutomationBuilder } from "./automationbuilder/builder";
import { createTray } from "./tray";
+import { AudioRecorder } from "./audio/recorder";
import { VideoRecorder } from "./video/recorder";
import { createLibraryWindow, createRecorderWindow, redockLibrary } from "./window";
@@ -23,6 +24,7 @@ const recorder = new RecorderController({
resolveConfig: () => ({ ...FULL_CAPTURE }),
buildCollectors: createCollectors,
createVideoRecorder: () => new VideoRecorder(),
+ createAudioRecorder: () => new AudioRecorder(),
postProcess: processSession,
});
diff --git a/electron/narration/transcribe.ts b/electron/narration/transcribe.ts
new file mode 100644
index 0000000..e28410c
--- /dev/null
+++ b/electron/narration/transcribe.ts
@@ -0,0 +1,185 @@
+import { execFile } from "node:child_process";
+import { createRequire } from "node:module";
+import { promisify } from "node:util";
+
+import type { NarrationSegment } from "../../common/narration";
+import { createLogger } from "../logger";
+import { getAsrPipeline, narrationModelId } from "./whisper";
+
+const log = createLogger("Narration/transcribe");
+const require = createRequire(import.meta.url);
+const execFileAsync = promisify(execFile);
+
+const ffmpegPath = require("ffmpeg-static") as string;
+
+// Whisper is trained on 16 kHz mono audio.
+const SAMPLE_RATE = 16_000;
+// f32le at 16 kHz mono is 64 KB per second; 1 GB covers ~4.5 h of narration.
+const MAX_PCM_BUFFER = 1024 * 1024 * 1024;
+
+/** A silence interval in the decoded audio's own clock (seconds). */
+interface SilenceSpan {
+ start: number;
+ end: number;
+}
+
+/**
+ * Transcribe one session's narration webm into session-clock segments.
+ *
+ * @param audioPath absolute path to `audio.webm`
+ * @param anchorDeltaMs `audio.startEpoch - session.startedAt` — added to every
+ * audio-local timestamp so segments land on the same clock as step offsets.
+ * @param durationMs recorded narration duration, used to close the final chunk.
+ */
+export async function transcribeNarration(
+ audioPath: string,
+ anchorDeltaMs: number,
+ durationMs: number,
+): Promise<{ model: string; segments: NarrationSegment[] } | null> {
+ const pipe = await getAsrPipeline();
+ if (!pipe) return null;
+
+ let samples: Float32Array;
+ try {
+ samples = await decodePcm(audioPath);
+ } catch (err) {
+ log.warn("failed to decode narration audio:", err instanceof Error ? err.message : err);
+ return null;
+ }
+ if (samples.length === 0) {
+ log.warn("narration audio decoded to zero samples");
+ return null;
+ }
+
+ // Silence detection runs on the same file and only reads the audio, so its
+ // intervals line up with the decoded samples and never shift timestamps.
+ const silences = await detectSilence(audioPath);
+
+ let result: { text: string; chunks?: Array<{ timestamp: [number, number | null]; text: string }> };
+ try {
+ result = await pipe(samples, {
+ return_timestamps: true,
+ chunk_length_s: 30,
+ stride_length_s: 5,
+ });
+ } catch (err) {
+ log.warn("whisper transcription failed:", err instanceof Error ? err.message : err);
+ return null;
+ }
+
+ const durationSec = durationMs > 0 ? durationMs / 1000 : samples.length / SAMPLE_RATE;
+ const raw = result.chunks?.length
+ ? result.chunks
+ : [{ timestamp: [0, durationSec] as [number, number | null], text: result.text ?? "" }];
+
+ const segments: NarrationSegment[] = [];
+ for (const chunk of raw) {
+ const text = (chunk.text ?? "").trim();
+ if (!isMeaningful(text)) continue;
+
+ const startSec = chunk.timestamp?.[0] ?? 0;
+ const endSec = chunk.timestamp?.[1] ?? Math.min(durationSec, startSec + 2);
+ if (isMostlySilent(startSec, endSec, silences)) continue;
+
+ segments.push({
+ atMs: Math.max(0, Math.round(startSec * 1000 + anchorDeltaMs)),
+ endMs: Math.max(0, Math.round(endSec * 1000 + anchorDeltaMs)),
+ text,
+ });
+ }
+
+ segments.sort((a, b) => a.atMs - b.atMs);
+ log.info(`transcribed ${raw.length} chunks -> ${segments.length} narration segments`);
+ return { model: narrationModelId(), segments };
+}
+
+/** Decode any container into raw 16 kHz mono float samples (no WAV/dep needed). */
+async function decodePcm(audioPath: string): Promise<Float32Array> {
+ const { stdout } = await execFileAsync(
+ ffmpegPath,
+ [
+ "-hide_banner",
+ "-loglevel", "error",
+ "-i", audioPath,
+ "-ac", "1",
+ "-ar", String(SAMPLE_RATE),
+ "-f", "f32le",
+ "pipe:1",
+ ],
+ { encoding: "buffer", maxBuffer: MAX_PCM_BUFFER },
+ );
+
+ const buf = stdout as unknown as Buffer;
+ // Copy into a fresh, 4-byte-aligned ArrayBuffer before viewing as Float32:
+ // execFile's Buffer offset is not guaranteed aligned.
+ const usableBytes = buf.byteLength - (buf.byteLength % 4);
+ const ab = new ArrayBuffer(usableBytes);
+ new Uint8Array(ab).set(buf.subarray(0, usableBytes));
+ return new Float32Array(ab);
+}
+
+/** Run ffmpeg's silencedetect and parse the intervals it logs to stderr. */
+async function detectSilence(audioPath: string): Promise<SilenceSpan[]> {
+ let stderr = "";
+ try {
+ const res = await execFileAsync(
+ ffmpegPath,
+ [
+ "-hide_banner",
+ "-i", audioPath,
+ "-af", "silencedetect=noise=-35dB:d=0.6",
+ "-f", "null",
+ "-",
+ ],
+ { maxBuffer: 8 * 1024 * 1024 },
+ );
+ stderr = res.stderr;
+ } catch (err) {
+ // silencedetect over the null muxer normally exits 0; treat any failure as
+ // "no silence info" rather than aborting transcription.
+ stderr = (err as { stderr?: string }).stderr ?? "";
+ }
+
+ const spans: SilenceSpan[] = [];
+ let pendingStart: number | null = null;
+ for (const m of stderr.matchAll(/silence_(start|end):\s*(-?[0-9.]+)/g)) {
+ const kind = m[1];
+ const value = Number(m[2]);
+ if (kind === "start") {
+ pendingStart = value;
+ } else if (pendingStart != null) {
+ spans.push({ start: pendingStart, end: value });
+ pendingStart = null;
+ }
+ }
+ return spans;
+}
+
+/** A chunk is dropped when its midpoint sits inside a detected silence span. */
+function isMostlySilent(startSec: number, endSec: number, silences: SilenceSpan[]): boolean {
+ const mid = (startSec + endSec) / 2;
+ return silences.some((s) => mid >= s.start && mid <= s.end);
+}
+
+// Whisper hallucinates stock phrases over silence/noise; drop the usual suspects
+// (and anything that carries no letters) so they never reach the analysis.
+const BOILERPLATE = new Set([
+ "you",
+ "thank you",
+ "thanks",
+ "thank you.",
+ "thanks for watching",
+ "thanks for watching!",
+ "please subscribe",
+ "subtitles by the amara.org community",
+ "bye",
+ "bye.",
+ ".",
+]);
+
+function isMeaningful(text: string): boolean {
+ if (text.length < 2) return false;
+ if (!/[a-z]/i.test(text)) return false;
+ const normalized = text.toLowerCase().replace(/\s+/g, " ").trim();
+ return !BOILERPLATE.has(normalized);
+}
diff --git a/electron/narration/whisper.ts b/electron/narration/whisper.ts
new file mode 100644
index 0000000..8af5557
--- /dev/null
+++ b/electron/narration/whisper.ts
@@ -0,0 +1,80 @@
+import path from "node:path";
+
+import { app } from "electron";
+
+import { createLogger } from "../logger";
+
+const log = createLogger("Narration/whisper");
+
+// Whisper small.en: English-only, int8 (q8) ONNX weights (~250 MB). Downloaded
+// once into the user-data models cache, then used fully offline. base.en is the
+// lighter fallback if this ever proves too heavy on low-end machines.
+const MODEL_ID = "Xenova/whisper-small.en";
+const DTYPE = "q8";
+
+/**
+ * The subset of the transformers.js ASR pipeline we actually call. Typing it as a
+ * plain callable avoids fighting the library's large pipeline union while keeping
+ * the return shape we depend on explicit.
+ */
+export type AsrPipeline = (
+ audio: Float32Array,
+ options?: Record<string, unknown>,
+) => Promise<{
+ text: string;
+ chunks?: Array<{ timestamp: [number, number | null]; text: string }>;
+}>;
+
+let pipePromise: Promise<AsrPipeline | null> | null = null;
+
+/** The model id transcripts are tagged with. */
+export function narrationModelId(): string {
+ return MODEL_ID;
+}
+
+async function build(): Promise<AsrPipeline | null> {
+ let tf: typeof import("@huggingface/transformers");
+ try {
+ // Dynamic import: transformers.js + onnxruntime-node are heavy native deps we
+ // only ever touch when a session actually has narration to transcribe.
+ tf = await import("@huggingface/transformers");
+ } catch (err) {
+ log.warn("transformers.js unavailable; narration disabled:", err instanceof Error ? err.message : err);
+ return null;
+ }
+
+ // Keep every downloaded model under the app's user-data dir so it survives
+ // upgrades and is trivial to locate or clear. First run needs the network once;
+ // every run after is offline from this cache. Nothing is bundled with the app.
+ tf.env.cacheDir = path.join(app.getPath("userData"), "models");
+ tf.env.allowLocalModels = false;
+
+ try {
+ const pipe = await tf.pipeline("automatic-speech-recognition", MODEL_ID, {
+ dtype: DTYPE,
+ // onnxruntime-node's CPU memory arena SIGTRAPs (hard native crash) under
+ // Electron's runtime, in both the main and utility processes. Disabling the
+ // arena is the single option that avoids it; unlike throttling threads or
+ // graph optimization it keeps multithreading and fusions on, so transcription
+ // stays near-native speed (~12s for 77s of audio) with no change in output.
+ // If this ever needs more isolation, move inference to a forked utility
+ // process (it still needs this flag) so a future native fault can't take the
+ // app down with it.
+ session_options: { enableCpuMemArena: false },
+ });
+ return pipe as unknown as AsrPipeline;
+ } catch (err) {
+ log.warn("failed to load whisper model (offline + not yet cached?):", err instanceof Error ? err.message : err);
+ return null;
+ }
+}
+
+/**
+ * Lazily builds and caches the ASR pipeline for the process. Returns null when
+ * the engine or model can't be loaded, so callers degrade cleanly to "no
+ * narration" instead of failing the analysis.
+ */
+export function getAsrPipeline(): Promise<AsrPipeline | null> {
+ if (!pipePromise) pipePromise = build();
+ return pipePromise;
+}
diff --git a/electron/pipeline.ts b/electron/pipeline.ts
index 30e939c..b1430f4 100644
--- a/electron/pipeline.ts
+++ b/electron/pipeline.ts
@@ -5,10 +5,13 @@ import { buildBundle } from "../common/bundle";
import { MEANINGFUL_EVENT_TYPES } from "../common/correlation";
import type { CorrelationResult } from "../common/correlation";
import { renderDescription } from "../common/describe";
+import { NARRATION_FILE } from "../common/narration";
import type { SessionMeta } from "../common/types";
+import type { AudioResult } from "./audio/recorder";
import { CorrelationEngine, readEvents } from "./frames/correlate";
import { FrameExtractor } from "./frames/extractor";
import { createLogger } from "./logger";
+import { transcribeNarration } from "./narration/transcribe";
import type { VideoResult } from "./video/recorder";
const log = createLogger("Pipeline");
@@ -74,6 +77,49 @@ async function runFrameStage(sessionDir: string): Promise<CorrelationResult | nu
}
}
+/**
+ * Optional narration stage: if the user opted into voice for this session, an
+ * `audio.webm` + `audio.json` pair exists. Transcribe it offline (Whisper via
+ * transformers.js) into session-clock segments and write `narration.json`. The
+ * transcript is the user's own stated intent; it is never merged into
+ * `events.jsonl` and is read only through the describer's `get_narration` tool.
+ * Best-effort: a missing model, engine, or mic degrades to no narration.
+ */
+async function runNarrationStage(sessionDir: string, meta: SessionMeta): Promise<void> {
+ const audioJsonPath = path.join(sessionDir, "audio.json");
+ if (!existsSync(audioJsonPath)) return;
+
+ let audio: AudioResult;
+ try {
+ audio = JSON.parse(readFileSync(audioJsonPath, "utf8")) as AudioResult;
+ } catch (err) {
+ log.warn("unreadable audio.json; skipping narration:", err instanceof Error ? err.message : err);
+ return;
+ }
+
+ const audioPath = path.join(sessionDir, audio.file);
+ if (!existsSync(audioPath)) {
+ log.warn("narration audio missing; skipping narration stage");
+ return;
+ }
+
+ try {
+ const transcript = await transcribeNarration(
+ audioPath,
+ audio.startEpoch - meta.startedAt,
+ audio.durationMs,
+ );
+ if (!transcript || transcript.segments.length === 0) {
+ log.info("no usable narration produced");
+ return;
+ }
+ writeFileSync(path.join(sessionDir, NARRATION_FILE), JSON.stringify(transcript, null, 2));
+ log.info(`narration: ${transcript.segments.length} segments → ${NARRATION_FILE}`);
+ } catch (err) {
+ log.warn("narration stage failed:", err instanceof Error ? err.message : err);
+ }
+}
+
/**
* Post-stop processing for a completed session. Always produces `bundle.json`
* (segmented steps) and `description.md` (baseline narrative) from the primary
@@ -88,7 +134,13 @@ export async function processSession(sessionDir: string): Promise<void> {
}
const events = readEvents(path.join(sessionDir, "events.jsonl"));
- const correlation = await runFrameStage(sessionDir);
+ // Frame correlation (ffmpeg) and narration transcription (Whisper) are
+ // independent; run them together so a first-run model download overlaps with
+ // frame extraction instead of adding to it.
+ const [correlation] = await Promise.all([
+ runFrameStage(sessionDir),
+ runNarrationStage(sessionDir, meta),
+ ]);
try {
const bundle = buildBundle({ meta, events, correlation });
diff --git a/electron/preload.cjs b/electron/preload.cjs
index 1f1beb5..648b0d4 100644
--- a/electron/preload.cjs
+++ b/electron/preload.cjs
@@ -34,7 +34,7 @@ const IPC = {
};
contextBridge.exposeInMainWorld("skillRecorder", {
- start: () => ipcRenderer.invoke(IPC.start),
+ start: (options) => ipcRenderer.invoke(IPC.start, options),
stop: () => ipcRenderer.invoke(IPC.stop),
status: () => ipcRenderer.invoke(IPC.status),
marker: (note) => ipcRenderer.invoke(IPC.marker, note),
diff --git a/electron/recorder/controller.ts b/electron/recorder/controller.ts
index 212cb91..b5434df 100644
--- a/electron/recorder/controller.ts
+++ b/electron/recorder/controller.ts
@@ -4,7 +4,7 @@ import { app } from "electron";
import type { CaptureConfig } from "../../common/config";
import { EventType } from "../../common/events";
-import type { MarkerResult, RecorderStatus, StartResult, StopResult } from "../../common/ipc";
+import type { MarkerResult, RecorderStatus, StartOptions, StartResult, StopResult } from "../../common/ipc";
import type { RecorderState, SessionMeta } from "../../common/types";
import { createLogger } from "../logger";
import type { Collector } from "./collector";
@@ -35,6 +35,8 @@ export interface RecorderDeps {
buildCollectors: (config: CaptureConfig) => readonly Collector[];
/** Optional factory for the video sidecar; only used when `config.video`. */
createVideoRecorder?: () => SessionMediaRecorder;
+ /** Optional factory for the narration sidecar; only used when the user opts in. */
+ createAudioRecorder?: () => SessionMediaRecorder;
/** Optional best-effort post-processing (frames + correlation) after stop. */
postProcess?: (sessionDir: string) => Promise<void>;
}
@@ -53,6 +55,7 @@ export class RecorderController {
private readonly host: CollectorHost;
private activeCollectors: readonly Collector[] = [];
private video: SessionMediaRecorder | null = null;
+ private audio: SessionMediaRecorder | null = null;
private processing: Promise<void> | null = null;
private lastCompleted: { id: string; dir: string } | null = null;
private lastProcessed = false;
@@ -108,7 +111,7 @@ export class RecorderController {
for (const cb of this.listeners) cb(s);
}
- async start(): Promise<StartResult> {
+ async start(options?: StartOptions): Promise<StartResult> {
if (this.store) return { ok: false, error: "Already recording" };
const meta: SessionMeta = {
id: makeSessionId(),
@@ -142,9 +145,25 @@ export class RecorderController {
this.video = null;
}
}
+
+ // Narration is strictly opt-in (the HUD toggle), off by default, and just as
+ // best-effort as video: a missing mic or denied permission never fails the run.
+ if (options?.narration && this.deps.createAudioRecorder) {
+ this.audio = this.deps.createAudioRecorder();
+ try {
+ await this.audio.start(store.dir);
+ } catch (err) {
+ log.warn("narration start failed:", err instanceof Error ? err.message : err);
+ this.audio = null;
+ }
+ }
return { ok: true, sessionId: meta.id };
}
+ // Marker capture is intentionally retained even though the "Add marker" HUD
+ // button was removed in favor of voice narration (see docs/future-features.md).
+ // The full path (IPC -> event bus -> bundle -> describer) stays wired so a
+ // future UI (e.g. a silent hotkey flag) can re-enable it without backend work.
marker(note: string): MarkerResult {
if (!this.store) return { ok: false, error: "Not recording" };
this.bus.publish({ type: EventType.Marker, source: "user", payload: { note } });
@@ -164,6 +183,14 @@ export class RecorderController {
}
this.video = null;
}
+ if (this.audio) {
+ try {
+ await this.audio.stop();
+ } catch (err) {
+ log.warn("narration stop failed:", err instanceof Error ? err.message : err);
+ }
+ this.audio = null;
+ }
this.bus.publish({ type: EventType.SessionStop, source: "recorder", payload: {} });
this.bus.detach();
this.store = null;
diff --git a/package-lock.json b/package-lock.json
index f36b6ac..b40c692 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -9,6 +9,7 @@
"version": "0.1.0",
"dependencies": {
"@github/copilot-sdk": "^1.0.6",
+ "@huggingface/transformers": "^4.2.0",
"ffmpeg-static": "^5.3.0",
"get-windows": "^9.3.0",
"react": "^19.2.5",
@@ -545,6 +546,34 @@
"copilot-win32-x64": "copilot.exe"
}
},
+ "node_modules/@huggingface/jinja": {
+ "version": "0.5.9",
+ "resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@huggingface/jinja/-/jinja-0.5.9.tgz",
+ "integrity": "sha1-KU90H6CYwrMXN4i0TKZRlYv/o20=",
+ "license": "MIT",
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@huggingface/tokenizers": {
+ "version": "0.1.3",
+ "resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@huggingface/tokenizers/-/tokenizers-0.1.3.tgz",
+ "integrity": "sha1-0bsrJTdeVQyCbkxxUdX3ZKFKamk=",
+ "license": "Apache-2.0"
+ },
+ "node_modules/@huggingface/transformers": {
+ "version": "4.2.0",
+ "resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@huggingface/transformers/-/transformers-4.2.0.tgz",
+ "integrity": "sha1-WlNCoaFIxdKX7YOE0hzuGCT6sDE=",
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@huggingface/jinja": "^0.5.6",
+ "@huggingface/tokenizers": "^0.1.3",
+ "onnxruntime-node": "1.24.3",
+ "onnxruntime-web": "1.26.0-dev.20260416-b7804b056c",
+ "sharp": "^0.34.5"
+ }
+ },
"node_modules/@img/colour": {
"version": "1.1.0",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@img/colour/-/colour-1.1.0.tgz",
@@ -1689,6 +1718,63 @@
"node": ">=14"
}
},
+ "node_modules/@protobufjs/aspromise": {
+ "version": "1.1.2",
+ "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",
+ "integrity": "sha1-m4sMxmPWaafY9vXQiToU00jzD78=",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/base64": {
+ "version": "1.1.2",
+ "resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/base64/-/base64-1.1.2.tgz",
+ "integrity": "sha1-TIVzDlm5ofHzSQR9vyQpYDS7JzU=",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/codegen": {
+ "version": "2.0.5",
+ "resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/codegen/-/codegen-2.0.5.tgz",
+ "integrity": "sha1-2TFa188/MKrHC9o8BoRD3G8UNlk=",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/eventemitter": {
+ "version": "1.1.1",
+ "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/eventemitter/-/eventemitter-1.1.1.tgz",
+ "integrity": "sha1-1RLLJsCuAmCR7iwRZ/G+b69chCo=",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/fetch": {
+ "version": "1.1.1",
+ "resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/fetch/-/fetch-1.1.1.tgz",
+ "integrity": "sha1-TW/ADI+2QBalyBtGnVSQRjUPEGU=",
+ "license": "BSD-3-Clause",
+ "dependencies": {
+ "@protobufjs/aspromise": "^1.1.1"
+ }
+ },
+ "node_modules/@protobufjs/float": {
+ "version": "1.0.2",
+ "resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/float/-/float-1.0.2.tgz",
+ "integrity": "sha1-Xp4avctz/Ap8uLKR33jIy9l7h9E=",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/path": {
+ "version": "1.1.2",
+ "resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/path/-/path-1.1.2.tgz",
+ "integrity": "sha1-bMKyDFya1q0NzP0hynZz2Nf79o0=",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/pool": {
+ "version": "1.1.0",
+ "resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/pool/-/pool-1.1.0.tgz",
+ "integrity": "sha1-Cf0V8tbTq/qbZbw2ZQbWrXhG/1Q=",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/utf8": {
+ "version": "1.1.2",
+ "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@protobufjs/utf8/-/utf8-1.1.2.tgz",
+ "integrity": "sha1-eNR2Mz2F1bHHkuJXvKdLoIDaSaQ=",
+ "license": "BSD-3-Clause"
+ },
"node_modules/@rolldown/binding-android-arm64": {
"version": "1.1.5",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@rolldown/binding-android-arm64/-/binding-android-arm64-1.1.5.tgz",
@@ -2062,7 +2148,6 @@
"version": "22.20.1",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/@types/node/-/node-22.20.1.tgz",
"integrity": "sha1-hOfN9jzaogwTSqMXzMkBqiHhbw4=",
- "dev": true,
"license": "MIT",
"dependencies": {
"undici-types": "~6.21.0"
@@ -2494,6 +2579,15 @@
"node": ">=0.4.0"
}
},
+ "node_modules/adm-zip": {
+ "version": "0.5.18",
+ "resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/adm-zip/-/adm-zip-0.5.18.tgz",
+ "integrity": "sha1-KDoF8r8eP9MV8PMc3im3puPBYZ0=",
+ "license": "MIT",
+ "engines": {
+ "node": ">=12.0"
+ }
+ },
"node_modules/agent-base": {
"version": "7.1.4",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/agent-base/-/agent-base-7.1.4.tgz",
@@ -2852,9 +2946,7 @@
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/boolean/-/boolean-3.2.0.tgz",
"integrity": "sha1-nlKUr06YMUSUy7F5efpUyhWfEWs=",
"deprecated": "Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.",
- "dev": true,
- "license": "MIT",
- "optional": true
+ "license": "MIT"
},
"node_modules/brace-expansion": {
"version": "5.0.7",
@@ -3486,9 +3578,7 @@
"version": "1.1.4",
"resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/define-data-property/-/define-data-property-1.1.4.tgz",
"integrity": "sha1-iU3BQbt9MGCuQ2b2oBB+aPvkjF4=",
- "dev": true,
"license": "MIT",
- "optional": true,
"dependencies": {
"es-define-property": "^1.0.0",
"es-errors": "^1.3.0",
@@ -3505,9 +3595,7 @@
"version": "1.2.1",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/define-properties/-/define-properties-1.2.1.tgz",
"integrity": "sha1-EHgcxhbrlRqAoDS6/Kpzd/avK2w=",
- "dev": true,
"license": "MIT",
- "optional": true,
"dependencies": {
"define-data-property": "^1.0.1",
"has-property-descriptors": "^1.0.0",
@@ -3550,9 +3638,7 @@
"version": "2.1.0",
"resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/detect-node/-/detect-node-2.1.0.tgz",
"integrity": "sha1-yccHdaScPQO8LAbZpzvlUPl4+LE=",
- "dev": true,
- "license": "MIT",
- "optional": true
+ "license": "MIT"
},
"node_modules/dir-compare": {
"version": "4.2.0",
@@ -3923,7 +4009,6 @@
"version": "1.0.1",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/es-define-property/-/es-define-property-1.0.1.tgz",
"integrity": "sha1-mD6y+aZyTpMD9hrd8BHHLgngsPo=",
- "dev": true,
"license": "MIT",
"engines": {
"node": ">= 0.4"
@@ -3933,7 +4018,6 @@
"version": "1.3.0",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/es-errors/-/es-errors-1.3.0.tgz",
"integrity": "sha1-BfdaJdq5jk+x3NXhRywFRtUFfI8=",
- "dev": true,
"license": "MIT",
"engines": {
"node": ">= 0.4"
@@ -3972,9 +4056,7 @@
"version": "4.1.1",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/es6-error/-/es6-error-4.1.1.tgz",
"integrity": "sha1-njr0B0Wd7tR+mpH5uIWoTrBcVh0=",
- "dev": true,
- "license": "MIT",
- "optional": true
+ "license": "MIT"
},
"node_modules/escalade": {
"version": "3.2.0",
@@ -3990,9 +4072,7 @@
"version": "4.0.0",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/escape-string-regexp/-/escape-string-regexp-4.0.0.tgz",
"integrity": "sha1-FLqDpdNz49MR5a/KKc9b+tllvzQ=",
- "dev": true,
"license": "MIT",
- "optional": true,
"engines": {
"node": ">=10"
},
@@ -4146,6 +4226,12 @@
"node": ">=10"
}
},
+ "node_modules/flatbuffers": {
+ "version": "25.9.23",
+ "resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/flatbuffers/-/flatbuffers-25.9.23.tgz",
+ "integrity": "sha1-NGgRVX/pMSq1ZHU155PHYenIHrE=",
+ "license": "Apache-2.0"
+ },
"node_modules/foreground-child": {
"version": "3.3.1",
"resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/foreground-child/-/foreground-child-3.3.1.tgz",
@@ -4656,9 +4742,7 @@
"version": "3.0.0",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/global-agent/-/global-agent-3.0.0.tgz",
"integrity": "sha1-rnzTG9NYO5PFoWQ3oa/ifMM6GrY=",
- "dev": true,
"license": "BSD-3-Clause",
- "optional": true,
"dependencies": {
"boolean": "^3.0.1",
"es6-error": "^4.1.1",
@@ -4675,9 +4759,7 @@
"version": "1.0.4",
"resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/globalthis/-/globalthis-1.0.4.tgz",
"integrity": "sha1-dDDtOpddl7+1m8zkH1yruvplEjY=",
- "dev": true,
"license": "MIT",
- "optional": true,
"dependencies": {
"define-properties": "^1.2.1",
"gopd": "^1.0.1"
@@ -4693,7 +4775,6 @@
"version": "1.2.0",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/gopd/-/gopd-1.2.0.tgz",
"integrity": "sha1-ifVrghe9vIgCvSmd9tfxCB1+UaE=",
- "dev": true,
"license": "MIT",
"engines": {
"node": ">= 0.4"
@@ -4735,6 +4816,12 @@
"devOptional": true,
"license": "ISC"
},
+ "node_modules/guid-typescript": {
+ "version": "1.0.9",
+ "resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/guid-typescript/-/guid-typescript-1.0.9.tgz",
+ "integrity": "sha1-4193ADU1sCl+oIVI9azmrbFIDdw=",
+ "license": "ISC"
+ },
"node_modules/has-flag": {
"version": "4.0.0",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/has-flag/-/has-flag-4.0.0.tgz",
@@ -4749,9 +4836,7 @@
"version": "1.0.2",
"resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/has-property-descriptors/-/has-property-descriptors-1.0.2.tgz",
"integrity": "sha1-lj7X0HHce/XwhMW/vg0bYiJYaFQ=",
- "dev": true,
"license": "MIT",
- "optional": true,
"dependencies": {
"es-define-property": "^1.0.0"
},
@@ -5078,9 +5163,7 @@
"version": "5.0.1",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/json-stringify-safe/-/json-stringify-safe-5.0.1.tgz",
"integrity": "sha1-Epai1Y/UXxmg9s4B1lcB4sc1tus=",
- "dev": true,
- "license": "ISC",
- "optional": true
+ "license": "ISC"
},
"node_modules/json5": {
"version": "2.2.3",
@@ -5438,6 +5521,12 @@
"dev": true,
"license": "MIT"
},
+ "node_modules/long": {
+ "version": "5.3.2",
+ "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/long/-/long-5.3.2.tgz",
+ "integrity": "sha1-HYRGMJWZkmLX17f4v9SozFUWf4M=",
+ "license": "Apache-2.0"
+ },
"node_modules/lowercase-keys": {
"version": "2.0.0",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/lowercase-keys/-/lowercase-keys-2.0.0.tgz",
@@ -5525,9 +5614,7 @@
"version": "3.0.0",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/matcher/-/matcher-3.0.0.tgz",
"integrity": "sha1-vZBg9MW3CqgEHMxvgDaHYJlPMMo=",
- "dev": true,
"license": "MIT",
- "optional": true,
"dependencies": {
"escape-string-regexp": "^4.0.0"
},
@@ -6072,9 +6159,7 @@
"version": "1.1.1",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/object-keys/-/object-keys-1.1.1.tgz",
"integrity": "sha1-HEfyct8nfzsdrwYWd9nILiMixg4=",
- "dev": true,
"license": "MIT",
- "optional": true,
"engines": {
"node": ">= 0.4"
}
@@ -6089,6 +6174,49 @@
"wrappy": "1"
}
},
+ "node_modules/onnxruntime-common": {
+ "version": "1.24.3",
+ "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/onnxruntime-common/-/onnxruntime-common-1.24.3.tgz",
+ "integrity": "sha1-+5Mxmi0EH8n+KiCRZ7pu8vGkf+4=",
+ "license": "MIT"
+ },
+ "node_modules/onnxruntime-node": {
+ "version": "1.24.3",
+ "resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/onnxruntime-node/-/onnxruntime-node-1.24.3.tgz",
+ "integrity": "sha1-7I2pnDNapeqVVZo6wa6RWolUHV4=",
+ "hasInstallScript": true,
+ "license": "MIT",
+ "os": [
+ "win32",
+ "darwin",
+ "linux"
+ ],
+ "dependencies": {
+ "adm-zip": "^0.5.16",
+ "global-agent": "^3.0.0",
+ "onnxruntime-common": "1.24.3"
+ }
+ },
+ "node_modules/onnxruntime-web": {
+ "version": "1.26.0-dev.20260416-b7804b056c",
+ "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/onnxruntime-web/-/onnxruntime-web-1.26.0-dev.20260416-b7804b056c.tgz",
+ "integrity": "sha1-RT8pdog2Mipri20dcwqEJrZYKuI=",
+ "license": "MIT",
+ "dependencies": {
+ "flatbuffers": "^25.1.24",
+ "guid-typescript": "^1.0.9",
+ "long": "^5.2.3",
+ "onnxruntime-common": "1.24.0-dev.20251116-b39e144322",
+ "platform": "^1.3.6",
+ "protobufjs": "^7.2.4"
+ }
+ },
+ "node_modules/onnxruntime-web/node_modules/onnxruntime-common": {
+ "version": "1.24.0-dev.20251116-b39e144322",
+ "resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/onnxruntime-common/-/onnxruntime-common-1.24.0-dev.20251116-b39e144322.tgz",
+ "integrity": "sha1-bt7PTlF4+ddpB9WrviQcNjXlKN0=",
+ "license": "MIT"
+ },
"node_modules/p-cancelable": {
"version": "2.1.1",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/p-cancelable/-/p-cancelable-2.1.1.tgz",
@@ -6272,6 +6400,12 @@
"url": "https://paulmillr.com/funding/"
}
},
+ "node_modules/platform": {
+ "version": "1.3.6",
+ "resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/platform/-/platform-1.3.6.tgz",
+ "integrity": "sha1-SLTOmDFksgnC1FoQetsx9HOm56c=",
+ "license": "MIT"
+ },
"node_modules/plist": {
"version": "3.1.0",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/plist/-/plist-3.1.0.tgz",
@@ -6398,6 +6532,29 @@
"signal-exit": "^3.0.2"
}
},
+ "node_modules/protobufjs": {
+ "version": "7.6.5",
+ "resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/protobufjs/-/protobufjs-7.6.5.tgz",
+ "integrity": "sha1-e5JQza9KBhOanw/kaKQNfU/rynE=",
+ "hasInstallScript": true,
+ "license": "BSD-3-Clause",
+ "dependencies": {
+ "@protobufjs/aspromise": "^1.1.2",
+ "@protobufjs/base64": "^1.1.2",
+ "@protobufjs/codegen": "^2.0.5",
+ "@protobufjs/eventemitter": "^1.1.1",
+ "@protobufjs/fetch": "^1.1.1",
+ "@protobufjs/float": "^1.0.2",
+ "@protobufjs/path": "^1.1.2",
+ "@protobufjs/pool": "^1.1.0",
+ "@protobufjs/utf8": "^1.1.1",
+ "@types/node": ">=13.7.0",
+ "long": "^5.3.2"
+ },
+ "engines": {
+ "node": ">=12.0.0"
+ }
+ },
"node_modules/pump": {
"version": "3.0.4",
"resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/pump/-/pump-3.0.4.tgz",
@@ -6596,9 +6753,7 @@
"version": "2.15.4",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/roarr/-/roarr-2.15.4.tgz",
"integrity": "sha1-9f55W3uDjM/jXcYI4Cgrnrouev0=",
- "dev": true,
"license": "BSD-3-Clause",
- "optional": true,
"dependencies": {
"boolean": "^3.0.1",
"detect-node": "^2.0.4",
@@ -6714,17 +6869,13 @@
"version": "1.0.0",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/semver-compare/-/semver-compare-1.0.0.tgz",
"integrity": "sha1-De4hahyUGrN+nvsXiPavxf9VN/w=",
- "dev": true,
- "license": "MIT",
- "optional": true
+ "license": "MIT"
},
"node_modules/serialize-error": {
"version": "7.0.1",
"resolved": "https://ms-feed-25.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/serialize-error/-/serialize-error-7.0.1.tgz",
"integrity": "sha1-8TYLBEf2H/tIPsQVfHN/q313jhg=",
- "dev": true,
"license": "MIT",
- "optional": true,
"dependencies": {
"type-fest": "^0.13.1"
},
@@ -6905,9 +7056,7 @@
"version": "1.1.3",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/sprintf-js/-/sprintf-js-1.1.3.tgz",
"integrity": "sha1-SRS5A6L4toXRf994pw6RfocuREo=",
- "dev": true,
- "license": "BSD-3-Clause",
- "optional": true
+ "license": "BSD-3-Clause"
},
"node_modules/ssri": {
"version": "10.0.6",
@@ -7178,9 +7327,7 @@
"version": "0.13.1",
"resolved": "https://ms-feed-12.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/type-fest/-/type-fest-0.13.1.tgz",
"integrity": "sha1-AXLLW86AsL1ULqNI21DH4hg02TQ=",
- "dev": true,
"license": "(MIT OR CC0-1.0)",
- "optional": true,
"engines": {
"node": ">=10"
},
@@ -7251,7 +7398,6 @@
"version": "6.21.0",
"resolved": "https://ms-feed-2.pkgs.visualstudio.com/1es-public/_packaging/npm-public/npm/registry/undici-types/-/undici-types-6.21.0.tgz",
"integrity": "sha1-aR0ArzkJvpOn+qE75hs6W1DvEss=",
- "dev": true,
"license": "MIT"
},
"node_modules/unique-filename": {
diff --git a/package.json b/package.json
index 11d2fce..55d8dcc 100644
--- a/package.json
+++ b/package.json
@@ -19,6 +19,7 @@
},
"dependencies": {
"@github/copilot-sdk": "^1.0.6",
+ "@huggingface/transformers": "^4.2.0",
"ffmpeg-static": "^5.3.0",
"get-windows": "^9.3.0",
"react": "^19.2.5",
@@ -49,14 +50,19 @@
"node_modules/get-windows/**",
"node_modules/sharp/**",
"node_modules/@img/**",
- "node_modules/ffmpeg-static/**"
+ "node_modules/ffmpeg-static/**",
+ "node_modules/@huggingface/transformers/**",
+ "node_modules/onnxruntime-node/**"
],
"mac": {
"target": [
"dmg",
"zip"
],
- "category": "public.app-category.developer-tools"
+ "category": "public.app-category.developer-tools",
+ "extendInfo": {
+ "NSMicrophoneUsageDescription": "Skill Recorder uses your microphone only while you turn on Narrate during a recording, to transcribe your spoken narration on this computer."
+ }
},
"win": {
"target": [
diff --git a/src/App.css b/src/App.css
index a44fe8f..fcf3923 100644
--- a/src/App.css
+++ b/src/App.css
@@ -430,17 +430,97 @@ button:focus-visible,
min-height: 15px;
}
-.marker {
+.narrate {
+ display: flex;
+ align-items: center;
+ gap: 12px;
+ width: 100%;
+ text-align: left;
background: transparent;
border: 1px solid var(--line);
- color: var(--muted);
- font-size: 13px;
- padding: 9px;
+ color: var(--text);
+ border-radius: 12px;
+ padding: 11px 13px;
+ transition: border-color 0.15s ease, background 0.15s ease;
}
-.marker:not(:disabled):hover {
+.narrate:not(:disabled):hover {
border-color: var(--line-strong);
+ background: var(--surface);
+}
+
+.narrate.on {
+ border-color: rgba(var(--brand-rgb), 0.45);
+ background: var(--signal-dim);
+}
+
+.narrate:disabled {
+ opacity: 0.9;
+ cursor: default;
+}
+
+.narrate-icon {
+ display: flex;
+ flex: none;
+ color: var(--muted);
+}
+
+.narrate.on .narrate-icon {
+ color: var(--brand-ink);
+}
+
+.narrate-text {
+ display: flex;
+ flex-direction: column;
+ gap: 1px;
+ flex: 1;
+ min-width: 0;
+}
+
+.narrate-label {
+ font-size: 12.5px;
+ font-weight: 600;
color: var(--text);
+ line-height: 1.25;
+}
+
+.narrate-sub {
+ font-size: 11px;
+ color: var(--muted);
+ line-height: 1.25;
+}
+
+.narrate.on .narrate-sub {
+ color: var(--brand-ink);
+}
+
+.narrate-switch {
+ flex: none;
+ box-sizing: border-box;
+ width: 34px;
+ height: 20px;
+ border-radius: 999px;
+ background: var(--line-strong);
+ padding: 2px;
+ transition: background 0.15s ease;
+}
+
+.narrate-switch.on {
+ background: var(--brand);
+}
+
+.narrate-knob {
+ display: block;
+ width: 16px;
+ height: 16px;
+ border-radius: 50%;
+ background: #fff;
+ box-shadow: 0 1px 2px rgba(0, 0, 0, 0.25);
+ transition: transform 0.15s ease;
+}
+
+.narrate-switch.on .narrate-knob {
+ transform: translateX(14px);
}
.privacy-note {
diff --git a/src/Recorder.tsx b/src/Recorder.tsx
index eb18bcf..3f5183c 100644
--- a/src/Recorder.tsx
+++ b/src/Recorder.tsx
@@ -12,6 +12,7 @@ export function Recorder() {
const [status, setStatus] = useState<RecorderStatus | null>(null);
const [doctor, setDoctor] = useState<DoctorReport | null>(null);
const [showPrivacy, setShowPrivacy] = useState(false);
+ const [narrate, setNarrate] = useState(false);
const [elapsed, setElapsed] = useState(0);
const [sessionCount, setSessionCount] = useState(0);
const [pendingCount, setPendingCount] = useState(0);
@@ -56,15 +57,12 @@ export function Recorder() {
}, [recording, startedAt]);
const toggle = useCallback(async () => {
- const res = recording ? await window.skillRecorder.stop() : await window.skillRecorder.start();
+ const res = recording
+ ? await window.skillRecorder.stop()
+ : await window.skillRecorder.start({ narration: narrate });
if (!res.ok) window.alert(res.error ?? "Action failed");
setStatus(await window.skillRecorder.status());
- }, [recording]);
-
- const addMarker = useCallback(async () => {
- const note = window.prompt("Marker: what are you doing right now?");
- if (note) await window.skillRecorder.marker(note);
- }, []);
+ }, [recording, narrate]);
const openLibrary = useCallback(() => {
void window.skillRecorder.openLibrary();
@@ -103,8 +101,40 @@ export function Recorder() {
</div>
</div>
- <button className="marker" onClick={addMarker} disabled={!recording}>
- Add marker
+ <button
+ className={`narrate ${narrate ? "on" : ""}`}
+ role="switch"
+ aria-checked={narrate}
+ onClick={() => setNarrate((v) => !v)}
+ disabled={recording}
+ >
+ <span className="narrate-icon" aria-hidden>
+ <svg width="18" height="18" viewBox="0 0 20 20" fill="none">
+ <rect x="7.5" y="2.5" width="5" height="9" rx="2.5" stroke="currentColor" strokeWidth="1.4" />
+ <path
+ d="M5 9.2a5 5 0 0 0 10 0"
+ stroke="currentColor"
+ strokeWidth="1.4"
+ strokeLinecap="round"
+ />
+ <path d="M10 14.2v3" stroke="currentColor" strokeWidth="1.4" strokeLinecap="round" />
+ </svg>
+ </span>
+ <span className="narrate-text">
+ <span className="narrate-label">Narrate</span>
+ <span className="narrate-sub">
+ {recording
+ ? narrate
+ ? "Listening to your voice"
+ : "Voice off for this recording"
+ : narrate
+ ? "Your voice will be included"
+ : "Explain out loud (optional)"}
+ </span>
+ </span>
+ <span className={`narrate-switch ${narrate ? "on" : ""}`} aria-hidden>
+ <span className="narrate-knob" />
+ </span>
</button>
<button className="privacy-note" onClick={() => setShowPrivacy(true)}>
diff --git a/src/WhatsRecorded.tsx b/src/WhatsRecorded.tsx
index b02893f..4507902 100644
--- a/src/WhatsRecorded.tsx
+++ b/src/WhatsRecorded.tsx
@@ -61,7 +61,7 @@ export function WhatsRecorded({ onClose }: { onClose: () => void }) {
<li>Which apps you switch to, and their window and document titles.</li>
<li>Web addresses of the pages you open.</li>
<li>A short preview of text you copy, up to 120 characters.</li>
- <li>A silent video of your screen at a low frame rate. No microphone or audio.</li>
+ <li>A silent video of your screen at a low frame rate.</li>
</ul>
<p className="sheet-caution">
If you copy something sensitive like a password while recording, its preview is captured.
@@ -69,6 +69,16 @@ export function WhatsRecorded({ onClose }: { onClose: () => void }) {
</p>
</section>
+ <section className="sheet-block">
+ <h3>Voice narration (only if you turn it on)</h3>
+ <p className="sheet-note">Off by default. The Narrate switch is your choice, per recording.</p>
+ <ul>
+ <li>Your microphone is used only while Narrate is on and a recording is running.</li>
+ <li>The recording is turned into text on this computer, using an offline model.</li>
+ <li>Turn Narrate off and your microphone is never touched.</li>
+ </ul>
+ </section>
+
<section className="sheet-block">
<h3>Where it's stored</h3>
<ul>
@@ -82,8 +92,9 @@ export function WhatsRecorded({ onClose }: { onClose: () => void }) {
<ul>
<li>Nothing leaves your computer while you record.</li>
<li>
- When you choose Analyze, the timeline, the recorded events, and screenshots from your
- video are sent to a Copilot AI model so it can describe what you did.
+ When you choose Analyze, the timeline, the recorded events, screenshots from your
+ video, and your narration text (if you recorded any) are sent to a Copilot AI model so
+ it can describe what you did.
</li>
</ul>
</section>
diff --git a/vite.config.ts b/vite.config.ts
index cd0d777..2b46ac3 100644
--- a/vite.config.ts
+++ b/vite.config.ts
@@ -14,6 +14,7 @@ function copyStaticAssets(): void {
const out = path.join(rootDir, "dist-electron");
mkdirSync(out, { recursive: true });
mkdirSync(path.join(out, "video"), { recursive: true });
+ mkdirSync(path.join(out, "audio"), { recursive: true });
copyFileSync(path.join(rootDir, "electron", "preload.cjs"), path.join(out, "preload.cjs"));
copyFileSync(
path.join(rootDir, "electron", "video", "capture.html"),
@@ -23,6 +24,14 @@ function copyStaticAssets(): void {
path.join(rootDir, "electron", "video", "capture-preload.cjs"),
path.join(out, "video", "capture-preload.cjs"),
);
+ copyFileSync(
+ path.join(rootDir, "electron", "audio", "capture.html"),
+ path.join(out, "audio", "capture.html"),
+ );
+ copyFileSync(
+ path.join(rootDir, "electron", "audio", "capture-preload.cjs"),
+ path.join(out, "audio", "capture-preload.cjs"),
+ );
}
export default defineConfig({
@@ -45,7 +54,15 @@ export default defineConfig({
rolldownOptions: {
// Native / binary deps must be required from node_modules at runtime,
// never inlined into the main bundle.
- external: ["electron", "@github/copilot-sdk", "get-windows", "sharp", "ffmpeg-static"],
+ external: [
+ "electron",
+ "@github/copilot-sdk",
+ "get-windows",
+ "sharp",
+ "ffmpeg-static",
+ "@huggingface/transformers",
+ "onnxruntime-node",
+ ],
},
},
plugins: [