Skip to content

Desktop app that records your on-screen work session and uses the GitHub Copilot SDK to reconstruct it as an intent + ordered steps, then builds a reusable Skill or Automation for Microsoft Autopilots like Scout, Microsoft Copilot Cowork, or Copilot Studio.

Add Windows event-capture parity: UIA URL provider, doctor observability, shell-hook installer, packaging

adileibef6be495b53
diff --git a/common/ipc.ts b/common/ipc.ts
index b20cdd2..aceaac1 100644
--- a/common/ipc.ts
+++ b/common/ipc.ts
@@ -136,13 +136,58 @@ export interface CopilotInfo {
path: string | null;
}
+/** Whether the native window-tracking addon (get-windows) loaded for this platform. */
+export interface ActiveWindowInfo {
+ ok: boolean;
+ /** The prebuilt binding path we resolved, for troubleshooting. */
+ bindingPath: string | null;
+ error?: string;
+}
+
+/** How, and whether, active-tab URLs can be read on this platform. */
+export interface BrowserUrlInfo {
+ kind: "applescript" | "uia" | "none";
+ supported: boolean;
+}
+
+/** Shell-hook install status for terminal capture. */
+export interface ShellHookInfo {
+ shell: "zsh" | "bash" | "pwsh" | null;
+ profilePath: string | null;
+ installed: boolean;
+}
+
+/** One capture source in the doctor report, annotated with platform support. */
+export interface DoctorSource {
+ key: string;
+ label: string;
+ tier: number;
+ cost: string;
+ /** False when this source can't work on the current platform. */
+ supported: boolean;
+ /** Short reason shown when unsupported, or a setup nudge. */
+ note?: string;
+}
+
export interface DoctorReport {
platform: NodeJS.Platform;
ffmpeg: FfmpegInfo;
copilotCli: CopilotInfo;
+ activeWindow: ActiveWindowInfo;
+ browserUrl: BrowserUrlInfo;
+ shellHook: ShellHookInfo;
sessionsDir: string;
captureLevel: CaptureLevel;
- activeSources: { key: string; label: string; tier: number; cost: string }[];
+ activeSources: DoctorSource[];
+}
+
+/** Result of installing the terminal shell hook. */
+export interface ShellHookInstallResult {
+ ok: boolean;
+ shell: "zsh" | "bash" | "pwsh" | null;
+ profilePath: string | null;
+ alreadyInstalled: boolean;
+ error?: string;
}
/** Current capture configuration plus its resolved named level. */
@@ -158,6 +203,7 @@ export const IPC = {
status: "recorder:status",
marker: "recorder:marker",
doctor: "doctor:check",
+ installShellHook: "doctor:install-shell-hook",
statusChanged: "recorder:status-changed",
getCapture: "capture:get",
setLevel: "capture:set-level",
@@ -187,6 +233,8 @@ export interface SkillRecorderApi {
status(): Promise<RecorderStatus>;
marker(note: string): Promise<MarkerResult>;
doctor(): Promise<DoctorReport>;
+ /** Install the terminal shell hook into the user's shell profile (idempotent). */
+ installShellHook(): Promise<ShellHookInstallResult>;
getCapture(): Promise<CaptureState>;
setLevel(level: Exclude<CaptureLevel, "custom">): Promise<CaptureState>;
setConfig(config: CaptureConfig): Promise<CaptureState>;
diff --git a/docs/windows-capture.md b/docs/windows-capture.md
new file mode 100644
index 0000000..33065fd
--- /dev/null
+++ b/docs/windows-capture.md
@@ -0,0 +1,107 @@
+# Windows event capture
+
+Skill Recorder is cross platform (macOS + Windows). The core (recorder controller,
+event bus, session store, collector host, capture tiers, describer, skill builder)
+is platform agnostic. This doc covers the parts that are OS specific: how each
+event source behaves on Windows, how to set it up, and a live smoke test to run
+before trusting a Windows build.
+
+## What captures what on Windows
+
+| Source | Mechanism on Windows | Parity vs macOS | Permission |
+|--------|----------------------|-----------------|------------|
+| App switches | `get-windows` native addon | Full | None |
+| Window titles | `get-windows` native addon | Full (better: no grant needed) | None |
+| Browser URLs | UI Automation address bar read (`powershell.exe` host) | Functional, not byte exact | None |
+| Clipboard | Electron clipboard | Full | None |
+| Terminal commands | PowerShell `$PROFILE` hook writing `~/.skill-recorder/shell.jsonl` | Parity for pwsh (no cmd.exe) | Install hook once |
+| Screen video + frames | `desktopCapturer` + ffmpeg + sharp | Full | Screen capture |
+
+Notes:
+
+- **Browser URLs** read the *omnibox display value* through UI Automation
+ (`electron/collectors/windows-url-provider.ts`), not the exact active tab URL
+ the macOS AppleScript provider gets. That is enough for host level step
+ segmentation, and unlike macOS it also reads **Firefox**. Values that look like
+ a search term (whitespace, or no dot) are dropped rather than emitted as noise.
+- **Window titles** need no OS permission on Windows (macOS needs Accessibility).
+- **Terminal** capture is pwsh only. The hook overrides the `prompt` function, so
+ a heavily customized prompt (oh-my-posh etc.) may need manual merging.
+
+## Prerequisites
+
+1. **Native window addon.** `get-windows` ships a prebuilt N-API binary per
+ platform. On a Windows build/dev machine, `npm install` fetches
+ `napi-*-win32-*-x64/node-get-windows.node`. If it is missing, `get-windows`
+ silently returns no-op stubs and app-switch tracking dies with no error, so
+ the doctor check below exists specifically to catch that.
+2. **Windows PowerShell.** `powershell.exe` (Windows PowerShell 5.1, in-box on
+ every Windows 10/11) is used to host the UI Automation URL reader. No install
+ needed.
+3. **Copilot CLI** on `PATH` for the describer (`copilot`).
+4. **ffmpeg** is bundled via `ffmpeg-static`; no system install needed.
+
+## Doctor signals
+
+Open the recorder HUD and read the doctor rows (or call `doctor()` over IPC).
+On Windows, confirm:
+
+- **window tracking** = `native` (not `addon missing`). `addon missing` means the
+ `get-windows` `.node` did not resolve, which is a packaging or install problem.
+- **browser URLs** = `uia` when the capture level includes URLs.
+- **terminal hook** shows `install` until the PowerShell hook is added; click it
+ (or run the installer) and it flips to `pwsh`.
+
+## Live smoke test
+
+Run a real recording on Windows and verify each source lands in the session's
+`events.jsonl`. Set the capture level to **Full** first so every source is on.
+
+1. **Start** capture from the HUD (or `Ctrl+Shift+R`).
+2. **App switches / titles.** Alt-Tab between two apps (e.g. Edge and Notepad).
+ Expect `app.activate` events with the right `owner.name`, and
+ `app.title-change` as titles change.
+3. **Browser URLs.** In Edge or Chrome, navigate to two different sites. Expect
+ `browser.url` events with the address bar URL and `host`. Try Firefox too.
+ Typing a partial URL or a search term should not emit a bogus event.
+4. **Clipboard.** Copy some text. Expect a `clipboard.change` event with a
+ preview and hash.
+5. **Terminal.** Ensure the hook is installed (doctor row shows `pwsh`), open a
+ new PowerShell window, run a couple of commands. Expect `terminal.command`
+ events tailing `~/.skill-recorder/shell.jsonl`.
+6. **Video.** Confirm a `.webm` is written and `frame.captured` events appear.
+7. **Stop.** The recording should show up in the library as `recorded`, and
+ analysis should produce a coherent intent + ordered steps.
+
+If a source produces nothing, check the doctor row for it first, then the main
+process log for the one-time warnings (e.g. "Browser URL capture is on but
+unavailable on this platform", or a reduced-capture notice).
+
+## Packaging
+
+`package.json` `build` configures electron-builder for both `mac` and `win`
+(nsis x64). Native modules (`get-windows`, `sharp`, `@img/*`, `ffmpeg-static`)
+are listed under `asarUnpack` so their binaries load from disk rather than from
+inside the asar archive.
+
+Build the Windows installer on Windows (or a Windows CI runner) so the native
+binaries for `win32-x64` are present:
+
+```powershell
+npm ci
+npm run dist
+```
+
+Cross building the Windows target from macOS will not fetch the Windows native
+binaries and is not supported here.
+
+## Known limitations
+
+- Browser URLs are best effort display strings, not the exact tab URL.
+- Terminal capture is pwsh only (no cmd.exe).
+- Semantic UI events (focus/invoke/value via UI Automation) are not implemented
+ on either platform yet.
+- The Windows paths above are validated by typecheck, a PowerShell parse check of
+ the UIA script, and a `win32` describer eval (`evals/scenarios/windows-deploy.ts`).
+ Live capture must still be verified on a real Windows machine using the checklist
+ above.
diff --git a/electron/collectors/active-window.ts b/electron/collectors/active-window.ts
index 2c4ff3d..4c2a34b 100644
--- a/electron/collectors/active-window.ts
+++ b/electron/collectors/active-window.ts
@@ -83,6 +83,11 @@ export class ActiveWindowCollector implements Collector {
start(ctx: CollectorContext): void {
this.ctx = ctx;
+ if (this.captureUrls && !this.urlProvider) {
+ // Honest degradation: the level asks for URLs but this platform has no
+ // provider (e.g. Linux). Say so once instead of silently emitting nothing.
+ ctx.log.warn("Browser URL capture is on but unavailable on this platform; skipping URLs.");
+ }
void this.poll();
}
@@ -91,6 +96,7 @@ export class ActiveWindowCollector implements Collector {
clearTimeout(this.timer);
this.timer = null;
}
+ this.urlProvider?.dispose?.();
this.ctx = null;
}
diff --git a/electron/collectors/shell-hook.ts b/electron/collectors/shell-hook.ts
new file mode 100644
index 0000000..864140c
--- /dev/null
+++ b/electron/collectors/shell-hook.ts
@@ -0,0 +1,143 @@
+import { execFileSync } from "node:child_process";
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+
+import { SHELL_HOOKS } from "./terminal-hooks";
+
+/**
+ * Terminal capture needs a one-time shell hook installed into the user's shell
+ * profile (see terminal-hooks.ts for the snippets). This module detects the
+ * active shell, resolves its profile file, reports whether our hook is present,
+ * and installs it idempotently. Cross-platform: zsh/bash on macOS + Linux,
+ * PowerShell on Windows.
+ *
+ * The snippet is wrapped in a marked block so install is idempotent and the
+ * status check is a simple substring test.
+ */
+
+export type SupportedShell = "zsh" | "bash" | "pwsh";
+
+const MARKER = ">>> skill-recorder";
+
+export interface ShellHookStatus {
+ /** Detected shell, or null when it isn't one we can hook. */
+ shell: SupportedShell | null;
+ /** Absolute path to the profile we'd install into, when known. */
+ profilePath: string | null;
+ /** True when our marked hook block is already present in the profile. */
+ installed: boolean;
+}
+
+export interface ShellHookInstallResult {
+ ok: boolean;
+ shell: SupportedShell | null;
+ profilePath: string | null;
+ /** True when the block was already there and we made no change. */
+ alreadyInstalled: boolean;
+ error?: string;
+}
+
+/** Detect the shell we'd install a hook for on this platform. */
+export function detectShell(): SupportedShell | null {
+ if (process.platform === "win32") return "pwsh";
+ const shellPath = process.env.SHELL ?? "";
+ const base = path.basename(shellPath).toLowerCase();
+ if (base.includes("zsh")) return "zsh";
+ if (base.includes("bash")) return "bash";
+ // A sensible default per platform when $SHELL is unset or exotic.
+ return process.platform === "darwin" ? "zsh" : "bash";
+}
+
+/**
+ * Resolve the profile file for a shell. For pwsh we ask PowerShell for its real
+ * `$PROFILE.CurrentUserAllHosts` (locale- / OneDrive-aware), falling back to the
+ * conventional path when no interpreter is reachable.
+ */
+export function resolveProfilePath(shell: SupportedShell): string | null {
+ const home = os.homedir();
+ if (shell === "zsh") return path.join(home, ".zshrc");
+ if (shell === "bash") return path.join(home, ".bashrc");
+ return resolvePwshProfile(home);
+}
+
+function resolvePwshProfile(home: string): string {
+ for (const exe of ["pwsh", "powershell.exe"]) {
+ try {
+ const out = execFileSync(
+ exe,
+ ["-NoProfile", "-NonInteractive", "-Command", "$PROFILE.CurrentUserAllHosts"],
+ { encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] },
+ )
+ .trim()
+ .split(/\r?\n/)[0];
+ if (out) return out;
+ } catch {
+ // try the next interpreter
+ }
+ }
+ // Conventional PowerShell 7 location when no interpreter answered.
+ return path.join(home, "Documents", "PowerShell", "Microsoft.PowerShell_profile.ps1");
+}
+
+function readProfile(profilePath: string): string | null {
+ try {
+ return existsSync(profilePath) ? readFileSync(profilePath, "utf8") : "";
+ } catch {
+ return null;
+ }
+}
+
+/** Whether our marked hook block is present in the resolved profile. */
+export function shellHookStatus(): ShellHookStatus {
+ const shell = detectShell();
+ if (!shell) return { shell: null, profilePath: null, installed: false };
+ const profilePath = resolveProfilePath(shell);
+ const contents = profilePath ? readProfile(profilePath) : null;
+ return {
+ shell,
+ profilePath,
+ installed: contents != null && contents.includes(MARKER),
+ };
+}
+
+/** Idempotently append our shell hook to the detected shell's profile. */
+export function installShellHook(): ShellHookInstallResult {
+ const shell = detectShell();
+ if (!shell) {
+ return {
+ ok: false,
+ shell: null,
+ profilePath: null,
+ alreadyInstalled: false,
+ error: "No supported shell detected (zsh, bash, or PowerShell).",
+ };
+ }
+ const profilePath = resolveProfilePath(shell);
+ if (!profilePath) {
+ return { ok: false, shell, profilePath: null, alreadyInstalled: false, error: "Could not resolve a shell profile path." };
+ }
+
+ const existing = readProfile(profilePath);
+ if (existing == null) {
+ return { ok: false, shell, profilePath, alreadyInstalled: false, error: `Could not read ${profilePath}.` };
+ }
+ if (existing.includes(MARKER)) {
+ return { ok: true, shell, profilePath, alreadyInstalled: true };
+ }
+
+ try {
+ mkdirSync(path.dirname(profilePath), { recursive: true });
+ const sep = existing.length && !existing.endsWith("\n") ? "\n\n" : existing.length ? "\n" : "";
+ writeFileSync(profilePath, `${existing}${sep}${SHELL_HOOKS[shell]}\n`, "utf8");
+ return { ok: true, shell, profilePath, alreadyInstalled: false };
+ } catch (err) {
+ return {
+ ok: false,
+ shell,
+ profilePath,
+ alreadyInstalled: false,
+ error: err instanceof Error ? err.message : String(err),
+ };
+ }
+}
diff --git a/electron/collectors/url-provider.ts b/electron/collectors/url-provider.ts
index 0e3f9b6..e33a2f8 100644
--- a/electron/collectors/url-provider.ts
+++ b/electron/collectors/url-provider.ts
@@ -1,5 +1,7 @@
import { execFile } from "node:child_process";
+import { WindowsUrlProvider } from "./windows-url-provider";
+
/**
* Reads the *active tab URL* of the frontmost browser — the single richest
* non-video signal we can't get from window titles alone. macOS exposes it only
@@ -21,6 +23,8 @@ export interface UrlProvider {
supports(app: string): boolean;
/** Best-effort active-tab URL; resolves null on any failure (never throws). */
get(app: string): Promise<ActiveUrl | null>;
+ /** Release any long-lived resources (e.g. a helper process). Optional. */
+ dispose?(): void;
}
// Browsers whose scripting dictionary exposes `active tab of front window`.
@@ -119,6 +123,17 @@ export class MacUrlProvider implements UrlProvider {
/** The URL provider for the current platform, or null where unsupported. */
export function createUrlProvider(): UrlProvider | null {
if (process.platform === "darwin") return new MacUrlProvider();
- // Windows: UI Automation address-bar read is a later milestone.
+ if (process.platform === "win32") return new WindowsUrlProvider();
return null;
}
+
+/** How the active-tab URL is read on a given platform, for honest reporting. */
+export type BrowserUrlKind = "applescript" | "uia" | "none";
+
+export function browserUrlProviderKind(
+ platform: NodeJS.Platform = process.platform,
+): BrowserUrlKind {
+ if (platform === "darwin") return "applescript";
+ if (platform === "win32") return "uia";
+ return "none";
+}
diff --git a/electron/collectors/windows-url-provider.ts b/electron/collectors/windows-url-provider.ts
new file mode 100644
index 0000000..69dbe82
--- /dev/null
+++ b/electron/collectors/windows-url-provider.ts
@@ -0,0 +1,286 @@
+import { spawn, type ChildProcess } from "node:child_process";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { createInterface, type Interface } from "node:readline";
+import os from "node:os";
+import path from "node:path";
+
+import type { ActiveUrl, UrlProvider } from "./url-provider";
+
+/**
+ * Windows active-tab URL provider via **UI Automation** — the Windows parallel to
+ * the macOS AppleScript provider. Windows exposes no scripting bridge for the
+ * address bar, so we read the frontmost window's UIA tree and pull the omnibox
+ * (address bar) Edit control's value.
+ *
+ * UIA lives in the .NET `UIAutomationClient` / `UIAutomationTypes` assemblies,
+ * which are only reliably present in **Windows PowerShell 5.1** (`powershell.exe`,
+ * shipped in-box on every Windows 10/11), not necessarily in PowerShell 7. So we
+ * host the reader in `powershell.exe`. Loading those assemblies costs ~0.5–1s, so
+ * spawning per URL read would blow any reasonable timeout — instead we keep a
+ * **persistent host process** that loads UIA once and then answers one request
+ * (frontmost window → url) per stdin line with one stdout line. The read walks a
+ * bounded slice of the control tree and prunes `Document` subtrees so it never
+ * descends into the (huge, lazily-realized) web-page accessibility tree.
+ *
+ * Strictly best-effort and isolated behind {@link UrlProvider}: any failure
+ * (no host, timeout, no address bar, permission quirk) resolves `null`, never
+ * throws, and never blocks the poll loop. Only ever invoked when a browser is
+ * frontmost.
+ */
+
+const SEP = String.fromCharCode(30); // ASCII record separator; can't occur in a URL/title
+const READY = "READY";
+const REQUEST_TIMEOUT_MS = 2000;
+
+// get-windows reports the browser's product name as the window owner on Windows
+// (e.g. "Google Chrome", "Microsoft Edge", "Firefox"). Match loosely on the
+// recognizable family token — unlike macOS we can read Firefox's address bar too.
+const WINDOWS_BROWSER_TOKENS = [
+ "chrome",
+ "chromium",
+ "edge",
+ "brave",
+ "opera",
+ "vivaldi",
+ "firefox",
+ "arc",
+];
+
+/**
+ * The persistent PowerShell reader. Emits `READY` once UIA is loaded, then for
+ * each line on stdin writes one line: `<url><SEP><title>` (or empty on failure).
+ * Uses plain string tests (no regex) to sidestep escaping surprises, prunes web
+ * content, and is bounded in nodes + depth so it can never run away on a heavy
+ * page tree.
+ */
+const UIA_SCRIPT = `
+$ErrorActionPreference = 'SilentlyContinue'
+Add-Type -AssemblyName UIAutomationClient
+Add-Type -AssemblyName UIAutomationTypes
+Add-Type @"
+using System;
+using System.Runtime.InteropServices;
+public static class SkrNative {
+ [DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
+}
+"@
+$SEP = [char]30
+$AE = [System.Windows.Automation.AutomationElement]
+$editType = [System.Windows.Automation.ControlType]::Edit
+$docType = [System.Windows.Automation.ControlType]::Document
+$valuePattern = [System.Windows.Automation.ValuePattern]::Pattern
+$walker = [System.Windows.Automation.TreeWalker]::ControlViewWalker
+
+function Read-Value($el) {
+ try {
+ $vp = $el.GetCurrentPattern($valuePattern)
+ if ($vp -ne $null) { return $vp.Current.Value }
+ } catch {}
+ return ""
+}
+
+function Looks-Url($v) {
+ if ([string]::IsNullOrWhiteSpace($v)) { return $false }
+ if ($v.Contains('://')) { return $true }
+ if ((-not $v.Contains(' ')) -and $v.Contains('.')) { return $true }
+ return $false
+}
+
+function Read-Url {
+ $h = [SkrNative]::GetForegroundWindow()
+ if ($h -eq [IntPtr]::Zero) { return "" }
+ $root = $AE::FromHandle($h)
+ if ($root -eq $null) { return "" }
+ $title = ""
+ try { $title = $root.Current.Name } catch {}
+
+ $queue = New-Object System.Collections.Generic.Queue[object]
+ $queue.Enqueue(@($root, 0))
+ $seen = 0
+ $best = ""
+ while ($queue.Count -gt 0 -and $seen -lt 600) {
+ $item = $queue.Dequeue()
+ $el = $item[0]; $depth = $item[1]
+ $seen++
+ $ct = $null
+ try { $ct = $el.Current.ControlType } catch {}
+ if ($ct -eq $docType) { continue } # prune the web-page a11y tree
+ if ($ct -eq $editType) {
+ $v = Read-Value $el
+ if (Looks-Url $v) {
+ $n = ""; $id = ""
+ try { $n = $el.Current.Name } catch {}
+ try { $id = $el.Current.AutomationId } catch {}
+ if ($id -eq 'urlbar-input' -or $n -like '*address*' -or $n -like '*enter address*' -or $n -like '*search or enter*') {
+ return "$v$SEP$title"
+ }
+ if ($best -eq "") { $best = $v }
+ }
+ }
+ if ($depth -lt 8) {
+ $child = $walker.GetFirstChild($el)
+ while ($child -ne $null) {
+ $queue.Enqueue(@($child, $depth + 1))
+ $child = $walker.GetNextSibling($child)
+ }
+ }
+ }
+ if ($best -ne "") { return "$best$SEP$title" }
+ return ""
+}
+
+[Console]::Out.WriteLine('${READY}')
+[Console]::Out.Flush()
+while ($true) {
+ $line = [Console]::In.ReadLine()
+ if ($line -eq $null) { break }
+ $out = ""
+ try { $out = Read-Url } catch { $out = "" }
+ [Console]::Out.WriteLine($out)
+ [Console]::Out.Flush()
+}
+`;
+
+function isWindowsBrowser(app: string): boolean {
+ const a = app.toLowerCase();
+ return WINDOWS_BROWSER_TOKENS.some((t) => a.includes(t));
+}
+
+/** Normalize an omnibox value into a parseable URL, or null if it's not one. */
+function normalizeUrl(raw: string): string | null {
+ const v = raw.trim();
+ if (!v) return null;
+ if (/^[a-z][a-z0-9+.-]*:\/\//i.test(v)) return v;
+ // Address bars usually drop the scheme ("github.com/foo"). Restore it so the
+ // consumer can parse a host. Reject anything that still looks like a search
+ // (has whitespace, or no dot) rather than emit noise.
+ if (!/\s/.test(v) && v.includes(".")) return `https://${v}`;
+ return null;
+}
+
+export class WindowsUrlProvider implements UrlProvider {
+ private proc: ChildProcess | null = null;
+ private rl: Interface | null = null;
+ private tmpDir: string | null = null;
+ private ready = false;
+ private busy = false;
+ private disposed = false;
+ private waiter: ((line: string) => void) | null = null;
+
+ supports(app: string): boolean {
+ return isWindowsBrowser(app);
+ }
+
+ async get(app: string): Promise<ActiveUrl | null> {
+ if (this.disposed || !isWindowsBrowser(app)) return null;
+ this.ensureHost();
+ if (!this.proc || !this.ready || this.busy) return null;
+
+ const line = await this.request();
+ if (line == null) return null;
+ const [rawUrl, title] = line.split(SEP);
+ const url = normalizeUrl(rawUrl ?? "");
+ if (!url) return null;
+ return { url, title: title?.trim() || undefined };
+ }
+
+ dispose(): void {
+ this.disposed = true;
+ this.killHost();
+ }
+
+ private request(): Promise<string | null> {
+ return new Promise((resolve) => {
+ const proc = this.proc;
+ if (!proc?.stdin?.writable) return resolve(null);
+ this.busy = true;
+ let settled = false;
+ const done = (line: string | null) => {
+ if (settled) return;
+ settled = true;
+ clearTimeout(timer);
+ this.waiter = null;
+ this.busy = false;
+ resolve(line);
+ };
+ const timer = setTimeout(() => {
+ // Resolve first, then recycle the host: a late/desynced response would
+ // otherwise corrupt the next read.
+ done(null);
+ this.killHost();
+ }, REQUEST_TIMEOUT_MS);
+ this.waiter = (line) => done(line);
+ try {
+ proc.stdin.write("get\n");
+ } catch {
+ done(null);
+ }
+ });
+ }
+
+ private ensureHost(): void {
+ if (this.disposed || this.proc) return;
+ try {
+ this.tmpDir = mkdtempSync(path.join(os.tmpdir(), "skr-uia-"));
+ const scriptPath = path.join(this.tmpDir, "uia-url.ps1");
+ writeFileSync(scriptPath, UIA_SCRIPT, "utf8");
+
+ const proc = spawn(
+ "powershell.exe",
+ ["-NoProfile", "-NonInteractive", "-ExecutionPolicy", "Bypass", "-File", scriptPath],
+ { stdio: ["pipe", "pipe", "ignore"], windowsHide: true },
+ );
+ this.proc = proc;
+ this.ready = false;
+
+ if (proc.stdout) {
+ this.rl = createInterface({ input: proc.stdout });
+ this.rl.on("line", (line) => this.onLine(line));
+ }
+ proc.on("error", () => this.killHost());
+ proc.on("exit", () => this.killHost());
+ } catch {
+ this.killHost();
+ }
+ }
+
+ private onLine(line: string): void {
+ if (!this.ready) {
+ // First line is the readiness sentinel emitted once UIA is loaded.
+ this.ready = line.trim() === READY;
+ return;
+ }
+ const w = this.waiter;
+ this.waiter = null;
+ if (w) w(line);
+ }
+
+ private killHost(): void {
+ const w = this.waiter;
+ this.waiter = null;
+ if (w) w(""); // unblock any in-flight request (resolves to no-url)
+ this.ready = false;
+ this.busy = false;
+ if (this.rl) {
+ this.rl.close();
+ this.rl = null;
+ }
+ if (this.proc) {
+ this.proc.removeAllListeners();
+ try {
+ this.proc.kill();
+ } catch {
+ // already gone
+ }
+ this.proc = null;
+ }
+ if (this.tmpDir) {
+ try {
+ rmSync(this.tmpDir, { recursive: true, force: true });
+ } catch {
+ // best-effort temp cleanup
+ }
+ this.tmpDir = null;
+ }
+ }
+}
diff --git a/electron/doctor.ts b/electron/doctor.ts
index b6d78bd..224a832 100644
--- a/electron/doctor.ts
+++ b/electron/doctor.ts
@@ -1,9 +1,19 @@
import { execFileSync } from "node:child_process";
import { existsSync } from "node:fs";
import { createRequire } from "node:module";
+import path from "node:path";
import { CAPTURE_SOURCES } from "../common/config";
-import type { CopilotInfo, DoctorReport, FfmpegInfo } from "../common/ipc";
+import type {
+ ActiveWindowInfo,
+ BrowserUrlInfo,
+ CopilotInfo,
+ DoctorReport,
+ DoctorSource,
+ FfmpegInfo,
+} from "../common/ipc";
+import { shellHookStatus } from "./collectors/shell-hook";
+import { browserUrlProviderKind } from "./collectors/url-provider";
import { sessionsRoot } from "./recorder/session-store";
import type { SettingsStore } from "./settings";
@@ -40,19 +50,62 @@ function checkCopilot(): CopilotInfo {
return { ok: Boolean(p), path: p };
}
+/**
+ * Verify the get-windows native addon is actually present for this platform.
+ * get-windows silently falls back to no-op stubs when its prebuilt `.node` is
+ * missing (e.g. a packaging miss), which would make app-switch tracking die
+ * with no error — so we resolve the exact binding path and check it exists.
+ */
+function checkActiveWindow(): ActiveWindowInfo {
+ try {
+ const gwIndex = require.resolve("get-windows");
+ // Resolve node-pre-gyp from get-windows's own scope so we find the exact copy
+ // it uses, whether the dependency is hoisted or nested.
+ const gwRequire = createRequire(gwIndex);
+ const preGyp = gwRequire("@mapbox/node-pre-gyp") as { find(pkg: string): string };
+ const pkgJson = path.join(path.dirname(gwIndex), "package.json");
+ const bindingPath = preGyp.find(pkgJson);
+ return { ok: existsSync(bindingPath), bindingPath };
+ } catch (err) {
+ return { ok: false, bindingPath: null, error: err instanceof Error ? err.message : String(err) };
+ }
+}
+
+function checkBrowserUrl(): BrowserUrlInfo {
+ const kind = browserUrlProviderKind();
+ return { kind, supported: kind !== "none" };
+}
+
+/** Whether a capture source can work at all on the current platform. */
+function sourceSupport(key: string, browserUrl: BrowserUrlInfo): { supported: boolean; note?: string } {
+ if (key === "browserUrls" && !browserUrl.supported) {
+ return { supported: false, note: "Not available on this platform" };
+ }
+ return { supported: true };
+}
+
/** Environment readiness check surfaced in the UI and (later) a CLI `doctor` command. */
export function runDoctor(settings: SettingsStore): DoctorReport {
const config = settings.resolve();
- const activeSources = CAPTURE_SOURCES.filter((s) => config[s.key]).map((s) => ({
- key: s.key,
- label: s.label,
- tier: s.tier,
- cost: s.cost,
- }));
+ const browserUrl = checkBrowserUrl();
+ const shellHook = shellHookStatus();
+
+ const activeSources: DoctorSource[] = CAPTURE_SOURCES.filter((s) => config[s.key]).map((s) => {
+ const support = sourceSupport(s.key, browserUrl);
+ let note = support.note;
+ if (s.key === "terminal" && support.supported && !shellHook.installed) {
+ note = "Shell hook not installed";
+ }
+ return { key: s.key, label: s.label, tier: s.tier, cost: s.cost, supported: support.supported, note };
+ });
+
return {
platform: process.platform,
ffmpeg: checkFfmpeg(),
copilotCli: checkCopilot(),
+ activeWindow: checkActiveWindow(),
+ browserUrl,
+ shellHook,
sessionsDir: sessionsRoot(),
captureLevel: settings.level,
activeSources,
diff --git a/electron/ipc.ts b/electron/ipc.ts
index 4daef06..1fea0b6 100644
--- a/electron/ipc.ts
+++ b/electron/ipc.ts
@@ -15,6 +15,7 @@ import type {
import { IPC } from "../common/ipc";
import { Describer, loadPersistedAnalysis } from "./describer/describer";
import { runDoctor } from "./doctor";
+import { installShellHook } from "./collectors/shell-hook";
import { createLogger } from "./logger";
import type { RecorderController } from "./recorder/controller";
import { isValidSessionId } from "./recorder/session-store";
@@ -41,6 +42,7 @@ export function registerIpc(
ipcMain.handle(IPC.status, () => recorder.status());
ipcMain.handle(IPC.marker, (_event, note: string) => recorder.marker(note));
ipcMain.handle(IPC.doctor, () => runDoctor(settings));
+ ipcMain.handle(IPC.installShellHook, () => installShellHook());
ipcMain.handle(IPC.getCapture, () => captureState());
ipcMain.handle(IPC.setLevel, (_event, level: Exclude<CaptureLevel, "custom">) => {
settings.setLevel(level);
diff --git a/electron/preload.cjs b/electron/preload.cjs
index 86e429b..29eb810 100644
--- a/electron/preload.cjs
+++ b/electron/preload.cjs
@@ -8,6 +8,7 @@ const IPC = {
status: "recorder:status",
marker: "recorder:marker",
doctor: "doctor:check",
+ installShellHook: "doctor:install-shell-hook",
statusChanged: "recorder:status-changed",
getCapture: "capture:get",
setLevel: "capture:set-level",
@@ -36,6 +37,7 @@ contextBridge.exposeInMainWorld("skillRecorder", {
status: () => ipcRenderer.invoke(IPC.status),
marker: (note) => ipcRenderer.invoke(IPC.marker, note),
doctor: () => ipcRenderer.invoke(IPC.doctor),
+ installShellHook: () => ipcRenderer.invoke(IPC.installShellHook),
getCapture: () => ipcRenderer.invoke(IPC.getCapture),
setLevel: (level) => ipcRenderer.invoke(IPC.setLevel, level),
setConfig: (config) => ipcRenderer.invoke(IPC.setConfig, config),
diff --git a/evals/README.md b/evals/README.md
index 570a9be..b036861 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -84,6 +84,7 @@ Business, repeatable knowledge-work patterns (`evals/scenarios/`):
| `expense-report` | Reconcile card charges against receipts and file an expense report (Chrome + Preview + Expensify) |
| `release-notes` | Compile release notes from merged PRs, then version + deploy (Terminal + GitHub + editor) |
| `lead-to-crm` | Qualify inbound leads and enter them into the CRM (Mail + LinkedIn + Salesforce) |
+| `windows-deploy` | Deploy a web app to Azure and log the live URL, on Windows (Edge + Windows Terminal/pwsh + Excel) |
The last three are longer, multi-app **business processes** — they loop over several
records, mix a native app with the browser and/or terminal, and end in a submit /
diff --git a/evals/scenarios/index.ts b/evals/scenarios/index.ts
index 03c0f06..dfeec13 100644
--- a/evals/scenarios/index.ts
+++ b/evals/scenarios/index.ts
@@ -6,6 +6,7 @@ import { leadToCrm } from "./lead-to-crm";
import { releaseNotes } from "./release-notes";
import { researchCompile } from "./research-compile";
import { webToSpreadsheet } from "./web-to-spreadsheet";
+import { windowsDeploy } from "./windows-deploy";
/** All eval scenarios, in run order. */
export const scenarios: Scenario[] = [
@@ -17,4 +18,6 @@ export const scenarios: Scenario[] = [
expenseReport,
releaseNotes,
leadToCrm,
+ // Windows-shaped capture (win32 app names + browser.url + pwsh):
+ windowsDeploy,
];
diff --git a/evals/scenarios/windows-deploy.ts b/evals/scenarios/windows-deploy.ts
new file mode 100644
index 0000000..ab60931
--- /dev/null
+++ b/evals/scenarios/windows-deploy.ts
@@ -0,0 +1,67 @@
+// Scenario: deploy a web app to Azure from Windows, then log the live URL.
+// A Windows-shaped devops flow that exercises the describer on Windows app
+// names ("Microsoft Edge", "Windows Terminal", "Microsoft Excel"), PowerShell
+// terminal commands, and browser.url events shaped like the Windows UI
+// Automation provider emits them (address-bar URLs, Chromium + the omnibox).
+// Marked platform: "win32" so the session meta is stamped as Windows.
+
+import { appActivate, clipboard, marker, recorder, terminal, visit, type Scenario } from "../scenario";
+
+const EDGE = "Microsoft Edge";
+const TERMINAL = "Windows Terminal";
+const EXCEL = "Microsoft Excel";
+const REPO = "C:\\repos\\contoso-crm";
+const TERM_TITLE = "contoso-crm - pwsh";
+const PORTAL = "https://portal.azure.com/#home";
+const PORTAL_TITLE = "Home - Microsoft Azure";
+const LIVE_URL = "https://contoso-crm.azurewebsites.net/";
+const LIVE_TITLE = "Contoso CRM";
+const LOG_TITLE = "Deployments.xlsx - Excel";
+
+export const windowsDeploy: Scenario = {
+ id: "windows-deploy",
+ platform: "win32",
+ title: "Deploy a web app to Azure and log the live URL (Windows)",
+ truth:
+ "On Windows, the user opened the Azure Portal in Microsoft Edge, then used Windows Terminal " +
+ "(PowerShell) to sign in with the Azure CLI and deploy the Contoso CRM web app with `az webapp up`. " +
+ "They copied the resulting site URL (https://contoso-crm.azurewebsites.net), opened it in Edge to " +
+ "verify the deployment was live, and pasted the URL into a Deployments.xlsx tracking sheet in Excel.",
+ build: () => [
+ recorder(0),
+ ...visit(900, EDGE, PORTAL, PORTAL_TITLE), // check the resource group in the portal
+ appActivate(3000, TERMINAL, TERM_TITLE), // focus the terminal to run the CLI
+ terminal(3300, "az login", REPO, { shell: "pwsh" }),
+ terminal(6000, "az webapp up --name contoso-crm --resource-group web-prod --runtime NODE:20-lts", REPO, {
+ shell: "pwsh",
+ durationMs: 52000,
+ }),
+ clipboard(9000, "https://contoso-crm.azurewebsites.net"), // copy the deployed URL
+ ...visit(11000, EDGE, LIVE_URL, LIVE_TITLE), // open it to verify it is live
+ appActivate(14000, EXCEL, LOG_TITLE), // paste the URL into the deployment log (inferred)
+ marker(15500, "contoso-crm deployed and verified live"),
+ recorder(17000),
+ ],
+ rubric: {
+ intentKeywordsAny: [
+ ["deploy", "deployment", "publish", "ship"],
+ ["azure"],
+ ["web app", "webapp", "app", "site"],
+ ],
+ minSteps: 4,
+ maxSteps: 9,
+ expectedApps: ["edge", "terminal"],
+ orderedActions: [
+ ["azure portal", "portal", "azure"],
+ ["az webapp", "webapp up", "deploy", "az cli", "azure cli", "az "],
+ ["copy", "copied", "clipboard"],
+ ["verify", "verified", "live", "azurewebsites", "open"],
+ ["excel", "spreadsheet", "sheet", "log", "tracking"],
+ ],
+ mustMentionAny: [
+ ["azure"],
+ ["url", "link", "address", "azurewebsites", "contoso-crm"],
+ ],
+ forbidden: ["skill recorder"],
+ },
+};
diff --git a/package.json b/package.json
index d89a32f..c531985 100644
--- a/package.json
+++ b/package.json
@@ -2,6 +2,7 @@
"name": "skill-recorder",
"version": "0.1.0",
"description": "Records a user's on-screen work session and derives a structured description for building AI-agent skills.",
+ "author": "Skill Recorder",
"private": true,
"type": "module",
"main": "dist-electron/main.js",
@@ -34,5 +35,42 @@
"typescript": "^7.0.0",
"vite": "^8.0.16",
"vite-plugin-electron": "1.0.4"
+ },
+ "build": {
+ "appId": "com.skillrecorder.app",
+ "productName": "Skill Recorder",
+ "files": [
+ "dist/**",
+ "dist-electron/**",
+ "package.json"
+ ],
+ "asarUnpack": [
+ "node_modules/get-windows/**",
+ "node_modules/sharp/**",
+ "node_modules/@img/**",
+ "node_modules/ffmpeg-static/**"
+ ],
+ "mac": {
+ "target": [
+ "dmg",
+ "zip"
+ ],
+ "category": "public.app-category.developer-tools"
+ },
+ "win": {
+ "target": [
+ {
+ "target": "nsis",
+ "arch": [
+ "x64"
+ ]
+ }
+ ]
+ },
+ "nsis": {
+ "oneClick": false,
+ "perMachine": false,
+ "allowToChangeInstallationDirectory": true
+ }
}
}
diff --git a/src/App.css b/src/App.css
index f04069d..0f2762a 100644
--- a/src/App.css
+++ b/src/App.css
@@ -47,6 +47,7 @@
--signal-dim: rgba(var(--brand-rgb), 0.13);
--go: hsl(158, 100%, 26%); /* Scout success-loud (dark green) */
--go-rgb: 0, 133, 84;
+ --warn: hsl(35, 92%, 42%); /* amber for "needs setup" (not an error) */
--sans: "Segoe Sans", Aptos, Calibri, -apple-system, BlinkMacSystemFont,
"Segoe UI", Roboto, sans-serif;
@@ -632,6 +633,30 @@ button:focus-visible,
background: var(--rec);
}
+.badge.warn {
+ background: var(--warn);
+}
+
+.row-action {
+ font-family: var(--mono);
+ font-size: 11px;
+ color: var(--brand-ink);
+ background: var(--signal-dim);
+ border: 1px solid rgba(var(--brand-rgb), 0.3);
+ border-radius: 5px;
+ padding: 2px 8px;
+ cursor: pointer;
+}
+
+.row-action:hover:not(:disabled) {
+ background: rgba(var(--brand-rgb), 0.2);
+}
+
+.row-action:disabled {
+ opacity: 0.55;
+ cursor: default;
+}
+
.hint {
margin: 0;
font-size: 11px;
diff --git a/src/Recorder.tsx b/src/Recorder.tsx
index f403d8d..400ad5c 100644
--- a/src/Recorder.tsx
+++ b/src/Recorder.tsx
@@ -15,6 +15,7 @@ export function Recorder() {
const [elapsed, setElapsed] = useState(0);
const [sessionCount, setSessionCount] = useState(0);
const [pendingCount, setPendingCount] = useState(0);
+ const [installingHook, setInstallingHook] = useState(false);
const refreshCount = useCallback(async () => {
const list = await window.skillRecorder.listSessions();
@@ -77,6 +78,25 @@ export function Recorder() {
void window.skillRecorder.openLibrary();
}, []);
+ const installHook = useCallback(async () => {
+ setInstallingHook(true);
+ try {
+ const res = await window.skillRecorder.installShellHook();
+ if (!res.ok) {
+ window.alert(res.error ?? "Could not install the shell hook.");
+ } else if (res.alreadyInstalled) {
+ window.alert("The shell hook is already installed.");
+ } else {
+ window.alert(
+ `Installed the ${res.shell} hook. Open a new terminal to start capturing commands.`,
+ );
+ }
+ setDoctor(await window.skillRecorder.doctor());
+ } finally {
+ setInstallingHook(false);
+ }
+ }, []);
+
return (
<div className="hud">
<header className="hud-head">
@@ -181,12 +201,39 @@ export function Recorder() {
{doctor && (
<div className="doctor">
+ <Row
+ label="window tracking"
+ ok={doctor.activeWindow.ok}
+ note={doctor.activeWindow.ok ? "native" : "addon missing"}
+ />
+ {doctor.activeSources.some((s) => s.key === "browserUrls") && (
+ <Row
+ label="browser URLs"
+ ok={doctor.browserUrl.supported}
+ note={doctor.browserUrl.supported ? doctor.browserUrl.kind : "not on this OS"}
+ />
+ )}
<Row
label="ffmpeg"
ok={doctor.ffmpeg.ok}
note={doctor.ffmpeg.ok ? doctor.ffmpeg.source : "missing"}
/>
<Row label="copilot CLI" ok={doctor.copilotCli.ok} note={doctor.copilotCli.ok ? "found" : "missing"} />
+ {doctor.activeSources.some((s) => s.key === "terminal") && (
+ <div className="row">
+ <span className={`badge ${doctor.shellHook.installed ? "good" : "warn"}`}>
+ {doctor.shellHook.installed ? "✓" : "!"}
+ </span>
+ <span className="row-label">terminal hook</span>
+ {doctor.shellHook.installed ? (
+ <span className="row-note">{doctor.shellHook.shell}</span>
+ ) : (
+ <button className="row-action" onClick={installHook} disabled={installingHook}>
+ {installingHook ? "installing" : "install"}
+ </button>
+ )}
+ </div>
+ )}
</div>
)}