YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
fix(mcp): prefer screen for accessibility and OCR text searches
Louis Beaumont9060f6cec7b3
diff --git a/packages/screenpipe-mcp/README.md b/packages/screenpipe-mcp/README.mdindex 3b7ae41cf..2a2af0b41 100644--- a/packages/screenpipe-mcp/README.md+++ b/packages/screenpipe-mcp/README.md@@ -374,16 +374,21 @@ npx @modelcontextprotocol/inspector npx screenpipe-mcp### search-contentSearch through recorded content with content type filtering:-- `all` — OCR + Audio + Accessibility (default)-- `ocr` — Screen text from screenshots+- `all` — Screen text + Audio + Accessibility + Input (default)+- `screen` — Screen text, primarily from accessibility with OCR fallback (preferred)+- `ocr` — Legacy alias for `screen`, including accessibility-derived text- `audio` — Audio transcriptions- `input` — User actions (clicks, keystrokes, clipboard, app switches)-- `accessibility` — Accessibility tree text+- `accessibility` — Accessibility-only text, excluding OCR fallback- `parsed` — Compact app-specific messages, emails, tasks, documents, and code review (experimental; may be empty when parsing is disabled or unsupported)- Time range, app/window, and speaker filtering- Parsed data can also be filtered by `frame_id` or resolved `actor_id`- Pagination support+Screen-text results show their capture source: `[Screen·a11y]` for accessibility,+`[Screen·ocr]` for OCR fallback, or `[Screen]` when unknown. The MCP server maps+`screen` to the API's legacy `ocr` value, so it also works with existing backends.+### export-videoExport screen recordings as video files:- Specify time range with start/end timesdiff --git a/packages/screenpipe-mcp/src/index.ts b/packages/screenpipe-mcp/src/index.tsindex 0f5fc61c5..fc9aa4849 100644--- a/packages/screenpipe-mcp/src/index.ts+++ b/packages/screenpipe-mcp/src/index.ts@@ -345,9 +345,9 @@ const TOOLS: Tool[] = [},content_type: {type: "string",- enum: ["all", "ocr", "audio", "input", "accessibility", "memory", "parsed"],+ enum: ["all", "screen", "ocr", "audio", "input", "accessibility", "memory", "parsed"],description:- "Filter by content type. Use 'parsed' for compact app-specific records such as messages, emails, tasks, documents, and code review; it is experimental and may be empty when parsing is disabled or unsupported. NOTE on screen text: 'ocr' is a legacy label — it returns ALL screen-text rows, which are accessibility-derived for most apps (the result tag [Screen·a11y] vs [Screen·ocr] tells you which). Use 'ocr' for screen text (covers both paths), 'audio' for transcriptions, 'input' for keyboard/mouse events, 'memory' for stored facts. Default: 'all'.",+ "Filter by content type. Use 'screen' for screen text: accessibility is the primary capture path, with OCR fallback. Results identify the actual source as [Screen·a11y] or [Screen·ocr]; [Screen] means the source is unknown. 'ocr' is a legacy alias for 'screen', not an OCR-only filter. Use 'accessibility' only when you explicitly need accessibility-only results; it excludes OCR fallback. Use 'parsed' for compact app-specific messages, emails, tasks, documents, and code review (experimental; may be empty when parsing is disabled or unsupported), 'audio' for transcriptions, 'input' for keyboard/mouse events, 'memory' for stored facts. Default: 'all'.",default: "all",},limit: { type: "integer", description: "Max results (default 10, max 20). Start with 5 for exploration.", default: 10 },@@ -370,7 +370,7 @@ const TOOLS: Tool[] = [max_length: { type: "integer", description: "Max content length in characters" },include_frames: {type: "boolean",- description: "Include base64 screenshots (OCR only). Warning: large response.",+ description: "Include base64 screenshots for screen-text results, including accessibility-derived captures. Warning: large response.",default: false,},speaker_ids: { type: "string", description: "Comma-separated speaker IDs to filter audio" },@@ -378,7 +378,7 @@ const TOOLS: Tool[] = [tags: {type: "string",description:- "Comma-separated tags; returns only items carrying ALL of them (e.g. 'person:ada,project:atlas'). Works for screen + audio (content_type 'ocr'/'audio'/'all', tags written by add-tags) AND memories (content_type 'memory', tags written by update-memory). Same tag string links across all three, so two items sharing a tag are connected. Use namespaced tags (person:, project:, topic:) to link people/projects/topics. content_type 'input' and 'accessibility' have no tags and return nothing when this is set; 'parsed' does not support tags.",+ "Comma-separated tags; returns only items carrying ALL of them (e.g. 'person:ada,project:atlas'). Works for screen + audio (content_type 'screen'/'ocr'/'audio'/'all', tags written by add-tags) AND memories (content_type 'memory', tags written by update-memory). Same tag string links across all three, so two items sharing a tag are connected. Use namespaced tags (person:, project:, topic:) to link people/projects/topics. content_type 'input' and 'accessibility' have no tags and return nothing when this is set; 'parsed' does not support tags.",},include_related: {type: "boolean",@@ -1170,7 +1170,7 @@ current machine. Synced results must be attributed with their device and timesta- **Use max_content_length=500** to keep responses compact- **Don't use q for audio** — transcriptions are noisy, q filters too aggressively. Search audio by time range and speaker instead- **app_name is case-sensitive** — use exact names: "Google Chrome" not "chrome"-- **Screen text is mostly accessibility-derived, not OCR.** Screenpipe walks the OS accessibility tree first; OCR is only a fallback (terminals, canvas-rendered apps, games). \`content_type=ocr\` returns both paths — the result label \`[Screen·a11y]\` vs \`[Screen·ocr]\` tells you which produced the row. Don't pre-filter to a11y/ocr unless you specifically need one or the other+- **Use content_type=screen for screen text.** Screenpipe walks the OS accessibility tree first, with OCR fallback. The result label \`[Screen·a11y]\` or \`[Screen·ocr]\` identifies the actual capture source; \`[Screen]\` means unknown. \`content_type=ocr\` is a legacy alias for the same search, not an OCR-only filter. Use \`accessibility\` only for an explicit accessibility-only request, since it excludes OCR fallback.## Common Patterns@@ -1613,6 +1613,9 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => observeCall(rcase "search-content": {const includeFrames = args.include_frames === true;const normalized = normalizeTimeFields(args);+ // Keep the clearer MCP name compatible with APIs that still call all+ // screen-text rows "ocr", including accessibility-derived captures.+ if (normalized.content_type === "screen") normalized.content_type = "ocr";// Default text cap if the caller didn't pass max_content_length.// Keeps single calls under Claude Code's per-tool output limit.const userCap = normalized.max_content_length;diff --git a/packages/screenpipe-mcp/src/screen-search.test.ts b/packages/screenpipe-mcp/src/screen-search.test.tsnew file mode 100644index 000000000..f9d7fb6bc--- /dev/null+++ b/packages/screenpipe-mcp/src/screen-search.test.ts@@ -0,0 +1,80 @@+// screenpipe — AI that knows everything you've seen, said, or heard+// https://screenpipe.com+import { it, expect } from "vitest";+import { Client } from "@modelcontextprotocol/sdk/client/index.js";+import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";+import { createServer } from "node:http";+import { execFileSync } from "node:child_process";+import path from "node:path";++it("advertises screen text and preserves legacy search behavior over the built MCP transport", async () => {+ const root = path.resolve(__dirname, "..");+ execFileSync("bun", ["run", "build"], { cwd: root, stdio: "pipe", timeout: 120000 });+ const requests: URL[] = [];+ const api = createServer((req, res) => {+ const url = new URL(req.url || "/", "http://localhost");+ requests.push(url);+ res.setHeader("Content-Type", "application/json");+ if (req.headers.authorization !== "Bearer fixture-key") {+ res.writeHead(403).end("{}");+ return;+ }+ // A backend that only accepts the legacy name proves MCP translates it.+ const sources = url.searchParams.get("content_type") === "ocr"+ ? ["accessibility", "ocr", undefined] : [];+ res.end(JSON.stringify({+ data: sources.map((source, index) => ({ type: "OCR", content: {+ frame_id: index + 1, app_name: "Fixture", window_name: "Screen text",+ timestamp: "2026-01-01T12:00:00Z", text: `Captured text ${index + 1}`,+ text_source: source, tags: ["project:fixture"], frame: "c3ludGhldGlj",+ } })),+ pagination: { total: sources.length ? 10 : 0, limit: 3, offset: Number(url.searchParams.get("offset") || 0) },+ }));+ });+ await new Promise<void>(resolve => api.listen(0, "127.0.0.1", resolve));+ const address = api.address();+ if (!address || typeof address === "string") throw new Error("No fixture API address");+ const client = new Client({ name: "screen-search-contract", version: "1.0" });+ const transport = new StdioClientTransport({+ command: process.execPath, args: [path.join(root, "dist/cli.js")],+ env: { PATH: process.env.PATH || "", SCREENPIPE_LOCAL_API_URL: `http://127.0.0.1:${address.port}`,+ SCREENPIPE_LOCAL_API_KEY: "fixture-key", SCREENPIPE_DISABLE_TELEMETRY: "1" }, stderr: "pipe",+ });+ try {+ await client.connect(transport);+ const { tools } = await client.listTools();+ const property = tools.find(tool => tool.name === "search-content")!.inputSchema.properties!.content_type as { enum: string[]; description: string };+ expect(property.enum).toEqual(expect.arrayContaining(["screen", "ocr", "accessibility", "all", "audio", "parsed"]));+ expect(property.description).toContain("Use 'screen' for screen text");+ expect(property.description).toContain("legacy alias");+ for (const contentType of ["screen", "ocr"]) {+ const filters = { app_name: "Fixture", window_name: "Screen text", q: "Captured", tags: "project:fixture",+ include_related: true, limit: 3, offset: 3, max_content_length: 100, include_frames: true };+ const result = await client.callTool({ name: "search-content", arguments: {+ content_type: contentType, start_time: "2026-01-01T00:00:00Z", end_time: "2026-01-02T00:00:00Z", ...filters,+ } });+ expect(result.isError).not.toBe(true);+ const text = JSON.stringify(result.content);+ expect(text).toContain("[Screen·a11y]");+ expect(text).toContain("[Screen·ocr]");+ expect(text).toContain("[Screen] Fixture");+ expect(text).toContain("offset=6");+ expect(result.content).toEqual(expect.arrayContaining([expect.objectContaining({ type: "image", data: "c3ludGhldGlj" })]));+ const params = requests.at(-1)!.searchParams;+ expect(params.get("content_type")).toBe("ocr");+ for (const [key, value] of Object.entries(filters)) expect(params.get(key)).toBe(String(value));+ expect(Date.parse(params.get("start_time")!)).toBe(Date.parse("2026-01-01T00:00:00Z"));+ expect(Date.parse(params.get("end_time")!)).toBe(Date.parse("2026-01-02T00:00:00Z"));+ }+ for (const contentType of ["all", "accessibility", "audio", "input", "memory", "parsed", undefined]) {+ await client.callTool({ name: "search-content", arguments: {+ start_time: "2026-01-01", ...(contentType ? { content_type: contentType } : {}),+ } });+ expect(requests.at(-1)!.searchParams.get("content_type")).toBe(contentType ?? null);+ }+ } finally {+ await client.close();+ await transport.close();+ await new Promise<void>(resolve => api.close(() => resolve()));+ }+}, 15000);