Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

feat(meetings): tell the summarizer to also query the screen (OCR)

Louis Beaumont0289575f49fa
diff --git a/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts b/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts
index e714c311c..44ada605b 100644
--- a/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts
+++ b/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts
@@ -531,6 +531,7 @@ export function buildMeetingSummarizeInstructions(
`meeting id: ${meetingId}`,
`primary transcript source: GET "http://localhost:3030/meetings/${meetingId}/transcript" and use each row's "transcript", "speakerName", "capturedAt", and "source" fields. sort rows by capturedAt before summarizing.`,
`fallback transcript source: /search?content_type=audio for the meeting time window. audio rows use content.transcription (not content.text); content.text may be missing for audio and should not be treated as an empty transcript.`,
+ `also read the screenpipe-api skill and query the screen for what was *shown* during the meeting: GET /search?content_type=ocr for the meeting window (this returns the frame's on-screen text — accessibility tree + OCR merged, not just OCR) — shared slides, docs, code, demos, and the on-screen name tags video-call apps render for participants. fold anything useful into the summary, and use on-screen names to fill in attendees who never spoke.`,
`if your summary is worth saving, append it to the meeting note (and refresh the title in the same call) via:`,
` curl -s -X PATCH "http://localhost:3030/meetings/${meetingId}" \\`,
` -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \\`,
diff --git a/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md b/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md
index bfaad40a3..c1743b137 100644
--- a/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md
+++ b/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md
@@ -28,6 +28,8 @@ the most recent row is the one that just ended. capture its `id`, `meeting_start
step 2 — search screenpipe for what happened during this meeting and summarize it: key topics, decisions, action items. scope your searches to the meeting's `meeting_start`/`meeting_end` window. prefer `content_type=audio` for transcripts.
+step 2b — also query the screen for what was *shown*: `content_type=ocr` over the same window (this returns the frame's on-screen text — accessibility tree + OCR merged, not just OCR) — shared slides, docs, code, demos, and the on-screen name tags video-call apps render for participants. fold anything useful into the summary, and use on-screen names to fill in attendees who never spoke.
+
step 3 — if your summary is worth saving, append it to the meeting note (and refresh the title in the same call) via:
curl -s -X PATCH "http://localhost:3030/meetings/<MEETING_ID>" \