YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
feat(meetings): tell the summarizer to also query the screen (OCR)
Louis Beaumont0289575f49fa
diff --git a/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts b/apps/screenpipe-app-tauri/lib/utils/meeting-context.tsindex e714c311c..44ada605b 100644--- a/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts+++ b/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts@@ -531,6 +531,7 @@ export function buildMeetingSummarizeInstructions(`meeting id: ${meetingId}`,`primary transcript source: GET "http://localhost:3030/meetings/${meetingId}/transcript" and use each row's "transcript", "speakerName", "capturedAt", and "source" fields. sort rows by capturedAt before summarizing.`,`fallback transcript source: /search?content_type=audio for the meeting time window. audio rows use content.transcription (not content.text); content.text may be missing for audio and should not be treated as an empty transcript.`,+ `also read the screenpipe-api skill and query the screen for what was *shown* during the meeting: GET /search?content_type=ocr for the meeting window (this returns the frame's on-screen text — accessibility tree + OCR merged, not just OCR) — shared slides, docs, code, demos, and the on-screen name tags video-call apps render for participants. fold anything useful into the summary, and use on-screen names to fill in attendees who never spoke.`,`if your summary is worth saving, append it to the meeting note (and refresh the title in the same call) via:`,` curl -s -X PATCH "http://localhost:3030/meetings/${meetingId}" \\`,` -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \\`,diff --git a/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md b/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.mdindex bfaad40a3..c1743b137 100644--- a/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md+++ b/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md@@ -28,6 +28,8 @@ the most recent row is the one that just ended. capture its `id`, `meeting_startstep 2 — search screenpipe for what happened during this meeting and summarize it: key topics, decisions, action items. scope your searches to the meeting's `meeting_start`/`meeting_end` window. prefer `content_type=audio` for transcripts.+step 2b — also query the screen for what was *shown*: `content_type=ocr` over the same window (this returns the frame's on-screen text — accessibility tree + OCR merged, not just OCR) — shared slides, docs, code, demos, and the on-screen name tags video-call apps render for participants. fold anything useful into the summary, and use on-screen names to fill in attendees who never spoke.+step 3 — if your summary is worth saving, append it to the meeting note (and refresh the title in the same call) via:curl -s -X PATCH "http://localhost:3030/meetings/<MEETING_ID>" \