YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
feat(meetings): auto-name speakers from screen + ask to push summary to a connection (#4341)
Louis Beaumontdae0db7dc642
diff --git a/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts b/apps/screenpipe-app-tauri/lib/utils/meeting-context.tsindex 9689c32f3..963e92a14 100644--- a/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts+++ b/apps/screenpipe-app-tauri/lib/utils/meeting-context.ts@@ -532,6 +532,7 @@ export function buildMeetingSummarizeInstructions(`primary transcript source: GET "http://localhost:3030/meetings/${meetingId}/transcript" and use each row's "transcript", "speakerName", "capturedAt", and "source" fields. sort rows by capturedAt before summarizing.`,`fallback transcript source: /search?content_type=audio for the meeting time window. audio rows use content.transcription (not content.text); content.text may be missing for audio and should not be treated as an empty transcript.`,`also read the screenpipe-api skill and query the screen for what was *shown* during the meeting: GET /search?content_type=ocr for the meeting window (this returns the frame's on-screen text — accessibility tree + OCR merged, not just OCR) — shared slides, docs, code, demos, and the on-screen name tags video-call apps render for participants. fold anything useful into the summary, and use on-screen names to fill in attendees who never spoke.`,+ `then name the speakers from the screen (do this every run, don't ask first): for every speaker still unnamed or generic ("speaker 1", "unknown", "") in the transcript above, line up when they were talking with the on-screen name tag showing at that moment, then GET /speakers/unnamed?limit=20 and POST /speakers/update {"id": <SPEAKER_ID>, "name": "<NAME_FROM_SCREEN>"} for each confident match. only rename when the on-screen evidence is unambiguous — never guess from voice alone. note which speakers you renamed (and which you left as-is) in your reply.`,`*if available*, use the cloud media (video/audio) model for what text alone can't capture: the screenpipe-api skill includes a "Cloud media analysis" block (the gemma4-e4b multimodal model) only when cloud media analysis is enabled and the user is signed into screenpipe cloud. when it's there, use it for visual-only content the transcript and OCR miss — diagrams, charts, whiteboards, slide figures, UI demos, screen-shared video. export the meeting (POST /export with this meeting_id) or ffmpeg-sample a few keyframes, then send up to 4 frames per request as image_url[] to POST /v1/chat/completions with "model": "gemma4-e4b" and ask what they show; it can also take audio_url directly for hard-to-transcribe spans. optional — if the block isn't in the skill, or the call returns 503 cloud_token_missing (signed out), skip it and summarize from transcript + OCR.`,`if your summary is worth saving, append it to the meeting note (and refresh the title in the same call) via:`,` curl -s -X PUT "http://localhost:3030/meetings/${meetingId}" \\`,@@ -544,7 +545,7 @@ export function buildMeetingSummarizeInstructions(if (options?.followUpAsk) {lines.push(``,- `after the PUT, ask the user — in one short message — whether they'd like you to (a) update speaker assignments for any of the audio segments above, or (b) push this summary into one of the apps they were using during the meeting (use the "apps used during meeting" + "tabs/docs visited" sections to list 2-3 plausible targets like Notion, Linear, GitHub, etc.). don't act on either until they reply.`,+ `after the PUT, offer to push the summary into one of the user's *connected* apps — ask first, never push on your own. don't guess at the integration list: GET http://localhost:3030/connections and keep only the ones with "connected": true, then ask in one short message which of those (if any) to push to. rank them by relevance — an app used during the meeting (see "apps used during meeting" / "tabs/docs visited") comes first. if nothing is connected, say so in one line (connecting Notion/Slack/Telegram/… would let you push next time) and stop. when they pick one, push via that connection's endpoint (POST /connections/<id>/send for slack/gmail/telegram/discord, POST /connections/<id>/proxy/... for notion/linear/etc.) and confirm specifics (channel, parent page) before anything leaves the machine.`,);}diff --git a/apps/screenpipe-app-tauri/lib/utils/meeting-format.ts b/apps/screenpipe-app-tauri/lib/utils/meeting-format.tsindex aa1d8d81e..d73d5cfef 100644--- a/apps/screenpipe-app-tauri/lib/utils/meeting-format.ts+++ b/apps/screenpipe-app-tauri/lib/utils/meeting-format.ts@@ -72,7 +72,7 @@ export function buildSummarizePrompt(meeting: MeetingRecord): string {if (meeting.attendees) parts.push(`attendees: ${meeting.attendees}`);if (meeting.note) parts.push(`notes: ${meeting.note}`);- return `search screenpipe for what happened during this meeting and summarize it: key topics, decisions, action items. then suggest which of my connected integrations would be useful to share this with and draft a message for each.\n\nmeeting:\n${parts.join("\n")}`;+ return `search screenpipe for what happened during this meeting and summarize it: key topics, decisions, action items. name any unnamed/generic speakers from the on-screen name tags video-call apps render (GET /search?content_type=ocr over the meeting window, then POST /speakers/update) — do this without asking, only on unambiguous evidence. then offer to push the summary to one of my *connected* apps: GET http://localhost:3030/connections (keep "connected": true), rank by the apps used during the meeting, and ask which to push to before sending anything.\n\nmeeting:\n${parts.join("\n")}`;}/**diff --git a/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md b/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.mdindex 3f9d8981f..e839fe05a 100644--- a/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md+++ b/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md@@ -32,6 +32,19 @@ step 2b — also query the screen for what was *shown*: `content_type=ocr` overstep 2c — *if available*, use the cloud media (video/audio) model for what text alone can't capture: the screenpipe-api skill includes a "Cloud media analysis" block (the `gemma4-e4b` multimodal model) only when cloud media analysis is enabled and the user is signed into screenpipe cloud. when it's there, use it for visual-only content the transcript and OCR miss — diagrams, charts, whiteboards, slide figures, UI demos, screen-shared video. export the meeting (`POST /export` with this `meeting_id`) or ffmpeg-sample a few keyframes, then send up to 4 frames per request as `image_url[]` to `POST /v1/chat/completions` with `"model": "gemma4-e4b"` and ask what they show; it can also take `audio_url` directly for hard-to-transcribe spans. this step is optional — if the block isn't in the skill, or the call returns `503 cloud_token_missing` (signed out), just skip it and summarize from transcript + OCR.+step 2d — name the speakers from the screen (do this every run, don't ask first): video-call apps render each participant's name on their tile, and that text is already in the `content_type=ocr` frames from step 2b. for every speaker still unnamed or generic ("speaker 1", "unknown", "") in the transcript, line up when they were talking with the on-screen name tag showing at that moment and rename them:++ # speakers with no name yet+ curl -s -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \+ "http://localhost:3030/speakers/unnamed?limit=20"+ # apply a confident match+ curl -s -X POST "http://localhost:3030/speakers/update" \+ -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \+ -H "Content-Type: application/json" \+ -d '{"id": <SPEAKER_ID>, "name": "<NAME_FROM_SCREEN>"}'++only rename when the on-screen evidence is unambiguous — never guess from voice alone. note which speakers you renamed (and which you left as-is) in your final message.+step 3 — if your summary is worth saving, append it to the meeting note (and refresh the title in the same call) via:curl -s -X PUT "http://localhost:3030/meetings/<MEETING_ID>" \@@ -40,3 +53,19 @@ step 3 — if your summary is worth saving, append it to the meeting note (and r-d '{"title": "<NEW_TITLE_OR_OMIT>", "note": "<EXISTING_NOTE>\n\n## Summary\n<YOUR_SUMMARY>"}'replace `<EXISTING_NOTE>` with the meeting's current `note` field (empty string if none) so you don't overwrite the user's work; just append your summary under a `## Summary` heading. for the title: if the current title is missing, generic ("untitled", "meeting", just the app name) or doesn't capture what actually happened, replace it with a 5-8 word plain-english title (no quotes, no "meeting about…" prefix) — otherwise omit the field so a user-set title is left alone. if there's nothing useful to summarize (empty transcript, irrelevant audio), say so out loud and skip the PUT — don't write a placeholder.++step 4 — offer to push the summary into one of the user's connected apps (ask, never push on your own). list what's actually connected, then let them pick with one click:++ curl -s -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030/connections" # keep only "connected": true++rank the connected targets by relevance — an app used during the meeting first (Notion, Slack, Linear, …). then post a desktop notification whose action buttons are those targets, so the ask renders as buttons in the UI:++ curl -s -X POST "http://localhost:11435/notify" \+ -H "Content-Type: application/json" \+ -d '{"title": "<TITLE> summarized", "body": "<one-line recap> — push it somewhere?", "actions": [+ {"label": "push to notion", "type": "api", "method": "POST", "url": "http://localhost:3030/connections/notion/proxy/v1/pages", "body": { /* page payload built from the summary */ }},+ {"label": "review in chat", "type": "pipe", "pipe": "meeting-summary", "open_in_chat": true, "context": {"meeting_id": <ID>}},+ {"label": "dismiss", "type": "dismiss"}+ ]}'++each button maps to a connection's endpoint from its `/connections` `description` (`POST /connections/<id>/send` for slack/gmail/telegram/discord, `POST /connections/<id>/proxy/...` for notion/linear/etc.). when a target needs a destination you can't infer (a Notion parent page, a Slack channel), make that button `"review in chat"` so the user confirms specifics before anything leaves the machine. if nothing is connected, skip the notification and just say that connecting an app would let you push summaries next time.