YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
feat(search): include_related co-occurring tags + memory frame_id (#4429) (#4430)
Louis Beaumont66f580f237d3
diff --git a/.claude/skills/screenpipe-api/SKILL.md b/.claude/skills/screenpipe-api/SKILL.mdindex f792ce663..aa2c3b099 100644--- a/.claude/skills/screenpipe-api/SKILL.md+++ b/.claude/skills/screenpipe-api/SKILL.md@@ -46,6 +46,7 @@ curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030| `speaker_name` | string | No | Filter audio by speaker (case-insensitive partial) || `focused` | boolean | No | Only focused windows || `tags` | string | No | Comma-separated; return only items carrying ALL of them (e.g. `person:ada,project:atlas`). Works for screen/audio and, with `content_type=memory`, memories. See Tags below. |+| `include_related` | boolean | No | With `tags`, also return a `related` map of co-occurring tags (people/projects/workflows seen alongside yours), most-frequent first. One call for the surrounding context instead of several. See Tags below. || `max_content_length` | integer | No | Truncate each result's text (middle-truncation) || `format` | string | No | `json` (default), `csv`, `tsv`/`table`, or `outline`/`tree` (element endpoints only). CSV/TSV return a columnar table (column names written once) instead of one JSON object per row. `outline` returns a deduped indented text tree of the text-bearing UI nodes — the cheapest read for "what's on screen?" (~91% fewer tokens). CSV is lossless; TSV collapses newlines (worse for long `ocr` text). || `fields` | string | No | Comma-separated column allowlist of dotted paths, e.g. `type,content.app_name,content.text`. Returns only those columns (handy for dropping the repeated absolute `content.file_path`). Works for `json` too (sparse objects). |@@ -78,7 +79,15 @@ Tags are a shared label layer across screen, audio, and memories under one strin- Add to a memory: include `tags` in `POST /memories` (or `PUT /memories/{id}`).- Retrieve by tag: `GET /search?tags=person:ada&start_time=30d%20ago` (screen+audio), or add `content_type=memory` for memories. Multiple tags AND together; matching is exact, not substring.-Frames are pruned by retention, so for a durable link tag a memory (memories also carry `created_at` and a `frame_id` back to the moment). To pull everything about a person across time: one call for captures (`content_type=all&tags=person:ada`) plus one for facts (`content_type=memory&tags=person:ada`).+Frames are pruned by retention, so for a durable link tag a memory (memories also carry `created_at` and a `frame_id` back to the moment — jump there with `GET /frames/{frame_id}`). To pull everything about a person across time: one call for captures (`content_type=all&tags=person:ada`) plus one for facts (`content_type=memory&tags=person:ada`).++Add `include_related=true` to a tag query to get the surrounding context in the same response — the tags that co-occur with yours, grouped by namespace (prefix pluralized: `person:`→`people`, `project:`→`projects`) and ranked by frequency. Replaces the 2-3 follow-up "who/what else" calls with one:++```bash+curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \+ "http://localhost:3030/search?tags=person:ada&include_related=true&limit=5"+# data: [...], related: { "people": ["connor","drew"], "projects": ["atlas"], "workflows": ["planning"] }+```### Critical Rulesdiff --git a/crates/screenpipe-core/assets/skills/screenpipe-api/SKILL.md b/crates/screenpipe-core/assets/skills/screenpipe-api/SKILL.mdindex b5e55aec8..45ba103aa 100644--- a/crates/screenpipe-core/assets/skills/screenpipe-api/SKILL.md+++ b/crates/screenpipe-core/assets/skills/screenpipe-api/SKILL.md@@ -64,6 +64,7 @@ curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030| `speaker_name` | string | No | Filter audio by speaker (case-insensitive partial) || `focused` | boolean | No | Only focused windows || `tags` | string | No | Comma-separated; return only items carrying ALL of them (e.g. `person:ada,project:atlas`). Works for screen/audio and, with `content_type=memory`, memories. See Tags below. |+| `include_related` | boolean | No | With `tags`, also return a `related` map of co-occurring tags (people/projects/workflows seen alongside yours), most-frequent first. One call for the surrounding context instead of several. See Tags below. || `max_content_length` | integer | No | Truncate each result's text (middle-truncation) || `format` | string | No | `json` (default), `csv`, or `tsv`/`table`. CSV/TSV return a columnar table (column names written once) instead of one JSON object per row, much cheaper to read on list-shaped results. CSV is lossless; TSV collapses newlines (worse for long `ocr` text). || `fields` | string | No | Comma-separated column allowlist of dotted paths, e.g. `type,content.app_name,content.text`. Returns only those columns (handy for dropping the repeated absolute `content.file_path`). Works for `json` too (sparse objects). |@@ -76,7 +77,15 @@ Tags are a shared label layer across screen, audio, and memories under one strin- Add to a memory: include `tags` in `POST /memories` (or `PUT /memories/{id}`).- Retrieve by tag: `GET /search?tags=person:ada&start_time=30d%20ago` (screen+audio), or add `content_type=memory` for memories. Multiple tags AND together; matching is exact, not substring.-Frames are pruned by retention, so for a durable link tag a memory (memories also carry `created_at` and a `frame_id` back to the moment). Tag frames/audio for short-term recall. To pull everything about a person across time: one call for captures (`content_type=all&tags=person:ada`) plus one for facts (`content_type=memory&tags=person:ada`).+Frames are pruned by retention, so for a durable link tag a memory (memories also carry `created_at` and a `frame_id` back to the moment — jump there with `GET /frames/{frame_id}`). Tag frames/audio for short-term recall. To pull everything about a person across time: one call for captures (`content_type=all&tags=person:ada`) plus one for facts (`content_type=memory&tags=person:ada`).++Add `include_related=true` to a tag query to get the surrounding context in the same response — the tags that co-occur with yours, grouped by namespace (prefix pluralized: `person:`→`people`, `project:`→`projects`) and ranked by frequency. Replaces the 2-3 follow-up "who/what else" calls with one:++```bash+curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \+ "http://localhost:3030/search?tags=person:ada&include_related=true&limit=5"+# data: [...], related: { "people": ["connor","drew"], "projects": ["atlas"], "workflows": ["planning"] }+```### Critical Rules@@ -500,7 +509,7 @@ curl -X DELETE http://localhost:3030/memories/1Parameters for `GET /memories`: `q` (FTS search), `source`, `tags`, `min_importance`, `start_time`, `end_time`, `limit`, `offset`.-Use `/memories` directly — `/search` does not currently surface `content_type=memory` results.+`GET /memories` is the direct read (richest filters: `min_importance`, `source`). Memories also come back via `GET /search?content_type=memory` (note: `content_type=all` does NOT include them — ask for `memory` explicitly), which is the path that supports `tags` + `include_related` for tag-linked recall.### Creating Memoriesdiff --git a/crates/screenpipe-db/src/db/search.rs b/crates/screenpipe-db/src/db/search.rsindex b062f91aa..336d904de 100644--- a/crates/screenpipe-db/src/db/search.rs+++ b/crates/screenpipe-db/src/db/search.rs@@ -1748,4 +1748,102 @@ impl DatabaseManager {Ok((latest_frame.map(|f| f.0), latest_audio.map(|a| a.0), None))}++ /// Tags that co-occur with ALL of the given `tags`, most-frequent first,+ /// excluding the input tags themselves. Spans the same three stores as the+ /// tag filter on [`search_with_tags`](Self::search_with_tags): the screen+ /// (`vision_tags`) and audio (`audio_tags`) junction tables plus the+ /// `memories.tags` JSON array.+ ///+ /// Powers `GET /search?...&include_related=true`: one query surfaces the+ /// people / projects / workflows that appear alongside a tag so an AI+ /// caller gets the surrounding context without N follow-up requests.+ /// Returns each co-occurring tag's full namespaced name and its count.+ /// An empty `tags` slice returns an empty vec; duplicate inputs are folded+ /// (the `DISTINCT` in the `input` CTE) so they match like a single tag.+ ///+ /// Cost: the vision/audio legs ride the tag indexes (`idx_*_tags_tag_id` ++ /// `tags.name`), but the memories leg full-scans + `json_each` because+ /// `memories.tags` is an unindexed JSON column — the same linear cost the+ /// tag *filter* already pays (see `tests/tag_filter_bench.rs`). Measured on+ /// a 200k-frame / 250k-vision_tag / 50k-memory in-memory DB: ~21 ms for a+ /// realistic tag, ~150 ms worst-case for a hot tag on 50k items with wide+ /// fan-out. The HTTP handler bounds it with a timeout and treats it as+ /// optional. If memory counts ever reach millions, give them a+ /// `memory_tags` junction table mirroring `vision_tags`.+ pub async fn related_tags(+ &self,+ tags: &[String],+ limit: u32,+ ) -> Result<Vec<(String, i64)>, sqlx::Error> {+ if tags.is_empty() {+ return Ok(Vec::new());+ }++ // Same JSON-array binding trick as the tag filter: pass the tags as a+ // JSON array and expand with `json_each`. `n.c` is the input cardinality+ // so the `HAVING` clauses keep only items carrying ALL requested tags.+ // The `memories.tags` reads wrap the column in `CASE WHEN json_valid`+ // because a single legacy/sync row that isn't valid JSON would+ // otherwise make `json_each` raise "malformed JSON" and 500 the whole+ // query (same guard as `list_memory_tags`).+ let tags_json = serde_json::to_string(tags).unwrap_or_else(|_| "[]".to_string());++ let rows: Vec<(String, i64)> = sqlx::query_as(+ r#"+ WITH input(name) AS (SELECT DISTINCT value FROM json_each(?1)),+ n(c) AS (SELECT COUNT(*) FROM input),+ vision_matches(id) AS (+ SELECT vt.vision_id+ FROM vision_tags vt JOIN tags t ON vt.tag_id = t.id+ WHERE t.name IN (SELECT name FROM input)+ GROUP BY vt.vision_id+ HAVING COUNT(DISTINCT t.name) = (SELECT c FROM n)+ ),+ audio_matches(id) AS (+ SELECT aud.audio_chunk_id+ FROM audio_tags aud JOIN tags t ON aud.tag_id = t.id+ WHERE t.name IN (SELECT name FROM input)+ GROUP BY aud.audio_chunk_id+ HAVING COUNT(DISTINCT t.name) = (SELECT c FROM n)+ ),+ memory_matches(id) AS (+ SELECT m.id+ FROM memories m+ WHERE (+ SELECT COUNT(DISTINCT j.value)+ FROM json_each(CASE WHEN json_valid(m.tags) THEN m.tags ELSE '[]' END) j+ WHERE j.value IN (SELECT name FROM input)+ ) = (SELECT c FROM n)+ ),+ co(name) AS (+ SELECT t.name+ FROM vision_tags vt JOIN tags t ON vt.tag_id = t.id+ WHERE vt.vision_id IN (SELECT id FROM vision_matches)+ UNION ALL+ SELECT t.name+ FROM audio_tags aud JOIN tags t ON aud.tag_id = t.id+ WHERE aud.audio_chunk_id IN (SELECT id FROM audio_matches)+ UNION ALL+ SELECT j.value+ FROM memories m,+ json_each(CASE WHEN json_valid(m.tags) THEN m.tags ELSE '[]' END) j+ WHERE m.id IN (SELECT id FROM memory_matches)+ )+ SELECT name, COUNT(*) AS count+ FROM co+ WHERE name IS NOT NULL AND name != ''+ AND name NOT IN (SELECT name FROM input)+ GROUP BY name+ ORDER BY count DESC, name ASC+ LIMIT ?2+ "#,+ )+ .bind(&tags_json)+ .bind(limit as i64)+ .fetch_all(&self.pool)+ .await?;++ Ok(rows)+ }}diff --git a/crates/screenpipe-db/tests/db.rs b/crates/screenpipe-db/tests/db.rsindex 3c8ae5efe..a23159b3d 100644--- a/crates/screenpipe-db/tests/db.rs+++ b/crates/screenpipe-db/tests/db.rs@@ -484,6 +484,245 @@ mod tests {assert_eq!(n, 2);}+ /// `related_tags` returns the tags that co-occur with the requested ones,+ /// counted across all three stores (vision frames, audio chunks, memory+ /// JSON), most-frequent first, with the inputs themselves excluded and+ /// AND-semantics on multiple inputs.+ #[tokio::test]+ async fn test_related_tags_co_occurrence() {+ let db = setup_test_db().await;+ db.insert_video_chunk("v.mp4", "dev").await.unwrap();++ // Frame carrying person:ada alongside a project and a workflow.+ let f_a = db+ .insert_frame("dev", None, None, Some("app"), Some(""), false, None)+ .await+ .unwrap();+ db.add_tags(+ f_a,+ TagContentType::Vision,+ vec![+ "person:ada".to_string(),+ "project:atlas".to_string(),+ "workflow:planning".to_string(),+ ],+ )+ .await+ .unwrap();++ // Frame carrying person:ada with a second person and the same project.+ let f_b = db+ .insert_frame("dev", None, None, Some("app"), Some(""), false, None)+ .await+ .unwrap();+ db.add_tags(+ f_b,+ TagContentType::Vision,+ vec![+ "person:ada".to_string(),+ "person:connor".to_string(),+ "project:atlas".to_string(),+ ],+ )+ .await+ .unwrap();++ // Unrelated frame — its tag must never surface for person:ada.+ let f_x = db+ .insert_frame("dev", None, None, Some("app"), Some(""), false, None)+ .await+ .unwrap();+ db.add_tags(f_x, TagContentType::Vision, vec!["person:bob".to_string()])+ .await+ .unwrap();++ // Audio chunk carrying person:ada + the same project (third hit).+ let ac = db.insert_audio_chunk("a.mp4", None).await.unwrap();+ db.add_tags(+ ac,+ TagContentType::Audio,+ vec!["person:ada".to_string(), "project:atlas".to_string()],+ )+ .await+ .unwrap();++ // Memory carrying person:ada alongside a different person.+ db.insert_memory(+ "ada + drew planning",+ "user",+ None,+ Some(r#"["person:ada","person:drew"]"#),+ 0.5,+ None,+ )+ .await+ .unwrap();++ // Single input tag: project:atlas co-occurs 3× (f_a, f_b, ac); the+ // three singletons tie and break by name ascending. person:ada (input)+ // and person:bob (unrelated frame) must be absent.+ let related = db+ .related_tags(&["person:ada".to_string()], 50)+ .await+ .unwrap();+ let counts: std::collections::HashMap<&str, i64> = related+ .iter()+ .map(|(n, c)| (n.as_str(), *c))+ .collect();+ assert_eq!(related.len(), 4, "got {related:?}");+ assert_eq!(related[0], ("project:atlas".to_string(), 3));+ assert_eq!(counts.get("project:atlas"), Some(&3));+ assert_eq!(counts.get("workflow:planning"), Some(&1));+ assert_eq!(counts.get("person:connor"), Some(&1));+ assert_eq!(counts.get("person:drew"), Some(&1));+ assert!(!counts.contains_key("person:ada"), "input tag leaked");+ assert!(!counts.contains_key("person:bob"), "unrelated tag leaked");++ // The `limit` truncates to the top-N by count.+ let top1 = db.related_tags(&["person:ada".to_string()], 1).await.unwrap();+ assert_eq!(top1, vec![("project:atlas".to_string(), 3)]);++ // Multiple inputs → AND: only items carrying BOTH person:ada AND+ // project:atlas (f_a, f_b, ac — not the memory, which lacks the+ // project). Co-occurring extras: workflow:planning and person:connor.+ let both = db+ .related_tags(+ &["person:ada".to_string(), "project:atlas".to_string()],+ 50,+ )+ .await+ .unwrap();+ let both_names: std::collections::HashSet<&str> =+ both.iter().map(|(n, _)| n.as_str()).collect();+ assert_eq!(both.len(), 2, "got {both:?}");+ assert!(both_names.contains("workflow:planning"));+ assert!(both_names.contains("person:connor"));+ assert!(!both_names.contains("person:ada"));+ assert!(!both_names.contains("project:atlas"));++ // Empty input → empty result (no tags to relate against).+ assert!(db.related_tags(&[], 50).await.unwrap().is_empty());+ }++ /// Adversarial inputs for `related_tags`: malformed memory JSON, colon-rich+ /// tag values, duplicate inputs, store isolation, limit edges, unicode, and+ /// quote/`%`/`_` injection-shaped strings. None may error or leak.+ #[tokio::test]+ async fn test_related_tags_edge_cases() {+ let db = setup_test_db().await;+ db.insert_video_chunk("v.mp4", "dev").await.unwrap();++ // A frame carrying the anchor + a value that itself contains colons+ // (e.g. a URL tag) — the split into a namespace happens in the route,+ // so the DB must return the full name verbatim.+ let f = db+ .insert_frame("dev", None, None, Some("app"), Some(""), false, None)+ .await+ .unwrap();+ db.add_tags(+ f,+ TagContentType::Vision,+ vec![+ "person:ada".to_string(),+ "url:https://example.com:8080/x".to_string(),+ "emoji:🦀".to_string(),+ ],+ )+ .await+ .unwrap();++ // A memory carrying the anchor + a SQL-injection-shaped value. Bound as+ // a JSON param, so it's inert; it must come back as data, not break out.+ db.insert_memory(+ "weird tags",+ "user",+ None,+ Some(r#"["person:ada","weird:a' OR 1=1 --","like:50%_x"]"#),+ 0.5,+ None,+ )+ .await+ .unwrap();++ // A memory whose `tags` column is NOT valid JSON. The `json_valid`+ // guard must skip it instead of letting `json_each` raise and 500.+ db.insert_memory("legacy", "user", None, Some("not valid json"), 0.5, None)+ .await+ .unwrap();+ // ...and one carrying the anchor twice (deliberately) — the value must+ // still be counted once per memory, not double.+ db.insert_memory(+ "dupe-in-row",+ "user",+ None,+ Some(r#"["person:ada","person:ada","only:here"]"#),+ 0.5,+ None,+ )+ .await+ .unwrap();++ let related = db+ .related_tags(&["person:ada".to_string()], 50)+ .await+ .unwrap();+ let names: std::collections::HashSet<&str> =+ related.iter().map(|(n, _)| n.as_str()).collect();+ // Colon-rich and unicode values survive intact.+ assert!(names.contains("url:https://example.com:8080/x"));+ assert!(names.contains("emoji:🦀"));+ // Injection-shaped strings come back as plain data.+ assert!(names.contains("weird:a' OR 1=1 --"));+ assert!(names.contains("like:50%_x"));+ assert!(names.contains("only:here"));+ // The anchor itself is never echoed back.+ assert!(!names.contains("person:ada"));++ // Duplicate input tags must behave like a single input (the DISTINCT in+ // the input CTE), NOT silently match nothing.+ let deduped = db+ .related_tags(+ &["person:ada".to_string(), "person:ada".to_string()],+ 50,+ )+ .await+ .unwrap();+ let deduped_names: std::collections::HashSet<&str> =+ deduped.iter().map(|(n, _)| n.as_str()).collect();+ assert_eq!(deduped_names, names, "duplicate inputs must equal single");++ // limit = 0 → no rows (not an error).+ assert!(db+ .related_tags(&["person:ada".to_string()], 0)+ .await+ .unwrap()+ .is_empty());++ // A tag that exists on nothing → empty, no error.+ assert!(db+ .related_tags(&["person:ghost".to_string()], 50)+ .await+ .unwrap()+ .is_empty());++ // Store isolation: an anchor present ONLY in audio still finds its+ // audio-side co-tags and nothing from unrelated frames/memories.+ let ac = db.insert_audio_chunk("a.mp4", None).await.unwrap();+ db.add_tags(+ ac,+ TagContentType::Audio,+ vec!["call:standup".to_string(), "person:bob".to_string()],+ )+ .await+ .unwrap();+ let audio_only = db+ .related_tags(&["call:standup".to_string()], 50)+ .await+ .unwrap();+ assert_eq!(audio_only.len(), 1);+ assert_eq!(audio_only[0], ("person:bob".to_string(), 1));+ }+#[tokio::test]async fn test_recent_output_audio_detects_deferred_output_chunk() {let db = setup_test_db().await;diff --git a/crates/screenpipe-db/tests/tag_filter_bench.rs b/crates/screenpipe-db/tests/tag_filter_bench.rsindex ed89840f7..77e3b3540 100644--- a/crates/screenpipe-db/tests/tag_filter_bench.rs+++ b/crates/screenpipe-db/tests/tag_filter_bench.rs@@ -314,3 +314,228 @@ async fn bench_tag_filter_scaling() {);}}++/// Scaling check for `related_tags` (powers `?include_related=true`).+///+/// Ignored by default. Run explicitly:+/// cargo test -p screenpipe-db --test tag_filter_bench related_tags -- --ignored --nocapture+///+/// `related_tags` is the inverse-shaped sibling of the tag filter: instead of+/// "items carrying this tag", it returns "every OTHER tag on those items,+/// counted". Two cost drivers to pin down:+/// 1. a HOT input tag — on many items, each carrying many other tags — makes+/// the `co` CTE materialize (items × tags-per-item) rows before GROUP BY.+/// 2. memories have no tag index, so the memory leg full-scans + json_each+/// (same linear cost the tag-filter bench already documents).+/// This seed maximizes (1): `person:hot` is on 50k frames that EACH carry 4+/// extra co-tags (so 50k×5 = 250k rows feed the aggregation) plus a 50k-memory+/// store for (2). `person:cold` (200 items) is the realistic case.+#[tokio::test]+#[ignore]+async fn bench_related_tags_scaling() {+ const N_FRAMES_R: i64 = 200_000;+ const HOT_FRAMES: i64 = 50_000; // person:hot is on the first 50k frames+ const CO_POOL: i64 = 50; // each hot frame carries 4 of these → wide fan-out+ const COLD_FRAMES: i64 = 200; // realistic tag+ const N_MEM_R: i64 = 50_000;+ const HOT_MEM: i64 = 50_000; // every memory carries person:hot + a co tag++ let db = migrated_db().await;+ let seed = Instant::now();++ exec(+ &db,+ "INSERT INTO video_chunks (file_path, device_name) VALUES ('v.mp4','dev')",+ )+ .await;+ let chunk_id: i64 = sqlx::query_scalar("SELECT id FROM video_chunks LIMIT 1")+ .fetch_one(&db.pool)+ .await+ .unwrap();+ exec(+ &db,+ &format!(+ "INSERT INTO frames (video_chunk_id, offset_index, timestamp, name, app_name, window_name, focused, device_name) \+ WITH RECURSIVE seq(i) AS (SELECT 0 UNION ALL SELECT i+1 FROM seq WHERE i < {n}-1) \+ SELECT {chunk}, i, datetime('2026-01-01 00:00:00', '+'||i||' seconds'), 'f'||i, 'app', 'win', 0, 'dev' FROM seq",+ n = N_FRAMES_R,+ chunk = chunk_id+ ),+ )+ .await;++ // Tags: person:hot (1), person:cold (2), then CO_POOL co:N tags (3..).+ exec(+ &db,+ "INSERT INTO tags (name) VALUES ('person:hot'), ('person:cold')",+ )+ .await;+ exec(+ &db,+ &format!(+ "INSERT INTO tags (name) WITH RECURSIVE s(i) AS (SELECT 0 UNION ALL SELECT i+1 FROM s WHERE i<{p}-1) SELECT 'co:'||i FROM s",+ p = CO_POOL+ ),+ )+ .await;+ let hot: i64 = sqlx::query_scalar("SELECT id FROM tags WHERE name='person:hot'")+ .fetch_one(&db.pool)+ .await+ .unwrap();+ let cold: i64 = sqlx::query_scalar("SELECT id FROM tags WHERE name='person:cold'")+ .fetch_one(&db.pool)+ .await+ .unwrap();+ let co_base: i64 = sqlx::query_scalar("SELECT id FROM tags WHERE name='co:0'")+ .fetch_one(&db.pool)+ .await+ .unwrap();++ // person:hot on the first HOT_FRAMES frames.+ exec(+ &db,+ &format!(+ "INSERT INTO vision_tags (vision_id, tag_id) \+ WITH RECURSIVE s(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<{k}) SELECT i, {hot} FROM s",+ k = HOT_FRAMES,+ hot = hot+ ),+ )+ .await;+ // Each hot frame also carries 4 co-tags → wide co-occurrence fan-out.+ for off in 0..4i64 {+ exec(+ &db,+ &format!(+ "INSERT OR IGNORE INTO vision_tags (vision_id, tag_id) \+ WITH RECURSIVE s(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<{k}) \+ SELECT i, {base} + ((i + {off}) % {pool}) FROM s",+ k = HOT_FRAMES,+ base = co_base,+ off = off,+ pool = CO_POOL+ ),+ )+ .await;+ }+ // person:cold on the first COLD_FRAMES frames (they already carry co-tags).+ exec(+ &db,+ &format!(+ "INSERT INTO vision_tags (vision_id, tag_id) \+ WITH RECURSIVE s(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<{k}) SELECT i, {cold} FROM s",+ k = COLD_FRAMES,+ cold = cold+ ),+ )+ .await;++ // Memories: every one carries person:hot + a rotating co:N (full-scan leg).+ exec(+ &db,+ &format!(+ "INSERT INTO memories (content, tags, importance) \+ WITH RECURSIVE s(i) AS (SELECT 0 UNION ALL SELECT i+1 FROM s WHERE i<{n}-1) \+ SELECT 'mem '||i, \+ CASE WHEN i<{k} THEN '[\"person:hot\",\"co:'||(i%{pool})||'\"]' ELSE '[\"noise:'||(i%500)||'\"]' END, \+ 0.5 FROM s",+ n = N_MEM_R,+ k = HOT_MEM,+ pool = CO_POOL+ ),+ )+ .await;++ let counts: (i64, i64, i64) = sqlx::query_as(+ "SELECT (SELECT COUNT(*) FROM frames), (SELECT COUNT(*) FROM vision_tags), (SELECT COUNT(*) FROM memories)",+ )+ .fetch_one(&db.pool)+ .await+ .unwrap();+ println!(+ "seeded in {:?}: frames={} vision_tags={} memories={}",+ seed.elapsed(),+ counts.0,+ counts.1,+ counts.2+ );++ // ---- query plan: confirm the vision/audio legs ride the tag indexes and+ // only the memories leg scans (the documented, accepted linear cost). ----+ let related_sql = r#"+ WITH input(name) AS (SELECT DISTINCT value FROM json_each(?1)),+ n(c) AS (SELECT COUNT(*) FROM input),+ vision_matches(id) AS (+ SELECT vt.vision_id FROM vision_tags vt JOIN tags t ON vt.tag_id = t.id+ WHERE t.name IN (SELECT name FROM input)+ GROUP BY vt.vision_id HAVING COUNT(DISTINCT t.name) = (SELECT c FROM n)+ ),+ audio_matches(id) AS (+ SELECT aud.audio_chunk_id FROM audio_tags aud JOIN tags t ON aud.tag_id = t.id+ WHERE t.name IN (SELECT name FROM input)+ GROUP BY aud.audio_chunk_id HAVING COUNT(DISTINCT t.name) = (SELECT c FROM n)+ ),+ memory_matches(id) AS (+ SELECT m.id FROM memories m+ WHERE (SELECT COUNT(DISTINCT j.value)+ FROM json_each(CASE WHEN json_valid(m.tags) THEN m.tags ELSE '[]' END) j+ WHERE j.value IN (SELECT name FROM input)) = (SELECT c FROM n)+ ),+ co(name) AS (+ SELECT t.name FROM vision_tags vt JOIN tags t ON vt.tag_id = t.id+ WHERE vt.vision_id IN (SELECT id FROM vision_matches)+ UNION ALL+ SELECT t.name FROM audio_tags aud JOIN tags t ON aud.tag_id = t.id+ WHERE aud.audio_chunk_id IN (SELECT id FROM audio_matches)+ UNION ALL+ SELECT j.value FROM memories m,+ json_each(CASE WHEN json_valid(m.tags) THEN m.tags ELSE '[]' END) j+ WHERE m.id IN (SELECT id FROM memory_matches)+ )+ SELECT name, COUNT(*) AS count FROM co+ WHERE name IS NOT NULL AND name != '' AND name NOT IN (SELECT name FROM input)+ GROUP BY name ORDER BY count DESC, name ASC LIMIT ?2"#;+ let plan: Vec<(i64, i64, i64, String)> =+ sqlx::query_as(&format!("EXPLAIN QUERY PLAN {related_sql}"))+ .bind("[\"person:hot\"]")+ .bind(30i64)+ .fetch_all(&db.pool)+ .await+ .unwrap();+ println!("\n--- PLAN: related_tags(person:hot) ---");+ for (_, _, _, detail) in plan {+ println!(" {detail}");+ }++ // ---- timings (best of 3 after warm-up) ----+ let bench = |label: &'static str, tags: Vec<String>| {+ let db = &db;+ async move {+ let _ = db.related_tags(&tags, 30).await.unwrap(); // warm-up+ let mut best = std::time::Duration::MAX;+ let mut rows = 0usize;+ let mut top = String::new();+ for _ in 0..3 {+ let t = Instant::now();+ let r = db.related_tags(&tags, 30).await.unwrap();+ best = best.min(t.elapsed());+ rows = r.len();+ top = r+ .first()+ .map(|(n, c)| format!("{n}={c}"))+ .unwrap_or_default();+ }+ println!("related {label:<40} -> {rows:>3} groups (top {top}), best {best:?}");+ }+ };++ println!("\n--- timings ---");+ bench("person:hot (50k items, wide fan-out)", vec!["person:hot".to_string()]).await;+ bench("person:cold (200 items, realistic)", vec!["person:cold".to_string()]).await;+ bench(+ "person:hot+co:0 (AND, two inputs)",+ vec!["person:hot".to_string(), "co:0".to_string()],+ )+ .await;+ bench("nonexistent:tag (no matches)", vec!["nonexistent:tag".to_string()]).await;+}diff --git a/crates/screenpipe-engine/src/routes/content.rs b/crates/screenpipe-engine/src/routes/content.rsindex 0025b6e9a..bbcc45a40 100644--- a/crates/screenpipe-engine/src/routes/content.rs+++ b/crates/screenpipe-engine/src/routes/content.rs@@ -50,6 +50,10 @@ pub struct MemoryContent {pub source_context: Option<serde_json::Value>,pub tags: Vec<String>,pub importance: f64,+ /// First frame of the workflow segment this memory describes, when known.+ /// Lets a caller jump straight from a memory to the exact captured moment+ /// (`GET /frames/{frame_id}`). Null for memories with no frame provenance.+ pub frame_id: Option<i64>,pub created_at: String,pub updated_at: String,}diff --git a/crates/screenpipe-engine/src/routes/search.rs b/crates/screenpipe-engine/src/routes/search.rsindex d6709d339..a4840e947 100644--- a/crates/screenpipe-engine/src/routes/search.rs+++ b/crates/screenpipe-engine/src/routes/search.rs@@ -146,6 +146,14 @@ pub(crate) struct SearchQuery {/// nothing when this is set. Omit for no tag filtering.#[serde(default, deserialize_with = "from_comma_separated_string_array")]tags: Option<Vec<String>>,+ /// When `true` and a `tags` filter is set, attach a `related` block to the+ /// response: the tags that co-occur with the requested ones, grouped by+ /// namespace (`people`, `projects`, `workflows`, …) and ordered+ /// most-frequent first. Lets an AI caller pull the surrounding context+ /// (who/what/which-workflow showed up alongside a tag) in one request+ /// instead of several follow-up queries. No-op without `tags`.+ #[serde(default, deserialize_with = "deserialize_flexible_bool")]+ include_related: bool,/// Output format: `json` (default), `csv`, or `tsv`/`table`. CSV/TSV emit a/// columnar table (column names written once) instead of one JSON object/// per row. For text-heavy `ocr`/`audio` results the `text` blob dominates@@ -223,6 +231,60 @@ pub struct SearchResponse {/// Metadata about cloud search availability (only present when cloud sync is available)#[serde(skip_serializing_if = "Option::is_none")]pub cloud: Option<crate::cloud_search::CloudSearchMetadata>,+ /// Tags that co-occur with the requested `tags`, grouped by namespace+ /// (`people`, `projects`, `workflows`, …) and ordered most-frequent+ /// first. Present only when `include_related=true` and a `tags` filter+ /// yielded co-occurring tags; omitted otherwise.+ #[serde(skip_serializing_if = "Option::is_none")]+ pub related: Option<std::collections::HashMap<String, Vec<String>>>,+}++/// How many co-occurring tags to pull for the `related` block. Spread across+/// a few namespaces this is plenty of context while staying token-cheap.+const RELATED_TAGS_LIMIT: u32 = 30;++/// Upper bound on the (auxiliary, opt-in) related-tags query. The memories leg+/// full-scans (no tag index) and a hot tag fans out to items×tags rows, so on a+/// pathological store this could run long; past this we drop the `related`+/// block rather than dragging out the whole search response. Measured cost on a+/// 200k-frame / 50k-memory DB is ~20ms (cold tag) to ~150ms (hot tag), so 5s is+/// a generous safety net, not a normal-path limit.+const RELATED_TAGS_TIMEOUT_SECS: u64 = 5;++/// Pluralize a tag namespace into the `related` map key. Mirrors the shape+/// callers expect (`person:` → `people`, `project:` → `projects`,+/// `workflow:` → `workflows`); unknown namespaces just get a trailing `s`.+fn related_namespace_key(ns: &str) -> String {+ match ns {+ "person" => "people".to_string(),+ "company" => "companies".to_string(),+ other => format!("{other}s"),+ }+}++/// Group flat co-occurring `(tag, count)` rows (already count-desc) into a+/// `namespace → values` map for the `related` block. `person:louis` lands as+/// `{"people": ["louis"]}`; a namespace-less tag lands under `"tags"` with its+/// full value. Values keep the most-frequent-first order from the query and+/// are de-duplicated within each bucket.+fn group_related_tags(+ rows: Vec<(String, i64)>,+) -> std::collections::HashMap<String, Vec<String>> {+ let mut grouped: std::collections::HashMap<String, Vec<String>> =+ std::collections::HashMap::new();+ for (name, _count) in rows {+ let (key, value) = match name.split_once(':') {+ Some((ns, val)) if !ns.is_empty() && !val.is_empty() => {+ (related_namespace_key(ns), val.to_string())+ }+ _ => ("tags".to_string(), name),+ };+ let bucket = grouped.entry(key).or_default();+ if !bucket.contains(&value) {+ bucket.push(value);+ }+ }+ grouped}/// Middle-truncate a string to at most `max_chars` characters.@@ -359,6 +421,7 @@ pub fn search_result_to_content_item(.and_then(|t| serde_json::from_str(t).ok()).unwrap_or_default(),importance: m.importance,+ frame_id: m.frame_id,created_at: m.created_at.clone(),updated_at: m.updated_at.clone(),}),@@ -395,6 +458,10 @@ pub(crate) fn compute_search_cache_key(query: &SearchQuery) -> u64 {// Tags change the result set materially — must be in the cache key so a// cached untagged response can't be served for a tag-filtered query.query.tags.hash(&mut hasher);+ // `include_related` adds the `related` block to the cached body, so a+ // request without it must not be served a related-bearing cache entry+ // (and vice-versa).+ query.include_related.hash(&mut hasher);hasher.finish()}@@ -731,6 +798,38 @@ pub(crate) async fn search(None};+ // Co-occurring tags ("related" context). Only meaningful when the caller+ // both opted in and supplied a tag filter to relate against. This is+ // auxiliary — neither an error nor a slow query may sink an otherwise-good+ // search, so we bound it with a timeout and degrade to `None` (logging)+ // rather than propagating the failure or blocking the response.+ let related = if query.include_related && !tags.is_empty() {+ match timeout(+ Duration::from_secs(RELATED_TAGS_TIMEOUT_SECS),+ state.db.related_tags(tags, RELATED_TAGS_LIMIT),+ )+ .await+ {+ Ok(Ok(rows)) => {+ let grouped = group_related_tags(rows);+ (!grouped.is_empty()).then_some(grouped)+ }+ Ok(Err(e)) => {+ warn!("related-tags query failed: {}", e);+ None+ }+ Err(_) => {+ warn!(+ "related-tags query exceeded {}s; omitting related block",+ RELATED_TAGS_TIMEOUT_SECS+ );+ None+ }+ }+ } else {+ None+ };+let response = SearchResponse {data: content_items,pagination: PaginationInfo {@@ -739,6 +838,7 @@ pub(crate) async fn search(total: total as i64,},cloud,+ related,};// Cache the result (only for queries without frame extraction)@@ -958,6 +1058,7 @@ mod tests {machine_id: None,filter_pii: false,tags: None,+ include_related: false,format: None,fields: None,};@@ -988,6 +1089,7 @@ mod tests {machine_id: None,filter_pii: false,tags: None,+ include_related: false,format: None,fields: None,};@@ -1026,6 +1128,7 @@ mod tests {machine_id: None,filter_pii: false,tags: None,+ include_related: false,format: None,fields: None,};@@ -1056,6 +1159,7 @@ mod tests {machine_id: None,filter_pii: false,tags: None,+ include_related: false,format: None,fields: None,};@@ -1101,6 +1205,7 @@ mod tests {machine_id: None,filter_pii: false,tags: None,+ include_related: false,format: None,fields: None,};@@ -1112,6 +1217,89 @@ mod tests {assert_ne!(yes, no, "Some(true) vs Some(false) must hash differently");}+ /// `include_related` must invalidate the cache: a body computed without the+ /// `related` block can't be served to a caller asking for it (or vice-versa).+ #[test]+ fn test_search_cache_key_distinguishes_include_related() {+ let mk = |include_related: bool| SearchQuery {+ q: Some("test".to_string()),+ pagination: PaginationQuery {+ limit: 10,+ offset: 0,+ },+ content_type: ContentType::All,+ start_time: None,+ end_time: None,+ app_name: None,+ window_name: None,+ frame_name: None,+ include_frames: false,+ min_length: None,+ max_length: None,+ speaker_ids: None,+ focused: None,+ on_screen: None,+ browser_url: None,+ speaker_name: None,+ include_cloud: false,+ max_content_length: None,+ device_name: None,+ machine_id: None,+ filter_pii: false,+ tags: Some(vec!["person:ada".to_string()]),+ include_related,+ format: None,+ fields: None,+ };+ assert_ne!(+ compute_search_cache_key(&mk(false)),+ compute_search_cache_key(&mk(true)),+ "include_related must change the cache key"+ );+ }++ #[test]+ fn test_related_namespace_key_pluralizes() {+ assert_eq!(related_namespace_key("person"), "people");+ assert_eq!(related_namespace_key("company"), "companies");+ assert_eq!(related_namespace_key("project"), "projects");+ assert_eq!(related_namespace_key("workflow"), "workflows");+ assert_eq!(related_namespace_key("app"), "apps");+ }++ #[test]+ fn test_group_related_tags_buckets_by_namespace() {+ // Count-desc input; grouping must preserve that order per bucket.+ let rows = vec![+ ("person:drew".to_string(), 9),+ ("project:screenpipe".to_string(), 7),+ ("person:connor".to_string(), 5),+ ("workflow:ai-coding".to_string(), 4),+ ("project:screenpipe-finance".to_string(), 3),+ ("standalone".to_string(), 2),+ ];+ let grouped = group_related_tags(rows);++ assert_eq!(grouped.get("people").unwrap(), &vec!["drew", "connor"]);+ assert_eq!(+ grouped.get("projects").unwrap(),+ &vec!["screenpipe", "screenpipe-finance"]+ );+ assert_eq!(grouped.get("workflows").unwrap(), &vec!["ai-coding"]);+ // Namespace-less tags fall under "tags" with their full value.+ assert_eq!(grouped.get("tags").unwrap(), &vec!["standalone"]);+ }++ #[test]+ fn test_group_related_tags_dedupes_within_bucket() {+ let rows = vec![+ ("person:ada".to_string(), 3),+ ("person:ada".to_string(), 1),+ ];+ let grouped = group_related_tags(rows);+ assert_eq!(grouped.get("people").unwrap(), &vec!["ada"]);+ }+#[test]fn test_truncate_middle_short_text() {assert_eq!(truncate_middle("hello", 10), "hello");diff --git a/docs/mintlify/docs-mintlify-mig-tmp/cli-reference.mdx b/docs/mintlify/docs-mintlify-mig-tmp/cli-reference.mdxindex 6833bc81c..ff7d4da1c 100644--- a/docs/mintlify/docs-mintlify-mig-tmp/cli-reference.mdx+++ b/docs/mintlify/docs-mintlify-mig-tmp/cli-reference.mdx@@ -62,6 +62,35 @@ curl "http://localhost:3030/search?q=meeting&limit=10&content_type=all"| `browser_url` | string | filter by browser URL || `min_length` | int | minimum text length || `max_length` | int | maximum text length |+| `tags` | string | comma-separated; return only items carrying **all** of these tags, e.g. `tags=person:ada,project:atlas` |+| `include_related` | bool | with `tags`, attach a `related` block of co-occurring tags grouped by namespace |++### related context++pass `include_related=true` alongside a `tags` filter to get the tags that+co-occur with the ones you asked for — the people, projects, and workflows that+show up in the same frames, calls, and memories — in a single call instead of+several follow-up queries:++```bash+curl "http://localhost:3030/search?tags=person:ada&include_related=true&limit=5"+```++```json+{+ "data": [ "...frames, audio, and memories..." ],+ "pagination": { "limit": 5, "offset": 0, "total": 42 },+ "related": {+ "people": ["connor", "drew"],+ "projects": ["atlas", "atlas-finance"],+ "workflows": ["planning"]+ }+}+```++namespaces are pluralized from the tag prefix (`person:` → `people`,+`project:` → `projects`); values are ordered most-frequent first. omit+`tags` and the block is skipped.### content type guidediff --git a/packages/screenpipe-mcp/src/index.ts b/packages/screenpipe-mcp/src/index.tsindex 83ce8696b..60a2a2866 100644--- a/packages/screenpipe-mcp/src/index.ts+++ b/packages/screenpipe-mcp/src/index.ts@@ -307,6 +307,12 @@ const TOOLS: Tool[] = [description:"Comma-separated tags; returns only items carrying ALL of them (e.g. 'person:ada,project:atlas'). Works for screen + audio (content_type 'ocr'/'audio'/'all', tags written by add-tags) AND memories (content_type 'memory', tags written by update-memory). Same tag string links across all three, so two items sharing a tag are connected. Use namespaced tags (person:, project:, topic:) to link people/projects/topics. content_type 'input' and 'accessibility' have no tags and return nothing when this is set.",},+ include_related: {+ type: "boolean",+ description:+ "With tags set, also return the co-occurring tags (the people/projects/topics seen alongside yours, ranked by frequency) as a 'Related:' line. One call for the surrounding context instead of several follow-ups. Ignored without tags.",+ default: false,+ },max_content_length: {type: "integer",description: "Truncate each result's text via middle-truncation. Use 200-500 to keep responses compact.",@@ -1383,8 +1389,11 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {const tagsStr = content.tags?.length ? ` [${content.tags.join(", ")}]` : "";const importance =content.importance != null ? ` (importance: ${content.importance})` : "";+ // frame_id links a memory back to the exact moment — jump there with+ // frame-context / get-frame-elements (frame_id=N).+ const frameRef = content.frame_id != null ? ` frame:${content.frame_id}` : "";formattedResults.push(- `[Memory #${content.id}]${tagsStr}${importance}\n` ++ `[Memory #${content.id}]${tagsStr}${importance}${frameRef}\n` +`${content.created_at || ""}\n` +`${truncateMiddle(content.content || "", effectiveCap)}`);@@ -1397,9 +1406,20 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {? ` (use offset=${(pagination.offset || 0) + results.length} for more)`: "");+ // Co-occurring tags (only present when include_related=true + tags set).+ // Compact one-liner per namespace so it's cheap to read.+ const related = data.related as Record<string, string[]> | undefined;+ const relatedStr =+ related && Object.keys(related).length > 0+ ? "\n\nRelated tags: " ++ Object.entries(related)+ .map(([ns, vals]) => `${ns}: ${(Array.isArray(vals) ? vals : []).join(", ")}`)+ .join(" | ")+ : "";+contentItems.push({type: "text",- text: header + "\n\n" + formattedResults.join("\n---\n"),+ text: header + "\n\n" + formattedResults.join("\n---\n") + relatedStr,});for (const img of images) {