Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

feat(search): include_related co-occurring tags + memory frame_id (#4429) (#4430)

Louis Beaumont66f580f237d3
diff --git a/.claude/skills/screenpipe-api/SKILL.md b/.claude/skills/screenpipe-api/SKILL.md
index f792ce663..aa2c3b099 100644
--- a/.claude/skills/screenpipe-api/SKILL.md
+++ b/.claude/skills/screenpipe-api/SKILL.md
@@ -46,6 +46,7 @@ curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030
| `speaker_name` | string | No | Filter audio by speaker (case-insensitive partial) |
| `focused` | boolean | No | Only focused windows |
| `tags` | string | No | Comma-separated; return only items carrying ALL of them (e.g. `person:ada,project:atlas`). Works for screen/audio and, with `content_type=memory`, memories. See Tags below. |
+| `include_related` | boolean | No | With `tags`, also return a `related` map of co-occurring tags (people/projects/workflows seen alongside yours), most-frequent first. One call for the surrounding context instead of several. See Tags below. |
| `max_content_length` | integer | No | Truncate each result's text (middle-truncation) |
| `format` | string | No | `json` (default), `csv`, `tsv`/`table`, or `outline`/`tree` (element endpoints only). CSV/TSV return a columnar table (column names written once) instead of one JSON object per row. `outline` returns a deduped indented text tree of the text-bearing UI nodes — the cheapest read for "what's on screen?" (~91% fewer tokens). CSV is lossless; TSV collapses newlines (worse for long `ocr` text). |
| `fields` | string | No | Comma-separated column allowlist of dotted paths, e.g. `type,content.app_name,content.text`. Returns only those columns (handy for dropping the repeated absolute `content.file_path`). Works for `json` too (sparse objects). |
@@ -78,7 +79,15 @@ Tags are a shared label layer across screen, audio, and memories under one strin
- Add to a memory: include `tags` in `POST /memories` (or `PUT /memories/{id}`).
- Retrieve by tag: `GET /search?tags=person:ada&start_time=30d%20ago` (screen+audio), or add `content_type=memory` for memories. Multiple tags AND together; matching is exact, not substring.
-Frames are pruned by retention, so for a durable link tag a memory (memories also carry `created_at` and a `frame_id` back to the moment). To pull everything about a person across time: one call for captures (`content_type=all&tags=person:ada`) plus one for facts (`content_type=memory&tags=person:ada`).
+Frames are pruned by retention, so for a durable link tag a memory (memories also carry `created_at` and a `frame_id` back to the moment — jump there with `GET /frames/{frame_id}`). To pull everything about a person across time: one call for captures (`content_type=all&tags=person:ada`) plus one for facts (`content_type=memory&tags=person:ada`).
+
+Add `include_related=true` to a tag query to get the surrounding context in the same response — the tags that co-occur with yours, grouped by namespace (prefix pluralized: `person:`→`people`, `project:`→`projects`) and ranked by frequency. Replaces the 2-3 follow-up "who/what else" calls with one:
+
+```bash
+curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \
+ "http://localhost:3030/search?tags=person:ada&include_related=true&limit=5"
+# data: [...], related: { "people": ["connor","drew"], "projects": ["atlas"], "workflows": ["planning"] }
+```
### Critical Rules
diff --git a/crates/screenpipe-core/assets/skills/screenpipe-api/SKILL.md b/crates/screenpipe-core/assets/skills/screenpipe-api/SKILL.md
index b5e55aec8..45ba103aa 100644
--- a/crates/screenpipe-core/assets/skills/screenpipe-api/SKILL.md
+++ b/crates/screenpipe-core/assets/skills/screenpipe-api/SKILL.md
@@ -64,6 +64,7 @@ curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030
| `speaker_name` | string | No | Filter audio by speaker (case-insensitive partial) |
| `focused` | boolean | No | Only focused windows |
| `tags` | string | No | Comma-separated; return only items carrying ALL of them (e.g. `person:ada,project:atlas`). Works for screen/audio and, with `content_type=memory`, memories. See Tags below. |
+| `include_related` | boolean | No | With `tags`, also return a `related` map of co-occurring tags (people/projects/workflows seen alongside yours), most-frequent first. One call for the surrounding context instead of several. See Tags below. |
| `max_content_length` | integer | No | Truncate each result's text (middle-truncation) |
| `format` | string | No | `json` (default), `csv`, or `tsv`/`table`. CSV/TSV return a columnar table (column names written once) instead of one JSON object per row, much cheaper to read on list-shaped results. CSV is lossless; TSV collapses newlines (worse for long `ocr` text). |
| `fields` | string | No | Comma-separated column allowlist of dotted paths, e.g. `type,content.app_name,content.text`. Returns only those columns (handy for dropping the repeated absolute `content.file_path`). Works for `json` too (sparse objects). |
@@ -76,7 +77,15 @@ Tags are a shared label layer across screen, audio, and memories under one strin
- Add to a memory: include `tags` in `POST /memories` (or `PUT /memories/{id}`).
- Retrieve by tag: `GET /search?tags=person:ada&start_time=30d%20ago` (screen+audio), or add `content_type=memory` for memories. Multiple tags AND together; matching is exact, not substring.
-Frames are pruned by retention, so for a durable link tag a memory (memories also carry `created_at` and a `frame_id` back to the moment). Tag frames/audio for short-term recall. To pull everything about a person across time: one call for captures (`content_type=all&tags=person:ada`) plus one for facts (`content_type=memory&tags=person:ada`).
+Frames are pruned by retention, so for a durable link tag a memory (memories also carry `created_at` and a `frame_id` back to the moment — jump there with `GET /frames/{frame_id}`). Tag frames/audio for short-term recall. To pull everything about a person across time: one call for captures (`content_type=all&tags=person:ada`) plus one for facts (`content_type=memory&tags=person:ada`).
+
+Add `include_related=true` to a tag query to get the surrounding context in the same response — the tags that co-occur with yours, grouped by namespace (prefix pluralized: `person:`→`people`, `project:`→`projects`) and ranked by frequency. Replaces the 2-3 follow-up "who/what else" calls with one:
+
+```bash
+curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \
+ "http://localhost:3030/search?tags=person:ada&include_related=true&limit=5"
+# data: [...], related: { "people": ["connor","drew"], "projects": ["atlas"], "workflows": ["planning"] }
+```
### Critical Rules
@@ -500,7 +509,7 @@ curl -X DELETE http://localhost:3030/memories/1
Parameters for `GET /memories`: `q` (FTS search), `source`, `tags`, `min_importance`, `start_time`, `end_time`, `limit`, `offset`.
-Use `/memories` directly — `/search` does not currently surface `content_type=memory` results.
+`GET /memories` is the direct read (richest filters: `min_importance`, `source`). Memories also come back via `GET /search?content_type=memory` (note: `content_type=all` does NOT include them — ask for `memory` explicitly), which is the path that supports `tags` + `include_related` for tag-linked recall.
### Creating Memories
diff --git a/crates/screenpipe-db/src/db/search.rs b/crates/screenpipe-db/src/db/search.rs
index b062f91aa..336d904de 100644
--- a/crates/screenpipe-db/src/db/search.rs
+++ b/crates/screenpipe-db/src/db/search.rs
@@ -1748,4 +1748,102 @@ impl DatabaseManager {
Ok((latest_frame.map(|f| f.0), latest_audio.map(|a| a.0), None))
}
+
+ /// Tags that co-occur with ALL of the given `tags`, most-frequent first,
+ /// excluding the input tags themselves. Spans the same three stores as the
+ /// tag filter on [`search_with_tags`](Self::search_with_tags): the screen
+ /// (`vision_tags`) and audio (`audio_tags`) junction tables plus the
+ /// `memories.tags` JSON array.
+ ///
+ /// Powers `GET /search?...&include_related=true`: one query surfaces the
+ /// people / projects / workflows that appear alongside a tag so an AI
+ /// caller gets the surrounding context without N follow-up requests.
+ /// Returns each co-occurring tag's full namespaced name and its count.
+ /// An empty `tags` slice returns an empty vec; duplicate inputs are folded
+ /// (the `DISTINCT` in the `input` CTE) so they match like a single tag.
+ ///
+ /// Cost: the vision/audio legs ride the tag indexes (`idx_*_tags_tag_id` +
+ /// `tags.name`), but the memories leg full-scans + `json_each` because
+ /// `memories.tags` is an unindexed JSON column — the same linear cost the
+ /// tag *filter* already pays (see `tests/tag_filter_bench.rs`). Measured on
+ /// a 200k-frame / 250k-vision_tag / 50k-memory in-memory DB: ~21 ms for a
+ /// realistic tag, ~150 ms worst-case for a hot tag on 50k items with wide
+ /// fan-out. The HTTP handler bounds it with a timeout and treats it as
+ /// optional. If memory counts ever reach millions, give them a
+ /// `memory_tags` junction table mirroring `vision_tags`.
+ pub async fn related_tags(
+ &self,
+ tags: &[String],
+ limit: u32,
+ ) -> Result<Vec<(String, i64)>, sqlx::Error> {
+ if tags.is_empty() {
+ return Ok(Vec::new());
+ }
+
+ // Same JSON-array binding trick as the tag filter: pass the tags as a
+ // JSON array and expand with `json_each`. `n.c` is the input cardinality
+ // so the `HAVING` clauses keep only items carrying ALL requested tags.
+ // The `memories.tags` reads wrap the column in `CASE WHEN json_valid`
+ // because a single legacy/sync row that isn't valid JSON would
+ // otherwise make `json_each` raise "malformed JSON" and 500 the whole
+ // query (same guard as `list_memory_tags`).
+ let tags_json = serde_json::to_string(tags).unwrap_or_else(|_| "[]".to_string());
+
+ let rows: Vec<(String, i64)> = sqlx::query_as(
+ r#"
+ WITH input(name) AS (SELECT DISTINCT value FROM json_each(?1)),
+ n(c) AS (SELECT COUNT(*) FROM input),
+ vision_matches(id) AS (
+ SELECT vt.vision_id
+ FROM vision_tags vt JOIN tags t ON vt.tag_id = t.id
+ WHERE t.name IN (SELECT name FROM input)
+ GROUP BY vt.vision_id
+ HAVING COUNT(DISTINCT t.name) = (SELECT c FROM n)
+ ),
+ audio_matches(id) AS (
+ SELECT aud.audio_chunk_id
+ FROM audio_tags aud JOIN tags t ON aud.tag_id = t.id
+ WHERE t.name IN (SELECT name FROM input)
+ GROUP BY aud.audio_chunk_id
+ HAVING COUNT(DISTINCT t.name) = (SELECT c FROM n)
+ ),
+ memory_matches(id) AS (
+ SELECT m.id
+ FROM memories m
+ WHERE (
+ SELECT COUNT(DISTINCT j.value)
+ FROM json_each(CASE WHEN json_valid(m.tags) THEN m.tags ELSE '[]' END) j
+ WHERE j.value IN (SELECT name FROM input)
+ ) = (SELECT c FROM n)
+ ),
+ co(name) AS (
+ SELECT t.name
+ FROM vision_tags vt JOIN tags t ON vt.tag_id = t.id
+ WHERE vt.vision_id IN (SELECT id FROM vision_matches)
+ UNION ALL
+ SELECT t.name
+ FROM audio_tags aud JOIN tags t ON aud.tag_id = t.id
+ WHERE aud.audio_chunk_id IN (SELECT id FROM audio_matches)
+ UNION ALL
+ SELECT j.value
+ FROM memories m,
+ json_each(CASE WHEN json_valid(m.tags) THEN m.tags ELSE '[]' END) j
+ WHERE m.id IN (SELECT id FROM memory_matches)
+ )
+ SELECT name, COUNT(*) AS count
+ FROM co
+ WHERE name IS NOT NULL AND name != ''
+ AND name NOT IN (SELECT name FROM input)
+ GROUP BY name
+ ORDER BY count DESC, name ASC
+ LIMIT ?2
+ "#,
+ )
+ .bind(&tags_json)
+ .bind(limit as i64)
+ .fetch_all(&self.pool)
+ .await?;
+
+ Ok(rows)
+ }
}
diff --git a/crates/screenpipe-db/tests/db.rs b/crates/screenpipe-db/tests/db.rs
index 3c8ae5efe..a23159b3d 100644
--- a/crates/screenpipe-db/tests/db.rs
+++ b/crates/screenpipe-db/tests/db.rs
@@ -484,6 +484,245 @@ mod tests {
assert_eq!(n, 2);
}
+ /// `related_tags` returns the tags that co-occur with the requested ones,
+ /// counted across all three stores (vision frames, audio chunks, memory
+ /// JSON), most-frequent first, with the inputs themselves excluded and
+ /// AND-semantics on multiple inputs.
+ #[tokio::test]
+ async fn test_related_tags_co_occurrence() {
+ let db = setup_test_db().await;
+ db.insert_video_chunk("v.mp4", "dev").await.unwrap();
+
+ // Frame carrying person:ada alongside a project and a workflow.
+ let f_a = db
+ .insert_frame("dev", None, None, Some("app"), Some(""), false, None)
+ .await
+ .unwrap();
+ db.add_tags(
+ f_a,
+ TagContentType::Vision,
+ vec![
+ "person:ada".to_string(),
+ "project:atlas".to_string(),
+ "workflow:planning".to_string(),
+ ],
+ )
+ .await
+ .unwrap();
+
+ // Frame carrying person:ada with a second person and the same project.
+ let f_b = db
+ .insert_frame("dev", None, None, Some("app"), Some(""), false, None)
+ .await
+ .unwrap();
+ db.add_tags(
+ f_b,
+ TagContentType::Vision,
+ vec![
+ "person:ada".to_string(),
+ "person:connor".to_string(),
+ "project:atlas".to_string(),
+ ],
+ )
+ .await
+ .unwrap();
+
+ // Unrelated frame — its tag must never surface for person:ada.
+ let f_x = db
+ .insert_frame("dev", None, None, Some("app"), Some(""), false, None)
+ .await
+ .unwrap();
+ db.add_tags(f_x, TagContentType::Vision, vec!["person:bob".to_string()])
+ .await
+ .unwrap();
+
+ // Audio chunk carrying person:ada + the same project (third hit).
+ let ac = db.insert_audio_chunk("a.mp4", None).await.unwrap();
+ db.add_tags(
+ ac,
+ TagContentType::Audio,
+ vec!["person:ada".to_string(), "project:atlas".to_string()],
+ )
+ .await
+ .unwrap();
+
+ // Memory carrying person:ada alongside a different person.
+ db.insert_memory(
+ "ada + drew planning",
+ "user",
+ None,
+ Some(r#"["person:ada","person:drew"]"#),
+ 0.5,
+ None,
+ )
+ .await
+ .unwrap();
+
+ // Single input tag: project:atlas co-occurs 3× (f_a, f_b, ac); the
+ // three singletons tie and break by name ascending. person:ada (input)
+ // and person:bob (unrelated frame) must be absent.
+ let related = db
+ .related_tags(&["person:ada".to_string()], 50)
+ .await
+ .unwrap();
+ let counts: std::collections::HashMap<&str, i64> = related
+ .iter()
+ .map(|(n, c)| (n.as_str(), *c))
+ .collect();
+ assert_eq!(related.len(), 4, "got {related:?}");
+ assert_eq!(related[0], ("project:atlas".to_string(), 3));
+ assert_eq!(counts.get("project:atlas"), Some(&3));
+ assert_eq!(counts.get("workflow:planning"), Some(&1));
+ assert_eq!(counts.get("person:connor"), Some(&1));
+ assert_eq!(counts.get("person:drew"), Some(&1));
+ assert!(!counts.contains_key("person:ada"), "input tag leaked");
+ assert!(!counts.contains_key("person:bob"), "unrelated tag leaked");
+
+ // The `limit` truncates to the top-N by count.
+ let top1 = db.related_tags(&["person:ada".to_string()], 1).await.unwrap();
+ assert_eq!(top1, vec![("project:atlas".to_string(), 3)]);
+
+ // Multiple inputs → AND: only items carrying BOTH person:ada AND
+ // project:atlas (f_a, f_b, ac — not the memory, which lacks the
+ // project). Co-occurring extras: workflow:planning and person:connor.
+ let both = db
+ .related_tags(
+ &["person:ada".to_string(), "project:atlas".to_string()],
+ 50,
+ )
+ .await
+ .unwrap();
+ let both_names: std::collections::HashSet<&str> =
+ both.iter().map(|(n, _)| n.as_str()).collect();
+ assert_eq!(both.len(), 2, "got {both:?}");
+ assert!(both_names.contains("workflow:planning"));
+ assert!(both_names.contains("person:connor"));
+ assert!(!both_names.contains("person:ada"));
+ assert!(!both_names.contains("project:atlas"));
+
+ // Empty input → empty result (no tags to relate against).
+ assert!(db.related_tags(&[], 50).await.unwrap().is_empty());
+ }
+
+ /// Adversarial inputs for `related_tags`: malformed memory JSON, colon-rich
+ /// tag values, duplicate inputs, store isolation, limit edges, unicode, and
+ /// quote/`%`/`_` injection-shaped strings. None may error or leak.
+ #[tokio::test]
+ async fn test_related_tags_edge_cases() {
+ let db = setup_test_db().await;
+ db.insert_video_chunk("v.mp4", "dev").await.unwrap();
+
+ // A frame carrying the anchor + a value that itself contains colons
+ // (e.g. a URL tag) — the split into a namespace happens in the route,
+ // so the DB must return the full name verbatim.
+ let f = db
+ .insert_frame("dev", None, None, Some("app"), Some(""), false, None)
+ .await
+ .unwrap();
+ db.add_tags(
+ f,
+ TagContentType::Vision,
+ vec![
+ "person:ada".to_string(),
+ "url:https://example.com:8080/x".to_string(),
+ "emoji:🦀".to_string(),
+ ],
+ )
+ .await
+ .unwrap();
+
+ // A memory carrying the anchor + a SQL-injection-shaped value. Bound as
+ // a JSON param, so it's inert; it must come back as data, not break out.
+ db.insert_memory(
+ "weird tags",
+ "user",
+ None,
+ Some(r#"["person:ada","weird:a' OR 1=1 --","like:50%_x"]"#),
+ 0.5,
+ None,
+ )
+ .await
+ .unwrap();
+
+ // A memory whose `tags` column is NOT valid JSON. The `json_valid`
+ // guard must skip it instead of letting `json_each` raise and 500.
+ db.insert_memory("legacy", "user", None, Some("not valid json"), 0.5, None)
+ .await
+ .unwrap();
+ // ...and one carrying the anchor twice (deliberately) — the value must
+ // still be counted once per memory, not double.
+ db.insert_memory(
+ "dupe-in-row",
+ "user",
+ None,
+ Some(r#"["person:ada","person:ada","only:here"]"#),
+ 0.5,
+ None,
+ )
+ .await
+ .unwrap();
+
+ let related = db
+ .related_tags(&["person:ada".to_string()], 50)
+ .await
+ .unwrap();
+ let names: std::collections::HashSet<&str> =
+ related.iter().map(|(n, _)| n.as_str()).collect();
+ // Colon-rich and unicode values survive intact.
+ assert!(names.contains("url:https://example.com:8080/x"));
+ assert!(names.contains("emoji:🦀"));
+ // Injection-shaped strings come back as plain data.
+ assert!(names.contains("weird:a' OR 1=1 --"));
+ assert!(names.contains("like:50%_x"));
+ assert!(names.contains("only:here"));
+ // The anchor itself is never echoed back.
+ assert!(!names.contains("person:ada"));
+
+ // Duplicate input tags must behave like a single input (the DISTINCT in
+ // the input CTE), NOT silently match nothing.
+ let deduped = db
+ .related_tags(
+ &["person:ada".to_string(), "person:ada".to_string()],
+ 50,
+ )
+ .await
+ .unwrap();
+ let deduped_names: std::collections::HashSet<&str> =
+ deduped.iter().map(|(n, _)| n.as_str()).collect();
+ assert_eq!(deduped_names, names, "duplicate inputs must equal single");
+
+ // limit = 0 → no rows (not an error).
+ assert!(db
+ .related_tags(&["person:ada".to_string()], 0)
+ .await
+ .unwrap()
+ .is_empty());
+
+ // A tag that exists on nothing → empty, no error.
+ assert!(db
+ .related_tags(&["person:ghost".to_string()], 50)
+ .await
+ .unwrap()
+ .is_empty());
+
+ // Store isolation: an anchor present ONLY in audio still finds its
+ // audio-side co-tags and nothing from unrelated frames/memories.
+ let ac = db.insert_audio_chunk("a.mp4", None).await.unwrap();
+ db.add_tags(
+ ac,
+ TagContentType::Audio,
+ vec!["call:standup".to_string(), "person:bob".to_string()],
+ )
+ .await
+ .unwrap();
+ let audio_only = db
+ .related_tags(&["call:standup".to_string()], 50)
+ .await
+ .unwrap();
+ assert_eq!(audio_only.len(), 1);
+ assert_eq!(audio_only[0], ("person:bob".to_string(), 1));
+ }
+
#[tokio::test]
async fn test_recent_output_audio_detects_deferred_output_chunk() {
let db = setup_test_db().await;
diff --git a/crates/screenpipe-db/tests/tag_filter_bench.rs b/crates/screenpipe-db/tests/tag_filter_bench.rs
index ed89840f7..77e3b3540 100644
--- a/crates/screenpipe-db/tests/tag_filter_bench.rs
+++ b/crates/screenpipe-db/tests/tag_filter_bench.rs
@@ -314,3 +314,228 @@ async fn bench_tag_filter_scaling() {
);
}
}
+
+/// Scaling check for `related_tags` (powers `?include_related=true`).
+///
+/// Ignored by default. Run explicitly:
+/// cargo test -p screenpipe-db --test tag_filter_bench related_tags -- --ignored --nocapture
+///
+/// `related_tags` is the inverse-shaped sibling of the tag filter: instead of
+/// "items carrying this tag", it returns "every OTHER tag on those items,
+/// counted". Two cost drivers to pin down:
+/// 1. a HOT input tag — on many items, each carrying many other tags — makes
+/// the `co` CTE materialize (items × tags-per-item) rows before GROUP BY.
+/// 2. memories have no tag index, so the memory leg full-scans + json_each
+/// (same linear cost the tag-filter bench already documents).
+/// This seed maximizes (1): `person:hot` is on 50k frames that EACH carry 4
+/// extra co-tags (so 50k×5 = 250k rows feed the aggregation) plus a 50k-memory
+/// store for (2). `person:cold` (200 items) is the realistic case.
+#[tokio::test]
+#[ignore]
+async fn bench_related_tags_scaling() {
+ const N_FRAMES_R: i64 = 200_000;
+ const HOT_FRAMES: i64 = 50_000; // person:hot is on the first 50k frames
+ const CO_POOL: i64 = 50; // each hot frame carries 4 of these → wide fan-out
+ const COLD_FRAMES: i64 = 200; // realistic tag
+ const N_MEM_R: i64 = 50_000;
+ const HOT_MEM: i64 = 50_000; // every memory carries person:hot + a co tag
+
+ let db = migrated_db().await;
+ let seed = Instant::now();
+
+ exec(
+ &db,
+ "INSERT INTO video_chunks (file_path, device_name) VALUES ('v.mp4','dev')",
+ )
+ .await;
+ let chunk_id: i64 = sqlx::query_scalar("SELECT id FROM video_chunks LIMIT 1")
+ .fetch_one(&db.pool)
+ .await
+ .unwrap();
+ exec(
+ &db,
+ &format!(
+ "INSERT INTO frames (video_chunk_id, offset_index, timestamp, name, app_name, window_name, focused, device_name) \
+ WITH RECURSIVE seq(i) AS (SELECT 0 UNION ALL SELECT i+1 FROM seq WHERE i < {n}-1) \
+ SELECT {chunk}, i, datetime('2026-01-01 00:00:00', '+'||i||' seconds'), 'f'||i, 'app', 'win', 0, 'dev' FROM seq",
+ n = N_FRAMES_R,
+ chunk = chunk_id
+ ),
+ )
+ .await;
+
+ // Tags: person:hot (1), person:cold (2), then CO_POOL co:N tags (3..).
+ exec(
+ &db,
+ "INSERT INTO tags (name) VALUES ('person:hot'), ('person:cold')",
+ )
+ .await;
+ exec(
+ &db,
+ &format!(
+ "INSERT INTO tags (name) WITH RECURSIVE s(i) AS (SELECT 0 UNION ALL SELECT i+1 FROM s WHERE i<{p}-1) SELECT 'co:'||i FROM s",
+ p = CO_POOL
+ ),
+ )
+ .await;
+ let hot: i64 = sqlx::query_scalar("SELECT id FROM tags WHERE name='person:hot'")
+ .fetch_one(&db.pool)
+ .await
+ .unwrap();
+ let cold: i64 = sqlx::query_scalar("SELECT id FROM tags WHERE name='person:cold'")
+ .fetch_one(&db.pool)
+ .await
+ .unwrap();
+ let co_base: i64 = sqlx::query_scalar("SELECT id FROM tags WHERE name='co:0'")
+ .fetch_one(&db.pool)
+ .await
+ .unwrap();
+
+ // person:hot on the first HOT_FRAMES frames.
+ exec(
+ &db,
+ &format!(
+ "INSERT INTO vision_tags (vision_id, tag_id) \
+ WITH RECURSIVE s(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<{k}) SELECT i, {hot} FROM s",
+ k = HOT_FRAMES,
+ hot = hot
+ ),
+ )
+ .await;
+ // Each hot frame also carries 4 co-tags → wide co-occurrence fan-out.
+ for off in 0..4i64 {
+ exec(
+ &db,
+ &format!(
+ "INSERT OR IGNORE INTO vision_tags (vision_id, tag_id) \
+ WITH RECURSIVE s(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<{k}) \
+ SELECT i, {base} + ((i + {off}) % {pool}) FROM s",
+ k = HOT_FRAMES,
+ base = co_base,
+ off = off,
+ pool = CO_POOL
+ ),
+ )
+ .await;
+ }
+ // person:cold on the first COLD_FRAMES frames (they already carry co-tags).
+ exec(
+ &db,
+ &format!(
+ "INSERT INTO vision_tags (vision_id, tag_id) \
+ WITH RECURSIVE s(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM s WHERE i<{k}) SELECT i, {cold} FROM s",
+ k = COLD_FRAMES,
+ cold = cold
+ ),
+ )
+ .await;
+
+ // Memories: every one carries person:hot + a rotating co:N (full-scan leg).
+ exec(
+ &db,
+ &format!(
+ "INSERT INTO memories (content, tags, importance) \
+ WITH RECURSIVE s(i) AS (SELECT 0 UNION ALL SELECT i+1 FROM s WHERE i<{n}-1) \
+ SELECT 'mem '||i, \
+ CASE WHEN i<{k} THEN '[\"person:hot\",\"co:'||(i%{pool})||'\"]' ELSE '[\"noise:'||(i%500)||'\"]' END, \
+ 0.5 FROM s",
+ n = N_MEM_R,
+ k = HOT_MEM,
+ pool = CO_POOL
+ ),
+ )
+ .await;
+
+ let counts: (i64, i64, i64) = sqlx::query_as(
+ "SELECT (SELECT COUNT(*) FROM frames), (SELECT COUNT(*) FROM vision_tags), (SELECT COUNT(*) FROM memories)",
+ )
+ .fetch_one(&db.pool)
+ .await
+ .unwrap();
+ println!(
+ "seeded in {:?}: frames={} vision_tags={} memories={}",
+ seed.elapsed(),
+ counts.0,
+ counts.1,
+ counts.2
+ );
+
+ // ---- query plan: confirm the vision/audio legs ride the tag indexes and
+ // only the memories leg scans (the documented, accepted linear cost). ----
+ let related_sql = r#"
+ WITH input(name) AS (SELECT DISTINCT value FROM json_each(?1)),
+ n(c) AS (SELECT COUNT(*) FROM input),
+ vision_matches(id) AS (
+ SELECT vt.vision_id FROM vision_tags vt JOIN tags t ON vt.tag_id = t.id
+ WHERE t.name IN (SELECT name FROM input)
+ GROUP BY vt.vision_id HAVING COUNT(DISTINCT t.name) = (SELECT c FROM n)
+ ),
+ audio_matches(id) AS (
+ SELECT aud.audio_chunk_id FROM audio_tags aud JOIN tags t ON aud.tag_id = t.id
+ WHERE t.name IN (SELECT name FROM input)
+ GROUP BY aud.audio_chunk_id HAVING COUNT(DISTINCT t.name) = (SELECT c FROM n)
+ ),
+ memory_matches(id) AS (
+ SELECT m.id FROM memories m
+ WHERE (SELECT COUNT(DISTINCT j.value)
+ FROM json_each(CASE WHEN json_valid(m.tags) THEN m.tags ELSE '[]' END) j
+ WHERE j.value IN (SELECT name FROM input)) = (SELECT c FROM n)
+ ),
+ co(name) AS (
+ SELECT t.name FROM vision_tags vt JOIN tags t ON vt.tag_id = t.id
+ WHERE vt.vision_id IN (SELECT id FROM vision_matches)
+ UNION ALL
+ SELECT t.name FROM audio_tags aud JOIN tags t ON aud.tag_id = t.id
+ WHERE aud.audio_chunk_id IN (SELECT id FROM audio_matches)
+ UNION ALL
+ SELECT j.value FROM memories m,
+ json_each(CASE WHEN json_valid(m.tags) THEN m.tags ELSE '[]' END) j
+ WHERE m.id IN (SELECT id FROM memory_matches)
+ )
+ SELECT name, COUNT(*) AS count FROM co
+ WHERE name IS NOT NULL AND name != '' AND name NOT IN (SELECT name FROM input)
+ GROUP BY name ORDER BY count DESC, name ASC LIMIT ?2"#;
+ let plan: Vec<(i64, i64, i64, String)> =
+ sqlx::query_as(&format!("EXPLAIN QUERY PLAN {related_sql}"))
+ .bind("[\"person:hot\"]")
+ .bind(30i64)
+ .fetch_all(&db.pool)
+ .await
+ .unwrap();
+ println!("\n--- PLAN: related_tags(person:hot) ---");
+ for (_, _, _, detail) in plan {
+ println!(" {detail}");
+ }
+
+ // ---- timings (best of 3 after warm-up) ----
+ let bench = |label: &'static str, tags: Vec<String>| {
+ let db = &db;
+ async move {
+ let _ = db.related_tags(&tags, 30).await.unwrap(); // warm-up
+ let mut best = std::time::Duration::MAX;
+ let mut rows = 0usize;
+ let mut top = String::new();
+ for _ in 0..3 {
+ let t = Instant::now();
+ let r = db.related_tags(&tags, 30).await.unwrap();
+ best = best.min(t.elapsed());
+ rows = r.len();
+ top = r
+ .first()
+ .map(|(n, c)| format!("{n}={c}"))
+ .unwrap_or_default();
+ }
+ println!("related {label:<40} -> {rows:>3} groups (top {top}), best {best:?}");
+ }
+ };
+
+ println!("\n--- timings ---");
+ bench("person:hot (50k items, wide fan-out)", vec!["person:hot".to_string()]).await;
+ bench("person:cold (200 items, realistic)", vec!["person:cold".to_string()]).await;
+ bench(
+ "person:hot+co:0 (AND, two inputs)",
+ vec!["person:hot".to_string(), "co:0".to_string()],
+ )
+ .await;
+ bench("nonexistent:tag (no matches)", vec!["nonexistent:tag".to_string()]).await;
+}
diff --git a/crates/screenpipe-engine/src/routes/content.rs b/crates/screenpipe-engine/src/routes/content.rs
index 0025b6e9a..bbcc45a40 100644
--- a/crates/screenpipe-engine/src/routes/content.rs
+++ b/crates/screenpipe-engine/src/routes/content.rs
@@ -50,6 +50,10 @@ pub struct MemoryContent {
pub source_context: Option<serde_json::Value>,
pub tags: Vec<String>,
pub importance: f64,
+ /// First frame of the workflow segment this memory describes, when known.
+ /// Lets a caller jump straight from a memory to the exact captured moment
+ /// (`GET /frames/{frame_id}`). Null for memories with no frame provenance.
+ pub frame_id: Option<i64>,
pub created_at: String,
pub updated_at: String,
}
diff --git a/crates/screenpipe-engine/src/routes/search.rs b/crates/screenpipe-engine/src/routes/search.rs
index d6709d339..a4840e947 100644
--- a/crates/screenpipe-engine/src/routes/search.rs
+++ b/crates/screenpipe-engine/src/routes/search.rs
@@ -146,6 +146,14 @@ pub(crate) struct SearchQuery {
/// nothing when this is set. Omit for no tag filtering.
#[serde(default, deserialize_with = "from_comma_separated_string_array")]
tags: Option<Vec<String>>,
+ /// When `true` and a `tags` filter is set, attach a `related` block to the
+ /// response: the tags that co-occur with the requested ones, grouped by
+ /// namespace (`people`, `projects`, `workflows`, …) and ordered
+ /// most-frequent first. Lets an AI caller pull the surrounding context
+ /// (who/what/which-workflow showed up alongside a tag) in one request
+ /// instead of several follow-up queries. No-op without `tags`.
+ #[serde(default, deserialize_with = "deserialize_flexible_bool")]
+ include_related: bool,
/// Output format: `json` (default), `csv`, or `tsv`/`table`. CSV/TSV emit a
/// columnar table (column names written once) instead of one JSON object
/// per row. For text-heavy `ocr`/`audio` results the `text` blob dominates
@@ -223,6 +231,60 @@ pub struct SearchResponse {
/// Metadata about cloud search availability (only present when cloud sync is available)
#[serde(skip_serializing_if = "Option::is_none")]
pub cloud: Option<crate::cloud_search::CloudSearchMetadata>,
+ /// Tags that co-occur with the requested `tags`, grouped by namespace
+ /// (`people`, `projects`, `workflows`, …) and ordered most-frequent
+ /// first. Present only when `include_related=true` and a `tags` filter
+ /// yielded co-occurring tags; omitted otherwise.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub related: Option<std::collections::HashMap<String, Vec<String>>>,
+}
+
+/// How many co-occurring tags to pull for the `related` block. Spread across
+/// a few namespaces this is plenty of context while staying token-cheap.
+const RELATED_TAGS_LIMIT: u32 = 30;
+
+/// Upper bound on the (auxiliary, opt-in) related-tags query. The memories leg
+/// full-scans (no tag index) and a hot tag fans out to items×tags rows, so on a
+/// pathological store this could run long; past this we drop the `related`
+/// block rather than dragging out the whole search response. Measured cost on a
+/// 200k-frame / 50k-memory DB is ~20ms (cold tag) to ~150ms (hot tag), so 5s is
+/// a generous safety net, not a normal-path limit.
+const RELATED_TAGS_TIMEOUT_SECS: u64 = 5;
+
+/// Pluralize a tag namespace into the `related` map key. Mirrors the shape
+/// callers expect (`person:` → `people`, `project:` → `projects`,
+/// `workflow:` → `workflows`); unknown namespaces just get a trailing `s`.
+fn related_namespace_key(ns: &str) -> String {
+ match ns {
+ "person" => "people".to_string(),
+ "company" => "companies".to_string(),
+ other => format!("{other}s"),
+ }
+}
+
+/// Group flat co-occurring `(tag, count)` rows (already count-desc) into a
+/// `namespace → values` map for the `related` block. `person:louis` lands as
+/// `{"people": ["louis"]}`; a namespace-less tag lands under `"tags"` with its
+/// full value. Values keep the most-frequent-first order from the query and
+/// are de-duplicated within each bucket.
+fn group_related_tags(
+ rows: Vec<(String, i64)>,
+) -> std::collections::HashMap<String, Vec<String>> {
+ let mut grouped: std::collections::HashMap<String, Vec<String>> =
+ std::collections::HashMap::new();
+ for (name, _count) in rows {
+ let (key, value) = match name.split_once(':') {
+ Some((ns, val)) if !ns.is_empty() && !val.is_empty() => {
+ (related_namespace_key(ns), val.to_string())
+ }
+ _ => ("tags".to_string(), name),
+ };
+ let bucket = grouped.entry(key).or_default();
+ if !bucket.contains(&value) {
+ bucket.push(value);
+ }
+ }
+ grouped
}
/// Middle-truncate a string to at most `max_chars` characters.
@@ -359,6 +421,7 @@ pub fn search_result_to_content_item(
.and_then(|t| serde_json::from_str(t).ok())
.unwrap_or_default(),
importance: m.importance,
+ frame_id: m.frame_id,
created_at: m.created_at.clone(),
updated_at: m.updated_at.clone(),
}),
@@ -395,6 +458,10 @@ pub(crate) fn compute_search_cache_key(query: &SearchQuery) -> u64 {
// Tags change the result set materially — must be in the cache key so a
// cached untagged response can't be served for a tag-filtered query.
query.tags.hash(&mut hasher);
+ // `include_related` adds the `related` block to the cached body, so a
+ // request without it must not be served a related-bearing cache entry
+ // (and vice-versa).
+ query.include_related.hash(&mut hasher);
hasher.finish()
}
@@ -731,6 +798,38 @@ pub(crate) async fn search(
None
};
+ // Co-occurring tags ("related" context). Only meaningful when the caller
+ // both opted in and supplied a tag filter to relate against. This is
+ // auxiliary — neither an error nor a slow query may sink an otherwise-good
+ // search, so we bound it with a timeout and degrade to `None` (logging)
+ // rather than propagating the failure or blocking the response.
+ let related = if query.include_related && !tags.is_empty() {
+ match timeout(
+ Duration::from_secs(RELATED_TAGS_TIMEOUT_SECS),
+ state.db.related_tags(tags, RELATED_TAGS_LIMIT),
+ )
+ .await
+ {
+ Ok(Ok(rows)) => {
+ let grouped = group_related_tags(rows);
+ (!grouped.is_empty()).then_some(grouped)
+ }
+ Ok(Err(e)) => {
+ warn!("related-tags query failed: {}", e);
+ None
+ }
+ Err(_) => {
+ warn!(
+ "related-tags query exceeded {}s; omitting related block",
+ RELATED_TAGS_TIMEOUT_SECS
+ );
+ None
+ }
+ }
+ } else {
+ None
+ };
+
let response = SearchResponse {
data: content_items,
pagination: PaginationInfo {
@@ -739,6 +838,7 @@ pub(crate) async fn search(
total: total as i64,
},
cloud,
+ related,
};
// Cache the result (only for queries without frame extraction)
@@ -958,6 +1058,7 @@ mod tests {
machine_id: None,
filter_pii: false,
tags: None,
+ include_related: false,
format: None,
fields: None,
};
@@ -988,6 +1089,7 @@ mod tests {
machine_id: None,
filter_pii: false,
tags: None,
+ include_related: false,
format: None,
fields: None,
};
@@ -1026,6 +1128,7 @@ mod tests {
machine_id: None,
filter_pii: false,
tags: None,
+ include_related: false,
format: None,
fields: None,
};
@@ -1056,6 +1159,7 @@ mod tests {
machine_id: None,
filter_pii: false,
tags: None,
+ include_related: false,
format: None,
fields: None,
};
@@ -1101,6 +1205,7 @@ mod tests {
machine_id: None,
filter_pii: false,
tags: None,
+ include_related: false,
format: None,
fields: None,
};
@@ -1112,6 +1217,89 @@ mod tests {
assert_ne!(yes, no, "Some(true) vs Some(false) must hash differently");
}
+ /// `include_related` must invalidate the cache: a body computed without the
+ /// `related` block can't be served to a caller asking for it (or vice-versa).
+ #[test]
+ fn test_search_cache_key_distinguishes_include_related() {
+ let mk = |include_related: bool| SearchQuery {
+ q: Some("test".to_string()),
+ pagination: PaginationQuery {
+ limit: 10,
+ offset: 0,
+ },
+ content_type: ContentType::All,
+ start_time: None,
+ end_time: None,
+ app_name: None,
+ window_name: None,
+ frame_name: None,
+ include_frames: false,
+ min_length: None,
+ max_length: None,
+ speaker_ids: None,
+ focused: None,
+ on_screen: None,
+ browser_url: None,
+ speaker_name: None,
+ include_cloud: false,
+ max_content_length: None,
+ device_name: None,
+ machine_id: None,
+ filter_pii: false,
+ tags: Some(vec!["person:ada".to_string()]),
+ include_related,
+ format: None,
+ fields: None,
+ };
+ assert_ne!(
+ compute_search_cache_key(&mk(false)),
+ compute_search_cache_key(&mk(true)),
+ "include_related must change the cache key"
+ );
+ }
+
+ #[test]
+ fn test_related_namespace_key_pluralizes() {
+ assert_eq!(related_namespace_key("person"), "people");
+ assert_eq!(related_namespace_key("company"), "companies");
+ assert_eq!(related_namespace_key("project"), "projects");
+ assert_eq!(related_namespace_key("workflow"), "workflows");
+ assert_eq!(related_namespace_key("app"), "apps");
+ }
+
+ #[test]
+ fn test_group_related_tags_buckets_by_namespace() {
+ // Count-desc input; grouping must preserve that order per bucket.
+ let rows = vec![
+ ("person:drew".to_string(), 9),
+ ("project:screenpipe".to_string(), 7),
+ ("person:connor".to_string(), 5),
+ ("workflow:ai-coding".to_string(), 4),
+ ("project:screenpipe-finance".to_string(), 3),
+ ("standalone".to_string(), 2),
+ ];
+ let grouped = group_related_tags(rows);
+
+ assert_eq!(grouped.get("people").unwrap(), &vec!["drew", "connor"]);
+ assert_eq!(
+ grouped.get("projects").unwrap(),
+ &vec!["screenpipe", "screenpipe-finance"]
+ );
+ assert_eq!(grouped.get("workflows").unwrap(), &vec!["ai-coding"]);
+ // Namespace-less tags fall under "tags" with their full value.
+ assert_eq!(grouped.get("tags").unwrap(), &vec!["standalone"]);
+ }
+
+ #[test]
+ fn test_group_related_tags_dedupes_within_bucket() {
+ let rows = vec![
+ ("person:ada".to_string(), 3),
+ ("person:ada".to_string(), 1),
+ ];
+ let grouped = group_related_tags(rows);
+ assert_eq!(grouped.get("people").unwrap(), &vec!["ada"]);
+ }
+
#[test]
fn test_truncate_middle_short_text() {
assert_eq!(truncate_middle("hello", 10), "hello");
diff --git a/docs/mintlify/docs-mintlify-mig-tmp/cli-reference.mdx b/docs/mintlify/docs-mintlify-mig-tmp/cli-reference.mdx
index 6833bc81c..ff7d4da1c 100644
--- a/docs/mintlify/docs-mintlify-mig-tmp/cli-reference.mdx
+++ b/docs/mintlify/docs-mintlify-mig-tmp/cli-reference.mdx
@@ -62,6 +62,35 @@ curl "http://localhost:3030/search?q=meeting&limit=10&content_type=all"
| `browser_url` | string | filter by browser URL |
| `min_length` | int | minimum text length |
| `max_length` | int | maximum text length |
+| `tags` | string | comma-separated; return only items carrying **all** of these tags, e.g. `tags=person:ada,project:atlas` |
+| `include_related` | bool | with `tags`, attach a `related` block of co-occurring tags grouped by namespace |
+
+### related context
+
+pass `include_related=true` alongside a `tags` filter to get the tags that
+co-occur with the ones you asked for — the people, projects, and workflows that
+show up in the same frames, calls, and memories — in a single call instead of
+several follow-up queries:
+
+```bash
+curl "http://localhost:3030/search?tags=person:ada&include_related=true&limit=5"
+```
+
+```json
+{
+ "data": [ "...frames, audio, and memories..." ],
+ "pagination": { "limit": 5, "offset": 0, "total": 42 },
+ "related": {
+ "people": ["connor", "drew"],
+ "projects": ["atlas", "atlas-finance"],
+ "workflows": ["planning"]
+ }
+}
+```
+
+namespaces are pluralized from the tag prefix (`person:` → `people`,
+`project:` → `projects`); values are ordered most-frequent first. omit
+`tags` and the block is skipped.
### content type guide
diff --git a/packages/screenpipe-mcp/src/index.ts b/packages/screenpipe-mcp/src/index.ts
index 83ce8696b..60a2a2866 100644
--- a/packages/screenpipe-mcp/src/index.ts
+++ b/packages/screenpipe-mcp/src/index.ts
@@ -307,6 +307,12 @@ const TOOLS: Tool[] = [
description:
"Comma-separated tags; returns only items carrying ALL of them (e.g. 'person:ada,project:atlas'). Works for screen + audio (content_type 'ocr'/'audio'/'all', tags written by add-tags) AND memories (content_type 'memory', tags written by update-memory). Same tag string links across all three, so two items sharing a tag are connected. Use namespaced tags (person:, project:, topic:) to link people/projects/topics. content_type 'input' and 'accessibility' have no tags and return nothing when this is set.",
},
+ include_related: {
+ type: "boolean",
+ description:
+ "With tags set, also return the co-occurring tags (the people/projects/topics seen alongside yours, ranked by frequency) as a 'Related:' line. One call for the surrounding context instead of several follow-ups. Ignored without tags.",
+ default: false,
+ },
max_content_length: {
type: "integer",
description: "Truncate each result's text via middle-truncation. Use 200-500 to keep responses compact.",
@@ -1383,8 +1389,11 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
const tagsStr = content.tags?.length ? ` [${content.tags.join(", ")}]` : "";
const importance =
content.importance != null ? ` (importance: ${content.importance})` : "";
+ // frame_id links a memory back to the exact moment — jump there with
+ // frame-context / get-frame-elements (frame_id=N).
+ const frameRef = content.frame_id != null ? ` frame:${content.frame_id}` : "";
formattedResults.push(
- `[Memory #${content.id}]${tagsStr}${importance}\n` +
+ `[Memory #${content.id}]${tagsStr}${importance}${frameRef}\n` +
`${content.created_at || ""}\n` +
`${truncateMiddle(content.content || "", effectiveCap)}`
);
@@ -1397,9 +1406,20 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
? ` (use offset=${(pagination.offset || 0) + results.length} for more)`
: "");
+ // Co-occurring tags (only present when include_related=true + tags set).
+ // Compact one-liner per namespace so it's cheap to read.
+ const related = data.related as Record<string, string[]> | undefined;
+ const relatedStr =
+ related && Object.keys(related).length > 0
+ ? "\n\nRelated tags: " +
+ Object.entries(related)
+ .map(([ns, vals]) => `${ns}: ${(Array.isArray(vals) ? vals : []).join(", ")}`)
+ .join(" | ")
+ : "";
+
contentItems.push({
type: "text",
- text: header + "\n\n" + formattedResults.join("\n---\n"),
+ text: header + "\n\n" + formattedResults.join("\n---\n") + relatedStr,
});
for (const img of images) {