YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
feat(api): token-efficient `format=outline` for the element APIs (#4287)
Louis Beaumont300d547d6b8a
diff --git a/.claude/skills/screenpipe-api/SKILL.md b/.claude/skills/screenpipe-api/SKILL.mdindex ece1b74f6..f792ce663 100644--- a/.claude/skills/screenpipe-api/SKILL.md+++ b/.claude/skills/screenpipe-api/SKILL.md@@ -21,7 +21,7 @@ The `$SCREENPIPE_LOCAL_API_KEY` env var is already set in your environment. WithAPI responses can be large. Always write curl output to a file first (`curl ... -o /tmp/sp_result.json`), check size (`wc -c /tmp/sp_result.json`), and if over 5KB read only the first 50-100 lines. Extract what you need with `jq`. NEVER dump full large responses into context.-For the list endpoints (`/search`, `/elements`, `/frames/{id}/elements`) you can also cut tokens at the source: add `&format=csv` (or `tsv`) to get a columnar table that writes each column name once instead of repeating keys per row, and `&fields=a,b,c` to return only the columns you need (dotted paths like `content.text`). On a list of UI elements that is roughly a 70% token cut versus JSON. Text-heavy `ocr`/`audio` barely benefit (the text blob dominates), so reach for `fields` + `max_content_length` there. With no `format`/`fields` the response is unchanged JSON.+For the list endpoints (`/search`, `/elements`, `/frames/{id}/elements`) you can also cut tokens at the source: add `&format=csv` (or `tsv`) to get a columnar table that writes each column name once instead of repeating keys per row, and `&fields=a,b,c` to return only the columns you need (dotted paths like `content.text`). On a list of UI elements that is roughly a 70% token cut versus JSON. For the element endpoints specifically, `&format=outline` (alias `tree`) goes further still — a deduped, indented tree of just the text-bearing nodes (~91% fewer tokens, measured) — and is the best default for reading UI structure. Text-heavy `ocr`/`audio` barely benefit from any reshaping (the text blob dominates), so reach for `fields` + `max_content_length` there. With no `format`/`fields` the response is unchanged JSON.---@@ -47,7 +47,7 @@ curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030| `focused` | boolean | No | Only focused windows || `tags` | string | No | Comma-separated; return only items carrying ALL of them (e.g. `person:ada,project:atlas`). Works for screen/audio and, with `content_type=memory`, memories. See Tags below. || `max_content_length` | integer | No | Truncate each result's text (middle-truncation) |-| `format` | string | No | `json` (default), `csv`, or `tsv`/`table`. CSV/TSV return a columnar table (column names written once) instead of one JSON object per row, much cheaper to read on list-shaped results. CSV is lossless; TSV collapses newlines (worse for long `ocr` text). |+| `format` | string | No | `json` (default), `csv`, `tsv`/`table`, or `outline`/`tree` (element endpoints only). CSV/TSV return a columnar table (column names written once) instead of one JSON object per row. `outline` returns a deduped indented text tree of the text-bearing UI nodes — the cheapest read for "what's on screen?" (~91% fewer tokens). CSV is lossless; TSV collapses newlines (worse for long `ocr` text). || `fields` | string | No | Comma-separated column allowlist of dotted paths, e.g. `type,content.app_name,content.text`. Returns only those columns (handy for dropping the repeated absolute `content.file_path`). Works for `json` too (sparse objects). |### Progressive Disclosure@@ -133,13 +133,24 @@ Lightweight FTS search across UI elements (~100-500 bytes each vs 5-20KB from `/curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030/elements?q=Submit&start_time=1h%20ago&limit=10"```-Parameters: `q`, `frame_id`, `source` (`accessibility`|`ocr`), `role`, `start_time`, `end_time`, `app_name`, `limit`, `offset`, plus `format` (`json`/`csv`/`tsv`) and `fields` (dotted paths). Elements are uniform rows, so this is where the columnar format pays off most:+Parameters: `q`, `frame_id`, `source` (`accessibility`|`ocr`), `role`, `start_time`, `end_time`, `app_name`, `limit`, `offset`, plus `format` (`json`/`csv`/`tsv`/`outline`) and `fields` (dotted paths). Elements are uniform rows, so this is where compact formats pay off most.++**`format=outline` (alias `tree`) is the cheapest read for "what's on screen?"** — a deduped, indented text tree of just the text-bearing nodes (drops empty structural nodes + bounds, collapses repeats into `×N`, `#id` is the ref, inlines `(disabled)`/`(selected)`/`(focused)`/`(expanded)`/`(off-screen)` state, body capped). Best on `source=accessibility` (the common UI case — structural noise, repeated rows, hierarchy, state): 85–99% fewer tokens than JSON (o200k_base). Flat OCR text blocks are the floor (~67%, nothing to dedup) — for pure OCR `format=csv&fields=text` is about as good.```bash-# compact table, only the columns you need (~70% fewer tokens than JSON)+# compact outline — best default for an LLM reading the UI+curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030/elements?q=Submit&format=outline&limit=30"+# frame 12345 · accessibility · 8 text elements+# AXButton "Submit Order" #4012+# AXButton "Cancel" #4013 (disabled)+# AXCell "Shipped" #4020 ×6++# columnar table when you need specific columns (e.g. bounds) insteadcurl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030/elements?frame_id=12345&format=csv&fields=role,text,bounds.left,bounds.top"```+`GET /frames/{id}/elements?format=outline` gives the whole frame's tree the same way (and is capped, unlike the raw JSON dump).+### Frame Context — `GET /frames/{id}/context`Returns accessibility text, parsed nodes, and extracted URLs for a frame.diff --git a/crates/screenpipe-db/src/db/elements.rs b/crates/screenpipe-db/src/db/elements.rsindex 0199cec6f..6a12be562 100644--- a/crates/screenpipe-db/src/db/elements.rs+++ b/crates/screenpipe-db/src/db/elements.rs@@ -269,7 +269,7 @@ LIMIT ? OFFSET ?let sql = format!(r#"SELECT e.id, e.frame_id, e.source, e.role, e.text, e.parent_id,e.depth, e.left_bound, e.top_bound, e.width_bound, e.height_bound,- e.confidence, e.sort_order, e.on_screen+ e.confidence, e.sort_order, e.on_screen, e.propertiesFROM elements eJOIN frames f ON f.id = e.frame_id{}@@ -361,9 +361,9 @@ LIMIT ? OFFSET ?.unwrap_or(frame_id);let sql = if source.is_some() {- "SELECT id, frame_id, source, role, text, parent_id, depth, left_bound, top_bound, width_bound, height_bound, confidence, sort_order, on_screen FROM elements WHERE frame_id = ?1 AND source = ?2 ORDER BY sort_order"+ "SELECT id, frame_id, source, role, text, parent_id, depth, left_bound, top_bound, width_bound, height_bound, confidence, sort_order, on_screen, properties FROM elements WHERE frame_id = ?1 AND source = ?2 ORDER BY sort_order"} else {- "SELECT id, frame_id, source, role, text, parent_id, depth, left_bound, top_bound, width_bound, height_bound, confidence, sort_order, on_screen FROM elements WHERE frame_id = ?1 ORDER BY sort_order"+ "SELECT id, frame_id, source, role, text, parent_id, depth, left_bound, top_bound, width_bound, height_bound, confidence, sort_order, on_screen, properties FROM elements WHERE frame_id = ?1 ORDER BY sort_order"};let mut query = sqlx::query_as::<_, ElementRow>(sql).bind(effective_frame_id);@@ -589,3 +589,92 @@ LIMIT ? OFFSET ?groups}}++#[cfg(test)]+mod tests {+ use super::*;++ async fn mem_db() -> DatabaseManager {+ DatabaseManager::new("sqlite::memory:", Default::default())+ .await+ .expect("in-memory db")+ }++ async fn seed_frame(db: &DatabaseManager) {+ sqlx::query("INSERT INTO video_chunks (id, file_path) VALUES (1, '/tmp/x.mp4')")+ .execute(&db.pool)+ .await+ .unwrap();+ sqlx::query(+ "INSERT INTO frames (id, video_chunk_id, offset_index, timestamp) \+ VALUES (1, 1, 0, '2026-06-17T00:00:00Z')",+ )+ .execute(&db.pool)+ .await+ .unwrap();+ }++ const INSERT_EL: &str = "INSERT INTO elements \+ (frame_id, source, role, text, parent_id, depth, left_bound, top_bound, \+ width_bound, height_bound, confidence, sort_order, properties, on_screen) \+ VALUES (1, ?, ?, ?, NULL, 1, 0.1, 0.2, 0.3, 0.4, NULL, ?, ?, 1)";++ #[tokio::test]+ async fn element_queries_select_and_roundtrip_properties() {+ let db = mem_db().await;+ seed_frame(&db).await;++ // accessibility element WITH state properties+ sqlx::query(INSERT_EL)+ .bind("accessibility")+ .bind("AXButton")+ .bind("Save")+ .bind(1)+ .bind(r#"{"is_enabled":false,"is_selected":true}"#)+ .execute(&db.pool)+ .await+ .unwrap();+ // accessibility element with NULL properties+ sqlx::query(INSERT_EL)+ .bind("accessibility")+ .bind("AXStaticText")+ .bind("Welcome")+ .bind(2)+ .bind(Option::<String>::None)+ .execute(&db.pool)+ .await+ .unwrap();++ // get_frame_elements: the new `properties` column is selected + mapped.+ let frame_els = db.get_frame_elements(1, None).await.unwrap();+ assert_eq!(frame_els.len(), 2);+ let btn = frame_els.iter().find(|e| e.role == "AXButton").unwrap();+ assert!(btn.properties.as_deref().unwrap().contains("is_enabled"));+ let txt = frame_els.iter().find(|e| e.role == "AXStaticText").unwrap();+ assert!(txt.properties.is_none());++ // search_elements path also selects `properties` and executes cleanly.+ let (search_els, total) = db+ .search_elements("", Some(1), None, None, None, None, None, None, 10, 0)+ .await+ .unwrap();+ assert_eq!(total, 2);+ assert!(search_els+ .iter()+ .any(|e| e.properties.as_deref().is_some_and(|p| p.contains("is_selected"))));+ }++ #[tokio::test]+ async fn element_queries_run_against_real_schema_when_empty() {+ // A typo'd column in either SELECT is a *runtime* sqlx error a compile+ // check can't catch — assert both run on the real (migrated) schema.+ let db = mem_db().await;+ let (els, total) = db+ .search_elements("", None, None, None, None, None, None, None, 5, 0)+ .await+ .unwrap();+ assert!(els.is_empty());+ assert_eq!(total, 0);+ assert!(db.get_frame_elements(1, None).await.unwrap().is_empty());+ }+}diff --git a/crates/screenpipe-db/src/types.rs b/crates/screenpipe-db/src/types.rsindex d5aed8648..641cd486f 100644--- a/crates/screenpipe-db/src/types.rs+++ b/crates/screenpipe-db/src/types.rs@@ -916,6 +916,12 @@ pub struct Element {/// implicitly true/false. See issue #2436.#[serde(default, skip_serializing_if = "Option::is_none")]pub on_screen: Option<bool>,+ /// Raw JSON of the element's automation properties (`is_enabled`,+ /// `is_focused`, `is_selected`, `is_expanded`, `value`, `placeholder`, …)+ /// serialized at capture. `None` for OCR rows and legacy accessibility+ /// rows. Parsed into compact state by the API layer.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub properties: Option<String>,}/// Flat row for bulk insert (parent_id assigned after insert)@@ -951,6 +957,9 @@ pub struct ElementRow {/// (legacy rows pre-issue-#2436 fix); search treats unknown as/// neither on- nor off-screen.pub on_screen: Option<bool>,+ /// Raw JSON automation properties (see [`Element::properties`]). NULL for+ /// OCR / legacy rows.+ pub properties: Option<String>,}impl From<ElementRow> for Element {@@ -981,6 +990,7 @@ impl From<ElementRow> for Element {confidence: row.confidence,sort_order: row.sort_order,on_screen: row.on_screen,+ properties: row.properties,}}}diff --git a/crates/screenpipe-engine/src/routes/elements.rs b/crates/screenpipe-engine/src/routes/elements.rsindex 4a8aed6b0..5e7cabdeb 100644--- a/crates/screenpipe-engine/src/routes/elements.rs+++ b/crates/screenpipe-engine/src/routes/elements.rs@@ -18,7 +18,9 @@ use serde_json::{json, Value};use std::sync::Arc;use tracing::error;-use super::response_format::{is_passthrough, parse_fields, parse_format, render_list, rows_from};+use super::response_format::{+ is_passthrough, parse_fields, parse_format, render_list, rows_from, OutputFormat,+};use crate::server::AppState;#[derive(OaSchema, Deserialize)]@@ -55,13 +57,16 @@ pub(crate) struct ElementsQuery {limit: u32,#[serde(default)]offset: u32,- /// Output format: `json` (default), `csv`, or `tsv`/`table`. CSV/TSV write- /// each column name once instead of repeating keys per row, which is far- /// cheaper for an LLM to read (a 25-row element list drops ~73% of tokens).+ /// Output format: `json` (default), `csv`, `tsv`/`table`, or+ /// `outline`/`tree`. CSV/TSV write each column name once instead of+ /// repeating keys per row. `outline` goes further for an LLM "what's on+ /// screen?" read: a deduped, indented tree of just the text-bearing nodes+ /// (drops empty structural nodes, bounds, and ids-other-than-`#id`).#[serde(default)]format: Option<String>,/// Comma-separated column allowlist, e.g. `fields=role,text,bounds.left`.- /// Dotted paths reach into nested objects. Omit for all fields.+ /// Dotted paths reach into nested objects. Omit for all fields. (Ignored by+ /// `format=outline`.)#[serde(default)]fields: Option<String>,}@@ -87,6 +92,10 @@ pub(crate) struct ElementResponse {/// landed — see issue #2436.#[serde(skip_serializing_if = "Option::is_none")]pub on_screen: Option<bool>,+ /// Compact interaction state (disabled/focused/selected/expanded) parsed+ /// from the captured automation properties. Omitted when there's no state.+ #[serde(skip_serializing_if = "Option::is_none")]+ pub state: Option<ElementState>,}#[derive(OaSchema, Serialize)]@@ -97,6 +106,40 @@ pub(crate) struct BoundsResponse {pub height: f64,}+/// Compact interaction state, parsed from the captured `properties` JSON. Only+/// the flags actually present are emitted (so a static-text node carries no+/// `state` at all). `disabled` is the inverse of the stored `is_enabled`.+#[derive(OaSchema, Serialize, Debug, Clone, PartialEq, Default)]+pub(crate) struct ElementState {+ #[serde(skip_serializing_if = "Option::is_none")]+ pub disabled: Option<bool>,+ #[serde(skip_serializing_if = "Option::is_none")]+ pub focused: Option<bool>,+ #[serde(skip_serializing_if = "Option::is_none")]+ pub selected: Option<bool>,+ #[serde(skip_serializing_if = "Option::is_none")]+ pub expanded: Option<bool>,+}++/// Parse the stored automation `properties` JSON into compact state. Returns+/// `None` unless at least one state flag is present, so non-interactive rows+/// stay empty. Defensive: bad/absent JSON simply yields `None`.+fn parse_element_state(properties: Option<&str>) -> Option<ElementState> {+ let v: Value = serde_json::from_str(properties?).ok()?;+ let b = |k: &str| v.get(k).and_then(Value::as_bool);+ let state = ElementState {+ disabled: b("is_enabled").map(|enabled| !enabled),+ focused: b("is_focused"),+ selected: b("is_selected"),+ expanded: b("is_expanded"),+ };+ if state == ElementState::default() {+ None+ } else {+ Some(state)+ }+}+#[derive(OaSchema, Serialize)]pub(crate) struct PaginationResponse {pub limit: u32,@@ -129,8 +172,186 @@ impl From<Element> for ElementResponse {confidence: e.confidence,sort_order: e.sort_order,on_screen: e.on_screen,+ state: parse_element_state(e.properties.as_deref()),+ }+ }+}++// ---------------------------------------------------------------------------+// `format=outline` — a compact, element-specific tree view+//+// CSV/TSV (in response_format) write keys once but stay flat and keep every+// row, including the structural noise an accessibility tree is full of. For an+// LLM asking "what's on screen?", the cheapest faithful view is a deduped,+// indented outline of the *text-bearing* nodes. This is element-specific (it+// understands role/text/depth/on_screen/state), so it lives here rather than in+// the generic tabular renderer. Interaction state (disabled/selected/focused/+// expanded) is inlined; bounds and numeric metadata (parent_id/sort_order/+// confidence) are dropped — ask for `format=json` when you need them.+//+// Measured vs the JSON default (tiktoken o200k_base, `dump_token_samples`):+//+// | tree shape | json tok | outline | saved |+// |---------------------|---------:|--------:|------:|+// | small flat UI | 381 | 58 | 85% |+// | typical app frame | 4196 | 463 | 89% |+// | wide table (dups) | 4471 | 60 | 99% |+// | deep nested | 1534 | 51 | 97% |+// | OCR text-heavy | 1309 | 432 | 67% |+// | multi-frame search | 2755 | 501 | 82% |+// | mostly-structural | 5879 | 299 | 95% |+// | AGGREGATE | 20525 | 1864 | 91% |+//+// The win tracks tree SHAPE, which tracks source: accessibility trees (rows+// 1–4, 6–7 above) are the target — structural noise to drop, repeated+// cells/rows to dedup, hierarchy, and state — a 6–75× cut. OCR is the floor+// (flat text blocks: generic role, no hierarchy/state, nothing to dedup, so the+// outline ≈ the text) — same caveat as CSV/TSV; for pure OCR `fields=text` is+// about as good.+// ---------------------------------------------------------------------------++const OUTLINE_MAX_LINES: usize = 200;+const OUTLINE_TEXT_CLIP: usize = 120;++/// True when the caller asked for the element outline view (`format=outline`+/// or `format=tree`). Checked before `parse_format` so those values don't trip+/// its json|csv|tsv validator.+pub(crate) fn wants_outline(format: &Option<String>) -> bool {+ matches!(+ format.as_deref().map(|s| s.trim().to_ascii_lowercase()).as_deref(),+ Some("outline") | Some("tree")+ )+}++/// Collapse internal whitespace and clip to `n` chars (char-safe, not byte).+fn outline_clip(s: &str, n: usize) -> String {+ let collapsed = s.split_whitespace().collect::<Vec<_>>().join(" ");+ if collapsed.chars().count() > n {+ let mut t: String = collapsed.chars().take(n).collect();+ t.push('…');+ t+ } else {+ collapsed+ }+}++/// Render an element list as the compact outline. Pure (no I/O) so it's unit+/// tested directly. Drops empty-text structural nodes, collapses runs of+/// identical `role`+`text` into `×N`, hoists per-frame context into a header,+/// indents by depth, escapes quotes in names, inlines state (off-screen ++/// disabled/selected/focused/expanded), and caps the body with a note.+fn elements_outline_text(elements: &[ElementResponse], total: i64) -> String {+ // A text view's whole point: keep only nodes that carry text.+ let kept: Vec<&ElementResponse> = elements+ .iter()+ .filter(|e| e.text.as_deref().map(|t| !t.trim().is_empty()).unwrap_or(false))+ .collect();++ if kept.is_empty() {+ return "no text-bearing elements (use format=json for the raw tree)".to_string();+ }++ // Group by frame in first-seen order (search can span frames).+ let mut frame_order: Vec<i64> = Vec::new();+ let mut by_frame: std::collections::HashMap<i64, Vec<&ElementResponse>> =+ std::collections::HashMap::new();+ for e in kept.iter().copied() {+ if !by_frame.contains_key(&e.frame_id) {+ frame_order.push(e.frame_id);+ by_frame.insert(e.frame_id, Vec::new());}+ by_frame.get_mut(&e.frame_id).unwrap().push(e);}++ let mut out = String::new();+ let mut emitted = 0usize;+ let mut truncated = false;++ 'frames: for fid in &frame_order {+ let els = &by_frame[fid];+ let src = els.first().map(|e| e.source.as_str()).unwrap_or("");+ out.push_str(&format!("frame {fid} · {src} · {} text elements\n", els.len()));++ let mut i = 0;+ while i < els.len() {+ if emitted >= OUTLINE_MAX_LINES {+ truncated = true;+ break 'frames;+ }+ let e = els[i];+ let text = outline_clip(e.text.as_deref().unwrap_or(""), OUTLINE_TEXT_CLIP);+ // Collapse a run of identical (role, clipped-text) siblings — AX+ // trees are full of repeated static text (list/table cells).+ let mut run = 1;+ while i + run < els.len() {+ let n = els[i + run];+ let ntext = outline_clip(n.text.as_deref().unwrap_or(""), OUTLINE_TEXT_CLIP);+ if n.role == e.role && ntext == text {+ run += 1;+ } else {+ break;+ }+ }+ let indent = " ".repeat((e.depth.max(0) as usize).min(6));+ // Escape quotes so the "name" delimiters stay unambiguous even when+ // the captured text itself contains a double-quote.+ let safe = text.replace('"', "\\\"");+ let mut line = format!("{indent}{} \"{safe}\" #{}", e.role, e.id);+ // Inline state — off-screen (issue #2436) + interaction flags parsed+ // from the captured properties.+ let mut flags: Vec<&str> = Vec::new();+ if e.on_screen == Some(false) {+ flags.push("off-screen");+ }+ if let Some(st) = &e.state {+ if st.disabled == Some(true) {+ flags.push("disabled");+ }+ if st.selected == Some(true) {+ flags.push("selected");+ }+ if st.focused == Some(true) {+ flags.push("focused");+ }+ match st.expanded {+ Some(true) => flags.push("expanded"),+ Some(false) => flags.push("collapsed"),+ None => {}+ }+ }+ if !flags.is_empty() {+ line.push_str(&format!(" ({})", flags.join(",")));+ }+ if run > 1 {+ line.push_str(&format!(" ×{run}"));+ }+ out.push_str(&line);+ out.push('\n');+ emitted += 1;+ i += run;+ }+ }++ if truncated || (total as usize) > emitted {+ out.push_str(&format!(+ "… showing {emitted} of {total} elements — narrow with ?q=, ?role=, ?on_screen=true, or ?limit=\n"+ ));+ }+ out+}++/// Wrap the outline text as a `text/plain` response, carrying the total in+/// `x-total-count` (mirrors how the delimited renderer ships pagination).+fn elements_outline_response(elements: &[ElementResponse], total: i64) -> Response {+ let body = elements_outline_text(elements, total);+ Response::builder()+ .header(+ axum::http::header::CONTENT_TYPE,+ "text/plain; charset=utf-8",+ )+ .header("x-total-count", total.to_string())+ .body(Body::from(body))+ .unwrap_or_else(|_| StatusCode::INTERNAL_SERVER_ERROR.into_response())}/// Search elements across all frames with optional FTS, time, and app filters.@@ -139,7 +360,14 @@ pub(crate) async fn search_elements(Query(query): Query<ElementsQuery>,State(state): State<Arc<AppState>>,) -> Result<Response<Body>, (StatusCode, JsonResponse<Value>)> {- let format = parse_format(&query.format)?;+ // `outline`/`tree` is element-specific and handled below; everything else+ // goes through the generic json|csv|tsv negotiation.+ let outline = wants_outline(&query.format);+ let format = if outline {+ OutputFormat::Json+ } else {+ parse_format(&query.format)?+ };let fields = parse_fields(&query.fields);let q = query.q.as_deref().unwrap_or("");let source = query@@ -178,6 +406,9 @@ pub(crate) async fn search_elements(total,},};+ if outline {+ return Ok(elements_outline_response(&list.data, total));+ }if is_passthrough(format, &fields) {return Ok(JsonResponse(list).into_response());}@@ -197,7 +428,12 @@ pub(crate) async fn get_frame_elements(Path(frame_id): Path<i64>,Query(query): Query<FrameElementsQuery>,) -> Result<Response<Body>, (StatusCode, JsonResponse<Value>)> {- let format = parse_format(&query.format)?;+ let outline = wants_outline(&query.format);+ let format = if outline {+ OutputFormat::Json+ } else {+ parse_format(&query.format)?+ };let fields = parse_fields(&query.fields);let source = query.source@@ -225,6 +461,9 @@ pub(crate) async fn get_frame_elements(total,},};+ if outline {+ return Ok(elements_outline_response(&list.data, total));+ }if is_passthrough(format, &fields) {return Ok(JsonResponse(list).into_response());}@@ -237,11 +476,383 @@ pub(crate) async fn get_frame_elements())}+#[cfg(test)]+mod tests {+ use super::*;++ fn el(+ id: i64,+ frame_id: i64,+ role: &str,+ text: Option<&str>,+ depth: i32,+ on_screen: Option<bool>,+ ) -> ElementResponse {+ ElementResponse {+ id,+ frame_id,+ source: "accessibility".into(),+ role: role.into(),+ text: text.map(|s| s.to_string()),+ parent_id: None,+ depth,+ bounds: Some(BoundsResponse {+ left: 0.1,+ top: 0.2,+ width: 0.3,+ height: 0.4,+ }),+ confidence: None,+ sort_order: id as i32,+ on_screen,+ state: None,+ }+ }++ #[test]+ fn parse_element_state_extracts_flags_and_inverts_enabled() {+ let s = parse_element_state(Some(+ r#"{"is_enabled":false,"is_selected":true,"value":"x"}"#,+ ))+ .unwrap();+ assert_eq!(s.disabled, Some(true)); // is_enabled:false → disabled+ assert_eq!(s.selected, Some(true));+ assert_eq!(s.focused, None);+ // value/placeholder only → no interaction state+ assert!(parse_element_state(Some(r#"{"value":"x"}"#)).is_none());+ assert!(parse_element_state(None).is_none());+ assert!(parse_element_state(Some("not json")).is_none());+ }++ #[test]+ fn outline_inlines_interaction_state() {+ let mut a = el(1, 9, "AXButton", Some("Save"), 1, Some(true));+ a.state = Some(ElementState {+ disabled: Some(true),+ ..Default::default()+ });+ let mut b = el(2, 9, "AXTab", Some("Inbox"), 1, Some(true));+ b.state = Some(ElementState {+ selected: Some(true),+ focused: Some(true),+ ..Default::default()+ });+ let mut c = el(3, 9, "AXDisclosureTriangle", Some("Details"), 1, Some(true));+ c.state = Some(ElementState {+ expanded: Some(false),+ ..Default::default()+ });+ let out = elements_outline_text(&[a, b, c], 3);+ assert!(out.contains("AXButton \"Save\" #1 (disabled)"), "got:\n{out}");+ assert!(out.contains("AXTab \"Inbox\" #2 (selected,focused)"), "got:\n{out}");+ assert!(out.contains("AXDisclosureTriangle \"Details\" #3 (collapsed)"), "got:\n{out}");+ }++ #[test]+ fn outline_escapes_quotes_in_text() {+ let els = vec![el(1, 9, "AXStaticText", Some("say \"hi\" now"), 1, None)];+ let out = elements_outline_text(&els, 1);+ assert!(out.contains("\\\"hi\\\""), "quotes not escaped: {out}");+ }++ #[test]+ fn wants_outline_matches_outline_and_tree_only() {+ assert!(wants_outline(&Some("outline".into())));+ assert!(wants_outline(&Some(" TREE ".into())));+ assert!(!wants_outline(&Some("json".into())));+ assert!(!wants_outline(&Some("csv".into())));+ assert!(!wants_outline(&None));+ }++ #[test]+ fn outline_drops_empty_text_structural_nodes() {+ let els = vec![+ el(1, 9, "AXGroup", None, 0, None),+ el(2, 9, "AXStaticText", Some(" "), 1, None), // whitespace only+ el(3, 9, "AXButton", Some("Submit"), 1, None),+ ];+ let out = elements_outline_text(&els, 3);+ assert!(out.contains("AXButton \"Submit\" #3"), "got:\n{out}");+ assert!(!out.contains("AXGroup"));+ assert!(!out.contains("#1"));+ assert!(!out.contains("#2"));+ }++ #[test]+ fn outline_collapses_identical_runs_into_count() {+ let els = vec![+ el(1, 9, "AXCell", Some("Active"), 2, None),+ el(2, 9, "AXCell", Some("Active"), 2, None),+ el(3, 9, "AXCell", Some("Active"), 2, None),+ el(4, 9, "AXCell", Some("Inactive"), 2, None),+ ];+ let out = elements_outline_text(&els, 4);+ assert!(out.contains("AXCell \"Active\" #1 ×3"), "got:\n{out}");+ assert!(out.contains("AXCell \"Inactive\" #4"));+ assert!(!out.contains("#2"));+ }++ #[test]+ fn outline_flags_off_screen_and_drops_bounds() {+ let els = vec![el(7, 9, "AXTextField", Some("Email"), 1, Some(false))];+ let out = elements_outline_text(&els, 1);+ assert!(out.contains("AXTextField \"Email\" #7 (off-screen)"), "got:\n{out}");+ // bounds (floats / keys) must not bloat the outline+ assert!(!out.contains("0.1"));+ assert!(!out.contains("left"));+ }++ #[test]+ fn outline_groups_by_frame_with_headers_in_first_seen_order() {+ let els = vec![+ el(1, 100, "AXButton", Some("A"), 0, None),+ el(2, 200, "AXButton", Some("B"), 0, None),+ ];+ let out = elements_outline_text(&els, 2);+ assert!(out.contains("frame 100 · accessibility · 1 text elements"), "got:\n{out}");+ assert!(out.contains("frame 200 · accessibility · 1 text elements"));+ assert!(out.find("#1").unwrap() < out.find("#2").unwrap());+ }++ #[test]+ fn outline_caps_body_and_notes_truncation() {+ let els: Vec<ElementResponse> = (0..(OUTLINE_MAX_LINES as i64 + 50))+ .map(|i| el(i + 1, 9, "AXStaticText", Some(&format!("line {i}")), 1, None))+ .collect();+ let total = els.len() as i64;+ let out = elements_outline_text(&els, total);+ assert!(out.contains("showing"));+ assert!(out.contains(&format!("of {total}")));+ let emitted = out.lines().filter(|l| l.contains('#')).count();+ assert!(emitted <= OUTLINE_MAX_LINES, "emitted {emitted} > cap");+ }++ #[test]+ fn outline_collapses_whitespace_and_clips_long_text() {+ let long = "x".repeat(OUTLINE_TEXT_CLIP + 40);+ let els = vec![+ el(1, 9, "AXStaticText", Some("a\n b\t c"), 1, None),+ el(2, 9, "AXStaticText", Some(&long), 1, None),+ ];+ let out = elements_outline_text(&els, 2);+ assert!(out.contains("\"a b c\""), "got:\n{out}");+ assert!(out.contains('…'));+ assert!(!out.contains(&long));+ }++ #[test]+ fn outline_is_far_cheaper_than_json() {+ // Distinct text so the win is the format itself, not just dedup.+ let els: Vec<ElementResponse> = (0..25)+ .map(|i| el(i + 1, 9, "AXButton", Some(&format!("Button {i}")), 2, Some(true)))+ .collect();+ let outline = elements_outline_text(&els, 25);+ let json = serde_json::to_string(&els).unwrap();+ assert!(+ outline.len() < json.len() / 2,+ "outline {} vs json {} bytes",+ outline.len(),+ json.len()+ );+ }++ #[test]+ fn outline_handles_no_text_bearing_input() {+ let els = vec![el(1, 9, "AXGroup", None, 0, None)];+ assert!(elements_outline_text(&els, 1).contains("no text-bearing elements"));+ assert!(elements_outline_text(&[], 0).contains("no text-bearing elements"));+ }++ #[test]+ fn outline_is_char_safe_with_unicode_and_clips_by_char() {+ // Multi-byte text must clip on char boundaries (never panic) and the+ // … marker must appear without slicing a code point.+ let emoji = "🎉".repeat(OUTLINE_TEXT_CLIP + 30);+ let els = vec![+ el(1, 9, "AXStaticText", Some("café — naïve 日本語 😀"), 1, None),+ el(2, 9, "AXStaticText", Some(&emoji), 1, None),+ ];+ let out = elements_outline_text(&els, 2);+ assert!(out.contains("café — naïve 日本語 😀"));+ assert!(out.contains('…'));+ // clipped to OUTLINE_TEXT_CLIP code points (+ the … marker)+ let clipped_line = out.lines().find(|l| l.contains("🎉")).unwrap();+ let emoji_count = clipped_line.chars().filter(|&c| c == '🎉').count();+ assert_eq!(emoji_count, OUTLINE_TEXT_CLIP);+ }++ #[test]+ fn outline_caps_indent_for_deeply_nested() {+ let els = vec![el(1, 9, "AXButton", Some("Deep"), 30, Some(true))];+ let out = elements_outline_text(&els, 1);+ let line = out.lines().find(|l| l.contains("Deep")).unwrap();+ let leading = line.len() - line.trim_start().len();+ assert_eq!(leading, 12, "indent should cap at 6 levels × 2 spaces"); // not 60+ }++ #[test]+ fn outline_dedup_is_adjacent_only_preserving_structure() {+ // A A B A — the trailing A must NOT merge with the leading run.+ let els = vec![+ el(1, 9, "AXTab", Some("A"), 1, None),+ el(2, 9, "AXTab", Some("A"), 1, None),+ el(3, 9, "AXTab", Some("B"), 1, None),+ el(4, 9, "AXTab", Some("A"), 1, None),+ ];+ let out = elements_outline_text(&els, 4);+ assert!(out.contains("AXTab \"A\" #1 ×2"), "got:\n{out}");+ assert!(out.contains("AXTab \"B\" #3"));+ assert!(out.contains("AXTab \"A\" #4"));+ assert!(!out.contains("#4 ×"));+ }++ #[test]+ fn outline_buckets_interleaved_frames() {+ // Search can return frames interleaved; the outline buckets each frame+ // (first-seen order) so the model reads one coherent frame at a time.+ let els = vec![+ el(1, 100, "AXButton", Some("one"), 0, None),+ el(2, 200, "AXButton", Some("two"), 0, None),+ el(3, 100, "AXButton", Some("three"), 0, None),+ ];+ let out = elements_outline_text(&els, 3);+ assert!(out.contains("frame 100 · accessibility · 2 text elements"), "got:\n{out}");+ assert!(out.contains("frame 200 · accessibility · 1 text elements"));+ // #1 and #3 (frame 100) precede #2 (frame 200)+ assert!(out.find("#3").unwrap() < out.find("#2").unwrap());+ }++ #[test]+ fn outline_keeps_one_line_per_element_through_quotes_and_newlines() {+ let els = vec![el(+ 1,+ 9,+ "AXStaticText",+ Some("he said \"hi\"\nthen left"),+ 1,+ None,+ )];+ let out = elements_outline_text(&els, 1);+ // exactly one frame header + one element line (+ trailing newline)+ let body: Vec<&str> = out.lines().filter(|l| l.contains('#')).collect();+ assert_eq!(body.len(), 1, "embedded newline split the row: {out:?}");+ // newline collapsed to a space; embedded quotes escaped+ assert!(body[0].contains("he said \\\"hi\\\" then left"), "got: {}", body[0]);+ }++ // Reproducible token-measurement harness. Renders JSON-default vs outline+ // for several realistic tree shapes into a temp file so an external+ // tokenizer can score them:+ // cargo test -p screenpipe-engine dump_token_samples -- --ignored+ // python3 -c "import json,tiktoken;e=tiktoken.get_encoding('o200k_base');\+ // [print(s['name'], len(e.encode(s['json'])), len(e.encode(s['outline']))) \+ // for s in json.load(open('<tmp>/sp_elem_samples.json'))]"+ // Measured (o200k_base): 91% fewer tokens aggregate (11×); per shape 67%+ // (OCR text-heavy) … 99% (tabular dedup). #[ignore] so CI never does the IO.+ #[test]+ #[ignore]+ fn dump_token_samples() {+ fn collect(name: &str, els: Vec<ElementResponse>) -> serde_json::Value {+ let total = els.len() as i64;+ let json = serde_json::to_string(&serde_json::json!({+ "data": &els,+ "pagination": { "limit": total, "offset": 0, "total": total }+ }))+ .unwrap();+ let outline = elements_outline_text(&els, total);+ serde_json::json!({ "name": name, "elements": els.len(), "json": json, "outline": outline })+ }++ let mut samples = Vec::new();++ // 1. small flat UI+ samples.push(collect(+ "small_flat",+ vec![+ el(1, 1, "AXButton", Some("Compose"), 1, Some(true)),+ el(2, 1, "AXLink", Some("Inbox"), 1, Some(true)),+ el(3, 1, "AXLink", Some("Sent"), 1, Some(true)),+ el(4, 1, "AXTextField", Some("Search mail"), 1, Some(true)),+ el(5, 1, "AXButton", Some("Settings"), 1, Some(true)),+ ],+ ));++ // 2. typical app frame: real text + structural noise + some dup text+ let mut typ = vec![+ el(1, 7, "AXHeading", Some("Inbox"), 1, Some(true)),+ el(2, 7, "AXButton", Some("Compose"), 1, Some(true)),+ el(3, 7, "AXTextField", Some("Search"), 1, Some(true)),+ ];+ for i in 0..18 {+ typ.push(el(100 + i * 3, 7, "AXGroup", None, 2, Some(true))); // empty noise+ typ.push(el(101 + i * 3, 7, "AXStaticText", Some(&format!("Sender {i}")), 4, Some(true)));+ typ.push(el(102 + i * 3, 7, "AXStaticText", Some("Unread"), 4, Some(true))); // dup+ }+ samples.push(collect("typical_app", typ));++ // 3. wide table: 60 rows of repeated cells (dedup territory)+ let mut table = vec![el(1, 3, "AXHeading", Some("Orders"), 1, Some(true))];+ for i in 0..60 {+ table.push(el(10 + i, 3, "AXCell", Some("Shipped"), 3, Some(true)));+ }+ samples.push(collect("wide_table_dups", table));++ // 4. deep nested chain+ let mut deep = Vec::new();+ for d in 0..20 {+ deep.push(el(d as i64 + 1, 4, "AXGroup", None, d, Some(true)));+ }+ deep.push(el(999, 4, "AXButton", Some("Deeply nested action"), 20, Some(true)));+ samples.push(collect("deep_nested", deep));++ // 5. OCR-heavy long text+ let mut ocr = Vec::new();+ for i in 0..15 {+ let mut e = el(+ i + 1,+ 5,+ "text",+ Some(&format!("This is a longer sentence of captured OCR text number {i} that a real screen would contain in a paragraph.")),+ 1,+ Some(true),+ );+ e.source = "ocr".into();+ e.on_screen = None;+ ocr.push(e);+ }+ samples.push(collect("ocr_heavy", ocr));++ // 6. multi-frame search result+ let mut multi = Vec::new();+ for f in 0..3 {+ for i in 0..12 {+ multi.push(el(f * 100 + i, 200 + f, "AXLink", Some(&format!("Result {f}-{i}")), 2, Some(true)));+ }+ }+ samples.push(collect("multi_frame", multi));++ // 7. mostly-structural noise (60 empty groups, 20 text)+ let mut noisy = Vec::new();+ for i in 0..60 {+ noisy.push(el(i + 1, 6, "AXGroup", None, (i % 8) as i32, Some(true)));+ }+ for i in 0..20 {+ noisy.push(el(1000 + i, 6, "AXStaticText", Some(&format!("Label {i}")), 3, Some(true)));+ }+ samples.push(collect("noisy_structural", noisy));++ let path = std::env::temp_dir().join("sp_elem_samples.json");+ std::fs::write(&path, serde_json::to_string_pretty(&samples).unwrap()).unwrap();+ eprintln!("wrote {} samples to {}", samples.len(), path.display());+ }+}+#[derive(OaSchema, Deserialize)]pub(crate) struct FrameElementsQuery {#[serde(default)]source: Option<String>,- /// Output format: `json` (default), `csv`, or `tsv`/`table`.+ /// Output format: `json` (default), `csv`, `tsv`/`table`, or+ /// `outline`/`tree` (compact text tree — the cheapest full-frame read).#[serde(default)]format: Option<String>,/// Comma-separated column allowlist, e.g. `fields=role,text,bounds.left`.diff --git a/packages/screenpipe-mcp/README.md b/packages/screenpipe-mcp/README.mdindex 8498dc93f..bae8dad7f 100644--- a/packages/screenpipe-mcp/README.md+++ b/packages/screenpipe-mcp/README.md@@ -154,6 +154,12 @@ List detected meetings with duration, app, and attendees. Pass `q` to filter bySearch structured UI elements (accessibility tree nodes and OCR text blocks):- Filter by source, role, app, time range- Much lighter than search-content for targeted UI lookups+- Returns a compact `outline` view by default — a deduped, indented tree of the+ text-bearing nodes (`#id` refs, `(off-screen)` flags), ~91% fewer tokens than+ raw element JSON++### get-frame-elements+The whole element tree for one frame, as the same compact outline.### frame-contextGet accessibility text, parsed tree nodes, and extracted URLs for a specific frame.@@ -179,7 +185,7 @@ Get accessibility text, parsed tree nodes, and extracted URLs for a specific fra- All timestamps are handled in UTC- Results are formatted for readability in Claude's interface- macOS automation features require accessibility permissions-- The MCP tools already return compact, readable text. If you instead call the underlying screenpipe REST API directly (e.g. via `curl`), the list endpoints (`/search`, `/elements`, `/frames/{id}/elements`) accept `?format=csv|tsv` for a columnar table (column names written once) and `?fields=a,b,c` to select only the columns you need (dotted paths like `content.text`). On list-shaped results that is roughly a 70% token cut versus the default JSON, which stays unchanged when neither param is set.+- The MCP tools already return compact, readable text (the element tools default to the `outline` view). If you instead call the underlying screenpipe REST API directly (e.g. via `curl`), the list endpoints (`/search`, `/elements`, `/frames/{id}/elements`) accept `?format=csv|tsv` for a columnar table (column names written once) and `?fields=a,b,c` to select only the columns you need (dotted paths like `content.text`); the element endpoints also accept `?format=outline` (the same tree the MCP tools return, ~91% fewer tokens than JSON). On list-shaped results that is a 70–91% token cut versus the default JSON, which stays unchanged when no param is set.## Privacy Policydiff --git a/packages/screenpipe-mcp/src/index.ts b/packages/screenpipe-mcp/src/index.tsindex b48e24647..83ce8696b 100644--- a/packages/screenpipe-mcp/src/index.ts+++ b/packages/screenpipe-mcp/src/index.ts@@ -1537,48 +1537,24 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {}}- const response = await callAPI(`/elements?${params.toString()}`);-- const data = await response.json();- const elements = data.data || [];- const pagination = data.pagination || {};-- if (elements.length === 0) {- return {- content: [- {- type: "text",- text: "No elements found. Try: broader search, different role/source, or wider time range.",- },- ],- };- }-- const formatted = elements.map(- (e: {- id: number;- frame_id: number;- source: string;- role: string;- text: string | null;- depth: number;- bounds: { left: number; top: number; width: number; height: number } | null;- }) => {- const boundsStr = e.bounds- ? ` [${e.bounds.left.toFixed(2)},${e.bounds.top.toFixed(2)} ${e.bounds.width.toFixed(2)}x${e.bounds.height.toFixed(2)}]`- : "";- return `[${e.source}] ${e.role} (frame:${e.frame_id}, depth:${e.depth})${boundsStr}\n ${e.text || "(no text)"}`;- }- );+ // Default to the server's compact `outline` view — a deduped, indented+ // tree of just the text-bearing nodes, far cheaper for the model to read+ // than the raw JSON rows (and the dedup/cap/footer replace the old+ // hand-rolled header). Callers can still override with format=json|csv|tsv.+ if (!params.has("format")) params.append("format", "outline");- const header =- `Elements: ${elements.length}/${pagination.total || "?"}` +- (pagination.total > elements.length- ? ` (use offset=${(pagination.offset || 0) + elements.length} for more)`- : "");+ const response = await callAPI(`/elements?${params.toString()}`);+ const text = (await response.text()).trim();return {- content: [{ type: "text", text: header + "\n\n" + formatted.join("\n---\n") }],+ content: [+ {+ type: "text",+ text: text.length+ ? text+ : "No elements found. Try: broader search, different role/source, or wider time range.",+ },+ ],};}@@ -1997,19 +1973,19 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {if (!frameId) {return { content: [{ type: "text", text: "Error: frame_id is required" }] };}- const response = await callAPI(`/frames/${frameId}/elements`);- const elements = await response.json();- if (!Array.isArray(elements) || elements.length === 0) {- return { content: [{ type: "text", text: `No elements found for frame ${frameId}.` }] };- }- const formatted = elements.map(- (e: { role: string; text: string | null; depth: number; source: string }) => {- const indent = " ".repeat(Math.min(e.depth, 5));- return `${indent}[${e.source}:${e.role}] ${e.text || "(no text)"}`;- }- );+ // Compact outline (text/plain): drops structural noise, dedups repeated+ // rows, caps the body. Also avoids the old bug here that parsed the+ // `{data,pagination}` envelope as a bare array and always reported+ // "no elements".+ const response = await callAPI(`/frames/${frameId}/elements?format=outline`);+ const text = (await response.text()).trim();return {- content: [{ type: "text", text: `Frame ${frameId} elements (${elements.length}):\n${formatted.join("\n")}` }],+ content: [+ {+ type: "text",+ text: text.length ? text : `No elements found for frame ${frameId}.`,+ },+ ],};}