Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

feat(api): token-efficient `format=outline` for the element APIs (#4287)

Louis Beaumont300d547d6b8a
diff --git a/.claude/skills/screenpipe-api/SKILL.md b/.claude/skills/screenpipe-api/SKILL.md
index ece1b74f6..f792ce663 100644
--- a/.claude/skills/screenpipe-api/SKILL.md
+++ b/.claude/skills/screenpipe-api/SKILL.md
@@ -21,7 +21,7 @@ The `$SCREENPIPE_LOCAL_API_KEY` env var is already set in your environment. With
API responses can be large. Always write curl output to a file first (`curl ... -o /tmp/sp_result.json`), check size (`wc -c /tmp/sp_result.json`), and if over 5KB read only the first 50-100 lines. Extract what you need with `jq`. NEVER dump full large responses into context.
-For the list endpoints (`/search`, `/elements`, `/frames/{id}/elements`) you can also cut tokens at the source: add `&format=csv` (or `tsv`) to get a columnar table that writes each column name once instead of repeating keys per row, and `&fields=a,b,c` to return only the columns you need (dotted paths like `content.text`). On a list of UI elements that is roughly a 70% token cut versus JSON. Text-heavy `ocr`/`audio` barely benefit (the text blob dominates), so reach for `fields` + `max_content_length` there. With no `format`/`fields` the response is unchanged JSON.
+For the list endpoints (`/search`, `/elements`, `/frames/{id}/elements`) you can also cut tokens at the source: add `&format=csv` (or `tsv`) to get a columnar table that writes each column name once instead of repeating keys per row, and `&fields=a,b,c` to return only the columns you need (dotted paths like `content.text`). On a list of UI elements that is roughly a 70% token cut versus JSON. For the element endpoints specifically, `&format=outline` (alias `tree`) goes further still — a deduped, indented tree of just the text-bearing nodes (~91% fewer tokens, measured) — and is the best default for reading UI structure. Text-heavy `ocr`/`audio` barely benefit from any reshaping (the text blob dominates), so reach for `fields` + `max_content_length` there. With no `format`/`fields` the response is unchanged JSON.
---
@@ -47,7 +47,7 @@ curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030
| `focused` | boolean | No | Only focused windows |
| `tags` | string | No | Comma-separated; return only items carrying ALL of them (e.g. `person:ada,project:atlas`). Works for screen/audio and, with `content_type=memory`, memories. See Tags below. |
| `max_content_length` | integer | No | Truncate each result's text (middle-truncation) |
-| `format` | string | No | `json` (default), `csv`, or `tsv`/`table`. CSV/TSV return a columnar table (column names written once) instead of one JSON object per row, much cheaper to read on list-shaped results. CSV is lossless; TSV collapses newlines (worse for long `ocr` text). |
+| `format` | string | No | `json` (default), `csv`, `tsv`/`table`, or `outline`/`tree` (element endpoints only). CSV/TSV return a columnar table (column names written once) instead of one JSON object per row. `outline` returns a deduped indented text tree of the text-bearing UI nodes — the cheapest read for "what's on screen?" (~91% fewer tokens). CSV is lossless; TSV collapses newlines (worse for long `ocr` text). |
| `fields` | string | No | Comma-separated column allowlist of dotted paths, e.g. `type,content.app_name,content.text`. Returns only those columns (handy for dropping the repeated absolute `content.file_path`). Works for `json` too (sparse objects). |
### Progressive Disclosure
@@ -133,13 +133,24 @@ Lightweight FTS search across UI elements (~100-500 bytes each vs 5-20KB from `/
curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030/elements?q=Submit&start_time=1h%20ago&limit=10"
```
-Parameters: `q`, `frame_id`, `source` (`accessibility`|`ocr`), `role`, `start_time`, `end_time`, `app_name`, `limit`, `offset`, plus `format` (`json`/`csv`/`tsv`) and `fields` (dotted paths). Elements are uniform rows, so this is where the columnar format pays off most:
+Parameters: `q`, `frame_id`, `source` (`accessibility`|`ocr`), `role`, `start_time`, `end_time`, `app_name`, `limit`, `offset`, plus `format` (`json`/`csv`/`tsv`/`outline`) and `fields` (dotted paths). Elements are uniform rows, so this is where compact formats pay off most.
+
+**`format=outline` (alias `tree`) is the cheapest read for "what's on screen?"** — a deduped, indented text tree of just the text-bearing nodes (drops empty structural nodes + bounds, collapses repeats into `×N`, `#id` is the ref, inlines `(disabled)`/`(selected)`/`(focused)`/`(expanded)`/`(off-screen)` state, body capped). Best on `source=accessibility` (the common UI case — structural noise, repeated rows, hierarchy, state): 85–99% fewer tokens than JSON (o200k_base). Flat OCR text blocks are the floor (~67%, nothing to dedup) — for pure OCR `format=csv&fields=text` is about as good.
```bash
-# compact table, only the columns you need (~70% fewer tokens than JSON)
+# compact outline — best default for an LLM reading the UI
+curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030/elements?q=Submit&format=outline&limit=30"
+# frame 12345 · accessibility · 8 text elements
+# AXButton "Submit Order" #4012
+# AXButton "Cancel" #4013 (disabled)
+# AXCell "Shipped" #4020 ×6
+
+# columnar table when you need specific columns (e.g. bounds) instead
curl -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" "http://localhost:3030/elements?frame_id=12345&format=csv&fields=role,text,bounds.left,bounds.top"
```
+`GET /frames/{id}/elements?format=outline` gives the whole frame's tree the same way (and is capped, unlike the raw JSON dump).
+
### Frame Context — `GET /frames/{id}/context`
Returns accessibility text, parsed nodes, and extracted URLs for a frame.
diff --git a/crates/screenpipe-db/src/db/elements.rs b/crates/screenpipe-db/src/db/elements.rs
index 0199cec6f..6a12be562 100644
--- a/crates/screenpipe-db/src/db/elements.rs
+++ b/crates/screenpipe-db/src/db/elements.rs
@@ -269,7 +269,7 @@ LIMIT ? OFFSET ?
let sql = format!(
r#"SELECT e.id, e.frame_id, e.source, e.role, e.text, e.parent_id,
e.depth, e.left_bound, e.top_bound, e.width_bound, e.height_bound,
- e.confidence, e.sort_order, e.on_screen
+ e.confidence, e.sort_order, e.on_screen, e.properties
FROM elements e
JOIN frames f ON f.id = e.frame_id
{}
@@ -361,9 +361,9 @@ LIMIT ? OFFSET ?
.unwrap_or(frame_id);
let sql = if source.is_some() {
- "SELECT id, frame_id, source, role, text, parent_id, depth, left_bound, top_bound, width_bound, height_bound, confidence, sort_order, on_screen FROM elements WHERE frame_id = ?1 AND source = ?2 ORDER BY sort_order"
+ "SELECT id, frame_id, source, role, text, parent_id, depth, left_bound, top_bound, width_bound, height_bound, confidence, sort_order, on_screen, properties FROM elements WHERE frame_id = ?1 AND source = ?2 ORDER BY sort_order"
} else {
- "SELECT id, frame_id, source, role, text, parent_id, depth, left_bound, top_bound, width_bound, height_bound, confidence, sort_order, on_screen FROM elements WHERE frame_id = ?1 ORDER BY sort_order"
+ "SELECT id, frame_id, source, role, text, parent_id, depth, left_bound, top_bound, width_bound, height_bound, confidence, sort_order, on_screen, properties FROM elements WHERE frame_id = ?1 ORDER BY sort_order"
};
let mut query = sqlx::query_as::<_, ElementRow>(sql).bind(effective_frame_id);
@@ -589,3 +589,92 @@ LIMIT ? OFFSET ?
groups
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ async fn mem_db() -> DatabaseManager {
+ DatabaseManager::new("sqlite::memory:", Default::default())
+ .await
+ .expect("in-memory db")
+ }
+
+ async fn seed_frame(db: &DatabaseManager) {
+ sqlx::query("INSERT INTO video_chunks (id, file_path) VALUES (1, '/tmp/x.mp4')")
+ .execute(&db.pool)
+ .await
+ .unwrap();
+ sqlx::query(
+ "INSERT INTO frames (id, video_chunk_id, offset_index, timestamp) \
+ VALUES (1, 1, 0, '2026-06-17T00:00:00Z')",
+ )
+ .execute(&db.pool)
+ .await
+ .unwrap();
+ }
+
+ const INSERT_EL: &str = "INSERT INTO elements \
+ (frame_id, source, role, text, parent_id, depth, left_bound, top_bound, \
+ width_bound, height_bound, confidence, sort_order, properties, on_screen) \
+ VALUES (1, ?, ?, ?, NULL, 1, 0.1, 0.2, 0.3, 0.4, NULL, ?, ?, 1)";
+
+ #[tokio::test]
+ async fn element_queries_select_and_roundtrip_properties() {
+ let db = mem_db().await;
+ seed_frame(&db).await;
+
+ // accessibility element WITH state properties
+ sqlx::query(INSERT_EL)
+ .bind("accessibility")
+ .bind("AXButton")
+ .bind("Save")
+ .bind(1)
+ .bind(r#"{"is_enabled":false,"is_selected":true}"#)
+ .execute(&db.pool)
+ .await
+ .unwrap();
+ // accessibility element with NULL properties
+ sqlx::query(INSERT_EL)
+ .bind("accessibility")
+ .bind("AXStaticText")
+ .bind("Welcome")
+ .bind(2)
+ .bind(Option::<String>::None)
+ .execute(&db.pool)
+ .await
+ .unwrap();
+
+ // get_frame_elements: the new `properties` column is selected + mapped.
+ let frame_els = db.get_frame_elements(1, None).await.unwrap();
+ assert_eq!(frame_els.len(), 2);
+ let btn = frame_els.iter().find(|e| e.role == "AXButton").unwrap();
+ assert!(btn.properties.as_deref().unwrap().contains("is_enabled"));
+ let txt = frame_els.iter().find(|e| e.role == "AXStaticText").unwrap();
+ assert!(txt.properties.is_none());
+
+ // search_elements path also selects `properties` and executes cleanly.
+ let (search_els, total) = db
+ .search_elements("", Some(1), None, None, None, None, None, None, 10, 0)
+ .await
+ .unwrap();
+ assert_eq!(total, 2);
+ assert!(search_els
+ .iter()
+ .any(|e| e.properties.as_deref().is_some_and(|p| p.contains("is_selected"))));
+ }
+
+ #[tokio::test]
+ async fn element_queries_run_against_real_schema_when_empty() {
+ // A typo'd column in either SELECT is a *runtime* sqlx error a compile
+ // check can't catch — assert both run on the real (migrated) schema.
+ let db = mem_db().await;
+ let (els, total) = db
+ .search_elements("", None, None, None, None, None, None, None, 5, 0)
+ .await
+ .unwrap();
+ assert!(els.is_empty());
+ assert_eq!(total, 0);
+ assert!(db.get_frame_elements(1, None).await.unwrap().is_empty());
+ }
+}
diff --git a/crates/screenpipe-db/src/types.rs b/crates/screenpipe-db/src/types.rs
index d5aed8648..641cd486f 100644
--- a/crates/screenpipe-db/src/types.rs
+++ b/crates/screenpipe-db/src/types.rs
@@ -916,6 +916,12 @@ pub struct Element {
/// implicitly true/false. See issue #2436.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub on_screen: Option<bool>,
+ /// Raw JSON of the element's automation properties (`is_enabled`,
+ /// `is_focused`, `is_selected`, `is_expanded`, `value`, `placeholder`, …)
+ /// serialized at capture. `None` for OCR rows and legacy accessibility
+ /// rows. Parsed into compact state by the API layer.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub properties: Option<String>,
}
/// Flat row for bulk insert (parent_id assigned after insert)
@@ -951,6 +957,9 @@ pub struct ElementRow {
/// (legacy rows pre-issue-#2436 fix); search treats unknown as
/// neither on- nor off-screen.
pub on_screen: Option<bool>,
+ /// Raw JSON automation properties (see [`Element::properties`]). NULL for
+ /// OCR / legacy rows.
+ pub properties: Option<String>,
}
impl From<ElementRow> for Element {
@@ -981,6 +990,7 @@ impl From<ElementRow> for Element {
confidence: row.confidence,
sort_order: row.sort_order,
on_screen: row.on_screen,
+ properties: row.properties,
}
}
}
diff --git a/crates/screenpipe-engine/src/routes/elements.rs b/crates/screenpipe-engine/src/routes/elements.rs
index 4a8aed6b0..5e7cabdeb 100644
--- a/crates/screenpipe-engine/src/routes/elements.rs
+++ b/crates/screenpipe-engine/src/routes/elements.rs
@@ -18,7 +18,9 @@ use serde_json::{json, Value};
use std::sync::Arc;
use tracing::error;
-use super::response_format::{is_passthrough, parse_fields, parse_format, render_list, rows_from};
+use super::response_format::{
+ is_passthrough, parse_fields, parse_format, render_list, rows_from, OutputFormat,
+};
use crate::server::AppState;
#[derive(OaSchema, Deserialize)]
@@ -55,13 +57,16 @@ pub(crate) struct ElementsQuery {
limit: u32,
#[serde(default)]
offset: u32,
- /// Output format: `json` (default), `csv`, or `tsv`/`table`. CSV/TSV write
- /// each column name once instead of repeating keys per row, which is far
- /// cheaper for an LLM to read (a 25-row element list drops ~73% of tokens).
+ /// Output format: `json` (default), `csv`, `tsv`/`table`, or
+ /// `outline`/`tree`. CSV/TSV write each column name once instead of
+ /// repeating keys per row. `outline` goes further for an LLM "what's on
+ /// screen?" read: a deduped, indented tree of just the text-bearing nodes
+ /// (drops empty structural nodes, bounds, and ids-other-than-`#id`).
#[serde(default)]
format: Option<String>,
/// Comma-separated column allowlist, e.g. `fields=role,text,bounds.left`.
- /// Dotted paths reach into nested objects. Omit for all fields.
+ /// Dotted paths reach into nested objects. Omit for all fields. (Ignored by
+ /// `format=outline`.)
#[serde(default)]
fields: Option<String>,
}
@@ -87,6 +92,10 @@ pub(crate) struct ElementResponse {
/// landed — see issue #2436.
#[serde(skip_serializing_if = "Option::is_none")]
pub on_screen: Option<bool>,
+ /// Compact interaction state (disabled/focused/selected/expanded) parsed
+ /// from the captured automation properties. Omitted when there's no state.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub state: Option<ElementState>,
}
#[derive(OaSchema, Serialize)]
@@ -97,6 +106,40 @@ pub(crate) struct BoundsResponse {
pub height: f64,
}
+/// Compact interaction state, parsed from the captured `properties` JSON. Only
+/// the flags actually present are emitted (so a static-text node carries no
+/// `state` at all). `disabled` is the inverse of the stored `is_enabled`.
+#[derive(OaSchema, Serialize, Debug, Clone, PartialEq, Default)]
+pub(crate) struct ElementState {
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub disabled: Option<bool>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub focused: Option<bool>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub selected: Option<bool>,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub expanded: Option<bool>,
+}
+
+/// Parse the stored automation `properties` JSON into compact state. Returns
+/// `None` unless at least one state flag is present, so non-interactive rows
+/// stay empty. Defensive: bad/absent JSON simply yields `None`.
+fn parse_element_state(properties: Option<&str>) -> Option<ElementState> {
+ let v: Value = serde_json::from_str(properties?).ok()?;
+ let b = |k: &str| v.get(k).and_then(Value::as_bool);
+ let state = ElementState {
+ disabled: b("is_enabled").map(|enabled| !enabled),
+ focused: b("is_focused"),
+ selected: b("is_selected"),
+ expanded: b("is_expanded"),
+ };
+ if state == ElementState::default() {
+ None
+ } else {
+ Some(state)
+ }
+}
+
#[derive(OaSchema, Serialize)]
pub(crate) struct PaginationResponse {
pub limit: u32,
@@ -129,8 +172,186 @@ impl From<Element> for ElementResponse {
confidence: e.confidence,
sort_order: e.sort_order,
on_screen: e.on_screen,
+ state: parse_element_state(e.properties.as_deref()),
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// `format=outline` — a compact, element-specific tree view
+//
+// CSV/TSV (in response_format) write keys once but stay flat and keep every
+// row, including the structural noise an accessibility tree is full of. For an
+// LLM asking "what's on screen?", the cheapest faithful view is a deduped,
+// indented outline of the *text-bearing* nodes. This is element-specific (it
+// understands role/text/depth/on_screen/state), so it lives here rather than in
+// the generic tabular renderer. Interaction state (disabled/selected/focused/
+// expanded) is inlined; bounds and numeric metadata (parent_id/sort_order/
+// confidence) are dropped — ask for `format=json` when you need them.
+//
+// Measured vs the JSON default (tiktoken o200k_base, `dump_token_samples`):
+//
+// | tree shape | json tok | outline | saved |
+// |---------------------|---------:|--------:|------:|
+// | small flat UI | 381 | 58 | 85% |
+// | typical app frame | 4196 | 463 | 89% |
+// | wide table (dups) | 4471 | 60 | 99% |
+// | deep nested | 1534 | 51 | 97% |
+// | OCR text-heavy | 1309 | 432 | 67% |
+// | multi-frame search | 2755 | 501 | 82% |
+// | mostly-structural | 5879 | 299 | 95% |
+// | AGGREGATE | 20525 | 1864 | 91% |
+//
+// The win tracks tree SHAPE, which tracks source: accessibility trees (rows
+// 1–4, 6–7 above) are the target — structural noise to drop, repeated
+// cells/rows to dedup, hierarchy, and state — a 6–75× cut. OCR is the floor
+// (flat text blocks: generic role, no hierarchy/state, nothing to dedup, so the
+// outline ≈ the text) — same caveat as CSV/TSV; for pure OCR `fields=text` is
+// about as good.
+// ---------------------------------------------------------------------------
+
+const OUTLINE_MAX_LINES: usize = 200;
+const OUTLINE_TEXT_CLIP: usize = 120;
+
+/// True when the caller asked for the element outline view (`format=outline`
+/// or `format=tree`). Checked before `parse_format` so those values don't trip
+/// its json|csv|tsv validator.
+pub(crate) fn wants_outline(format: &Option<String>) -> bool {
+ matches!(
+ format.as_deref().map(|s| s.trim().to_ascii_lowercase()).as_deref(),
+ Some("outline") | Some("tree")
+ )
+}
+
+/// Collapse internal whitespace and clip to `n` chars (char-safe, not byte).
+fn outline_clip(s: &str, n: usize) -> String {
+ let collapsed = s.split_whitespace().collect::<Vec<_>>().join(" ");
+ if collapsed.chars().count() > n {
+ let mut t: String = collapsed.chars().take(n).collect();
+ t.push('…');
+ t
+ } else {
+ collapsed
+ }
+}
+
+/// Render an element list as the compact outline. Pure (no I/O) so it's unit
+/// tested directly. Drops empty-text structural nodes, collapses runs of
+/// identical `role`+`text` into `×N`, hoists per-frame context into a header,
+/// indents by depth, escapes quotes in names, inlines state (off-screen +
+/// disabled/selected/focused/expanded), and caps the body with a note.
+fn elements_outline_text(elements: &[ElementResponse], total: i64) -> String {
+ // A text view's whole point: keep only nodes that carry text.
+ let kept: Vec<&ElementResponse> = elements
+ .iter()
+ .filter(|e| e.text.as_deref().map(|t| !t.trim().is_empty()).unwrap_or(false))
+ .collect();
+
+ if kept.is_empty() {
+ return "no text-bearing elements (use format=json for the raw tree)".to_string();
+ }
+
+ // Group by frame in first-seen order (search can span frames).
+ let mut frame_order: Vec<i64> = Vec::new();
+ let mut by_frame: std::collections::HashMap<i64, Vec<&ElementResponse>> =
+ std::collections::HashMap::new();
+ for e in kept.iter().copied() {
+ if !by_frame.contains_key(&e.frame_id) {
+ frame_order.push(e.frame_id);
+ by_frame.insert(e.frame_id, Vec::new());
}
+ by_frame.get_mut(&e.frame_id).unwrap().push(e);
}
+
+ let mut out = String::new();
+ let mut emitted = 0usize;
+ let mut truncated = false;
+
+ 'frames: for fid in &frame_order {
+ let els = &by_frame[fid];
+ let src = els.first().map(|e| e.source.as_str()).unwrap_or("");
+ out.push_str(&format!("frame {fid} · {src} · {} text elements\n", els.len()));
+
+ let mut i = 0;
+ while i < els.len() {
+ if emitted >= OUTLINE_MAX_LINES {
+ truncated = true;
+ break 'frames;
+ }
+ let e = els[i];
+ let text = outline_clip(e.text.as_deref().unwrap_or(""), OUTLINE_TEXT_CLIP);
+ // Collapse a run of identical (role, clipped-text) siblings — AX
+ // trees are full of repeated static text (list/table cells).
+ let mut run = 1;
+ while i + run < els.len() {
+ let n = els[i + run];
+ let ntext = outline_clip(n.text.as_deref().unwrap_or(""), OUTLINE_TEXT_CLIP);
+ if n.role == e.role && ntext == text {
+ run += 1;
+ } else {
+ break;
+ }
+ }
+ let indent = " ".repeat((e.depth.max(0) as usize).min(6));
+ // Escape quotes so the "name" delimiters stay unambiguous even when
+ // the captured text itself contains a double-quote.
+ let safe = text.replace('"', "\\\"");
+ let mut line = format!("{indent}{} \"{safe}\" #{}", e.role, e.id);
+ // Inline state — off-screen (issue #2436) + interaction flags parsed
+ // from the captured properties.
+ let mut flags: Vec<&str> = Vec::new();
+ if e.on_screen == Some(false) {
+ flags.push("off-screen");
+ }
+ if let Some(st) = &e.state {
+ if st.disabled == Some(true) {
+ flags.push("disabled");
+ }
+ if st.selected == Some(true) {
+ flags.push("selected");
+ }
+ if st.focused == Some(true) {
+ flags.push("focused");
+ }
+ match st.expanded {
+ Some(true) => flags.push("expanded"),
+ Some(false) => flags.push("collapsed"),
+ None => {}
+ }
+ }
+ if !flags.is_empty() {
+ line.push_str(&format!(" ({})", flags.join(",")));
+ }
+ if run > 1 {
+ line.push_str(&format!(" ×{run}"));
+ }
+ out.push_str(&line);
+ out.push('\n');
+ emitted += 1;
+ i += run;
+ }
+ }
+
+ if truncated || (total as usize) > emitted {
+ out.push_str(&format!(
+ "… showing {emitted} of {total} elements — narrow with ?q=, ?role=, ?on_screen=true, or ?limit=\n"
+ ));
+ }
+ out
+}
+
+/// Wrap the outline text as a `text/plain` response, carrying the total in
+/// `x-total-count` (mirrors how the delimited renderer ships pagination).
+fn elements_outline_response(elements: &[ElementResponse], total: i64) -> Response {
+ let body = elements_outline_text(elements, total);
+ Response::builder()
+ .header(
+ axum::http::header::CONTENT_TYPE,
+ "text/plain; charset=utf-8",
+ )
+ .header("x-total-count", total.to_string())
+ .body(Body::from(body))
+ .unwrap_or_else(|_| StatusCode::INTERNAL_SERVER_ERROR.into_response())
}
/// Search elements across all frames with optional FTS, time, and app filters.
@@ -139,7 +360,14 @@ pub(crate) async fn search_elements(
Query(query): Query<ElementsQuery>,
State(state): State<Arc<AppState>>,
) -> Result<Response<Body>, (StatusCode, JsonResponse<Value>)> {
- let format = parse_format(&query.format)?;
+ // `outline`/`tree` is element-specific and handled below; everything else
+ // goes through the generic json|csv|tsv negotiation.
+ let outline = wants_outline(&query.format);
+ let format = if outline {
+ OutputFormat::Json
+ } else {
+ parse_format(&query.format)?
+ };
let fields = parse_fields(&query.fields);
let q = query.q.as_deref().unwrap_or("");
let source = query
@@ -178,6 +406,9 @@ pub(crate) async fn search_elements(
total,
},
};
+ if outline {
+ return Ok(elements_outline_response(&list.data, total));
+ }
if is_passthrough(format, &fields) {
return Ok(JsonResponse(list).into_response());
}
@@ -197,7 +428,12 @@ pub(crate) async fn get_frame_elements(
Path(frame_id): Path<i64>,
Query(query): Query<FrameElementsQuery>,
) -> Result<Response<Body>, (StatusCode, JsonResponse<Value>)> {
- let format = parse_format(&query.format)?;
+ let outline = wants_outline(&query.format);
+ let format = if outline {
+ OutputFormat::Json
+ } else {
+ parse_format(&query.format)?
+ };
let fields = parse_fields(&query.fields);
let source = query
.source
@@ -225,6 +461,9 @@ pub(crate) async fn get_frame_elements(
total,
},
};
+ if outline {
+ return Ok(elements_outline_response(&list.data, total));
+ }
if is_passthrough(format, &fields) {
return Ok(JsonResponse(list).into_response());
}
@@ -237,11 +476,383 @@ pub(crate) async fn get_frame_elements(
))
}
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn el(
+ id: i64,
+ frame_id: i64,
+ role: &str,
+ text: Option<&str>,
+ depth: i32,
+ on_screen: Option<bool>,
+ ) -> ElementResponse {
+ ElementResponse {
+ id,
+ frame_id,
+ source: "accessibility".into(),
+ role: role.into(),
+ text: text.map(|s| s.to_string()),
+ parent_id: None,
+ depth,
+ bounds: Some(BoundsResponse {
+ left: 0.1,
+ top: 0.2,
+ width: 0.3,
+ height: 0.4,
+ }),
+ confidence: None,
+ sort_order: id as i32,
+ on_screen,
+ state: None,
+ }
+ }
+
+ #[test]
+ fn parse_element_state_extracts_flags_and_inverts_enabled() {
+ let s = parse_element_state(Some(
+ r#"{"is_enabled":false,"is_selected":true,"value":"x"}"#,
+ ))
+ .unwrap();
+ assert_eq!(s.disabled, Some(true)); // is_enabled:false → disabled
+ assert_eq!(s.selected, Some(true));
+ assert_eq!(s.focused, None);
+ // value/placeholder only → no interaction state
+ assert!(parse_element_state(Some(r#"{"value":"x"}"#)).is_none());
+ assert!(parse_element_state(None).is_none());
+ assert!(parse_element_state(Some("not json")).is_none());
+ }
+
+ #[test]
+ fn outline_inlines_interaction_state() {
+ let mut a = el(1, 9, "AXButton", Some("Save"), 1, Some(true));
+ a.state = Some(ElementState {
+ disabled: Some(true),
+ ..Default::default()
+ });
+ let mut b = el(2, 9, "AXTab", Some("Inbox"), 1, Some(true));
+ b.state = Some(ElementState {
+ selected: Some(true),
+ focused: Some(true),
+ ..Default::default()
+ });
+ let mut c = el(3, 9, "AXDisclosureTriangle", Some("Details"), 1, Some(true));
+ c.state = Some(ElementState {
+ expanded: Some(false),
+ ..Default::default()
+ });
+ let out = elements_outline_text(&[a, b, c], 3);
+ assert!(out.contains("AXButton \"Save\" #1 (disabled)"), "got:\n{out}");
+ assert!(out.contains("AXTab \"Inbox\" #2 (selected,focused)"), "got:\n{out}");
+ assert!(out.contains("AXDisclosureTriangle \"Details\" #3 (collapsed)"), "got:\n{out}");
+ }
+
+ #[test]
+ fn outline_escapes_quotes_in_text() {
+ let els = vec![el(1, 9, "AXStaticText", Some("say \"hi\" now"), 1, None)];
+ let out = elements_outline_text(&els, 1);
+ assert!(out.contains("\\\"hi\\\""), "quotes not escaped: {out}");
+ }
+
+ #[test]
+ fn wants_outline_matches_outline_and_tree_only() {
+ assert!(wants_outline(&Some("outline".into())));
+ assert!(wants_outline(&Some(" TREE ".into())));
+ assert!(!wants_outline(&Some("json".into())));
+ assert!(!wants_outline(&Some("csv".into())));
+ assert!(!wants_outline(&None));
+ }
+
+ #[test]
+ fn outline_drops_empty_text_structural_nodes() {
+ let els = vec![
+ el(1, 9, "AXGroup", None, 0, None),
+ el(2, 9, "AXStaticText", Some(" "), 1, None), // whitespace only
+ el(3, 9, "AXButton", Some("Submit"), 1, None),
+ ];
+ let out = elements_outline_text(&els, 3);
+ assert!(out.contains("AXButton \"Submit\" #3"), "got:\n{out}");
+ assert!(!out.contains("AXGroup"));
+ assert!(!out.contains("#1"));
+ assert!(!out.contains("#2"));
+ }
+
+ #[test]
+ fn outline_collapses_identical_runs_into_count() {
+ let els = vec![
+ el(1, 9, "AXCell", Some("Active"), 2, None),
+ el(2, 9, "AXCell", Some("Active"), 2, None),
+ el(3, 9, "AXCell", Some("Active"), 2, None),
+ el(4, 9, "AXCell", Some("Inactive"), 2, None),
+ ];
+ let out = elements_outline_text(&els, 4);
+ assert!(out.contains("AXCell \"Active\" #1 ×3"), "got:\n{out}");
+ assert!(out.contains("AXCell \"Inactive\" #4"));
+ assert!(!out.contains("#2"));
+ }
+
+ #[test]
+ fn outline_flags_off_screen_and_drops_bounds() {
+ let els = vec![el(7, 9, "AXTextField", Some("Email"), 1, Some(false))];
+ let out = elements_outline_text(&els, 1);
+ assert!(out.contains("AXTextField \"Email\" #7 (off-screen)"), "got:\n{out}");
+ // bounds (floats / keys) must not bloat the outline
+ assert!(!out.contains("0.1"));
+ assert!(!out.contains("left"));
+ }
+
+ #[test]
+ fn outline_groups_by_frame_with_headers_in_first_seen_order() {
+ let els = vec![
+ el(1, 100, "AXButton", Some("A"), 0, None),
+ el(2, 200, "AXButton", Some("B"), 0, None),
+ ];
+ let out = elements_outline_text(&els, 2);
+ assert!(out.contains("frame 100 · accessibility · 1 text elements"), "got:\n{out}");
+ assert!(out.contains("frame 200 · accessibility · 1 text elements"));
+ assert!(out.find("#1").unwrap() < out.find("#2").unwrap());
+ }
+
+ #[test]
+ fn outline_caps_body_and_notes_truncation() {
+ let els: Vec<ElementResponse> = (0..(OUTLINE_MAX_LINES as i64 + 50))
+ .map(|i| el(i + 1, 9, "AXStaticText", Some(&format!("line {i}")), 1, None))
+ .collect();
+ let total = els.len() as i64;
+ let out = elements_outline_text(&els, total);
+ assert!(out.contains("showing"));
+ assert!(out.contains(&format!("of {total}")));
+ let emitted = out.lines().filter(|l| l.contains('#')).count();
+ assert!(emitted <= OUTLINE_MAX_LINES, "emitted {emitted} > cap");
+ }
+
+ #[test]
+ fn outline_collapses_whitespace_and_clips_long_text() {
+ let long = "x".repeat(OUTLINE_TEXT_CLIP + 40);
+ let els = vec![
+ el(1, 9, "AXStaticText", Some("a\n b\t c"), 1, None),
+ el(2, 9, "AXStaticText", Some(&long), 1, None),
+ ];
+ let out = elements_outline_text(&els, 2);
+ assert!(out.contains("\"a b c\""), "got:\n{out}");
+ assert!(out.contains('…'));
+ assert!(!out.contains(&long));
+ }
+
+ #[test]
+ fn outline_is_far_cheaper_than_json() {
+ // Distinct text so the win is the format itself, not just dedup.
+ let els: Vec<ElementResponse> = (0..25)
+ .map(|i| el(i + 1, 9, "AXButton", Some(&format!("Button {i}")), 2, Some(true)))
+ .collect();
+ let outline = elements_outline_text(&els, 25);
+ let json = serde_json::to_string(&els).unwrap();
+ assert!(
+ outline.len() < json.len() / 2,
+ "outline {} vs json {} bytes",
+ outline.len(),
+ json.len()
+ );
+ }
+
+ #[test]
+ fn outline_handles_no_text_bearing_input() {
+ let els = vec![el(1, 9, "AXGroup", None, 0, None)];
+ assert!(elements_outline_text(&els, 1).contains("no text-bearing elements"));
+ assert!(elements_outline_text(&[], 0).contains("no text-bearing elements"));
+ }
+
+ #[test]
+ fn outline_is_char_safe_with_unicode_and_clips_by_char() {
+ // Multi-byte text must clip on char boundaries (never panic) and the
+ // … marker must appear without slicing a code point.
+ let emoji = "🎉".repeat(OUTLINE_TEXT_CLIP + 30);
+ let els = vec![
+ el(1, 9, "AXStaticText", Some("café — naïve 日本語 😀"), 1, None),
+ el(2, 9, "AXStaticText", Some(&emoji), 1, None),
+ ];
+ let out = elements_outline_text(&els, 2);
+ assert!(out.contains("café — naïve 日本語 😀"));
+ assert!(out.contains('…'));
+ // clipped to OUTLINE_TEXT_CLIP code points (+ the … marker)
+ let clipped_line = out.lines().find(|l| l.contains("🎉")).unwrap();
+ let emoji_count = clipped_line.chars().filter(|&c| c == '🎉').count();
+ assert_eq!(emoji_count, OUTLINE_TEXT_CLIP);
+ }
+
+ #[test]
+ fn outline_caps_indent_for_deeply_nested() {
+ let els = vec![el(1, 9, "AXButton", Some("Deep"), 30, Some(true))];
+ let out = elements_outline_text(&els, 1);
+ let line = out.lines().find(|l| l.contains("Deep")).unwrap();
+ let leading = line.len() - line.trim_start().len();
+ assert_eq!(leading, 12, "indent should cap at 6 levels × 2 spaces"); // not 60
+ }
+
+ #[test]
+ fn outline_dedup_is_adjacent_only_preserving_structure() {
+ // A A B A — the trailing A must NOT merge with the leading run.
+ let els = vec![
+ el(1, 9, "AXTab", Some("A"), 1, None),
+ el(2, 9, "AXTab", Some("A"), 1, None),
+ el(3, 9, "AXTab", Some("B"), 1, None),
+ el(4, 9, "AXTab", Some("A"), 1, None),
+ ];
+ let out = elements_outline_text(&els, 4);
+ assert!(out.contains("AXTab \"A\" #1 ×2"), "got:\n{out}");
+ assert!(out.contains("AXTab \"B\" #3"));
+ assert!(out.contains("AXTab \"A\" #4"));
+ assert!(!out.contains("#4 ×"));
+ }
+
+ #[test]
+ fn outline_buckets_interleaved_frames() {
+ // Search can return frames interleaved; the outline buckets each frame
+ // (first-seen order) so the model reads one coherent frame at a time.
+ let els = vec![
+ el(1, 100, "AXButton", Some("one"), 0, None),
+ el(2, 200, "AXButton", Some("two"), 0, None),
+ el(3, 100, "AXButton", Some("three"), 0, None),
+ ];
+ let out = elements_outline_text(&els, 3);
+ assert!(out.contains("frame 100 · accessibility · 2 text elements"), "got:\n{out}");
+ assert!(out.contains("frame 200 · accessibility · 1 text elements"));
+ // #1 and #3 (frame 100) precede #2 (frame 200)
+ assert!(out.find("#3").unwrap() < out.find("#2").unwrap());
+ }
+
+ #[test]
+ fn outline_keeps_one_line_per_element_through_quotes_and_newlines() {
+ let els = vec![el(
+ 1,
+ 9,
+ "AXStaticText",
+ Some("he said \"hi\"\nthen left"),
+ 1,
+ None,
+ )];
+ let out = elements_outline_text(&els, 1);
+ // exactly one frame header + one element line (+ trailing newline)
+ let body: Vec<&str> = out.lines().filter(|l| l.contains('#')).collect();
+ assert_eq!(body.len(), 1, "embedded newline split the row: {out:?}");
+ // newline collapsed to a space; embedded quotes escaped
+ assert!(body[0].contains("he said \\\"hi\\\" then left"), "got: {}", body[0]);
+ }
+
+ // Reproducible token-measurement harness. Renders JSON-default vs outline
+ // for several realistic tree shapes into a temp file so an external
+ // tokenizer can score them:
+ // cargo test -p screenpipe-engine dump_token_samples -- --ignored
+ // python3 -c "import json,tiktoken;e=tiktoken.get_encoding('o200k_base');\
+ // [print(s['name'], len(e.encode(s['json'])), len(e.encode(s['outline']))) \
+ // for s in json.load(open('<tmp>/sp_elem_samples.json'))]"
+ // Measured (o200k_base): 91% fewer tokens aggregate (11×); per shape 67%
+ // (OCR text-heavy) … 99% (tabular dedup). #[ignore] so CI never does the IO.
+ #[test]
+ #[ignore]
+ fn dump_token_samples() {
+ fn collect(name: &str, els: Vec<ElementResponse>) -> serde_json::Value {
+ let total = els.len() as i64;
+ let json = serde_json::to_string(&serde_json::json!({
+ "data": &els,
+ "pagination": { "limit": total, "offset": 0, "total": total }
+ }))
+ .unwrap();
+ let outline = elements_outline_text(&els, total);
+ serde_json::json!({ "name": name, "elements": els.len(), "json": json, "outline": outline })
+ }
+
+ let mut samples = Vec::new();
+
+ // 1. small flat UI
+ samples.push(collect(
+ "small_flat",
+ vec![
+ el(1, 1, "AXButton", Some("Compose"), 1, Some(true)),
+ el(2, 1, "AXLink", Some("Inbox"), 1, Some(true)),
+ el(3, 1, "AXLink", Some("Sent"), 1, Some(true)),
+ el(4, 1, "AXTextField", Some("Search mail"), 1, Some(true)),
+ el(5, 1, "AXButton", Some("Settings"), 1, Some(true)),
+ ],
+ ));
+
+ // 2. typical app frame: real text + structural noise + some dup text
+ let mut typ = vec![
+ el(1, 7, "AXHeading", Some("Inbox"), 1, Some(true)),
+ el(2, 7, "AXButton", Some("Compose"), 1, Some(true)),
+ el(3, 7, "AXTextField", Some("Search"), 1, Some(true)),
+ ];
+ for i in 0..18 {
+ typ.push(el(100 + i * 3, 7, "AXGroup", None, 2, Some(true))); // empty noise
+ typ.push(el(101 + i * 3, 7, "AXStaticText", Some(&format!("Sender {i}")), 4, Some(true)));
+ typ.push(el(102 + i * 3, 7, "AXStaticText", Some("Unread"), 4, Some(true))); // dup
+ }
+ samples.push(collect("typical_app", typ));
+
+ // 3. wide table: 60 rows of repeated cells (dedup territory)
+ let mut table = vec![el(1, 3, "AXHeading", Some("Orders"), 1, Some(true))];
+ for i in 0..60 {
+ table.push(el(10 + i, 3, "AXCell", Some("Shipped"), 3, Some(true)));
+ }
+ samples.push(collect("wide_table_dups", table));
+
+ // 4. deep nested chain
+ let mut deep = Vec::new();
+ for d in 0..20 {
+ deep.push(el(d as i64 + 1, 4, "AXGroup", None, d, Some(true)));
+ }
+ deep.push(el(999, 4, "AXButton", Some("Deeply nested action"), 20, Some(true)));
+ samples.push(collect("deep_nested", deep));
+
+ // 5. OCR-heavy long text
+ let mut ocr = Vec::new();
+ for i in 0..15 {
+ let mut e = el(
+ i + 1,
+ 5,
+ "text",
+ Some(&format!("This is a longer sentence of captured OCR text number {i} that a real screen would contain in a paragraph.")),
+ 1,
+ Some(true),
+ );
+ e.source = "ocr".into();
+ e.on_screen = None;
+ ocr.push(e);
+ }
+ samples.push(collect("ocr_heavy", ocr));
+
+ // 6. multi-frame search result
+ let mut multi = Vec::new();
+ for f in 0..3 {
+ for i in 0..12 {
+ multi.push(el(f * 100 + i, 200 + f, "AXLink", Some(&format!("Result {f}-{i}")), 2, Some(true)));
+ }
+ }
+ samples.push(collect("multi_frame", multi));
+
+ // 7. mostly-structural noise (60 empty groups, 20 text)
+ let mut noisy = Vec::new();
+ for i in 0..60 {
+ noisy.push(el(i + 1, 6, "AXGroup", None, (i % 8) as i32, Some(true)));
+ }
+ for i in 0..20 {
+ noisy.push(el(1000 + i, 6, "AXStaticText", Some(&format!("Label {i}")), 3, Some(true)));
+ }
+ samples.push(collect("noisy_structural", noisy));
+
+ let path = std::env::temp_dir().join("sp_elem_samples.json");
+ std::fs::write(&path, serde_json::to_string_pretty(&samples).unwrap()).unwrap();
+ eprintln!("wrote {} samples to {}", samples.len(), path.display());
+ }
+}
+
#[derive(OaSchema, Deserialize)]
pub(crate) struct FrameElementsQuery {
#[serde(default)]
source: Option<String>,
- /// Output format: `json` (default), `csv`, or `tsv`/`table`.
+ /// Output format: `json` (default), `csv`, `tsv`/`table`, or
+ /// `outline`/`tree` (compact text tree — the cheapest full-frame read).
#[serde(default)]
format: Option<String>,
/// Comma-separated column allowlist, e.g. `fields=role,text,bounds.left`.
diff --git a/packages/screenpipe-mcp/README.md b/packages/screenpipe-mcp/README.md
index 8498dc93f..bae8dad7f 100644
--- a/packages/screenpipe-mcp/README.md
+++ b/packages/screenpipe-mcp/README.md
@@ -154,6 +154,12 @@ List detected meetings with duration, app, and attendees. Pass `q` to filter by
Search structured UI elements (accessibility tree nodes and OCR text blocks):
- Filter by source, role, app, time range
- Much lighter than search-content for targeted UI lookups
+- Returns a compact `outline` view by default — a deduped, indented tree of the
+ text-bearing nodes (`#id` refs, `(off-screen)` flags), ~91% fewer tokens than
+ raw element JSON
+
+### get-frame-elements
+The whole element tree for one frame, as the same compact outline.
### frame-context
Get accessibility text, parsed tree nodes, and extracted URLs for a specific frame.
@@ -179,7 +185,7 @@ Get accessibility text, parsed tree nodes, and extracted URLs for a specific fra
- All timestamps are handled in UTC
- Results are formatted for readability in Claude's interface
- macOS automation features require accessibility permissions
-- The MCP tools already return compact, readable text. If you instead call the underlying screenpipe REST API directly (e.g. via `curl`), the list endpoints (`/search`, `/elements`, `/frames/{id}/elements`) accept `?format=csv|tsv` for a columnar table (column names written once) and `?fields=a,b,c` to select only the columns you need (dotted paths like `content.text`). On list-shaped results that is roughly a 70% token cut versus the default JSON, which stays unchanged when neither param is set.
+- The MCP tools already return compact, readable text (the element tools default to the `outline` view). If you instead call the underlying screenpipe REST API directly (e.g. via `curl`), the list endpoints (`/search`, `/elements`, `/frames/{id}/elements`) accept `?format=csv|tsv` for a columnar table (column names written once) and `?fields=a,b,c` to select only the columns you need (dotted paths like `content.text`); the element endpoints also accept `?format=outline` (the same tree the MCP tools return, ~91% fewer tokens than JSON). On list-shaped results that is a 70–91% token cut versus the default JSON, which stays unchanged when no param is set.
## Privacy Policy
diff --git a/packages/screenpipe-mcp/src/index.ts b/packages/screenpipe-mcp/src/index.ts
index b48e24647..83ce8696b 100644
--- a/packages/screenpipe-mcp/src/index.ts
+++ b/packages/screenpipe-mcp/src/index.ts
@@ -1537,48 +1537,24 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
}
}
- const response = await callAPI(`/elements?${params.toString()}`);
-
- const data = await response.json();
- const elements = data.data || [];
- const pagination = data.pagination || {};
-
- if (elements.length === 0) {
- return {
- content: [
- {
- type: "text",
- text: "No elements found. Try: broader search, different role/source, or wider time range.",
- },
- ],
- };
- }
-
- const formatted = elements.map(
- (e: {
- id: number;
- frame_id: number;
- source: string;
- role: string;
- text: string | null;
- depth: number;
- bounds: { left: number; top: number; width: number; height: number } | null;
- }) => {
- const boundsStr = e.bounds
- ? ` [${e.bounds.left.toFixed(2)},${e.bounds.top.toFixed(2)} ${e.bounds.width.toFixed(2)}x${e.bounds.height.toFixed(2)}]`
- : "";
- return `[${e.source}] ${e.role} (frame:${e.frame_id}, depth:${e.depth})${boundsStr}\n ${e.text || "(no text)"}`;
- }
- );
+ // Default to the server's compact `outline` view — a deduped, indented
+ // tree of just the text-bearing nodes, far cheaper for the model to read
+ // than the raw JSON rows (and the dedup/cap/footer replace the old
+ // hand-rolled header). Callers can still override with format=json|csv|tsv.
+ if (!params.has("format")) params.append("format", "outline");
- const header =
- `Elements: ${elements.length}/${pagination.total || "?"}` +
- (pagination.total > elements.length
- ? ` (use offset=${(pagination.offset || 0) + elements.length} for more)`
- : "");
+ const response = await callAPI(`/elements?${params.toString()}`);
+ const text = (await response.text()).trim();
return {
- content: [{ type: "text", text: header + "\n\n" + formatted.join("\n---\n") }],
+ content: [
+ {
+ type: "text",
+ text: text.length
+ ? text
+ : "No elements found. Try: broader search, different role/source, or wider time range.",
+ },
+ ],
};
}
@@ -1997,19 +1973,19 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
if (!frameId) {
return { content: [{ type: "text", text: "Error: frame_id is required" }] };
}
- const response = await callAPI(`/frames/${frameId}/elements`);
- const elements = await response.json();
- if (!Array.isArray(elements) || elements.length === 0) {
- return { content: [{ type: "text", text: `No elements found for frame ${frameId}.` }] };
- }
- const formatted = elements.map(
- (e: { role: string; text: string | null; depth: number; source: string }) => {
- const indent = " ".repeat(Math.min(e.depth, 5));
- return `${indent}[${e.source}:${e.role}] ${e.text || "(no text)"}`;
- }
- );
+ // Compact outline (text/plain): drops structural noise, dedups repeated
+ // rows, caps the body. Also avoids the old bug here that parsed the
+ // `{data,pagination}` envelope as a bare array and always reported
+ // "no elements".
+ const response = await callAPI(`/frames/${frameId}/elements?format=outline`);
+ const text = (await response.text()).trim();
return {
- content: [{ type: "text", text: `Frame ${frameId} elements (${elements.length}):\n${formatted.join("\n")}` }],
+ content: [
+ {
+ type: "text",
+ text: text.length ? text : `No elements found for frame ${frameId}.`,
+ },
+ ],
};
}