Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

eval: cover redaction progress across missing schema targets (#7485)

Louis Beaumonta8a2c3748713
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.md
index ed0e372fe..c5d572e84 100644
--- a/evals/coding-agent/README.md
+++ b/evals/coding-agent/README.md
@@ -67,7 +67,7 @@ The hidden fixture appears only at grading and uses no dependency links.
This does not establish native recording continuity, real authentication,
account-switch race safety, browser integration, isolation or model performance.
-The current app corpus contains 132 git-mined regressions. See
+The current app corpus contains 133 git-mined regressions. See
[DESIGN.md](./DESIGN.md) for the Anthropic guidance, source contract, and
history-mining workflow. The companion website manifest uses this same harness.
@@ -1668,3 +1668,22 @@ for correct, broken, equivalent, disconnected, partial-repair, ignored-filter,
blanket-filter and missing-source controls. Hidden fixtures appear only at grading.
No build-cache links are declared. This does not establish sync ingestion, HTTP
routing, hybrid storage, concurrent writes, execution isolation or model quality.
+
+## Redaction across missing schema targets
+
+`app-redact-missing-schema-target` runs the actual native Worker with local regex
+redaction and synthetic temporary SQLite data. Six outcomes cover missing tables
+and columns, later-row progress, ordinary redaction, clean-text preservation,
+pause/resume, transient-write retry and corruption-error backoff with shutdown.
+The parent fails the two missing-schema outcomes and preserves four; the
+historical reference and current source pass all six. The task discloses the
+small-workload timing bounds and does not require an exact implementation.
+
+Run the shared runner with `--case app-redact-missing-schema-target --verify`.
+The hidden fixture appears only at grading; no dependency or target-cache links
+are declared. Calibration rejects unused correct code, blanket skipping, treating
+all errors as missing schema and lost pause behavior, while accepting an
+equivalent missing-object predicate. Compilation/setup failures are infrastructure
+errors. The malformed-database error is supplied by a synthetic SQLite trigger,
+not an actually corrupt database. This does not establish migration safety,
+capture continuity, real corruption recovery, host isolation or model capability.
diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.json
index f7085b3ef..0354f0427 100644
--- a/evals/coding-agent/cases.json
+++ b/evals/coding-agent/cases.json
@@ -5478,6 +5478,42 @@
],
"command": "cargo test --offline --locked -p screenpipe-db --test eval_memory_tag_filter -- --test-threads=1"
}
+ },
+ {
+ "id": "app-redact-missing-schema-target",
+ "name": "Keep redaction progressing across missing schema targets",
+ "source": {
+ "kind": "git_regression",
+ "fix_commit": "ac94fa0e26073ca9854a3695a308fb76d917172e",
+ "pull_request": "https://github.com/screenpipe/screenpipe/pull/4315"
+ },
+ "base_ref": "84143f2402bb9d64331427233d3ab321798b7e19",
+ "oracle_ref": "ac94fa0e26073ca9854a3695a308fb76d917172e",
+ "oracle_paths": [
+ "crates/screenpipe-redact/src/worker/mod.rs"
+ ],
+ "gate": "advisory",
+ "trigger_paths": [
+ "crates/screenpipe-redact/src/**"
+ ],
+ "tags": [
+ "regression",
+ "escaped-failure",
+ "privacy",
+ "schema-compatibility"
+ ],
+ "prompt": "The background redaction Worker stalls otherwise valid targets when an earlier target table or required column is absent from the database schema. Repair the actual worker so missing-schema targets do not delay redaction of unaffected queued rows or rows inserted during later sweeps. Tests use a small synthetic SQLite database, local regex redaction, a 20ms poll interval and a 1ms minimum batch idle. With a missing elements table or its completion column before full-text work, existing and newly added full-text rows must lose their synthetic email and gain a completion marker within 700ms; preserve the nearby contact word and leave clean text unchanged while marking it complete. Keep ordinary full-text redaction working. Pause before startup must prevent effects while paused, and resume must allow pending work within 700ms. A transient rejected update must retain its original retry text and unset completion marker; once the refusal is removed after 250ms, the same worker must successfully retry within another 2400ms. Preserve the existing database-corruption backoff: a SQLite error reporting a malformed database must prevent subsequent audio-target processing during the initial 700ms, while a shutdown notification must still stop the worker within one second. Do not classify every error as a missing object, drop all work or disable pause controls. No particular helper, disabled-target collection, log wording or replacement token is required. This task concerns background reconciliation of existing synthetic rows, not schema migration, real corruption recovery or capture admission. Preserve recording authorization and durable capture; do not block or discard captures to satisfy background work. Do not contact live model services.",
+ "agent_timeout_seconds": 1800,
+ "grader": {
+ "timeout_seconds": 180,
+ "fixtures": [
+ {
+ "local_path": "graders/redaction-schema-progress.rs",
+ "destination_path": "crates/screenpipe-redact/tests/eval_missing_schema.rs"
+ }
+ ],
+ "command": "cargo test --locked -p screenpipe-redact --test eval_missing_schema -- --test-threads=1"
+ }
}
]
}
diff --git a/evals/coding-agent/graders/redaction-schema-progress.rs b/evals/coding-agent/graders/redaction-schema-progress.rs
new file mode 100644
index 000000000..f336d32f9
--- /dev/null
+++ b/evals/coding-agent/graders/redaction-schema-progress.rs
@@ -0,0 +1,172 @@
+// screenpipe — AI that knows everything you've seen, said, or heard
+// https://screenpipe.com
+use screenpipe_redact::{
+ adapters::regex::RegexRedactor,
+ worker::{TargetTable, Worker, WorkerConfig},
+ Redactor,
+};
+use sqlx::{
+ sqlite::{SqliteConnectOptions, SqlitePoolOptions},
+ Row,
+};
+use std::{sync::Arc, time::Duration};
+use tempfile::TempDir;
+use tokio::sync::Notify;
+
+async fn db(elements: &'static str) -> (sqlx::SqlitePool, TempDir) {
+ let temp = tempfile::tempdir().unwrap();
+ let pool = SqlitePoolOptions::new()
+ .max_connections(2)
+ .connect_with(
+ SqliteConnectOptions::new()
+ .filename(temp.path().join("synthetic.sqlite"))
+ .create_if_missing(true),
+ )
+ .await
+ .unwrap();
+ sqlx::query("CREATE TABLE frames(id INTEGER PRIMARY KEY, full_text TEXT, full_text_redacted_at INTEGER, accessibility_text TEXT, accessibility_redacted_at INTEGER, accessibility_tree_json TEXT, accessibility_tree_redacted_at INTEGER, window_name TEXT, window_name_redacted_at INTEGER, browser_url TEXT, browser_url_redacted_at INTEGER, text_json TEXT, text_json_redacted_at INTEGER); CREATE TABLE audio_transcriptions(id INTEGER PRIMARY KEY, transcription TEXT, redacted_at INTEGER);").execute(&pool).await.unwrap();
+ if !elements.is_empty() {
+ sqlx::query(elements).execute(&pool).await.unwrap();
+ }
+ sqlx::query("INSERT INTO frames(id,full_text) VALUES(1,'contact [email protected] today'),(2,'ordinary clean words'); INSERT INTO audio_transcriptions VALUES(1,'audio [email protected]',NULL);").execute(&pool).await.unwrap();
+ (pool, temp)
+}
+fn worker(pool: &sqlx::SqlitePool, tables: Vec<TargetTable>) -> Worker {
+ Worker::new(
+ pool.clone(),
+ Arc::new(RegexRedactor::new()) as Arc<dyn Redactor>,
+ WorkerConfig {
+ tables,
+ batch_size: 16,
+ idle_between_batches: Duration::from_millis(1),
+ poll_interval: Duration::from_millis(20),
+ ..Default::default()
+ },
+ )
+}
+async fn frame(pool: &sqlx::SqlitePool, id: i64) -> (String, Option<i64>) {
+ let r = sqlx::query("SELECT full_text,full_text_redacted_at FROM frames WHERE id=?")
+ .bind(id)
+ .fetch_one(pool)
+ .await
+ .unwrap();
+ (r.get(0), r.get(1))
+}
+async fn stop(n: Arc<Notify>, h: tokio::task::JoinHandle<()>) {
+ n.notify_waiters();
+ tokio::time::timeout(Duration::from_secs(1), h)
+ .await
+ .expect("shutdown must finish")
+ .unwrap();
+}
+async fn assert_missing(elements: &'static str) {
+ let (pool, _temp) = db(elements).await;
+ let w = worker(&pool, vec![TargetTable::Elements, TargetTable::FullText]);
+ let n = Arc::new(Notify::new());
+ let h = w.spawn_with_shutdown(n.clone());
+ tokio::time::sleep(Duration::from_millis(700)).await;
+ let first = frame(&pool, 1).await;
+ let clean = frame(&pool, 2).await;
+ sqlx::query("INSERT INTO frames(id,full_text) VALUES(3,'later [email protected]')")
+ .execute(&pool)
+ .await
+ .unwrap();
+ tokio::time::sleep(Duration::from_millis(700)).await;
+ let later = frame(&pool, 3).await;
+ stop(n, h).await;
+ assert!(
+ !first.0.contains("[email protected]")
+ && first.0.contains("contact")
+ && first.1.is_some(),
+ "missing target delayed initial redaction: {first:?}"
+ );
+ assert_eq!(clean.0, "ordinary clean words");
+ assert!(clean.1.is_some());
+ assert!(
+ !later.0.contains("[email protected]") && later.1.is_some(),
+ "later sweep lost progress: {later:?}"
+ );
+}
+#[tokio::test]
+async fn missing_table_preserves_progress() {
+ assert_missing("").await;
+}
+#[tokio::test]
+async fn missing_column_preserves_progress() {
+ assert_missing("CREATE TABLE elements(id INTEGER PRIMARY KEY, text TEXT, properties TEXT);")
+ .await;
+}
+#[tokio::test]
+async fn ordinary_redaction_and_clean_words() {
+ let (pool, _temp) = db("").await;
+ let w = worker(&pool, vec![TargetTable::FullText]);
+ let n = Arc::new(Notify::new());
+ let h = w.spawn_with_shutdown(n.clone());
+ tokio::time::sleep(Duration::from_millis(700)).await;
+ let a = frame(&pool, 1).await;
+ let b = frame(&pool, 2).await;
+ stop(n, h).await;
+ assert!(!a.0.contains("[email protected]") && a.1.is_some());
+ assert_eq!(b.0, "ordinary clean words");
+ assert!(b.1.is_some());
+}
+#[tokio::test]
+async fn pause_resume_preserved() {
+ let (pool, _temp) = db("").await;
+ let w = worker(&pool, vec![TargetTable::FullText]);
+ w.pause();
+ let n = Arc::new(Notify::new());
+ let h = w.clone().spawn_with_shutdown(n.clone());
+ tokio::time::sleep(Duration::from_millis(150)).await;
+ let paused = frame(&pool, 1).await;
+ w.resume();
+ tokio::time::sleep(Duration::from_millis(700)).await;
+ let resumed = frame(&pool, 1).await;
+ stop(n, h).await;
+ assert!(paused.0.contains("[email protected]") && paused.1.is_none());
+ assert!(!resumed.0.contains("[email protected]") && resumed.1.is_some());
+}
+#[tokio::test]
+async fn transient_write_error_retries() {
+ let (pool, _temp) = db("").await;
+ sqlx::query("CREATE TRIGGER fail_write BEFORE UPDATE ON frames BEGIN SELECT RAISE(ABORT,'synthetic transient write refusal'); END;").execute(&pool).await.unwrap();
+ let w = worker(&pool, vec![TargetTable::FullText]);
+ let n = Arc::new(Notify::new());
+ let h = w.spawn_with_shutdown(n.clone());
+ tokio::time::sleep(Duration::from_millis(250)).await;
+ let refused = frame(&pool, 1).await;
+ sqlx::query("DROP TRIGGER fail_write")
+ .execute(&pool)
+ .await
+ .unwrap();
+ tokio::time::sleep(Duration::from_millis(2400)).await;
+ let retry = frame(&pool, 1).await;
+ stop(n, h).await;
+ assert!(refused.0.contains("[email protected]") && refused.1.is_none());
+ assert!(
+ !retry.0.contains("[email protected]") && retry.1.is_some(),
+ "transient target lost retry: {retry:?}"
+ );
+}
+#[tokio::test]
+async fn hard_fault_preserves_backoff_and_shutdown() {
+ let (pool, _temp) = db("").await;
+ sqlx::query("CREATE TRIGGER fail_write BEFORE UPDATE ON frames BEGIN SELECT RAISE(ABORT,'database disk image is malformed'); END;").execute(&pool).await.unwrap();
+ let w = worker(
+ &pool,
+ vec![TargetTable::FullText, TargetTable::AudioTranscription],
+ );
+ let n = Arc::new(Notify::new());
+ let h = w.spawn_with_shutdown(n.clone());
+ tokio::time::sleep(Duration::from_millis(700)).await;
+ let audio: Option<i64> =
+ sqlx::query_scalar("SELECT redacted_at FROM audio_transcriptions WHERE id=1")
+ .fetch_one(&pool)
+ .await
+ .unwrap();
+ stop(n, h).await;
+ assert!(
+ audio.is_none(),
+ "hard fault must not silently continue to the next target"
+ );
+}