Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

fix(pipes): run event-triggered pipes once per event

Anshgrover23ad6760368779
diff --git a/TESTING.md b/TESTING.md
index 012912fb5..4c218f95e 100644
--- a/TESTING.md
+++ b/TESTING.md
@@ -699,6 +699,8 @@ commits: `fa887407`, `815f52e6`, `60840155`, `e66c3ff8`, `c905ffbf`, `01147096`,
- [ ] **Pi agent & search timeouts** — Run a long-running search or Pi agent task. Verify it doesn't timeout prematurely at 60s (should allow up to 120s for search). (`f01213cf5`)
- [ ] **allow schedule + triggers together in pipe config UI** — Verify that both schedule and triggers can be configured together in the pipe config UI without conflicts. (`f32bf9230`)
- [ ] **local event triggers for pipes** — Verify that pipes can be triggered by local events such as meeting start/end and pipe chaining. (`508b68ff7`, `776587aa7`)
+- [ ] **one event-triggered run per meeting** — With meeting-summary enabled, join a call, leave, rejoin the same app within 120s (this reopens the same meeting row), then leave again. `meeting_ended` fires twice for the same id; the pipe must run once. verify: `grep "suppressed duplicate 'meeting_ended'" ~/.screenpipe/screenpipe-app.*.log` and `sqlite3 ~/.screenpipe/db.sqlite "SELECT id, trigger_event, trigger_key FROM pipe_executions WHERE pipe_name='meeting-summary' ORDER BY id DESC LIMIT 5"` shows one row per meeting id. (#5481)
+- [ ] **failed event run retries** — Force an event-triggered run to fail (e.g. sign out of the preset's provider mid-run). Its claim must be released so the next delivery of that event runs. verify: `sqlite3 ~/.screenpipe/db.sqlite "SELECT * FROM pipe_event_runs"` no longer holds the failed key. (#5481)
- [ ] **tool call indicators in pipe run output** — Verify that tool call indicators are displayed in the pipe run output. (`dda33a6a7`)
- [ ] **align copy and chat icons in pipe run history** — Verify that copy and chat icons are properly aligned in the pipe run history. (`f8fe5cff4`)
- [ ] **Live pipe output streaming** — Open the "Runs" tab for a running pipe. Verify that the output streams live as the pipe executes. (`4c1fab276`)
diff --git a/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md b/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md
index ff7cf63eb..37fbb3328 100644
--- a/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md
+++ b/crates/screenpipe-core/assets/pipes/meeting-summary/pipe.md
@@ -30,12 +30,19 @@ keep the wording of this prompt in sync with `buildMeetingSummarizeInstructions`
read the screenpipe skill first so you know the meetings + search endpoints.
-step 1 — find the meeting that just ended:
+step 1 — find the meeting that just ended. when the scheduler woke you for an event it wrote `./.trigger-context.json` in this pipe's folder; read it first and use the meeting id it names:
+
+ cat ./.trigger-context.json # {"event": "meeting_ended", "key": "<MEETING_ID>", ...}
+
+ curl -s -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \
+ "http://localhost:3030/meetings/<MEETING_ID>"
+
+only if that file is missing (a manual run) fall back to the most recent row:
curl -s -H "Authorization: Bearer $SCREENPIPE_LOCAL_API_KEY" \
"http://localhost:3030/meetings?limit=1"
-the most recent row is the one that just ended. capture its `id`, `meeting_start`, `meeting_end`, `title`, `note`, `meeting_app`, and `attendees`.
+either way, capture the meeting's `id`, `meeting_start`, `meeting_end`, `title`, `note`, `meeting_app`, and `attendees`. prefer the id from the trigger file — "most recent" picks the wrong meeting when two end close together.
step 2 — search screenpipe for what happened during this meeting and summarize it: key topics, decisions, action items. scope your searches to the meeting's `meeting_start`/`meeting_end` window. prefer `content_type=audio` for transcripts.
diff --git a/crates/screenpipe-core/src/pipes/mod.rs b/crates/screenpipe-core/src/pipes/mod.rs
index 49b0c8d37..fd34f5ac6 100644
--- a/crates/screenpipe-core/src/pipes/mod.rs
+++ b/crates/screenpipe-core/src/pipes/mod.rs
@@ -125,6 +125,16 @@ pub fn install_bundled_pipe(pipes_dir: &Path, name: &str) -> Result<bool> {
/// reported on `pipe_scheduled_run` so the limit's behavior is observable.
const EVENT_TRIGGERED_CONCURRENCY_LIMIT: usize = 4;
+/// How long an event-run claim is remembered. Claims exist to suppress duplicate
+/// deliveries of one event, so this only has to outlive the delivery window —
+/// a week is generous and keeps the table small.
+const EVENT_CLAIM_RETENTION_DAYS: i64 = 7;
+
+/// Cap on triggers held back for a later tick. Reached only if a pipe stays busy
+/// while its events keep arriving; beyond this the oldest are dropped rather than
+/// growing without bound.
+const MAX_CARRYOVER_EVENTS: usize = 64;
+
// ---------------------------------------------------------------------------
// Config & log types
// ---------------------------------------------------------------------------
@@ -1161,6 +1171,10 @@ pub struct PipeExecution {
pub error_message: Option<String>,
pub duration_ms: Option<i64>,
pub session_path: Option<String>,
+ /// Event that triggered this run, and that event's identity (e.g. a meeting
+ /// id). Both None for scheduled and manual runs.
+ pub trigger_event: Option<String>,
+ pub trigger_key: Option<String>,
}
/// Compact activity row for sidebar/history inventory.
@@ -1201,6 +1215,51 @@ pub trait PipeStore: Send + Sync {
provider: Option<&str>,
) -> Result<i64>;
+ /// Same, but records which event instance started the run so duplicate runs
+ /// can be traced back to the events that caused them.
+ async fn create_execution_with_trigger(
+ &self,
+ pipe_name: &str,
+ trigger_type: &str,
+ model: &str,
+ provider: Option<&str>,
+ _trigger_event: Option<&str>,
+ _trigger_key: Option<&str>,
+ ) -> Result<i64> {
+ self.create_execution(pipe_name, trigger_type, model, provider)
+ .await
+ }
+
+ /// Claim `(pipe, event, key)` for an event-triggered run. Returns false if
+ /// the claim is already held, meaning this is a duplicate delivery.
+ ///
+ /// Default is "always won" so stores without persistence (CLI mode) keep the
+ /// previous best-effort behaviour rather than blocking every run.
+ async fn claim_event_run(
+ &self,
+ _pipe_name: &str,
+ _event_name: &str,
+ _event_key: &str,
+ ) -> Result<bool> {
+ Ok(true)
+ }
+
+ /// Release a claim so the event can trigger the pipe again. Called when the
+ /// run fails, so a failure isn't remembered as a completed run.
+ async fn release_event_run(
+ &self,
+ _pipe_name: &str,
+ _event_name: &str,
+ _event_key: &str,
+ ) -> Result<()> {
+ Ok(())
+ }
+
+ /// Drop claims older than `before`.
+ async fn prune_event_runs(&self, _before: DateTime<Utc>) -> Result<()> {
+ Ok(())
+ }
+
/// Transition an execution to 'running' and record its PID.
async fn set_execution_running(&self, id: i64, pid: Option<u32>) -> Result<()>;
@@ -4754,6 +4813,19 @@ impl PipeManager {
std::collections::HashMap::new();
const CHAIN_COOLDOWN: std::time::Duration = std::time::Duration::from_secs(60);
+ // Triggers that matched a pipe which was already busy. They used to be
+ // dropped on the floor; instead they wait for the next tick, where the
+ // idempotency claim decides whether they still need to run.
+ let mut carryover: Vec<(String, serde_json::Value)> = Vec::new();
+
+ // Claims are pruned once per process rather than every tick.
+ if let Some(ref store) = store {
+ let cutoff = Utc::now() - chrono::Duration::days(EVENT_CLAIM_RETENTION_DAYS);
+ if let Err(e) = store.prune_event_runs(cutoff).await {
+ warn!("scheduler: failed to prune event claims: {}", e);
+ }
+ }
+
loop {
// Check for shutdown
if *rx.borrow() {
@@ -4775,23 +4847,33 @@ impl PipeManager {
// Check each pipe
let pipe_snapshot: Vec<(String, PipeConfig, String)> = {
let p = pipes.lock().await;
- p.iter()
+ let mut snapshot: Vec<(String, PipeConfig, String)> = p
+ .iter()
.map(|(n, (c, b, _))| (n.clone(), c.clone(), b.clone()))
- .collect()
+ .collect();
+ // HashMap order is randomized per process, which made the start
+ // order of pipes sharing a trigger unpredictable. Sorting makes
+ // it reproducible; runs still overlap (see event_semaphore).
+ snapshot.sort_by(|a, b| a.0.cmp(&b.0));
+ snapshot
};
// Drain pending events and mark matching pipes for immediate execution.
// Collect events from targeted subscriptions to avoid processing
// high-frequency system events (ui_frame, window_ocr, etc.).
- let mut event_triggered: std::collections::HashSet<String> =
- std::collections::HashSet::new();
+ let mut event_triggered: std::collections::HashMap<String, EventTrigger> =
+ std::collections::HashMap::new();
let mut connection_triggered: std::collections::HashSet<String> =
std::collections::HashSet::new();
+ // Triggers held back this tick, carried into the next one.
+ let mut deferred: Vec<(String, serde_json::Value)> = Vec::new();
{
use futures::FutureExt;
- // Collect events from all targeted subscriptions into one vec
- let mut pending_events: Vec<(String, serde_json::Value)> = Vec::new();
+ // Collect events from all targeted subscriptions into one vec,
+ // starting with anything held back from the previous tick.
+ let mut pending_events: Vec<(String, serde_json::Value)> =
+ std::mem::take(&mut carryover);
while let Some(e) = meeting_start_rx.next().now_or_never().flatten() {
pending_events.push((e.name, e.data));
}
@@ -4825,7 +4907,18 @@ impl PipeManager {
if name == target && config.enabled {
info!("scheduler: connection trigger fired pipe '{}'", name);
last_run.remove(name);
- event_triggered.insert(name.clone());
+ // No key: connection triggers already have their
+ // own committed-cursor dedupe in
+ // connection_triggers.rs, so they must not be
+ // suppressed a second time here.
+ event_triggered.insert(
+ name.clone(),
+ EventTrigger {
+ name: e.name.clone(),
+ key: None,
+ data: e.data.clone(),
+ },
+ );
connection_triggered.insert(name.clone());
}
}
@@ -4835,7 +4928,7 @@ impl PipeManager {
// Expire old chain cooldowns
recent_chain.retain(|_, ts| ts.elapsed() < CHAIN_COOLDOWN);
- for (event_name, _data) in &pending_events {
+ for (event_name, data) in &pending_events {
for (name, config, _body) in &pipe_snapshot {
if !config.enabled {
continue;
@@ -4871,11 +4964,15 @@ impl PipeManager {
let r = running.lock().await;
r.contains_key(name)
};
- if already_running {
+ // Busy, or another event already claimed this pipe's
+ // slot this tick. Either way the trigger waits for
+ // the next tick instead of being lost.
+ if already_running || event_triggered.contains_key(name) {
debug!(
- "scheduler: event '{}' skipped pipe '{}' (already running)",
+ "scheduler: event '{}' deferred for pipe '{}' (busy)",
event_name, name
);
+ deferred.push((event_name.clone(), data.clone()));
continue;
}
info!(
@@ -4883,7 +4980,14 @@ impl PipeManager {
event_name, name
);
last_run.remove(name);
- event_triggered.insert(name.clone());
+ event_triggered.insert(
+ name.clone(),
+ EventTrigger {
+ name: event_name.clone(),
+ key: event_identity_key(data),
+ data: data.clone(),
+ },
+ );
}
}
}
@@ -4894,7 +4998,7 @@ impl PipeManager {
continue;
}
- let triggered_by_event = event_triggered.contains(name);
+ let triggered_by_event = event_triggered.contains_key(name);
let trigger = if connection_triggered.contains(name) {
"connection"
} else if triggered_by_event {
@@ -4947,6 +5051,17 @@ impl PipeManager {
{
let qr = queued_or_running.lock().await;
if qr.contains(name) {
+ // Hold an event trigger back rather than dropping it —
+ // the run in flight may be for a different event.
+ // Connection triggers are excluded: they are addressed
+ // to a pipe by name, so replaying them through event
+ // matching wouldn't fire anything, and their watcher
+ // already retries uncommitted fires.
+ if !connection_triggered.contains(name) {
+ if let Some(t) = event_triggered.get(name) {
+ deferred.push((t.name.clone(), t.data.clone()));
+ }
+ }
continue;
}
}
@@ -5097,6 +5212,37 @@ impl PipeManager {
}
}
+ // Claim this (pipe, event, key) before committing to a run. A
+ // second delivery of the same logical event — `meeting_ended`
+ // fires again for a meeting the user rejoined — loses the claim
+ // and is suppressed here instead of starting a duplicate run
+ // (#5481).
+ //
+ // Claimed after the guards above, not when the event was
+ // matched, so a trigger that lost one of those races is still
+ // free to retry on a later tick. Must stay ahead of the queue
+ // mark below: suppressing after that would leave the pipe
+ // marked as queued forever.
+ let mut event_claim: Option<(String, String)> = None;
+ if let (Some(t), Some(store)) = (event_triggered.get(name), store.as_ref()) {
+ if let Some(key) = t.key.clone() {
+ match store.claim_event_run(name, &t.name, &key).await {
+ Ok(true) => event_claim = Some((t.name.clone(), key)),
+ Ok(false) => {
+ info!(
+ "scheduler: suppressed duplicate '{}' for pipe '{}' (key {}), already ran",
+ t.name, name, key
+ );
+ continue;
+ }
+ // Fail open: a store error must not stop the run.
+ Err(e) => {
+ warn!("scheduler: could not claim event for '{}': {}", name, e)
+ }
+ }
+ }
+ }
+
// Mark as queued so the next tick doesn't double-queue
{
let mut qr = queued_or_running.lock().await;
@@ -5186,6 +5332,19 @@ impl PipeManager {
Some("ai_preset_unavailable"),
);
}
+ // The run never happened, so don't let the claim
+ // suppress the next delivery of this event.
+ if let (Some((event, key)), Some(store)) =
+ (&event_claim, store.as_ref())
+ {
+ if let Err(e) = store.release_event_run(name, event, key).await
+ {
+ warn!(
+ "scheduler: could not release event claim for '{}': {}",
+ name, e
+ );
+ }
+ }
continue;
}
None => (
@@ -5293,6 +5452,22 @@ impl PipeManager {
let queued_ref = queued_or_running.clone();
let mcp_server_allowlist = selected_mcp_server_ids(config);
+ // Tell the pipe which event fired it, so it acts on that
+ // meeting instead of guessing at the most recent one.
+ //
+ // Skipped for connection triggers: their watcher already wrote
+ // the same file with the items it detected, and overwriting it
+ // would take that payload away from the pipe.
+ if !connection_triggered.contains(name) {
+ if let Some(t) = event_triggered.get(name) {
+ write_event_trigger_context(&pipe_dir, t);
+ }
+ }
+
+ let claim_for_release = event_claim.clone();
+ let trigger_event_name = event_claim.as_ref().map(|(e, _)| e.clone());
+ let trigger_event_key = event_claim.as_ref().map(|(_, k)| k.clone());
+
tokio::spawn(async move {
// Scheduled pipes wait for the previous one to finish
// (semaphore of 1). Event-triggered pipes skip that queue
@@ -5336,7 +5511,14 @@ impl PipeManager {
// Create DB execution row
let exec_id = if let Some(ref store) = store_ref {
match store
- .create_execution(&pipe_name, &trigger, &model, provider.as_deref())
+ .create_execution_with_trigger(
+ &pipe_name,
+ &trigger,
+ &model,
+ provider.as_deref(),
+ trigger_event_name.as_deref(),
+ trigger_event_key.as_deref(),
+ )
.await
{
Ok(id) => {
@@ -5637,6 +5819,24 @@ impl PipeManager {
let duration_secs =
(finished_at - started_at).num_milliseconds() as f64 / 1000.0;
let success = log.success;
+
+ // A failed run shouldn't be remembered as one that happened —
+ // drop the claim so the event can trigger the pipe again.
+ if !success {
+ if let (Some((event, key)), Some(ref store)) =
+ (&claim_for_release, &store_ref)
+ {
+ if let Err(e) =
+ store.release_event_run(&pipe_name, event, key).await
+ {
+ warn!(
+ "scheduler: could not release event claim for '{}': {}",
+ pipe_name, e
+ );
+ }
+ }
+ }
+
let name_for_cb = log.pipe_name.clone();
let mut l = logs_ref.lock().await;
let entry = l.entry(log.pipe_name.clone()).or_insert_with(VecDeque::new);
@@ -5715,6 +5915,20 @@ impl PipeManager {
last_cleanup = Instant::now();
}
+ // Retry deferred triggers next tick, newest last so the oldest are
+ // the ones dropped if a pipe stays busy indefinitely.
+ if !deferred.is_empty() {
+ carryover.extend(deferred);
+ if carryover.len() > MAX_CARRYOVER_EVENTS {
+ let dropped = carryover.len() - MAX_CARRYOVER_EVENTS;
+ carryover.drain(..dropped);
+ warn!(
+ "scheduler: dropped {} deferred trigger(s), backlog over {}",
+ dropped, MAX_CARRYOVER_EVENTS
+ );
+ }
+ }
+
// Sleep 30s between checks
tokio::select! {
_ = tokio::time::sleep(std::time::Duration::from_secs(30)) => {},
@@ -6275,6 +6489,58 @@ fn validate_one_off_freshness(schedule: &str) -> Result<()> {
Ok(())
}
+/// Name of the file a pipe reads to find out what triggered it. One file per run,
+/// written by whichever trigger fired: connection triggers write their detected
+/// items here, event triggers write the event below.
+const EVENT_TRIGGER_CONTEXT_FILE: &str = ".trigger-context.json";
+
+/// Record what fired the pipe, next to the pipe itself.
+///
+/// Without this a `meeting_ended` pipe has to guess which meeting it was woken
+/// for, which goes wrong exactly when two meetings end close together.
+fn write_event_trigger_context(pipe_dir: &Path, trigger: &EventTrigger) {
+ if !pipe_dir.is_dir() {
+ return;
+ }
+ let ctx = serde_json::json!({
+ "event": trigger.name,
+ "key": trigger.key,
+ "data": trigger.data,
+ "triggered_at": Utc::now().to_rfc3339(),
+ });
+ if let Ok(s) = serde_json::to_string_pretty(&ctx) {
+ let _ = atomic_write(&pipe_dir.join(EVENT_TRIGGER_CONTEXT_FILE), &s);
+ }
+}
+
+/// One event delivery that matched a pipe's `trigger.events`.
+///
+/// The payload is kept so the trigger can be held back for a later tick if the
+/// pipe is busy, and so the pipe can be told what fired it.
+#[derive(Clone, Debug)]
+struct EventTrigger {
+ /// Event name as matched against `trigger.events`.
+ name: String,
+ /// The event's own identity, if it has one — a meeting id for `meeting_*`.
+ /// Events without one can only be deduplicated within a single tick.
+ key: Option<String>,
+ data: serde_json::Value,
+}
+
+/// Read the identity of an event out of its payload.
+///
+/// Events that describe a specific thing carry its id (`meeting_ended` carries
+/// `meeting_id`), which is what makes "already ran for this one" answerable.
+fn event_identity_key(data: &serde_json::Value) -> Option<String> {
+ ["meeting_id", "event_id", "id"]
+ .iter()
+ .find_map(|field| match data.get(field) {
+ Some(serde_json::Value::String(s)) if !s.is_empty() => Some(s.clone()),
+ Some(serde_json::Value::Number(n)) => Some(n.to_string()),
+ _ => None,
+ })
+}
+
fn pipe_completed_source(event_name: &str) -> Option<&str> {
event_name.strip_prefix(PIPE_COMPLETED_EVENT_PREFIX)
}
@@ -7144,6 +7410,71 @@ mod tests {
PipeManager::new(dir, HashMap::new(), None, 0)
}
+ // -- event trigger identity (#5481) -------------------------------------
+
+ /// Both deliveries of a rejoined meeting's end must produce the same key,
+ /// otherwise the claim can't recognize the second one as a duplicate.
+ #[test]
+ fn duplicate_meeting_ended_yields_same_key() {
+ let first = serde_json::json!({ "meeting_id": 42 });
+ let second = serde_json::json!({ "meeting_id": 42 });
+ assert_eq!(event_identity_key(&first).as_deref(), Some("42"));
+ assert_eq!(event_identity_key(&first), event_identity_key(&second));
+ }
+
+ #[test]
+ fn different_meetings_yield_different_keys() {
+ let a = serde_json::json!({ "meeting_id": 42 });
+ let b = serde_json::json!({ "meeting_id": 43 });
+ assert_ne!(event_identity_key(&a), event_identity_key(&b));
+ }
+
+ #[test]
+ fn event_identity_key_accepts_string_ids() {
+ let data = serde_json::json!({ "event_id": "abc-123" });
+ assert_eq!(event_identity_key(&data).as_deref(), Some("abc-123"));
+ }
+
+ /// Events with no identity fall back to the per-tick guard, so they must not
+ /// produce a key that would collide across unrelated deliveries.
+ #[test]
+ fn keyless_events_have_no_identity() {
+ assert_eq!(event_identity_key(&serde_json::json!({})), None);
+ assert_eq!(
+ event_identity_key(&serde_json::json!({ "meeting_id": null })),
+ None
+ );
+ assert_eq!(
+ event_identity_key(&serde_json::json!({ "meeting_id": "" })),
+ None
+ );
+ }
+
+ #[test]
+ fn event_trigger_context_records_what_fired_the_pipe() {
+ let dir = std::env::temp_dir().join(format!(
+ "screenpipe-trigger-ctx-{}-{}",
+ std::process::id(),
+ line!()
+ ));
+ std::fs::create_dir_all(&dir).unwrap();
+
+ let trigger = EventTrigger {
+ name: "meeting_ended".to_string(),
+ key: Some("42".to_string()),
+ data: serde_json::json!({ "meeting_id": 42 }),
+ };
+ write_event_trigger_context(&dir, &trigger);
+
+ let written = std::fs::read_to_string(dir.join(EVENT_TRIGGER_CONTEXT_FILE)).unwrap();
+ let ctx: serde_json::Value = serde_json::from_str(&written).unwrap();
+ assert_eq!(ctx["event"], "meeting_ended");
+ assert_eq!(ctx["key"], "42");
+ assert_eq!(ctx["data"]["meeting_id"], 42);
+
+ let _ = std::fs::remove_dir_all(&dir);
+ }
+
fn pipe_source(template: bool, body: &str) -> String {
format!(
"---\nschedule: manual\nenabled: true\ntemplate: {}\n---\n\n{}\n",
@@ -9186,6 +9517,8 @@ mod tests {
error_message: None,
duration_ms: Some(60000),
session_path: None,
+ trigger_event: Some("meeting_ended".to_string()),
+ trigger_key: Some("42".to_string()),
};
let json = serde_json::to_string(&exec).unwrap();
let parsed: PipeExecution = serde_json::from_str(&json).unwrap();
@@ -9193,6 +9526,8 @@ mod tests {
assert_eq!(parsed.status, "completed");
assert_eq!(parsed.pid, Some(1234));
assert_eq!(parsed.duration_ms, Some(60000));
+ assert_eq!(parsed.trigger_event.as_deref(), Some("meeting_ended"));
+ assert_eq!(parsed.trigger_key.as_deref(), Some("42"));
}
#[test]
diff --git a/crates/screenpipe-db/src/db/write_ops.rs b/crates/screenpipe-db/src/db/write_ops.rs
index ebb072423..90a4e0f10 100644
--- a/crates/screenpipe-db/src/db/write_ops.rs
+++ b/crates/screenpipe-db/src/db/write_ops.rs
@@ -237,12 +237,17 @@ impl DatabaseManager {
}
/// Create a pipe execution via the write queue. Returns the new row ID.
+ ///
+ /// `trigger_event` / `trigger_key` record which event instance started the
+ /// run (e.g. `meeting_ended` / meeting id); both None for scheduled runs.
pub async fn pipe_create_execution_queued(
&self,
pipe_name: &str,
trigger_type: &str,
model: &str,
provider: Option<&str>,
+ trigger_event: Option<&str>,
+ trigger_key: Option<&str>,
) -> Result<i64, sqlx::Error> {
use crate::write_queue::{WriteOp, WriteResult};
let result = self
@@ -253,6 +258,8 @@ impl DatabaseManager {
model: model.to_string(),
provider: provider.map(|s| s.to_string()),
started_at: chrono::Utc::now().to_rfc3339(),
+ trigger_event: trigger_event.map(|s| s.to_string()),
+ trigger_key: trigger_key.map(|s| s.to_string()),
})
.await?;
match result {
@@ -261,6 +268,64 @@ impl DatabaseManager {
}
}
+ /// Claim `(pipe, event, key)` for an event-triggered run.
+ ///
+ /// Returns true if this caller won the claim, false if it was already held —
+ /// i.e. a duplicate delivery of the same logical event.
+ pub async fn pipe_claim_event_run_queued(
+ &self,
+ pipe_name: &str,
+ event_name: &str,
+ event_key: &str,
+ ) -> Result<bool, sqlx::Error> {
+ use crate::write_queue::{WriteOp, WriteResult};
+ let result = self
+ .write_queue
+ .submit(WriteOp::PipeClaimEventRun {
+ pipe_name: pipe_name.to_string(),
+ event_name: event_name.to_string(),
+ event_key: event_key.to_string(),
+ claimed_at: chrono::Utc::now().to_rfc3339(),
+ })
+ .await?;
+ match result {
+ WriteResult::Id(rows) => Ok(rows > 0),
+ _ => unreachable!(),
+ }
+ }
+
+ /// Release a claim so the event can trigger the pipe again.
+ pub async fn pipe_release_event_run_queued(
+ &self,
+ pipe_name: &str,
+ event_name: &str,
+ event_key: &str,
+ ) -> Result<(), sqlx::Error> {
+ use crate::write_queue::WriteOp;
+ self.write_queue
+ .submit(WriteOp::PipeReleaseEventRun {
+ pipe_name: pipe_name.to_string(),
+ event_name: event_name.to_string(),
+ event_key: event_key.to_string(),
+ })
+ .await?;
+ Ok(())
+ }
+
+ /// Drop claims older than `before`.
+ pub async fn pipe_prune_event_runs_queued(
+ &self,
+ before: chrono::DateTime<chrono::Utc>,
+ ) -> Result<(), sqlx::Error> {
+ use crate::write_queue::WriteOp;
+ self.write_queue
+ .submit(WriteOp::PipePruneEventRuns {
+ before: before.to_rfc3339(),
+ })
+ .await?;
+ Ok(())
+ }
+
/// Execute a pipe UPDATE/INSERT/DELETE via the write queue.
pub async fn pipe_execute_write_queued(
&self,
diff --git a/crates/screenpipe-db/src/migrations/20260728000000_create_pipe_event_runs.sql b/crates/screenpipe-db/src/migrations/20260728000000_create_pipe_event_runs.sql
new file mode 100644
index 000000000..8ff3f7002
--- /dev/null
+++ b/crates/screenpipe-db/src/migrations/20260728000000_create_pipe_event_runs.sql
@@ -0,0 +1,24 @@
+-- Idempotency for event-triggered pipe runs (#5481).
+--
+-- `meeting_ended` is not once-per-meeting: the 120s rejoin window reopens the
+-- same meetings row, so ending a rejoined call emits the same meeting_id twice.
+-- The scheduler's in-memory guard is keyed on pipe slug and resets every tick,
+-- so two ends landing in different ticks each started a run.
+--
+-- One row per (pipe, event, key) the scheduler has claimed. Claims are inserted
+-- with INSERT OR IGNORE, so the primary key is what makes the claim atomic.
+CREATE TABLE IF NOT EXISTS pipe_event_runs (
+ pipe_name TEXT NOT NULL,
+ event_name TEXT NOT NULL,
+ event_key TEXT NOT NULL,
+ claimed_at TEXT NOT NULL,
+ PRIMARY KEY (pipe_name, event_name, event_key)
+);
+
+-- Supports pruning by age.
+CREATE INDEX IF NOT EXISTS idx_pipe_event_runs_claimed_at ON pipe_event_runs(claimed_at);
+
+-- Which event instance produced a run, so duplicates are diagnosable after the
+-- fact. trigger_type only says "event"; these say which one.
+ALTER TABLE pipe_executions ADD COLUMN trigger_event TEXT;
+ALTER TABLE pipe_executions ADD COLUMN trigger_key TEXT;
diff --git a/crates/screenpipe-db/src/write_queue.rs b/crates/screenpipe-db/src/write_queue.rs
index 5b04dc601..e0b35fe74 100644
--- a/crates/screenpipe-db/src/write_queue.rs
+++ b/crates/screenpipe-db/src/write_queue.rs
@@ -517,6 +517,10 @@ pub(crate) enum WriteOp {
model: String,
provider: Option<String>,
started_at: String,
+ /// Event that triggered the run, and the event's identity (e.g. a
+ /// meeting id). Both None for scheduled and manual runs.
+ trigger_event: Option<String>,
+ trigger_key: Option<String>,
},
PipeUpdateExecution {
sql: String,
@@ -525,6 +529,24 @@ pub(crate) enum WriteOp {
PipeDeleteOldExecutions {
keep_per_pipe: i32,
},
+ /// Claim a (pipe, event, key) tuple for an event-triggered run. Returns the
+ /// number of rows inserted: 1 on a fresh claim, 0 if already claimed.
+ PipeClaimEventRun {
+ pipe_name: String,
+ event_name: String,
+ event_key: String,
+ claimed_at: String,
+ },
+ /// Drop a claim so the event can be retried (used when the run fails).
+ PipeReleaseEventRun {
+ pipe_name: String,
+ event_name: String,
+ event_key: String,
+ },
+ /// Prune claims older than `before`, bounding the table's growth.
+ PipePruneEventRuns {
+ before: String,
+ },
/// Insert a synced frame (video_chunk + frame row) from cloud sync import.
SyncInsertFrame {
sync_id: String,
@@ -2000,10 +2022,12 @@ async fn execute_single_write(
model,
provider,
started_at,
+ trigger_event,
+ trigger_key,
} => {
let row = sqlx::query_scalar::<_, i64>(
- r#"INSERT INTO pipe_executions (pipe_name, status, trigger_type, model, provider, started_at)
- VALUES (?, 'queued', ?, ?, ?, ?)
+ r#"INSERT INTO pipe_executions (pipe_name, status, trigger_type, model, provider, started_at, trigger_event, trigger_key)
+ VALUES (?, 'queued', ?, ?, ?, ?, ?, ?)
RETURNING id"#,
)
.bind(pipe_name)
@@ -2011,11 +2035,60 @@ async fn execute_single_write(
.bind(model)
.bind(provider)
.bind(started_at)
+ .bind(trigger_event)
+ .bind(trigger_key)
.fetch_one(&mut **conn)
.await?;
Ok(WriteResult::Id(row))
}
+ WriteOp::PipeClaimEventRun {
+ pipe_name,
+ event_name,
+ event_key,
+ claimed_at,
+ } => {
+ // OR IGNORE + the primary key is the whole idempotency guarantee:
+ // a duplicate claim inserts 0 rows instead of erroring.
+ let result = sqlx::query(
+ r#"INSERT OR IGNORE INTO pipe_event_runs
+ (pipe_name, event_name, event_key, claimed_at)
+ VALUES (?, ?, ?, ?)"#,
+ )
+ .bind(pipe_name)
+ .bind(event_name)
+ .bind(event_key)
+ .bind(claimed_at)
+ .execute(&mut **conn)
+ .await?;
+ Ok(WriteResult::Id(result.rows_affected() as i64))
+ }
+
+ WriteOp::PipeReleaseEventRun {
+ pipe_name,
+ event_name,
+ event_key,
+ } => {
+ sqlx::query(
+ r#"DELETE FROM pipe_event_runs
+ WHERE pipe_name = ? AND event_name = ? AND event_key = ?"#,
+ )
+ .bind(pipe_name)
+ .bind(event_name)
+ .bind(event_key)
+ .execute(&mut **conn)
+ .await?;
+ Ok(WriteResult::Unit)
+ }
+
+ WriteOp::PipePruneEventRuns { before } => {
+ sqlx::query("DELETE FROM pipe_event_runs WHERE claimed_at < ?")
+ .bind(before)
+ .execute(&mut **conn)
+ .await?;
+ Ok(WriteResult::Unit)
+ }
+
WriteOp::PipeUpdateExecution { sql, binds } => {
// Pipe update SQL is assembled internally with bind placeholders;
// runtime values remain separate in `binds` below.
diff --git a/crates/screenpipe-engine/src/pipe_store.rs b/crates/screenpipe-engine/src/pipe_store.rs
index 71904eb8d..c8fde2169 100644
--- a/crates/screenpipe-engine/src/pipe_store.rs
+++ b/crates/screenpipe-engine/src/pipe_store.rs
@@ -35,10 +35,61 @@ impl PipeStore for SqlitePipeStore {
) -> Result<i64> {
Ok(self
.db
- .pipe_create_execution_queued(pipe_name, trigger_type, model, provider)
+ .pipe_create_execution_queued(pipe_name, trigger_type, model, provider, None, None)
.await?)
}
+ async fn create_execution_with_trigger(
+ &self,
+ pipe_name: &str,
+ trigger_type: &str,
+ model: &str,
+ provider: Option<&str>,
+ trigger_event: Option<&str>,
+ trigger_key: Option<&str>,
+ ) -> Result<i64> {
+ Ok(self
+ .db
+ .pipe_create_execution_queued(
+ pipe_name,
+ trigger_type,
+ model,
+ provider,
+ trigger_event,
+ trigger_key,
+ )
+ .await?)
+ }
+
+ async fn claim_event_run(
+ &self,
+ pipe_name: &str,
+ event_name: &str,
+ event_key: &str,
+ ) -> Result<bool> {
+ Ok(self
+ .db
+ .pipe_claim_event_run_queued(pipe_name, event_name, event_key)
+ .await?)
+ }
+
+ async fn release_event_run(
+ &self,
+ pipe_name: &str,
+ event_name: &str,
+ event_key: &str,
+ ) -> Result<()> {
+ self.db
+ .pipe_release_event_run_queued(pipe_name, event_name, event_key)
+ .await?;
+ Ok(())
+ }
+
+ async fn prune_event_runs(&self, before: DateTime<Utc>) -> Result<()> {
+ self.db.pipe_prune_event_runs_queued(before).await?;
+ Ok(())
+ }
+
async fn set_execution_running(&self, id: i64, pid: Option<u32>) -> Result<()> {
use screenpipe_db::write_queue::PipeBindValue;
let now = Utc::now().to_rfc3339();
@@ -116,7 +167,8 @@ impl PipeStore for SqlitePipeStore {
sqlx::query_as::<_, PipeExecutionRow>(
r#"SELECT id, pipe_name, status, trigger_type, pid, model, provider,
started_at, finished_at, stdout, stderr, exit_code,
- error_type, error_message, duration_ms, session_path
+ error_type, error_message, duration_ms, session_path,
+ trigger_event, trigger_key
FROM pipe_executions
WHERE pipe_name = ? AND id < ?
ORDER BY id DESC
@@ -131,7 +183,8 @@ impl PipeStore for SqlitePipeStore {
sqlx::query_as::<_, PipeExecutionRow>(
r#"SELECT id, pipe_name, status, trigger_type, pid, model, provider,
started_at, finished_at, stdout, stderr, exit_code,
- error_type, error_message, duration_ms, session_path
+ error_type, error_message, duration_ms, session_path,
+ trigger_event, trigger_key
FROM pipe_executions
WHERE pipe_name = ?
ORDER BY id DESC
@@ -157,7 +210,8 @@ impl PipeStore for SqlitePipeStore {
r#"SELECT id, pipe_name, status, trigger_type, pid, model, provider,
started_at, finished_at,
'' AS stdout, '' AS stderr,
- exit_code, error_type, error_message, duration_ms, session_path
+ exit_code, error_type, error_message, duration_ms, session_path,
+ trigger_event, trigger_key
FROM pipe_executions
WHERE pipe_name = ? AND id < ?
ORDER BY id DESC
@@ -173,7 +227,8 @@ impl PipeStore for SqlitePipeStore {
r#"SELECT id, pipe_name, status, trigger_type, pid, model, provider,
started_at, finished_at,
'' AS stdout, '' AS stderr,
- exit_code, error_type, error_message, duration_ms, session_path
+ exit_code, error_type, error_message, duration_ms, session_path,
+ trigger_event, trigger_key
FROM pipe_executions
WHERE pipe_name = ?
ORDER BY id DESC
@@ -338,7 +393,8 @@ impl PipeStore for SqlitePipeStore {
r#"SELECT id, pipe_name, status, trigger_type, pid, model, provider,
started_at, finished_at,
'' AS stdout, '' AS stderr,
- exit_code, error_type, error_message, duration_ms, session_path
+ exit_code, error_type, error_message, duration_ms, session_path,
+ trigger_event, trigger_key
FROM pipe_executions
WHERE pipe_name = ?
ORDER BY id DESC
@@ -447,6 +503,8 @@ struct PipeExecutionRow {
error_message: Option<String>,
duration_ms: Option<i64>,
session_path: Option<String>,
+ trigger_event: Option<String>,
+ trigger_key: Option<String>,
}
impl From<PipeExecutionRow> for PipeExecution {
@@ -468,6 +526,8 @@ impl From<PipeExecutionRow> for PipeExecution {
error_message: r.error_message,
duration_ms: r.duration_ms,
session_path: r.session_path,
+ trigger_event: r.trigger_event,
+ trigger_key: r.trigger_key,
}
}
}
@@ -600,6 +660,116 @@ mod tests {
.unwrap();
}
+ /// #5481: `meeting_ended` fires twice for one meeting when the user rejoins
+ /// within the 120s merge window. The second delivery must not win a claim.
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn test_duplicate_meeting_ended_claims_once() {
+ let (store, _tmp) = setup_test_store().await;
+
+ assert!(store
+ .claim_event_run("meeting-summary", "meeting_ended", "42")
+ .await
+ .unwrap());
+ assert!(
+ !store
+ .claim_event_run("meeting-summary", "meeting_ended", "42")
+ .await
+ .unwrap(),
+ "duplicate meeting_ended for meeting 42 should be suppressed"
+ );
+
+ // A different meeting is unrelated work.
+ assert!(store
+ .claim_event_run("meeting-summary", "meeting_ended", "43")
+ .await
+ .unwrap());
+ // So is the same meeting seen by a different pipe.
+ assert!(store
+ .claim_event_run("meeting-notes", "meeting_ended", "42")
+ .await
+ .unwrap());
+ }
+
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn test_released_claim_can_run_again() {
+ let (store, _tmp) = setup_test_store().await;
+
+ assert!(store
+ .claim_event_run("p", "meeting_ended", "7")
+ .await
+ .unwrap());
+ store
+ .release_event_run("p", "meeting_ended", "7")
+ .await
+ .unwrap();
+ assert!(
+ store
+ .claim_event_run("p", "meeting_ended", "7")
+ .await
+ .unwrap(),
+ "a failed run releases its claim so the event can be retried"
+ );
+ }
+
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn test_prune_event_runs_drops_old_claims() {
+ let (store, _tmp) = setup_test_store().await;
+
+ assert!(store
+ .claim_event_run("p", "meeting_ended", "1")
+ .await
+ .unwrap());
+ store
+ .prune_event_runs(Utc::now() + chrono::Duration::days(1))
+ .await
+ .unwrap();
+ assert!(
+ store
+ .claim_event_run("p", "meeting_ended", "1")
+ .await
+ .unwrap(),
+ "pruned claims no longer suppress"
+ );
+ }
+
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn test_execution_records_triggering_event() {
+ let (store, _tmp) = setup_test_store().await;
+
+ store
+ .create_execution_with_trigger(
+ "meeting-summary",
+ "event",
+ "opus",
+ None,
+ Some("meeting_ended"),
+ Some("42"),
+ )
+ .await
+ .unwrap();
+
+ let execs = store
+ .get_executions("meeting-summary", 10, None)
+ .await
+ .unwrap();
+ assert_eq!(execs[0].trigger_event.as_deref(), Some("meeting_ended"));
+ assert_eq!(execs[0].trigger_key.as_deref(), Some("42"));
+ }
+
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn test_scheduled_execution_has_no_trigger_event() {
+ let (store, _tmp) = setup_test_store().await;
+
+ store
+ .create_execution("p", "scheduled", "m", None)
+ .await
+ .unwrap();
+
+ let execs = store.get_executions("p", 10, None).await.unwrap();
+ assert_eq!(execs[0].trigger_event, None);
+ assert_eq!(execs[0].trigger_key, None);
+ }
+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_execution() {
let (store, _tmp) = setup_test_store().await;