YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
{
"schema_version": 1,
"suite": "screenpipe-app-coding-agent-regressions",
"dataset_version": "2026-10-09-manual-grant",
"suite_type": "regression",
"owner": "screenpipe engineering",
"policy": {
"stage": "advisory",
"promotion_requirement": "matched-environment repeated trials with reliable pass^k"
},
"cases": [
{
"id": "app-chat-concurrent-save",
"name": "Preserve both writers during concurrent chat saves",
"source": {
"kind": "git_regression",
"fix_commit": "1a74f9c68a19fdab0ba0d26409c111a79b3d5735"
},
"base_ref": "8c0affa6bd2476d84a226f85a7b18cab126d0a33",
"oracle_ref": "1a74f9c68a19fdab0ba0d26409c111a79b3d5735",
"tags": [
"regression",
"escaped-failure",
"chat",
"concurrency",
"data-loss"
],
"prompt": "A saved chat can lose messages when a later writer saves an older snapshot. Fix the conversation persistence API so stale snapshots contribute new messages without losing either suffix or repeating message IDs, and preserve the already-saved message order. For the same streaming message ID retain longer text, then more structured content blocks when text length ties. A stale snapshot must preserve newer saved scalar fields; a current edit must still persist its title and browser state. Concurrent message saves and concurrent updates to distinct flags must all survive. A missing-conversation flag update remains a no-op. Save errors must reject and leave later saves usable. Keep ordinary new saves and reloads working. Tests exercise actual persistence calls through synthetic filesystem ports, including overlapping calls in one runtime and an external commit between calls; independent native webview scheduling is outside this case. Keep the solution scoped to chat persistence and add or update tests as needed.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"local_path": "graders/chat-save-outcomes.fixture.ts",
"destination_path": "apps/screenpipe-app-tauri/lib/__tests__/eval-chat-save.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage node node_modules/vitest/vitest.mjs run --config vitest.config.ts lib/__tests__/eval-chat-save.test.ts --pool=forks --poolOptions.forks.singleFork=true"
},
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat-storage.ts",
"apps/screenpipe-app-tauri/lib/chat-merge.ts"
],
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
]
},
{
"id": "app-terminal-quota-retry",
"name": "Stop retrying terminal Pi quota failures",
"source": {
"kind": "git_regression",
"fix_commit": "58c6e837f"
},
"base_ref": "58c6e837f^",
"oracle_ref": "58c6e837f",
"tags": [
"regression",
"escaped-failure",
"chat",
"quota",
"retry"
],
"prompt": "The Pi chat transport receives provider 429 responses that mean the account has exhausted a daily allowance or the selected model is not permitted. The app currently treats these as transient and keeps retrying, leaving the user in a loop. Make terminal quota/plan errors stop the retry path and surface the assistant error, while ordinary transient failures remain retryable. Preserve existing silent prompt-timeout and connection-card behavior. Exercise the actual foreground event handling: terminal message errors followed by an agent-end event promising another retry must stop that session, finalize the assistant error, and clear loading/streaming state without submitting another prompt. Ordinary provider retry events stay active; a transient throttled response can retry after its wait, and successful answers still finalize. No live provider or native process is needed.",
"agent_timeout_seconds": 720,
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"local_path": "graders/app-terminal-quota.test.ts",
"destination_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/eval-terminal-quota.test.ts"
},
{
"source_ref": "58c6e837f^",
"source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/quota-errors.test.ts"
},
{
"source_ref": "58c6e837f^",
"source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/provider-errors.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/chat/__tests__/eval-terminal-quota.test.ts lib/chat/__tests__/quota-errors.test.ts lib/chat/__tests__/provider-errors.test.ts"
},
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
]
},
{
"id": "app-timeline-stream-reliability",
"name": "Keep native Timeline streams bounded and complete",
"source": {
"kind": "git_regression",
"fix_commit": "a4c1161d2"
},
"base_ref": "a4c1161d2^",
"oracle_ref": "a4c1161d2",
"tags": [
"regression",
"escaped-failure",
"timeline",
"websocket",
"native"
],
"prompt": "Native Timeline search becomes unreliable on large days: serialized WebSocket batches can exceed the native receiver ceiling, and navigation/search races can surface stale or incomplete frames. Fix the stream path so payloads are bounded by encoded bytes without dropping or duplicating rows, cancellation is respected, and the native search contract remains compatible. Do not solve this by imposing a small arbitrary row limit.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "target",
"destination_path": "target"
}
],
"grader": {
"timeout_seconds": 1200,
"fixtures": [
{
"source_path": "crates/screenpipe-engine/tests/stream_frames_test.rs"
}
],
"command": "cargo test -p screenpipe-engine --test stream_frames_test"
}
},
{
"id": "app-first-run-empty-summary",
"name": "Never render an empty first-run summary as success",
"source": {
"kind": "git_regression",
"fix_commit": "91fa9bd1b"
},
"base_ref": "91fa9bd1b^",
"oracle_ref": "91fa9bd1b",
"tags": [
"regression",
"escaped-failure",
"onboarding",
"truthfulness"
],
"prompt": "The first-run learning flow can finish with an empty or unusable generated summary and then render that as if capture produced a meaningful result. Fix the state and UI boundary so empty summaries stay out of the content surface, capture diagnostics remain actionable, and resume/dismiss behavior remains stable. A valid non-empty summary must still render normally.",
"agent_timeout_seconds": 900,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/lib/first-run/learning-window.test.ts"
},
{
"source_path": "apps/screenpipe-app-tauri/components/first-run/learning-banner.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/first-run/learning-window.test.ts components/first-run/learning-banner.test.tsx"
}
},
{
"id": "app-onboarding-checkout-confirmation",
"name": "Confirm onboarding checkout without false completion",
"source": {
"kind": "git_regression",
"fix_commit": "301535539"
},
"base_ref": "301535539^",
"oracle_ref": "301535539",
"tags": [
"regression",
"escaped-failure",
"onboarding",
"billing",
"state-machine"
],
"prompt": "Onboarding can advance after a canceled or unconfirmed checkout, submit duplicate checkout requests, or get stuck while entitlement hydration is incomplete. Make checkout confirmation bounded and fail closed: cancellation must not advance, an existing valid entitlement or payment must avoid another checkout, polling must stop deterministically, and repeated clicks must not create duplicate submissions. Preserve a successful checkout's normal onboarding transition.",
"agent_timeout_seconds": 1200,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/app/onboarding/page.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/components/onboarding/plan-selection-step.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/onboarding-checkout.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts app/onboarding/page.test.tsx components/onboarding/plan-selection-step.test.tsx lib/onboarding-checkout.test.ts"
}
},
{
"id": "app-activity-recent-control",
"name": "Keep recent activity navigation anchored to now",
"source": {
"kind": "git_regression",
"fix_commit": "1a854ab86"
},
"base_ref": "1a854ab86^",
"oracle_ref": "1a854ab86",
"tags": [
"regression",
"escaped-failure",
"activity",
"navigation",
"time"
],
"prompt": "The activity ledger lost its useful recent-activity control. Restore a bottom append control for Today and Last 24 hours when there is enough uncovered recent time, but do not roll the visible page clock forward or disturb historical/custom ranges. Keep the control hidden when the uncovered interval is too small and preserve existing paging behavior.",
"agent_timeout_seconds": 900,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/activity-ledger.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/activity-ledger.test.tsx"
}
},
{
"id": "app-chat-correction-boundaries",
"name": "Preserve user correction boundaries in chat context",
"source": {
"kind": "git_regression",
"fix_commit": "f177dc7bb"
},
"base_ref": "f177dc7bb^",
"oracle_ref": "f177dc7bb",
"tags": [
"regression",
"escaped-failure",
"chat",
"steering",
"prompt"
],
"prompt": "When a user corrects an in-progress Pi chat task, the app can flatten the correction into the original request or lose prior constraints. Preserve the original request and ordered steering boundaries so the latest correction wins only where it conflicts, unrelated constraints remain active, and the system prompt clearly distinguishes corrections, evidence requirements, and read-only boundaries.",
"agent_timeout_seconds": 900,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/hooks/__tests__/pi-event-handlers.test.ts"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/system-prompt.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/chat/standalone/hooks/__tests__/pi-event-handlers.test.ts lib/chat/__tests__/system-prompt.test.ts"
}
},
{
"id": "app-settings-clear-timeline-cache",
"name": "Clear timeline caches with backend data",
"source": {
"kind": "git_regression",
"fix_commit": "e12d48ef1"
},
"base_ref": "e12d48ef1^",
"oracle_ref": "e12d48ef1",
"tags": [
"regression",
"escaped-failure",
"settings",
"cache",
"privacy"
],
"prompt": "Clearing screenpipe data in Settings removes backend records but leaves Timeline caches behind, so deleted transcripts can reappear. Make the clear-cache action remove the current and legacy Timeline IndexedDB caches, await and report failures, and remain safe when no browser cache exists. Preserve the existing backend clear behavior and success/error UI semantics.",
"agent_timeout_seconds": 900,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/settings/storage-section.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/hooks/use-timeline-cache.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/settings/storage-section.test.tsx lib/hooks/use-timeline-cache.test.tsx"
}
},
{
"id": "app-optional-connection-credentials",
"name": "Allow connections whose credential fields are all optional",
"source": {
"kind": "git_regression",
"fix_commit": "4b0d119fd"
},
"base_ref": "4b0d119fd^",
"oracle_ref": "4b0d119fd",
"tags": [
"regression",
"escaped-failure",
"connections",
"credentials",
"form"
],
"prompt": "A connection definition whose credential fields are all optional cannot be connected because the submit action stays disabled when every field is blank. Allow that valid case and send the backend a usable non-empty credentials object, while still disabling submission when any required field is blank. Do not weaken validation for integrations with required credentials.",
"agent_timeout_seconds": 720,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/settings/__tests__/connection-credential-forms.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/settings/__tests__/connection-credential-forms.test.tsx"
}
},
{
"id": "app-pipe-store-load-error",
"name": "Distinguish pipe-store failures from an empty catalog",
"source": {
"kind": "git_regression",
"fix_commit": "8dbf6e47bac2bd033eaca3d7da649fce585b1c5f"
},
"base_ref": "e541a77536ccf33cd9668f18aad065a95bd19d42",
"oracle_ref": "8dbf6e47bac2bd033eaca3d7da649fce585b1c5f",
"tags": [
"regression",
"escaped-failure",
"pipes",
"error-state",
"retry"
],
"prompt": "When the pipe store request fails, the app renders the same empty state as a successfully loaded catalog with no pipes. Show a clear load-error state and a working retry for network failures, unsuccessful HTTP responses, and error payloads. Keep successful empty catalogs distinct, including bare arrays and lists wrapped in data or pipes. A successful retry to either populated or empty results must clear stale error and retry UI. Preserve the installed-tasks tab and successful catalog content. Wording and internal cache implementation may vary.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"local_path": "graders/pipe-store-error.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/__tests__/pipe-store-error-state.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts components/__tests__/pipe-store-error-state.test.tsx --maxWorkers=1 --minWorkers=1"
},
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"oracle_paths": [
"apps/screenpipe-app-tauri/components/pipe-store.tsx"
]
},
{
"id": "app-learning-handoff-always-finishes",
"name": "Always finish the first-run learning handoff",
"source": {
"kind": "git_regression",
"fix_commit": "e3587f5654"
},
"base_ref": "e3587f5654^",
"oracle_ref": "e3587f5654",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/first-run/**",
"apps/screenpipe-app-tauri/lib/first-run/**"
],
"tags": [
"regression",
"escaped-failure",
"onboarding",
"state-machine",
"rehydration"
],
"prompt": "The first-run learning handoff can remain in a fake in-progress state after an empty foreground result, disappear after reload, or reuse stale webview state after setup is completed again. Make every foreground attempt reach useful setup choices, keep background retries out of the interface, preserve unresolved choices across reload, and start a fresh learning window after setup is completed again. Valid completed summaries must keep their existing behavior.",
"agent_timeout_seconds": 1200,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/first-run/learning-banner.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/first-run/learning-window.test.ts"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/first-run/use-learning-window.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/first-run/learning-banner.test.tsx lib/first-run/learning-window.test.ts lib/first-run/use-learning-window.test.ts"
}
},
{
"id": "app-chat-activity-episode-content",
"name": "Analyze attached activity episode content instead of its title",
"source": {
"kind": "git_regression",
"fix_commit": "7fa6d1a956"
},
"base_ref": "7fa6d1a956^",
"oracle_ref": "7fa6d1a956",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/**",
"apps/screenpipe-app-tauri/lib/chat/**"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"retrieval",
"evidence"
],
"prompt": "When a user asks about an attached activity-history episode, the agent searches for the generated episode title instead of reading the bounded source content. Mark activity-history attachments as episodes and route questions to the exact attached interval and its content. Reject title-keyword searches and plans that drift outside the interval, while preserving ordinary search attachments and traceable evidence behavior.",
"agent_timeout_seconds": 1200,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/attached-context.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/activity-episode-retrieval-eval.test.ts"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/system-prompt.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/chat/standalone/attached-context.test.tsx lib/chat/__tests__/activity-episode-retrieval-eval.test.ts lib/chat/__tests__/system-prompt.test.ts"
}
},
{
"id": "app-recording-stale-permission-recovery",
"name": "Recover recording from stale screen-permission state",
"source": {
"kind": "git_regression",
"fix_commit": "8f79280560"
},
"base_ref": "8f79280560^",
"oracle_ref": "8f79280560",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/app/shortcut-reminder/**",
"apps/screenpipe-app-tauri/lib/hooks/use-health-check.ts"
],
"tags": [
"regression",
"escaped-failure",
"recording",
"permissions",
"recovery"
],
"prompt": "After screen permission is restored, stale permission state can leave the recording overlay showing a broken or restart-only state even though capture can recover. Reconcile permission and health state so the user gets a truthful recovery confirmation and recording resumes without an unnecessary restart. Keep genuine permission denial and real capture failures actionable.",
"agent_timeout_seconds": 900,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/app/shortcut-reminder/page.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts app/shortcut-reminder/page.test.tsx"
}
},
{
"id": "app-history-long-generation",
"name": "Keep legitimate slow history generation alive",
"source": {
"kind": "git_regression",
"fix_commit": "b402c7be54"
},
"base_ref": "b402c7be54^",
"oracle_ref": "b402c7be54",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/activity-ledger.tsx",
"apps/screenpipe-app-tauri/components/share-logs-button.tsx"
],
"tags": [
"regression",
"escaped-failure",
"activity",
"timeout",
"performance"
],
"prompt": "Generating a long activity-history result can take more than two minutes, but the app times it out as a failure. Keep legitimate slow generation running while retaining a bounded failure path, and load only the five conversations that can actually be attached instead of expanding work without limit. Preserve existing cancellation and successful short-generation behavior.",
"agent_timeout_seconds": 1200,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/activity-ledger.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/components/share-logs-button.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/activity-ledger.test.tsx components/share-logs-button.test.tsx"
}
},
{
"id": "app-live-view-cadence-write",
"name": "Make Live View cadence writes take effect",
"source": {
"kind": "git_regression",
"fix_commit": "f9f2445b94"
},
"base_ref": "f9f2445b94^",
"oracle_ref": "f9f2445b94",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/live-views/**",
"apps/screenpipe-app-tauri/components/settings/brain-overview.tsx"
],
"tags": [
"regression",
"escaped-failure",
"live-view",
"api-contract",
"silent-failure"
],
"prompt": "Refreshing a paused or manual Live View reports success but leaves its task disabled because the cadence update is wrapped in a config envelope that the API silently ignores. Send the flat configuration shape the endpoint consumes, omit fields that do not need changing, reject an invalid nested front-matter envelope, and surface partial write failures instead of pretending every task refreshed.",
"agent_timeout_seconds": 1200,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/settings/__tests__/brain-overview.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/live-views/__tests__/source-cadence.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/settings/__tests__/brain-overview.test.tsx lib/live-views/__tests__/source-cadence.test.ts"
}
},
{
"id": "app-mcp-traceable-activity-context",
"name": "Return bounded, traceable activity context from MCP",
"source": {
"kind": "git_regression",
"fix_commit": "e76dac2120"
},
"base_ref": "e76dac2120^",
"oracle_ref": "e76dac2120",
"gate": "advisory",
"trigger_paths": [
"packages/screenpipe-mcp/src/**"
],
"tags": [
"regression",
"escaped-failure",
"mcp",
"activity",
"evidence"
],
"prompt": "The MCP activity-summary tool returns aggregate time without enough traceable task context, and consumers can confuse missing parsed rows with missing activity. Add optional bounded parsed context while keeping duration authoritative, distinguish disabled context from fetch failure, retain the time result when parsing is unavailable, and expose source paths without making parsed context the default.",
"agent_timeout_seconds": 1200,
"dependency_links": [
{
"source_path": "packages/screenpipe-mcp/node_modules",
"destination_path": "packages/screenpipe-mcp/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "packages/screenpipe-mcp/src/activity-summary-format.test.ts"
},
{
"source_path": "packages/screenpipe-mcp/src/activity-summary-tool.test.ts"
},
{
"source_path": "packages/screenpipe-mcp/src/pack-contents.test.ts"
},
{
"source_path": "packages/screenpipe-mcp/src/stdio-startup.test.ts"
}
],
"command": "cd packages/screenpipe-mcp && node ./node_modules/vitest/vitest.mjs run src/activity-summary-format.test.ts src/activity-summary-tool.test.ts src/pack-contents.test.ts src/stdio-startup.test.ts"
}
},
{
"id": "app-chat-inspector-pipe-artifacts",
"name": "Keep pipe-run artifacts in the active chat inspector",
"source": {
"kind": "git_regression",
"fix_commit": "d3569717c0"
},
"base_ref": "d3569717c0^",
"oracle_ref": "d3569717c0",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/**",
"apps/screenpipe-app-tauri/lib/hooks/use-chat-inspector.ts"
],
"tags": [
"regression",
"escaped-failure",
"pipes",
"chat",
"artifacts"
],
"prompt": "Artifacts registered by a pipe run do not appear in the chat inspector unless a matching tool call exists, and stale artifacts can leak into a different task. Merge declared run artifacts into the active inspector, deduplicate files already represented by tool output, rewrite absolute artifact links safely, and keep pinned summaries open beside output previews without crossing execution boundaries.",
"agent_timeout_seconds": 1200,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/__tests__/markdown-viewer-link.test.ts"
},
{
"source_path": "apps/screenpipe-app-tauri/components/chat/chat-inspector.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/hooks/use-chat-inspector.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/__tests__/markdown-viewer-link.test.ts components/chat/chat-inspector.test.tsx lib/hooks/use-chat-inspector.test.ts"
}
},
{
"id": "app-sync-legacy-key-recovery",
"name": "Recover safely from legacy sync-key mismatches",
"source": {
"kind": "git_regression",
"fix_commit": "0f26a722d2"
},
"base_ref": "0f26a722d2^",
"oracle_ref": "0f26a722d2",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/settings/**sync**",
"apps/screenpipe-app-tauri/lib/hooks/use-cloud-sync.ts"
],
"tags": [
"regression",
"escaped-failure",
"sync",
"encryption",
"recovery"
],
"prompt": "A device with a legacy account-key mismatch cannot resume cloud sync and generic errors can incorrectly trigger destructive recovery UI. Detect only the specific legacy mismatch, keep recovery hidden otherwise, require explicit confirmation before the exact scoped start-fresh request, and leave recovery available when the remote reset fails. Do not weaken ordinary sync error handling.",
"agent_timeout_seconds": 900,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/settings/__tests__/sync-key-recovery.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/settings/__tests__/sync-key-recovery.test.tsx"
}
},
{
"id": "app-agent-local-calendar-days",
"name": "Use local calendar days for agent time ranges",
"source": {
"kind": "git_regression",
"fix_commit": "afed076026"
},
"base_ref": "afed076026^",
"oracle_ref": "afed076026",
"gate": "advisory",
"trigger_paths": [
"packages/screenpipe-mcp/src/**time**",
"apps/screenpipe-app-tauri/lib/chat/system-prompt.ts"
],
"tags": [
"regression",
"escaped-failure",
"agent",
"time",
"timezone"
],
"prompt": "Agent time ranges such as today and yesterday are normalized through UTC, so users near a day boundary can query the wrong local date. Normalize calendar literals from the runtime local day, use calendar arithmetic across daylight-saving changes and zones that skip midnight, preserve already supported values and input objects, and advertise the same local-calendar contract in every time field.",
"agent_timeout_seconds": 1200,
"dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
},
{
"source_path": "packages/screenpipe-mcp/node_modules",
"destination_path": "packages/screenpipe-mcp/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/system-prompt.test.ts"
},
{
"source_path": "packages/screenpipe-mcp/src/time-normalization.test.ts"
},
{
"source_path": "packages/screenpipe-mcp/src/pack-contents.test.ts"
},
{
"source_path": "packages/screenpipe-mcp/src/stdio-startup.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/chat/__tests__/system-prompt.test.ts && cd ../../packages/screenpipe-mcp && node ./node_modules/vitest/vitest.mjs run src/time-normalization.test.ts src/pack-contents.test.ts src/stdio-startup.test.ts"
}
},
{
"id": "app-privacy-category-rule-ownership",
"name": "Never delete user-owned privacy rules with a category",
"source": {
"kind": "git_regression",
"fix_commit": "4450aaa82d30d262a5375520a734f1dff143e81d"
},
"base_ref": "9ebcd29c6ada189922c02e4ab65f1a08e228e1a1",
"oracle_ref": "4450aaa82d30d262a5375520a734f1dff143e81d",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/settings/privacy-section.tsx",
"apps/screenpipe-app-tauri/components/settings/capture-filters/content-filters-card.tsx",
"apps/screenpipe-app-tauri/lib/settings/capture-categories.ts",
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"settings",
"data-loss"
],
"prompt": "Disabling a privacy capture category deletes an identical exclusion rule or domain the user created manually. Fix the actual Settings > Privacy category switches so enabling then disabling preserves hand-written exclusions and other categories, including after closing and reopening settings. Persist whatever ownership information the UI needs. Still add the category filters when enabled, clean up all filters it created when disabled, preserve unrelated include rules, and tolerate repeated disable. Keep a safe compatibility path so legacy enabled categories without provenance can still be disabled. No particular helper, callback or ownership field name is required. Preserve the existing category choices and privacy controls.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"local_path": "graders/app-privacy-ownership.test.js",
"destination_path": "evals/coding-agent/graders/app-privacy-ownership.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/app-privacy-ownership.test.js"
},
"oracle_paths": [
"apps/screenpipe-app-tauri/components/settings/privacy-section.tsx",
"apps/screenpipe-app-tauri/components/settings/capture-filters/content-filters-card.tsx",
"apps/screenpipe-app-tauri/lib/settings/capture-categories.ts",
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx"
]
},
{
"id": "app-acp-explicit-full-access-boundary",
"name": "Keep ACP full access explicit, reversible, and adapter-safe",
"source": {
"kind": "git_regression",
"fix_commit": "e83222db608f55b3931204e061262469411de94a",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6514"
},
"base_ref": "989a53b86af113e0e408b4ee6c5f7dd14d886d3d",
"oracle_ref": "e83222db608f55b3931204e061262469411de94a",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/**acp**",
"apps/screenpipe-app-tauri/lib/stores/acp-session-config.ts",
"apps/screenpipe-app-tauri/lib/utils/tauri.ts",
"apps/screenpipe-app-tauri/src-tauri/src/pi.rs",
"crates/screenpipe-core/src/agents/acp/**"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"acp",
"permissions",
"safety-boundary"
],
"prompt": "An ACP adapter can advertise working modes without an unrestricted one, leaving users unable to grant full access explicitly. Add a clearly warned opt-in full-access choice that restores a tool-capable mode and enables client-side automatic approval. Keep the default and unknown states approval-based, return to approval prompts when a prompted or read-only choice is selected, persist and reapply the policy for live sessions, and preserve adapter-owned unrestricted controls without duplicates.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/acp-permission-selector.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/stores/acp-session-config.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts components/chat/standalone/acp-permission-selector.test.tsx lib/stores/acp-session-config.test.ts"
}
},
{
"id": "app-sqlite-process-ownership",
"name": "Keep one live SQLite owner per physical database",
"source": {
"kind": "git_regression",
"fix_commit": "b68c120070351b90afb9e1df1e5f2c866f7fe396",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6517"
},
"base_ref": "6301ad06342d4bad420598db86db9e44423b05b6",
"oracle_ref": "b68c120070351b90afb9e1df1e5f2c866f7fe396",
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-db/src/db/**",
"crates/screenpipe-db/src/write_queue.rs",
"crates/screenpipe-sqlite-coordinator/src/lib.rs"
],
"tags": [
"regression",
"escaped-failure",
"database",
"sqlite",
"macos",
"data-integrity",
"process-ownership"
],
"prompt": "On macOS, a live capture database can lose its process-wide SQLite ownership lock when another independently managed database generation or helper opens the same physical file. A foreign writable SQLite process can then attach while the original pools are still live, risking WAL-index invalidation and persistent disk-I/O failures. Keep exactly one live owner for each canonical physical database path, reject a duplicate before it can disturb the lock, avoid reopening an existing live file through a different SQLite path, release ownership only after authoritative pool shutdown, and allow a clean replacement after close. Preserve in-memory databases and safe missing-file recovery.",
"agent_timeout_seconds": 1800,
"grader_dependency_links": [
{
"source_path": "target",
"destination_path": "target"
}
],
"grader": {
"timeout_seconds": 1800,
"fixtures": [
{
"local_path": "graders/app-sqlite-process-ownership.rs",
"destination_path": "crates/screenpipe-db/tests/eval_sqlite_process_ownership.rs"
},
{
"source_path": "crates/screenpipe-db/tests/sqlite_architecture_invariants_test.rs"
}
],
"command": "CARGO_TARGET_DIR=${SCREENPIPE_EVAL_CARGO_TARGET_DIR:-target} cargo test -p screenpipe-db --test eval_sqlite_process_ownership --test sqlite_architecture_invariants_test"
}
},
{
"id": "app-chat-unicode-context-pruning",
"name": "Keep trimmed chat context valid Unicode",
"source": {
"kind": "git_regression",
"fix_commit": "5c9a0831b0f7e0e6b39e916d0c9bfbd481e2db97",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6563"
},
"base_ref": "aff38b0634d41b2f7c2cda9e65a3c77b354b15fc",
"oracle_ref": "5c9a0831b0f7e0e6b39e916d0c9bfbd481e2db97",
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-core/assets/extensions/context-pruning.ts",
"apps/screenpipe-app-tauri/lib/__tests__/context-pruning.test.ts"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"unicode",
"context-pruning",
"cross-platform"
],
"prompt": "Long chat context can fail before reaching the model when trimming cuts an emoji or another supplementary Unicode character in half, leaving malformed serialized text. Make every history-tail and generic head/tail trim boundary produce valid Unicode without exceeding its existing character budget. Preserve the newest conversation history, the complete current question, both ends of generic oversized content, and unchanged behavior for text already within budget.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/lib/__tests__/context-pruning.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/__tests__/context-pruning.test.ts"
}
},
{
"id": "app-search-accessibility-metadata-match",
"name": "Keep visible metadata matches from accessibility-captured frames",
"source": {
"kind": "git_regression",
"fix_commit": "656a05c68163a4e554f7958a2f0dc690f366412b",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6647"
},
"base_ref": "845dc283e938bee87cd5ed8e21a4891145ac348e",
"oracle_ref": "656a05c68163a4e554f7958a2f0dc690f366412b",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-keyword-search-store.tsx",
"apps/screenpipe-app-tauri/lib/hooks/__tests__/use-keyword-search-store.test.ts"
],
"tags": [
"regression",
"escaped-failure",
"search",
"accessibility",
"metadata",
"false-empty"
],
"prompt": "Desktop keyword search can receive a backend-authoritative result from an accessibility-captured frame, then erase it and show no results even when the query matches visible app, window, or URL metadata. Keep those metadata matches without presenting accessibility element bounds as precise text highlights. Continue to omit accessibility-only semantic-label matches, preserve OCR, hybrid, legacy, and fuzzy-prefix results, and do not add fallback screenshot OCR. The filter path must settle normally rather than reporting a query failure.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/lib/hooks/__tests__/use-keyword-search-store.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/hooks/__tests__/use-keyword-search-store.test.ts"
}
},
{
"id": "app-chat-jump-to-latest",
"name": "Replace the chat new-content pill with a quiet jump control",
"source": {
"kind": "git_regression",
"fix_commit": "a245a91bfc9b2aa1a944eb9ca40cc6a66c490132",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6690"
},
"base_ref": "fed5c6e37603a3fc8aeb0b6fd1b4ef2b041ebd60",
"oracle_ref": "a245a91bfc9b2aa1a944eb9ca40cc6a66c490132",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/chat-main-pane.tsx",
"apps/screenpipe-app-tauri/components/chat/standalone/chat-composer.tsx",
"apps/screenpipe-app-tauri/components/chat/standalone/chat-jump-to-latest.tsx",
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-chat-scroll.ts"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"scroll",
"disclosure"
],
"prompt": "When a chat is long enough to scroll, the control that appears after the user moves up says \"new content\" even when nothing new arrived. Replace that labeled notification with a quiet jump-to-latest control parked on the composer: icon only, sharp corners, no visible copy, stay mounted and faded while the viewport is already at the latest message, and keep stick-to-bottom while reading unchanged. Do not show the control as interactive on a short chat that cannot scroll.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/chat-jump-to-latest.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/chat-main-pane.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-chat-scroll.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/chat/standalone/chat-jump-to-latest.test.tsx components/chat/standalone/chat-main-pane.test.tsx components/chat/standalone/hooks/use-chat-scroll.test.tsx"
}
},
{
"id": "app-chat-trailing-tool-answer",
"name": "Preserve a completed answer before a trailing tool",
"source": {
"kind": "git_regression",
"fix_commit": "be90ee8fe24646fbee6aa6c1aae90c0716557de8",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6691"
},
"base_ref": "1de85ceefcb83c18a7f94f013c1f5ef67e639c96",
"oracle_ref": "be90ee8fe24646fbee6aa6c1aae90c0716557de8",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/message-content.tsx",
"apps/screenpipe-app-tauri/lib/chat/message-rendering.ts"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"rendering",
"tool-call",
"answer-loss"
],
"prompt": "A completed assistant turn can contain earlier narration, tool activity, a final answer, and then a trailing tool event. The renderer hides all text before the last tool, leaving a tool receipt and an empty gap even though the answer is stored. Render the last non-empty answer for completed turns that end on a tool, keep earlier intermediate narration hidden, keep running turns compact, preserve text emitted after the final tool, and align the assistant action surface with the answer that is actually visible.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/message-content.answer-recovery.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/message-content.agent-action.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/message-content.structured-output.test.tsx"
},
{
"source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/message-rendering.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/chat/standalone/message-content.answer-recovery.test.tsx components/chat/standalone/message-content.agent-action.test.tsx components/chat/standalone/message-content.structured-output.test.tsx lib/chat/__tests__/message-rendering.test.ts"
}
},
{
"id": "app-chat-recents-pool-eviction",
"name": "Keep Recents stable when the agent pool evicts a chat",
"source": {
"kind": "git_regression",
"fix_commit": "3d10440e5ad238a8790a74a220ec3631a26ca2e1",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6741"
},
"base_ref": "a47bfd71075c1d8d63f3b76f4d45fe17d414a5c9",
"oracle_ref": "3d10440e5ad238a8790a74a220ec3631a26ca2e1",
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/stores/pi-event-router.ts",
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-chat-session-runtime.ts",
"apps/screenpipe-app-tauri/components/standalone-chat.tsx"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"recents",
"session-eviction",
"ordering",
"state-preservation"
],
"prompt": "When several chats are running, the agent process pool can evict one chat. The app then removes that conversation from its in-memory index, so it disappears from Recents and a later event can recreate it untitled with new sort timestamps; an active evicted chat can also jump to a blank conversation. Treat process eviction as a runtime state change rather than conversation deletion: keep the existing conversation identity, title, timestamps, ordering, messages, and on-disk history; settle its loading and streaming state; ignore evictions for unknown conversations; and let the next send restart the agent under the same conversation ID. Preserve normal clean termination and background event routing.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"local_path": "graders/app-chat-recents-pool-eviction.test.ts",
"destination_path": "apps/screenpipe-app-tauri/lib/__tests__/pi-event-router.pool-eviction-eval.test.ts"
},
{
"source_ref": "a47bfd71075c1d8d63f3b76f4d45fe17d414a5c9",
"source_path": "apps/screenpipe-app-tauri/lib/__tests__/pi-event-router.test.ts"
},
{
"source_ref": "a47bfd71075c1d8d63f3b76f4d45fe17d414a5c9",
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-chat-session-runtime.test.tsx"
},
{
"source_ref": "a47bfd71075c1d8d63f3b76f4d45fe17d414a5c9",
"source_path": "apps/screenpipe-app-tauri/lib/__tests__/chat-store.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts lib/__tests__/pi-event-router.pool-eviction-eval.test.ts lib/__tests__/pi-event-router.test.ts components/chat/standalone/hooks/use-chat-session-runtime.test.tsx lib/__tests__/chat-store.test.ts"
}
},
{
"id": "app-chat-overlapping-load-isolation",
"name": "Keep overlapping chat loads isolated",
"source": {
"kind": "git_regression",
"fix_commit": "3aa56c5a12d7797f83d523ffdd045fa17ac232cf",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6772"
},
"base_ref": "10ae49880fcb8d072bc17b2c08522f1e4ec1f1d1",
"oracle_ref": "3aa56c5a12d7797f83d523ffdd045fa17ac232cf",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/hooks/use-chat-conversations.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/hooks/use-chat-conversations.ts",
"apps/screenpipe-app-tauri/lib/stores/chat-store.ts",
"apps/screenpipe-app-tauri/lib/chat-storage.ts"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"navigation",
"concurrency",
"data-corruption",
"state-isolation"
],
"prompt": "Rapid chat navigation can start a second conversation load while the first still awaits metadata or disk work. The eager session reference then points at the requested chat while the rendered transcript still belongs to the outgoing chat, so the wrong messages can be saved under another conversation and a stale request can reclaim the foreground. Bind every outgoing snapshot to the conversation and messages still rendered, and allow only the newest navigation to commit panel, current-session, settings, and preset state after asynchronous work. Keep each conversation's stored messages unchanged, leave the newest chat in the foreground when an older load finishes late, and preserve single-load behavior, temporary side-chat boundaries, stable ID fallbacks, concurrent save merging, browser state, and user-message timestamps.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"local_path": "graders/chat-overlapping-load.fixture.tsx",
"destination_path": "apps/screenpipe-app-tauri/lib/__tests__/save-conversation-race.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage node node_modules/vitest/vitest.mjs run --config vitest.config.ts lib/__tests__/save-conversation-race.test.tsx --pool=forks --poolOptions.forks.singleFork=true"
}
},
{
"id": "app-chat-renderer-reload-hydration",
"name": "Hydrate a saved chat after renderer reload",
"source": {
"kind": "git_regression",
"fix_commit": "4bc33459becbdf0f824243a1600aa6263f174768",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6766"
},
"base_ref": "664233bf1df466feafdcea117fb26f2787df7786",
"oracle_ref": "4bc33459becbdf0f824243a1600aa6263f174768",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-chat-window-events.ts",
"apps/screenpipe-app-tauri/components/standalone-chat.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-chat-window-events.ts",
"apps/screenpipe-app-tauri/components/standalone-chat.tsx",
"apps/screenpipe-app-tauri/lib/stores/chat-store.ts",
"apps/screenpipe-app-tauri/lib/chat-storage.ts"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"renderer-reload",
"hydration",
"false-empty",
"state-recovery"
],
"prompt": "After a chat renderer reload, the selected tab and conversation metadata can recover before the message pane. Routing sees that the requested conversation ID already matches the current session and returns early forever, leaving a blank transcript even though the saved conversation still contains messages. Treat matching IDs as complete only when the current view has renderable state; otherwise hydrate the known saved conversation instead of replacing it with a fresh blank chat. Preserve the fast path and current-session notification when the transcript is already rendered.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-chat-window-events.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts components/chat/standalone/hooks/use-chat-window-events.test.tsx"
}
},
{
"id": "app-pipe-history-tool-call-rendering",
"name": "Render tool activity from saved pipe history",
"source": {
"kind": "git_regression",
"fix_commit": "ce59063b095bc54ddad910938b0c77bcd9a46a24"
},
"base_ref": "d0b2d298c1c5167eb843b32f8e08939ee54badce",
"oracle_ref": "ce59063b095bc54ddad910938b0c77bcd9a46a24",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/pipe-ndjson-to-chat.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/pipe-ndjson-to-chat.ts",
"apps/screenpipe-app-tauri/components/chat/**"
],
"tags": [
"regression",
"escaped-failure",
"pipes",
"chat",
"tool-call",
"rendering",
"false-empty",
"ndjson"
],
"prompt": "Saved pipe execution history can contain valid assistant tool calls and results, but the chat view renders empty bubbles instead of showing what the pipe did. Make each parsed completed tool call render with its name and structured arguments, and attach its result to the same tool activity. Recover the name and arguments when streaming deltas are thin but the completed event contains them, and keep oversized results bounded. Preserve ordinary text, thinking, multi-turn ordering, agent-end fallback, malformed or truncated input, empty input, tool-result deduplication, unique message IDs, and conversation metadata.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/components/__tests__/pipe-ndjson-to-chat.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/__tests__/pipe-ndjson-to-chat.test.ts"
}
},
{
"id": "app-log-share-redaction-false-positive",
"name": "Preserve accessibility telemetry while redacting shared logs",
"source": {
"kind": "git_regression",
"fix_commit": "08feb4df5db8c6d95f58f5f1fe02930a450885c3"
},
"base_ref": "3828e565d41de1df99a24c26e69da0121f8c71c1",
"oracle_ref": "08feb4df5db8c6d95f58f5f1fe02930a450885c3",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/share-logs-button.tsx",
"apps/screenpipe-app-tauri/lib/utils/redact-pii.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/share-logs-button.tsx",
"apps/screenpipe-app-tauri/lib/utils/redact-pii.ts",
"crates/screenpipe-core/src/pii_removal.rs"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"log-sharing",
"redaction",
"accessibility",
"false-positive",
"data-integrity"
],
"prompt": "A shared support bundle includes recent chat context after removing personal information, but its phone-number detector also replaces bare accessibility coordinates and numeric telemetry. Preserve structured bounds, frame and monitor identifiers, hashes, process identifiers, pixel dimensions, and other unformatted numeric values in the uploaded bundle. Continue to remove formatted phone numbers, email addresses, government identifiers, payment-card numbers, bearer tokens, API credentials, IP addresses, and home-directory usernames. Do not solve the problem by disabling redaction or omitting the chat context.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"local_path": "graders/app-log-share-redaction.test.tsx",
"destination_path": "apps/screenpipe-app-tauri/components/__tests__/app-log-share-redaction.eval.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts components/__tests__/app-log-share-redaction.eval.test.tsx"
}
},
{
"id": "app-startup-auth-not-required",
"name": "Start signup-free builds without a false sign-in gate",
"source": {
"kind": "git_regression",
"fix_commit": "27fb5d84707fc778b6d5740f54943ee25b6128b5",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6810"
},
"base_ref": "7ded3c132196a0ce3b347469843b4693d6765ff6",
"oracle_ref": "27fb5d84707fc778b6d5740f54943ee25b6128b5",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/app-entitlement-gate.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/app-entitlement-gate.tsx",
"apps/screenpipe-app-tauri/components/providers/app-init-provider.tsx",
"apps/screenpipe-app-tauri/src-tauri/src/startup_auth.rs",
"apps/screenpipe-app-tauri/src-tauri/src/main.rs"
],
"tags": [
"regression",
"escaped-failure",
"auth",
"startup",
"signup-free",
"false-gate",
"recording"
],
"prompt": "A signup-free desktop build has already resolved startup authentication as not required, but the protected app still renders a sign-in gate because no user account exists. Make an explicit no-auth-required startup reach the application without stopping the recorder. Ordinary signed-out builds must still require sign-in; authenticated, enterprise, onboarding, development-skip, entitlement, hydration, recovery, and non-primary-window behavior must remain unchanged.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_ref": "27fb5d84707fc778b6d5740f54943ee25b6128b5",
"source_path": "apps/screenpipe-app-tauri/components/app-entitlement-gate.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts components/app-entitlement-gate.test.tsx"
}
},
{
"id": "app-enterprise-account-handoff-recording",
"name": "Keep recording authorized during enterprise account handoff",
"source": {
"kind": "git_regression",
"fix_commit": "666dd855cd1e6539f700426990e90114d06e97d8",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6841"
},
"base_ref": "c6721aae2caefb59ce7d8a839c732107332bb773",
"oracle_ref": "666dd855cd1e6539f700426990e90114d06e97d8",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-enterprise-policy.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-enterprise-policy.ts",
"apps/screenpipe-app-tauri/lib/hooks/__tests__/use-enterprise-policy.test.tsx"
],
"tags": [
"regression",
"escaped-failure",
"enterprise",
"auth",
"recording",
"account-handoff",
"race",
"fail-closed"
],
"prompt": "An enterprise desktop has a previously verified device grant, a policy that now requires account sign-in, and a saved account credential ready to check. During startup, evaluating the device credential first revokes native recording authorization before the viable account credential finishes, stopping capture and racing the later account grant. Keep recording authorized through that account handoff, then grant account authorization when verification succeeds. Preserve immediate fail-closed behavior for key-only devices, explicit account denial, invalid or missing credentials, rotated keys, transient failures, policy changes, and ordinary non-enterprise startup.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 420,
"fixtures": [
{
"source_ref": "666dd855cd1e6539f700426990e90114d06e97d8",
"source_path": "apps/screenpipe-app-tauri/lib/hooks/__tests__/use-enterprise-policy.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts lib/hooks/__tests__/use-enterprise-policy.test.tsx"
}
},
{
"id": "ai-gateway-nested-json-schema",
"name": "Accept standard nested JSON Schema response formats",
"source": {
"kind": "git_regression",
"fix_commit": "28da95884640708ced5db107a4e08b587c2e32d6",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6873"
},
"base_ref": "05a801d6103426108817849f99b26edb94e5be42",
"oracle_ref": "28da95884640708ced5db107a4e08b587c2e32d6",
"oracle_paths": [
"packages/ai-gateway/src/providers/openai.ts",
"packages/ai-gateway/src/providers/openrouter.ts",
"packages/ai-gateway/src/types.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/openai.ts",
"packages/ai-gateway/src/providers/openrouter.ts",
"packages/ai-gateway/src/types.ts",
"packages/ai-gateway/src/index.ts"
],
"tags": [
"regression",
"escaped-failure",
"ai-gateway",
"structured-output",
"json-schema",
"openai",
"openrouter",
"compatibility"
],
"prompt": "Valid structured-output requests using the standard nested response_format.json_schema shape are rejected before provider dispatch, although the equivalent legacy flattened form works. Accept the nested form for both OpenAI and OpenRouter while preserving flattened compatibility. When both forms are present, nested fields must take precedence; json_object requests must stay unchanged; and incomplete schemas must still fail before dispatch.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"local_path": "graders/ai-gateway-nested-json-schema.test.ts",
"destination_path": "packages/ai-gateway/src/test/eval-nested-json-schema.test.ts"
},
{
"source_ref": "28da95884640708ced5db107a4e08b587c2e32d6",
"source_path": "packages/ai-gateway/src/test/openai-models.unit.test.ts"
}
],
"command": "cd packages/ai-gateway && bun test src/test/eval-nested-json-schema.test.ts src/test/openai-models.unit.test.ts --timeout=10000"
}
},
{
"id": "ai-gateway-gpt5-output-limit-retry",
"name": "Recover from retryable GPT-5 output-limit rejections",
"source": {
"kind": "git_regression",
"fix_commit": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6904"
},
"base_ref": "d375badd50f99e194c251c8dbc6ba90dc476231e",
"oracle_ref": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",
"oracle_paths": [
"packages/ai-gateway/src/providers/openai.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/openai.ts",
"packages/ai-gateway/src/handlers/chat.ts",
"packages/ai-gateway/src/types.ts"
],
"tags": [
"regression",
"escaped-failure",
"ai-gateway",
"openai",
"gpt-5",
"output-limit",
"retry",
"bounded-recovery"
],
"prompt": "GPT-5 requests can fail before producing a response when the provider explicitly says the configured output limit was reached and asks the caller to retry with a higher limit. Recover that request with a bounded higher-limit retry instead of letting the model chain fail. Preserve the caller's initial limit until this exact rejection occurs, keep the retry within the supported GPT-5 ceiling, avoid a futile retry when already at that ceiling, leave non-GPT-5 and unrelated errors untouched, and preserve the provider's existing parameter-repair behavior and loop bound.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_ref": "ae5f134d90acd4e38cf0e3a8a234f33df3c4806f",
"source_path": "packages/ai-gateway/src/test/openai-param-retry.unit.test.ts"
}
],
"command": "cd packages/ai-gateway && bun test src/test/openai-param-retry.unit.test.ts --timeout=10000"
}
},
{
"id": "ai-gateway-anthropic-tool-input-object",
"name": "Normalize Anthropic tool-use inputs to objects",
"source": {
"kind": "git_regression",
"fix_commit": "b06e1ffa467989bb075540e0d2e86ae36dc03683",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6905"
},
"base_ref": "1a645b0ba4d0c51e00a2665e871050800ae6fac5",
"oracle_ref": "b06e1ffa467989bb075540e0d2e86ae36dc03683",
"oracle_paths": [
"packages/ai-gateway/src/providers/anthropic.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/anthropic.ts",
"packages/ai-gateway/src/types.ts",
"packages/ai-gateway/src/handlers/chat.ts"
],
"tags": [
"regression",
"escaped-failure",
"ai-gateway",
"anthropic",
"tool-use",
"schema",
"compatibility"
],
"prompt": "OpenAI-compatible assistant history can contain tool-call arguments that parse as JSON null, an array, a number, or a string. Forwarding those values as Anthropic tool-use input causes the provider to reject the request because tool input must be an object. Normalize every parsed non-object tool input to an empty object before Anthropic dispatch. Preserve valid object arguments unchanged, keep malformed JSON fail-safe, and preserve tool call identifiers and names, tool results, ordinary text and image messages, streaming behavior, and provider routing.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_ref": "b06e1ffa467989bb075540e0d2e86ae36dc03683",
"source_path": "packages/ai-gateway/src/test/anthropic-proxy.unit.test.ts"
}
],
"command": "cd packages/ai-gateway && bun test src/test/anthropic-proxy.unit.test.ts --timeout=10000"
}
},
{
"id": "meeting-audio-fallback-before-process-tap",
"name": "Preserve meeting audio until process capture is ready",
"source": {
"kind": "git_regression",
"fix_commit": "c366c0747298cf919b767095c3b5f6632ffdfcde",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6937"
},
"base_ref": "018f1ecfa4c7bd0c28210607209c4bff19aa9bd8",
"oracle_ref": "c366c0747298cf919b767095c3b5f6632ffdfcde",
"oracle_paths": [
"crates/screenpipe-audio/src/audio_manager/meeting_piggyback.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-audio/src/audio_manager/meeting_piggyback.rs",
"crates/screenpipe-audio/src/audio_manager.rs",
"crates/screenpipe-audio/src/core/device.rs"
],
"tags": [
"regression",
"escaped-failure",
"audio",
"meeting",
"microphone",
"fallback",
"capture-reliability",
"data-loss",
"bluetooth"
],
"prompt": "A detected meeting can start before process-level audio capture opens or while process microphone resolution is temporarily empty. In those states, preserve the existing fallback microphone and normal configured capture so neither side of the meeting is silently lost. Retry process capture only within the existing bound. Once process capture has opened, retain meeting capture ownership through transient gaps. When the meeting ends or the feature is disabled, clean up session devices and resume suspended capture. Preserve ordinary non-meeting behavior.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "target",
"destination_path": "target"
}
],
"grader": {
"timeout_seconds": 600,
"fixtures": [
{
"local_path": "graders/meeting-audio-fallback.mjs",
"destination_path": "crates/screenpipe-audio/eval-meeting-audio-fallback.mjs"
}
],
"command": "node crates/screenpipe-audio/eval-meeting-audio-fallback.mjs && cargo test -p screenpipe-audio --lib eval_meeting_audio_fallback -- --test-threads=2"
}
},
{
"id": "app-referral-lookup-session-preservation",
"name": "Keep authenticated accounts signed in during referral lookup",
"source": {
"kind": "git_regression",
"fix_commit": "1494836d8f34d2a388ae3393ac0e2d802b1c9d7a",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6959",
"issue": "https://github.com/screenpipe/screenpipe/issues/6943"
},
"base_ref": "1d9aff4972dc25cc5984ce995ccfceb5ee95f6b9",
"oracle_ref": "1494836d8f34d2a388ae3393ac0e2d802b1c9d7a",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/settings/referral-card.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/settings/referral-card.tsx",
"apps/screenpipe-app-tauri/lib/auth-guard.tsx",
"apps/screenpipe-app-tauri/lib/web-url.ts"
],
"tags": [
"regression",
"escaped-failure",
"auth",
"session-preservation",
"settings",
"referral",
"false-sign-out"
],
"prompt": "Opening referral settings can clear a valid signed-in account when the referral lookup is rejected as unauthenticated. A valid account with no referral code should remain signed in and see the eligibility explanation. Make referral lookup work with the current account, while preserving genuine session-expiry handling and the distinction between authentication failures and unrelated service errors. Preserve backend-issued referral links, copying, authenticated invitations, retry after a transient failure, and the sign-in option when no account is present.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_ref": "1494836d8f34d2a388ae3393ac0e2d802b1c9d7a",
"source_path": "apps/screenpipe-app-tauri/components/settings/referral-card.test.tsx"
},
{
"source_ref": "1494836d8f34d2a388ae3393ac0e2d802b1c9d7a",
"source_path": "apps/screenpipe-app-tauri/lib/auth-guard.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts components/settings/referral-card.test.tsx lib/auth-guard.test.tsx"
}
},
{
"id": "ai-gateway-input-limit-classification",
"name": "Recognize current provider input and tool-call limits",
"source": {
"kind": "git_regression",
"fix_commit": "739090783bb1116dca26621522e8034134e65028",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6945"
},
"base_ref": "655dfc3ad50fedfa9ba9561ebdf3a1467bc55517",
"oracle_ref": "739090783bb1116dca26621522e8034134e65028",
"oracle_paths": [
"packages/ai-gateway/src/handlers/chat.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/handlers/chat.ts",
"packages/ai-gateway/src/test/chat-fallback.unit.test.ts"
],
"tags": [
"regression",
"escaped-failure",
"ai-gateway",
"input-limit",
"tool-call",
"error-classification",
"false-positive"
],
"prompt": "The gateway fails to recognize provider errors saying input tokens exceed the configured limit or a request exceeds the available context size. It also misses an excess-length tool-call array, leaving the caller without useful payload guidance. Recognize these input-size and tool-call-limit failures on their supported client-error statuses, and provide actionable tool-call guidance without leaking the raw nested error. Preserve older input-limit phrasings, image and empty-message guidance, transient status handling, terminal safety outcomes, and bounded ordered fallback attempts. Do not classify unrelated schemas, non-tool arrays, thin messages, authentication errors, or server failures as these client payload problems.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"local_path": "graders/ai-gateway-input-limits.test.ts",
"destination_path": "packages/ai-gateway/src/test/eval-input-limits.test.ts"
}
],
"command": "cd packages/ai-gateway && bun test src/test/eval-input-limits.test.ts --timeout=10000"
}
},
{
"id": "sqlite-operational-fault-not-corruption",
"name": "Keep SQLite operational faults distinct from corruption verdicts",
"source": {
"kind": "git_regression",
"fix_commit": "968141d50e944a625230e3df60f243eb4be5f9d5",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6967"
},
"base_ref": "db8bd7ee23c02cbfc0eed2f4b9d99c70fd4cfc2d",
"oracle_ref": "968141d50e944a625230e3df60f243eb4be5f9d5",
"oracle_paths": [
"crates/screenpipe-sqlite-coordinator/src/lib.rs",
"crates/screenpipe-sqlite-coordinator/src/quarantine.rs",
"crates/screenpipe-sqlite-coordinator/src/verification.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-sqlite-coordinator/src/**",
"crates/screenpipe-db/src/db/**"
],
"tags": [
"regression",
"escaped-failure",
"sqlite",
"data-integrity",
"availability",
"quarantine",
"false-positive"
],
"prompt": "A single SQLite I/O failure or full-disk result currently creates corruption quarantine even when database damage has not been verified, unnecessarily routing a healthy database toward repair. Fix this boundary in the shared SQLite coordinator: retire the failed writer generation and retain a durable admission barrier, but do not declare corruption from an operational result alone. Preserve the original database bytes and physical identity. Pending faults must still block writes after process restart; repeated faults or recreating a writer gate must not reopen the failed generation. Preserve the first recorded fault, existing quarantine and malformed-metadata safety, ordinary contention and pool-pressure handling, and independent admission for unrelated databases. Keep the existing public fault-entry and coordination interfaces usable. This task covers incident classification and safe admission, not automatic repair or the complete recording-recovery lifecycle.",
"agent_timeout_seconds": 1800,
"grader_dependency_links": [
{
"source_path": "target",
"destination_path": "target"
}
],
"grader": {
"timeout_seconds": 600,
"fixtures": [
{
"local_path": "graders/sqlite-operational-fault.rs",
"destination_path": "crates/screenpipe-sqlite-coordinator/tests/eval_operational_fault.rs"
}
],
"command": "cargo test -p screenpipe-sqlite-coordinator --locked --test eval_operational_fault -- --test-threads=1"
}
},
{
"id": "app-mcp-config-symlink-preservation",
"name": "Preserve linked config files during explicit MCP setup",
"source": {
"kind": "git_regression",
"fix_commit": "fa7ce9ab5fa8865b0a9c8c044368d8c76f4434cc",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6977"
},
"base_ref": "53049736dd78d2c12c1a326d89adb8f9b53ef831",
"oracle_ref": "fa7ce9ab5fa8865b0a9c8c044368d8c76f4434cc",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/ai-tools-mcp.ts",
"apps/screenpipe-app-tauri/lib/utils/tauri.ts",
"apps/screenpipe-app-tauri/src-tauri/src/skills.rs",
"crates/screenpipe-engine/src/cli/agent.rs"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/ai-tools-mcp.ts",
"apps/screenpipe-app-tauri/lib/utils/tauri.ts",
"apps/screenpipe-app-tauri/src-tauri/src/skills.rs",
"crates/screenpipe-engine/src/cli/agent.rs"
],
"tags": [
"regression",
"escaped-failure",
"mcp",
"configuration",
"symlink",
"data-integrity",
"false-positive"
],
"prompt": "Explicit MCP connect and disconnect in Settings replace a symlinked config with a regular file, leaving its dotfiles target unchanged. Preserve absolute and relative link chains when updating Codex TOML or Cursor JSON configurations: update the actual target, keep unrelated settings and servers, and back up the previous target contents beside that target. Dangling links, loops, and resolution errors must fail without replacing the link or creating a fresh config. Ordinary missing files must remain creatable, invalid or unreadable configs must remain untouched, backups must stay bounded, and disconnect must remain idempotent. Keep the existing public connect/remove interfaces usable. This task covers Settings-side configuration IO at the platform path-resolution boundary, not native desktop acceptance or launch-time reconciliation.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"source_path": "apps/screenpipe-app-tauri/lib/__tests__/ai-tools-mcp.test.ts"
},
{
"local_path": "graders/app-mcp-config-symlinks.test.ts",
"destination_path": "apps/screenpipe-app-tauri/lib/__tests__/eval-mcp-config-symlinks.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts lib/__tests__/ai-tools-mcp.test.ts lib/__tests__/eval-mcp-config-symlinks.test.ts"
}
},
{
"id": "app-provider-outage-vs-limit",
"name": "Separate hosted outages from usage limits",
"source": {
"kind": "git_regression",
"fix_commit": "42593c9b330a66ff39bc632bf87b2f34ad801b55",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6897"
},
"base_ref": "f04a0c8071ee38397cfe138e4ec3e1f433229ddd",
"oracle_ref": "42593c9b330a66ff39bc632bf87b2f34ad801b55",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat/provider-errors.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/chat/provider-errors.ts"
],
"tags": [
"regression",
"escaped-failure",
"classification",
"preserved-behavior"
],
"prompt": "Screenpipe hosted chat labels service-unavailable errors as usage throttling and suggests upgrading. Treat hosted unavailability, including differently cased and status-prefixed messages, as a retryable cloud outage with retry guidance and no quota or upgrade claim. Preserve explicit rate-limit guidance (wait/retry and upgrade options), real daily allowance limits and tomorrow guidance, ordinary cloud connectivity recovery, local Ollama recovery instructions with the configured model, and non-retryable model refusals. Keep the existing public provider error message/presentation interfaces. Grade user-visible semantics rather than exact sentences; no live provider calls are needed.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/app-provider-outage-vs-limit.test.ts",
"destination_path": "evals/coding-agent/graders/app-provider-outage-vs-limit.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test ./evals/coding-agent/graders/app-provider-outage-vs-limit.test.ts"
}
},
{
"id": "app-entitlement-plan-consistency",
"name": "Preserve verified entitlement plans when the API omits a plan label",
"source": {
"kind": "git_regression",
"fix_commit": "6ce45931c33a89cf27675e947bea3abec3d80159",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/5781"
},
"base_ref": "f0e9248ed600ee83e84e216e026ff98b6eead0e7",
"oracle_ref": "6ce45931c33a89cf27675e947bea3abec3d80159",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/app-entitlement.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/app-entitlement.ts"
],
"tags": [
"regression",
"escaped-failure",
"entitlement",
"normalization",
"preserved-behavior"
],
"prompt": "A paid account becomes unknown to local plan policy when an API response omits subscription_plan or returns it as null, even though the response includes a fresh, active entitlement for the correct plan. Preserve valid lifetime, pro_max and pro_ultra plan identity and verified paid consumer access when that top-level label is absent; cloud access must still follow the entitlement. Keep the normalized account identity and supplied session token. An explicitly supplied label, including an empty string or a conflicting plan, retains precedence; conflicting or empty evidence must not grant paid access. Explicit app denial with no cloud subscription must retain free policy. Preserve boolean-only legacy standard/pro fallbacks, and ignore null, whitespace-only or non-string entitlement plan labels as fallback candidates without treating malformed evidence as verified paid. Stale, future-dated or inactive non-lifetime evidence and accounts without stable identity must remain unknown. Enterprise entitlements must not become consumer subscriptions. Keep the existing normalization and policy interfaces. This task concerns pure account classification, not gate rendering, telemetry, live billing or recorder behavior.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/app-entitlement-plan-consistency.test.ts",
"destination_path": "evals/coding-agent/graders/app-entitlement-plan-consistency.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test ./evals/coding-agent/graders/app-entitlement-plan-consistency.test.ts"
}
},
{
"id": "ai-gateway-verified-identity",
"name": "Require verified account ownership before authenticated gateway tiers",
"source": {
"kind": "git_regression",
"fix_commit": "252b96c168e79a6e377f307cf1e54423d17e0db8"
},
"base_ref": "041f111810b334ac3667006907f1dc5b511273df",
"oracle_ref": "252b96c168e79a6e377f307cf1e54423d17e0db8",
"oracle_paths": [
"packages/ai-gateway/src/utils/auth.ts"
],
"tags": [
"regression",
"escaped-failure",
"security",
"identity",
"entitlement"
],
"prompt": "The gateway can treat a public database UUID or Clerk user ID supplied as a Bearer or Token credential as an authenticated account, even when no token verifier established ownership. Repair the authentication boundary so those public identifiers cannot grant logged-in or subscribed tiers or trigger account/subscription lookups. Preserve anonymous access and device identity for absent, invalid, or unproven credentials; preserve verified Clerk and successfully validated legacy website tokens at their proper logged-in/subscribed tier and canonical identity. A verifier result requires a nonempty string subject; a legacy website HTTP 200 alone is not proof of success and requires success=true with a user identity. Missing, rejected and unavailable verification must not manufacture an identity. Use synthetic credentials only. This task concerns validateAuth behavior; no particular helper structure is required.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/gateway-verified-identity.test.js",
"destination_path": "evals/coding-agent/graders/gateway-verified-identity.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/gateway-verified-identity.test.js"
}
},
{
"id": "ai-gateway-billing-capacity-compatibility",
"name": "Resolve canonical billing capacity without widening access",
"source": {
"kind": "git_regression",
"fix_commit": "d9ff95802e6566a7b3ea3242800aef59cacb9aa0",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/5779"
},
"base_ref": "8b455607ffbaef204a6f6a0a62e973bac6211e48",
"oracle_ref": "d9ff95802e6566a7b3ea3242800aef59cacb9aa0",
"oracle_paths": [
"packages/ai-gateway/src/utils/auth.ts"
],
"gate": "advisory",
"tags": [
"regression",
"ai-gateway",
"billing",
"entitlement",
"compatibility",
"identity"
],
"prompt": "Customers with Max or Ultra billing plans receive ordinary Business capacity when the website returns the older compatible pro/Business access labels. Repair validateAuth so canonical billing_plan refines a matching, active app entitlement to the purchased capacity. Preserve normalized aliases and legacy responses with no billing_plan, ordinary Basic/Business access and explicit app/cloud denial. Billing data must not independently establish access: reject null, empty, unknown, free, malformed or incompatible billing values as unknown plan truth, and require matching subscription/entitlement plans plus app_entitled, entitlement.active and features.app. Keep broad cloud subscription access separate from capacity, and do not issue a power usage tier without subscribed access. Bind plan truth to the verified Clerk subject; mismatched identity and unsuccessful or unavailable website responses must not transfer capacity. Retry unknown or mismatched plan responses on the next request; retain caching of valid paid plan truth. Preserve anonymous access without credentials and the dedicated service credential. Exercise synthetic credentials only; no live billing operations and no prescribed helper structure.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/gateway-billing-capacity.test.js",
"destination_path": "evals/coding-agent/graders/gateway-billing-capacity.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/gateway-billing-capacity.test.js"
}
},
{
"id": "mcp-team-credential-reload",
"name": "Reload enterprise credentials in the running MCP server",
"source": {
"kind": "git_regression",
"fix_commit": "909c9360a4379896df49507140017e9989d41428"
},
"base_ref": "e681fc3f44f44fc377c771988814db00021b2e84",
"oracle_ref": "909c9360a4379896df49507140017e9989d41428",
"oracle_paths": [
"packages/screenpipe-mcp/src/index.ts",
"packages/screenpipe-mcp/src/team-config.ts"
],
"tags": [
"regression",
"escaped-failure",
"mcp",
"enterprise",
"identity"
],
"prompt": "The desktop can save a replacement enterprise token or team gateway while its MCP server keeps running, but team requests continue using stale credentials. Make the next team-tool request and tool listing reflect the current saved configuration without restarting the process, including credentials saved for the first time. Keep token and gateway from the same configuration snapshot. Clearing, deleting or corrupting the saved configuration must hide unavailable team tools and reject their calls before any team request; restoring valid settings must recover. Preserve local tools and explicit environment-token overrides, and preserve gateway precedence: command-line override, environment override, saved gateway, then hosted default. An expired explicit token must stay authoritative and give recovery guidance identifying the environment override and enterprise.json settings without exposing the credential. Verify observable requests and results through the running MCP process using synthetic tokens and local test servers.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "packages/screenpipe-mcp/node_modules",
"destination_path": "packages/screenpipe-mcp/node_modules"
}
],
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/mcp-team-reload.fixture.ts",
"destination_path": "packages/screenpipe-mcp/src/eval-team-auth.test.ts"
}
],
"command": "cd packages/screenpipe-mcp && SCREENPIPE_DISABLE_TELEMETRY=1 node node_modules/vitest/vitest.mjs run src/eval-team-auth.test.ts"
}
},
{
"id": "grok-installer-explicit-consent",
"name": "Require explicit action before the Grok installer reads credentials",
"source": {
"kind": "git_regression",
"fix_commit": "b6b18e4fa628e53bbf7eccef974faf04a07a95cc"
},
"base_ref": "b247eb3c674af026fe28c628a098e9f8c748c8a5",
"oracle_ref": "b6b18e4fa628e53bbf7eccef974faf04a07a95cc",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/grokbot-installer.mjs"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"credentials",
"consent"
],
"prompt": "The Grok installer credential bridge currently accepts passive status requests and reads another application's credential files. Require an explicit connect or disconnect action before discovering the application files, reading protected credentials or contacting its gateway. Reject status and all other unsupported actions before those effects, whether or not the app is installed. Preserve explicit connect and disconnect: use the existing saved gateway credential only for its authorized gateway requests, verify installation, retry without creating duplicate skills, remove only this device's owned skill on disconnect, and preserve unrelated user workflows. Explicit requests with no installed app should report not detected/disconnected without credential or network access. Credential-command and network failures must not disclose credentials in returned errors. Verify the actual JavaScript installer entry point using synthetic files and provider/OS ports. Native status-cache behavior, UI consent and Windows credential handling are outside this task.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/grok-installer-consent.test.js",
"destination_path": "evals/coding-agent/graders/grok-installer-consent.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/grok-installer-consent.test.js"
}
},
{
"id": "app-private-reasoning-budget",
"name": "Bound Private model reasoning while reserving answer and tool output",
"source": {
"kind": "git_regression",
"fix_commit": "6666ed74c051bb536d255fb4520702ae520f9c3d"
},
"base_ref": "5e6d3712c4034c67e600a184dcde5b6e19e16ec2",
"oracle_ref": "6666ed74c051bb536d255fb4520702ae520f9c3d",
"oracle_paths": [
"crates/screenpipe-core/assets/extensions/lib/tinfoil-transport.ts"
],
"tags": [
"regression",
"escaped-failure",
"private-ai",
"transport",
"resource-bounds"
],
"prompt": "Private requests can consume their generation allowance on reasoning or lose the reasoning delimiter. Fix the real GLM request transport to enable the delimiter and translate ordinary reasoning settings into supported low/high/max values: high maps to high, xhigh/max map to max, all other values map to low. Bound reasoning per response at 512/2048/4096 tokens for low/high/max, while reserving 1024 tokens for answer or tool output and never producing a negative budget. Use max_completion_tokens before max_tokens and an 8192 default when neither is supplied; preserve supplied output limits. Body-supplied template or reasoning-budget overrides must not evade this policy. Preserve the existing verified-client boundary: verify before sending, no plaintext fallback, only the configured HTTPS endpoint/model/POST, propagate cancellation, forward the current authorization and content, refuse redirects, reuse a client for unchanged credentials and isolate its cache after credential rotation. Exercise the actual adapter with a synthetic verified-client port. Actual encryption, attestation cryptography and live model output are outside this task.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/private-reasoning-budget.test.ts",
"destination_path": "evals/coding-agent/graders/private-reasoning-budget.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/private-reasoning-budget.test.ts"
}
},
{
"id": "app-preset-deletion-save-recovery",
"name": "Reassign preset dependencies before committing deletion",
"source": {
"kind": "git_regression",
"fix_commit": "923294f6848b0aaf5800aee3b33b1f07a9eb9c3f"
},
"base_ref": "18661dfab746eb2d1f37e224ef8c4f91841473ff",
"oracle_ref": "923294f6848b0aaf5800aee3b33b1f07a9eb9c3f",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx",
"apps/screenpipe-app-tauri/lib/ai-preset-deletion.ts",
"apps/screenpipe-app-tauri/lib/chat-storage.ts"
],
"tags": [
"regression",
"escaped-failure",
"settings",
"automation",
"persistence",
"retry"
],
"prompt": "Deleting an AI preset leaves scheduled tasks and saved conversations referring to a preset that no longer exists. Fix the actual settings update path so affected task chains, saved conversations including hidden explicit task destinations, legacy conversation settings, activity settings, open chat selections and the active selection move to a surviving default before deletion is committed. Preserve unrelated presets, wildcard/default task references, conversation content and timestamps, and a newer conversation selection made after dependency discovery. Deduplicate rewritten task chains. Discovery, task-write or conversation-write failures must reject deletion, retain the original durable preset and support retry after partial dependency success without rewriting already-repaired task references. A failed final settings save must not announce committed changes or update active/open selections. Model edits and reordering must not trigger dependency work; empty preset lists must be refused. Concurrent queued unrelated settings updates must preserve both changes. No particular helper name or extraction is required. UI rendering, native filesystem guarantees and retry after a failed final store save are outside this task.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/preset-save-recovery.test.js",
"destination_path": "evals/coding-agent/graders/preset-save-recovery.test.js"
}
],
"command": "cd apps/screenpipe-app-tauri && \"${BUN_BIN:-bun}\" test ../../evals/coding-agent/graders/preset-save-recovery.test.js"
}
},
{
"id": "app-onboarding-fresh-assignment-login",
"name": "Resolve absent fresh flags and retain login completion across assignment",
"source": {
"kind": "git_regression",
"fix_commit": "b2a2b1a8c33cdba993c81d063065b86856f94acb",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6921"
},
"base_ref": "590f6e55f57e161d7e66e550bd39bb9e86384481",
"oracle_ref": "b2a2b1a8c33cdba993c81d063065b86856f94acb",
"oracle_paths": [
"apps/screenpipe-app-tauri/app/onboarding/page.tsx",
"apps/screenpipe-app-tauri/components/onboarding/login-gate.tsx"
],
"tags": [
"regression",
"escaped-failure",
"onboarding",
"auth",
"trial",
"recovery"
],
"prompt": "A fresh desktop login can lose its completion event when experiment assignment temporarily replaces the login screen, and a successful fresh flag response that omits the disabled experiment leaves setup waiting until timeout. Resolve an absent flag from the final authenticated identity immediately to the persisted control route without reporting assignment failure. Record onboarding_login_completed exactly once for a real signed-out to signed-in transition, both with and without assignment, even if the login screen unmounts; settings hydration, an already-authenticated resume, repeated callbacks, and advancing without a new auth transition must not count as another login. Preserve final-identity freshness against cached and earlier in-flight flag responses, explicit loading-error fallback, a pinned route after checkout navigation, and summary-first enrollment for eligible fresh installs with no plan label. Preserve the paid-account fresh-install bypass, never enroll upgraded existing installs after reset, and do not start the standard funnel for managed or resumed onboarding. Keep enrollment telemetry free of email. Exercise the real React page and login screen with synthetic provider/native ports; do not contact live services or change billing, native publication, or recording behavior. No particular internal state variable or telemetry-owner component is required.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 120,
"fixtures": [
{
"source_ref": "b2a2b1a8c33cdba993c81d063065b86856f94acb",
"source_path": "apps/screenpipe-app-tauri/app/onboarding/page.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts app/onboarding/page.test.tsx -t 'continues immediately when fresh authenticated flags|records a fresh login once|does not count an authenticated settings hydration|records one fresh login without|does not expose the control checkout|falls back explicitly to control|restores the pinned route|assigns a fresh-install treatment user|uses a paid account only|never enrolls an upgraded free install|does not start the standard funnel for managed|does not start a new funnel when standard onboarding resumes|leaves login completion analytics' --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "ai-gateway-glm-bounded-tool-history",
"name": "Bound compacted Pi tool history before GLM dispatch",
"source": {
"kind": "git_regression",
"fix_commit": "7f626da4ffde0e838a022044047a2835660b263d"
},
"base_ref": "0f2423d2976e04455ac51fde2879f00bbc491cd6",
"oracle_ref": "7f626da4ffde0e838a022044047a2835660b263d",
"oracle_paths": [
"packages/ai-gateway/src/providers/screenpipe-glm.ts"
],
"tags": [
"regression",
"escaped-failure",
"ai-gateway",
"glm",
"context",
"streaming"
],
"prompt": "Large Pi tool results still exceed the GLM context budget after the gateway compacts its skill catalog. Before both streaming and non-streaming GLM provider dispatch, bound oversized string tool results to at most 8,000 characters when the request has a system/developer available-skills catalog (including text-part arrays) or the subagent function tool. Preserve at least the first and last 128 characters, preserve tool-call/result pairing and message order, and mark the omission with guidance to reread a narrower range. Keep results at or below 8,000 characters unchanged. Preserve user and assistant prose, original input objects, normal completion/stream delivery, model routing, ordinary tools/tool-choice, existing subagent removal, and the useful minimum output allowance for compacted Pi requests. Ordinary clients and requests with catalog text only in a user message must retain their full tool results and intentional small output limits. Other OpenAI-compatible providers must retain their original context. Check the real provider methods and outgoing SDK request with synthetic external ports; no live model calls. Any head/tail allocation satisfying these outcomes is acceptable.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/ai-gateway-glm-tool-history.test.js",
"destination_path": "evals/coding-agent/graders/ai-gateway-glm-tool-history.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/ai-gateway-glm-tool-history.test.js"
}
},
{
"id": "ai-gateway-glm-tool-call-boundary",
"name": "Convert recognized GLM tool responses without treating ordinary text as actions",
"source": {
"kind": "git_regression",
"fix_commit": "da9ca5d1be7ef669cf7edec1d5fba7b99afc6de6"
},
"base_ref": "50a503a51e0b6f6a529aa2f4751df36ba4a756a7",
"oracle_ref": "da9ca5d1be7ef669cf7edec1d5fba7b99afc6de6",
"oracle_paths": [
"packages/ai-gateway/src/providers/screenpipe-glm.ts"
],
"tags": [
"regression",
"escaped-failure",
"tool-calls",
"streaming",
"gateway"
],
"prompt": "GLM returns native function-call text that Pi cannot execute. Repair the actual GLM completion and streaming provider paths so schema-known leading calls become OpenAI tool calls with nonempty distinct IDs, correct function names and argument objects, no duplicated raw call text, and a tool_calls finish reason. Support tagged XML values (string, number, boolean and object types, preserving string leading zeros and Unicode), XML containing JSON arguments, flat JSON call objects, nested function objects with JSON-string arguments, and multiple XML calls. Streaming must work across one-character and seven-character content fragments and retain a single completion terminator. Forward the caller\u2019s declared tools and tool choice and preserve input objects. Unknown tool names, prefixed explanatory prose, malformed JSON and array/null arguments remain unchanged ordinary text; no tools means no conversion. Preserve already-native tool-call IDs/arguments and other OpenAI providers. Use synthetic provider transport, without live calls or executing tools. No particular parser/helper name or ID format is required.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/glm-tool-call-boundary.test.js",
"destination_path": "evals/coding-agent/graders/glm-tool-call-boundary.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/glm-tool-call-boundary.test.js"
}
},
{
"id": "app-meeting-summary-handoff",
"name": "Keep completed meeting summaries visible until authoritative save catches up",
"source": {
"kind": "git_regression",
"fix_commit": "fc5a65ffbfc785d3b2e2240096ef33fa71d2f5c2",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6846"
},
"base_ref": "d86ad74d99fe6df68232ae04fcb6b7b5b3a99c40",
"oracle_ref": "fc5a65ffbfc785d3b2e2240096ef33fa71d2f5c2",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/meeting-notes/note-view.tsx",
"apps/screenpipe-app-tauri/components/meeting-notes/meeting-workspace.tsx"
],
"tags": [
"regression",
"escaped-failure",
"meetings",
"streaming",
"persistence",
"recovery"
],
"prompt": "A completed meeting summary disappears while its saved note is still catching up. Repair the real note view and summary surface so the matching streamed summary remains visible when execution changes from running to ready. With either no prior summary or an older saved summary, do not report a stale completed-note read as a successful save. Continue reading until the new summary is visible, including after a temporary failed read, and deliver the refreshed note without remounting. Once that execution has been refreshed, do not repeatedly reload its saved note. Preserve an already-saved summary when opening an initially completed meeting, ignore unrelated pipe/execution streams, do not refresh the saved note while execution is still running, and stop polling and event subscriptions on unmount. Preserve legitimate note autosave callbacks and existing title/content. Verify observable UI and save-boundary outcomes using synthetic local API, native and analytics ports. No live meeting data, provider calls or native builds are needed. No particular helper name or state layout is required.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 120,
"fixtures": [
{
"local_path": "graders/meeting-handoff.eval.test.tsx",
"destination_path": "apps/screenpipe-app-tauri/meeting-handoff.eval.test.tsx"
},
{
"local_path": "graders/meeting-handoff.config.ts",
"destination_path": "apps/screenpipe-app-tauri/meeting-handoff.config.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config meeting-handoff.config.ts"
}
},
{
"id": "ai-gateway-enterprise-allowance",
"name": "Carry verified enterprise allowance from authenticated account truth to gateway metadata",
"source": {
"kind": "git_regression",
"fix_commit": "8663ecbef34d7cd786b064a541c0e3740f4a3c14",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/6837"
},
"base_ref": "d52da0a6484f1b54a349224ed67c6ea52637dcb3",
"oracle_ref": "8663ecbef34d7cd786b064a541c0e3740f4a3c14",
"oracle_paths": [
"packages/ai-gateway/src/utils/auth.ts",
"packages/ai-gateway/src/services/cloudflare-ai-gateway.ts"
],
"gate": "advisory",
"tags": [
"regression",
"auth",
"enterprise",
"gateway",
"entitlement",
"capacity"
],
"prompt": "Verified enterprise users inherit a lower consumer allowance when authenticated website account truth includes both enterprise membership and a compatible consumer billing plan. Repair the authentication-to-gateway handoff so an explicit boolean enterprise flag produces enterprise account identity and business_ultra gateway metadata after the existing complete, active, matching entitlement checks pass. Preserve compatible Pro, Max and Ultra billing fields, and ensure the serialized metadata reaches both OpenAI and Anthropic gateway connection headers. A false, missing or string enterprise flag must not upgrade an ordinary consumer. Inactive or denied app entitlement, missing app features, conflicting entitlement/billing plans, wrong verified subject and unsuccessful website responses must not mint paid gateway capacity. Explicit app/cloud denial stays free despite stale enterprise labels. Preserve consumer Max/Ultra capacity, cached verified account identity across device changes, service internal allowance and anonymous refusal without account lookup. Metadata must use a stable hashed account identifier and retain auto lane, caller workload and trial status without exposing the account ID or bearer token. Keep payload logging disabled, remove direct provider authorization headers and keep provider retry amplification disabled. Exercise actual authentication, gateway context and connection functions using synthetic verification/website ports; no live identity, billing, provider or native operations. No particular helper name is required.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/enterprise-allowance.test.js",
"destination_path": "evals/coding-agent/graders/enterprise-allowance.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/enterprise-allowance.test.js"
}
},
{
"id": "app-restricted-enterprise-onboarding",
"name": "Gate restricted enterprise identity when onboarding resolves consumer build",
"source": {
"kind": "git_regression",
"fix_commit": "46a170f92ea371feecc5a8db139d2fc1e3cac8ba"
},
"base_ref": "e594a3c7ab8bed3e90310ff05af4c89cff32a166",
"oracle_ref": "46a170f92ea371feecc5a8db139d2fc1e3cac8ba",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/app-entitlement-gate.tsx"
],
"tags": [
"regression",
"onboarding",
"enterprise",
"access",
"identity"
],
"gate": "advisory",
"prompt": "A restricted enterprise member can continue consumer onboarding after sign-in because the onboarding route bypasses the enterprise-app gate. Once build detection confirms the consumer app, hide protected onboarding content and show the enterprise-app-required state for a signed-in member whose organization requires that app and restricts consumer access, even if the member also has a paid personal subscription. Observe both live sign-in and a cached identity while detection resolves. Preserve ordinary consumer onboarding, tokenless onboarding, paid personal access when the organization does not restrict it, and accounts whose organization does not require the enterprise app. Do not misclassify an unresolved or managed enterprise build; managed onboarding keeps ownership of its authentication flow. Preserve explicit not_required startup semantics, including cached restricted identities, and do not stop recording on these permitted onboarding paths. Test the real React gate with synthetic settings/managed-policy/native/provider ports. This case covers visible gate decisions and attempted command effects, not real recorder durability or native authentication. No helper or state variable name is required.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 120,
"fixtures": [
{
"local_path": "graders/restricted-onboarding.fixture.tsx",
"destination_path": "apps/screenpipe-app-tauri/components/restricted-onboarding.eval.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage ./node_modules/.bin/vitest run --config vitest.config.ts components/restricted-onboarding.eval.test.tsx --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "app-local-preset-preservation",
"name": "Preserve local-only presets through settings and account refresh",
"source": {
"kind": "git_regression",
"fix_commit": "5c3661a0b979e5cd6204ac20fd8872a274efe47a"
},
"base_ref": "45778f74227cbff7a83f65265e834b5985f7e72b",
"oracle_ref": "5c3661a0b979e5cd6204ac20fd8872a274efe47a",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx"
],
"tags": [
"regression",
"escaped-failure",
"settings",
"account",
"local-model"
],
"prompt": "A local-only AI preset configuration unexpectedly gains a cloud preset when settings reload or the account changes. Fix the actual settings persistence path so existing nonempty configurations, including custom model, prompt, URL, default selection and ordering, survive cold loads, repeated reloads, sign-in, subscription changes and sign-out unchanged. Explicit valid preset edits and unrelated settings changes must still persist, and intentional mixed cloud/local configurations must remain intact. Persisted missing, malformed or empty preset lists must recover usable defaults. Explicit empty or malformed preset updates must reject before any persistence or events, leaving prior settings intact. Preserve existing managed-policy behavior. No particular helper or internal structure is required. Rendered deletion controls, native durability and account network verification are outside this task.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/local-preset-preservation.test.js",
"destination_path": "evals/coding-agent/graders/local-preset-preservation.test.js"
}
],
"command": "cd apps/screenpipe-app-tauri && \"${BUN_BIN:-bun}\" test ../../evals/coding-agent/graders/local-preset-preservation.test.js"
}
},
{
"id": "mcp-live-database-boundary",
"name": "Keep MCP authentication outside the live recording database",
"source": {
"kind": "git_regression",
"fix_commit": "a43b9757f3133a842bbe0d7764fff97e31f199c9"
},
"base_ref": "c92d8e5fd67b724846f2b9db5cf32a17fe58f4a5",
"oracle_ref": "a43b9757f3133a842bbe0d7764fff97e31f199c9",
"oracle_paths": [
"packages/screenpipe-mcp/src/index.ts"
],
"tags": [
"regression",
"escaped-failure",
"mcp",
"recording",
"data-integrity",
"authentication"
],
"prompt": "The MCP authentication discovery fallback can bypass Screenpipe and read the live recording database when environment credentials and CLI discovery fail. Keep database ownership inside Screenpipe: the MCP process must not probe, open, copy or mutate db.sqlite or its WAL/shared-memory companions, or spawn an external SQLite client to recover credentials. With missing credentials or invalid CLI output, keep MCP initialization and tool listing available, surface the backend authentication failure clearly, and never invent a key or report recording results as success. Preserve local environment-key precedence over the legacy environment key, support the legacy key, and preserve successful discovery through the bundled CLI, node-adjacent npx and PATH-based npx. Repeated search calls should reuse discovery and send the resolved credential to the configured Screenpipe API. Verify observable tool responses, authenticated requests and forbidden filesystem/subprocess effects using synthetic inputs; do not access real recordings or credentials.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "packages/screenpipe-mcp/node_modules",
"destination_path": "packages/screenpipe-mcp/node_modules"
}
],
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/mcp-db-boundary.fixture.ts",
"destination_path": "packages/screenpipe-mcp/src/eval-db-boundary.test.ts"
},
{
"local_path": "graders/mcp-db-boundary.preload.cjs",
"destination_path": "packages/screenpipe-mcp/src/eval-db-boundary.preload.cjs"
}
],
"command": "cd packages/screenpipe-mcp && SCREENPIPE_DISABLE_TELEMETRY=1 node node_modules/vitest/vitest.mjs run src/eval-db-boundary.test.ts"
}
},
{
"id": "ai-gateway-image-cache-boundary",
"name": "Keep image payloads free of text-only prompt cache markers",
"source": {
"kind": "git_regression",
"fix_commit": "8696fbe7851f82f1bc1164929e860021790e928c"
},
"base_ref": "89ca135f648990b52655d5e21f3d334bb72ae3cc",
"oracle_ref": "8696fbe7851f82f1bc1164929e860021790e928c",
"oracle_paths": [
"packages/ai-gateway/src/providers/openai.ts"
],
"tags": [
"regression",
"escaped-failure",
"ai-gateway",
"media",
"provider-compatibility"
],
"prompt": "Explicit GPT-5.6 prompt caching can attach text-only cache breakpoint fields to image payloads and cause provider rejection. Fix the actual OpenAI provider request path for both streaming and non-streaming calls. Keep image data, detail and order intact through existing image_url, native image and base64-image normalization; do not add cache fields to image blocks or drop images to avoid rejection. Keep an instruction breakpoint and mark the last nonempty text in eligible multipart history, including text that precedes trailing images; image-only history should stay unchanged without a history marker. Preserve the final-text boundary, disabled history caching and older models without explicit cache fields. Successful provider content must still reach the caller. Verify sent request snapshots and returned responses with a synthetic SDK; do not make real model calls. Audio/file/refusal normalization is outside this task.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/openai-media-cache.test.js",
"destination_path": "evals/coding-agent/graders/openai-media-cache.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/openai-media-cache.test.js"
}
},
{
"id": "app-expired-certificate-recovery",
"name": "Recover from expired TLS certificates with provider-specific guidance",
"source": {
"kind": "git_regression",
"fix_commit": "bc2d014170e9cd8b81785ddf5fab6441c23d12f4"
},
"base_ref": "791c2018eecb6ff2d602bf6d9cff3ac5232b6df8",
"oracle_ref": "bc2d014170e9cd8b81785ddf5fab6441c23d12f4",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat/provider-errors.ts"
],
"tags": [
"regression",
"escaped-failure",
"provider-recovery",
"preserved-behavior"
],
"prompt": "Chat displays an unclassified raw Error: certificate has expired failure. Fix the public provider-error presentation and message interfaces so that error, including letter-case variations, becomes retryable provider-specific connection guidance. Hosted screenpipe-cloud and pi presets need Screenpipe Cloud retry guidance without a quota/upgrade claim; remote providers need connection/retry guidance; native Ollama needs its local service-start instruction and configured model. Do not echo the raw certificate error. Keep the message and presentation interfaces consistent. Preserve null fallback for unrecognized failures and ordinary prose mentioning expired certificates, ordinary cloud TLS recovery, distinct rate-limit guidance, nonretryable model safety refusals and custom-provider authentication guidance about the key and endpoint. No live providers, UI or native build are needed; grade observable classifications and guidance rather than exact sentences.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/certificate-error.test.ts",
"destination_path": "evals/coding-agent/graders/certificate-error.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/certificate-error.test.ts"
}
},
{
"id": "app-workflow-save-receipt-boundary",
"name": "Preserve workflow save receipts and expose actual operation failure",
"source": {
"kind": "git_regression",
"fix_commit": "1f8ecb0858b63e82dc324050338953e7e4b2eb17"
},
"base_ref": "52706da629809407bf151c09294b677b65a32a78",
"oracle_ref": "1f8ecb0858b63e82dc324050338953e7e4b2eb17",
"oracle_paths": [
"crates/screenpipe-core/assets/extensions/workflow-workspace.ts"
],
"tags": [
"regression",
"escaped-failure",
"workflow",
"retry-safety",
"context-integrity"
],
"prompt": "The workflow_workspace extension can report failed operations as successful tool results, omit remaining work after saves and lose medium-size context through the private transport. Execute the actual registered tool. Conflicts, missing capability, aborted operations and missing draft selectors must reject; missing drafts should report exact open ids and titles owned by the current task without substituting another draft or retrying a mutation. Preserve structured literal payloads and the current task. A successful save must retain its receipt and expose freshly read remaining work/current revisions in remaining (revision, catalogRevision, cycleStatus, canFinish, openDrafts); if that follow-up read fails, return the successful receipt with remaining.unavailable true and do not repeat the save. Keep context indexes bounded, explicit selectors complete and oversized context in a private readable snapshot with exact data and revision fields, including context between the private transport limit and the old larger inline threshold. Preserve compact index behavior and full draft/catalog reads. Use synthetic loopback recorder responses and temporary files; no real recorder, workflow publication, agent dispatch, model or native build. These are extension-boundary outcomes, not backend durability or full stop/resume behavior.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/workflow-receipt.fixture.ts",
"destination_path": "crates/screenpipe-core/assets/extensions/workflow-workspace.eval.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test crates/screenpipe-core/assets/extensions/workflow-workspace.eval.test.ts"
}
},
{
"id": "app-continued-pipe-recording-directory",
"name": "Keep continued Pipe chats with their active recording directory",
"source": {
"kind": "git_regression",
"fix_commit": "a6a611a159f6d8704acc6ad00d21a6b87db42a95"
},
"base_ref": "d300cfd9c355ab3568bffe802a5132bc40211615",
"oracle_ref": "a6a611a159f6d8704acc6ad00d21a6b87db42a95",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat/pi-project-dir.ts"
],
"tags": [
"regression",
"escaped-failure",
"workflow",
"compatibility"
],
"prompt": "Continued Pipe chat follow-ups can select the account settings directory instead of the recording directory, separating their conversation from scheduled runs. Resolve continued Pipe session working directories beneath the currently active recording root, retaining drive identity, spaces and the exact valid pipe name. Read a changed root for later calls. If the native root lookup throws, returns an error or an empty value, fall back to the default home/.screenpipe/pipes directory. Preserve generic home/.screenpipe/pi-chat routing for ordinary chats, per-execution sessions, malformed IDs and unsafe pipe names (dot segments, slash/backslash, NUL or surrounding whitespace); those generic paths must not query native account or recording directories. Use the actual exported directory resolver and existing session parser with synthetic native path/command ports. No particular helper extraction is required. No real chats, scheduler/agent dispatch, native build or filesystem operations; this grades directory selection, not native persistence or complete account-state relocation.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/pipe-directory.test.ts",
"destination_path": "evals/coding-agent/graders/pipe-directory.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/pipe-directory.test.ts"
}
},
{
"id": "app-device-settings-identity",
"name": "Preserve native device identity across frontend settings resets",
"source": {
"kind": "git_regression",
"fix_commit": "8a51bdac54a6b7ba8444404835fbcfd85161a1f0"
},
"base_ref": "1837c3daaeeecca1541b8b97d1205f6215c36f97",
"oracle_ref": "8a51bdac54a6b7ba8444404835fbcfd85161a1f0",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx"
],
"tags": [
"regression",
"escaped-failure",
"identity"
],
"prompt": "Frontend settings resets can replace the device identity owned by native startup. New frontend default settings must leave deviceId empty. Full resets must retain the currently persisted identity, including legacy IDs, stable IDs and the compatibility identity created by loading an empty persisted value, while resetting ordinary settings such as font size. Resetting deviceId individually must do nothing and must not persist a write or alter other settings. Individual resets of ordinary fields must still persist their default without changing identity. Repeated resets must read any later native-owned identity, not retain an old cached value. Exercise the actual settings provider and store actions, including disk-write completion and reload, through synthetic storage and native ports. Do not perform native builds or real filesystem/network operations. This grades frontend reset behavior only, not native identity generation or cross-database record collision prevention.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/device-settings-identity.test.js",
"destination_path": "evals/coding-agent/graders/device-settings-identity.test.js"
}
],
"command": "cd apps/screenpipe-app-tauri && \"${BUN_BIN:-bun}\" test ../../evals/coding-agent/graders/device-settings-identity.test.js"
}
},
{
"id": "app-migration-startup-retry",
"name": "Wait for storage readiness before offering explicit migration retry",
"source": {
"kind": "git_regression",
"fix_commit": "65b05ea2ad2cc5c82a92456482ced5da3c27ad10",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7029"
},
"base_ref": "69be03c40091bcb1974b12f5d124852f346f69cc",
"oracle_ref": "65b05ea2ad2cc5c82a92456482ced5da3c27ad10",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/storage-migration-prompt.tsx"
],
"tags": [
"regression",
"escaped-failure",
"migration",
"recovery",
"consent"
],
"prompt": "The visible desktop migration prompt can offer a retry while startup still owns the database. Hide the migration dialog and perform no start, cancel or deletion when native status is busy or has a blocking reason, even if a failed in-place migration is pending. Once that status is ready, allow exactly an explicit retry for the selected root. Busy activity also stays quiet. Preserve ordinary legacy-storage offers without automatic start or source deletion; Do later must survive a recreated webview in the same app process, while a new process may offer the choice again without automatically converting. Keep the prompt closed for empty/ineligible, already-converted, vault-blocked and completed-but-reopening storage. Interrupted in-place conversion permits explicit retry without offering the original-database fallback; where native status permits cancellation, that recovery action must call cancel for the selected root and never delete the original. Failed starts must display their error and permit another explicit retry. Repeated app launches after failure/interruption must never automatically retry. During an unresolved explicit start, disable repeated start and deferral until the request resolves. Exercise the actual React prompt and native command boundary with synthetic status and storage ports. No particular helper, wording or internal implementation is required. This case does not establish native recovery, durability, deployment exposure or managed migration authorization; do not broaden automatic conversion, change native code, build native targets, publish or contact live services.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 120,
"fixtures": [
{
"local_path": "graders/migration-startup-retry.fixture.tsx",
"destination_path": "apps/screenpipe-app-tauri/components/storage-migration-prompt.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts components/storage-migration-prompt.test.tsx --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "mcp-maintained-workflow-retrieval",
"name": "Retrieve maintained workflows without weakening ID request boundaries",
"source": {
"kind": "git_regression",
"fix_commit": "9c38b194f5e324b1fcf78fbc3ece1adf4af29274"
},
"base_ref": "ea0a6852e1bbb89b87535a1618a44d0bc752640b",
"oracle_ref": "9c38b194f5e324b1fcf78fbc3ece1adf4af29274",
"oracle_paths": [
"packages/screenpipe-mcp/src/workflow-tools.ts"
],
"tags": [
"regression",
"escaped-failure",
"mcp",
"workflow",
"input-validation"
],
"prompt": "The MCP workflow tools can list maintained workflows but reject their returned IDs when retrieving details. Make maintained wf-prefixed lowercase hexadecimal UUID IDs retrievable while preserving legacy wf-prefixed 64-character lowercase hexadecimal IDs. Preserve the exact workflow identity and complete returned JSON, including unknown nested fields and evidence. The automation-detail option defaults to true and respects explicit true or false; nonboolean options must fail before a backend call. Reject malformed IDs, wrong lengths, nonhex content, surrounding whitespace and path/query/fragment or encoded-path injection before requesting anything. Preserve list search encoding, empty catalogs and safe integer pagination (limit 1 through 100, offset zero or greater), rejecting invalid inputs and unknown tools without requests. Each valid invocation makes one backend read. Non-success HTTP responses, transport failures and malformed JSON must surface failure instead of an empty success; a later explicit retry can recover. Exercise the actual workflow tool function using a synthetic API port. Do not access real recordings, credentials or providers. Native catalog persistence, stdio transport, authorization and stalled-response timeouts are outside this task.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/mcp-workflow-ids.test.js",
"destination_path": "evals/coding-agent/graders/mcp-workflow-ids.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/mcp-workflow-ids.test.js"
}
},
{
"id": "app-localization-recovery-preservation",
"name": "Preserve accepted localization during cold recovery",
"source": {
"kind": "git_regression",
"fix_commit": "474f641f520b9057be862e3d2ed1cb9f03d1978b",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7172"
},
"base_ref": "90ebd317d61df0a81a3ac92ee29a1c25dffaa798",
"oracle_ref": "474f641f520b9057be862e3d2ed1cb9f03d1978b",
"oracle_paths": [
"apps/screenpipe-app-tauri/scripts/i18n/service.mjs"
],
"tags": [
"regression",
"escaped-failure",
"localization",
"recovery",
"data-preservation"
],
"prompt": "Cold localization recovery can report success while writing an incompatible legacy translation over a valid translation already recovered from another provider file. Fix the actual translation service so persisted locale catalogs preserve accepted rich-text structure and ICU placeholders when a later incompatible file is downloaded. Accept a later valid correction, recover regardless of whether the incompatible file arrives first or last, combine disjoint messages, and preserve unmatched rejected entries for diagnostics. Apply these rules independently to frontend and native catalogs and each locale. Fully recovered messages must not trigger translation, branch creation or uploads. Refuse downloaded results with mismatched branch, file, version, locale or format. Preserve the warm-cache path: valid cached frontend/native messages make no provider calls, caller-owned cache stays unchanged, and stale loose output is cleared. Use synthetic provider responses and disposable temporary files; do not contact a translation provider, translate real content, build or publish an app. No particular merge expression or internal helper is required. Native packaging, live provider availability and actual model quality are outside this task.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/localization-recovery.fixture.js",
"destination_path": "apps/screenpipe-app-tauri/scripts/i18n/eval-localization-recovery.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test apps/screenpipe-app-tauri/scripts/i18n/eval-localization-recovery.test.js"
}
},
{
"id": "app-mcp-home-config-scope",
"name": "Connect home-level MCP configuration without forbidden directory creation",
"source": {
"kind": "git_regression",
"fix_commit": "c06dc27999095443e8815c890d9c7989d53dc5fb"
},
"base_ref": "073b9d31ad8759db0fd8c33c5ac66972fe4431aa",
"oracle_ref": "c06dc27999095443e8815c890d9c7989d53dc5fb",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/ai-tools-mcp.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/ai-tools-mcp.ts"
],
"tags": [
"regression",
"escaped-failure",
"mcp",
"configuration",
"data-integrity"
],
"prompt": "Settings cannot connect Claude Code when filesystem policy permits its configuration file but refuses operations on the existing home directory itself. Restore explicit connect and disconnect for a home-level config, including trailing separators on POSIX and Windows and a native-resolved Windows verbatim path. Existing unrelated settings and servers must survive; preserve the previous file in a backup and leave no temporary files after successful updates. First-time configuration creation must work. Preserve nearby Cursor JSON and Codex TOML behavior: update the resolved target instead of replacing its link, refuse resolution failures and invalid or unreadable files without overwriting them, preserve the local API credential and tool identity, avoid duplicate differently-cased Screenpipe entries, keep at most the two newest backups, and keep disconnect idempotent. Keep the public connection interfaces usable. This task grades Settings-side file effects using synthetic platform ports; native filesystem permission acceptance is outside its scope. Do not access real credentials, configs or agents.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/mcp-home-scope.fixture.ts",
"destination_path": "apps/screenpipe-app-tauri/lib/__tests__/eval-mcp-home-scope.test.ts"
},
{
"local_path": "graders/mcp-home-scope.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-mcp-home-scope.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run lib/__tests__/eval-mcp-home-scope.test.ts --config eval-mcp-home-scope.config.mjs"
}
},
{
"id": "app-external-chat-scan-boundary",
"name": "Bound recent external chat discovery without losing eligible sources",
"source": {
"kind": "git_regression",
"fix_commit": "c03f24bba7f180adf95a84c4855da13886ef0375"
},
"base_ref": "d3517d8396a56488c40684b442e52901c6de2199",
"oracle_ref": "c03f24bba7f180adf95a84c4855da13886ef0375",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat/external-chat-import.ts"
],
"gate": "advisory",
"tags": [
"regression",
"chat",
"local-history",
"bounded-io",
"privacy"
],
"prompt": "Opening local chat import can traverse a large archive before showing recent conversations. Repair scanExternalChatHistory while preserving its discovery API. Only include sessions modified within the past seven days, including the exact cutoff, and report lookbackDays=7. Codex stores direct JSONL files in sessions/YYYY/MM/DD: avoid inspecting older date partitions and nested sessions. On a recent partition ordered by session filename, return the latest 100 eligible files and stop inspecting file metadata once 100 eligible files are found. Do not let files over 32 MiB consume that cap; exactly 32 MiB remains eligible. Report availableCount, omittedByLimit and skippedTooLarge consistently: 150 eligible recent files yields 100 candidates and 50 omitted; three newest oversized files followed by 105 eligible files yields 100 candidates, three oversized and five omitted. Preserve Claude direct project JSONL discovery, case-insensitive extensions and its recency/size filters, excluding subagents and non-JSONL files. Missing roots, missing dates, unreadable individual files or Claude projects must not hide other readable sources. Preserve candidate identity/path/metadata and total counts, retain independent sources and reflect newly added files on later scans. Discovery must only inspect directory and file metadata; do not read or parse transcript contents, write source or conversation data, invalidate caches, emit events or use the network. Use synthetic histories only. No particular private helper, sorting implementation or UI wording is required.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/app-external-chat-scan.test.js",
"destination_path": "evals/coding-agent/graders/app-external-chat-scan.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/app-external-chat-scan.test.js"
}
},
{
"id": "ai-gateway-tts-wav-response",
"name": "Return requested WAV audio without mislabelling provider bytes",
"source": {
"kind": "git_regression",
"fix_commit": "ffff7de62e09bcd7058e219dbc13620e38c7189a"
},
"base_ref": "b68c120070351b90afb9e1df1e5f2c866f7fe396",
"oracle_ref": "ffff7de62e09bcd7058e219dbc13620e38c7189a",
"oracle_paths": [
"packages/ai-gateway/src/utils/voice-utils.ts"
],
"gate": "advisory",
"tags": [
"regression",
"gateway",
"tts",
"binary-response",
"provider-failure"
],
"prompt": "The text-to-speech handler labels provider bytes audio/wav even when the provider sends another format. Repair the existing handleTextToSpeech/textToSpeech path. The HTTP handler must request linear16 audio in a WAV container from the existing speech endpoint, using the default voice or the explicitly supported voice, the configured provider authorization and the literal text. A successful provider WAV response must reach the caller byte-for-byte with status200 and audio/wav. Empty or shorter-than12-byte results, MP3 bytes, and buffers without both RIFF and WAVE signatures must produce a JSON error with status500, never successful audio. This is a format/signature boundary, not a full WAV decoder requirement. Provider HTTP rejection, transport rejection or body-read failure must similarly fail once without automatic retries, and a later explicit request must recover. Preserve rejection of missing/nonstring/empty text and unsupported voices with400 before provider access; whitespace-only text remains a500 error without provider access. Preserve the utility public API for explicitly requested non-WAV encoding such as mp3: forward that encoding, do not force a WAV container or WAV validation, and preserve successful bytes. Use only synthetic provider responses and text. No specific private helper name or implementation structure is required. This evaluates the actual TTS handler and utility, not gateway authentication/rate limits, transcription, real playback, provider connectivity or full container integrity.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/tts-wav.test.js",
"destination_path": "evals/coding-agent/graders/tts-wav.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/tts-wav.test.js"
}
},
{
"id": "ai-gateway-paid-tts-dispatch",
"name": "Enforce paid access before text-to-speech provider work",
"source": {
"kind": "git_regression",
"fix_commit": "6301ad06342d4bad420598db86db9e44423b05b6"
},
"base_ref": "8d2fd0135dbfe71e64aa9db2c4300601648c2077",
"oracle_ref": "6301ad06342d4bad420598db86db9e44423b05b6",
"oracle_paths": [
"packages/ai-gateway/src/index.ts"
],
"gate": "advisory",
"tags": [
"regression",
"billing",
"access",
"voice",
"provider-effects"
],
"prompt": "The text-to-speech gateway route can send speech to the provider without a paid account. Repair the actual POST /v1/text-to-speech boundary. For a valid text body, an anonymous caller must receive 401, a verified Free caller 403, and a verified caller with missing plan truth 503, all as JSON errors before any speech-provider request. Preserve paid Basic and Business callers: one provider request carrying the original text must return the exact provider audio bytes with audio/wav. Paid accounts denied good standing must still receive 403 before provider work; paid requests with invalid empty text must still receive 400 with no provider work. All refusals must be JSON errors. Use synthetic identity-verifier, entitlement API, rate-limit and audio-provider ports only. Do not assume a helper name or particular implementation. This case does not test authentication cryptography, live payment or provider delivery, transcription, native playback, database concurrency, agent isolation or model capability.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/paid-tts-dispatch.test.js",
"destination_path": "evals/coding-agent/graders/paid-tts-dispatch.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/paid-tts-dispatch.test.js"
}
},
{
"id": "app-onboarding-sdk-cold-start-assignment",
"name": "Resolve cold SDK assignment before timeout without accepting stale identities",
"source": {
"kind": "git_regression",
"fix_commit": "dd33e9cd83fb0a54e277dd802e9443bd807ce73a",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7200"
},
"base_ref": "41e66a4599a76bc5d04de45ef96ab8e75bca58c5",
"oracle_ref": "dd33e9cd83fb0a54e277dd802e9443bd807ce73a",
"oracle_paths": [
"apps/screenpipe-app-tauri/app/onboarding/page.tsx"
],
"tags": [
"regression",
"escaped-failure",
"onboarding",
"auth",
"trial",
"sdk"
],
"prompt": "Authenticated fresh-install onboarding can stay on Preparing your setup and fall back to control even though experiment responses arrive in time. With the repository-locked PostHog SDK, remote flag requests taking two seconds each must resolve the eligible authenticated route within 4.5 seconds, inside the existing five-second deadline. Persist summary_first for that fresh treatment; explicit control, false and absent flags must resolve to control without assignment-failure telemetry. Do not pin cached, local-override or earlier in-flight account values before fresh evidence for the current authenticated identity; account changes must not expose the previous account treatment. Preserve pinned control/treatment routes without new assignment requests, the five-second control fallback when responses are late, and immunity to late treatment after fallback. Preserve analytics opt-out without identifying the user or enrolling their machine assignment. After unmount, callbacks must not persist assignment or report failure. Exercise the real React page and locked SDK with synthetic transport and native/provider ports. No particular callback count, request count for unpinned assignment, internal variable or diagnostic format is required. Do not contact live services or alter billing, recording or release behavior.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 120,
"fixtures": [
{
"source_ref": "dd33e9cd83fb0a54e277dd802e9443bd807ce73a",
"source_path": "apps/screenpipe-app-tauri/app/onboarding/page.test.tsx"
},
{
"local_path": "graders/onboarding-sdk-outcomes.mjs",
"destination_path": ".eval-onboarding-sdk.mjs"
}
],
"command": "node .eval-onboarding-sdk.mjs && cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts app/onboarding/page.test.tsx -t 'eval SDK assignment outcomes' --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "app-abandoned-read-wal-cleanup",
"name": "Release abandoned SQLite read snapshots without losing writes",
"source": {
"kind": "git_regression",
"fix_commit": "2978ef82da358544450998f81dd5a687651cffee",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7194"
},
"base_ref": "b15733e7b1a252f71a037a968d4f63af578aac47",
"oracle_ref": "2978ef82da358544450998f81dd5a687651cffee",
"oracle_paths": [
"crates/screenpipe-db/src/cancellable_query.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-db/src/cancellable_query.rs",
"crates/screenpipe-db/src/storage/snapshot.rs"
],
"tags": [
"regression",
"escaped-failure",
"sqlite",
"cancellation",
"durable-recording"
],
"prompt": "A cancelled read request can leave a raw SQLite read transaction on a pooled connection. Later checkpoints remain pinned even after that request is gone. Fix cancellable read cleanup so dropping the guard, explicitly releasing it, or aborting its owning task releases the abandoned snapshot before returning pool capacity. A single-slot read pool must become usable within three seconds; a passive checkpoint must drain once the abandoned reader is gone. Writes made during the read and after cleanup must remain visible, survive database reopen and retain database integrity. Preserve an active reader's original snapshot while concurrent writes continue, ordinary non-transaction release, interruption of long SQLite work within three seconds when given a 20 ms deadline, and a clean progress handler for the next borrower. Keep the public cancellable-read interfaces usable. This task is limited to read-connection cleanup; it does not change scheduler restart policy, native recovery, user pause or migration authority.",
"agent_timeout_seconds": 1800,
"grader_dependency_links": [
{
"source_path": "target",
"destination_path": "target"
}
],
"grader": {
"timeout_seconds": 600,
"fixtures": [
{
"local_path": "graders/wal-read-cleanup.rs",
"destination_path": "crates/screenpipe-db/tests/eval_wal_read_cleanup.rs"
}
],
"command": "cargo test --offline --locked -p screenpipe-db --test eval_wal_read_cleanup -- --test-threads=1"
}
},
{
"id": "app-daily-summary-terminal-error",
"name": "Preserve terminal provider failures in daily summaries",
"source": {
"kind": "git_regression",
"fix_commit": "ee91ade94484a349c98fd7ad9613f4a6018b50bf"
},
"base_ref": "e58cc923243682bb8fb9d87fbf5f33d41f122a2f",
"oracle_ref": "ee91ade94484a349c98fd7ad9613f4a6018b50bf",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/daily-summary-pi.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/daily-summary-pi.ts",
"apps/screenpipe-app-tauri/lib/events/error-text.ts"
],
"tags": [
"regression",
"escaped-failure",
"provider-errors",
"quota",
"retry"
],
"prompt": "Daily-summary generation can treat a terminal provider failure as a usable summary or lose the original allowance/rate-limit detail. Repair the actual runDailySummaryWithPi boundary. An assistant error in message_end without an explicit willRetry=true hint, or an assistant error in agent_end, must reject with the original provider detail, including errorMessage and legacy error fields. When detail is absent, return a non-empty failure explanation. A terminal error must beat partial or earlier assistant prose and must not trigger an empty-result reprompt. Preserve successful trimmed summary text, ignore error-like user/tool messages, and allow explicit retry hints to reach a later valid completion. Preserve one bounded empty-completion reprompt: it may recover valid text, but a second empty completion must reject as empty. Ordinary error events must retain provider detail. Unregister the event handler and stop the session when a started request completes or fails. Use synthetic native/event ports; do not call live providers or read personal activity. This task covers the summary orchestration boundary, not native recording, real provider delivery, the activity UI, model behavior or general runtime-start retry policy.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/daily-summary-errors.test.js",
"destination_path": "evals/coding-agent/graders/daily-summary-errors.test.js"
}
],
"command": "bun --no-env-file test evals/coding-agent/graders/daily-summary-errors.test.js"
}
},
{
"id": "app-pipe-run-registered-outputs",
"name": "Expose registered outputs for the active pipe run",
"source": {
"kind": "git_regression",
"fix_commit": "d3569717c055679ca232b822491cd72ac58ca06c"
},
"base_ref": "67f8cc7ceb74e5772ff44970419a664e7827fe74",
"oracle_ref": "d3569717c055679ca232b822491cd72ac58ca06c",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-chat-inspector.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-chat-inspector.ts",
"apps/screenpipe-app-tauri/lib/hooks/use-unified-artifacts.ts",
"apps/screenpipe-app-tauri/lib/source-citations.ts"
],
"tags": [
"regression",
"escaped-failure",
"pipe-outputs",
"execution-scope",
"recovery"
],
"prompt": "The public useChatInspector hook only exposes explicit artifact tool results, so a pipe run whose output was registered separately appears to have no output. Restore outputs for the supplied active pipe-run source even when the transcript has no artifact tool call. Query the artifact API with that source and exclude rows belonging to other runs. Preserve explicit tool outputs and avoid duplicate registered outputs when their original path already identifies an explicit output. Ordinary chat without a pipe-run source must preserve explicit outputs without reading the artifact API. Removing the pipe context must clear its registered outputs. An initial artifact API refusal must leave registered outputs empty and a later transcript update must recover the registered output after the API recovers. Preserve the public hook interface; evaluate observable outputs and requests rather than private helper names or UI wording. Use synthetic local API responses only. This task covers the actual hook and internal artifact/citation helpers, not full chat UI wiring, native registration, database persistence, or live services.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/pipe-run-outputs.fixture.ts",
"destination_path": "apps/screenpipe-app-tauri/eval-pipe-run-outputs.test.ts"
},
{
"local_path": "graders/pipe-run-outputs.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-pipe-run-outputs.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-pipe-run-outputs.config.mjs"
}
},
{
"id": "app-acp-billing-saved-recovery",
"name": "Preserve own-account ACP billing recovery without automatic retries",
"source": {
"kind": "git_regression",
"fix_commit": "02e8048c8138e8a2bb4fccb0a92ab945672d5b31"
},
"base_ref": "6661cc543625435bd8992efba6aa3277cb519883",
"oracle_ref": "02e8048c8138e8a2bb4fccb0a92ab945672d5b31",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat/provider-errors.ts",
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-pi-foreground-events.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/chat/provider-errors.ts",
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-pi-foreground-events.ts",
"apps/screenpipe-app-tauri/components/chat/standalone/message-content.tsx"
],
"tags": [
"regression",
"escaped-failure",
"billing-recovery",
"acp",
"retry",
"persistence"
],
"prompt": "Own-account ACP billing failures currently leave an opaque or misleading failed turn. When the selected Claude Code account has insufficient credits, Cursor reaches a spending limit, or Codex reaches a usage limit, retain actionable guidance naming that selected provider and the relevant account recovery (add credits, review or adjust spending limits, or review usage reset/plan options), plus the option to switch AI presets. This must survive message error events, message end, terminal agent end and saving/reloading the message as an accessible error alert. A billing failure prefixed with HTTP 429 is still a billing failure: do not offer an immediate retry button or automatically resend the prompt. Preserve an ordinary transient HTTP 429 response retry with the original prompt, and prevent a delayed retry from sending into a changed session. Preserve context-overflow recovery after earlier tool progress, retaining the progress blocks and a manual retry prompt, whether or not earlier error events were delivered. A subsequent successful answer must replace the error and clear its retry action. Keep public hook/message interfaces compatible. Exact prose, private helper names and logging format are not prescribed. Use synthetic local native, event, storage and network ports only. This task covers the foreground event hook, internal provider presentation and saved-message rendering, not native ACP execution, live provider billing or financial entitlement policy.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/acp-billing-recovery.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-acp-billing-semantic.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts eval-acp-billing-semantic.test.tsx"
}
},
{
"id": "ai-gateway-voice-transcription-budget",
"name": "Enforce transcription budgets before voice provider work",
"source": {
"kind": "git_regression",
"fix_commit": "dc35366d18ce0732e935cbfbdaa6093f6534406e"
},
"base_ref": "9064c3134d61bc3848ac3d84da24392f14d20ea5",
"oracle_ref": "dc35366d18ce0732e935cbfbdaa6093f6534406e",
"oracle_paths": [
"packages/ai-gateway/src/index.ts",
"packages/ai-gateway/src/handlers/realtime-transcription.ts",
"packages/ai-gateway/src/services/cost-tracker.ts",
"packages/ai-gateway/src/services/hosted-ai-cost-controls.ts",
"packages/ai-gateway/src/services/transcription-budget.ts",
"packages/ai-gateway/src/types.ts"
],
"gate": "advisory",
"tags": [
"regression",
"billing",
"voice",
"provider-effects",
"budget"
],
"prompt": "The hosted POST /v1/voice/transcribe route can contact its transcription provider despite exhausted spending limits. Repair the actual request boundary for an authenticated Business account in good standing. At the configured account daily, global hourly or global daily transcription cap, return 429 before any provider call. If the cost ledger cannot be read, return 503 before provider work. Use the existing cost ledger and configurable transcription limits; synthetic test caps are 103, 501 and 502 respectively, not production policy. Preserve an in-budget audio/wav request, including values immediately below every cap: send the original audio bytes to the provider exactly once and return 200 with its transcript in the transcription field. Preserve a 400 response with no provider call for application/json audio requests. Identity verification, entitlement service, rate limiter, D1 (SQLite-backed reads with inert telemetry writes) and transcription provider are synthetic ports; all other network access is refused. Any implementation achieving these outcomes is acceptable. This case does not establish atomic reservations or concurrent spending safety, duration-based billing, anonymous identity, realtime metering, live provider delivery, native behavior, agent isolation or model capability.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/voice-transcription-budget.test.js",
"destination_path": "evals/coding-agent/graders/voice-transcription-budget.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/voice-transcription-budget.test.js"
}
},
{
"id": "app-connection-probe-response",
"name": "Return valid connection-probe replies from streaming-default gateways",
"source": {
"kind": "git_regression",
"fix_commit": "9a4950abb4bbcddec07942609297b8d57de5e437"
},
"base_ref": "22d2053908e55e86d7b84365e3ba508896e63060",
"oracle_ref": "9a4950abb4bbcddec07942609297b8d57de5e437",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/utils/ai-preset-connection.ts",
"apps/screenpipe-app-tauri/lib/utils/chat-test-body.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"connection-test",
"provider-compatibility",
"recovery"
],
"prompt": "The public testAiPresetConnection service reports a connection failure for working OpenAI-compatible gateways that default to server-sent events. Restore the actual provider reply for custom, OpenAI, native Ollama and Anthropic connections, including gateways that require max_completion_tokens instead of max_tokens. Preserve configured model, endpoint and provider credentials, a nonnegative finite latency, ordinary JSON replies, existing empty-text message compatibility and the 100-character displayed reply limit. Missing messages, provider authorization errors, array-shaped provider errors and transport failures must remain failures with their meaningful details; caller cancellation must reach the provider request. Any implementation that achieves these outcomes is acceptable, including successful SSE handling. Preserve the public service interface. Use synthetic provider ports only. This covers the connection service and its internal request/error helpers, not settings UI persistence, native networking or live provider acceptance.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/connection-probe.fixture.ts",
"destination_path": "apps/screenpipe-app-tauri/eval-connection-probe.test.ts"
},
{
"local_path": "graders/connection-probe.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-connection-probe.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-connection-probe.config.mjs"
},
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/utils/ai-preset-connection.ts",
"apps/screenpipe-app-tauri/lib/utils/chat-test-body.ts",
"apps/screenpipe-app-tauri/lib/utils/ai-endpoint-url.ts",
"apps/screenpipe-app-tauri/lib/utils/validation.ts"
]
},
{
"id": "app-business-capacity-plan-picker",
"name": "Expose capacity plans and route each selection to its billing target",
"source": {
"kind": "git_regression",
"fix_commit": "9d7866f0eec57826f3af7660e4c2a224e3f53dbc"
},
"base_ref": "e86a6d71a79ba302f9a8e6a765e1d4d1342404c3",
"oracle_ref": "9d7866f0eec57826f3af7660e4c2a224e3f53dbc",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/settings/account-plan-options.tsx",
"apps/screenpipe-app-tauri/components/settings/account-section.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/settings/account-plan-options.tsx",
"apps/screenpipe-app-tauri/components/settings/account-section.tsx"
],
"tags": [
"regression",
"escaped-failure",
"billing",
"plan-picker"
],
"prompt": "Account settings hides Business Max and Ultra behind an unset feature flag. Make all five plan choices visible without enabling a feature flag: Free $0, Basic $25, Business $50, Business Max $100 and Business Ultra $200 per month. Describe Max and Ultra as 2x and 4x daily query/request capacity respectively, retaining the same 400 credits and frontier models, not extra credit allowances. A signed-in Business owner selecting Max must open the screenpipe.com/account/billing page with target_plan=pro_max and interval=month; a Max owner selecting Ultra must use target_plan=pro_ultra and interval=month. These capacity changes must not start a new in-app checkout. Mark each Max or Ultra owner on the matching priced plan as current, without a purchase button for that current plan or a current marker on ordinary Business. Recommend Max as the next step for Business, Ultra for Max, and no next step above Ultra or for team/enterprise accounts. Preserve signed-out stale-account gating: an account without a token, even with a stale cloud subscription flag, must show sign-in guidance and no active subscription. Signed-in subscribers must retain their active account display and working web-account link. Preserve the public AccountSection interface. Exact layout, test ids, CSS, internal helper names, analytics names and URL query ordering are not prescribed. Use synthetic native/provider ports only. This covers account settings presentation and outgoing billing navigation, not live financial operations, backend proration, native behavior or model performance.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/capacity-plan-picker.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-capacity-plan-picker.test.tsx"
},
{
"local_path": "graders/capacity-plan-picker.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-capacity-plan-picker.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-capacity-plan-picker.config.mjs"
}
},
{
"id": "app-pipe-write-permission-hook",
"name": "Enforce pipe write boundaries at the registered tool hook",
"source": {
"kind": "git_regression",
"fix_commit": "c630f18d186b62ab826f3956e7963a42ce9f9909"
},
"base_ref": "47b59ce096d37843b4b04f7401e2ec246ca2e5e3",
"oracle_ref": "c630f18d186b62ab826f3956e7963a42ce9f9909",
"oracle_paths": [
"crates/screenpipe-core/assets/extensions/screenpipe-permissions.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"permissions",
"pipe-tools"
],
"prompt": "A configured pipe can write outside its permitted directory through the extension tool-call hook. Enforce its configured write boundary for built-in write and edit tools, including both tool/name event fields and file_path/path targets, traversal and missing targets. For shell calls, reject an outside target after a permitted first rm or touch operand, runtime-expanded redirect destinations, opaque Python scripts, outside writes inside a nested shell and outside writes after a permitted compound-command segment. Preserve relative file writes, absolute in-directory edits, reads outside the directory, in-directory shell redirects, multiple permitted write targets, literal dollar characters in single-quoted in-directory filenames, read-only shell commands and interpreter version queries. Any implementation achieving these outcomes is acceptable; no helper name, parser architecture or error wording is required. Exercise the actual registered extension hook using synthetic configuration and event inputs. Do not execute the supplied commands or access real user files. This case grades hook decisions for these command forms, not complete shell analysis, native filesystem enforcement, symlink handling or agent isolation.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/pipe-write-hook.fixture.ts",
"destination_path": "eval-pipe-write-hook.test.ts"
}
],
"command": "bun test eval-pipe-write-hook.test.ts"
},
"trigger_paths": [
"crates/screenpipe-core/assets/extensions/screenpipe-permissions.ts"
]
},
{
"id": "app-card-ask-login-hydration",
"name": "Preserve an eligible login prompt after account hydration",
"source": {
"kind": "git_regression",
"fix_commit": "1eb83b93c716ca27d7c91bbd74f886ff8e429872"
},
"base_ref": "36d84b89d54597c1a672eaa00bbfc653f1873ffc",
"oracle_ref": "1eb83b93c716ca27d7c91bbd74f886ff8e429872",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/card-ask-provider.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/card-ask-provider.tsx",
"apps/screenpipe-app-tauri/lib/hooks/use-card-ask.tsx"
],
"tags": [
"regression",
"escaped-failure",
"billing",
"account-hydration"
],
"prompt": "The Home login card prompt can disappear for an entire session when its timing variant resolves before account settings or when a user signs in after Home mounts. For the configured at_login variant, show the existing dialog once an eligible signed-in account without a payment method is available, whether loaded initially or later. Keep it hidden until settings and account identity are available. Preserve once-per-install suppression across unmount and remount with persisted browser storage, suppression for accounts with a payment method, silence in control, silence for a timing variant that does not own login, and silence while the variant is unresolved. Preserve the public CardAskProvider component interface. No particular effect dependencies, private names, data-test attributes, storage representation or modal wording are required. Exercise real provider/controller/dialog behavior with synthetic settings, feature-flag, native-platform and telemetry ports. This is the historical configured login flow; it does not authorize enabling any current rollout or changing payment, recording or native behavior.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/card-login-hydration.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/__tests__/eval-card-login-hydration.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage node node_modules/vitest/vitest.mjs run --config vitest.config.ts components/__tests__/eval-card-login-hydration.test.tsx --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "app-complete-free-entitlement-evidence",
"name": "Accept complete legacy free evidence while rejecting partial account truth",
"source": {
"kind": "git_regression",
"fix_commit": "e3adc22d1fbd54a0637a3f2fae7ccb379f72ebd0"
},
"base_ref": "4cc85bdf22649300f41b2e67d492de649ac124cb",
"oracle_ref": "e3adc22d1fbd54a0637a3f2fae7ccb379f72ebd0",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/app-entitlement.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/app-entitlement.ts"
],
"tags": [
"entitlement",
"identity",
"free-policy",
"normalization"
],
"prompt": "An authenticated older API response containing a stable account identity and cloud_subscribed=false, with app_entitled, subscription_plan and entitlement omitted, is incorrectly classified as unknown. Normalize that complete legacy denial into verified-free policy, preserving account identity and the supplied session token, without granting paid app access. Partial newer evidence (plan-only or no explicit app-feature boolean) stays unknown; explicit null newer fields must not be treated as legacy omissions. Complete newer free evidence requires explicit cloud_subscribed=false, matching none plan labels, source none, boolean active and app feature evidence, a non-future checked_at, no cloud feature and no future grace. Either true or false app-feature/active values may describe a complete free account; old verified-free evidence remains free offline beyond 72 hours. Missing/null/true cloud denial and subscription,unknown,manual,enterprise,lifetime or dev sources cannot establish free policy. Empty, whitespace-only and null tokens cannot authenticate cached free policy, though cached free classification itself remains valid. Require a nonblank id or clerk_id; preserve clerk identity fallback. Preserve explicit app denial overriding stale paid labels and valid fresh standard,pro and lifetime paid classification. Keep the public normalization/classification interfaces; no private helper name or implementation shape is required. This task covers pure classification, not native recording, UI gates, provider verification or production entitlement safety.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/free-evidence.test.ts",
"destination_path": "evals/coding-agent/graders/free-evidence.test.ts"
}
],
"command": "bun test ./evals/coding-agent/graders/free-evidence.test.ts"
}
},
{
"id": "ai-gateway-account-standing-provenance",
"name": "Keep temporary sign-in locks and stale account IDs distinct from revocation",
"source": {
"kind": "git_regression",
"fix_commit": "e8732d64bb543cd17b7cd798e74027fcf168573c"
},
"base_ref": "a9ddc64913d5a0cee60de9a582cb5293298f6ec7",
"oracle_ref": "e8732d64bb543cd17b7cd798e74027fcf168573c",
"oracle_paths": [
"packages/ai-gateway/src/utils/auth.ts",
"packages/ai-gateway/src/utils/clerk-standing.ts",
"packages/ai-gateway/src/utils/rate-limiter.ts",
"packages/ai-gateway/src/types.ts"
],
"gate": "advisory",
"tags": [
"regression",
"authentication",
"account-standing",
"gateway"
],
"prompt": "The gateway can refuse an otherwise entitled account because its identity provider temporarily locks new sign-ins, and can treat a stale database account ID as proof of permanent deletion. Repair actual HTTP admission for paid accounts. On GET /anthropic/v1/models, both good and temporarily locked accounts must return 200 with the provider's nonempty model list and exactly one model-list provider call. On that route and POST /v1/chat/completions, a banned account must receive 403 with account_not_in_good_standing and reason banned. A deleted subject verified from this request's signed credential must receive the same 403 error with reason deleted. A provider 404 for a subject obtained only from the website account row must instead return retryable 503 with account_status_unavailable. Preserve the established JSON error envelope. Refused requests must not reach the model provider. Exercise both request-verified and website-row identity paths through the real router, authentication and durable-object admission logic. Identity verification, entitlement/standing responses, provider model listing and durable storage are synthetic ports; other network traffic is refused. Any implementation achieving these outcomes is acceptable, without requiring particular private field names. This case does not cover cryptographic verification, cache migration, live provider behavior, native transcription retries, concurrent admission, model quality or agent isolation.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/account-standing.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-standing-http.test.ts"
}
],
"command": "cd packages/ai-gateway && \"${BUN_BIN:-bun}\" test src/test/eval-standing-http.test.ts"
}
},
{
"id": "app-attachment-owner-handoff",
"name": "Keep asynchronous file attachments with their initiating chat",
"source": {
"kind": "git_regression",
"fix_commit": "f1bcd5bbbb9d838e471cb73e3ec0e88316ef89e7"
},
"base_ref": "1f35d982415a6ebb5509dcbd8243772cd8a128b9",
"oracle_ref": "f1bcd5bbbb9d838e471cb73e3ec0e88316ef89e7",
"tags": [
"regression",
"escaped-failure",
"chat",
"attachments",
"ownership"
],
"prompt": "File attachments can disappear or reach the wrong chat when file reads finish after a pane closes or the foreground chat changes. Fix the attachment hook so a secondary pane file completion remains in its initiating session draft after unmount. A pending foreground read must complete in its original chat, clear its pending entry, and leave the newly selected chat and its local attachment list untouched. Window-level drops must reach only the pane under their pointer, using physical coordinates and device pixel ratio; keep every file in a multiple-file foreground drop. Cancelled pickers must perform no file read or draft creation. Failed reads must clear pending work without adding an attachment or disturbing another draft. Late completion must not recreate a deleted or hidden chat draft. Keep ordinary successful attachment handling working. Checks exercise the actual hook and in-memory chat store with synthetic native dialog/file/document-extraction and locale ports. Native webview delivery, disk persistence, and rendered composer integration are outside this task. Add or update focused tests as needed.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/attachment-owner.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/chat/standalone/hooks/eval-attachment-owner.test.tsx"
},
{
"local_path": "graders/attachment-owner.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-attachment-owner.config.mjs"
},
{
"local_path": "graders/attachment-owner-locale.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-attachment-locale.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x --no-install vitest run --config eval-attachment-owner.config.mjs"
},
"oracle_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-chat-attachments.ts",
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-session-draft-field.ts"
],
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
]
},
{
"id": "ai-gateway-background-request-preservation",
"name": "Preserve the complete request when recovering an exhausted background allowance",
"source": {
"kind": "git_regression",
"fix_commit": "1c0a271403a4128a2238c355e5820b7f18c98eac"
},
"base_ref": "5eb2bd80952a7302f20f46f561abdbb136950f86",
"oracle_ref": "1c0a271403a4128a2238c355e5820b7f18c98eac",
"oracle_paths": [
"packages/ai-gateway/src/handlers/chat.ts",
"packages/ai-gateway/src/services/background-limit-fallback.ts",
"packages/ai-gateway/src/index.ts"
],
"gate": "advisory",
"tags": [
"regression",
"background",
"allowance",
"request-integrity"
],
"prompt": "An already-authorized background request exhausts its hosted allowance. Its recovery attempt can silently lose developer instructions, older context, tool descriptions and schema metadata, or most of its output budget. Repair the existing chat completion handler so an enabled, configured text-only recovery makes one alternate provider attempt with the complete message sequence, roles, tool-call associations, tool schemas, requested output budget, response format and other request options preserved. Do not mutate the caller request. Preserve the real provider response and streamed tool-call result. A model substitution is allowed; no particular private helper name or provider model is required. A successful primary needs no recovery. Disabled or unconfigured recovery and requests with images or files must retain the original allowance refusal without dropping input. If recovery is unavailable, preserve the original hosted_ai_allowance_exceeded 429 response. Ordinary requests-per-minute throttling must not activate allowance recovery; an empty message list must return 400 before provider work. Tests use the existing handler entrypoint with both legacy and current option spellings, synthetic provider and telemetry ports, and denied external network. Authorization and paid-plan HTTP dispatch, safety-refusal recovery policy, actual provider compatibility, concurrent spend accounting and model quality are outside this case. Add focused tests for the outcomes.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/background-request.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-background-request.test.ts"
}
],
"command": "cd packages/ai-gateway && \"${BUN_BIN:-bun}\" test src/test/eval-background-request.test.ts"
}
},
{
"id": "app-canvas-preview-persistence",
"name": "Keep unaccepted canvas positions out of saved Live Views",
"source": {
"kind": "git_regression",
"fix_commit": "d8621a14179173c7eb4df82085d9a209251f4454"
},
"base_ref": "85ae3c99fc1369b7261667cd5c083730d361ea0e",
"oracle_ref": "d8621a14179173c7eb4df82085d9a209251f4454",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/settings/brain-overview.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/settings/brain-overview.tsx",
"apps/screenpipe-app-tauri/components/settings/live-view-canvas.tsx",
"apps/screenpipe-app-tauri/lib/live-views/canvas-layout.ts"
],
"tags": [
"regression",
"escaped-failure",
"canvas",
"persistence"
],
"prompt": "A Live View with no saved canvas yet can preview an AI-proposed new block before the user accepts it. Canvas viewport changes during that review currently send positions for the unaccepted block to native persistence, which rejects them. Keep previews visible, but send and persist canvas positions only for blocks in the saved view until acceptance. Resetting the zoom and then zooming in during review must successfully persist the viewport without a save-error notification. Preserve the saved block positions and correct view ownership. Keep ordinary viewport editing without proposals working. For an existing canvas, preserve stored notes and allow note-text edits during review without saving the proposed block. Rejecting all proposals must leave the saved view and annotations intact, make no view-definition save, and allow later canvas editing. Accepting all proposals must persist the added block, leave annotations intact and allow later canvas edits with the newly accepted block positions. Preserve the public BrainOverview interface and the synthetic native command boundary. Exact save-call counts, DOM identities, test IDs, CSS and private helper names are not prescribed. Use synthetic data and external ports only; do not perform live model calls or native/customer storage operations.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/canvas-outcome.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-canvas-outcome.test.tsx"
},
{
"local_path": "graders/canvas-outcome.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-canvas-outcome.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-canvas-outcome.config.mjs"
}
},
{
"id": "app-onboarding-notification-transport",
"name": "Deliver onboarding follow-ups through the app-control server",
"source": {
"kind": "git_regression",
"fix_commit": "35caddb4562d3bd15fdb0aa3f7928ce20dfed93c",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/5844"
},
"base_ref": "a7d74fb9adf6c9c286ce9cfa33b2e8c9c62135bc",
"oracle_ref": "35caddb4562d3bd15fdb0aa3f7928ce20dfed93c",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/live-views/onboarding-follow-up.ts",
"apps/screenpipe-app-tauri/lib/notifications/app-server.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/live-views/onboarding-follow-up.ts",
"apps/screenpipe-app-tauri/lib/notifications/app-server.ts"
],
"tags": [
"regression",
"escaped-failure",
"notifications",
"retry",
"transport"
],
"prompt": "An already scheduled onboarding follow-up dispatches its permitted dashboard work but never delivers its notification because notification requests go to the recording engine. Repair the existing runDueOnboardingLiveViewFollowUp default transport. Keep Pipe runs on the engine transport, deduplicate shared Pipe bindings, and POST notifications to the app-control server using its native runtime port; use localhost:11435 if native configuration lookup fails. Preserve the notification link to the originating Live View. Persist sent status only after notification acceptance, and suppress later duplicate delivery. A rejected notification must remain scheduled for a later retry that can recover and persist sent status. If all engine dispatches fail, schedule a retry without sending a notification. A future deadline or a missing view must perform no transport work. Use synthetic native configuration, storage and network ports. This task fixes delivery of an existing scheduled follow-up; it does not authorize new scheduling, activation of paused work, claim completed Pipe results, or change notification wording or notification policy. Native display, engine completion, multi-window concurrency, recording and live service behavior are outside scope.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/notification-transport.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-notification-transport.test.ts"
},
{
"local_path": "graders/notification-transport.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-notification-transport.config.mjs"
},
{
"local_path": "graders/notification-localization.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-notification-localization.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-notification-transport.config.mjs"
}
},
{
"id": "ai-gateway-lazy-stream-initial-error",
"name": "Expose initial lazy-stream failures before successful stream delivery",
"source": {
"kind": "git_regression",
"fix_commit": "e8ed10afed8400e943c7ab2cb9c7a18f7b83e5b6",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/5840"
},
"base_ref": "bfaeef6183c6b757f9779b25635d3687466f56e3",
"oracle_ref": "e8ed10afed8400e943c7ab2cb9c7a18f7b83e5b6",
"oracle_paths": [
"packages/ai-gateway/src/providers/openai.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/openai.ts"
],
"tags": [
"regression",
"escaped-failure",
"ai-gateway",
"streaming",
"error-handling"
],
"prompt": "An OpenAI-compatible SDK can return a stream successfully and raise its upstream error only when iteration begins. The provider currently turns that initial failure into an already-successful stream, preventing callers from handling it before delivery. Repair the actual OpenAIProvider streaming boundary so initial quota and service-unavailable errors reject before returning a successful stream, preserving their message, status and code. Preserve SDK creation errors as well. Successful streams must deliver the first and later text or tool-call fragments exactly once and in order, preserve tool identity and argument assembly, completion reasons and usage including cached tokens, and end with one [DONE]. Empty upstream streams must complete normally without invented content. After content has begun, later errors must remain SSE errors with original message and numeric status code, network_error completion and [DONE], retaining the partial content. Preserve the input object and outgoing model/messages/stream request. Use synthetic SDK and telemetry ports and no external network. No particular iterator, private helper or priming implementation is required. Caller fallback selection, retired model-routing policy, cancellation, live provider behavior and model quality are outside this task.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/gateway-stream-initial-error.test.js",
"destination_path": "evals/coding-agent/graders/gateway-stream-initial-error.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/gateway-stream-initial-error.test.js"
}
},
{
"id": "ai-gateway-settlement-replay-atomicity",
"name": "Apply replayed hosted-AI settlements once without partial accounting",
"source": {
"kind": "git_regression",
"fix_commit": "d9ff95802e6566a7b3ea3242800aef59cacb9aa0"
},
"base_ref": "8b455607ffbaef204a6f6a0a62e973bac6211e48",
"oracle_ref": "d9ff95802e6566a7b3ea3242800aef59cacb9aa0",
"oracle_paths": [
"packages/ai-gateway/src/services/cost-tracker.ts",
"packages/ai-gateway/src/services/hosted-ai-settlement-ledger.ts"
],
"gate": "advisory",
"tags": [
"accounting",
"retry",
"concurrency",
"ownership"
],
"trigger_paths": [
"packages/ai-gateway/src/services/cost-tracker.ts",
"packages/ai-gateway/src/services/hosted-ai-settlement-ledger.ts"
],
"prompt": "The existing hosted-AI logCost entrypoint can apply the same priced request repeatedly and report success after only part of its accounting succeeds. Repair its budgeted text accounting for calls carrying a stable settlement_id. Sequential or concurrent identical replays must return true and apply the account cost and cost_daily request/cost aggregate exactly once. Different IDs still apply independently. Reusing an applied ID with a different cost or account owner must return false without changing either account or the aggregate. A failed dependent aggregate write must return false with no partial account charge; a later successful retry applies once. If the accounting transaction committed but its acknowledgement was lost, an identical retry must succeed without another charge. Zero-cost replays record one request and no spend. Preserve existing legacy calls without settlement_id: each call still accounts independently. Preserve budgeted:false telemetry-only calls and transcription_budgeted:true calls: they retain their repeated telemetry behavior, with unbudgeted calls not charging accounts and speech calls charging only their existing separate transcription account bucket. Exercise the real logCost entrypoint with synthetic local D1 data, never production services. Any implementation meeting these outcomes is acceptable; do not depend on particular helper names or a particular idempotency-table design. This case does not prove request admission, hold release, all global/background/trial budget windows, deployment activation, or provider billing correctness. Add focused tests for these outcomes.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/settlement-replay.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-settlement-replay.test.ts"
}
],
"command": "cd packages/ai-gateway && \"${BUN_BIN:-bun}\" test src/test/eval-settlement-replay.test.ts --timeout=10000"
}
},
{
"id": "app-consumer-stale-managed-settings",
"name": "Release stale managed settings only on confirmed consumer builds",
"source": {
"kind": "git_regression",
"fix_commit": "9712ee210f7efc7f85e98bcc1e80c4c4173450dc"
},
"base_ref": "335ca8d842d1ac0311c038a72a5c3bf084081340",
"oracle_ref": "9712ee210f7efc7f85e98bcc1e80c4c4173450dc",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx",
"apps/screenpipe-app-tauri/lib/hooks/use-is-enterprise-build.ts"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"settings"
],
"prompt": "Consumer and enterprise installations can share a settings store. A confirmed consumer build currently keeps enforcing stale enterpriseManagedSettings, silently undoing personal choices. Repair actual settings loading, updating and resetting so an authoritative native consumer result releases the stale managed values and durably removes that cached policy. Loading must preserve the existing personal screenshot and font choices. Writes must persist new choices, survive reload and retain unrelated settings. Reset must restore ordinary defaults without restoring the stale policy. An enterprise build or failed native build check must continue enforcing and retaining managed screenshot and font values on read, write and reset; unrelated settings must remain editable. With no cached policy, ordinary edits must still persist in all three situations. Exercise the real settings provider and store actions, their queued disk-write completion and reload using synthetic native, storage and external ports. Do not perform native builds, network operations or actual user-file writes. This case checks frontend persistence and its build-authority gate; it does not authorize recording, migration, changes to native policy ownership or real device enforcement. No helper name or implementation structure is required.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/stale-policy.test.js",
"destination_path": "evals/coding-agent/graders/stale-policy.test.js"
},
{
"local_path": "graders/stale-policy.mjs",
"destination_path": "evals/coding-agent/graders/stale-policy.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && \"${BUN_BIN:-bun}\" ../../evals/coding-agent/graders/stale-policy.mjs"
}
},
{
"id": "app-public-hosted-ai-allowance",
"name": "Keep actual hosted text admission inside public included credits",
"source": {
"kind": "git_regression",
"fix_commit": "a448803c30e4220d60e8d55003a796b425d9e349"
},
"base_ref": "f073b89db6894546cf72fddfb2ce39cbe86dfec6",
"oracle_ref": "a448803c30e4220d60e8d55003a796b425d9e349",
"oracle_paths": [
"packages/ai-gateway/src/services/hosted-ai-cost-controls.ts",
"packages/ai-gateway/src/services/hosted-ai-policy.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"billing",
"admission",
"concurrency"
],
"trigger_paths": [
"packages/ai-gateway/src/services/cost-cap.ts",
"packages/ai-gateway/src/services/hosted-ai-cost-controls.ts"
],
"prompt": "The existing hosted text reserveDailyCostCap entrypoint can admit spending beyond the public included-credit ceiling when private controls are permissive. Repair actual local-D1 request reservation. With synthetic private request/daily/monthly and trial windows set to10 USD, a request whose estimated hold would cross the public monthly ceiling must return allowed:false and429 with monthly_cost_limit_exceeded, without changing existing spend or leaving a reservation. Public monthly provider-cost ceilings are Free0.10USD, Basic1.50USD and Business/BusinessMax/BusinessUltra4USD; an active Business trial has a4USD total ceiling and must similarly refuse with trial_cost_limit_exceeded. A request with room for two estimated holds must still be admitted for every listed plan/period, persist its positive hold in the existing usage ledger, preserve prior spend, and release exactly that hold through releaseDailyCostReservation. Two simultaneous Basic requests with room for only one-and-a-half holds must admit exactly one; the other must receive429, and releasing the accepted hold must restore prior ledger state. Do not prescribe a specific monthly-versus-temporary-capacity error code for that concurrent refusal. Preserve a stricter private0.80USD monthly ceiling, and fail closed with503/cost_control_unavailable and no hold for malformed private configuration. Use the existing entrypoint, ledger and release interface with a fixed clock and synthetic workerd D1. No particular private helper name, SQL formatting or algorithm is required. This case does not execute HTTP authentication/entitlement, provider dispatch, final cost settlement, global or background windows, incident resets, deployment activation, real financial operations, native behavior, execution isolation or model trials.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/public-allowance.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/services/eval-public-allowance.test.ts"
}
],
"command": "cd packages/ai-gateway && bun test src/services/eval-public-allowance.test.ts"
}
},
{
"id": "app-reservation-admission-proof",
"name": "Use persisted reservation evidence despite ambiguous write metadata",
"source": {
"kind": "git_regression",
"fix_commit": "a5749d6d922fec0f642720dbb9a0cbbc3719ea82"
},
"base_ref": "7a1462b5d0684f5ad4a7c5059f8070ee8a8f2eb6",
"oracle_ref": "a5749d6d922fec0f642720dbb9a0cbbc3719ea82",
"oracle_paths": [
"packages/ai-gateway/src/services/cost-cap.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"billing",
"admission",
"concurrency"
],
"trigger_paths": [
"packages/ai-gateway/src/services/cost-cap.ts"
],
"prompt": "Repair the existing reserveDailyCostCap request-admission boundary. A successful local-D1 write may store a reservation while its change-count metadata is zero or absent. Healthy requests in those conditions must be admitted with a positive stored hold matching the returned key, account and reserved amount, and releaseDailyCostReservation must remove exactly that hold. Conversely, positive change-count metadata is not sufficient when no row was inserted: a full account with three active interactive reservations must reject the next request with 429 and preserve every existing hold; a reported successful write that stores nothing must refuse with 429 or 503. With a configured interactive limit of three, six concurrent requests under ambiguous write metadata must yield exactly three allowed responses and three 429 responses, matching exactly three stored holds. Preserve ordinary healthy admission and exact release without deleting sibling reservations; a storage write exception must refuse with 503 and leave no hold. The fixture uses a fixed clock, synthetic Business requests for gpt-5.6-luna, the existing usage ledger, and real local workerd D1 with faults restricted to write execution or returned metadata. Keep the public reservation/release entrypoints; no specific private name, SQL formatting or row-reading method is required. This case does not test HTTP authentication, provider dispatch, actual settlement, production D1 fault prevalence, global or background limits, native behavior, execution isolation or model capability.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/admission-proof.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/services/eval-admission-proof.test.ts"
}
],
"command": "cd packages/ai-gateway && bun test src/services/eval-admission-proof.test.ts"
}
},
{
"id": "ai-gateway-provider-cap-guidance",
"name": "Explain provider spend-cap outages without blaming client input",
"source": {
"kind": "git_regression",
"fix_commit": "4e6c53848e337eacb8e5eff18843604800be05d8",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/5653"
},
"base_ref": "6b8f11c0c1ed933f4db6342d9b38c6160572cdff",
"oracle_ref": "4e6c53848e337eacb8e5eff18843604800be05d8",
"oracle_paths": [
"packages/ai-gateway/src/handlers/chat.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/handlers/chat.ts"
],
"tags": [
"regression",
"escaped-failure",
"provider-error",
"preserved-behavior"
],
"prompt": "Repair the existing chat-completions handler when a provider rejects work with HTTP 400 because its organization API usage limit has been reached. For explicit single-model JSON and streaming requests, and when every model in an explicit fallback chain has this cap, retain the 400 response but explain the provider capacity/usage limit and recovery through retrying later, a reset time or a different provider. Do not return nested provider error JSON, provider request identifiers, or blame malformed tools or unsupported parameters. Suppress redundant exception reports for this known provider cap while retaining failed-model health records. Preserve a successful fallback response, its original messages/tools and the caller request; fallback already works in the broken revision. Preserve healthy explicit replies, malformed-image client errors, rejection of empty messages before provider work, provider authentication failures with 401 and their API-key detail, and ordinary 429 throttling with retry guidance rather than a provider-cap explanation. Keep the public handler entrypoint. Grade observable response and provider/telemetry effects, not private helper names or exact advisory wording. Tests use synthetic provider and telemetry ports, no live requests. HTTP authentication, budget accounting, actual provider availability, log redaction beyond returned error payloads and execution isolation are outside this case.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/provider-cap-guidance.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-provider-cap.test.ts"
}
],
"command": "cd packages/ai-gateway && \"${BUN_BIN:-bun}\" --no-env-file test src/test/eval-provider-cap.test.ts"
},
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
]
},
{
"id": "ai-gateway-active-ip-quota-retention",
"name": "Preserve active IP quotas during runtime cleanup",
"source": {
"kind": "git_regression",
"fix_commit": "6b92dc24ee785857a06a2d39188c8831e98e865f"
},
"base_ref": "a687dbcb28d83d0691c169fe10aa2863297a0820",
"oracle_ref": "6b92dc24ee785857a06a2d39188c8831e98e865f",
"oracle_paths": [
"packages/ai-gateway/src/services/runtime-state-maintenance.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/services/runtime-state-maintenance.ts",
"packages/ai-gateway/src/services/usage-tracker.ts"
],
"tags": [
"regression",
"escaped-failure",
"quota",
"preserved-behavior"
],
"prompt": "Repair runtime-state cleanup so active IP usage counters are not erased when their update metadata is stale. For ip_tracking rows, retain last_reset dates from seven days ago onward, including the exact seven-day boundary and future dates, preserving the entire row and daily count. Remove older reset dates even when updated_at is recent. Repeating cleanup must neither reset active counts nor resurrect expired rows. Preserve the separate updated_at-based seven-day retention for free_chat_in_flight_v1, free_chat_turn_v2:* and free_chat_budget_v2:* rows, and ninety-day retention for ordinary usage. A single invocation must make progress while deleting no more than 5000 expired IP rows. Retain existing ninety-day aggregate cleanup and one-hour model-health cleanup, including recent data. Keep the public pruneRuntimeState entrypoint. Grade observable stored rows using real SQLite executing production SQL through a sequential synthetic D1 adapter, not SQL spelling, private names, statement counts or helper architecture. This case does not establish native capture, live D1 atomicity/concurrency, cron delivery, current gateway admission policy or execution isolation.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/ip-quota-retention.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-ip-quota-retention.test.ts"
}
],
"command": "cd packages/ai-gateway && \"${BUN_BIN:-bun}\" --no-env-file test src/test/eval-ip-quota-retention.test.ts"
}
},
{
"id": "app-settings-restart-persistence",
"name": "Persist accepted settings before update restart",
"source": {
"kind": "git_regression",
"fix_commit": "46402fe054406730ef291fc7de18a7ded6f975ce"
},
"base_ref": "4e7b00fdb6317594150daefacb60d69eab8baaba",
"oracle_ref": "46402fe054406730ef291fc7de18a7ded6f975ce",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx",
"apps/screenpipe-app-tauri/components/update-banner.tsx"
],
"tags": [
"regression",
"escaped-failure",
"settings",
"update",
"persistence"
],
"prompt": "A user enables Auto-update and immediately chooses Restart to update, but the restarted app can retain the old setting. Repair the actual settings persistence and update-banner flow so accepted writes finish saving before the macOS/Linux native restart handoff. Include another settings edit accepted while the first save is pending: both changes must be saved before the handoff. A failed save must prevent the handoff and leave the banner usable; a later successful save must allow a retry. Restart with no pending writes must still proceed and preserve already saved Auto-update and font-size settings. No particular helper names, queue structure or private implementation is required. Native installation, OS persistence guarantees, encryption, Windows update behavior and cross-window synchronization are outside this task.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/settings-restart-persistence.test.js",
"destination_path": "evals/coding-agent/graders/settings-restart-persistence.test.js"
}
],
"command": "cd apps/screenpipe-app-tauri && \"${BUN_BIN:-bun}\" test ../../evals/coding-agent/graders/settings-restart-persistence.test.js"
}
},
{
"id": "app-chat-foreground-save-ownership",
"name": "Keep foreground event saves within the active conversation",
"source": {
"kind": "git_regression",
"fix_commit": "a5230a0d7a993a1a9a20781ca5abd6ace69e77b4"
},
"base_ref": "4c2f43f5c7b519dab72564a66541e4078a38963f",
"oracle_ref": "a5230a0d7a993a1a9a20781ca5abd6ace69e77b4",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-pi-foreground-events.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-pi-foreground-events.ts",
"apps/screenpipe-app-tauri/components/hooks/use-chat-conversations.ts"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"persistence",
"session-ownership"
],
"prompt": "After switching the foreground chat session, an echoed user message can replace the previous conversation's saved messages. Fix the foreground event path so a nonempty user message-start saves its user row and one assistant placeholder into the active session only, whether that target is new or already exists as an empty conversation. Preserve the previous conversation's contents and avoid creating extra conversation records. Preserve ordinary stable-session saves. An empty user event after switching must neither write a conversation nor add visible messages. Keep the public hooks and save callback interfaces compatible. This task covers the real React foreground event lifecycle with synthetic event, storage and native ports; it does not require native filesystem execution. No particular private helper name, callback-refresh strategy or implementation is prescribed.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/foreground-save-ownership.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-stale-session.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts eval-stale-session.test.tsx"
}
},
{
"id": "app-settings-telemetry-identity",
"name": "Respect runtime and user refusal before settings identity effects",
"source": {
"kind": "git_regression",
"fix_commit": "d57b2fb92d84f45a2b2374146a7fc0300ae95def"
},
"base_ref": "0bae3b170b3cd603381df4278d5e6311ba857734",
"oracle_ref": "d57b2fb92d84f45a2b2374146a7fc0300ae95def",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx",
"apps/screenpipe-app-tauri/lib/telemetry-env.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx",
"apps/screenpipe-app-tauri/lib/telemetry-env.ts"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"telemetry",
"async-lifecycle"
],
"prompt": "The settings provider can alias and identify an account even when the saved analytics preference is explicitly disabled or the native runtime reports telemetry disabled. Repair the actual SettingsProvider identity effect. Neither alias nor identify may run for either refusal, or while the native runtime decision is pending. Once that decision permits telemetry and the user explicitly opted in, preserve account aliasing and identification by the Clerk account ID, including the saved email and stable per-install analytics ID. Unmounting the provider while the app-version lookup is pending must prevent a late identify call. Keep the existing public provider/settings interfaces and native command contract compatible. Tests use actual React lifecycle with synthetic native, settings-store and PostHog ports. No particular private helper or gating structure is required. Native environment parsing, capture transport, unknown preferences and failed IPC policy are outside this task.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/telemetry-identity.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-telemetry-caller.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts eval-telemetry-caller.test.tsx"
}
},
{
"id": "app-chat-timeline-offset",
"name": "Preserve captured instants when opening chat timeline citations",
"source": {
"kind": "git_regression",
"fix_commit": "e6df687c7f674fbdda546925be17e1115a8f2fa1"
},
"base_ref": "afcee637bb263ada9ac34911052f693553c4333a",
"oracle_ref": "e6df687c7f674fbdda546925be17e1115a8f2fa1",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/chat/markdown-block.tsx",
"apps/screenpipe-app-tauri/lib/timeline-deeplink.ts",
"apps/screenpipe-app-tauri/lib/timeline-navigation.ts"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/markdown-block.tsx",
"apps/screenpipe-app-tauri/lib/timeline-deeplink.ts",
"apps/screenpipe-app-tauri/lib/timeline-navigation.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"chat",
"timeline",
"navigation"
],
"prompt": "Clicking an AI chat timeline citation can silently do nothing when its ISO timestamp has a raw positive timezone offset. Repair the rendered MarkdownBlock click path so timestamp links preserve the captured instant to millisecond precision across raw +00:00 and nonzero positive offsets, percent-encoded offsets, negative offsets and Z timestamps. Support start_time links, with timestamp taking precedence when both are present. Valid links must update pending timeline navigation, show the Main window and emit the existing navigate-to-timestamp event for the same instant; semantically equivalent timestamp formatting is allowed. Invalid timestamps must produce no navigation or viewer effects. Preserve frame navigation, local artifact in-chat preview callbacks and ordinary web-link destinations. Keep public component and native event/command contracts compatible. No particular parser or private helper is required. The task concerns chat clicks with actual React/Markdown rendering and synthetic native/store ports; native event delivery, the separate OS deeplink handler, timeline frame retrieval and sub-millisecond precision are outside this task.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/chat-timeline-offset.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/chat/eval-timeline-offset.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts components/chat/eval-timeline-offset.test.tsx"
}
},
{
"id": "app-local-api-authority",
"name": "Keep local API credentials within the configured authority",
"source": {
"kind": "git_regression",
"fix_commit": "107d957aca3523cd309f56d93bd5ab20ce6519bf"
},
"base_ref": "ebeef3e7249ebcc2d1a9d1824c3b2fb68a1cfacb",
"oracle_ref": "107d957aca3523cd309f56d93bd5ab20ce6519bf",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/api.ts",
"apps/screenpipe-app-tauri/lib/notifications/actions.ts"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/api.ts",
"apps/screenpipe-app-tauri/lib/notifications/actions.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"privacy",
"credentials",
"notifications"
],
"prompt": "The local API fetch wrapper and global fetch interceptor can attach the local bearer key to an external URL, and the shared notification API action executor can dispatch to an external target. Repair these actual request paths. Only HTTP localhost or 127.0.0.1 at the configured API port may receive automatically supplied local authentication; do not mistake URL credentials, host suffixes or path/query text for authority. localFetch must continue passing absolute non-local URLs through without adding its key. Resolve localFetch relative paths against the configured local API, and global fetch relative paths against document.baseURI. Preserve authenticated local requests, disabled authentication, explicit caller Authorization, and a local401 key-refresh retry. Shared notification API actions must refuse non-local targets (including protocol-relative URLs) with a rejected operation and zero transport calls, while preserving local POST method/body. Check non-default configured ports and wrong-protocol/wrong-port targets. Use existing public entrypoints and contracts; no particular helper or parser is required. This task covers the actual JavaScript wrapper/interceptor/shared action executor with synthetic native configuration and transport, not the separate toast component, browser redirects/cookies, native network delivery or server authorization.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/local-api-authority.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/lib/eval-local-api-authority.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts lib/eval-local-api-authority.test.ts"
}
},
{
"id": "app-chat-raw-html-sanitization",
"name": "Keep untrusted chat HTML inert without losing safe formatting",
"source": {
"kind": "git_regression",
"fix_commit": "bd90b1b682fabce2e938bda0f0cd2ae6fb74aba3"
},
"base_ref": "ba342ea4889862cea52adae856225dd60d1a8eaf",
"oracle_ref": "bd90b1b682fabce2e938bda0f0cd2ae6fb74aba3",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/chat/markdown-block.tsx"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/components/chat/markdown-block.tsx"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"chat",
"untrusted-html",
"preserved-behavior"
],
"prompt": "Chat messages can turn untrusted HTML into active embedded content inside the application. Repair the rendered MarkdownBlock for both user and assistant messages. Embedded iframes, forms/password inputs, meta refresh, object/embed content, style sheets and base URL elements must not survive as live elements. Remove inline styling from otherwise allowed formatting, and neutralize javascript URLs in links and image long descriptions. Keep the surrounding ordinary message text. Preserve collapsible details with their summary and nested Markdown emphasis, kbd/sub/sup/b/i/u/mark/small formatting, ordinary safe images, and bold text with its styling removed. GFM footnotes and raw in-page anchors must still resolve to their rendered targets; no particular ID prefix is required. Preserve ordinary web-link clicks in the owning chat browser, and local file links routed through the supplied in-chat preview callback without opening a separate viewer. Keep the public component and native port contracts compatible. No particular sanitizer or internal implementation is required. This task checks actual React/Markdown rendering with synthetic native/browser ports; it does not establish a complete browser exploit defense, native isolation or external navigation security.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
},
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "packages/workflows-ui/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/chat-html-outcomes.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/chat/eval-html-outcome.test.tsx"
},
{
"local_path": "graders/chat-html-outcomes.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-html.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-html.config.mjs components/chat/eval-html-outcome.test.tsx"
}
},
{
"id": "app-markdown-remote-images",
"name": "Render only local images and media from untrusted Markdown and meeting notes",
"source": {
"kind": "git_regression",
"fix_commit": "ba2985a684cb83b11be7b2dbf17db3032e205b8e"
},
"base_ref": "7274f29b6b1bb940332b4e912b9cfcb20d591d3d",
"oracle_ref": "ba2985a684cb83b11be7b2dbf17db3032e205b8e",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/markdown.tsx",
"apps/screenpipe-app-tauri/components/markdown/image-alt-text.tsx",
"apps/screenpipe-app-tauri/components/markdown/local-markdown-image.tsx",
"apps/screenpipe-app-tauri/lib/utils/media-file-path.ts",
"apps/screenpipe-app-tauri/components/chat/markdown-block.tsx",
"apps/screenpipe-app-tauri/app/notification-panel/page.tsx",
"apps/screenpipe-app-tauri/components/notification-bell.tsx",
"apps/screenpipe-app-tauri/components/announcement-body.tsx",
"apps/screenpipe-app-tauri/components/meeting-notes/note-editor.tsx",
"apps/screenpipe-app-tauri/components/meeting-notes/image-utils.ts",
"apps/screenpipe-app-tauri/components/rewind/mermaid-diagram.tsx",
"apps/screenpipe-app-tauri/components/rewind/mermaid-sandbox.ts"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/components/markdown.tsx",
"apps/screenpipe-app-tauri/components/markdown/local-markdown-image.tsx",
"apps/screenpipe-app-tauri/components/chat/markdown-block.tsx",
"apps/screenpipe-app-tauri/components/meeting-notes/note-editor.tsx",
"apps/screenpipe-app-tauri/lib/utils/media-file-path.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"chat",
"markdown",
"meeting-notes",
"privacy",
"preserved-behavior"
],
"prompt": "Rendered Markdown and meeting notes make the application load remote resources the moment they render. Chat replies, summaries and AI meeting notes are built from captured outside content, so steered text can put private data in an image URL that leaks without any click. Repair the chat MarkdownBlock for both user and assistant messages, the shared MemoizedReactMarkdown renderer, and the meeting note editor (NoteEditor, built from createMeetingNoteEditorExtensions). Only local files may display as images or media in Markdown: absolute paths, file URLs and Windows drive paths for images, plus home-relative paths for media, must still be read through the supplied native media port and displayed; a Windows drive path whose backslashes are doubled, the way Markdown escapes them, is still a local path. No other image source (http, https, protocol-relative, relative, reference-style, raw HTML img, or picture/source srcset) may produce an element that loads it; keep its alt text visible instead, or its address when it has no alt text, and keep the alt text when the source is removed for safety. A local image the native media port cannot read keeps its alt text and path visible. Media images, links and code that name a remote URL or a network-share path must never reach the native media reader, including a home-relative path that resolves to a share or a local-looking string from which the reader would take a share path; keep their alt text, link or code visible. In a meeting note, a remote image written by the AI or arriving with a later update, including in place of an embedded image, must not load: keep its alt text visible, keep images embedded as data: URLs displaying, and keep the note's Markdown, including the remote image reference, so viewing, saving, or copying a note and pasting it into a note loses nothing; pasted embedded images keep their alt text and size, and a web image pasted into a note that cannot be downloaded through the native HTTP port stays in the note's Markdown as its address without loading. Keep the surrounding text. Keep the public component, editor extension and native port contracts compatible. No particular sanitizer, helper or internal implementation is required. This task checks actual React/Markdown and editor rendering with synthetic native ports; it does not establish webview CSP, Mermaid diagrams, pasting from other apps, native file access policy or network isolation.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
},
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "packages/workflows-ui/node_modules"
}
],
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/markdown-remote-image-outcomes.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/chat/eval-remote-image-outcome.test.tsx"
},
{
"local_path": "graders/markdown-remote-image-outcomes.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-remote-images.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-remote-images.config.mjs components/chat/eval-remote-image-outcome.test.tsx"
}
},
{
"id": "ai-gateway-trusted-runner-admission",
"name": "Admit verified machine chat while preserving access refusals",
"source": {
"kind": "git_regression",
"fix_commit": "25716cb17c4a11a75377df02b830891cc83229db"
},
"base_ref": "2168f5a3e118c28b79f6ba2ab15fff437c9cec33",
"oracle_ref": "25716cb17c4a11a75377df02b830891cc83229db",
"oracle_paths": [
"packages/ai-gateway/src/index.ts",
"packages/ai-gateway/src/services/free-chat-limit.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"access",
"machine-identity",
"background"
],
"prompt": "A trusted backend runner supplies the configured service bearer but hosted chat rejects it because it has no human user ID. Repair the existing POST /v1/chat/completions path so that exact configured machine credential can complete both interactive and background requests for its requested paid model. The actual chat/provider path must make one successful request carrying the original user message and requested model, then return the provider answer. Preserve successful paid human chat. Missing, incorrect, public-user-ID, unsupported-scheme and unconfigured service credentials must return 401 before provider work, even with runner-like device, workload and background headers. A verified Free human background request must remain 403; missing human plan truth must remain 503. Machine and paid-human rate-limit refusals must remain 429 without provider work. A machine request with an empty model or empty message list must return 400 without provider work. Use synthetic credentials, entitlement, rate-limit, storage and provider ports only. Do not assume any particular private helper or guard structure. This case exercises authentication through HTTP dispatch and provider-response propagation, not JWT cryptography, live inference, SQL/concurrent spend accounting, full current entitlement integration, agent isolation or model quality.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/trusted-runner-admission.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-trusted-runner.test.ts"
}
],
"command": "cd packages/ai-gateway && \"${BUN_BIN:-bun}\" test src/test/eval-trusted-runner.test.ts"
}
},
{
"id": "app-migration-unicode-search-probe",
"name": "Preserve Unicode history during explicitly requested storage conversion",
"source": {
"kind": "git_regression",
"fix_commit": "72e09b7dd7425fcb0387715d42782704f9de677a"
},
"base_ref": "1a8e311dc8b82725449cd59ae80f814d03bf5c62",
"oracle_ref": "72e09b7dd7425fcb0387715d42782704f9de677a",
"oracle_paths": [
"crates/screenpipe-db/src/storage/lifecycle.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-db/src/storage/lifecycle.rs"
],
"tags": [
"regression",
"escaped-failure",
"migration",
"data-preservation"
],
"prompt": "An explicitly requested offline storage conversion fails on otherwise valid histories containing control characters, diagram glyphs or non-Latin text. Make these histories convert successfully, including symbol-only histories, while retaining exact frame payloads and search results after reopening the converted store. Preserve ordinary-text conversion, frame counts, a verified completion receipt, absence of an unfinished journal, and storage-integrity verification. Do not silently bypass genuine missing source search-index errors: refuse those conversions while retaining the original frame contents and leaving no success receipt or unfinished journal. Low disk headroom and external readers must likewise preserve the original history on refusal; once the external reader releases its transaction, explicit retry must succeed with payloads intact. Exercise actual temporary SQLite histories and reopened storage. No particular helper, token-extraction algorithm or internal call path is required. This task concerns conversion correctness only; it does not authorize automatic migration, broaden recording or migration permissions, alter pause/cohort/stop controls, delete retained recovery copies, contact live services, or publish releases. Preserve public storage interfaces.",
"agent_timeout_seconds": 1800,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/migration-unicode.rs",
"destination_path": "crates/screenpipe-db/tests/eval_migration_unicode.rs"
}
],
"command": "cargo test --offline --locked -p screenpipe-db --test eval_migration_unicode -- --test-threads=1"
}
},
{
"id": "app-trial-upgrade-expiration",
"name": "Paid upgrade replaces stale trial state",
"source": {
"kind": "git_regression",
"fix_commit": "29f4477d049b37b295184e50016a4c16c31fa452"
},
"base_ref": "e57c61cd78ffd6bf6519af5e08cfcca668826066",
"oracle_ref": "29f4477d049b37b295184e50016a4c16c31fa452",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/settings/account-section.tsx",
"apps/screenpipe-app-tauri/components/plan-expiration-notice.tsx",
"apps/screenpipe-app-tauri/lib/app-entitlement.ts"
],
"gate": "advisory",
"tags": [
"trial",
"entitlement"
],
"prompt": "Account settings can continue showing a profile-trial countdown after billing activates a subscription. Make the displayed and saved account state agree with the verified subscription entitlement. A subscription entitlement must suppress leftover profile-trial expiry, while an active manual trial must retain its countdown and billing navigation. After checkout is confirmed active, persist the cleared trial expiry and refreshed subscription entitlement, preserving account identity and the signed-in token. An unconfirmed checkout must retain the manual trial and expiry. Preserve the active Business account and its cloud settings, the active Basic badge with cloud controls locked, login-first behavior without an active badge or enabled cloud controls for signed-out or tokenless accounts, and routing existing Basic upgrades to billing without a duplicate checkout. Use synthetic service responses for tests. The task concerns Account settings; do not change product policy or native capture.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/trial-upgrade.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/settings/__tests__/eval-trial-upgrade.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts components/settings/__tests__/eval-trial-upgrade.test.tsx --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "app-capture-start-rejection-recovery",
"name": "Retry capture startup after a transient refusal",
"source": {
"kind": "git_regression",
"fix_commit": "e5a3f4859b9a9bc99867597c649e4c1767b69f29"
},
"base_ref": "0acfbf0da25f2ab007931c68c61335b300c47200",
"oracle_ref": "e5a3f4859b9a9bc99867597c649e4c1767b69f29",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/onboarding/engine-startup.tsx"
],
"gate": "advisory",
"tags": [
"onboarding",
"capture",
"retry"
],
"prompt": "Onboarding can remain stuck after a transient native capture-start failure even after the underlying condition recovers. Allow a later readiness check to retry a failed start, whether the native call returns an error or rejects. Advance exactly once after recovery. With promptly settling synthetic native and health responses, keep completion within 2.5 seconds; once a delayed successful native call settles, advance within one more second. Preserve one in-flight native startup at a time, wait for it before advancing, and reuse an already successful session without stopping capture or respawning a listening engine. Persistent startup refusal must show the original refusal reason, retain the failure screen and must not advance. Successful native initialization may advance when media is intentionally idle, including disabled screenshots, or when the local health endpoint requires authentication. An unrelated health response must not establish capture readiness. Use synthetic native and health ports for tests; do not change recording permission policy or perform live capture. No particular helper, promise layout, log text or telemetry schema is required.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/capture-recovery.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/onboarding/eval-capture-recovery.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts components/onboarding/eval-capture-recovery.test.tsx --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "app-auth-verification-schedule",
"name": "Keep session verification on schedule across account refreshes",
"source": {
"kind": "git_regression",
"fix_commit": "89c104ce80f4dbebe23323d6dabab2fc873ecd0c"
},
"base_ref": "b3ac026dbef83a15dc6f93deb98883609ea1320d",
"oracle_ref": "89c104ce80f4dbebe23323d6dabab2fc873ecd0c",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/auth-guard.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/auth-guard.tsx",
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx"
],
"tags": [
"regression",
"escaped-failure",
"auth",
"scheduling",
"session-preservation"
],
"prompt": "Account refreshes and unrelated settings renders can repeatedly restart or postpone desktop session verification. Keep the startup check at five seconds and periodic checks every ten minutes from the start of each token session, even when refresh callbacks change. Use the latest refresh callback. Changing the account token must start a fresh schedule and cancel the old one. Stop checks on sign-out and unmount, while preserving rendered children. Preserve eager verification on visible focus or visibility changes with a 30-second cooldown, and suppress it while hidden. A current-session 401 must clear the full saved account and native cloud token. Permission errors and transient network failures must preserve the account, and an old account's delayed 401 must never clear a newly selected account.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/auth-schedule.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/lib/auth-schedule.eval.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/auth-schedule.eval.test.tsx --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "app-bounded-first-run-preview",
"name": "Bound native-authorized first-run activity previews",
"source": {
"kind": "git_regression",
"fix_commit": "dd652a6dc5db63832e8b7b913af8151c502608cc"
},
"base_ref": "34fb4aec4995c3b0874db8565c198df9283968e0",
"oracle_ref": "dd652a6dc5db63832e8b7b913af8151c502608cc",
"tags": [
"regression",
"escaped-failure",
"onboarding",
"polling",
"cancellation"
],
"prompt": "The first-run activity preview can repeatedly query history from restored browser state, overlap slow reads and continue after native summary generation takes over. Fix the actual learning-window hook. Preview reads require a completed native onboarding status in its learning phase and the native start time. Restored state alone, pending native status, idle/writing phases and an expired window must not authorize reads. Keep one read in flight; after a successful completion, wait the existing three-second preview interval before the next read. A null read must back off to six seconds, then twelve seconds for the next consecutive null result. The existing learning deadline must not restart after navigation or reload. Cancel pending reads when the hook unmounts, the existing deadline expires or native generation starts writing, and ignore late results without changing the preview or starting more reads. Successful previews must still display their returned app counts and minutes and use the native time window. Preserve native ready-chat restoration, unread/open summaries across reload, explicit dismissal across remount, native writing projection and the summary-first trial paywall transition only after rendering. A restored summary behind an already persisted treatment paywall must remain saved. Tests exercise the actual hook and storage helpers with synthetic native status/event, analytics, clock and activity-read ports. Native recording, SQL history queries and server generation are outside this task. Add or update focused tests as needed.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/bounded-preview.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/lib/first-run/eval-bounded-preview.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && bun x --no-install vitest run --config vitest.config.ts lib/first-run/eval-bounded-preview.test.ts --maxWorkers=1 --minWorkers=1"
},
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/first-run/use-learning-window.ts"
],
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
]
},
{
"id": "app-gateway-response-lifetime-cost-settlement",
"name": "Retain completed gateway usage through request lifetime",
"source": {
"kind": "git_regression",
"fix_commit": "7a320188f55127d91762c336483adcd3336a46a6"
},
"base_ref": "1f7880caf6b4c0faa0f4e9805f8002f794f4d0d2",
"oracle_ref": "7a320188f55127d91762c336483adcd3336a46a6",
"oracle_paths": [
"packages/ai-gateway/src/index.ts"
],
"gate": "advisory",
"tags": [
"regression",
"billing",
"gateway",
"request-lifetime"
],
"prompt": "Successful authorized chat completions can return their answer while their deferred usage recording is lost when the request lifetime ends. Repair the existing gateway HTTP handler so successful JSON and SSE completions retain exactly one daily usage record with the actual input, output, and cached-input token counts. Preserve the successful status and tool-call response body for both formats, and continue rejecting unauthenticated requests without recording cost. Work may finish before the response completes or be retained as background work; no specific helper or scheduling API is required. Tests use the real handler and cost-recording source with synthetic provider, authentication, rate-limit, and database ports. The database write completes on a later scheduler turn and survives only while the handler/body or registered execution-lifetime work remains active. External network is denied. Missing usage, provider failure, retries, quota reservations, concurrency, runtime termination behavior, live D1 durability, and model quality are outside this case. Add focused tests.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/cost-lifetime.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-cost-lifetime.test.ts"
}
],
"command": "cd packages/ai-gateway && \"${BUN_BIN:-bun}\" test src/test/eval-cost-lifetime.test.ts"
}
},
{
"id": "app-workflow-voice-immutable-response",
"name": "Preserve workflow voice responses across Workers fetch",
"source": {
"kind": "git_regression",
"fix_commit": "d9c66f9bb4143fde99331a1b29dc58ff86cfc580"
},
"base_ref": "b09141e87b263098d12394901f799220c53b247c",
"oracle_ref": "d9c66f9bb4143fde99331a1b29dc58ff86cfc580",
"oracle_paths": [
"packages/ai-gateway/src/index.ts"
],
"gate": "advisory",
"tags": [
"regression",
"gateway",
"voice",
"workers"
],
"prompt": "The authenticated workflow voice endpoint returns an internal error while forwarding Durable Object responses. Repair the actual gateway boundary so an invalid voice request retains its 400 JSON error, a valid session creation retains 201 with the provider SDP and a usable call token, and terminating that session retains empty 204. All three responses must include no-store and permissive CORS headers. Preserve one synthetic provider session creation and one termination, and reject a service-only caller with 401 before provider work. Tests run the existing local gateway harness with a real Workers Durable Object boundary, local D1, and synthetic account/provider replies; unexpected outbound requests are refused. Any implementation preserving these observable outcomes is valid. No live provider, deployment, billing safety, concurrency, native audio, runtime isolation or model-quality claim is required. Add focused tests.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/ai-gateway/node_modules",
"destination_path": "packages/ai-gateway/node_modules"
}
],
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/workflow-voice-response.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-workflow-voice-response.test.ts"
}
],
"command": "cd packages/ai-gateway && WRANGLER_SEND_METRICS=false \"${BUN_BIN:-bun}\" test src/test/eval-workflow-voice-response.test.ts --timeout 45000"
}
},
{
"id": "app-workflow-sharing-entry-consent",
"name": "Offer independent account-scoped workflow sharing consent",
"source": {
"kind": "git_regression",
"fix_commit": "05b3f08ae1574c728ba7af952b12bdc954df1e06"
},
"base_ref": "97adca652c426d95bcb192c76013852158446804",
"oracle_ref": "05b3f08ae1574c728ba7af952b12bdc954df1e06",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/workflows/workflow-tasks-prompt.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/workflows/workflow-tasks-prompt.tsx",
"apps/screenpipe-app-tauri/components/workflows/workflow-sharing-controls.tsx"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"consent",
"account-boundary"
],
"prompt": "Entering Workflows does not offer the optional sharing choice when automatic updates are off or the recorder is still connecting. Make the real workspace prompt and sharing controls offer an authenticated account its unseen sharing notice independently of recorder readiness and update allowance. Merely opening or skipping must not grant sharing, change redaction/recording preferences, or enable schedules. Skipping records the current notice for that account while preserving other accounts; returning to Workflows must not repeat it. Wait for account hydration, offer a fresh choice when switching to an unseen account, and close on sign-out. An unanswered choice must survive recorder disconnect/reconnect. Preserve no prompt or service work while Workflows is inactive, and no prompt for an already-seen account. Preserve explicit successful sharing through the existing consent service, including both sharing and training permission, account/epoch persistence and cloud redaction without changing recording preferences. An unavailable account must not be allowed to grant sharing and must still be able to skip. After skipping, normal schedule confirmation must enable updates without reopening sharing or granting it. Keep public component and service contracts compatible; preserve existing accessible action labels. No particular hook structure is required. Tests execute both real React components with synthetic settings, consent-service and native-token ports. Server consent enforcement, durable native settings and real data upload are outside this task.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/sharing-entry.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/workflows/eval-sharing-entry.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts components/workflows/eval-sharing-entry.test.tsx --maxWorkers=1 --minWorkers=1"
}
},
{
"id": "app-capture-cadence-durable-before-restart",
"name": "Persist capture cadence before recording restart",
"source": {
"kind": "git_regression",
"fix_commit": "110664b4481146d47312bda1e0789f73d83e8086"
},
"base_ref": "d62a64784956504e8a952d201546dddb336a4c54",
"oracle_ref": "110664b4481146d47312bda1e0789f73d83e8086",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/settings/recording-settings.tsx",
"apps/screenpipe-app-tauri/components/settings/settings-write-queue.ts"
],
"tags": [
"regression",
"escaped-failure",
"recording",
"settings",
"persistence"
],
"prompt": "In Screen settings, an already-authorized active capture can restart with the old capture frequency when a user changes the slider and immediately chooses Apply & restart. Repair the actual recording settings change/apply flow so every accepted cadence save completes before either capture stop or start is requested. A second cadence edit accepted while the first save is pending must also finish, with the latest value visible at both handoffs. A failed save must prevent both handoffs and leave Apply usable; a later successful edit and retry must restart normally with its saved value. Preserve Auto cadence as a saved null value and the normal capture stop/start sequence. Do not replace the cadence-only restart with a full server restart. The task concerns the frontend callback handoff with controlled settings and native ports. Native disk durability, pause/authorization enforcement, DOM presentation, React mount effects, background capture and audio are outside this task. No particular helper names, queue structure or test IDs are required.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/capture-cadence.test.js",
"destination_path": "evals/coding-agent/graders/capture-cadence.test.js"
}
],
"command": "cd apps/screenpipe-app-tauri && \"${BUN_BIN:-bun}\" test ../../evals/coding-agent/graders/capture-cadence.test.js"
}
},
{
"id": "app-shared-running-pipes-lifecycle",
"name": "Stop abandoned running-pipe trackers without disrupting active consumers",
"source": {
"kind": "git_regression",
"fix_commit": "94b670bc0cc3fd09a7b3fa02e1ae7fe93586599a",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/5116"
},
"base_ref": "9df282bf7daa7efb2e4e23759b7752eee445cefd",
"oracle_ref": "94b670bc0cc3fd09a7b3fa02e1ae7fe93586599a",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-running-pipes.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-running-pipes.ts"
],
"tags": [
"regression",
"escaped-failure",
"lifecycle",
"shared-state",
"stale-response"
],
"prompt": "The running-pipes sidebar tracker keeps polling and listening after its final consumer unmounts. Fix the existing useRunningPipes hook so active consumers share one tracker, removing one consumer preserves the others, and removing the last stops polling and event subscriptions. A later mount must refresh its state. Fetches and event-bus setup that finish after their original mount was abandoned must not start background work, overwrite a newer mount, or cancel the newer tracker. Preserve the existing 30-second recovery poll, recovery after temporary HTTP failures, running-only filtering, start-time ordering, pipe event updates and terminal removal; unrelated or malformed session events must remain ignored. Keep the existing hook interface. This task concerns React lifecycle with controlled HTTP and event ports, not native event delivery.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/running-pipes-lifecycle.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-running-pipes.test.tsx"
},
{
"local_path": "graders/running-pipes-lifecycle.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-running-pipes.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-running-pipes.config.mjs"
}
},
{
"id": "app-model-catalog-session-freshness",
"name": "Keep model catalogs tied to the current hydrated session",
"source": {
"kind": "git_regression",
"fix_commit": "757808f1043865752a05013e1a77973ce880f6f7",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/5139"
},
"base_ref": "074e5512469da7a74fa08711484a5b7f859706fb",
"oracle_ref": "757808f1043865752a05013e1a77973ce880f6f7",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-pi-models.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-pi-models.ts"
],
"tags": [
"regression",
"escaped-failure",
"session",
"stale-response",
"model-catalog"
],
"prompt": "The model picker can load before settings hydrate and later replace a new session\u2019s catalog with an older request. Fix the existing usePiModels hook. Do not request models until settings are loaded. Once loaded, support anonymous requests and include the current bearer token when present; refresh after token changes. Keep the last successful catalog visible during refresh and after HTTP or network failures. Responses from an abandoned session, including delayed response bodies, must never replace the current catalog or clear a newer request\u2019s loading state. Unloading settings must invalidate pending work and report no loading. Preserve the existing hook interface, model metadata, name fallback to ID, and first-occurrence deduplication by model ID. This task concerns React behavior with controlled settings and HTTP ports; native credential persistence is outside scope.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/model-catalog.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-model-catalog.test.tsx"
},
{
"local_path": "graders/model-catalog.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-model-catalog.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-model-catalog.config.mjs"
}
},
{
"id": "app-recording-global-device-resume",
"name": "Resume globally stopped recording and individually paused devices",
"source": {
"kind": "git_regression",
"fix_commit": "cdc3afb06cf62a6570cd196d1e9d0df87742498e"
},
"base_ref": "66936aa58874d10020132c80b679d42f47a63a3e",
"oracle_ref": "cdc3afb06cf62a6570cd196d1e9d0df87742498e",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/recording-status.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/recording-status.tsx"
],
"tags": [
"regression",
"escaped-failure",
"recording",
"user-pause"
],
"prompt": "The recording status popover has no working resume-all action when every device is paused. Preserve the RecordingStatus public interface and provide an accessible resume-all control. When the entire session is globally paused, resume through onResumeRecording once without issuing individual device requests. When the session is running but every device is individually paused, resume those devices through their existing POST /vision/device/start (monitor_id) and /audio/device/start (device_name) routes without invoking global callbacks. Keep pause-all routed through onPauseRecording for active or mixed device states. A legacy monitor without an id must issue no device request, while eligible audio devices still resume. Hide the global control when neither callback exists; when globally paused without a resume callback, the unavailable resume action must be disabled or absent. Preserve individual device request failure rollback. Keep the observable action names pause all recording, resume all recording, and resume accessible to assistive technology. No particular private function names, test attributes, or implementation structure are required. This task covers the historical frontend control and its synthetic callback/API boundaries, not native recording, the Home wiring, current rollout activation, or operating-system capture.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/recording-resume.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-hidden/recording-resume.test.tsx"
},
{
"local_path": "graders/recording-resume-config.mjs.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-hidden/recording-resume.config.mjs"
},
{
"local_path": "graders/recording-resume-translation.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-hidden/recording-resume-translation.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config .eval-hidden/recording-resume.config.mjs"
}
},
{
"id": "app-permission-recovery-audio-policy",
"name": "Respect audio-off policy in permission interfaces",
"source": {
"kind": "git_regression",
"fix_commit": "7d176bc2ed68cf331a83d0be76e1859cbabb06e1"
},
"base_ref": "4340a80723e50ff2192d6f3543a79ff7d6dd5462",
"oracle_ref": "7d176bc2ed68cf331a83d0be76e1859cbabb06e1",
"tags": [
"regression",
"escaped-failure",
"permissions",
"privacy",
"onboarding"
],
"prompt": "Audio-disabled users still encounter microphone prompts in permission recovery, onboarding and the status banner. Make these interfaces respect disableAudio and an audioCaptureMode of disabled, case-insensitively. Recovery must withhold the microphone action until settings finish loading. Audio-off users must still be able to repair denied accessibility and screen access, including a screen denial alongside a denied microphone. Remove an existing microphone banner immediately when audio is turned off. In onboarding, do not query or offer the unneeded microphone permission, advance while settings are loading, or complete after unmount. If audio is re-enabled before delayed completion, require a fresh microphone permission result, including when a previous result was granted; complete once after all required permissions are granted. Preserve ordinary audio-on microphone requests and the recovery wheel: only the first denied permission is actionable, granting it enables the next, and already granted rows stay inactive. Keep existing public permission-row identifiers and accessible action labels compatible. The checks render the actual three interfaces with synthetic settings and native command ports. Native recording restart, pause enforcement, OS permissions and deployment are outside this case. Keep the change scoped to these permission behaviors.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/audio-policy-recovery.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/app/permission-recovery/audio-policy.eval.test.tsx"
},
{
"local_path": "graders/audio-policy-onboarding.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/onboarding/audio-policy.eval.test.tsx"
},
{
"local_path": "graders/audio-policy-banner.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/status/audio-policy.eval.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage node node_modules/vitest/vitest.mjs run --config vitest.config.ts app/permission-recovery/audio-policy.eval.test.tsx components/onboarding/audio-policy.eval.test.tsx components/status/audio-policy.eval.test.tsx --pool=forks --poolOptions.forks.singleFork=true"
},
"oracle_paths": [
"apps/screenpipe-app-tauri/app/permission-recovery/page.tsx",
"apps/screenpipe-app-tauri/components/onboarding/permissions-step.tsx",
"apps/screenpipe-app-tauri/components/status/permission-banner.tsx",
"apps/screenpipe-app-tauri/lib/utils/permission-requirements.ts"
],
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
]
},
{
"id": "app-openai-stream-tool-choice",
"name": "Preserve explicit tool-selection policy on OpenAI streams and retries",
"source": {
"kind": "git_regression",
"fix_commit": "33d5637dda01c704e5436f0be477effbd18c1e4b"
},
"base_ref": "45105b5ee837501d0c10020a016b16ca2c4c4083",
"oracle_ref": "33d5637dda01c704e5436f0be477effbd18c1e4b",
"oracle_paths": [
"packages/ai-gateway/src/providers/openai.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/openai.ts"
],
"tags": [
"regression",
"gateway",
"tool-policy",
"streaming",
"preserved-behavior"
],
"agent_timeout_seconds": 1200,
"prompt": "Streaming requests through the OpenAI provider lose the caller's tool-selection policy, although ordinary completions preserve it. Repair the existing provider behavior so required, auto, none and named-function tool_choice values reach the provider unchanged on streaming and non-streaming requests. An unspecified choice must remain unspecified, including requests without tools. Preserve the selected model, messages, exact tool schemas and input object. If an OpenAI-compatible server rejects stream_options.include_usage, the supported retry must omit that option while retaining the original tools and tool choice. Preserve streamed content, native tool-call identity/name/arguments, tool_calls finish reason, prompt/completion/total/cached usage and exactly one completion terminator. Do not change routing/model policy or authentication. Use synthetic SDK transport without live API calls or executing tools; no specific implementation shape is required. This task concerns provider request/response behavior, not native Pi configuration, model adherence to the tool policy or full gateway authentication. All streamed content, tool fragments, finish reason and usage must arrive before the single completion terminator; clients stop consuming at that terminator.",
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/openai-tool-choice.test.js",
"destination_path": "evals/coding-agent/graders/openai-tool-choice.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/openai-tool-choice.test.js"
}
},
{
"id": "ai-gateway-auth-log-privacy",
"name": "Keep gateway authentication credentials and identities out of logs",
"source": {
"kind": "git_regression",
"fix_commit": "33d5637dda01c704e5436f0be477effbd18c1e4b"
},
"base_ref": "45105b5ee837501d0c10020a016b16ca2c4c4083",
"oracle_ref": "33d5637dda01c704e5436f0be477effbd18c1e4b",
"oracle_paths": [
"packages/ai-gateway/src/utils/auth.ts"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"authentication"
],
"prompt": "Gateway authentication writes credentials and account details to operational logs. Repair verifyClerkToken and validateAuth so successful and rejected verification, verified-account lookup, legacy website-token validation and provider exceptions do not log raw credentials, account IDs, email addresses, secret keys or upstream error details. Generic diagnostics or silence are both acceptable. Preserve actual token verification with the supplied verifier key, successful verified and legacy paid/free account tiers and canonical identity, Bearer and Token support, and anonymous access with the supplied device identity when credentials are missing, invalid, rejected or unavailable. Missing credentials must not trigger verification or account lookup. Use synthetic credentials only. This task does not require preserving public-identifier authentication or any particular log message or implementation structure.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/gateway-auth-log-privacy.test.js",
"destination_path": "evals/coding-agent/graders/gateway-auth-log-privacy.test.js"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/gateway-auth-log-privacy.test.js"
}
},
{
"id": "app-search-focus-lifecycle",
"name": "Suspend hidden search focus work and preserve reopening",
"source": {
"kind": "git_regression",
"fix_commit": "27fef88bae7c56bb6bce58e98d8b97c28d7f7aac"
},
"base_ref": "073c54cba78b42719b909733fe151083baff3fed",
"oracle_ref": "27fef88bae7c56bb6bce58e98d8b97c28d7f7aac",
"oracle_paths": [
"apps/screenpipe-app-tauri/app/search/page.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/app/search/page.tsx",
"apps/screenpipe-app-tauri/components/rewind/hooks/use-search-focus.ts",
"apps/screenpipe-app-tauri/lib/hooks/use-event-listener.ts"
],
"tags": [
"regression",
"escaped-failure",
"search",
"lifecycle",
"focus"
],
"prompt": "The retained search webview keeps reclaiming input focus while hidden. Fix the React search-page lifecycle: a page opened at /search?prewarm=1 must remain inactive until the native search-reset event; search-hidden must suspend focus work, even before the first show. An ordinary /search open must still focus the input and recover focus from a non-input element while visible. Reopening after hiding must restore usable focus and the supplied query, including Unicode and URL-special characters; a null or absent reset query clears the old query. Preserve result navigation with its timestamp, frame ID and search context, explicit native window-close handoff, and removal of listeners and focus work on unmount. Preserve public component/hook interfaces and any intentional recent-chat cache warming. This task covers the actual page and focus hook with synthetic native events and a small search-view shell, not native window-server execution or the full search-results UI. No particular mount strategy or private helper is prescribed.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/search-focus-lifecycle.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-search-lifecycle.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts eval-search-lifecycle.test.tsx"
}
},
{
"id": "app-mixed-note-paste",
"name": "Preserve text when pasting an image into meeting notes",
"source": {
"kind": "git_regression",
"fix_commit": "7c4e89fb03d9e39f680a440bef37152f9207fd2e"
},
"base_ref": "46fe476a01841e3b20b4ded99d87d6abd20dcfbc",
"oracle_ref": "7c4e89fb03d9e39f680a440bef37152f9207fd2e",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/meeting-notes/note-editor.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/meeting-notes/note-editor.tsx",
"apps/screenpipe-app-tauri/components/meeting-notes/image-utils.ts"
],
"tags": [
"regression",
"escaped-failure",
"meeting-notes",
"clipboard",
"data-loss"
],
"prompt": "Pasting a clipboard image together with readable text into a meeting note loses the text. Fix the actual editor paste path so the resulting document and emitted markdown retain both the image and the accompanying text. Preserve ordinary text paste, image-only file paste, and HTML-only image paste without duplicating the image URL as visible text. If the meeting changes while an image is being converted, the old paste must not insert into the newly keyed note. Preserve public editor interfaces. Checks mount the real React and TipTap editor with synthetic image conversion, inert menus and localization; native clipboard delivery, image encoding and disk autosave are outside this task. Any implementation satisfying these outcomes is accepted.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/mixed-note-paste.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-mixed-note-paste.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config vitest.config.ts eval-mixed-note-paste.test.tsx"
}
},
{
"id": "app-chat-repeat-prompt-identity",
"name": "Keep independent same-prompt chats visible in the sidebar",
"source": {
"kind": "git_regression",
"fix_commit": "c2051b6d02bed075e3acc97a9dc6d89c55b58db3"
},
"base_ref": "4c2f43f5c7b519dab72564a66541e4078a38963f",
"oracle_ref": "c2051b6d02bed075e3acc97a9dc6d89c55b58db3",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat-dedup.ts",
"apps/screenpipe-app-tauri/lib/stores/chat-store.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/chat-dedup.ts",
"apps/screenpipe-app-tauri/lib/stores/chat-store.ts"
],
"tags": [
"regression",
"escaped-failure",
"chat",
"sidebar",
"identity"
],
"prompt": "Starting two chats with the same opening prompt can make one disappear from the sidebar. Repair the existing in-memory sidebar selection behavior. Independent sends must remain separate, including messages with different IDs at the same timestamp or the same ID at different timestamps. Preserve legacy conversations when a first user message lacks its ID or timestamp. Actual cross-window copies of the same first user message should still collapse within the existing thirty-minute creation window, including copies whose text carries connections context. Keep the copy with a completed reply instead of a Processing... placeholder; a visible copy must survive a hidden twin. Preserve distinct text, pipe runs, branches and conversations outside the creation window. Return the surviving records intact without mutating the input records or deleting stored conversations. Preserve the existing public selector interface. This task concerns in-memory sidebar selection; disk history/search and native UI transport are outside scope.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/chat-repeat-identity.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-chat-repeat-identity.test.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && \"${BUN_BIN:-bun}\" --no-env-file test eval-chat-repeat-identity.test.ts"
}
},
{
"id": "app-ocr-derived-copy-retry",
"name": "Redact persisted OCR words and retain retry after failed writes",
"source": {
"kind": "git_regression",
"fix_commit": "e5600a3376e55806fd70ac406f2f5e353358fdc4",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/4631"
},
"base_ref": "a2e58a5c9d323219dee86d8623221065c120d19d",
"oracle_ref": "e5600a3376e55806fd70ac406f2f5e353358fdc4",
"oracle_paths": [
"crates/screenpipe-redact/src/lib.rs",
"crates/screenpipe-redact/src/ocr_json.rs",
"crates/screenpipe-redact/src/worker/columns.rs",
"crates/screenpipe-redact/src/worker/mod.rs",
"crates/screenpipe-redact/src/worker/tables.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-redact/src/**"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"ocr",
"durable-retry"
],
"prompt": "The background redaction worker can finish a frame while the same sensitive text survives in its persisted per-word OCR JSON. Fix the actual worker so single-token sensitive values are scrubbed from both full_text and text_json, for map-producing and spanless redactors. Keep the OCR blocks, their order, all geometry and metadata, clean words and surrounding non-sensitive full text intact; replace sensitive words without discarding their blocks or emptying the word. OCR redaction must still occur when optional window-title and browser-URL redaction is disabled, and those opted-out fields must stay unchanged. Clean frames must finish processing without text or geometry changes. If SQLite rejects an OCR update, preserve the original full-text retry source and leave its completion marker unset. After that write failure clears and the worker is restarted on the same database, finish redaction successfully. Use the existing public Worker and redactor interfaces; no particular helper, replacement label, JSON formatting, detection count or write strategy is required. The task concerns background reconciliation of existing rows with the full-text, OCR and per-copy completion columns available. It does not require old-row backfill, whitespace-split entity reconstruction, a schema migration or capture-path changes. Preserve recording authorization and pause controls; do not block or discard captures to satisfy redaction work. Do not contact live model services.",
"agent_timeout_seconds": 1800,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/ocr-derived-copy.rs",
"destination_path": "crates/screenpipe-redact/tests/eval_ocr_derived.rs"
}
],
"command": "cargo test --locked -p screenpipe-redact --test eval_ocr_derived -- --test-threads=1"
}
},
{
"id": "app-enterprise-local-policy-waits",
"name": "Finish enterprise verification despite stalled local policy work",
"source": {
"kind": "git_regression",
"fix_commit": "6b48cdbc6bd445f6ce0389aa00fd40c642872f65",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7435"
},
"base_ref": "959f7e46559620b9df4666553379d26d21da8760",
"oracle_ref": "6b48cdbc6bd445f6ce0389aa00fd40c642872f65",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-enterprise-policy.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-enterprise-policy.ts"
],
"tags": [
"regression",
"escaped-failure",
"enterprise",
"auth",
"deadline",
"preserved-behavior"
],
"prompt": "Enterprise startup can stay on checking access forever when local policy work stalls, even though the service can verify the saved license key. Ensure verification finishes within 10 seconds when one store-open, settings-read, AI-preset write/save, install-metadata lookup, or app-update settings-save operation never settles and the other operations respond promptly. Continue real policy verification and native authorization before reporting success. Keep key requests free of account authorization headers. Ignore install metadata that arrives after its wait has ended so it cannot overwrite settings or a newly selected account. Preserve credential choice with no credentials, denied and expired credentials even after a local read stall, account-login requirements, administrator recording pauses, and native authorization failure. Healthy policy application must still persist update policy and timely install metadata while retaining allowed employee presets, account identity, and unrelated settings.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/enterprise-policy-waits.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/lib/hooks/__tests__/eval-enterprise-policy-waits.test.tsx"
}
],
"command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts lib/hooks/__tests__/eval-enterprise-policy-waits.test.tsx"
}
},
{
"id": "app-learning-save-readback",
"name": "Verify the saved skill before reporting learning success",
"source": {
"kind": "git_regression",
"fix_commit": "ebfae02d4464e87983c1516bb1f4e95a40197397",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7434"
},
"base_ref": "fffff2a42ef5584969e7dd680b54bafde59803c4",
"oracle_ref": "ebfae02d4464e87983c1516bb1f4e95a40197397",
"oracle_paths": [
"crates/screenpipe-core/assets/extensions/skill-learning.ts"
],
"tags": [
"regression",
"escaped-failure",
"local-skills",
"write-verification",
"preserved-behavior"
],
"prompt": "The opted-in learning task can report a successful skill save even when the service stored a different method or identity. Repair the active learning extension so a save succeeds only when read-back has matching nonempty write/read hashes and the requested key, name, agent origin, description and instructions. Accept the storage service trimming surrounding whitespace. A mismatched result or failed write must report an error, keep a durable pending receipt, leave consumed evidence, ownership and change count unchanged, and avoid a success report or another write in that run. A pending receipt must also block a save after restart. Preserve successful creation and updating of an owned skill, the existing optimistic concurrency hash, evidence consumption, ownership, one-change limit and the review report including the previous method for updates. Use synthetic activity and service replies; no real recordings or private data are needed.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/learning-save-readback.test.ts",
"destination_path": "evals/coding-agent/graders/learning-save-readback.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/learning-save-readback.test.ts"
}
},
{
"id": "app-active-recording-storage-volume",
"name": "Check low storage on the active recording volume",
"source": {
"kind": "git_regression",
"fix_commit": "8e51453598d2587ac57828e1238d3cec0a92f05c",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7460"
},
"base_ref": "b93f07684b845a3baaedafc288d18d79b642fbf0",
"oracle_ref": "8e51453598d2587ac57828e1238d3cec0a92f05c",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-recording-storage.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-recording-storage.ts"
],
"tags": [
"regression",
"escaped-failure",
"recording",
"storage",
"volume"
],
"prompt": "Paused recording can show a low-storage warning for the saved settings directory while the running engine uses another volume. Fix the existing useRecordingStorage hook so its warning reflects the currently active recording directory, including after manual refresh. A healthy saved volume must not hide low space on the active volume, and a low saved volume must not block a healthy active one. If the active directory or disk probe is unavailable, report an error with no warning instead of treating failure as zero free space or using a different volume. Preserve the hook interface, native reserve and inclusive low-space boundary, invalid-native-value errors, manual recovery, and checks only while paused with the low-disk guard enabled. Keep checking state accurate and prevent late old-directory results or resumed-recording results from restoring stale warnings. This task concerns React behavior with controlled native ports, not filesystem operations or native recording policy.",
"agent_timeout_seconds": 1200,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/active-storage-volume.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-active-storage-volume.test.tsx"
},
{
"local_path": "graders/active-storage-volume.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-active-storage-volume.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-active-storage-volume.config.mjs"
}
},
{
"id": "ai-gateway-wif-token-retry",
"name": "Recover transient WIF authentication failures",
"source": {
"kind": "git_regression",
"fix_commit": "a82653675fcdaca022c88b7c3599bac3e3542d89"
},
"base_ref": "b6e2cb2249d75d60a92c85d1f02938fe840e52cf",
"oracle_ref": "a82653675fcdaca022c88b7c3599bac3e3542d89",
"oracle_paths": [
"packages/ai-gateway/src/providers/vertex.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/vertex.ts"
],
"tags": [
"regression",
"escaped-failure",
"authentication",
"retry"
],
"prompt": "The public VertexAIProvider.getAccessToken method fails an entire request when either Workload Identity Federation token exchange encounters a temporary outage. Repair the actual STS exchange and service-account impersonation paths so 522/other 5xx, 429 and thrown transport errors can recover on the third attempt after two transient failures. Bound each exchange to three attempts; persistent failures must reject instead of returning or caching a token. Permanent 400/403 responses must reject immediately without retry, and failed STS must not reach impersonation. Preserve the original POST exchange data, signed JWT issuer/audience/subject, selected service-account URL and STS bearer credential during retry. Successful WIF tokens must be returned and reused from the existing valid-token cache without new requests. Preserve successful legacy service-account authentication and its cached token. Keep the existing public class, constructor and getAccessToken contract. No particular helper or retry implementation is required; exact delay values are not prescribed. Use synthetic credentials and intercepted transport only. Real provider availability, gateway HTTP admission, concurrent or cross-account cache isolation, cancellation, expiry-boundary policy and live model calls are outside this historical task.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/wif-token-retry.fixture.ts.txt",
"destination_path": "packages/ai-gateway/eval-wif-token-retry.test.ts"
}
],
"command": "bun --no-env-file test packages/ai-gateway/eval-wif-token-retry.test.ts"
}
},
{
"id": "app-pipe-run-terminal-preservation",
"name": "Keep complete pipe transcripts across intermediate turns",
"source": {
"kind": "git_regression",
"fix_commit": "4f58c512232267cc66eb5bf76846107afff11931"
},
"base_ref": "523615534af1bc9174d777ba917a3603354a164f",
"oracle_ref": "4f58c512232267cc66eb5bf76846107afff11931",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/events/pipe-run-recorder.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/events/pipe-run-recorder.ts",
"apps/screenpipe-app-tauri/lib/pipe-ndjson-to-chat.ts"
],
"tags": [
"regression",
"escaped-failure",
"data-preservation",
"pipe-history"
],
"prompt": "The pipe-run recorder can save an incomplete conversation between model turns, losing later output. Repair the actual mounted recorder so intermediate turn_end events do not persist an incomplete run. When agent_end or pipe_done finishes a run, save one complete conversation containing its original user prompt and all streamed assistant turns. Preserve ordinary single-turn completion, the execution identity in saved pipeContext, rejection of non-pipe events, disabled chat-history settings, and the foreground pipe-watch writer's ownership. Repeated terminal events must not overwrite the saved conversation. Keep the public mount/unmount lifecycle and normal event-envelope contract; no particular terminal guard or internal helper is required. Use synthetic events and intercepted storage/settings/bus ports. Native delivery, actual disk durability, crash recovery, continued-chat merging, and presentation labels are outside this historical task.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/pipe-run-terminal.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-pipe-run-terminal.test.ts"
}
],
"command": "bun --no-env-file test apps/screenpipe-app-tauri/eval-pipe-run-terminal.test.ts"
}
},
{
"id": "app-compressed-pending-sqlx-upgrades",
"name": "Upgrade existing compressed databases without losing history",
"source": {
"kind": "git_regression",
"fix_commit": "9d7d2cd3bd5cbb8b29d40a1b74af4cd606244d49"
},
"base_ref": "9123d79bd8fa117174c3a256f92873bc4b785e1b",
"oracle_ref": "9d7d2cd3bd5cbb8b29d40a1b74af4cd606244d49",
"oracle_paths": [
"crates/screenpipe-db/src/db/setup.rs",
"crates/screenpipe-db/src/storage/schema.rs",
"crates/screenpipe-db/src/storage/read_schema.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-db/src/db/setup.rs",
"crates/screenpipe-db/src/storage/",
"crates/screenpipe-db/src/migrations/"
],
"tags": [
"regression",
"escaped-failure",
"migration",
"data-preservation",
"privacy"
],
"prompt": "Existing compressed databases created before starred sessions were introduced reopen without applying the pending application schema migration. Make normal startup apply pending feature migrations and record successful completion once, preserving older recordings, exact sealed frame payloads, search results, existing starred sessions and their revisions. Star listing, timestamp membership (including empty input), saving and search ranges must work after restart; new captures must remain writable and searchable. Already-applied migrations must preserve existing data. Resident feature tables introduced after conversion must receive storage revision tracking for insert/update/delete and invalidate previously acquired read-snapshot tokens on deletion or redaction while preserving their admission across ordinary edits. Repair missing tracking hooks on existing resident tables without duplicating their effects across repeated startup. Preserve bulk storage behavior. A failed multi-statement feature migration must roll back its partial schema and completion record; after the conflict is removed, normal startup must retry successfully while preserving recorded history. Ordinary SQLite starred sessions and already-current compressed histories must retain their behavior. Exercise actual disposable databases and public storage methods; no particular Rust helper or implementation is required. This concerns upgrades of already-converted synthetic data. It does not authorize migration rollout, broadening recording or migration permissions, altering pause/cohort/stop controls, deleting retained recovery copies, contacting live services or publishing releases.",
"agent_timeout_seconds": 1800,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/compressed-sqlx-upgrade.rs",
"destination_path": "crates/screenpipe-db/tests/eval_compressed_sqlx_upgrade.rs"
}
],
"command": "cargo test --offline --locked -p screenpipe-db --features storage-fault-injection --test eval_compressed_sqlx_upgrade -- --test-threads=1"
}
},
{
"id": "app-cloud-auth-host-boundary",
"name": "Preserve sessions across local and unrelated HTTP authorization failures",
"source": {
"kind": "git_regression",
"fix_commit": "aad18c86ab9ca120840b8048f025b7ed7c7839ab"
},
"base_ref": "84e6faba1affb73707239f217a7c379e4a41c014",
"oracle_ref": "aad18c86ab9ca120840b8048f025b7ed7c7839ab",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/auth-guard.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/auth-guard.tsx"
],
"tags": [
"regression",
"escaped-failure",
"auth",
"session-preservation"
],
"prompt": "The desktop fetch interceptor can clear a valid cloud session when a local connection returns 401 and its URL happens to contain a Screenpipe domain. Preserve the cloud session for localhost, IPv4 and IPv6 loopback responses, and for unrelated hosts mentioning screenpi.pe or screenpipe.com in their query or a misleading hostname. Keep session expiry for a signed-in request returning 401 from the genuine screenpipe.com or screenpi.pe website. Successful responses and requests without a signed-in token must not clear a session. Preserve fetch arguments and the original response object, and propagate network errors without clearing the session. Support string URLs, URL objects and Request objects.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 45,
"fixtures": [
{
"local_path": "graders/auth-host.fixture.tsx.txt",
"destination_path": "auth-host.eval.test.tsx"
}
],
"command": "bun --no-env-file test auth-host.eval.test.tsx"
}
},
{
"id": "app-unmarked-html-file-preview",
"name": "Render unmarked HTML files without losing source or ordinary previews",
"source": {
"kind": "git_regression",
"fix_commit": "93c0de3c6b9b872056987ef70bf46c9af2d9d890"
},
"base_ref": "6ea84d80e5ab4041aa855014ab71542a9d353206",
"oracle_ref": "93c0de3c6b9b872056987ef70bf46c9af2d9d890",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/file-viewer.tsx",
"apps/screenpipe-app-tauri/lib/utils/html-sandbox.ts"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/components/file-viewer.tsx",
"apps/screenpipe-app-tauri/components/file-viewer-html-frame.tsx",
"apps/screenpipe-app-tauri/lib/utils/html-sandbox.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"html",
"preserved-behavior"
],
"prompt": "An HTML report without a producer marker is only shown as source in the file viewer. Repair the actual ViewerFileContent preview so complete unmarked HTML documents open rendered, including upper-case .HTM names and Unicode content. Nonempty bare snippets must offer rendering too; their initial tab is unspecified. Preserve rendering for marked snippets. Keep meaningful document content inside a sandboxed iframe, with no payload style/script/heading elements inserted into the host. Grant at most script execution in the sandbox, and retain actual default-deny CSP metadata for default sources, connections and form submissions. Commented-out content or policy text is not a rendered document or policy. The source/render switch must show the unchanged source and return to the preview. Keep a chosen source tab when the same path is rerendered; a new path must clear old content while loading and display its own document afterward. Truncated, empty and loading HTML must not render or offer a render toggle. Preserve literal plain-text previews and ordinary Markdown headings, bold text, lists and safe web links. Keep the public component and existing source/render control contracts compatible. No specific internal helper or default-selection implementation is required. This task checks actual React/Markdown DOM output and iframe configuration with synthetic native/media ports; it does not establish browser network or IPC enforcement, Brain-view behavior or native security.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/html-viewer.fixture.ts.txt",
"destination_path": "evals/coding-agent/graders/html-viewer.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/html-viewer.test.ts"
}
},
{
"id": "app-transcription-nova-language",
"name": "Preserve language selection through cloud transcription",
"source": {
"kind": "git_regression",
"fix_commit": "a52a6c640457308fec92daa270f423b79ce7a34a"
},
"base_ref": "066291bc4ccdd986e3a29e339c48b610d8fd2a2f",
"oracle_ref": "a52a6c640457308fec92daa270f423b79ce7a34a",
"oracle_paths": [
"packages/ai-gateway/src/services/transcription-ab.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/handlers/transcription.ts",
"packages/ai-gateway/src/services/transcription-ab.ts"
],
"tags": [
"regression",
"escaped-failure",
"transcription",
"language",
"gateway"
],
"prompt": "Cloud batch transcription loses the selected language and returns English even when another language was requested. Repair the actual handleFileTranscription request path so the outgoing Nova-3 request uses a language parameter: a single selected code, trimmed and lowercased, or multi when no language or multiple languages are selected. Do not send the obsolete detect_language parameter to the provider. Preserve parsing of language/languages query values, repeated or comma-separated detect_language query codes, and legacy language/detect_language headers. Query codes take precedence over headers; boolean detection flags are not codes and must allow a usable legacy header fallback. Empty or auto-detection-only requests use multi. Preserve the default Deepgram POST endpoint, Nova-3 model, explicit sample rate, original audio bytes, audio content type and provider credential. With alternate-provider traffic disabled, make one provider call for a successful request or a permanent 400 response. Pass through the successful transcript payload unchanged, including an empty transcript; preserve HTTP 500 with the provider's error detail for permanent rejection. Use the existing public handler contract. No particular helper, export, or implementation structure is required. Use synthetic audio and intercepted provider transport only. Actual speech recognition quality, native language settings persistence, alternate providers, real credentials, deployed gateway admission, concurrent requests and infrastructure isolation are outside this task.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/transcription-language.fixture.ts.txt",
"destination_path": "packages/ai-gateway/eval-transcription-language.test.ts"
}
],
"command": "bun --no-env-file test packages/ai-gateway/eval-transcription-language.test.ts"
}
},
{
"id": "app-manual-sync-http-failure",
"name": "Report manual sync HTTP failures without false success",
"source": {
"kind": "git_regression",
"fix_commit": "ac201302bffe38e978efe574fc7033436ae96939"
},
"base_ref": "b2ff035b626e7a4a5e117fb1013aa1501d689300",
"oracle_ref": "ac201302bffe38e978efe574fc7033436ae96939",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/settings/account-section.tsx",
"apps/screenpipe-app-tauri/lib/sync-fetch.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/settings/account-section.tsx",
"apps/screenpipe-app-tauri/lib/sync-fetch.ts"
],
"tags": [
"regression",
"escaped-failure",
"sync",
"error-feedback"
],
"prompt": "The account settings manual sync actions can report success when the local server rejects a request. For each enabled pipe/scheduled-task, memory and connection sync action, POST pull before push. A failed pull must stop before push; a failed push must report failure after the successful pull. On HTTP errors, show one destructive failure notification containing the nonempty server error message, or the HTTP status when the body is not JSON. Do not show success on rejection. After a failure, restore the enabled manual action and allow a later successful retry. Preserve one success notification after successful pull and push, with a title indicating synced. Preserve network-error feedback explaining that the server is unreachable and restore the action afterward. Disabled sync choices must expose no manual action and issue no sync request just from opening settings. Keep the existing public AccountSection interface, recognizable sync section labels and Sync now actions. Do not require any particular helper, layout, DOM order or exact notification title. Tests mount the actual component with synthetic account settings, local HTTP responses and inert native/translation ports. This task does not establish native sync persistence, encryption, real account access or backend delivery.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/manual-sync.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-manual-sync.test.tsx"
},
{
"local_path": "graders/manual-sync.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-manual-sync.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-manual-sync.config.mjs"
}
},
{
"id": "app-native-calendar-unavailable-poll",
"name": "Keep unavailable native calendars disconnected without repeated HTTP probes",
"source": {
"kind": "git_regression",
"fix_commit": "f5b1f3d29246bdcb0bdb3f156721b80c71f367e5"
},
"base_ref": "166bdc1dd78a7767ceabe20495c2faad0e7ef866",
"oracle_ref": "f5b1f3d29246bdcb0bdb3f156721b80c71f367e5",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/utils/calendar.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/utils/calendar.ts"
],
"tags": [
"regression",
"escaped-failure",
"calendar",
"provider-availability"
],
"prompt": "The upcoming-calendar frontend poller repeatedly requests native events on machines whose native calendar is known to be unavailable, and can present an unavailable provider as connected. Keep the public fetchUpcomingCalendarSnapshot API. Across repeated polls, a successful native status reporting unavailable must produce no native events request and no native connection or failure entry. If status is unknown and the HTTP response explicitly reports connected:false with empty data, keep native disconnected and without a failed-source entry. Preserve HTTP recovery when native status cannot be obtained or returns an error, and when the calendar is available but its authorization status is stale: a successful event response must still appear as connected, healthy native events. Forward caller hoursBack and hoursAhead to native HTTP queries. An authorized native calendar with a successful empty response stays connected and healthy; its HTTP failure stays connected and failed. Preserve disconnected Google accounts making no events request, both connected Google accounts contributing their events, and enabled ICS events while native is unavailable. Preserve event titles and provider source labels. Tests execute the actual frontend module with synthetic HTTP and native-command ports. Native HTTP implementation, OS permissions, OAuth token refresh and publisher retry timing are outside this task.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/calendar-unavailable.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-calendar-unavailable.test.ts"
},
{
"local_path": "graders/calendar-unavailable.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-calendar-unavailable.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-calendar-unavailable.config.mjs"
}
},
{
"id": "app-meeting-stop-save-refusal",
"name": "Preserve explicit meeting Stop when saving the current note fails",
"source": {
"kind": "git_regression",
"fix_commit": "f3cd3f8b8d462a8df686b236e75a702db8cad033"
},
"base_ref": "c89551d84a2f2c8065aa0e08dc706ef467643f74",
"oracle_ref": "f3cd3f8b8d462a8df686b236e75a702db8cad033",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/meeting-notes/note-view.tsx"
],
"tags": [
"regression",
"meetings",
"user-control",
"persistence",
"escaped-failure"
],
"gate": "advisory",
"prompt": "The live meeting note view prevents Stop when saving the latest draft fails. Repair the actual NoteView Stop action so a rejected pre-stop save warns the user and still invokes the supplied Stop callback exactly once, without reporting that the note was saved. Preserve successful-save ordering: wait for an in-flight save of the current draft before stopping. An unchanged note must stop without an unnecessary save. Preserve ordinary autosave and ensure an older in-flight save cannot overwrite a newer draft after overlapping edits, including the visible note and persisted request data. Keep the public component interfaces compatible. This task checks the real React component with synthetic editor-input, HTTP and native ports; actual editor internals, native meeting-stop/deferral release, database durability and global recording pause are outside scope. No particular queue implementation, callback name or warning copy is required.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/meeting-stop.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/note-save-probe.test.tsx"
},
{
"local_path": "graders/meeting-stop.config.ts",
"destination_path": "apps/screenpipe-app-tauri/note-save.config.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config note-save.config.ts"
},
"trigger_paths": [
"apps/screenpipe-app-tauri/components/meeting-notes/note-view.tsx",
"apps/screenpipe-app-tauri/components/meeting-notes/note-save-queue.ts"
]
},
{
"id": "app-memory-tag-filter-malformed-json",
"name": "Keep memory tag filtering usable around malformed stored tags",
"source": {
"kind": "git_regression",
"fix_commit": "b3ce2f4d751950a85446ac7ca92366687efb4b3d"
},
"base_ref": "06b856d7395859a38b249eb2bf921524c148f921",
"oracle_ref": "b3ce2f4d751950a85446ac7ca92366687efb4b3d",
"oracle_paths": [
"crates/screenpipe-db/src/db/memories.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-db/src/db/memories.rs"
],
"tags": [
"regression",
"escaped-failure",
"database",
"search"
],
"prompt": "Memory listing and counting fail when an unrelated stored memory has empty, null or malformed JSON tags. Make exact tag-filtered reads tolerate those rows while returning and counting the correct valid memories. Requested tags use exact AND semantics; unknown tags match nothing. Preserve combined full-text search, source and minimum-importance filters, descending importance order and pagination. Empty tag filters must retain all memories, including rows with malformed tags. Preserve ordinary valid-tag behavior and do not delete or rewrite stored memories to make reads succeed. Exercise the actual database methods using disposable synthetic databases. No specific helper, query spelling or implementation is required. Do not access user databases or live services.",
"agent_timeout_seconds": 1800,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/memory-tag-filter.rs",
"destination_path": "crates/screenpipe-db/tests/eval_memory_tag_filter.rs"
}
],
"command": "cargo test --offline --locked -p screenpipe-db --test eval_memory_tag_filter -- --test-threads=1"
}
},
{
"id": "app-redact-missing-schema-target",
"name": "Keep redaction progressing across missing schema targets",
"source": {
"kind": "git_regression",
"fix_commit": "ac94fa0e26073ca9854a3695a308fb76d917172e",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/4315"
},
"base_ref": "84143f2402bb9d64331427233d3ab321798b7e19",
"oracle_ref": "ac94fa0e26073ca9854a3695a308fb76d917172e",
"oracle_paths": [
"crates/screenpipe-redact/src/worker/mod.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-redact/src/**"
],
"tags": [
"regression",
"escaped-failure",
"privacy",
"schema-compatibility"
],
"prompt": "The background redaction Worker stalls otherwise valid targets when an earlier target table or required column is absent from the database schema. Repair the actual worker so missing-schema targets do not delay redaction of unaffected queued rows or rows inserted during later sweeps. Tests use a small synthetic SQLite database, local regex redaction, a 20ms poll interval and a 1ms minimum batch idle. With a missing elements table or its completion column before full-text work, existing and newly added full-text rows must lose their synthetic email and gain a completion marker within 700ms; preserve the nearby contact word and leave clean text unchanged while marking it complete. Keep ordinary full-text redaction working. Pause before startup must prevent effects while paused, and resume must allow pending work within 700ms. A transient rejected update must retain its original retry text and unset completion marker; once the refusal is removed after 250ms, the same worker must successfully retry within another 2400ms. Preserve the existing database-corruption backoff: a SQLite error reporting a malformed database must prevent subsequent audio-target processing during the initial 700ms, while a shutdown notification must still stop the worker within one second. Do not classify every error as a missing object, drop all work or disable pause controls. No particular helper, disabled-target collection, log wording or replacement token is required. This task concerns background reconciliation of existing synthetic rows, not schema migration, real corruption recovery or capture admission. Preserve recording authorization and durable capture; do not block or discard captures to satisfy background work. Do not contact live model services.",
"agent_timeout_seconds": 1800,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/redaction-schema-progress.rs",
"destination_path": "crates/screenpipe-redact/tests/eval_missing_schema.rs"
}
],
"command": "cargo test --locked -p screenpipe-redact --test eval_missing_schema -- --test-threads=1"
}
},
{
"id": "app-audio-distinct-segment-search",
"name": "Preserve distinct speech segments in audio search",
"source": {
"kind": "git_regression",
"fix_commit": "acf844b85625df212436a7db8c68b370def720fe"
},
"base_ref": "6e9f48bfd69751428ec5c4b10483689845058e4c",
"oracle_ref": "acf844b85625df212436a7db8c68b370def720fe",
"oracle_paths": [
"crates/screenpipe-db/src/db/search.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-db/src/db/search.rs"
],
"tags": [
"regression",
"escaped-failure",
"audio",
"search",
"pagination"
],
"prompt": "Audio search collapses distinct recorded speech segments when they share an audio chunk, offset and capture timestamp. Repair the real database search path so every segment remains independently visible, including identical words in separate chunks with distinct segment start/end times. Preserve transcription text, segment boundaries and stored rows. Joining multiple chunk tags must not repeat result rows or tags. With capture timestamp, chunk and offset tied, ascending order follows insertion order and descending order reverses it; offset/limit pagination must visit each segment once. Preserve ordinary distinct-offset records, exact all-requested-tag filtering, absent-tag and unknown-device refusal, combined device and minimum/maximum text length filters, unmatched text search, date bounds, zero limit and exhausted offsets. Reads must not rewrite or delete stored transcriptions. Use disposable synthetic SQLite databases, never user recordings or live services. No particular SQL grouping, helper or query spelling is required. The scope is database search, not recording admission, HTTP delivery, hybrid storage or speech recognition quality.",
"agent_timeout_seconds": 1800,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/audio-segment-search.rs",
"destination_path": "crates/screenpipe-db/tests/eval_audio_segments.rs"
}
],
"command": "cargo test --offline --locked -p screenpipe-db --test eval_audio_segments -- --test-threads=1"
}
},
{
"id": "app-replay-unavailable-images",
"name": "Recover meeting replay from unavailable frame images",
"source": {
"kind": "git_regression",
"fix_commit": "5c1632dda6f5129f22f1b43b2ee250d12ec55f2f"
},
"base_ref": "335651f8047977e6f4c4819070abd0f991c6280e",
"oracle_ref": "5c1632dda6f5129f22f1b43b2ee250d12ec55f2f",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/meeting-notes/replay-strip.tsx"
],
"tags": [
"regression",
"meetings",
"images",
"recovery",
"escaped-failure"
],
"gate": "advisory",
"prompt": "Meeting replay keeps broken images visible when a sampled frame cannot render. Repair the actual ReplayStrip so a load error in either the active image or a thumbnail removes every displayed occurrence of that frame and excludes it from subsequent active-image selection, while keeping healthy images usable. Keep the active image accessible through nonempty alternative text and thumbnails decorative. When all samples are unavailable, show a visible no-screen-images-available message and no broken image elements. Keep the displayed number of available frames truthful. A new time range must retry frames that failed in the previous range. Preserve successful image loads, transcript captions, and the Open in timeline action, including the selected timestamp and /home?section=timeline destination, even without screen images. Keep existing public component interfaces. The checks render the actual React component with synthetic frame/transcript responses, native/navigation ports and image load/error events. Native image decoding, network delivery, audio playback and full browser layout are outside this task. No particular state structure or private helper name is required.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/replay-images.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/replay-probe.test.tsx"
},
{
"local_path": "graders/replay-images.config.ts",
"destination_path": "apps/screenpipe-app-tauri/replay-probe.config.ts"
},
{
"local_path": "graders/replay-images-i18n.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/replay-i18n.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config replay-probe.config.ts"
},
"trigger_paths": [
"apps/screenpipe-app-tauri/components/meeting-notes/replay-strip.tsx"
]
},
{
"id": "app-chat-saved-user-recency",
"name": "Recover saved chat recency from the latest user message",
"source": {
"kind": "git_regression",
"fix_commit": "fa7c1db8d6d9ca9e945aca1d522a37d2b5158f16"
},
"base_ref": "5d3a539aab4134b9dbaa7467a44c31d096644e00",
"oracle_ref": "fa7c1db8d6d9ca9e945aca1d522a37d2b5158f16",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat-storage.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"chat",
"recents",
"persistence"
],
"prompt": "Reloading saved chats can leave a conversation with an old Recents timestamp even though its saved transcript contains newer user messages. Fix the saved-chat listing API so lastUserMessageAt reflects the greatest numeric user-message timestamp or an even newer numeric persisted lastUserMessageAt. Transcript order must not matter. Ignore non-numeric persisted timestamps and non-numeric message timestamps; assistant messages must not advance user recency. When there are no numeric user-message timestamps, preserve a numeric persisted value, otherwise leave user recency absent. Legacy files without recency metadata must still derive it from user messages. Preserve conversation identity, title, createdAt, updatedAt, pinned state, message count and reloadable message contents. Keep the public listing and load APIs compatible. Checks call the actual APIs with synthetic filesystem/native ports. Native persistence, foreground/background save callbacks and sidebar rendering are outside this task; no private helper or specific implementation is required.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/recency.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/recency-probe.test.ts"
},
{
"local_path": "graders/recency.config.ts",
"destination_path": "apps/screenpipe-app-tauri/recency-probe.config.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config recency-probe.config.ts"
}
},
{
"id": "app-timeline-keyboard-filter-freshness",
"name": "Keep timeline arrow navigation on the current filter and state receiver",
"source": {
"kind": "git_regression",
"fix_commit": "a3f49b7c66bcc80d97d99abed54fca406db4fb91"
},
"base_ref": "f5d0c1550f8c37ed0cb034ba93248b2bb4116639",
"oracle_ref": "a3f49b7c66bcc80d97d99abed54fca406db4fb91",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/rewind/hooks/use-timeline-keyboard.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"timeline",
"keyboard",
"preserved-behavior"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/components/rewind/hooks/use-timeline-keyboard.ts"
],
"prompt": "Timeline arrow navigation can keep using the previous filter until new frames arrive. Fix the actual mounted keyboard hook so ArrowLeft and ArrowRight immediately use the current filter/navigation callback after a rerender, even when the frame array stays unchanged. Do not invoke the stale selection callback. If the index state receiver changes, the next key must reach the new receiver exactly once, not the old receiver. Keep the selected frame and index aligned with the current callback result. Preserve ordinary older/newer navigation, no-successor behavior, and keyboard-listener cleanup on unmount. Text inputs, textareas, editable content and an open search modal retain their arrow keys without moving the timeline. In search review, arrows navigate results and pause playback without ordinary frame navigation. While ordinary playback is running, arrow navigation seeks to the selected frame timestamp without pausing it. Public hook interfaces remain compatible; no dependency-list shape or private implementation is required. Checks mount the actual React hook and dispatch DOM keyboard events with synthetic navigation/native/translation ports. This task does not cover the separate live-capture prepend index repair, full timeline rendering, native event delivery, capture persistence or execution isolation.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/keyboard.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/keyboard-probe.test.tsx"
},
{
"local_path": "graders/keyboard.config.ts",
"destination_path": "apps/screenpipe-app-tauri/keyboard-probe.config.ts"
},
{
"local_path": "graders/keyboard-translation.ts",
"destination_path": "apps/screenpipe-app-tauri/keyboard-translation.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config keyboard-probe.config.ts"
}
},
{
"id": "app-media-file-url-path",
"name": "Preserve absolute file URL paths through the media player",
"source": {
"kind": "git_regression",
"fix_commit": "60d896613c64396e059c07ca03c5c8bb8ceade59"
},
"base_ref": "0d5a3e726a71f8c22c9162a0db1ea0a9bc86a806",
"oracle_ref": "60d896613c64396e059c07ca03c5c8bb8ceade59",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/utils/media-file-path.ts"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/utils/media-file-path.ts",
"apps/screenpipe-app-tauri/components/rewind/media.tsx",
"apps/screenpipe-app-tauri/lib/actions/video-actions.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"media",
"preserved-behavior"
],
"prompt": "The media player loses the beginning of absolute Unix file paths when it receives a file URL, so local recordings cannot be read. Repair the actual MediaComponent-to-native-media-reader flow. Unix file URLs with one, three or four slashes after file: must retain the full absolute path, including its leading slash, and decode percent-encoded spaces. Outer whitespace and surrounding quotes must not corrupt the path. Preserve ordinary absolute paths containing spaces, Windows drive file URLs with encoded spaces, and paths containing malformed percent escapes without dropping text or throwing. Each successful read must request the intended local path through the existing native media command contract and display the returned bytes with audio controls for WAV files and video controls for ordinary MP4 video. An empty path must not invoke native file reading. Keep the public component and native command contracts compatible. No particular helper or implementation is required. Synthetic native commands and a DOM environment assess player behavior; actual filesystem permission, native decoding, browser playback, Markdown source filtering, cache policy and isolation are separate.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/media-file-url.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/rewind/eval-media-file-url.test.tsx"
},
{
"local_path": "graders/media-file-url.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-media-file-url.config.mjs"
},
{
"local_path": "graders/media-file-url-translation.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-media-translation.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-media-file-url.config.mjs"
}
},
{
"id": "app-auth-logout-pending-load",
"name": "Keep sign-out authoritative over a pending account refresh",
"source": {
"kind": "git_regression",
"fix_commit": "b19f9c2ae6a55994f5b1057c11ed89c7522e9304"
},
"base_ref": "32a3db993ea7ac0dc71a7805f20f875ca1281567",
"oracle_ref": "b19f9c2ae6a55994f5b1057c11ed89c7522e9304",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"authentication",
"preserved-behavior"
],
"prompt": "A slow successful account refresh can restore a session after the user signs out. Repair the actual SettingsProvider account-loading flow. Once this window explicitly clears its user, a previously pending account response must not persist that user or reinstall its native cloud token. The same must hold when another window clears the shared user and delivers the existing screenpipe-auth-signout notification. Preserve successful fresh sign-in: save the returned identity and pass the supplied token to the existing native token setter. Preserve explicit verification requests, a fresh sign-in begun after completed sign-out, and successful pending refreshes across unrelated preference changes. Keep the existing provider, account-request and native command contracts compatible. No particular implementation is required. Grading mounts the provider with synthetic HTTP, native event/storage/token and translation ports; real multi-window event delivery, native credential durability, identity-provider revocation and execution isolation are separate.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/auth-logout.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-auth-logout.test.tsx"
},
{
"local_path": "graders/auth-logout.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-auth-logout.config.mjs"
},
{
"local_path": "graders/auth-logout-translation.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-auth-translation.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-auth-logout.config.mjs"
}
},
{
"id": "app-imported-agent-settings",
"name": "Persist portable agent settings when importing external chats",
"source": {
"kind": "git_regression",
"fix_commit": "0f7e2a87d51cde5950aa2160df6fc16774cc78f6"
},
"base_ref": "ae84199d25c4494ba1ac671a3adcddb7a7e413f5",
"oracle_ref": "0f7e2a87d51cde5950aa2160df6fc16774cc78f6",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat/external-chat-parser.ts",
"apps/screenpipe-app-tauri/lib/chat/external-chat-import.ts",
"apps/screenpipe-app-tauri/lib/hooks/use-settings.tsx"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/chat/external-chat-parser.ts",
"apps/screenpipe-app-tauri/lib/chat/external-chat-import.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"chat-import",
"preserved-behavior"
],
"prompt": "Imported external chats lose their portable agent settings before they can be restored. Repair the actual transcript import-and-save flow. For Codex, retain the latest recorded model and reasoning effort, accepting direct effort/reasoning_effort fields and collaboration-mode settings when direct values are absent. For Claude, retain the assistant model and permission mode; synthetic wrapper models and sidechain messages must not override the main conversation. Save these values using the existing importedFrom.config model/reasoningEffort/modeId contract. Do not copy local directories, environment values, sandbox rules or approval authority into that portable configuration. A settings-only change must persist even when automatic sync sees unchanged messages and timestamps. Preserve user-owned titles, selected presets, pins and local continuation messages. A transcript without configuration must not acquire invented settings; a completely unchanged record must avoid unnecessary writes; an unreadable input must report failure without saving. Keep the existing importer and conversation contracts compatible. No particular implementation is required. Grading invokes the importer and real parser with synthetic filesystem, persistence and event ports. Real transcript discovery, filesystem durability, composer selection, enterprise policy enforcement, provider execution and agent isolation are separate.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/imported-agent-settings.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-import-settings.test.ts"
},
{
"local_path": "graders/imported-agent-settings.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-import-settings.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-import-settings.config.mjs"
}
},
{
"id": "app-codex-embedded-image-import",
"name": "Preserve embedded image attachments during Codex transcript import",
"source": {
"kind": "git_regression",
"fix_commit": "a8ef9734ad2e852593b787c6272a2a11b85e12b4"
},
"base_ref": "21b59f57bd16e20196349287de7cbc2da99262eb",
"oracle_ref": "a8ef9734ad2e852593b787c6272a2a11b85e12b4",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/chat/external-chat-parser.ts",
"apps/screenpipe-app-tauri/lib/chat/external-chat-import.ts"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/chat/external-chat-parser.ts",
"apps/screenpipe-app-tauri/lib/chat/external-chat-import.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"chat-import",
"images",
"preserved-behavior"
],
"prompt": "Codex transcript imports lose embedded image attachments. Repair the actual import-and-save flow so accepted base64 raster image data reaches the existing saved-message images contract in original order, alongside user text. Retain image-only user messages. Automatic sync must persist missing or changed image bytes even if text, timestamps and image count are unchanged, while avoiding writes for fully unchanged text or image conversations. Matching trailing image transport envelopes must not appear in the saved user text or title; preserve quoted transport syntax as authored text. Reject remote image URLs, non-raster data such as HTML/SVG, and malformed base64 syntax from saved attachments without losing the accompanying text. Preserve chosen titles, presets, pins, local continuation messages and ordinary Claude text imports. Use the existing importer and conversation interfaces; no particular implementation is required. Grading invokes actual import-and-save code with synthetic filesystem, storage and event ports. Path-only attachments, renderer behavior, image decoding, real transcript discovery, native durability, execution isolation and provider access are separate.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/codex-images.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-codex-images.test.ts"
},
{
"local_path": "graders/codex-images.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/eval-codex-images.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-codex-images.config.mjs"
}
},
{
"id": "app-zero-channel-downmix",
"name": "Keep degenerate audio channel metadata from crashing sample conversion",
"source": {
"kind": "git_regression",
"fix_commit": "7c3b846ba35d6b90448b4d72b302c3fbb8f85bcc"
},
"base_ref": "e3cbb57f61c0805accedf97cf0c062880566ad68",
"oracle_ref": "7c3b846ba35d6b90448b4d72b302c3fbb8f85bcc",
"oracle_paths": [
"crates/screenpipe-audio/src/utils/audio/convert.rs"
],
"trigger_paths": [
"crates/screenpipe-audio/src/utils/audio/convert.rs"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"audio",
"preserved-behavior"
],
"prompt": "Audio conversion can panic when a device reports zero channels. Repair the existing interleaved-f32-to-mono converter so zero-channel metadata returns the original samples unchanged, including an empty buffer, without panicking. Preserve mono passthrough. For positive multichannel counts, average each frame across the declared channel count, preserve sample order and signed values, and retain real silence from cancellation. Preserve high-channel inputs and historical partial-frame behavior: emit the trailing partial frame using the full declared channel count as the denominator, even when the whole input is shorter than a frame. Empty buffers remain empty. Use the existing converter interface; no particular algorithm is required. Grading compiles the actual complete converter module with synthetic buffers. Device configuration rejection, capture callbacks, Windows COM, operating-system recovery, real recording and execution isolation are outside this narrow conversion task. Do not access recording hardware, user recordings or live services.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/zero-channel-downmix.fixture.rs",
"destination_path": ".eval-downmix/outcomes.rs"
},
{
"local_path": "graders/zero-channel-downmix.lib.rs",
"destination_path": ".eval-downmix/lib.rs"
},
{
"local_path": "graders/zero-channel-downmix.Cargo.toml.txt",
"destination_path": ".eval-downmix/Cargo.toml"
}
],
"command": "cargo test --offline --manifest-path .eval-downmix/Cargo.toml --test outcomes -- --nocapture"
}
},
{
"id": "app-mcp-search-source-evidence",
"name": "Preserve source evidence through HTTP MCP search",
"source": {
"kind": "git_regression",
"fix_commit": "fd797a4607f7def147171ac23a6c061cf4920d9f"
},
"base_ref": "0d082079f354e42ef8b51baafe4bc258b30cded2",
"oracle_ref": "fd797a4607f7def147171ac23a6c061cf4920d9f",
"oracle_paths": [
"packages/screenpipe-mcp/src/http-server.ts",
"packages/screenpipe-mcp/src/search-result.ts",
"packages/screenpipe-mcp/src/call-observation.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/screenpipe-mcp/src/**"
],
"tags": [
"regression",
"mcp",
"evidence",
"pagination"
],
"prompt": "HTTP MCP search_content loses the source references needed to inspect returned evidence and can discard input-event text. Repair the existing public tool. Screen and input results with valid backend frame IDs must retain frame_id=<id> and screenpipe://frame/<id>; audio results must retain chunk_id=<id> and transcription. Preserve zero and other nonnegative safe integer IDs, and never invent references for missing, negative or nonnumeric IDs. Keep source text in the response. Default long screen text must be bounded while preserving its beginning, end and source reference: for one 3008-character result, the complete response must stay below 1500 characters. An explicitly scoped max_content_length:0 read must preserve the full text. Preserve next-page offset hints when more results remain, but do not suggest another offset on a final nonempty page. Keep empty-result messages and upstream HTTP403 failures distinct. Exercise the actual HTTP MCP server with its SDK client and synthetic loopback backend. No private helper name, extraction or formatting outside these public references is required. This task does not establish stdio behavior, native telemetry, real customer retrieval, live authentication or client-disconnection cleanup. Do not call live services.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/screenpipe-mcp/node_modules",
"destination_path": "packages/screenpipe-mcp/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/mcp-search-evidence.fixture.ts.txt",
"destination_path": "packages/screenpipe-mcp/src/eval-evidence.test.ts"
}
],
"command": "cd packages/screenpipe-mcp && SCREENPIPE_MCP_TELEMETRY_DISABLED=1 DO_NOT_TRACK=1 node node_modules/vitest/vitest.mjs run src/eval-evidence.test.ts"
}
},
{
"id": "ai-gateway-null-model-accounting",
"name": "Preserve accounting fallbacks when the model name is absent",
"source": {
"kind": "git_regression",
"fix_commit": "982e1b2849b81966153cba1cd7d30e07a26ed25b"
},
"base_ref": "4b8326d778d7ea2ced2a80ea9cf92b63ba59795e",
"oracle_ref": "982e1b2849b81966153cba1cd7d30e07a26ed25b",
"oracle_paths": [
"packages/ai-gateway/src/services/cost-tracker.ts"
],
"gate": "advisory",
"tags": [
"regression",
"accounting",
"missing-input",
"preserved-behavior"
],
"prompt": "Gateway accounting crashes when the model name is absent. Repair the existing exported getModelCost, isZeroCostModel and inferProvider behavior for undefined and null model names: return the existing unknown-model estimate of 0.01 USD, false for free-model classification, and unknown for provider inference without throwing. The unknown estimate applies with either absent or measured token counts. Preserve the same behavior for empty, whitespace-only and unrecognized model strings. Preserve known-model accounting: the existing gpt-5.4-nano table uses 0.20 USD per million input tokens and 1.25 per million output tokens, gives 0.00045 for 1000 input/200 output, zero for zero tokens, and 0.001025 when both token counts are null. A lone null token count contributes zero when the other is measured. Preserve case-insensitive versioned-name matching and the longest matching price entry, OpenAI provider inference, and the existing screenpipe-event-classifier free/zero-cost/screenpipe-vllm behavior. These are repository pricing fixtures, not a claim about current external prices. Any implementation meeting the exported behavior is acceptable; do not require a specific private helper. The grader executes the complete accounting module without database or provider calls. This case does not establish HTTP admission, live billing, settlement durability, deployment, agent isolation or model capability. Add focused tests.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 30,
"fixtures": [
{
"local_path": "graders/null-model-accounting.fixture.ts.txt",
"destination_path": "packages/ai-gateway/src/test/eval-null-model-accounting.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test packages/ai-gateway/src/test/eval-null-model-accounting.test.ts"
}
},
{
"id": "app-manual-grant-payment-prompts",
"name": "Honor active fixed-term grants across trial and card eligibility",
"source": {
"kind": "git_regression",
"fix_commit": "b1c2736e8c0d6218be4d8053c84a3920912956cb"
},
"base_ref": "2670901fe575db4715830fdaa9e39512356d32c0",
"oracle_ref": "b1c2736e8c0d6218be4d8053c84a3920912956cb",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/app-entitlement.ts",
"apps/screenpipe-app-tauri/lib/card-ask/gating.ts",
"apps/screenpipe-app-tauri/lib/first-run/trial-activation.ts"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/app-entitlement.ts",
"apps/screenpipe-app-tauri/lib/card-ask/gating.ts",
"apps/screenpipe-app-tauri/lib/first-run/trial-activation.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"entitlement",
"preserved-behavior"
],
"prompt": "An account with verified, active fixed-term manual app access and no payment card is still sent through trial activation and contextual card requests. Repair the existing account eligibility flow so a consistent paid plan with fresh entitlement evidence, active app feature and a future subscription period end exempts the account from trial activation, new-trial enrollment and contextual card requests. Accept whitespace/case-normalized manual source values. Profile signup trials use expires_at rather than a subscription period end and must retain their ordinary trial/card eligibility. Do not exempt inactive, expired (including exactly at the deadline), invalid or absent period ends, stale checked evidence without grace, missing app features, non-manual or conflicting unpaid source evidence, inconsistent plan truth or explicit app denial. Preserve card-backed subscription and enterprise-account exemptions, signed-out refusal, settings-loading refusal and existing-install trial refusal. Keep existing public interfaces compatible; no particular helper or implementation is required. Grading exercises actual eligibility functions with synthetic accounts and a frozen clock, not rendered UI, native recording, live billing, provider authentication or execution isolation.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/manual-grant.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/lib/first-run/manual-grant.eval.test.ts"
},
{
"local_path": "graders/manual-grant.config.mjs",
"destination_path": "apps/screenpipe-app-tauri/manual-grant.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config manual-grant.config.mjs"
}
},
{
"id": "ai-gateway-vertex-string-tool-calls",
"name": "Preserve tool-call history across Vertex message content shapes",
"source": {
"kind": "git_regression",
"fix_commit": "85b0f37c6e7efbcdde83bfeea21b601f8da79ca7"
},
"base_ref": "9371579352e01a088bc153d287323d54601a954f",
"oracle_ref": "85b0f37c6e7efbcdde83bfeea21b601f8da79ca7",
"oracle_paths": [
"packages/ai-gateway/src/providers/vertex-maas.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/vertex-maas.ts"
],
"tags": [
"regression",
"escaped-failure",
"tool-calling",
"provider-compatibility"
],
"prompt": "Valid tool conversations sent through Vertex MaaS fail after an assistant response whose content is a string, even though it includes tool calls and the following tool outputs reference those calls. Preserve every valid call identifier, function name, serialized arguments and matching tool result in outbound JSON and streaming requests. Support empty, whitespace, null and nonempty assistant content; preserve nonempty text and accept either empty-string or null representation for absent text. Preserve array text with top-level calls, ordinary system/user/assistant text and image messages, request tools and tool choice, zero temperature and token limits, and returned response content. Preserve the existing model endpoint, POST method and bearer authentication header, and send one completion request on ordinary success. Tool outputs appearing before their matching call, or with missing, empty or unknown identifiers, must still be filtered when no missing-id call can be repaired. Preserve valid neighboring pairs and do not mutate the caller message array or nested calls. Exercise the existing public completion methods. No particular private helper or implementation is required. Use synthetic transport and credentials only; do not call a live model, change authentication policy or publish releases.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/vertex-tool-content.fixture.ts.txt",
"destination_path": "packages/ai-gateway/eval-vertex-tool-content.test.ts"
}
],
"command": "bun --no-env-file test packages/ai-gateway/eval-vertex-tool-content.test.ts"
}
},
{
"id": "app-mcp-session-lifecycle",
"name": "Preserve HTTP MCP sessions and explicit deletion",
"source": {
"kind": "git_regression",
"fix_commit": "5e0b1bdbe72f1628bd9b0a82acb63c3d79aba136"
},
"base_ref": "5aa887630cb04307c4734366b18b9d8ddccca45a",
"oracle_ref": "5e0b1bdbe72f1628bd9b0a82acb63c3d79aba136",
"oracle_paths": [
"packages/screenpipe-mcp/src/http-server.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/screenpipe-mcp/src/http-server.ts"
],
"tags": [
"regression",
"mcp",
"sessions",
"lifecycle"
],
"prompt": "HTTP MCP clients initialize successfully but cannot reliably reuse or delete their sessions. Repair the public HTTP session lifecycle. Initialization must return a nonempty session ID; repeated tools/list requests using it must succeed and include search_content. Independently initialized clients need distinct IDs and must remain usable independently. The unauthenticated health endpoint must report status ok and the accurate number of active initialized sessions, beginning at zero. An explicit DELETE of an initialized session succeeds, removes it from the active count and makes that ID unusable without disturbing other sessions. Closing every session returns the count to zero; a later initialization gets a fresh usable identity. Unknown or deleted session IDs must receive an HTTP error instead of tools, without entering the active count. Preserve unauthenticated loopback access, HTTP 404 for unknown routes and HTTP 204 for OPTIONS. Exercise synthetic loopback requests only; no actual screen recordings, credentials, live services or particular helper/storage implementation is required. This task checks public responses, active-session accounting and logical invalidation, not heap reclamation, dropped TCP connections, LAN authentication, stdio or concurrency guarantees.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/screenpipe-mcp/node_modules",
"destination_path": "packages/screenpipe-mcp/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/mcp-session-lifecycle.fixture.ts.txt",
"destination_path": "packages/screenpipe-mcp/src/eval-session.test.ts"
}
],
"command": "cd packages/screenpipe-mcp && SCREENPIPE_DISABLE_TELEMETRY=1 SCREENPIPE_MCP_TELEMETRY_DISABLED=1 DO_NOT_TRACK=1 node node_modules/vitest/vitest.mjs run src/eval-session.test.ts"
}
},
{
"id": "app-vault-lock-fresh-process-key",
"name": "Lock and unlock an offline vault from fresh CLI processes",
"source": {
"kind": "git_regression",
"fix_commit": "7d171cc2de988c92b0d38a940fecd8dfb27816b0"
},
"base_ref": "5dd6c7f0db9a38aa1ffbf71ae9bf5ef46f420a6d",
"oracle_ref": "7d171cc2de988c92b0d38a940fecd8dfb27816b0",
"oracle_paths": [
"crates/screenpipe-engine/src/cli/vault.rs",
"crates/screenpipe-engine/src/routes/vault.rs",
"crates/screenpipe-vault/src/lib.rs",
"crates/screenpipe-vault/src/manager.rs"
],
"trigger_paths": [
"crates/screenpipe-engine/src/cli/vault.rs",
"crates/screenpipe-vault"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"vault",
"data-integrity",
"preserved-behavior"
],
"prompt": "Offline vault setup succeeds, but a later CLI invocation cannot lock the existing vault. Repair the actual CLI flow so setup, lock and unlock can run in independent processes. Setup must leave database, WAL, SHM and nested media bytes unchanged and report unlocked. Lock with the correct password must succeed, report locked, encrypt all those files using the existing SPVAULT1-compatible format without leaving their plaintext content, and allow a fresh unlock process to restore identical original bytes and report unlocked. A wrong lock password must fail while preserving plaintext and unlocked state. A wrong unlock password and a second lock of an already-locked vault must fail without changing encrypted bytes or locked state. Lock before setup must fail without creating vault metadata or altering data. Repeated setup must fail without changing metadata or plaintext. Preserve command status behavior and compatible existing callers; no particular helper or key-storage implementation is required. Grading executes the actual CLI handler and complete vault crate in a minimal offline Cargo workspace, with synthetic filesystem paths and password input, separate processes, and an offline daemon port. It does not run the packaged engine, terminal input, HTTP routes, live recordings or customer files. Do not broaden online locking or migration policy; current daemon and hybrid-storage restrictions remain separate preserved contracts.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 240,
"fixtures": [
{
"local_path": "graders/vault-cli.fixture.js",
"destination_path": "vault-cli.eval.js"
}
],
"command": "bun --no-env-file vault-cli.eval.js"
}
},
{
"id": "app-mcp-package-http-dispatch",
"name": "Dispatch the published package HTTP flag while preserving stdio",
"source": {
"kind": "git_regression",
"fix_commit": "2c687b0b530623c750fdafda7bcc8985ee755a87"
},
"base_ref": "e4341ef7c9bc4ade2dfb5ea3bcb6a149bae28146",
"oracle_ref": "2c687b0b530623c750fdafda7bcc8985ee755a87",
"oracle_paths": [
"packages/screenpipe-mcp/package.json",
"packages/screenpipe-mcp/src/cli.ts",
"packages/screenpipe-mcp/src/http-server.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/screenpipe-mcp/package.json",
"packages/screenpipe-mcp/src/cli.ts",
"packages/screenpipe-mcp/src/index.ts",
"packages/screenpipe-mcp/src/http-server.ts"
],
"tags": [
"regression",
"mcp",
"cli",
"transport"
],
"prompt": "Invoking the screenpipe-mcp package with --http still starts its stdio transport. Repair the package entry point. After compiling its TypeScript, launching the screenpipe-mcp bin declared in package.json with --http either before or after --port must open HTTP on that requested loopback port. Preserve the separate screenpipe-mcp-http bin without requiring --http. Both HTTP entry points must return unauthenticated /health with status ok and zero initial sessions, HTTP 404 for unknown routes, and a successful MCP initialize response advertising tools. Without --http, including the unrelated --httpish flag, the primary bin must preserve newline JSON-RPC stdio initialization and tools/list containing search-content. HTTP requests with --port 0 or --listen-on-lan without --api-key must refuse startup with exit 2 and a diagnostic. Use synthetic loopback probes only, disable telemetry and supply a synthetic local API key with an empty home. No particular dispatch helper or source filename is required. This task covers compiled package bins, not registry installation, packaged installers, external API calls, HTTP session reuse, authorized LAN serving or deployed recordings.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "packages/screenpipe-mcp/node_modules",
"destination_path": "packages/screenpipe-mcp/node_modules"
}
],
"grader": {
"timeout_seconds": 90,
"fixtures": [
{
"local_path": "graders/mcp-package-dispatch.fixture.js.txt",
"destination_path": "packages/screenpipe-mcp/src/eval-package-dispatch.test.js"
}
],
"command": "cd packages/screenpipe-mcp && bun test src/eval-package-dispatch.test.js"
}
},
{
"id": "ai-gateway-outbound-chat-compatibility",
"name": "Preserve outbound chat compatibility and recover rejected sampling",
"source": {
"kind": "git_regression",
"fix_commit": "ee62f61129503c9400c845b9c25b88e37fc510ea"
},
"base_ref": "ece9d58bb90d88a4a232d0277bb74bbce11714ab",
"oracle_ref": "ee62f61129503c9400c845b9c25b88e37fc510ea",
"oracle_paths": [
"packages/ai-gateway/src/providers/openai.ts"
],
"gate": "advisory",
"tags": [
"regression",
"hosted-ai",
"tools",
"recovery"
],
"trigger_paths": [
"packages/ai-gateway/src/providers/openai.ts"
],
"prompt": "Chat requests can be rejected after conversation history changes or when an upstream model refuses an explicitly supplied temperature. Repair the existing OpenAIProvider public completion and streaming methods. Outbound messages must omit the response-only refusal field, remove tool outputs whose tool_call_id has no preceding assistant call, and preserve ordinary conversation text plus valid assistant calls and their matching tool output IDs/content. When an HTTP400 Unsupported value error names the supplied temperature, recover in both streaming and non-streaming calls by retrying without that parameter; retain messages and token limits and return the resulting answer. Do not retry HTTP500 failures or errors naming a parameter that was not sent. A repeated identical temperature refusal must terminate with the original upstream error after at most one retry. Preserve ordinary text and image_url content, including image detail. Keep the existing public interfaces compatible; no private helper, field, or implementation structure is required. Grading runs the actual provider with synthetic SDK and telemetry ports and no live network. This historical task does not establish HTTP gateway admission, other providers, billing, SDK transport behavior, production activation, agent isolation or model performance. Current support for repairing several different upstream rejections is a separate contract.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/outbound-chat.fixture.ts.txt",
"destination_path": "packages/ai-gateway/eval-outbound-chat.test.ts"
}
],
"command": "bun --no-env-file test packages/ai-gateway/eval-outbound-chat.test.ts"
}
},
{
"id": "app-meeting-background-recovery",
"name": "Keep recovered meeting speech beside partial live transcripts",
"source": {
"kind": "git_regression",
"fix_commit": "2b83241eee061b953edcbac3e66c70199bf5d8ca"
},
"base_ref": "ed2496416c20fc0b1d68c1355481036ddf79975f",
"oracle_ref": "2b83241eee061b953edcbac3e66c70199bf5d8ca",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/utils/meeting-context.ts"
],
"gate": "advisory",
"tags": [
"regression",
"meeting",
"transcript",
"recovery"
],
"trigger_paths": [
"apps/screenpipe-app-tauri/lib/utils/meeting-context.ts"
],
"prompt": "A partial routed meeting transcript hides speech recovered by background transcription. Repair the existing fetchMeetingAudio function so it returns both routed and recovered speech in chronological order, up to the requested positive cap. Remove identical text/time/device/speaker duplicates across sources while preserving distinct speech and separate input/output devices. Preserve background-only recovery when the routed request fails, retain available routed words when search fails, and return an empty list when both sources are unavailable. Filter malformed background rows and preserve valid chunk identity, audio path, timestamp, speaker and device metadata. Keep the public interface compatible; no particular helper, state layout or merge algorithm is required. Tests run the actual complete utility module with synthetic local API responses. This task does not establish native capture, live transcription delivery, database reconciliation, current cross-source speaker-label normalization, recording authority, browser rendering, agent isolation or model performance. Do not contact live services.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/meeting-background.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/lib/utils/meeting-background.eval.test.ts"
}
],
"command": "bun --no-env-file test apps/screenpipe-app-tauri/lib/utils/meeting-background.eval.test.ts"
}
},
{
"id": "ai-gateway-vertex-upstream-json",
"name": "Classify malformed Vertex success bodies as upstream failures",
"source": {
"kind": "git_regression",
"fix_commit": "b4563754cf43d182d4f57147a315817cf435efd6"
},
"base_ref": "d7952bd90b412705b1d5f6133ec75d8e61858214",
"oracle_ref": "b4563754cf43d182d4f57147a315817cf435efd6",
"oracle_paths": [
"packages/ai-gateway/src/providers/vertex-maas.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/vertex-maas.ts"
],
"tags": [
"regression",
"escaped-failure",
"provider-compatibility",
"error-classification"
],
"prompt": "The public Vertex MaaS JSON completion method leaks unclassified parse failures when the provider returns HTTP 200 with an empty, whitespace-only, truncated JSON or HTML response body. Report these as upstream errors carrying status 502 so callers can identify a bad gateway response. Preserve valid completion content and metadata, reasoning-only responses supplying visible content, ordinary upstream HTTP 400 refusal status and readable streaming completions. For the existing glm-4.7 endpoint, preserve POST and bearer-token authentication, user messages, zero temperature, max_tokens and the streaming flag, without mutating caller input. Each ordinary success or malformed-success response uses one completion request. Do not make valid requests fail or turn client refusals into 502 errors. Exercise the public completion methods with synthetic credentials and transport only; do not contact live providers. No particular parser, helper name, error class or error wording is required.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/vertex-upstream-json.fixture.ts.txt",
"destination_path": "packages/ai-gateway/eval-vertex-upstream-json.test.ts"
}
],
"command": "bun --no-env-file test packages/ai-gateway/eval-vertex-upstream-json.test.ts"
}
},
{
"id": "app-acp-declined-warmup",
"name": "Keep declined automatic agent warmups quiet across chat remounts",
"source": {
"kind": "git_regression",
"fix_commit": "6a443fb1ff9e0a11848f7dbddd923ec6748be87a"
},
"base_ref": "67f8fdf8222305796a9cafe31cdb74c289a8121d",
"oracle_ref": "6a443fb1ff9e0a11848f7dbddd923ec6748be87a",
"tags": [
"regression",
"escaped-failure",
"authentication",
"chat",
"react"
],
"prompt": "The chat view can repeatedly reopen an agent sign-in after the user declined an automatic warmup. Fix the existing useAcpWarmup hook so a pending or unsuccessful automatic start for the same preset and ACP agent stays quiet across new conversations and chat remounts within the same webview lifetime. Non-running replies, error replies and rejected starts all count as unsuccessful. A successful running handshake must still permit automatic warming for a later conversation. Failure for one preset/agent must not suppress another preset or agent. Preserve disabled-feature, hosted-provider, missing preset/agent, already-running and in-flight start guards. A process-liveness update must not release a handshake guard before its call completes. Release the guard on settlement or unmount; ignore late UI/config/thinking updates after unmount, and only update them after successful running handshakes. Preserve session, project directory, token and provider config passed to the start command, and successful state/config/thinking synchronization. The hidden checks mount the actual hook with synthetic native command responses; native sign-in UI, cross-webview persistence and manual send/retry paths are outside this case. Keep changes scoped to automatic warmup and add or update tests as needed.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 120,
"fixtures": [
{
"local_path": "graders/acp-declined-warmup.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-acp-warmup.test.ts"
},
{
"local_path": "graders/acp-warmup.config.mjs.txt",
"destination_path": "apps/screenpipe-app-tauri/eval-acp-warmup.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-acp-warmup.config.mjs"
},
"oracle_paths": [
"apps/screenpipe-app-tauri/components/chat/standalone/hooks/use-acp-warmup.ts"
],
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
]
},
{
"id": "app-meeting-speaker-source",
"name": "Keep per-stream speaker labels distinct in meeting transcript text",
"source": {
"kind": "git_regression",
"fix_commit": "13d960d27f7c4b95b1e35c23c76a45ea241d912b"
},
"base_ref": "b9cff1112aff37a2e461c620db4e03209c164b54",
"oracle_ref": "13d960d27f7c4b95b1e35c23c76a45ea241d912b",
"oracle_paths": [
"apps/screenpipe-app-tauri/lib/utils/meeting-context.ts"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"meetings",
"transcript",
"speaker-identity"
],
"prompt": "Meeting transcript text merges microphone and meeting-audio turns when independent streams reuse generic speaker labels. Repair the actual transcript renderer and the transcript section of full meeting Markdown. Generic labels speaker, speaker followed by a space and digits, or speaker followed by an underscore and digits are case-insensitive: qualify them as Microphone for input, Meeting audio for output, and Unidentified for an unknown direction. Preserve the original label alongside that source using the existing bracketed speaker format, for example [Microphone · speaker 1]. Adjacent generic turns from different sources must remain separate and retain all words. Preserve explicit names, including names that only resemble generic labels; an explicit matching name may still coalesce across sources. Keep same-source consecutive turns grouped below a ten-second gap, but separate turns at exactly ten seconds. Preserve chronological sorting, whitespace normalization, empty/null inputs, omission of blank speech and omission of the unknown speaker label. Never mutate input chunks or their stored identities. Full meeting Markdown must retain notes. The grader invokes the complete existing module with synthetic chunks and refuses local API work. Any implementation achieving these outputs is acceptable. This task covers rendered transcript text only, not live announcements, the transcript panel, native clipboard delivery, audio diarization, storage identity, execution isolation or model capability.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/meeting-speaker-source.fixture.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/lib/utils/meeting-speaker-source.eval.test.ts"
}
],
"command": "bun --no-env-file test apps/screenpipe-app-tauri/lib/utils/meeting-speaker-source.eval.test.ts"
}
},
{
"id": "app-historical-analytics-false-persistence",
"name": "Preserve explicit falsy preferences across historical settings reload",
"source": {
"kind": "git_regression",
"fix_commit": "648c89abeb770ab5a2630fa90d9fcd80e1bbc813"
},
"base_ref": "500c8fc3e658fd8045e19f35b814dc310b5c65a7",
"oracle_ref": "648c89abeb770ab5a2630fa90d9fcd80e1bbc813",
"oracle_paths": [
"screenpipe-app-tauri/lib/hooks/use-settings.tsx"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"privacy",
"settings",
"historical-contract"
],
"prompt": "In this historical app revision, loading saved settings replaces explicit false, zero and empty-string choices with defaults. Repair the actual useSettings hook while retaining its public interface. Loading must preserve false analyticsEnabled, enableFrameCache and isFirstTimeUser; zero fps and aiMaxContextChars; and empty aiModel and customPrompt. Accepted analyticsEnabled=false, fps=0 and customPrompt=\"\" updates must be visible immediately and survive unmount/remount from the synthetic store, preserving unrelated port and model values. Absent or null fields must retain this revision's defaults, including analytics enabled and macOS fps 0.2 and apple-native OCR. Loading explicit preferences, ordinary values, empty ignoredWindows or absent fields must not write the store. Ordinary values and empty ignoredWindows must be preserved. Custom dataDir must still be returned by getDataDir. Tests mount the real React hook with synthetic native storage, platform and path ports; no network or native builds are needed. The storage port verifies accepted values across hook reload, not physical disk durability or native process restart. These scalar fields and default rules belong to this historical revision; do not apply them to a later settings schema. No particular private helper or implementation path is required.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/settings-falsy.fixture.tsx.txt",
"destination_path": "screenpipe-app-tauri/lib/hooks/eval-settings-falsy.test.tsx"
},
{
"local_path": "graders/settings-falsy.config.mjs",
"destination_path": "screenpipe-app-tauri/eval-settings-falsy.config.mjs"
}
],
"command": "cd screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config eval-settings-falsy.config.mjs"
}
},
{
"id": "app-browser-native-consent-navigation",
"name": "Show native browser session consent without losing ownership safeguards",
"source": {
"kind": "git_regression",
"fix_commit": "0cf9a64a5d7684b00c318154b249400bc007fd38"
},
"base_ref": "a71e821de98615c9bafde1574298cb592b664634",
"oracle_ref": "0cf9a64a5d7684b00c318154b249400bc007fd38",
"tags": [
"regression",
"escaped-failure",
"privacy",
"browser",
"react"
],
"prompt": "The browser sidebar can stay on a loading view when native navigation pauses for permission to use a browser login. Fix the existing consent flow so a valid session-access request for the current chat and navigation presents the explicit Use browser session and Continue logged out choices. Accept native snake_case navigation_id and the existing camelCase navigationId event format, for both the default browser and user-created tabs. Pause loading while permission is pending and hide the native user tab so it cannot cover the choices. Do not resolve the request or change cookie-access/settings state before the user chooses. Each explicit choice must resolve the original request once with the selected boolean, set cookie access to that choice (mark denial when declined), update browserCookieAccessGranted, and dismiss the prompt. Preserve rejection of stale or absent navigation IDs and foreign or absent owners, with no prompt or permission/settings effects. The hidden checks mount the actual component with synthetic native events, persistence and presentation ports. Native webview geometry and cookie extraction are outside this case. Keep the existing ownership protections and add or update relevant tests.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 120,
"fixtures": [
{
"local_path": "graders/browser-consent.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/components/.eval-browser-consent.test.tsx"
},
{
"local_path": "graders/browser-consent.bootstrap.mjs.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-browser-consent-bootstrap.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node .eval-browser-consent-bootstrap.mjs"
},
"oracle_paths": [
"apps/screenpipe-app-tauri/components/browser-sidebar.tsx"
],
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
]
},
{
"id": "ai-gateway-vertex-stream-fallback",
"name": "Keep reasoning-only Vertex streams visibly readable",
"source": {
"kind": "git_regression",
"fix_commit": "65f7928a294cc847534cb1adfe693e8553addfa1"
},
"base_ref": "dec32b804203013868d99a87ba94b6bac61d00e9",
"oracle_ref": "65f7928a294cc847534cb1adfe693e8553addfa1",
"oracle_paths": [
"packages/ai-gateway/src/providers/vertex-maas.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/vertex-maas.ts"
],
"tags": [
"regression",
"escaped-failure",
"streaming",
"provider-compatibility"
],
"prompt": "The public Vertex MaaS streaming completion can return no visible answer when a thinking model supplies only reasoning_content. Preserve the original reasoning deltas and expose their complete text as content exactly once when the stream finishes, sends [DONE], or closes without either, provided no nonempty content was delivered. Handle empty content deltas, CRLF lines and arbitrary byte boundaries including split Unicode. Preserve response id, model and choice index; fallback content must precede an existing [DONE] marker. Ordinary content and reasoning followed by a real answer must remain exact without appended reasoning. Empty streams must not invent text. Preserve tool-call deltas, usage records, comments, malformed data lines and ordinary upstream client-error refusal. For the existing glm-4.7 endpoint, retain one completion POST, bearer authorization, user messages, zero temperature, max_tokens and stream:true without changing caller input. Use synthetic authentication and transport only; never call live providers. No particular helper or transform implementation is required. This task covers one-choice streaming transport, not non-streaming JSON parsing, multiple-choice multiplexing, live model output, billing or desktop rendering.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/vertex-stream-fallback.fixture.ts.txt",
"destination_path": "packages/ai-gateway/eval-vertex-stream-fallback.test.ts"
}
],
"command": "bun --no-env-file test packages/ai-gateway/eval-vertex-stream-fallback.test.ts"
}
},
{
"id": "app-recording-status-authority",
"name": "Recover unavailable recording status without changing capture authority",
"source": {
"kind": "git_regression",
"fix_commit": "dd332af338ba8302b3df12ef424d8a4c7e7bf291"
},
"base_ref": "bbc17e34d453abb9449be47e0cfc9d7a9ecd8565",
"oracle_ref": "dd332af338ba8302b3df12ef424d8a4c7e7bf291",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/recording-status.tsx"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/recording-status.tsx"
],
"tags": [
"regression",
"escaped-failure",
"recording",
"user-pause",
"status-recovery"
],
"prompt": "The recording status popover offers the wrong controls when device information is empty or stale. Preserve the RecordingStatus public interface. For a running session with no reported devices, provide accessible Check again and Open settings actions, without offering a global pause/resume action or changing capture. Check again calls onRefreshDevices once, keeps an accessible Check again or Checking button disabled while pending, prevents duplicate requests, and becomes usable again after either success or rejection. Open settings invokes onOpenRecordingSettings. Global pause must take precedence over an empty device list or stale active devices: show an accessible Resume all recording action that invokes onResumeRecording once, with no individual Pause/Resume controls or device requests. When settings disable all capture, show no capture or status-retry controls, but keep Open settings available, regardless of reported device or global-pause state. Preserve existing behavior: a live session whose devices are individually paused resumes those devices through POST /vision/device/start with monitor_id and POST /audio/device/start with device_name, without global callbacks; active or mixed devices pause through onPauseRecording; a globally paused session resumes through the global callback. A monitor without an id must issue no request while eligible audio devices still resume. Failed individual requests must restore their previous paused state. Hide the global control when both callbacks are absent; an unavailable global resume callback must leave its action disabled or absent. Preserve accessible action names pause all recording, resume all recording and resume (case-insensitive). No private function names, test attributes or implementation structure are required. This task covers frontend controls with synthetic callback/API boundaries, not Home wiring or native recording.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/recording-status-authority.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-hidden/recording-resume.test.tsx"
},
{
"local_path": "graders/recording-resume-config.mjs.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-hidden/recording-resume.config.mjs"
},
{
"local_path": "graders/recording-resume-translation.ts.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-hidden/recording-resume-translation.ts"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config .eval-hidden/recording-resume.config.mjs"
}
},
{
"id": "app-pipe-watch-switch-preservation",
"name": "Keep authoritative watched-pipe text when switching conversations",
"source": {
"kind": "git_regression",
"fix_commit": "3a427b22626db983406d917b43abe4888c647dfc"
},
"base_ref": "6b4edca9b022ddfdc5c489e6175507f626298f71",
"oracle_ref": "3a427b22626db983406d917b43abe4888c647dfc",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/hooks/use-chat-conversations.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/hooks/use-chat-conversations.ts",
"apps/screenpipe-app-tauri/lib/stores/chat-store.ts"
],
"tags": [
"regression",
"escaped-failure",
"data-preservation",
"pipe-history"
],
"prompt": "Switching conversations can replace a watched pipe's newer streamed text with the panel's stale copy when both copies have the same message count. Repair the actual useChatConversations loadConversation path so leaving a pipe-watch session retains its authoritative store messages. Preserve normal chat ownership: an equally sized panel transcript must still replace the normal chat's older store transcript, while a shorter panel snapshot must not discard additional stored messages. Loading a warm incoming chat must restore its stored messages and update both the current store identity and the panel session reference to that incoming chat. Keep the public hook and session-store interfaces compatible. Checks mount the real React hook with its real store and synthetic native, storage and translation ports. No particular guard, helper, or reducer architecture is required. New-chat creation, stream-event delivery, native rendering, disk durability and crash recovery are outside this historical task.",
"agent_timeout_seconds": 900,
"grader_dependency_links": [
{
"source_path": "apps/screenpipe-app-tauri/node_modules",
"destination_path": "apps/screenpipe-app-tauri/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/pipe-watch-switch.fixture.tsx.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-hidden/pipe-watch-switch.test.tsx"
},
{
"local_path": "graders/pipe-watch-switch-config.mjs.txt",
"destination_path": "apps/screenpipe-app-tauri/.eval-hidden/pipe-watch-switch.config.mjs"
}
],
"command": "cd apps/screenpipe-app-tauri && node node_modules/vitest/vitest.mjs run --config .eval-hidden/pipe-watch-switch.config.mjs"
}
},
{
"id": "app-telemetry-full-ax-wire",
"name": "Preserve complete accessibility trees through the telemetry wire format",
"source": {
"kind": "git_regression",
"fix_commit": "449156e009be5580b0235d52c8543c56e29e37be",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7659"
},
"base_ref": "ada8f2fab2f78ea6a9ac8d59f97c932063418a24",
"oracle_ref": "449156e009be5580b0235d52c8543c56e29e37be",
"oracle_paths": [
"crates/screenpipe-telemetry-wire/src/records.rs"
],
"trigger_paths": [
"crates/screenpipe-telemetry-wire/src/records.rs"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"telemetry",
"preserved-behavior"
],
"prompt": "Enterprise frame telemetry loses retained accessibility evidence while encoding or decoding a batch. Repair the existing telemetry wire boundary so an optional accessibility_tree survives JSON input, the public batch encoder and the public batch decoder without losing nodes, order, empty text, nested or unknown properties, false/zero/null values, Unicode, multiline text or long values. An empty tree must remain distinct from an absent tree. Older frames without a tree must remain readable; missing and null trees must be omitted from encoded output. Preserve the existing flattened frame fields, timestamps, device identity and labels, visible summary and nullable browser URL. Mixed frame/audio batches must retain both records and speech. Repeated frame IDs must retain input order and distinct payloads; device identities must not merge. Keep tolerant parsing: blank lines are ignored, each malformed or unknown-kind nonblank line is counted and skipped, and valid neighboring records survive. Empty batches remain empty. Use existing public wire interfaces; no particular implementation is required. Grading compiles the complete records module with synthetic inputs. This task does not require native capture, PII redaction, upload authorization, database hydration or persistence, live services, customer data or execution-isolation changes.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/telemetry-ax.fixture.rs",
"destination_path": ".eval-telemetry-ax/outcomes.rs"
},
{
"local_path": "graders/telemetry-ax.lib.rs",
"destination_path": ".eval-telemetry-ax/lib.rs"
},
{
"local_path": "graders/telemetry-ax.Cargo.toml.txt",
"destination_path": ".eval-telemetry-ax/Cargo.toml"
}
],
"command": "cargo test --offline --manifest-path .eval-telemetry-ax/Cargo.toml --test outcomes -- --nocapture"
}
},
{
"id": "app-audio-drop-control-dispatch",
"name": "Dispatch audio shutdown before dropping the stream",
"source": {
"kind": "git_regression",
"fix_commit": "cd1fe138819e573b89e975283e4d19c8307748ed"
},
"base_ref": "e53b0f0938e47d7d161e4e50a088c43ac04a402c",
"oracle_ref": "cd1fe138819e573b89e975283e4d19c8307748ed",
"oracle_paths": [
"crates/screenpipe-audio/src/core/stream.rs"
],
"trigger_paths": [
"crates/screenpipe-audio/src/core/stream.rs"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"audio",
"resource-lifecycle"
],
"prompt": "Dropping an AudioStream outside an asynchronous runtime can leave its shutdown request unsent and shared disconnect state unchanged. Repair the actual stream destructor so dropping a synthetic stream synchronously marks it disconnected and dispatches its stop request before returning, even when no executor is polling. If the control receiver is already closed, dropping must still mark it disconnected without panicking. A live stream must remain connected until stopped or dropped, and explicit stop with the synthetic closed receiver must remain successful and mark it disconnected. Preserve the existing AudioStream interfaces and synthetic constructor. The hidden checks use the existing internal control sender and shared disconnect flag to observe these outcomes; no particular shutdown helper, task implementation or control-message representation is required. Use synthetic streams only. Native worker completion, device teardown timing, capture hardware, durable recording, migration authority and real customer data are outside this task.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/audio-drop.mjs",
"destination_path": ".eval-audio-drop/audio-drop.mjs"
},
{
"local_path": "graders/audio-drop.fixture.rs",
"destination_path": ".eval-audio-drop/audio-drop.fixture.rs"
}
],
"command": "node .eval-audio-drop/audio-drop.mjs && cargo test --locked -p screenpipe-audio --lib eval_audio_drop -- --test-threads=1"
}
},
{
"id": "app-mcp-browser-boundary",
"name": "Reject foreign browser authority at the HTTP MCP boundary",
"source": {
"kind": "git_regression",
"fix_commit": "52297f8edee9c85ada55581f6ae7287b0643df82"
},
"base_ref": "6bb753f85c6a01cc3a2fc26890ce68cef5cbad85",
"oracle_ref": "52297f8edee9c85ada55581f6ae7287b0643df82",
"oracle_paths": [
"packages/screenpipe-mcp/src/http-server.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/screenpipe-mcp/src/http-server.ts"
],
"tags": [
"regression",
"security",
"mcp",
"privacy",
"preserved-behavior"
],
"agent_timeout_seconds": 900,
"prompt": "The HTTP MCP listener trusts every loopback TCP peer, allowing browser requests with attacker-controlled Host or Origin headers to initialize or use sessions. Repair the actual HTTP boundary. Without a matching configured bearer token, reject foreign authorities, misleading localhost/IP suffixes, userinfo-like authorities, numeric IP aliases, foreign origins and opaque null origins with HTTP 401 and no new session. Apply that protection to initialization, existing-session tools/list and DELETE, including /mcp query URLs; rejected deletion must leave the legitimate session usable. Preserve native clients using literal 127.0.0.1, localhost (case insensitive) or bracketed ::1 authorities with optional ports, and local HTTP/HTTPS origins using those literals. Explicit matching bearer credentials must still authorize otherwise untrusted authorities/origins; wrong or malformed bearer values must not. Successful initialization must return a real usable session and advertise tools, including search_content on listing. Preserve health status/session accounting, unauthenticated /health and OPTIONS, and 404 for unknown authorized routes. Use synthetic loopback traffic with telemetry disabled and no real tool/backend calls. No particular private helper or implementation path is required. This task does not test a real browser/DNS resolver, remote network reachability, all header-parser ambiguities, native recording, telemetry delivery or registry packaging.",
"grader_dependency_links": [
{
"source_path": "packages/screenpipe-mcp/node_modules",
"destination_path": "packages/screenpipe-mcp/node_modules"
}
],
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/mcp-browser-boundary.fixture.ts.txt",
"destination_path": "packages/screenpipe-mcp/src/eval-browser-boundary.test.ts"
}
],
"command": "cd packages/screenpipe-mcp && SCREENPIPE_DISABLE_TELEMETRY=1 SCREENPIPE_MCP_TELEMETRY_DISABLED=1 DO_NOT_TRACK=1 SCREENPIPE_API_URL=http://127.0.0.1:1 node node_modules/vitest/vitest.mjs run src/eval-browser-boundary.test.ts"
}
},
{
"id": "app-shared-inventory-duplicate-devices",
"name": "Return a unique audio device inventory without losing cache behavior",
"source": {
"kind": "git_regression",
"fix_commit": "c6c6382768caf87a83758f65b13e5f6bf53ffb5a"
},
"base_ref": "5ce7edfcda145fa1bb074574925b290a1356bf2c",
"oracle_ref": "c6c6382768caf87a83758f65b13e5f6bf53ffb5a",
"oracle_paths": [
"crates/screenpipe-audio/src/core/device.rs"
],
"trigger_paths": [
"crates/screenpipe-audio/src/core/device.rs"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"audio",
"device-inventory"
],
"prompt": "The shared audio inventory can repeat the same device when platform enumeration returns duplicate entries. Make every fresh or cached inventory contain each exact device name and input/output direction once, preserving first-seen order. Same-named inputs and outputs remain distinct, as do names differing in case or whitespace. Empty and single-device inventories remain valid. Preserve the existing cache contract: unexpired same-generation reads reuse the inventory, concurrent readers share one enumeration, TTL expiry or a topology-generation change refreshes it, refresh replaces removed devices, enumeration errors propagate and remain retryable, and a topology change during enumeration forces another refresh on the next read. Preserve the existing AudioDevice and list_audio_devices_with_cache interfaces so synthetic enumeration can exercise the real cache function. No particular deduplication algorithm is required. Use synthetic devices only; do not access recording hardware, customer data or external services. This task does not require changing capture, pause, migration or HTTP behavior.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/audio-inventory.mjs",
"destination_path": ".eval-audio-inventory/audio-inventory.mjs"
},
{
"local_path": "graders/audio-inventory.fixture.rs",
"destination_path": ".eval-audio-inventory/audio-inventory.fixture.rs"
}
],
"command": "node .eval-audio-inventory/audio-inventory.mjs && cargo test --locked -p screenpipe-audio --lib eval_audio_inventory -- --test-threads=1"
}
},
{
"id": "ai-gateway-anthropic-json-usage",
"name": "Preserve reported Anthropic usage through JSON completions",
"source": {
"kind": "git_regression",
"fix_commit": "3ccae296d1c99840669ed1472306907eac093198"
},
"base_ref": "7f4f1494042e31968fb8062936a0b971283dcf84",
"oracle_ref": "3ccae296d1c99840669ed1472306907eac093198",
"oracle_paths": [
"packages/ai-gateway/src/providers/anthropic.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/anthropic.ts"
],
"tags": [
"regression",
"escaped-failure",
"billing",
"provider-usage"
],
"prompt": "Successful non-streaming Anthropic completions return an answer but lose the provider-reported token usage. Repair the existing AnthropicProvider.createCompletion entrypoint so its JSON response includes prompt_tokens, completion_tokens and total_tokens for uncached input/output usage supplied by the upstream SDK. Preserve exact nonnegative counts, including either or both counts being zero and large counts; total_tokens is their sum. Successive calls on the same provider must use each response's own counts. Preserve HTTP 200 and JSON content type, visible Unicode text and assistant role, tool names and structured arguments beside text or in tool-only responses with empty text. Each completion must make one upstream request, retain the requested message, generation limit and temperature, normalize the existing @date model spelling, and leave the caller's request unchanged. Upstream rejection must remain a failure rather than a fabricated completion. Use synthetic provider responses; do not call live services. Any implementation meeting these outcomes is acceptable. Streaming, absent usage, prompt-cache accounting, downstream monetary settlement, provider billing and live model behavior are outside this task.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/anthropic-json-usage.fixture.ts.txt",
"destination_path": "packages/ai-gateway/eval-anthropic-json-usage.test.ts"
}
],
"command": "bun --no-env-file test packages/ai-gateway/eval-anthropic-json-usage.test.ts"
}
},
{
"id": "ai-gateway-openrouter-stream-envelope",
"name": "Preserve OpenRouter streaming response fields",
"source": {
"kind": "git_regression",
"fix_commit": "4678cc32842a2992a338016557a7c14e6335a18e"
},
"base_ref": "f9fc6e488643a798920a1cd72b4dcdf6208ef3e1",
"oracle_ref": "4678cc32842a2992a338016557a7c14e6335a18e",
"oracle_paths": [
"packages/ai-gateway/src/providers/openrouter.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers/openrouter.ts"
],
"tags": [
"regression",
"escaped-failure",
"ai-gateway",
"streaming"
],
"prompt": "OpenRouter streaming responses lose fields and omit events without visible text. Repair the existing public streaming completion entrypoint so its SSE output retains response identity, object, creation time, model and each choice index alongside text. Preserve assistant role-only deltas, tool-only identity/name/argument fragments in order, finish-only deltas and finish reasons, usage-only events including zero counts, explicit empty text, and multiple choices. Preserve ordinary Unicode text order, exactly one final [DONE] marker, and empty-stream termination without fabricated chunks. Make one upstream streaming request preserving model, messages, temperature and tools without mutating the caller. An initial provider rejection must still reject; an iterator failure after partial output must retain that text and emit error message/type/string status code before termination. Use synthetic provider responses; no live services. Any implementation meeting these observable outcomes is acceptable. SDK transport, Pi execution, monetary settlement, cancellation and model behavior are outside scope.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/openrouter-stream.fixture.ts.txt",
"destination_path": "packages/ai-gateway/eval-openrouter-stream.test.ts"
}
],
"command": "bun --no-env-file test packages/ai-gateway/eval-openrouter-stream.test.ts"
}
},
{
"id": "app-manual-pipe-instruction-preservation",
"name": "Preserve pipe instructions across manual and scheduled execution",
"source": {
"kind": "git_regression",
"fix_commit": "047d2286dbae0b5123f413bb6ae11991ac62789e"
},
"base_ref": "1b0e072719a157c83050ac92792cba4def7a33ae",
"oracle_ref": "047d2286dbae0b5123f413bb6ae11991ac62789e",
"oracle_paths": [
"crates/screenpipe-core/src/pipes/mod.rs"
],
"trigger_paths": [
"crates/screenpipe-core/src/pipes/**",
"crates/screenpipe-core/src/agents/**"
],
"gate": "advisory",
"tags": [
"regression",
"escaped-failure",
"pipes",
"instruction-preservation"
],
"prompt": "A manually started pipe can reach its executor without the instructions authored in pipe.md. Repair both the blocking manual entrypoint and the nonblocking background entrypoint. Preserve scheduled execution. The executor must receive the complete multiline and Unicode pipe body, the selected preset instructions and the configured local API endpoint. These may be carried in either prompt channel; no channel split or rendering helper is required. Preserve extra integration context, the current local date and the existing fresh ISO8601 time range (the preceding hour for an every-1h pipe), selected model/provider/URL/key, explicit history continuation and the default fresh-session behavior. Each invocation must complete once, successfully relay executor output and use that pipe directory. Empty bodies and pipes without a selected preset remain usable without inheriting another preset or body. Keep the existing PipeManager and AgentExecutor interfaces so synthetic executors can exercise the actual foreground, background and scheduler callers. Use only synthetic files and executors; do not run a real model, provider, capture hardware or customer data. Native Pi transport, permission enforcement, durable database effects and model quality are outside this task.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 180,
"fixtures": [
{
"local_path": "graders/pipe-prompt.mjs",
"destination_path": ".eval-pipe-prompt/pipe-prompt.mjs"
},
{
"local_path": "graders/pipe-prompt.fixture.rs",
"destination_path": ".eval-pipe-prompt/pipe-prompt.fixture.rs"
}
],
"command": "node .eval-pipe-prompt/pipe-prompt.mjs && cargo test --locked -p screenpipe-core --lib eval_pipe_prompt -- --test-threads=1"
}
},
{
"id": "app-cloud-procedure-terminal-routes",
"name": "Retain deliberate terminal destinations when saving a cloud procedure",
"source": {
"kind": "git_regression",
"fix_commit": "d1d5c4cb020159628c0319527b4f7583a883e1e4"
},
"base_ref": "fea47cbda9708ea6cd84959e145a734e4f4478d9",
"oracle_ref": "d1d5c4cb020159628c0319527b4f7583a883e1e4",
"oracle_paths": [
"packages/workflows-ui/src/cloud-workflow.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/workflows-ui/src/cloud-workflow.ts"
],
"tags": [
"regression",
"escaped-failure",
"workflows",
"data-integrity",
"routing"
],
"prompt": "Saving an existing cloud workflow can erase a deliberate stop and change where the procedure continues. Repair the existing cloud procedure edit API so an ordinary step and either decision branch retain an explicit `end` destination. Preserve links to retained step IDs when steps are reordered, and clear only destinations whose step is missing or deleted. Keep absent routes absent, retain unrelated step, branch and document metadata, apply authored content edits and the expected revision, and leave the original procedure and edit input unchanged.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/cloud-terminal-routes.test.ts",
"destination_path": "packages/workflows-ui/src/cloud-terminal-routes.eval.test.ts"
}
],
"command": "bun test ./packages/workflows-ui/src/cloud-terminal-routes.eval.test.ts"
}
},
{
"id": "app-retention-capture-during-cleanup",
"name": "Keep durable live captures available during local retention cleanup",
"source": {
"kind": "git_regression",
"fix_commit": "d5046990b77565ad45db3587277d9e765b391dfd",
"pull_request": "https://github.com/screenpipe/screenpipe/pull/7734"
},
"base_ref": "9334c1f6368bda3c28a8f694b6b5624b092d9f7b",
"oracle_ref": "d5046990b77565ad45db3587277d9e765b391dfd",
"oracle_paths": [
"crates/screenpipe-engine/src/retention.rs",
"crates/screenpipe-db/src/db/maintenance.rs"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-engine/src/retention.rs"
],
"tags": [
"regression",
"escaped-failure",
"durable-recording",
"retention",
"sqlite"
],
"prompt": "When local retention removes an old archive from the recording database, live captures must remain available and durable throughout the cleanup. In All mode, seed 1,000 old frames with 100,000 associated accessibility elements, then run cleanup while continuing to insert captures. Cleanup must begin database processing within 15 seconds. Each capture must complete within the existing 15-second request deadline, at least one capture must be accepted during cleanup, and cleanup must remove the old frames and elements and reach its requested watermark. After closing and reopening the database, every accepted live capture must remain present and the latest display layout must remain preserved. Use a disposable synthetic database. Any implementation that meets these observable results is acceptable. This case does not cover migration authority, user pause policy, search behavior or live customer recordings.",
"agent_timeout_seconds": 1800,
"grader_dependency_links": [
{
"source_path": "target",
"destination_path": "target"
}
],
"grader": {
"timeout_seconds": 300,
"fixtures": [
{
"local_path": "graders/retention-capture-durable.rs.txt",
"destination_path": ".eval-retention/retention-capture-durable.rs"
},
{
"local_path": "graders/retention-capture-durable.mjs",
"destination_path": ".eval-retention/retention-capture-durable.mjs"
}
],
"command": "node .eval-retention/retention-capture-durable.mjs"
}
},
{
"id": "app-zero-weight-over-quota",
"name": "Allow free requests after paid usage exceeds the daily quota",
"source": {
"kind": "git_regression",
"fix_commit": "e7d14bd22073151bc127c10751d9533c80e38349"
},
"base_ref": "71a8da699db22094eb3d0546008279111e337b08",
"oracle_ref": "e7d14bd22073151bc127c10751d9533c80e38349",
"oracle_paths": [
"packages/ai-gateway/src/services/usage-tracker.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/services/usage-tracker.ts"
],
"tags": [
"regression",
"escaped-failure",
"gateway",
"quota",
"preserved-abuse-limit"
],
"prompt": "Repair the existing exported trackUsage behavior. The zero-weight auto model must be allowed even when earlier paid usage left a same-day device counter above its daily quota. With the logged-in daily limit configured to 50, auto requests at counts 49, 50, 51 and 500 must preserve the count, report that used count and limit 50, and report remaining quota as max(0, 50 minus the used count). A new auto device reports used 0/remaining 50. Preserve rejection of a paid gpt-4o request at count 50 without increasing its counter. Free requests must still obey the independent anonymous IP abuse limit: a current-day IP counter 1500 rejects an anonymous auto request and leaves its device count 51 unchanged. These no-user-identity sequences must make no credit, auto-reload or other external fetch calls. Preserve auto as zero weight and paid request limits. Any implementation meeting these observable outcomes is acceptable; do not require a particular helper or SQL arrangement. Use synthetic data and focused tests. This case exercises the accounting module through an in-memory SQLite adapter for its D1 port, not HTTP admission, real D1 concurrency, credit settlement, native behavior or live services.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/zero-weight-quota.test.js.txt",
"destination_path": "packages/ai-gateway/src/services/eval-hidden-zero-weight.test.js"
}
],
"command": "bun test packages/ai-gateway/src/services/eval-hidden-zero-weight.test.js"
}
},
{
"id": "app-transcription-diagnostic-custom-headers",
"name": "Honor custom authentication headers in transcription diagnostics",
"source": {
"kind": "git_regression",
"fix_commit": "624edb18bc1db0fb76c9852402e1ad6ff5b36911"
},
"base_ref": "459b30a7bb4b518da7da33793740ac8637db9781",
"oracle_ref": "624edb18bc1db0fb76c9852402e1ad6ff5b36911",
"oracle_paths": [
"apps/screenpipe-app-tauri/components/settings/hooks/use-transcription-diagnostics.ts"
],
"gate": "advisory",
"trigger_paths": [
"apps/screenpipe-app-tauri/components/settings/hooks/use-transcription-diagnostics.ts"
],
"tags": [
"regression",
"escaped-failure",
"transcription",
"custom-auth",
"preserved-refusal"
],
"prompt": "Repair the existing exported useTranscriptionDiagnostics hook so configured custom headers reach both model/auth diagnostics and transcription requests. Support a custom X-Fixture-Key with no bearer key and a configured Authorization value overriding the default bearer, alongside other custom headers. Preserve bearer-only and public endpoints. An auth refusal must report auth failure, skip transcription and make no transcription request, including custom-header authentication. A transcription refusal must remain a failed transcription. Successful requests must complete with passing auth/transcription. Use the configured endpoint and model; preserve multipart POST with a nonempty audio/wav file on the historical webview transport. Do not require a particular helper or merge syntax. Tests execute the actual exported hook with deterministic state/callback and synthetic HTTP/native transport ports. They inspect outgoing requests and reported outcomes, not real React lifecycle, browser CORS, native audio/HTTP or any live provider. Do not use real keys or services.",
"agent_timeout_seconds": 1200,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/transcription-custom-headers.test.js.txt",
"destination_path": "apps/screenpipe-app-tauri/components/settings/hooks/eval-hidden-custom-headers.test.js"
}
],
"command": "bun test apps/screenpipe-app-tauri/components/settings/hooks/eval-hidden-custom-headers.test.js"
}
},
{
"id": "gateway-fable-tool-continuation",
"name": "Complete Fable tool-result continuations without replay",
"source": {
"kind": "git_regression",
"fix_commit": "34c8d0939c8865dd04e20fddf3c61adf2c7db9a5"
},
"base_ref": "b44633101f62d14427a8ef9b18640cd3808b4af5",
"oracle_ref": "34c8d0939c8865dd04e20fddf3c61adf2c7db9a5",
"oracle_paths": [
"packages/ai-gateway/src/providers/index.ts",
"packages/ai-gateway/src/providers/experiential-fable.ts"
],
"gate": "advisory",
"trigger_paths": [
"packages/ai-gateway/src/providers"
],
"tags": [
"regression",
"gateway",
"preserved-behavior"
],
"prompt": "A hosted Fable conversation can end with no assistant content after a tool result when further tool use is disabled. Repair streamed and JSON continuations so the available provider response reaches the caller. Select a working request before dispatch; do not replay a paid request to hide an empty response. Preserve the requested output-token budget, server policy headers, reported input/output token usage and supplied input objects. Ordinary tool-choice turns must still complete and explicit native-provider rollback must keep using its requested native protocol. Keep existing provider interfaces usable. Add focused tests using synthetic transport responses, with no live model or customer requests.",
"agent_timeout_seconds": 900,
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/fable-tool-continuation.test.ts.txt",
"destination_path": "evals/coding-agent/graders/fable-tool-continuation.test.ts"
}
],
"command": "\"${BUN_BIN:-bun}\" test evals/coding-agent/graders/fable-tool-continuation.test.ts"
},
"grader_dependency_links": [
{
"source_path": "node_modules",
"destination_path": "node_modules"
}
]
},
{
"id": "app-orphan-guard-one-shot-stdin",
"name": "Keep one-shot work alive after prompt stdin closes",
"source": {
"kind": "git_regression",
"fix_commit": "686a2c9058f702ae9f6acfbf97c306281272d7b8"
},
"base_ref": "3aea8df68f66d3a20e19093be47432b3bee476cf",
"oracle_ref": "686a2c9058f702ae9f6acfbf97c306281272d7b8",
"oracle_paths": [
"crates/screenpipe-core/assets/extensions/orphan-guard.ts"
],
"gate": "advisory",
"trigger_paths": [
"crates/screenpipe-core/assets/extensions/orphan-guard.ts"
],
"prompt": "One-shot pipe runs terminate before producing work when their prompt input closes. Preserve one-shot work after stdin end or close. RPC sessions must still exit when their stdin ends or closes. In both modes preserve work while the original parent remains healthy, and exit when the parent PID changes or the parent no longer exists.",
"grader": {
"timeout_seconds": 60,
"fixtures": [
{
"local_path": "graders/orphan-guard-stdin.test.ts.txt",
"destination_path": "eval-hidden-orphan.test.ts"
}
],
"command": "bun test ./eval-hidden-orphan.test.ts"
},
"tags": [
"regression",
"escaped-failure",
"pipes",
"lifecycle",
"stdin"
]
}
]
}