Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

test(evals): cover fresh onboarding assignment and login completion

louis65d555899b79
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.md
index 00288be72..95e744714 100644
--- a/evals/coding-agent/README.md
+++ b/evals/coding-agent/README.md
@@ -9,10 +9,31 @@ This is an agent eval suite, not a unit-test suite. Every case contains:
- saved prompt, transcript, candidate patch, grader output, runtime fingerprint, and result;
- repeated-trial reporting with success rate, `pass@k`, and `pass^k`.
-The current app corpus contains 49 git-mined regressions. See
+The current app corpus contains 50 git-mined regressions. See
[DESIGN.md](./DESIGN.md) for the Anthropic guidance, source contract, and
history-mining workflow. The companion website manifest uses this same harness.
+`app-onboarding-fresh-assignment-login` selects thirteen historical React outcome
+tests for disabled/absent experiment flags, final-identity assignment, pinned
+routes, fresh-login completion across login-screen unmounts, hydration/resume
+exclusion and preserved treatment/control eligibility. The actual page and, for
+the login-transition cases, real login gate run against synthetic settings,
+managed-policy, analytics and native-command ports. Other slides are test doubles.
+The parent fails two intended outcomes and preserves eleven; the reference and
+the matching thirteen tests in current product source pass. Unselected product
+tests are explicitly skipped, not counted as verified outcomes.
+
+Run `bun test evals/coding-agent/calibrate-onboarding-assignment.test.js` with the
+desktop's JavaScript test dependencies installed. Controls reject an unused
+correct page, stale callback acceptance, duplicate or lost completion, hydration
+miscounting, blanket control assignment and false error telemetry, while accepting
+equivalent state naming and distinguishing missing-source setup errors. Only the
+page and login gate are applied by the historical oracle. Dependencies are linked
+only when grading begins; this does not establish execution isolation. No live
+login, PostHog delivery, checkout, recorder or native build is exercised, and no
+model trial is implied. Newer workflow/auth-restore behavior remains outside the
+historical case.
+
The MCP config symlink case grades Settings-side IO with synthetic files and
real links on a symlink-capable host. Its platform path-resolution port is
substituted; passing it does not establish native bridge or desktop acceptance.
diff --git a/evals/coding-agent/calibrate-onboarding-assignment.test.js b/evals/coding-agent/calibrate-onboarding-assignment.test.js
new file mode 100644
index 000000000..cb517a3df
--- /dev/null
+++ b/evals/coding-agent/calibrate-onboarding-assignment.test.js
@@ -0,0 +1,96 @@
+// screenpipe — AI that knows everything you've seen, said, or heard
+// https://screenpipe.com
+import { afterAll, expect, test } from 'bun:test';
+import { execFileSync, spawnSync } from 'node:child_process';
+import { dirname, join, resolve } from 'node:path';
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+
+const repo = resolve(import.meta.dir, '../..');
+const app = 'apps/screenpipe-app-tauri';
+const page = `${app}/app/onboarding/page.tsx`;
+const gate = `${app}/components/onboarding/login-gate.tsx`;
+const fixture = `${app}/app/onboarding/page.test.tsx`;
+const item = JSON.parse(readFileSync(join(import.meta.dir, 'cases.json'), 'utf8')).cases
+ .find(c => c.id === 'app-onboarding-fresh-assignment-login');
+const root = mkdtempSync(join(tmpdir(), 'onboarding-assignment-calibration-'));
+const archives = new Map();
+const fixtureSource = execFileSync('git', ['show', `${item.oracle_ref}:${fixture}`], { cwd: repo });
+const paths = ['app/onboarding', 'components', 'lib', 'scripts', 'vitest.config.ts',
+ 'vitest.setup.ts', 'package.json', 'tsconfig.json', 'postcss.config.js', 'tailwind.config.js']
+ .filter(path => spawnSync('git', ['cat-file', '-e', `${item.oracle_ref}:${app}/${path}`], { cwd: repo }).status === 0)
+ .map(path => `${app}/${path}`);
+afterAll(() => rmSync(root, { recursive: true, force: true }));
+
+function change(cwd, path, from, to) {
+ const file = join(cwd, path), source = readFileSync(file, 'utf8');
+ expect(source.split(from)).toHaveLength(2);
+ writeFileSync(file, source.replace(from, to));
+}
+function grade(name, ref = item.oracle_ref, mutate = () => {}) {
+ const cwd = join(root, name); mkdirSync(cwd);
+ if (!archives.has(ref)) archives.set(ref, execFileSync('git',
+ ['-c', 'core.commitGraph=false', 'archive', ref, ...paths], { cwd: repo, maxBuffer: 128 * 1024 * 1024 }));
+ execFileSync('tar', ['-x', '-C', cwd], { input: archives.get(ref) });
+ writeFileSync(join(cwd, fixture), fixtureSource);
+ symlinkSync(join(repo, app, 'node_modules'), join(cwd, app, 'node_modules'), 'dir');
+ mutate(cwd);
+ return spawnSync('/bin/bash', ['-c', item.grader.command], { cwd, encoding: 'utf8',
+ timeout: 120_000, env: { ...process.env, PATH: `${dirname(process.execPath)}:${process.env.PATH}` },
+ maxBuffer: 4 * 1024 * 1024 });
+}
+function fails(result) {
+ expect(result.error).toBeUndefined(); expect(result.signal).toBeNull();
+ expect(result.status).toBe(1);
+ expect(result.stdout + result.stderr).toMatch(/expected|Unable to find an element/);
+ expect(result.stdout + result.stderr).not.toMatch(/Failed to load url|Cannot find module|Failed to resolve import/);
+}
+test('parent exposes both intended failures and eleven preserved outcomes', () => {
+ const r = grade('parent', item.base_ref); fails(r);
+ expect(r.stdout).toContain('2 failed | 11 passed');
+}, 120_000);
+test('historical reference passes all thirteen selected outcomes', () => {
+ const r = grade('reference'); expect(r.status).toBe(0); expect(r.stdout).toContain('13 passed');
+}, 120_000);
+test('equivalent state variable naming is accepted', () => {
+ const r = grade('renamed', item.oracle_ref, cwd => {
+ const file = join(cwd, page);
+ writeFileSync(file, readFileSync(file, 'utf8').replaceAll('previousLoginStateRef', 'lastHydratedAuthState'));
+ }); expect(r.status).toBe(0);
+}, 120_000);
+test('an unused correct page cannot hide the unchanged broken caller', () => {
+ fails(grade('unused-page', item.oracle_ref, cwd => {
+ writeFileSync(join(cwd, dirname(page), 'unused-correct-page.tsx'), readFileSync(join(cwd, page)));
+ writeFileSync(join(cwd, page), execFileSync('git', ['show', `${item.base_ref}:${page}`], { cwd: repo }));
+ }));
+}, 120_000);
+test('restoring the old login gate rejects duplicate completion effects', () => {
+ fails(grade('old-gate', item.oracle_ref, cwd => writeFileSync(join(cwd, gate),
+ execFileSync('git', ['show', `${item.base_ref}:${gate}`], { cwd: repo }))));
+}, 120_000);
+test('discarding login completion is rejected', () => {
+ fails(grade('lost-login', item.oracle_ref, cwd => change(cwd, page,
+ 'if (loginCompleted && currentSlide === "login" && !isManagedDeployment)',
+ 'if (false && loginCompleted && currentSlide === "login" && !isManagedDeployment)')));
+}, 120_000);
+test('counting hydration as a fresh login is rejected', () => {
+ fails(grade('hydration-login', item.oracle_ref, cwd => change(cwd, page,
+ 'previousLoginStateRef.current === false && isLoggedIn', 'isLoggedIn')));
+}, 120_000);
+test('forcing every assignment to control is rejected by treatment preservation', () => {
+ fails(grade('control-only', item.oracle_ref, cwd => change(cwd, page,
+ 'variant: typeof value === "string" ? value : "control",', 'variant: "control",')));
+}, 120_000);
+test('accepting earlier matching-identity callbacks is rejected', () => {
+ fails(grade('stale-flags', item.oracle_ref, cwd => change(cwd, page,
+ 'if (matchingIdentityResponses < 3)', 'if (matchingIdentityResponses < 1)')));
+}, 120_000);
+test('reporting a disabled flag as a load failure is rejected', () => {
+ fails(grade('false-error', item.oracle_ref, cwd => change(cwd, page,
+ 'settle({ variant: "control", source: "posthog" });', 'fallBackToControl("load_error");')));
+}, 120_000);
+test('missing page remains a setup failure rather than a reproduced regression', () => {
+ const r = grade('missing', item.oracle_ref, cwd => rmSync(join(cwd, page)));
+ expect(r.status).toBe(1); expect(r.stdout + r.stderr).toMatch(/Failed to resolve import|Failed to load url/);
+ expect(r.stdout).not.toContain('13 passed');
+}, 120_000);
diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.json
index 892a8b92e..14c02c3b1 100644
--- a/evals/coding-agent/cases.json
+++ b/evals/coding-agent/cases.json
@@ -1,7 +1,7 @@
{
"schema_version": 1,
"suite": "screenpipe-app-coding-agent-regressions",
- "dataset_version": "2026-09-19.6",
+ "dataset_version": "2026-09-19.7",
"suite_type": "regression",
"owner": "screenpipe engineering",
"policy": {
@@ -1583,6 +1583,47 @@
],
"command": "cd apps/screenpipe-app-tauri && \"${BUN_BIN:-bun}\" test ../../evals/coding-agent/graders/preset-save-recovery.test.js"
}
+ },
+ {
+ "id": "app-onboarding-fresh-assignment-login",
+ "name": "Resolve absent fresh flags and retain login completion across assignment",
+ "source": {
+ "kind": "git_regression",
+ "fix_commit": "b2a2b1a8c33cdba993c81d063065b86856f94acb",
+ "pull_request": "https://github.com/screenpipe/screenpipe/pull/6921"
+ },
+ "base_ref": "590f6e55f57e161d7e66e550bd39bb9e86384481",
+ "oracle_ref": "b2a2b1a8c33cdba993c81d063065b86856f94acb",
+ "oracle_paths": [
+ "apps/screenpipe-app-tauri/app/onboarding/page.tsx",
+ "apps/screenpipe-app-tauri/components/onboarding/login-gate.tsx"
+ ],
+ "tags": [
+ "regression",
+ "escaped-failure",
+ "onboarding",
+ "auth",
+ "trial",
+ "recovery"
+ ],
+ "prompt": "A fresh desktop login can lose its completion event when experiment assignment temporarily replaces the login screen, and a successful fresh flag response that omits the disabled experiment leaves setup waiting until timeout. Resolve an absent flag from the final authenticated identity immediately to the persisted control route without reporting assignment failure. Record onboarding_login_completed exactly once for a real signed-out to signed-in transition, both with and without assignment, even if the login screen unmounts; settings hydration, an already-authenticated resume, repeated callbacks, and advancing without a new auth transition must not count as another login. Preserve final-identity freshness against cached and earlier in-flight flag responses, explicit loading-error fallback, a pinned route after checkout navigation, and summary-first enrollment for eligible fresh installs with no plan label. Preserve the paid-account fresh-install bypass, never enroll upgraded existing installs after reset, and do not start the standard funnel for managed or resumed onboarding. Keep enrollment telemetry free of email. Exercise the real React page and login screen with synthetic provider/native ports; do not contact live services or change billing, native publication, or recording behavior. No particular internal state variable or telemetry-owner component is required.",
+ "agent_timeout_seconds": 1200,
+ "grader_dependency_links": [
+ {
+ "source_path": "apps/screenpipe-app-tauri/node_modules",
+ "destination_path": "apps/screenpipe-app-tauri/node_modules"
+ }
+ ],
+ "grader": {
+ "timeout_seconds": 120,
+ "fixtures": [
+ {
+ "source_ref": "b2a2b1a8c33cdba993c81d063065b86856f94acb",
+ "source_path": "apps/screenpipe-app-tauri/app/onboarding/page.test.tsx"
+ }
+ ],
+ "command": "cd apps/screenpipe-app-tauri && NODE_OPTIONS=--localstorage-file=.eval-localstorage bun x vitest run --config vitest.config.ts app/onboarding/page.test.tsx -t 'continues immediately when fresh authenticated flags|records a fresh login once|does not count an authenticated settings hydration|records one fresh login without|does not expose the control checkout|falls back explicitly to control|restores the pinned route|assigns a fresh-install treatment user|uses a paid account only|never enrolls an upgraded free install|does not start the standard funnel for managed|does not start a new funnel when standard onboarding resumes|leaves login completion analytics' --maxWorkers=1 --minWorkers=1"
+ }
}
]
}