An open source design system that's fully customizable and agent ready
TypeScript
feat(vibe-tests): add execution provenance sidecars (#5500)
nynexman4464fa6756fe3bbf
diff --git a/CLAUDE.md b/CLAUDE.mdindex 1a7b6a36a5..e98e33c8d7 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -48,6 +48,8 @@ Probes at turns 0, 6, 8, 10 to measure quality degradation. Results show a line}```+Runners may also write an optional `<promptId>.provenance.json` sidecar beside the result metadata. The versioned, executor-neutral contract and fallback behavior are documented in `internal/vibe-tests/docs/execution-provenance.md`.+## AI ContextFor architectural context, decisions, and research, see the **[GitHub Wiki](https://github.com/facebook/astryx/wiki)**:diff --git a/internal/vibe-tests/README.md b/internal/vibe-tests/README.mdindex 2ec472f72e..9188db8ff7 100644--- a/internal/vibe-tests/README.md+++ b/internal/vibe-tests/README.md@@ -126,6 +126,12 @@ These are intentional and documented; they slightly favor baseline, making Astry- **Maintainability:** Tailwind scale values (`p-4`, `text-sm`) count as semantic, which is generous compared to how raw `16px` is counted for HTML- **Astryx+Tailwind scoring:** The hybrid target counts styling decisions from both Astryx props and Tailwind classes. This may inflate its decision count relative to pure Astryx, but accurately reflects the code's actual styling surface area+## Execution Provenance++Runners may write an optional `<promptId>.provenance.json` sidecar beside each result. It records versioned, executor-neutral task, fixture, condition, executor, timing, and token metadata without embedding prompts or filesystem paths. Collection copies the sidecar, and universal aggregation preserves per-run harness/model labels, supports provenance filters, and marks measured, derived, estimated, complete, and incomplete cost data explicitly.++See [`docs/execution-provenance.md`](docs/execution-provenance.md) for the schema, example, fallback rules, and runner migration contract.+## Directory Structure```diff --git a/internal/vibe-tests/docs/execution-provenance.md b/internal/vibe-tests/docs/execution-provenance.mdnew file mode 100644index 0000000000..c3ce5e3477--- /dev/null+++ b/internal/vibe-tests/docs/execution-provenance.md@@ -0,0 +1,69 @@+# Execution provenance++Vibe-test runners may emit an optional, executor-neutral provenance sidecar for each result. The canonical filename is `<promptId>.provenance.json`, beside `<promptId>.tsx` and `<promptId>.json`.++The version 1 contract is defined by:++- `schema/execution-provenance-v1.schema.json` for JSON Schema consumers+- `src/provenance.ts` for the TypeScript type and parser++Unknown fields are preserved so producers can add data without breaking version 1 readers. A reader rejects an unknown `schemaVersion`. Missing sidecars and valid partial sidecars use the legacy aggregation fallbacks. A sidecar that exists but is malformed is an error.++## Example++```json+{+ "schemaVersion": 1,+ "task": {+ "id": "profile-card",+ "sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"+ },+ "fixture": {+ "id": "starter-project",+ "commit": "0123456789abcdef"+ },+ "condition": "setup",+ "rep": 1,+ "executor": {+ "harness": "local-cli",+ "model": "example-model",+ "effort": "standard",+ "harnessVersion": "2.0.0",+ "runnerVersion": "1.4.0"+ },+ "execution": {+ "status": "succeeded",+ "startedAt": "2026-01-02T03:04:05.000Z",+ "finishedAt": "2026-01-02T03:04:09.000Z",+ "durationMs": 4000,+ "attempt": 1,+ "retry": 0+ },+ "usage": {+ "inputTokens": 1200,+ "outputTokens": 340,+ "source": "runner",+ "complete": true+ }+}+```++Labels under `executor` are opaque. The framework does not interpret or restrict runner, harness, model, or effort names.++## Aggregation++`universal-aggregate.ts` preserves a per-run summary and groups results by:++- harness and model together+- fixture, when present+- condition, when present++It accepts `--harness`, `--model`, `--fixture`, and `--condition` filters. Old results are grouped under `unknown` harness and model labels.++Sidecar duration and token usage take precedence over legacy metadata and estimates. Every selected value carries a source and quality. Token totals are comparable only when every run in the group has `usage.complete: true`; otherwise grouped totals are `null` and the complete/incomplete run counts explain why.++## Runner migration++Existing runners do not need to change their result JSON. Add the sidecar beside each result and let `collect-results.mjs` copy it. Use `expandExecutionMatrix` from `src/provenance-matrix.ts` when a setup or adoption runner needs the Cartesian product of harness, model, effort, condition, and repetition.++This is an additive adapter contract: specialized runner branches can rebase, emit version 1 sidecars, and keep their current task/result formats. Do not put raw prompts, filesystem paths, credentials, or tool transcripts in provenance; use stable IDs and hashes instead.diff --git a/internal/vibe-tests/schema/execution-provenance-v1.schema.json b/internal/vibe-tests/schema/execution-provenance-v1.schema.jsonnew file mode 100644index 0000000000..c2895e9d2d--- /dev/null+++ b/internal/vibe-tests/schema/execution-provenance-v1.schema.json@@ -0,0 +1,82 @@+{+ "$schema": "https://json-schema.org/draft/2020-12/schema",+ "$id": "https://github.com/facebook/astryx/blob/main/internal/vibe-tests/schema/execution-provenance-v1.schema.json",+ "title": "ExecutionProvenanceV1",+ "description": "Executor-neutral provenance for one vibe-test result.",+ "type": "object",+ "required": ["schemaVersion"],+ "properties": {+ "schemaVersion": {"const": 1},+ "task": {+ "type": "object",+ "required": ["id", "sha256"],+ "properties": {+ "id": {"type": "string", "minLength": 1},+ "sha256": {"$ref": "#/$defs/sha256"}+ },+ "additionalProperties": true+ },+ "fixture": {+ "type": "object",+ "required": ["id"],+ "properties": {+ "id": {"type": "string", "minLength": 1},+ "sha256": {"$ref": "#/$defs/sha256"},+ "commit": {"type": "string", "minLength": 1}+ },+ "additionalProperties": true+ },+ "condition": {"type": "string", "minLength": 1},+ "rep": {"type": "integer", "minimum": 1},+ "executor": {+ "type": "object",+ "required": ["harness", "model"],+ "properties": {+ "harness": {"type": "string", "minLength": 1},+ "model": {"type": "string", "minLength": 1},+ "effort": {"type": "string", "minLength": 1},+ "harnessVersion": {"type": "string", "minLength": 1},+ "runnerVersion": {"type": "string", "minLength": 1}+ },+ "additionalProperties": true+ },+ "execution": {+ "type": "object",+ "properties": {+ "status": {"type": "string", "minLength": 1},+ "startedAt": {"type": "string", "format": "date-time"},+ "finishedAt": {"type": "string", "format": "date-time"},+ "durationMs": {"type": "number", "minimum": 0},+ "attempt": {"type": "integer", "minimum": 1},+ "retry": {"type": "integer", "minimum": 0}+ },+ "additionalProperties": true+ },+ "usage": {+ "type": "object",+ "required": ["complete"],+ "properties": {+ "inputTokens": {"type": "integer", "minimum": 0},+ "outputTokens": {"type": "integer", "minimum": 0},+ "source": {"type": "string", "minLength": 1},+ "complete": {"type": "boolean"}+ },+ "allOf": [+ {+ "if": {"properties": {"complete": {"const": true}}},+ "then": {"required": ["inputTokens", "outputTokens"]}+ }+ ],+ "additionalProperties": true+ },+ "environmentHash": {"$ref": "#/$defs/sha256"},+ "toolPolicyHash": {"$ref": "#/$defs/sha256"}+ },+ "$defs": {+ "sha256": {+ "type": "string",+ "pattern": "^[a-fA-F0-9]{64}$"+ }+ },+ "additionalProperties": true+}diff --git a/internal/vibe-tests/src/collect-results.mjs b/internal/vibe-tests/src/collect-results.mjsindex d0952ce207..f79a9f27b3 100644--- a/internal/vibe-tests/src/collect-results.mjs+++ b/internal/vibe-tests/src/collect-results.mjs@@ -12,6 +12,8 @@** Copies: results/<id>/projects/<pid>/<pid>.tsx → results/<id>/results/<pid>.tsx* results/<id>/projects/<pid>/<pid>.json → results/<id>/results/<pid>.json+ * results/<id>/projects/<pid>/<pid>.provenance.json+ * → results/<id>/results/<pid>.provenance.json (when present)*/import * as fs from 'node:fs';@@ -21,41 +23,69 @@ import {fileURLToPath} from 'node:url';const __dirname = path.dirname(fileURLToPath(import.meta.url));const RESULTS_DIR = path.resolve(__dirname, '..', 'results');-const iterationId = process.argv[2];-if (!iterationId) {- console.error('Usage: node collect-results.mjs <iteration-id>');- process.exit(1);-}+export function collectResults(iterDir) {+ const projectsDir = path.join(iterDir, 'projects');+ const resultsDir = path.join(iterDir, 'results');-const iterDir = path.join(RESULTS_DIR, iterationId);-const projectsDir = path.join(iterDir, 'projects');-const resultsDir = path.join(iterDir, 'results');+ if (!fs.existsSync(projectsDir)) {+ throw new Error(`No projects/ directory found at ${projectsDir}`);+ }-if (!fs.existsSync(projectsDir)) {- console.error(`No projects/ directory found at ${projectsDir}`);- process.exit(1);-}+ fs.mkdirSync(resultsDir, {recursive: true});-fs.mkdirSync(resultsDir, {recursive: true});+ let copied = 0;+ let missing = 0;+ const projectIds = fs.readdirSync(projectsDir);-let copied = 0;-let missing = 0;+ for (const pid of projectIds) {+ const projectDir = path.join(projectsDir, pid);+ if (!fs.statSync(projectDir).isDirectory()) continue;-for (const pid of fs.readdirSync(projectsDir)) {- const projectDir = path.join(projectsDir, pid);- if (!fs.statSync(projectDir).isDirectory()) continue;+ for (const ext of ['.tsx', '.json']) {+ const src = path.join(projectDir, `${pid}${ext}`);+ const dest = path.join(resultsDir, `${pid}${ext}`);+ if (fs.existsSync(src)) {+ fs.copyFileSync(src, dest);+ copied++;+ } else {+ missing++;+ console.warn(` ⚠ Missing: ${pid}${ext}`);+ }+ }- for (const ext of ['.tsx', '.json']) {- const src = path.join(projectDir, `${pid}${ext}`);- const dest = path.join(resultsDir, `${pid}${ext}`);- if (fs.existsSync(src)) {- fs.copyFileSync(src, dest);+ const provenanceName = `${pid}.provenance.json`;+ const provenanceSrc = path.join(projectDir, provenanceName);+ if (fs.existsSync(provenanceSrc)) {+ fs.copyFileSync(provenanceSrc, path.join(resultsDir, provenanceName));copied++;- } else {- missing++;- console.warn(` ⚠ Missing: ${pid}${ext}`);}}++ return {copied, missing, projects: projectIds.length};}-console.log(`✓ Collected ${copied} files from ${fs.readdirSync(projectsDir).length} projects${missing ? ` (${missing} missing)` : ''}`);+function main() {+ const iterationId = process.argv[2];+ if (!iterationId) {+ console.error('Usage: node collect-results.mjs <iteration-id>');+ process.exitCode = 1;+ return;+ }++ try {+ const summary = collectResults(path.join(RESULTS_DIR, iterationId));+ console.log(+ `✓ Collected ${summary.copied} files from ${summary.projects} projects${summary.missing ? ` (${summary.missing} missing)` : ''}`,+ );+ } catch (error) {+ console.error(error instanceof Error ? error.message : error);+ process.exitCode = 1;+ }+}++if (+ process.argv[1] &&+ path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)+) {+ main();+}diff --git a/internal/vibe-tests/src/collect-results.test.mjs b/internal/vibe-tests/src/collect-results.test.mjsnew file mode 100644index 0000000000..307e629bf6--- /dev/null+++ b/internal/vibe-tests/src/collect-results.test.mjs@@ -0,0 +1,60 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++import * as fs from 'node:fs';+import * as os from 'node:os';+import * as path from 'node:path';+import {afterEach, describe, expect, it} from 'vitest';+import {collectResults} from './collect-results.mjs';++const tempDirs = [];+afterEach(() => {+ for (const dir of tempDirs.splice(0)) {+ fs.rmSync(dir, {recursive: true, force: true});+ }+});++describe('collectResults', () => {+ it('copies an optional canonical provenance sidecar beside result metadata', () => {+ const iterDir = fs.mkdtempSync(path.join(os.tmpdir(), 'vibe-collect-'));+ tempDirs.push(iterDir);+ const projectDir = path.join(iterDir, 'projects', 'prompt-a');+ fs.mkdirSync(projectDir, {recursive: true});+ fs.writeFileSync(+ path.join(projectDir, 'prompt-a.tsx'),+ 'export default null;',+ );+ fs.writeFileSync(path.join(projectDir, 'prompt-a.json'), '{}');+ fs.writeFileSync(+ path.join(projectDir, 'prompt-a.provenance.json'),+ '{"schemaVersion":1}',+ );++ const summary = collectResults(iterDir);++ expect(summary).toEqual({copied: 3, missing: 0, projects: 1});+ expect(+ fs.readFileSync(+ path.join(iterDir, 'results', 'prompt-a.provenance.json'),+ 'utf-8',+ ),+ ).toBe('{"schemaVersion":1}');+ });++ it('keeps old results without sidecars valid', () => {+ const iterDir = fs.mkdtempSync(path.join(os.tmpdir(), 'vibe-collect-'));+ tempDirs.push(iterDir);+ const projectDir = path.join(iterDir, 'projects', 'prompt-old');+ fs.mkdirSync(projectDir, {recursive: true});+ fs.writeFileSync(+ path.join(projectDir, 'prompt-old.tsx'),+ 'export default null;',+ );+ fs.writeFileSync(path.join(projectDir, 'prompt-old.json'), '{}');++ expect(collectResults(iterDir)).toEqual({+ copied: 2,+ missing: 0,+ projects: 1,+ });+ });+});diff --git a/internal/vibe-tests/src/index.ts b/internal/vibe-tests/src/index.tsindex 18c52bf838..01ebab384d 100644--- a/internal/vibe-tests/src/index.ts+++ b/internal/vibe-tests/src/index.ts@@ -6,4 +6,6 @@*/export * from './types.js';+export * from './provenance.js';+export * from './provenance-matrix.js';export {stratifiedSample} from './sampling.js';diff --git a/internal/vibe-tests/src/provenance-aggregation.test.ts b/internal/vibe-tests/src/provenance-aggregation.test.tsnew file mode 100644index 0000000000..d8cbd24414--- /dev/null+++ b/internal/vibe-tests/src/provenance-aggregation.test.ts@@ -0,0 +1,203 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++import * as fs from 'node:fs';+import * as os from 'node:os';+import * as path from 'node:path';+import {afterEach, describe, expect, it} from 'vitest';+import type {UniversalRunSummary, UniversalScore} from './types';+import {+ buildExecutionBreakdown,+ loadOptionalExecutionProvenance,+ matchesExecutionFilter,+ resolveDuration,+ resolveUsage,+} from './provenance-aggregation';++const tempDirs: string[] = [];+afterEach(() => {+ for (const dir of tempDirs.splice(0)) {+ fs.rmSync(dir, {recursive: true, force: true});+ }+});++function score(value: number): UniversalScore {+ return {+ correctness: {score: value},+ accessibility: {score: value},+ codeQuality: {score: value},+ efficiency: {score: value},+ maintainability: {score: value},+ };+}++function run(+ promptId: string,+ harness: string,+ model: string,+ usageComplete = true,+): UniversalRunSummary {+ return {+ promptId,+ taskId: promptId,+ harness,+ model,+ score: score(80),+ duration: {valueMs: 100, source: 'provenance', quality: 'measured'},+ usage: {+ inputTokens: 10,+ outputTokens: 5,+ source: 'runner',+ quality: usageComplete ? 'complete' : 'incomplete',+ complete: usageComplete,+ },+ };+}++describe('provenance aggregation', () => {+ it('treats an absent sidecar as an explicit estimated fallback', () => {+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vibe-provenance-'));+ tempDirs.push(dir);+ expect(+ loadOptionalExecutionProvenance(path.join(dir, 'missing.json')),+ ).toBe(undefined);+ expect(+ resolveUsage({estimatedInputTokens: 30, estimatedOutputTokens: 20}),+ ).toEqual({+ inputTokens: 30,+ outputTokens: 20,+ source: 'estimate',+ quality: 'estimated',+ complete: false,+ });+ });++ it('fails on malformed present sidecars instead of ignoring them', () => {+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vibe-provenance-'));+ tempDirs.push(dir);+ const file = path.join(dir, 'prompt.provenance.json');+ fs.writeFileSync(file, '{not-json');+ expect(() => loadOptionalExecutionProvenance(file)).toThrow(+ 'Invalid execution provenance in prompt.provenance.json',+ );+ });++ it('prefers sidecar duration and usage over legacy estimates', () => {+ const provenance = {+ schemaVersion: 1 as const,+ execution: {durationMs: 42},+ usage: {+ inputTokens: 7,+ outputTokens: 3,+ source: 'runner',+ complete: true,+ },+ };+ expect(+ resolveDuration({+ provenance,+ legacyStartedAt: '2026-08-25T10:00:00.000Z',+ legacyFinishedAt: '2026-08-25T10:00:10.000Z',+ }),+ ).toEqual({valueMs: 42, source: 'provenance', quality: 'measured'});+ expect(+ resolveUsage({+ provenance,+ estimatedInputTokens: 100,+ estimatedOutputTokens: 50,+ }),+ ).toEqual({+ inputTokens: 7,+ outputTokens: 3,+ source: 'runner',+ quality: 'complete',+ complete: true,+ });+ });++ it('groups the same model across two harnesses without collapsing them', () => {+ const breakdown = buildExecutionBreakdown([+ run('a', 'harness-a', 'model-a'),+ run('b', 'harness-b', 'model-a'),+ ]);+ expect(breakdown.byHarnessModel.map(group => group.dimensions)).toEqual([+ {harness: 'harness-a', model: 'model-a'},+ {harness: 'harness-b', model: 'model-a'},+ ]);+ });++ it('groups the same harness across two models without collapsing them', () => {+ const breakdown = buildExecutionBreakdown([+ run('a', 'harness-a', 'model-a'),+ run('b', 'harness-a', 'model-b'),+ ]);+ expect(breakdown.byHarnessModel.map(group => group.dimensions)).toEqual([+ {harness: 'harness-a', model: 'model-a'},+ {harness: 'harness-a', model: 'model-b'},+ ]);+ expect(breakdown.runs[0]).toMatchObject({+ harness: 'harness-a',+ model: 'model-a',+ });+ });++ it('groups fixture and condition when present', () => {+ const first = run('a', 'harness-a', 'model-a');+ first.fixture = 'fixture-a';+ first.condition = 'adoption';+ const breakdown = buildExecutionBreakdown([first]);+ expect(breakdown.byFixture[0]?.dimensions).toEqual({fixture: 'fixture-a'});+ expect(breakdown.byCondition[0]?.dimensions).toEqual({+ condition: 'adoption',+ });+ });++ it('filters by executor, fixture, and condition labels', () => {+ const provenance = {+ schemaVersion: 1 as const,+ executor: {harness: 'harness-a', model: 'model-a'},+ fixture: {id: 'fixture-a'},+ condition: 'setup',+ };+ expect(+ matchesExecutionFilter(provenance, {+ harness: 'harness-a',+ model: 'model-a',+ fixture: 'fixture-a',+ condition: 'setup',+ }),+ ).toBe(true);+ expect(matchesExecutionFilter(provenance, {model: 'model-b'})).toBe(false);+ expect(matchesExecutionFilter(undefined, {harness: 'unknown'})).toBe(true);+ expect(matchesExecutionFilter(undefined, {model: 'unknown'})).toBe(true);+ });++ it('does not report incomplete usage as comparable totals', () => {+ expect(+ resolveUsage({+ provenance: {+ schemaVersion: 1,+ usage: {+ inputTokens: 10,+ outputTokens: 5,+ source: 'runner',+ complete: false,+ },+ },+ estimatedInputTokens: 100,+ estimatedOutputTokens: 50,+ }),+ ).toMatchObject({quality: 'incomplete', complete: false});++ const breakdown = buildExecutionBreakdown([+ run('a', 'harness-a', 'model-a', true),+ run('b', 'harness-a', 'model-a', false),+ ]);+ expect(breakdown.byHarnessModel[0]?.usage).toEqual({+ complete: false,+ completeRuns: 1,+ incompleteRuns: 1,+ inputTokens: null,+ outputTokens: null,+ });+ });+});diff --git a/internal/vibe-tests/src/provenance-aggregation.ts b/internal/vibe-tests/src/provenance-aggregation.tsnew file mode 100644index 0000000000..cd4f776175--- /dev/null+++ b/internal/vibe-tests/src/provenance-aggregation.ts@@ -0,0 +1,240 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file Provenance-aware fallback and grouping helpers for universal aggregation.+ * @position internal/vibe-tests/src/provenance-aggregation.ts+ */++import * as fs from 'node:fs';+import * as path from 'node:path';+import type {+ DurationMetric,+ ExecutionBreakdown,+ ExecutionGroupSummary,+ ExecutionProvenanceFilter,+ TokenMetric,+ UniversalDimension,+ UniversalRunSummary,+ UniversalScore,+} from './types.js';+import {+ parseExecutionProvenanceV1,+ type ExecutionProvenanceV1,+} from './provenance.js';++export function loadOptionalExecutionProvenance(+ filePath: string,+): ExecutionProvenanceV1 | undefined {+ if (!fs.existsSync(filePath)) {+ return undefined;+ }++ let value: unknown;+ try {+ value = JSON.parse(fs.readFileSync(filePath, 'utf-8'));+ } catch (error) {+ throw new Error(+ `Invalid execution provenance in ${path.basename(filePath)}: ${String(error)}`,+ {cause: error},+ );+ }++ try {+ return parseExecutionProvenanceV1(value);+ } catch (error) {+ throw new Error(+ `Invalid execution provenance in ${path.basename(filePath)}: ${String(error)}`,+ {cause: error},+ );+ }+}++export function matchesExecutionFilter(+ provenance: ExecutionProvenanceV1 | undefined,+ filter: ExecutionProvenanceFilter,+): boolean {+ const harness = provenance?.executor?.harness ?? 'unknown';+ const model = provenance?.executor?.model ?? 'unknown';+ return (+ (!filter.harness || harness === filter.harness) &&+ (!filter.model || model === filter.model) &&+ (!filter.fixture || provenance?.fixture?.id === filter.fixture) &&+ (!filter.condition || provenance?.condition === filter.condition)+ );+}++export function resolveDuration(options: {+ provenance?: ExecutionProvenanceV1;+ legacyStartedAt?: string;+ legacyFinishedAt?: string;+ taskMtimeMs?: number;+ resultMtimeMs?: number;+}): DurationMetric {+ const explicit = options.provenance?.execution?.durationMs;+ if (explicit !== undefined) {+ return {valueMs: explicit, source: 'provenance', quality: 'measured'};+ }++ const provenanceStartedAt = options.provenance?.execution?.startedAt;+ const provenanceFinishedAt = options.provenance?.execution?.finishedAt;+ if (provenanceStartedAt && provenanceFinishedAt) {+ const valueMs =+ new Date(provenanceFinishedAt).getTime() -+ new Date(provenanceStartedAt).getTime();+ if (valueMs >= 0) {+ return {valueMs, source: 'provenance-timestamps', quality: 'derived'};+ }+ }++ if (options.legacyStartedAt && options.legacyFinishedAt) {+ const valueMs =+ new Date(options.legacyFinishedAt).getTime() -+ new Date(options.legacyStartedAt).getTime();+ if (Number.isFinite(valueMs) && valueMs >= 0) {+ return {valueMs, source: 'result-metadata', quality: 'derived'};+ }+ }++ if (+ options.taskMtimeMs !== undefined &&+ options.resultMtimeMs !== undefined &&+ options.resultMtimeMs >= options.taskMtimeMs+ ) {+ return {+ valueMs: Math.round(options.resultMtimeMs - options.taskMtimeMs),+ source: 'filesystem',+ quality: 'estimated',+ };+ }++ return {valueMs: null, source: 'unavailable', quality: 'unavailable'};+}++export function resolveUsage(options: {+ provenance?: ExecutionProvenanceV1;+ estimatedInputTokens: number;+ estimatedOutputTokens: number;+}): TokenMetric {+ const usage = options.provenance?.usage;+ if (usage) {+ return {+ inputTokens: usage.inputTokens ?? null,+ outputTokens: usage.outputTokens ?? null,+ source: usage.source ?? 'provenance',+ quality: usage.complete ? 'complete' : 'incomplete',+ complete: usage.complete,+ };+ }++ return {+ inputTokens: options.estimatedInputTokens,+ outputTokens: options.estimatedOutputTokens,+ source: 'estimate',+ quality: 'estimated',+ complete: false,+ };+}++function averageScore(score: UniversalScore): number {+ const dimensions: UniversalDimension[] = [+ 'correctness',+ 'accessibility',+ 'codeQuality',+ 'efficiency',+ 'maintainability',+ 'design',+ ];+ const values = dimensions+ .map(dimension => score[dimension]?.score)+ .filter((value): value is number => value !== undefined);+ return values.length === 0+ ? 0+ : Math.round(values.reduce((sum, value) => sum + value, 0) / values.length);+}++function summarizeGroup(+ dimensions: ExecutionGroupSummary['dimensions'],+ runs: UniversalRunSummary[],+): ExecutionGroupSummary {+ const durationValues = runs+ .map(run => run.duration.valueMs)+ .filter((value): value is number => value !== null);+ const completeUsageRuns = runs.filter(run => run.usage.complete).length;+ const usageComplete = completeUsageRuns === runs.length;+ return {+ dimensions,+ runCount: runs.length,+ averageScore: Math.round(+ runs.reduce((sum, run) => sum + averageScore(run.score), 0) / runs.length,+ ),+ averageDurationMs:+ durationValues.length > 0+ ? Math.round(+ durationValues.reduce((sum, value) => sum + value, 0) /+ durationValues.length,+ )+ : null,+ usage: {+ complete: usageComplete,+ completeRuns: completeUsageRuns,+ incompleteRuns: runs.length - completeUsageRuns,+ inputTokens: usageComplete+ ? runs.reduce((sum, run) => sum + (run.usage.inputTokens ?? 0), 0)+ : null,+ outputTokens: usageComplete+ ? runs.reduce((sum, run) => sum + (run.usage.outputTokens ?? 0), 0)+ : null,+ },+ };+}++function groupRuns(+ runs: UniversalRunSummary[],+ dimensionsFor: (+ run: UniversalRunSummary,+ ) => ExecutionGroupSummary['dimensions'] | undefined,+): ExecutionGroupSummary[] {+ const groups = new Map<+ string,+ {+ dimensions: ExecutionGroupSummary['dimensions'];+ runs: UniversalRunSummary[];+ }+ >();+ for (const run of runs) {+ const dimensions = dimensionsFor(run);+ if (!dimensions) {+ continue;+ }+ const key = JSON.stringify(dimensions);+ const existing = groups.get(key);+ if (existing) {+ existing.runs.push(run);+ } else {+ groups.set(key, {dimensions, runs: [run]});+ }+ }+ return [...groups.values()]+ .map(group => summarizeGroup(group.dimensions, group.runs))+ .sort((a, b) =>+ JSON.stringify(a.dimensions).localeCompare(JSON.stringify(b.dimensions)),+ );+}++export function buildExecutionBreakdown(+ runs: UniversalRunSummary[],+): ExecutionBreakdown {+ return {+ runs,+ byHarnessModel: groupRuns(runs, run => ({+ harness: run.harness,+ model: run.model,+ })),+ byFixture: groupRuns(runs, run =>+ run.fixture ? {fixture: run.fixture} : undefined,+ ),+ byCondition: groupRuns(runs, run =>+ run.condition ? {condition: run.condition} : undefined,+ ),+ };+}diff --git a/internal/vibe-tests/src/provenance-matrix.test.ts b/internal/vibe-tests/src/provenance-matrix.test.tsnew file mode 100644index 0000000000..b6e5640263--- /dev/null+++ b/internal/vibe-tests/src/provenance-matrix.test.ts@@ -0,0 +1,18 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++import {describe, expect, it} from 'vitest';+import {expandExecutionMatrix} from './provenance-matrix';++describe('expandExecutionMatrix', () => {+ it('enumerates executor, condition, and repetition dimensions generically', () => {+ expect(+ expandExecutionMatrix({+ harnesses: ['harness-a', 'harness-b'],+ models: ['model-a'],+ efforts: ['standard'],+ conditions: ['setup', 'adoption'],+ reps: 2,+ }),+ ).toHaveLength(8);+ });+});diff --git a/internal/vibe-tests/src/provenance-matrix.ts b/internal/vibe-tests/src/provenance-matrix.tsnew file mode 100644index 0000000000..85807023db--- /dev/null+++ b/internal/vibe-tests/src/provenance-matrix.ts@@ -0,0 +1,60 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file Generic matrix expansion for provenance-aware vibe-test runners.+ * @position internal/vibe-tests/src/provenance-matrix.ts+ */++export interface ExecutionMatrixConfig {+ harnesses: readonly string[];+ models: readonly string[];+ efforts?: readonly string[];+ conditions?: readonly string[];+ reps?: number;+}++export interface ExecutionMatrixEntry {+ harness: string;+ model: string;+ effort?: string;+ condition?: string;+ rep: number;+}++export function expandExecutionMatrix(+ config: ExecutionMatrixConfig,+): ExecutionMatrixEntry[] {+ if (config.harnesses.length === 0 || config.models.length === 0) {+ throw new Error(+ 'Execution matrices require at least one harness and model',+ );+ }+ const efforts = config.efforts?.length ? config.efforts : [undefined];+ const conditions = config.conditions?.length+ ? config.conditions+ : [undefined];+ const reps = config.reps ?? 1;+ if (!Number.isInteger(reps) || reps < 1) {+ throw new Error('Execution matrix reps must be a positive integer');+ }++ const entries: ExecutionMatrixEntry[] = [];+ for (const harness of config.harnesses) {+ for (const model of config.models) {+ for (const effort of efforts) {+ for (const condition of conditions) {+ for (let rep = 1; rep <= reps; rep++) {+ entries.push({+ harness,+ model,+ ...(effort ? {effort} : {}),+ ...(condition ? {condition} : {}),+ rep,+ });+ }+ }+ }+ }+ }+ return entries;+}diff --git a/internal/vibe-tests/src/provenance.test.ts b/internal/vibe-tests/src/provenance.test.tsnew file mode 100644index 0000000000..204acf8a5a--- /dev/null+++ b/internal/vibe-tests/src/provenance.test.ts@@ -0,0 +1,82 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++import {describe, expect, it} from 'vitest';+import {+ ExecutionProvenanceValidationError,+ parseExecutionProvenanceV1,+ provenanceFilename,+} from './provenance';++const SHA = 'a'.repeat(64);++describe('ExecutionProvenanceV1', () => {+ it('parses a full sidecar and preserves unknown fields', () => {+ const value = parseExecutionProvenanceV1({+ schemaVersion: 1,+ task: {id: 'prompt-a', sha256: SHA, futureTaskField: true},+ fixture: {id: 'fixture-a', sha256: SHA, commit: 'abc123'},+ condition: 'setup',+ rep: 2,+ executor: {+ harness: 'shell-runner',+ model: 'model-a',+ effort: 'high',+ harnessVersion: '1.2.3',+ runnerVersion: '4.5.6',+ futureExecutorField: 'kept',+ },+ execution: {+ status: 'succeeded',+ startedAt: '2026-08-25T10:00:00.000Z',+ finishedAt: '2026-08-25T10:00:05.000Z',+ durationMs: 5000,+ attempt: 2,+ retry: 1,+ },+ usage: {+ inputTokens: 120,+ outputTokens: 45,+ source: 'runner',+ complete: true,+ },+ environmentHash: SHA,+ toolPolicyHash: SHA,+ futureTopLevelField: {enabled: true},+ });++ expect(value.executor?.harness).toBe('shell-runner');+ expect(value.futureTopLevelField).toEqual({enabled: true});+ expect(value.executor?.futureExecutorField).toBe('kept');+ });++ it('accepts a partial sidecar and canonicalizes its filename', () => {+ expect(parseExecutionProvenanceV1({schemaVersion: 1})).toEqual({+ schemaVersion: 1,+ });+ expect(provenanceFilename('prompt-a')).toBe('prompt-a.provenance.json');+ });++ it('rejects malformed present fields', () => {+ expect(() =>+ parseExecutionProvenanceV1({schemaVersion: 1, rep: 0}),+ ).toThrow(ExecutionProvenanceValidationError);+ expect(() =>+ parseExecutionProvenanceV1({+ schemaVersion: 1,+ execution: {startedAt: '2026-08-25'},+ }),+ ).toThrow('must be a valid date-time string');+ expect(() =>+ parseExecutionProvenanceV1({+ schemaVersion: 1,+ usage: {complete: true, inputTokens: 1},+ }),+ ).toThrow('complete usage requires both inputTokens and outputTokens');+ });++ it('rejects unknown schema versions clearly', () => {+ expect(() => parseExecutionProvenanceV1({schemaVersion: 2})).toThrow(+ 'unsupported provenance schema version 2; expected 1',+ );+ });+});diff --git a/internal/vibe-tests/src/provenance.ts b/internal/vibe-tests/src/provenance.tsnew file mode 100644index 0000000000..48fee21486--- /dev/null+++ b/internal/vibe-tests/src/provenance.ts@@ -0,0 +1,253 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file Versioned, executor-neutral provenance contract for vibe-test results.+ * @position internal/vibe-tests/src/provenance.ts+ */++export interface ExecutionProvenanceTaskV1 {+ id: string;+ sha256: string;+ [key: string]: unknown;+}++export interface ExecutionProvenanceFixtureV1 {+ id: string;+ sha256?: string;+ commit?: string;+ [key: string]: unknown;+}++export interface ExecutionProvenanceExecutorV1 {+ harness: string;+ model: string;+ effort?: string;+ harnessVersion?: string;+ runnerVersion?: string;+ [key: string]: unknown;+}++export interface ExecutionProvenanceExecutionV1 {+ status?: string;+ startedAt?: string;+ finishedAt?: string;+ durationMs?: number;+ attempt?: number;+ retry?: number;+ [key: string]: unknown;+}++export interface ExecutionProvenanceUsageV1 {+ inputTokens?: number;+ outputTokens?: number;+ source?: string;+ complete: boolean;+ [key: string]: unknown;+}++export interface ExecutionProvenanceV1 {+ schemaVersion: 1;+ task?: ExecutionProvenanceTaskV1;+ fixture?: ExecutionProvenanceFixtureV1;+ condition?: string;+ rep?: number;+ executor?: ExecutionProvenanceExecutorV1;+ execution?: ExecutionProvenanceExecutionV1;+ usage?: ExecutionProvenanceUsageV1;+ environmentHash?: string;+ toolPolicyHash?: string;+ [key: string]: unknown;+}++export class ExecutionProvenanceValidationError extends Error {+ constructor(+ message: string,+ readonly field = '$',+ ) {+ super(`${field}: ${message}`);+ this.name = 'ExecutionProvenanceValidationError';+ }+}++const SHA256 = /^[a-fA-F0-9]{64}$/;+const RFC3339_DATE_TIME =+ /^\d{4}-\d{2}-\d{2}[Tt]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:[Zz]|[+-]\d{2}:\d{2})$/;++function fail(field: string, message: string): never {+ throw new ExecutionProvenanceValidationError(message, field);+}++function objectAt(value: unknown, field: string): Record<string, unknown> {+ if (value == null || typeof value !== 'object' || Array.isArray(value)) {+ fail(field, 'must be an object');+ }+ return value as Record<string, unknown>;+}++function optionalString(+ object: Record<string, unknown>,+ key: string,+ field: string,+): void {+ const value = object[key];+ if (+ value !== undefined &&+ (typeof value !== 'string' || value.length === 0)+ ) {+ fail(`${field}.${key}`, 'must be a non-empty string');+ }+}++function requiredString(+ object: Record<string, unknown>,+ key: string,+ field: string,+): void {+ optionalString(object, key, field);+ if (object[key] === undefined) {+ fail(`${field}.${key}`, 'is required');+ }+}++function optionalInteger(+ object: Record<string, unknown>,+ key: string,+ field: string,+ minimum: number,+): void {+ const value = object[key];+ if (+ value !== undefined &&+ (typeof value !== 'number' || !Number.isInteger(value) || value < minimum)+ ) {+ fail(+ `${field}.${key}`,+ `must be an integer greater than or equal to ${minimum}`,+ );+ }+}++function optionalNumber(+ object: Record<string, unknown>,+ key: string,+ field: string,+): void {+ const value = object[key];+ if (+ value !== undefined &&+ (typeof value !== 'number' || !Number.isFinite(value) || value < 0)+ ) {+ fail(`${field}.${key}`, 'must be a non-negative finite number');+ }+}++function optionalSha256(+ object: Record<string, unknown>,+ key: string,+ field: string,+): void {+ const value = object[key];+ if (+ value !== undefined &&+ (typeof value !== 'string' || !SHA256.test(value))+ ) {+ fail(+ `${field}.${key}`,+ 'must be a 64-character hexadecimal SHA-256 digest',+ );+ }+}++function optionalDateTime(+ object: Record<string, unknown>,+ key: string,+ field: string,+): void {+ const value = object[key];+ if (+ value !== undefined &&+ (typeof value !== 'string' ||+ !RFC3339_DATE_TIME.test(value) ||+ Number.isNaN(Date.parse(value)))+ ) {+ fail(`${field}.${key}`, 'must be a valid date-time string');+ }+}++export function parseExecutionProvenanceV1(+ input: unknown,+): ExecutionProvenanceV1 {+ const value = objectAt(input, '$');+ if (value.schemaVersion !== 1) {+ if (value.schemaVersion === undefined) {+ fail('$.schemaVersion', 'is required');+ }+ fail(+ '$.schemaVersion',+ `unsupported provenance schema version ${String(value.schemaVersion)}; expected 1`,+ );+ }++ if (value.task !== undefined) {+ const task = objectAt(value.task, '$.task');+ requiredString(task, 'id', '$.task');+ requiredString(task, 'sha256', '$.task');+ optionalSha256(task, 'sha256', '$.task');+ }++ if (value.fixture !== undefined) {+ const fixture = objectAt(value.fixture, '$.fixture');+ requiredString(fixture, 'id', '$.fixture');+ optionalSha256(fixture, 'sha256', '$.fixture');+ optionalString(fixture, 'commit', '$.fixture');+ }++ optionalString(value, 'condition', '$');+ optionalInteger(value, 'rep', '$', 1);++ if (value.executor !== undefined) {+ const executor = objectAt(value.executor, '$.executor');+ requiredString(executor, 'harness', '$.executor');+ requiredString(executor, 'model', '$.executor');+ optionalString(executor, 'effort', '$.executor');+ optionalString(executor, 'harnessVersion', '$.executor');+ optionalString(executor, 'runnerVersion', '$.executor');+ }++ if (value.execution !== undefined) {+ const execution = objectAt(value.execution, '$.execution');+ optionalString(execution, 'status', '$.execution');+ optionalDateTime(execution, 'startedAt', '$.execution');+ optionalDateTime(execution, 'finishedAt', '$.execution');+ optionalNumber(execution, 'durationMs', '$.execution');+ optionalInteger(execution, 'attempt', '$.execution', 1);+ optionalInteger(execution, 'retry', '$.execution', 0);+ }++ if (value.usage !== undefined) {+ const usage = objectAt(value.usage, '$.usage');+ optionalInteger(usage, 'inputTokens', '$.usage', 0);+ optionalInteger(usage, 'outputTokens', '$.usage', 0);+ optionalString(usage, 'source', '$.usage');+ if (typeof usage.complete !== 'boolean') {+ fail('$.usage.complete', 'must be a boolean');+ }+ if (+ usage.complete &&+ (usage.inputTokens === undefined || usage.outputTokens === undefined)+ ) {+ fail(+ '$.usage',+ 'complete usage requires both inputTokens and outputTokens',+ );+ }+ }++ optionalSha256(value, 'environmentHash', '$');+ optionalSha256(value, 'toolPolicyHash', '$');+ return value as unknown as ExecutionProvenanceV1;+}++export function provenanceFilename(promptId: string): string {+ return `${promptId}.provenance.json`;+}diff --git a/internal/vibe-tests/src/types.ts b/internal/vibe-tests/src/types.tsindex dd2b25a1b2..6efd500bd2 100644--- a/internal/vibe-tests/src/types.ts+++ b/internal/vibe-tests/src/types.ts@@ -78,10 +78,7 @@ export interface TestResult {}export type RefinementTarget =- | 'skill_doc'- | 'component_api'- | 'component_naming'- | 'examples';+ 'skill_doc' | 'component_api' | 'component_naming' | 'examples';export type EffortEstimate = 'trivial' | 'moderate' | 'significant';@@ -182,10 +179,7 @@ export interface DesignSystemIssue {export interface CodeQualityIssue {severity: 'critical' | 'moderate' | 'minor';category:- | 'state-management'- | 'event-handling'- | 'typescript'- | 'performance';+ 'state-management' | 'event-handling' | 'typescript' | 'performance';issue: string;recommendation: string;codeSnippet?: string;@@ -287,25 +281,108 @@ export interface UniversalScore {design?: DimensionScore<DesignMetrics> | null;}+export type MetricQuality =+ 'measured' | 'derived' | 'estimated' | 'unavailable';++export interface DurationMetric {+ valueMs: number | null;+ source:+ | 'provenance'+ | 'provenance-timestamps'+ | 'result-metadata'+ | 'filesystem'+ | 'unavailable';+ quality: MetricQuality;+}++export interface TokenMetric {+ inputTokens: number | null;+ outputTokens: number | null;+ source: string;+ quality: 'complete' | 'incomplete' | 'estimated';+ complete: boolean;+}++export interface ExecutionProvenanceFilter {+ harness?: string;+ model?: string;+ fixture?: string;+ condition?: string;+}++export interface UniversalRunSummary {+ promptId: string;+ taskId: string;+ harness: string;+ model: string;+ effort?: string;+ fixture?: string;+ condition?: string;+ rep?: number;+ executionStatus?: string;+ score: UniversalScore;+ duration: DurationMetric;+ usage: TokenMetric;+}++export interface ExecutionGroupSummary {+ dimensions: {+ harness?: string;+ model?: string;+ fixture?: string;+ condition?: string;+ };+ runCount: number;+ averageScore: number;+ averageDurationMs: number | null;+ usage: {+ complete: boolean;+ completeRuns: number;+ incompleteRuns: number;+ inputTokens: number | null;+ outputTokens: number | null;+ };+}++export interface ExecutionBreakdown {+ runs: UniversalRunSummary[];+ byHarnessModel: ExecutionGroupSummary[];+ byFixture: ExecutionGroupSummary[];+ byCondition: ExecutionGroupSummary[];+}++export interface PromptCostMetrics {+ durationMs: number;+ durationSource: DurationMetric['source'];+ durationQuality: MetricQuality;+ outputChars: number;+ outputLines: number;+ docsRead: string[];+ inputTokens: number | null;+ outputTokens: number | null;+ tokenSource: string;+ tokenQuality: TokenMetric['quality'];+ usageComplete: boolean;+ /** Legacy estimates retained independently of selected provenance usage. */+ estimatedInputTokens: number;+ estimatedOutputTokens: number;+}+export interface CostMetrics {totalDurationMs: number;avgDurationMs: number;avgOutputChars: number;avgOutputLines: number;avgDocsRead: number;+ inputTokens: number | null;+ outputTokens: number | null;+ usageComplete: boolean;+ completeUsageRuns: number;+ incompleteUsageRuns: number;+ durationSources: Record<string, number>;estimatedInputTokens: number;estimatedOutputTokens: number;- byPrompt: Record<- string,- {- durationMs: number;- outputChars: number;- outputLines: number;- docsRead: string[];- estimatedInputTokens: number;- estimatedOutputTokens: number;- }- >;+ byPrompt: Record<string, PromptCostMetrics>;}export interface UniversalAggregate {@@ -315,6 +392,7 @@ export interface UniversalAggregate {byCategory: Record<string, Record<UniversalDimension, number>>;darkModeRate: number;cost?: CostMetrics;+ execution?: ExecutionBreakdown;}export type TargetName = 'astryx' | 'astryx-tailwind' | 'baseline' | 'html';diff --git a/internal/vibe-tests/src/universal-aggregate.ts b/internal/vibe-tests/src/universal-aggregate.tsindex 52d90c28e1..4bdd43331f 100644--- a/internal/vibe-tests/src/universal-aggregate.ts+++ b/internal/vibe-tests/src/universal-aggregate.ts@@ -14,12 +14,23 @@import * as fs from 'node:fs';import * as path from 'node:path';import type {+ ExecutionProvenanceFilter,+ PromptCostMetrics,UniversalDimension,+ UniversalRunSummary,UniversalScore,UniversalAggregate,} from './types.js';import {getResultsDir, writeJson, ensureTsxFiles} from './utils.js';import {evaluate, getDimensionNames} from './universal-eval.js';+import {provenanceFilename} from './provenance.js';+import {+ buildExecutionBreakdown,+ loadOptionalExecutionProvenance,+ matchesExecutionFilter,+ resolveDuration,+ resolveUsage,+} from './provenance-aggregation.js';const DIMENSION_LABELS: Partial<Record<UniversalDimension, string>> = {correctness: 'Correctness',@@ -30,10 +41,15 @@ const DIMENSION_LABELS: Partial<Record<UniversalDimension, string>> = {design: 'Design',};-function parseArgs(): {iteration: string; json: boolean} {+function parseArgs(): {+ iteration: string;+ json: boolean;+ filter: ExecutionProvenanceFilter;+} {const args = process.argv.slice(2);let iteration = '';let json = false;+ const filter: ExecutionProvenanceFilter = {};for (let i = 0; i < args.length; i++) {if (args[i] === '--iteration' && args[i + 1]) {@@ -41,21 +57,29 @@ function parseArgs(): {iteration: string; json: boolean} {i++;} else if (args[i] === '--json') {json = true;+ } else if (args[i] === '--harness' && args[i + 1]) {+ filter.harness = args[++i];+ } else if (args[i] === '--model' && args[i + 1]) {+ filter.model = args[++i];+ } else if (args[i] === '--fixture' && args[i + 1]) {+ filter.fixture = args[++i];+ } else if (args[i] === '--condition' && args[i + 1]) {+ filter.condition = args[++i];}}if (!iteration) {console.error(- 'Usage: tsx src/universal-aggregate.ts --iteration <id> [--json]',+ 'Usage: tsx src/universal-aggregate.ts --iteration <id> [--json] [--harness <label>] [--model <label>] [--fixture <id>] [--condition <label>]',);process.exit(1);}- return {iteration, json};+ return {iteration, json, filter};}async function main() {- const {iteration, json} = parseArgs();+ const {iteration, json, filter} = parseArgs();const resultsDir = getResultsDir();const iterDir = path.join(resultsDir, iteration);const codeDir = path.join(iterDir, 'results');@@ -97,18 +121,9 @@ async function main() {> = {};let darkModeCount = 0;- // Cost tracking- const costByPrompt: Record<- string,- {- durationMs: number;- outputChars: number;- outputLines: number;- docsRead: string[];- estimatedInputTokens: number;- estimatedOutputTokens: number;- }- > = {};+ // Cost and provenance tracking+ const costByPrompt: Record<string, PromptCostMetrics> = {};+ const runs: UniversalRunSummary[] = [];/** Rough doc size estimates in chars (for input token calculation) */const DOC_CHAR_SIZES: Record<string, number> = {@@ -122,6 +137,12 @@ async function main() {for (const file of files) {const promptId = path.basename(file, '.tsx');const codePath = path.join(codeDir, file);+ const provenance = loadOptionalExecutionProvenance(+ path.join(codeDir, provenanceFilename(promptId)),+ );+ if (!matchesExecutionFilter(provenance, filter)) {+ continue;+ }const code = fs.readFileSync(codePath, 'utf-8');const score = evaluate(code, target, {iterDir, promptId});byPrompt[promptId] = score;@@ -148,67 +169,90 @@ async function main() {}}- // --- Cost data ---- let durationMs = 0;+ // --- Cost and execution data ---let docsRead: string[] = [];-- // Read result metadata (docsRead, completedAt)- const jsonPath = path.join(codeDir, `${promptId}.json`);let completedAt: string | undefined;+ const jsonPath = path.join(codeDir, `${promptId}.json`);if (fs.existsSync(jsonPath)) {try {const meta = JSON.parse(fs.readFileSync(jsonPath, 'utf-8'));- docsRead = meta.docsRead || [];+ docsRead = Array.isArray(meta.docsRead) ? meta.docsRead : [];completedAt = meta.completedAt;} catch {- // ignore parse errors+ // Legacy metadata remains best-effort. Provenance sidecars are validated.}}- // Duration: prefer createdAt/completedAt timestamps, fall back to file mtimesconst taskPath = path.join(iterDir, 'tasks', `${promptId}.json`);+ let createdAt: string | undefined;+ let taskMtimeMs: number | undefined;if (fs.existsSync(taskPath)) {+ taskMtimeMs = fs.statSync(taskPath).mtimeMs;try {const taskMeta = JSON.parse(fs.readFileSync(taskPath, 'utf-8'));- if (taskMeta.createdAt && completedAt) {- const start = new Date(taskMeta.createdAt).getTime();- const end = new Date(completedAt).getTime();- if (end > start) {- durationMs = Math.round(end - start);- }- }+ createdAt = taskMeta.createdAt;} catch {- // ignore- }-- // Fall back to file timestamps if no explicit timestamps- if (durationMs === 0) {- const taskStat = fs.statSync(taskPath);- const resultStat = fs.statSync(codePath);- const inferred = resultStat.mtimeMs - taskStat.mtimeMs;- if (inferred > 0) {- durationMs = Math.round(inferred);- }+ // Legacy task metadata remains best-effort.}}- // Estimate input tokens from docs read- let estimatedInputChars = 0;+ let estimatedInputChars = 1500;for (const doc of docsRead) {const key = doc.endsWith('.md') ? doc : `${doc}.md`;estimatedInputChars += DOC_CHAR_SIZES[key] || DEFAULT_DOC_SIZE;}- // Add prompt overhead (~1500 chars)- estimatedInputChars += 1500;+ const estimatedInputTokens = Math.round(estimatedInputChars / 4);+ const estimatedOutputTokens = Math.round(code.length / 4);+ const duration = resolveDuration({+ provenance,+ legacyStartedAt: createdAt,+ legacyFinishedAt: completedAt,+ taskMtimeMs,+ resultMtimeMs: fs.statSync(codePath).mtimeMs,+ });+ const usage = resolveUsage({+ provenance,+ estimatedInputTokens,+ estimatedOutputTokens,+ });costByPrompt[promptId] = {- durationMs,+ durationMs: duration.valueMs ?? 0,+ durationSource: duration.source,+ durationQuality: duration.quality,outputChars: code.length,outputLines: code.split('\n').length,docsRead,- estimatedInputTokens: Math.round(estimatedInputChars / 4),- estimatedOutputTokens: Math.round(code.length / 4),+ inputTokens: usage.inputTokens,+ outputTokens: usage.outputTokens,+ tokenSource: usage.source,+ tokenQuality: usage.quality,+ usageComplete: usage.complete,+ estimatedInputTokens,+ estimatedOutputTokens,};+ runs.push({+ promptId,+ taskId: provenance?.task?.id ?? promptId,+ harness: provenance?.executor?.harness ?? 'unknown',+ model: provenance?.executor?.model ?? 'unknown',+ ...(provenance?.executor?.effort+ ? {effort: provenance.executor.effort}+ : {}),+ ...(provenance?.fixture?.id ? {fixture: provenance.fixture.id} : {}),+ ...(provenance?.condition ? {condition: provenance.condition} : {}),+ ...(provenance?.rep ? {rep: provenance.rep} : {}),+ ...(provenance?.execution?.status+ ? {executionStatus: provenance.execution.status}+ : {}),+ score,+ duration,+ usage,+ });+ }++ if (runs.length === 0) {+ throw new Error('No results matched the requested provenance filters');}// --- Merge design scores if available ---@@ -305,20 +349,34 @@ async function main() {const darkModeRate = Math.round((darkModeCount / promptCount) * 100);- // Compute cost aggregates+ // Compute cost aggregates. Selected token totals are only comparable when+ // every run reports complete usage; estimates remain available separately.const costEntries = Object.values(costByPrompt);const totalDurationMs = costEntries.reduce((s, c) => s + c.durationMs, 0);const totalOutputChars = costEntries.reduce((s, c) => s + c.outputChars, 0);const totalOutputLines = costEntries.reduce((s, c) => s + c.outputLines, 0);const totalDocsRead = costEntries.reduce((s, c) => s + c.docsRead.length, 0);- const totalInputTokens = costEntries.reduce(+ const completeUsageRuns = costEntries.filter(c => c.usageComplete).length;+ const usageComplete = completeUsageRuns === costEntries.length;+ const selectedInputTokens = usageComplete+ ? costEntries.reduce((s, c) => s + (c.inputTokens ?? 0), 0)+ : null;+ const selectedOutputTokens = usageComplete+ ? costEntries.reduce((s, c) => s + (c.outputTokens ?? 0), 0)+ : null;+ const totalEstimatedInputTokens = costEntries.reduce((s, c) => s + c.estimatedInputTokens,0,);- const totalOutputTokens = costEntries.reduce(+ const totalEstimatedOutputTokens = costEntries.reduce((s, c) => s + c.estimatedOutputTokens,0,);+ const durationSources: Record<string, number> = {};+ for (const entry of costEntries) {+ durationSources[entry.durationSource] =+ (durationSources[entry.durationSource] ?? 0) + 1;+ }const aggregate: UniversalAggregate = {averages,@@ -332,10 +390,17 @@ async function main() {avgOutputChars: Math.round(totalOutputChars / promptCount),avgOutputLines: Math.round(totalOutputLines / promptCount),avgDocsRead: Math.round((totalDocsRead / promptCount) * 10) / 10,- estimatedInputTokens: totalInputTokens,- estimatedOutputTokens: totalOutputTokens,+ inputTokens: selectedInputTokens,+ outputTokens: selectedOutputTokens,+ usageComplete,+ completeUsageRuns,+ incompleteUsageRuns: costEntries.length - completeUsageRuns,+ durationSources,+ estimatedInputTokens: totalEstimatedInputTokens,+ estimatedOutputTokens: totalEstimatedOutputTokens,byPrompt: costByPrompt,},+ execution: buildExecutionBreakdown(runs),};// Save@@ -402,7 +467,7 @@ async function main() {}// Cost metrics- if (aggregate.cost && aggregate.cost.totalDurationMs > 0) {+ if (aggregate.cost) {const c = aggregate.cost;console.log(`\n💰 Cost:`);console.log(@@ -412,9 +477,48 @@ async function main() {` Output: ${c.avgOutputLines} lines avg (${c.avgOutputChars} chars)`,);console.log(` Docs read: ${c.avgDocsRead} avg per prompt`);- console.log(- ` Tokens: ~${c.estimatedInputTokens} input, ~${c.estimatedOutputTokens} output`,- );+ if (c.usageComplete) {+ console.log(+ ` Tokens: ${c.inputTokens ?? 0} input, ${c.outputTokens ?? 0} output (complete)`,+ );+ } else {+ console.log(+ ` Tokens: not comparable (${c.completeUsageRuns}/${promptCount} runs complete); estimates: ~${c.estimatedInputTokens} input, ~${c.estimatedOutputTokens} output`,+ );+ }+ }++ const harnessModelGroups = aggregate.execution?.byHarnessModel ?? [];+ if (harnessModelGroups.length > 0) {+ console.log('\n⚙️ By Harness + Model:');+ for (const group of harnessModelGroups) {+ const {harness = 'unknown', model = 'unknown'} = group.dimensions;+ const usage = group.usage.complete+ ? `${(group.usage.inputTokens ?? 0) + (group.usage.outputTokens ?? 0)} tokens`+ : `usage ${group.usage.completeRuns}/${group.runCount} complete`;+ console.log(+ ` ${harness} / ${model}: ${group.runCount} runs, score ${group.averageScore}, ${usage}`,+ );+ }+ }++ for (const [label, groups] of [+ ['Fixture', aggregate.execution?.byFixture ?? []],+ ['Condition', aggregate.execution?.byCondition ?? []],+ ] as const) {+ if (groups.length === 0) {+ continue;+ }+ console.log(`\n📎 By ${label}:`);+ for (const group of groups) {+ const value =+ label === 'Fixture'+ ? group.dimensions.fixture+ : group.dimensions.condition;+ console.log(+ ` ${value}: ${group.runCount} runs, score ${group.averageScore}`,+ );+ }}// Category breakdowndiff --git a/internal/vibe-tests/src/universal-compare.ts b/internal/vibe-tests/src/universal-compare.tsindex b66126b0d1..a96a083890 100644--- a/internal/vibe-tests/src/universal-compare.ts+++ b/internal/vibe-tests/src/universal-compare.ts@@ -527,13 +527,28 @@ async function main() {t => `${t.label} ${t.data.cost?.avgDocsRead ?? 0}`,);console.log(` Avg docs read: ${docsParts.join(' | ')}`);- const tokenParts = costTargets.map(t => {- const total =- (t.data.cost?.estimatedInputTokens ?? 0) +- (t.data.cost?.estimatedOutputTokens ?? 0);- return `${t.label} ~${total}`;- });- console.log(` Est. tokens: ${tokenParts.join(' | ')}`);+ const usageComparable = costTargets.every(+ target => target.data.cost?.usageComplete === true,+ );+ if (usageComparable) {+ const tokenParts = costTargets.map(target => {+ const total =+ (target.data.cost?.inputTokens ?? 0) ++ (target.data.cost?.outputTokens ?? 0);+ return `${target.label} ${total}`;+ });+ console.log(` Tokens: ${tokenParts.join(' | ')}`);+ } else {+ const coverageParts = costTargets.map(target => {+ const cost = target.data.cost;+ const total =+ (cost?.completeUsageRuns ?? 0) + (cost?.incompleteUsageRuns ?? 0);+ return `${target.label} ${cost?.completeUsageRuns ?? 0}/${total}`;+ });+ console.log(+ ` Tokens: not comparable; complete runs ${coverageParts.join(' | ')}`,+ );+ }}// Per-prompt wins