:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
perf(ci): size a narrowed Storybook run from the timing manifest (#127960)
James Greenhill59e12285f9fa
diff --git a/.github/workflows/ci-scripts.yml b/.github/workflows/ci-scripts.ymlindex 02629e6ba339..792acd6c2c5a 100644--- a/.github/workflows/ci-scripts.yml+++ b/.github/workflows/ci-scripts.yml@@ -8,6 +8,8 @@ on:paths:- .github/scripts/**- .github/auto-assign-labels.json+ - bin/find-affected-stories+ - bin/find-affected-stories.test.mjs- .github/workflows/ci-scripts.yml- .github/workflows/cd-sandbox-base-image.yml- .github/workflows/update-sandbox-agent-version.yml@@ -58,6 +60,8 @@ jobs:fetch-depth: 1sparse-checkout: |/.github/+ /bin/find-affected-stories+ /bin/find-affected-stories.test.mjs/bin/render-jest-timings-example/bin/report-jest-timings/.depot/@@ -118,3 +122,10 @@ jobs:# preinstalled python3.- name: Node script testsrun: node --test .github/scripts/*.test.js .github/scripts/*.test.mjs++ # find-affected-stories decides how much of the Storybook matrix a PR+ # runs, and its test has run nowhere since it was added. The job has no+ # checkout of the pnpm workspace, so the script's own imports must stay+ # standard library only.+ - name: Story selection tests+ run: node --test bin/find-affected-stories.test.mjsdiff --git a/.github/workflows/ci-storybook.yml b/.github/workflows/ci-storybook.ymlindex 5488bbcb5557..67244fec4d53 100644--- a/.github/workflows/ci-storybook.yml+++ b/.github/workflows/ci-storybook.yml@@ -1413,7 +1413,13 @@ jobs:path: story-selectorref: ${{ github.event.pull_request.base.ref }}repository: ${{ github.event.pull_request.base.repo.full_name }}- sparse-checkout: bin/find-affected-stories+ # The timing manifest sizes the shard count, so a narrow diff is not+ # given a quarter of the runners to do the full run's work. It comes+ # from the base branch with the selector: the nightly refresh lands+ # there, and a fork cannot steer the shard count through it.+ sparse-checkout: |+ bin/find-affected-stories+ common/storybook/storybook-timings.jsonsparse-checkout-cone-mode: false- name: Mint setup-action GitHub token@@ -1455,7 +1461,9 @@ jobs:RESULT=$(story-selector/bin/find-affected-stories \--graph /tmp/storybook-graph/module-graph.json \- --base-ref "base/$BASE_REF")+ --base-ref "base/$BASE_REF" \+ --timings story-selector/common/storybook/storybook-timings.json \+ --max-shards "$CHROMIUM_SHARDS")echo "$RESULT" > /tmp/selection-result.json# Stash compact form in step output (single line) for the summary step.COMPACT=$(node -e 'const d = JSON.parse(require("fs").readFileSync("/tmp/selection-result.json", "utf8")); delete d.affected_stories; delete d.unresolved_changed_files; process.stdout.write(JSON.stringify(d))')@@ -1529,7 +1537,9 @@ jobs:echo "affected_files_json=$AFFECTED_FILES_JSON" >> "$GITHUB_OUTPUT"# Est. minutes saved vs the full chromium run — same formula as the- # "Write summary" step (SHARD_OVERHEAD_MINUTES=3).+ # "Write summary" step (SHARD_OVERHEAD_MINUTES=3). Runner minutes, not+ # wall: the story time sums to the same total either way, so the saving+ # is the shards that no longer pay their own setup.AFFECTED_DUR=$(jq -r '.affected_duration_seconds' /tmp/selection-result.json)TOTAL_DUR=$(jq -r '.total_duration_seconds' /tmp/selection-result.json)FULL_MINUTES=$(( (CHROMIUM_SHARDS * 3) + (TOTAL_DUR / 60) ))@@ -1558,6 +1568,10 @@ jobs:FULL_MINUTES=$(( (CURRENT_SHARDS * SHARD_OVERHEAD_MINUTES) + (TOTAL_DUR / 60) ))SELECTIVE_MINUTES=$(( (SHARDS * SHARD_OVERHEAD_MINUTES) + (AFFECTED_DUR / 60) ))SAVED_MINUTES=$((FULL_MINUTES - SELECTIVE_MINUTES))+ # A shard runs two workers, so its test wall is its share of the story+ # time over two. This is the number a person waits for.+ FULL_WALL_MINUTES=$(( TOTAL_DUR / (2 * CURRENT_SHARDS) / 60 ))+ SELECTIVE_WALL_MINUTES=$(( AFFECTED_DUR / (2 * SHARDS) / 60 ))cat >> "$GITHUB_STEP_SUMMARY" << EOF## Story selection@@ -1566,6 +1580,7 @@ jobs:|---|---|---|---|| Story files | $TOTAL | $COUNT | **$SAVED_STORIES** || Shards | $CURRENT_SHARDS | $SHARDS | **$SAVED_SHARDS** |+ | Est. shard wall | ~${FULL_WALL_MINUTES}m | ~${SELECTIVE_WALL_MINUTES}m | **~$((FULL_WALL_MINUTES - SELECTIVE_WALL_MINUTES))m** || Est. CI minutes | ~${FULL_MINUTES} | ~${SELECTIVE_MINUTES} | **~${SAVED_MINUTES}** || Est. render time | ${TOTAL_DUR}s | ${AFFECTED_DUR}s | **$((TOTAL_DUR - AFFECTED_DUR))s** |diff --git a/bin/find-affected-stories b/bin/find-affected-storiesindex 822722b0cb23..4ed11d0d8586 100755--- a/bin/find-affected-stories+++ b/bin/find-affected-stories@@ -67,24 +67,39 @@ const NON_RUNTIME_FILE_RE =/(^|\/)(__tests__|tests?)\/|\.(test|spec)\.[cm]?[jt]sx?$|(^|\/)README\.md$|^(frontend|playwright)\/snapshots\.yml$/const MAX_CHANGED_FILES = 100-// Rough per-story render cost (see ci-storybook.yml matrix); used only for-// the savings-estimate row in the job summary.+// Fallback per-story render cost, used only when the timing manifest is absent.+// The manifest is the real source; this keeps the estimate in the right order of+// magnitude rather than sizing from zero.const AVG_STORY_DURATION_SECONDS = 6-const TARGET_SHARD_SECONDS = 10 * 60+// Shards the full chromium matrix uses, i.e. CHROMIUM_SHARDS in ci-storybook.yml.+// Overridden by --max-shards, which the workflow passes.+const DEFAULT_FULL_SHARDS = 28const STORY_RE = /\.stories\.(tsx?|mdx|jsx?)$/+// Per-story render times, keyed by story path, refreshed nightly by+// ci-storybook-update-test-timing.yml. The same file the test sequencer balances with.+const DEFAULT_TIMINGS_PATH = 'common/storybook/storybook-timings.json'// -----------------------------------------------------------------------------// CLI// -----------------------------------------------------------------------------function parseArgs(argv) {- const args = { graph: null, baseRef: null }+ const args = { graph: null, baseRef: null, timings: DEFAULT_TIMINGS_PATH, maxShards: DEFAULT_FULL_SHARDS }for (let i = 0; i < argv.length; i++) {const a = argv[i]if (a === '--graph') args.graph = argv[++i]else if (a === '--base-ref') args.baseRef = argv[++i]+ else if (a === '--timings') args.timings = argv[++i]+ else if (a === '--max-shards') {+ // A missing or non-numeric value must not reach the matrix as NaN.+ const value = Number(argv[++i])+ args.maxShards = Number.isFinite(value) && value >= 1 ? Math.floor(value) : DEFAULT_FULL_SHARDS+ }else if (a === '--help' || a === '-h') {- process.stderr.write('Usage: find-affected-stories --graph <path> --base-ref <git-ref>\n')+ process.stderr.write(+ 'Usage: find-affected-stories --graph <path> --base-ref <git-ref> ' ++ '[--timings <path>] [--max-shards <n>]\n'+ )process.exit(0)} else {process.stderr.write(`Unknown argument: ${a}\n`)@@ -115,11 +130,53 @@ function emitFull(reason, totalStories) {process.stdout.write(JSON.stringify({ mode: 'full', reason, total_story_count: totalStories }) + '\n')}-function emitSelective(affected, totalStories, unresolved) {- const affectedDuration = affected.length * AVG_STORY_DURATION_SECONDS- const totalDuration = totalStories * AVG_STORY_DURATION_SECONDS- // 2x safety factor keeps selective shards below the target under variance.- const suggestedShards = Math.max(1, Math.floor((affectedDuration * 2) / TARGET_SHARD_SECONDS) + 1)+export function loadTimings(path) {+ try {+ const data = JSON.parse(fs.readFileSync(path, 'utf-8'))+ if (!data || typeof data !== 'object' || Array.isArray(data)) {+ return null+ }+ const entries = Object.entries(data).filter(([, v]) => v && typeof v.chromium === 'number')+ return entries.length > 0 ? new Map(entries) : null+ } catch {+ return null+ }+}++// Render seconds for a set of stories. A story the nightly refresh has not timed+// yet costs the manifest median, which is far closer than the old flat constant.+export function workloadSeconds(stories, timings) {+ if (!timings) {+ return stories.length * AVG_STORY_DURATION_SECONDS+ }+ const known = [...timings.values()]+ .map((v) => v.chromium)+ .sort((a, b) => a - b)+ const fallback = known[Math.floor(known.length / 2)]+ let total = 0+ for (const story of stories) {+ const entry = timings.get(story)+ total += entry ? entry.chromium : fallback+ }+ return total+}++export function suggestShards(affectedDuration, totalDuration, maxShards) {+ // Give the narrowed run the shard density the full matrix has, so a selective+ // run finishes its tests in about the wall time a full run does. Sizing from a+ // flat per-story constant instead put the full run's story time on a quarter of+ // its runners, and a narrowed run took twice as long per shard as a full one.+ if (!(totalDuration > 0) || !(affectedDuration > 0)) {+ return 1+ }+ const proportional = Math.ceil((affectedDuration / totalDuration) * maxShards)+ return Math.max(1, Math.min(maxShards, proportional))+}++function emitSelective(affected, totalStories, unresolved, allStories, maxShards, timings) {+ const affectedDuration = workloadSeconds(affected, timings)+ const totalDuration = workloadSeconds(allStories, timings)+ const suggestedShards = suggestShards(affectedDuration, totalDuration, maxShards)process.stdout.write(JSON.stringify({mode: 'selective',@@ -275,7 +332,20 @@ export function main() {return}process.stderr.write(`Affected stories: ${affected.length} / ${totalStories} (unresolved: ${unresolved.length})\n`)- emitSelective(affected, totalStories, unresolved)++ const timings = loadTimings(args.timings)+ if (!timings) {+ process.stderr.write(`No timing manifest at ${args.timings}; sizing shards from the flat estimate\n`)+ } else {+ // A key format drift would send every story to the median without failing,+ // so the hit rate is printed where the job log shows it.+ const timed = affected.filter((story) => timings.has(story)).length+ process.stderr.write(+ `Timed ${timed} of ${affected.length} affected stories from ${args.timings}; the rest cost the median\n`+ )+ }+ const allStories = graph.modules.map((m) => normalize(m.name)).filter((name) => STORY_RE.test(name))+ emitSelective(affected, totalStories, unresolved, allStories, args.maxShards, timings)}if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {diff --git a/bin/find-affected-stories.test.mjs b/bin/find-affected-stories.test.mjsindex 226fc7372979..435e4b3cae94 100644--- a/bin/find-affected-stories.test.mjs+++ b/bin/find-affected-stories.test.mjs@@ -1,10 +1,16 @@import assert from 'node:assert/strict'+import fs from 'node:fs'+import os from 'node:os'+import path from 'node:path'import { describe, test } from 'node:test'import {affectedStoriesFor,+ loadTimings,requiresFullRun,+ suggestShards,unresolvedRuntimeFiles,+ workloadSeconds,} from './find-affected-stories'describe('find-affected-stories', () => {@@ -61,4 +67,50 @@ describe('find-affected-stories', () => {[])})++ // A selective run that gets too few shards spends more wall time on its short+ // story list than a full run spends on all of them, which makes the narrowed+ // run the slow lane it was meant to avoid.+ test('sizes a narrowed run to the shard density of the full matrix', () => {+ // Half the story seconds of the full matrix, so half its shards.+ assert.equal(suggestShards(5000, 10000, 28), 14)+ // A change set that touches every story must not ask for more than the full matrix.+ assert.equal(suggestShards(10000, 10000, 28), 28)+ assert.equal(suggestShards(20000, 10000, 28), 28)+ // A heavy dozen stories still earns more than a single shard.+ assert.equal(suggestShards(2000, 10000, 28), 6)+ assert.equal(suggestShards(10, 10000, 28), 1)+ })++ test('reads chromium times from the manifest and rejects unusable files', () => {+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'story-timings-'))+ const good = path.join(dir, 'good.json')+ fs.writeFileSync(+ good,+ JSON.stringify({+ 'frontend/src/A.stories.tsx': { chromium: 10, webkit: 3 },+ 'frontend/src/B.stories.tsx': { webkit: 4 },+ })+ )+ const timings = loadTimings(good)+ assert.deepEqual([...timings.keys()], ['frontend/src/A.stories.tsx'])++ const list = path.join(dir, 'list.json')+ fs.writeFileSync(list, '[]')+ assert.equal(loadTimings(list), null)+ assert.equal(loadTimings(path.join(dir, 'missing.json')), null)+ })++ test('costs an untimed story the manifest median', () => {+ const timings = new Map([+ ['a.stories.tsx', { chromium: 10 }],+ ['b.stories.tsx', { chromium: 20 }],+ ['c.stories.tsx', { chromium: 90 }],+ ])+ assert.equal(workloadSeconds(['a.stories.tsx', 'c.stories.tsx'], timings), 100)+ // Not in the manifest, so it costs the median of 10, 20, 90.+ assert.equal(workloadSeconds(['new.stories.tsx'], timings), 20)+ // No manifest at all falls back to the flat per-story estimate.+ assert.equal(workloadSeconds(['a.stories.tsx', 'b.stories.tsx'], null), 12)+ })})diff --git a/docs/internal/ci-things-already-tried.md b/docs/internal/ci-things-already-tried.mdindex 7c76af6d4622..ad25393baec1 100644--- a/docs/internal/ci-things-already-tried.md+++ b/docs/internal/ci-things-already-tried.md@@ -480,7 +480,7 @@ OpenAPI codegen tests keep root files, every package manifest, patches, the pnpmIf you propose this again, name the jobs and prove that each one reads only the included paths. A test job can read more of the tree than an exclusion list expects.-Storybook's `select-stories` job checks out only `.nvmrc` from the PR head. It keeps the bounded blobless history for changed-path detection, downloads the module graph, and runs the selector from a separate trusted base checkout. Source files do not need to exist in the PR worktree for this job.+Storybook's `select-stories` job checks out only `.nvmrc` from the PR head. It keeps the bounded blobless history for changed-path detection, downloads the module graph, and runs the selector from a separate trusted base checkout, which also carries `common/storybook/storybook-timings.json`. Source files do not need to exist in the PR worktree for this job. The manifest is there because it sizes the shard count, and a file count cannot size that: the stories a frontend diff affects carry about twice the render time per story that the average story does. It is read from the base checkout so the nightly refresh applies to branches behind master and a fork cannot steer the shard count._Also asked as:_ sparse-checkout, partial clone, do not check out the whole repo, speed up the checkout step