Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

test(vibe-tests): List edge naming vibe test (#7629)

Kenton Quatman305cc61b5d3a
diff --git a/internal/vibe-tests/list-edge-naming-test/.gitignore b/internal/vibe-tests/list-edge-naming-test/.gitignore
new file mode 100644
index 0000000000..c217d12988
--- /dev/null
+++ b/internal/vibe-tests/list-edge-naming-test/.gitignore
@@ -0,0 +1,3 @@
+results/
+# Generated by gen-docs.mjs; the first rerun step rebuilds them byte for byte.
+docs/
diff --git a/internal/vibe-tests/list-edge-naming-test/PLAN.md b/internal/vibe-tests/list-edge-naming-test/PLAN.md
new file mode 100644
index 0000000000..c2425a6b13
--- /dev/null
+++ b/internal/vibe-tests/list-edge-naming-test/PLAN.md
@@ -0,0 +1,285 @@
+# List edge naming vibe test
+
+**Status:** ad-hoc · rounds 1–3 complete · recommendation shipped in PR #7592
+**Related:** [`../README.md`](../README.md) (Checker Protocol), [API Arbitration](https://github.com/facebook/astryx/wiki/API-Arbitration), [Designing Vibe Tests](https://github.com/facebook/astryx/wiki/Designing-Vibe-Tests), [PR #7592](https://github.com/facebook/astryx/pull/7592)
+
+---
+
+## 1. The question
+
+[PR #7592](https://github.com/facebook/astryx/pull/7592) gives `List` one prop that
+picks where each row's text, highlight, and dividers sit relative to the list's sides
+and a surrounding `Card`, `Section`, or `Dialog`. There are four looks:
+
+| look | row text | highlight and dividers |
+| ----------- | ----------------------------------------- | ---------------------------------------------------------- |
+| 1 (default) | a short distance in from the list's sides | span the list, a little beyond the text |
+| 2 | at the list's sides, on a heading's line | stop at the text |
+| 3 | on a heading's line | a little beyond the text, short of the container's borders |
+| 4 | on a heading's line | reach the container's borders, like table rows |
+
+**What should the prop and its four values be called?** Rounds 1 and 2 tested the PR's
+first names, `edge` with `inset · flush · outset · bleed`, against alternatives. Round 3
+tests the names it ships, `padded · flush · outset · bleed`, against the released API and
+against no API at all.
+
+## 2. Design
+
+Four-ingredient intake per [Designing Vibe Tests](https://github.com/facebook/astryx/wiki/Designing-Vibe-Tests).
+
+**Goal and measure.** A naive agent is told what a layout needs and never told a prop or
+value name. Scored on correctness, invented props or styles, directness, and confidence
+calibration. Tie-break: fewer misreads and invented props, then plainer words.
+
+**Three probes.**
+
+| probe | what the agent gets | what it measures |
+| ------- | ---------------------------------------------------- | --------------------------------------------- |
+| use | its arm's reference doc and one layout prompt | can it pick the right value from the docs? |
+| recall | a doc that describes the four looks but names no API | what API does it expect? one prop or several? |
+| reading | no doc, only code with one value in a `Card` | does the name alone predict the rendering? |
+
+**Use battery.** Seven prompts in [`prompts.json`](prompts.json). None name a prop, a
+value, or the words padding, inset, edge, margin, or align.
+
+| | layout | correct look |
+| --- | -------------------------------------------------------------------------------- | ---------------------------------------- |
+| p1 | static list with dividers under a page heading, no card | 2 |
+| p2 | static list in a card, lines touch the card's borders | 4 |
+| p3 | static list in a card, lines stop short of the borders | 2 |
+| p4 | clickable folders in a card, highlight with room that stays clear of the borders | 3 |
+| p5 | clickable menu in a card, highlight touching the card's sides | 4 |
+| p6 | recent-files popup, nothing to line up with | **leave it unset** (negative control) |
+| p7 | sidebar list under a heading, no card, aligned text and a roomy highlight | **no single value does both** (conflict) |
+
+**Arms.**
+
+| arm | round | prop | look 1 | look 2 | look 3 | look 4 |
+| ------------- | ----- | --------------------------------------------------- | -------- | ------- | ---------- | ------- |
+| A | 1 | `edge` | `inset` | `flush` | `outset` | `bleed` |
+| B | 1 | `edge` | `padded` | `flush` | `hanging` | `bleed` |
+| C | 2 | `rowAlignment` | `inset` | `flush` | `outset` | `bleed` |
+| D | 3 | `edge` | `padded` | `flush` | `outset` | `bleed` |
+| Both names | 3 | `edge`, plus deprecated `edgeCompensation="inline"` | `padded` | `flush` | `outset` | `bleed` |
+| Main baseline | 3 | `edgeCompensation` | unset | — | `'inline'` | — |
+| Natural reach | 3 | none | — | — | — | — |
+
+Arm B swaps the two names most likely to mislead: `outset` is rare, and `inset`
+already names a different kind of list on some platforms. Arm C is round 1's most common recalled
+prop name, added in round 2 as planned.
+
+Round 3 adds the three arms a review asked for, each run on the same seven prompts:
+
+- **Main baseline:** the released `List`, whose only edge prop is
+ `edgeCompensation="inline"` (look 3). Its API cannot produce looks 2 and 4, so p1, p2,
+ p3, and p5 measure what agents do when the API falls short.
+- **Natural reach:** no edge API at all, and no anti-pattern section, so nothing steers
+ agents away from the workaround being measured. The doc describes only the default look.
+- **Both names:** the shipped names plus the deprecated `edgeCompensation`, as the docs
+ read during its compatibility window. Does the old name attract use?
+- **Arm D**, the shipped names alone, is the control the both-names arm is compared
+ against. It is also the first use run of the shipped set as a whole.
+
+**Orchestration.** 91 isolated agents, one per task, all on one model at one setting:
+round 1 had 14 use, 6 recall, and 10 reading agents; round 2 had 7 use (arm C) and 26
+reading agents; round 3 had 28 use agents (seven prompts in each of four arms). Round 2's reading probe adds samples where round 1 split or was thin,
+tries both prop names, and adds a condition that shows the type an editor displays on
+hover. A scripted checker ([`check.mjs`](check.mjs)) scores every value choice and
+multiple-choice answer. One judge agent, seeing all arms side by side, scored round 1's
+use replies on the rubric; rounds 2 and 3 were scored by the checker, with round 3's
+workarounds read from each reply's code and notes.
+
+### Fairness (Checker Protocol)
+
+| Invariant | How this honors it |
+| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| §1 Fair evaluators | The checker applies one rule to every arm. The judge scored round 1 on one rubric with all arms side by side. Blinding is impossible because the value names show the arm, so the judge was told to score the rubric, not its own taste, and to audit doc parity first. It found none |
+| §2 Only the SUT varies | [`gen-docs.mjs`](gen-docs.mjs) renders every arm from one template and fails unless arms B, C, and D with their differing names mapped back equal arm A byte for byte (507 words each), and the both-names doc without its deprecated row and paragraph equals arm D. The main baseline and natural-reach docs differ only in their edge section, which is the system under test: main's second example sits in a `Card` because its one value needs one, and natural reach has no anti-pattern. The prose never uses a word that echoes one arm's names (pad, inset, inside, outset, outside, hang), and the recall doc's prose never uses a word that would name the prop |
+| §3 Never leak the answer | Ground truth lives in `prompts.json` and never reached a generating agent. The prompts were scanned for padding, inset, edge, margin, align, and indent |
+| §4 Representative environment | A reference doc is what a consumer's agent reads. The reading probe is what a reviewer sees in code, with and without the editor's type hover |
+| §5 Context-free | Each agent started fresh and received only the exact path of its own task file, in a directory it could not list. Each of the 91 test agents' logs shows exactly one tool call: reading that file |
+
+---
+
+## 3. Findings
+
+### 3.1 With the docs, the names do not change what agents build
+
+**18 of 18 scored use prompts were correct across all three arms**, with no invented props
+and no custom styles. All three p6 agents left the prop unset. All three p7 agents saw
+that look 3 needs a container: arms A and C chose look 2 and said the highlight would be
+tight (confidence 2/5), and arm B wrapped the heading and list in a `Section` (3/5).
+
+As in earlier naming tests, once the API is documented, agents pick by the look
+descriptions ("like table rows", "without reaching those borders"), not by the names.
+Everything below is about what agents expect and what they read.
+
+### 3.2 One prop with four values is what agents expect
+
+All six recall agents wrote one prop with four values. None wrote booleans, and all six
+rejected them unprompted: a yes/no switch cannot pick among four looks, and several
+switches allow combinations like "both on" that mean nothing.
+
+### 3.3 `inset` is misread in code; `padded` is read better
+
+| value | bare code | with the editor's type |
+| -------- | --------- | ---------------------- |
+| `inset` | **0/9** | 2/3 |
+| `padded` | 5/9 | 3/3 |
+
+Every bare `inset` reader said the text lines up with the heading, which is the opposite
+of what look 1 does. They read it as "inset inside the card's padding" (round 1:
+_"'inset' suggests the list sits inside the card's padding, so the row text lines up with
+the heading"_). The four `padded` readers who missed made the same placement mistake by
+another route: they assumed the rows grow outward around their padding, so the text stays
+on the heading's line.
+
+Recall points the other way: five of six recall agents named look 1 `inset`, and three
+chose `outset` for look 3 because it pairs with `inset`. The sixth chose `padded` only
+because its prop was already called `rowInset`, and it produced `padded · flush · outset ·
+bleed` exactly. Agents guess `inset`, but they misread it.
+
+### 3.4 `outset` and `bleed` are read right every time
+
+`outset` 4/4, `hanging` 2/2, and `bleed` 3/3 were fully right from code alone. In
+recall, `outset` named look 3 in four of six samples (`hanging` and `bleed` once each).
+Four recall agents considered `hang` or `hanging` and rejected it: two as typography terms
+(hanging punctuation, a hanging indent) and two because `outset` pairs with `inset`.
+`bleed` named look 4 in four samples (`edgeToEdge` and `fullBleed` once each).
+
+### 3.5 `flush` is what everyone reaches for, but its highlight is a guess
+
+All six recall agents named look 2 `flush`, and every use agent applied it correctly. From
+code alone, all three readers put the text in the right place, but **none** put the
+highlight at the text: one put it a little beyond the text, and two ran it and the dividers
+to the card's borders. The docs have to say plainly that look 2's highlight and dividers
+stop at the text.
+
+### 3.6 Agents guess alignment-style prop names, but those do no better
+
+Recall produced `rowAlignment` 3, `rowAlign` 1, `rowInset` 1, and `edge` 1. But arm C
+(`rowAlignment`) scored the same as `edge` in use (6/6), and read no better:
+
+| value | `edge` | `rowAlignment` |
+| -------- | ------ | -------------- |
+| `inset` | 0/6 | 0/3 |
+| `padded` | 4/6 | 1/3 |
+| `outset` | 2/2 | 2/2 |
+| `bleed` | 1/1 | 2/2 |
+
+Recall agents themselves rejected bare `align` and `alignment` as left, center, or right
+text alignment. In this codebase, `Item` already has an `align` prop for the vertical
+alignment of its start and end content, so `rowAlignment` on `List` would sit next to an
+unrelated `align`.
+
+### 3.7 Without the prop, agents hack around it; the deprecated name draws no use
+
+| arm | right look (prompts its API can reach) | custom styles | deprecated prop written | confidence, p1–p5 |
+| ----------------- | -------------------------------------- | ------------- | ----------------------- | ------------------------------------------ |
+| D (shipped names) | 6/6 | 0/7 | — | 4/5 |
+| Both names | 6/6 | 0/7 | 0/7 | 4–5/5 |
+| Main baseline | 2/2 (p4, p6) | 0/7 | — | 4/5 on p4; 2–3/5 where its API falls short |
+| Natural reach | 1/1 (p6) | 5/7 | — | 1–2/5 |
+
+- **Shipped names.** Arm D used the shipped set correctly every time, as arms A, B, and C
+ did. Its p7 agent chose look 2 and named the tight highlight, as arms A and C did.
+- **Both names.** No agent wrote `edgeCompensation`; one said it left the deprecated prop
+ out on purpose. Its p7 agent wrapped the heading and list in a `Section` and used
+ `outset`, as arm B did in round 1.
+- **Main baseline.** For the four layouts its API cannot produce (p1, p2, p3, p5), every
+ agent still wrote `edgeCompensation="inline"`, wrapping p1 in a `Section` to make it
+ apply. None added custom styles, and all four said the layout was out of reach: lines
+ that overshoot the text, or a highlight that stops short of the card's borders. One
+ asked for "something like `edgeCompensation=\"bleed\"`". Four of six layouts ship
+ visibly wrong, and the agents know it.
+- **Natural reach.** Every alignment prompt (p1–p5) ended in a workaround. Four agents
+ shifted the list or the heading by guessed pixel values (12px, 16px), and one dropped
+ `List` for a hand-built `<ul>` with its own borders. Each wrote that the numbers were
+ guesses. The p7 agent put the heading in `List`'s `header` slot, hoping it would line up.
+
+### 3.8 Doc gaps the use probe surfaced (not naming)
+
+1. **Which containers count.** All three p6 agents asked whether a popup counts as a
+ container for looks 3 and 4.
+2. **Whether start content counts as text.** All three p4 agents were unsure whether the
+ icon in `startContent` lines up with the heading or the label does.
+3. **Sidebars.** No look aligns the text and keeps highlight room without a container. One
+ p7 agent in three used a `Section` wrapper; the other two considered it and held back
+ because the docs do not say whether a `Section` looks like a box. The docs could show
+ the pattern.
+
+---
+
+## 4. Recommendation
+
+**Keep `edge` and one prop with four values. Rename look 1 from `inset` to `padded`. Keep
+`flush`, `outset`, and `bleed`.** Applied in [PR #7592](https://github.com/facebook/astryx/pull/7592),
+with the List docs covering §3.5 and the three gaps in §3.8.
+
+- One prop with four values, because recall produced it 6/6 and the looks exclude each
+ other (§3.2).
+- `edge`, because the alternative agents guess performs no better and collides with
+ `Item`'s `align` (§3.6), and `edge` matches the existing edge-compensation vocabulary.
+- `padded` for look 1, because `inset` was misread 0/9 from code and `padded` 5/9; the
+ gap narrows, to 2/3 against 3/3, only when the editor shows the full type (§3.3). The
+ cost is the `inset`/`outset` pair that recall agents liked.
+- `outset` and `bleed`, because they were read right every time and recalled most
+ (§3.4).
+- `flush`, because everyone reaches for it, with docs that state where its highlight and
+ dividers stop (§3.5).
+
+Round 3 supports this. The shipped set scored 6/6 in use. With the deprecated name
+beside it, no agent picked the old name. Without the prop, agents either ship four of six
+layouts wrong (the released API) or shift rows by guessed pixel values (no API), which the
+List docs name as an anti-pattern (§3.7).
+
+## 5. Caveats
+
+- **Small samples.** Reading has 2 to 9 samples per condition and recall has 6. The
+ `inset`/`padded` split (0/9 against 5/9) is the only reading result wide enough to
+ matter on its own.
+- **One model.** All agents ran on one model at one setting, so they share priors and are
+ not independent people.
+- **The use probe hit its ceiling.** The prompts echo the look descriptions, so use cannot
+ separate the arms.
+- **One composition.** Reading covers one value at a time in a `Card` under a heading.
+- **Recall wording.** The recall doc's phrases ("a short distance", "extend a little
+ beyond") may prime some words.
+- **The shipped set was never read from code as a set.** Round 3 used it as a whole (6/6
+ in arm D and in the both-names arm), but the reading probe tested values one at a time.
+- **Natural reach is one reading of the request.** It is run here as "no edge API and no
+ anti-pattern". A run with natural-language prompts instead of geometric ones would
+ answer a different question: whether agents reach for the prop when not told the
+ geometry.
+- **The main-baseline doc is the test template, not the released List docs.** It swaps
+ in the released prop but keeps the template's anti-pattern section ("Don't shift rows
+ with your own styles … Use `edgeCompensation` instead.") and omits the released
+ `xstyle`, `listStyle`, and `start` props. The natural-reach doc has no anti-pattern
+ section, so main's 0/7 custom styles against natural reach's 5/7 partly reflects that
+ warning, not only the prop.
+- **One author** wrote the battery and the candidates.
+
+## 6. Folder layout
+
+```
+internal/vibe-tests/list-edge-naming-test/
+├── PLAN.md # this file: design, findings, recommendation
+├── prompts.json # battery, ground truth, recall and reading probes, rounds 2 and 3
+├── gen-docs.mjs # renders every arm's doc from one template and proves parity
+├── build-tasks.mjs # assembles one self-contained task per agent
+├── check.mjs # scripted scoring of value choices and reading answers
+├── docs/ # (gitignored) one doc per arm, generated by gen-docs.mjs
+└── results/ # (gitignored) tasks, the agents' replies, check.md, judge
+```
+
+To rerun:
+
+1. Run `node gen-docs.mjs`, then `node build-tasks.mjs` (round 1),
+ `node build-tasks.mjs --round=2`, and `node build-tasks.mjs --round=3`. Each writes
+ `results/tasks[-rN]/<task id>.md` and a `manifest.json` that maps each task id to a
+ random file name.
+2. Copy each task file to its manifest name in a directory agents can open files in but
+ cannot list (for example, files `chmod 0444` in a directory `chmod 0311`).
+3. Start one fresh agent per task. Give it only the exact path of its own file, with the
+ instruction to read that file and use no other tool.
+4. Save each reply verbatim to `results/out/<task id>.md`, then run `node check.mjs`.
diff --git a/internal/vibe-tests/list-edge-naming-test/build-tasks.mjs b/internal/vibe-tests/list-edge-naming-test/build-tasks.mjs
new file mode 100644
index 0000000000..2a5d905ef2
--- /dev/null
+++ b/internal/vibe-tests/list-edge-naming-test/build-tasks.mjs
@@ -0,0 +1,101 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+// Assembles one self-contained task per agent from prompts.json and the
+// generated docs, so every agent in an arm reads byte-identical material.
+// `node build-tasks.mjs` builds round 1, `--round=2` and `--round=3` build
+// those rounds. Writes results/tasks[-rN]/<id>.md plus manifest.json, which
+// maps each task id to a random file name for delivery: agents receive only
+// their own file's exact path.
+
+import {randomBytes} from 'node:crypto';
+import {mkdirSync, readFileSync, writeFileSync} from 'node:fs';
+import {dirname, join} from 'node:path';
+import {fileURLToPath} from 'node:url';
+
+const HERE = dirname(fileURLToPath(import.meta.url));
+const round = Number(
+ process.argv.find(arg => arg.startsWith('--round='))?.slice(8) ?? 1,
+);
+const OUT = join(HERE, 'results', round === 1 ? 'tasks' : `tasks-r${round}`);
+const spec = JSON.parse(readFileSync(join(HERE, 'prompts.json'), 'utf8'));
+const doc = path => readFileSync(join(HERE, path), 'utf8').trim();
+
+/** @type {Record<string, string>} */
+const tasks = {};
+
+function useTasks(armIds) {
+ for (const arm of armIds) {
+ for (const prompt of spec.generation.prompts) {
+ tasks[`gen-${prompt.id}-${arm}`] = [
+ spec.preamble,
+ `## Reference\n\n${doc(spec.arms[arm].doc)}`,
+ `## Task\n\n${prompt.text}`,
+ spec.generation.output,
+ ].join('\n\n');
+ }
+ }
+}
+
+function readingTask({prop, value, union}) {
+ const hint =
+ union == null
+ ? []
+ : [
+ spec.round2.unionHint
+ .replace('__PROP__', prop)
+ .replace('__UNION__', union.map(v => `'${v}'`).join(' | ')),
+ ];
+ return [
+ spec.preamble,
+ `## Code\n\n${spec.reading.snippet.replace('__PROP__', prop).replace('__VALUE__', value)}`,
+ ...hint,
+ `## Questions\n\n${spec.reading.questions.replace('__PROP__', prop)}`,
+ ].join('\n\n');
+}
+
+if (round === 1) {
+ useTasks(Object.keys(spec.arms).filter(arm => spec.arms[arm].round == null));
+
+ const byId = Object.fromEntries(spec.generation.prompts.map(p => [p.id, p]));
+ const recallTasks = spec.recall.scenarios
+ .map((id, i) => `### Task ${i + 1}\n\n${byId[id].text}`)
+ .join('\n\n');
+ for (let i = 1; i <= spec.recall.samples; i++) {
+ tasks[`recall-${i}`] = [
+ spec.preamble,
+ `## Reference\n\n${doc(spec.recall.doc)}`,
+ `## Tasks\n\nThe reference describes four row looks but does not list the API that chooses one. Write the API you would expect, then use it.\n\n${recallTasks}`,
+ spec.recall.output,
+ ].join('\n\n');
+ }
+
+ for (const [value, count] of Object.entries(spec.reading.samples)) {
+ for (let i = 1; i <= count; i++) {
+ tasks[`read-${value}-${i}`] = readingTask({prop: 'edge', value});
+ }
+ }
+} else if (round === 3) {
+ useTasks(spec.round3.use.arms);
+} else {
+ useTasks(spec.round2.use.arms);
+ for (const condition of spec.round2.reading) {
+ const id = `read2-${condition.prop}-${condition.value}${condition.union ? '-union' : ''}`;
+ for (let i = 1; i <= condition.samples; i++) {
+ tasks[`${id}-${i}`] = readingTask(condition);
+ }
+ }
+}
+
+mkdirSync(OUT, {recursive: true});
+const manifest = {};
+for (const [id, text] of Object.entries(tasks)) {
+ writeFileSync(join(OUT, `${id}.md`), `${text}\n`);
+ manifest[id] = `${randomBytes(12).toString('hex')}.md`;
+}
+writeFileSync(
+ join(OUT, 'manifest.json'),
+ `${JSON.stringify(manifest, null, 2)}\n`,
+);
+console.log(
+ `${Object.keys(tasks).length} round ${round} tasks written to ${OUT}`,
+);
diff --git a/internal/vibe-tests/list-edge-naming-test/check.mjs b/internal/vibe-tests/list-edge-naming-test/check.mjs
new file mode 100644
index 0000000000..ed3308b9fa
--- /dev/null
+++ b/internal/vibe-tests/list-edge-naming-test/check.mjs
@@ -0,0 +1,318 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+// Scores what can be scored mechanically, from results/out/<task id>.md:
+// - use: the `edge` value each agent put on <List>, against prompts.json's
+// ground truth, plus undocumented props and custom styles;
+// - recall: the prop name and the four value names each agent wrote;
+// - reading: each multiple-choice answer, against the real geometry.
+// p7 (a deliberate conflict) and every fuzzy dimension go to the judge.
+// Prints markdown tables; run after the agents' replies are saved.
+
+import {existsSync, readFileSync} from 'node:fs';
+import {dirname, join} from 'node:path';
+import {fileURLToPath} from 'node:url';
+
+const HERE = dirname(fileURLToPath(import.meta.url));
+const OUT = join(HERE, 'results', 'out');
+const spec = JSON.parse(readFileSync(join(HERE, 'prompts.json'), 'utf8'));
+const read = id => {
+ const path = join(OUT, `${id}.md`);
+ return existsSync(path) ? readFileSync(path, 'utf8') : null;
+};
+
+// List props every arm's doc documents. Each arm adds its own edge prop and,
+// for the both-names arm, the deprecated one, so a prop only another arm
+// documents is reported as undocumented.
+const SHARED_LIST_PROPS = [
+ 'children',
+ 'header',
+ 'density',
+ 'hasDividers',
+ 'key',
+];
+
+/** The List props an arm's doc documents. */
+function listPropsFor({prop, deprecated}) {
+ return new Set(
+ [...SHARED_LIST_PROPS, prop, deprecated?.prop].filter(p => p != null),
+ );
+}
+const ITEM_PROPS = new Set([
+ 'label',
+ 'description',
+ 'startContent',
+ 'endContent',
+ 'onClick',
+ 'href',
+ 'isSelected',
+ 'isDisabled',
+ 'key',
+]);
+const STYLE_PROPS = /\b(style|xstyle|className)=/;
+
+/**
+ * Each opening tag of `name` in the reply's code, as its attributes:
+ * `{name, value}`, where `value` is the raw source of a quoted or braced
+ * value, or null for a bare boolean attribute. Values are skipped with a
+ * quote- and brace-aware walk, so JSX inside an earlier attribute
+ * (`header={<Heading level={2}>…</Heading>}`) cannot end the tag early.
+ */
+function tagAttributes(code, name) {
+ const found = [];
+ const opener = new RegExp(`<${name}\\b`, 'g');
+ let match;
+ while ((match = opener.exec(code)) != null) {
+ const attributes = [];
+ let i = match.index + name.length + 1;
+ while (i < code.length) {
+ const char = code[i];
+ if (char === '>' || (char === '/' && code[i + 1] === '>')) break;
+ const attr = /^[A-Za-z][\w-]*/.exec(code.slice(i));
+ if (attr == null) {
+ i++;
+ continue;
+ }
+ i += attr[0].length;
+ if (code[i] !== '=') {
+ attributes.push({name: attr[0], value: null});
+ continue;
+ }
+ i++;
+ const start = i;
+ if (code[i] === '"' || code[i] === "'") {
+ const close = code.indexOf(code[i], i + 1);
+ i = close === -1 ? code.length : close + 1;
+ } else if (code[i] === '{') {
+ for (let depth = 0; i < code.length; i++) {
+ if (code[i] === '{') depth++;
+ else if (code[i] === '}' && --depth === 0) {
+ i++;
+ break;
+ }
+ }
+ }
+ attributes.push({name: attr[0], value: code.slice(start, i)});
+ }
+ found.push(attributes);
+ }
+ return found;
+}
+
+/** Props written on each opening tag of `name`, from the reply's code. */
+function tagProps(code, name) {
+ return tagAttributes(code, name).map(attributes =>
+ attributes.map(attribute => attribute.name),
+ );
+}
+
+/**
+ * The value of `prop` on the first <List> that sets it: a string literal's
+ * text, `?<source>` for any other value (so a computed value is reported, not
+ * read as unset), or null when no <List> sets it.
+ */
+function edgeValue(code, prop) {
+ if (prop == null) return null;
+ for (const attributes of tagAttributes(code, 'List')) {
+ const attribute = attributes.find(a => a.name === prop);
+ if (attribute == null) continue;
+ const literal =
+ /^(?:"([^"]*)"|'([^']*)'|\{\s*(?:"([^"]*)"|'([^']*)')\s*\})$/.exec(
+ attribute.value ?? '',
+ );
+ if (literal) return literal[1] ?? literal[2] ?? literal[3] ?? literal[4];
+ return `?${attribute.value ?? 'true'}`;
+ }
+ return null;
+}
+
+const rows = [];
+const correctCount = Object.fromEntries(
+ Object.keys(spec.arms).map(arm => [arm, 0]),
+);
+const reachable = Object.fromEntries(
+ Object.keys(spec.arms).map(arm => [arm, 0]),
+);
+const deprecatedUse = {};
+for (const prompt of spec.generation.prompts) {
+ for (const [arm, armSpec] of Object.entries(spec.arms)) {
+ const {values, prop, deprecated} = armSpec;
+ const listProps = listPropsFor(armSpec);
+ const reply = read(`gen-${prompt.id}-${arm}`);
+ if (reply == null) continue;
+ const code = (reply.match(/```tsx\n([\s\S]*?)```/) ?? [, ''])[1];
+ const edge = edgeValue(code, prop);
+ const deprecatedValue =
+ deprecated == null ? null : edgeValue(code, deprecated.prop);
+ if (deprecatedValue != null)
+ deprecatedUse[arm] = (deprecatedUse[arm] ?? 0) + 1;
+ const value =
+ edge ??
+ (deprecatedValue != null
+ ? `${deprecated.prop}=${deprecatedValue}`
+ : null);
+ // A defined `edge` wins over the deprecated prop, as List resolves them.
+ let canonical = 'unset';
+ if (edge != null) {
+ canonical =
+ Object.keys(values).find(k => values[k] === edge) ??
+ `?${edge.replace(/^\?/, '')}`;
+ } else if (deprecatedValue != null) {
+ canonical =
+ deprecatedValue === deprecated.value
+ ? deprecated.look
+ : `?${deprecatedValue.replace(/^\?/, '')}`;
+ }
+ // Whether the arm's API can produce the prompt's look at all (the main
+ // baseline has no flush or bleed; the natural-reach arm has no values).
+ const canReach =
+ prompt.correct === 'unset' ||
+ prompt.correct === 'conflict' ||
+ prompt.correct in values;
+ const invented = [
+ ...tagProps(code, 'List')
+ .flat()
+ .filter(p => !listProps.has(p)),
+ ...tagProps(code, 'ListItem')
+ .flat()
+ .filter(p => !ITEM_PROPS.has(p)),
+ ];
+ const styled = STYLE_PROPS.test(code);
+ const wrapped = /<(Section|Card|Dialog)\b/.exec(code)?.[1] ?? '—';
+ const confidence = (reply.match(/Confidence:\s*(\d)/) ?? [, '?'])[1];
+ let verdict;
+ if (prompt.correct === 'conflict') verdict = 'judge';
+ else if (prompt.correct === 'unset')
+ verdict =
+ canonical === 'unset'
+ ? 'yes'
+ : canonical === 'inset'
+ ? 'yes (explicit default)'
+ : 'no';
+ else if (!canReach) verdict = 'no API';
+ else verdict = canonical === prompt.correct ? 'yes' : 'no';
+ if (verdict.startsWith('yes')) correctCount[arm]++;
+ if (canReach && prompt.correct !== 'conflict') reachable[arm]++;
+ const usesList = /<List\b/.test(code);
+ rows.push(
+ `| ${prompt.id} | ${arm.toUpperCase()} | ${prompt.correct} | ${value ?? '(unset)'} | ${verdict} | ${invented.length ? invented.join(', ') : 'none'} | ${styled ? 'yes' : 'no'} | ${usesList ? wrapped : `${wrapped}, no List`} | ${confidence}/5 |`,
+ );
+ }
+}
+
+console.log('## Use (scripted)\n');
+console.log(
+ '| prompt | arm | correct look | value written | correct | undocumented props | custom styles | wrapper | confidence |',
+);
+console.log('| --- | --- | --- | --- | --- | --- | --- | --- | --- |');
+console.log(rows.join('\n'));
+console.log(
+ `\nScripted correct, excluding p7 (out of the prompts each arm's API can reach): ${Object.entries(
+ correctCount,
+ )
+ .map(
+ ([arm, n]) =>
+ `${arm.toUpperCase()} (${spec.arms[arm].prop ?? 'no prop'}) ${n}/${reachable[arm]}`,
+ )
+ .join(', ')}.`,
+);
+console.log(
+ `Deprecated prop written: ${
+ Object.entries(deprecatedUse)
+ .map(([arm, n]) => `${arm.toUpperCase()} ${n}/7`)
+ .join(', ') || 'none'
+ }.\n`,
+);
+
+console.log('## Recall (scripted)\n');
+console.log('| sample | prop | look 1 | look 2 | look 3 | look 4 | shape |');
+console.log('| --- | --- | --- | --- | --- | --- | --- |');
+for (let i = 1; i <= spec.recall.samples; i++) {
+ const reply = read(`recall-${i}`);
+ if (reply == null) continue;
+ const api = (reply.match(/```ts\n([\s\S]*?)```/) ?? [, ''])[1];
+ const union = api.match(/(\w+)\??:\s*((?:'[^']+'\s*\|\s*)+'[^']+')/);
+ const alias = api.match(/type\s+\w+\s*=\s*((?:'[^']+'\s*\|\s*)+'[^']+')/);
+ const propLine = api.match(/^\s*(\w+)\??:\s*(\w+);/m);
+ const prop = union?.[1] ?? propLine?.[1] ?? '?';
+ const values =
+ (union?.[2] ?? alias?.[1] ?? '')
+ .match(/'([^']+)'/g)
+ ?.map(v => v.slice(1, -1)) ?? [];
+ const booleans = (api.match(/\w+\??:\s*boolean/g) ?? []).length;
+ const shape =
+ values.length === 4
+ ? 'one prop, four values'
+ : booleans > 0
+ ? `${booleans} booleans`
+ : '?';
+ console.log(
+ `| ${i} | \`${prop}\` | ${values.map(v => `\`${v}\``).join(' | ') || '? | ? | ? | ?'} | ${shape} |`,
+ );
+}
+
+console.log('\n## Reading (scripted)\n');
+console.log(
+ '| prop | value | editor type shown | sample | text | highlight | dividers | all three right | confidence |',
+);
+console.log('| --- | --- | --- | --- | --- | --- | --- | --- | --- |');
+const CANON = {
+ inset: 'inset',
+ padded: 'inset',
+ flush: 'flush',
+ outset: 'outset',
+ hanging: 'outset',
+ bleed: 'bleed',
+};
+const conditions = [
+ ...Object.entries(spec.reading.samples).map(([value, samples]) => ({
+ id: `read-${value}`,
+ prop: 'edge',
+ value,
+ samples,
+ union: null,
+ })),
+ ...(spec.round2?.reading ?? []).map(c => ({
+ ...c,
+ id: `read2-${c.prop}-${c.value}${c.union ? '-union' : ''}`,
+ union: c.union ?? null,
+ })),
+];
+/** @type {Map<string, {right: number, total: number, textRight: number}>} */
+const tally = new Map();
+for (const c of conditions) {
+ for (let i = 1; i <= c.samples; i++) {
+ const reply = read(`${c.id}-${i}`);
+ if (reply == null) continue;
+ const answer = JSON.parse(
+ reply.slice(reply.indexOf('{'), reply.lastIndexOf('}') + 1),
+ );
+ const truth = spec.reading.correct[CANON[c.value]];
+ const mark = key =>
+ answer[key] === truth[key]
+ ? `${answer[key]} ✓`
+ : `${answer[key]} ✗ (${truth[key]})`;
+ const all = ['text', 'highlight', 'dividers'].every(
+ k => answer[k] === truth[k],
+ );
+ console.log(
+ `| \`${c.prop}\` | \`${c.value}\` | ${c.union ? 'yes' : 'no'} | ${c.id.startsWith('read2') ? 'r2' : 'r1'}·${i} | ${mark('text')} | ${mark('highlight')} | ${mark('dividers')} | ${all ? 'yes' : 'no'} | ${answer.confidence}/5 |`,
+ );
+ const key = `${c.prop}|${c.value}|${c.union ? 'type shown' : 'bare'}`;
+ const t = tally.get(key) ?? {right: 0, total: 0, textRight: 0};
+ t.total++;
+ if (all) t.right++;
+ if (answer.text === truth.text) t.textRight++;
+ tally.set(key, t);
+ }
+}
+console.log('\n### Reading totals\n');
+console.log(
+ '| prop | value | editor type shown | all three right | text placement right |',
+);
+console.log('| --- | --- | --- | --- | --- |');
+for (const [key, t] of tally) {
+ const [prop, value, shown] = key.split('|');
+ console.log(
+ `| \`${prop}\` | \`${value}\` | ${shown === 'type shown' ? 'yes' : 'no'} | ${t.right}/${t.total} | ${t.textRight}/${t.total} |`,
+ );
+}
diff --git a/internal/vibe-tests/list-edge-naming-test/gen-docs.mjs b/internal/vibe-tests/list-edge-naming-test/gen-docs.mjs
new file mode 100644
index 0000000000..5caf9269ee
--- /dev/null
+++ b/internal/vibe-tests/list-edge-naming-test/gen-docs.mjs
@@ -0,0 +1,328 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+// Generates one reference doc per arm from a single template.
+//
+// Checker Protocol §2 ("only the system under test varies") is enforced here
+// rather than by inspection:
+// - arm B's doc, with its two differing value names mapped back to arm A's,
+// must equal arm A's doc byte for byte, and so must arm C's with its prop
+// name mapped back and arm D's with its default's name mapped back;
+// - the both-names doc, with its deprecated-prop row and paragraph removed,
+// must equal arm D's doc byte for byte;
+// - the prose outside code spans must not use a word that echoes one arm's
+// value names (pad, inset, inside, outset, outside, hang), and the recall
+// doc's prose must not use a word that would name the prop for the agent.
+// The script exits non-zero when either check fails.
+
+import {mkdirSync, writeFileSync} from 'node:fs';
+import {dirname, join} from 'node:path';
+import {fileURLToPath} from 'node:url';
+
+const HERE = dirname(fileURLToPath(import.meta.url));
+const OUT = join(HERE, 'docs');
+
+/** Arm prop and value names, values in the order inset · flush · outset · bleed. */
+const ARMS = {
+ 'arm-a': {
+ prop: 'edge',
+ inset: 'inset',
+ flush: 'flush',
+ outset: 'outset',
+ bleed: 'bleed',
+ },
+ 'arm-b': {
+ prop: 'edge',
+ inset: 'padded',
+ flush: 'flush',
+ outset: 'hanging',
+ bleed: 'bleed',
+ },
+ // Round 2: the recall probe's most common prop name, with arm A's values.
+ 'arm-c': {
+ prop: 'rowAlignment',
+ inset: 'inset',
+ flush: 'flush',
+ outset: 'outset',
+ bleed: 'bleed',
+ },
+ // Round 3: the names the pull request ships.
+ 'arm-d': {
+ prop: 'edge',
+ inset: 'padded',
+ flush: 'flush',
+ outset: 'outset',
+ bleed: 'bleed',
+ },
+};
+
+const TEMPLATE = `# List
+
+\`\`\`
+import {List, ListItem} from '@astryxdesign/core/List';
+\`\`\`
+
+\`List\` renders a vertical list of rows. Each \`ListItem\` shows a \`label\`, an
+optional \`description\` beneath it, and optional \`startContent\` (an icon or
+avatar) and \`endContent\` (a badge or a price). A row becomes clickable with
+\`onClick\` or \`href\` and shows a highlight while the pointer is over it;
+\`isSelected\` keeps the highlight on.
+
+A list often sits under a heading, and often in a \`Card\`, \`Section\`, or
+\`Dialog\`, each of which keeps its content a set distance from its own
+borders. They come from \`@astryxdesign/core/Card\`, \`@astryxdesign/core/Section\`,
+and \`@astryxdesign/core/Dialog\`; \`Heading\` comes from \`@astryxdesign/core/Text\`.
+
+## Props
+
+### List
+
+| prop | type | default |
+| --- | --- | --- |
+| \`children\` | \`ReactNode\` | — |
+| \`header\` | \`ReactNode\` | — |
+| \`density\` | \`'compact' \\| 'balanced' \\| 'spacious'\` | \`'balanced'\` |
+| \`hasDividers\` | \`boolean\` | \`false\` |
+__EDGE_ROW__
+
+\`header\` renders a label above the rows. \`hasDividers\` draws a thin line
+between rows.
+
+__EDGE_PROSE__
+
+### ListItem
+
+| prop | type |
+| --- | --- |
+| \`label\` | \`ReactNode\` (required) |
+| \`description\` | \`ReactNode\` |
+| \`startContent\` | \`ReactNode\` |
+| \`endContent\` | \`ReactNode\` |
+| \`onClick\` | \`() => void\` |
+| \`href\` | \`string\` |
+| \`isSelected\` | \`boolean\` |
+| \`isDisabled\` | \`boolean\` |
+
+## Examples
+
+Default, no configuration:
+
+\`\`\`tsx
+<List>
+ <ListItem label="Notifications" description="Manage your alerts" />
+ <ListItem label="Privacy" description="Control your data" />
+</List>
+\`\`\`
+
+__EXAMPLE_2__
+
+## Anti-pattern
+
+Don't shift rows with your own styles:
+
+\`\`\`tsx
+/* Don't do this */
+<List style={{marginInline: -8}}>…</List>
+\`\`\`
+
+The highlight and dividers stop matching the row text. __ANTIPATTERN_TAIL__
+`;
+
+/**
+ * The four looks, described once. `label(key)` renders how each look is
+ * named: a value in the arms, "look N" in the recall doc.
+ */
+const LOOK_TEXT = {
+ inset: `text sits a short distance away from the list's sides.
+ The highlight and dividers span the list and extend a little beyond the
+ text, and the text does not line up with a heading above the list: it
+ starts a short distance after it.`,
+ flush: `text, highlight, and dividers all start and end at the
+ list's sides, so the text is in line with a heading above the list. Works
+ anywhere.`,
+ outset: `text is in line with a heading above the list. The
+ highlight and dividers extend a little beyond the text, into the space
+ between the list and the borders of the surrounding \`Card\`, \`Section\`, or
+ \`Dialog\`, without reaching those borders.`,
+ bleed: `text is in line with a heading above the list. The
+ highlight and dividers stretch to the borders of the surrounding \`Card\`,
+ \`Section\`, or \`Dialog\`, like table rows.`,
+};
+
+function looks(label, keys = ['inset', 'flush', 'outset', 'bleed']) {
+ return keys.map(key => `- ${label(key, true)}: ${LOOK_TEXT[key]}`).join('\n');
+}
+
+const NO_CONTAINER = label =>
+ `Without a surrounding \`Card\`, \`Section\`, or \`Dialog\`, ${label('outset')} and
+${label('bleed')} place text, highlight, and dividers as ${label('inset')} does.`;
+
+function armSubs(names) {
+ const value = key => `\`'${names[key]}'\``;
+ const union = ['inset', 'flush', 'outset', 'bleed']
+ .map(key => `'${names[key]}'`)
+ .join(' \\| ');
+ return {
+ EDGE_ROW: `| \`${names.prop}\` | \`${union}\` | \`'${names.inset}'\` |`,
+ EDGE_PROSE: `\`${names.prop}\` sets where each row's text, highlight, and divider sit, relative to
+the list's sides and to the borders of a surrounding \`Card\`, \`Section\`, or
+\`Dialog\`. A list uses one value:
+
+${looks(value)}
+
+${NO_CONTAINER(value)}`,
+ EXAMPLE_2: `Row text in line with a heading above the list:
+
+\`\`\`tsx
+<Heading level={3}>Shortcuts</Heading>
+<List ${names.prop}="${names.flush}">
+ <ListItem label="Search" endContent="⌘K" />
+ <ListItem label="New file" endContent="⌘N" />
+</List>
+\`\`\``,
+ ANTIPATTERN_TAIL: `Use \`${names.prop}\` instead.`,
+ };
+}
+
+// Round 3, both names: the shipped names plus the deprecated prop as the
+// docs show it during its compatibility window.
+const DEPRECATED_ROW = "| `edgeCompensation` | `'inline'` (deprecated) | — |";
+const DEPRECATED_PROSE = `\`edgeCompensation="inline"\` is deprecated. It gives the same look as
+\`edge="outset"\`; use \`edge\` instead. When both are set, \`edge\` wins.`;
+
+function bothSubs() {
+ const subs = armSubs(ARMS['arm-d']);
+ return {
+ ...subs,
+ EDGE_ROW: `${subs.EDGE_ROW}\n${DEPRECATED_ROW}`,
+ EDGE_PROSE: `${subs.EDGE_PROSE}\n\n${DEPRECATED_PROSE}`,
+ };
+}
+
+// Round 3, main baseline: the one edge prop the released List has.
+const MAIN_LABEL = (key, isBullet) =>
+ key === 'inset'
+ ? isBullet
+ ? 'Unset (the default)'
+ : 'the default'
+ : '`edgeCompensation="inline"`';
+const MAIN_SUBS = {
+ EDGE_ROW: "| `edgeCompensation` | `'inline'` | — |",
+ EDGE_PROSE: `\`edgeCompensation\` sets where each row's text, highlight, and divider sit,
+relative to the list's sides and to the borders of a surrounding \`Card\`,
+\`Section\`, or \`Dialog\`. A list leaves it unset or sets \`'inline'\`:
+
+${looks(MAIN_LABEL, ['inset', 'outset'])}
+
+Without a surrounding \`Card\`, \`Section\`, or \`Dialog\`, ${MAIN_LABEL('outset')} places
+text, highlight, and dividers as ${MAIN_LABEL('inset')} does.`,
+ EXAMPLE_2: `Row text in line with a heading above the list, in a card:
+
+\`\`\`tsx
+<Card>
+ <Heading level={3}>Shortcuts</Heading>
+ <List edgeCompensation="inline">
+ <ListItem label="Search" endContent="⌘K" />
+ <ListItem label="New file" endContent="⌘N" />
+ </List>
+</Card>
+\`\`\``,
+ ANTIPATTERN_TAIL: 'Use `edgeCompensation` instead.',
+};
+
+// Round 3, natural reach: no edge API at all, and no anti-pattern steering
+// away from the workaround being measured. Only the default look is described.
+const NONE_SUBS = {
+ EDGE_ROW: '',
+ EDGE_PROSE: `${LOOK_TEXT.inset[0].toUpperCase()}${LOOK_TEXT.inset.slice(1).replaceAll('\n ', '\n')}`,
+ EXAMPLE_2: '',
+ ANTIPATTERN: '',
+};
+
+const LOOK_NUMBERS = {inset: 1, flush: 2, outset: 3, bleed: 4};
+
+const RECALL_SUBS = {
+ EDGE_ROW: '',
+ EDGE_PROSE: `Each list uses one of four row looks. The looks set where each row's text,
+highlight, and divider sit, relative to the list's sides and to the borders of
+a surrounding \`Card\`, \`Section\`, or \`Dialog\`:
+
+${looks(
+ (key, isBullet) =>
+ `${isBullet ? 'Look' : 'look'} ${LOOK_NUMBERS[key]}${key === 'inset' && isBullet ? ' (the default)' : ''}`,
+)}
+
+${NO_CONTAINER(key => `look ${LOOK_NUMBERS[key]}`)}
+
+**The API for choosing a look is not listed in this reference.**`,
+ EXAMPLE_2: '',
+ ANTIPATTERN_TAIL: "Use the component's own API instead.",
+};
+
+function render(subs) {
+ let out = TEMPLATE;
+ if (subs.ANTIPATTERN === '') {
+ out = `${out.slice(0, out.indexOf('## Anti-pattern')).trimEnd()}\n`;
+ }
+ for (const [key, value] of Object.entries(subs)) {
+ out = out.replace(`__${key}__`, value);
+ }
+ return out.replace(/\n{3,}/g, '\n\n');
+}
+
+/** The prose a reader sees outside code fences and inline code. */
+function prose(doc) {
+ return doc.replace(/```[\s\S]*?```/g, ' ').replace(/`[^`]*`/g, ' ');
+}
+
+const ECHO = /pad|inset|inside|outset|outside|hang/i;
+const NAMES_THE_PROP = /edge|inset|bleed|flush|gutter|spacing|padding|align/i;
+
+const docs = {
+ 'arm-a': render(armSubs(ARMS['arm-a'])),
+ 'arm-b': render(armSubs(ARMS['arm-b'])),
+ 'arm-c': render(armSubs(ARMS['arm-c'])),
+ 'arm-d': render(armSubs(ARMS['arm-d'])),
+ 'arm-both': render(bothSubs()),
+ 'arm-main': render(MAIN_SUBS),
+ 'arm-none': render(NONE_SUBS),
+ 'arm-recall': render(RECALL_SUBS),
+};
+
+const problems = [];
+const mappedB = docs['arm-b']
+ .replaceAll("'padded'", "'inset'")
+ .replaceAll("'hanging'", "'outset'");
+if (mappedB !== docs['arm-a']) {
+ problems.push('arm-b differs from arm-a beyond the two value names');
+}
+if (docs['arm-c'].replaceAll('rowAlignment', 'edge') !== docs['arm-a']) {
+ problems.push('arm-c differs from arm-a beyond the prop name');
+}
+if (docs['arm-d'].replaceAll("'padded'", "'inset'") !== docs['arm-a']) {
+ problems.push('arm-d differs from arm-a beyond the default name');
+}
+if (
+ docs['arm-both']
+ .replace(`\n${DEPRECATED_ROW}`, '')
+ .replace(`\n\n${DEPRECATED_PROSE}`, '') !== docs['arm-d']
+) {
+ problems.push('arm-both differs from arm-d beyond the deprecated prop');
+}
+for (const [name, doc] of Object.entries(docs)) {
+ const echo = prose(doc).match(ECHO);
+ if (echo) problems.push(`${name}: prose echoes a value name ("${echo[0]}")`);
+}
+const naming = prose(docs['arm-recall']).match(NAMES_THE_PROP);
+if (naming) problems.push(`arm-recall: prose names the prop ("${naming[0]}")`);
+
+mkdirSync(OUT, {recursive: true});
+for (const [name, doc] of Object.entries(docs)) {
+ writeFileSync(join(OUT, `${name}.md`), doc);
+ const words = doc.split(/\s+/).filter(Boolean).length;
+ console.log(`${name.padEnd(12)} ${String(words).padStart(4)} words`);
+}
+if (problems.length > 0) {
+ console.error(problems.join('\n'));
+ process.exit(1);
+}
+console.log('parity and echo checks passed');
diff --git a/internal/vibe-tests/list-edge-naming-test/prompts.json b/internal/vibe-tests/list-edge-naming-test/prompts.json
new file mode 100644
index 0000000000..2ce3eafb50
--- /dev/null
+++ b/internal/vibe-tests/list-edge-naming-test/prompts.json
@@ -0,0 +1,238 @@
+{
+ "note": "Identical across arms. No generation prompt names a prop, a value, or the words padding, inset, edge, margin, or align. Ground truth (`correct`) is for scoring only and never reaches a generating agent. Canonical look keys: inset, flush, outset, bleed; arm B names inset 'padded' and outset 'hanging'.",
+ "arms": {
+ "a": {
+ "doc": "docs/arm-a.md",
+ "values": {
+ "inset": "inset",
+ "flush": "flush",
+ "outset": "outset",
+ "bleed": "bleed"
+ },
+ "prop": "edge"
+ },
+ "b": {
+ "doc": "docs/arm-b.md",
+ "values": {
+ "inset": "padded",
+ "flush": "flush",
+ "outset": "hanging",
+ "bleed": "bleed"
+ },
+ "prop": "edge"
+ },
+ "c": {
+ "doc": "docs/arm-c.md",
+ "prop": "rowAlignment",
+ "round": 2,
+ "values": {
+ "inset": "inset",
+ "flush": "flush",
+ "outset": "outset",
+ "bleed": "bleed"
+ }
+ },
+ "d": {
+ "doc": "docs/arm-d.md",
+ "prop": "edge",
+ "round": 3,
+ "values": {
+ "inset": "padded",
+ "flush": "flush",
+ "outset": "outset",
+ "bleed": "bleed"
+ }
+ },
+ "both": {
+ "doc": "docs/arm-both.md",
+ "prop": "edge",
+ "round": 3,
+ "values": {
+ "inset": "padded",
+ "flush": "flush",
+ "outset": "outset",
+ "bleed": "bleed"
+ },
+ "deprecated": {
+ "prop": "edgeCompensation",
+ "value": "inline",
+ "look": "outset"
+ }
+ },
+ "main": {
+ "doc": "docs/arm-main.md",
+ "prop": "edgeCompensation",
+ "round": 3,
+ "values": {
+ "outset": "inline"
+ }
+ },
+ "none": {
+ "doc": "docs/arm-none.md",
+ "prop": null,
+ "round": 3,
+ "values": {}
+ }
+ },
+ "preamble": "You are running a vibe test.\n\nYou have NO prior knowledge of the system under test. Do NOT use prior knowledge of any specific library, tool, product, or convention. Do NOT use any other tool: do not search, read other files, browse, run code, or look anything up. Everything you may use is in this task.",
+ "generation": {
+ "output": "## Output\n\nReply with exactly three parts:\n1. Your code, in one ```tsx fenced block.\n2. One line: `Confidence: N/5`.\n3. `Notes:` what you reached for first, where you hesitated, anything you wanted but couldn't find, and any workarounds you used.",
+ "prompts": [
+ {
+ "id": "p1",
+ "correct": "flush",
+ "why": "No container; text and dividers on the heading's line. Only flush does that outside a container.",
+ "text": "Build the ingredients section of a recipe page: an 'Ingredients' heading followed by a list of six ingredients, each with its amount as smaller secondary text, with a thin line between ingredients. The section sits directly on the page, not inside a card. The ingredient names and the lines between them must start exactly where the heading's text starts, so the heading and the list read as one tidy column."
+ },
+ {
+ "id": "p2",
+ "correct": "bleed",
+ "why": "Lines touch the card's borders while names stay on the title's line.",
+ "text": "Build an order summary card: a card titled 'Items' listing the five products in an order, each with its price on the right, with a thin line between products. Like rows in a table, each line should run all the way across the card, touching the card's left and right borders, while the product names stay lined up exactly with the card's title."
+ },
+ {
+ "id": "p3",
+ "correct": "flush",
+ "why": "In a card, names and lines start on the title's line and stop short of the borders.",
+ "text": "Build an 'Emergency contacts' card: a card with that title and a list of three people, each with their phone number as smaller secondary text, with a thin line between people. The names and the lines between them should both start exactly where the title's text starts, and the lines must stop the same distance short of the card's right border as they start from its left border; they should not touch the card's borders."
+ },
+ {
+ "id": "p4",
+ "correct": "outset",
+ "why": "Names on the title's line; the highlight reaches a little past them without touching the borders.",
+ "text": "Build a folder picker inside a card titled 'Folders': Inbox, Drafts, Sent, and Archive, each with an icon, each clickable, with the current folder highlighted. The folder icons and names must line up exactly with the card's title. The highlight behind a folder should extend a little past the icon and name on both sides, so it doesn't look cramped, but it must not touch the card's borders."
+ },
+ {
+ "id": "p5",
+ "correct": "bleed",
+ "why": "The highlight touches the card's borders while labels stay on the title's line.",
+ "text": "Build an account settings menu inside a card titled 'Account': Profile, Security, Notifications, and Billing, each clickable to open its page. When the pointer is over an item, its highlight should stretch the full width of the card, touching both of the card's side borders, while the item labels stay lined up exactly with the card's title."
+ },
+ {
+ "id": "p6",
+ "correct": "unset",
+ "why": "Negative control: nothing to line up with, so the default is right. Writing the default value explicitly is acceptable but not direct.",
+ "text": "Build the contents of a 'Recent files' popup: a list of the five most recently opened files, each with its last-opened date as smaller secondary text, each clickable to open the file. Assume the popup itself already exists; build only what goes inside it. Nothing else in the popup sits next to the list: there is no title or other text it needs to line up with."
+ },
+ {
+ "id": "p7",
+ "correct": "conflict",
+ "why": "No container, but the names must be on the heading's line and the highlight must reach past them. No value alone does both: outset needs a container. Credit for noticing and choosing a documented answer (wrap the heading and list in a Section or Card and use outset, or pick flush or the default and name the tradeoff). Claiming outset works here, or custom margins, is wrong.",
+ "text": "Build the project navigation for a page sidebar: a 'Projects' heading directly on the sidebar (not inside a card), followed by five projects, each clickable, with the current project highlighted. The project names must line up exactly with the heading's text, and the highlight should extend a little past each name on both sides so it doesn't look cramped."
+ }
+ ]
+ },
+ "recall": {
+ "doc": "docs/arm-recall.md",
+ "samples": 6,
+ "note": "One agent writes the API it expects and uses it for three tasks, because naming a coherent set is the realistic act.",
+ "scenarios": ["p1", "p4", "p5"],
+ "output": "## Output\n\nReply with:\n1. `API:` how you would expect this library to let you choose a look: the TypeScript prop declaration(s) on `List`, with types and defaults, and one line per look saying what selects it.\n2. One ```tsx block per task, labeled Task 1, Task 2, and Task 3.\n3. `Naming notes:` the names you considered and rejected, and why."
+ },
+ "reading": {
+ "note": "No reference. The agent meets one value in code and predicts the result. Two samples for each arm-specific value, one each for the shared flush and bleed.",
+ "samples": {
+ "inset": 2,
+ "padded": 2,
+ "outset": 2,
+ "hanging": 2,
+ "flush": 1,
+ "bleed": 1
+ },
+ "snippet": "```tsx\nimport {Card} from '@astryxdesign/core/Card';\nimport {Heading} from '@astryxdesign/core/Text';\nimport {List, ListItem} from '@astryxdesign/core/List';\n\nexport function TeamCard({onOpen}: {onOpen: (id: string) => void}) {\n return (\n <Card>\n <Heading level={3}>Team</Heading>\n <List __PROP__=\"__VALUE__\" hasDividers>\n <ListItem label=\"Ada Lovelace\" description=\"Engineer\" onClick={() => onOpen('ada')} />\n <ListItem label=\"Grace Hopper\" description=\"Admiral\" onClick={() => onOpen('grace')} isSelected />\n </List>\n </Card>\n );\n}\n```",
+ "questions": "You have no documentation for this design system. From the code alone, predict what it renders.\n\n1. text: Do the names \"Ada Lovelace\" and \"Grace Hopper\" start (a) exactly in line with the heading \"Team\", (b) a little after it, or (c) a little before it?\n2. highlight: Where does the selected row's highlight start and end horizontally: (a) exactly at the start and end of the row's text, (b) a little beyond the text but not reaching the card's borders, or (c) at the card's left and right borders?\n3. dividers: Where does the line between the two rows start and end: (a) exactly at the start and end of the row's text, (b) a little beyond the text but not reaching the card's borders, or (c) at the card's left and right borders?\n4. otherValues: Which other values do you think `__PROP__` accepts?\n\nReply with one JSON object and nothing else: {\"text\": \"a|b|c\", \"highlight\": \"a|b|c\", \"dividers\": \"a|b|c\", \"otherValues\": [\"...\"], \"confidence\": 1-5, \"reasoning\": \"one or two sentences\"}",
+ "correct": {
+ "inset": {
+ "text": "b",
+ "highlight": "b",
+ "dividers": "b"
+ },
+ "flush": {
+ "text": "a",
+ "highlight": "a",
+ "dividers": "a"
+ },
+ "outset": {
+ "text": "a",
+ "highlight": "b",
+ "dividers": "b"
+ },
+ "bleed": {
+ "text": "a",
+ "highlight": "c",
+ "dividers": "c"
+ }
+ }
+ },
+ "judge": {
+ "dimensions": {
+ "correctness": "The rendered result matches `correct` for the prompt.",
+ "hallucination": "No props, values, or components the reference does not document. 5 = none.",
+ "directness": "No unrequested code, wrappers, or custom styles; does not restate defaults.",
+ "calibration": "Stated confidence matches correctness; doubts named where the reference was unclear."
+ },
+ "tieBreak": "Fewer misreads and invented props, then plainer words."
+ },
+ "round2": {
+ "note": "Round 2 adds the recall probe's most common prop name (arm C, rowAlignment with arm A's values) to the use probe, and gives the reading probe more samples where round 1 split or was thin: the default's name, the prop name, and flush. The `union` condition adds the type an editor shows on hover.",
+ "use": {
+ "arms": ["c"]
+ },
+ "reading": [
+ {
+ "prop": "edge",
+ "value": "inset",
+ "samples": 4
+ },
+ {
+ "prop": "edge",
+ "value": "padded",
+ "samples": 4
+ },
+ {
+ "prop": "edge",
+ "value": "inset",
+ "union": ["inset", "flush", "outset", "bleed"],
+ "samples": 3
+ },
+ {
+ "prop": "edge",
+ "value": "padded",
+ "union": ["padded", "flush", "outset", "bleed"],
+ "samples": 3
+ },
+ {
+ "prop": "edge",
+ "value": "flush",
+ "samples": 2
+ },
+ {
+ "prop": "rowAlignment",
+ "value": "inset",
+ "samples": 3
+ },
+ {
+ "prop": "rowAlignment",
+ "value": "padded",
+ "samples": 3
+ },
+ {
+ "prop": "rowAlignment",
+ "value": "outset",
+ "samples": 2
+ },
+ {
+ "prop": "rowAlignment",
+ "value": "bleed",
+ "samples": 2
+ }
+ ],
+ "unionHint": "Your editor shows this type when you hover the prop: `__PROP__?: __UNION__`. There is no other documentation."
+ },
+ "round3": {
+ "note": "Round 3 adds the arms a review asked for, with the same seven use prompts. Main baseline: the released List, whose only edge prop is edgeCompensation. Natural reach: no edge API and no anti-pattern, to see what agents reach for unaided. Both names: the shipped names plus the deprecated edgeCompensation, as the docs read during its compatibility window. Arm D, the shipped names alone, is the control the both-names arm is compared against.",
+ "use": {
+ "arms": ["d", "both", "main", "none"]
+ }
+ }
+}