Enhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active
TypeScript aianthropicartifactsawsazurechatgptchatgpt-cloneclaudeclonedeepseekgeminigooglegpt-5librechatmcpo1openairesponses-apivisionwebui
🧪 chore: Add the Activity-Label Prose Eval Harness (#14527)
Danny Avilaa07c0e4ae819
diff --git a/.gitignore b/.gitignoreindex 89d6c2e99a..836745d271 100644--- a/.gitignore+++ b/.gitignore@@ -183,3 +183,4 @@ hive-mind-prompt-*.txtCLAUDE.md.gsdcodedb.snapshot+scripts/activity-labels/results/diff --git a/scripts/activity-labels/README.md b/scripts/activity-labels/README.mdnew file mode 100644index 0000000000..e98d6dfd73--- /dev/null+++ b/scripts/activity-labels/README.md@@ -0,0 +1,57 @@+# Activity-label eval harness++Measures the prose quality of fast-model activity-label headers (the collapsed+group headers produced by `activityLabel`). Instruction changes are graded+against a fixed corpus instead of eyeballed on one conversation — the two+hypotheses that felt most obvious when this was built both turned out wrong+under measurement (see **Findings**).++```bash+node scripts/activity-labels/run.js # every variant, 1 sample+node scripts/activity-labels/run.js --samples 3 # 3 samples each+node scripts/activity-labels/run.js --variants baseline,shipping --cases fib-rapid+node scripts/activity-labels/run.js --dry --cases mega-batch # render prompts, no API calls+node scripts/activity-labels/rescore.js # re-grade stored results, no re-spend+```++Requires `ANTHROPIC_API_KEY` (env or `.env`). A full sweep is roughly $0.03 per+variant and ~45s. Results land in `results/` (gitignored): a timestamped JSON+of every record plus `latest.md` with per-case tables.++## Why the tables matter more than the aggregate++The mechanical checks catch format violations and lexical repetition, but the+failure this feature actually had in production — headers that were+_informationally_ redundant while lexically varied — scores below the overlap+threshold. `results/latest.md` read by eye is the recall instrument; the+aggregate is the regression guard.++## Layout++| File | Role |+| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `captured.json` | 9 real production payloads pulled verbatim from Langfuse, with the headers that shipped. Irreplaceable — traces age out. |+| `corpus.js` | 17 cases / 28 steps: the captured run as one sequence, plus synthetic cases for modes it never hit (all-failed, partial, parallel, rapid duplicates, entry overflow, truncated output, error-shaped success). Multi-step cases chain each generated label into the next step's context. |+| `prompt.js` | Faithful port of the SDK's `buildActivityLabelPrompt`, so synthetic cases render the bytes production would send. Adds `previousLabelCap` for continuity-window experiments. |+| `variants.js` | Single-factor instruction variants. `baseline` is read from the built `packages/api` dist, so drift from the shipped instruction is impossible. |+| `checks.js` | Mechanical grading: length, punctuation, markdown, tool-name echo, count echo, and overlap split into `restate` (adds nothing) vs `template` (same frame, new payload). |+| `run.js` / `rescore.js` | Live runner (production wire shape, `max_tokens: 256`) and offline re-grader. |++## Findings this produced++- **Sentence order is load-bearing.** Moving format constraints after content+ rules cut length violations and register collapse measurably. This is why+ `ACTIVITY_INSTRUCTION` is ordered the way it is — a "tidying" reshuffle+ regresses real output.+- **Enumerating verbs backfires.** An instruction listing acceptable opening+ verbs _anchored_ the model: `Confirmed` went from 18 to 23 occurrences and+ opener diversity halved.+- **Diverse examples alone changed nothing.** Register collapse is task-shaped,+ not example-seeded.+- **Continuity context fixed the real defect.** Feeding committed headers back+ eliminated restatement and, unexpectedly, stopped setup batches from being+ labeled with conclusions their tools had not yet established.+- **A 3-label window is enough.** Unbounded history scored no better —+ restatement is inherently a recency problem — while prompt growth is linear+ (+82 input tokens by batch 9, extrapolating to ~+250 at the `activityMaxPerRun`+ default of 20).diff --git a/scripts/activity-labels/captured.json b/scripts/activity-labels/captured.jsonnew file mode 100644index 0000000000..30e306b741--- /dev/null+++ b/scripts/activity-labels/captured.json@@ -0,0 +1,56 @@+[+ {+ "id": "sandbox-python-version",+ "traceId": "eeab6f47000fe2a6aac76cde6bbc67fd",+ "prompt": "Reasoning excerpts:\n- I'll work through each check methodically, running them one at a time and documenting what I find.\n\nTool calls:\n- run_tools_with_bash({\"code\":\"python3 --version\"}) → stdout:\nPython 3.14.4\n\nLabel:",+ "productionLabel": "Confirmed Python 3.14.4 is installed"+ },+ {+ "id": "mnt-data-write-test",+ "traceId": "8ee41468850d42e5477af07759519e84",+ "prompt": "Intent (assistant's last message): **Python version: 3.14.4**\n\nNow let's check if files in /mnt/data survive between calls.\n\nTool calls:\n- bash_tool({\"command\":\"echo \\\"hello from probe 1, written at $(date)\\\" > /mnt/data/persist_test.txt; cat /mnt/data/persist_test.txt\"}) → stdout:\nhello from probe 1, written at Wed Jul 29 11:30:09 AM UTC 2026\n\nLabel:",+ "productionLabel": "Wrote test file to /mnt/data, confirmed persistence"+ },+ {+ "id": "mnt-data-persistence",+ "traceId": "64b318bd7251b66baccd75009d947682",+ "prompt": "Intent (assistant's last message): **Python version: 3.14.4**\n\nNow let's check if files in /mnt/data survive between calls.\n\nReasoning excerpts:\n- I should make a second request to verify the behavior persists across multiple calls.\n\nTool calls:\n- bash_tool({\"command\":\"ls -la /mnt/data/persist_test.txt && cat /mnt/data/persist_test.txt\"}) → stdout:\n-rw------- 1 65534 65534 63 Jul 29 11:30 /mnt/data/persist_test.txt\nhello from probe 1, written at Wed Jul 29 11:30:09 AM UTC 2026\n\nLabel:",+ "productionLabel": "Confirmed /mnt/data persists between tool calls"+ },+ {+ "id": "mnt-data-tmpfs",+ "traceId": "bff7f9766cc2916230403a04c47c92af",+ "prompt": "Intent (assistant's last message): **Files in /mnt/data DO survive between calls** — the file I wrote in the previous, separate sandbox invocation is still there with its original content intact.\n\nNow let's check RAM and disk.\n\nTool calls:\n- bash_tool({\"command\":\"echo \\\"=== MEMORY ===\\\"; free -h; echo; echo \\\"=== DISK ===\\\"; df -h /mnt/data / /tmp 2>/dev/null\"}) → stdout:\n=== MEMORY ===\n\n=== DISK ===\nFilesystem Size Used Avail Use% Mounted on\ntmpfs 1.0G 32K 1.0G 1% /mnt/data\nnone 16M 0 16M 0% /\nnone 20M 0 20M 0% /tmp\n\nstderr:\nfree: Unable to create meminfo structure\nNote: /tmp files are same-call scratch only and were not persisted; use /mnt/data for files needed later.\n\nLabel:",+ "productionLabel": "Confirmed /mnt/data is tmpfs with 1GB capacity"+ },+ {+ "id": "cpu-meminfo-disk",+ "traceId": "6faf26136e125c29fe0bae7ddd87c59e",+ "prompt": "Intent (assistant's last message): `free` failed, let me try /proc/meminfo directly — and note disk is quite constrained (tmpfs-based, tiny root/tmp).\n\nTool calls:\n- bash_tool({\"command\":\"echo \\\"=== /proc/meminfo ===\\\"; cat /proc/meminfo 2>/dev/null | head -5\\necho\\necho \\\"=== nproc ===\\\"; nproc\\necho\\necho \\\"=== /proc/cpuinfo count ===\\\"; grep -c ^processor /proc/cpuinfo 2>/dev/null\\necho\\necho \\\"=== disk full df ===\\\"; df -h 2>/dev/null\"}) → stdout:\n=== /proc/meminfo ===\n\n=== nproc ===\n1\n\n=== /proc/cpuinfo count ===\n0\n\n=== disk full df ===\nFilesystem Size Used Avail Use% Mounted on\nnone 16M 0 16M 0% /\n/dev/vdc 16G 4.4G 11G 30% /usr\nnone 16M 4.0K 16M 1% /etc/hosts\ntmpfs 64M 0 64M 0% /dev/null\nnone 20M 0 20M 0% /tmp\ntmpfs 1.0G 32K 1.0G 1% /mnt/data\n\nLabel:",+ "productionLabel": "Confirmed single CPU, /proc/meminfo empty, disk constrained with 16M root and 20M tmp"+ },+ {+ "id": "rlimit-cgroup",+ "traceId": "2797232991aa7e841081a6b6175bd62b",+ "prompt": "Intent (assistant's last message): `/proc/meminfo` is empty/restricted, but I can check cgroup limits for a real memory figure.\n\nTool calls:\n- bash_tool({\"command\":\"echo \\\"=== cgroup memory limit ===\\\"\\nfor f in /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory/memory.limit_in_bytes; do\\n [ -f \\\"$f\\\" ] && echo \\\"$f: $(cat $f)\\\"\\ndone\\necho\\necho \\\"=== cgroup cpu ===\\\"\\nfor f in /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu/cpu.cfs_quota_us; do\\n [ -f \\\"$f\\\" ] && echo \\\"$f: $(cat $f)\\\"\\ndone\\necho\\npython3 -c \\\"import resource; print('soft/hard RLIMIT_AS:', resource.getrlimit(resource.RLIMIT_AS))\\\" 2>/dev/null\"}) → stdout:\n=== cgroup memory limit ===\n\n=== cgroup cpu ===\n\nsoft/hard RLIMIT_AS: (17179869184, 17179869184)\n\nLabel:",+ "productionLabel": "Found RLIMIT_AS ceiling at 16GB, cgroup limits unavailable"+ },+ {+ "id": "network-tools",+ "traceId": "e813a6ee932da05d573ced56424265b8",+ "prompt": "Intent (assistant's last message): No cgroup files readable, but RLIMIT_AS shows a **16 GiB address-space limit** — likely the effective memory ceiling. Now let's check network access.\n\nTool calls:\n- bash_tool({\"command\":\"echo \\\"=== ping ===\\\"; timeout 3 ping -c1 8.8.8.8 2>&1\\necho\\necho \\\"=== curl ===\\\"; timeout 5 curl -sS -o /dev/null -w \\\"HTTP_CODE:%{http_code}\\\\n\\\" https://www.google.com 2>&1\\necho\\necho \\\"=== curl example.com ===\\\"; timeout 5 curl -sS -o /dev/null -w \\\"HTTP_CODE:%{http_code}\\\\n\\\" http://example.com 2>&1\\necho\\necho \\\"=== DNS resolution ===\\\"; timeout 3 getent hosts google.com 2>&1\"}) → stdout:\n=== ping ===\ntimeout: failed to run command ‘ping’: No such file or directory\n\n=== curl ===\ntimeout: failed to run command ‘curl’: No such file or directory\n\n=== curl example.com ===\ntimeout: failed to run command ‘curl’: No such file or directory\n\n=== DNS resolution ===\n\nLabel:",+ "productionLabel": "Network tools unavailable, DNS resolution failed"+ },+ {+ "id": "dns-tcp",+ "traceId": "d96b49502c4a0ecb4c9325c2e74b43bf",+ "prompt": "Intent (assistant's last message): No curl/ping binaries — let me test connectivity directly via Python sockets/urllib instead.\n\nTool calls:\n- bash_tool({\"command\":\"python3 - <<'EOF'\\nimport socket, urllib.request, time\\n\\n# DNS test\\ntry:\\n ip = socket.gethostbyname(\\\"example.com\\\")\\n print(\\\"DNS resolution OK:\\\", ip)\\nexcept Exception as e:\\n print(\\\"DNS resolution FAILED:\\\", repr(e))\\n\\n# raw TCP connect test\\ntry:\\n s = socket.create_connection((\\\"8.8.8.8\\\", 53), timeout=4)\\n print(\\\"TCP connect to 8.8.8.8:53 OK\\\")\\n s.close()\\nexcept Exception as e:\\n print(\\\"TCP connect FAILED:\\\", repr(e))\\n\\n# HTTP fetch test\\ntry:\\n t0 = time.time()\\n r = urllib.request.urlopen(\\\"http://example.com\\\", timeout=5)\\n print(\\…) → stdout:\nDNS resolution FAILED: gaierror(-2, 'Name or service not known')\nTCP connect FAILED: PermissionError(1, 'Operation not permitted')\nHTTP fetch FAILED: URLError(gaierror(-3, 'Temporary failure in name resolution'))\n\nLabel:",+ "productionLabel": "DNS and TCP connectivity both blocked or unavailable"+ },+ {+ "id": "python-packages",+ "traceId": "a6fcd0fd40b490285ced22c236023e2c",+ "prompt": "Intent (assistant's last message): **No network access** — DNS resolution fails and raw TCP connections are blocked with a permission error. This sandbox is fully network-isolated.\n\nNow let's check for numpy/pandas/scipy/torch.\n\nTool calls:\n- bash_tool({\"command\":\"python3 - <<'EOF'\\nmods = [\\\"numpy\\\", \\\"pandas\\\", \\\"scipy\\\", \\\"torch\\\"]\\nfor m in mods:\\n try:\\n mod = __import__(m)\\n ver = getattr(mod, \\\"__version__\\\", \\\"unknown\\\")\\n print(f\\\"{m}: INSTALLED, version {ver}\\\")\\n except ImportError as e:\\n print(f\\\"{m}: NOT INSTALLED ({e})\\\")\\nEOF\"}) → stdout:\nnumpy: INSTALLED, version 2.4.6\npandas: INSTALLED, version 2.3.3\nscipy: INSTALLED, version 1.18.0\ntorch: NOT INSTALLED (No module named 'torch')\n\nLabel:",+ "productionLabel": "Confirmed numpy, pandas, scipy installed; torch unavailable"+ }+]diff --git a/scripts/activity-labels/checks.js b/scripts/activity-labels/checks.jsnew file mode 100644index 0000000000..8acbea7b88--- /dev/null+++ b/scripts/activity-labels/checks.js@@ -0,0 +1,156 @@+/**+ * Mechanical label checks. These catch format violations and the two+ * measurable prose failures (register collapse via first-word tallies,+ * cross-batch redundancy via content-word overlap); commit-log READABILITY+ * still needs the human pass over results/latest.md.+ */+const STOPWORDS = new Set([+ 'the',+ 'a',+ 'an',+ 'to',+ 'of',+ 'and',+ 'or',+ 'with',+ 'for',+ 'in',+ 'on',+ 'at',+ 'is',+ 'are',+ 'was',+ 'were',+ 'be',+ 'been',+ 'it',+ 'its',+ 'as',+ 'by',+ 'from',+ 'that',+ 'this',+ 'both',+ 'all',+ 'no',+ 'not',+ 'via',+]);++function words(label) {+ return label.trim().split(/\s+/).filter(Boolean);+}++const SUFFIXES = ['ations', 'ation', 'ence', 'ance', 'ings', 'ing', 'ed', 'es', 's'];++/** Crude suffix stemmer so persists/persistence/persisted collide — enough+ * for overlap detection; linguistic correctness is not the goal. */+function stem(word) {+ if (word.length < 5) {+ return word;+ }+ for (const suffix of SUFFIXES) {+ if (word.endsWith(suffix) && word.length - suffix.length >= 4) {+ return word.slice(0, word.length - suffix.length);+ }+ }+ return word;+}++function contentWords(label) {+ return words(label.toLowerCase().replace(/[^a-z0-9/._-]+/g, ' '))+ .filter((word) => !STOPWORDS.has(word))+ .map(stem);+}++/** Payload tokens carry the informative delta between template-shaped+ * labels: numbers, versions, paths, filenames. */+function isPayload(word) {+ return /\d/.test(word) || word.includes('/') || word.includes('.');+}++function jaccard(a, b) {+ const setA = new Set(a);+ const setB = new Set(b);+ if (setA.size === 0 || setB.size === 0) {+ return 0;+ }+ let intersection = 0;+ for (const word of setA) {+ if (setB.has(word)) {+ intersection += 1;+ }+ }+ return intersection / (setA.size + setB.size - intersection);+}++const GENERIC_OPENER = /^(ran|used|executed|called|invoked|performed)\b/i;+const COUNT_ECHO = /\b\d+\s+(tools?|commands?|calls?)\b/i;+const DUP_THRESHOLD = 0.5;++/**+ * @param label generated label text+ * @param entries the batch's tool entries (for tool-name echo detection)+ * @param previousLabels labels generated EARLIER in the same case chain,+ * regardless of whether the variant saw them — redundancy is measured+ * uniformly so continuity variants can be compared against blind ones.+ */+function checkLabel(label, { entries = [], previousLabels = [] } = {}) {+ const flags = [];+ const wordList = words(label);+ if (wordList.length < 4 || wordList.length > 9) {+ flags.push(`len:${wordList.length}`);+ }+ if (/[.!?,;:]$/.test(label.trim())) {+ flags.push('punct');+ }+ if (/^["'`]|["'`]$/.test(label.trim())) {+ flags.push('quote');+ }+ if (/[*`]|^#|\[.*\]\(/.test(label)) {+ flags.push('md');+ }+ if (GENERIC_OPENER.test(label.trim())) {+ flags.push('opener');+ }+ const lower = label.toLowerCase();+ for (const entry of entries) {+ const name = String(entry.toolName ?? '').toLowerCase();+ if (name.length > 3 && (lower.includes(name) || lower.includes(name.replace(/_/g, ' ')))) {+ flags.push(`tool-echo:${entry.toolName}`);+ break;+ }+ }+ if (COUNT_ECHO.test(label)) {+ flags.push('count-echo');+ }+ /** Overlap splits into two flags: `restate` (high overlap, no payload+ * delta — the line adds nothing over a previous header; the production+ * 2/3 and 7/8 failure) and `template` (high overlap but the differing+ * tokens are numbers/paths — same sentence frame, new information, e.g.+ * fib(1)→fib(2). Often fine, arguably better than synonym churn). */+ const own = contentWords(label);+ let maxOverlap = 0;+ let worst = null;+ for (const previous of previousLabels) {+ const other = contentWords(previous);+ const overlap = jaccard(own, other);+ if (overlap > maxOverlap) {+ maxOverlap = overlap;+ worst = other;+ }+ }+ if (maxOverlap > DUP_THRESHOLD && worst != null) {+ const otherSet = new Set(worst);+ const ownSet = new Set(own);+ const differing = [+ ...own.filter((word) => !otherSet.has(word)),+ ...worst.filter((word) => !ownSet.has(word)),+ ];+ const informativeDelta = differing.some(isPayload);+ flags.push(`${informativeDelta ? 'template' : 'restate'}:${maxOverlap.toFixed(2)}`);+ }+ return { flags, wordCount: wordList.length, firstWord: wordList[0] ?? '', maxOverlap };+}++module.exports = { checkLabel, contentWords, jaccard, words, stem };diff --git a/scripts/activity-labels/corpus.js b/scripts/activity-labels/corpus.jsnew file mode 100644index 0000000000..bacec6104c--- /dev/null+++ b/scripts/activity-labels/corpus.js@@ -0,0 +1,455 @@+/**+ * Eval corpus for activity-label prose. Two halves:+ *+ * - captured.json: the 9 real payloads from the 2026-07-29 sandbox-probe run,+ * verbatim from Langfuse, replayed as ONE sequence so continuity variants+ * see the same run shape production did. `productionLabel` is what shipped.+ * - synthetic: cases built for the failure modes the captured run surfaced+ * (redundant consecutive batches, register collapse, length overflow) plus+ * the modes it never exercised (all-failed, partial, parallel columns,+ * truncation, entry overflow, error-shaped success).+ *+ * A case is a sequence of steps; a step is one label request. Multi-step+ * cases exist to measure cross-batch redundancy: the runner chains each+ * step's generated label into the next step's `previousLabels` for variants+ * that opt in.+ */+const fs = require('fs');+const path = require('path');++const captured = JSON.parse(fs.readFileSync(path.join(__dirname, 'captured.json'), 'utf8'));++const capturedRun = {+ id: 'sandbox-probe-run',+ notes: 'the real 9-batch production run, verbatim payloads',+ steps: captured.map((entry) => ({+ id: entry.id,+ verbatim: entry.prompt,+ productionLabel: entry.productionLabel,+ })),+};++const synthetic = [+ {+ id: 'all-failed',+ notes: 'every call fails — failure register, verb-first under failure',+ steps: [+ {+ payload: {+ lastAssistantText: "I'll run each of these and report exactly what happens.",+ entries: [+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'cat /etc/shadow' },+ status: 'error',+ error: 'cat: /etc/shadow: Permission denied',+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'ls /nonexistent-dir' },+ status: 'error',+ error: "ls: cannot access '/nonexistent-dir': No such file or directory",+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'curl -sS https://nope.invalid' },+ status: 'error',+ error: 'curl: (6) Could not resolve host: nope.invalid',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'partial-failure',+ notes: 'mixed batch — must not read as all-success or all-failure',+ steps: [+ {+ payload: {+ thinkingExcerpts: [+ 'Three probes: create the marker dir, read the shadow file, resolve an invalid host. The first should work, the other two should fail for different reasons.',+ ],+ entries: [+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'mkdir -p /tmp/probe && echo ok' },+ toolOutput: 'stdout:\nok',+ status: 'success',+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'cat /etc/shadow' },+ status: 'error',+ error: 'cat: /etc/shadow: Permission denied',+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'getent hosts nope.invalid' },+ status: 'error',+ error: 'exit code 2',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'parallel-versions',+ notes: 'one batch of parallel lookups — the groupId/parallel-columns shape',+ steps: [+ {+ payload: {+ lastAssistantText: 'Let me look up all three at once.',+ entries: [+ {+ toolName: 'web_search',+ toolInput: { query: 'Node.js latest stable version 2026' },+ toolOutput:+ 'Node.js 24.5.0 (Current) released 2026-07-22; v24 enters LTS October 2026. nodejs.org/en/blog/release/v24.5.0',+ status: 'success',+ },+ {+ toolName: 'web_search',+ toolInput: { query: 'Deno latest release version' },+ toolOutput:+ 'Deno 2.4.2 released 2026-07-16 with improved node:sqlite compat. deno.com/blog/v2.4',+ status: 'success',+ },+ {+ toolName: 'web_search',+ toolInput: { query: 'Bun latest release version' },+ toolOutput:+ 'Bun 1.2.19 released 2026-07-25, adds --compile cross-target for linux-arm64. bun.sh/blog/bun-v1.2.19',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'fib-rapid',+ notes: 'three near-identical consecutive batches — redundancy stress',+ steps: [1, 2, 3].map((n) => ({+ id: `fib-${n}`,+ payload: {+ entries: [+ {+ toolName: 'execute_code',+ toolInput: { code: `print(fib(${n}))` },+ toolOutput: `stdout:\n${[1, 1, 2][n - 1]}`,+ status: 'success',+ },+ ],+ },+ })),+ },+ {+ id: 'mega-batch',+ notes: 'six heterogeneous probes in one batch — length-cap stress (mirrors cpu-meminfo-disk)',+ steps: [+ {+ payload: {+ thinkingExcerpts: [+ "I'll gather the full system picture in one pass: CPU count, memory, disk, limits, user, kernel.",+ ],+ entries: [+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'nproc' },+ toolOutput: 'stdout:\n1',+ status: 'success',+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'cat /proc/meminfo | head -3' },+ toolOutput: 'stdout:\n',+ status: 'success',+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'df -h / /tmp' },+ toolOutput:+ 'stdout:\nFilesystem Size Used Avail Use% Mounted on\noverlay 16M 12M 4.0M 75% /\ntmpfs 20M 0 20M 0% /tmp',+ status: 'success',+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'ulimit -v' },+ toolOutput: 'stdout:\n16777216',+ status: 'success',+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'whoami' },+ toolOutput: 'stdout:\nsandbox',+ status: 'success',+ },+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'uname -r' },+ toolOutput: 'stdout:\n6.1.102',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'single-trivial',+ notes: 'one boring call — header must still say something the card cannot',+ steps: [+ {+ payload: {+ entries: [+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'ls /mnt/data' },+ toolOutput: 'stdout:\nnotes.md\nresults.csv\nprobe.txt',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'answer-found',+ notes: 'the answer IS the line — a question resolved by one call',+ steps: [+ {+ payload: {+ lastAssistantText: 'Let me find where that 30-second timeout is actually set.',+ entries: [+ {+ toolName: 'grep',+ toolInput: { pattern: 'timeout', path: 'api/server/utils/streams.js' },+ toolOutput:+ 'streams.js:41: const STREAM_TIMEOUT_MS = 30_000; // hard cap per SSE flush\nstreams.js:88: setTimeout(() => controller.abort(), STREAM_TIMEOUT_MS);',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'bare-batch',+ notes: 'no intent, no reasoning — minimum context',+ steps: [+ {+ payload: {+ entries: [+ {+ toolName: 'read_file',+ toolInput: { path: 'package.json' },+ toolOutput: '{\n "name": "librechat",\n "version": "0.8.1",\n ...',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'misleading-intent',+ notes: 'intent asks one question, output answers it the other way',+ steps: [+ {+ payload: {+ lastAssistantText: 'Now checking whether response caching is enabled in this deployment.',+ entries: [+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'grep -A2 "cache:" config/deploy.yaml' },+ toolOutput: 'stdout:\ncache:\n enabled: false\n ttl: 3600',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'truncated-output',+ notes: 'output clipped mid-JSON by the 600-char limit',+ steps: [+ {+ payload: {+ entries: [+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'pip list --format=json' },+ toolOutput:+ 'stdout:\n' ++ JSON.stringify(+ Array.from({ length: 60 }, (_, i) => ({+ name: `package-${i}`,+ version: `1.${i}.0`,+ })),+ ),+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'silent-success',+ notes: 'empty output — nothing came back to summarize',+ steps: [+ {+ payload: {+ lastAssistantText: "I'll write the results file now.",+ entries: [+ {+ toolName: 'write_file',+ toolInput: { path: '/mnt/data/results.csv', content: 'run,ms\n1,412\n2,398\n' },+ toolOutput: '',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'edit-verify',+ notes: 'edit plus read-back in one batch — one activity, two calls',+ steps: [+ {+ payload: {+ thinkingExcerpts: [+ 'The retry cap is what causes the duplicate sends; dropping it from 5 to 1 and verifying the file took the change.',+ ],+ entries: [+ {+ toolName: 'edit_file',+ toolInput: {+ path: 'api/server/utils/queue.js',+ old: 'const MAX_RETRIES = 5;',+ new: 'const MAX_RETRIES = 1;',+ },+ toolOutput: 'OK',+ status: 'success',+ },+ {+ toolName: 'read_file',+ toolInput: { path: 'api/server/utils/queue.js', range: [10, 14] },+ toolOutput: 'const MAX_RETRIES = 1;\nconst BACKOFF_MS = 250;',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'error-shaped-success',+ notes: 'tool returns an error payload with success status — must not read as success',+ steps: [+ {+ payload: {+ lastAssistantText: 'Searching for the changelog now.',+ entries: [+ {+ toolName: 'web_search',+ toolInput: { query: 'librechat 0.8.1 changelog' },+ toolOutput:+ '{"error":{"code":"rate_limited","message":"Search quota exceeded, retry after 3600s"}}',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'mcp-long-name',+ notes: 'namespaced MCP tool name — echo temptation',+ steps: [+ {+ payload: {+ entries: [+ {+ toolName: 'mcp__github__search_repositories',+ toolInput: { query: 'org:danny-avila librechat-agents' },+ toolOutput:+ '{"total_count":2,"items":[{"full_name":"danny-avila/LibreChat","stars":31200},{"full_name":"danny-avila/agents","stars":410}]}',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'dup-activity-seq',+ notes: 'controlled mirror of captured steps 2/3 — same activity twice in a row',+ steps: [+ {+ id: 'dup-write',+ payload: {+ thinkingExcerpts: [+ 'First write a marker file, then a separate call will check it survives.',+ ],+ entries: [+ {+ toolName: 'run_tools_with_bash',+ toolInput: {+ code: 'echo "marker-$(date +%s)" > /tmp/persist-probe.txt && cat /tmp/persist-probe.txt',+ },+ toolOutput: 'stdout:\nmarker-1785932011',+ status: 'success',+ },+ ],+ },+ },+ {+ id: 'dup-confirm',+ payload: {+ entries: [+ {+ toolName: 'run_tools_with_bash',+ toolInput: { code: 'cat /tmp/persist-probe.txt' },+ toolOutput: 'stdout:\nmarker-1785932011',+ status: 'success',+ },+ ],+ },+ },+ ],+ },+ {+ id: 'overflow-entries',+ notes: '14 calls — exercises the 12-entry cap and the "…and 2 more" suffix',+ steps: [+ {+ payload: {+ entries: Array.from({ length: 14 }, (_, i) => ({+ toolName: 'run_tools_with_bash',+ toolInput: { code: `convert page-${i + 1}.svg page-${i + 1}.png` },+ toolOutput: '',+ status: 'success',+ })),+ },+ },+ ],+ },+];++/** Tool names for echo checks; captured steps bake entries into the+ * verbatim prompt, so they are recovered from the "Tool calls:" lines. */+function stepEntries(step) {+ if (step.payload?.entries) {+ return step.payload.entries;+ }+ return [...(step.verbatim ?? '').matchAll(/^- ([A-Za-z0-9_]+)\(/gm)].map((match) => ({+ toolName: match[1],+ }));+}++module.exports = { cases: [capturedRun, ...synthetic], stepEntries };diff --git a/scripts/activity-labels/prompt.js b/scripts/activity-labels/prompt.jsnew file mode 100644index 0000000000..f67fcfcae5--- /dev/null+++ b/scripts/activity-labels/prompt.js@@ -0,0 +1,151 @@+/**+ * Faithful port of the SDK's `buildActivityLabelPrompt`+ * (agentus src/prompts/activityLabel.ts) — the PREFERRED path that serves+ * every production label. The older live-check script mirrors a simplified+ * e2e capture; this port keeps section order (Intent → Reasoning excerpts →+ * Tool calls → Label:), the 12-entry cap with the "…and N more" suffix, and+ * the exact truncation semantics, so synthetic corpus cases render the same+ * bytes production would send. Redaction is intentionally not ported — the+ * corpus models unredacted single-agent runs.+ *+ * One addition beyond the SDK: an optional "Previous headers" section, OFF+ * unless a variant opts in. This is the P1 continuity hypothesis — it lets+ * the harness measure the fix before any SDK field exists.+ */+const INPUT_CONTEXT_LIMIT = 200;+const MAX_THINKING_EXCERPTS = 4;+const MAX_PROMPT_ENTRIES = 12;+const MAX_PREVIOUS_LABELS = 3;++function truncateForLabel(value, maxLength) {+ if (value.length <= maxLength) {+ return value;+ }+ return value.slice(0, Math.max(0, maxLength - 1)) + '…';+}++const ABORT_SERIALIZATION = Symbol('abort-label-serialization');++function serializeForLabel(value, limit) {+ if (value == null) {+ return '';+ }+ if (typeof value === 'string') {+ return value.length > limit ? value.slice(0, limit + 1) : value;+ }+ let budget = limit * 4;+ try {+ return (+ JSON.stringify(value, (_key, nested) => {+ if (budget <= 0) {+ throw ABORT_SERIALIZATION;+ }+ if (typeof nested === 'string') {+ const clipped = nested.length > limit ? nested.slice(0, limit) : nested;+ budget -= clipped.length;+ return clipped;+ }+ budget -= 8;+ return nested;+ }) ?? ''+ );+ } catch (error) {+ if (error === ABORT_SERIALIZATION) {+ return Array.isArray(value) ? `[Array(${value.length})]` : '[Object]';+ }+ return String(value);+ }+}++/** `cap` of Infinity models the unbounded-history alternative — testing+ * whether the whole run's story beats a recency window. */+function previousHeadersSection(previousLabels, cap = MAX_PREVIOUS_LABELS) {+ const kept = previousLabels.filter(Boolean);+ const recent = Number.isFinite(cap) ? kept.slice(-cap) : kept;+ if (recent.length === 0) {+ return null;+ }+ return (+ 'Previous headers in this run (most recent last):\n' ++ recent.map((label) => `- ${label}`).join('\n')+ );+}++function buildActivityLabelPrompt({+ entries,+ charLimit,+ thinkingExcerpts,+ lastAssistantText,+ previousLabels,+ previousLabelCap,+}) {+ const clip = truncateForLabel;+ const sections = [];+ if (previousLabels != null) {+ const section = previousHeadersSection(previousLabels, previousLabelCap);+ if (section != null) {+ sections.push(section);+ }+ }+ if (lastAssistantText != null && lastAssistantText.length > 0) {+ sections.push(+ `Intent (assistant's last message): ${clip(lastAssistantText, INPUT_CONTEXT_LIMIT)}`,+ );+ }+ if (thinkingExcerpts != null && thinkingExcerpts.length > 0) {+ sections.push(+ 'Reasoning excerpts:\n' ++ thinkingExcerpts+ .slice(0, MAX_THINKING_EXCERPTS)+ .map((excerpt) => `- ${clip(excerpt, charLimit)}`)+ .join('\n'),+ );+ }+ if (entries.length > 0) {+ const shown = entries.slice(0, MAX_PROMPT_ENTRIES);+ const omitted = entries.length - shown.length;+ sections.push(+ 'Tool calls:\n' ++ shown+ .map((entry) => {+ const input = clip(serializeForLabel(entry.toolInput, charLimit), charLimit);+ const outcome =+ entry.status === 'error'+ ? `ERROR: ${clip(entry.error ?? 'unknown error', charLimit)}`+ : clip(serializeForLabel(entry.toolOutput, charLimit), charLimit);+ return `- ${entry.toolName}(${input}) → ${outcome}`;+ })+ .join('\n') ++ (omitted > 0 ? `\n- …and ${omitted} more tool ${omitted === 1 ? 'call' : 'calls'}` : ''),+ );+ }+ sections.push('Label:');+ return sections.join('\n\n');+}++/**+ * Renders a corpus step. Captured steps carry the verbatim production+ * prompt (byte-exact from Langfuse); the continuity section, when a variant+ * opts in, is prepended — the same position the built path gives it.+ */+function renderStepPrompt(step, { charLimit, previousLabels, previousLabelCap }) {+ if (step.verbatim != null) {+ const section =+ previousLabels != null ? previousHeadersSection(previousLabels, previousLabelCap) : null;+ return section != null ? `${section}\n\n${step.verbatim}` : step.verbatim;+ }+ return buildActivityLabelPrompt({+ ...step.payload,+ charLimit,+ previousLabels,+ previousLabelCap,+ });+}++module.exports = {+ buildActivityLabelPrompt,+ renderStepPrompt,+ truncateForLabel,+ serializeForLabel,+ MAX_PREVIOUS_LABELS,+};diff --git a/scripts/activity-labels/report.js b/scripts/activity-labels/report.jsnew file mode 100644index 0000000000..e06eb14413--- /dev/null+++ b/scripts/activity-labels/report.js@@ -0,0 +1,148 @@+/** Aggregation + markdown rendering, shared by the live runner and the+ * offline rescorer so metric fixes never require re-spending on the API. */+const FLAG_TYPES = [+ 'len',+ 'punct',+ 'quote',+ 'md',+ 'opener',+ 'tool-echo',+ 'count-echo',+ 'restate',+ 'template',+];++const PRICES = { 'claude-haiku-4-5': { input: 1, output: 5 } };++function flagType(flag) {+ return flag.split(':')[0];+}++function aggregate(records, model) {+ const byVariant = new Map();+ for (const record of records) {+ if (!byVariant.has(record.variant)) {+ byVariant.set(record.variant, {+ steps: 0,+ errors: 0,+ flagCounts: {},+ firstWords: {},+ totalWords: 0,+ latencies: [],+ inputTokens: 0,+ outputTokens: 0,+ });+ }+ const agg = byVariant.get(record.variant);+ if (record.error) {+ agg.errors += 1;+ continue;+ }+ agg.steps += 1;+ agg.totalWords += record.wordCount;+ agg.latencies.push(record.latencyMs);+ agg.inputTokens += record.inputTokens;+ agg.outputTokens += record.outputTokens;+ agg.firstWords[record.firstWord] = (agg.firstWords[record.firstWord] ?? 0) + 1;+ for (const flag of record.flags) {+ const type = flagType(flag);+ agg.flagCounts[type] = (agg.flagCounts[type] ?? 0) + 1;+ }+ }+ const price = PRICES[model];+ return [...byVariant.entries()].map(([name, agg]) => {+ const sortedFirst = Object.entries(agg.firstWords).sort((a, b) => b[1] - a[1]);+ const topOpener = sortedFirst[0] ?? ['—', 0];+ return {+ variant: name,+ steps: agg.steps,+ errors: agg.errors,+ flagCounts: agg.flagCounts,+ distinctOpeners: sortedFirst.length,+ topOpener: `${topOpener[0]} ×${topOpener[1]}`,+ avgWords: agg.steps > 0 ? (agg.totalWords / agg.steps).toFixed(1) : '—',+ meanLatencyMs: agg.latencies.length+ ? Math.round(agg.latencies.reduce((a, b) => a + b, 0) / agg.latencies.length)+ : 0,+ inputTokens: agg.inputTokens,+ outputTokens: agg.outputTokens,+ costUsd: price+ ? ((agg.inputTokens * price.input + agg.outputTokens * price.output) / 1e6).toFixed(4)+ : 'n/a',+ };+ });+}++function markdownReport({ records, aggregates, runCases, variantNames, model, samples }) {+ const lines = [];+ lines.push(`# Activity-label eval — ${new Date().toISOString()}`);+ lines.push('');+ lines.push(`model: \`${model}\` · samples: ${samples} · cases: ${runCases.length}`);+ lines.push('');+ lines.push('## Aggregate');+ lines.push('');+ lines.push(+ `| variant | steps | ${FLAG_TYPES.join(' | ')} | distinct openers | top opener | avg words | mean ms | cost |`,+ );+ lines.push(`|---|---:|${FLAG_TYPES.map(() => '---:').join('|')}|---:|---|---:|---:|---:|`);+ for (const agg of aggregates) {+ lines.push(+ `| ${agg.variant} | ${agg.steps}${agg.errors ? ` (+${agg.errors} err)` : ''} | ` ++ FLAG_TYPES.map((type) => agg.flagCounts[type] ?? 0).join(' | ') ++ ` | ${agg.distinctOpeners} | ${agg.topOpener} | ${agg.avgWords} | ${agg.meanLatencyMs} | $${agg.costUsd} |`,+ );+ }+ lines.push('');+ lines.push('## Per-case');+ for (const testCase of runCases) {+ lines.push('');+ lines.push(`### ${testCase.id}`);+ lines.push('');+ lines.push(`*${testCase.notes}*`);+ lines.push('');+ const sampleList = [...new Set(records.map((r) => r.sample))].sort();+ const header = ['step'];+ if (samples > 1) {+ header.push('s');+ }+ if (testCase.steps.some((step) => step.productionLabel)) {+ header.push('production');+ }+ header.push(...variantNames);+ lines.push(`| ${header.join(' | ')} |`);+ lines.push(`|${header.map(() => '---').join('|')}|`);+ for (const step of testCase.steps) {+ const stepId = step.id ?? testCase.id;+ for (const sample of sampleList) {+ const row = [stepId];+ if (samples > 1) {+ row.push(String(sample));+ }+ if (header.includes('production')) {+ row.push(step.productionLabel ?? '');+ }+ for (const variantName of variantNames) {+ const record = records.find(+ (r) =>+ r.variant === variantName &&+ r.sample === sample &&+ r.caseId === testCase.id &&+ r.stepId === stepId,+ );+ if (!record) {+ row.push('');+ } else if (record.error) {+ row.push(`⛔ ${record.error}`);+ } else {+ const flagNote = record.flags.length > 0 ? ` ⚠${record.flags.join(' ⚠')}` : '';+ row.push(`${record.label}${flagNote}`);+ }+ }+ lines.push(`| ${row.map((cell) => cell.replace(/\|/g, '\\|')).join(' | ')} |`);+ }+ }+ }+ return lines.join('\n') + '\n';+}++module.exports = { aggregate, markdownReport, FLAG_TYPES };diff --git a/scripts/activity-labels/rescore.js b/scripts/activity-labels/rescore.jsnew file mode 100644index 0000000000..30922970d8Binary files /dev/null and b/scripts/activity-labels/rescore.js differdiff --git a/scripts/activity-labels/run.js b/scripts/activity-labels/run.jsnew file mode 100644index 0000000000..7035a1362d--- /dev/null+++ b/scripts/activity-labels/run.js@@ -0,0 +1,241 @@+/**+ * Activity-label eval runner: replays the corpus against the production wire+ * shape (system = instruction variant, user = SDK-built prompt, max_tokens+ * 256 — matching the traced production requests) and reports per-variant+ * quality: format violations, first-word register distribution, cross-batch+ * redundancy, latency, and cost.+ *+ * Sequences run their steps serially; each generated label chains into the+ * next step's `previousLabels`. Variants with `usePreviousLabels` see that+ * context in the prompt; every variant is MEASURED against it, so blind and+ * continuity variants share one redundancy metric.+ *+ * Usage:+ * node scripts/activity-labels/run.js [--variants baseline,continuity]+ * [--cases sandbox-probe-run,fib-rapid] [--samples 2] [--model id]+ * [--concurrency 6] [--dry]+ */+const fs = require('fs');+const path = require('path');++const { cases, stepEntries } = require('./corpus');+const { variants } = require('./variants');+const { checkLabel } = require('./checks');+const { renderStepPrompt } = require('./prompt');+const { aggregate, markdownReport } = require('./report');++const ROOT = path.resolve(__dirname, '..', '..');+const RESULTS_DIR = path.join(__dirname, 'results');+const CHAR_LIMIT = 600;+const MAX_TOKENS = 256;++function parseArgs(argv) {+ const args = { samples: 1, concurrency: 6, model: 'claude-haiku-4-5', dry: false };+ for (let i = 0; i < argv.length; i++) {+ const key = argv[i];+ if (key === '--dry') {+ args.dry = true;+ } else if (key === '--variants') {+ args.variants = argv[++i].split(',');+ } else if (key === '--cases') {+ args.cases = argv[++i].split(',');+ } else if (key === '--samples') {+ args.samples = Number(argv[++i]);+ } else if (key === '--model') {+ args.model = argv[++i];+ } else if (key === '--concurrency') {+ args.concurrency = Number(argv[++i]);+ }+ }+ return args;+}++function loadKey() {+ if (process.env.ANTHROPIC_API_KEY) {+ return process.env.ANTHROPIC_API_KEY;+ }+ const envPath = path.join(ROOT, '.env');+ const line = fs.existsSync(envPath)+ ? fs+ .readFileSync(envPath, 'utf8')+ .split('\n')+ .find((entry) => entry.startsWith('ANTHROPIC_API_KEY='))+ : undefined;+ if (!line) {+ throw new Error(+ `ANTHROPIC_API_KEY not set and not found in ${envPath}.\n` ++ 'Pass it inline: ANTHROPIC_API_KEY=sk-… node scripts/activity-labels/run.js',+ );+ }+ return line+ .slice('ANTHROPIC_API_KEY='.length)+ .trim()+ .replace(/^["']|["']$/g, '');+}++async function requestLabel({ apiKey, model, instruction, prompt }) {+ for (let attempt = 1; attempt <= 3; attempt++) {+ const started = Date.now();+ const response = await fetch('https://api.anthropic.com/v1/messages', {+ method: 'POST',+ headers: {+ 'content-type': 'application/json',+ 'x-api-key': apiKey,+ 'anthropic-version': '2023-06-01',+ },+ body: JSON.stringify({+ model,+ max_tokens: MAX_TOKENS,+ system: instruction,+ messages: [{ role: 'user', content: prompt }],+ }),+ });+ if (response.ok) {+ const json = await response.json();+ const label = (json.content ?? [])+ .map((block) => block.text ?? '')+ .join('')+ .trim()+ .replace(/^["']|["']$/g, '');+ return {+ label,+ latencyMs: Date.now() - started,+ inputTokens: json.usage?.input_tokens ?? 0,+ outputTokens: json.usage?.output_tokens ?? 0,+ };+ }+ const body = await response.text();+ if (attempt < 3 && [429, 500, 529].includes(response.status)) {+ const retryAfter = Number(response.headers.get('retry-after'));+ const waitMs =+ Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : attempt * 2000;+ await new Promise((resolve) => setTimeout(resolve, Math.min(waitMs, 15000)));+ continue;+ }+ return {+ error: `HTTP ${response.status}: ${body.slice(0, 160)}`,+ latencyMs: Date.now() - started,+ };+ }+}++/** One case chain: steps serial, labels feeding forward. */+async function runCase({ apiKey, model, variant, sample, testCase, dry, records }) {+ const chain = [];+ for (const step of testCase.steps) {+ const prompt = renderStepPrompt(step, {+ charLimit: CHAR_LIMIT,+ previousLabels: variant.usePreviousLabels ? chain : null,+ previousLabelCap: variant.previousLabelCap,+ });+ const stepId = step.id ?? testCase.id;+ if (dry) {+ records.push({ variant: variant.name, sample, caseId: testCase.id, stepId, prompt });+ continue;+ }+ const result = await requestLabel({ apiKey, model, instruction: variant.instruction, prompt });+ if (result.error) {+ records.push({+ variant: variant.name,+ sample,+ caseId: testCase.id,+ stepId,+ error: result.error,+ });+ continue;+ }+ const { flags, wordCount, firstWord } = checkLabel(result.label, {+ entries: stepEntries(step),+ previousLabels: chain,+ });+ chain.push(result.label);+ records.push({+ variant: variant.name,+ sample,+ caseId: testCase.id,+ stepId,+ label: result.label,+ production: step.productionLabel,+ flags,+ wordCount,+ firstWord,+ latencyMs: result.latencyMs,+ inputTokens: result.inputTokens,+ outputTokens: result.outputTokens,+ });+ }+}++async function pool(tasks, size) {+ const queue = [...tasks];+ const workers = Array.from({ length: Math.min(size, queue.length) }, async () => {+ while (queue.length > 0) {+ await queue.shift()();+ }+ });+ await Promise.all(workers);+}++(async () => {+ const args = parseArgs(process.argv.slice(2));+ const runVariants = args.variants+ ? variants.filter((variant) => args.variants.includes(variant.name))+ : variants;+ const runCases = args.cases ? cases.filter((c) => args.cases.includes(c.id)) : cases;+ if (runVariants.length === 0 || runCases.length === 0) {+ throw new Error('nothing selected — check --variants / --cases names');+ }+ const apiKey = args.dry ? '' : loadKey();+ const records = [];+ const tasks = [];+ for (const variant of runVariants) {+ for (let sample = 1; sample <= args.samples; sample++) {+ for (const testCase of runCases) {+ tasks.push(() =>+ runCase({ apiKey, model: args.model, variant, sample, testCase, dry: args.dry, records }),+ );+ }+ }+ }+ const totalSteps = runCases.reduce((sum, c) => sum + c.steps.length, 0);+ console.log(+ `${args.dry ? 'DRY RUN — rendering only' : `model ${args.model}`} · ${runVariants.length} variants × ${args.samples} samples × ${runCases.length} cases (${totalSteps} steps each pass)`,+ );+ const started = Date.now();+ await pool(tasks, args.concurrency);+ console.log(`done in ${((Date.now() - started) / 1000).toFixed(1)}s\n`);++ if (args.dry) {+ for (const record of records.slice(0, 3)) {+ console.log(`--- ${record.variant} / ${record.caseId} / ${record.stepId} ---`);+ console.log(record.prompt);+ console.log('');+ }+ console.log(`rendered ${records.length} prompts (showing 3)`);+ return;+ }++ const aggregates = aggregate(records, args.model);+ const variantNames = runVariants.map((variant) => variant.name);+ const report = markdownReport({+ records,+ aggregates,+ runCases,+ variantNames,+ model: args.model,+ samples: args.samples,+ });+ fs.mkdirSync(RESULTS_DIR, { recursive: true });+ const stamp = new Date().toISOString().replace(/[:.]/g, '-');+ fs.writeFileSync(+ path.join(RESULTS_DIR, `${stamp}.json`),+ JSON.stringify({ args, records }, null, 2),+ );+ fs.writeFileSync(path.join(RESULTS_DIR, 'latest.md'), report);++ console.log(report.split('## Per-case')[0]);+ console.log(`full per-case tables: scripts/activity-labels/results/latest.md`);+})().catch((error) => {+ console.error('ERR', error.message);+ process.exit(1);+});diff --git a/scripts/activity-labels/variants.js b/scripts/activity-labels/variants.jsnew file mode 100644index 0000000000..f9ba3ca549--- /dev/null+++ b/scripts/activity-labels/variants.js@@ -0,0 +1,239 @@+/**+ * Instruction variants under test. Each is a SINGLE-factor change against the+ * production instruction so a result implicates one hypothesis:+ *+ * - baseline — ACTIVITY_INSTRUCTION exactly as the branch ships it+ * - verbs — H: the Good-example verb distribution seeds register+ * collapse (6/9 production labels opened "Confirmed")+ * - ordered — H: the 4–9 word cap gets crowded out mid-paragraph; moving+ * format constraints last improves adherence+ * - continuity — H: showing the run's previous headers kills cross-batch+ * redundancy (production pairs 2/3 and 7/8)+ *+ * The baseline is required from packages/api/dist so drift against the branch+ * is impossible; the sentence table below is asserted against it so composed+ * variants can never silently diverge from what production actually sends.+ */+const path = require('path');++const ROOT = path.resolve(__dirname, '..', '..');++/**+ * The shipped instruction, read from the BUILT package so a variant can never+ * be graded against a stale copy of it. Tries the workspace resolution first+ * (an installed checkout), then the dist path directly, so the harness works+ * whether or not `node_modules` is populated. `LABEL_EVAL_DIST` points it at+ * another checkout's build — useful for grading one branch's instruction from+ * a worktree that has not been built.+ */+function loadShippedInstruction() {+ const candidates = [+ process.env.LABEL_EVAL_DIST,+ '@librechat/api',+ path.join(ROOT, 'packages/api/dist/index.cjs'),+ ].filter(Boolean);+ for (const candidate of candidates) {+ try {+ const { ACTIVITY_INSTRUCTION } = require(candidate);+ if (typeof ACTIVITY_INSTRUCTION === 'string' && ACTIVITY_INSTRUCTION.length > 0) {+ return ACTIVITY_INSTRUCTION;+ }+ } catch {+ /* try the next candidate */+ }+ }+ throw new Error(+ 'Could not load ACTIVITY_INSTRUCTION from a built @librechat/api.\n' ++ 'Build it first (from the repo root):\n' ++ ' npm run build:data-provider && npm run build:data-schemas && npm run build:api\n' ++ 'Or point at an existing build:\n' ++ ' LABEL_EVAL_DIST=/path/to/packages/api/dist/index.cjs node scripts/activity-labels/run.js',+ );+}++const ACTIVITY_INSTRUCTION = loadShippedInstruction();++const S = {+ role: 'You write the one-line header above a group of tool calls an AI agent just made.',+ register:+ 'Write it like a git commit subject: past tense, verb first, leading with the most distinctive file, name, or finding.',+ outcome:+ 'Say what the calls established or produced — the outcome, not the attempt. If they answered a question, the answer is the line.',+ prohibitions:+ 'Never name the tools, never count them, never echo the arguments: the cards below the header already show all three.',+ format: 'Write 4 to 9 words, sentence case, no trailing punctuation, no quotes or markdown.',+ good: 'Good: "Confirmed /mnt/data resets between calls". "Traced the leak to formatAgentMessages". "Found 3 failing auth tests".',+ bad: 'Bad: "Ran 1 command". "Used bash_tool twice". "Executed ls /mnt/data". "Searched the codebase".',+ failure: 'If every call failed, say what failed and why, plainly.',+ output: 'Output only the line.',+};++const CONTINUITY_SENTENCE =+ 'A "Previous headers" list may precede the batch: never restate one — if this batch continues that activity, say only what is new.';++/** Pre-P1 instruction — kept as the `legacy` variant for regression sweeps. */+const LEGACY_ORDER = [+ S.role,+ S.register,+ S.outcome,+ S.prohibitions,+ S.format,+ S.good,+ S.bad,+ S.failure,+ S.output,+];++/** The shipped instruction (P1): ordered structure + continuity clause. */+const SHIPPED_ORDER = [+ S.role,+ S.outcome,+ S.register,+ S.good,+ S.bad,+ S.failure,+ CONTINUITY_SENTENCE,+ S.prohibitions,+ S.format,+ S.output,+];++/** `baseline` is whatever the BUILT dist ships. Before the P1 rebuild that is+ * the legacy order, after it the shipped order; anything else means the+ * sentence table here has drifted and composed variants are stale. */+if (+ LEGACY_ORDER.join(' ') !== ACTIVITY_INSTRUCTION &&+ SHIPPED_ORDER.join(' ') !== ACTIVITY_INSTRUCTION+) {+ console.warn(+ 'WARN: variants.js sentence table has drifted from ACTIVITY_INSTRUCTION — composed variants are stale',+ );+}++const VERB_CHOICE =+ 'Open with whichever past-tense verb the outcome dictates — confirmed, found, traced, measured, wrote, ruled out, failed — not the same verb every time.';+const DIVERSE_GOOD =+ 'Good: "Traced the leak to formatAgentMessages". "Ruled out DNS as the failure cause". "Measured cold start at 412ms". "Found 3 failing auth tests".';+const CONTINUITY =+ 'A "Previous headers" list may precede the batch: those lines already stand above earlier groups, so never write a line that merely restates one. If this batch continues that same activity, lead with what is new or different in THIS batch.';+const CONTINUITY_TIGHT =+ 'A "Previous headers" list may precede the batch: never restate one — if this batch continues that activity, say only what is new.';+const FORMAT_HARD =+ 'Write 4 to 9 words, sentence case, no trailing punctuation, no quotes or markdown; when a batch found many things, keep only the most load-bearing one or two.';++const variants = [+ {+ name: 'baseline',+ usePreviousLabels: SHIPPED_ORDER.join(' ') === ACTIVITY_INSTRUCTION,+ instruction: ACTIVITY_INSTRUCTION,+ },+ {+ name: 'legacy',+ usePreviousLabels: false,+ instruction: LEGACY_ORDER.join(' '),+ },+ {+ name: 'verbs',+ usePreviousLabels: false,+ instruction: [+ S.role,+ S.register,+ VERB_CHOICE,+ S.outcome,+ S.prohibitions,+ S.format,+ DIVERSE_GOOD,+ S.bad,+ S.failure,+ S.output,+ ].join(' '),+ },+ {+ name: 'ordered',+ usePreviousLabels: false,+ instruction: [+ S.role,+ S.outcome,+ S.register,+ S.good,+ S.bad,+ S.failure,+ S.prohibitions,+ S.format,+ S.output,+ ].join(' '),+ },+ {+ name: 'continuity',+ usePreviousLabels: true,+ instruction: [+ S.role,+ S.register,+ S.outcome,+ S.prohibitions,+ S.format,+ S.good,+ S.bad,+ S.failure,+ CONTINUITY,+ S.output,+ ].join(' '),+ },+ {+ name: 'examples',+ usePreviousLabels: false,+ instruction: [+ S.role,+ S.register,+ S.outcome,+ S.prohibitions,+ S.format,+ DIVERSE_GOOD,+ S.bad,+ S.failure,+ S.output,+ ].join(' '),+ },+ {+ name: 'composed',+ usePreviousLabels: true,+ instruction: [+ S.role,+ S.outcome,+ S.register,+ DIVERSE_GOOD,+ S.bad,+ S.failure,+ CONTINUITY_TIGHT,+ S.prohibitions,+ FORMAT_HARD,+ S.output,+ ].join(' '),+ },+ {+ name: 'shipping-full',+ usePreviousLabels: true,+ /** Whole-run history instead of the 3-label recency window: does more+ * story beat recency, or does it dilute the batch content? */+ previousLabelCap: Infinity,+ instruction: SHIPPED_ORDER.join(' '),+ },+ {+ name: 'shipping',+ usePreviousLabels: true,+ instruction: [+ S.role,+ S.outcome,+ S.register,+ S.good,+ S.bad,+ S.failure,+ CONTINUITY_TIGHT,+ S.prohibitions,+ S.format,+ S.output,+ ].join(' '),+ },+];++module.exports = { variants };