Enhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active
TypeScript aianthropicartifactsawsazurechatgptchatgpt-cloneclaudeclonedeepseekgeminigooglegpt-5librechatmcpo1openairesponses-apivisionwebui
🎰 ci: Vote on the Full Mock Suite to End Phantom Spec Trials (#15162)
Danny Avilaf10fcd7d19ba
diff --git a/.github/workflows/codegraph-e2e-votes.yml b/.github/workflows/codegraph-e2e-votes.ymlindex fdcbe79626..35637a3041 100644--- a/.github/workflows/codegraph-e2e-votes.yml+++ b/.github/workflows/codegraph-e2e-votes.yml@@ -1,15 +1,22 @@# Codegraph e2e VOTES — observe-only, post-merge, time-boxed.## Playwright never runs on pushes to dev, so evidence for the e2e skip election would-# otherwise wait on rare organic PR spec failures. This workflow runs EXACTLY the skippable-# tier the merged PR's selection computed — every merge becomes a direct trial of "would-# skipping these specs have missed a failure". A green run is a confirmation vote; a failing-# spec here is a tier-miss vote counted AGAINST enabling skipping. The shadow evaluator on-# the codegraph droplet harvests these runs and attributes them back to the merged PR.+# otherwise wait on rare organic PR spec failures. This workflow runs the FULL mock suite on+# every merge: each run is one graduation trial for every spec it executes, and doubles as+# the post-merge safety net the jest workflows already have via their dev-push triggers.#-# It cannot fail the branch: the tier lookup exits 0 on every path and the test step is-# continue-on-error. The newest merge cancels older vote runs. The whole campaign switches-# off by setting repo variable CODEGRAPH_E2E_VOTES=off once the election passes.+# It previously ran only the merged PR's skippable tier, passing the tier as CLI path+# filters. playwright.config.mock.ts scopes discovery to testDir specs/mock/, so tier+# entries outside that directory matched nothing — and the covered-list log line still+# claimed them, minting graduation trials for specs that never executed (run 32701691037:+# a11y/keys/messages in the covered list, zero of their tests run). The covered list below+# is therefore derived from the run's EXECUTED results — discovery is not enough either,+# since env-gated suites self-skip under this job's default env — and the run takes no+# path filters at all.+#+# It cannot fail the branch: the test step is continue-on-error. The newest merge cancels+# older vote runs. The whole campaign switches off by setting repo variable+# CODEGRAPH_E2E_VOTES=off once the election passes.name: Codegraph E2E Voteson:@@ -20,6 +27,7 @@ on:- '**'- '!**.md'- '!.github/workflows/**'+ - '.github/workflows/codegraph-e2e-votes.yml'permissions:contents: read@@ -34,10 +42,10 @@ env:jobs:vote:- name: vote (skippable tier)+ name: vote (full suite)if: vars.CODEGRAPH_E2E_VOTES != 'off'runs-on: ubuntu-latest- timeout-minutes: 30+ timeout-minutes: 45env:CI: 'true'E2E_CHROMIUM_CHANNEL: chrome@@ -45,56 +53,12 @@ jobs:steps:- uses: actions/checkout@v5- - name: Ask codegraph for this merge's skippable tier- id: tiers- env:- URL: ${{ secrets.CODEGRAPH_URL }}- TOKEN: ${{ secrets.CODEGRAPH_TOKEN }}- GH_TOKEN: ${{ github.token }}- run: |- set +e- N=0- if [ -n "$URL" ] && [ -n "$TOKEN" ]; then- gh api "repos/${{ github.repository }}/commits/${{ github.sha }}" \- --jq '[.files[] | {path: .filename, status: .status}]' > files.json 2>/dev/null- if [ -s files.json ]; then- jq -c '{files: .}' files.json > body.json- RESP=$(curl -sS -m 45 -H "Authorization: Bearer $TOKEN" \- -H 'content-type: application/json' --data-binary @body.json "$URL/v1/select")- # fail_open reflects the JEST floors (root config, lockfile, stale graph); the- # e2e tiers come from the testid bridge and are valid whenever they computed at- # all. The old fail-open skip silently excused exactly the big backend merges- # whose trials matter most (LibreChat#14957's merge produced no vote because- # api/package.json tripped the jest floor). Tiers present => vote.- if ! echo "$RESP" | jq -e '.e2e.skippable' >/dev/null 2>&1; then- echo "codegraph unavailable or no tiers; skipping"- else- echo "$RESP" | jq -r '.e2e.skippable[]' | sed 's|^e2e/||' > skippable.txt- N=$(wc -l < skippable.txt | tr -d ' ')- fi- else- echo "could not read merge commit files; skipping"- fi- else- echo "no codegraph config; skipping"- fi- echo "codegraph-votes: running $N skippable specs"- # The exact list, one log line: the shadow's per-spec graduation ledger counts a clean- # trial for every spec a green vote run covered, and until this line existed it had to- # approximate coverage from the decision event's tier (drift: the tier is recomputed- # here at the merge commit against a possibly newer graph head).- if [ "$N" != "0" ]; then echo "codegraph-votes-specs: $(tr '\n' ' ' < skippable.txt)"; fi- echo "count=$N" >> "$GITHUB_OUTPUT"- exit 0-- name: Use Node.js 24.16.0- if: steps.tiers.outputs.count != '0'uses: actions/setup-node@v5with:node-version: '24.16.0'- name: Restore node_modules cache- if: steps.tiers.outputs.count != '0'id: cache-node-modulesuses: actions/cache@v5with:@@ -109,11 +73,10 @@ jobs:key: node-modules-e2e-${{ runner.os }}-24.16.0-${{ hashFiles('package-lock.json') }}- name: Install dependencies- if: steps.tiers.outputs.count != '0' && steps.cache-node-modules.outputs.cache-hit != 'true'+ if: steps.cache-node-modules.outputs.cache-hit != 'true'run: npm ci- name: Restore data-provider build cache- if: steps.tiers.outputs.count != '0'id: cache-data-provideruses: actions/cache@v5with:@@ -121,11 +84,10 @@ jobs:key: build-data-provider-${{ runner.os }}-${{ hashFiles('package.json', 'package-lock.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json') }}- name: Build data-provider- if: steps.tiers.outputs.count != '0' && steps.cache-data-provider.outputs.cache-hit != 'true'+ if: steps.cache-data-provider.outputs.cache-hit != 'true'run: npm run build:data-provider- name: Restore data-schemas build cache- if: steps.tiers.outputs.count != '0'id: cache-data-schemasuses: actions/cache@v5with:@@ -133,11 +95,10 @@ jobs:key: build-data-schemas-${{ runner.os }}-${{ hashFiles('package.json', 'package-lock.json', 'packages/data-schemas/src/**', 'packages/data-schemas/tsconfig*.json', 'packages/data-schemas/tsdown.config.mjs', 'packages/data-schemas/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json') }}- name: Build data-schemas- if: steps.tiers.outputs.count != '0' && steps.cache-data-schemas.outputs.cache-hit != 'true'+ if: steps.cache-data-schemas.outputs.cache-hit != 'true'run: npm run build:data-schemas- name: Restore api build cache- if: steps.tiers.outputs.count != '0'id: cache-apiuses: actions/cache@v5with:@@ -145,11 +106,10 @@ jobs:key: build-api-${{ runner.os }}-${{ hashFiles('package.json', 'package-lock.json', 'packages/api/src/**', 'packages/api/tsconfig*.json', 'packages/api/tsdown.config.mjs', 'packages/api/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json', 'packages/data-schemas/src/**', 'packages/data-schemas/tsconfig*.json', 'packages/data-schemas/tsdown.config.mjs', 'packages/data-schemas/package.json') }}- name: Build api- if: steps.tiers.outputs.count != '0' && steps.cache-api.outputs.cache-hit != 'true'+ if: steps.cache-api.outputs.cache-hit != 'true'run: npm run build:api- name: Restore client-package build cache- if: steps.tiers.outputs.count != '0'id: cache-client-packageuses: actions/cache@v5with:@@ -157,11 +117,10 @@ jobs:key: build-client-package-${{ runner.os }}-${{ hashFiles('package.json', 'package-lock.json', 'packages/client/src/**', 'packages/client/tsconfig*.json', 'packages/client/tsdown.config.mjs', 'packages/client/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json') }}- name: Build client-package- if: steps.tiers.outputs.count != '0' && steps.cache-client-package.outputs.cache-hit != 'true'+ if: steps.cache-client-package.outputs.cache-hit != 'true'run: npm run build:client-package- name: Restore client app build cache- if: steps.tiers.outputs.count != '0'id: cache-client-appuses: actions/cache@v5with:@@ -169,24 +128,21 @@ jobs:key: build-client-app-e2e-${{ runner.os }}-${{ hashFiles('package.json', 'package-lock.json', 'client/src/**', 'client/public/**', 'client/scripts/post-build.cjs', 'client/index.html', 'client/package.json', 'client/vite.config.*', 'client/tsconfig*.json', 'client/tailwind.config.*', 'client/postcss.config.*', 'packages/client/src/**', 'packages/client/tailwind.preset.cjs', 'packages/client/tsconfig*.json', 'packages/client/tsdown.config.mjs', 'packages/client/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json') }}- name: Build client app- if: steps.tiers.outputs.count != '0' && steps.cache-client-app.outputs.cache-hit != 'true'+ if: steps.cache-client-app.outputs.cache-hit != 'true'run: npm run build:client- name: Verify Chrome is present- if: steps.tiers.outputs.count != '0'run: google-chrome --version# ffmpeg for retry video — see the note in playwright-mock.yml.- name: Resolve Playwright versionid: playwright-version- if: steps.tiers.outputs.count != '0'run: |version=$(node -p "require('./package-lock.json').packages['node_modules/playwright-core'].version")echo "version=${version}" >> "$GITHUB_OUTPUT"- name: Restore Playwright ffmpeg cacheid: cache-ffmpeg- if: steps.tiers.outputs.count != '0'uses: actions/cache/restore@v5with:path: ~/.cache/ms-playwright@@ -194,7 +150,7 @@ jobs:- name: Install Playwright ffmpeg (best effort)id: install-ffmpeg- if: steps.tiers.outputs.count != '0' && steps.cache-ffmpeg.outputs.cache-hit != 'true'+ if: steps.cache-ffmpeg.outputs.cache-hit != 'true'timeout-minutes: 3continue-on-error: truerun: |@@ -202,7 +158,7 @@ jobs:.github/scripts/verify-playwright-ffmpeg.sh- name: Save Playwright ffmpeg cache- if: steps.tiers.outputs.count != '0' && steps.install-ffmpeg.outcome == 'success'+ if: steps.install-ffmpeg.outcome == 'success'continue-on-error: trueuses: actions/cache/save@v5with:@@ -211,15 +167,36 @@ jobs:# Optional fonts only — see the note in playwright-mock.yml.- name: Install optional Playwright font dependencies (best effort)- if: steps.tiers.outputs.count != '0'timeout-minutes: 4continue-on-error: truerun: .github/scripts/install-playwright-fonts.sh- - name: Vote — run the skippable tier (cannot fail the branch)- if: steps.tiers.outputs.count != '0'+ - name: Vote — run the full mock suite (cannot fail the branch)continue-on-error: true- run: npx playwright test --config=e2e/playwright.config.mock.ts $(tr '\n' ' ' < skippable.txt)+ env:+ PLAYWRIGHT_JSON_OUTPUT_NAME: pw-results.json+ run: npx playwright test --config=e2e/playwright.config.mock.ts --reporter=line,json++ - name: Ledger — log the specs that actually executed+ run: |+ set +e+ # The shadow's per-spec graduation ledger counts a clean trial for every spec a green+ # run covered, so the covered list must come from EXECUTED tests, not from discovery:+ # env-gated suites (mcp-tool-list-changed needs E2E_MCP_LIST_CHANGED, enforced-model-+ # specs needs E2E_MODEL_SPECS_ENFORCE) are discovered by --list yet skip every test+ # under this job's default env — counting them as covered would mint phantom trials,+ # the exact bug this workflow revision exists to kill (Codex P1 on #15162). A spec is+ # covered iff at least one of its tests reached a non-skipped outcome.+ if jq -e '.suites' pw-results.json >/dev/null 2>&1; then+ jq -r '[.suites[] | recurse(.suites[]?) | .specs[]? | select([.tests[]?.status] | any(. != "skipped")) | .file] | unique | .[]' pw-results.json \+ | sed 's|^|specs/mock/|' > covered.txt+ N=$(wc -l < covered.txt | tr -d ' ')+ echo "codegraph-votes: running $N specs (executed, full suite)"+ if [ "$N" != "0" ]; then echo "codegraph-votes-specs: $(tr '\n' ' ' < covered.txt)"; fi+ else+ echo "codegraph-votes: no results json — run crashed before reporting; no trials logged"+ fi+ exit 0- name: Doneif: always()