Enhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active
TypeScript aianthropicartifactsawsazurechatgptchatgpt-cloneclaudeclonedeepseekgeminigooglegpt-5librechatmcpo1openairesponses-apivisionwebui
🗳️ ci: Shard the Codegraph E2E Vote Suite Across Ten Runners (#17339)
Danny Avilafb8f71ab1497
diff --git a/.github/workflows/codegraph-e2e-votes.yml b/.github/workflows/codegraph-e2e-votes.ymlindex 0666338148..e46b18f372 100644--- a/.github/workflows/codegraph-e2e-votes.yml+++ b/.github/workflows/codegraph-e2e-votes.yml@@ -20,6 +20,23 @@# of 499 runs from 2026-09-24 never finished and graduation evidence stopped at 09-30. One# run proceeds and the newest merge waits as the single pending run. The whole campaign# switches off by setting repo variable CODEGRAPH_E2E_VOTES=off once the election passes.+#+# The suite runs as duration-planned shards, one runner each, the way playwright-mock.yml's+# memory shards do: on one runner with one worker it grew from 375 tests in 39.1 min+# (2026-09-30) to 1,321 tests, and run 39682542347 reached 570 of them when its 75-minute+# timeout cancelled it, so no vote finished after 09-30. Every shard plans the same disjoint+# partition (.github/scripts/playwright-shards.mjs over e2e/timings.mock.json), so the+# shards together still run the whole suite and the run is still ONE trial: the codegraph+# shadow unions the specs each shard's ledger logged. A shard that never logs its ledger+# (setup failed) adds no spec to the trial, so it can mint no clean trial for the specs it+# would have run. Each shard's test step has its own time budget below the job's, so a slow+# shard ends as a shard that ran out of time; a job timeout would cancel the whole run, and+# the shadow reads a cancelled run as no trial. A shard that ran out of time still logs the+# specs that FAILED: failing tests run three times, so a regression is what pushes a shard+# past its budget, and dropping that shard would drop exactly the failure the votes exist to+# catch. Its clean specs are not logged (not all of their tests ran), and the shadow ages+# each spec by its own newest observation, so a shard that never finishes stops its specs'+# skips instead of hiding behind the other shards' fresh votes.name: Codegraph E2E Voteson:@@ -45,15 +62,27 @@ env:jobs:vote:- name: vote (full suite)+ name: vote (shard ${{ matrix.shard }}/10)if: vars.CODEGRAPH_E2E_VOTES != 'off'runs-on: ubuntu-latest- # The full suite took 35.8-45.0 min in the runs that finished; 45 cut 153 runs short.- timeout-minutes: 75+ # Sized from e2e/timings.mock.json: 6,151 recorded seconds over 10 shards plans ~10.3 min+ # of tests per shard. PR CI's memory shards (same plan, same workers: 1) ran their test+ # step in 8.3-16.5 min on 2026-10-21/22, up to 1.6x the plan, and the single vote job ran+ # ~7.7 s per test (17 min for a tenth of 1,321). VOTE_TEST_MINUTES gives the tests 28 min+ # (2.7x the plan); the job adds setup (about 2.5 min with npm ci, 70 s of it) and margin.+ timeout-minutes: 40+ # Separate runners isolate each shard's user, MongoDB and fixture state. Keep workers: 1+ # in Playwright; shared-user cleanup is not worker-safe.+ strategy:+ fail-fast: false+ matrix:+ shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]env:CI: 'true'E2E_CHROMIUM_CHANNEL: chromeE2E_STREAM_STORE: memory+ VOTE_SHARD: ${{ matrix.shard }}/10+ VOTE_TEST_MINUTES: '28'steps:- uses: actions/checkout@v5@@ -180,15 +209,39 @@ jobs:continue-on-error: truerun: .github/scripts/install-playwright-fonts.sh- - name: Vote — run the full mock suite (cannot fail the branch)+ # The plan is playwright-mock.yml's: list the suite, assign whole spec files longest+ # first to the least-loaded shard by recorded duration. No --skip (votes run every spec,+ # graduated or not: they are the net that catches a wrong skip) and no --reserve (vote+ # shards run nothing after the suite).+ #+ # The tests run under `timeout`, not the step's timeout-minutes, so the ledger can tell+ # a shard that ran out of time from one whose tests failed: an interrupted shard's+ # results would claim specs whose tests never all ran.+ - name: Vote — run this shard of the mock suite (cannot fail the branch)continue-on-error: true- env:- # Absolute on purpose: Playwright resolves a relative PLAYWRIGHT_JSON_OUTPUT_NAME- # against the CONFIG directory (e2e/), not the working directory — the first live run- # wrote e2e/pw-results.json while the ledger looked in the repo root and logged zero- # trials (fail-safe, but a silent no-op).- PLAYWRIGHT_JSON_OUTPUT_NAME: ${{ github.workspace }}/pw-results.json- run: npx playwright test --config=e2e/playwright.config.mock.ts --reporter=line,json+ run: |+ PLAYWRIGHT_JSON_OUTPUT_FILE=e2e/.generated/playwright-list.json \+ npx playwright test --config=e2e/playwright.config.mock.ts --list --reporter=json > /dev/null+ node .github/scripts/playwright-shards.mjs plan \+ --list e2e/.generated/playwright-list.json \+ --timings e2e/timings.mock.json \+ --shard "$VOTE_SHARD" \+ --out e2e/.generated/shard-tests.txt+ # PLAYWRIGHT_JSON_OUTPUT_NAME is absolute on purpose: Playwright resolves a relative+ # one against the CONFIG directory (e2e/), not the working directory — the first live+ # run wrote e2e/pw-results.json while the ledger looked in the repo root and logged+ # zero trials (fail-safe, but a silent no-op).+ set +e+ PLAYWRIGHT_JSON_OUTPUT_NAME="$GITHUB_WORKSPACE/pw-results.json" \+ timeout --signal=INT --kill-after=60 "${VOTE_TEST_MINUTES}m" \+ npx playwright test --config=e2e/playwright.config.mock.ts \+ --test-list=e2e/.generated/shard-tests.txt --reporter=line,json+ rc=$?+ if [ "$rc" -eq 124 ] || [ "$rc" -eq 137 ]; then+ touch "$GITHUB_WORKSPACE/vote-timed-out"+ echo "::warning::shard $VOTE_SHARD ran out of its ${VOTE_TEST_MINUTES}-minute test budget"+ fi+ exit "$rc"- name: Ledger — log the specs that actually executedrun: |@@ -199,15 +252,26 @@ jobs:# specs needs E2E_MODEL_SPECS_ENFORCE) are discovered by --list yet skip every test# under this job's default env — counting them as covered would mint phantom trials,# the exact bug this workflow revision exists to kill (Codex P1 on #15162). A spec is- # covered iff at least one of its tests reached a non-skipped outcome.- if jq -e '.suites' "$GITHUB_WORKSPACE/pw-results.json" >/dev/null 2>&1; then+ # covered iff at least one of its tests reached a non-skipped outcome. Each shard logs+ # only what it executed; the shadow unions the shards of one run into one trial. A+ # shard that ran out of time logs no clean spec (not all of their tests ran), only+ # the specs with a test Playwright counts as failed (outcome unexpected: every+ # attempt that finished failed), so the shadow records them as failed trials.+ if [ -e "$GITHUB_WORKSPACE/vote-timed-out" ]; then+ echo "codegraph-votes: shard $VOTE_SHARD ran out of time — only its failures logged"+ if jq -e '.suites' "$GITHUB_WORKSPACE/pw-results.json" >/dev/null 2>&1; then+ jq -r '[.suites[] | recurse(.suites[]?) | .specs[]? | select(any(.tests[]?; .status == "unexpected")) | .file] | unique | .[]' "$GITHUB_WORKSPACE/pw-results.json" \+ | sed 's|^|specs/mock/|' > failed.txt+ if [ -s failed.txt ]; then echo "codegraph-votes-failed: $(tr '\n' ' ' < failed.txt)"; fi+ fi+ elif jq -e '.suites' "$GITHUB_WORKSPACE/pw-results.json" >/dev/null 2>&1; thenjq -r '[.suites[] | recurse(.suites[]?) | .specs[]? | select([.tests[]?.status] | any(. != "skipped")) | .file] | unique | .[]' "$GITHUB_WORKSPACE/pw-results.json" \| sed 's|^|specs/mock/|' > covered.txtN=$(wc -l < covered.txt | tr -d ' ')- echo "codegraph-votes: running $N specs (executed, full suite)"+ echo "codegraph-votes: running $N specs (executed, shard $VOTE_SHARD)"if [ "$N" != "0" ]; then echo "codegraph-votes-specs: $(tr '\n' ' ' < covered.txt)"; fielse- echo "codegraph-votes: no results json — run crashed before reporting; no trials logged"+ echo "codegraph-votes: no results json — shard $VOTE_SHARD crashed before reporting; no trials logged"fiexit 0