~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
ci: always run full pytest and image build on PRs (#7372)
LearningCircuitdbd56ead57f0
diff --git a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/pull_request_template.mdindex ffc95e90af..e03e79b599 100644--- a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md+++ b/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md@@ -9,7 +9,7 @@ regression test that fails on `main` and passes on this branch. -->## CI test coverage-Lint, security and focused checks run automatically. The shared test image is built only when full pytest or a selected focused check needs it. When this PR is nearly ready, add `test:pytest`, `code-ready`, `code-ready-preliminary`, or `auto-merge` to request the full pytest suite. Passing full pytest remains required before merge; until requested, that check reports a short failure explaining the missing label. Later pushes rerun the suite while any of those labels remains.+Lint, security checks, the shared test-image build, and the full pytest suite run automatically on every push. Passing full pytest is required before merge; no label is needed to start it, and a red check always means a failure, not a missing request.Additional heavy suites use these labels:diff --git a/.github/labels.yml b/.github/labels.ymlindex 6daac9febe..5b41ee1596 100644--- a/.github/labels.yml+++ b/.github/labels.yml@@ -189,12 +189,9 @@# Workflow/control labels. These are deliberately excluded from the AI# reviewer's output allowlist.-- name: 'test:pytest'- color: '0e8a16'- description: 'Maintainer request for full pytest before merge; reruns on later pushes while the label remains.'- name: code-ready-preliminarycolor: 'c2e0c6'- description: 'Preliminary readiness; requests full pytest. Not human-verified; re-check before code-ready.'+ description: 'Preliminary readiness. Not human-verified; re-check before code-ready.'- name: ai_code_reviewcolor: '6f42c1'description: 'Maintainer-only: apply or re-add to run AI code review. Release notes: 🔄 Automation (18/20).'@@ -206,7 +203,7 @@description: 'Branch synchronization PR. Release notes: 🔄 Branch Syncs & Automation (18/20).'- name: auto-mergecolor: 'ededed'- description: 'PR approved for automatic merging; requests full pytest. Release notes: 🔄 Automation (18/20).'+ description: 'PR approved for automatic merging. Release notes: 🔄 Automation (18/20).'- name: merge-blockedcolor: 'ededed'description: 'PR is blocked from merging. Release notes: 🚧 Review & Blocked Items (19/20).'@@ -229,4 +226,4 @@description: PR needs substantive rework before it can be reviewed in depth- name: code-readycolor: '006b75'- description: Code looks ready; requests full pytest, still needs approval. Apply manually — never auto-applied.+ description: Code looks ready; still needs approval. Apply manually — never auto-applied.diff --git a/.github/workflows/README.md b/.github/workflows/README.mdindex 50995fdac1..65c29532ed 100644--- a/.github/workflows/README.md+++ b/.github/workflows/README.md@@ -5,61 +5,35 @@ This directory contains GitHub Actions workflows for automated development tasks## Full pytest scheduling`docker-tests.yml` keeps **All Pytest Tests + Coverage** required before merge.-A PR requests full pytest when it carries any of these labels:--- `test:pytest` — explicit full-suite request.-- `code-ready` — code is ready for CI and approval.-- `code-ready-preliminary` — preliminary readiness; full validation is needed.-- `auto-merge` — the PR has been marked for merging.--Adding any request label starts full pytest against the PR's test merge commit;-later pushes and reopen events rerun it while at least one remains. Remove all-request labels to stop full pytest on future pushes. Removing and re-adding one-requests another run. These labels request tests; they do not approve or merge-a PR. A label added by a workflow using `GITHUB_TOKEN` does not itself start-another workflow; apply it with a maintainer token or GitHub App, or validation-will wait for a later eligible PR event.--Without a request label, the required job fails a short guard before checkout or test-setup. This is an intentional merge block, not a failing test. A skipped job-would satisfy GitHub's required-check rule, so the job must not be skipped for-an unrequested PR. Failed or cancelled image builds also cannot make it pass.-Unrelated label events skip work under separate, non-required check names and-cannot cancel a real PR run or overwrite its required checks.--The shared test image is built only when full pytest is requested or an-automatic focused check needs it (LLM, infrastructure, or accessibility paths).-A PR without request labels or any of those checks skips the image build while the-required pytest request guard still blocks merging. Production-image smoke-tests retain their separate path-based build and do not need the test image.--Main retains automatic full pytest on pushes as a second check of the code that-landed, plus a daily run at **03:23 UTC**. Both can publish validated coverage for-the current main commit. Manual dispatch and reusable callers (including-mandatory release validation and the full UI label) also run the full suite.-The full UI label already requests full pytest through its reusable strict-workflow; it is not also a direct trigger, which would run the suite twice.-Topic and review-request labels (such as `python`, `tests`, `security`, or-`ai_code_review`) do not request full pytest.-Manual runs are useful for diagnosis; request PR validation with a label so-the run uses a `pull_request` event and satisfies required checks.--New direct PR runs cancel obsolete predecessors. The required pytest job uses-`!cancelled()` at job level so cancellation can release it while it is queued,-without waiting for a runner to execute a cancellation step. This explicit-status function still runs the guard after a failed or skipped image build;-ordinary dependency failure cannot skip the required check. A cancelled check-does not satisfy the merge requirement.--Main pushes share one concurrency group: the active run finishes, and only the-newest pending run is retained. Intermediate queued main commits may therefore-be superseded; the required pre-merge validation still applies to every PR.-Reusable release runs and the daily run have isolated concurrency groups.--The request labels are declared in `.github/labels.yml` and maintained by label sync.-The existing required check name is preserved; no branch-rule change is-needed. Existing PRs receive the policy when their workflow includes this-change; merging this change does not cancel or rewrite already queued runs.+The shared test image and the full pytest suite run unconditionally on every+pull request event (opened, synchronize, reopened) and every push to main. No+label requests, defers, or suppresses them: a red check means the suite+failed, never that validation was merely not requested. Superseded direct PR+runs are cancelled automatically, so pushing twice in quick succession does+not pay for the suite twice.++If the image build fails or is cancelled, the pytest job is skipped — a+skipped job would satisfy GitHub's required-check rule — and merging is+blocked by the required **Build Test Image** check instead. Do not remove+that check from the repository ruleset.++The focused suites (LLM, accessibility, production-image smoke) keep their+path filters through `detect-changes`; only the shared image build ignores+them. Manual dispatch and reusable callers (including mandatory release+validation and the full UI label) also run the full suite. Release runs use+isolated concurrency groups so they cannot cancel or queue behind direct PR+runs; main pushes share one group where the active run finishes and only the+newest pending run is retained.++Direct main runs publish validated coverage for the current main commit.+Manual runs are useful for diagnosis; PR validation happens on the PR's own+pushes, so required checks are always satisfied by a `pull_request` event.++The #7101 label gate (`test:pytest` / `code-ready` / `code-ready-preliminary` /+`auto-merge` request labels, the failing guard, and the daily main run) was+reverted: every PR paid a red required check until a maintainer applied a+label, and bot-applied labels could not even start the suite. Readiness+labels remain as review markers; they no longer have any CI effect.## Update NPM Dependencies Workflowdiff --git a/.github/workflows/docker-tests.yml b/.github/workflows/docker-tests.ymlindex bb7e3f55f3..36a7fe46e5 100644--- a/.github/workflows/docker-tests.yml+++ b/.github/workflows/docker-tests.yml@@ -2,13 +2,10 @@ name: Docker Tests (Consolidated)on:pull_request:- types: [opened, synchronize, reopened, labeled]+ types: [opened, synchronize, reopened]branches: [ main ]push:branches: [ main ]- schedule:- # Full pytest on main once daily, away from the top-of-hour queue.- - cron: '23 3 * * *'workflow_call: # Called by ci-gate.yml for release pipelineinputs:strict-mode:@@ -30,25 +27,23 @@ on:required: falseworkflow_dispatch:-# Full pytest request labels: test:pytest, code-ready, code-ready-preliminary,-# and auto-merge. Keep event filters and PR-label checks aligned.+# The image build and the full pytest suite run unconditionally on every PR+# and main push (the #7101 label gate is reverted: a red pytest check means+# the suite failed, never that validation was merely not requested).# Cancel obsolete direct PR runs before they consume build/test capacity.# Main pushes share one group: finish the active run and retain only the newest# pending run (GitHub's default), so frequent merges do not starve validation.-# Unrelated label events get isolated groups and distinct skipped check names:-# they must neither cancel a real run nor replace a required check with a skip.# Reusable invocations retain the caller's event context, so the explicit# workflow_call marker keeps them on unique run-id groups instead of allowing# them to cancel or queue behind the direct PR workflow.concurrency:group: >-${{ !inputs.reusable-invocation && github.event_name == 'pull_request' &&- (github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name)) &&format('docker-tests-direct-pr-{0}', github.event.pull_request.number) ||!inputs.reusable-invocation && github.event_name == 'push' &&github.ref == 'refs/heads/main' && 'docker-tests-direct-main' ||format('docker-tests-isolated-run-{0}', github.run_id) }}- cancel-in-progress: ${{ !inputs.reusable-invocation && github.event_name == 'pull_request' && (github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name)) }}+ cancel-in-progress: ${{ !inputs.reusable-invocation && github.event_name == 'pull_request' }}# Top-level permissions set to minimum (OSSF Scorecard Token-Permissions)permissions: {}@@ -60,8 +55,7 @@ env:jobs:# Detect which paths changed to conditionally run certain jobsdetect-changes:- name: ${{ github.event_name == 'pull_request' && github.event.action == 'labeled' && !contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) && !inputs.reusable-invocation && 'Ignored label (change detection)' || 'detect-changes' }}- if: github.event_name != 'pull_request' || github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) || inputs.reusable-invocation+ name: detect-changesruns-on: ubuntu-latestoutputs:llm: ${{ steps.filter.outputs.llm }}@@ -140,22 +134,19 @@ jobs:- 'lighthouserc.json'- '.github/workflows/docker-tests.yml'- # Build once when a selected test needs the image. Unrequested PRs with no- # Docker-based focused checks must not spend capacity building an unused image.+ # Build the shared test image once per run. Every PR, main push and manual+ # dispatch (on any branch) runs the full pytest suite, so the image is always+ # needed; no label or path filter may suppress it (it is a required check in+ # the repository ruleset). Without it a dispatch on a feature branch would+ # skip pytest and still report success.build-test-image:runs-on: ubuntu-latest- needs: detect-changes- name: ${{ github.event_name == 'pull_request' && github.event.action == 'labeled' && !contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) && !inputs.reusable-invocation && 'Ignored label (image build)' || 'Build Test Image' }}+ name: Build Test Imageif: |- (github.event_name != 'pull_request' || github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) || inputs.reusable-invocation) &&- (github.event_name != 'pull_request' || inputs.reusable-invocation ||- (contains(github.event.pull_request.labels.*.name, 'test:pytest') ||- contains(github.event.pull_request.labels.*.name, 'code-ready') ||- contains(github.event.pull_request.labels.*.name, 'code-ready-preliminary') ||- contains(github.event.pull_request.labels.*.name, 'auto-merge')) ||- needs.detect-changes.outputs.llm == 'true' ||- needs.detect-changes.outputs.infrastructure == 'true' ||- needs.detect-changes.outputs.accessibility == 'true')+ github.event_name == 'pull_request' ||+ github.event_name == 'workflow_dispatch' ||+ github.ref == 'refs/heads/main' ||+ inputs.strict-mode == truesteps:- name: Harden the runner (Audit all outbound calls)@@ -188,17 +179,13 @@ jobs:path: /tmp/ldr-test.tarretention-days: 1- # Keep the existing required check. An unrequested PR fails a cheap guard;- # skipping this job for missing labels would satisfy branch protection.- # !cancelled() bypasses the implicit success() dependency check, so a failed- # or skipped image build still reaches the guard. Unlike always(), it lets- # cancellation release an obsolete queued job without first needing a runner.+ # Required check on every PR and main push. It carries no condition: if the+ # image build fails or is cancelled this job skips, and the required+ # "Build Test Image" check in the repository ruleset is what blocks merging+ # (a skipped job reports success, so this job must not try to guard itself).pytest-tests:runs-on: ubuntu-latest- name: ${{ github.event_name == 'pull_request' && github.event.action == 'labeled' && !contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) && !inputs.reusable-invocation && 'Ignored label (pytest)' || 'All Pytest Tests + Coverage' }}- if: >-- !cancelled() &&- (github.event_name != 'pull_request' || github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) || inputs.reusable-invocation)+ name: All Pytest Tests + Coverage# Cap exists so a hung run doesn't inherit GitHub's 360-minute default# (observed 2026-06-10 on PR #4491: a flaked run sat in the main pytest# step for 2+ hours before being cancelled manually). The original@@ -221,28 +208,6 @@ jobs:pull-requests: write # Needed for PR commentssteps:- - name: Require a pytest request and successful image build- id: pytest_request- env:- PYTEST_REQUESTED: >-- ${{ github.event_name != 'pull_request' || inputs.reusable-invocation ||- contains(github.event.pull_request.labels.*.name, 'test:pytest') ||- contains(github.event.pull_request.labels.*.name, 'code-ready') ||- contains(github.event.pull_request.labels.*.name, 'code-ready-preliminary') ||- contains(github.event.pull_request.labels.*.name, 'auto-merge') }}- TEST_IMAGE_RESULT: ${{ needs.build-test-image.result }}- run: |- set -euo pipefail- if [ "$PYTEST_REQUESTED" != 'true' ]; then- echo 'Full pytest has not been requested. Add test:pytest, code-ready, code-ready-preliminary, or auto-merge when this PR is ready for full validation.' >> "$GITHUB_STEP_SUMMARY"- echo '::error::Full pytest is required before merging. Add test:pytest or a readiness label (code-ready, code-ready-preliminary, auto-merge) to run it.'- exit 1- fi- if [ "$TEST_IMAGE_RESULT" != 'success' ]; then- echo '::error::Full pytest cannot pass without a successful test image build.'- exit 1- fi-- name: Harden the runner (Audit all outbound calls)uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1with:@@ -382,7 +347,7 @@ jobs:# collected") and this step fails hard (cf. #4411). If the last# marked module goes away, delete this step in the same change.- name: Run fd_canary FD-leak tests (serial, no coverage)- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}+ if: ${{ !cancelled() }}run: |docker run --rm -t \-v "$PWD":/app \@@ -461,7 +426,7 @@ jobs:# parity; the news scheduler is only read in the FastAPI lifespan,# which a pytest lane never enters, so this is a no-op here today.- name: Run serial + nonroot (process-global state, permission-bit) tests- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}+ if: ${{ !cancelled() }}run: |docker run --rm -t \--user ldruser \@@ -490,17 +455,17 @@ jobs:--ignore=tests/fuzz"- name: Run JavaScript infrastructure tests- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}+ if: ${{ !cancelled() }}run: |cd tests/infrastructure_tests && npm test- name: Run Vitest JS unit tests- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}+ if: ${{ !cancelled() }}run: |npm ci && npm test- name: Validate coverage data- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}+ if: ${{ !cancelled() }}run: |docker run --rm \-v "$PWD":/app \@@ -510,7 +475,7 @@ jobs:coverage/coverage.xml coverage/coverage-summary.json- name: Generate coverage summary- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}+ if: ${{ !cancelled() }}run: |{echo "## Coverage Report"@@ -536,7 +501,7 @@ jobs:} >> "$GITHUB_STEP_SUMMARY"- name: Upload coverage reports- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' && hashFiles('coverage/**') != '' }}+ if: ${{ !cancelled() && hashFiles('coverage/**') != '' }}uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2with:name: coverage-report@@ -544,7 +509,7 @@ jobs:# Fix permissions on coverage directory (Docker creates files as root)- name: Fix coverage directory permissions- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}+ if: ${{ !cancelled() }}run: |if [ -d "coverage" ]; thensudo chown -R "$(id -u):$(id -g)" coverage/@@ -564,7 +529,7 @@ jobs:# Add coverage summary as PR comment for easy visibility- name: Comment coverage on PR- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' && github.event_name == 'pull_request' }}+ if: ${{ !cancelled() && github.event_name == 'pull_request' }}continue-on-error: true # Don't fail the job if GitHub API is throttleduses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0with:@@ -745,7 +710,7 @@ jobs:});}- # Publish only a complete, validated report from a direct main push or daily run.+ # Publish only a complete, validated report from a direct main push.# Release-gate's nested strict-mode invocation produces its own# coverage artifact but must not race this job for gh-pages or the badge.publish-coverage:@@ -756,7 +721,7 @@ jobs:${{!cancelled() &&needs.pytest-tests.result == 'success' &&- (github.event_name == 'push' || github.event_name == 'schedule') &&+ github.event_name == 'push' &&github.ref == 'refs/heads/main' &&!inputs.reusable-invocation &&inputs.strict-mode != truediff --git a/changelog.d/+always-run-full-pytest.misc.md b/changelog.d/+always-run-full-pytest.misc.mdnew file mode 100644index 0000000000..2f1271df1c--- /dev/null+++ b/changelog.d/+always-run-full-pytest.misc.md@@ -0,0 +1 @@+Full pytest and the shared test-image build run automatically on every pull request and main push again. The request labels (`test:pytest`, `code-ready`, `code-ready-preliminary`, `auto-merge`), the failing "add a label" guard, and the daily main run are gone: a red "All Pytest Tests + Coverage" check now always means the suite failed, never that validation was not requested. Superseded direct PR runs are cancelled automatically; main-queue and release validation behavior is unchanged.diff --git a/changelog.d/+pytest-readiness-labels.misc.md b/changelog.d/+pytest-readiness-labels.misc.mddeleted file mode 100644index a3dda14905..0000000000--- a/changelog.d/+pytest-readiness-labels.misc.md+++ /dev/null@@ -1 +0,0 @@-Full pytest now starts automatically for PRs labeled `code-ready`, `code-ready-preliminary`, or `auto-merge`, as well as `test:pytest`, and reruns on later pushes while any request label remains. Full validation remains required before merging; unrelated labels and existing release validation retain their behavior.diff --git a/changelog.d/7101.misc.md b/changelog.d/7101.misc.mddeleted file mode 100644index b20e17e1d7..0000000000--- a/changelog.d/7101.misc.md+++ /dev/null@@ -1,5 +0,0 @@-Full pytest for pull requests is requested with the `test:pytest` label and-remains required before merge; unused test-image builds are skipped. Main-push,-daily, and release validation still run automatically, with the main queue-retaining one active run and only the newest pending run. Superseded PR pytest-jobs can be cancelled while queued without waiting for a runner.diff --git a/tests/ci/standalone/test_pytest_scheduling.py b/tests/ci/standalone/test_pytest_scheduling.pyindex db17718120..7364706d33 100644--- a/tests/ci/standalone/test_pytest_scheduling.py+++ b/tests/ci/standalone/test_pytest_scheduling.py@@ -1,21 +1,25 @@-"""Exercise the actual workflow conditions and guard without app dependencies.+"""Exercise the actual workflow conditions without app dependencies.++Policy under test (revert of the #7101 label gate): the shared test image and+the full pytest suite run unconditionally on every pull request and every+main push. No label can request, defer, or suppress them; a red+"All Pytest Tests + Coverage" check therefore means the suite failed, not+that validation was never requested. Merge enforcement for a failed image+build lives in the repository ruleset, which lists "Build Test Image" as a+required check; this file can only verify the workflow side of that contract.Actionlint validates YAML and Actions syntax separately. The small expressioninterpreter below accepts only the operators/functions used by these conditions-and rejects unknown syntax or context. Conditions also account for GitHub's-implicit success() check. These are policy regressions, not a simulation of-GitHub's scheduler or proof of a hosted run's final check conclusion.+and rejects unknown syntax or context. These are policy regressions, not a+simulation of GitHub's scheduler or proof of a hosted run's final check+conclusion."""import astimport json-import osimport re-import subprocess-import tempfileimport textwrapimport unittest-from itertools import productfrom pathlib import Path@@ -25,17 +29,29 @@ TOKEN = re.compile(r"\s*('(?:[^']|'')*'|&&|\|\||==|!=|!|[(),]|"r"[A-Za-z_][A-Za-z0-9_.*-]*|[0-9]+)")-REQUEST_LABELS = (+# Every #7101 gating mechanism that must stay out of the workflow: the+# request labels, the label-event special cases, and the failing guard.+GATE_ARTIFACTS = ("test:pytest","code-ready","code-ready-preliminary","auto-merge",+ "github.event.label",+ "github.event.action",+ "PYTEST_REQUESTED",+ "Full pytest is required before merging",)REQUIRED = {"detect-changes": "detect-changes","build-test-image": "Build Test Image","pytest-tests": "All Pytest Tests + Coverage",}+FOCUSED_JOBS = (+ ("llm-unit-tests", "llm"),+ ("llm-example-tests", "llm"),+ ("accessibility-tests", "accessibility"),+ ("ldr-production-smoke-test", "docker"),+)def job(name):@@ -70,14 +86,6 @@ def field(text, key, indent):raise AssertionError(f"Missing field: {key}")-def steps():- return re.split(r"^ - ", job("pytest-tests"), flags=re.MULTILINE)[1:]---def step_with_id(step_id):- return next(step for step in steps() if f" id: {step_id}\n" in step)--def evaluate(expression, context, *, cancelled=False, success=True):expression = expression.strip().removeprefix("${{").removesuffix("}}")translated = []@@ -160,433 +168,161 @@ def evaluate(expression, context, *, cancelled=False, success=True):return visit(ast.parse(" ".join(translated).strip(), mode="eval").body)-def evaluate_condition(expression, context, *, cancelled=False, success=True):- # GitHub adds success() unless the condition uses a status function. Without- # this rule a removed !cancelled() could silently skip the required guard- # after a failed/skipped build while these tests still pass.- if not re.search(- r"\b(?:always|cancelled|failure|success)\s*\(", expression- ):- if not success:- return False- return evaluate(expression, context, cancelled=cancelled, success=success)--def context(event="pull_request",action="synchronize",- label="",*,- requested=False,+ ref=None,reusable=False,- labels=None,+ strict=None,):+ if ref is None:+ ref = (+ "refs/pull/42/merge"+ if event == "pull_request"+ else "refs/heads/main"+ )+ if strict is None:+ strict = reusable # workflow_call defaults strict-mode to truereturn {"github.event_name": event,"github.event.action": action,- "github.event.label.name": label,- "github.event.pull_request.labels.*.name": (- list(labels)- if labels is not None- else (["test:pytest"] if requested else [])- ),"github.event.pull_request.number": 42,- "github.ref": "refs/pull/42/merge"- if event == "pull_request"- else "refs/heads/main",+ "github.ref": ref,"github.run_id": 123,"inputs.reusable-invocation": reusable,- "inputs.strict-mode": reusable,+ "inputs.strict-mode": strict,"needs.pytest-tests.result": "success",+ "needs.build-test-image.result": "success","needs.detect-changes.outputs.llm": "false","needs.detect-changes.outputs.infrastructure": "false",- "needs.detect-changes.outputs.accessibility": "false","needs.detect-changes.outputs.docker": "false",- "steps.pytest_request.outcome": "success",+ "needs.detect-changes.outputs.accessibility": "false",}class PytestSchedulingTests(unittest.TestCase):- def test_pr_lifecycle_keeps_pytest_required_and_skips_unused_images(self):- for action in ("opened", "synchronize", "reopened"):- for requested in (False, True):- ctx = context(action=action, requested=requested)- with self.subTest(action=action, requested=requested):- for name, required in REQUIRED.items():- self.assertEqual(- bool(evaluate(field(job(name), "if", 4), ctx)),- name != "build-test-image" or requested,- )- self.assertEqual(- evaluate(field(job(name), "name", 4), ctx), required- )- self.assertEqual(- evaluate(- field(job("pytest-tests"), "PYTEST_REQUESTED", 10),- ctx,- ),- requested,- )-- def test_only_request_labels_run_or_replace_required_checks(self):- for label in (- *REQUEST_LABELS,- "bugfix",- "test:notes",- "test:ui-full-shards",- ):- for requested in (False, True):- ctx = context(- action="labeled", label=label, requested=requested- )- relevant = label in REQUEST_LABELS- with self.subTest(label=label, requested=requested):- for name, required in REQUIRED.items():- self.assertEqual(- bool(evaluate(field(job(name), "if", 4), ctx)),- relevant- and (name != "build-test-image" or requested),- )- self.assertEqual(- evaluate(field(job(name), "name", 4), ctx)- == required,- relevant,- )+ def test_pr_trigger_excludes_label_events_and_daily_schedule(self):+ self.assertEqual(+ field(WORKFLOW, "types", 4),+ "[opened, synchronize, reopened]",+ )+ self.assertNotRegex(WORKFLOW, r"(?m)^\s*schedule:")- def test_each_readiness_label_requests_full_pytest_on_events_and_later_pushes(- self,- ):- for label, action in product(- REQUEST_LABELS, ("opened", "labeled", "synchronize", "reopened")- ):- ctx = context(action=action, label=label, labels=[label])- with self.subTest(label=label, action=action):- for name, required in REQUIRED.items():- self.assertTrue(- evaluate_condition(field(job(name), "if", 4), ctx)- )- self.assertEqual(- evaluate(field(job(name), "name", 4), ctx), required- )- self.assertTrue(- evaluate(- field(job("pytest-tests"), "PYTEST_REQUESTED", 10), ctx- )- )- self.assertTrue(- evaluate(field(WORKFLOW, "cancel-in-progress", 2), ctx)- )- self.assertEqual(- evaluate(field(WORKFLOW, "group", 2), ctx),- evaluate(field(WORKFLOW, "group", 2), context()),- )+ def test_no_label_gate_remains_in_the_workflow(self):+ for artifact in GATE_ARTIFACTS:+ with self.subTest(artifact=artifact):+ self.assertNotIn(artifact, WORKFLOW)- def test_unrelated_labels_do_not_interrupt_a_readiness_requested_run(self):- for requested_label, event_label in product(- REQUEST_LABELS,- (- "bugfix",- "python",- "security",- "tests",- "test:notes",- "test:ui-full-shards",- ),- ):- ctx = context(- action="labeled",- label=event_label,- labels=[requested_label, event_label],- )+ def test_required_checks_run_on_every_pr_and_main_push(self):+ contexts = [+ context(action=a) for a in ("opened", "synchronize", "reopened")+ ]+ contexts += [+ context("push"),+ context(reusable=True),+ context("workflow_dispatch"),+ ]+ for ctx in contexts:with self.subTest(- requested_label=requested_label, event_label=event_label+ event=ctx["github.event_name"],+ action=ctx["github.event.action"],+ reusable=ctx["inputs.reusable-invocation"],):- for name, required in REQUIRED.items():- self.assertFalse(- evaluate_condition(field(job(name), "if", 4), ctx)- )- self.assertNotEqual(- evaluate(field(job(name), "name", 4), ctx), required- )- self.assertFalse(- evaluate(field(WORKFLOW, "cancel-in-progress", 2), ctx)- )- self.assertNotEqual(- evaluate(field(WORKFLOW, "group", 2), ctx),- evaluate(field(WORKFLOW, "group", 2), context()),- )-- def test_readiness_requests_still_fail_closed_and_allow_cancellation(self):- for label, result in product(- REQUEST_LABELS, ("failure", "skipped", "cancelled")- ):- ctx = context(action="labeled", label=label, labels=[label])- ctx["needs.build-test-image.result"] = result- with self.subTest(label=label, image_result=result):+ for job_id, check in REQUIRED.items():+ self.assertEqual(field(job(job_id), "name", 4), check)self.assertTrue(- evaluate_condition(- field(job("pytest-tests"), "if", 4), ctx, success=False- )- )- self.assertFalse(- evaluate_condition(- field(job("pytest-tests"), "if", 4),- ctx,- cancelled=True,- success=False,- )+ evaluate(field(job("build-test-image"), "if", 4), ctx))-- def test_main_push_runs_full_pytest_without_a_pr_label(self):- ctx = context("push")+ # A manual dispatch on a non-main branch also builds the image, so its+ # pytest runs instead of skipping (a skipped job reports success); a+ # push to any other branch does not trigger the workflow.self.assertTrue(- evaluate(field(job("pytest-tests"), "PYTEST_REQUESTED", 10), ctx)- )- for name, required in REQUIRED.items():- self.assertTrue(evaluate(field(job(name), "if", 4), ctx))- self.assertEqual(- evaluate(field(job(name), "name", 4), ctx), required+ evaluate(+ field(job("build-test-image"), "if", 4),+ context("workflow_dispatch", ref="refs/heads/feature"),)-- def test_image_is_built_exactly_when_pytest_or_a_focused_consumer_needs_it(- self,- ):- job_names = re.findall(- r"^ ([\w-]+):$", WORKFLOW.split("\njobs:\n", 1)[1], re.MULTILINE)- focused_consumers = [- name- for name in job_names- if name != "pytest-tests"- and re.search(- r"^ needs:.*\bbuild-test-image\b", job(name), re.MULTILINE- )- ]- self.assertTrue(focused_consumers)++ def test_detect_changes_and_pytest_carry_no_condition(self):+ # The pre-#7101 shape: these jobs run whenever their dependencies+ # allow. A reintroduced `if:` here is how the #7101 gate came back.+ for job_id in ("detect-changes", "pytest-tests"):+ with self.subTest(job=job_id), self.assertRaises(AssertionError):+ field(job(job_id), "if", 4)++ def test_pytest_depends_on_the_image_build_without_a_request_guard(self):self.assertEqual(- field(job("build-test-image"), "needs", 4), "detect-changes"+ field(job("pytest-tests"), "needs", 4), "build-test-image")- for requested, llm, infrastructure, accessibility, docker in product(- (False, True), repeat=5- ):- ctx = context(requested=requested)- for path, changed in (- ("llm", llm),- ("infrastructure", infrastructure),- ("accessibility", accessibility),- ("docker", docker),- ):- ctx[f"needs.detect-changes.outputs.{path}"] = str(- changed- ).lower()- selected = any(- evaluate(field(job(name), "if", 4), ctx)- for name in focused_consumers- )- with self.subTest(- requested=requested,- llm=llm,- infrastructure=infrastructure,- accessibility=accessibility,- docker=docker,- ):- self.assertEqual(- bool(- evaluate(field(job("build-test-image"), "if", 4), ctx)- ),- requested or selected,- )-- def test_daily_manual_and_release_callers_run_full_pytest(self):- cases = [context("schedule"), context("workflow_dispatch")]- cases += [- context(- event,- action="labeled",- label="test:ui-full-shards",- reusable=True,- )- for event in (- "push",- "pull_request",- "workflow_dispatch",- "schedule",- )- ]- for ctx in cases:- with self.subTest(context=ctx):- self.assertTrue(- evaluate(field(job("build-test-image"), "if", 4), ctx)- )- self.assertTrue(- evaluate(field(job("pytest-tests"), "if", 4), ctx)- )- self.assertTrue(- evaluate(- field(job("pytest-tests"), "PYTEST_REQUESTED", 10), ctx- )- )- self.assertEqual(- evaluate(field(job("pytest-tests"), "name", 4), ctx),- REQUIRED["pytest-tests"],+ self.assertNotIn("pytest_request", job("pytest-tests"))+ self.assertNotIn("PYTEST_REQUESTED", job("pytest-tests"))++ def test_image_build_is_unconditional_and_path_filter_free(self):+ build = job("build-test-image")+ with self.assertRaises(AssertionError):+ field(build, "needs", 4)+ self.assertNotIn("detect-changes", build)+ self.assertNotIn("labels", build)++ def test_focused_checks_keep_their_path_filters(self):+ # Path filters still gate the focused suites themselves; only the+ # shared image build stopped consulting them.+ for job_id, output in FOCUSED_JOBS:+ with self.subTest(job=job_id):+ condition = field(job(job_id), "if", 4)+ self.assertIn(+ f"needs.detect-changes.outputs.{output}", condition)- self.assertRegex(- WORKFLOW, r"schedule:\s*\n\s*#.*\n\s*- cron: '23 3 \* \* \*'"- )- release_caller = (ROOT / ".github/workflows/ci-gate.yml").read_text()- self.assertRegex(- release_caller,- r"uses: \$/\.github/workflows/docker-tests.yml\s+with:\s+strict-mode: true",- )- def test_concurrency_cancels_only_relevant_direct_pr_runs(self):+ def test_concurrency_cancels_superseded_direct_pr_runs_only(self):group = field(WORKFLOW, "group", 2)cancel = field(WORKFLOW, "cancel-in-progress", 2)- direct = context()- label = context(action="labeled", label="test:pytest", requested=True)- self.assertEqual(evaluate(group, direct), evaluate(group, label))- for ctx in (direct, label):+ direct = [+ context(action=a) for a in ("opened", "synchronize", "reopened")+ ]+ self.assertEqual(len({evaluate(group, ctx) for ctx in direct}), 1)+ for ctx in direct:self.assertTrue(evaluate(cancel, ctx))+ push, newer = context("push"), {**context("push"), "github.run_id": 456}+ self.assertEqual(evaluate(group, push), evaluate(group, newer))+ self.assertFalse(evaluate(cancel, push))for ctx in (- context(action="labeled", label="bugfix"),- context("schedule"),+ context("workflow_dispatch", ref="refs/heads/feature"),context(reusable=True),context("push", reusable=True),):self.assertFalse(evaluate(cancel, ctx))- self.assertNotEqual(evaluate(group, ctx), evaluate(group, direct))self.assertNotEqual(- evaluate(group, ctx), evaluate(group, context("push"))+ evaluate(group, ctx), evaluate(group, direct[0]))-- def test_main_pushes_keep_active_run_and_share_single_pending_slot(self):- group = field(WORKFLOW, "group", 2)- cancel = field(WORKFLOW, "cancel-in-progress", 2)- first, latest = context("push"), context("push")- latest["github.run_id"] = 456- self.assertEqual(evaluate(group, first), evaluate(group, latest))- for ctx in (first, latest):- self.assertFalse(evaluate(cancel, ctx))- # GitHub retains just the newest pending run by default. Do not opt- # into queue: max, which would restore the main-push backlog.+ self.assertNotEqual(evaluate(group, ctx), evaluate(group, push))+ # Main keeps GitHub's default single-pending-slot behavior; queueing+ # every push would restore the main-push backlog.concurrency = WORKFLOW.split("\nconcurrency:\n", 1)[1].split("\npermissions:", 1)[0]- self.assertNotRegex(concurrency, r"\bqueue:\s*max\b")+ self.assertNotRegex(concurrency, r"\bqueue:")- def test_only_successful_direct_main_runs_publish_coverage(self):+ def test_publish_coverage_only_from_a_successful_direct_main_push(self):condition = field(job("publish-coverage"), "if", 4)- for event in ("push", "pull_request", "schedule", "workflow_dispatch"):- for reusable in (False, True):- for result in ("success", "failure", "cancelled", "skipped"):- ctx = context(event, reusable=reusable)- ctx["needs.pytest-tests.result"] = result- expected = (- event in {"push", "schedule"}- and not reusable- and result == "success"- )- self.assertEqual(bool(evaluate(condition, ctx)), expected)- self.assertFalse(evaluate(condition, ctx, cancelled=True))-- def test_guard_fails_closed_before_checkout_or_testing(self):- pytest = job("pytest-tests")- guard = step_with_id("pytest_request")- script = field(guard, "run", 8)- self.assertNotIn("continue-on-error", guard)- self.assertLess(- pytest.index("id: pytest_request"),- pytest.index("actions/checkout@"),- )- for requested in ("true", "false", "", "unexpected"):- for image_result in (- "success",- "failure",- "cancelled",- "skipped",- "",- ):- with (- self.subTest(- requested=requested, image_result=image_result- ),- tempfile.TemporaryDirectory() as temp,- ):- result = subprocess.run(- ["bash", "-c", script],- env={- **os.environ,- "PYTEST_REQUESTED": requested,- "TEST_IMAGE_RESULT": image_result,- "GITHUB_STEP_SUMMARY": str(Path(temp) / "summary"),- },- capture_output=True,- text=True,- check=False,- )- self.assertEqual(- result.returncode == 0,- requested == "true" and image_result == "success",- )- if requested == "false":- self.assertIn("test:pytest", result.stdout)-- def test_unsuccessful_image_build_cannot_skip_required_guard(self):- condition = field(job("pytest-tests"), "if", 4)- for requested, image_result in product(- (False, True), ("failure", "skipped", "cancelled")- ):- ctx = context(requested=requested)- ctx["needs.build-test-image.result"] = image_result- with self.subTest(requested=requested, image_result=image_result):- self.assertTrue(- evaluate_condition(condition, ctx, success=False)- )-- def test_cancellation_releases_pytest_without_waiting_for_a_runner(self):- condition = field(job("pytest-tests"), "if", 4)- cases = [- context(),- context(requested=True),- context(action="labeled", label="test:pytest", requested=True),- context("push"),- context("schedule"),- context("workflow_dispatch"),+ self.assertNotIn("schedule", condition)+ self.assertTrue(evaluate(condition, context("push")))+ for ctx in (+ {**context("push"), "needs.pytest-tests.result": "failure"},+ context("pull_request"),context(reusable=True),- ]- for ctx, image_result in product(- cases, ("success", "failure", "cancelled", "skipped")+ context("workflow_dispatch", ref="refs/heads/feature"),):- ctx = {**ctx, "needs.build-test-image.result": image_result}with self.subTest(context=ctx):- # Cancellation is evaluated at the job, before a runner can- # execute any step. A cancelled() shell guard cannot release- # an obsolete queued job from its workflow concurrency group.- self.assertFalse(- evaluate_condition(- condition,- ctx,- cancelled=True,- success=image_result == "success",- )- )-- def test_failure_cleanup_stops_after_guard_failure_or_cancellation(self):- guarded_steps = 0- for step in steps():- if " if:" not in step:- continue # GitHub's default success() already stops these.- condition = field(step, "if", 8)- guarded_steps += 1- for result in ("failure", "cancelled", "skipped"):- ctx = context(requested=False)- ctx["steps.pytest_request.outcome"] = resultself.assertFalse(evaluate(condition, ctx))- self.assertTrue(evaluate(condition, context(requested=True)))- self.assertFalse(- evaluate(condition, context(requested=True), cancelled=True)- )- self.assertGreater(guarded_steps, 0)++ def test_release_pipeline_still_calls_the_strict_suite(self):+ release_caller = (ROOT / ".github/workflows/ci-gate.yml").read_text()+ self.assertRegex(+ release_caller,+ r"uses: \$/\.github/workflows/docker-tests.yml\s+with:\s+strict-mode: true",+ )if __name__ == "__main__":