Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

ci: always run full pytest and image build on PRs (#7372)

LearningCircuitdbd56ead57f0
diff --git a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md
index ffc95e90af..e03e79b599 100644
--- a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md
+++ b/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md
@@ -9,7 +9,7 @@ regression test that fails on `main` and passes on this branch. -->
## CI test coverage
-Lint, security and focused checks run automatically. The shared test image is built only when full pytest or a selected focused check needs it. When this PR is nearly ready, add `test:pytest`, `code-ready`, `code-ready-preliminary`, or `auto-merge` to request the full pytest suite. Passing full pytest remains required before merge; until requested, that check reports a short failure explaining the missing label. Later pushes rerun the suite while any of those labels remains.
+Lint, security checks, the shared test-image build, and the full pytest suite run automatically on every push. Passing full pytest is required before merge; no label is needed to start it, and a red check always means a failure, not a missing request.
Additional heavy suites use these labels:
diff --git a/.github/labels.yml b/.github/labels.yml
index 6daac9febe..5b41ee1596 100644
--- a/.github/labels.yml
+++ b/.github/labels.yml
@@ -189,12 +189,9 @@
# Workflow/control labels. These are deliberately excluded from the AI
# reviewer's output allowlist.
-- name: 'test:pytest'
- color: '0e8a16'
- description: 'Maintainer request for full pytest before merge; reruns on later pushes while the label remains.'
- name: code-ready-preliminary
color: 'c2e0c6'
- description: 'Preliminary readiness; requests full pytest. Not human-verified; re-check before code-ready.'
+ description: 'Preliminary readiness. Not human-verified; re-check before code-ready.'
- name: ai_code_review
color: '6f42c1'
description: 'Maintainer-only: apply or re-add to run AI code review. Release notes: 🔄 Automation (18/20).'
@@ -206,7 +203,7 @@
description: 'Branch synchronization PR. Release notes: 🔄 Branch Syncs & Automation (18/20).'
- name: auto-merge
color: 'ededed'
- description: 'PR approved for automatic merging; requests full pytest. Release notes: 🔄 Automation (18/20).'
+ description: 'PR approved for automatic merging. Release notes: 🔄 Automation (18/20).'
- name: merge-blocked
color: 'ededed'
description: 'PR is blocked from merging. Release notes: 🚧 Review & Blocked Items (19/20).'
@@ -229,4 +226,4 @@
description: PR needs substantive rework before it can be reviewed in depth
- name: code-ready
color: '006b75'
- description: Code looks ready; requests full pytest, still needs approval. Apply manually — never auto-applied.
+ description: Code looks ready; still needs approval. Apply manually — never auto-applied.
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 50995fdac1..65c29532ed 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -5,61 +5,35 @@ This directory contains GitHub Actions workflows for automated development tasks
## Full pytest scheduling
`docker-tests.yml` keeps **All Pytest Tests + Coverage** required before merge.
-A PR requests full pytest when it carries any of these labels:
-
-- `test:pytest` — explicit full-suite request.
-- `code-ready` — code is ready for CI and approval.
-- `code-ready-preliminary` — preliminary readiness; full validation is needed.
-- `auto-merge` — the PR has been marked for merging.
-
-Adding any request label starts full pytest against the PR's test merge commit;
-later pushes and reopen events rerun it while at least one remains. Remove all
-request labels to stop full pytest on future pushes. Removing and re-adding one
-requests another run. These labels request tests; they do not approve or merge
-a PR. A label added by a workflow using `GITHUB_TOKEN` does not itself start
-another workflow; apply it with a maintainer token or GitHub App, or validation
-will wait for a later eligible PR event.
-
-Without a request label, the required job fails a short guard before checkout or test
-setup. This is an intentional merge block, not a failing test. A skipped job
-would satisfy GitHub's required-check rule, so the job must not be skipped for
-an unrequested PR. Failed or cancelled image builds also cannot make it pass.
-Unrelated label events skip work under separate, non-required check names and
-cannot cancel a real PR run or overwrite its required checks.
-
-The shared test image is built only when full pytest is requested or an
-automatic focused check needs it (LLM, infrastructure, or accessibility paths).
-A PR without request labels or any of those checks skips the image build while the
-required pytest request guard still blocks merging. Production-image smoke
-tests retain their separate path-based build and do not need the test image.
-
-Main retains automatic full pytest on pushes as a second check of the code that
-landed, plus a daily run at **03:23 UTC**. Both can publish validated coverage for
-the current main commit. Manual dispatch and reusable callers (including
-mandatory release validation and the full UI label) also run the full suite.
-The full UI label already requests full pytest through its reusable strict
-workflow; it is not also a direct trigger, which would run the suite twice.
-Topic and review-request labels (such as `python`, `tests`, `security`, or
-`ai_code_review`) do not request full pytest.
-Manual runs are useful for diagnosis; request PR validation with a label so
-the run uses a `pull_request` event and satisfies required checks.
-
-New direct PR runs cancel obsolete predecessors. The required pytest job uses
-`!cancelled()` at job level so cancellation can release it while it is queued,
-without waiting for a runner to execute a cancellation step. This explicit
-status function still runs the guard after a failed or skipped image build;
-ordinary dependency failure cannot skip the required check. A cancelled check
-does not satisfy the merge requirement.
-
-Main pushes share one concurrency group: the active run finishes, and only the
-newest pending run is retained. Intermediate queued main commits may therefore
-be superseded; the required pre-merge validation still applies to every PR.
-Reusable release runs and the daily run have isolated concurrency groups.
-
-The request labels are declared in `.github/labels.yml` and maintained by label sync.
-The existing required check name is preserved; no branch-rule change is
-needed. Existing PRs receive the policy when their workflow includes this
-change; merging this change does not cancel or rewrite already queued runs.
+The shared test image and the full pytest suite run unconditionally on every
+pull request event (opened, synchronize, reopened) and every push to main. No
+label requests, defers, or suppresses them: a red check means the suite
+failed, never that validation was merely not requested. Superseded direct PR
+runs are cancelled automatically, so pushing twice in quick succession does
+not pay for the suite twice.
+
+If the image build fails or is cancelled, the pytest job is skipped — a
+skipped job would satisfy GitHub's required-check rule — and merging is
+blocked by the required **Build Test Image** check instead. Do not remove
+that check from the repository ruleset.
+
+The focused suites (LLM, accessibility, production-image smoke) keep their
+path filters through `detect-changes`; only the shared image build ignores
+them. Manual dispatch and reusable callers (including mandatory release
+validation and the full UI label) also run the full suite. Release runs use
+isolated concurrency groups so they cannot cancel or queue behind direct PR
+runs; main pushes share one group where the active run finishes and only the
+newest pending run is retained.
+
+Direct main runs publish validated coverage for the current main commit.
+Manual runs are useful for diagnosis; PR validation happens on the PR's own
+pushes, so required checks are always satisfied by a `pull_request` event.
+
+The #7101 label gate (`test:pytest` / `code-ready` / `code-ready-preliminary` /
+`auto-merge` request labels, the failing guard, and the daily main run) was
+reverted: every PR paid a red required check until a maintainer applied a
+label, and bot-applied labels could not even start the suite. Readiness
+labels remain as review markers; they no longer have any CI effect.
## Update NPM Dependencies Workflow
diff --git a/.github/workflows/docker-tests.yml b/.github/workflows/docker-tests.yml
index bb7e3f55f3..36a7fe46e5 100644
--- a/.github/workflows/docker-tests.yml
+++ b/.github/workflows/docker-tests.yml
@@ -2,13 +2,10 @@ name: Docker Tests (Consolidated)
on:
pull_request:
- types: [opened, synchronize, reopened, labeled]
+ types: [opened, synchronize, reopened]
branches: [ main ]
push:
branches: [ main ]
- schedule:
- # Full pytest on main once daily, away from the top-of-hour queue.
- - cron: '23 3 * * *'
workflow_call: # Called by ci-gate.yml for release pipeline
inputs:
strict-mode:
@@ -30,25 +27,23 @@ on:
required: false
workflow_dispatch:
-# Full pytest request labels: test:pytest, code-ready, code-ready-preliminary,
-# and auto-merge. Keep event filters and PR-label checks aligned.
+# The image build and the full pytest suite run unconditionally on every PR
+# and main push (the #7101 label gate is reverted: a red pytest check means
+# the suite failed, never that validation was merely not requested).
# Cancel obsolete direct PR runs before they consume build/test capacity.
# Main pushes share one group: finish the active run and retain only the newest
# pending run (GitHub's default), so frequent merges do not starve validation.
-# Unrelated label events get isolated groups and distinct skipped check names:
-# they must neither cancel a real run nor replace a required check with a skip.
# Reusable invocations retain the caller's event context, so the explicit
# workflow_call marker keeps them on unique run-id groups instead of allowing
# them to cancel or queue behind the direct PR workflow.
concurrency:
group: >-
${{ !inputs.reusable-invocation && github.event_name == 'pull_request' &&
- (github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name)) &&
format('docker-tests-direct-pr-{0}', github.event.pull_request.number) ||
!inputs.reusable-invocation && github.event_name == 'push' &&
github.ref == 'refs/heads/main' && 'docker-tests-direct-main' ||
format('docker-tests-isolated-run-{0}', github.run_id) }}
- cancel-in-progress: ${{ !inputs.reusable-invocation && github.event_name == 'pull_request' && (github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name)) }}
+ cancel-in-progress: ${{ !inputs.reusable-invocation && github.event_name == 'pull_request' }}
# Top-level permissions set to minimum (OSSF Scorecard Token-Permissions)
permissions: {}
@@ -60,8 +55,7 @@ env:
jobs:
# Detect which paths changed to conditionally run certain jobs
detect-changes:
- name: ${{ github.event_name == 'pull_request' && github.event.action == 'labeled' && !contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) && !inputs.reusable-invocation && 'Ignored label (change detection)' || 'detect-changes' }}
- if: github.event_name != 'pull_request' || github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) || inputs.reusable-invocation
+ name: detect-changes
runs-on: ubuntu-latest
outputs:
llm: ${{ steps.filter.outputs.llm }}
@@ -140,22 +134,19 @@ jobs:
- 'lighthouserc.json'
- '.github/workflows/docker-tests.yml'
- # Build once when a selected test needs the image. Unrequested PRs with no
- # Docker-based focused checks must not spend capacity building an unused image.
+ # Build the shared test image once per run. Every PR, main push and manual
+ # dispatch (on any branch) runs the full pytest suite, so the image is always
+ # needed; no label or path filter may suppress it (it is a required check in
+ # the repository ruleset). Without it a dispatch on a feature branch would
+ # skip pytest and still report success.
build-test-image:
runs-on: ubuntu-latest
- needs: detect-changes
- name: ${{ github.event_name == 'pull_request' && github.event.action == 'labeled' && !contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) && !inputs.reusable-invocation && 'Ignored label (image build)' || 'Build Test Image' }}
+ name: Build Test Image
if: |
- (github.event_name != 'pull_request' || github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) || inputs.reusable-invocation) &&
- (github.event_name != 'pull_request' || inputs.reusable-invocation ||
- (contains(github.event.pull_request.labels.*.name, 'test:pytest') ||
- contains(github.event.pull_request.labels.*.name, 'code-ready') ||
- contains(github.event.pull_request.labels.*.name, 'code-ready-preliminary') ||
- contains(github.event.pull_request.labels.*.name, 'auto-merge')) ||
- needs.detect-changes.outputs.llm == 'true' ||
- needs.detect-changes.outputs.infrastructure == 'true' ||
- needs.detect-changes.outputs.accessibility == 'true')
+ github.event_name == 'pull_request' ||
+ github.event_name == 'workflow_dispatch' ||
+ github.ref == 'refs/heads/main' ||
+ inputs.strict-mode == true
steps:
- name: Harden the runner (Audit all outbound calls)
@@ -188,17 +179,13 @@ jobs:
path: /tmp/ldr-test.tar
retention-days: 1
- # Keep the existing required check. An unrequested PR fails a cheap guard;
- # skipping this job for missing labels would satisfy branch protection.
- # !cancelled() bypasses the implicit success() dependency check, so a failed
- # or skipped image build still reaches the guard. Unlike always(), it lets
- # cancellation release an obsolete queued job without first needing a runner.
+ # Required check on every PR and main push. It carries no condition: if the
+ # image build fails or is cancelled this job skips, and the required
+ # "Build Test Image" check in the repository ruleset is what blocks merging
+ # (a skipped job reports success, so this job must not try to guard itself).
pytest-tests:
runs-on: ubuntu-latest
- name: ${{ github.event_name == 'pull_request' && github.event.action == 'labeled' && !contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) && !inputs.reusable-invocation && 'Ignored label (pytest)' || 'All Pytest Tests + Coverage' }}
- if: >-
- !cancelled() &&
- (github.event_name != 'pull_request' || github.event.action != 'labeled' || contains(fromJSON('["test:pytest", "code-ready", "code-ready-preliminary", "auto-merge"]'), github.event.label.name) || inputs.reusable-invocation)
+ name: All Pytest Tests + Coverage
# Cap exists so a hung run doesn't inherit GitHub's 360-minute default
# (observed 2026-06-10 on PR #4491: a flaked run sat in the main pytest
# step for 2+ hours before being cancelled manually). The original
@@ -221,28 +208,6 @@ jobs:
pull-requests: write # Needed for PR comments
steps:
- - name: Require a pytest request and successful image build
- id: pytest_request
- env:
- PYTEST_REQUESTED: >-
- ${{ github.event_name != 'pull_request' || inputs.reusable-invocation ||
- contains(github.event.pull_request.labels.*.name, 'test:pytest') ||
- contains(github.event.pull_request.labels.*.name, 'code-ready') ||
- contains(github.event.pull_request.labels.*.name, 'code-ready-preliminary') ||
- contains(github.event.pull_request.labels.*.name, 'auto-merge') }}
- TEST_IMAGE_RESULT: ${{ needs.build-test-image.result }}
- run: |
- set -euo pipefail
- if [ "$PYTEST_REQUESTED" != 'true' ]; then
- echo 'Full pytest has not been requested. Add test:pytest, code-ready, code-ready-preliminary, or auto-merge when this PR is ready for full validation.' >> "$GITHUB_STEP_SUMMARY"
- echo '::error::Full pytest is required before merging. Add test:pytest or a readiness label (code-ready, code-ready-preliminary, auto-merge) to run it.'
- exit 1
- fi
- if [ "$TEST_IMAGE_RESULT" != 'success' ]; then
- echo '::error::Full pytest cannot pass without a successful test image build.'
- exit 1
- fi
-
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
@@ -382,7 +347,7 @@ jobs:
# collected") and this step fails hard (cf. #4411). If the last
# marked module goes away, delete this step in the same change.
- name: Run fd_canary FD-leak tests (serial, no coverage)
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}
+ if: ${{ !cancelled() }}
run: |
docker run --rm -t \
-v "$PWD":/app \
@@ -461,7 +426,7 @@ jobs:
# parity; the news scheduler is only read in the FastAPI lifespan,
# which a pytest lane never enters, so this is a no-op here today.
- name: Run serial + nonroot (process-global state, permission-bit) tests
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}
+ if: ${{ !cancelled() }}
run: |
docker run --rm -t \
--user ldruser \
@@ -490,17 +455,17 @@ jobs:
--ignore=tests/fuzz"
- name: Run JavaScript infrastructure tests
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}
+ if: ${{ !cancelled() }}
run: |
cd tests/infrastructure_tests && npm test
- name: Run Vitest JS unit tests
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}
+ if: ${{ !cancelled() }}
run: |
npm ci && npm test
- name: Validate coverage data
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}
+ if: ${{ !cancelled() }}
run: |
docker run --rm \
-v "$PWD":/app \
@@ -510,7 +475,7 @@ jobs:
coverage/coverage.xml coverage/coverage-summary.json
- name: Generate coverage summary
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}
+ if: ${{ !cancelled() }}
run: |
{
echo "## Coverage Report"
@@ -536,7 +501,7 @@ jobs:
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload coverage reports
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' && hashFiles('coverage/**') != '' }}
+ if: ${{ !cancelled() && hashFiles('coverage/**') != '' }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: coverage-report
@@ -544,7 +509,7 @@ jobs:
# Fix permissions on coverage directory (Docker creates files as root)
- name: Fix coverage directory permissions
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' }}
+ if: ${{ !cancelled() }}
run: |
if [ -d "coverage" ]; then
sudo chown -R "$(id -u):$(id -g)" coverage/
@@ -564,7 +529,7 @@ jobs:
# Add coverage summary as PR comment for easy visibility
- name: Comment coverage on PR
- if: ${{ !cancelled() && steps.pytest_request.outcome == 'success' && github.event_name == 'pull_request' }}
+ if: ${{ !cancelled() && github.event_name == 'pull_request' }}
continue-on-error: true # Don't fail the job if GitHub API is throttled
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
@@ -745,7 +710,7 @@ jobs:
});
}
- # Publish only a complete, validated report from a direct main push or daily run.
+ # Publish only a complete, validated report from a direct main push.
# Release-gate's nested strict-mode invocation produces its own
# coverage artifact but must not race this job for gh-pages or the badge.
publish-coverage:
@@ -756,7 +721,7 @@ jobs:
${{
!cancelled() &&
needs.pytest-tests.result == 'success' &&
- (github.event_name == 'push' || github.event_name == 'schedule') &&
+ github.event_name == 'push' &&
github.ref == 'refs/heads/main' &&
!inputs.reusable-invocation &&
inputs.strict-mode != true
diff --git a/changelog.d/+always-run-full-pytest.misc.md b/changelog.d/+always-run-full-pytest.misc.md
new file mode 100644
index 0000000000..2f1271df1c
--- /dev/null
+++ b/changelog.d/+always-run-full-pytest.misc.md
@@ -0,0 +1 @@
+Full pytest and the shared test-image build run automatically on every pull request and main push again. The request labels (`test:pytest`, `code-ready`, `code-ready-preliminary`, `auto-merge`), the failing "add a label" guard, and the daily main run are gone: a red "All Pytest Tests + Coverage" check now always means the suite failed, never that validation was not requested. Superseded direct PR runs are cancelled automatically; main-queue and release validation behavior is unchanged.
diff --git a/changelog.d/+pytest-readiness-labels.misc.md b/changelog.d/+pytest-readiness-labels.misc.md
deleted file mode 100644
index a3dda14905..0000000000
--- a/changelog.d/+pytest-readiness-labels.misc.md
+++ /dev/null
@@ -1 +0,0 @@
-Full pytest now starts automatically for PRs labeled `code-ready`, `code-ready-preliminary`, or `auto-merge`, as well as `test:pytest`, and reruns on later pushes while any request label remains. Full validation remains required before merging; unrelated labels and existing release validation retain their behavior.
diff --git a/changelog.d/7101.misc.md b/changelog.d/7101.misc.md
deleted file mode 100644
index b20e17e1d7..0000000000
--- a/changelog.d/7101.misc.md
+++ /dev/null
@@ -1,5 +0,0 @@
-Full pytest for pull requests is requested with the `test:pytest` label and
-remains required before merge; unused test-image builds are skipped. Main-push,
-daily, and release validation still run automatically, with the main queue
-retaining one active run and only the newest pending run. Superseded PR pytest
-jobs can be cancelled while queued without waiting for a runner.
diff --git a/tests/ci/standalone/test_pytest_scheduling.py b/tests/ci/standalone/test_pytest_scheduling.py
index db17718120..7364706d33 100644
--- a/tests/ci/standalone/test_pytest_scheduling.py
+++ b/tests/ci/standalone/test_pytest_scheduling.py
@@ -1,21 +1,25 @@
-"""Exercise the actual workflow conditions and guard without app dependencies.
+"""Exercise the actual workflow conditions without app dependencies.
+
+Policy under test (revert of the #7101 label gate): the shared test image and
+the full pytest suite run unconditionally on every pull request and every
+main push. No label can request, defer, or suppress them; a red
+"All Pytest Tests + Coverage" check therefore means the suite failed, not
+that validation was never requested. Merge enforcement for a failed image
+build lives in the repository ruleset, which lists "Build Test Image" as a
+required check; this file can only verify the workflow side of that contract.
Actionlint validates YAML and Actions syntax separately. The small expression
interpreter below accepts only the operators/functions used by these conditions
-and rejects unknown syntax or context. Conditions also account for GitHub's
-implicit success() check. These are policy regressions, not a simulation of
-GitHub's scheduler or proof of a hosted run's final check conclusion.
+and rejects unknown syntax or context. These are policy regressions, not a
+simulation of GitHub's scheduler or proof of a hosted run's final check
+conclusion.
"""
import ast
import json
-import os
import re
-import subprocess
-import tempfile
import textwrap
import unittest
-from itertools import product
from pathlib import Path
@@ -25,17 +29,29 @@ TOKEN = re.compile(
r"\s*('(?:[^']|'')*'|&&|\|\||==|!=|!|[(),]|"
r"[A-Za-z_][A-Za-z0-9_.*-]*|[0-9]+)"
)
-REQUEST_LABELS = (
+# Every #7101 gating mechanism that must stay out of the workflow: the
+# request labels, the label-event special cases, and the failing guard.
+GATE_ARTIFACTS = (
"test:pytest",
"code-ready",
"code-ready-preliminary",
"auto-merge",
+ "github.event.label",
+ "github.event.action",
+ "PYTEST_REQUESTED",
+ "Full pytest is required before merging",
)
REQUIRED = {
"detect-changes": "detect-changes",
"build-test-image": "Build Test Image",
"pytest-tests": "All Pytest Tests + Coverage",
}
+FOCUSED_JOBS = (
+ ("llm-unit-tests", "llm"),
+ ("llm-example-tests", "llm"),
+ ("accessibility-tests", "accessibility"),
+ ("ldr-production-smoke-test", "docker"),
+)
def job(name):
@@ -70,14 +86,6 @@ def field(text, key, indent):
raise AssertionError(f"Missing field: {key}")
-def steps():
- return re.split(r"^ - ", job("pytest-tests"), flags=re.MULTILINE)[1:]
-
-
-def step_with_id(step_id):
- return next(step for step in steps() if f" id: {step_id}\n" in step)
-
-
def evaluate(expression, context, *, cancelled=False, success=True):
expression = expression.strip().removeprefix("${{").removesuffix("}}")
translated = []
@@ -160,433 +168,161 @@ def evaluate(expression, context, *, cancelled=False, success=True):
return visit(ast.parse(" ".join(translated).strip(), mode="eval").body)
-def evaluate_condition(expression, context, *, cancelled=False, success=True):
- # GitHub adds success() unless the condition uses a status function. Without
- # this rule a removed !cancelled() could silently skip the required guard
- # after a failed/skipped build while these tests still pass.
- if not re.search(
- r"\b(?:always|cancelled|failure|success)\s*\(", expression
- ):
- if not success:
- return False
- return evaluate(expression, context, cancelled=cancelled, success=success)
-
-
def context(
event="pull_request",
action="synchronize",
- label="",
*,
- requested=False,
+ ref=None,
reusable=False,
- labels=None,
+ strict=None,
):
+ if ref is None:
+ ref = (
+ "refs/pull/42/merge"
+ if event == "pull_request"
+ else "refs/heads/main"
+ )
+ if strict is None:
+ strict = reusable # workflow_call defaults strict-mode to true
return {
"github.event_name": event,
"github.event.action": action,
- "github.event.label.name": label,
- "github.event.pull_request.labels.*.name": (
- list(labels)
- if labels is not None
- else (["test:pytest"] if requested else [])
- ),
"github.event.pull_request.number": 42,
- "github.ref": "refs/pull/42/merge"
- if event == "pull_request"
- else "refs/heads/main",
+ "github.ref": ref,
"github.run_id": 123,
"inputs.reusable-invocation": reusable,
- "inputs.strict-mode": reusable,
+ "inputs.strict-mode": strict,
"needs.pytest-tests.result": "success",
+ "needs.build-test-image.result": "success",
"needs.detect-changes.outputs.llm": "false",
"needs.detect-changes.outputs.infrastructure": "false",
- "needs.detect-changes.outputs.accessibility": "false",
"needs.detect-changes.outputs.docker": "false",
- "steps.pytest_request.outcome": "success",
+ "needs.detect-changes.outputs.accessibility": "false",
}
class PytestSchedulingTests(unittest.TestCase):
- def test_pr_lifecycle_keeps_pytest_required_and_skips_unused_images(self):
- for action in ("opened", "synchronize", "reopened"):
- for requested in (False, True):
- ctx = context(action=action, requested=requested)
- with self.subTest(action=action, requested=requested):
- for name, required in REQUIRED.items():
- self.assertEqual(
- bool(evaluate(field(job(name), "if", 4), ctx)),
- name != "build-test-image" or requested,
- )
- self.assertEqual(
- evaluate(field(job(name), "name", 4), ctx), required
- )
- self.assertEqual(
- evaluate(
- field(job("pytest-tests"), "PYTEST_REQUESTED", 10),
- ctx,
- ),
- requested,
- )
-
- def test_only_request_labels_run_or_replace_required_checks(self):
- for label in (
- *REQUEST_LABELS,
- "bugfix",
- "test:notes",
- "test:ui-full-shards",
- ):
- for requested in (False, True):
- ctx = context(
- action="labeled", label=label, requested=requested
- )
- relevant = label in REQUEST_LABELS
- with self.subTest(label=label, requested=requested):
- for name, required in REQUIRED.items():
- self.assertEqual(
- bool(evaluate(field(job(name), "if", 4), ctx)),
- relevant
- and (name != "build-test-image" or requested),
- )
- self.assertEqual(
- evaluate(field(job(name), "name", 4), ctx)
- == required,
- relevant,
- )
+ def test_pr_trigger_excludes_label_events_and_daily_schedule(self):
+ self.assertEqual(
+ field(WORKFLOW, "types", 4),
+ "[opened, synchronize, reopened]",
+ )
+ self.assertNotRegex(WORKFLOW, r"(?m)^\s*schedule:")
- def test_each_readiness_label_requests_full_pytest_on_events_and_later_pushes(
- self,
- ):
- for label, action in product(
- REQUEST_LABELS, ("opened", "labeled", "synchronize", "reopened")
- ):
- ctx = context(action=action, label=label, labels=[label])
- with self.subTest(label=label, action=action):
- for name, required in REQUIRED.items():
- self.assertTrue(
- evaluate_condition(field(job(name), "if", 4), ctx)
- )
- self.assertEqual(
- evaluate(field(job(name), "name", 4), ctx), required
- )
- self.assertTrue(
- evaluate(
- field(job("pytest-tests"), "PYTEST_REQUESTED", 10), ctx
- )
- )
- self.assertTrue(
- evaluate(field(WORKFLOW, "cancel-in-progress", 2), ctx)
- )
- self.assertEqual(
- evaluate(field(WORKFLOW, "group", 2), ctx),
- evaluate(field(WORKFLOW, "group", 2), context()),
- )
+ def test_no_label_gate_remains_in_the_workflow(self):
+ for artifact in GATE_ARTIFACTS:
+ with self.subTest(artifact=artifact):
+ self.assertNotIn(artifact, WORKFLOW)
- def test_unrelated_labels_do_not_interrupt_a_readiness_requested_run(self):
- for requested_label, event_label in product(
- REQUEST_LABELS,
- (
- "bugfix",
- "python",
- "security",
- "tests",
- "test:notes",
- "test:ui-full-shards",
- ),
- ):
- ctx = context(
- action="labeled",
- label=event_label,
- labels=[requested_label, event_label],
- )
+ def test_required_checks_run_on_every_pr_and_main_push(self):
+ contexts = [
+ context(action=a) for a in ("opened", "synchronize", "reopened")
+ ]
+ contexts += [
+ context("push"),
+ context(reusable=True),
+ context("workflow_dispatch"),
+ ]
+ for ctx in contexts:
with self.subTest(
- requested_label=requested_label, event_label=event_label
+ event=ctx["github.event_name"],
+ action=ctx["github.event.action"],
+ reusable=ctx["inputs.reusable-invocation"],
):
- for name, required in REQUIRED.items():
- self.assertFalse(
- evaluate_condition(field(job(name), "if", 4), ctx)
- )
- self.assertNotEqual(
- evaluate(field(job(name), "name", 4), ctx), required
- )
- self.assertFalse(
- evaluate(field(WORKFLOW, "cancel-in-progress", 2), ctx)
- )
- self.assertNotEqual(
- evaluate(field(WORKFLOW, "group", 2), ctx),
- evaluate(field(WORKFLOW, "group", 2), context()),
- )
-
- def test_readiness_requests_still_fail_closed_and_allow_cancellation(self):
- for label, result in product(
- REQUEST_LABELS, ("failure", "skipped", "cancelled")
- ):
- ctx = context(action="labeled", label=label, labels=[label])
- ctx["needs.build-test-image.result"] = result
- with self.subTest(label=label, image_result=result):
+ for job_id, check in REQUIRED.items():
+ self.assertEqual(field(job(job_id), "name", 4), check)
self.assertTrue(
- evaluate_condition(
- field(job("pytest-tests"), "if", 4), ctx, success=False
- )
- )
- self.assertFalse(
- evaluate_condition(
- field(job("pytest-tests"), "if", 4),
- ctx,
- cancelled=True,
- success=False,
- )
+ evaluate(field(job("build-test-image"), "if", 4), ctx)
)
-
- def test_main_push_runs_full_pytest_without_a_pr_label(self):
- ctx = context("push")
+ # A manual dispatch on a non-main branch also builds the image, so its
+ # pytest runs instead of skipping (a skipped job reports success); a
+ # push to any other branch does not trigger the workflow.
self.assertTrue(
- evaluate(field(job("pytest-tests"), "PYTEST_REQUESTED", 10), ctx)
- )
- for name, required in REQUIRED.items():
- self.assertTrue(evaluate(field(job(name), "if", 4), ctx))
- self.assertEqual(
- evaluate(field(job(name), "name", 4), ctx), required
+ evaluate(
+ field(job("build-test-image"), "if", 4),
+ context("workflow_dispatch", ref="refs/heads/feature"),
)
-
- def test_image_is_built_exactly_when_pytest_or_a_focused_consumer_needs_it(
- self,
- ):
- job_names = re.findall(
- r"^ ([\w-]+):$", WORKFLOW.split("\njobs:\n", 1)[1], re.MULTILINE
)
- focused_consumers = [
- name
- for name in job_names
- if name != "pytest-tests"
- and re.search(
- r"^ needs:.*\bbuild-test-image\b", job(name), re.MULTILINE
- )
- ]
- self.assertTrue(focused_consumers)
+
+ def test_detect_changes_and_pytest_carry_no_condition(self):
+ # The pre-#7101 shape: these jobs run whenever their dependencies
+ # allow. A reintroduced `if:` here is how the #7101 gate came back.
+ for job_id in ("detect-changes", "pytest-tests"):
+ with self.subTest(job=job_id), self.assertRaises(AssertionError):
+ field(job(job_id), "if", 4)
+
+ def test_pytest_depends_on_the_image_build_without_a_request_guard(self):
self.assertEqual(
- field(job("build-test-image"), "needs", 4), "detect-changes"
+ field(job("pytest-tests"), "needs", 4), "build-test-image"
)
- for requested, llm, infrastructure, accessibility, docker in product(
- (False, True), repeat=5
- ):
- ctx = context(requested=requested)
- for path, changed in (
- ("llm", llm),
- ("infrastructure", infrastructure),
- ("accessibility", accessibility),
- ("docker", docker),
- ):
- ctx[f"needs.detect-changes.outputs.{path}"] = str(
- changed
- ).lower()
- selected = any(
- evaluate(field(job(name), "if", 4), ctx)
- for name in focused_consumers
- )
- with self.subTest(
- requested=requested,
- llm=llm,
- infrastructure=infrastructure,
- accessibility=accessibility,
- docker=docker,
- ):
- self.assertEqual(
- bool(
- evaluate(field(job("build-test-image"), "if", 4), ctx)
- ),
- requested or selected,
- )
-
- def test_daily_manual_and_release_callers_run_full_pytest(self):
- cases = [context("schedule"), context("workflow_dispatch")]
- cases += [
- context(
- event,
- action="labeled",
- label="test:ui-full-shards",
- reusable=True,
- )
- for event in (
- "push",
- "pull_request",
- "workflow_dispatch",
- "schedule",
- )
- ]
- for ctx in cases:
- with self.subTest(context=ctx):
- self.assertTrue(
- evaluate(field(job("build-test-image"), "if", 4), ctx)
- )
- self.assertTrue(
- evaluate(field(job("pytest-tests"), "if", 4), ctx)
- )
- self.assertTrue(
- evaluate(
- field(job("pytest-tests"), "PYTEST_REQUESTED", 10), ctx
- )
- )
- self.assertEqual(
- evaluate(field(job("pytest-tests"), "name", 4), ctx),
- REQUIRED["pytest-tests"],
+ self.assertNotIn("pytest_request", job("pytest-tests"))
+ self.assertNotIn("PYTEST_REQUESTED", job("pytest-tests"))
+
+ def test_image_build_is_unconditional_and_path_filter_free(self):
+ build = job("build-test-image")
+ with self.assertRaises(AssertionError):
+ field(build, "needs", 4)
+ self.assertNotIn("detect-changes", build)
+ self.assertNotIn("labels", build)
+
+ def test_focused_checks_keep_their_path_filters(self):
+ # Path filters still gate the focused suites themselves; only the
+ # shared image build stopped consulting them.
+ for job_id, output in FOCUSED_JOBS:
+ with self.subTest(job=job_id):
+ condition = field(job(job_id), "if", 4)
+ self.assertIn(
+ f"needs.detect-changes.outputs.{output}", condition
)
- self.assertRegex(
- WORKFLOW, r"schedule:\s*\n\s*#.*\n\s*- cron: '23 3 \* \* \*'"
- )
- release_caller = (ROOT / ".github/workflows/ci-gate.yml").read_text()
- self.assertRegex(
- release_caller,
- r"uses: \$/\.github/workflows/docker-tests.yml\s+with:\s+strict-mode: true",
- )
- def test_concurrency_cancels_only_relevant_direct_pr_runs(self):
+ def test_concurrency_cancels_superseded_direct_pr_runs_only(self):
group = field(WORKFLOW, "group", 2)
cancel = field(WORKFLOW, "cancel-in-progress", 2)
- direct = context()
- label = context(action="labeled", label="test:pytest", requested=True)
- self.assertEqual(evaluate(group, direct), evaluate(group, label))
- for ctx in (direct, label):
+ direct = [
+ context(action=a) for a in ("opened", "synchronize", "reopened")
+ ]
+ self.assertEqual(len({evaluate(group, ctx) for ctx in direct}), 1)
+ for ctx in direct:
self.assertTrue(evaluate(cancel, ctx))
+ push, newer = context("push"), {**context("push"), "github.run_id": 456}
+ self.assertEqual(evaluate(group, push), evaluate(group, newer))
+ self.assertFalse(evaluate(cancel, push))
for ctx in (
- context(action="labeled", label="bugfix"),
- context("schedule"),
+ context("workflow_dispatch", ref="refs/heads/feature"),
context(reusable=True),
context("push", reusable=True),
):
self.assertFalse(evaluate(cancel, ctx))
- self.assertNotEqual(evaluate(group, ctx), evaluate(group, direct))
self.assertNotEqual(
- evaluate(group, ctx), evaluate(group, context("push"))
+ evaluate(group, ctx), evaluate(group, direct[0])
)
-
- def test_main_pushes_keep_active_run_and_share_single_pending_slot(self):
- group = field(WORKFLOW, "group", 2)
- cancel = field(WORKFLOW, "cancel-in-progress", 2)
- first, latest = context("push"), context("push")
- latest["github.run_id"] = 456
- self.assertEqual(evaluate(group, first), evaluate(group, latest))
- for ctx in (first, latest):
- self.assertFalse(evaluate(cancel, ctx))
- # GitHub retains just the newest pending run by default. Do not opt
- # into queue: max, which would restore the main-push backlog.
+ self.assertNotEqual(evaluate(group, ctx), evaluate(group, push))
+ # Main keeps GitHub's default single-pending-slot behavior; queueing
+ # every push would restore the main-push backlog.
concurrency = WORKFLOW.split("\nconcurrency:\n", 1)[1].split(
"\npermissions:", 1
)[0]
- self.assertNotRegex(concurrency, r"\bqueue:\s*max\b")
+ self.assertNotRegex(concurrency, r"\bqueue:")
- def test_only_successful_direct_main_runs_publish_coverage(self):
+ def test_publish_coverage_only_from_a_successful_direct_main_push(self):
condition = field(job("publish-coverage"), "if", 4)
- for event in ("push", "pull_request", "schedule", "workflow_dispatch"):
- for reusable in (False, True):
- for result in ("success", "failure", "cancelled", "skipped"):
- ctx = context(event, reusable=reusable)
- ctx["needs.pytest-tests.result"] = result
- expected = (
- event in {"push", "schedule"}
- and not reusable
- and result == "success"
- )
- self.assertEqual(bool(evaluate(condition, ctx)), expected)
- self.assertFalse(evaluate(condition, ctx, cancelled=True))
-
- def test_guard_fails_closed_before_checkout_or_testing(self):
- pytest = job("pytest-tests")
- guard = step_with_id("pytest_request")
- script = field(guard, "run", 8)
- self.assertNotIn("continue-on-error", guard)
- self.assertLess(
- pytest.index("id: pytest_request"),
- pytest.index("actions/checkout@"),
- )
- for requested in ("true", "false", "", "unexpected"):
- for image_result in (
- "success",
- "failure",
- "cancelled",
- "skipped",
- "",
- ):
- with (
- self.subTest(
- requested=requested, image_result=image_result
- ),
- tempfile.TemporaryDirectory() as temp,
- ):
- result = subprocess.run(
- ["bash", "-c", script],
- env={
- **os.environ,
- "PYTEST_REQUESTED": requested,
- "TEST_IMAGE_RESULT": image_result,
- "GITHUB_STEP_SUMMARY": str(Path(temp) / "summary"),
- },
- capture_output=True,
- text=True,
- check=False,
- )
- self.assertEqual(
- result.returncode == 0,
- requested == "true" and image_result == "success",
- )
- if requested == "false":
- self.assertIn("test:pytest", result.stdout)
-
- def test_unsuccessful_image_build_cannot_skip_required_guard(self):
- condition = field(job("pytest-tests"), "if", 4)
- for requested, image_result in product(
- (False, True), ("failure", "skipped", "cancelled")
- ):
- ctx = context(requested=requested)
- ctx["needs.build-test-image.result"] = image_result
- with self.subTest(requested=requested, image_result=image_result):
- self.assertTrue(
- evaluate_condition(condition, ctx, success=False)
- )
-
- def test_cancellation_releases_pytest_without_waiting_for_a_runner(self):
- condition = field(job("pytest-tests"), "if", 4)
- cases = [
- context(),
- context(requested=True),
- context(action="labeled", label="test:pytest", requested=True),
- context("push"),
- context("schedule"),
- context("workflow_dispatch"),
+ self.assertNotIn("schedule", condition)
+ self.assertTrue(evaluate(condition, context("push")))
+ for ctx in (
+ {**context("push"), "needs.pytest-tests.result": "failure"},
+ context("pull_request"),
context(reusable=True),
- ]
- for ctx, image_result in product(
- cases, ("success", "failure", "cancelled", "skipped")
+ context("workflow_dispatch", ref="refs/heads/feature"),
):
- ctx = {**ctx, "needs.build-test-image.result": image_result}
with self.subTest(context=ctx):
- # Cancellation is evaluated at the job, before a runner can
- # execute any step. A cancelled() shell guard cannot release
- # an obsolete queued job from its workflow concurrency group.
- self.assertFalse(
- evaluate_condition(
- condition,
- ctx,
- cancelled=True,
- success=image_result == "success",
- )
- )
-
- def test_failure_cleanup_stops_after_guard_failure_or_cancellation(self):
- guarded_steps = 0
- for step in steps():
- if " if:" not in step:
- continue # GitHub's default success() already stops these.
- condition = field(step, "if", 8)
- guarded_steps += 1
- for result in ("failure", "cancelled", "skipped"):
- ctx = context(requested=False)
- ctx["steps.pytest_request.outcome"] = result
self.assertFalse(evaluate(condition, ctx))
- self.assertTrue(evaluate(condition, context(requested=True)))
- self.assertFalse(
- evaluate(condition, context(requested=True), cancelled=True)
- )
- self.assertGreater(guarded_steps, 0)
+
+ def test_release_pipeline_still_calls_the_strict_suite(self):
+ release_caller = (ROOT / ".github/workflows/ci-gate.yml").read_text()
+ self.assertRegex(
+ release_caller,
+ r"uses: \$/\.github/workflows/docker-tests.yml\s+with:\s+strict-mode: true",
+ )
if __name__ == "__main__":