Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

Shard the Python 3.13 backend pytest leg three ways (#10920)

Daniel Han80181969801d
diff --git a/.github/workflows/studio-backend-ci.yml b/.github/workflows/studio-backend-ci.yml
index 43c30625f3d..655b064f63a 100644
--- a/.github/workflows/studio-backend-ci.yml
+++ b/.github/workflows/studio-backend-ci.yml
@@ -92,7 +92,7 @@ env:
jobs:
pytest:
- name: (Python ${{ matrix.python }})
+ name: (Python ${{ matrix.python }}, ${{ matrix.shard }})
runs-on: ubuntu-24.04
# 45 guards against a hang; it is not a performance budget. The comment this replaces
# claimed 14.1 to 15.0 minutes against a limit of 30, but this step now measures 22.5 to
@@ -114,8 +114,16 @@ jobs:
# its work in six minutes. The per-test --timeout below is what actually names a hang;
# this stays as the backstop for a wedge the per-test timer cannot see, such as one in
# collection or in a worker that never comes up.
+ #
+ # Unchanged by the split below, deliberately. A shard does less work than the leg it
+ # came from, so 45 only gets looser, and retuning it on a prediction is the failure
+ # this number was raised to fix: the sharded spread has not been measured on a runner
+ # yet. Retune it once there are runs to retune it against.
timeout-minutes: 45
strategy:
+ # fail-fast off: three shards of one suite, and cancelling the other two on the first
+ # failure would hide which of them were also red. It was already off for the
+ # 3.13/3.11 pair, for the same reason.
fail-fast: false
matrix:
# 3.13 only, on pull requests and on main alike.
@@ -138,22 +146,81 @@ jobs:
# away: two interpreters that both accept a line can still behave differently on
# it, and a sys.version_info branch is only ever parsed, never taken.
#
- # So the branches were counted rather than assumed. Seven files in the backend
- # carry one, at 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by
- # the old matrix, whose oldest leg was 3.10, so every leg took the same side of
- # them and removing legs loses nothing there. 3.14 is above every leg there has
- # ever been. What is genuinely lost is the PRE-3.12 side of three files, and that
- # is small enough to keep executing directly.
+ # So the branches were counted rather than assumed. Seven backend files carry one, at
+ # 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by the old matrix
+ # (oldest leg 3.10), and 3.14 is above every leg there has ever been. What is
+ # genuinely lost is the PRE-3.12 side of three files.
+ #
+ # Hence the second entry: 3.11, the newest version still on that side, running those
+ # three files and nothing else. Roughly 40 tests in under ten seconds, beside the full
+ # leg rather than in front of it, so the critical path is unchanged.
+ #
+ # The 3.13 leg is three shards of one selection, balanced on measured time rather than
+ # file count, for the reason repo-cpu-tests was split below: it is now the longest
+ # thing a pull request waits for, at 763-822s parallel plus 201-234s serial.
+ #
+ # Summed test time is 1886.1s across 922 files (`--durations=0 --durations-min=0`,
+ # -n 16; the default 0.005s cutoff hides ~105,000 durations and is unusable here),
+ # split on the first letter after `test_`:
+ #
+ # a-k 542.8s 396 files 28.8%
+ # l-r 693.7s 318 files 36.8%
+ # s-z + every subdirectory 649.6s 208 files 34.4%
#
- # Hence the second entry: 3.11, the newest version that still takes that side,
- # running those three files and nothing else. Roughly 40 tests in under ten
- # seconds, not a second copy of the suite, and it runs beside the full leg rather
- # than in front of it, so the critical path is unchanged.
+ # so the longest shard is 1.103x an ideal third. The letters are lumpy (l is 217.2s,
+ # t is 280.4s, multi_account's 159.3s is stuck in the catch-all) and no contiguous
+ # split does better. A hand-picked letter SET reaches 1.001x and is not worth it: an
+ # allowlist tuned to one measurement is what the shape below exists to avoid.
+ #
+ # The shape. All three shards root at `tests/` and differ only in what they ignore:
+ # `--ignore=FILE` does NOT filter a file passed as an explicit argument (checked
+ # against pytest), so a shard rooted at an expanded glob would drag the twelve serial
+ # files back in past the shared --ignore list below. Shards 1 and 2 name their range
+ # and exclude everything else, including names outside a-z and anything below a
+ # subdirectory. Shard 3 is "tests/ minus the other two ranges", never an allowlist, so
+ # a new file lands in exactly one shard with no workflow edit: test_apple.py in 1,
+ # test_monkey.py in 2, and test_snake.py, tests/a_new_dir/test_x.py, foo_test.py and
+ # test_Zebra.py in 3. test_ci_backend_pytest_shards.py fails if that stops holding,
+ # and test_backend_ci_parallel_isolation.py fails if a shard collects one of the twelve.
+ #
+ # Two name shapes the split cannot absorb, forbidden by name in that guard rather than
+ # left to be discovered: a subdirectory called test_*, and a file matching BOTH
+ # discovery patterns (test_api_test.py). Each lands in no shard, because fnmatch's *
+ # crosses / and the suffix rule cannot say "unless it starts with test_" in one pattern.
+ # Diverging per prefix character was tried and reverted: it fixes test_api_test.py but
+ # over-consumes te_test.py into BOTH ranged shards, and completing it needs nine globs
+ # per shard for a filename this repo does not use.
+ #
+ # Verified at the level of the test id, not the count: the unsharded selection and the
+ # three shards run with --junitxml and the reports diffed gave 13,073 + 17,483 + 12,053
+ # = 42,609 against the baseline's 42,609, no id in two shards or in neither, and the
+ # same 332 skipped and 386 errored.
include:
- python: '3.13'
scope: full
+ shard: a-k
+ selection: >-
+ tests/
+ --ignore-glob=tests/*/*
+ --ignore-glob=tests/*_test.py
+ --ignore-glob=tests/test_[!a-k]*.py
+ - python: '3.13'
+ scope: full
+ shard: l-r
+ selection: >-
+ tests/
+ --ignore-glob=tests/*/*
+ --ignore-glob=tests/*_test.py
+ --ignore-glob=tests/test_[!l-r]*.py
+ - python: '3.13'
+ scope: full
+ shard: rest
+ selection: >-
+ tests/
+ --ignore-glob=tests/test_[a-r]*.py
- python: '3.11'
scope: floor-spot-check
+ shard: floor
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
@@ -284,9 +351,20 @@ jobs:
# Reproduced with a test forking a child that never exits: unwrapped it hangs
# silently, wrapped it exits 124 through execnet safe_terminate -> workerpool.waitall.
# Expect a stuck FRAME, not always a test name. 1500s is ~3x the 8-10 minute norm.
+ #
+ # The thirteen --ignore flags and the -k filter stay HERE rather than moving into
+ # the matrix: they are the selection every shard inherits, and one copy of them is
+ # what makes "the twelve serial files are ignored by every parallel shard" a
+ # property of the step instead of three lists that have to agree. The shards differ
+ # only in `${{ matrix.selection }}`.
+ #
+ # The outer 1500s is not retuned on a prediction, for the reason repo-cpu-tests
+ # gives below: a bound that lands inside the real spread is the failure this kind of
+ # number exists to prevent, and the sharded spread has not been measured on a runner
+ # yet. It only ever gets looser as the work per shard drops.
run: |
timeout --signal=INT --kill-after=60 1500 \
- python -m pytest tests/ -q --tb=short -n 4 --timeout=330 \
+ python -m pytest ${{ matrix.selection }} -q --tb=short -n 4 --timeout=330 \
--ignore=tests/test_studio_api.py \
--ignore=tests/test_streaming_stripper.py \
--ignore=tests/test_llama_cpp_wait_for_vram_settle.py \
@@ -303,7 +381,12 @@ jobs:
-k 'not llama_cpp_load_progress_live and not TestGpuAutoSelection and not TestPreSpawnGpuResolution and not TestPerGpuFitGuardAllCounts and not TestTransformersIntrospection and not test_returns_cuda_when_cuda_available and not test_calls_cuda_cache_when_cuda'
- name: Backend tests that cannot share a worker
- if: matrix.scope == 'full'
+ # One shard, one process, the same twelve files. Not split and not redistributed:
+ # whether any of them could go back into the parallel run was asked separately and
+ # the answer was no, so this step is carried unchanged and only has to be carried
+ # ONCE. a-k because it is the shortest shard at 28.8% of summed test time, and this
+ # step is 201-234s: it lands there rather than on the 36.8% shard.
+ if: matrix.scope == 'full' && matrix.shard == 'a-k'
working-directory: studio/backend
# Relative timing against a reference measured in the same process. Serial, so the
# comparison is between two implementations rather than between two schedulings.
diff --git a/studio/backend/tests/test_native_audio.py b/studio/backend/tests/test_native_audio.py
index b86f3bd6471..d2617678aad 100644
--- a/studio/backend/tests/test_native_audio.py
+++ b/studio/backend/tests/test_native_audio.py
@@ -13,6 +13,46 @@ from types import SimpleNamespace
import pytest
import torch
+import importlib
+import types
+from unittest.mock import MagicMock
+
+
+# This file reaches core/inference/inference.py, which imports unsloth at module scope, from
+# inside a helper, so test_backend_tests_stub_heavy_imports.py's source scan never saw it. The
+# import only ever worked when another file on the same xdist worker had already stubbed
+# unsloth; sharding changed who shares a worker, so the stub is explicit here now.
+_STUBBED: list[str] = []
+
+
+def _stub_if_missing(name, attrs):
+ """Stub a dep the backend pytest matrix does not install, as test_trainer_stdout_quiet.py does.
+
+ That matrix stops at studio.txt plus torch and transformers; repo-cpu-tests is the job that
+ installs unsloth_zoo. Unstubbed, this module fails COLLECTION and takes the job down. A real
+ install is left alone, and __spec__ = None keeps the namespace-shadow guard a no-op here.
+ """
+ if name in sys.modules:
+ return
+ try:
+ importlib.import_module(name)
+ return
+ except Exception: # noqa: BLE001 - unusable here either way, so stub it
+ pass
+ _STUBBED.append(name)
+ mod = types.ModuleType(name)
+ mod.__spec__ = None
+ for attr in attrs:
+ setattr(mod, attr, MagicMock())
+ sys.modules[name] = mod
+ parent, _, child = name.rpartition(".")
+ if parent and parent in sys.modules:
+ setattr(sys.modules[parent], child, mod)
+
+
+_stub_if_missing("unsloth", ("FastLanguageModel", "FastVisionModel", "is_bfloat16_supported"))
+_stub_if_missing("unsloth.chat_templates", ("get_chat_template",))
+
from core.inference.native_audio import (
HIGGS_TTS2_CODEC_REPO,
HIGGS_TTS3_CODEC_REPO,
@@ -28,6 +68,16 @@ from core.inference.native_audio import (
native_audio_type_from_local_path,
)
+# Bind the dependency while the stubs stand, then drop them, as test_trainer_stdout_quiet.py
+# does. Left in place they outlive this module, and a later file's _stub_if_missing returns
+# before recording ownership, so nobody can clean them up. Concretely, _shared_policy branches
+# on `"unsloth" in sys.modules` and returns None off the stub instead of reaching its disk
+# fallback. A real install stubs nothing, so this is a no-op there.
+import core.inference.inference # noqa: F401,E402 - imported to bind it under the stubs
+
+for _name in reversed(_STUBBED):
+ sys.modules.pop(_name, None)
+
def _backend(audio_type: str, **entry):
backend = NativeAudioBackend.__new__(NativeAudioBackend)
diff --git a/studio/backend/tests/test_native_tool_token_provenance.py b/studio/backend/tests/test_native_tool_token_provenance.py
index 6959828139f..630fb661d21 100644
--- a/studio/backend/tests/test_native_tool_token_provenance.py
+++ b/studio/backend/tests/test_native_tool_token_provenance.py
@@ -8,6 +8,47 @@ import json
import pytest
import torch
+import importlib
+import sys
+import types
+from unittest.mock import MagicMock
+
+
+# This file reaches core/inference/inference.py, which imports unsloth at module scope, from
+# inside a helper, so test_backend_tests_stub_heavy_imports.py's source scan never saw it. The
+# import only ever worked when another file on the same xdist worker had already stubbed
+# unsloth; sharding changed who shares a worker, so the stub is explicit here now.
+_STUBBED: list[str] = []
+
+
+def _stub_if_missing(name, attrs):
+ """Stub a dep the backend pytest matrix does not install, as test_trainer_stdout_quiet.py does.
+
+ That matrix stops at studio.txt plus torch and transformers; repo-cpu-tests is the job that
+ installs unsloth_zoo. Unstubbed, this module fails COLLECTION and takes the job down. A real
+ install is left alone, and __spec__ = None keeps the namespace-shadow guard a no-op here.
+ """
+ if name in sys.modules:
+ return
+ try:
+ importlib.import_module(name)
+ return
+ except Exception: # noqa: BLE001 - unusable here either way, so stub it
+ pass
+ _STUBBED.append(name)
+ mod = types.ModuleType(name)
+ mod.__spec__ = None
+ for attr in attrs:
+ setattr(mod, attr, MagicMock())
+ sys.modules[name] = mod
+ parent, _, child = name.rpartition(".")
+ if parent and parent in sys.modules:
+ setattr(sys.modules[parent], child, mod)
+
+
+_stub_if_missing("unsloth", ("FastLanguageModel", "FastVisionModel", "is_bfloat16_supported"))
+_stub_if_missing("unsloth.chat_templates", ("get_chat_template",))
+
from core.inference.native_tool_tokens import (
NATIVE_TOOL_CONTROL_TOKENS,
NativeToolTokenDecoder,
@@ -18,6 +59,16 @@ from core.inference.native_tool_tokens import (
from core.inference.safetensors_agentic import run_safetensors_tool_loop
from core.inference.tool_call_parser import parse_tool_calls_from_text
+# Bind the dependency while the stubs stand, then drop them, as test_trainer_stdout_quiet.py
+# does. Left in place they outlive this module, and a later file's _stub_if_missing returns
+# before recording ownership, so nobody can clean them up. Concretely, _shared_policy branches
+# on `"unsloth" in sys.modules` and returns None off the stub instead of reaching its disk
+# fallback. A real install stubs nothing, so this is a no-op there.
+import core.inference.inference # noqa: F401,E402 - imported to bind it under the stubs
+
+for _name in reversed(_STUBBED):
+ sys.modules.pop(_name, None)
+
class _GemmaTokenDecoder:
_pieces = {
diff --git a/tests/studio/test_backend_ci_matrix.py b/tests/studio/test_backend_ci_matrix.py
index 89471757ed6..5b2d39fdd9d 100644
--- a/tests/studio/test_backend_ci_matrix.py
+++ b/tests/studio/test_backend_ci_matrix.py
@@ -47,13 +47,25 @@ CEILING = "3.13"
def _legs() -> dict[str, str]:
- """Each leg the matrix runs, as scope -> interpreter."""
+ """Each scope the matrix runs, as scope -> interpreter.
+
+ A scope maps to one interpreter but no longer to one entry: `full` is three shards of
+ the same suite on the same 3.13, split because the leg had become the longest thing a
+ pull request waits for. What this has to keep unambiguous is the SCOPE-to-interpreter
+ mapping the tests below ask about, so that is what is asserted -- counting entries
+ would now be counting shards, which is a different question and not this file's.
+ """
document = yaml.safe_load(WORKFLOW.read_text(encoding = "utf-8"))
matrix = document["jobs"]["pytest"]["strategy"]["matrix"]
entries = matrix.get("include")
assert entries, f"the matrix no longer lists its legs by scope: {matrix!r}"
- legs = {str(entry["scope"]): str(entry["python"]) for entry in entries}
- assert len(legs) == len(entries), f"two legs share a scope: {entries!r}"
+ legs = {}
+ for entry in entries:
+ scope, python = str(entry["scope"]), str(entry["python"])
+ assert legs.setdefault(scope, python) == python, (
+ f"scope {scope!r} runs on more than one interpreter, so scope no longer says "
+ f"which one a leg is: {entries!r}"
+ )
return legs
diff --git a/tests/studio/test_backend_ci_parallel_isolation.py b/tests/studio/test_backend_ci_parallel_isolation.py
index a0af9b3568f..3a3a7edc750 100644
--- a/tests/studio/test_backend_ci_parallel_isolation.py
+++ b/tests/studio/test_backend_ci_parallel_isolation.py
@@ -22,6 +22,7 @@ ignore appears without a step that runs the same path, or the other way round.
"""
import ast
+import fnmatch
import importlib.util
import re
from pathlib import Path
@@ -41,51 +42,80 @@ ISOLATED = [
]
-def _pytest_commands(text: str) -> list[str]:
- """Every `python -m pytest ...` invocation in the workflow, line joins resolved.
+def _jobs() -> dict:
+ return yaml.safe_load(WORKFLOW.read_text(encoding = "utf-8"))["jobs"]
- Read off the raw text rather than the parsed YAML: a `run:` block is one
- scalar and the interesting structure is inside it, so parsing buys nothing and
- would make this depend on the job/step layout instead of the commands.
- The one thing that cannot be read off the text is `${{ matrix.selection }}`: since
- repo-cpu-tests became three shards, its parallel command is a template and its paths
- live in the matrix. Those are substituted in, one command per shard, so a scan of the
- commands is still a scan of what actually runs.
+def _selections(job_name: str) -> list[str]:
+ """The `selection` of each matrix entry of one job, whitespace-normalised.
+
+ Not every entry has one: the `pytest` job's 3.11 floor-spot-check leg names three
+ files directly and is not a shard of anything, so it carries no selection.
"""
- joined = re.sub(r"\\\s*\n\s*", " ", text)
- commands = [
- line.strip()
- for line in joined.splitlines()
- if "python -m pytest" in line and not line.lstrip().startswith("#")
- ]
- expanded = []
- for command in commands:
- if "${{ matrix.selection }}" not in command:
- expanded.append(command)
- continue
- for selection in _shard_selections():
- expanded.append(command.replace("${{ matrix.selection }}", selection))
- return expanded
+ job = _jobs()[job_name]
+ include = job.get("strategy", {}).get("matrix", {}).get("include", [])
+ return [" ".join(entry["selection"].split()) for entry in include if "selection" in entry]
+
+
+def _commands_in(job_name: str) -> list[str]:
+ """Every `python -m pytest ...` invocation of one job, line joins and matrix resolved.
+
+ Read off the raw text of each `run:` scalar rather than off more parsed YAML: a
+ `run:` block is one scalar and the interesting structure is inside it, so parsing
+ further buys nothing and would make this depend on step layout instead of commands.
+
+ The one thing that cannot be read off the text is `${{ matrix.selection }}`, because
+ the paths live in the matrix rather than in the command. BOTH parallel jobs are
+ sharded now and both write their step that way, so the substitution has to know whose
+ matrix to read: expanding a command with the other job's selections would build runs
+ that are not in the workflow and ask the isolation questions below of those instead.
+ Hence per job, which is also how the caller knows the owner without guessing from a
+ flag that happens to appear in one of them.
+ """
+ job = _jobs()[job_name]
+ selections = _selections(job_name)
+ commands = []
+ for step in job.get("steps", []):
+ joined = re.sub(r"\\\s*\n\s*", " ", str(step.get("run", "")))
+ for line in joined.splitlines():
+ line = line.strip()
+ if "python -m pytest" not in line or line.startswith("#"):
+ continue
+ if "${{ matrix.selection }}" in line:
+ commands.extend(
+ line.replace("${{ matrix.selection }}", selection) for selection in selections
+ )
+ else:
+ commands.append(line)
+ return commands
-def _shard_selections() -> list[str]:
- job = yaml.safe_load(WORKFLOW.read_text(encoding = "utf-8"))["jobs"]["repo-cpu-tests"]
- return [" ".join(entry["selection"].split()) for entry in job["strategy"]["matrix"]["include"]]
+def _pytest_commands() -> list[str]:
+ """Every pytest invocation in the workflow, across every job."""
+ return [command for job_name in _jobs() for command in _commands_in(job_name)]
def _collects(command: str, path: str) -> bool:
- """Whether a pytest command would collect `path`, by its roots and its --ignore flags.
+ """Whether a pytest command would collect `path`, by its roots and its ignore flags.
An isolated path used to be kept out of the parallel run by naming it in an --ignore.
A shard that does not name its directory at all keeps it out just as effectively, so
- the question the guard asks is whether the run reaches the path, not how."""
+ the question the guard asks is whether the run reaches the path, not how.
+
+ --ignore-glob is read as well as --ignore, because the backend shards are told apart
+ by nothing else: all three root at `tests/` and differ only in which glob they
+ exclude. Treating those flags as noise would have every backend shard appear to
+ collect every backend file, and "not collected by any parallel run" would then be
+ unfalsifiable for the whole suite.
+ """
tokens = command.split()
- roots, ignores = [], []
+ roots, ignores, globs = [], [], []
index = 0
while index < len(tokens):
token = tokens[index]
- if token.startswith("--ignore="):
+ if token.startswith("--ignore-glob="):
+ globs.append(token.split("=", 1)[1].strip("'\""))
+ elif token.startswith("--ignore="):
ignores.append(token.split("=", 1)[1].rstrip("/"))
elif token == "--deselect":
index += 1
@@ -98,19 +128,39 @@ def _collects(command: str, path: str) -> bool:
if any(under(ignore, path) for ignore in ignores):
return False
+ if any(_ignore_glob_hits(pattern, path) for pattern in globs):
+ return False
return any(under(root, path) for root in roots)
+def _ignore_glob_hits(pattern: str, path: str) -> bool:
+ """pytest's own --ignore-glob rule, for a repo-relative path.
+
+ pytest matches with `_pytest.pathlib.fnmatch_ex`, which fnmatches the ABSOLUTE path
+ against the pattern with `*/` prepended when the pattern contains a separator and is
+ relative. fnmatch's `*` crosses `/`, so the prefix is free and matching the suffix of
+ the relative path is the same question. Verified against pytest itself rather than
+ assumed: `--ignore-glob=tests/test_[a-h]*.py` does exclude tests/test_a.py and does
+ NOT exclude tests/sub/test_a.py, which is the property the shards below rely on.
+ """
+ return fnmatch.fnmatch(path, pattern) or fnmatch.fnmatch(path, f"*/{pattern}")
+
+
# Two different trees are run in parallel now, and the isolation below belongs to exactly one of them. The repo-root
# job runs `tests/` from the checkout; the matrix job runs the backend's own suite with
# `working-directory: studio/backend`, so `tests/studio/...` is not a path that exists for it.
# Told apart by the selection they carry. This used to be one flag, `--ignore=tests/qlora`, which only the repo-root
# run had; the repo-root run is three shards now and only one of them carries that flag, so the test is membership of
# the matrix instead. The backend matrix run is not in it.
+#
+# Both sides are membership now. The backend run used to be told apart by one of its --ignore flags, which stopped
+# identifying it once it was sharded the same way and its flags moved onto a shared step. Reading each side out of its
+# own job's matrix is the same question asked the same way twice, and it cannot start matching the other job by
+# accident the way a shared flag can; the shape test below asserts the two sets stay disjoint.
def _over_the_repo_tests(command: str) -> bool:
- return any(selection in command for selection in _shard_selections())
+ return any(selection in command for selection in _selections("repo-cpu-tests"))
# The same pairing, for the backend matrix run. Ignoring a file from the parallel run and running it again serially is
@@ -128,10 +178,29 @@ BACKEND_ISOLATED = [
("tests/test_web_fetch_extraction.py", "compares parse time at two input sizes"),
("tests/test_tool_call_parser_strict.py", "compares parse time at two nesting depths"),
# Found by staging rather than by the scan, and the scan cannot find it: see below.
- ("tests/test_tunnel_safe_long_post.py", "work sleeps 0.2s past a 0.05s keepalive timer"),
+ (
+ "tests/test_tunnel_safe_long_post.py",
+ ":101 requires len(chunks) > 2, which one 100ms stall falsifies",
+ ),
("tests/test_scan_loras_off_event_loop.py", "counts heartbeats during a 0.3s sleep"),
("tests/test_anthropic_messages.py", "counts SSE keepalives emitted during a 0.24s stall"),
- ("tests/test_profile_stats.py", "counts event-loop ticks during a 0.5s blocking call"),
+ # The tick count is not the tight part of this file: it has 5x margin. :400 is,
+ # `assert elapsed < 0.2` around a join that already costs 0.03s on an idle box.
+ ("tests/test_profile_stats.py", ":400 asserts elapsed < 0.2 around a 0.03s join"),
+ # Was ignored by the parallel run and rerun serially and named in NEITHER direction
+ # here, so the file was the one thing this guard cannot see: deleting it from the
+ # serial step would have left it running nowhere with the job green. It qualifies
+ # twice over. Timing-tight at :1101 `assert started.is_set()` under a patched
+ # _SWITCH_BUDGET_S = 0.3, against a cold path the file itself records at 1.98s -- and
+ # like test_tunnel_safe_long_post, the assertion is on a RESULT rather than on a
+ # duration, so the scan below cannot reach it. State-mutating as well: it writes
+ # keepwarm globals directly at :2628-2630. It has already failed this way once on
+ # 3.13 while 3.10 passed the same commit.
+ (
+ "tests/test_media_auto_switch.py",
+ ":1101 asserts started.is_set() under a patched 0.3s budget on a 1.98s cold path, "
+ "and writes keepwarm globals at :2628-2630",
+ ),
]
# What the scan above does NOT cover, recorded because the gap is structural rather than a missing case. It finds
@@ -163,14 +232,12 @@ BACKEND_ISOLATED = [
# vCPUs it measures the scheduler as much as the code. Above it there is enough headroom to survive being descheduled.
# Twenty-two backend files assert some elapsed bound and serialising all of them would give back most of what -n 4
# buys, so the line is drawn where the measurement stops being about the code.
-BACKEND_MARKER = "--ignore=tests/test_studio_api.py"
+TIGHT_BOUND_S = 0.1
def _over_the_backend(command: str) -> bool:
- return BACKEND_MARKER in command
-
+ return any(selection in command for selection in _selections("pytest"))
-TIGHT_BOUND_S = 0.1
BACKEND_TESTS = Path(__file__).resolve().parents[2] / "studio" / "backend" / "tests"
_CLOCKS = ("monotonic", "perf_counter", "process_time", "time")
@@ -380,7 +447,7 @@ def _fragile_timing_asserts(path: Path) -> list:
@pytest.mark.parametrize("path, reason", ISOLATED, ids = [p for p, _ in ISOLATED])
def test_an_isolated_path_is_ignored_by_every_parallel_pytest_run(path, reason):
- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8")):
+ for command in _pytest_commands():
if " -n " not in f" {command} " or not _over_the_repo_tests(command):
continue
assert not _collects(command, path), (
@@ -394,7 +461,7 @@ def test_an_isolated_path_still_runs_in_a_serial_step(path, reason):
"""Ignoring it is half the change. Without this, the tests silently stop running."""
serial = [
command
- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))
+ for command in _pytest_commands()
if " -n " not in f" {command} "
and re.search(rf"(?<![\w/]){re.escape(path)}(?![\w/])", command)
]
@@ -406,23 +473,42 @@ def test_an_isolated_path_still_runs_in_a_serial_step(path, reason):
def test_the_command_scan_sees_the_parallel_run_and_the_serial_steps():
"""Pin the parser: a scan that matched nothing would pass both tests above."""
- commands = _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))
+ commands = _pytest_commands()
parallel = [command for command in commands if " -n " in f" {command} "]
- assert len(parallel) == 4, (
- f"expected four parallel pytest runs, the backend matrix and repo-cpu-tests' three "
- f"shards, got {parallel}. If a job stopped running in parallel, or a shard was added "
- f"or removed, say so here rather than letting this scan quietly cover fewer runs."
+ assert len(parallel) == 6, (
+ f"expected six parallel pytest runs, three shards of the backend matrix and three "
+ f"of repo-cpu-tests, got {parallel}. If a job stopped running in parallel, or a "
+ f"shard was added or removed, say so here rather than letting this scan quietly "
+ f"cover fewer runs."
)
root = [command for command in parallel if _over_the_repo_tests(command)]
assert len(root) == 3, (
f"expected the three repo-root shards, got {root}. The isolation checks above apply "
f"to those, and a scan that matched none of them would pass on nothing."
)
+ backend = [command for command in parallel if _over_the_backend(command)]
+ assert len(backend) == 3, (
+ f"expected the three backend shards, got {backend}. Same reason: the backend "
+ f"isolation checks apply to those."
+ )
+ # The two jobs are told apart by whose matrix a command's paths came out of, so the sets have to be disjoint or
+ # each job's isolation rules would be applied to the other's runs -- which is how tests/studio/... would come to be
+ # asked of a run whose working directory is studio/backend.
+ assert not set(root) & set(
+ backend
+ ), f"a command reads as belonging to both jobs: {set(root) & set(backend)}"
+ assert len(parallel) == len(root) + len(backend), (
+ f"a parallel run belongs to neither job's matrix, so nothing below checks it: "
+ f"{[command for command in parallel if command not in root + backend]}"
+ )
# The line joins and the matrix substitution both have to be resolved, or a shard command reads as
# `pytest ${{ matrix.selection }} -q` with no paths at all and the first test above passes on nothing.
- assert all("${{" not in command for command in root)
+ assert all("${{" not in command for command in root + backend)
assert any("--ignore=" in command for command in root)
assert {command for command in root} == set(root), "a shard selection appears twice"
+ assert len(set(backend)) == 3, "a backend shard selection appears twice"
+ # Non-vacuous for the backend side too: the shards between them must reach the backend suite.
+ assert any(_collects(command, "tests/test_account_contract.py") for command in backend)
# Non-vacuous the other way too: the shards between them must reach the repo's test root, or "not collected by any
# parallel run" would be true of every path in the repo.
assert any(_collects(command, "tests/test_model_registry.py") for command in root)
@@ -437,16 +523,13 @@ def test_the_backend_matrix_still_runs_in_parallel():
depends on the order it runs in. Asserted here because dropping the flag would show up
only as CI slowly getting slower again, which nothing reports.
"""
- backend = [
- command
- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))
- if "--ignore=tests/test_studio_api.py" in command
- ]
+ backend = [command for command in _pytest_commands() if _over_the_backend(command)]
assert backend, "the backend matrix pytest step is gone or was renamed past this scan"
- assert " -n " in f" {backend[0]} ", (
- f"the backend matrix leg is running serially again, which costs about 17 minutes "
- f"per leg on every pull request and every push to main: {backend[0]}"
- )
+ for command in backend:
+ assert " -n " in f" {command} ", (
+ f"a backend matrix shard is running serially again, which costs about 17 "
+ f"minutes on every pull request and every push to main: {command}"
+ )
@pytest.mark.parametrize("path, reason", BACKEND_ISOLATED, ids = [p for p, _ in BACKEND_ISOLATED])
@@ -459,14 +542,17 @@ def test_a_backend_isolated_path_is_ignored_by_the_parallel_run(path, reason):
"""
parallel = [
command
- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))
+ for command in _pytest_commands()
if " -n " in f" {command} " and _over_the_backend(command)
]
assert parallel, "the backend parallel run is gone or was renamed past this scan"
- assert f"--ignore={path}" in parallel[0], (
- f"{path} ({reason}) is back in the backend parallel run, where its measurements "
- f"compare a descheduled worker against an undescheduled one: {parallel[0]}"
- )
+ # Asked of EVERY shard, as "does this run reach the path" rather than "does it spell
+ # this --ignore", so a shard that quietly grew its own root still has to get past it.
+ for command in parallel:
+ assert not _collects(command, path), (
+ f"{path} ({reason}) is back in a backend parallel run, where its measurements "
+ f"compare a descheduled worker against an undescheduled one: {command}"
+ )
@pytest.mark.parametrize("path, reason", BACKEND_ISOLATED, ids = [p for p, _ in BACKEND_ISOLATED])
@@ -474,7 +560,7 @@ def test_a_backend_isolated_path_still_runs_serially(path, reason):
"""Ignoring it is half the change; without this it runs nowhere and the job is green."""
serial = [
command
- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))
+ for command in _pytest_commands()
if " -n " not in f" {command} "
and re.search(rf"(?<![\w/]){re.escape(path)}(?![\w/])", command)
]
@@ -487,9 +573,8 @@ def test_a_backend_isolated_path_still_runs_serially(path, reason):
def test_every_tight_elapsed_bound_is_isolated():
"""The rule, applied by scanning rather than by memory.
- Two of the entries above were found by review rather than by CI: they passed on
- staging and would have flaked later. A new test asserting a 20ms bound would do the
- same. This finds them, so adding one forces the isolation instead of buying a flake.
+ Two entries above were found by review, not CI: they passed on staging and would have
+ flaked later. This finds them, so adding one forces the isolation instead of a flake.
"""
isolated = {path for path, _ in BACKEND_ISOLATED}
stray = {}
@@ -579,32 +664,24 @@ def test_the_scan_finds_all_three_shapes(tmp_path):
)
assert not _fragile_timing_asserts(roomy), _fragile_timing_asserts(roomy)
- # Deliberately nothing about the live suite here. The synthetic files above already
- # exercise every shape the scanner knows, and a "the live suite still has some"
- # assertion would turn cleaning the last one into a failure, which is the same
- # dependency on real files this rewrite exists to remove.
+ # Deliberately nothing about the live suite: the synthetic files cover every shape, and
+ # asserting the suite still has some would turn cleaning the last one into a failure.
def test_an_isolated_file_never_shadows_an_installed_library_with_a_stub():
"""A stub may stand in for a MISSING library, never for an installed one.
- `sys.modules.setdefault("httpx", stub)` reads as deferring to the real library and
- does not: sys.modules holds what has been IMPORTED, not what is installed, so in a
- process where nothing has touched httpx yet the stub wins and shadows it for the rest
- of the session. These stubs carry no Response, starlette.testclient reads
- httpx.Response at import, and every module collected afterwards that reaches
- fastapi.testclient or routes.inference dies on it.
-
- In a 26,000-test run something always imports httpx first, so this was invisible for
- as long as the suite ran as one process. The serial step collects ten files and
- nothing else, and the 3.10 leg failed collection on two of them the first time it
- ran.
-
- Scoped to the isolated files on purpose. Roughly fifty other backend modules stub
- structlog the same way, and they are load-bearing in a run that also imports the real
- one; rewriting them is a separate change with its own risk, and the full parallel run
- is not the process where a small file list makes the shadowing decisive. What has to
- hold here is that anything moved OUT of that run stands on its own.
+ `sys.modules.setdefault("httpx", stub)` reads as deferring to the real library and does
+ not: sys.modules holds what has been IMPORTED, not what is installed, so where nothing
+ has touched httpx yet the stub wins for the rest of the session. These stubs carry no
+ Response, starlette.testclient reads httpx.Response at import, and every module after it
+ reaching fastapi.testclient or routes.inference dies on it. In a 26,000-test run
+ something always imports httpx first, so this stayed invisible while the suite was one
+ process; the serial step collects ten files, and the 3.10 leg failed collection on two.
+
+ Scoped to the isolated files on purpose: ~fifty other backend modules stub structlog the
+ same way and are load-bearing in a run that also imports the real one. What has to hold
+ here is that anything moved OUT of the parallel run stands on its own.
"""
offenders = {}
for name, _reason in BACKEND_ISOLATED:
diff --git a/tests/test_ci_backend_pytest_shards.py b/tests/test_ci_backend_pytest_shards.py
new file mode 100644
index 00000000000..d70b0e9ec7e
--- /dev/null
+++ b/tests/test_ci_backend_pytest_shards.py
@@ -0,0 +1,395 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
+
+"""Guards that the sharded backend `pytest` job still runs every test it used to.
+
+`(Python 3.13)` was one job discovering all of studio/backend/tests; it is now three shards
+on three runners, and the whole risk is a file landing in no shard: the job goes green
+faster having checked less, and nothing says so.
+
+So the split is not three allowlists. studio/backend/tests is flat (923 files, plus 17 under
+multi_account), so there are no directory roots to divide and the division is by the first
+letter after `test_`. Shards 1 and 2 name their range and exclude everything else; shard 3 is
+"tests/ except those two ranges", which makes exactly-one-shard a property of the shape
+rather than of anyone remembering to edit the workflow. These tests hold that shape, and hold
+that the inherited selection -- the -k filter, the two timeouts, and the twelve files the
+serial step reruns -- came through intact.
+
+All three shards root at `tests/` and differ only in ignores, because `--ignore=FILE` does
+not filter a file passed as an explicit argument (checked against pytest): a shard rooted at
+a shell-expanded glob would name the twelve serial files and run them anyway.
+
+The per-test-id partition was checked directly when the split landed, via --junitxml diffs:
+13,073 + 17,483 + 12,053 = 42,609 against the baseline's 42,609, no overlap, no gap, every id
+in the same state bar three failure-to-passed flips, none caused by the split (two flip
+between two runs of the UNSHARDED selection over the same tree; the third asserts four
+concurrent sizings cannot all claim the same 8GB, which fails on a loaded box). That measured
+one tree; these tests keep it true of the next one.
+"""
+
+import fnmatch
+from pathlib import Path
+
+import pytest
+import yaml
+
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+
+_BACKEND_CI = REPO_ROOT / ".github" / "workflows" / "studio-backend-ci.yml"
+_BACKEND_TESTS = REPO_ROOT / "studio" / "backend" / "tests"
+
+# Named once so the tests can say "and only there", and so a rename fails here rather than
+# silently weakening them.
+_CATCH_ALL = "rest"
+
+# The twelve files the parallel run ignores and the serial step reruns. Read out of the
+# workflow rather than listed here, so moving one between the two cannot make it look
+# dropped, as tests/test_ci_repo_cpu_shards.py does for its isolated paths.
+_SERIAL_STEP = "Backend tests that cannot share a worker"
+
+# Files this job runs nowhere: a lost file would otherwise look like one of these. Only one,
+# predating the split -- the thirteenth --ignore on the parallel step, which no step reruns.
+_NOT_IN_ANY_SHARD = {
+ "tests/test_studio_api.py": (
+ "end-to-end against a live model and a GGUF download, which a GPU-less runner "
+ "cannot do; the workflow's own header says so"
+ ),
+}
+
+
+def _backend_ci() -> dict:
+ return yaml.safe_load(_BACKEND_CI.read_text(encoding = "utf-8"))
+
+
+def _job() -> dict:
+ return _backend_ci()["jobs"]["pytest"]
+
+
+def _parse_selection(tokens: list) -> tuple:
+ """(roots, ignores, ignore globs) out of a pytest argument list."""
+ roots, ignores, globs = [], [], []
+ index = 0
+ while index < len(tokens):
+ token = tokens[index]
+ if token.startswith("--ignore-glob="):
+ globs.append(token.split("=", 1)[1])
+ elif token.startswith("--ignore="):
+ ignores.append(token.split("=", 1)[1].rstrip("/"))
+ elif token == "--deselect":
+ index += 1 # a test id, not a path
+ elif token.startswith("-"):
+ pass
+ elif token.startswith("tests/") or token == "tests":
+ roots.append(token.rstrip("/"))
+ index += 1
+ return roots, ignores, globs
+
+
+def _shards() -> dict:
+ """{shard name: (roots, ignores, ignore globs)} for the command each shard really runs.
+
+ The matrix carries only what the shards DIFFER by. The thirteen --ignore flags they
+ all share sit on the step, so a model built from the matrix alone would have the
+ catch-all shard collecting the twelve files the serial step reruns -- it does not, and
+ reporting that it does would either fail honestly-written guards or push the flags into
+ three copies that have to agree. Both halves of the command, then.
+
+ The 3.11 floor-spot-check leg is not a shard of anything and carries no selection, so
+ it is not in here.
+ """
+ _, shared_ignores, shared_globs = _parse_selection(_shared_pytest_step()["run"].split())
+ shards = {}
+ for entry in _job()["strategy"]["matrix"]["include"]:
+ if "selection" not in entry:
+ continue
+ roots, ignores, globs = _parse_selection(entry["selection"].split())
+ shards[entry["shard"]] = (roots, ignores + shared_ignores, globs + shared_globs)
+ return shards
+
+
+def _covers(prefix: str, path: str) -> bool:
+ prefix = prefix.rstrip("/")
+ return path == prefix or path.startswith(prefix + "/")
+
+
+def _glob_hits(pattern: str, path: str) -> bool:
+ """pytest's own --ignore-glob rule, asked of a path relative to studio/backend.
+
+ pytest matches with `_pytest.pathlib.fnmatch_ex`, which fnmatches the ABSOLUTE path
+ against the pattern with `*/` prepended when the pattern contains a separator and is
+ relative. fnmatch's `*` crosses `/`, so the prepended prefix is free and matching the
+ tail of the relative path asks the same question.
+ """
+ return fnmatch.fnmatch(path, pattern) or fnmatch.fnmatch(path, f"*/{pattern}")
+
+
+def _claiming_shards(path: str, shards: dict) -> list:
+ """Which shards would collect `path`, by pytest's own root-and-ignore rules."""
+ claiming = []
+ for name, (roots, ignores, globs) in shards.items():
+ if any(_covers(ignore, path) for ignore in ignores):
+ continue
+ if any(_glob_hits(pattern, path) for pattern in globs):
+ continue
+ if any(_covers(root, path) for root in roots):
+ claiming.append(name)
+ return sorted(claiming)
+
+
+def _shared_pytest_step() -> dict:
+ for step in _job()["steps"]:
+ if "${{ matrix.selection }}" in str(step.get("run", "")):
+ return step
+ raise AssertionError("the sharded pytest step is gone from the backend pytest job")
+
+
+def _serial_paths() -> list:
+ """The files the serial step names, read out of the step itself."""
+ for step in _job()["steps"]:
+ if step.get("name") == _SERIAL_STEP:
+ return [token for token in str(step["run"]).split() if token.startswith("tests/")]
+ raise AssertionError(f"the {_SERIAL_STEP!r} step is gone from the backend pytest job")
+
+
+def _test_files() -> list:
+ return sorted(
+ str(path.relative_to(_BACKEND_TESTS.parent))
+ for path in _BACKEND_TESTS.rglob("test_*.py")
+ if "__pycache__" not in path.parts
+ )
+
+
+class TestEveryTestFileLandsInExactlyOneShard:
+ def test_no_file_is_dropped_or_run_twice(self):
+ shards = _shards()
+ serial = _serial_paths()
+ dropped, doubled = [], []
+ for path in _test_files():
+ if path in serial or path in _NOT_IN_ANY_SHARD:
+ continue # rerun by the serial step, or excluded from the job on purpose
+ claiming = _claiming_shards(path, shards)
+ if not claiming:
+ dropped.append(path)
+ elif len(claiming) > 1:
+ doubled.append((path, claiming))
+ assert not dropped, (
+ "these files are collected by no shard, so the backend pytest job no longer "
+ f"runs them: {dropped}"
+ )
+ assert not doubled, f"these files are collected by more than one shard: {doubled}"
+
+ @pytest.mark.parametrize("path", sorted(_NOT_IN_ANY_SHARD))
+ def test_the_files_no_shard_runs_are_the_expected_ones(self, path):
+ """Excluded on purpose, and from every shard rather than from one: without this, a
+ dropped file and a deliberately-skipped one look identical to the test above."""
+ claiming = _claiming_shards(path, _shards())
+ assert (
+ claiming == []
+ ), f"{path} is {_NOT_IN_ANY_SHARD[path]}, but shard(s) {claiming} now collect it"
+ assert (
+ _BACKEND_TESTS.parent / path
+ ).is_file(), f"{path} no longer exists, so this exclusion is stale"
+
+ @pytest.mark.parametrize(
+ "path, expected",
+ [
+ ("tests/test_apple.py", "a-k"),
+ ("tests/test_kiwi.py", "a-k"),
+ ("tests/test_lemon.py", "l-r"),
+ ("tests/test_raisin.py", "l-r"),
+ ("tests/test_squash.py", _CATCH_ALL),
+ ("tests/test_yam.py", _CATCH_ALL),
+ ],
+ )
+ def test_a_brand_new_file_is_claimed_by_exactly_one_shard(self, path, expected):
+ """The property the split exists for: a file nobody has heard of yet runs, in one
+ place, without a workflow edit."""
+ claiming = _claiming_shards(path, _shards())
+ assert claiming == [
+ expected
+ ], f"{path} must be collected by {expected} and only there; got {claiming}"
+
+ @pytest.mark.parametrize(
+ "path",
+ [
+ "tests/a_directory_added_tomorrow/test_new.py",
+ "tests/multi_account/test_alice_bob_matrix.py",
+ "tests/test_Zebra.py",
+ "tests/test_9_regression.py",
+ "tests/an_odd_name_test.py",
+ ],
+ ids = ["new-subdir", "existing-subdir", "uppercase", "digit", "underscore-test-suffix"],
+ )
+ def test_the_names_the_ranges_do_not_describe_land_in_the_catch_all(self, path):
+ """Why shards 1 and 2 exclude their complement rather than name their range.
+
+ Not every name starts with a lowercase letter: a subdirectory, an uppercase or digit
+ first character, and pytest's other discovery pattern `*_test.py` are outside both
+ ranges, and each would be claimed by BOTH ranged shards had they said "not l-z" and
+ "not a-k". Only the catch-all may be open-ended, which is what makes exactly-one hold
+ for names nobody anticipated.
+ """
+ claiming = _claiming_shards(path, _shards())
+ assert claiming == [_CATCH_ALL], (
+ f"{path} is not described by either range, so it must land in the "
+ f"{_CATCH_ALL!r} shard and only there; got {claiming}"
+ )
+
+ def test_a_name_matching_both_discovery_patterns_falls_out_of_every_shard(self):
+ """`test_api_test.py` matches `test_*.py` AND `*_test.py`, and lands nowhere.
+
+ The ranged shards drop it for the suffix, the catch-all for the `test_[a-r]*` range;
+ measured against real pytest, all three collect it zero times. Fixing it would need
+ the suffix rule to mean "ends in _test.py unless it starts with test_", which fnmatch
+ cannot say: diverging per prefix character over-consumes `te_test.py` into BOTH ranged
+ shards (tried, and this guard caught it), and completing it needs nine globs per shard
+ for a shape this repo does not use. Forbidden by name below instead.
+ """
+ assert _claiming_shards("tests/test_api_test.py", _shards()) == [], (
+ "this is a known limitation; if it now lands in a shard the patterns have "
+ "changed and the naming rule below can be relaxed"
+ )
+
+ def test_a_subdirectory_named_like_a_test_file_falls_out_of_every_shard(self):
+ """The one shape the catch-all cannot absorb. Recorded, not hidden.
+
+ fnmatch's `*` crosses `/` and fnmatch_ex matches the whole path, so the catch-all's
+ `tests/test_[a-r]*.py` also excludes `tests/test_api/test_auth.py`, which the ranged
+ shards already exclude via `tests/*/*`. Measured against real pytest: zero collections
+ in all three. Both obvious repairs were measured and fail -- `[!/]` cannot stop `*`
+ crossing a separator, and an extra root does not bypass --ignore-glob -- so the test
+ below enforces the invariant by name instead.
+ """
+ assert _claiming_shards("tests/test_api/test_auth.py", _shards()) == [], (
+ "this is a known limitation of the split; if it now lands in a shard the "
+ "patterns have changed and the naming rule below can be dropped"
+ )
+
+ def test_no_test_subdirectory_is_named_like_a_test_file(self):
+ """Enforces the invariant the split depends on, so the hole above stays unreachable."""
+ dirs = sorted(
+ path.name
+ for path in _BACKEND_TESTS.iterdir()
+ if path.is_dir() and path.name.startswith("test_")
+ )
+ assert not dirs, (
+ f"{dirs} would be collected by no shard, because the catch-all's "
+ "tests/test_[a-r]*.py excludes nested paths too (fnmatch * crosses /). Rename "
+ "the directory, or give the catch-all an explicit root for it."
+ )
+ hybrids = sorted(
+ path.name
+ for path in _BACKEND_TESTS.iterdir()
+ if path.is_file() and path.name.startswith("test_") and path.name.endswith("_test.py")
+ )
+ assert not hybrids, (
+ f"{hybrids} match both default discovery patterns, so the ranged shards drop "
+ "them for the _test.py suffix and the catch-all drops them for the test_[a-r]* "
+ "range, leaving them in no shard. Drop one of the two markers from the name."
+ )
+
+
+class TestTheSplitKeepsTheSelectionItInherited:
+ def test_every_shard_keeps_the_marker_filter(self):
+ """The environment-specific deselections: live GPU introspection and a real
+ llama.cpp process, none of which exist on a GPU-less runner. The filter is on the
+ shared step rather than per shard, so it is checked there."""
+ run = _shared_pytest_step()["run"]
+ for fragment in (
+ "not llama_cpp_load_progress_live",
+ "not TestGpuAutoSelection",
+ "not TestPreSpawnGpuResolution",
+ "not TestPerGpuFitGuardAllCounts",
+ "not TestTransformersIntrospection",
+ "not test_returns_cuda_when_cuda_available",
+ "not test_calls_cuda_cache_when_cuda",
+ ):
+ assert fragment in run, f"the -k filter lost {fragment!r}"
+
+ def test_every_shard_keeps_the_timeouts(self):
+ """A shard that loses these reports "cancelled" and names nothing, which is what
+ #9515 / #9530 were about."""
+ run = _shared_pytest_step()["run"]
+ assert "--timeout=330" in run
+ assert "timeout --signal=INT --kill-after=60" in run
+
+ def test_the_serial_files_are_ignored_by_every_shard(self):
+ """The twelve cannot share a worker, and sharding does not change that: three
+ runners are still four workers each. They are held out by the shared --ignore list
+ on the step, so this asks the question of the shard AND the step together."""
+ run = _shared_pytest_step()["run"]
+ shards = _shards()
+ for path in _serial_paths():
+ assert f"--ignore={path}" in run, (
+ f"{path} is rerun by the serial step and no longer ignored by the parallel "
+ f"one, so it runs twice and its timing assertions run under four workers"
+ )
+ claiming = _claiming_shards(path, shards)
+ assert (
+ claiming == []
+ ), f"{path} is rerun by the serial step and shard(s) {claiming} collect it too"
+
+ def test_the_serial_step_runs_in_one_process_in_one_shard(self):
+ """Splitting it would put its relative timings back on two machines, and running it
+ on every shard would run it three times."""
+ for step in _job()["steps"]:
+ if step.get("name") != _SERIAL_STEP:
+ continue
+ assert " -n " not in f" {step['run']} ", "the serial step is running under xdist"
+ condition = str(step.get("if", ""))
+ assert "matrix.shard ==" in condition, (
+ f"the serial step is no longer pinned to one shard, so it runs once per "
+ f"shard: {condition!r}"
+ )
+ pinned = [name for name in _shards() if f"matrix.shard == '{name}'" in condition]
+ assert len(pinned) == 1, f"expected exactly one shard named in {condition!r}"
+ return
+ raise AssertionError(f"the {_SERIAL_STEP!r} step is gone")
+
+ def test_the_serial_step_still_runs_the_same_twelve_files(self):
+ """Not eleven and not thirteen. Whether any of them could go back into the parallel
+ run was audited separately and the answer was no, so the set is carried across the
+ split unchanged; a file leaving it here is a coverage change wearing a refactor's
+ clothes."""
+ paths = _serial_paths()
+ assert len(paths) == 12, f"the serial step runs {len(paths)} files, not 12: {paths}"
+ assert len(set(paths)) == 12, f"the serial step names a file twice: {paths}"
+ missing = [path for path in paths if not (_BACKEND_TESTS.parent / path).is_file()]
+ assert not missing, f"the serial step names files that do not exist: {missing}"
+
+
+class TestTheGuardIsNotVacuous:
+ def test_a_gap_is_reported(self):
+ """Shard 3 stops being a catch-all and starts being an allowlist."""
+ broken = {
+ "a-k": (["tests/"], [], ["tests/*/*", "tests/*_test.py", "tests/test_[!a-k]*.py"]),
+ "l-r": (["tests/"], [], ["tests/*/*", "tests/*_test.py", "tests/test_[!l-r]*.py"]),
+ _CATCH_ALL: (["tests/test_squash.py"], [], []),
+ }
+ assert _claiming_shards("tests/test_yam.py", broken) == []
+ assert _claiming_shards("tests/multi_account/test_alice_bob_matrix.py", broken) == []
+
+ def test_an_overlap_is_reported(self):
+ """The mistake the ranges are written to avoid: shard 1 excluding `l-z` rather than
+ excluding `not a-k`, which leaves every name outside a-z owned by both."""
+ broken = {
+ "a-k": (["tests/"], [], ["tests/*/*", "tests/test_[l-z]*.py"]),
+ "l-r": (["tests/"], [], ["tests/*/*", "tests/test_[a-k]*.py", "tests/test_[s-z]*.py"]),
+ _CATCH_ALL: (["tests/"], [], ["tests/test_[a-r]*.py"]),
+ }
+ # All three, in fact: a name outside a-z is outside every range, so the catch-all
+ # takes it as well and the ranged shards no longer exclude it. The names in a-z are
+ # still partitioned correctly, which is why this is the mistake that survives review.
+ assert _claiming_shards("tests/test_Zebra.py", broken) == ["a-k", "l-r", _CATCH_ALL]
+ assert _claiming_shards("tests/test_apple.py", broken) == ["a-k"]
+ assert _claiming_shards("tests/test_monkey.py", broken) == ["l-r"]
+
+ def test_the_glob_rule_matches_pytest(self):
+ """`tests/test_[a-k]*.py` must not reach below a subdirectory, or shard 3 would lose
+ multi_account to a shard that never names it. Measured against pytest directly when
+ this landed; asserted here so a rewrite of _glob_hits cannot quietly change it."""
+ assert _glob_hits("tests/test_[a-k]*.py", "tests/test_apple.py")
+ assert not _glob_hits("tests/test_[a-k]*.py", "tests/multi_account/test_alice.py")
+ assert _glob_hits("tests/*/*", "tests/multi_account/test_alice.py")
+ assert not _glob_hits("tests/*/*", "tests/test_apple.py")