Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
Python agentaichatgptdeepseekfine-tuninggemmaimage-generationllamallmllmsopenaipythonqwenreinforcement-learningself-hostedstable-diffusiontext-to-speechttsuiunsloth
Shard the Python 3.13 backend pytest leg three ways (#10920)
Daniel Han80181969801d
diff --git a/.github/workflows/studio-backend-ci.yml b/.github/workflows/studio-backend-ci.ymlindex 43c30625f3d..655b064f63a 100644--- a/.github/workflows/studio-backend-ci.yml+++ b/.github/workflows/studio-backend-ci.yml@@ -92,7 +92,7 @@ env:jobs:pytest:- name: (Python ${{ matrix.python }})+ name: (Python ${{ matrix.python }}, ${{ matrix.shard }})runs-on: ubuntu-24.04# 45 guards against a hang; it is not a performance budget. The comment this replaces# claimed 14.1 to 15.0 minutes against a limit of 30, but this step now measures 22.5 to@@ -114,8 +114,16 @@ jobs:# its work in six minutes. The per-test --timeout below is what actually names a hang;# this stays as the backstop for a wedge the per-test timer cannot see, such as one in# collection or in a worker that never comes up.+ #+ # Unchanged by the split below, deliberately. A shard does less work than the leg it+ # came from, so 45 only gets looser, and retuning it on a prediction is the failure+ # this number was raised to fix: the sharded spread has not been measured on a runner+ # yet. Retune it once there are runs to retune it against.timeout-minutes: 45strategy:+ # fail-fast off: three shards of one suite, and cancelling the other two on the first+ # failure would hide which of them were also red. It was already off for the+ # 3.13/3.11 pair, for the same reason.fail-fast: falsematrix:# 3.13 only, on pull requests and on main alike.@@ -138,22 +146,81 @@ jobs:# away: two interpreters that both accept a line can still behave differently on# it, and a sys.version_info branch is only ever parsed, never taken.#- # So the branches were counted rather than assumed. Seven files in the backend- # carry one, at 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by- # the old matrix, whose oldest leg was 3.10, so every leg took the same side of- # them and removing legs loses nothing there. 3.14 is above every leg there has- # ever been. What is genuinely lost is the PRE-3.12 side of three files, and that- # is small enough to keep executing directly.+ # So the branches were counted rather than assumed. Seven backend files carry one, at+ # 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by the old matrix+ # (oldest leg 3.10), and 3.14 is above every leg there has ever been. What is+ # genuinely lost is the PRE-3.12 side of three files.+ #+ # Hence the second entry: 3.11, the newest version still on that side, running those+ # three files and nothing else. Roughly 40 tests in under ten seconds, beside the full+ # leg rather than in front of it, so the critical path is unchanged.+ #+ # The 3.13 leg is three shards of one selection, balanced on measured time rather than+ # file count, for the reason repo-cpu-tests was split below: it is now the longest+ # thing a pull request waits for, at 763-822s parallel plus 201-234s serial.+ #+ # Summed test time is 1886.1s across 922 files (`--durations=0 --durations-min=0`,+ # -n 16; the default 0.005s cutoff hides ~105,000 durations and is unusable here),+ # split on the first letter after `test_`:+ #+ # a-k 542.8s 396 files 28.8%+ # l-r 693.7s 318 files 36.8%+ # s-z + every subdirectory 649.6s 208 files 34.4%#- # Hence the second entry: 3.11, the newest version that still takes that side,- # running those three files and nothing else. Roughly 40 tests in under ten- # seconds, not a second copy of the suite, and it runs beside the full leg rather- # than in front of it, so the critical path is unchanged.+ # so the longest shard is 1.103x an ideal third. The letters are lumpy (l is 217.2s,+ # t is 280.4s, multi_account's 159.3s is stuck in the catch-all) and no contiguous+ # split does better. A hand-picked letter SET reaches 1.001x and is not worth it: an+ # allowlist tuned to one measurement is what the shape below exists to avoid.+ #+ # The shape. All three shards root at `tests/` and differ only in what they ignore:+ # `--ignore=FILE` does NOT filter a file passed as an explicit argument (checked+ # against pytest), so a shard rooted at an expanded glob would drag the twelve serial+ # files back in past the shared --ignore list below. Shards 1 and 2 name their range+ # and exclude everything else, including names outside a-z and anything below a+ # subdirectory. Shard 3 is "tests/ minus the other two ranges", never an allowlist, so+ # a new file lands in exactly one shard with no workflow edit: test_apple.py in 1,+ # test_monkey.py in 2, and test_snake.py, tests/a_new_dir/test_x.py, foo_test.py and+ # test_Zebra.py in 3. test_ci_backend_pytest_shards.py fails if that stops holding,+ # and test_backend_ci_parallel_isolation.py fails if a shard collects one of the twelve.+ #+ # Two name shapes the split cannot absorb, forbidden by name in that guard rather than+ # left to be discovered: a subdirectory called test_*, and a file matching BOTH+ # discovery patterns (test_api_test.py). Each lands in no shard, because fnmatch's *+ # crosses / and the suffix rule cannot say "unless it starts with test_" in one pattern.+ # Diverging per prefix character was tried and reverted: it fixes test_api_test.py but+ # over-consumes te_test.py into BOTH ranged shards, and completing it needs nine globs+ # per shard for a filename this repo does not use.+ #+ # Verified at the level of the test id, not the count: the unsharded selection and the+ # three shards run with --junitxml and the reports diffed gave 13,073 + 17,483 + 12,053+ # = 42,609 against the baseline's 42,609, no id in two shards or in neither, and the+ # same 332 skipped and 386 errored.include:- python: '3.13'scope: full+ shard: a-k+ selection: >-+ tests/+ --ignore-glob=tests/*/*+ --ignore-glob=tests/*_test.py+ --ignore-glob=tests/test_[!a-k]*.py+ - python: '3.13'+ scope: full+ shard: l-r+ selection: >-+ tests/+ --ignore-glob=tests/*/*+ --ignore-glob=tests/*_test.py+ --ignore-glob=tests/test_[!l-r]*.py+ - python: '3.13'+ scope: full+ shard: rest+ selection: >-+ tests/+ --ignore-glob=tests/test_[a-r]*.py- python: '3.11'scope: floor-spot-check+ shard: floorsteps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1with:@@ -284,9 +351,20 @@ jobs:# Reproduced with a test forking a child that never exits: unwrapped it hangs# silently, wrapped it exits 124 through execnet safe_terminate -> workerpool.waitall.# Expect a stuck FRAME, not always a test name. 1500s is ~3x the 8-10 minute norm.+ #+ # The thirteen --ignore flags and the -k filter stay HERE rather than moving into+ # the matrix: they are the selection every shard inherits, and one copy of them is+ # what makes "the twelve serial files are ignored by every parallel shard" a+ # property of the step instead of three lists that have to agree. The shards differ+ # only in `${{ matrix.selection }}`.+ #+ # The outer 1500s is not retuned on a prediction, for the reason repo-cpu-tests+ # gives below: a bound that lands inside the real spread is the failure this kind of+ # number exists to prevent, and the sharded spread has not been measured on a runner+ # yet. It only ever gets looser as the work per shard drops.run: |timeout --signal=INT --kill-after=60 1500 \- python -m pytest tests/ -q --tb=short -n 4 --timeout=330 \+ python -m pytest ${{ matrix.selection }} -q --tb=short -n 4 --timeout=330 \--ignore=tests/test_studio_api.py \--ignore=tests/test_streaming_stripper.py \--ignore=tests/test_llama_cpp_wait_for_vram_settle.py \@@ -303,7 +381,12 @@ jobs:-k 'not llama_cpp_load_progress_live and not TestGpuAutoSelection and not TestPreSpawnGpuResolution and not TestPerGpuFitGuardAllCounts and not TestTransformersIntrospection and not test_returns_cuda_when_cuda_available and not test_calls_cuda_cache_when_cuda'- name: Backend tests that cannot share a worker- if: matrix.scope == 'full'+ # One shard, one process, the same twelve files. Not split and not redistributed:+ # whether any of them could go back into the parallel run was asked separately and+ # the answer was no, so this step is carried unchanged and only has to be carried+ # ONCE. a-k because it is the shortest shard at 28.8% of summed test time, and this+ # step is 201-234s: it lands there rather than on the 36.8% shard.+ if: matrix.scope == 'full' && matrix.shard == 'a-k'working-directory: studio/backend# Relative timing against a reference measured in the same process. Serial, so the# comparison is between two implementations rather than between two schedulings.diff --git a/studio/backend/tests/test_native_audio.py b/studio/backend/tests/test_native_audio.pyindex b86f3bd6471..d2617678aad 100644--- a/studio/backend/tests/test_native_audio.py+++ b/studio/backend/tests/test_native_audio.py@@ -13,6 +13,46 @@ from types import SimpleNamespaceimport pytestimport torch+import importlib+import types+from unittest.mock import MagicMock+++# This file reaches core/inference/inference.py, which imports unsloth at module scope, from+# inside a helper, so test_backend_tests_stub_heavy_imports.py's source scan never saw it. The+# import only ever worked when another file on the same xdist worker had already stubbed+# unsloth; sharding changed who shares a worker, so the stub is explicit here now.+_STUBBED: list[str] = []+++def _stub_if_missing(name, attrs):+ """Stub a dep the backend pytest matrix does not install, as test_trainer_stdout_quiet.py does.++ That matrix stops at studio.txt plus torch and transformers; repo-cpu-tests is the job that+ installs unsloth_zoo. Unstubbed, this module fails COLLECTION and takes the job down. A real+ install is left alone, and __spec__ = None keeps the namespace-shadow guard a no-op here.+ """+ if name in sys.modules:+ return+ try:+ importlib.import_module(name)+ return+ except Exception: # noqa: BLE001 - unusable here either way, so stub it+ pass+ _STUBBED.append(name)+ mod = types.ModuleType(name)+ mod.__spec__ = None+ for attr in attrs:+ setattr(mod, attr, MagicMock())+ sys.modules[name] = mod+ parent, _, child = name.rpartition(".")+ if parent and parent in sys.modules:+ setattr(sys.modules[parent], child, mod)+++_stub_if_missing("unsloth", ("FastLanguageModel", "FastVisionModel", "is_bfloat16_supported"))+_stub_if_missing("unsloth.chat_templates", ("get_chat_template",))+from core.inference.native_audio import (HIGGS_TTS2_CODEC_REPO,HIGGS_TTS3_CODEC_REPO,@@ -28,6 +68,16 @@ from core.inference.native_audio import (native_audio_type_from_local_path,)+# Bind the dependency while the stubs stand, then drop them, as test_trainer_stdout_quiet.py+# does. Left in place they outlive this module, and a later file's _stub_if_missing returns+# before recording ownership, so nobody can clean them up. Concretely, _shared_policy branches+# on `"unsloth" in sys.modules` and returns None off the stub instead of reaching its disk+# fallback. A real install stubs nothing, so this is a no-op there.+import core.inference.inference # noqa: F401,E402 - imported to bind it under the stubs++for _name in reversed(_STUBBED):+ sys.modules.pop(_name, None)+def _backend(audio_type: str, **entry):backend = NativeAudioBackend.__new__(NativeAudioBackend)diff --git a/studio/backend/tests/test_native_tool_token_provenance.py b/studio/backend/tests/test_native_tool_token_provenance.pyindex 6959828139f..630fb661d21 100644--- a/studio/backend/tests/test_native_tool_token_provenance.py+++ b/studio/backend/tests/test_native_tool_token_provenance.py@@ -8,6 +8,47 @@ import jsonimport pytestimport torch+import importlib+import sys+import types+from unittest.mock import MagicMock+++# This file reaches core/inference/inference.py, which imports unsloth at module scope, from+# inside a helper, so test_backend_tests_stub_heavy_imports.py's source scan never saw it. The+# import only ever worked when another file on the same xdist worker had already stubbed+# unsloth; sharding changed who shares a worker, so the stub is explicit here now.+_STUBBED: list[str] = []+++def _stub_if_missing(name, attrs):+ """Stub a dep the backend pytest matrix does not install, as test_trainer_stdout_quiet.py does.++ That matrix stops at studio.txt plus torch and transformers; repo-cpu-tests is the job that+ installs unsloth_zoo. Unstubbed, this module fails COLLECTION and takes the job down. A real+ install is left alone, and __spec__ = None keeps the namespace-shadow guard a no-op here.+ """+ if name in sys.modules:+ return+ try:+ importlib.import_module(name)+ return+ except Exception: # noqa: BLE001 - unusable here either way, so stub it+ pass+ _STUBBED.append(name)+ mod = types.ModuleType(name)+ mod.__spec__ = None+ for attr in attrs:+ setattr(mod, attr, MagicMock())+ sys.modules[name] = mod+ parent, _, child = name.rpartition(".")+ if parent and parent in sys.modules:+ setattr(sys.modules[parent], child, mod)+++_stub_if_missing("unsloth", ("FastLanguageModel", "FastVisionModel", "is_bfloat16_supported"))+_stub_if_missing("unsloth.chat_templates", ("get_chat_template",))+from core.inference.native_tool_tokens import (NATIVE_TOOL_CONTROL_TOKENS,NativeToolTokenDecoder,@@ -18,6 +59,16 @@ from core.inference.native_tool_tokens import (from core.inference.safetensors_agentic import run_safetensors_tool_loopfrom core.inference.tool_call_parser import parse_tool_calls_from_text+# Bind the dependency while the stubs stand, then drop them, as test_trainer_stdout_quiet.py+# does. Left in place they outlive this module, and a later file's _stub_if_missing returns+# before recording ownership, so nobody can clean them up. Concretely, _shared_policy branches+# on `"unsloth" in sys.modules` and returns None off the stub instead of reaching its disk+# fallback. A real install stubs nothing, so this is a no-op there.+import core.inference.inference # noqa: F401,E402 - imported to bind it under the stubs++for _name in reversed(_STUBBED):+ sys.modules.pop(_name, None)+class _GemmaTokenDecoder:_pieces = {diff --git a/tests/studio/test_backend_ci_matrix.py b/tests/studio/test_backend_ci_matrix.pyindex 89471757ed6..5b2d39fdd9d 100644--- a/tests/studio/test_backend_ci_matrix.py+++ b/tests/studio/test_backend_ci_matrix.py@@ -47,13 +47,25 @@ CEILING = "3.13"def _legs() -> dict[str, str]:- """Each leg the matrix runs, as scope -> interpreter."""+ """Each scope the matrix runs, as scope -> interpreter.++ A scope maps to one interpreter but no longer to one entry: `full` is three shards of+ the same suite on the same 3.13, split because the leg had become the longest thing a+ pull request waits for. What this has to keep unambiguous is the SCOPE-to-interpreter+ mapping the tests below ask about, so that is what is asserted -- counting entries+ would now be counting shards, which is a different question and not this file's.+ """document = yaml.safe_load(WORKFLOW.read_text(encoding = "utf-8"))matrix = document["jobs"]["pytest"]["strategy"]["matrix"]entries = matrix.get("include")assert entries, f"the matrix no longer lists its legs by scope: {matrix!r}"- legs = {str(entry["scope"]): str(entry["python"]) for entry in entries}- assert len(legs) == len(entries), f"two legs share a scope: {entries!r}"+ legs = {}+ for entry in entries:+ scope, python = str(entry["scope"]), str(entry["python"])+ assert legs.setdefault(scope, python) == python, (+ f"scope {scope!r} runs on more than one interpreter, so scope no longer says "+ f"which one a leg is: {entries!r}"+ )return legsdiff --git a/tests/studio/test_backend_ci_parallel_isolation.py b/tests/studio/test_backend_ci_parallel_isolation.pyindex a0af9b3568f..3a3a7edc750 100644--- a/tests/studio/test_backend_ci_parallel_isolation.py+++ b/tests/studio/test_backend_ci_parallel_isolation.py@@ -22,6 +22,7 @@ ignore appears without a step that runs the same path, or the other way round."""import ast+import fnmatchimport importlib.utilimport refrom pathlib import Path@@ -41,51 +42,80 @@ ISOLATED = []-def _pytest_commands(text: str) -> list[str]:- """Every `python -m pytest ...` invocation in the workflow, line joins resolved.+def _jobs() -> dict:+ return yaml.safe_load(WORKFLOW.read_text(encoding = "utf-8"))["jobs"]- Read off the raw text rather than the parsed YAML: a `run:` block is one- scalar and the interesting structure is inside it, so parsing buys nothing and- would make this depend on the job/step layout instead of the commands.- The one thing that cannot be read off the text is `${{ matrix.selection }}`: since- repo-cpu-tests became three shards, its parallel command is a template and its paths- live in the matrix. Those are substituted in, one command per shard, so a scan of the- commands is still a scan of what actually runs.+def _selections(job_name: str) -> list[str]:+ """The `selection` of each matrix entry of one job, whitespace-normalised.++ Not every entry has one: the `pytest` job's 3.11 floor-spot-check leg names three+ files directly and is not a shard of anything, so it carries no selection."""- joined = re.sub(r"\\\s*\n\s*", " ", text)- commands = [- line.strip()- for line in joined.splitlines()- if "python -m pytest" in line and not line.lstrip().startswith("#")- ]- expanded = []- for command in commands:- if "${{ matrix.selection }}" not in command:- expanded.append(command)- continue- for selection in _shard_selections():- expanded.append(command.replace("${{ matrix.selection }}", selection))- return expanded+ job = _jobs()[job_name]+ include = job.get("strategy", {}).get("matrix", {}).get("include", [])+ return [" ".join(entry["selection"].split()) for entry in include if "selection" in entry]+++def _commands_in(job_name: str) -> list[str]:+ """Every `python -m pytest ...` invocation of one job, line joins and matrix resolved.++ Read off the raw text of each `run:` scalar rather than off more parsed YAML: a+ `run:` block is one scalar and the interesting structure is inside it, so parsing+ further buys nothing and would make this depend on step layout instead of commands.++ The one thing that cannot be read off the text is `${{ matrix.selection }}`, because+ the paths live in the matrix rather than in the command. BOTH parallel jobs are+ sharded now and both write their step that way, so the substitution has to know whose+ matrix to read: expanding a command with the other job's selections would build runs+ that are not in the workflow and ask the isolation questions below of those instead.+ Hence per job, which is also how the caller knows the owner without guessing from a+ flag that happens to appear in one of them.+ """+ job = _jobs()[job_name]+ selections = _selections(job_name)+ commands = []+ for step in job.get("steps", []):+ joined = re.sub(r"\\\s*\n\s*", " ", str(step.get("run", "")))+ for line in joined.splitlines():+ line = line.strip()+ if "python -m pytest" not in line or line.startswith("#"):+ continue+ if "${{ matrix.selection }}" in line:+ commands.extend(+ line.replace("${{ matrix.selection }}", selection) for selection in selections+ )+ else:+ commands.append(line)+ return commands-def _shard_selections() -> list[str]:- job = yaml.safe_load(WORKFLOW.read_text(encoding = "utf-8"))["jobs"]["repo-cpu-tests"]- return [" ".join(entry["selection"].split()) for entry in job["strategy"]["matrix"]["include"]]+def _pytest_commands() -> list[str]:+ """Every pytest invocation in the workflow, across every job."""+ return [command for job_name in _jobs() for command in _commands_in(job_name)]def _collects(command: str, path: str) -> bool:- """Whether a pytest command would collect `path`, by its roots and its --ignore flags.+ """Whether a pytest command would collect `path`, by its roots and its ignore flags.An isolated path used to be kept out of the parallel run by naming it in an --ignore.A shard that does not name its directory at all keeps it out just as effectively, so- the question the guard asks is whether the run reaches the path, not how."""+ the question the guard asks is whether the run reaches the path, not how.++ --ignore-glob is read as well as --ignore, because the backend shards are told apart+ by nothing else: all three root at `tests/` and differ only in which glob they+ exclude. Treating those flags as noise would have every backend shard appear to+ collect every backend file, and "not collected by any parallel run" would then be+ unfalsifiable for the whole suite.+ """tokens = command.split()- roots, ignores = [], []+ roots, ignores, globs = [], [], []index = 0while index < len(tokens):token = tokens[index]- if token.startswith("--ignore="):+ if token.startswith("--ignore-glob="):+ globs.append(token.split("=", 1)[1].strip("'\""))+ elif token.startswith("--ignore="):ignores.append(token.split("=", 1)[1].rstrip("/"))elif token == "--deselect":index += 1@@ -98,19 +128,39 @@ def _collects(command: str, path: str) -> bool:if any(under(ignore, path) for ignore in ignores):return False+ if any(_ignore_glob_hits(pattern, path) for pattern in globs):+ return Falsereturn any(under(root, path) for root in roots)+def _ignore_glob_hits(pattern: str, path: str) -> bool:+ """pytest's own --ignore-glob rule, for a repo-relative path.++ pytest matches with `_pytest.pathlib.fnmatch_ex`, which fnmatches the ABSOLUTE path+ against the pattern with `*/` prepended when the pattern contains a separator and is+ relative. fnmatch's `*` crosses `/`, so the prefix is free and matching the suffix of+ the relative path is the same question. Verified against pytest itself rather than+ assumed: `--ignore-glob=tests/test_[a-h]*.py` does exclude tests/test_a.py and does+ NOT exclude tests/sub/test_a.py, which is the property the shards below rely on.+ """+ return fnmatch.fnmatch(path, pattern) or fnmatch.fnmatch(path, f"*/{pattern}")++# Two different trees are run in parallel now, and the isolation below belongs to exactly one of them. The repo-root# job runs `tests/` from the checkout; the matrix job runs the backend's own suite with# `working-directory: studio/backend`, so `tests/studio/...` is not a path that exists for it.# Told apart by the selection they carry. This used to be one flag, `--ignore=tests/qlora`, which only the repo-root# run had; the repo-root run is three shards now and only one of them carries that flag, so the test is membership of# the matrix instead. The backend matrix run is not in it.+#+# Both sides are membership now. The backend run used to be told apart by one of its --ignore flags, which stopped+# identifying it once it was sharded the same way and its flags moved onto a shared step. Reading each side out of its+# own job's matrix is the same question asked the same way twice, and it cannot start matching the other job by+# accident the way a shared flag can; the shape test below asserts the two sets stay disjoint.def _over_the_repo_tests(command: str) -> bool:- return any(selection in command for selection in _shard_selections())+ return any(selection in command for selection in _selections("repo-cpu-tests"))# The same pairing, for the backend matrix run. Ignoring a file from the parallel run and running it again serially is@@ -128,10 +178,29 @@ BACKEND_ISOLATED = [("tests/test_web_fetch_extraction.py", "compares parse time at two input sizes"),("tests/test_tool_call_parser_strict.py", "compares parse time at two nesting depths"),# Found by staging rather than by the scan, and the scan cannot find it: see below.- ("tests/test_tunnel_safe_long_post.py", "work sleeps 0.2s past a 0.05s keepalive timer"),+ (+ "tests/test_tunnel_safe_long_post.py",+ ":101 requires len(chunks) > 2, which one 100ms stall falsifies",+ ),("tests/test_scan_loras_off_event_loop.py", "counts heartbeats during a 0.3s sleep"),("tests/test_anthropic_messages.py", "counts SSE keepalives emitted during a 0.24s stall"),- ("tests/test_profile_stats.py", "counts event-loop ticks during a 0.5s blocking call"),+ # The tick count is not the tight part of this file: it has 5x margin. :400 is,+ # `assert elapsed < 0.2` around a join that already costs 0.03s on an idle box.+ ("tests/test_profile_stats.py", ":400 asserts elapsed < 0.2 around a 0.03s join"),+ # Was ignored by the parallel run and rerun serially and named in NEITHER direction+ # here, so the file was the one thing this guard cannot see: deleting it from the+ # serial step would have left it running nowhere with the job green. It qualifies+ # twice over. Timing-tight at :1101 `assert started.is_set()` under a patched+ # _SWITCH_BUDGET_S = 0.3, against a cold path the file itself records at 1.98s -- and+ # like test_tunnel_safe_long_post, the assertion is on a RESULT rather than on a+ # duration, so the scan below cannot reach it. State-mutating as well: it writes+ # keepwarm globals directly at :2628-2630. It has already failed this way once on+ # 3.13 while 3.10 passed the same commit.+ (+ "tests/test_media_auto_switch.py",+ ":1101 asserts started.is_set() under a patched 0.3s budget on a 1.98s cold path, "+ "and writes keepwarm globals at :2628-2630",+ ),]# What the scan above does NOT cover, recorded because the gap is structural rather than a missing case. It finds@@ -163,14 +232,12 @@ BACKEND_ISOLATED = [# vCPUs it measures the scheduler as much as the code. Above it there is enough headroom to survive being descheduled.# Twenty-two backend files assert some elapsed bound and serialising all of them would give back most of what -n 4# buys, so the line is drawn where the measurement stops being about the code.-BACKEND_MARKER = "--ignore=tests/test_studio_api.py"+TIGHT_BOUND_S = 0.1def _over_the_backend(command: str) -> bool:- return BACKEND_MARKER in command-+ return any(selection in command for selection in _selections("pytest"))-TIGHT_BOUND_S = 0.1BACKEND_TESTS = Path(__file__).resolve().parents[2] / "studio" / "backend" / "tests"_CLOCKS = ("monotonic", "perf_counter", "process_time", "time")@@ -380,7 +447,7 @@ def _fragile_timing_asserts(path: Path) -> list:@pytest.mark.parametrize("path, reason", ISOLATED, ids = [p for p, _ in ISOLATED])def test_an_isolated_path_is_ignored_by_every_parallel_pytest_run(path, reason):- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8")):+ for command in _pytest_commands():if " -n " not in f" {command} " or not _over_the_repo_tests(command):continueassert not _collects(command, path), (@@ -394,7 +461,7 @@ def test_an_isolated_path_still_runs_in_a_serial_step(path, reason):"""Ignoring it is half the change. Without this, the tests silently stop running."""serial = [command- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))+ for command in _pytest_commands()if " -n " not in f" {command} "and re.search(rf"(?<![\w/]){re.escape(path)}(?![\w/])", command)]@@ -406,23 +473,42 @@ def test_an_isolated_path_still_runs_in_a_serial_step(path, reason):def test_the_command_scan_sees_the_parallel_run_and_the_serial_steps():"""Pin the parser: a scan that matched nothing would pass both tests above."""- commands = _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))+ commands = _pytest_commands()parallel = [command for command in commands if " -n " in f" {command} "]- assert len(parallel) == 4, (- f"expected four parallel pytest runs, the backend matrix and repo-cpu-tests' three "- f"shards, got {parallel}. If a job stopped running in parallel, or a shard was added "- f"or removed, say so here rather than letting this scan quietly cover fewer runs."+ assert len(parallel) == 6, (+ f"expected six parallel pytest runs, three shards of the backend matrix and three "+ f"of repo-cpu-tests, got {parallel}. If a job stopped running in parallel, or a "+ f"shard was added or removed, say so here rather than letting this scan quietly "+ f"cover fewer runs.")root = [command for command in parallel if _over_the_repo_tests(command)]assert len(root) == 3, (f"expected the three repo-root shards, got {root}. The isolation checks above apply "f"to those, and a scan that matched none of them would pass on nothing.")+ backend = [command for command in parallel if _over_the_backend(command)]+ assert len(backend) == 3, (+ f"expected the three backend shards, got {backend}. Same reason: the backend "+ f"isolation checks apply to those."+ )+ # The two jobs are told apart by whose matrix a command's paths came out of, so the sets have to be disjoint or+ # each job's isolation rules would be applied to the other's runs -- which is how tests/studio/... would come to be+ # asked of a run whose working directory is studio/backend.+ assert not set(root) & set(+ backend+ ), f"a command reads as belonging to both jobs: {set(root) & set(backend)}"+ assert len(parallel) == len(root) + len(backend), (+ f"a parallel run belongs to neither job's matrix, so nothing below checks it: "+ f"{[command for command in parallel if command not in root + backend]}"+ )# The line joins and the matrix substitution both have to be resolved, or a shard command reads as# `pytest ${{ matrix.selection }} -q` with no paths at all and the first test above passes on nothing.- assert all("${{" not in command for command in root)+ assert all("${{" not in command for command in root + backend)assert any("--ignore=" in command for command in root)assert {command for command in root} == set(root), "a shard selection appears twice"+ assert len(set(backend)) == 3, "a backend shard selection appears twice"+ # Non-vacuous for the backend side too: the shards between them must reach the backend suite.+ assert any(_collects(command, "tests/test_account_contract.py") for command in backend)# Non-vacuous the other way too: the shards between them must reach the repo's test root, or "not collected by any# parallel run" would be true of every path in the repo.assert any(_collects(command, "tests/test_model_registry.py") for command in root)@@ -437,16 +523,13 @@ def test_the_backend_matrix_still_runs_in_parallel():depends on the order it runs in. Asserted here because dropping the flag would show uponly as CI slowly getting slower again, which nothing reports."""- backend = [- command- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))- if "--ignore=tests/test_studio_api.py" in command- ]+ backend = [command for command in _pytest_commands() if _over_the_backend(command)]assert backend, "the backend matrix pytest step is gone or was renamed past this scan"- assert " -n " in f" {backend[0]} ", (- f"the backend matrix leg is running serially again, which costs about 17 minutes "- f"per leg on every pull request and every push to main: {backend[0]}"- )+ for command in backend:+ assert " -n " in f" {command} ", (+ f"a backend matrix shard is running serially again, which costs about 17 "+ f"minutes on every pull request and every push to main: {command}"+ )@pytest.mark.parametrize("path, reason", BACKEND_ISOLATED, ids = [p for p, _ in BACKEND_ISOLATED])@@ -459,14 +542,17 @@ def test_a_backend_isolated_path_is_ignored_by_the_parallel_run(path, reason):"""parallel = [command- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))+ for command in _pytest_commands()if " -n " in f" {command} " and _over_the_backend(command)]assert parallel, "the backend parallel run is gone or was renamed past this scan"- assert f"--ignore={path}" in parallel[0], (- f"{path} ({reason}) is back in the backend parallel run, where its measurements "- f"compare a descheduled worker against an undescheduled one: {parallel[0]}"- )+ # Asked of EVERY shard, as "does this run reach the path" rather than "does it spell+ # this --ignore", so a shard that quietly grew its own root still has to get past it.+ for command in parallel:+ assert not _collects(command, path), (+ f"{path} ({reason}) is back in a backend parallel run, where its measurements "+ f"compare a descheduled worker against an undescheduled one: {command}"+ )@pytest.mark.parametrize("path, reason", BACKEND_ISOLATED, ids = [p for p, _ in BACKEND_ISOLATED])@@ -474,7 +560,7 @@ def test_a_backend_isolated_path_still_runs_serially(path, reason):"""Ignoring it is half the change; without this it runs nowhere and the job is green."""serial = [command- for command in _pytest_commands(WORKFLOW.read_text(encoding = "utf-8"))+ for command in _pytest_commands()if " -n " not in f" {command} "and re.search(rf"(?<![\w/]){re.escape(path)}(?![\w/])", command)]@@ -487,9 +573,8 @@ def test_a_backend_isolated_path_still_runs_serially(path, reason):def test_every_tight_elapsed_bound_is_isolated():"""The rule, applied by scanning rather than by memory.- Two of the entries above were found by review rather than by CI: they passed on- staging and would have flaked later. A new test asserting a 20ms bound would do the- same. This finds them, so adding one forces the isolation instead of buying a flake.+ Two entries above were found by review, not CI: they passed on staging and would have+ flaked later. This finds them, so adding one forces the isolation instead of a flake."""isolated = {path for path, _ in BACKEND_ISOLATED}stray = {}@@ -579,32 +664,24 @@ def test_the_scan_finds_all_three_shapes(tmp_path):)assert not _fragile_timing_asserts(roomy), _fragile_timing_asserts(roomy)- # Deliberately nothing about the live suite here. The synthetic files above already- # exercise every shape the scanner knows, and a "the live suite still has some"- # assertion would turn cleaning the last one into a failure, which is the same- # dependency on real files this rewrite exists to remove.+ # Deliberately nothing about the live suite: the synthetic files cover every shape, and+ # asserting the suite still has some would turn cleaning the last one into a failure.def test_an_isolated_file_never_shadows_an_installed_library_with_a_stub():"""A stub may stand in for a MISSING library, never for an installed one.- `sys.modules.setdefault("httpx", stub)` reads as deferring to the real library and- does not: sys.modules holds what has been IMPORTED, not what is installed, so in a- process where nothing has touched httpx yet the stub wins and shadows it for the rest- of the session. These stubs carry no Response, starlette.testclient reads- httpx.Response at import, and every module collected afterwards that reaches- fastapi.testclient or routes.inference dies on it.-- In a 26,000-test run something always imports httpx first, so this was invisible for- as long as the suite ran as one process. The serial step collects ten files and- nothing else, and the 3.10 leg failed collection on two of them the first time it- ran.-- Scoped to the isolated files on purpose. Roughly fifty other backend modules stub- structlog the same way, and they are load-bearing in a run that also imports the real- one; rewriting them is a separate change with its own risk, and the full parallel run- is not the process where a small file list makes the shadowing decisive. What has to- hold here is that anything moved OUT of that run stands on its own.+ `sys.modules.setdefault("httpx", stub)` reads as deferring to the real library and does+ not: sys.modules holds what has been IMPORTED, not what is installed, so where nothing+ has touched httpx yet the stub wins for the rest of the session. These stubs carry no+ Response, starlette.testclient reads httpx.Response at import, and every module after it+ reaching fastapi.testclient or routes.inference dies on it. In a 26,000-test run+ something always imports httpx first, so this stayed invisible while the suite was one+ process; the serial step collects ten files, and the 3.10 leg failed collection on two.++ Scoped to the isolated files on purpose: ~fifty other backend modules stub structlog the+ same way and are load-bearing in a run that also imports the real one. What has to hold+ here is that anything moved OUT of the parallel run stands on its own."""offenders = {}for name, _reason in BACKEND_ISOLATED:diff --git a/tests/test_ci_backend_pytest_shards.py b/tests/test_ci_backend_pytest_shards.pynew file mode 100644index 00000000000..d70b0e9ec7e--- /dev/null+++ b/tests/test_ci_backend_pytest_shards.py@@ -0,0 +1,395 @@+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0++"""Guards that the sharded backend `pytest` job still runs every test it used to.++`(Python 3.13)` was one job discovering all of studio/backend/tests; it is now three shards+on three runners, and the whole risk is a file landing in no shard: the job goes green+faster having checked less, and nothing says so.++So the split is not three allowlists. studio/backend/tests is flat (923 files, plus 17 under+multi_account), so there are no directory roots to divide and the division is by the first+letter after `test_`. Shards 1 and 2 name their range and exclude everything else; shard 3 is+"tests/ except those two ranges", which makes exactly-one-shard a property of the shape+rather than of anyone remembering to edit the workflow. These tests hold that shape, and hold+that the inherited selection -- the -k filter, the two timeouts, and the twelve files the+serial step reruns -- came through intact.++All three shards root at `tests/` and differ only in ignores, because `--ignore=FILE` does+not filter a file passed as an explicit argument (checked against pytest): a shard rooted at+a shell-expanded glob would name the twelve serial files and run them anyway.++The per-test-id partition was checked directly when the split landed, via --junitxml diffs:+13,073 + 17,483 + 12,053 = 42,609 against the baseline's 42,609, no overlap, no gap, every id+in the same state bar three failure-to-passed flips, none caused by the split (two flip+between two runs of the UNSHARDED selection over the same tree; the third asserts four+concurrent sizings cannot all claim the same 8GB, which fails on a loaded box). That measured+one tree; these tests keep it true of the next one.+"""++import fnmatch+from pathlib import Path++import pytest+import yaml+++REPO_ROOT = Path(__file__).resolve().parents[1]++_BACKEND_CI = REPO_ROOT / ".github" / "workflows" / "studio-backend-ci.yml"+_BACKEND_TESTS = REPO_ROOT / "studio" / "backend" / "tests"++# Named once so the tests can say "and only there", and so a rename fails here rather than+# silently weakening them.+_CATCH_ALL = "rest"++# The twelve files the parallel run ignores and the serial step reruns. Read out of the+# workflow rather than listed here, so moving one between the two cannot make it look+# dropped, as tests/test_ci_repo_cpu_shards.py does for its isolated paths.+_SERIAL_STEP = "Backend tests that cannot share a worker"++# Files this job runs nowhere: a lost file would otherwise look like one of these. Only one,+# predating the split -- the thirteenth --ignore on the parallel step, which no step reruns.+_NOT_IN_ANY_SHARD = {+ "tests/test_studio_api.py": (+ "end-to-end against a live model and a GGUF download, which a GPU-less runner "+ "cannot do; the workflow's own header says so"+ ),+}+++def _backend_ci() -> dict:+ return yaml.safe_load(_BACKEND_CI.read_text(encoding = "utf-8"))+++def _job() -> dict:+ return _backend_ci()["jobs"]["pytest"]+++def _parse_selection(tokens: list) -> tuple:+ """(roots, ignores, ignore globs) out of a pytest argument list."""+ roots, ignores, globs = [], [], []+ index = 0+ while index < len(tokens):+ token = tokens[index]+ if token.startswith("--ignore-glob="):+ globs.append(token.split("=", 1)[1])+ elif token.startswith("--ignore="):+ ignores.append(token.split("=", 1)[1].rstrip("/"))+ elif token == "--deselect":+ index += 1 # a test id, not a path+ elif token.startswith("-"):+ pass+ elif token.startswith("tests/") or token == "tests":+ roots.append(token.rstrip("/"))+ index += 1+ return roots, ignores, globs+++def _shards() -> dict:+ """{shard name: (roots, ignores, ignore globs)} for the command each shard really runs.++ The matrix carries only what the shards DIFFER by. The thirteen --ignore flags they+ all share sit on the step, so a model built from the matrix alone would have the+ catch-all shard collecting the twelve files the serial step reruns -- it does not, and+ reporting that it does would either fail honestly-written guards or push the flags into+ three copies that have to agree. Both halves of the command, then.++ The 3.11 floor-spot-check leg is not a shard of anything and carries no selection, so+ it is not in here.+ """+ _, shared_ignores, shared_globs = _parse_selection(_shared_pytest_step()["run"].split())+ shards = {}+ for entry in _job()["strategy"]["matrix"]["include"]:+ if "selection" not in entry:+ continue+ roots, ignores, globs = _parse_selection(entry["selection"].split())+ shards[entry["shard"]] = (roots, ignores + shared_ignores, globs + shared_globs)+ return shards+++def _covers(prefix: str, path: str) -> bool:+ prefix = prefix.rstrip("/")+ return path == prefix or path.startswith(prefix + "/")+++def _glob_hits(pattern: str, path: str) -> bool:+ """pytest's own --ignore-glob rule, asked of a path relative to studio/backend.++ pytest matches with `_pytest.pathlib.fnmatch_ex`, which fnmatches the ABSOLUTE path+ against the pattern with `*/` prepended when the pattern contains a separator and is+ relative. fnmatch's `*` crosses `/`, so the prepended prefix is free and matching the+ tail of the relative path asks the same question.+ """+ return fnmatch.fnmatch(path, pattern) or fnmatch.fnmatch(path, f"*/{pattern}")+++def _claiming_shards(path: str, shards: dict) -> list:+ """Which shards would collect `path`, by pytest's own root-and-ignore rules."""+ claiming = []+ for name, (roots, ignores, globs) in shards.items():+ if any(_covers(ignore, path) for ignore in ignores):+ continue+ if any(_glob_hits(pattern, path) for pattern in globs):+ continue+ if any(_covers(root, path) for root in roots):+ claiming.append(name)+ return sorted(claiming)+++def _shared_pytest_step() -> dict:+ for step in _job()["steps"]:+ if "${{ matrix.selection }}" in str(step.get("run", "")):+ return step+ raise AssertionError("the sharded pytest step is gone from the backend pytest job")+++def _serial_paths() -> list:+ """The files the serial step names, read out of the step itself."""+ for step in _job()["steps"]:+ if step.get("name") == _SERIAL_STEP:+ return [token for token in str(step["run"]).split() if token.startswith("tests/")]+ raise AssertionError(f"the {_SERIAL_STEP!r} step is gone from the backend pytest job")+++def _test_files() -> list:+ return sorted(+ str(path.relative_to(_BACKEND_TESTS.parent))+ for path in _BACKEND_TESTS.rglob("test_*.py")+ if "__pycache__" not in path.parts+ )+++class TestEveryTestFileLandsInExactlyOneShard:+ def test_no_file_is_dropped_or_run_twice(self):+ shards = _shards()+ serial = _serial_paths()+ dropped, doubled = [], []+ for path in _test_files():+ if path in serial or path in _NOT_IN_ANY_SHARD:+ continue # rerun by the serial step, or excluded from the job on purpose+ claiming = _claiming_shards(path, shards)+ if not claiming:+ dropped.append(path)+ elif len(claiming) > 1:+ doubled.append((path, claiming))+ assert not dropped, (+ "these files are collected by no shard, so the backend pytest job no longer "+ f"runs them: {dropped}"+ )+ assert not doubled, f"these files are collected by more than one shard: {doubled}"++ @pytest.mark.parametrize("path", sorted(_NOT_IN_ANY_SHARD))+ def test_the_files_no_shard_runs_are_the_expected_ones(self, path):+ """Excluded on purpose, and from every shard rather than from one: without this, a+ dropped file and a deliberately-skipped one look identical to the test above."""+ claiming = _claiming_shards(path, _shards())+ assert (+ claiming == []+ ), f"{path} is {_NOT_IN_ANY_SHARD[path]}, but shard(s) {claiming} now collect it"+ assert (+ _BACKEND_TESTS.parent / path+ ).is_file(), f"{path} no longer exists, so this exclusion is stale"++ @pytest.mark.parametrize(+ "path, expected",+ [+ ("tests/test_apple.py", "a-k"),+ ("tests/test_kiwi.py", "a-k"),+ ("tests/test_lemon.py", "l-r"),+ ("tests/test_raisin.py", "l-r"),+ ("tests/test_squash.py", _CATCH_ALL),+ ("tests/test_yam.py", _CATCH_ALL),+ ],+ )+ def test_a_brand_new_file_is_claimed_by_exactly_one_shard(self, path, expected):+ """The property the split exists for: a file nobody has heard of yet runs, in one+ place, without a workflow edit."""+ claiming = _claiming_shards(path, _shards())+ assert claiming == [+ expected+ ], f"{path} must be collected by {expected} and only there; got {claiming}"++ @pytest.mark.parametrize(+ "path",+ [+ "tests/a_directory_added_tomorrow/test_new.py",+ "tests/multi_account/test_alice_bob_matrix.py",+ "tests/test_Zebra.py",+ "tests/test_9_regression.py",+ "tests/an_odd_name_test.py",+ ],+ ids = ["new-subdir", "existing-subdir", "uppercase", "digit", "underscore-test-suffix"],+ )+ def test_the_names_the_ranges_do_not_describe_land_in_the_catch_all(self, path):+ """Why shards 1 and 2 exclude their complement rather than name their range.++ Not every name starts with a lowercase letter: a subdirectory, an uppercase or digit+ first character, and pytest's other discovery pattern `*_test.py` are outside both+ ranges, and each would be claimed by BOTH ranged shards had they said "not l-z" and+ "not a-k". Only the catch-all may be open-ended, which is what makes exactly-one hold+ for names nobody anticipated.+ """+ claiming = _claiming_shards(path, _shards())+ assert claiming == [_CATCH_ALL], (+ f"{path} is not described by either range, so it must land in the "+ f"{_CATCH_ALL!r} shard and only there; got {claiming}"+ )++ def test_a_name_matching_both_discovery_patterns_falls_out_of_every_shard(self):+ """`test_api_test.py` matches `test_*.py` AND `*_test.py`, and lands nowhere.++ The ranged shards drop it for the suffix, the catch-all for the `test_[a-r]*` range;+ measured against real pytest, all three collect it zero times. Fixing it would need+ the suffix rule to mean "ends in _test.py unless it starts with test_", which fnmatch+ cannot say: diverging per prefix character over-consumes `te_test.py` into BOTH ranged+ shards (tried, and this guard caught it), and completing it needs nine globs per shard+ for a shape this repo does not use. Forbidden by name below instead.+ """+ assert _claiming_shards("tests/test_api_test.py", _shards()) == [], (+ "this is a known limitation; if it now lands in a shard the patterns have "+ "changed and the naming rule below can be relaxed"+ )++ def test_a_subdirectory_named_like_a_test_file_falls_out_of_every_shard(self):+ """The one shape the catch-all cannot absorb. Recorded, not hidden.++ fnmatch's `*` crosses `/` and fnmatch_ex matches the whole path, so the catch-all's+ `tests/test_[a-r]*.py` also excludes `tests/test_api/test_auth.py`, which the ranged+ shards already exclude via `tests/*/*`. Measured against real pytest: zero collections+ in all three. Both obvious repairs were measured and fail -- `[!/]` cannot stop `*`+ crossing a separator, and an extra root does not bypass --ignore-glob -- so the test+ below enforces the invariant by name instead.+ """+ assert _claiming_shards("tests/test_api/test_auth.py", _shards()) == [], (+ "this is a known limitation of the split; if it now lands in a shard the "+ "patterns have changed and the naming rule below can be dropped"+ )++ def test_no_test_subdirectory_is_named_like_a_test_file(self):+ """Enforces the invariant the split depends on, so the hole above stays unreachable."""+ dirs = sorted(+ path.name+ for path in _BACKEND_TESTS.iterdir()+ if path.is_dir() and path.name.startswith("test_")+ )+ assert not dirs, (+ f"{dirs} would be collected by no shard, because the catch-all's "+ "tests/test_[a-r]*.py excludes nested paths too (fnmatch * crosses /). Rename "+ "the directory, or give the catch-all an explicit root for it."+ )+ hybrids = sorted(+ path.name+ for path in _BACKEND_TESTS.iterdir()+ if path.is_file() and path.name.startswith("test_") and path.name.endswith("_test.py")+ )+ assert not hybrids, (+ f"{hybrids} match both default discovery patterns, so the ranged shards drop "+ "them for the _test.py suffix and the catch-all drops them for the test_[a-r]* "+ "range, leaving them in no shard. Drop one of the two markers from the name."+ )+++class TestTheSplitKeepsTheSelectionItInherited:+ def test_every_shard_keeps_the_marker_filter(self):+ """The environment-specific deselections: live GPU introspection and a real+ llama.cpp process, none of which exist on a GPU-less runner. The filter is on the+ shared step rather than per shard, so it is checked there."""+ run = _shared_pytest_step()["run"]+ for fragment in (+ "not llama_cpp_load_progress_live",+ "not TestGpuAutoSelection",+ "not TestPreSpawnGpuResolution",+ "not TestPerGpuFitGuardAllCounts",+ "not TestTransformersIntrospection",+ "not test_returns_cuda_when_cuda_available",+ "not test_calls_cuda_cache_when_cuda",+ ):+ assert fragment in run, f"the -k filter lost {fragment!r}"++ def test_every_shard_keeps_the_timeouts(self):+ """A shard that loses these reports "cancelled" and names nothing, which is what+ #9515 / #9530 were about."""+ run = _shared_pytest_step()["run"]+ assert "--timeout=330" in run+ assert "timeout --signal=INT --kill-after=60" in run++ def test_the_serial_files_are_ignored_by_every_shard(self):+ """The twelve cannot share a worker, and sharding does not change that: three+ runners are still four workers each. They are held out by the shared --ignore list+ on the step, so this asks the question of the shard AND the step together."""+ run = _shared_pytest_step()["run"]+ shards = _shards()+ for path in _serial_paths():+ assert f"--ignore={path}" in run, (+ f"{path} is rerun by the serial step and no longer ignored by the parallel "+ f"one, so it runs twice and its timing assertions run under four workers"+ )+ claiming = _claiming_shards(path, shards)+ assert (+ claiming == []+ ), f"{path} is rerun by the serial step and shard(s) {claiming} collect it too"++ def test_the_serial_step_runs_in_one_process_in_one_shard(self):+ """Splitting it would put its relative timings back on two machines, and running it+ on every shard would run it three times."""+ for step in _job()["steps"]:+ if step.get("name") != _SERIAL_STEP:+ continue+ assert " -n " not in f" {step['run']} ", "the serial step is running under xdist"+ condition = str(step.get("if", ""))+ assert "matrix.shard ==" in condition, (+ f"the serial step is no longer pinned to one shard, so it runs once per "+ f"shard: {condition!r}"+ )+ pinned = [name for name in _shards() if f"matrix.shard == '{name}'" in condition]+ assert len(pinned) == 1, f"expected exactly one shard named in {condition!r}"+ return+ raise AssertionError(f"the {_SERIAL_STEP!r} step is gone")++ def test_the_serial_step_still_runs_the_same_twelve_files(self):+ """Not eleven and not thirteen. Whether any of them could go back into the parallel+ run was audited separately and the answer was no, so the set is carried across the+ split unchanged; a file leaving it here is a coverage change wearing a refactor's+ clothes."""+ paths = _serial_paths()+ assert len(paths) == 12, f"the serial step runs {len(paths)} files, not 12: {paths}"+ assert len(set(paths)) == 12, f"the serial step names a file twice: {paths}"+ missing = [path for path in paths if not (_BACKEND_TESTS.parent / path).is_file()]+ assert not missing, f"the serial step names files that do not exist: {missing}"+++class TestTheGuardIsNotVacuous:+ def test_a_gap_is_reported(self):+ """Shard 3 stops being a catch-all and starts being an allowlist."""+ broken = {+ "a-k": (["tests/"], [], ["tests/*/*", "tests/*_test.py", "tests/test_[!a-k]*.py"]),+ "l-r": (["tests/"], [], ["tests/*/*", "tests/*_test.py", "tests/test_[!l-r]*.py"]),+ _CATCH_ALL: (["tests/test_squash.py"], [], []),+ }+ assert _claiming_shards("tests/test_yam.py", broken) == []+ assert _claiming_shards("tests/multi_account/test_alice_bob_matrix.py", broken) == []++ def test_an_overlap_is_reported(self):+ """The mistake the ranges are written to avoid: shard 1 excluding `l-z` rather than+ excluding `not a-k`, which leaves every name outside a-z owned by both."""+ broken = {+ "a-k": (["tests/"], [], ["tests/*/*", "tests/test_[l-z]*.py"]),+ "l-r": (["tests/"], [], ["tests/*/*", "tests/test_[a-k]*.py", "tests/test_[s-z]*.py"]),+ _CATCH_ALL: (["tests/"], [], ["tests/test_[a-r]*.py"]),+ }+ # All three, in fact: a name outside a-z is outside every range, so the catch-all+ # takes it as well and the ranged shards no longer exclude it. The names in a-z are+ # still partitioned correctly, which is why this is the mistake that survives review.+ assert _claiming_shards("tests/test_Zebra.py", broken) == ["a-k", "l-r", _CATCH_ALL]+ assert _claiming_shards("tests/test_apple.py", broken) == ["a-k"]+ assert _claiming_shards("tests/test_monkey.py", broken) == ["l-r"]++ def test_the_glob_rule_matches_pytest(self):+ """`tests/test_[a-k]*.py` must not reach below a subdirectory, or shard 3 would lose+ multi_account to a shard that never names it. Measured against pytest directly when+ this landed; asserted here so a rewrite of _glob_hits cannot quietly change it."""+ assert _glob_hits("tests/test_[a-k]*.py", "tests/test_apple.py")+ assert not _glob_hits("tests/test_[a-k]*.py", "tests/multi_account/test_alice.py")+ assert _glob_hits("tests/*/*", "tests/multi_account/test_alice.py")+ assert not _glob_hits("tests/*/*", "tests/test_apple.py")