Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

Backend CI: run tensor and pipeline parallel gloo suites in separate shards (#16634)

Daniel Han624cd342c52b
diff --git a/.github/workflows/studio-backend-ci.yml b/.github/workflows/studio-backend-ci.yml
index 405899f54f2..66f86c82b3d 100644
--- a/.github/workflows/studio-backend-ci.yml
+++ b/.github/workflows/studio-backend-ci.yml
@@ -416,9 +416,14 @@ jobs:
# file, native tensor parallel plus pipeline parallel are about 51% of it, everything
# else about 49%. A *gloo* file named test_t[!p]* or test_tp[!_]* would match neither
# shard; tests/test_ci_repo_cpu_shards.py reports it as dropped.
- - shard: gloo-tp-pp
+ # Pipeline parallel then kept growing (#15655, #15656, #16167, #16417): by 2026-11-04 the
+ # combined shard ran 31 to 36 minutes per job and one run was interrupted at the 2100 s
+ # cap with 174 tests passed, so tensor and pipeline parallel run in separate shards.
+ - shard: gloo-tp
selection: >-
tests/test_tp_n*gloo*.py
+ - shard: gloo-pp
+ selection: >-
tests/test_pp*gloo*.py
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
diff --git a/tests/studio/test_backend_ci_parallel_isolation.py b/tests/studio/test_backend_ci_parallel_isolation.py
index 0c2bfe5aa8e..55465b6072c 100644
--- a/tests/studio/test_backend_ci_parallel_isolation.py
+++ b/tests/studio/test_backend_ci_parallel_isolation.py
@@ -342,15 +342,15 @@ def test_the_command_scan_sees_the_parallel_run_and_the_serial_steps():
"""Pin the parser: a scan that matched nothing would pass both tests above."""
commands = _pytest_commands()
parallel = [command for command in commands if " -n " in f" {command} "]
- assert len(parallel) == 12, (
- f"expected twelve parallel pytest runs, three shards of the backend matrix and nine "
+ assert len(parallel) == 13, (
+ f"expected thirteen parallel pytest runs, three shards of the backend matrix and ten "
f"of repo-cpu-tests, got {parallel}. If a job stopped running in parallel, or a "
f"shard was added or removed, say so here rather than letting this scan quietly "
f"cover fewer runs."
)
root = [command for command in parallel if _over_the_repo_tests(command)]
- assert len(root) == 9, (
- f"expected the nine repo-root shards, got {root}. The isolation checks above apply "
+ assert len(root) == 10, (
+ f"expected the ten repo-root shards, got {root}. The isolation checks above apply "
f"to those, and a scan that matched none of them would pass on nothing."
)
backend = [command for command in parallel if _over_the_backend(command)]