Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

Pair the linearity measurements, so a busy runner stops reading as quadratic (#10868)

Daniel Hanb3dfd088b5b4
diff --git a/studio/backend/tests/test_pr5624_regressions.py b/studio/backend/tests/test_pr5624_regressions.py
index edc13de474c..2e7c51fc989 100644
--- a/studio/backend/tests/test_pr5624_regressions.py
+++ b/studio/backend/tests/test_pr5624_regressions.py
@@ -230,31 +230,60 @@ def _growth(
) -> tuple[float, float, list]:
"""How much more `factor` times the input costs. Returns (ratio, big_seconds, big_result).
- Best-of-`repeats` at each size, not a mean: contention on a shared runner only ever ADDS
- time, so the minimum is the closest each size gets to its own true cost, and dividing one
- minimum by the other cancels the machine out.
+ The MEDIAN of `repeats` PAIRED ratios: each pair times the small input and then the big
+ one back to back, and the ratio is formed inside the pair before anything is aggregated.
This replaces an absolute ``elapsed < 1.0`` budget at one size. That budget read 0.20s on
a quiet runner and 1.41s on a busy one, so it failed for the wrong reason on unrelated
PRs, and it also sat close enough to the line that a REAL regression (#10507, which made
the R1 path quadratic again) only tipped it over some of the time. A ratio answers the
question these tests are named for: linear is ~`factor`, quadratic is ~`factor ** 2`.
+
+ It also replaces ``min(big over repeats) / min(small over repeats)``, which looked like
+ it cancelled the machine out and did not. Those two minima come from batches run at
+ different times, so a quiet window during the small batch and a busy one during the big
+ batch multiply instead of cancelling, and the quotient of two separately-taken minima is
+ not a minimum of anything. Measured on a 2-vCPU box under load, 15 trials per shape:
+
+ strategy R1 R1-distant GLM V3 >= 6.0
+ min/min max 8.69 max 8.23 max 8.27 max 8.59 7 of 60
+ paired median max 4.26 max 4.83 max 4.83 max 5.42 0 of 60
+
+ That 12% false-fail rate is not hypothetical: it is why this test failed on #10825 and
+ again on #10864 at 6.56, both times on branches that touch none of this code.
+
+ Pairing is what fixes it. Contention hits both halves of a pair roughly equally and
+ divides out, which is what the old comment claimed for the unpaired form. The median
+ then discards a pair that got unlucky, in either direction, rather than trusting one
+ reading.
+
+ Detection power is kept, which is the half worth checking before loosening anything. On
+ a synthetic path with a true 16x profile this reports 8 of 8 over the bar, same as the
+ old form. On the marginal 6.7x shape (#10832's partially fixed sweep) it reports 9 of 12
+ against the old form's 10 of 12, a difference well inside the noise at that sample size,
+ and that shape's sibling test measures 12.2x when broken, so the suite still catches it.
"""
+ import statistics as _statistics
import time as _time
- def cost(n: int) -> tuple[float, list]:
- text, best, result = build(n), None, None
- for _ in range(repeats):
- start = _time.perf_counter()
- result = parse_tool_calls_from_text(text)
- elapsed = _time.perf_counter() - start
- best = elapsed if best is None else min(best, elapsed)
- return best, result
-
- small, _ = cost(units)
- big, result = cost(units * factor)
- # A timer's own resolution must not read as superlinear growth on a very fast machine.
- return big / max(small, 1e-4), big, result
+ def once(text: str) -> tuple[float, list]:
+ start = _time.perf_counter()
+ result = parse_tool_calls_from_text(text)
+ return _time.perf_counter() - start, result
+
+ small_text, big_text = build(units), build(units * factor)
+ ratios, big, result = [], None, None
+ for _ in range(repeats):
+ small_elapsed, _ = once(small_text)
+ big_elapsed, result = once(big_text)
+ # The backstop below reads the BEST big time, not the worst. Contention only adds, so
+ # the minimum is the closest this size got to its own cost, and the backstop should
+ # fire on a path that is genuinely too slow rather than on a runner that stalled once.
+ big = big_elapsed if big is None else min(big, big_elapsed)
+ # A timer's own resolution must not read as superlinear growth on a very fast machine.
+ ratios.append(big_elapsed / max(small_elapsed, 1e-4))
+
+ return _statistics.median(ratios), big, result
def _assert_linear(