Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(tests): make codex scheduler stress test machine-independent

chopratejasf82d61b1636c
diff --git a/tests/test_codex_ws_compression_scheduler.py b/tests/test_codex_ws_compression_scheduler.py
index 4d55fdb5b..eed3fe046 100644
--- a/tests/test_codex_ws_compression_scheduler.py
+++ b/tests/test_codex_ws_compression_scheduler.py
@@ -286,24 +286,31 @@ def test_concurrent_compression_has_no_semaphore_tail() -> None:
p99 = elapsed[int(len(elapsed) * 0.99)]
errors = [r for r in results if r.error]
- # Pre-fix baseline on the same machine (12-CPU, 30c × 30f):
- # p50 91ms, p99 2433ms, wall 7.5s.
- # Post-fix targets from the design doc — these are the regression
- # ratchet:
- assert not errors, f"Got {len(errors)} errors; first: {errors[0].error}"
- assert p99 < 1000, (
- f"p99 per-frame elapsed_ms = {p99:.0f}; expected < 1000 after the "
- f"semaphore fix (pre-fix baseline was 2433). Either the fix "
- f"regressed or your machine is much slower than expected."
+ # Print full distribution so CI logs always show numbers — useful
+ # both for diagnosing failures and tracking drift across runs.
+ print(
+ f"\n[stress] frames={len(results)} wall={wall_s:.2f}s "
+ f"p50={p50:.0f}ms p99={p99:.0f}ms ratio={p99 / max(p50, 1):.2f}× errors={len(errors)}"
)
- # Contention-tail ratio test. Pre-fix this was 27× (2433/91); the
- # fix should bring it under 5×.
- assert p99 < max(p50 * 5, 500), (
- f"p99/p50 ratio is {p99 / max(p50, 1):.1f}× (p50={p50:.0f}, "
- f"p99={p99:.0f}). Expected < 5× — a higher ratio means the "
- f"contention tail is back."
+
+ # The KEY regression assertion is the *ratio*, not absolute latency.
+ # The bug being guarded against creates a bimodal latency
+ # distribution (most fast, some catastrophic) via the deleted
+ # `_CODEX_WS_UNIT_ROUTER_SEMAPHORE`. Pre-fix on a 12-CPU dev box:
+ # p50=91ms, p99=2433ms → ratio=27×. CI runners are 5–50× slower in
+ # absolute terms (observed p99=214,000ms during the first attempt at
+ # this test) but the contention pattern is invariant — if the
+ # semaphore tail comes back, the ratio explodes regardless of CPU
+ # speed. The ratio is therefore the right machine-independent test.
+ #
+ # Absolute thresholds (p99<1000ms, wall<5s) intentionally removed —
+ # they're sane on dev hardware but force CI either to skip this test
+ # entirely or to use thresholds so loose they stop catching the bug.
+ # The ratio catches the bug everywhere.
+ assert not errors, f"Got {len(errors)} errors; first: {errors[0].error}"
+ ratio = p99 / max(p50, 1)
+ assert ratio < 5.0, (
+ f"p99/p50 ratio is {ratio:.1f}× (p50={p50:.0f}ms, p99={p99:.0f}ms). "
+ f"Expected < 5× — a higher ratio means the contention tail is back. "
+ f"Pre-fix baseline ratio was ~27× regardless of machine speed."
)
- # Wall-time sanity. 30c × 12 frames = 360 frames. With 32 frame-pool
- # workers and most frames < 200ms, total wall should be well under
- # the pre-fix 7.5s.
- assert wall_s < 5.0, f"Wall time {wall_s:.1f}s; expected < 5s after fix."