Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(agent): group batch inputs by encoded length (#294)

Alphin Tomdd9a4df7d102
diff --git a/README.md b/README.md
index 4b408e54..7d0ce4cd 100644
--- a/README.md
+++ b/README.md
@@ -50,7 +50,7 @@ Three checkpoints, and a `Router` that picks between them per request:
* **About 10x faster loading.** Checkpoints are built without the throwaway random weight initialisation, so `laya.load()` drops from about 22 s to about 2 s on CPU with bit-identical answers. This also skips the pass that crashed on Windows with Python 3.14 (#123). A second `Agent` for the same checkpoint reuses its parsed tokenizer and loads in about 0.5 s.
* **`import laya` no longer loads torch.** Routing, language detection and e-mail cleaning work in lightweight processes; torch loads on first use of a model.
-* **Batch scoring.** `agent.predict_batch(states, questions)` scores many states in shared forward passes, with answers identical to calling `predict` one state at a time. See [Batch Mode](#batch-mode-score-many-states-in-one-forward-pass).
+* **Batch scoring.** `agent.predict_batch(states, questions)` scores many states in shared forward passes and returns results in input order. See [Batch Mode](#batch-mode-score-many-states-in-one-forward-pass).
* **Faster paths, all opt-in.** `laya.load(..., fast=True)` uses a TileLang GPU fast path that matches the stock bf16 forward within rounding (see [GPU Fast Path](#gpu-fast-path-tilelang)). `Agent(compile=True)` enables `torch.compile`, and `laya.onnx_agent.ONNXAgent` runs an exported model on ONNX Runtime.
* **Run it your way, locally.** A self-hosted Jev-compatible HTTP server (`pip install "laya[serve]"`, then `laya-serve`, see [Self-Hosting](#self-hosting-http-server-jev-compatible)), a `laya` [command](#command-line) for quick local tests, an optional [MCP server](#mcp-server-optional) (`pip install "laya[mcp]"`), [LangChain and LangGraph](#langchain-and-langgraph-integration) routing, guardrails, triage and evaluation (`pip install "laya[langchain]"`), a Docker quickstart under `docs/docker.md`, and `laya-ts/`, a TypeScript package for Node and the browser that gives the same answers as the Python package.
* **Better routing.** Plain-ASCII Spanish, Italian, Portuguese and French, Brazilian Portuguese support text, CJK text containing Latin brand names, romanized Bangla and Azerbaijani now reach the multilingual checkpoint. Scripts the router has no range for no longer fall through to English, and URLs, e-mail addresses and dotted names no longer count as words. Checked on 20,000 English texts: at most 5 English sentences move, all quoting long native-script names.
@@ -458,18 +458,29 @@ a log slice — `predict_batch` packs them into shared forward passes:
states = [{"body": t} for t in ticket_texts] # a list of states
results = agent.predict_batch(states, questions) # one forward pass for the whole list
-# results[i] is exactly what agent.predict(states[i], questions) would return
+# results[i] corresponds to states[i], with the same output shape as predict
# Bound peak memory when the list (or the texts) are large — chunk into passes of N:
results = agent.predict_batch(states, questions, batch_size=64)
+
+# Reduce padding when input lengths vary; results still follow the original state order:
+results = agent.predict_batch(states, questions, batch_size=64, sort_by_length=True)
```
-Results are aligned with `states` by index and identical in shape to `predict`. Decisions match the
-one-at-a-time path exactly (numbers are bit-identical on CPU; on GPU they can differ in the 4th
-decimal because fp16 autocast reorders reductions across padding widths). Batching is a **GPU
+`sort_by_length=True` groups states by their longest encoded question row, after truncation.
+It looks ahead at most eight batches and reuses the encoded rows for sorting. This uses more temporary
+CPU memory for tokenized inputs, and takes effect only when `1 < batch_size < len(states)`.
+Benchmark it on your workload and backend: uniform lengths offer little benefit, and changed
+batch shapes can cause small floating-point differences, including near decision thresholds.
+Hooks still see states and final results in input order. The option is available on `Agent`.
+
+Results are aligned with `states` by index and identical in shape to `predict`. Changing batch
+shapes can introduce floating-point differences on CPU and GPU; check decision thresholds on
+your workload, particularly with mixed precision. Batching is a **GPU
throughput win** — on an RTX 5060 Ti, per-decision latency drops from ~10 ms one-by-one to ~1 ms
-batched (measured ~9–10×). On CPU the model is already compute-bound, so batching does not speed it
-up; use it there only for API convenience.
+batched (measured ~9–10×). On CPU, increasing batch size alone may not speed up inference;
+length grouping can help by reducing the padded work in a mixed-length workload. See the
+[CPU measurements and reproduction commands](research/README.md#length-batching).
---
diff --git a/docs/hooks/api.md b/docs/hooks/api.md
index ac5b3a84..5072179b 100644
--- a/docs/hooks/api.md
+++ b/docs/hooks/api.md
@@ -156,7 +156,7 @@ load(..., hooks=None, on_predict_start=None, on_predict_end=None,
agent.predict_batch(states, questions, batch_size=None,
hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
- max_len=None, head_max_len=None)
+ max_len=None, head_max_len=None, sort_by_length=False)
agent.system_one(state, questions,
hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
diff --git a/laya/agent.py b/laya/agent.py
index 330417de..19520f5f 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -602,7 +602,8 @@ class Agent(HookRegistry):
on_predict_start=None, on_predict_end=None,
hooks_raise: Optional[bool] = None,
max_len: Optional[int] = None,
- head_max_len: Optional[int] = None) -> List[Dict[str, Any]]:
+ head_max_len: Optional[int] = None,
+ sort_by_length: bool = False) -> List[Dict[str, Any]]:
"""Evaluate the same questions over many states, packing them into shared forward passes.
This is the throughput path. `system_one`/`predict` handle one state per forward pass; on a
@@ -623,6 +624,11 @@ class Agent(HookRegistry):
hooks_raise: Override the Agent's `hooks_raise` for this call.
max_len, head_max_len: Override the agent config for this call. A start hook may also
set `ctx.max_len` / `ctx.head_max_len` to shape the token budget.
+ sort_by_length: Group similarly sized encoded states within windows of eight batches
+ to reduce padding. Requires an explicit `batch_size` greater than one and
+ smaller than the number of states; otherwise it has no effect. Results retain
+ input order. This buffers up to eight batches of tokenized states instead of
+ one. Changing batch shapes can slightly change floating-point predictions.
Returns:
A list of per-state result dicts, each identical in shape to `system_one`'s output and
@@ -668,25 +674,37 @@ class Agent(HookRegistry):
overrides["head_max_len"] = ctx.head_max_len
results: List[Dict[str, Any]] = []
- for start in range(0, len(states), chunk):
- part = states[start:start + chunk]
- per_state_items = [self._encode_state(st, ids, internal, **overrides) for st in part]
-
- b = collate_items(per_state_items, self.tok.pad_token_id)
- logits, act = self._forward(b)
- att = b["attention_mask"]
-
- row = 0
- for items in per_state_items:
- nrows = len(items)
- n_tokens = int(att[row:row + nrows].sum())
- answers = self._decode_answers(logits, act, items, ids, internal, row)
- results.append({
- "model": "laya-rl-agent",
- "answers": answers,
- "usage": {"input_tokens": n_tokens, "output_tokens": 0},
- })
- row += nrows
+ reorder = sort_by_length and 1 < chunk < len(states)
+ # Bound tokenized lookahead independently of the full input size. Use
+ # actual post-truncation lengths, with each state's questions kept together.
+ window = chunk * 8 if reorder else chunk
+ for start in range(0, len(states), window):
+ part = states[start:start + window]
+ encoded = [self._encode_state(st, ids, internal, **overrides) for st in part]
+ order = list(range(len(encoded)))
+ if reorder:
+ order.sort(key=lambda i: max(len(item["ids"]) for item in encoded[i]))
+ window_results = [None] * len(encoded)
+ for offset in range(0, len(order), chunk):
+ indices = order[offset:offset + chunk]
+ per_state_items = [encoded[i] for i in indices]
+
+ b = collate_items(per_state_items, self.tok.pad_token_id)
+ logits, act = self._forward(b)
+ att = b["attention_mask"]
+
+ row = 0
+ for index, items in zip(indices, per_state_items):
+ nrows = len(items)
+ n_tokens = int(att[row:row + nrows].sum())
+ answers = self._decode_answers(logits, act, items, ids, internal, row)
+ window_results[index] = {
+ "model": "laya-rl-agent",
+ "answers": answers,
+ "usage": {"input_tokens": n_tokens, "output_tokens": 0},
+ }
+ row += nrows
+ results.extend(window_results)
ctx.results = results
except BaseException as exc:
ctx.error = exc
diff --git a/research/README.md b/research/README.md
index a9add1a5..44e9d04e 100644
--- a/research/README.md
+++ b/research/README.md
@@ -16,11 +16,51 @@ the numbers quoted in the main README — nothing here is imported by the `laya`
| `scripts/bench_local.py` | CPU sweep: MASSIVE intent across **all 51 languages**, plus typed-decisions on all three checkpoints |
| `scripts/bench_apps.py` | the six application workflows (support triage, email + phishing, guardrails, RAG relevance, moderation, model routing) plus the datasets where public Jev numbers exist |
| `scripts/bench_latency.py` | inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several `max_loaded` settings |
+| `scripts/bench_length_batching.py` | compare upstream contiguous batches with optional length sorting on synthetic tickets, including output consistency and optional fresh-process memory profiles (`psutil` required for memory mode) |
| `scripts/make_plots.py` | renders `assets/laya_benchmark.png` from the result JSONs |
Everything runs with `USE_TF=0` — `transformers` probes for TensorFlow at import, and when TF is
installed its abseil runtime can deadlock model construction on macOS/Python 3.9.
+## Length batching
+
+[Local CPU measurements](results/length_batching_cpu_20260924.json) compare the original
+`predict_batch` method at `1e28ac20c0896b1c37a744cd11f740eb98f8b178` with `sort_by_length=True`.
+On 10,000 synthetic English support tickets, the multilingual checkpoint took **1775.71 s
+before and 825.09 s after (2.15x)**, including tokenization, collation, inference and decoding.
+There were zero choice/action decision changes, zero usage mismatches, and a maximum returned
+numeric difference of **0.0001**. This is a throughput workload, not a labeled accuracy test.
+
+The machine was a Ryzen 9 5950X with 64 GiB RAM, Windows 11, CPU float32 and 16 Torch threads.
+An unrelated adaptation experiment was running on the GPU. Small tests alternate execution
+order over repeated rounds; the 1,000 and 10,000 input tests each have one full timing pair.
+The similar-length control showed no regression. GPU, MPS, compiled and TileLang performance
+were not measured. Results depend on input lengths, checkpoint, batch size and hardware.
+
+To reproduce, use a fresh Python 3.12 environment and the pinned model snapshot:
+
+```sh
+python -m pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cpu
+python -m pip install transformers==4.57.1 huggingface_hub==0.36.2 numpy==2.3.5 psutil==7.2.2
+python -c "from huggingface_hub import snapshot_download; print(snapshot_download('convaiinnovations/laya', revision='aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8', allow_patterns=['multilingual/*']) + '/multilingual')"
+```
+
+Replace `MODEL_DIR` below with the printed checkpoint directory. Start with `--count 16
+--rounds 2 --questions 3`, then increase the count. The full CPU comparison takes tens of minutes:
+
+```sh
+python research/scripts/bench_length_batching.py MODEL_DIR --count 10000 --batch-size 8 --threads 16 --rounds 1 --output length-10000.json
+```
+
+Memory measurements run each mode in a separate process after model warm-up. They sample
+process RSS every 10 ms during 64-input inference, excluding model-load transients; short
+allocation peaks can be missed. They do not measure the peak memory of the 10,000-input run:
+
+```sh
+python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-size 8 --threads 16 --memory-mode original --output memory-original.json
+python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-size 8 --threads 16 --memory-mode grouped --output memory-grouped.json
+```
+
## Results
| file | contents |
diff --git a/research/results/length_batching_cpu_20260924.json b/research/results/length_batching_cpu_20260924.json
new file mode 100644
index 00000000..6d80f7d7
--- /dev/null
+++ b/research/results/length_batching_cpu_20260924.json
@@ -0,0 +1,194 @@
+{
+ "model": "convaiinnovations/laya",
+ "revision": "aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8",
+ "upstream_baseline": "1e28ac20c0896b1c37a744cd11f740eb98f8b178",
+ "agent_source_sha256": "67b5dd563004bcc2a2413dfcc11168de070b9d16fc28eacf2eae1902618aa33b",
+ "hardware": "AMD Ryzen 9 5950X, 64 GiB RAM",
+ "device": "cpu",
+ "precision": "float32",
+ "platform": "Windows-11-10.0.26200-SP0",
+ "python": "3.12.10",
+ "torch": "2.9.1+cpu",
+ "transformers": "4.57.1",
+ "numpy": "2.3.5",
+ "huggingface_hub": "0.36.2",
+ "psutil": "7.2.2",
+ "method": "Warm model; elapsed prediction call includes tokenization, collation, forward and decoding. Input generation and model loading excluded. Repeated small runs alternate order; 1000 and 10000 inputs each have one full before/after pair.",
+ "workload": "Synthetic English support tickets, six templates with seeded varying context repetition and unique reference IDs. This is a padding/throughput workload, not a labeled accuracy benchmark.",
+ "host_load": "An unrelated adaptation experiment was running on the GPU; these runs used CPU only.",
+ "memory_method": "Separate fresh processes, warmed model, sampled process RSS every 10 ms during 64-input inference. Model-load transients excluded. Sampled peaks can miss short transients.",
+ "cases": [
+ {
+ "name": "english, mixed lengths",
+ "subfolder": "",
+ "states": 16,
+ "questions_per_state": 1,
+ "batch_size": 4,
+ "threads": 16,
+ "rounds": 2,
+ "workload": "mixed",
+ "before_seconds": [
+ 6.145877099996142,
+ 6.290177799994126
+ ],
+ "after_seconds": [
+ 4.295132200000808,
+ 4.140591199997289
+ ],
+ "speedup": 1.4742132132963337,
+ "parity": {
+ "decision_flips": 0,
+ "max_numeric_delta": 0.0,
+ "usage_mismatches": 0,
+ "results_equal": true
+ },
+ "before_padded_slots": 3964,
+ "after_padded_slots": 2540,
+ "before_peak_batch_slots": 1464,
+ "after_peak_batch_slots": 1464,
+ "states_sha256": "a2ed1e7467103a0850df1f80adb0ea12b82252ed7c6dfeb4ca79ec6a656f200c"
+ },
+ {
+ "name": "english, similar lengths",
+ "subfolder": "",
+ "states": 16,
+ "questions_per_state": 1,
+ "batch_size": 4,
+ "threads": 16,
+ "rounds": 3,
+ "workload": "uniform",
+ "before_seconds": [
+ 2.667264800002158,
+ 2.655638900003396,
+ 2.6071328000034555
+ ],
+ "after_seconds": [
+ 2.5969467000031727,
+ 2.5523272000064026,
+ 2.5502405000006547
+ ],
+ "speedup": 1.0404774513223596,
+ "parity": {
+ "decision_flips": 0,
+ "max_numeric_delta": 0.0,
+ "usage_mismatches": 0,
+ "results_equal": true
+ },
+ "before_padded_slots": 1600,
+ "after_padded_slots": 1580,
+ "before_peak_batch_slots": 404,
+ "after_peak_batch_slots": 404,
+ "states_sha256": "7e9a5edbc40f5bdfbc398dca672ec495f12401383e8a8b5ea2830fb9004a672b"
+ },
+ {
+ "name": "multilingual, three question types",
+ "subfolder": "multilingual",
+ "states": 16,
+ "questions_per_state": 3,
+ "batch_size": 4,
+ "threads": 16,
+ "rounds": 2,
+ "workload": "mixed",
+ "before_seconds": [
+ 6.941053100003046,
+ 7.0523381999955745
+ ],
+ "after_seconds": [
+ 4.3698996000021,
+ 4.389237299998058
+ ],
+ "speedup": 1.597576503228117,
+ "parity": {
+ "decision_flips": 0,
+ "max_numeric_delta": 0.0,
+ "usage_mismatches": 0,
+ "results_equal": true
+ },
+ "before_padded_slots": 11832,
+ "after_padded_slots": 7536,
+ "before_peak_batch_slots": 4380,
+ "after_peak_batch_slots": 4380,
+ "states_sha256": "a2ed1e7467103a0850df1f80adb0ea12b82252ed7c6dfeb4ca79ec6a656f200c"
+ },
+ {
+ "name": "multilingual, 1000 inputs",
+ "subfolder": "multilingual",
+ "states": 1000,
+ "questions_per_state": 1,
+ "batch_size": 8,
+ "threads": 16,
+ "rounds": 1,
+ "workload": "mixed",
+ "before_seconds": [
+ 179.09358500000235
+ ],
+ "after_seconds": [
+ 83.7743122000029
+ ],
+ "speedup": 2.1378102701987465,
+ "parity": {
+ "decision_flips": 0,
+ "max_numeric_delta": 0.0001000000000000445,
+ "usage_mismatches": 0,
+ "results_equal": false
+ },
+ "before_padded_slots": 305128,
+ "after_padded_slots": 149160,
+ "before_peak_batch_slots": 2968,
+ "after_peak_batch_slots": 2968,
+ "states_sha256": "c3b9032cbd29c4aec917735e8cff7916d1a96d968b780edfa2931c7d36b8888a"
+ },
+ {
+ "name": "multilingual, 10000 inputs",
+ "subfolder": "multilingual",
+ "states": 10000,
+ "questions_per_state": 1,
+ "batch_size": 8,
+ "threads": 16,
+ "rounds": 1,
+ "workload": "mixed",
+ "before_seconds": [
+ 1775.7104209000026
+ ],
+ "after_seconds": [
+ 825.0943344999978
+ ],
+ "speedup": 2.152130182757918,
+ "parity": {
+ "decision_flips": 0,
+ "max_numeric_delta": 0.00010000000000010001,
+ "usage_mismatches": 0,
+ "results_equal": false
+ },
+ "before_padded_slots": 3044328,
+ "after_padded_slots": 1505560,
+ "before_peak_batch_slots": 2976,
+ "after_peak_batch_slots": 2976,
+ "states_sha256": "35530c4a8e1fde6cd6256587be8f95e37f3cf8c70999268471eb6ebd39f589b4"
+ }
+ ],
+ "memory_profiles": [
+ {
+ "mode": "original",
+ "states": 64,
+ "batch_size": 8,
+ "threads": 16,
+ "sampled_peak_rss_bytes": 2028224512,
+ "start_rss_bytes": 2005831680,
+ "sample_interval_ms": 10,
+ "samples": 707,
+ "result_count": 64
+ },
+ {
+ "mode": "grouped",
+ "states": 64,
+ "batch_size": 8,
+ "threads": 16,
+ "sampled_peak_rss_bytes": 2030919680,
+ "start_rss_bytes": 2005139456,
+ "sample_interval_ms": 10,
+ "samples": 341,
+ "result_count": 64
+ }
+ ]
+}
diff --git a/research/scripts/bench_length_batching.py b/research/scripts/bench_length_batching.py
new file mode 100644
index 00000000..273f0d64
--- /dev/null
+++ b/research/scripts/bench_length_batching.py
@@ -0,0 +1,215 @@
+"""Compare contiguous and length-grouped real-model batches on synthetic tickets.
+
+This is a throughput experiment, not an accuracy benchmark. All timing includes
+tokenization, collation, inference and decoding, but excludes model loading.
+"""
+import argparse
+import ast
+import hashlib
+import json
+import os
+import platform
+import random
+import statistics
+import subprocess
+import sys
+import threading
+import time
+from pathlib import Path
+
+os.environ.setdefault("USE_TF", "0")
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import torch
+import transformers
+
+from laya import Agent
+from laya import agent as agent_module
+
+QUESTIONS = {
+ "team": {"type": "choice", "instructions": "Which team should handle this support ticket?",
+ "criteria": {"billing": "payments, charges and refunds", "technical": "bugs and outages",
+ "sales": "pricing and new purchases"}},
+ "urgent": {"type": "noul", "instructions": "Does the customer need urgent help?"},
+ "sentiment": {"type": "score", "instructions": "How satisfied is the customer?",
+ "criteria": ["very dissatisfied", "neutral", "very satisfied"]},
+}
+TICKETS = [
+ "I was charged twice. Please refund the duplicate payment today.",
+ "The app crashes at login and I cannot access my work. Please help immediately.",
+ "Could you share pricing for twenty seats? We are planning next quarter's budget.",
+ "Thank you for resolving my invoice problem. Everything works well now.",
+ "I am unable to upload a document. There is no rush; I can try tomorrow.",
+ "I need a quote for the enterprise plan. I am happy with the trial so far.",
+]
+DETAIL = "For context, I checked the account details and reviewed the earlier support conversation. "
+
+
+def states_for(count, workload):
+ rng = random.Random(912)
+ states = []
+ for i in range(count):
+ repeats = rng.choice([0, 0, 0, 1, 2, 4, 8, 20]) if workload == "mixed" else 2
+ states.append({"ticket": TICKETS[i % len(TICKETS)], "reference": i,
+ "context": DETAIL * repeats})
+ rng.shuffle(states)
+ return states
+
+
+def compare(before, after):
+ assert len(before) == len(after)
+ max_delta = 0.0
+ flips = 0
+ usage_mismatches = 0
+ def numeric(a, b):
+ nonlocal max_delta
+ if isinstance(a, dict):
+ assert a.keys() == b.keys()
+ for key in a:
+ numeric(a[key], b[key])
+ elif isinstance(a, (int, float)):
+ max_delta = max(max_delta, abs(a - b))
+ elif a != b:
+ assert isinstance(a, str) and isinstance(b, str)
+ for a, b in zip(before, after):
+ usage_mismatches += a["usage"] != b["usage"]
+ for qid, ans in a["answers"].items():
+ new = b["answers"][qid]
+ flips += (ans["action"]["act_probability"] >= 0.5) != (new["action"]["act_probability"] >= 0.5)
+ if ans["type"] == "choice":
+ flips += ans["choice"] != new["choice"]
+ elif ans["type"] == "noul":
+ flips += (ans["noul"] >= 0.5) != (new["noul"] >= 0.5)
+ else:
+ flips += max(ans["probabilities"], key=ans["probabilities"].get) != max(
+ new["probabilities"], key=new["probabilities"].get)
+ numeric(ans, new)
+ return {"decision_flips": flips, "max_numeric_delta": max_delta,
+ "usage_mismatches": usage_mismatches, "results_equal": before == after}
+
+
+def main():
+ p = argparse.ArgumentParser(description=__doc__)
+ p.add_argument("model")
+ p.add_argument("--count", type=int, default=32)
+ p.add_argument("--batch-size", type=int, default=8)
+ p.add_argument("--rounds", type=int, default=3)
+ p.add_argument("--threads", type=int, default=4)
+ p.add_argument("--device", choices=["cpu", "cuda"], default="cpu")
+ p.add_argument("--questions", type=int, default=1, choices=[1, 3])
+ p.add_argument("--workload", choices=["mixed", "uniform"], default="mixed")
+ p.add_argument("--output", required=True)
+ p.add_argument("--baseline", default="1e28ac20c0896b1c37a744cd11f740eb98f8b178")
+ p.add_argument("--memory-mode", choices=["original", "grouped"],
+ help="Profile one mode in a fresh process (requires psutil), then exit.")
+ args = p.parse_args()
+ if min(args.count, args.batch_size, args.rounds, args.threads) < 1:
+ p.error("count, batch size, rounds and threads must be positive")
+ torch.set_num_threads(args.threads)
+ torch.set_num_interop_threads(1)
+ if args.device == "cuda" and not torch.cuda.is_available():
+ p.error("CUDA requested but unavailable")
+ ag = Agent(args.model, device=args.device)
+ # Execute the actual upstream method, so added default-path overhead is included.
+ source = subprocess.check_output(["git", "show", args.baseline + ":laya/agent.py"], encoding="utf-8")
+ cls = next(n for n in ast.parse(source).body if isinstance(n, ast.ClassDef) and n.name == "Agent")
+ method = next(n for n in cls.body if isinstance(n, ast.FunctionDef) and n.name == "predict_batch")
+ scope = dict(vars(agent_module))
+ exec(compile(ast.Module(body=[method], type_ignores=[]), "<upstream predict_batch>", "exec"), scope) # noqa: S102
+ baseline = scope["predict_batch"].__get__(ag)
+ questions = dict(list(QUESTIONS.items())[:args.questions])
+ states = states_for(args.count, args.workload)
+ ag.predict_batch(states[:args.batch_size], questions, batch_size=args.batch_size)
+ def require_device():
+ if ag.device.type != args.device:
+ raise RuntimeError("Device fallback occurred; this benchmark run is invalid")
+ require_device()
+ if args.memory_mode:
+ # Run each mode in a fresh process: allocator retention from a previous mode
+ # otherwise makes RSS peaks incomparable. This excludes model-load transients.
+ import psutil
+ process = psutil.Process()
+ samples = [process.memory_info().rss]
+ stop = threading.Event()
+ def sample_rss():
+ while not stop.wait(0.01):
+ samples.append(process.memory_info().rss)
+ sampler = threading.Thread(target=sample_rss, daemon=True)
+ sampler.start()
+ try:
+ if args.memory_mode == "grouped":
+ results = ag.predict_batch(states, questions, batch_size=args.batch_size,
+ sort_by_length=True)
+ else:
+ results = baseline(states, questions, batch_size=args.batch_size)
+ require_device()
+ samples.append(process.memory_info().rss)
+ finally:
+ stop.set()
+ sampler.join()
+ report = {"args": vars(args), "start_rss_bytes": samples[0],
+ "sampled_peak_rss_bytes": max(samples), "sample_interval_ms": 10,
+ "samples": len(samples), "results": len(results),
+ "results_sha256": hashlib.sha256(json.dumps(results, sort_keys=True).encode()).hexdigest()}
+ Path(args.output).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+ print(json.dumps(report), flush=True)
+ return 0
+ original_forward = ag._forward
+ stats = {}
+ def measured_forward(batch):
+ stats["padded_slots"] += batch["input_ids"].numel()
+ stats["max_batch_slots"] = max(stats["max_batch_slots"], batch["input_ids"].numel())
+ start = time.perf_counter()
+ out = original_forward(batch)
+ stats["forward_seconds"].append(time.perf_counter() - start)
+ completed = len(stats["forward_seconds"])
+ if completed % 25 == 0:
+ print(json.dumps({"forward_batches": completed, "sorted": sorted_,
+ "elapsed_seconds": time.perf_counter() - pass_started}), flush=True)
+ return out
+ ag._forward = measured_forward
+ times = {False: [], True: []}
+ runs = []
+ gate_failed = False
+ for r in range(args.rounds):
+ outputs = {}
+ for sorted_ in ([False, True] if r % 2 == 0 else [True, False]):
+ stats = {"padded_slots": 0, "max_batch_slots": 0, "forward_seconds": []}
+ if args.device == "cuda":
+ torch.cuda.synchronize()
+ torch.cuda.reset_peak_memory_stats()
+ pass_started = start = time.perf_counter()
+ if sorted_:
+ outputs[sorted_] = ag.predict_batch(states, questions, batch_size=args.batch_size,
+ sort_by_length=True)
+ else:
+ outputs[sorted_] = baseline(states, questions, batch_size=args.batch_size)
+ seconds = time.perf_counter() - start
+ require_device()
+ if args.device == "cuda":
+ stats["peak_cuda_allocated_bytes"] = torch.cuda.max_memory_allocated()
+ times[sorted_].append(seconds)
+ run = {"round": r, "sorted": sorted_, "seconds": seconds, **stats}
+ runs.append(run)
+ print(json.dumps({k: v for k, v in run.items() if k != "forward_seconds"}), flush=True)
+ parity = compare(outputs[False], outputs[True])
+ print(json.dumps(parity), flush=True)
+ if parity["decision_flips"] or parity["usage_mismatches"] or parity["max_numeric_delta"] > 0.001:
+ print("CONSISTENCY GATE FAILED", flush=True)
+ gate_failed = True
+ break
+ report = {"args": vars(args), "torch": torch.__version__, "device": str(ag.device),
+ "transformers": transformers.__version__, "python": platform.python_version(),
+ "platform": platform.platform(), "processor": platform.processor(),
+ "states_sha256": hashlib.sha256(json.dumps(states, sort_keys=True).encode()).hexdigest(),
+ "agent_source_sha256": hashlib.sha256(Path(agent_module.__file__).read_bytes()).hexdigest(),
+ "runs": runs, "parity": parity, "consistency_gate_passed": not gate_failed,
+ "speedup": statistics.median(times[False]) / statistics.median(times[True])}
+ Path(args.output).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+ print(json.dumps({k: v for k, v in report.items() if k != "runs"}), flush=True)
+ return int(gate_failed)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tests/test_batch.py b/tests/test_batch.py
index 88d8c143..d5ab02e9 100644
--- a/tests/test_batch.py
+++ b/tests/test_batch.py
@@ -93,6 +93,103 @@ def make_fake():
return fake
+# --------------------------------------------------------------- length grouping and restoration
+def make_length_fake():
+ fake = make_fake()
+ fake.encoded = []
+ fake.batch_rows = []
+ fake.slots = 0
+ fake.encoded_at_forward = []
+
+ def encode(state, ids, internal, **overrides):
+ fake.encoded.append(state)
+ limit = overrides.get("max_len", 100)
+ # Question rows have different lengths; sorting must use the longest.
+ return [{"ids": [state["id"] + 1] * min(limit, size), "markers": [0, 1], "qtype": 2}
+ for size in (3, state["length"])]
+
+ def forward(batch):
+ fake.slots += batch["input_ids"].numel()
+ fake.batch_rows.append(batch["input_ids"][:, 0].tolist())
+ fake.encoded_at_forward.append(len(fake.encoded))
+ return batch["input_ids"][:, :1].numpy(), None
+
+ def decode(logits, act, items, ids, internal, offset):
+ return {qid: int(logits[offset + i, 0]) - 1 for i, qid in enumerate(ids)}
+
+ fake._encode_state = encode
+ fake._forward = forward
+ fake._decode_answers = decode
+ return fake
+
+
+states = [{"id": i, "length": [40, 5, 25, 8][i % 4]} for i in range(37)]
+plain, grouped = make_length_fake(), make_length_fake()
+expected = plain.predict_batch(states, QUESTIONS, batch_size=2)
+events = []
+actual = grouped.predict_batch(states, QUESTIONS, batch_size=2, sort_by_length=True,
+ on_predict_end=lambda ctx: events.append((ctx.results, ctx.usage)))
+check("length/answers and usage retain input order across windows", actual, expected)
+check("length/each state encoded once in input order", grouped.encoded, states)
+check_true("length/grouping reduces padded slots", grouped.slots < plain.slots)
+check("length/same number of forward passes", len(grouped.batch_rows), len(plain.batch_rows))
+check_true("length/batch size caps all forward passes", all(len(rows) <= 4 for rows in grouped.batch_rows))
+check("length/partial final batch", len(grouped.batch_rows[-1]), 2)
+check("length/first forward buffers at most eight batches", grouped.encoded_at_forward[0], 16)
+check("length/end hook sees restored results", events[0][0], expected)
+check("length/end hook usage excludes padding", events[0][1]["input_tokens"],
+ sum(3 + st["length"] for st in states))
+check("length/does not mutate caller states", [st["id"] for st in states], list(range(37)))
+
+# A hook may replace both inputs and the length budget before grouping occurs.
+def rewrite_length_inputs(ctx):
+ ctx.states = states
+ ctx.max_len = 10
+
+
+rewritten = make_length_fake().predict_batch(["unused"], QUESTIONS, batch_size=2,
+ sort_by_length=True, on_predict_start=rewrite_length_inputs)
+reference = make_length_fake().predict_batch(states, QUESTIONS, batch_size=2, max_len=10)
+check("length/hook rewrites and truncation budgets precede sorting", rewritten, reference)
+
+for batch_size in (None, 0, -1, 1, 100):
+ plain, grouped = make_length_fake(), make_length_fake()
+ check("length/no-op results for batch_size=%r" % batch_size,
+ grouped.predict_batch(states, QUESTIONS, batch_size=batch_size, sort_by_length=True),
+ plain.predict_batch(states, QUESTIONS, batch_size=batch_size))
+ check("length/no-op grouping for batch_size=%r" % batch_size, grouped.batch_rows, plain.batch_rows)
+
+f = make_length_fake()
+check("length/empty inputs", f.predict_batch([], QUESTIONS, sort_by_length=True), [])
+check("length/empty questions", f.predict_batch(states[:1], {}, sort_by_length=True)[0]["answers"], {})
+check("length/empty cases do not encode", f.encoded, [])
+f.predict_batch(states, QUESTIONS, sort_by_length=True, on_predict_start=lambda ctx: ctx.skip([]))
+check("length/cache skip does not encode", f.encoded, [])
+
+plain, grouped = make_length_fake(), make_length_fake()
+equal_lengths = [{"id": i, "length": 10} for i in range(20)]
+plain.predict_batch(equal_lengths, QUESTIONS, batch_size=3)
+grouped.predict_batch(equal_lengths, QUESTIONS, batch_size=3, sort_by_length=True)
+check("length/equal length ties preserve row order", grouped.batch_rows, plain.batch_rows)
+
+errors = []
+class RecordFailure:
+ def on_error(self, ctx):
+ errors.append(ctx.error)
+
+
+def fail_forward(batch):
+ raise RuntimeError("forward failed")
+
+
+f = make_length_fake()
+f._forward = fail_forward
+check_raises("length/forward error propagates", RuntimeError,
+ lambda: f.predict_batch(states, QUESTIONS, batch_size=2,
+ sort_by_length=True, hooks=[RecordFailure()]))
+check("length/error hook receives failure once", [str(e) for e in errors], ["forward failed"])
+
+
f = make_fake()
check("orchestration/empty states -> empty list", f.predict_batch([], QUESTIONS), [])