Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(agent): add an opt-in persistent Laya compile cache
cklxx130875ed8e5b
diff --git a/README.md b/README.mdindex aeb2b6c9..ba575e05 100644--- a/README.md+++ b/README.md@@ -879,7 +879,10 @@ On CUDA, compiled inference pads the masked end of each sequence to a multiple oavoids extra SDPA graph specialisations on PyTorch versions that distinguish lengths modulo eight;reported token usage still counts the original, unpadded sequence.Inductor caches compiled graphs under `TORCHINDUCTOR_CACHE_DIR` (by default in `/tmp`); point it at a-persistent directory to keep them across restarts.+persistent directory to keep them across restarts. Alternatively, load with+`compile=True, compile_cache=True` to use `$XDG_CACHE_HOME/laya/torchinductor` (or+`~/.cache/laya/torchinductor`). This opt-in sets the process-wide environment variable only if+absent; an existing value always wins. It does nothing on eager or TileLang loads.---diff --git a/benchmarks/bench_compile_defaults.py b/benchmarks/bench_compile_defaults.pyindex b178a237..91a79a82 100644--- a/benchmarks/bench_compile_defaults.py+++ b/benchmarks/bench_compile_defaults.py@@ -22,7 +22,6 @@ def main():parser.add_argument("--mode", default=None)args = parser.parse_args()import torch- from torch._dynamo.utils import countersfrom laya import Agentdef smi():@@ -37,10 +36,19 @@ def main():kw["compile_cache"] = Trueif args.mode:kw["compile_mode"] = args.mode+ initial_cache = os.environ.get("TORCHINDUCTOR_CACHE_DIR")t = time.perf_counter()agent = Agent(args.model, device="cuda", compile=True, **kw)torch.cuda.synchronize()load_s = time.perf_counter() - t+ from torch._dynamo.utils import counters+ if args.cache:+ root = os.environ.get("XDG_CACHE_HOME", "")+ if not os.path.isabs(root):+ root = os.path.expanduser("~/.cache")+ expected = initial_cache or os.path.join(root, "laya", "torchinductor")+ assert os.environ["TORCHINDUCTOR_CACHE_DIR"] == expected+ assert os.path.isdir(expected)print("load_s", load_s, "cache", os.environ.get("TORCHINDUCTOR_CACHE_DIR"), flush=True)times = []# Repeat the sequence to expose CUDA graph recording/replay and retained-output issues.diff --git a/benchmarks/results/compile-defaults/README.md b/benchmarks/results/compile-defaults/README.mdindex 76d7dbd6..6e13b44b 100644--- a/benchmarks/results/compile-defaults/README.md+++ b/benchmarks/results/compile-defaults/README.md@@ -51,3 +51,33 @@ Output: `Using Python 3.12.13 environment at: /home/ckl/projects/S/laya/.venv`;Baseline-cold ran before code edits; use `--no-warmup` to reproduce its lazy behavior on this tree.Warm-up shifts cost into load; it does not eliminate compilation or all possible guard specializations.The two warm-up runs were contended. Cross-process reuse yielded two FX hits and zero misses.++## Item 2: opt-in persistent Laya cache++`compile_cache=True` selects `$XDG_CACHE_HOME/laya/torchinductor` or+`~/.cache/laya/torchinductor` only if `TORCHINDUCTOR_CACHE_DIR` is absent.+The default is false, existing settings win, and non-compiled/TileLang loads do nothing.+The choice is explicitly process-wide; PyTorch owns cache compatibility/invalidation.++The `cache-verified-*` experiments unset `TORCHINDUCTOR_CACHE_DIR`, set an isolated persistent+XDG root, assert the selected path, and test reuse in separate processes (see `.command` files).+The earlier `cache-cold`, `cache-restart`, and `cache-quiet-*` logs are diagnostic attempts,+NOT measurements of the Laya cache: an early Dynamo counters import populated the default+Inductor environment variable before the opt-in ran. This was caught by auditing the actual+printed path. The benchmark now imports counters after load, asserts the path, and the agent+configures the cache before model construction can import Dynamo. A regression pins that order. Full GPU state is attached to each run, not inferred from+allocator counters. Tests cover opt-in behavior, XDG/fallback paths, explicit overrides,+repeated setup, and constructor forwarding. `cache-verified-validation.log` records every gate and core suite after the correction.+The final ordering assertion is rerun with `python tests/test_compile.py`+(`cache-order-tests.log`, exit 0).++| Run | Load s | Request 1 ms | Request 8 ms | Peak alloc/reserved MiB | FX hits/misses |+|---|---:|---:|---:|---:|---:|+| cache-verified-cold | 111.643 | 22.86 | 8.69 | 1685.6 / 1820.0 | 0 / 2 |+| cache-verified-restart | 51.292 | 24.43 | 10.84 | 1671.0 / 1796.0 | 2 / 0 |++Both verified processes selected `/home/ckl/.cache/laya-compile-defaults-verified-20261003/laya/torchinductor`.+The second process hit both FX graphs, reducing load from 111.64 s to 51.29 s.+The cache location adds no meaningful GPU memory requirement; both peaks match the earlier+warm-up experiments. These final runs had no other compute jobs in their GPU snapshots.+Reboot/container persistence was not exercised; that depends on retaining the directory.diff --git a/benchmarks/results/compile-defaults/cache-cold.command b/benchmarks/results/compile-defaults/cache-cold.commandnew file mode 100644index 00000000..c88316b8--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-cold.command@@ -0,0 +1 @@+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-measure-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cachediff --git a/benchmarks/results/compile-defaults/cache-cold.log b/benchmarks/results/compile-defaults/cache-cold.lognew file mode 100644index 00000000..d8d34551--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-cold.log@@ -0,0 +1,109 @@+Sat Oct 3 18:46:26 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 30% 33C P8 12W / 295W | 3917MiB / 16376MiB | 0% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 146MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |+| 0 N/A N/A 4072238 C ...jects/S/laya/.venv/bin/python 3184MiB |++-----------------------------------------------------------------------------------------+++torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.+ agent = Agent(args.model, device="cuda", compile=True, **kw)+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.+ warnings.warn(+W1003 18:47:07.365000 4074699 torch/_inductor/utils.py:1731] [0/0_1] Not enough SMs to use max_autotune_gemm mode+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/lowering.py:7836: UserWarning:+Online softmax is disabled on the fly since Inductor decides to+split the reduction. Cut an issue to PyTorch if this is an+important use case and you want to speed it up with online+softmax.++ warnings.warn(+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/lowering.py:7836: UserWarning:+Online softmax is disabled on the fly since Inductor decides to+split the reduction. Cut an issue to PyTorch if this is an+important use case and you want to speed it up with online+softmax.++ warnings.warn(+load_s 114.72476257698145 cache /home/ckl/.hermes/cache/scratch/torchinductor_ckl+request 1 seconds 0.02641500800382346 graphs 2+request 2 seconds 0.013872356968931854 graphs 2+request 3 seconds 0.011120865936391056 graphs 2+request 4 seconds 0.016031821025535464 graphs 2+request 5 seconds 0.011700992938131094 graphs 2+request 6 seconds 0.02680087205953896 graphs 2+request 7 seconds 0.01864045101683587 graphs 2+request 8 seconds 0.010053643956780434 graphs 2+request 9 seconds 0.032923111924901605 graphs 2+request 10 seconds 0.01651303598191589 graphs 2+request 11 seconds 0.01252508198376745 graphs 2+request 12 seconds 0.014541615033522248 graphs 2+request 13 seconds 0.010531489038839936 graphs 2+request 14 seconds 0.015025408938527107 graphs 2+request 15 seconds 0.01098875398747623 graphs 2+request 16 seconds 0.02744422003161162 graphs 2+request 17 seconds 0.018546530045568943 graphs 2+request 18 seconds 0.00946029694750905 graphs 2+request 19 seconds 0.03218311397358775 graphs 2+request 20 seconds 0.016114933067001402 graphs 2+request 21 seconds 0.012362029985524714 graphs 2+request 22 seconds 0.014607945107854903 graphs 2+request 23 seconds 0.012195498100481927 graphs 2+request 24 seconds 0.014966299058869481 graphs 2+request 25 seconds 0.010161984013393521 graphs 2+request 26 seconds 0.02779080392792821 graphs 2+request 27 seconds 0.018532140064053237 graphs 2+request 28 seconds 0.009995712898671627 graphs 2+request 29 seconds 0.03149076597765088 graphs 2+request 30 seconds 0.01549176499247551 graphs 2+RESULT {"load_s": 114.72476257698145, "requests_s": [0.02641500800382346, 0.013872356968931854, 0.011120865936391056, 0.016031821025535464, 0.011700992938131094, 0.02680087205953896, 0.01864045101683587, 0.010053643956780434, 0.032923111924901605, 0.01651303598191589, 0.01252508198376745, 0.014541615033522248, 0.010531489038839936, 0.015025408938527107, 0.01098875398747623, 0.02744422003161162, 0.018546530045568943, 0.00946029694750905, 0.03218311397358775, 0.016114933067001402, 0.012362029985524714, 0.014607945107854903, 0.012195498100481927, 0.014966299058869481, 0.010161984013393521, 0.02779080392792821, 0.018532140064053237, 0.009995712898671627, 0.03149076597765088, 0.01549176499247551], "allocated_mib": 1616.48291015625, "reserved_mib": 1820.0, "peak_allocated_mib": 1685.626953125, "peak_reserved_mib": 1820.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"pattern_matcher_count": 786, "pattern_matcher_nodes": 910, "benchmarking.InductorBenchmarker.benchmark_gpu": 105, "pad_mm_bench": 4, "fxgraph_cache_miss": 2, "async_compile_cache_miss": 162, "extern_calls": 384, "async_compile_cache_hit": 81, "triton_bundler_save_kernel": 1048, "triton_bundler_save_static_autotuner": 2, "benchmarking.InductorBenchmarker.benchmark": 97}}}+Sat Oct 3 18:48:21 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 44C P2 152W / 295W | 6631MiB / 16376MiB | 77% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 156MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |+| 0 N/A N/A 4072238 C ...jects/S/laya/.venv/bin/python 3802MiB |+| 0 N/A N/A 4074699 C ...jects/S/laya/.venv/bin/python 2080MiB |++-----------------------------------------------------------------------------------------++diff --git a/benchmarks/results/compile-defaults/cache-final-tests.log b/benchmarks/results/compile-defaults/cache-final-tests.lognew file mode 100644index 00000000..fc14b254--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-final-tests.log@@ -0,0 +1,9 @@+ok test_cuda_compile_padding_masks_only_the_new_tail+ok test_one_graph_across_shapes_and_same_outputs+ok test_warmup_builds_every_graph_before_the_first_request+ok test_duck_shape_restored_after_errors_and_nesting+ok test_duck_shape_restored_across_threads+ok test_eager_agent_leaves_the_setting_alone+ok test_constructor_warms_only_the_active_compiled_path+ok test_persistent_cache_is_opt_in_and_respects_the_environment+all compile tests passeddiff --git a/benchmarks/results/compile-defaults/cache-order-tests.log b/benchmarks/results/compile-defaults/cache-order-tests.lognew file mode 100644index 00000000..fc14b254--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-order-tests.log@@ -0,0 +1,9 @@+ok test_cuda_compile_padding_masks_only_the_new_tail+ok test_one_graph_across_shapes_and_same_outputs+ok test_warmup_builds_every_graph_before_the_first_request+ok test_duck_shape_restored_after_errors_and_nesting+ok test_duck_shape_restored_across_threads+ok test_eager_agent_leaves_the_setting_alone+ok test_constructor_warms_only_the_active_compiled_path+ok test_persistent_cache_is_opt_in_and_respects_the_environment+all compile tests passeddiff --git a/benchmarks/results/compile-defaults/cache-quiet-cold.command b/benchmarks/results/compile-defaults/cache-quiet-cold.commandnew file mode 100644index 00000000..ab595d56--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-quiet-cold.command@@ -0,0 +1 @@+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-quiet-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cachediff --git a/benchmarks/results/compile-defaults/cache-quiet-cold.log b/benchmarks/results/compile-defaults/cache-quiet-cold.lognew file mode 100644index 00000000..dacde636--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-quiet-cold.log@@ -0,0 +1,90 @@+Sat Oct 3 18:49:49 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 34C P8 12W / 295W | 728MiB / 16376MiB | 0% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 146MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |++-----------------------------------------------------------------------------------------+++torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.+ agent = Agent(args.model, device="cuda", compile=True, **kw)+load_s 49.42939719394781 cache /home/ckl/.hermes/cache/scratch/torchinductor_ckl+request 1 seconds 0.12283430597744882 graphs 2+request 2 seconds 0.013316553900949657 graphs 2+request 3 seconds 0.00988961395341903 graphs 2+request 4 seconds 0.01398233207874 graphs 2+request 5 seconds 0.009096584981307387 graphs 2+request 6 seconds 0.024636785965412855 graphs 2+request 7 seconds 0.016743125044740736 graphs 2+request 8 seconds 0.00864292006008327 graphs 2+request 9 seconds 0.029934056918136775 graphs 2+request 10 seconds 0.014817061950452626 graphs 2+request 11 seconds 0.009821613901294768 graphs 2+request 12 seconds 0.012280132039450109 graphs 2+request 13 seconds 0.009488470037467778 graphs 2+request 14 seconds 0.013466925942339003 graphs 2+request 15 seconds 0.008513677981682122 graphs 2+request 16 seconds 0.022905955906026065 graphs 2+request 17 seconds 0.015729162958450615 graphs 2+request 18 seconds 0.008210705011151731 graphs 2+request 19 seconds 0.028184838010929525 graphs 2+request 20 seconds 0.01407369296066463 graphs 2+request 21 seconds 0.009481230052188039 graphs 2+request 22 seconds 0.011742135975509882 graphs 2+request 23 seconds 0.009388239006511867 graphs 2+request 24 seconds 0.013468245975673199 graphs 2+request 25 seconds 0.00856755895074457 graphs 2+request 26 seconds 0.022970746038481593 graphs 2+request 27 seconds 0.015777833061292768 graphs 2+request 28 seconds 0.008236306020990014 graphs 2+request 29 seconds 0.028220248059369624 graphs 2+request 30 seconds 0.014039832982234657 graphs 2+RESULT {"load_s": 49.42939719394781, "requests_s": [0.12283430597744882, 0.013316553900949657, 0.00988961395341903, 0.01398233207874, 0.009096584981307387, 0.024636785965412855, 0.016743125044740736, 0.00864292006008327, 0.029934056918136775, 0.014817061950452626, 0.009821613901294768, 0.012280132039450109, 0.009488470037467778, 0.013466925942339003, 0.008513677981682122, 0.022905955906026065, 0.015729162958450615, 0.008210705011151731, 0.028184838010929525, 0.01407369296066463, 0.009481230052188039, 0.011742135975509882, 0.009388239006511867, 0.013468245975673199, 0.00856755895074457, 0.022970746038481593, 0.015777833061292768, 0.008236306020990014, 0.028220248059369624, 0.014039832982234657], "allocated_mib": 1616.48291015625, "reserved_mib": 1796.0, "peak_allocated_mib": 1671.017578125, "peak_reserved_mib": 1796.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"triton_bundler_load_static_autotuner": 68, "async_compile_cache_miss": 243, "async_compile_cache_hit": 149, "pattern_matcher_count": 20, "pattern_matcher_nodes": 26, "extern_calls": 384, "fxgraph_cache_hit": 2}}}+Sat Oct 3 18:50:39 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 43C P2 65W / 295W | 2789MiB / 16376MiB | 82% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 146MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |+| 0 N/A N/A 4079010 C ...jects/S/laya/.venv/bin/python 2056MiB |++-----------------------------------------------------------------------------------------++diff --git a/benchmarks/results/compile-defaults/cache-quiet-restart.command b/benchmarks/results/compile-defaults/cache-quiet-restart.commandnew file mode 100644index 00000000..ab595d56--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-quiet-restart.command@@ -0,0 +1 @@+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-quiet-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cachediff --git a/benchmarks/results/compile-defaults/cache-quiet-restart.log b/benchmarks/results/compile-defaults/cache-quiet-restart.lognew file mode 100644index 00000000..8cd7293a--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-quiet-restart.log@@ -0,0 +1,91 @@+Sat Oct 3 18:50:43 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 35C P3 19W / 295W | 728MiB / 16376MiB | 0% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 146MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |++-----------------------------------------------------------------------------------------+++torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.+ agent = Agent(args.model, device="cuda", compile=True, **kw)+load_s 49.84901165601332 cache /home/ckl/.hermes/cache/scratch/torchinductor_ckl+request 1 seconds 0.021522802067920566 graphs 2+request 2 seconds 0.0127951999893412 graphs 2+request 3 seconds 0.009552222094498575 graphs 2+request 4 seconds 0.013322615996003151 graphs 2+request 5 seconds 0.008758343057706952 graphs 2+request 6 seconds 0.02338546304963529 graphs 2+request 7 seconds 0.015908485976979136 graphs 2+request 8 seconds 0.008505699923262 graphs 2+request 9 seconds 0.02822929003741592 graphs 2+request 10 seconds 0.014193495968356729 graphs 2+request 11 seconds 0.009512242046184838 graphs 2+request 12 seconds 0.011779058026149869 graphs 2+request 13 seconds 0.009392678970471025 graphs 2+request 14 seconds 0.013497408013790846 graphs 2+request 15 seconds 0.008576749940402806 graphs 2+request 16 seconds 0.022936348104849458 graphs 2+request 17 seconds 0.01577090402133763 graphs 2+request 18 seconds 0.0081936459755525 graphs 2+request 19 seconds 0.028569523943588138 graphs 2+request 20 seconds 0.015004775952547789 graphs 2+request 21 seconds 0.009698834037408233 graphs 2+request 22 seconds 0.011769518023356795 graphs 2+request 23 seconds 0.009374139015562832 graphs 2+request 24 seconds 0.013569338945671916 graphs 2+request 25 seconds 0.008530530030839145 graphs 2+request 26 seconds 0.02300787903368473 graphs 2+request 27 seconds 0.015862375032156706 graphs 2+request 28 seconds 0.00838116800878197 graphs 2+request 29 seconds 0.028341430937871337 graphs 2+request 30 seconds 0.014030663995072246 graphs 2+RESULT {"load_s": 49.84901165601332, "requests_s": [0.021522802067920566, 0.0127951999893412, 0.009552222094498575, 0.013322615996003151, 0.008758343057706952, 0.02338546304963529, 0.015908485976979136, 0.008505699923262, 0.02822929003741592, 0.014193495968356729, 0.009512242046184838, 0.011779058026149869, 0.009392678970471025, 0.013497408013790846, 0.008576749940402806, 0.022936348104849458, 0.01577090402133763, 0.0081936459755525, 0.028569523943588138, 0.015004775952547789, 0.009698834037408233, 0.011769518023356795, 0.009374139015562832, 0.013569338945671916, 0.008530530030839145, 0.02300787903368473, 0.015862375032156706, 0.00838116800878197, 0.028341430937871337, 0.014030663995072246], "allocated_mib": 1616.48291015625, "reserved_mib": 1796.0, "peak_allocated_mib": 1671.017578125, "peak_reserved_mib": 1796.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"triton_bundler_load_static_autotuner": 68, "async_compile_cache_miss": 243, "async_compile_cache_hit": 149, "pattern_matcher_count": 20, "pattern_matcher_nodes": 26, "extern_calls": 384, "fxgraph_cache_hit": 2}}}+Sat Oct 3 18:51:33 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 48C P2 105W / 295W | 3134MiB / 16376MiB | 82% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 146MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |+| 0 N/A N/A 4079529 C ...jects/S/laya/.venv/bin/python 2056MiB |+| 0 N/A N/A 4079725 C ...jects/S/laya/.venv/bin/python 340MiB |++-----------------------------------------------------------------------------------------++diff --git a/benchmarks/results/compile-defaults/cache-restart.command b/benchmarks/results/compile-defaults/cache-restart.commandnew file mode 100644index 00000000..c88316b8--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-restart.command@@ -0,0 +1 @@+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-measure-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cachediff --git a/benchmarks/results/compile-defaults/cache-restart.log b/benchmarks/results/compile-defaults/cache-restart.lognew file mode 100644index 00000000..da18d671--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-restart.log@@ -0,0 +1,91 @@+Sat Oct 3 18:48:27 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 33C P8 12W / 295W | 4545MiB / 16376MiB | 26% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 156MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |+| 0 N/A N/A 4072238 C ...jects/S/laya/.venv/bin/python 3802MiB |++-----------------------------------------------------------------------------------------+++torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.+ agent = Agent(args.model, device="cuda", compile=True, **kw)+load_s 53.14927826402709 cache /home/ckl/.hermes/cache/scratch/torchinductor_ckl+request 1 seconds 0.02442872989922762 graphs 2+request 2 seconds 0.014553737011738122 graphs 2+request 3 seconds 0.009722160990349948 graphs 2+request 4 seconds 0.015148414066061378 graphs 2+request 5 seconds 0.010848594014532864 graphs 2+request 6 seconds 0.02579907502513379 graphs 2+request 7 seconds 0.01758231094572693 graphs 2+request 8 seconds 0.010467909974977374 graphs 2+request 9 seconds 0.03215115796774626 graphs 2+request 10 seconds 0.016135973972268403 graphs 2+request 11 seconds 0.01121161796618253 graphs 2+request 12 seconds 0.011908155982382596 graphs 2+request 13 seconds 0.01099822495598346 graphs 2+request 14 seconds 0.01532110502012074 graphs 2+request 15 seconds 0.010279048001393676 graphs 2+request 16 seconds 0.02483839401975274 graphs 2+request 17 seconds 0.017773673054762185 graphs 2+request 18 seconds 0.009426687960512936 graphs 2+request 19 seconds 0.03183845500461757 graphs 2+request 20 seconds 0.015550486976280808 graphs 2+request 21 seconds 0.009742891998030245 graphs 2+request 22 seconds 0.01376086799427867 graphs 2+request 23 seconds 0.011247409041970968 graphs 2+request 24 seconds 0.014977620914578438 graphs 2+request 25 seconds 0.00899228302296251 graphs 2+request 26 seconds 0.02644445199985057 graphs 2+request 27 seconds 0.01784457394387573 graphs 2+request 28 seconds 0.008444887003861368 graphs 2+request 29 seconds 0.031590171973221004 graphs 2+request 30 seconds 0.01599324296694249 graphs 2+RESULT {"load_s": 53.14927826402709, "requests_s": [0.02442872989922762, 0.014553737011738122, 0.009722160990349948, 0.015148414066061378, 0.010848594014532864, 0.02579907502513379, 0.01758231094572693, 0.010467909974977374, 0.03215115796774626, 0.016135973972268403, 0.01121161796618253, 0.011908155982382596, 0.01099822495598346, 0.01532110502012074, 0.010279048001393676, 0.02483839401975274, 0.017773673054762185, 0.009426687960512936, 0.03183845500461757, 0.015550486976280808, 0.009742891998030245, 0.01376086799427867, 0.011247409041970968, 0.014977620914578438, 0.00899228302296251, 0.02644445199985057, 0.01784457394387573, 0.008444887003861368, 0.031590171973221004, 0.01599324296694249], "allocated_mib": 1616.48291015625, "reserved_mib": 1796.0, "peak_allocated_mib": 1671.017578125, "peak_reserved_mib": 1796.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"triton_bundler_load_static_autotuner": 68, "async_compile_cache_miss": 243, "async_compile_cache_hit": 149, "pattern_matcher_count": 20, "pattern_matcher_nodes": 26, "extern_calls": 384, "fxgraph_cache_hit": 2}}}+Sat Oct 3 18:49:21 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 44C P2 87W / 295W | 2799MiB / 16376MiB | 81% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 156MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |+| 0 N/A N/A 4077174 C ...jects/S/laya/.venv/bin/python 2056MiB |++-----------------------------------------------------------------------------------------++diff --git a/benchmarks/results/compile-defaults/cache-validation.log b/benchmarks/results/compile-defaults/cache-validation.lognew file mode 100644index 00000000..55018529--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-validation.log@@ -0,0 +1,57 @@+PATH=/home/ckl/projects/S/laya/.venv/bin:$PATH++$ ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120+All checks passed!++exit=0++$ python -m compileall -q laya/ tests/++exit=0++$ python tests/test_compile.py+ok test_cuda_compile_padding_masks_only_the_new_tail+ok test_one_graph_across_shapes_and_same_outputs+ok test_warmup_builds_every_graph_before_the_first_request+ok test_duck_shape_restored_after_errors_and_nesting+ok test_duck_shape_restored_across_threads+ok test_eager_agent_leaves_the_setting_alone+ok test_constructor_warms_only_the_active_compiled_path+ok test_persistent_cache_is_opt_in_and_respects_the_environment+all compile tests passed++exit=0++$ python tests/test_hooks_api.py++401 passed, 0 failed+all hook API tests passed++exit=0++$ python tests/test_router.py++703 passed, 0 failed+all routing tests passed++exit=0++$ python tests/test_criteria.py++198 passed, 0 failed++exit=0++$ python tests/test_hooks.py++240 passed, 0 failed+all hook tests passed++exit=0++$ zensical build --strict --clean+Build started+No issues found+Build finished in 2.06s++exit=0diff --git a/benchmarks/results/compile-defaults/cache-verified-cold.command b/benchmarks/results/compile-defaults/cache-verified-cold.commandnew file mode 100644index 00000000..2d578752--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-verified-cold.command@@ -0,0 +1 @@+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-verified-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cachediff --git a/benchmarks/results/compile-defaults/cache-verified-cold.log b/benchmarks/results/compile-defaults/cache-verified-cold.lognew file mode 100644index 00000000..c63e4b6a--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-verified-cold.log@@ -0,0 +1,107 @@+Sat Oct 3 18:53:03 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 36C P8 12W / 295W | 738MiB / 16376MiB | 28% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 156MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |++-----------------------------------------------------------------------------------------+++torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.+ agent = Agent(args.model, device="cuda", compile=True, **kw)+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.+ warnings.warn(+W1003 18:53:44.221000 4081427 torch/_inductor/utils.py:1731] [0/0_1] Not enough SMs to use max_autotune_gemm mode+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/lowering.py:7836: UserWarning:+Online softmax is disabled on the fly since Inductor decides to+split the reduction. Cut an issue to PyTorch if this is an+important use case and you want to speed it up with online+softmax.++ warnings.warn(+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/lowering.py:7836: UserWarning:+Online softmax is disabled on the fly since Inductor decides to+split the reduction. Cut an issue to PyTorch if this is an+important use case and you want to speed it up with online+softmax.++ warnings.warn(+load_s 111.64288430591114 cache /home/ckl/.cache/laya-compile-defaults-verified-20261003/laya/torchinductor+request 1 seconds 0.02286072401329875 graphs 2+request 2 seconds 0.012921674991957843 graphs 2+request 3 seconds 0.009732725913636386 graphs 2+request 4 seconds 0.01357605296652764 graphs 2+request 5 seconds 0.008746225037612021 graphs 2+request 6 seconds 0.023331869044341147 graphs 2+request 7 seconds 0.01587031001690775 graphs 2+request 8 seconds 0.008692333940416574 graphs 2+request 9 seconds 0.0283058590721339 graphs 2+request 10 seconds 0.013749895035289228 graphs 2+request 11 seconds 0.009646686026826501 graphs 2+request 12 seconds 0.012101444997824728 graphs 2+request 13 seconds 0.009533305070362985 graphs 2+request 14 seconds 0.013106117956340313 graphs 2+request 15 seconds 0.008559482987038791 graphs 2+request 16 seconds 0.023173767956905067 graphs 2+request 17 seconds 0.015705847996287048 graphs 2+request 18 seconds 0.008421211037784815 graphs 2+request 19 seconds 0.028346410021185875 graphs 2+request 20 seconds 0.013642733916640282 graphs 2+request 21 seconds 0.009569393936544657 graphs 2+request 22 seconds 0.011785740964114666 graphs 2+request 23 seconds 0.009457602980546653 graphs 2+request 24 seconds 0.01304219604935497 graphs 2+request 25 seconds 0.008564352057874203 graphs 2+request 26 seconds 0.023065467015840113 graphs 2+request 27 seconds 0.015712207998149097 graphs 2+request 28 seconds 0.008489751955494285 graphs 2+request 29 seconds 0.02870765409898013 graphs 2+request 30 seconds 0.016941983019933105 graphs 2+RESULT {"load_s": 111.64288430591114, "requests_s": [0.02286072401329875, 0.012921674991957843, 0.009732725913636386, 0.01357605296652764, 0.008746225037612021, 0.023331869044341147, 0.01587031001690775, 0.008692333940416574, 0.0283058590721339, 0.013749895035289228, 0.009646686026826501, 0.012101444997824728, 0.009533305070362985, 0.013106117956340313, 0.008559482987038791, 0.023173767956905067, 0.015705847996287048, 0.008421211037784815, 0.028346410021185875, 0.013642733916640282, 0.009569393936544657, 0.011785740964114666, 0.009457602980546653, 0.01304219604935497, 0.008564352057874203, 0.023065467015840113, 0.015712207998149097, 0.008489751955494285, 0.02870765409898013, 0.016941983019933105], "allocated_mib": 1616.48291015625, "reserved_mib": 1820.0, "peak_allocated_mib": 1685.626953125, "peak_reserved_mib": 1820.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"pattern_matcher_count": 786, "pattern_matcher_nodes": 910, "benchmarking.InductorBenchmarker.benchmark_gpu": 105, "pad_mm_bench": 4, "fxgraph_cache_miss": 2, "async_compile_cache_miss": 162, "extern_calls": 384, "async_compile_cache_hit": 81, "triton_bundler_save_kernel": 1048, "triton_bundler_save_static_autotuner": 2, "benchmarking.InductorBenchmarker.benchmark": 97}}}+Sat Oct 3 18:54:55 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 47C P2 158W / 295W | 2823MiB / 16376MiB | 83% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 156MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |+| 0 N/A N/A 4081427 C ...jects/S/laya/.venv/bin/python 2080MiB |++-----------------------------------------------------------------------------------------++diff --git a/benchmarks/results/compile-defaults/cache-verified-restart.command b/benchmarks/results/compile-defaults/cache-verified-restart.commandnew file mode 100644index 00000000..2d578752--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-verified-restart.command@@ -0,0 +1 @@+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-verified-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cachediff --git a/benchmarks/results/compile-defaults/cache-verified-restart.log b/benchmarks/results/compile-defaults/cache-verified-restart.lognew file mode 100644index 00000000..71042c88--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-verified-restart.log@@ -0,0 +1,90 @@+Sat Oct 3 18:55:00 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 36C P3 19W / 295W | 738MiB / 16376MiB | 10% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 156MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |++-----------------------------------------------------------------------------------------+++torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.+ agent = Agent(args.model, device="cuda", compile=True, **kw)+load_s 51.29203179304022 cache /home/ckl/.cache/laya-compile-defaults-verified-20261003/laya/torchinductor+request 1 seconds 0.024434805032797158 graphs 2+request 2 seconds 0.013582332991063595 graphs 2+request 3 seconds 0.011464047944173217 graphs 2+request 4 seconds 0.015985592966899276 graphs 2+request 5 seconds 0.010889400960877538 graphs 2+request 6 seconds 0.026401367969810963 graphs 2+request 7 seconds 0.018746355082839727 graphs 2+request 8 seconds 0.010839381022378802 graphs 2+request 9 seconds 0.03360046504531056 graphs 2+request 10 seconds 0.01667865097988397 graphs 2+request 11 seconds 0.011249065981246531 graphs 2+request 12 seconds 0.012481120997108519 graphs 2+request 13 seconds 0.00973973807413131 graphs 2+request 14 seconds 0.015411366010084748 graphs 2+request 15 seconds 0.010465405997820199 graphs 2+request 16 seconds 0.02478666801471263 graphs 2+request 17 seconds 0.017652343027293682 graphs 2+request 18 seconds 0.009124230011366308 graphs 2+request 19 seconds 0.031999476021155715 graphs 2+request 20 seconds 0.016068443073891103 graphs 2+request 21 seconds 0.009568715933710337 graphs 2+request 22 seconds 0.013553263037465513 graphs 2+request 23 seconds 0.011064652935601771 graphs 2+request 24 seconds 0.015312475035898387 graphs 2+request 25 seconds 0.008710375055670738 graphs 2+request 26 seconds 0.026597779942676425 graphs 2+request 27 seconds 0.01785830594599247 graphs 2+request 28 seconds 0.008381210966035724 graphs 2+request 29 seconds 0.03149348997976631 graphs 2+request 30 seconds 0.01582511095330119 graphs 2+RESULT {"load_s": 51.29203179304022, "requests_s": [0.024434805032797158, 0.013582332991063595, 0.011464047944173217, 0.015985592966899276, 0.010889400960877538, 0.026401367969810963, 0.018746355082839727, 0.010839381022378802, 0.03360046504531056, 0.01667865097988397, 0.011249065981246531, 0.012481120997108519, 0.00973973807413131, 0.015411366010084748, 0.010465405997820199, 0.02478666801471263, 0.017652343027293682, 0.009124230011366308, 0.031999476021155715, 0.016068443073891103, 0.009568715933710337, 0.013553263037465513, 0.011064652935601771, 0.015312475035898387, 0.008710375055670738, 0.026597779942676425, 0.01785830594599247, 0.008381210966035724, 0.03149348997976631, 0.01582511095330119], "allocated_mib": 1616.48291015625, "reserved_mib": 1796.0, "peak_allocated_mib": 1671.017578125, "peak_reserved_mib": 1796.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"triton_bundler_load_static_autotuner": 68, "async_compile_cache_miss": 243, "async_compile_cache_hit": 149, "pattern_matcher_count": 20, "pattern_matcher_nodes": 26, "extern_calls": 384, "fxgraph_cache_hit": 2}}}+Sat Oct 3 18:55:52 2026++-----------------------------------------------------------------------------------------++| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |++-----------------------------------------+------------------------+----------------------++| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |+| | | MIG M. |+|=========================================+========================+======================|+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |+| 0% 47C P2 74W / 295W | 2795MiB / 16376MiB | 84% Default |+| | | N/A |++-----------------------------------------+------------------------+----------------------++++-----------------------------------------------------------------------------------------++| Processes: |+| GPU GI CI PID Type Process name GPU Memory |+| ID ID Usage |+|=========================================================================================|+| 0 N/A N/A 892 G Hyprland 156MiB |+| 0 N/A N/A 952 G quickshell 38MiB |+| 0 N/A N/A 995 G Xwayland 4MiB |+| 0 N/A N/A 1168 G monocloud 3MiB |+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 31MiB |+| 0 N/A N/A 4083959 C ...jects/S/laya/.venv/bin/python 2056MiB |++-----------------------------------------------------------------------------------------++diff --git a/benchmarks/results/compile-defaults/cache-verified-validation.log b/benchmarks/results/compile-defaults/cache-verified-validation.lognew file mode 100644index 00000000..022075f3--- /dev/null+++ b/benchmarks/results/compile-defaults/cache-verified-validation.log@@ -0,0 +1,57 @@+PATH=/home/ckl/projects/S/laya/.venv/bin:$PATH++$ ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120+All checks passed!++exit=0++$ python -m compileall -q laya/ tests/++exit=0++$ python tests/test_compile.py+ok test_cuda_compile_padding_masks_only_the_new_tail+ok test_one_graph_across_shapes_and_same_outputs+ok test_warmup_builds_every_graph_before_the_first_request+ok test_duck_shape_restored_after_errors_and_nesting+ok test_duck_shape_restored_across_threads+ok test_eager_agent_leaves_the_setting_alone+ok test_constructor_warms_only_the_active_compiled_path+ok test_persistent_cache_is_opt_in_and_respects_the_environment+all compile tests passed++exit=0++$ python tests/test_hooks_api.py++401 passed, 0 failed+all hook API tests passed++exit=0++$ python tests/test_router.py++703 passed, 0 failed+all routing tests passed++exit=0++$ python tests/test_criteria.py++198 passed, 0 failed++exit=0++$ python tests/test_hooks.py++240 passed, 0 failed+all hook tests passed++exit=0++$ zensical build --strict --clean+Build started+No issues found+Build finished in 2.09s++exit=0diff --git a/docs/compile-and-fast-path.md b/docs/compile-and-fast-path.mdindex c3ea2205..0076af49 100644--- a/docs/compile-and-fast-path.md+++ b/docs/compile-and-fast-path.md@@ -31,6 +31,14 @@ by sequence length, so it has no such buffer.container restart. Set it to a persistent directory, or a volume in a container, and a secondprocess loads the graphs instead of compiling them. In #472's measurement that took warm-up fromabout 120 s to about 50 s.+- **Laya cache opt-in.** `laya.load(..., compile=True, compile_cache=True)` sets the process-wide+ `TORCHINDUCTOR_CACHE_DIR` only when absent, to `$XDG_CACHE_HOME/laya/torchinductor` or+ `~/.cache/laya/torchinductor` when XDG is unset or not absolute. An existing setting, including+ one set by an earlier PyTorch compile, wins. The directory is created at load; filesystem+ errors propagate. `compile_cache=False` (default), eager, and TileLang loads leave the+ environment alone. This does not move or delete old caches. Containers still need a persistent+ home/volume. Cache compatibility and invalidation are managed by PyTorch; a GPU, torch,+ compiler, model, or input guard change can require compilation again.## AOTInductor: not yetdiff --git a/laya/_compile.py b/laya/_compile.pyindex aa1e5326..852a8442 100644--- a/laya/_compile.py+++ b/laya/_compile.py@@ -26,6 +26,7 @@ Mode and device are unchanged from what `compile=True` has always done: the defaon whatever device the agent runs on, CPU included."""import threading+import osfrom contextlib import contextmanagerfrom contextvars import ContextVar@@ -41,6 +42,16 @@ def compile_model(model, **kwargs):return torch.compile(model, dynamic=True, **kwargs)+def configure_cache():+ """Respect an existing Inductor directory; otherwise use the user's Laya cache."""+ root = os.environ.get("XDG_CACHE_HOME", "")+ if not os.path.isabs(root):+ root = os.path.expanduser("~/.cache")+ directory = os.environ.setdefault("TORCHINDUCTOR_CACHE_DIR", os.path.join(root, "laya", "torchinductor"))+ os.makedirs(directory, exist_ok=True)+ return directory++def _fx_config():try:from torch.fx.experimental import _configdiff --git a/laya/agent.py b/laya/agent.pyindex e9a523d9..354b99c4 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -13,7 +13,7 @@ from typing import Any, Dict, List, Optional, Unionimport numpy as npimport torch-from ._compile import compile_model, independent_dims+from ._compile import compile_model, independent_dims, configure_cachefrom .calibrate import (_install_temperatures,apply_calibration_payload,@@ -493,6 +493,7 @@ class Agent(HookRegistry):hooks_timeout: Optional[float] = None,calibration: Optional[str] = None,compile_warmup: bool = True,+ compile_cache: bool = False,):"""Load a Laya checkpoint.@@ -513,6 +514,8 @@ class Agent(HookRegistry):TileLang path replaces the forward that would be compiled.Compiled agents run `warmup()` before returning; `compile_warmup=False` defers thatwork to requests or a manual `warmup()` call. Eager and fast agents are unchanged.+ `compile_cache=True` opts into a persistent Laya Inductor directory (process-wide),+ respecting any existing `TORCHINDUCTOR_CACHE_DIR`; see the compile engineering notes.`subfolder` selects one checkpoint from a repo that bundles several, e.g.`Agent("convaiinnovations/laya", subfolder="multilingual")`. Only that subfolder is@@ -527,6 +530,9 @@ class Agent(HookRegistry):`hooks_concurrent=False` serialises hooks that are not safe to run in parallel, and`hooks_timeout` bounds each hook call in seconds (None means no limit)."""+ # Model loading can import Dynamo, which populates Inductor's default directory.+ if compile and not fast and compile_cache:+ configure_cache()self.hooks = normalise_hooks(hooks, on_predict_start, on_predict_end)self.hooks_raise = bool(hooks_raise)self.hooks_concurrent = bool(hooks_concurrent)@@ -1913,7 +1919,8 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]hooks=None, on_predict_start=None, on_predict_end=None,hooks_raise: bool = True, hooks_concurrent: bool = True,hooks_timeout: Optional[float] = None,- calibration: Optional[str] = None, compile_warmup: bool = True) -> Agent:+ calibration: Optional[str] = None, compile_warmup: bool = True,+ compile_cache: bool = False) -> Agent:"""Load a Laya agent.`subfolder` picks one checkpoint out of a repo that bundles several:@@ -1949,7 +1956,7 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]if spec is not None:model_id_or_path, subfolder = specreturn Agent(model_id_or_path, device=device, token=token, subfolder=subfolder, fast=fast,- compile=compile, compile_warmup=compile_warmup,+ compile=compile, compile_warmup=compile_warmup, compile_cache=compile_cache,revision=revision, expected_sha256=expected_sha256,lang_temperatures=lang_temperatures,hooks=hooks, on_predict_start=on_predict_start, on_predict_end=on_predict_end,diff --git a/tests/test_compile.py b/tests/test_compile.pyindex e8801162..f3426aac 100644--- a/tests/test_compile.py+++ b/tests/test_compile.py@@ -182,14 +182,16 @@ def test_constructor_warms_only_the_active_compiled_path():from laya import loadmodel = tiny_model()+ events = []with tempfile.TemporaryDirectory() as directory:root = Path(directory)(root / "rl_agent_config.json").write_text(json.dumps({"encoder": "unused", "head_layers": 1}))(root / "model.safetensors").touch()- with patch.object(module, "build_model", return_value=model), \+ with patch.object(module, "build_model", side_effect=lambda *a, **kw: events.append("model") or model), \patch.object(module, "_load_tokenizer", return_value=SimpleNamespace(cls_token_id=1)), \patch("safetensors.torch.load_file", return_value=model.state_dict()), \- patch.object(module, "compile_model", side_effect=lambda m: m) as compile_spy, \+ patch.object(module, "compile_model", side_effect=lambda m, **kw: m) as compile_spy, \+ patch.object(module, "configure_cache", side_effect=lambda: events.append("cache")) as cache_spy, \patch.object(Agent, "accelerate"), \patch.object(Agent, "warmup", autospec=True) as warm:agent = load(directory, device="cpu", compile=True)@@ -203,6 +205,46 @@ def test_constructor_warms_only_the_active_compiled_path():assert compile_spy.call_count == 2agent.warmup()warm.assert_called_once_with(agent)+ cache_spy.assert_not_called()+ events.clear()+ load(directory, device="cpu", compile=True, compile_cache=True, compile_warmup=False)+ assert events == ["cache", "model"]+ cache_spy.assert_called_once_with()+ cache_spy.reset_mock()+ compile_spy.reset_mock()+ load(directory, device="cpu", compile=False, compile_cache=True)+ load(directory, device="cpu", compile=True, fast=True, compile_cache=True)+ compile_spy.assert_not_called()+ cache_spy.assert_not_called()+++def test_persistent_cache_is_opt_in_and_respects_the_environment():+ import tempfile+ from pathlib import Path+ from unittest.mock import patch+ from laya._compile import configure_cache++ with tempfile.TemporaryDirectory() as directory:+ with patch.dict(os.environ, {"XDG_CACHE_HOME": directory + "/xdg"}, clear=True), \+ patch("os.path.expanduser", return_value=directory + "/.cache"), \+ patch("torch.compile") as compiler:+ compile_model(object())+ assert "TORCHINDUCTOR_CACHE_DIR" not in os.environ+ configure_cache()+ expected = str(Path(directory) / "xdg/laya/torchinductor")+ assert os.environ["TORCHINDUCTOR_CACHE_DIR"] == expected+ assert Path(expected).is_dir()+ assert compiler.call_args.kwargs == {"dynamic": True}+ # A caller-selected directory always wins, including after repeated loads.+ os.environ["TORCHINDUCTOR_CACHE_DIR"] = directory + "/explicit"+ assert configure_cache() == directory + "/explicit"+ assert configure_cache() == directory + "/explicit"+ del os.environ["TORCHINDUCTOR_CACHE_DIR"]+ os.environ["XDG_CACHE_HOME"] = "relative-is-invalid"+ assert configure_cache() == directory + "/.cache/laya/torchinductor"+ del os.environ["TORCHINDUCTOR_CACHE_DIR"]+ del os.environ["XDG_CACHE_HOME"]+ assert configure_cache() == directory + "/.cache/laya/torchinductor"if __name__ == "__main__":diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex 3e889433..528527a3 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -77,6 +77,7 @@ for label, fn in (("Agent.__init__", Agent.__init__), ("load", load),for label, fn in (("Agent.__init__", Agent.__init__), ("load", load)):check_param(label, fn, "compile_warmup", True)+ check_param(label, fn, "compile_cache", False)# Router keeps lang_guess and explicit per-model revisions toocheck_param("Router.__init__", Router.__init__, "lang_guess", None)