Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(agent): add an opt-in persistent Laya compile cache

cklxx130875ed8e5b
diff --git a/README.md b/README.md
index aeb2b6c9..ba575e05 100644
--- a/README.md
+++ b/README.md
@@ -879,7 +879,10 @@ On CUDA, compiled inference pads the masked end of each sequence to a multiple o
avoids extra SDPA graph specialisations on PyTorch versions that distinguish lengths modulo eight;
reported token usage still counts the original, unpadded sequence.
Inductor caches compiled graphs under `TORCHINDUCTOR_CACHE_DIR` (by default in `/tmp`); point it at a
-persistent directory to keep them across restarts.
+persistent directory to keep them across restarts. Alternatively, load with
+`compile=True, compile_cache=True` to use `$XDG_CACHE_HOME/laya/torchinductor` (or
+`~/.cache/laya/torchinductor`). This opt-in sets the process-wide environment variable only if
+absent; an existing value always wins. It does nothing on eager or TileLang loads.
---
diff --git a/benchmarks/bench_compile_defaults.py b/benchmarks/bench_compile_defaults.py
index b178a237..91a79a82 100644
--- a/benchmarks/bench_compile_defaults.py
+++ b/benchmarks/bench_compile_defaults.py
@@ -22,7 +22,6 @@ def main():
parser.add_argument("--mode", default=None)
args = parser.parse_args()
import torch
- from torch._dynamo.utils import counters
from laya import Agent
def smi():
@@ -37,10 +36,19 @@ def main():
kw["compile_cache"] = True
if args.mode:
kw["compile_mode"] = args.mode
+ initial_cache = os.environ.get("TORCHINDUCTOR_CACHE_DIR")
t = time.perf_counter()
agent = Agent(args.model, device="cuda", compile=True, **kw)
torch.cuda.synchronize()
load_s = time.perf_counter() - t
+ from torch._dynamo.utils import counters
+ if args.cache:
+ root = os.environ.get("XDG_CACHE_HOME", "")
+ if not os.path.isabs(root):
+ root = os.path.expanduser("~/.cache")
+ expected = initial_cache or os.path.join(root, "laya", "torchinductor")
+ assert os.environ["TORCHINDUCTOR_CACHE_DIR"] == expected
+ assert os.path.isdir(expected)
print("load_s", load_s, "cache", os.environ.get("TORCHINDUCTOR_CACHE_DIR"), flush=True)
times = []
# Repeat the sequence to expose CUDA graph recording/replay and retained-output issues.
diff --git a/benchmarks/results/compile-defaults/README.md b/benchmarks/results/compile-defaults/README.md
index 76d7dbd6..6e13b44b 100644
--- a/benchmarks/results/compile-defaults/README.md
+++ b/benchmarks/results/compile-defaults/README.md
@@ -51,3 +51,33 @@ Output: `Using Python 3.12.13 environment at: /home/ckl/projects/S/laya/.venv`;
Baseline-cold ran before code edits; use `--no-warmup` to reproduce its lazy behavior on this tree.
Warm-up shifts cost into load; it does not eliminate compilation or all possible guard specializations.
The two warm-up runs were contended. Cross-process reuse yielded two FX hits and zero misses.
+
+## Item 2: opt-in persistent Laya cache
+
+`compile_cache=True` selects `$XDG_CACHE_HOME/laya/torchinductor` or
+`~/.cache/laya/torchinductor` only if `TORCHINDUCTOR_CACHE_DIR` is absent.
+The default is false, existing settings win, and non-compiled/TileLang loads do nothing.
+The choice is explicitly process-wide; PyTorch owns cache compatibility/invalidation.
+
+The `cache-verified-*` experiments unset `TORCHINDUCTOR_CACHE_DIR`, set an isolated persistent
+XDG root, assert the selected path, and test reuse in separate processes (see `.command` files).
+The earlier `cache-cold`, `cache-restart`, and `cache-quiet-*` logs are diagnostic attempts,
+NOT measurements of the Laya cache: an early Dynamo counters import populated the default
+Inductor environment variable before the opt-in ran. This was caught by auditing the actual
+printed path. The benchmark now imports counters after load, asserts the path, and the agent
+configures the cache before model construction can import Dynamo. A regression pins that order. Full GPU state is attached to each run, not inferred from
+allocator counters. Tests cover opt-in behavior, XDG/fallback paths, explicit overrides,
+repeated setup, and constructor forwarding. `cache-verified-validation.log` records every gate and core suite after the correction.
+The final ordering assertion is rerun with `python tests/test_compile.py`
+(`cache-order-tests.log`, exit 0).
+
+| Run | Load s | Request 1 ms | Request 8 ms | Peak alloc/reserved MiB | FX hits/misses |
+|---|---:|---:|---:|---:|---:|
+| cache-verified-cold | 111.643 | 22.86 | 8.69 | 1685.6 / 1820.0 | 0 / 2 |
+| cache-verified-restart | 51.292 | 24.43 | 10.84 | 1671.0 / 1796.0 | 2 / 0 |
+
+Both verified processes selected `/home/ckl/.cache/laya-compile-defaults-verified-20261003/laya/torchinductor`.
+The second process hit both FX graphs, reducing load from 111.64 s to 51.29 s.
+The cache location adds no meaningful GPU memory requirement; both peaks match the earlier
+warm-up experiments. These final runs had no other compute jobs in their GPU snapshots.
+Reboot/container persistence was not exercised; that depends on retaining the directory.
diff --git a/benchmarks/results/compile-defaults/cache-cold.command b/benchmarks/results/compile-defaults/cache-cold.command
new file mode 100644
index 00000000..c88316b8
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-cold.command
@@ -0,0 +1 @@
+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-measure-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cache
diff --git a/benchmarks/results/compile-defaults/cache-cold.log b/benchmarks/results/compile-defaults/cache-cold.log
new file mode 100644
index 00000000..d8d34551
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-cold.log
@@ -0,0 +1,109 @@
+Sat Oct 3 18:46:26 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 30% 33C P8 12W / 295W | 3917MiB / 16376MiB | 0% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 146MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |
+| 0 N/A N/A 4072238 C ...jects/S/laya/.venv/bin/python 3184MiB |
++-----------------------------------------------------------------------------------------+
+
+torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851
+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.
+ agent = Agent(args.model, device="cuda", compile=True, **kw)
+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
+ warnings.warn(
+W1003 18:47:07.365000 4074699 torch/_inductor/utils.py:1731] [0/0_1] Not enough SMs to use max_autotune_gemm mode
+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/lowering.py:7836: UserWarning:
+Online softmax is disabled on the fly since Inductor decides to
+split the reduction. Cut an issue to PyTorch if this is an
+important use case and you want to speed it up with online
+softmax.
+
+ warnings.warn(
+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/lowering.py:7836: UserWarning:
+Online softmax is disabled on the fly since Inductor decides to
+split the reduction. Cut an issue to PyTorch if this is an
+important use case and you want to speed it up with online
+softmax.
+
+ warnings.warn(
+load_s 114.72476257698145 cache /home/ckl/.hermes/cache/scratch/torchinductor_ckl
+request 1 seconds 0.02641500800382346 graphs 2
+request 2 seconds 0.013872356968931854 graphs 2
+request 3 seconds 0.011120865936391056 graphs 2
+request 4 seconds 0.016031821025535464 graphs 2
+request 5 seconds 0.011700992938131094 graphs 2
+request 6 seconds 0.02680087205953896 graphs 2
+request 7 seconds 0.01864045101683587 graphs 2
+request 8 seconds 0.010053643956780434 graphs 2
+request 9 seconds 0.032923111924901605 graphs 2
+request 10 seconds 0.01651303598191589 graphs 2
+request 11 seconds 0.01252508198376745 graphs 2
+request 12 seconds 0.014541615033522248 graphs 2
+request 13 seconds 0.010531489038839936 graphs 2
+request 14 seconds 0.015025408938527107 graphs 2
+request 15 seconds 0.01098875398747623 graphs 2
+request 16 seconds 0.02744422003161162 graphs 2
+request 17 seconds 0.018546530045568943 graphs 2
+request 18 seconds 0.00946029694750905 graphs 2
+request 19 seconds 0.03218311397358775 graphs 2
+request 20 seconds 0.016114933067001402 graphs 2
+request 21 seconds 0.012362029985524714 graphs 2
+request 22 seconds 0.014607945107854903 graphs 2
+request 23 seconds 0.012195498100481927 graphs 2
+request 24 seconds 0.014966299058869481 graphs 2
+request 25 seconds 0.010161984013393521 graphs 2
+request 26 seconds 0.02779080392792821 graphs 2
+request 27 seconds 0.018532140064053237 graphs 2
+request 28 seconds 0.009995712898671627 graphs 2
+request 29 seconds 0.03149076597765088 graphs 2
+request 30 seconds 0.01549176499247551 graphs 2
+RESULT {"load_s": 114.72476257698145, "requests_s": [0.02641500800382346, 0.013872356968931854, 0.011120865936391056, 0.016031821025535464, 0.011700992938131094, 0.02680087205953896, 0.01864045101683587, 0.010053643956780434, 0.032923111924901605, 0.01651303598191589, 0.01252508198376745, 0.014541615033522248, 0.010531489038839936, 0.015025408938527107, 0.01098875398747623, 0.02744422003161162, 0.018546530045568943, 0.00946029694750905, 0.03218311397358775, 0.016114933067001402, 0.012362029985524714, 0.014607945107854903, 0.012195498100481927, 0.014966299058869481, 0.010161984013393521, 0.02779080392792821, 0.018532140064053237, 0.009995712898671627, 0.03149076597765088, 0.01549176499247551], "allocated_mib": 1616.48291015625, "reserved_mib": 1820.0, "peak_allocated_mib": 1685.626953125, "peak_reserved_mib": 1820.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"pattern_matcher_count": 786, "pattern_matcher_nodes": 910, "benchmarking.InductorBenchmarker.benchmark_gpu": 105, "pad_mm_bench": 4, "fxgraph_cache_miss": 2, "async_compile_cache_miss": 162, "extern_calls": 384, "async_compile_cache_hit": 81, "triton_bundler_save_kernel": 1048, "triton_bundler_save_static_autotuner": 2, "benchmarking.InductorBenchmarker.benchmark": 97}}}
+Sat Oct 3 18:48:21 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 44C P2 152W / 295W | 6631MiB / 16376MiB | 77% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 156MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |
+| 0 N/A N/A 4072238 C ...jects/S/laya/.venv/bin/python 3802MiB |
+| 0 N/A N/A 4074699 C ...jects/S/laya/.venv/bin/python 2080MiB |
++-----------------------------------------------------------------------------------------+
+
diff --git a/benchmarks/results/compile-defaults/cache-final-tests.log b/benchmarks/results/compile-defaults/cache-final-tests.log
new file mode 100644
index 00000000..fc14b254
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-final-tests.log
@@ -0,0 +1,9 @@
+ok test_cuda_compile_padding_masks_only_the_new_tail
+ok test_one_graph_across_shapes_and_same_outputs
+ok test_warmup_builds_every_graph_before_the_first_request
+ok test_duck_shape_restored_after_errors_and_nesting
+ok test_duck_shape_restored_across_threads
+ok test_eager_agent_leaves_the_setting_alone
+ok test_constructor_warms_only_the_active_compiled_path
+ok test_persistent_cache_is_opt_in_and_respects_the_environment
+all compile tests passed
diff --git a/benchmarks/results/compile-defaults/cache-order-tests.log b/benchmarks/results/compile-defaults/cache-order-tests.log
new file mode 100644
index 00000000..fc14b254
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-order-tests.log
@@ -0,0 +1,9 @@
+ok test_cuda_compile_padding_masks_only_the_new_tail
+ok test_one_graph_across_shapes_and_same_outputs
+ok test_warmup_builds_every_graph_before_the_first_request
+ok test_duck_shape_restored_after_errors_and_nesting
+ok test_duck_shape_restored_across_threads
+ok test_eager_agent_leaves_the_setting_alone
+ok test_constructor_warms_only_the_active_compiled_path
+ok test_persistent_cache_is_opt_in_and_respects_the_environment
+all compile tests passed
diff --git a/benchmarks/results/compile-defaults/cache-quiet-cold.command b/benchmarks/results/compile-defaults/cache-quiet-cold.command
new file mode 100644
index 00000000..ab595d56
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-quiet-cold.command
@@ -0,0 +1 @@
+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-quiet-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cache
diff --git a/benchmarks/results/compile-defaults/cache-quiet-cold.log b/benchmarks/results/compile-defaults/cache-quiet-cold.log
new file mode 100644
index 00000000..dacde636
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-quiet-cold.log
@@ -0,0 +1,90 @@
+Sat Oct 3 18:49:49 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 34C P8 12W / 295W | 728MiB / 16376MiB | 0% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 146MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |
++-----------------------------------------------------------------------------------------+
+
+torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851
+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.
+ agent = Agent(args.model, device="cuda", compile=True, **kw)
+load_s 49.42939719394781 cache /home/ckl/.hermes/cache/scratch/torchinductor_ckl
+request 1 seconds 0.12283430597744882 graphs 2
+request 2 seconds 0.013316553900949657 graphs 2
+request 3 seconds 0.00988961395341903 graphs 2
+request 4 seconds 0.01398233207874 graphs 2
+request 5 seconds 0.009096584981307387 graphs 2
+request 6 seconds 0.024636785965412855 graphs 2
+request 7 seconds 0.016743125044740736 graphs 2
+request 8 seconds 0.00864292006008327 graphs 2
+request 9 seconds 0.029934056918136775 graphs 2
+request 10 seconds 0.014817061950452626 graphs 2
+request 11 seconds 0.009821613901294768 graphs 2
+request 12 seconds 0.012280132039450109 graphs 2
+request 13 seconds 0.009488470037467778 graphs 2
+request 14 seconds 0.013466925942339003 graphs 2
+request 15 seconds 0.008513677981682122 graphs 2
+request 16 seconds 0.022905955906026065 graphs 2
+request 17 seconds 0.015729162958450615 graphs 2
+request 18 seconds 0.008210705011151731 graphs 2
+request 19 seconds 0.028184838010929525 graphs 2
+request 20 seconds 0.01407369296066463 graphs 2
+request 21 seconds 0.009481230052188039 graphs 2
+request 22 seconds 0.011742135975509882 graphs 2
+request 23 seconds 0.009388239006511867 graphs 2
+request 24 seconds 0.013468245975673199 graphs 2
+request 25 seconds 0.00856755895074457 graphs 2
+request 26 seconds 0.022970746038481593 graphs 2
+request 27 seconds 0.015777833061292768 graphs 2
+request 28 seconds 0.008236306020990014 graphs 2
+request 29 seconds 0.028220248059369624 graphs 2
+request 30 seconds 0.014039832982234657 graphs 2
+RESULT {"load_s": 49.42939719394781, "requests_s": [0.12283430597744882, 0.013316553900949657, 0.00988961395341903, 0.01398233207874, 0.009096584981307387, 0.024636785965412855, 0.016743125044740736, 0.00864292006008327, 0.029934056918136775, 0.014817061950452626, 0.009821613901294768, 0.012280132039450109, 0.009488470037467778, 0.013466925942339003, 0.008513677981682122, 0.022905955906026065, 0.015729162958450615, 0.008210705011151731, 0.028184838010929525, 0.01407369296066463, 0.009481230052188039, 0.011742135975509882, 0.009388239006511867, 0.013468245975673199, 0.00856755895074457, 0.022970746038481593, 0.015777833061292768, 0.008236306020990014, 0.028220248059369624, 0.014039832982234657], "allocated_mib": 1616.48291015625, "reserved_mib": 1796.0, "peak_allocated_mib": 1671.017578125, "peak_reserved_mib": 1796.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"triton_bundler_load_static_autotuner": 68, "async_compile_cache_miss": 243, "async_compile_cache_hit": 149, "pattern_matcher_count": 20, "pattern_matcher_nodes": 26, "extern_calls": 384, "fxgraph_cache_hit": 2}}}
+Sat Oct 3 18:50:39 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 43C P2 65W / 295W | 2789MiB / 16376MiB | 82% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 146MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |
+| 0 N/A N/A 4079010 C ...jects/S/laya/.venv/bin/python 2056MiB |
++-----------------------------------------------------------------------------------------+
+
diff --git a/benchmarks/results/compile-defaults/cache-quiet-restart.command b/benchmarks/results/compile-defaults/cache-quiet-restart.command
new file mode 100644
index 00000000..ab595d56
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-quiet-restart.command
@@ -0,0 +1 @@
+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-quiet-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cache
diff --git a/benchmarks/results/compile-defaults/cache-quiet-restart.log b/benchmarks/results/compile-defaults/cache-quiet-restart.log
new file mode 100644
index 00000000..8cd7293a
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-quiet-restart.log
@@ -0,0 +1,91 @@
+Sat Oct 3 18:50:43 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 35C P3 19W / 295W | 728MiB / 16376MiB | 0% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 146MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |
++-----------------------------------------------------------------------------------------+
+
+torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851
+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.
+ agent = Agent(args.model, device="cuda", compile=True, **kw)
+load_s 49.84901165601332 cache /home/ckl/.hermes/cache/scratch/torchinductor_ckl
+request 1 seconds 0.021522802067920566 graphs 2
+request 2 seconds 0.0127951999893412 graphs 2
+request 3 seconds 0.009552222094498575 graphs 2
+request 4 seconds 0.013322615996003151 graphs 2
+request 5 seconds 0.008758343057706952 graphs 2
+request 6 seconds 0.02338546304963529 graphs 2
+request 7 seconds 0.015908485976979136 graphs 2
+request 8 seconds 0.008505699923262 graphs 2
+request 9 seconds 0.02822929003741592 graphs 2
+request 10 seconds 0.014193495968356729 graphs 2
+request 11 seconds 0.009512242046184838 graphs 2
+request 12 seconds 0.011779058026149869 graphs 2
+request 13 seconds 0.009392678970471025 graphs 2
+request 14 seconds 0.013497408013790846 graphs 2
+request 15 seconds 0.008576749940402806 graphs 2
+request 16 seconds 0.022936348104849458 graphs 2
+request 17 seconds 0.01577090402133763 graphs 2
+request 18 seconds 0.0081936459755525 graphs 2
+request 19 seconds 0.028569523943588138 graphs 2
+request 20 seconds 0.015004775952547789 graphs 2
+request 21 seconds 0.009698834037408233 graphs 2
+request 22 seconds 0.011769518023356795 graphs 2
+request 23 seconds 0.009374139015562832 graphs 2
+request 24 seconds 0.013569338945671916 graphs 2
+request 25 seconds 0.008530530030839145 graphs 2
+request 26 seconds 0.02300787903368473 graphs 2
+request 27 seconds 0.015862375032156706 graphs 2
+request 28 seconds 0.00838116800878197 graphs 2
+request 29 seconds 0.028341430937871337 graphs 2
+request 30 seconds 0.014030663995072246 graphs 2
+RESULT {"load_s": 49.84901165601332, "requests_s": [0.021522802067920566, 0.0127951999893412, 0.009552222094498575, 0.013322615996003151, 0.008758343057706952, 0.02338546304963529, 0.015908485976979136, 0.008505699923262, 0.02822929003741592, 0.014193495968356729, 0.009512242046184838, 0.011779058026149869, 0.009392678970471025, 0.013497408013790846, 0.008576749940402806, 0.022936348104849458, 0.01577090402133763, 0.0081936459755525, 0.028569523943588138, 0.015004775952547789, 0.009698834037408233, 0.011769518023356795, 0.009374139015562832, 0.013569338945671916, 0.008530530030839145, 0.02300787903368473, 0.015862375032156706, 0.00838116800878197, 0.028341430937871337, 0.014030663995072246], "allocated_mib": 1616.48291015625, "reserved_mib": 1796.0, "peak_allocated_mib": 1671.017578125, "peak_reserved_mib": 1796.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"triton_bundler_load_static_autotuner": 68, "async_compile_cache_miss": 243, "async_compile_cache_hit": 149, "pattern_matcher_count": 20, "pattern_matcher_nodes": 26, "extern_calls": 384, "fxgraph_cache_hit": 2}}}
+Sat Oct 3 18:51:33 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 48C P2 105W / 295W | 3134MiB / 16376MiB | 82% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 146MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |
+| 0 N/A N/A 4079529 C ...jects/S/laya/.venv/bin/python 2056MiB |
+| 0 N/A N/A 4079725 C ...jects/S/laya/.venv/bin/python 340MiB |
++-----------------------------------------------------------------------------------------+
+
diff --git a/benchmarks/results/compile-defaults/cache-restart.command b/benchmarks/results/compile-defaults/cache-restart.command
new file mode 100644
index 00000000..c88316b8
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-restart.command
@@ -0,0 +1 @@
+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-measure-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cache
diff --git a/benchmarks/results/compile-defaults/cache-restart.log b/benchmarks/results/compile-defaults/cache-restart.log
new file mode 100644
index 00000000..da18d671
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-restart.log
@@ -0,0 +1,91 @@
+Sat Oct 3 18:48:27 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 33C P8 12W / 295W | 4545MiB / 16376MiB | 26% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 156MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |
+| 0 N/A N/A 4072238 C ...jects/S/laya/.venv/bin/python 3802MiB |
++-----------------------------------------------------------------------------------------+
+
+torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851
+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.
+ agent = Agent(args.model, device="cuda", compile=True, **kw)
+load_s 53.14927826402709 cache /home/ckl/.hermes/cache/scratch/torchinductor_ckl
+request 1 seconds 0.02442872989922762 graphs 2
+request 2 seconds 0.014553737011738122 graphs 2
+request 3 seconds 0.009722160990349948 graphs 2
+request 4 seconds 0.015148414066061378 graphs 2
+request 5 seconds 0.010848594014532864 graphs 2
+request 6 seconds 0.02579907502513379 graphs 2
+request 7 seconds 0.01758231094572693 graphs 2
+request 8 seconds 0.010467909974977374 graphs 2
+request 9 seconds 0.03215115796774626 graphs 2
+request 10 seconds 0.016135973972268403 graphs 2
+request 11 seconds 0.01121161796618253 graphs 2
+request 12 seconds 0.011908155982382596 graphs 2
+request 13 seconds 0.01099822495598346 graphs 2
+request 14 seconds 0.01532110502012074 graphs 2
+request 15 seconds 0.010279048001393676 graphs 2
+request 16 seconds 0.02483839401975274 graphs 2
+request 17 seconds 0.017773673054762185 graphs 2
+request 18 seconds 0.009426687960512936 graphs 2
+request 19 seconds 0.03183845500461757 graphs 2
+request 20 seconds 0.015550486976280808 graphs 2
+request 21 seconds 0.009742891998030245 graphs 2
+request 22 seconds 0.01376086799427867 graphs 2
+request 23 seconds 0.011247409041970968 graphs 2
+request 24 seconds 0.014977620914578438 graphs 2
+request 25 seconds 0.00899228302296251 graphs 2
+request 26 seconds 0.02644445199985057 graphs 2
+request 27 seconds 0.01784457394387573 graphs 2
+request 28 seconds 0.008444887003861368 graphs 2
+request 29 seconds 0.031590171973221004 graphs 2
+request 30 seconds 0.01599324296694249 graphs 2
+RESULT {"load_s": 53.14927826402709, "requests_s": [0.02442872989922762, 0.014553737011738122, 0.009722160990349948, 0.015148414066061378, 0.010848594014532864, 0.02579907502513379, 0.01758231094572693, 0.010467909974977374, 0.03215115796774626, 0.016135973972268403, 0.01121161796618253, 0.011908155982382596, 0.01099822495598346, 0.01532110502012074, 0.010279048001393676, 0.02483839401975274, 0.017773673054762185, 0.009426687960512936, 0.03183845500461757, 0.015550486976280808, 0.009742891998030245, 0.01376086799427867, 0.011247409041970968, 0.014977620914578438, 0.00899228302296251, 0.02644445199985057, 0.01784457394387573, 0.008444887003861368, 0.031590171973221004, 0.01599324296694249], "allocated_mib": 1616.48291015625, "reserved_mib": 1796.0, "peak_allocated_mib": 1671.017578125, "peak_reserved_mib": 1796.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"triton_bundler_load_static_autotuner": 68, "async_compile_cache_miss": 243, "async_compile_cache_hit": 149, "pattern_matcher_count": 20, "pattern_matcher_nodes": 26, "extern_calls": 384, "fxgraph_cache_hit": 2}}}
+Sat Oct 3 18:49:21 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 44C P2 87W / 295W | 2799MiB / 16376MiB | 81% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 156MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |
+| 0 N/A N/A 4077174 C ...jects/S/laya/.venv/bin/python 2056MiB |
++-----------------------------------------------------------------------------------------+
+
diff --git a/benchmarks/results/compile-defaults/cache-validation.log b/benchmarks/results/compile-defaults/cache-validation.log
new file mode 100644
index 00000000..55018529
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-validation.log
@@ -0,0 +1,57 @@
+PATH=/home/ckl/projects/S/laya/.venv/bin:$PATH
+
+$ ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
+All checks passed!
+
+exit=0
+
+$ python -m compileall -q laya/ tests/
+
+exit=0
+
+$ python tests/test_compile.py
+ok test_cuda_compile_padding_masks_only_the_new_tail
+ok test_one_graph_across_shapes_and_same_outputs
+ok test_warmup_builds_every_graph_before_the_first_request
+ok test_duck_shape_restored_after_errors_and_nesting
+ok test_duck_shape_restored_across_threads
+ok test_eager_agent_leaves_the_setting_alone
+ok test_constructor_warms_only_the_active_compiled_path
+ok test_persistent_cache_is_opt_in_and_respects_the_environment
+all compile tests passed
+
+exit=0
+
+$ python tests/test_hooks_api.py
+
+401 passed, 0 failed
+all hook API tests passed
+
+exit=0
+
+$ python tests/test_router.py
+
+703 passed, 0 failed
+all routing tests passed
+
+exit=0
+
+$ python tests/test_criteria.py
+
+198 passed, 0 failed
+
+exit=0
+
+$ python tests/test_hooks.py
+
+240 passed, 0 failed
+all hook tests passed
+
+exit=0
+
+$ zensical build --strict --clean
+Build started
+No issues found
+Build finished in 2.06s
+
+exit=0
diff --git a/benchmarks/results/compile-defaults/cache-verified-cold.command b/benchmarks/results/compile-defaults/cache-verified-cold.command
new file mode 100644
index 00000000..2d578752
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-verified-cold.command
@@ -0,0 +1 @@
+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-verified-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cache
diff --git a/benchmarks/results/compile-defaults/cache-verified-cold.log b/benchmarks/results/compile-defaults/cache-verified-cold.log
new file mode 100644
index 00000000..c63e4b6a
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-verified-cold.log
@@ -0,0 +1,107 @@
+Sat Oct 3 18:53:03 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 36C P8 12W / 295W | 738MiB / 16376MiB | 28% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 156MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 36MiB |
++-----------------------------------------------------------------------------------------+
+
+torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851
+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.
+ agent = Agent(args.model, device="cuda", compile=True, **kw)
+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
+ warnings.warn(
+W1003 18:53:44.221000 4081427 torch/_inductor/utils.py:1731] [0/0_1] Not enough SMs to use max_autotune_gemm mode
+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/lowering.py:7836: UserWarning:
+Online softmax is disabled on the fly since Inductor decides to
+split the reduction. Cut an issue to PyTorch if this is an
+important use case and you want to speed it up with online
+softmax.
+
+ warnings.warn(
+/home/ckl/projects/S/laya/.venv/lib/python3.12/site-packages/torch/_inductor/lowering.py:7836: UserWarning:
+Online softmax is disabled on the fly since Inductor decides to
+split the reduction. Cut an issue to PyTorch if this is an
+important use case and you want to speed it up with online
+softmax.
+
+ warnings.warn(
+load_s 111.64288430591114 cache /home/ckl/.cache/laya-compile-defaults-verified-20261003/laya/torchinductor
+request 1 seconds 0.02286072401329875 graphs 2
+request 2 seconds 0.012921674991957843 graphs 2
+request 3 seconds 0.009732725913636386 graphs 2
+request 4 seconds 0.01357605296652764 graphs 2
+request 5 seconds 0.008746225037612021 graphs 2
+request 6 seconds 0.023331869044341147 graphs 2
+request 7 seconds 0.01587031001690775 graphs 2
+request 8 seconds 0.008692333940416574 graphs 2
+request 9 seconds 0.0283058590721339 graphs 2
+request 10 seconds 0.013749895035289228 graphs 2
+request 11 seconds 0.009646686026826501 graphs 2
+request 12 seconds 0.012101444997824728 graphs 2
+request 13 seconds 0.009533305070362985 graphs 2
+request 14 seconds 0.013106117956340313 graphs 2
+request 15 seconds 0.008559482987038791 graphs 2
+request 16 seconds 0.023173767956905067 graphs 2
+request 17 seconds 0.015705847996287048 graphs 2
+request 18 seconds 0.008421211037784815 graphs 2
+request 19 seconds 0.028346410021185875 graphs 2
+request 20 seconds 0.013642733916640282 graphs 2
+request 21 seconds 0.009569393936544657 graphs 2
+request 22 seconds 0.011785740964114666 graphs 2
+request 23 seconds 0.009457602980546653 graphs 2
+request 24 seconds 0.01304219604935497 graphs 2
+request 25 seconds 0.008564352057874203 graphs 2
+request 26 seconds 0.023065467015840113 graphs 2
+request 27 seconds 0.015712207998149097 graphs 2
+request 28 seconds 0.008489751955494285 graphs 2
+request 29 seconds 0.02870765409898013 graphs 2
+request 30 seconds 0.016941983019933105 graphs 2
+RESULT {"load_s": 111.64288430591114, "requests_s": [0.02286072401329875, 0.012921674991957843, 0.009732725913636386, 0.01357605296652764, 0.008746225037612021, 0.023331869044341147, 0.01587031001690775, 0.008692333940416574, 0.0283058590721339, 0.013749895035289228, 0.009646686026826501, 0.012101444997824728, 0.009533305070362985, 0.013106117956340313, 0.008559482987038791, 0.023173767956905067, 0.015705847996287048, 0.008421211037784815, 0.028346410021185875, 0.013642733916640282, 0.009569393936544657, 0.011785740964114666, 0.009457602980546653, 0.01304219604935497, 0.008564352057874203, 0.023065467015840113, 0.015712207998149097, 0.008489751955494285, 0.02870765409898013, 0.016941983019933105], "allocated_mib": 1616.48291015625, "reserved_mib": 1820.0, "peak_allocated_mib": 1685.626953125, "peak_reserved_mib": 1820.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"pattern_matcher_count": 786, "pattern_matcher_nodes": 910, "benchmarking.InductorBenchmarker.benchmark_gpu": 105, "pad_mm_bench": 4, "fxgraph_cache_miss": 2, "async_compile_cache_miss": 162, "extern_calls": 384, "async_compile_cache_hit": 81, "triton_bundler_save_kernel": 1048, "triton_bundler_save_static_autotuner": 2, "benchmarking.InductorBenchmarker.benchmark": 97}}}
+Sat Oct 3 18:54:55 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 47C P2 158W / 295W | 2823MiB / 16376MiB | 83% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 156MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |
+| 0 N/A N/A 4081427 C ...jects/S/laya/.venv/bin/python 2080MiB |
++-----------------------------------------------------------------------------------------+
+
diff --git a/benchmarks/results/compile-defaults/cache-verified-restart.command b/benchmarks/results/compile-defaults/cache-verified-restart.command
new file mode 100644
index 00000000..2d578752
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-verified-restart.command
@@ -0,0 +1 @@
+env -u TORCHINDUCTOR_CACHE_DIR XDG_CACHE_HOME=/home/ckl/.cache/laya-compile-defaults-verified-20261003 HF_HUB_OFFLINE=1 /home/ckl/projects/S/laya/.venv/bin/python benchmarks/bench_compile_defaults.py --model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 --cache
diff --git a/benchmarks/results/compile-defaults/cache-verified-restart.log b/benchmarks/results/compile-defaults/cache-verified-restart.log
new file mode 100644
index 00000000..71042c88
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-verified-restart.log
@@ -0,0 +1,90 @@
+Sat Oct 3 18:55:00 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 36C P3 19W / 295W | 738MiB / 16376MiB | 10% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 156MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 35MiB |
++-----------------------------------------------------------------------------------------+
+
+torch 2.11.0+cu130 model /home/ckl/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851
+/home/ckl/projects/S/laya-wt/compile-defaults/benchmarks/bench_compile_defaults.py:41: RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.
+ agent = Agent(args.model, device="cuda", compile=True, **kw)
+load_s 51.29203179304022 cache /home/ckl/.cache/laya-compile-defaults-verified-20261003/laya/torchinductor
+request 1 seconds 0.024434805032797158 graphs 2
+request 2 seconds 0.013582332991063595 graphs 2
+request 3 seconds 0.011464047944173217 graphs 2
+request 4 seconds 0.015985592966899276 graphs 2
+request 5 seconds 0.010889400960877538 graphs 2
+request 6 seconds 0.026401367969810963 graphs 2
+request 7 seconds 0.018746355082839727 graphs 2
+request 8 seconds 0.010839381022378802 graphs 2
+request 9 seconds 0.03360046504531056 graphs 2
+request 10 seconds 0.01667865097988397 graphs 2
+request 11 seconds 0.011249065981246531 graphs 2
+request 12 seconds 0.012481120997108519 graphs 2
+request 13 seconds 0.00973973807413131 graphs 2
+request 14 seconds 0.015411366010084748 graphs 2
+request 15 seconds 0.010465405997820199 graphs 2
+request 16 seconds 0.02478666801471263 graphs 2
+request 17 seconds 0.017652343027293682 graphs 2
+request 18 seconds 0.009124230011366308 graphs 2
+request 19 seconds 0.031999476021155715 graphs 2
+request 20 seconds 0.016068443073891103 graphs 2
+request 21 seconds 0.009568715933710337 graphs 2
+request 22 seconds 0.013553263037465513 graphs 2
+request 23 seconds 0.011064652935601771 graphs 2
+request 24 seconds 0.015312475035898387 graphs 2
+request 25 seconds 0.008710375055670738 graphs 2
+request 26 seconds 0.026597779942676425 graphs 2
+request 27 seconds 0.01785830594599247 graphs 2
+request 28 seconds 0.008381210966035724 graphs 2
+request 29 seconds 0.03149348997976631 graphs 2
+request 30 seconds 0.01582511095330119 graphs 2
+RESULT {"load_s": 51.29203179304022, "requests_s": [0.024434805032797158, 0.013582332991063595, 0.011464047944173217, 0.015985592966899276, 0.010889400960877538, 0.026401367969810963, 0.018746355082839727, 0.010839381022378802, 0.03360046504531056, 0.01667865097988397, 0.011249065981246531, 0.012481120997108519, 0.00973973807413131, 0.015411366010084748, 0.010465405997820199, 0.02478666801471263, 0.017652343027293682, 0.009124230011366308, 0.031999476021155715, 0.016068443073891103, 0.009568715933710337, 0.013553263037465513, 0.011064652935601771, 0.015312475035898387, 0.008710375055670738, 0.026597779942676425, 0.01785830594599247, 0.008381210966035724, 0.03149348997976631, 0.01582511095330119], "allocated_mib": 1616.48291015625, "reserved_mib": 1796.0, "peak_allocated_mib": 1671.017578125, "peak_reserved_mib": 1796.0, "counters": {"stats": {"calls_captured": 6700, "unique_graphs": 2}, "inductor": {"triton_bundler_load_static_autotuner": 68, "async_compile_cache_miss": 243, "async_compile_cache_hit": 149, "pattern_matcher_count": 20, "pattern_matcher_nodes": 26, "extern_calls": 384, "fxgraph_cache_hit": 2}}}
+Sat Oct 3 18:55:52 2026
++-----------------------------------------------------------------------------------------+
+| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
++-----------------------------------------+------------------------+----------------------+
+| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
+| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
+| | | MIG M. |
+|=========================================+========================+======================|
+| 0 NVIDIA GeForce RTX 4070 ... On | 00000000:26:00.0 Off | N/A |
+| 0% 47C P2 74W / 295W | 2795MiB / 16376MiB | 84% Default |
+| | | N/A |
++-----------------------------------------+------------------------+----------------------+
+
++-----------------------------------------------------------------------------------------+
+| Processes: |
+| GPU GI CI PID Type Process name GPU Memory |
+| ID ID Usage |
+|=========================================================================================|
+| 0 N/A N/A 892 G Hyprland 156MiB |
+| 0 N/A N/A 952 G quickshell 38MiB |
+| 0 N/A N/A 995 G Xwayland 4MiB |
+| 0 N/A N/A 1168 G monocloud 3MiB |
+| 0 N/A N/A 3324 C+G /usr/bin/sunshine 215MiB |
+| 0 N/A N/A 12883 G /usr/bin/ghostty 100MiB |
+| 0 N/A N/A 3906709 C+G ...rack-uuid=3190708988185955192 31MiB |
+| 0 N/A N/A 4083959 C ...jects/S/laya/.venv/bin/python 2056MiB |
++-----------------------------------------------------------------------------------------+
+
diff --git a/benchmarks/results/compile-defaults/cache-verified-validation.log b/benchmarks/results/compile-defaults/cache-verified-validation.log
new file mode 100644
index 00000000..022075f3
--- /dev/null
+++ b/benchmarks/results/compile-defaults/cache-verified-validation.log
@@ -0,0 +1,57 @@
+PATH=/home/ckl/projects/S/laya/.venv/bin:$PATH
+
+$ ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
+All checks passed!
+
+exit=0
+
+$ python -m compileall -q laya/ tests/
+
+exit=0
+
+$ python tests/test_compile.py
+ok test_cuda_compile_padding_masks_only_the_new_tail
+ok test_one_graph_across_shapes_and_same_outputs
+ok test_warmup_builds_every_graph_before_the_first_request
+ok test_duck_shape_restored_after_errors_and_nesting
+ok test_duck_shape_restored_across_threads
+ok test_eager_agent_leaves_the_setting_alone
+ok test_constructor_warms_only_the_active_compiled_path
+ok test_persistent_cache_is_opt_in_and_respects_the_environment
+all compile tests passed
+
+exit=0
+
+$ python tests/test_hooks_api.py
+
+401 passed, 0 failed
+all hook API tests passed
+
+exit=0
+
+$ python tests/test_router.py
+
+703 passed, 0 failed
+all routing tests passed
+
+exit=0
+
+$ python tests/test_criteria.py
+
+198 passed, 0 failed
+
+exit=0
+
+$ python tests/test_hooks.py
+
+240 passed, 0 failed
+all hook tests passed
+
+exit=0
+
+$ zensical build --strict --clean
+Build started
+No issues found
+Build finished in 2.09s
+
+exit=0
diff --git a/docs/compile-and-fast-path.md b/docs/compile-and-fast-path.md
index c3ea2205..0076af49 100644
--- a/docs/compile-and-fast-path.md
+++ b/docs/compile-and-fast-path.md
@@ -31,6 +31,14 @@ by sequence length, so it has no such buffer.
container restart. Set it to a persistent directory, or a volume in a container, and a second
process loads the graphs instead of compiling them. In #472's measurement that took warm-up from
about 120 s to about 50 s.
+- **Laya cache opt-in.** `laya.load(..., compile=True, compile_cache=True)` sets the process-wide
+ `TORCHINDUCTOR_CACHE_DIR` only when absent, to `$XDG_CACHE_HOME/laya/torchinductor` or
+ `~/.cache/laya/torchinductor` when XDG is unset or not absolute. An existing setting, including
+ one set by an earlier PyTorch compile, wins. The directory is created at load; filesystem
+ errors propagate. `compile_cache=False` (default), eager, and TileLang loads leave the
+ environment alone. This does not move or delete old caches. Containers still need a persistent
+ home/volume. Cache compatibility and invalidation are managed by PyTorch; a GPU, torch,
+ compiler, model, or input guard change can require compilation again.
## AOTInductor: not yet
diff --git a/laya/_compile.py b/laya/_compile.py
index aa1e5326..852a8442 100644
--- a/laya/_compile.py
+++ b/laya/_compile.py
@@ -26,6 +26,7 @@ Mode and device are unchanged from what `compile=True` has always done: the defa
on whatever device the agent runs on, CPU included.
"""
import threading
+import os
from contextlib import contextmanager
from contextvars import ContextVar
@@ -41,6 +42,16 @@ def compile_model(model, **kwargs):
return torch.compile(model, dynamic=True, **kwargs)
+def configure_cache():
+ """Respect an existing Inductor directory; otherwise use the user's Laya cache."""
+ root = os.environ.get("XDG_CACHE_HOME", "")
+ if not os.path.isabs(root):
+ root = os.path.expanduser("~/.cache")
+ directory = os.environ.setdefault("TORCHINDUCTOR_CACHE_DIR", os.path.join(root, "laya", "torchinductor"))
+ os.makedirs(directory, exist_ok=True)
+ return directory
+
+
def _fx_config():
try:
from torch.fx.experimental import _config
diff --git a/laya/agent.py b/laya/agent.py
index e9a523d9..354b99c4 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -13,7 +13,7 @@ from typing import Any, Dict, List, Optional, Union
import numpy as np
import torch
-from ._compile import compile_model, independent_dims
+from ._compile import compile_model, independent_dims, configure_cache
from .calibrate import (
_install_temperatures,
apply_calibration_payload,
@@ -493,6 +493,7 @@ class Agent(HookRegistry):
hooks_timeout: Optional[float] = None,
calibration: Optional[str] = None,
compile_warmup: bool = True,
+ compile_cache: bool = False,
):
"""Load a Laya checkpoint.
@@ -513,6 +514,8 @@ class Agent(HookRegistry):
TileLang path replaces the forward that would be compiled.
Compiled agents run `warmup()` before returning; `compile_warmup=False` defers that
work to requests or a manual `warmup()` call. Eager and fast agents are unchanged.
+ `compile_cache=True` opts into a persistent Laya Inductor directory (process-wide),
+ respecting any existing `TORCHINDUCTOR_CACHE_DIR`; see the compile engineering notes.
`subfolder` selects one checkpoint from a repo that bundles several, e.g.
`Agent("convaiinnovations/laya", subfolder="multilingual")`. Only that subfolder is
@@ -527,6 +530,9 @@ class Agent(HookRegistry):
`hooks_concurrent=False` serialises hooks that are not safe to run in parallel, and
`hooks_timeout` bounds each hook call in seconds (None means no limit).
"""
+ # Model loading can import Dynamo, which populates Inductor's default directory.
+ if compile and not fast and compile_cache:
+ configure_cache()
self.hooks = normalise_hooks(hooks, on_predict_start, on_predict_end)
self.hooks_raise = bool(hooks_raise)
self.hooks_concurrent = bool(hooks_concurrent)
@@ -1913,7 +1919,8 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]
hooks=None, on_predict_start=None, on_predict_end=None,
hooks_raise: bool = True, hooks_concurrent: bool = True,
hooks_timeout: Optional[float] = None,
- calibration: Optional[str] = None, compile_warmup: bool = True) -> Agent:
+ calibration: Optional[str] = None, compile_warmup: bool = True,
+ compile_cache: bool = False) -> Agent:
"""Load a Laya agent.
`subfolder` picks one checkpoint out of a repo that bundles several:
@@ -1949,7 +1956,7 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]
if spec is not None:
model_id_or_path, subfolder = spec
return Agent(model_id_or_path, device=device, token=token, subfolder=subfolder, fast=fast,
- compile=compile, compile_warmup=compile_warmup,
+ compile=compile, compile_warmup=compile_warmup, compile_cache=compile_cache,
revision=revision, expected_sha256=expected_sha256,
lang_temperatures=lang_temperatures,
hooks=hooks, on_predict_start=on_predict_start, on_predict_end=on_predict_end,
diff --git a/tests/test_compile.py b/tests/test_compile.py
index e8801162..f3426aac 100644
--- a/tests/test_compile.py
+++ b/tests/test_compile.py
@@ -182,14 +182,16 @@ def test_constructor_warms_only_the_active_compiled_path():
from laya import load
model = tiny_model()
+ events = []
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
(root / "rl_agent_config.json").write_text(json.dumps({"encoder": "unused", "head_layers": 1}))
(root / "model.safetensors").touch()
- with patch.object(module, "build_model", return_value=model), \
+ with patch.object(module, "build_model", side_effect=lambda *a, **kw: events.append("model") or model), \
patch.object(module, "_load_tokenizer", return_value=SimpleNamespace(cls_token_id=1)), \
patch("safetensors.torch.load_file", return_value=model.state_dict()), \
- patch.object(module, "compile_model", side_effect=lambda m: m) as compile_spy, \
+ patch.object(module, "compile_model", side_effect=lambda m, **kw: m) as compile_spy, \
+ patch.object(module, "configure_cache", side_effect=lambda: events.append("cache")) as cache_spy, \
patch.object(Agent, "accelerate"), \
patch.object(Agent, "warmup", autospec=True) as warm:
agent = load(directory, device="cpu", compile=True)
@@ -203,6 +205,46 @@ def test_constructor_warms_only_the_active_compiled_path():
assert compile_spy.call_count == 2
agent.warmup()
warm.assert_called_once_with(agent)
+ cache_spy.assert_not_called()
+ events.clear()
+ load(directory, device="cpu", compile=True, compile_cache=True, compile_warmup=False)
+ assert events == ["cache", "model"]
+ cache_spy.assert_called_once_with()
+ cache_spy.reset_mock()
+ compile_spy.reset_mock()
+ load(directory, device="cpu", compile=False, compile_cache=True)
+ load(directory, device="cpu", compile=True, fast=True, compile_cache=True)
+ compile_spy.assert_not_called()
+ cache_spy.assert_not_called()
+
+
+def test_persistent_cache_is_opt_in_and_respects_the_environment():
+ import tempfile
+ from pathlib import Path
+ from unittest.mock import patch
+ from laya._compile import configure_cache
+
+ with tempfile.TemporaryDirectory() as directory:
+ with patch.dict(os.environ, {"XDG_CACHE_HOME": directory + "/xdg"}, clear=True), \
+ patch("os.path.expanduser", return_value=directory + "/.cache"), \
+ patch("torch.compile") as compiler:
+ compile_model(object())
+ assert "TORCHINDUCTOR_CACHE_DIR" not in os.environ
+ configure_cache()
+ expected = str(Path(directory) / "xdg/laya/torchinductor")
+ assert os.environ["TORCHINDUCTOR_CACHE_DIR"] == expected
+ assert Path(expected).is_dir()
+ assert compiler.call_args.kwargs == {"dynamic": True}
+ # A caller-selected directory always wins, including after repeated loads.
+ os.environ["TORCHINDUCTOR_CACHE_DIR"] = directory + "/explicit"
+ assert configure_cache() == directory + "/explicit"
+ assert configure_cache() == directory + "/explicit"
+ del os.environ["TORCHINDUCTOR_CACHE_DIR"]
+ os.environ["XDG_CACHE_HOME"] = "relative-is-invalid"
+ assert configure_cache() == directory + "/.cache/laya/torchinductor"
+ del os.environ["TORCHINDUCTOR_CACHE_DIR"]
+ del os.environ["XDG_CACHE_HOME"]
+ assert configure_cache() == directory + "/.cache/laya/torchinductor"
if __name__ == "__main__":
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
index 3e889433..528527a3 100644
--- a/tests/test_hooks_api.py
+++ b/tests/test_hooks_api.py
@@ -77,6 +77,7 @@ for label, fn in (("Agent.__init__", Agent.__init__), ("load", load),
for label, fn in (("Agent.__init__", Agent.__init__), ("load", load)):
check_param(label, fn, "compile_warmup", True)
+ check_param(label, fn, "compile_cache", False)
# Router keeps lang_guess and explicit per-model revisions too
check_param("Router.__init__", Router.__init__, "lang_guess", None)