Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(compile): set use_duck_shape per thread on torch that keeps config per thread
cklxxe580aaa532a6
diff --git a/laya/_compile.py b/laya/_compile.pyindex a3ad5cd6..aa1e5326 100644--- a/laya/_compile.py+++ b/laya/_compile.py@@ -15,16 +15,19 @@ switches duck sizing off while a compiled forward runs, which removes the secondof Laya's inputs is independent, so each gets its own symbol. What is left is torch's own 0/1specialisation, one extra graph the first time a single-row batch arrives.-`use_duck_shape` is a process-global torch setting read when a graph is traced, which happens lazily-inside a call, so it cannot be set once at load and restored. It is set for the duration of each-compiled forward instead and put back to whatever it was when the last one returns (refcounted, so-concurrent calls on several threads do not restore it early). Nothing is left changed between calls.+`use_duck_shape` is read when a graph is traced, which happens lazily inside a call, so it cannot be+set once at load and restored. It is set for the duration of each compiled forward instead. In older+torch (2.11, for one) the setting is process-global, so it is put back to whatever it was when the+last call returns (refcounted, so concurrent calls on several threads do not restore it early); newer+torch (2.14) keeps config overrides per thread (a `ContextVar` per entry), so each call sets and+restores its own thread's value with the config's `patch`. Nothing is left changed between calls.Mode and device are unchanged from what `compile=True` has always done: the default inductor mode,on whatever device the agent runs on, CPU included."""import threadingfrom contextlib import contextmanager+from contextvars import ContextVarimport torch@@ -46,6 +49,12 @@ def _fx_config():return _config if hasattr(_config, "use_duck_shape") else None+def _per_thread(cfg):+ """True where a config override only applies to the thread that set it (torch 2.14; 2.11 is global)."""+ entry = getattr(cfg, "_config", {}).get("use_duck_shape")+ return isinstance(getattr(entry, "user_override", None), ContextVar)++@contextmanagerdef independent_dims():"""Trace with `use_duck_shape = False` inside the block, restoring the prior value afterwards."""@@ -54,6 +63,11 @@ def independent_dims():if cfg is None:yieldreturn+ if _per_thread(cfg):+ # each thread traces with its own settings, so each call sets and restores its own+ with cfg.patch(use_duck_shape=False):+ yield+ returnwith _lock:if _depth == 0:_saved = cfg.use_duck_shapediff --git a/tests/test_compile.py b/tests/test_compile.pyindex fe48d3c5..8abc4983 100644--- a/tests/test_compile.py+++ b/tests/test_compile.py@@ -96,14 +96,16 @@ def test_duck_shape_restored_after_errors_and_nesting():def test_duck_shape_restored_across_threads():before = fx_config.use_duck_shape- inside, release = threading.Barrier(4), threading.Event()- seen = []+ inside, left, release = threading.Barrier(4), threading.Barrier(4), threading.Event()+ seen, after = [], []def call():with independent_dims():inside.wait()seen.append(fx_config.use_duck_shape)release.wait()+ left.wait() # read once every call has returned, global or per-thread setting alike+ after.append(fx_config.use_duck_shape)threads = [threading.Thread(target=call) for _ in range(4)]for t in threads:@@ -114,6 +116,7 @@ def test_duck_shape_restored_across_threads():for t in threads:t.join()assert seen == [False] * 4+ assert after == [before] * 4assert fx_config.use_duck_shape is before