Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Merge pull request #810 from Swatantra-66/fix/concurrent-gpu-oom-fallback

Nandakishor8584c3525960
diff --cc laya/agent.py
index 1f73a35c,b376706c..4780b379
--- a/laya/agent.py
+++ b/laya/agent.py
@@@ -664,12 -612,12 +713,13 @@@ class Agent(HookRegistry)
self.amp_enabled = True
self.dtype = torch.bfloat16
elif self.device.type == "cpu":
- if os.environ.get("LAYA_CPU_AMP", "").lower() in ("bf16", "bfloat16"):
+ cpu_amp = _cpu_amp_dtype()
+ if cpu_amp is not None:
self.amp_enabled = True
- self.dtype = torch.bfloat16
+ self.dtype = cpu_amp
self._fast = None
+ self._gate = _InferenceGate()
# 2. Place on device with graceful fallback to CPU on memory error
fell_back_from = fell_back_why = None