Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

release: 0.4.26, laya[vision] and laya[audio]

Nandakishor3293116c8942
diff --git a/README.md b/README.md
index cc79bb8b..427ab7ed 100644
--- a/README.md
+++ b/README.md
@@ -29,7 +29,7 @@ python -m pip install laya
With [uv](https://docs.astral.sh/uv/), run `uv add laya` in a uv project or `uv pip install laya` in a virtual environment.
-Python 3.10 or newer. Optional extras: `laya[serve]` (HTTP server), `laya[mcp]` (MCP server), `laya[langchain]` (LangChain and LangGraph), `laya[llamaindex]` (LlamaIndex selectors), `laya[crewai]` (CrewAI routing), `laya[onnx]` (ONNX Runtime), `laya[fast]` (TileLang GPU fast path), `laya[structured]` (pydantic models in `decide`). Step-by-step setup for each platform, CPU-only or GPU PyTorch builds, and troubleshooting are in [Installation details](#installation-details).
+Python 3.10 or newer. Optional extras: `laya[serve]` (HTTP server), `laya[mcp]` (MCP server), `laya[langchain]` (LangChain and LangGraph), `laya[llamaindex]` (LlamaIndex selectors), `laya[crewai]` (CrewAI routing), `laya[onnx]` (ONNX Runtime), `laya[fast]` (TileLang GPU fast path), `laya[structured]` (pydantic models in `decide`), `laya[vision]` (Pillow and torchvision, for `predict(image=...)` and `predict(video=...)`), `laya[audio]` (soundfile, to decode audio files). Step-by-step setup for each platform, CPU-only or GPU PyTorch builds, and troubleshooting are in [Installation details](#installation-details).
For TypeScript / Node.js / browser, see [`laya-ts/`](https://github.com/NandhaKishorM/laya/tree/main/laya-ts/). npm releases (`npm install laya-ts`) are published from this repository's `laya-ts-v*` release tags.
@@ -107,8 +107,9 @@ The rule, in order: an image or audio goes to vega; text longer than the fast mo
# text: up to 73,728 tokens in one pass; usage["truncated"] says if a state hit the limit
agent.predict_long(seventy_page_contract, questions)
# image: PIL image, path or HxWx3 array, any format Pillow opens; scaled to ~1 megapixel, no file-size cap
+# needs pip install 'laya[vision]' (Pillow and torchvision), as does video
agent.predict(image="invoice.png", questions=questions)
-# audio: path or bytes (wav, flac, ogg, mp3; needs soundfile) or a 1-D waveform with sampling_rate=;
+# audio: path or bytes (wav, flac, ogg, mp3; needs pip install 'laya[audio]') or a 1-D waveform with sampling_rate=;
# any length, resampled to 16 kHz, transcribed in 30 s windows, transcript counts toward the 73,728 tokens
agent.predict(audio=wave, questions=questions, sampling_rate=16000)
# video: path or bytes, any container ffmpeg opens, any length and resolution (needs the ffmpeg binary);
@@ -243,6 +244,15 @@ The shipped checkpoints work zero-shot, but fine-tuning on decisions from your o
**[nandhakishorm.github.io/laya](https://nandhakishorm.github.io/laya/)**: guides for [prediction hooks](https://nandhakishorm.github.io/laya/hooks/), [schema-driven decisions](https://nandhakishorm.github.io/laya/structured/), [Docker](https://nandhakishorm.github.io/laya/docker/) and [LangChain and LangGraph](https://nandhakishorm.github.io/laya/langchain/), plus a full [API reference](https://nandhakishorm.github.io/laya/reference/).
+## What's new in 0.4.26
+
+* **`laya[vision]` and `laya[audio]`.** Images and video through the vega lane need Pillow and
+ torchvision, and audio files need soundfile; none of them is a core dependency, and a fresh
+ install used to fail inside transformers with a message about an image processor backend. The
+ two extras install exactly what each path needs, and `predict(image=...)`, `predict(video=...)`
+ and `predict(audio=path)` now name the extra when it is missing. Text and a numpy waveform need
+ nothing beyond `pip install laya`.
+
## What's new in 0.4.25
* **The `Router` has a vega lane.** When vegaml is installed, `Router()` routes what the encoders
diff --git a/docs/routing.md b/docs/routing.md
index f52b96a9..555a3ba6 100644
--- a/docs/routing.md
+++ b/docs/routing.md
@@ -108,6 +108,9 @@ shares one forward pass on one checkpoint; send media and long states one at a t
server refuses `image`, `audio` and `video` in a body with a message saying where media goes, since
the JSON surface has no media encoding yet.
+Images and video through the lane need `pip install 'laya[vision]'` (Pillow and torchvision) and
+audio files `pip install 'laya[audio]'` (soundfile); text and a numpy waveform need nothing more.
+
Answers from the lane carry vega's fields on top of laya's: `prediction_set`, `abstain`, `unbound`
and `route`; see [vega inside laya](vegaml.md).
diff --git a/docs/vegaml.md b/docs/vegaml.md
index ff6196cd..49ba4ee0 100644
--- a/docs/vegaml.md
+++ b/docs/vegaml.md
@@ -7,6 +7,8 @@ is `laya.VegaAgent`, in `laya/vega.py`.
```bash
pip install -U laya # vegaml ships with laya 0.4.23 and newer
+pip install 'laya[vision]' # images and video: Pillow and torchvision
+pip install 'laya[audio]' # audio files: soundfile; a numpy waveform needs nothing
```
```python
diff --git a/laya/__init__.py b/laya/__init__.py
index b795750d..853ef7ce 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -35,7 +35,7 @@ from .presets import (
from .router import DEFAULT_MODELS, RouteDecision, Router
from .structured import DecisionResult, decide, decide_batch
-__version__ = "0.4.25"
+__version__ = "0.4.26"
# Routing, language detection and email cleaning are pure Python. The torch-backed names are
# resolved lazily so that `import laya` -- and therefore `from laya import Router` or
diff --git a/laya/vega.py b/laya/vega.py
index 6e13a7ea..994d13c3 100644
--- a/laya/vega.py
+++ b/laya/vega.py
@@ -334,12 +334,22 @@ class VegaAgent(Agent):
started = time.perf_counter()
with self._lock:
if image is not None:
- raw = self.vega.decide_image(image, vq, mode=mode, alpha=self.alpha, risk=self.risk)
+ try:
+ raw = self.vega.decide_image(image, vq, mode=mode, alpha=self.alpha, risk=self.risk)
+ except ImportError as exc:
+ # the reader's image processor wants Pillow and torchvision, which core laya
+ # does not carry; say which extra, not which transformers backend
+ raise ImportError("images and video need Pillow and torchvision: "
+ "pip install 'laya[vision]' (%s)" % exc) from exc
modality = "image"
elif audio is not None:
- raw = self.vega.decide_audio(audio, vq, asr=asr, language=language,
- sampling_rate=sampling_rate, mode=mode,
- alpha=self.alpha, views=views)
+ try:
+ raw = self.vega.decide_audio(audio, vq, asr=asr, language=language,
+ sampling_rate=sampling_rate, mode=mode,
+ alpha=self.alpha, views=views)
+ except ImportError as exc:
+ raise ImportError("an audio file needs soundfile to decode: pip install 'laya[audio]'; "
+ "a numpy waveform with sampling_rate= needs nothing extra (%s)" % exc) from exc
modality = "audio"
else:
raw = self.vega.decide(state, vq, mode=mode, alpha=self.alpha, views=views)
diff --git a/nix/package.nix b/nix/package.nix
index a94767f8..7ad1d4c8 100644
--- a/nix/package.nix
+++ b/nix/package.nix
@@ -12,7 +12,7 @@ let
# side, where tests/test_packaging.py compares all three declarations (this one,
# pyproject.toml and laya.__version__) and fails CI when they diverge. Nothing
# did, which is how this sat at 0.3.4 while the package reached 0.3.20.
- version = "0.4.25";
+ version = "0.4.26";
src = ../.;
format = "setuptools";
propagatedBuildInputs = with python3Packages; [
diff --git a/pyproject.toml b/pyproject.toml
index 2b6ce84a..40b59024 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "laya"
-version = "0.4.25"
+version = "0.4.26"
description = "Fast, non-autoregressive System 1 decision engine with calibrated probabilities"
readme = "README.md"
requires-python = ">=3.10"
@@ -57,6 +57,14 @@ mcp = ["mcp>=2.2.0"]
# `pip install laya[structured]` adds pydantic for the typed-model helpers (laya/structured.py).
structured = ["pydantic>=2.0.0"]
+# `pip install laya[vision]` is what `Agent(vegaml=True).predict(image=...)`, `predict(video=...)` and the
+# Router's `--image`/`--video` need: the reader's image processor is built on Pillow and torchvision.
+# Text and speech need neither, which is why these are an extra and not core.
+vision = ["pillow>=9.0", "torchvision>=0.19"]
+# `pip install laya[audio]` decodes audio FILES (wav, flac, ogg, mp3) for `predict(audio=path)`; a numpy
+# waveform with `sampling_rate=` needs nothing extra.
+audio = ["soundfile>=0.12"]
+
# `pip install laya[onnx]` adds ONNX Runtime for `laya.onnx_agent` and `onnxscript`, which
# `torch.onnx.export` requires (`scripts/export_onnx.py`, `tests/test_onnx.py`).
#
diff --git a/tests/test_audit_scope.py b/tests/test_audit_scope.py
index dd304472..dfb8e431 100644
--- a/tests/test_audit_scope.py
+++ b/tests/test_audit_scope.py
@@ -86,15 +86,16 @@ if len(od) > 1:
check_true("pyproject/core parses", len(core) == 6, core)
check("pyproject/extra tables found", sorted(extras),
- ["agno", "crewai", "fast", "langchain", "langgraph", "llamaindex", "mcp", "onnx", "serve",
- "structured"])
+ ["agno", "audio", "crewai", "fast", "langchain", "langgraph", "llamaindex", "mcp", "onnx", "serve",
+ "structured", "vision"])
declared = list(core) + [s for names in extras.values() for s in names]
expected = sorted(set(declared))
check("pyproject/extras add names the core does not have",
sorted({re.split(r"[<>=!;\[ ]", s)[0] for s in declared}
- {re.split(r"[<>=!;\[ ]", s)[0] for s in core}),
["agno", "crewai", "fastapi", "langchain-core", "langgraph", "llama-index-core", "mcp", "onnx",
- "onnxruntime", "onnxscript", "pydantic", "python-multipart", "tilelang", "uvicorn"])
+ "onnxruntime", "onnxscript", "pillow", "pydantic", "python-multipart", "soundfile", "tilelang",
+ "torchvision", "uvicorn"])
# The two scopes: shipped (blocking) is the core plus [serve]; everything else is advisory.
shipped = sorted(set(core) | set(extras.get("serve", [])))