Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

release: 0.4.24, video through the same call

Nandakishora6a50371e025
diff --git a/README.md b/README.md
index b3db4df6..6405bb65 100644
--- a/README.md
+++ b/README.md
@@ -90,15 +90,16 @@ print(out["routing"]["model"]) # vega: non-English goes to the reader
The rule, in order: an image or audio goes to vega; text longer than the fast model's context goes to vega; non-English or non-Latin text goes to vega; a question with a fitted head goes to vega; everything else goes to the encoder. `model="vega"` or `model="fast"` forces one.
-**Images, speech, long documents.** The questions and the answer shape do not change with the input:
+**Images, speech, video, long documents.** The questions and the answer shape do not change with the input:
```python
agent.predict(image="invoice.png", questions=questions) # PIL image, path or array
agent.predict(audio=wave, questions=questions, sampling_rate=16000) # transcribed, then decided; the transcript rides along
+agent.predict(video="call.mp4", questions=questions) # speech transcribed if there is any, else frames tiled into one image
agent.predict_long(seventy_page_contract, questions) # up to 73,728 tokens, read whole
```
-Image, audio and video, in one paragraph: an image is a PIL image, a path or an HxWx3 array in any format Pillow opens, and it is scaled to about one megapixel (1,048,576 pixels) before the reader sees it, so a larger or sharper file costs nothing more and gains nothing; there is no file-size cap of its own, memory is the bound. Audio is a path or raw file bytes (wav, flac, ogg, mp3, with `pip install soundfile`) or a 1-D numpy waveform with `sampling_rate=`; any length works, stereo is averaged, the wave is resampled to 16 kHz and transcribed in 30-second windows, and the transcript then counts toward the 73,728-token context. The default transcriber is English-only; pass `asr="openai/whisper-tiny", language="hi"` for other languages. Video is not read by laya: split it yourself, frames go in as images and the sound track as audio.
+Image, audio and video, in one paragraph: an image is a PIL image, a path or an HxWx3 array in any format Pillow opens, and it is scaled to about one megapixel (1,048,576 pixels) before the reader sees it, so a larger or sharper file costs nothing more and gains nothing; there is no file-size cap of its own, memory is the bound. Audio is a path or raw file bytes (wav, flac, ogg, mp3, with `pip install soundfile`) or a 1-D numpy waveform with `sampling_rate=`; any length works, stereo is averaged, the wave is resampled to 16 kHz and transcribed in 30-second windows, and the transcript then counts toward the 73,728-token context. The default transcriber is English-only; pass `asr="openai/whisper-tiny", language="hi"` for other languages. Video is a path or raw file bytes in any container ffmpeg opens (mp4, mov, webm, mkv), decoded through the `ffmpeg` binary on your PATH, with no Python dependency; the sound track is extracted at 16 kHz and transcribed, and if the transcript holds real words the decision is made on it through the text path, with heads, calibration, set and route all applying. A silent clip, or one whose sound is music or noise, has `frames` (4 by default) evenly spaced frames tiled into one contact sheet and decided through the image path, which is zero-shot. Any length and resolution work, since only the frames sampled and the transcript reach the reader; `out["video"]` says which path decided, the frame times or the transcript, and the duration.
**Fit and calibrate on your own labels.** Seconds, no gradient step, and the route line is placed for you:
@@ -157,6 +158,19 @@ The shipped checkpoints work zero-shot, but fine-tuning on decisions from your o
**[nandhakishorm.github.io/laya](https://nandhakishorm.github.io/laya/)**: guides for [prediction hooks](https://nandhakishorm.github.io/laya/hooks/), [schema-driven decisions](https://nandhakishorm.github.io/laya/structured/), [Docker](https://nandhakishorm.github.io/laya/docker/) and [LangChain and LangGraph](https://nandhakishorm.github.io/laya/langchain/), plus a full [API reference](https://nandhakishorm.github.io/laya/reference/).
+## What's new in 0.4.24
+
+* **Video, through the same call: `agent.predict(video="call.mp4", questions=questions)`** on
+ `Agent(vegaml=True)`. The clip is split into what the engine can already read, and the path is
+ chosen by its content: the sound track is extracted at 16 kHz and transcribed, and when the
+ transcript holds real words the decision is made on it through the text path, so fitted heads,
+ the refit calibration, the conformal set and the route all apply and the transcript rides along.
+ A silent clip, or one whose sound is music or noise, has four evenly spaced frames tiled into
+ one contact sheet and decided through the image path. `out["video"]` says which path decided,
+ with the frame times or the transcript. Decoding goes through the `ffmpeg` binary on your
+ PATH; laya takes on no Python dependency for it, and `laya.video` holds the helpers.
+ `laya.vega.Router` sends a clip to vega like an image or a recording.
+
## What's new in 0.4.23
* **`laya.Agent(vegaml=True)`: vega's engine behind laya's surface.** One argument swaps the
@@ -1108,7 +1122,7 @@ ans["route"], ans["route_reason"] # "auto" or "review", decided fo
| learning from your labels | gradient fine-tune | `agent.fit(rows)`: a head per question over frozen features, seconds |
| calibration | shipped temperature map | refit on your rows: temperature, conformal threshold, escalation band |
| refusal | `min_confidence` you choose | `abstain`, `unbound`, the conformal set, the fitted band |
-| images, speech | no | `agent.predict(image=img, ...)`, `agent.predict(audio=wav, ...)` |
+| images, speech, video | no | `agent.predict(image=img, ...)`, `agent.predict(audio=wav, ...)`, `agent.predict(video=clip, ...)` |
| speed on text (T4) | ~16 ms a decision batched | ~72 ms a decision |
```python
@@ -1118,6 +1132,7 @@ agent.save_calibration("fit/"); agent.load_calibration("fit/")
agent.predict_long(document, questions) # no windowing up to 73k tokens
agent.predict(image=img, questions=questions) # the reader's vision tower
agent.predict(audio=wav, questions=questions, sampling_rate=16000) # transcribed, then decided
+agent.predict(video="call.mp4", questions=questions) # speech if any, else frames
from laya.vega import Router
router = Router(fast=laya.Agent(), vega=agent) # laya for short English text, vega otherwise
diff --git a/docs/vegaml.md b/docs/vegaml.md
index d568f18f..26465722 100644
--- a/docs/vegaml.md
+++ b/docs/vegaml.md
@@ -34,7 +34,7 @@ ans["route"], ans["route_reason"] # "auto" or "review", decided f
| learning from your labels | gradient fine-tune | `agent.fit(rows)`: a head per question over the frozen features, seconds |
| calibration | a shipped temperature map | refit on your rows: temperature, conformal threshold, escalation band |
| refusal | `min_confidence` you choose | `abstain`, `unbound`, the conformal set, the fitted band |
-| images and speech | no | `predict(image=...)`, `predict(audio=...)` |
+| images, speech, video | no | `predict(image=...)`, `predict(audio=...)`, `predict(video=...)` |
| speed on text (T4) | ~16 ms a decision batched | ~72 ms a decision; each served head adds a read |
| memory | under 1 GB | 2 to 4 GB |
@@ -76,15 +76,16 @@ Gate on `route`, not on `confidence`. A fitted head can report 1.000 on an answe
`route` on the same answer is still right, because it comes from the set and the band rather
than the top probability.
-## Images, speech, long documents
+## Images, speech, video, long documents
```python
agent.predict(image=img, questions=questions) # PIL image, path, or array
agent.predict(audio=wave, questions=questions, sampling_rate=16000) # transcribed, then decided
+agent.predict(video="call.mp4", questions=questions) # speech if any, else frames
agent.predict_long(document, questions) # one pass up to 73k tokens
```
-Image, audio and video, in one paragraph: an image is a PIL image, a path or an HxWx3 array in any format Pillow opens, and it is scaled to about one megapixel (1,048,576 pixels) before the reader sees it, so a larger or sharper file costs nothing more and gains nothing; there is no file-size cap of its own, memory is the bound. Audio is a path or raw file bytes (wav, flac, ogg, mp3, with `pip install soundfile`) or a 1-D numpy waveform with `sampling_rate=`; any length works, stereo is averaged, the wave is resampled to 16 kHz and transcribed in 30-second windows, and the transcript then counts toward the 73,728-token context. The default transcriber is English-only; pass `asr="openai/whisper-tiny", language="hi"` for other languages. Video is not read by laya: split it yourself, frames go in as images and the sound track as audio.
+Image, audio and video, in one paragraph: an image is a PIL image, a path or an HxWx3 array in any format Pillow opens, and it is scaled to about one megapixel (1,048,576 pixels) before the reader sees it, so a larger or sharper file costs nothing more and gains nothing; there is no file-size cap of its own, memory is the bound. Audio is a path or raw file bytes (wav, flac, ogg, mp3, with `pip install soundfile`) or a 1-D numpy waveform with `sampling_rate=`; any length works, stereo is averaged, the wave is resampled to 16 kHz and transcribed in 30-second windows, and the transcript then counts toward the 73,728-token context. The default transcriber is English-only; pass `asr="openai/whisper-tiny", language="hi"` for other languages. Video is a path or raw file bytes in any container ffmpeg opens (mp4, mov, webm, mkv), decoded through the `ffmpeg` binary on your PATH, with no Python dependency; the sound track is extracted at 16 kHz and transcribed, and if the transcript holds real words the decision is made on it through the text path, with heads, calibration, set and route all applying. A silent clip, or one whose sound is music or noise, has `frames` (4 by default) evenly spaced frames tiled into one contact sheet and decided through the image path, which is zero-shot. Any length and resolution work, since only the frames sampled and the transcript reach the reader; `out["video"]` says which path decided, the frame times or the transcript, and the duration.
The modality is chosen by which argument you passed; the questions and the answer shape are the
same. An audio answer carries `transcript` and `asr`, because a decision made on words you never
diff --git a/laya/__init__.py b/laya/__init__.py
index dd953400..998e42fc 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -35,7 +35,7 @@ from .presets import (
from .router import DEFAULT_MODELS, RouteDecision, Router
from .structured import DecisionResult, decide, decide_batch
-__version__ = "0.4.23"
+__version__ = "0.4.24"
# Routing, language detection and email cleaning are pure Python. The torch-backed names are
# resolved lazily so that `import laya` -- and therefore `from laya import Router` or
diff --git a/laya/vega.py b/laya/vega.py
index e80cfdc0..4eae0367 100644
--- a/laya/vega.py
+++ b/laya/vega.py
@@ -16,6 +16,7 @@
agent.fit(rows, questions) # heads + temperature + conformal + band, one call
agent.predict(image=img, questions=questions) # pictures, through the reader's vision tower
agent.predict(audio=wav, questions=questions) # speech, transcribed then decided
+ agent.predict(video=clip, questions=questions) # the sound track if it carries speech, frames if not
agent.predict_long(very_long_state, questions) # 73k tokens, no windowing
agent.save_calibration("fit/") # heads and calibration, reloadable
laya.vega.Router(fast=laya.Agent(), ...) # model switching: laya for short English text,
@@ -292,14 +293,15 @@ class VegaAgent(Agent):
# ------------------------------------------------------------------ predicting
def predict(self, state: Any = None, questions: Optional[Mapping] = None,
lang: Optional[str] = None, *, image: Any = None, audio: Any = None,
+ video: Any = None, frames: int = 4,
mode: Optional[str] = None, views: int = 1, max_len: Optional[int] = None,
head_max_len: Optional[int] = None, min_confidence: Optional[float] = None,
asr: Optional[str] = None, language: Optional[str] = None,
sampling_rate: Optional[int] = None, hooks=None, on_predict_start=None,
on_predict_end=None, **ignored) -> Dict[str, Any]:
- """Answer typed questions about text, an image, or a recording, in laya's shape.
+ """Answer typed questions about text, an image, a recording or a clip, in laya's shape.
- One of ``state``, ``image`` or ``audio``. The modality is chosen by which you passed,
+ One of ``state``, ``image``, ``audio`` or ``video``. The modality is chosen by which you passed,
which is the whole of "modal switching": the same questions, the same answer shape,
the same route. ``lang`` is accepted for compatibility and recorded; the universal
reader is multilingual and needs no checkpoint switch.
@@ -308,12 +310,23 @@ class VegaAgent(Agent):
options at 32 tokens and instructions at 256 by itself, so they are recorded and not
applied. ``min_confidence`` is applied on top of the automatic route when given, the
way laya's gate applies it, and nothing here needs it.
+
+ ``video`` is a path or raw file bytes. Its sound track is transcribed and, when the
+ transcript holds real words, decided through the text path with everything that path
+ has (heads, calibration, set, route); otherwise ``frames`` evenly spaced frames are
+ tiled into one image and decided through the image path. ``out["video"]`` says which,
+ and why. See :mod:`laya.video`.
"""
if questions is None:
raise ValueError("predict needs `questions`")
- given = sum(x is not None for x in (state, image, audio))
+ given = sum(x is not None for x in (state, image, audio, video))
if given != 1:
- raise ValueError("predict takes exactly one of state=, image=, audio=; got %d" % given)
+ raise ValueError("predict takes exactly one of state=, image=, audio=, video=; got %d"
+ % given)
+ if video is not None:
+ return self._predict_video(video, questions, frames=frames, asr=asr, language=language,
+ mode=mode, views=views, min_confidence=min_confidence,
+ lang=lang)
if not questions:
return {"answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0},
"latency_ms": 0.0, "modality": "text"}
@@ -359,6 +372,60 @@ class VegaAgent(Agent):
def predict_image(self, image: Any, questions: Mapping, **kw) -> Dict[str, Any]:
return self.predict(image=image, questions=questions, **kw)
+ def predict_video(self, video: Any, questions: Mapping, **kw) -> Dict[str, Any]:
+ return self.predict(video=video, questions=questions, **kw)
+
+ def _predict_video(self, video: Any, questions: Mapping, *, frames: int = 4,
+ asr: Optional[str] = None, language: Optional[str] = None,
+ **kw) -> Dict[str, Any]:
+ """The clip split into what the engine can read, and the path chosen by its content.
+
+ The speech test is made on the transcript the audio path produced, not before it, so a
+ sound track that is loud but wordless (music, noise) costs one audio decision and then
+ falls through to the frames. That is cheaper than a second transcriber and keeps one
+ definition of "speech" for the whole library: :func:`laya.video.has_speech`.
+ """
+ from . import video as _video
+
+ started = time.perf_counter()
+ path, tmp = _video.as_path(video)
+ try:
+ info = _video.probe(path)
+ wave = _video.extract_audio(path)
+ out = None
+ detail: Dict[str, Any] = {"duration_s": round(info["duration_s"], 3),
+ "has_audio": bool(info["has_audio"])}
+ if _video.has_speech(wave):
+ heard = self.predict(audio=wave, questions=questions, sampling_rate=_video.TARGET_SR,
+ asr=asr, language=language, **kw)
+ transcript = heard.get("transcript") or ""
+ if _video.has_speech(wave, transcript):
+ out = heard
+ detail.update(path="speech", transcript=transcript, asr=heard.get("asr"))
+ else:
+ detail["transcript_rejected"] = transcript
+ if out is None:
+ if not info.get("has_video", True):
+ raise ValueError("the clip has no video stream and no speech on its sound track; "
+ "there is nothing to decide on")
+ sampled, times = _video.sample_frames(path, frames, info["duration_s"])
+ sheet = _video.contact_sheet(sampled)
+ out = self.predict(image=sheet, questions=questions,
+ **{k: v for k, v in kw.items() if k != "views"})
+ out.pop("transcript", None); out.pop("asr", None)
+ detail.update(path="frames", frame_times=[round(t, 3) for t in times],
+ frames=len(sampled))
+ finally:
+ if tmp:
+ try:
+ os.unlink(tmp)
+ except OSError:
+ pass
+ out["modality"] = "video"
+ out["video"] = detail
+ out["latency_ms"] = round(1000 * (time.perf_counter() - started), 1)
+ return out
+
def predict_image_text(self, image: Any, questions: Mapping, **kw) -> Dict[str, Any]:
"""Parity check: the vision path with the image withheld, to see what the picture
itself contributed."""
@@ -612,7 +679,7 @@ class Router:
Router(fast=laya.Agent(), vega=laya.Agent(vegaml=True))
router.predict(state, questions) # -> picks, and says which under "routing"
- The rule, in order: an ``image`` or ``audio`` goes to vega (the encoder has no eyes or
+ The rule, in order: an ``image``, ``audio`` or ``video`` goes to vega (the encoder has no eyes or
ears); a state longer than the fast model's context goes to vega (73k against 512 or
1,024); non-English or non-Latin text goes to vega (the universal reader is multilingual,
and laya's own English checkpoint collapses on non-Latin scripts); a question with a
@@ -636,11 +703,11 @@ class Router:
return self._vega
def choose(self, state: Any = None, questions: Optional[Mapping] = None, *, image=None,
- audio=None, lang: Optional[str] = None) -> Tuple[str, str]:
+ audio=None, video=None, lang: Optional[str] = None) -> Tuple[str, str]:
from .lang import is_english
- if image is not None or audio is not None:
- return "vega", "image or audio: the encoder has no vision or speech path"
+ if image is not None or audio is not None or video is not None:
+ return "vega", "image, audio or video: the encoder has no vision or speech path"
if self.fast is None:
return "vega", "no fast model attached"
text = state if isinstance(state, str) else json.dumps(state, ensure_ascii=False)
@@ -660,16 +727,18 @@ class Router:
return "fast", "short English text, no fitted head: the fast encoder is enough"
def predict(self, state: Any = None, questions: Optional[Mapping] = None, *,
- model: Optional[str] = None, image=None, audio=None, lang: Optional[str] = None,
- **kw) -> Dict[str, Any]:
+ model: Optional[str] = None, image=None, audio=None, video=None,
+ lang: Optional[str] = None, **kw) -> Dict[str, Any]:
if model is None:
- model, reason = self.choose(state, questions, image=image, audio=audio, lang=lang)
+ model, reason = self.choose(state, questions, image=image, audio=audio, video=video,
+ lang=lang)
else:
reason = "explicit model=%s" % model
if model == "vega":
- out = self.vega.predict(state, questions, lang=lang, image=image, audio=audio, **kw)
+ out = self.vega.predict(state, questions, lang=lang, image=image, audio=audio,
+ video=video, **kw)
elif model == "fast":
- if image is not None or audio is not None:
+ if image is not None or audio is not None or video is not None:
raise ValueError("the fast laya model takes text only; drop model= or use vega")
out = self.fast.predict(state, questions, lang=lang, **kw)
else:
diff --git a/laya/video.py b/laya/video.py
new file mode 100644
index 00000000..c1bce30d
--- /dev/null
+++ b/laya/video.py
@@ -0,0 +1,193 @@
+"""Video in, the same typed decision out: the sound track if it carries speech, frames if not.
+
+No open model under a billion parameters watches a clip and reads a question sheet, so the
+video path does not pretend to be one. It splits the clip into the two things the engine can
+already read, and chooses between them by what the clip contains:
+
+* **Speech.** The sound track is pulled out at 16 kHz mono and transcribed. If the transcript
+ holds real words, the decision is made on it through the text path, so fitted heads, the
+ refit calibration, the conformal set and the route all apply, exactly as for audio. The
+ transcript rides along, because a decision made on words the caller never saw cannot be
+ audited.
+* **Frames.** A silent clip (or one whose sound track is music, noise, or nothing) has a few
+ evenly spaced frames sampled and tiled into one contact sheet, which goes through the image
+ path. That is the vision tower's zero-shot readout: heads and calibration do not apply to
+ pixels, and the answer says so with ``calibrated: False``.
+
+Decoding goes through the ``ffmpeg`` binary when it is on the path. laya takes on no Python
+dependency for video, and a machine without ffmpeg is told exactly that in one sentence rather
+than failing somewhere inside a codec. Everything runs locally: the clip never leaves the
+process.
+
+ from laya.video import probe, extract_audio, sample_frames, contact_sheet
+"""
+from __future__ import annotations
+
+import io
+import json
+import os
+import re
+import shutil
+import subprocess
+import tempfile
+import wave as _wave
+from typing import List, Optional, Sequence, Tuple
+
+import numpy as np
+
+__all__ = ["ffmpeg_available", "probe", "extract_audio", "sample_frames", "contact_sheet",
+ "has_speech", "as_path", "TARGET_SR"]
+
+#: What whisper-family transcribers expect.
+TARGET_SR = 16_000
+
+#: Below this root-mean-square level the sound track is treated as silent and never sent to the
+#: transcriber, which would otherwise invent words for it.
+SILENCE_RMS = 1e-3
+
+#: A transcript with fewer words than this is noise the transcriber dressed up, not speech.
+MIN_WORDS = 3
+
+_NO_FFMPEG = ("video needs the ffmpeg binary on the PATH (brew install ffmpeg, apt install "
+ "ffmpeg, or https://ffmpeg.org/download.html); laya takes no Python dependency "
+ "for it. Images and audio work without it.")
+
+
+def ffmpeg_available() -> bool:
+ return shutil.which("ffmpeg") is not None
+
+
+def _need_ffmpeg() -> str:
+ exe = shutil.which("ffmpeg")
+ if exe is None:
+ raise RuntimeError(_NO_FFMPEG)
+ return exe
+
+
+def _run(args: Sequence[str], what: str) -> bytes:
+ proc = subprocess.run(list(args), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ if proc.returncode != 0:
+ tail = proc.stderr.decode("utf-8", "replace").strip().splitlines()[-3:]
+ raise RuntimeError("ffmpeg could not %s: %s" % (what, " | ".join(tail) or "no output"))
+ return proc.stdout
+
+
+def as_path(video) -> Tuple[str, Optional[str]]:
+ """A readable path for the clip, and the temporary file to delete afterwards, if any.
+
+ ffmpeg needs a seekable input for most containers, so raw bytes are written to a temporary
+ file rather than piped. A path or PathLike is used as given.
+ """
+ if isinstance(video, (bytes, bytearray)):
+ fd, path = tempfile.mkstemp(suffix=".video")
+ with os.fdopen(fd, "wb") as f:
+ f.write(bytes(video))
+ return path, path
+ if isinstance(video, (str, os.PathLike)):
+ path = os.fspath(video)
+ if not os.path.isfile(path):
+ raise FileNotFoundError("no video file at %r" % path)
+ return path, None
+ raise TypeError("video must be a path or raw video-file bytes, got %r" % type(video).__name__)
+
+
+def probe(path: str) -> dict:
+ """Duration in seconds and whether the clip has an audio stream, from ffprobe when it is
+ there and from ffmpeg's own banner when it is not."""
+ _need_ffmpeg()
+ ffprobe = shutil.which("ffprobe")
+ if ffprobe is not None:
+ out = _run([ffprobe, "-v", "error", "-show_entries", "format=duration",
+ "-show_entries", "stream=codec_type", "-of", "json", path], "probe the clip")
+ info = json.loads(out.decode("utf-8", "replace") or "{}")
+ dur = float((info.get("format") or {}).get("duration") or 0.0)
+ kinds = {s.get("codec_type") for s in info.get("streams", [])}
+ return {"duration_s": dur, "has_audio": "audio" in kinds, "has_video": "video" in kinds}
+ proc = subprocess.run([shutil.which("ffmpeg"), "-i", path], stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ text = proc.stderr.decode("utf-8", "replace")
+ dur = 0.0
+ for line in text.splitlines():
+ line = line.strip()
+ if line.startswith("Duration:"):
+ h, m, s = line.split(",")[0].split("Duration:")[1].strip().split(":")
+ dur = int(h) * 3600 + int(m) * 60 + float(s)
+ return {"duration_s": dur, "has_audio": "Audio:" in text, "has_video": "Video:" in text}
+
+
+def extract_audio(path: str) -> Optional[np.ndarray]:
+ """The sound track as a 1-D float32 waveform at 16 kHz mono, or None when the clip has
+ no audio stream."""
+ exe = _need_ffmpeg()
+ if not probe(path)["has_audio"]:
+ return None
+ out = _run([exe, "-v", "error", "-i", path, "-vn", "-ac", "1", "-ar", str(TARGET_SR),
+ "-f", "wav", "-acodec", "pcm_s16le", "-"], "extract the sound track")
+ with _wave.open(io.BytesIO(out)) as w:
+ frames = w.readframes(w.getnframes())
+ pcm = np.frombuffer(frames, dtype="<i2").astype(np.float32) / 32768.0
+ return pcm
+
+
+def has_speech(wave: Optional[np.ndarray], transcript: Optional[str] = None) -> bool:
+ """The gate that chooses the path: a sound track quiet enough to be silence never reaches
+ the transcriber, and a transcript shorter than a few words is not treated as speech."""
+ if wave is None or wave.size == 0:
+ return False
+ if float(np.sqrt(np.mean(wave.astype(np.float64) ** 2))) < SILENCE_RMS:
+ return False
+ if transcript is None:
+ return True # loud enough to be worth transcribing; the caller decides after
+ # Words with letters or digits in them. A tone or music comes back from whisper as rows of
+ # dots or dashes, which split() counts as words and this does not.
+ return len(re.findall(r"[^\W_]+", transcript)) >= MIN_WORDS
+
+
+def sample_frames(path: str, n: int = 4, duration_s: Optional[float] = None) -> Tuple[List, List[float]]:
+ """``n`` frames at evenly spaced times, as PIL RGB images, with the times they came from.
+
+ Times sit at the centres of ``n`` equal slices, so a two-frame sample of a ten-second clip
+ reads 2.5 s and 7.5 s rather than the first and last frame, which are often black.
+ """
+ exe = _need_ffmpeg()
+ try:
+ from PIL import Image
+ except ImportError as e:
+ raise ImportError("video frames need Pillow: pip install pillow") from e
+ if duration_s is None:
+ duration_s = probe(path)["duration_s"]
+ n = max(1, int(n))
+ if duration_s <= 0:
+ times = [0.0]
+ else:
+ times = [duration_s * (i + 0.5) / n for i in range(n)]
+ frames = []
+ for t in times:
+ png = _run([exe, "-v", "error", "-ss", "%.3f" % t, "-i", path, "-frames:v", "1",
+ "-f", "image2pipe", "-vcodec", "png", "-"], "read a frame at %.1fs" % t)
+ if not png:
+ continue
+ with Image.open(io.BytesIO(png)) as im:
+ frames.append(im.convert("RGB"))
+ if not frames:
+ raise RuntimeError("no frame could be read from %r" % path)
+ return frames, times[:len(frames)]
+
+
+def contact_sheet(frames: Sequence, cell: int = 512):
+ """The frames tiled left to right, top to bottom, into one image the vision tower reads
+ at once. Each frame is fitted into a ``cell`` square, keeping its aspect ratio."""
+ from PIL import Image
+ if not frames:
+ raise ValueError("no frames to tile")
+ n = len(frames)
+ cols = int(np.ceil(np.sqrt(n)))
+ rows = int(np.ceil(n / cols))
+ sheet = Image.new("RGB", (cols * cell, rows * cell), (0, 0, 0))
+ for i, fr in enumerate(frames):
+ fr = fr.copy()
+ fr.thumbnail((cell, cell))
+ x = (i % cols) * cell + (cell - fr.width) // 2
+ y = (i // cols) * cell + (cell - fr.height) // 2
+ sheet.paste(fr, (x, y))
+ return sheet
diff --git a/nix/package.nix b/nix/package.nix
index 78914d38..0a7dca73 100644
--- a/nix/package.nix
+++ b/nix/package.nix
@@ -12,7 +12,7 @@ let
# side, where tests/test_packaging.py compares all three declarations (this one,
# pyproject.toml and laya.__version__) and fails CI when they diverge. Nothing
# did, which is how this sat at 0.3.4 while the package reached 0.3.20.
- version = "0.4.23";
+ version = "0.4.24";
src = ../.;
format = "setuptools";
propagatedBuildInputs = with python3Packages; [
diff --git a/pyproject.toml b/pyproject.toml
index 56bbbb5d..ccfa994d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "laya"
-version = "0.4.23"
+version = "0.4.24"
description = "Fast, non-autoregressive System 1 decision engine with calibrated probabilities"
readme = "README.md"
requires-python = ">=3.10"
diff --git a/tests/test_vega.py b/tests/test_vega.py
index 0b3665ae..bb764975 100644
--- a/tests/test_vega.py
+++ b/tests/test_vega.py
@@ -14,8 +14,6 @@ Two layers, so the suite is useful on every machine that runs it:
Run: python -m pytest tests/test_vega.py -q
"""
-import json
-import math
import os
import sys
import tempfile
@@ -333,5 +331,111 @@ def test_image_path(agent):
assert agent.predict_image_text(img, q)["modality"] == "image_withheld"
+# ----------------------------------------------------------------------------- video
+# The clip is split into what the engine can read (laya/video.py) and the path is chosen by
+# content. The helpers need the ffmpeg binary and nothing else, so they are tested on synthetic
+# clips ffmpeg itself generates; the model-backed tests ride the LAYA_VEGA_MEDIA_TESTS gate.
+
+needs_ffmpeg = pytest.mark.skipif(
+ __import__("shutil").which("ffmpeg") is None, reason="ffmpeg is not on the PATH")
+
+
+def _clip(tmp_path, name, audio_src, video_src="color=c=orange:s=160x120:d=3"):
+ import subprocess
+ out = str(tmp_path / ("%s.mp4" % name))
+ subprocess.run(["ffmpeg", "-v", "error", "-y", "-f", "lavfi", "-i", video_src,
+ "-f", "lavfi", "-i", audio_src, "-t", "3", "-shortest", "-pix_fmt", "yuv420p", out],
+ check=True)
+ return out
+
+
+@needs_ffmpeg
+def test_video_helpers_on_synthetic_clips(tmp_path):
+ import numpy as np
+ from laya import video as v
+
+ silent = _clip(tmp_path, "silent", "anullsrc=r=16000:cl=mono")
+ tone = _clip(tmp_path, "tone", "sine=f=440:r=16000", "testsrc=s=160x120:d=3")
+ for path in (silent, tone):
+ info = v.probe(path)
+ assert info["has_video"] and info["has_audio"] and 2.5 < info["duration_s"] < 3.5
+ quiet = v.extract_audio(silent)
+ loud = v.extract_audio(tone)
+ assert quiet.dtype == np.float32 and loud.dtype == np.float32
+ assert abs(len(loud) - 3 * v.TARGET_SR) < v.TARGET_SR // 10, "resampled to 16 kHz, 3 s"
+ assert not v.has_speech(quiet), "silence never reaches the transcriber"
+ assert v.has_speech(loud), "a loud track is worth transcribing"
+ # the word gate: a tone comes back from whisper as rows of dots, which are not words
+ assert not v.has_speech(loud, ". . . . . . . . . . . .")
+ assert not v.has_speech(loud, "thank you")
+ assert v.has_speech(loud, "refund the duplicate today")
+ frames, times = v.sample_frames(tone, 4)
+ assert len(frames) == 4 and [round(t, 2) for t in times] == [0.38, 1.12, 1.88, 2.62]
+ assert frames[0].size == (160, 120)
+ sheet = v.contact_sheet(frames, cell=128)
+ assert sheet.size == (256, 256), "four frames tile 2x2"
+
+
+@needs_ffmpeg
+def test_video_bytes_go_through_a_temporary_file(tmp_path):
+ from laya import video as v
+
+ path = _clip(tmp_path, "silent", "anullsrc=r=16000:cl=mono")
+ raw = open(path, "rb").read()
+ got, tmp = v.as_path(raw)
+ try:
+ assert tmp is not None and got == tmp and os.path.getsize(got) == len(raw)
+ finally:
+ os.unlink(tmp)
+ assert v.as_path(path) == (path, None)
+ with pytest.raises(TypeError, match="path or raw video-file bytes"):
+ v.as_path(3.5)
+ with pytest.raises(FileNotFoundError):
+ v.as_path(str(tmp_path / "missing.mp4"))
+
+
+def test_video_is_one_of_the_exclusive_inputs():
+ a = object.__new__(lv.VegaAgent)
+ with pytest.raises(ValueError, match="exactly one of state=, image=, audio=, video="):
+ a.predict("text", {"q": {"type": "noul", "instructions": "x"}}, video="clip.mp4")
+
+
+def test_router_sends_video_to_vega():
+ r = lv.Router(fast=_StubAgent("fast"), vega=None)
+ assert r.choose(video="clip.mp4")[0] == "vega"
+ with pytest.raises(ValueError, match="text only"):
+ r.predict(video="clip.mp4", questions={"q": {"type": "noul", "instructions": "x"}}, model="fast")
+
+
+@needs_ffmpeg
[email protected](not MEDIA_TESTS, reason="set LAYA_VEGA_MEDIA_TESTS=1 (pulls the vision tower)")
+def test_video_silent_clip_goes_to_frames(agent, tmp_path):
+ q = {"q": {"type": "noul", "instructions": "Is the picture mostly one flat colour?"}}
+ out = agent.predict(video=_clip(tmp_path, "silent", "anullsrc=r=16000:cl=mono"), questions=q)
+ assert out["modality"] == "video" and out["video"]["path"] == "frames"
+ assert len(out["video"]["frame_times"]) == 4 and "transcript" not in out
+ assert set(out["answers"]["q"]) >= {"noul", "route", "prediction_set"}
+
+
+@needs_ffmpeg
[email protected](not MEDIA_TESTS or __import__("shutil").which("say") is None,
+ reason="needs LAYA_VEGA_MEDIA_TESTS=1 and macOS `say` to make a speech clip")
+def test_video_with_speech_decides_on_the_transcript(agent, tmp_path):
+ import subprocess
+ aiff = str(tmp_path / "speech.aiff")
+ subprocess.run(["say", "-o", aiff, "We were billed twice for March. Please refund the "
+ "duplicate today or we will cancel our plan."], check=True)
+ clip = str(tmp_path / "speech.mp4")
+ subprocess.run(["ffmpeg", "-v", "error", "-y", "-f", "lavfi", "-i", "color=c=white:s=160x120",
+ "-i", aiff, "-shortest", "-pix_fmt", "yuv420p", "-c:a", "aac", clip], check=True)
+ q = {"dept": {"type": "choice", "instructions": "Which department should handle this?",
+ "criteria": {"billing": "invoices, payments, refunds",
+ "technical": "bugs, outages", "other": "everything else"}}}
+ out = agent.predict(video=clip, questions=q)
+ assert out["modality"] == "video" and out["video"]["path"] == "speech"
+ assert "billed" in out["video"]["transcript"].lower()
+ assert out["answers"]["dept"]["choice"] == "billing"
+
+
if __name__ == "__main__":
sys.exit(pytest.main([__file__, "-q"]))