Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(ci): refuse an incomplete Hugging Face snapshot at the fetch

Adeshina Lasisie82399c7539d
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index af943426..28d71a39 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -306,7 +306,17 @@ jobs:
python-version: "3.11"
- run: pip install ruff
- name: Ruff
- run: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
+ # Two invocations, because the SDK script directories carry pre-existing F401/F811
+ # import debt that is not worth churning in an unrelated change. They DO get `F82`,
+ # the undefined-name group, which is a real bug class rather than style: a
+ # `verify_snapshot(..., name)` whose enclosing parameter was `name_or_path` reached
+ # review in this repository, and it would have raised `NameError` on every Hugging
+ # Face download and failed all four .NET parity cells. ruff finds it in under a
+ # second. Widen the second list to the first's rule set once that debt is paid.
+ run: |
+ ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
+ ruff check laya-java/scripts/ laya-dotnet/tools/ laya-ts/scripts/ \
+ --select=E9,F63,F7,F82 --line-length=120
- name: Byte-compile every module
run: python -m compileall -q laya/ tests/
diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml
index cc91afdf..9042e251 100644
--- a/.github/workflows/dotnet.yml
+++ b/.github/workflows/dotnet.yml
@@ -58,6 +58,16 @@ jobs:
- name: Test regeneration cache and skip guard
run: python laya-dotnet/tools/test_ci_tools.py
- name: Regenerate the routing goldens from Python
+ env:
+ # Scoped to this step, not the workflow: a workflow-level `env` reaches every step of
+ # every job, including `pip`, `gradle`, `semgrep` and third-party actions that have no
+ # business holding a credential. Only the steps that fetch from the hub get it.
+ #
+ # Unauthenticated requests are rate-limited, and a throttled fetch is what arrived
+ # TRUNCATED here. Unset expands to "", which the implicit token path treats as
+ # anonymous -- the explicit `token=` call sites normalise "" to None themselves,
+ # because an empty Bearer header makes httpx refuse the request outright.
+ HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: python laya-dotnet/tools/regen_golden.py --checkpoint routing
- name: Golden drift vs. committed (informational)
run: |
@@ -101,6 +111,13 @@ jobs:
timeout-minutes: 60
strategy:
fail-fast: false
+ # One cell at a time. Every cell that needs weights fetches them from the same Hugging
+ # Face repo, and four in parallel -- alongside `java.yml`'s parity matrix against the same
+ # repo in the same run -- is what provoked the throttle. `regen_golden.py` now refuses an
+ # incomplete snapshot at the fetch, so this is no longer the only thing between a
+ # truncated download and a confusing failure; not provoking the throttle is simply
+ # cheaper than diagnosing it, and this lane is advisory.
+ max-parallel: 1
matrix:
include:
- name: english
@@ -169,6 +186,16 @@ jobs:
key: laya-onnx-${{ runner.os }}-typed-decisions-${{ hashFiles('laya-dotnet/tools/regen_golden.py', 'laya-dotnet/tools/export_onnx.py', 'laya-ts/scripts/export_onnx.py', 'laya-dotnet/tools/requirements-regen.txt', 'laya/common.py') }}
- name: Export the checkpoints and regenerate their goldens from Python
+ env:
+ # Scoped to this step, not the workflow: a workflow-level `env` reaches every step of
+ # every job, including `pip`, `gradle`, `semgrep` and third-party actions that have no
+ # business holding a credential. Only the steps that fetch from the hub get it.
+ #
+ # Unauthenticated requests are rate-limited, and a throttled fetch is what arrived
+ # TRUNCATED here. Unset expands to "", which the implicit token path treats as
+ # anonymous -- the explicit `token=` call sites normalise "" to None themselves,
+ # because an empty Bearer header makes httpx refuse the request outright.
+ HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
args=()
for c in ${{ matrix.checkpoints }}; do args+=(--checkpoint "$c"); done
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index e8a9617e..ad89b336 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -405,10 +405,11 @@ jobs:
# same job passed on a branch that adds no cell.
#
# Serialising costs wall-clock on an advisory lane and removes the pressure this change
- # added. It is a MITIGATION, not the fix: the real defect is that nothing verifies a
- # download before using it, and the real fix is an `HF_TOKEN` for the higher rate limit or
- # a size/digest check at the fetch. Both live outside this directory, so they are not
- # bundled here.
+ # added. It is one of three parts, not the whole fix: `prepare_checkpoint.py` now refuses an
+ # incomplete snapshot at the fetch -- a download that produced nothing, a missing file, a
+ # 0-byte or unparseable one -- and the fetching step carries `HF_TOKEN` so the higher rate
+ # limit applies where the secret exists. The guard is what matters; not provoking the
+ # throttle is merely cheaper than diagnosing it.
max-parallel: 1
matrix:
include:
@@ -542,6 +543,16 @@ jobs:
# a boolean, because `graph: true` could not say WHICH checkpoint to trace, and a cell that
# names two would otherwise have traced both.
- name: Prepare the checkpoints this cell owns
+ env:
+ # Scoped to this step, not the workflow: a workflow-level `env` reaches every step of
+ # every job, including `pip`, `gradle`, `semgrep` and third-party actions that have no
+ # business holding a credential. Only the steps that fetch from the hub get it.
+ #
+ # Unauthenticated requests are rate-limited, and a throttled fetch is what arrived
+ # TRUNCATED here. Unset expands to "", which the implicit token path treats as
+ # anonymous -- the explicit `token=` call sites normalise "" to None themselves,
+ # because an empty Bearer header makes httpx refuse the request outright.
+ HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
for c in ${{ matrix.checkpoints }}; do
if [ "$c" = "${{ matrix.graph-model }}" ]; then
diff --git a/laya-dotnet/tools/export_onnx.py b/laya-dotnet/tools/export_onnx.py
index 0487ce93..f2d15694 100644
--- a/laya-dotnet/tools/export_onnx.py
+++ b/laya-dotnet/tools/export_onnx.py
@@ -83,7 +83,10 @@ def _resolve_spec(name_or_path: str, token: str, revision: str = None):
prefix = (subfolder + "/") if subfolder else ""
kw = {
- "token": token or os.environ.get("HF_TOKEN"),
+ # `or None`: an unset repo secret expands to "", and an explicit empty token makes
+ # httpx refuse the request outright -- `LocalProtocolError: Illegal header value
+ # b'Bearer '`. Only the IMPLICIT path treats "" as anonymous.
+ "token": token or os.environ.get("HF_TOKEN") or None,
"revision": revision,
"allow_patterns": [
prefix + n for n in (
@@ -96,9 +99,63 @@ def _resolve_spec(name_or_path: str, token: str, revision: str = None):
}
print(" downloading %s/%s from HuggingFace Hub..." % (repo, subfolder or "(root)"))
model_dir = huggingface_hub.snapshot_download(repo, **kw)
+ verify_snapshot(os.path.join(model_dir, subfolder) if subfolder else model_dir,
+ name_or_path, required=("rl_agent_config.json",))
return model_dir, subfolder, name_or_path
+def verify_snapshot(directory, label, required=()):
+ """Refuses a snapshot that is missing files, or whose files arrived empty or unparseable.
+
+ `snapshot_download` reports success on a TRUNCATED fetch. Measured in CI: the hub
+ rate-limited an unauthenticated request and `hf-xet` wrote a 0-byte 34 MB
+ `tokenizer.json` without raising, and nothing noticed until a tokenizer constructor failed
+ with `EOF while parsing a value at line 1 column 0` ninety seconds later, in a test, with
+ no mention of a download in the message. Checked here, where the cause is still visible.
+
+ `required` is the point, not the size checks. Walking what is present cannot see a file
+ that never arrived -- an empty directory passes a walk -- so the caller names what it
+ asked the hub for and absence is an error.
+
+ `.cache/` is skipped: with `local_dir=`, `huggingface_hub` keeps its own bookkeeping in
+ `<dir>/.cache/huggingface/download/`, and the `.lock` files there are legitimately
+ 0 bytes. A first version of this check walked them and rejected a HEALTHY english
+ checkpoint, which is the one failure mode a fail-closed guard must not have.
+ """
+ problems = []
+ for name in required:
+ if not os.path.exists(os.path.join(directory, name)):
+ problems.append("%s: missing -- it was requested but never arrived" % name)
+ for base, dirs, files in os.walk(directory):
+ dirs[:] = [d for d in dirs if d != ".cache"]
+ for name in files:
+ path = os.path.join(base, name)
+ shown = os.path.relpath(path, directory)
+ try:
+ size = os.path.getsize(path)
+ except OSError as problem:
+ problems.append("%s: cannot stat (%s)" % (shown, problem))
+ continue
+ if size == 0:
+ problems.append("%s: 0 bytes" % shown)
+ continue
+ if name.endswith(".json"):
+ try:
+ with open(path, "rb") as handle:
+ json.load(handle)
+ except Exception as problem:
+ problems.append("%s: %d bytes, not valid JSON (%s)" % (shown, size, problem))
+ elif name.endswith(".safetensors") and size < 1024:
+ problems.append("%s: %d bytes, too small to hold a header" % (shown, size))
+ if problems:
+ raise SystemExit(
+ "%s: the %s snapshot is incomplete, so nothing downstream can be trusted:\n"
+ " %s\n\nThis is what a rate-limited or interrupted Hugging Face fetch looks "
+ "like. Re-run; if it persists, set HF_TOKEN for the higher rate limit, or reduce "
+ "how many jobs fetch this repo at once."
+ % (os.path.basename(__file__), label, "\n ".join(problems)))
+
+
# ─── fixture cases ──────────────────────────────────────────────────────────
# Two synthetic test cases, chosen to cover noul (2 markers) and a mixed
diff --git a/laya-dotnet/tools/regen_golden.py b/laya-dotnet/tools/regen_golden.py
index fa84f7e7..7c7248fc 100644
--- a/laya-dotnet/tools/regen_golden.py
+++ b/laya-dotnet/tools/regen_golden.py
@@ -131,6 +131,58 @@ def run(cmd: list, env: dict = None):
print(" (%.1f s)" % (time.perf_counter() - t0), flush=True)
+def verify_snapshot(directory, label, required=()):
+ """Refuses a snapshot that is missing files, or whose files arrived empty or unparseable.
+
+ `snapshot_download` reports success on a TRUNCATED fetch. Measured in CI: the hub
+ rate-limited an unauthenticated request and `hf-xet` wrote a 0-byte 34 MB
+ `tokenizer.json` without raising, and nothing noticed until a tokenizer constructor failed
+ with `EOF while parsing a value at line 1 column 0` ninety seconds later, in a test, with
+ no mention of a download in the message. Checked here, where the cause is still visible.
+
+ `required` is the point, not the size checks. Walking what is present cannot see a file
+ that never arrived -- an empty directory passes a walk -- so the caller names what it
+ asked the hub for and absence is an error.
+
+ `.cache/` is skipped: with `local_dir=`, `huggingface_hub` keeps its own bookkeeping in
+ `<dir>/.cache/huggingface/download/`, and the `.lock` files there are legitimately
+ 0 bytes. A first version of this check walked them and rejected a HEALTHY english
+ checkpoint, which is the one failure mode a fail-closed guard must not have.
+ """
+ problems = []
+ for name in required:
+ if not os.path.exists(os.path.join(directory, name)):
+ problems.append("%s: missing -- it was requested but never arrived" % name)
+ for base, dirs, files in os.walk(directory):
+ dirs[:] = [d for d in dirs if d != ".cache"]
+ for name in files:
+ path = os.path.join(base, name)
+ shown = os.path.relpath(path, directory)
+ try:
+ size = os.path.getsize(path)
+ except OSError as problem:
+ problems.append("%s: cannot stat (%s)" % (shown, problem))
+ continue
+ if size == 0:
+ problems.append("%s: 0 bytes" % shown)
+ continue
+ if name.endswith(".json"):
+ try:
+ with open(path, "rb") as handle:
+ json.load(handle)
+ except Exception as problem:
+ problems.append("%s: %d bytes, not valid JSON (%s)" % (shown, size, problem))
+ elif name.endswith(".safetensors") and size < 1024:
+ problems.append("%s: %d bytes, too small to hold a header" % (shown, size))
+ if problems:
+ raise SystemExit(
+ "%s: the %s snapshot is incomplete, so nothing downstream can be trusted:\n"
+ " %s\n\nThis is what a rate-limited or interrupted Hugging Face fetch looks "
+ "like. Re-run; if it persists, set HF_TOKEN for the higher rate limit, or reduce "
+ "how many jobs fetch this repo at once."
+ % (os.path.basename(__file__), label, "\n ".join(problems)))
+
+
def snapshot(name: str, revision: str) -> str:
"""The pinned checkpoint directory in the Hugging Face cache (downloaded on first use)."""
from huggingface_hub import snapshot_download
@@ -150,7 +202,9 @@ def snapshot(name: str, revision: str) -> str:
"tokenizer.json", "tokenizer/*", "encoder/*",
)]
root = snapshot_download(repo, revision=revision, allow_patterns=allow)
- return os.path.join(root, subfolder) if subfolder else root
+ directory = os.path.join(root, subfolder) if subfolder else root
+ verify_snapshot(directory, name, required=("rl_agent_config.json",))
+ return directory
def publish(partial: str, final: str, stamp: str):
diff --git a/laya-java/scripts/prepare_checkpoint.py b/laya-java/scripts/prepare_checkpoint.py
index c558f571..6e5e746e 100755
--- a/laya-java/scripts/prepare_checkpoint.py
+++ b/laya-java/scripts/prepare_checkpoint.py
@@ -14,6 +14,7 @@ Usage:
python laya-java/scripts/prepare_checkpoint.py --checkpoint multilingual
"""
import argparse
+import json
import os
import shutil
import subprocess
@@ -114,6 +115,8 @@ def fetch(name, revision, destination, with_weights=True):
raise SystemExit(
"the download produced no rl_agent_config.json at %s; the pinned revision's layout "
"may have changed" % source)
+ # A present file is not a complete one -- see verify_snapshot.
+ verify_snapshot(source, name, required=("rl_agent_config.json",))
# Moved rather than copied, so a checkpoint is never on disk twice.
os.replace(source, destination)
if os.path.exists(staging):
@@ -122,6 +125,58 @@ def fetch(name, revision, destination, with_weights=True):
return destination
+def verify_snapshot(directory, label, required=()):
+ """Refuses a snapshot that is missing files, or whose files arrived empty or unparseable.
+
+ `snapshot_download` reports success on a TRUNCATED fetch. Measured in CI: the hub
+ rate-limited an unauthenticated request and `hf-xet` wrote a 0-byte 34 MB
+ `tokenizer.json` without raising, and nothing noticed until a tokenizer constructor failed
+ with `EOF while parsing a value at line 1 column 0` ninety seconds later, in a test, with
+ no mention of a download in the message. Checked here, where the cause is still visible.
+
+ `required` is the point, not the size checks. Walking what is present cannot see a file
+ that never arrived -- an empty directory passes a walk -- so the caller names what it
+ asked the hub for and absence is an error.
+
+ `.cache/` is skipped: with `local_dir=`, `huggingface_hub` keeps its own bookkeeping in
+ `<dir>/.cache/huggingface/download/`, and the `.lock` files there are legitimately
+ 0 bytes. A first version of this check walked them and rejected a HEALTHY english
+ checkpoint, which is the one failure mode a fail-closed guard must not have.
+ """
+ problems = []
+ for name in required:
+ if not os.path.exists(os.path.join(directory, name)):
+ problems.append("%s: missing -- it was requested but never arrived" % name)
+ for base, dirs, files in os.walk(directory):
+ dirs[:] = [d for d in dirs if d != ".cache"]
+ for name in files:
+ path = os.path.join(base, name)
+ shown = os.path.relpath(path, directory)
+ try:
+ size = os.path.getsize(path)
+ except OSError as problem:
+ problems.append("%s: cannot stat (%s)" % (shown, problem))
+ continue
+ if size == 0:
+ problems.append("%s: 0 bytes" % shown)
+ continue
+ if name.endswith(".json"):
+ try:
+ with open(path, "rb") as handle:
+ json.load(handle)
+ except Exception as problem:
+ problems.append("%s: %d bytes, not valid JSON (%s)" % (shown, size, problem))
+ elif name.endswith(".safetensors") and size < 1024:
+ problems.append("%s: %d bytes, too small to hold a header" % (shown, size))
+ if problems:
+ raise SystemExit(
+ "%s: the %s snapshot is incomplete, so nothing downstream can be trusted:\n"
+ " %s\n\nThis is what a rate-limited or interrupted Hugging Face fetch looks "
+ "like. Re-run; if it persists, set HF_TOKEN for the higher rate limit, or reduce "
+ "how many jobs fetch this repo at once."
+ % (os.path.basename(__file__), label, "\n ".join(problems)))
+
+
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--checkpoint", required=True, help="english, multilingual, ...")