Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(ci): refuse an incomplete Hugging Face snapshot at the fetch
Adeshina Lasisie82399c7539d
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex af943426..28d71a39 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -306,7 +306,17 @@ jobs:python-version: "3.11"- run: pip install ruff- name: Ruff- run: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120+ # Two invocations, because the SDK script directories carry pre-existing F401/F811+ # import debt that is not worth churning in an unrelated change. They DO get `F82`,+ # the undefined-name group, which is a real bug class rather than style: a+ # `verify_snapshot(..., name)` whose enclosing parameter was `name_or_path` reached+ # review in this repository, and it would have raised `NameError` on every Hugging+ # Face download and failed all four .NET parity cells. ruff finds it in under a+ # second. Widen the second list to the first's rule set once that debt is paid.+ run: |+ ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120+ ruff check laya-java/scripts/ laya-dotnet/tools/ laya-ts/scripts/ \+ --select=E9,F63,F7,F82 --line-length=120- name: Byte-compile every modulerun: python -m compileall -q laya/ tests/diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.ymlindex cc91afdf..9042e251 100644--- a/.github/workflows/dotnet.yml+++ b/.github/workflows/dotnet.yml@@ -58,6 +58,16 @@ jobs:- name: Test regeneration cache and skip guardrun: python laya-dotnet/tools/test_ci_tools.py- name: Regenerate the routing goldens from Python+ env:+ # Scoped to this step, not the workflow: a workflow-level `env` reaches every step of+ # every job, including `pip`, `gradle`, `semgrep` and third-party actions that have no+ # business holding a credential. Only the steps that fetch from the hub get it.+ #+ # Unauthenticated requests are rate-limited, and a throttled fetch is what arrived+ # TRUNCATED here. Unset expands to "", which the implicit token path treats as+ # anonymous -- the explicit `token=` call sites normalise "" to None themselves,+ # because an empty Bearer header makes httpx refuse the request outright.+ HF_TOKEN: ${{ secrets.HF_TOKEN }}run: python laya-dotnet/tools/regen_golden.py --checkpoint routing- name: Golden drift vs. committed (informational)run: |@@ -101,6 +111,13 @@ jobs:timeout-minutes: 60strategy:fail-fast: false+ # One cell at a time. Every cell that needs weights fetches them from the same Hugging+ # Face repo, and four in parallel -- alongside `java.yml`'s parity matrix against the same+ # repo in the same run -- is what provoked the throttle. `regen_golden.py` now refuses an+ # incomplete snapshot at the fetch, so this is no longer the only thing between a+ # truncated download and a confusing failure; not provoking the throttle is simply+ # cheaper than diagnosing it, and this lane is advisory.+ max-parallel: 1matrix:include:- name: english@@ -169,6 +186,16 @@ jobs:key: laya-onnx-${{ runner.os }}-typed-decisions-${{ hashFiles('laya-dotnet/tools/regen_golden.py', 'laya-dotnet/tools/export_onnx.py', 'laya-ts/scripts/export_onnx.py', 'laya-dotnet/tools/requirements-regen.txt', 'laya/common.py') }}- name: Export the checkpoints and regenerate their goldens from Python+ env:+ # Scoped to this step, not the workflow: a workflow-level `env` reaches every step of+ # every job, including `pip`, `gradle`, `semgrep` and third-party actions that have no+ # business holding a credential. Only the steps that fetch from the hub get it.+ #+ # Unauthenticated requests are rate-limited, and a throttled fetch is what arrived+ # TRUNCATED here. Unset expands to "", which the implicit token path treats as+ # anonymous -- the explicit `token=` call sites normalise "" to None themselves,+ # because an empty Bearer header makes httpx refuse the request outright.+ HF_TOKEN: ${{ secrets.HF_TOKEN }}run: |args=()for c in ${{ matrix.checkpoints }}; do args+=(--checkpoint "$c"); donediff --git a/.github/workflows/java.yml b/.github/workflows/java.ymlindex e8a9617e..ad89b336 100644--- a/.github/workflows/java.yml+++ b/.github/workflows/java.yml@@ -405,10 +405,11 @@ jobs:# same job passed on a branch that adds no cell.## Serialising costs wall-clock on an advisory lane and removes the pressure this change- # added. It is a MITIGATION, not the fix: the real defect is that nothing verifies a- # download before using it, and the real fix is an `HF_TOKEN` for the higher rate limit or- # a size/digest check at the fetch. Both live outside this directory, so they are not- # bundled here.+ # added. It is one of three parts, not the whole fix: `prepare_checkpoint.py` now refuses an+ # incomplete snapshot at the fetch -- a download that produced nothing, a missing file, a+ # 0-byte or unparseable one -- and the fetching step carries `HF_TOKEN` so the higher rate+ # limit applies where the secret exists. The guard is what matters; not provoking the+ # throttle is merely cheaper than diagnosing it.max-parallel: 1matrix:include:@@ -542,6 +543,16 @@ jobs:# a boolean, because `graph: true` could not say WHICH checkpoint to trace, and a cell that# names two would otherwise have traced both.- name: Prepare the checkpoints this cell owns+ env:+ # Scoped to this step, not the workflow: a workflow-level `env` reaches every step of+ # every job, including `pip`, `gradle`, `semgrep` and third-party actions that have no+ # business holding a credential. Only the steps that fetch from the hub get it.+ #+ # Unauthenticated requests are rate-limited, and a throttled fetch is what arrived+ # TRUNCATED here. Unset expands to "", which the implicit token path treats as+ # anonymous -- the explicit `token=` call sites normalise "" to None themselves,+ # because an empty Bearer header makes httpx refuse the request outright.+ HF_TOKEN: ${{ secrets.HF_TOKEN }}run: |for c in ${{ matrix.checkpoints }}; doif [ "$c" = "${{ matrix.graph-model }}" ]; thendiff --git a/laya-dotnet/tools/export_onnx.py b/laya-dotnet/tools/export_onnx.pyindex 0487ce93..f2d15694 100644--- a/laya-dotnet/tools/export_onnx.py+++ b/laya-dotnet/tools/export_onnx.py@@ -83,7 +83,10 @@ def _resolve_spec(name_or_path: str, token: str, revision: str = None):prefix = (subfolder + "/") if subfolder else ""kw = {- "token": token or os.environ.get("HF_TOKEN"),+ # `or None`: an unset repo secret expands to "", and an explicit empty token makes+ # httpx refuse the request outright -- `LocalProtocolError: Illegal header value+ # b'Bearer '`. Only the IMPLICIT path treats "" as anonymous.+ "token": token or os.environ.get("HF_TOKEN") or None,"revision": revision,"allow_patterns": [prefix + n for n in (@@ -96,9 +99,63 @@ def _resolve_spec(name_or_path: str, token: str, revision: str = None):}print(" downloading %s/%s from HuggingFace Hub..." % (repo, subfolder or "(root)"))model_dir = huggingface_hub.snapshot_download(repo, **kw)+ verify_snapshot(os.path.join(model_dir, subfolder) if subfolder else model_dir,+ name_or_path, required=("rl_agent_config.json",))return model_dir, subfolder, name_or_path+def verify_snapshot(directory, label, required=()):+ """Refuses a snapshot that is missing files, or whose files arrived empty or unparseable.++ `snapshot_download` reports success on a TRUNCATED fetch. Measured in CI: the hub+ rate-limited an unauthenticated request and `hf-xet` wrote a 0-byte 34 MB+ `tokenizer.json` without raising, and nothing noticed until a tokenizer constructor failed+ with `EOF while parsing a value at line 1 column 0` ninety seconds later, in a test, with+ no mention of a download in the message. Checked here, where the cause is still visible.++ `required` is the point, not the size checks. Walking what is present cannot see a file+ that never arrived -- an empty directory passes a walk -- so the caller names what it+ asked the hub for and absence is an error.++ `.cache/` is skipped: with `local_dir=`, `huggingface_hub` keeps its own bookkeeping in+ `<dir>/.cache/huggingface/download/`, and the `.lock` files there are legitimately+ 0 bytes. A first version of this check walked them and rejected a HEALTHY english+ checkpoint, which is the one failure mode a fail-closed guard must not have.+ """+ problems = []+ for name in required:+ if not os.path.exists(os.path.join(directory, name)):+ problems.append("%s: missing -- it was requested but never arrived" % name)+ for base, dirs, files in os.walk(directory):+ dirs[:] = [d for d in dirs if d != ".cache"]+ for name in files:+ path = os.path.join(base, name)+ shown = os.path.relpath(path, directory)+ try:+ size = os.path.getsize(path)+ except OSError as problem:+ problems.append("%s: cannot stat (%s)" % (shown, problem))+ continue+ if size == 0:+ problems.append("%s: 0 bytes" % shown)+ continue+ if name.endswith(".json"):+ try:+ with open(path, "rb") as handle:+ json.load(handle)+ except Exception as problem:+ problems.append("%s: %d bytes, not valid JSON (%s)" % (shown, size, problem))+ elif name.endswith(".safetensors") and size < 1024:+ problems.append("%s: %d bytes, too small to hold a header" % (shown, size))+ if problems:+ raise SystemExit(+ "%s: the %s snapshot is incomplete, so nothing downstream can be trusted:\n"+ " %s\n\nThis is what a rate-limited or interrupted Hugging Face fetch looks "+ "like. Re-run; if it persists, set HF_TOKEN for the higher rate limit, or reduce "+ "how many jobs fetch this repo at once."+ % (os.path.basename(__file__), label, "\n ".join(problems)))++# ─── fixture cases ──────────────────────────────────────────────────────────# Two synthetic test cases, chosen to cover noul (2 markers) and a mixeddiff --git a/laya-dotnet/tools/regen_golden.py b/laya-dotnet/tools/regen_golden.pyindex fa84f7e7..7c7248fc 100644--- a/laya-dotnet/tools/regen_golden.py+++ b/laya-dotnet/tools/regen_golden.py@@ -131,6 +131,58 @@ def run(cmd: list, env: dict = None):print(" (%.1f s)" % (time.perf_counter() - t0), flush=True)+def verify_snapshot(directory, label, required=()):+ """Refuses a snapshot that is missing files, or whose files arrived empty or unparseable.++ `snapshot_download` reports success on a TRUNCATED fetch. Measured in CI: the hub+ rate-limited an unauthenticated request and `hf-xet` wrote a 0-byte 34 MB+ `tokenizer.json` without raising, and nothing noticed until a tokenizer constructor failed+ with `EOF while parsing a value at line 1 column 0` ninety seconds later, in a test, with+ no mention of a download in the message. Checked here, where the cause is still visible.++ `required` is the point, not the size checks. Walking what is present cannot see a file+ that never arrived -- an empty directory passes a walk -- so the caller names what it+ asked the hub for and absence is an error.++ `.cache/` is skipped: with `local_dir=`, `huggingface_hub` keeps its own bookkeeping in+ `<dir>/.cache/huggingface/download/`, and the `.lock` files there are legitimately+ 0 bytes. A first version of this check walked them and rejected a HEALTHY english+ checkpoint, which is the one failure mode a fail-closed guard must not have.+ """+ problems = []+ for name in required:+ if not os.path.exists(os.path.join(directory, name)):+ problems.append("%s: missing -- it was requested but never arrived" % name)+ for base, dirs, files in os.walk(directory):+ dirs[:] = [d for d in dirs if d != ".cache"]+ for name in files:+ path = os.path.join(base, name)+ shown = os.path.relpath(path, directory)+ try:+ size = os.path.getsize(path)+ except OSError as problem:+ problems.append("%s: cannot stat (%s)" % (shown, problem))+ continue+ if size == 0:+ problems.append("%s: 0 bytes" % shown)+ continue+ if name.endswith(".json"):+ try:+ with open(path, "rb") as handle:+ json.load(handle)+ except Exception as problem:+ problems.append("%s: %d bytes, not valid JSON (%s)" % (shown, size, problem))+ elif name.endswith(".safetensors") and size < 1024:+ problems.append("%s: %d bytes, too small to hold a header" % (shown, size))+ if problems:+ raise SystemExit(+ "%s: the %s snapshot is incomplete, so nothing downstream can be trusted:\n"+ " %s\n\nThis is what a rate-limited or interrupted Hugging Face fetch looks "+ "like. Re-run; if it persists, set HF_TOKEN for the higher rate limit, or reduce "+ "how many jobs fetch this repo at once."+ % (os.path.basename(__file__), label, "\n ".join(problems)))++def snapshot(name: str, revision: str) -> str:"""The pinned checkpoint directory in the Hugging Face cache (downloaded on first use)."""from huggingface_hub import snapshot_download@@ -150,7 +202,9 @@ def snapshot(name: str, revision: str) -> str:"tokenizer.json", "tokenizer/*", "encoder/*",)]root = snapshot_download(repo, revision=revision, allow_patterns=allow)- return os.path.join(root, subfolder) if subfolder else root+ directory = os.path.join(root, subfolder) if subfolder else root+ verify_snapshot(directory, name, required=("rl_agent_config.json",))+ return directorydef publish(partial: str, final: str, stamp: str):diff --git a/laya-java/scripts/prepare_checkpoint.py b/laya-java/scripts/prepare_checkpoint.pyindex c558f571..6e5e746e 100755--- a/laya-java/scripts/prepare_checkpoint.py+++ b/laya-java/scripts/prepare_checkpoint.py@@ -14,6 +14,7 @@ Usage:python laya-java/scripts/prepare_checkpoint.py --checkpoint multilingual"""import argparse+import jsonimport osimport shutilimport subprocess@@ -114,6 +115,8 @@ def fetch(name, revision, destination, with_weights=True):raise SystemExit("the download produced no rl_agent_config.json at %s; the pinned revision's layout ""may have changed" % source)+ # A present file is not a complete one -- see verify_snapshot.+ verify_snapshot(source, name, required=("rl_agent_config.json",))# Moved rather than copied, so a checkpoint is never on disk twice.os.replace(source, destination)if os.path.exists(staging):@@ -122,6 +125,58 @@ def fetch(name, revision, destination, with_weights=True):return destination+def verify_snapshot(directory, label, required=()):+ """Refuses a snapshot that is missing files, or whose files arrived empty or unparseable.++ `snapshot_download` reports success on a TRUNCATED fetch. Measured in CI: the hub+ rate-limited an unauthenticated request and `hf-xet` wrote a 0-byte 34 MB+ `tokenizer.json` without raising, and nothing noticed until a tokenizer constructor failed+ with `EOF while parsing a value at line 1 column 0` ninety seconds later, in a test, with+ no mention of a download in the message. Checked here, where the cause is still visible.++ `required` is the point, not the size checks. Walking what is present cannot see a file+ that never arrived -- an empty directory passes a walk -- so the caller names what it+ asked the hub for and absence is an error.++ `.cache/` is skipped: with `local_dir=`, `huggingface_hub` keeps its own bookkeeping in+ `<dir>/.cache/huggingface/download/`, and the `.lock` files there are legitimately+ 0 bytes. A first version of this check walked them and rejected a HEALTHY english+ checkpoint, which is the one failure mode a fail-closed guard must not have.+ """+ problems = []+ for name in required:+ if not os.path.exists(os.path.join(directory, name)):+ problems.append("%s: missing -- it was requested but never arrived" % name)+ for base, dirs, files in os.walk(directory):+ dirs[:] = [d for d in dirs if d != ".cache"]+ for name in files:+ path = os.path.join(base, name)+ shown = os.path.relpath(path, directory)+ try:+ size = os.path.getsize(path)+ except OSError as problem:+ problems.append("%s: cannot stat (%s)" % (shown, problem))+ continue+ if size == 0:+ problems.append("%s: 0 bytes" % shown)+ continue+ if name.endswith(".json"):+ try:+ with open(path, "rb") as handle:+ json.load(handle)+ except Exception as problem:+ problems.append("%s: %d bytes, not valid JSON (%s)" % (shown, size, problem))+ elif name.endswith(".safetensors") and size < 1024:+ problems.append("%s: %d bytes, too small to hold a header" % (shown, size))+ if problems:+ raise SystemExit(+ "%s: the %s snapshot is incomplete, so nothing downstream can be trusted:\n"+ " %s\n\nThis is what a rate-limited or interrupted Hugging Face fetch looks "+ "like. Re-run; if it persists, set HF_TOKEN for the higher rate limit, or reduce "+ "how many jobs fetch this repo at once."+ % (os.path.basename(__file__), label, "\n ".join(problems)))++def main(argv=None):parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])parser.add_argument("--checkpoint", required=True, help="english, multilingual, ...")