Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(java): the parity lane cannot be a per-checkpoint matrix

Adeshina Lasisi310b0dd2fdd9
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index 846b8325..610d30b9 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -87,14 +87,18 @@ jobs:
path: laya-java/laya-java/build/reports
parity:
- name: jvm parity (${{ matrix.checkpoint }})
+ # ONE job, not a per-checkpoint matrix. The .NET lane matrixes over checkpoints because its
+ # goldens are per-checkpoint directories; these fixtures are not -- `tokenizer_ids.json`,
+ # `sequences.json` and `decode.json` each carry BOTH checkpoints' sections in one document, so
+ # a job holding only one of them can never regenerate them and `--strict` fails by
+ # construction. That is what happened on the first run of this lane.
+ #
+ # Only the checkpoint the end-to-end golden was recorded from needs a graph, so the other is
+ # fetched without its weights.
+ name: jvm parity
runs-on: ubuntu-latest
continue-on-error: true # advisory: see the header
timeout-minutes: 75
- strategy:
- fail-fast: false
- matrix:
- checkpoint: [english, multilingual]
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
@@ -117,24 +121,29 @@ jobs:
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/huggingface/hub
- key: laya-java-hf-${{ runner.os }}-${{ matrix.checkpoint }}-${{ hashFiles('laya-dotnet/tools/regen_golden.py') }}
+ key: laya-java-hf-${{ runner.os }}-${{ hashFiles('laya-java/scripts/prepare_checkpoint.py') }}
- name: Cache the exported graph
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
- path: onnx-java/${{ matrix.checkpoint }}
- key: laya-java-onnx-${{ runner.os }}-${{ matrix.checkpoint }}-${{ hashFiles('scripts/export_onnx.py', 'laya/common.py', 'laya-dotnet/tools/requirements-regen.txt') }}
+ path: laya-java/.work/onnx
+ key: laya-java-onnx-${{ runner.os }}-${{ hashFiles('scripts/export_onnx.py', 'laya/common.py', 'laya-dotnet/tools/requirements-regen.txt') }}
- - name: Download the checkpoint and export the fused graph
- run: python laya-java/scripts/prepare_checkpoint.py --checkpoint ${{ matrix.checkpoint }}
+ # english: tokenizer and config only. The tokenizer and sequence fixtures need it, nothing
+ # here traces it, so its weights would be a few gigabytes of nothing.
+ - name: Prepare the english checkpoint (no weights, no graph)
+ run: python laya-java/scripts/prepare_checkpoint.py --checkpoint english --no-graph
+ # multilingual: the checkpoint the end-to-end golden is recorded from, so it needs the graph.
+ - name: Prepare the multilingual checkpoint and export its graph
+ run: python laya-java/scripts/prepare_checkpoint.py --checkpoint multilingual
# Re-recorded from the Python code at THIS commit, and `--strict` makes a family this job
# could not regenerate an error: here, a skip means the download or export failed rather than
# that the machine has no model.
- name: Regenerate every fixture from Python
env:
- LAYA_FIXTURE_CHECKPOINTS: ${{ github.workspace }}/checkpoints-java
- LAYA_ONNX_GRAPH: ${{ github.workspace }}/onnx-java/${{ matrix.checkpoint }}/laya.onnx
- LAYA_PREDICT_MODEL: ${{ matrix.checkpoint }}
+ LAYA_FIXTURE_CHECKPOINTS: ${{ github.workspace }}/laya-java/.work/checkpoints
+ LAYA_ONNX_GRAPH: ${{ github.workspace }}/laya-java/.work/onnx/multilingual/laya.onnx
+ LAYA_PREDICT_MODEL: multilingual
run: python laya-java/scripts/gen_fixtures.py --strict
# Informational: float noise across platforms is expected in the model-backed family, and the
# Java test's tolerance is the gate rather than byte equality of the file.
@@ -146,8 +155,8 @@ jobs:
- name: Run every test against the fresh fixtures
working-directory: laya-java
env:
- LAYA_CHECKPOINTS: ${{ github.workspace }}/checkpoints-java
- LAYA_ONNX_GRAPH: ${{ github.workspace }}/onnx-java/${{ matrix.checkpoint }}/laya.onnx
+ LAYA_CHECKPOINTS: ${{ github.workspace }}/laya-java/.work/checkpoints
+ LAYA_ONNX_GRAPH: ${{ github.workspace }}/laya-java/.work/onnx/multilingual/laya.onnx
run: ./gradlew --no-daemon test
# A checkpoint-backed factory that aborts is not a pass. With the checkpoints present nothing
# may abort, and the floor catches a suite that silently stopped registering its cases.
@@ -160,7 +169,7 @@ jobs:
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
with:
- name: java-parity-diagnostics-${{ matrix.checkpoint }}
+ name: java-parity-diagnostics
path: |
laya-java/laya-java/build/reports
laya-java/fixtures
diff --git a/laya-java/.gitignore b/laya-java/.gitignore
index 2c75b472..2bee34ed 100644
--- a/laya-java/.gitignore
+++ b/laya-java/.gitignore
@@ -2,3 +2,7 @@
# committed, because a build that cannot be reproduced from a clean checkout is not a build.
.gradle/
build/
+
+# Downloaded checkpoints and exported graphs, written by scripts/prepare_checkpoint.py. Gigabytes,
+# and reproducible from the pinned revision, so they are never committed.
+.work/
diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.py
index 026b1e51..422d58cc 100644
--- a/laya-java/scripts/gen_fixtures.py
+++ b/laya-java/scripts/gen_fixtures.py
@@ -20,7 +20,6 @@ Phase 0 families (no model weights needed -- these are the tables and the text t
import argparse
import json
import os
-import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
@@ -402,7 +401,6 @@ def decode_answers():
magnitudes that would overflow a naive `exp`, and the language override -- and so the fixture
does not need a 1.2 GB graph to regenerate.
"""
- import math
import os
import numpy as np
diff --git a/laya-java/scripts/prepare_checkpoint.py b/laya-java/scripts/prepare_checkpoint.py
index 7df92639..0a384e5e 100755
--- a/laya-java/scripts/prepare_checkpoint.py
+++ b/laya-java/scripts/prepare_checkpoint.py
@@ -6,9 +6,9 @@ part of a CI run (minutes), while regenerating the fixtures from the Python code
must happen on every commit (seconds). Splitting them lets the workflow cache the first and never
cache the second.
-Writes
- <repo>/checkpoints-java/<name>/ the checkpoint, as `Agent.open` expects it
- <repo>/onnx-java/<name>/laya.onnx the fused graph, as `LayaSession.open` expects it
+Writes, under `laya-java/.work/` so the repository root stays clean and one `.gitignore` covers it
+ laya-java/.work/checkpoints/<name>/ the checkpoint, as `Agent.open` expects it
+ laya-java/.work/onnx/<name>/laya.onnx the fused graph, as `LayaSession.open` expects it
Usage:
python laya-java/scripts/prepare_checkpoint.py --checkpoint multilingual
@@ -29,13 +29,27 @@ REPO = os.path.dirname(os.path.dirname(HERE))
HF_REPO = "convaiinnovations/laya"
HF_REVISION = "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851"
-# What `Agent.open` reads, plus what the exporter needs to trace the model.
-WANTED = ("rl_agent_config.json", "model.safetensors", "config.json",
- "tokenizer.json", "tokenizer/*", "encoder/*")
+# What `Agent.open` and the fixture generator read: the budgets, the fitted temperatures and the
+# tokenizer. No weights.
+WANTED_LIGHT = ("rl_agent_config.json", "config.json", "tokenizer.json", "tokenizer/*")
+# Plus what the exporter needs to trace the model.
+WANTED = WANTED_LIGHT + ("model.safetensors", "encoder/*")
-def snapshot(name, revision):
- """The pinned checkpoint directory in the Hugging Face cache, downloading on first use."""
+def fetch(name, revision, destination, with_weights=True):
+ """Materialise exactly the wanted files of one checkpoint at `destination`.
+
+ `local_dir`, not the shared cache. `snapshot_download` returns the cache ROOT, and
+ `allow_patterns` governs only what it DOWNLOADS -- anything already cached from another run is
+ still sitting there. The english checkpoint has no subfolder, so its directory IS the
+ repository root and contains `multilingual/` and `typed-decisions/` as children: copying that
+ root produced a 2.2 GB "tokenizer only" checkpoint, which is how this was found.
+
+ `with_weights=False` fetches only what the tokenizer and sequence fixtures need. Three of the
+ fixture families cover BOTH checkpoints in one document, so a parity run has to hold both -- but
+ only the one the end-to-end golden was recorded from needs a graph, and therefore weights. The
+ other is a few megabytes this way instead of a few gigabytes.
+ """
from huggingface_hub import snapshot_download
sys.path.insert(0, REPO)
@@ -51,9 +65,25 @@ def snapshot(name, revision):
"HF_REVISION here and in laya-dotnet/tools/regen_golden.py together"
% (name, repo, HF_REPO))
prefix = (subfolder + "/") if subfolder else ""
- root = snapshot_download(repo, revision=revision,
- allow_patterns=[prefix + pattern for pattern in WANTED])
- return os.path.join(root, subfolder) if subfolder else root
+ wanted = WANTED if with_weights else WANTED_LIGHT
+
+ staging = destination + ".partial"
+ for path in (staging, destination):
+ if os.path.exists(path):
+ shutil.rmtree(path)
+ os.makedirs(os.path.dirname(destination), exist_ok=True)
+ snapshot_download(repo, revision=revision, local_dir=staging,
+ allow_patterns=[prefix + pattern for pattern in wanted])
+ source = os.path.join(staging, subfolder) if subfolder else staging
+ if not os.path.isfile(os.path.join(source, "rl_agent_config.json")):
+ raise SystemExit(
+ "the download produced no rl_agent_config.json at %s; the pinned revision's layout "
+ "may have changed" % source)
+ # Moved rather than copied, so a checkpoint is never on disk twice.
+ os.replace(source, destination)
+ if os.path.exists(staging):
+ shutil.rmtree(staging)
+ return destination
def main(argv=None):
@@ -62,21 +92,28 @@ def main(argv=None):
parser.add_argument("--revision", default=HF_REVISION)
parser.add_argument("--force", action="store_true",
help="re-export even when the graph is already present")
+ parser.add_argument("--no-graph", action="store_true",
+ help="fetch only the tokenizer and config, and do not export a graph: "
+ "enough for the tokenizer and sequence fixtures, which cover both "
+ "checkpoints, without paying for weights this run will not trace")
args = parser.parse_args(argv)
- model_dir = os.path.join(REPO, "checkpoints-java", args.checkpoint)
- graph_dir = os.path.join(REPO, "onnx-java", args.checkpoint)
+ work = os.path.join(REPO, "laya-java", ".work")
+ model_dir = os.path.join(work, "checkpoints", args.checkpoint)
+ graph_dir = os.path.join(work, "onnx", args.checkpoint)
graph = os.path.join(graph_dir, "laya.onnx")
- print("downloading %s at %s" % (args.checkpoint, args.revision[:12]), flush=True)
- cached = snapshot(args.checkpoint, args.revision)
- # Copied out of the cache rather than used in place: the cache path contains the revision and
- # the tests take a plain directory, and a copy keeps a cached export from pointing into a
- # directory a later `snapshot_download` may garbage-collect.
- if os.path.exists(model_dir):
- shutil.rmtree(model_dir)
- shutil.copytree(cached, model_dir, symlinks=False)
- print(" checkpoint at %s" % os.path.relpath(model_dir, REPO))
+ print("downloading %s at %s%s" % (args.checkpoint, args.revision[:12],
+ " (tokenizer and config only)" if args.no_graph else ""),
+ flush=True)
+ fetch(args.checkpoint, args.revision, model_dir, with_weights=not args.no_graph)
+ size = sum(os.path.getsize(os.path.join(root, f))
+ for root, _dirs, files in os.walk(model_dir) for f in files)
+ print(" checkpoint at %s (%.1f MB)" % (os.path.relpath(model_dir, REPO), size / 1e6))
+
+ if args.no_graph:
+ print(" --no-graph: no export, as asked")
+ return 0
if os.path.exists(graph) and os.path.getsize(graph) > 0 and not args.force:
print(" graph already exported at %s (use --force to redo it)"