Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(java): Tokenizer.decode, and the four rules the slices found

Adeshina Lasisi561c496508ed
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index 2522b3fa..8219b2e3 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -70,7 +70,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 585 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 625 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -226,7 +226,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 585 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 625 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
@@ -263,13 +263,28 @@ jobs:
- name: tokenizer
checkpoints: english multilingual
graph: false
- families: --only tokenizer_ids.json --only sequences.json --only decode.json
+ families: >-
+ --only tokenizer_ids.json --only sequences.json --only decode.json
+ --only decode_text.json
tests: >-
--tests com.convaiinnovations.laya.tokenizer.TokenizerParityTest
--tests com.convaiinnovations.laya.sequence.SequenceParityTest
--tests com.convaiinnovations.laya.decode.DecoderParityTest
--tests com.convaiinnovations.laya.tokenizer.PreTokenizerTablesTest
- min-tests: 175
+ --tests com.convaiinnovations.laya.tokenizer.TokenDecoderTest
+ # Measured locally. The margin is deliberately smaller than the 8 the 183/175 pair
+ # used, because the smallest class in this cell is PreTokenizerTablesTest at 5 tests
+ # and a margin of 8 let that whole class be dropped with the floor still green
+ # (measured: 369 of 374, exit 0). A floor below the total minus the smallest class
+ # does not bind any single class.
+ #
+ # What this does NOT protect, despite an earlier version of this comment saying so:
+ # the window SLICES. A count floor cannot see them -- the 403 slice assertions live
+ # inside 46 of the ~195 dynamic tests, because the factory bundles all of one text's
+ # slices into a single test, and a slice list that emptied would still register its
+ # test. The slices are guarded inside the suite instead, by the `total >= 400`
+ # assertion in TokenDecoderTest and the `cases.size() >= 23` corpus floor beside it.
+ min-tests: 374
timeout: 25
# The one cell that needs the traced graph, and the only checkpoint the end-to-end
# golden was recorded from.
diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml
index d2cf118f..6f1895f2 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -58,7 +58,10 @@ jobs:
- name: The suite was not vacuous
run: >
python3 scripts/check_test_results.py laya-java/build/test-results/test
- --min-tests 440 --allow-aborted
+ # 631 of the 640 registered tests run without a checkpoint. 440 left 191 of
+ # slack, so the whole decode family could have vanished from this lane with the
+ # gate still green -- and a floor that does not bind is not a gate.
+ --min-tests 625 --allow-aborted
- name: Build the jars
run: ./gradlew --no-daemon build -x test
@@ -103,7 +106,11 @@ jobs:
# jobs disagreed about what was being released.
case "$GITHUB_REF" in
refs/tags/laya-java-v*) : ;;
- *) echo "::error::refusing to publish from $GITHUB_REF; tag the release as laya-java-vX.Y.Z and run this on the tag" ; exit 1 ;;
+ *)
+ echo "::error::refusing to publish from $GITHUB_REF; tag the release as" \
+ "laya-java-vX.Y.Z and run this on the tag"
+ exit 1
+ ;;
esac
- name: Version from the tag
run: |
@@ -125,8 +132,20 @@ jobs:
# All four, not two. Without the passphrase the build calls useInMemoryPgpKeys(key,
# null) and uploads UNSIGNED artifacts, which Central rejects only after the upload --
# a much worse place to find out.
- test -n "$LAYA_SIGNING_KEY" || { echo "::error::LAYA_SIGNING_KEY is unset; see the header"; exit 1; }
- test -n "$LAYA_SIGNING_PASSWORD" || { echo "::error::LAYA_SIGNING_PASSWORD is unset; an unsigned bundle is rejected by Central after upload"; exit 1; }
- test -n "$MAVEN_CENTRAL_USERNAME" || { echo "::error::MAVEN_CENTRAL_USERNAME is unset; see the header"; exit 1; }
- test -n "$MAVEN_CENTRAL_PASSWORD" || { echo "::error::MAVEN_CENTRAL_PASSWORD is unset; see the header"; exit 1; }
+ #
+ # A loop rather than four `test -n` lines. Written out, the four were 111 to 161 columns
+ # and no longer read as the parallel checks they are, and each one exited on the FIRST
+ # thing it found -- so a release missing two secrets cost two tagged runs to discover.
+ # This names all of them at once.
+ missing=""
+ for secret in LAYA_SIGNING_KEY LAYA_SIGNING_PASSWORD \
+ MAVEN_CENTRAL_USERNAME MAVEN_CENTRAL_PASSWORD; do
+ [ -n "${!secret:-}" ] || missing="$missing $secret"
+ done
+ if [ -n "$missing" ]; then
+ echo "::error::unset secret(s):$missing -- see this workflow's header for how each" \
+ "one is produced. Note the signing passphrase in particular: without it the" \
+ "bundle is built UNSIGNED and Central rejects it only after the upload."
+ exit 1
+ fi
./gradlew --no-daemon publishAllPublicationsToCentralRepository
diff --git a/laya-java/CHANGELOG.md b/laya-java/CHANGELOG.md
index 17bcbc30..6f0f6b7a 100644
--- a/laya-java/CHANGELOG.md
+++ b/laya-java/CHANGELOG.md
@@ -20,6 +20,13 @@ see [Installing](README.md#installing).
`confidence` alongside `answerConfidence`.
- Python-compatible tokenization (BPE and Metaspace, added tokens, normalizers), sequence
building, collation, answer decoding and calibration.
+- `Tokenizer.decode(int[])` and `decode(int[], boolean skipSpecialTokens)`: ids back to text,
+ matching the `tokenizers` crate's `ByteLevel`, `Replace`, `ByteFallback`, `Fuse`, `Strip` and
+ `Sequence` decoders. Byte-exact against the reference over both shipped checkpoints, including
+ 201,279 window slices — the shape that cuts a multi-byte character in half, where a decoder that
+ agrees on every whole text can still be wrong. Decoding on the multilingual checkpoint is lossy
+ by the reference's own behaviour: `Metaspace` prepends its marker and the round trip gains a
+ leading space, which this port reproduces rather than corrects.
- `lang.LanguageDetection`: script and language detection, the eight public fields and the
fourteen intermediate steps the reference exposes.
- `Presets`: the five preset question sets, compared against the reference word for word.
diff --git a/laya-java/README.md b/laya-java/README.md
index 886de5de..52404bb7 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -296,9 +296,16 @@ import java.nio.file.Path;
Tokenizer tok = Tokenizer.fromModelDirectory(Path.of("./checkpoint"));
int[] ids = tok.encode("charged twice");
int[] capped = tok.encode(longText, 48); // stops early; same prefix as the full encoding
+String text = tok.decode(ids); // ids back to text; special tokens dropped
+String withSpecials = tok.decode(ids, false); // ...kept
AgentConfig cfg = AgentConfig.fromModelDirectory(Path.of("./checkpoint"));
```
+`decode` follows the reference rather than tidying after it, and on the multilingual checkpoint
+that means it is **lossy**: `Metaspace` prepends its marker, so `decode(encode("Hello world"))` is
+`" Hello world"`. The English checkpoint happens to round-trip. Correcting the space would make
+every window of a long document tokenize differently from the reference, so the loss is kept.
+
`Agent.using(tokenizer, config, session)` assembles an agent from parts — for a caller that already
holds them, or to drive the batching and usage accounting through a stub
`infer.InferenceSession` instead of a 1.2 GB graph.
diff --git a/laya-java/fixtures/decode_text.json b/laya-java/fixtures/decode_text.json
new file mode 100644
index 00000000..a3ae5981
--- /dev/null
+++ b/laya-java/fixtures/decode_text.json
@@ -0,0 +1,3964 @@
+{
+ "texts": [
+ "Hello world",
+ "",
+ " ",
+ " ",
+ "\t",
+ "a\tb\nc",
+ "café résumé",
+ "注文をキャンセル",
+ "مرحبا",
+ "क्‌ष",
+ "😀",
+ "emoji 😀 here",
+ "🇬🇧",
+ "▁literal marker",
+ "pre▁fix",
+ " nbsp figure",
+ "<0x41>",
+ "a<0xFF>b",
+ "\u0001\u0002",
+ "",
+ "󰀀",
+ "I cannot log in and need a password reset.",
+ "Esta mensagem e confidencial."
+ ],
+ "by": {
+ "english": {
+ "decoder": {
+ "type": "ByteLevel"
+ },
+ "cases": [
+ {
+ "text": "Hello world",
+ "ids": [
+ 12092,
+ 1533
+ ],
+ "decoded": "Hello world",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "Hello"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " world"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "Hello world"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "Hello world"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "Hello world"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 12092,
+ 1533,
+ 50282
+ ],
+ "skipping_specials": "Hello world",
+ "kept": "[CLS]Hello world[SEP]"
+ }
+ },
+ {
+ "text": "",
+ "ids": [],
+ "decoded": "",
+ "slices": [],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 50282
+ ],
+ "skipping_specials": "",
+ "kept": "[CLS][SEP]"
+ }
+ },
+ {
+ "text": " ",
+ "ids": [
+ 209
+ ],
+ "decoded": " ",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " "
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 209,
+ 50282
+ ],
+ "skipping_specials": " ",
+ "kept": "[CLS] [SEP]"
+ }
+ },
+ {
+ "text": " ",
+ "ids": [
+ 50275
+ ],
+ "decoded": " ",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " "
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 50275,
+ 50282
+ ],
+ "skipping_specials": " ",
+ "kept": "[CLS] [SEP]"
+ }
+ },
+ {
+ "text": "\t",
+ "ids": [
+ 186
+ ],
+ "decoded": "\t",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "\t"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "\t"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "\t"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "\t"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 186,
+ 50282
+ ],
+ "skipping_specials": "\t",
+ "kept": "[CLS]\t[SEP]"
+ }
+ },
+ {
+ "text": "a\tb\nc",
+ "ids": [
+ 66,
+ 186,
+ 67,
+ 187,
+ 68
+ ],
+ "decoded": "a\tb\nc",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "a"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "\t"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "b"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "\n"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "c"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "a\t"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "b\n"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "c"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "a\tb"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "\nc"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "a\tb\nc"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 66,
+ 186,
+ 67,
+ 187,
+ 68,
+ 50282
+ ],
+ "skipping_specials": "a\tb\nc",
+ "kept": "[CLS]a\tb\nc[SEP]"
+ }
+ },
+ {
+ "text": "café résumé",
+ "ids": [
+ 68,
+ 2320,
+ 860,
+ 9568,
+ 2204,
+ 860
+ ],
+ "decoded": "café résumé",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "c"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "af"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "é"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": " ré"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "sum"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": "é"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "caf"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "é ré"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "sumé"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "café"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": " résumé"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "café résum"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": "é"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 68,
+ 2320,
+ 860,
+ 9568,
+ 2204,
+ 860,
+ 50282
+ ],
+ "skipping_specials": "café résumé",
+ "kept": "[CLS]café résumé[SEP]"
+ }
+ },
+ {
+ "text": "注文をキャンセル",
+ "ids": [
+ 44712,
+ 20855,
+ 6449,
+ 33548,
+ 41760,
+ 9467,
+ 41662,
+ 16509
+ ],
+ "decoded": "注文をキャンセル",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "注"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "文"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "を"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "キ"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "ャ"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": "ン"
+ },
+ {
+ "start": 6,
+ "size": 1,
+ "decoded": "セ"
+ },
+ {
+ "start": 7,
+ "size": 1,
+ "decoded": "ル"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "注文"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "をキ"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "ャン"
+ },
+ {
+ "start": 6,
+ "size": 2,
+ "decoded": "セル"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "注文を"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "キャン"
+ },
+ {
+ "start": 6,
+ "size": 3,
+ "decoded": "セル"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "注文をキャ"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": "ンセル"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 44712,
+ 20855,
+ 6449,
+ 33548,
+ 41760,
+ 9467,
+ 41662,
+ 16509,
+ 50282
+ ],
+ "skipping_specials": "注文をキャンセル",
+ "kept": "[CLS]注文をキャンセル[SEP]"
+ }
+ },
+ {
+ "text": "مرحبا",
+ "ids": [
+ 5843,
+ 6900,
+ 21931,
+ 13621,
+ 3142
+ ],
+ "decoded": "مرحبا",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "م"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "ر"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "ح"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "ب"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "ا"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "مر"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "حب"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "ا"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "مرح"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "با"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "مرحبا"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 5843,
+ 6900,
+ 21931,
+ 13621,
+ 3142,
+ 50282
+ ],
+ "skipping_specials": "مرحبا",
+ "kept": "[CLS]مرحبا[SEP]"
+ }
+ },
+ {
+ "text": "क्‌ष",
+ "ids": [
+ 24381,
+ 17665,
+ 49003,
+ 2483,
+ 117
+ ],
+ "decoded": "क्‌ष",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "क"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "्"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "‌"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "क्"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "‌�"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "क्‌"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "ष"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "क्‌ष"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 24381,
+ 17665,
+ 49003,
+ 2483,
+ 117,
+ 50282
+ ],
+ "skipping_specials": "क्‌ष",
+ "kept": "[CLS]क्‌ष[SEP]"
+ }
+ },
+ {
+ "text": "😀",
+ "ids": [
+ 14931,
+ 235,
+ 211
+ ],
+ "decoded": "😀",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "😀"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "😀"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 14931,
+ 235,
+ 211,
+ 50282
+ ],
+ "skipping_specials": "😀",
+ "kept": "[CLS]😀[SEP]"
+ }
+ },
+ {
+ "text": "emoji 😀 here",
+ "ids": [
+ 43208,
+ 8020,
+ 49042,
+ 211,
+ 1060
+ ],
+ "decoded": "emoji 😀 here",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "emo"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "ji"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": " �"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": " here"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "emoji"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " 😀"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": " here"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "emoji �"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "� here"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "emoji 😀 here"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 43208,
+ 8020,
+ 49042,
+ 211,
+ 1060,
+ 50282
+ ],
+ "skipping_specials": "emoji 😀 here",
+ "kept": "[CLS]emoji 😀 here[SEP]"
+ }
+ },
+ {
+ "text": "🇬🇧",
+ "ids": [
+ 14931,
+ 218,
+ 107,
+ 14931,
+ 218,
+ 102
+ ],
+ "decoded": "🇬🇧",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "��"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "��"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "🇬"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "🇧"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "🇬�"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": "�"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 14931,
+ 218,
+ 107,
+ 14931,
+ 218,
+ 102,
+ 50282
+ ],
+ "skipping_specials": "🇬🇧",
+ "kept": "[CLS]🇬🇧[SEP]"
+ }
+ },
+ {
+ "text": "▁literal marker",
+ "ids": [
+ 10897,
+ 212,
+ 34943,
+ 10705
+ ],
+ "decoded": "▁literal marker",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "literal"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": " marker"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "▁"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "literal marker"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "▁literal"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": " marker"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "▁literal marker"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 10897,
+ 212,
+ 34943,
+ 10705,
+ 50282
+ ],
+ "skipping_specials": "▁literal marker",
+ "kept": "[CLS]▁literal marker[SEP]"
+ }
+ },
+ {
+ "text": "pre▁fix",
+ "ids": [
+ 3456,
+ 10897,
+ 212,
+ 11097
+ ],
+ "decoded": "pre▁fix",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "pre"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "fix"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "pre�"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "�fix"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "pre▁"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "fix"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "pre▁fix"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 3456,
+ 10897,
+ 212,
+ 11097,
+ 50282
+ ],
+ "skipping_specials": "pre▁fix",
+ "kept": "[CLS]pre▁fix[SEP]"
+ }
+ },
+ {
+ "text": " nbsp figure",
+ "ids": [
+ 575,
+ 6732,
+ 325,
+ 218,
+ 13206
+ ],
+ "decoded": " nbsp figure",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "nbsp"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "figure"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " nbsp"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " "
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "figure"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " nbsp�"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "�figure"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " nbsp figure"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 575,
+ 6732,
+ 325,
+ 218,
+ 13206,
+ 50282
+ ],
+ "skipping_specials": " nbsp figure",
+ "kept": "[CLS] nbsp figure[SEP]"
+ }
+ },
+ {
+ "text": "<0x41>",
+ "ids": [
+ 29,
+ 17,
+ 89,
+ 3156,
+ 31
+ ],
+ "decoded": "<0x41>",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "<"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "0"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "x"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "41"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": ">"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "<0"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "x41"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": ">"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "<0x"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "41>"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "<0x41>"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 29,
+ 17,
+ 89,
+ 3156,
+ 31,
+ 50282
+ ],
+ "skipping_specials": "<0x41>",
+ "kept": "[CLS]<0x41>[SEP]"
+ }
+ },
+ {
+ "text": "a<0xFF>b",
+ "ids": [
+ 66,
+ 29,
+ 17,
+ 30962,
+ 31,
+ 67
+ ],
+ "decoded": "a<0xFF>b",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "a"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "<"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "0"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "xFF"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": ">"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": "b"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "a<"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "0xFF"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": ">b"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "a<0"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "xFF>b"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "a<0xFF>"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": "b"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 66,
+ 29,
+ 17,
+ 30962,
+ 31,
+ 67,
+ 50282
+ ],
+ "skipping_specials": "a<0xFF>b",
+ "kept": "[CLS]a<0xFF>b[SEP]"
+ }
+ },
+ {
+ "text": "\u0001\u0002",
+ "ids": [
+ 178,
+ 179
+ ],
+ "decoded": "\u0001\u0002",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "\u0001"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "\u0002"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "\u0001\u0002"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "\u0001\u0002"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "\u0001\u0002"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 178,
+ 179,
+ 50282
+ ],
+ "skipping_specials": "\u0001\u0002",
+ "kept": "[CLS]\u0001\u0002[SEP]"
+ }
+ },
+ {
+ "text": "",
+ "ids": [
+ 170,
+ 211,
+ 211
+ ],
+ "decoded": "",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": ""
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": ""
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 170,
+ 211,
+ 211,
+ 50282
+ ],
+ "skipping_specials": "",
+ "kept": "[CLS][SEP]"
+ }
+ },
+ {
+ "text": "󰀀",
+ "ids": [
+ 175,
+ 110,
+ 211,
+ 211
+ ],
+ "decoded": "󰀀",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "��"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "�"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "󰀀"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 175,
+ 110,
+ 211,
+ 211,
+ 50282
+ ],
+ "skipping_specials": "󰀀",
+ "kept": "[CLS]󰀀[SEP]"
+ }
+ },
+ {
+ "text": "I cannot log in and need a password reset.",
+ "ids": [
+ 42,
+ 2550,
+ 2412,
+ 275,
+ 285,
+ 878,
+ 247,
+ 9868,
+ 14932,
+ 15
+ ],
+ "decoded": "I cannot log in and need a password reset.",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "I"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " cannot"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": " log"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": " in"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": " and"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": " need"
+ },
+ {
+ "start": 6,
+ "size": 1,
+ "decoded": " a"
+ },
+ {
+ "start": 7,
+ "size": 1,
+ "decoded": " password"
+ },
+ {
+ "start": 8,
+ "size": 1,
+ "decoded": " reset"
+ },
+ {
+ "start": 9,
+ "size": 1,
+ "decoded": "."
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "I cannot"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " log in"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": " and need"
+ },
+ {
+ "start": 6,
+ "size": 2,
+ "decoded": " a password"
+ },
+ {
+ "start": 8,
+ "size": 2,
+ "decoded": " reset."
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "I cannot log"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": " in and need"
+ },
+ {
+ "start": 6,
+ "size": 3,
+ "decoded": " a password reset"
+ },
+ {
+ "start": 9,
+ "size": 3,
+ "decoded": "."
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "I cannot log in and"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": " need a password reset."
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 42,
+ 2550,
+ 2412,
+ 275,
+ 285,
+ 878,
+ 247,
+ 9868,
+ 14932,
+ 15,
+ 50282
+ ],
+ "skipping_specials": "I cannot log in and need a password reset.",
+ "kept": "[CLS]I cannot log in and need a password reset.[SEP]"
+ }
+ },
+ {
+ "text": "Esta mensagem e confidencial.",
+ "ids": [
+ 38,
+ 18233,
+ 34939,
+ 31646,
+ 299,
+ 1461,
+ 301,
+ 49016,
+ 15
+ ],
+ "decoded": "Esta mensagem e confidencial.",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "E"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "sta"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": " mens"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "agem"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": " e"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": " conf"
+ },
+ {
+ "start": 6,
+ "size": 1,
+ "decoded": "id"
+ },
+ {
+ "start": 7,
+ "size": 1,
+ "decoded": "encial"
+ },
+ {
+ "start": 8,
+ "size": 1,
+ "decoded": "."
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "Esta"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " mensagem"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": " e conf"
+ },
+ {
+ "start": 6,
+ "size": 2,
+ "decoded": "idencial"
+ },
+ {
+ "start": 8,
+ "size": 2,
+ "decoded": "."
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "Esta mens"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "agem e conf"
+ },
+ {
+ "start": 6,
+ "size": 3,
+ "decoded": "idencial."
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "Esta mensagem e"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": " confidencial."
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 50281,
+ 38,
+ 18233,
+ 34939,
+ 31646,
+ 299,
+ 1461,
+ 301,
+ 49016,
+ 15,
+ 50282
+ ],
+ "skipping_specials": "Esta mensagem e confidencial.",
+ "kept": "[CLS]Esta mensagem e confidencial.[SEP]"
+ }
+ }
+ ]
+ },
+ "multilingual": {
+ "decoder": {
+ "type": "Sequence",
+ "decoders": [
+ {
+ "type": "Replace"
+ },
+ {
+ "type": "ByteFallback"
+ },
+ {
+ "type": "Fuse"
+ }
+ ]
+ },
+ "cases": [
+ {
+ "text": "Hello world",
+ "ids": [
+ 25957,
+ 2134
+ ],
+ "decoded": " Hello world",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " Hello"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " world"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " Hello world"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " Hello world"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " Hello world"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 25957,
+ 2134,
+ 1
+ ],
+ "skipping_specials": " Hello world",
+ "kept": "<bos> Hello world<eos>"
+ }
+ },
+ {
+ "text": "",
+ "ids": [],
+ "decoded": "",
+ "slices": [],
+ "with_specials": {
+ "ids": [
+ 2,
+ 1
+ ],
+ "skipping_specials": "",
+ "kept": "<bos><eos>"
+ }
+ },
+ {
+ "text": " ",
+ "ids": [
+ 235248
+ ],
+ "decoded": " ",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " "
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 235248,
+ 1
+ ],
+ "skipping_specials": " ",
+ "kept": "<bos> <eos>"
+ }
+ },
+ {
+ "text": " ",
+ "ids": [
+ 235248,
+ 235248,
+ 235248
+ ],
+ "decoded": " ",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " "
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " "
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 235248,
+ 235248,
+ 235248,
+ 1
+ ],
+ "skipping_specials": " ",
+ "kept": "<bos> <eos>"
+ }
+ },
+ {
+ "text": "\t",
+ "ids": [
+ 235248,
+ 226
+ ],
+ "decoded": " \t",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "\t"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " \t"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " \t"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " \t"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 235248,
+ 226,
+ 1
+ ],
+ "skipping_specials": " \t",
+ "kept": "<bos> \t<eos>"
+ }
+ },
+ {
+ "text": "a\tb\nc",
+ "ids": [
+ 476,
+ 226,
+ 235268,
+ 108,
+ 498
+ ],
+ "decoded": " a\tb\n c",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " a"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "\t"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "b"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "\n"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": " c"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " a\t"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "b\n"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": " c"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " a\tb"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "\n c"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " a\tb\n c"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 476,
+ 226,
+ 235268,
+ 108,
+ 498,
+ 1
+ ],
+ "skipping_specials": " a\tb\n c",
+ "kept": "<bos> a\tb\n c<eos>"
+ }
+ },
+ {
+ "text": "café résumé",
+ "ids": [
+ 18688,
+ 96537
+ ],
+ "decoded": " café résumé",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " café"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " résumé"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " café résumé"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " café résumé"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " café résumé"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 18688,
+ 96537,
+ 1
+ ],
+ "skipping_specials": " café résumé",
+ "kept": "<bos> café résumé<eos>"
+ }
+ },
+ {
+ "text": "注文をキャンセル",
+ "ids": [
+ 231293,
+ 235432,
+ 182129
+ ],
+ "decoded": " 注文をキャンセル",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " 注文"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "を"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "キャンセル"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " 注文を"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "キャンセル"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " 注文をキャンセル"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " 注文をキャンセル"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 231293,
+ 235432,
+ 182129,
+ 1
+ ],
+ "skipping_specials": " 注文をキャンセル",
+ "kept": "<bos> 注文をキャンセル<eos>"
+ }
+ },
+ {
+ "text": "مرحبا",
+ "ids": [
+ 89187,
+ 19833
+ ],
+ "decoded": " مرحبا",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " مرح"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "با"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " مرحبا"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " مرحبا"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " مرحبا"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 89187,
+ 19833,
+ 1
+ ],
+ "skipping_specials": " مرحبا",
+ "kept": "<bos> مرحبا<eos>"
+ }
+ },
+ {
+ "text": "क्‌ष",
+ "ids": [
+ 2280,
+ 235485,
+ 235762,
+ 236355
+ ],
+ "decoded": " क्‌ष",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " क"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "्"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "‌"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "ष"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " क्"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "‌ष"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " क्‌"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "ष"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " क्‌ष"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 2280,
+ 235485,
+ 235762,
+ 236355,
+ 1
+ ],
+ "skipping_specials": " क्‌ष",
+ "kept": "<bos> क्‌ष<eos>"
+ }
+ },
+ {
+ "text": "😀",
+ "ids": [
+ 79937
+ ],
+ "decoded": " 😀",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " 😀"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " 😀"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " 😀"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " 😀"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 79937,
+ 1
+ ],
+ "skipping_specials": " 😀",
+ "kept": "<bos> 😀<eos>"
+ }
+ },
+ {
+ "text": "emoji 😀 here",
+ "ids": [
+ 52810,
+ 79937,
+ 1517
+ ],
+ "decoded": " emoji 😀 here",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " emoji"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " 😀"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": " here"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " emoji 😀"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " here"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " emoji 😀 here"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " emoji 😀 here"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 52810,
+ 79937,
+ 1517,
+ 1
+ ],
+ "skipping_specials": " emoji 😀 here",
+ "kept": "<bos> emoji 😀 here<eos>"
+ }
+ },
+ {
+ "text": "🇬🇧",
+ "ids": [
+ 235248,
+ 242330,
+ 241461
+ ],
+ "decoded": " 🇬🇧",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "🇬"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "🇧"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " 🇬"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "🇧"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " 🇬🇧"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " 🇬🇧"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 235248,
+ 242330,
+ 241461,
+ 1
+ ],
+ "skipping_specials": " 🇬🇧",
+ "kept": "<bos> 🇬🇧<eos>"
+ }
+ },
+ {
+ "text": "▁literal marker",
+ "ids": [
+ 36414,
+ 24416
+ ],
+ "decoded": " literal marker",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " literal"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " marker"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " literal marker"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " literal marker"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " literal marker"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 36414,
+ 24416,
+ 1
+ ],
+ "skipping_specials": " literal marker",
+ "kept": "<bos> literal marker<eos>"
+ }
+ },
+ {
+ "text": "pre▁fix",
+ "ids": [
+ 953,
+ 6530
+ ],
+ "decoded": " pre fix",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " pre"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " fix"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " pre fix"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " pre fix"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " pre fix"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 953,
+ 6530,
+ 1
+ ],
+ "skipping_specials": " pre fix",
+ "kept": "<bos> pre fix<eos>"
+ }
+ },
+ {
+ "text": " nbsp figure",
+ "ids": [
+ 84306,
+ 11450,
+ 243499,
+ 6270
+ ],
+ "decoded": "  nbsp figure",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": "  "
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "nbsp"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "figure"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": "  nbsp"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " figure"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": "  nbsp "
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "figure"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": "  nbsp figure"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 84306,
+ 11450,
+ 243499,
+ 6270,
+ 1
+ ],
+ "skipping_specials": "  nbsp figure",
+ "kept": "<bos>  nbsp figure<eos>"
+ }
+ },
+ {
+ "text": "<0x41>",
+ "ids": [
+ 968,
+ 235276,
+ 235297,
+ 235310,
+ 235274,
+ 235313
+ ],
+ "decoded": " <0x41>",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " <"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "0"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "x"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "4"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "1"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": ">"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " <0"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "x4"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "1>"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " <0x"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "41>"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " <0x41"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": ">"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 968,
+ 235276,
+ 235297,
+ 235310,
+ 235274,
+ 235313,
+ 1
+ ],
+ "skipping_specials": " <0x41>",
+ "kept": "<bos> <0x41><eos>"
+ }
+ },
+ {
+ "text": "a<0xFF>b",
+ "ids": [
+ 476,
+ 235322,
+ 235276,
+ 16663,
+ 235313,
+ 235268
+ ],
+ "decoded": " a<0xFF>b",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " a"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "<"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "0"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "xFF"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": ">"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": "b"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " a<"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "0xFF"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": ">b"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " a<0"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "xFF>b"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " a<0xFF>"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": "b"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 476,
+ 235322,
+ 235276,
+ 16663,
+ 235313,
+ 235268,
+ 1
+ ],
+ "skipping_specials": " a<0xFF>b",
+ "kept": "<bos> a<0xFF>b<eos>"
+ }
+ },
+ {
+ "text": "\u0001\u0002",
+ "ids": [
+ 235248,
+ 238213,
+ 247977
+ ],
+ "decoded": " \u0001\u0002",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "\u0001"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "\u0002"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " \u0001"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "\u0002"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " \u0001\u0002"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " \u0001\u0002"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 235248,
+ 238213,
+ 247977,
+ 1
+ ],
+ "skipping_specials": " \u0001\u0002",
+ "kept": "<bos> \u0001\u0002<eos>"
+ }
+ },
+ {
+ "text": "",
+ "ids": [
+ 235248,
+ 240539
+ ],
+ "decoded": " ",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": ""
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " "
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " "
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 235248,
+ 240539,
+ 1
+ ],
+ "skipping_specials": " ",
+ "kept": "<bos> <eos>"
+ }
+ },
+ {
+ "text": "󰀀",
+ "ids": [
+ 235248,
+ 460,
+ 393,
+ 345,
+ 345
+ ],
+ "decoded": " 󰀀",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " "
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " �"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": "��"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "�"
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " ��"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": "��"
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " 󰀀"
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 235248,
+ 460,
+ 393,
+ 345,
+ 345,
+ 1
+ ],
+ "skipping_specials": " 󰀀",
+ "kept": "<bos> 󰀀<eos>"
+ }
+ },
+ {
+ "text": "I cannot log in and need a password reset.",
+ "ids": [
+ 590,
+ 2952,
+ 3846,
+ 575,
+ 578,
+ 1476,
+ 476,
+ 8003,
+ 13846,
+ 235265
+ ],
+ "decoded": " I cannot log in and need a password reset.",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " I"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " cannot"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": " log"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": " in"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": " and"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": " need"
+ },
+ {
+ "start": 6,
+ "size": 1,
+ "decoded": " a"
+ },
+ {
+ "start": 7,
+ "size": 1,
+ "decoded": " password"
+ },
+ {
+ "start": 8,
+ "size": 1,
+ "decoded": " reset"
+ },
+ {
+ "start": 9,
+ "size": 1,
+ "decoded": "."
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " I cannot"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " log in"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": " and need"
+ },
+ {
+ "start": 6,
+ "size": 2,
+ "decoded": " a password"
+ },
+ {
+ "start": 8,
+ "size": 2,
+ "decoded": " reset."
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " I cannot log"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": " in and need"
+ },
+ {
+ "start": 6,
+ "size": 3,
+ "decoded": " a password reset"
+ },
+ {
+ "start": 9,
+ "size": 3,
+ "decoded": "."
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " I cannot log in and"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": " need a password reset."
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 590,
+ 2952,
+ 3846,
+ 575,
+ 578,
+ 1476,
+ 476,
+ 8003,
+ 13846,
+ 235265,
+ 1
+ ],
+ "skipping_specials": " I cannot log in and need a password reset.",
+ "kept": "<bos> I cannot log in and need a password reset.<eos>"
+ }
+ },
+ {
+ "text": "Esta mensagem e confidencial.",
+ "ids": [
+ 22974,
+ 70816,
+ 526,
+ 2103,
+ 45972,
+ 235265
+ ],
+ "decoded": " Esta mensagem e confidencial.",
+ "slices": [
+ {
+ "start": 0,
+ "size": 1,
+ "decoded": " Esta"
+ },
+ {
+ "start": 1,
+ "size": 1,
+ "decoded": " mensagem"
+ },
+ {
+ "start": 2,
+ "size": 1,
+ "decoded": " e"
+ },
+ {
+ "start": 3,
+ "size": 1,
+ "decoded": " conf"
+ },
+ {
+ "start": 4,
+ "size": 1,
+ "decoded": "idencial"
+ },
+ {
+ "start": 5,
+ "size": 1,
+ "decoded": "."
+ },
+ {
+ "start": 0,
+ "size": 2,
+ "decoded": " Esta mensagem"
+ },
+ {
+ "start": 2,
+ "size": 2,
+ "decoded": " e conf"
+ },
+ {
+ "start": 4,
+ "size": 2,
+ "decoded": "idencial."
+ },
+ {
+ "start": 0,
+ "size": 3,
+ "decoded": " Esta mensagem e"
+ },
+ {
+ "start": 3,
+ "size": 3,
+ "decoded": " confidencial."
+ },
+ {
+ "start": 0,
+ "size": 5,
+ "decoded": " Esta mensagem e confidencial"
+ },
+ {
+ "start": 5,
+ "size": 5,
+ "decoded": "."
+ }
+ ],
+ "with_specials": {
+ "ids": [
+ 2,
+ 22974,
+ 70816,
+ 526,
+ 2103,
+ 45972,
+ 235265,
+ 1
+ ],
+ "skipping_specials": " Esta mensagem e confidencial.",
+ "kept": "<bos> Esta mensagem e confidencial.<eos>"
+ }
+ }
+ ]
+ }
+ },
+ "mixed_token": {
+ "spec": {
+ "version": "1.0",
+ "truncation": null,
+ "padding": null,
+ "added_tokens": [],
+ "normalizer": null,
+ "pre_tokenizer": {
+ "type": "ByteLevel",
+ "add_prefix_space": true,
+ "trim_offsets": true,
+ "use_regex": true
+ },
+ "post_processor": null,
+ "decoder": {
+ "type": "ByteLevel",
+ "add_prefix_space": true,
+ "trim_offsets": true,
+ "use_regex": true
+ },
+ "model": {
+ "type": "BPE",
+ "dropout": null,
+ "unk_token": null,
+ "continuing_subword_prefix": null,
+ "end_of_word_suffix": null,
+ "fuse_unk": false,
+ "byte_fallback": false,
+ "ignore_merges": true,
+ "vocab": {
+ "!": 0,
+ "\"": 1,
+ "#": 2,
+ "$": 3,
+ "%": 4,
+ "&": 5,
+ "'": 6,
+ "(": 7,
+ ")": 8,
+ "*": 9,
+ "+": 10,
+ ",": 11,
+ "-": 12,
+ ".": 13,
+ "/": 14,
+ "0": 15,
+ "1": 16,
+ "2": 17,
+ "3": 18,
+ "4": 19,
+ "5": 20,
+ "6": 21,
+ "7": 22,
+ "8": 23,
+ "9": 24,
+ ":": 25,
+ ";": 26,
+ "<": 27,
+ "=": 28,
+ ">": 29,
+ "?": 30,
+ "@": 31,
+ "A": 32,
+ "B": 33,
+ "C": 34,
+ "D": 35,
+ "E": 36,
+ "F": 37,
+ "G": 38,
+ "H": 39,
+ "I": 40,
+ "J": 41,
+ "K": 42,
+ "L": 43,
+ "M": 44,
+ "N": 45,
+ "O": 46,
+ "P": 47,
+ "Q": 48,
+ "R": 49,
+ "S": 50,
+ "T": 51,
+ "U": 52,
+ "V": 53,
+ "W": 54,
+ "X": 55,
+ "Y": 56,
+ "Z": 57,
+ "[": 58,
+ "\\": 59,
+ "]": 60,
+ "^": 61,
+ "_": 62,
+ "`": 63,
+ "a": 64,
+ "b": 65,
+ "c": 66,
+ "d": 67,
+ "e": 68,
+ "f": 69,
+ "g": 70,
+ "h": 71,
+ "i": 72,
+ "j": 73,
+ "k": 74,
+ "l": 75,
+ "m": 76,
+ "n": 77,
+ "o": 78,
+ "p": 79,
+ "q": 80,
+ "r": 81,
+ "s": 82,
+ "t": 83,
+ "u": 84,
+ "v": 85,
+ "w": 86,
+ "x": 87,
+ "y": 88,
+ "z": 89,
+ "{": 90,
+ "|": 91,
+ "}": 92,
+ "~": 93,
+ "¡": 94,
+ "¢": 95,
+ "£": 96,
+ "¤": 97,
+ "¥": 98,
+ "¦": 99,
+ "§": 100,
+ "¨": 101,
+ "©": 102,
+ "ª": 103,
+ "«": 104,
+ "¬": 105,
+ "®": 106,
+ "¯": 107,
+ "°": 108,
+ "±": 109,
+ "²": 110,
+ "³": 111,
+ "´": 112,
+ "µ": 113,
+ "¶": 114,
+ "·": 115,
+ "¸": 116,
+ "¹": 117,
+ "º": 118,
+ "»": 119,
+ "¼": 120,
+ "½": 121,
+ "¾": 122,
+ "¿": 123,
+ "À": 124,
+ "Á": 125,
+ "Â": 126,
+ "Ã": 127,
+ "Ä": 128,
+ "Å": 129,
+ "Æ": 130,
+ "Ç": 131,
+ "È": 132,
+ "É": 133,
+ "Ê": 134,
+ "Ë": 135,
+ "Ì": 136,
+ "Í": 137,
+ "Î": 138,
+ "Ï": 139,
+ "Ð": 140,
+ "Ñ": 141,
+ "Ò": 142,
+ "Ó": 143,
+ "Ô": 144,
+ "Õ": 145,
+ "Ö": 146,
+ "×": 147,
+ "Ø": 148,
+ "Ù": 149,
+ "Ú": 150,
+ "Û": 151,
+ "Ü": 152,
+ "Ý": 153,
+ "Þ": 154,
+ "ß": 155,
+ "à": 156,
+ "á": 157,
+ "â": 158,
+ "ã": 159,
+ "ä": 160,
+ "å": 161,
+ "æ": 162,
+ "ç": 163,
+ "è": 164,
+ "é": 165,
+ "ê": 166,
+ "ë": 167,
+ "ì": 168,
+ "í": 169,
+ "î": 170,
+ "ï": 171,
+ "ð": 172,
+ "ñ": 173,
+ "ò": 174,
+ "ó": 175,
+ "ô": 176,
+ "õ": 177,
+ "ö": 178,
+ "÷": 179,
+ "ø": 180,
+ "ù": 181,
+ "ú": 182,
+ "û": 183,
+ "ü": 184,
+ "ý": 185,
+ "þ": 186,
+ "ÿ": 187,
+ "Ā": 188,
+ "ā": 189,
+ "Ă": 190,
+ "ă": 191,
+ "Ą": 192,
+ "ą": 193,
+ "Ć": 194,
+ "ć": 195,
+ "Ĉ": 196,
+ "ĉ": 197,
+ "Ċ": 198,
+ "ċ": 199,
+ "Č": 200,
+ "č": 201,
+ "Ď": 202,
+ "ď": 203,
+ "Đ": 204,
+ "đ": 205,
+ "Ē": 206,
+ "ē": 207,
+ "Ĕ": 208,
+ "ĕ": 209,
+ "Ė": 210,
+ "ė": 211,
+ "Ę": 212,
+ "ę": 213,
+ "Ě": 214,
+ "ě": 215,
+ "Ĝ": 216,
+ "ĝ": 217,
+ "Ğ": 218,
+ "ğ": 219,
+ "Ġ": 220,
+ "ġ": 221,
+ "Ģ": 222,
+ "ģ": 223,
+ "Ĥ": 224,
+ "ĥ": 225,
+ "Ħ": 226,
+ "ħ": 227,
+ "Ĩ": 228,
+ "ĩ": 229,
+ "Ī": 230,
+ "ī": 231,
+ "Ĭ": 232,
+ "ĭ": 233,
+ "Į": 234,
+ "į": 235,
+ "İ": 236,
+ "ı": 237,
+ "IJ": 238,
+ "ij": 239,
+ "Ĵ": 240,
+ "ĵ": 241,
+ "Ķ": 242,
+ "ķ": 243,
+ "ĸ": 244,
+ "Ĺ": 245,
+ "ĺ": 246,
+ "Ļ": 247,
+ "ļ": 248,
+ "Ľ": 249,
+ "ľ": 250,
+ "Ŀ": 251,
+ "ŀ": 252,
+ "Ł": 253,
+ "ł": 254,
+ "Ń": 255,
+ "Ġ ": 256,
+ "Ġhi": 257,
+ " ": 258
+ },
+ "merges": []
+ }
+ },
+ "separating_token": "Ġ ",
+ "scope": "token",
+ "cases": [
+ {
+ "why": "the mixed token alone: the whole point",
+ "ids": [
+ 256
+ ],
+ "decoded": "Ġ "
+ },
+ {
+ "why": "entirely in the alphabet",
+ "ids": [
+ 257
+ ],
+ "decoded": " hi"
+ },
+ {
+ "why": "entirely literal",
+ "ids": [
+ 258
+ ],
+ "decoded": " "
+ },
+ {
+ "why": "a mixed token beside a plain one",
+ "ids": [
+ 256,
+ 257
+ ],
+ "decoded": "Ġ hi"
+ }
+ ]
+ },
+ "synthetic_decoders": {
+ "strip": {
+ "why": "Strip, which no shipped checkpoint declares",
+ "spec": {
+ "version": "1.0",
+ "truncation": null,
+ "padding": null,
+ "added_tokens": [],
+ "normalizer": null,
+ "pre_tokenizer": null,
+ "post_processor": null,
+ "model": {
+ "type": "BPE",
+ "dropout": null,
+ "unk_token": null,
+ "continuing_subword_prefix": null,
+ "end_of_word_suffix": null,
+ "fuse_unk": false,
+ "byte_fallback": true,
+ "ignore_merges": true,
+ "vocab": {
+ " hello ": 0,
+ " x": 1,
+ " ": 2,
+ "no": 3
+ },
+ "merges": []
+ },
+ "decoder": {
+ "type": "Strip",
+ "content": " ",
+ "start": 2,
+ "stop": 1
+ }
+ },
+ "crate_panics": [
+ [
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "cases": [
+ {
+ "ids": [
+ 0
+ ],
+ "decoded": "hello"
+ },
+ {
+ "ids": [
+ 1
+ ],
+ "decoded": "x"
+ },
+ {
+ "ids": [
+ 3
+ ],
+ "decoded": "no"
+ },
+ {
+ "ids": [
+ 0,
+ 3
+ ],
+ "decoded": "hellono"
+ }
+ ]
+ },
+ "bytefallback_hex_case": {
+ "why": "which byte-token spellings the crate reads as a byte",
+ "spec": {
+ "version": "1.0",
+ "truncation": null,
+ "padding": null,
+ "added_tokens": [],
+ "normalizer": null,
+ "pre_tokenizer": null,
+ "post_processor": null,
+ "model": {
+ "type": "BPE",
+ "dropout": null,
+ "unk_token": null,
+ "continuing_subword_prefix": null,
+ "end_of_word_suffix": null,
+ "fuse_unk": false,
+ "byte_fallback": true,
+ "ignore_merges": true,
+ "vocab": {
+ "<0xFF>": 0,
+ "<0xff>": 1,
+ "<0x41>": 2,
+ "<0x61>": 3,
+ "a": 4,
+ "<0x+5>": 5,
+ "<0x 5>": 6,
+ "<0XFF>": 7,
+ "<0xfF>": 8,
+ "<0xF3>": 9,
+ "<0xB0>": 10,
+ "<0x80>": 11
+ },
+ "merges": []
+ },
+ "decoder": {
+ "type": "ByteFallback"
+ }
+ },
+ "cases": [
+ {
+ "ids": [
+ 2
+ ],
+ "decoded": "A"
+ },
+ {
+ "ids": [
+ 3
+ ],
+ "decoded": "a"
+ },
+ {
+ "ids": [
+ 1
+ ],
+ "decoded": "�"
+ },
+ {
+ "ids": [
+ 0
+ ],
+ "decoded": "�"
+ },
+ {
+ "ids": [
+ 2,
+ 3
+ ],
+ "decoded": "Aa"
+ },
+ {
+ "ids": [
+ 0,
+ 0
+ ],
+ "decoded": "��"
+ },
+ {
+ "ids": [
+ 1,
+ 4
+ ],
+ "decoded": "�a"
+ },
+ {
+ "ids": [
+ 5
+ ],
+ "decoded": "\u0005"
+ },
+ {
+ "ids": [
+ 6
+ ],
+ "decoded": "<0x 5>"
+ },
+ {
+ "ids": [
+ 7
+ ],
+ "decoded": "<0XFF>"
+ },
+ {
+ "ids": [
+ 8
+ ],
+ "decoded": "�"
+ },
+ {
+ "ids": [
+ 5,
+ 4
+ ],
+ "decoded": "\u0005a"
+ },
+ {
+ "ids": [
+ 6,
+ 4
+ ],
+ "decoded": "<0x 5>a"
+ },
+ {
+ "ids": [
+ 7,
+ 4
+ ],
+ "decoded": "<0XFF>a"
+ },
+ {
+ "ids": [
+ 9,
+ 10
+ ],
+ "decoded": "��"
+ },
+ {
+ "ids": [
+ 9,
+ 10,
+ 11
+ ],
+ "decoded": "���"
+ },
+ {
+ "ids": [
+ 9,
+ 10,
+ 11,
+ 11
+ ],
+ "decoded": "󰀀"
+ },
+ {
+ "ids": [
+ 4,
+ 9,
+ 10,
+ 4
+ ],
+ "decoded": "a��a"
+ }
+ ]
+ },
+ "no_decoder": {
+ "why": "no decoder section: the separator the crate falls back to",
+ "spec": {
+ "version": "1.0",
+ "truncation": null,
+ "padding": null,
+ "added_tokens": [],
+ "normalizer": null,
+ "pre_tokenizer": null,
+ "post_processor": null,
+ "model": {
+ "type": "BPE",
+ "dropout": null,
+ "unk_token": null,
+ "continuing_subword_prefix": null,
+ "end_of_word_suffix": null,
+ "fuse_unk": false,
+ "byte_fallback": true,
+ "ignore_merges": true,
+ "vocab": {
+ "ab": 0,
+ "cd": 1,
+ "ef": 2,
+ " ": 3
+ },
+ "merges": []
+ }
+ },
+ "cases": [
+ {
+ "ids": [
+ 0
+ ],
+ "decoded": "ab"
+ },
+ {
+ "ids": [
+ 0,
+ 1
+ ],
+ "decoded": "ab cd"
+ },
+ {
+ "ids": [
+ 0,
+ 1,
+ 2
+ ],
+ "decoded": "ab cd ef"
+ },
+ {
+ "ids": [
+ 0,
+ 3,
+ 1
+ ],
+ "decoded": "ab cd"
+ },
+ {
+ "ids": [
+ 3,
+ 3
+ ],
+ "decoded": " "
+ }
+ ]
+ },
+ "nested_sequence": {
+ "why": "a Sequence nested in a Sequence keeps its pieces",
+ "spec": {
+ "version": "1.0",
+ "truncation": null,
+ "padding": null,
+ "added_tokens": [],
+ "normalizer": null,
+ "pre_tokenizer": null,
+ "post_processor": null,
+ "model": {
+ "type": "BPE",
+ "dropout": null,
+ "unk_token": null,
+ "continuing_subword_prefix": null,
+ "end_of_word_suffix": null,
+ "fuse_unk": false,
+ "byte_fallback": true,
+ "ignore_merges": true,
+ "vocab": {
+ "▁Hello": 0,
+ "▁world": 1,
+ "▁": 2,
+ "▁a": 3
+ },
+ "merges": []
+ },
+ "decoder": {
+ "type": "Sequence",
+ "decoders": [
+ {
+ "type": "Sequence",
+ "decoders": [
+ {
+ "type": "Replace",
+ "pattern": {
+ "String": "▁"
+ },
+ "content": " "
+ },
+ {
+ "type": "ByteFallback"
+ }
+ ]
+ },
+ {
+ "type": "Strip",
+ "content": " ",
+ "start": 1,
+ "stop": 0
+ }
+ ]
+ }
+ },
+ "cases": [
+ {
+ "ids": [
+ 0,
+ 1
+ ],
+ "decoded": "Helloworld"
+ },
+ {
+ "ids": [
+ 2,
+ 2,
+ 0
+ ],
+ "decoded": "Hello"
+ },
+ {
+ "ids": [
+ 2,
+ 3,
+ 2
+ ],
+ "decoded": "a"
+ },
+ {
+ "ids": [
+ 0
+ ],
+ "decoded": "Hello"
+ },
+ {
+ "ids": [
+ 3
+ ],
+ "decoded": "a"
+ }
+ ]
+ },
+ "bytefallback_piece_boundaries": {
+ "why": "a failed byte run is one piece per byte, not one piece of N",
+ "spec": {
+ "version": "1.0",
+ "truncation": null,
+ "padding": null,
+ "added_tokens": [],
+ "normalizer": null,
+ "pre_tokenizer": null,
+ "post_processor": null,
+ "model": {
+ "type": "BPE",
+ "dropout": null,
+ "unk_token": null,
+ "continuing_subword_prefix": null,
+ "end_of_word_suffix": null,
+ "fuse_unk": false,
+ "byte_fallback": true,
+ "ignore_merges": true,
+ "vocab": {
+ "<0xF3>": 0,
+ "<0xB0>": 1,
+ "<0x80>": 2,
+ "hi": 3
+ },
+ "merges": []
+ },
+ "decoder": {
+ "type": "Sequence",
+ "decoders": [
+ {
+ "type": "ByteFallback"
+ },
+ {
+ "type": "Replace",
+ "pattern": {
+ "String": "��"
+ },
+ "content": "X"
+ }
+ ]
+ }
+ },
+ "cases": [
+ {
+ "ids": [
+ 0,
+ 1
+ ],
+ "decoded": "��"
+ },
+ {
+ "ids": [
+ 0,
+ 1,
+ 2
+ ],
+ "decoded": "���"
+ },
+ {
+ "ids": [
+ 3,
+ 0,
+ 1,
+ 3
+ ],
+ "decoded": "hi��hi"
+ },
+ {
+ "ids": [
+ 0,
+ 1,
+ 2,
+ 2
+ ],
+ "decoded": "󰀀"
+ },
+ {
+ "ids": [
+ 3
+ ],
+ "decoded": "hi"
+ }
+ ]
+ }
+ }
+}
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/PreTokenizer.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/PreTokenizer.java
index b77de0a2..3e134873 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/PreTokenizer.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/PreTokenizer.java
@@ -32,6 +32,26 @@ abstract class PreTokenizer {
/** Cuts one span of normalized text into pieces for the BPE stage. */
abstract List<String> preTokenize(String text);
+ /**
+ * The byte a ByteLevel stand-in character represents, or -1 when it is not one of the 256.
+ *
+ * <p>Exposed on the enclosing class because {@link TokenDecoder} needs the inverse mapping and
+ * {@code ByteLevel} is private here. One table, inverted once -- a decoder with its own copy
+ * could disagree in one position and produce plausible text with one character wrong.
+ */
+ static int byteForChar(char c) {
+ return ByteLevel.byteForChar(c);
+ }
+
+ /**
+ * The stand-in character for a byte, which is what {@link #byteForChar} inverts.
+ *
+ * @param b a byte value, 0 to 255
+ */
+ static char charForByte(int b) {
+ return ByteLevel.charForByte(b);
+ }
+
/** Builds a pre-tokenizer from a checkpoint's {@code pre_tokenizer} node, which may be null. */
static PreTokenizer from(Map<String, Object> node) {
if (node == null) {
@@ -127,6 +147,48 @@ abstract class PreTokenizer {
this.useRegex = useRegex;
}
+ /**
+ * The stand-in character for a byte.
+ *
+ * @param b a byte value, 0 to 255
+ * @throws IllegalArgumentException for anything outside that range, rather than the
+ * {@code ArrayIndexOutOfBoundsException} a bare index would raise
+ */
+ static char charForByte(int b) {
+ if (b < 0 || b >= BYTE_TO_CHAR.length) {
+ throw new IllegalArgumentException(
+ "a byte is 0 to 255, got " + b);
+ }
+ return BYTE_TO_CHAR[b];
+ }
+
+ /**
+ * The byte a stand-in character represents, or -1 when it is not one of the 256.
+ *
+ * <p>Built by inverting {@link #BYTE_TO_CHAR} rather than by a second construction, so the
+ * two cannot disagree: a decode that mapped one character to the wrong byte would produce
+ * plausible text with one character wrong, and the window it belongs to would re-tokenize
+ * to different ids with nothing to show why.
+ */
+ static int byteForChar(char c) {
+ return c < CHAR_TO_BYTE.length ? CHAR_TO_BYTE[c] : -1;
+ }
+
+ private static final int[] CHAR_TO_BYTE = buildCharToByte();
+
+ private static int[] buildCharToByte() {
+ int highest = 0;
+ for (char c : BYTE_TO_CHAR) {
+ highest = Math.max(highest, c);
+ }
+ int[] table = new int[highest + 1];
+ java.util.Arrays.fill(table, -1);
+ for (int b = 0; b < BYTE_TO_CHAR.length; b++) {
+ table[BYTE_TO_CHAR[b]] = b;
+ }
+ return table;
+ }
+
/**
* The GPT-2 alphabet, built rather than transcribed.
*
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/TokenDecoder.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/TokenDecoder.java
new file mode 100644
index 00000000..2b9c8002
--- /dev/null
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/TokenDecoder.java
@@ -0,0 +1,514 @@
+package com.convaiinnovations.laya.tokenizer;
+
+import com.convaiinnovations.laya.json.Json;
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.CharacterCodingException;
+import java.nio.charset.CharsetDecoder;
+import java.nio.charset.CodingErrorAction;
+import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The inverse of tokenization: a list of token strings back into text.
+ *
+ * <p>Needed because {@code predictLong} scans a long state in overlapping token windows and hands
+ * each window back to {@code predictBatch} as TEXT, so every window has to be decoded. It is also
+ * worth having on its own -- {@code laya-ts} and {@code laya-dotnet} both expose a decode -- but
+ * the windowing is what made it necessary.
+ *
+ * <p>Unlike the pre-tokenizer, this has no Python reference to port: decoding lives in the Rust
+ * {@code tokenizers} crate, so the rules below were probed out of it against the two shipped
+ * checkpoints, and the recorded round-trips in {@code fixtures/decode_text.json} are the contract.
+ *
+ * <h2>Decoding is not lossless, and that is the contract</h2>
+ *
+ * <p>On the english checkpoint it happens to round-trip. On the multilingual one it does not, and
+ * the port must reproduce the loss exactly rather than improve on it:
+ *
+ * <pre>
+ * english "Hello world" -> "Hello world"
+ * multilingual "Hello world" -> " Hello world" &lt;- a leading space appears
+ * multilingual "a\tb\nc" -> " a\tb\n c" &lt;- and one after the newline
+ * </pre>
+ *
+ * <p>That is Metaspace: its pre-tokenizer prepends U+2581 to the text ({@code prepend_scheme:
+ * always}), and the decoder turns every U+2581 back into a space, including the prepended one. The
+ * reference lives with it -- {@code predict_long}'s own comment says the 50% window overlap
+ * absorbs the boundary drift -- so a port that trimmed the space would re-tokenize every window
+ * differently from the reference and diverge on every answer downstream.
+ */
+abstract class TokenDecoder {
+
+ /** The decoded text for these tokens, in order. */
+ abstract String decode(List<String> tokens);
+
+ /**
+ * The decoder a {@code tokenizer.json} declares, or a plain join when it declares none.
+ *
+ * <p>A checkpoint with no {@code decoder} section is not an error: the crate joins the tokens
+ * with a space, and refusing to load would reject a valid tokenizer over a decode path a
+ * caller may never use. See {@link Join} for the separator, which this got wrong.
+ */
+ static TokenDecoder from(Map<String, Object> node) {
+ if (node == null) {
+ return new Join();
+ }
+ String type = Json.string(node, "type", null);
+ if (type == null) {
+ throw new Json.JsonException("a decoder node must declare a type");
+ }
+ switch (type) {
+ case "ByteLevel":
+ return new ByteLevel();
+ case "Replace":
+ return new Replace(node);
+ case "ByteFallback":
+ return new ByteFallback();
+ case "Fuse":
+ return new Fuse();
+ case "Strip":
+ return new Strip(node);
+ case "Sequence": {
+ Object steps = node.get("decoders");
+ if (!(steps instanceof List)) {
+ throw new Json.JsonException("a Sequence decoder needs a decoders array");
+ }
+ List<TokenDecoder> parts = new ArrayList<>();
+ for (Object step : (List<?>) steps) {
+ if (!(step instanceof Map)) {
+ throw new Json.JsonException("every Sequence decoder step must be an object");
+ }
+ @SuppressWarnings("unchecked")
+ Map<String, Object> stepNode = (Map<String, Object>) step;
+ parts.add(from(stepNode));
+ }
+ return new Sequence(parts);
+ }
+ default:
+ // Refusing beats guessing. A WordPiece or Metaspace decoder would produce
+ // plausible-looking text with the wrong spaces, and the windows predictLong builds
+ // from it would re-tokenize to different ids with nothing to show it had happened.
+ throw new Json.JsonException(
+ "laya-java implements the ByteLevel, Replace, ByteFallback, Fuse, Strip and"
+ + " Sequence decoders; this checkpoint declares \"" + type + "\". Refusing"
+ + " to load rather than decode it wrongly.");
+ }
+ }
+
+ // ------------------------------------------------------------------ the steps
+
+ /**
+ * No decoder declared: the tokens joined with a SPACE.
+ *
+ * <p>A space, not nothing. The crate's fallback for an absent decoder is
+ * {@code tokens.join(" ")}, and this joined with nothing -- wrong for every multi-token
+ * decode, not for an edge case. Measured on the multilingual vocabulary with its
+ * {@code decoder} key removed: ids {@code [25957, 2134]} give {@code "\u2581Hello
+ * \u2581world"}, with a space between the pieces.
+ *
+ * <p>Unreachable from either shipped checkpoint, since both declare a decoder. That is the
+ * reason it was wrong and stayed wrong: this is the one branch of {@code from} that is
+ * permissive rather than refusing, so it is where a third-party checkpoint lands, and the
+ * recorded fixtures can never reach it. It has its own test for that reason.
+ */
+ private static final class Join extends TokenDecoder {
+ @Override
+ String decode(List<String> tokens) {
+ return String.join(" ", tokens);
+ }
+ }
+
+ /**
+ * The inverse of the ByteLevel pre-tokenizer: every stand-in character is one byte.
+ *
+ * <p>The bytes are then UTF-8, and a token boundary can fall INSIDE a multi-byte character --
+ * CJK is three bytes and routinely splits across two tokens -- so the whole token list is
+ * concatenated to bytes first and decoded once at the end. Decoding token by token would turn
+ * every split character into replacement characters.
+ */
+ private static final class ByteLevel extends TokenDecoder {
+ @Override
+ String decode(List<String> tokens) {
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ for (String token : tokens) {
+ // Whole-token fallback, not per character. The crate folds over a token's
+ // characters and, if ANY of them is outside the stand-in alphabet, uses the
+ // token's own UTF-8 bytes for the WHOLE token. Falling back per character
+ // coincides on a token that is entirely literal -- which is the only kind this
+ // vocabulary has -- and would differ on a mixed one, so the rule is written as
+ // the crate writes it rather than as the corpus happens to need.
+ byte[] mapped = mapAll(token);
+ if (mapped == null) {
+ byte[] raw = token.getBytes(StandardCharsets.UTF_8);
+ bytes.write(raw, 0, raw.length);
+ continue;
+ }
+ bytes.write(mapped, 0, mapped.length);
+ }
+ return lossyUtf8(bytes.toByteArray());
+ }
+
+ /** Every character as its byte, or null when one of them is not a stand-in. */
+ private static byte[] mapAll(String token) {
+ byte[] out = new byte[token.length()];
+ for (int i = 0; i < token.length(); i++) {
+ int b = PreTokenizer.byteForChar(token.charAt(i));
+ if (b < 0) {
+ return null;
+ }
+ out[i] = (byte) b;
+ }
+ return out;
+ }
+
+
+ }
+
+ /** {@code Replace}: a literal substring for another, applied to every token. */
+ private static final class Replace extends TokenDecoder {
+
+ private final String pattern;
+ private final String content;
+
+ Replace(Map<String, Object> node) {
+ Object patternNode = node.get("pattern");
+ String literal = null;
+ if (patternNode instanceof Map) {
+ @SuppressWarnings("unchecked")
+ Map<String, Object> p = (Map<String, Object>) patternNode;
+ literal = Json.string(p, "String", null);
+ if (literal == null && p.containsKey("Regex")) {
+ // A regex Replace would need the crate's regex dialect, and getting it subtly
+ // wrong moves spaces in decoded text. None of the shipped checkpoints uses
+ // one, so this refuses rather than approximating.
+ throw new Json.JsonException(
+ "laya-java implements only a literal String pattern in a Replace"
+ + " decoder; this checkpoint declares a Regex one");
+ }
+ } else if (patternNode instanceof String) {
+ literal = (String) patternNode;
+ }
+ if (literal == null) {
+ throw new Json.JsonException("a Replace decoder needs a String pattern");
+ }
+ this.pattern = literal;
+ this.content = Json.string(node, "content", "");
+ }
+
+ @Override
+ String decode(List<String> tokens) {
+ StringBuilder out = new StringBuilder();
+ for (String token : tokens) {
+ out.append(token.replace(pattern, content));
+ }
+ return out.toString();
+ }
+
+ @Override
+ List<String> transform(List<String> tokens) {
+ // One piece per token, NOT one joined string: ByteFallback runs after this in the
+ // multilingual checkpoint's Sequence and needs the boundaries to find a byte run.
+ List<String> out = new ArrayList<>(tokens.size());
+ for (String token : tokens) {
+ out.add(token.replace(pattern, content));
+ }
+ return out;
+ }
+ }
+
+ /**
+ * {@code ByteFallback}: a run of {@code <0xNN>} tokens is one UTF-8 sequence.
+ *
+ * <p>Reachable on the shipped multilingual checkpoint, which declares
+ * {@code byte_fallback: true} and carries all 255 {@code <0xNN>} tokens. U+F0000 encodes as
+ * four of them -- {@code <0xF3> <0xB0> <0x80> <0x80>} -- so a run has to be gathered and
+ * decoded together; one at a time gives four replacement characters.
+ */
+ private static final class ByteFallback extends TokenDecoder {
+ @Override
+ String decode(List<String> tokens) {
+ StringBuilder out = new StringBuilder();
+ ByteArrayOutputStream run = new ByteArrayOutputStream();
+ for (String token : tokens) {
+ int b = byteToken(token);
+ if (b >= 0) {
+ run.write(b);
+ continue;
+ }
+ flush(run, out);
+ out.append(token);
+ }
+ flush(run, out);
+ return out.toString();
+ }
+
+ @Override
+ List<String> transform(List<String> tokens) {
+ // A run that decodes is one piece; a run that does not becomes one piece PER BYTE.
+ // The crate pushes U+FFFD into its token list once per byte it could not use, and a
+ // later step in the Sequence sees those boundaries. Measured with
+ // `Sequence[ByteFallback, Replace("\ufffd\ufffd" -> "X")]` over the two bytes of a
+ // cut U+F0000: the crate leaves the pair alone because the pieces are separate, and
+ // emitting them as one piece of two characters matched the pattern and gave "X".
+ //
+ // Invisible on the shipped multilingual checkpoint because Fuse is the next step and
+ // re-joins everything, so the strings agree. The override exists to preserve
+ // boundaries, and this was the one shape where it did not.
+ List<String> out = new ArrayList<>();
+ ByteArrayOutputStream run = new ByteArrayOutputStream();
+ for (String token : tokens) {
+ int b = byteToken(token);
+ if (b >= 0) {
+ run.write(b);
+ continue;
+ }
+ flushPieces(run, out);
+ out.add(token);
+ }
+ flushPieces(run, out);
+ return out;
+ }
+
+ private static void flushPieces(ByteArrayOutputStream run, List<String> out) {
+ if (run.size() == 0) {
+ return;
+ }
+ byte[] bytes = run.toByteArray();
+ run.reset();
+ String text = strictUtf8OrNull(bytes);
+ if (text != null) {
+ out.add(text);
+ return;
+ }
+ for (int i = 0; i < bytes.length; i++) {
+ out.add("\ufffd");
+ }
+ }
+
+ private static void flush(ByteArrayOutputStream run, StringBuilder out) {
+ if (run.size() > 0) {
+ out.append(strictOrReplacementPerByte(run.toByteArray()));
+ run.reset();
+ }
+ }
+ }
+
+ /** {@code Fuse}: concatenate the pieces into one string. */
+ private static final class Fuse extends TokenDecoder {
+ @Override
+ String decode(List<String> tokens) {
+ StringBuilder out = new StringBuilder();
+ for (String token : tokens) {
+ out.append(token);
+ }
+ return out.toString();
+ }
+ }
+
+ /** {@code Strip}: remove up to n copies of a character from each end of every token. */
+ private static final class Strip extends TokenDecoder {
+
+ private final char content;
+ private final int start;
+ private final int stop;
+
+ Strip(Map<String, Object> node) {
+ String c = Json.string(node, "content", " ");
+ if (c.length() != 1) {
+ throw new Json.JsonException(
+ "a Strip decoder's content must be one character, got \"" + c + "\"");
+ }
+ this.content = c.charAt(0);
+ this.start = Json.integer(node, "start", 0);
+ this.stop = Json.integer(node, "stop", 0);
+ }
+
+ @Override
+ String decode(List<String> tokens) {
+ StringBuilder out = new StringBuilder();
+ for (String token : tokens) {
+ out.append(apply(token));
+ }
+ return out.toString();
+ }
+
+ @Override
+ List<String> transform(List<String> tokens) {
+ List<String> out = new ArrayList<>(tokens.size());
+ for (String token : tokens) {
+ out.add(apply(token));
+ }
+ return out;
+ }
+
+ String apply(String token) {
+ int from = 0;
+ int to = token.length();
+ for (int i = 0; i < start && from < to && token.charAt(from) == content; i++) {
+ from++;
+ }
+ for (int i = 0; i < stop && to > from && token.charAt(to - 1) == content; i++) {
+ to--;
+ }
+ return token.substring(from, to);
+ }
+ }
+
+ /**
+ * {@code Sequence}: each step transforms the token LIST, not the joined string.
+ *
+ * <p>That distinction is the whole behaviour of the multilingual checkpoint's
+ * {@code Sequence[Replace, ByteFallback, Fuse]}. Replace rewrites each token, ByteFallback
+ * needs to see token boundaries to find a {@code <0xNN>} run, and only Fuse joins. Running the
+ * steps over an already-joined string would make ByteFallback unable to tell a literal
+ * {@code "<0x41>"} in the text from a byte token.
+ */
+ private static final class Sequence extends TokenDecoder {
+
+ private final List<TokenDecoder> steps;
+
+ Sequence(List<TokenDecoder> steps) {
+ this.steps = List.copyOf(steps);
+ }
+
+ @Override
+ String decode(List<String> tokens) {
+ return String.join("", transform(tokens));
+ }
+
+ @Override
+ List<String> transform(List<String> tokens) {
+ // The chained list, NOT one joined piece. Without this override a Sequence inherits
+ // the base default, which re-joins -- so a Sequence NESTED in a Sequence handed one
+ // string to the outer sequence's next step where the crate hands it N pieces. The
+ // crate's `decode_chain` chains `decode_chain` through its steps and returns a list.
+ //
+ // Measured with `Sequence[Sequence[Replace, ByteFallback], Strip(" ", start=1)]`: the
+ // crate gives "Helloworld", because Strip sees two pieces and takes the leading space
+ // off each; re-joining gave "Hello world". Neither shipped checkpoint nests deeper
+ // than one level, which is why nothing caught it and why it has its own test.
+ List<String> current = new ArrayList<>(tokens);
+ for (TokenDecoder step : steps) {
+ current = step.transform(current);
+ }
+ return current;
+ }
+ }
+
+ /**
+ * This step's effect on a token list, which is what a {@code Sequence} composes.
+ *
+ * <p>The default is "whatever this decoder produces, as one piece". The steps that have to
+ * preserve boundaries for a later step override it.
+ */
+ List<String> transform(List<String> tokens) {
+ return List.of(decode(tokens));
+ }
+
+ // ------------------------------------------------------------------ helpers
+
+ /** The byte a {@code <0xNN>} token stands for, or -1 when the token is not one. */
+ private static int byteToken(String token) {
+ if (token == null || token.length() != 6
+ || token.charAt(0) != '<' || token.charAt(1) != '0' || token.charAt(2) != 'x'
+ || token.charAt(5) != '>') {
+ return -1;
+ }
+ return parseByte(token.charAt(3), token.charAt(4));
+ }
+
+ /**
+ * The two characters between {@code <0x} and {@code >} as a byte, following Rust's
+ * {@code u8::from_str_radix(.., 16)} -- which is the function the crate calls.
+ *
+ * <p>Case-INSENSITIVE, and a leading {@code +} is accepted. Both were measured, after this
+ * rejected lowercase on the stated grounds that "the crate writes these uppercase, and
+ * accepting both would make a literal {@code <0xff>} in ordinary text decode as a byte". That
+ * second clause describes what the crate actually does, rather than a hazard it avoids: asked
+ * to decode a vocabulary whose only token is {@code <0xff>}, the crate returns U+FFFD -- it
+ * read the byte -- and the same for {@code <0xfF>}. {@code <0x+5>} is byte 0x05.
+ *
+ * <p>{@code <0x 5>} and {@code <0XFF>} stay literal, because a space is not a digit and the
+ * {@code <0x} prefix is matched case-sensitively. Both measured too, so the boundary is
+ * recorded rather than assumed in the other direction this time.
+ *
+ * <p>No shipped vocabulary contains a lowercase byte token -- all 255 in the multilingual
+ * checkpoint are uppercase -- so this was not a live bug. It was a wrong rule with a
+ * confident comment, which is worse to leave in place than a wrong rule without one.
+ */
+ private static int parseByte(char high, char low) {
+ if (high == '+') {
+ return hex(low);
+ }
+ int h = hex(high);
+ int l = hex(low);
+ return h < 0 || l < 0 ? -1 : (h << 4) | l;
+ }
+
+ private static int hex(char c) {
+ if (c >= '0' && c <= '9') {
+ return c - '0';
+ }
+ if (c >= 'A' && c <= 'F') {
+ return c - 'A' + 10;
+ }
+ if (c >= 'a' && c <= 'f') {
+ return c - 'a' + 10;
+ }
+ return -1;
+ }
+
+ /**
+ * UTF-8 with replacement, as the crate's lossy decode does.
+ *
+ * <p>Not {@code new String(bytes, UTF_8)}, which is also lossy but has differed from Rust's
+ * {@code from_utf8_lossy} on how many replacement characters a truncated sequence produces.
+ * A window boundary cuts multi-byte characters routinely, so this is a path that runs, not an
+ * edge case: the decoder is configured explicitly to REPLACE rather than report, and one
+ * replacement per malformed input sequence is what both produce.
+ */
+ private static String lossyUtf8(byte[] bytes) {
+ CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder()
+ .onMalformedInput(CodingErrorAction.REPLACE)
+ .onUnmappableCharacter(CodingErrorAction.REPLACE);
+ try {
+ return decoder.decode(ByteBuffer.wrap(bytes)).toString();
+ } catch (CharacterCodingException impossible) {
+ throw new IllegalStateException("a REPLACE decoder cannot report an error", impossible);
+ }
+ }
+
+ /**
+ * {@code ByteFallback}'s rule, which is NOT lossy decoding: strict UTF-8, and on failure one
+ * U+FFFD per BYTE of the run.
+ *
+ * <p>Measured, after assuming otherwise cost 629 of 201,279 window slices. A window boundary
+ * cuts U+F0000 after two of its four bytes, leaving {@code F3 B0}; lossy decoding calls that
+ * one maximal subpart and emits ONE replacement character, and the crate emits TWO -- one for
+ * each byte it could not use. The two rules agree only when the run is valid, which is every
+ * case that does not straddle a boundary, which is why this looked right until the slices were
+ * compared.
+ *
+ * <p>The sibling {@code ByteLevel} decoder really does use lossy decoding, so the two paths in
+ * this file deliberately differ. That is the crate's shape, not an oversight.
+ */
+ private static String strictOrReplacementPerByte(byte[] bytes) {
+ String text = strictUtf8OrNull(bytes);
+ return text != null ? text : "\ufffd".repeat(bytes.length);
+ }
+
+ /** The run as strict UTF-8, or null when it is not valid UTF-8. */
+ private static String strictUtf8OrNull(byte[] bytes) {
+ CharsetDecoder strict = StandardCharsets.UTF_8.newDecoder()
+ .onMalformedInput(CodingErrorAction.REPORT)
+ .onUnmappableCharacter(CodingErrorAction.REPORT);
+ try {
+ return strict.decode(ByteBuffer.wrap(bytes)).toString();
+ } catch (CharacterCodingException invalid) {
+ return null;
+ }
+ }
+}
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/Tokenizer.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/Tokenizer.java
index 4015fe35..548743c5 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/Tokenizer.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/Tokenizer.java
@@ -42,6 +42,9 @@ public final class Tokenizer {
private final Map<String, Integer> specials;
private final Map<String, Integer> roles;
private final String maskToken;
+ private final TokenDecoder decoder;
+ /** Ids the reference's decode drops unless asked not to; see {@link #decode(int[])}. */
+ private final java.util.Set<Integer> specialIds;
/** The roles a checkpoint names in {@code tokenizer_config.json}, in sequence-building order. */
private static final String[] ROLE_KEYS =
@@ -136,6 +139,7 @@ public final class Tokenizer {
this.normalizer = Normalizer.from(objectOrNull(document, "normalizer"));
this.preTokenizer = PreTokenizer.from(objectOrNull(document, "pre_tokenizer"));
+ this.decoder = TokenDecoder.from(objectOrNull(document, "decoder"));
// --- the id map: model.vocab UNION added_tokens, with added_tokens authoritative.
//
@@ -153,6 +157,17 @@ public final class Tokenizer {
}
List<AddedToken> added = readAddedTokens(document);
+ // The ids a decode drops by default. Taken from each added token's own `special` flag
+ // rather than from the laya config's four roles: the english checkpoint declares 116
+ // added tokens and the config names only cls/sep/mask/pad, so the roles would leave
+ // every other control token in the decoded text.
+ java.util.Set<Integer> special = new java.util.LinkedHashSet<>();
+ for (AddedToken token : added) {
+ if (token.special()) {
+ special.add(token.id());
+ }
+ }
+ this.specialIds = java.util.Set.copyOf(special);
for (AddedToken token : added) {
Integer existing = vocab.get(token.content());
if (existing != null && existing != token.id()) {
@@ -340,6 +355,50 @@ public final class Tokenizer {
// ------------------------------------------------------------------ lookups
/** The number of distinct ids this tokenizer can emit, counting added tokens. */
+ /**
+ * These ids back into text, dropping special tokens -- the reference's default.
+ *
+ * <p>Used by {@code predictLong}, which splits a long state into overlapping token windows and
+ * hands each one back as text so it is re-tokenized as an ordinary state.
+ *
+ * <p><b>This does not round-trip, and is not meant to.</b> On the english checkpoint it
+ * happens to; on the multilingual one {@code "Hello world"} decodes to {@code " Hello world"},
+ * because Metaspace prepends U+2581 when encoding and the decoder turns every U+2581 back
+ * into a space. That is the reference's behaviour and the port reproduces it exactly -- a
+ * decode that trimmed the space would re-tokenize each window differently and move every
+ * answer downstream.
+ */
+ public String decode(int[] ids) {
+ return decode(ids, true);
+ }
+
+ /**
+ * These ids back into text.
+ *
+ * @param ids the token ids, in order
+ * @param skipSpecialTokens drop ids flagged {@code special} in {@code added_tokens}, which is
+ * what the reference does unless told otherwise
+ */
+ public String decode(int[] ids, boolean skipSpecialTokens) {
+ if (ids == null) {
+ throw new IllegalArgumentException("ids must not be null");
+ }
+ List<String> pieces = new ArrayList<>(ids.length);
+ for (int id : ids) {
+ if (skipSpecialTokens && specialIds.contains(id)) {
+ continue;
+ }
+ String token = idToToken(id);
+ if (token == null) {
+ // An id outside the vocabulary. The crate skips it; failing here would turn a
+ // caller's off-by-one into an exception from a method whose job is to be lossy.
+ continue;
+ }
+ pieces.add(token);
+ }
+ return decoder.decode(pieces);
+ }
+
public int vocabSize() {
return ids.size();
}
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/tokenizer/TokenDecoderTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/tokenizer/TokenDecoderTest.java
new file mode 100644
index 00000000..815a927a
--- /dev/null
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/tokenizer/TokenDecoderTest.java
@@ -0,0 +1,448 @@
+package com.convaiinnovations.laya.tokenizer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.DynamicTest.dynamicTest;
+
+import com.convaiinnovations.laya.Fixtures;
+import com.convaiinnovations.laya.json.Json;
+import com.convaiinnovations.laya.json.PythonJson;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.DynamicTest;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestFactory;
+
+/**
+ * {@code Tokenizer.decode}, against expectations recorded from the reference tokenizer.
+ *
+ * <p>Decoding has no Python reference to port -- it lives in the Rust {@code tokenizers} crate --
+ * so {@code fixtures/decode_text.json} is the only specification this port has, and the three
+ * rules it pins were each wrong on the first attempt. The counts in the comments below are what
+ * each wrong guess cost against the reference, so a later simplification has a number to argue
+ * with rather than a preference.
+ *
+ * <p>The SLICE cases carry the weight. {@code predictLong} cuts a tokenized state into windows at
+ * arbitrary offsets and decodes each one, so a boundary lands inside a multi-byte character
+ * routinely -- and every whole-text decode agreed while 629 of 201,279 slices did not. A fixture
+ * of whole texts would have shipped the bug.
+ */
+final class TokenDecoderTest {
+
+ @SuppressWarnings("unchecked")
+ private static Map<String, Object> family() {
+ return (Map<String, Object>) (Map<?, ?>) Fixtures.load("decode_text.json");
+ }
+
+ @SuppressWarnings("unchecked")
+ private static Map<String, Object> forCheckpoint(String name) {
+ Map<String, Object> by = (Map<String, Object>) family().get("by");
+ Map<String, Object> entry = (Map<String, Object>) by.get(name);
+ Assumptions.assumeTrue(entry != null && !entry.containsKey("skipped"),
+ "the " + name + " checkpoint was not present when the fixture was recorded");
+ return entry;
+ }
+
+ private static Tokenizer open(String name) {
+ Path root = Fixtures.checkpoint(name);
+ Assumptions.assumeTrue(root != null, "set LAYA_CHECKPOINTS to run the decode parity tests");
+ try {
+ return Tokenizer.fromFile(root.resolve("tokenizer").resolve("tokenizer.json"));
+ } catch (java.io.IOException problem) {
+ throw new IllegalStateException("cannot open the " + name + " tokenizer", problem);
+ }
+ }
+
+ @TestFactory
+ @DisplayName("every recorded decode matches, whole and sliced, on both checkpoints")
+ @SuppressWarnings("unchecked")
+ List<DynamicTest> recordedDecodesMatch() {
+ List<DynamicTest> tests = new ArrayList<>();
+ int sliceCases = 0;
+ for (String name : List.of("english", "multilingual")) {
+ Map<String, Object> entry = forCheckpoint(name);
+ List<Object> cases = (List<Object>) entry.get("cases");
+ assertTrue(cases.size() >= 23,
+ () -> "the decode corpus lost cases: " + cases.size() + " of at least 23");
+ Tokenizer tok = open(name);
+ for (Object raw : cases) {
+ Map<String, Object> row = (Map<String, Object>) raw;
+ String text = (String) row.get("text");
+ List<Object> wantIds = (List<Object>) row.get("ids");
+ String label = name + " " + describe(text);
+
+ tests.add(dynamicTest(label + " ids", () -> {
+ int[] ids = tok.encode(text);
+ assertEquals(wantIds.size(), ids.length, "id count for " + describe(text));
+ for (int i = 0; i < ids.length; i++) {
+ assertEquals(((Number) wantIds.get(i)).intValue(), ids[i],
+ "id " + i + " for " + describe(text));
+ }
+ }));
+
+ tests.add(dynamicTest(label + " whole", () ->
+ assertEquals(row.get("decoded"), tok.decode(ids(wantIds)),
+ "decode diverged on " + describe(text))));
+
+ List<Object> slices = (List<Object>) row.get("slices");
+ sliceCases += slices.size();
+ if (slices.isEmpty()) {
+ // No slices means no ids, which is true of exactly one corpus text. Asserting
+ // THAT is worth a test; registering a loop over nothing is not -- it passed
+ // unconditionally and still counted toward the suite's floor.
+ tests.add(dynamicTest(label + " has no ids to slice", () -> {
+ assertTrue(text.isEmpty(),
+ () -> "only the empty text should tokenize to nothing, but "
+ + describe(text) + " did");
+ assertEquals(0, ids(wantIds).length, "and it must have no ids");
+ }));
+ continue;
+ }
+ tests.add(dynamicTest(label + " slices (" + slices.size() + ")", () -> {
+ int[] ids = ids(wantIds);
+ for (Object s : slices) {
+ Map<String, Object> slice = (Map<String, Object>) s;
+ int start = ((Number) slice.get("start")).intValue();
+ int size = ((Number) slice.get("size")).intValue();
+ int end = Math.min(start + size, ids.length);
+ int[] chunk = new int[end - start];
+ System.arraycopy(ids, start, chunk, 0, chunk.length);
+ assertEquals(slice.get("decoded"), tok.decode(chunk),
+ () -> String.format("slice [%d,%d) of %s", start, end,
+ describe(text)));
+ }
+ }));
+
+ Map<String, Object> specials = (Map<String, Object>) row.get("with_specials");
+ tests.add(dynamicTest(label + " specials", () -> {
+ int[] ids = ids((List<Object>) specials.get("ids"));
+ assertEquals(specials.get("skipping_specials"), tok.decode(ids),
+ "decode must skip special tokens by default");
+ assertEquals(specials.get("kept"), tok.decode(ids, false),
+ "decode(ids, false) must keep them");
+ }));
+ }
+ }
+ // A floor on the slices specifically, because they are the cases that caught the
+ // ByteFallback rule and a corpus that lost them would still look healthy.
+ final int total = sliceCases;
+ assertTrue(total >= 400,
+ () -> "the slice cases are the point of this family and there are only " + total);
+ return tests;
+ }
+
+ @Test
+ @DisplayName("the fixture was recorded against the decoders this port implements")
+ @SuppressWarnings("unchecked")
+ void fixtureNamesItsDecoders() {
+ // Without this, a fixture regenerated against a checkpoint whose decoder changed would be
+ // asserted against as though nothing had moved -- the same guard tokenizer_ids applies.
+ Map<String, Object> en = (Map<String, Object>) forCheckpoint("english").get("decoder");
+ assertEquals("ByteLevel", en.get("type"), "the english decoder changed");
+
+ Map<String, Object> ml = (Map<String, Object>) forCheckpoint("multilingual").get("decoder");
+ assertEquals("Sequence", ml.get("type"), "the multilingual decoder changed");
+ List<Object> steps = (List<Object>) ml.get("decoders");
+ List<String> types = new ArrayList<>();
+ for (Object step : steps) {
+ types.add((String) ((Map<String, Object>) step).get("type"));
+ }
+ assertEquals(List.of("Replace", "ByteFallback", "Fuse"), types,
+ "the multilingual decoder chain changed");
+ }
+
+ // ------------------------------------------------- the three rules, each stated on its own
+
+ @Test
+ @DisplayName("a character outside the stand-in alphabet is kept, not dropped")
+ void unmappedCharactersSurvive() {
+ // Dropping them was the first guess and it got 414 of 5,312 texts wrong. This vocabulary
+ // mixes byte-level tokens with 23 added tokens whose content is LITERAL whitespace -- id
+ // 50275 is three real spaces, and 0x20 is lifted to U+0120 so a real space is not in the
+ // alphabet at all.
+ Tokenizer tok = open("english");
+ assertEquals(" ", tok.decode(new int[] {50275}),
+ "the literal-whitespace added token must survive decoding");
+ assertEquals(" ", tok.decode(tok.encode(" ")), "and a byte-level space still works");
+ assertEquals(" ", tok.decode(tok.encode(" ")));
+ }
+
+ /** Loads a synthetic spec recorded in the fixture, as the crate was asked about it. */
+ private static Tokenizer fromSpec(Object spec) throws IOException {
+ Path dir = Files.createTempDirectory("laya-synthetic");
+ Path file = dir.resolve("tokenizer.json");
+ Files.write(file, PythonJson.dumps(spec).getBytes(StandardCharsets.UTF_8));
+ try {
+ return Tokenizer.fromFile(file);
+ } finally {
+ Files.deleteIfExists(file);
+ Files.deleteIfExists(dir);
+ }
+ }
+
+ @TestFactory
+ @DisplayName("the decoder shapes no shipped checkpoint declares")
+ @SuppressWarnings("unchecked")
+ List<DynamicTest> syntheticDecoderShapes() {
+ // `TokenDecoder` implements six decoders, and the two shipped checkpoints between them use
+ // four: english is ByteLevel, multilingual is Sequence[Replace, ByteFallback, Fuse]. So
+ // the recorded corpus cannot reach Strip at all, cannot reach the no-decoder fallback, and
+ // cannot tell a nested Sequence or a split byte run from a joined one. An adversarial
+ // review gutted Strip entirely -- `apply` replaced by `return "MUTANT"` -- and all 191
+ // cases passed.
+ //
+ // Four real divergences were hiding in that gap, each confirmed against the crate before
+ // it was fixed: the no-decoder fallback joins with a SPACE (this joined with nothing), the
+ // hex parse is case-insensitive and takes a leading "+" (this rejected both), a nested
+ // Sequence keeps its pieces (this re-joined them), and a failed byte run is one piece PER
+ // BYTE (this made it one piece of N). Every expectation below comes from the crate.
+ Map<String, Object> shapes = (Map<String, Object>) family().get("synthetic_decoders");
+ Assumptions.assumeTrue(shapes != null && !shapes.containsKey("skipped"),
+ "the synthetic decoder shapes were not recorded");
+
+ List<DynamicTest> tests = new ArrayList<>();
+ for (Map.Entry<String, Object> entry : shapes.entrySet()) {
+ String name = entry.getKey();
+ Map<String, Object> shape = (Map<String, Object>) entry.getValue();
+ tests.add(dynamicTest(name, () -> {
+ Tokenizer tok = fromSpec(shape.get("spec"));
+ List<Object> cases = (List<Object>) shape.get("cases");
+ assertFalse(cases.isEmpty(), () -> name + " recorded no cases");
+ for (Object raw : cases) {
+ Map<String, Object> one = (Map<String, Object>) raw;
+ int[] ids = ids((List<Object>) one.get("ids"));
+ assertEquals(one.get("decoded"), tok.decode(ids, false),
+ () -> name + " diverged on ids " + one.get("ids"));
+ }
+
+ // Where the crate PANICS there is no reference answer, so what is asserted is
+ // that this port stays total: a Rust panic crossing the FFI boundary is not
+ // behaviour worth reproducing, and silently returning something is not either.
+ List<Object> panics = (List<Object>) shape.get("crate_panics");
+ if (panics != null) {
+ for (Object raw : panics) {
+ int[] ids = ids((List<Object>) raw);
+ assertEquals("", tok.decode(ids, false),
+ () -> name + ": the crate panics on " + raw
+ + " and this must return empty text, not throw");
+ }
+ }
+ }));
+ }
+ assertTrue(tests.size() >= 5,
+ () -> "five shapes were recorded and only " + tests.size() + " are here");
+ return tests;
+ }
+
+ @Test
+ @DisplayName("an unsupported or malformed decoder is refused, not guessed at")
+ void refusalsFire() {
+ // Every one of these threw nowhere in the suite before this test: an adversarial review
+ // replaced all four refusal paths at once -- the unknown-type throw by `return new
+ // Join()`, the Regex-pattern throw and the Strip content-length throw by `if (false)` --
+ // and all 191 cases passed. They are the guards whose whole stated purpose is to stop a
+ // future checkpoint being silently mis-decoded, so "nothing proves they fire" is the one
+ // thing they cannot afford. None of this needs a checkpoint, so it runs in every lane.
+ assertThrows(Json.JsonException.class,
+ () -> TokenDecoder.from(Map.of("type", "Metaspace")),
+ "a decoder this port does not implement must be refused, not approximated");
+ assertThrows(Json.JsonException.class,
+ () -> TokenDecoder.from(new LinkedHashMap<>()),
+ "a decoder node with no type must be refused");
+ // The MESSAGE, not just the type. Disabling the Regex check leaves `literal` null,
+ // so the NEXT guard throws Json.JsonException anyway -- and a mutant that removed
+ // the Regex refusal outright passed this assertion while it checked only the
+ // exception class. The two refusals say different things ("this port does not do
+ // regex" versus "your pattern is missing") and have to be told apart.
+ Json.JsonException regex = assertThrows(Json.JsonException.class,
+ () -> TokenDecoder.from(Map.of("type", "Replace",
+ "pattern", Map.of("Regex", "\\s+"), "content", " ")),
+ "a Regex Replace needs the crate's regex dialect and must be refused");
+ assertTrue(regex.getMessage().contains("Regex"),
+ () -> "the refusal must name the reason, got: " + regex.getMessage());
+ assertThrows(Json.JsonException.class,
+ () -> TokenDecoder.from(Map.of("type", "Replace", "content", " ")),
+ "a Replace with no pattern must be refused");
+ assertThrows(Json.JsonException.class,
+ () -> TokenDecoder.from(Map.of("type", "Sequence")),
+ "a Sequence with no decoders array must be refused");
+ assertThrows(Json.JsonException.class,
+ () -> TokenDecoder.from(Map.of("type", "Sequence",
+ "decoders", List.of("ByteLevel"))),
+ "a Sequence step that is not an object must be refused");
+ assertThrows(Json.JsonException.class,
+ () -> TokenDecoder.from(Map.of("type", "Strip", "content", "ab")),
+ "a Strip whose content is not one character must be refused");
+
+ // And the one branch that is deliberately permissive. Its separator is a space, which is
+ // what the crate falls back to; the recorded `no_decoder` shape is the parity check and
+ // this states that `from(null)` reaches the same code.
+ assertEquals("ab cd", TokenDecoder.from(null).decode(List.of("ab", "cd")),
+ "no decoder node means join with a space, as the crate does");
+
+ assertThrows(IllegalArgumentException.class, () -> PreTokenizer.charForByte(256),
+ "a byte is 0 to 255, and an out-of-range index must say so");
+ assertThrows(IllegalArgumentException.class, () -> PreTokenizer.charForByte(-1),
+ "including below the range");
+ }
+
+ @Test
+ @DisplayName("the ByteLevel fallback is scoped to a token, not to a character")
+ @SuppressWarnings("unchecked")
+ void byteLevelFallsBackPerToken() throws IOException {
+ // Neither real checkpoint can tell these rules apart. Their byte-level tokens are entirely
+ // stand-in alphabet and their added tokens are entirely literal whitespace, so a mutant
+ // that swapped per-token for per-character passed all 190 recorded cases -- the rule was
+ // correct and unpinned, which is the same as unverified.
+ //
+ // The rules differ only for a token that MIXES the two, so the generator builds a
+ // vocabulary containing one and records what the crate does. "Ġ " separates them:
+ // U+0120 is the stand-in for byte 0x20 and the literal space is not in the alphabet.
+ // per TOKEN -> every character's own UTF-8 bytes: C4 A0 20 -> "Ġ "
+ // per CHARACTER -> the mapped one's byte: 20 20 -> " "
+ Map<String, Object> scope = (Map<String, Object>) family().get("mixed_token");
+ Assumptions.assumeTrue(scope != null && !scope.containsKey("skipped"),
+ "the mixed-token scope was not recorded");
+ // Documentation, not a gate: the generator RAISES when the probed scope is not "token",
+ // so no other value can reach the fixture and this line can only fail on a hand-edited
+ // one. The real guard is that `raise` -- verified by changing the separating token so the
+ // crate answered differently, which failed the regeneration step rather than this test.
+ // The parity assertions below are what pin the behaviour.
+ assertEquals("token", scope.get("scope"),
+ "the crate's fallback scope changed and this port assumes per token");
+
+ // The spec travels in the fixture, so this loads the same tokenizer the crate was asked
+ // about instead of a second hand-written copy of it.
+ Path dir = Files.createTempDirectory("laya-mixed-token");
+ Path file = dir.resolve("tokenizer.json");
+ try {
+ Files.write(file, PythonJson.dumps(scope.get("spec"))
+ .getBytes(StandardCharsets.UTF_8));
+ Tokenizer tok = Tokenizer.fromFile(file);
+ for (Object raw : (List<Object>) scope.get("cases")) {
+ Map<String, Object> one = (Map<String, Object>) raw;
+ int[] ids = ids((List<Object>) one.get("ids"));
+ assertEquals(one.get("decoded"), tok.decode(ids, false),
+ (String) one.get("why"));
+ }
+ } finally {
+ Files.deleteIfExists(file);
+ Files.deleteIfExists(dir);
+ }
+ }
+
+ @Test
+ @DisplayName("ByteFallback emits one replacement character per byte, not per subpart")
+ void byteFallbackReplacesPerByte() {
+ // The rule that only the slices could see. U+F0000 is four ByteFallback tokens; cutting
+ // after two leaves F3 B0, which lossy decoding calls one maximal subpart and reports as
+ // ONE replacement character. The crate reports two -- one per byte it could not use.
+ Tokenizer tok = open("multilingual");
+ int[] ids = tok.encode("󰀀"); // U+F0000
+ assertTrue(ids.length >= 3,
+ () -> "expected a Metaspace marker plus a byte run, got " + ids.length + " ids");
+ int[] firstThree = {ids[0], ids[1], ids[2]};
+ assertEquals(" ��", tok.decode(firstThree),
+ "two bytes of a four-byte character must give two replacement characters");
+ // and the whole thing still decodes to the character itself
+ assertEquals(" 󰀀", tok.decode(ids),
+ "the complete byte run must decode to the character");
+ }
+
+ @Test
+ @DisplayName("decoding is lossy on the multilingual checkpoint, and that is the contract")
+ void metaspaceAddsALeadingSpace() {
+ // A port that "fixed" this would re-tokenize every predictLong window differently from the
+ // reference and move every answer downstream. The fixture records the loss; this states it
+ // where a reader will see it.
+ Tokenizer ml = open("multilingual");
+ assertEquals(" Hello world", ml.decode(ml.encode("Hello world")),
+ "Metaspace prepends U+2581 and the decoder turns it into a space");
+ Tokenizer en = open("english");
+ assertEquals("Hello world", en.decode(en.encode("Hello world")),
+ "the english checkpoint happens to round-trip");
+ }
+
+ @Test
+ @DisplayName("an unknown id and a null argument are handled, not thrown at")
+ void edges() {
+ Tokenizer tok = open("english");
+ assertEquals("", tok.decode(new int[0]), "no ids is empty text");
+ assertEquals("", tok.decode(new int[] {Integer.MAX_VALUE}),
+ "an id outside the vocabulary is skipped, as the crate skips it");
+ // A NEGATIVE id is not crate parity -- the crate's ids are u32, so the Python binding
+ // raises OverflowError rather than skipping it. Treating it like any other id outside the
+ // vocabulary is this port's own choice, made because `int[]` can hold one and a caller's
+ // off-by-one should not become an exception from a method whose job is to be lossy. Stated
+ // here rather than filed under "as the crate skips it", which is what it used to say.
+ assertEquals("a", tok.decode(new int[] {Integer.MAX_VALUE, tok.encode("a")[0], -1}),
+ "a negative id is skipped like any other unknown one, by this port's choice");
+ assertTrue(assertThrowsIllegalArgument(() -> tok.decode(null)),
+ "a null id array is a caller error, not a silent empty string");
+ }
+
+ private static boolean assertThrowsIllegalArgument(Runnable work) {
+ try {
+ work.run();
+ return false;
+ } catch (IllegalArgumentException expected) {
+ return true;
+ }
+ }
+
+ @Test
+ @DisplayName("the byte table and its inverse agree on all 256 values")
+ void byteTableRoundTrips() {
+ // What this catches, precisely: a table that is not INJECTIVE. If two bytes were given
+ // the same stand-in character, the inverse would lose one of them and the round trip
+ // would fail here. Measured, because the comment used to claim more than that: a mutant
+ // that SWAPS two entries (0x41 and 0x42) is still injective and passes this test and all
+ // 191 recorded cases. The table's actual values are pinned on the encode side instead,
+ // by the recorded tokenizer fixtures -- the same swap fails 10 cases in
+ // SequenceParityTest -- and none of the 23 texts in this family contains an "A" or a "B",
+ // which is why the decode side could not see it.
+ for (int value = 0; value < 256; value++) {
+ final int b = value;
+ char stand = PreTokenizer.charForByte(b);
+ assertEquals(b, PreTokenizer.byteForChar(stand),
+ () -> String.format("byte 0x%02X does not survive the round trip", b));
+ }
+ assertFalse(PreTokenizer.byteForChar(' ') == 0x20,
+ "a literal space must NOT be in the alphabet: 0x20 is lifted to U+0120, and the "
+ + "whole literal-token rule depends on it being absent");
+ }
+
+ private static int[] ids(List<Object> recorded) {
+ int[] out = new int[recorded.size()];
+ for (int i = 0; i < out.length; i++) {
+ out[i] = ((Number) recorded.get(i)).intValue();
+ }
+ return out;
+ }
+
+ /** A readable label that cannot contain a line break or a lone surrogate. */
+ private static String describe(String text) {
+ if (text.isEmpty()) {
+ return "<empty>";
+ }
+ StringBuilder out = new StringBuilder();
+ text.codePoints().limit(18).forEach(cp -> {
+ if (cp < 0x20 || cp > 0x7E) {
+ out.append(String.format("U+%04X", cp));
+ } else {
+ out.appendCodePoint(cp);
+ }
+ });
+ return out.toString();
+ }
+}
diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.py
index e0cced77..dc247d06 100644
--- a/laya-java/scripts/gen_fixtures.py
+++ b/laya-java/scripts/gen_fixtures.py
@@ -2088,6 +2088,349 @@ def email_clean():
}
+def decode_text():
+ """What `Tokenizer.decode` must return, per checkpoint, recorded from the reference.
+
+ Decoding has no Python reference to port -- it lives in the Rust `tokenizers` crate -- so this
+ family is the only specification the Java side has, and the three rules it pins were each
+ wrong on the first attempt:
+
+ * a character outside the ByteLevel stand-in alphabet is KEPT as its own UTF-8 bytes, not
+ dropped. The english vocabulary mixes byte-level tokens like "\\u0120world" with 23 added
+ tokens whose content is literal whitespace -- id 50275 is three real spaces -- and dropping
+ unmapped characters got 414 of 5,312 texts wrong.
+ * the ByteLevel fallback is per TOKEN, not per character: one unmapped character and the
+ whole token uses its raw bytes.
+ * ByteFallback does NOT decode lossily. It tries strict UTF-8 and, on failure, emits one
+ U+FFFD per BYTE of the run. Lossy decoding emits one per maximal subpart, which differs
+ the moment a window boundary cuts a multi-byte character -- 629 of 201,279 window slices.
+
+ The SLICES are the point of this family. `predictLong` cuts a tokenized state into windows at
+ arbitrary offsets and decodes each one, so a boundary lands inside a multi-byte character
+ routinely. Every whole-text decode agreed while 629 slices did not, so a fixture of whole
+ texts would have shipped the bug.
+ """
+ import os
+
+ rig = os.environ.get("LAYA_FIXTURE_CHECKPOINTS", DEFAULT_CHECKPOINT_ROOT)
+
+ # Chosen for where a decoder breaks, not for coverage of prose.
+ texts = [
+ "Hello world",
+ "",
+ " ",
+ " ", # an added token whose content is literal spaces
+ "\t",
+ "a\tb\nc", # multilingual inserts a space after the newline
+ "café résumé",
+ "注文をキャンセル", # 3-byte characters, split often
+ "مرحبا",
+ "क्‌ष", # ZWNJ, which is Cf and not a mark
+ "\U0001f600", # 4-byte, the shortest ByteFallback run
+ "emoji \U0001f600 here",
+ "\U0001f1ec\U0001f1e7", # a flag: two regional indicators
+ "▁literal marker", # the Metaspace marker as ORDINARY TEXT
+ "pre▁fix",
+ " nbsp figure",
+ "<0x41>", # a byte token spelled out in ordinary text
+ "a<0xFF>b",
+ "",
+ "", # private use
+ "\U000f0000", # unassigned: four ByteFallback tokens
+ "I cannot log in and need a password reset.",
+ "Esta mensagem e confidencial.",
+ ]
+
+ out = {"texts": texts, "by": {}}
+ for name in ("english", "multilingual"):
+ root = os.path.join(rig, name)
+ tokenizer_json = os.path.join(root, "tokenizer", "tokenizer.json")
+ if not os.path.isfile(tokenizer_json):
+ out["by"][name] = {"skipped": "checkpoint not present at %s" % root}
+ continue
+ from tokenizers import Tokenizer
+
+ tok = Tokenizer.from_file(tokenizer_json)
+ cases = []
+ for text in texts:
+ ids = tok.encode(text, add_special_tokens=False).ids
+ # Every cut point at several widths: this is the predictLong shape, and the only
+ # place the ByteFallback rule is observable.
+ slices = []
+ for size in (1, 2, 3, 5):
+ for start in range(0, max(1, len(ids)), max(1, size)):
+ chunk = ids[start:start + size]
+ if chunk:
+ slices.append({"start": start, "size": size,
+ "decoded": tok.decode(chunk)})
+ with_specials = tok.encode(text, add_special_tokens=True).ids
+ cases.append({
+ "text": text,
+ "ids": list(ids),
+ "decoded": tok.decode(ids),
+ "slices": slices,
+ "with_specials": {
+ "ids": list(with_specials),
+ # NOT "skipped": a lone `skipped` key is this script's marker for a family it
+ # could not regenerate, and naming a data field that way once made this very
+ # family look unregenerable.
+ "skipping_specials": tok.decode(with_specials),
+ "kept": tok.decode(with_specials, skip_special_tokens=False),
+ },
+ })
+ out["by"][name] = {
+ # What this tokenizer IS, so a fixture recorded against a different checkpoint cannot
+ # be mistaken for a matching one -- the same guard tokenizer_ids applies.
+ "decoder": _decoder_shape(tokenizer_json),
+ "cases": cases,
+ }
+ out["mixed_token"] = _mixed_token_scope()
+ out["synthetic_decoders"] = _synthetic_decoders()
+ return out
+
+
+def _synthetic_decoders():
+ """What the crate does with the decoder shapes no shipped checkpoint declares.
+
+ `TokenDecoder` implements six decoders because a checkpoint may declare any of them, but the
+ two shipped ones between them use four: english is `ByteLevel`, multilingual is
+ `Sequence[Replace, ByteFallback, Fuse]`. So `Strip` shipped with no coverage at all -- a 44
+ line decoder with a documented start/stop semantic that an adversarial mutant could gut while
+ all 191 recorded cases passed -- and the `hex` helper's refusal of lowercase byte tokens had
+ nothing exercising it either.
+
+ Rather than assert what the implementation was written to do, each shape below is a real
+ tokenizer the crate is asked to decode, exactly as `mixed_token` is. `byte_fallback` is set on
+ the model so the port's loader guard, which refuses a vocabulary that could drop a character
+ without trace, is satisfied by something other than a full byte-level alphabet.
+ """
+ import io
+ import os
+ import tempfile
+
+ try:
+ from tokenizers import Tokenizer
+ except ImportError:
+ return {"skipped": "the tokenizers package is not installed"}
+
+ def spec_for(decoder, vocab):
+ spec = {
+ "version": "1.0", "truncation": None, "padding": None,
+ "added_tokens": [], "normalizer": None, "pre_tokenizer": None,
+ "post_processor": None,
+ "model": {"type": "BPE", "dropout": None, "unk_token": None,
+ "continuing_subword_prefix": None, "end_of_word_suffix": None,
+ "fuse_unk": False, "byte_fallback": True, "ignore_merges": True,
+ "vocab": vocab, "merges": []},
+ }
+ # `decoder=None` means the key is ABSENT, which is its own code path -- not a decoder
+ # whose type is null.
+ if decoder is not None:
+ spec["decoder"] = decoder
+ return spec
+
+ shapes = {
+ # Strip removes up to `start` leading and up to `stop` trailing copies of `content`, per
+ # TOKEN. The crate serialises the Python `Strip(content, left, right)` as start/stop,
+ # which is why those are the field names the port reads.
+ "strip": {
+ "why": "Strip, which no shipped checkpoint declares",
+ "spec": spec_for(
+ {"type": "Strip", "content": " ", "start": 2, "stop": 1},
+ {" hello ": 0, " x": 1, " ": 2, "no": 3}),
+ "ids": [[0], [1], [3], [0, 3]],
+ # The crate PANICS on these, so there is no reference answer to record: stripping 2
+ # leading spaces from the two-space token leaves an empty slice and `stop` then
+ # indexes backwards through it -- "slice index starts at 2 but ends at 1", a Rust
+ # panic that crosses the FFI boundary as pyo3_runtime.PanicException. The port guards
+ # the same arithmetic (`to > from`) and returns "". That difference is deliberate and
+ # is asserted on the Java side: a decoder that aborts the process on a token that is
+ # all `content` is not behaviour worth reproducing.
+ "crate_panics": [[2], [2, 2]],
+ },
+ # Which `<0x..>` spellings the crate reads as a BYTE. Its predicate is a length-6 token
+ # starting "<0x" and ending ">", parsed by Rust's `u8::from_str_radix(.., 16)` -- which is
+ # case-INSENSITIVE and accepts a leading "+". This port had it backwards, rejecting
+ # lowercase and "+" with a comment asserting the crate wrote these uppercase only.
+ #
+ # The "+" case needs its own token: with only the upper/lower pair here, a mutant that
+ # re-broke "+" passed all 195 cases. "<0x 5>" and "<0XFF>" record the other side of the
+ # boundary -- a space is not a digit, and the "<0x" prefix IS case-sensitive -- so the
+ # rule is pinned from both directions rather than only where it says yes.
+ "bytefallback_hex_case": {
+ "why": "which byte-token spellings the crate reads as a byte",
+ "spec": spec_for(
+ {"type": "ByteFallback"},
+ {"<0xFF>": 0, "<0xff>": 1, "<0x41>": 2, "<0x61>": 3, "a": 4,
+ "<0x+5>": 5, "<0x 5>": 6, "<0XFF>": 7, "<0xfF>": 8,
+ "<0xF3>": 9, "<0xB0>": 10, "<0x80>": 11}),
+ # The last four are a TRUNCATED multi-byte run, which is the only shape that can tell
+ # lossy decoding from one replacement per byte: F3 B0 is a single maximal subpart
+ # (lossy gives ONE U+FFFD) and two unusable bytes (the crate gives TWO). Every other
+ # invalid run in this vocabulary is a single byte, where both rules agree -- so a
+ # mutant that made this decoder lossy passed all 195 cases.
+ "ids": [[2], [3], [1], [0], [2, 3], [0, 0], [1, 4],
+ [5], [6], [7], [8], [5, 4], [6, 4], [7, 4],
+ [9, 10], [9, 10, 11], [9, 10, 11, 11], [4, 9, 10, 4]],
+ },
+ # No `decoder` key at all. The crate's fallback is `tokens.join(" ")` -- a SPACE -- and
+ # this port joined with nothing, which was wrong for every multi-token decode. Neither
+ # shipped checkpoint can reach it, because both declare a decoder.
+ "no_decoder": {
+ "why": "no decoder section: the separator the crate falls back to",
+ "spec": spec_for(None, {"ab": 0, "cd": 1, "ef": 2, " ": 3}),
+ "ids": [[0], [0, 1], [0, 1, 2], [0, 3, 1], [3, 3]],
+ },
+ # A Sequence inside a Sequence. The crate chains `decode_chain` and returns a LIST, so the
+ # outer sequence's next step sees the inner one's pieces; re-joining them collapsed the
+ # boundaries. Strip is the step that can tell, because it acts per piece.
+ "nested_sequence": {
+ "why": "a Sequence nested in a Sequence keeps its pieces",
+ "spec": spec_for(
+ {"type": "Sequence", "decoders": [
+ {"type": "Sequence", "decoders": [
+ {"type": "Replace", "pattern": {"String": "\u2581"}, "content": " "},
+ {"type": "ByteFallback"}]},
+ {"type": "Strip", "content": " ", "start": 1, "stop": 0}]},
+ {"\u2581Hello": 0, "\u2581world": 1, "\u2581": 2, "\u2581a": 3}),
+ "ids": [[0, 1], [2, 2, 0], [2, 3, 2], [0], [3]],
+ },
+ # Whether a failed ByteFallback run is one piece of N replacement characters or N pieces
+ # of one. Only a per-piece step after it can tell, so Replace looks for the PAIR: it
+ # matches if the two characters ended up in the same piece.
+ "bytefallback_piece_boundaries": {
+ "why": "a failed byte run is one piece per byte, not one piece of N",
+ "spec": spec_for(
+ {"type": "Sequence", "decoders": [
+ {"type": "ByteFallback"},
+ {"type": "Replace", "pattern": {"String": "\ufffd\ufffd"},
+ "content": "X"}]},
+ {"<0xF3>": 0, "<0xB0>": 1, "<0x80>": 2, "hi": 3}),
+ "ids": [[0, 1], [0, 1, 2], [3, 0, 1, 3], [0, 1, 2, 2], [3]],
+ },
+ }
+
+ for name, shape in shapes.items():
+ path = os.path.join(tempfile.mkdtemp(), "tokenizer.json")
+ io.open(path, "w", encoding="utf-8").write(json.dumps(shape["spec"]))
+ tok = Tokenizer.from_file(path)
+ shape["cases"] = [{"ids": ids, "decoded": tok.decode(ids, skip_special_tokens=False)}
+ for ids in shape.pop("ids")]
+ # A shape whose cases all decoded to the same thing would pin nothing, and these are
+ # hand-chosen rather than swept, so the check is worth its one line.
+ distinct = {case["decoded"] for case in shape["cases"]}
+ if len(distinct) < 2:
+ raise AssertionError(
+ "the %s shape's cases all decode to %r, so they cannot tell any rule apart"
+ % (name, distinct.pop()))
+ return shapes
+
+
+def _mixed_token_scope():
+ """Whether the ByteLevel fallback is scoped to a TOKEN or to a CHARACTER, from the crate.
+
+ Neither real checkpoint can answer this. Their byte-level tokens are entirely stand-in
+ alphabet and their added tokens are entirely literal whitespace, so both rules produce the
+ same bytes for every token in both vocabularies -- a mutant that swaps per-token for
+ per-character passed all 190 recorded cases. The rules differ only for a token that MIXES
+ the two, so this builds a vocabulary that has one and records what the crate does with it.
+
+ The separating token is "Ġ ": U+0120 is the stand-in for byte 0x20, and the literal
+ space after it is not in the alphabet at all.
+
+ per TOKEN -> every character contributes its own UTF-8 bytes: C4 A0 20 -> "Ġ "
+ per CHARACTER -> the mapped one contributes its byte: 20 20 -> " "
+
+ The spec travels in the fixture rather than being written twice, so the Java test loads the
+ same bytes the crate was asked about.
+ """
+ import io
+ import os
+ import tempfile
+
+ try:
+ from tokenizers import Tokenizer
+ except ImportError:
+ return {"skipped": "the tokenizers package is not installed"}
+
+ mixed = "Ġ "
+
+ # Every single-byte alphabet symbol, from the crate's own table rather than a transcription.
+ # The port's loader refuses a vocabulary that cannot represent some text, and a vocabulary of
+ # three tokens cannot -- so the three interesting tokens join a complete byte-level one. They
+ # do not interact: decoding is driven by the ids this records, not by what else is present.
+ from tokenizers.pre_tokenizers import ByteLevel as ByteLevelPre
+
+ vocab = {}
+ for symbol in sorted(ByteLevelPre.alphabet()):
+ vocab[symbol] = len(vocab)
+ interesting = {}
+ for token in (mixed, "Ġhi", " "):
+ if token not in vocab:
+ vocab[token] = len(vocab)
+ interesting[token] = vocab[token]
+
+ spec = {
+ "version": "1.0", "truncation": None, "padding": None,
+ "added_tokens": [], "normalizer": None,
+ # The pre-tokenizer matches the decoder. It plays no part in decoding, but the port's
+ # loader refuses a vocabulary where a character could be dropped without trace -- no
+ # unk_token, no byte_fallback, no byte-level pre-tokenizer -- and that guard is right.
+ "pre_tokenizer": {"type": "ByteLevel", "add_prefix_space": True, "trim_offsets": True,
+ "use_regex": True},
+ "post_processor": None,
+ "decoder": {"type": "ByteLevel", "add_prefix_space": True, "trim_offsets": True,
+ "use_regex": True},
+ "model": {"type": "BPE", "dropout": None, "unk_token": None,
+ "continuing_subword_prefix": None, "end_of_word_suffix": None,
+ "fuse_unk": False, "byte_fallback": False, "ignore_merges": True,
+ "vocab": vocab, "merges": []},
+ }
+
+ path = os.path.join(tempfile.mkdtemp(), "tokenizer.json")
+ io.open(path, "w", encoding="utf-8").write(json.dumps(spec))
+ tok = Tokenizer.from_file(path)
+
+ cases = [
+ {"why": "the mixed token alone: the whole point",
+ "ids": [interesting[mixed]]},
+ {"why": "entirely in the alphabet", "ids": [interesting["Ġhi"]]},
+ {"why": "entirely literal", "ids": [interesting[" "]]},
+ {"why": "a mixed token beside a plain one",
+ "ids": [interesting[mixed], interesting["Ġhi"]]},
+ ]
+ for case in cases:
+ case["decoded"] = tok.decode(case["ids"], skip_special_tokens=False)
+
+ scope = "token" if cases[0]["decoded"] == mixed else (
+ "character" if cases[0]["decoded"] == " " else "neither")
+ if scope != "token":
+ raise AssertionError(
+ "the crate's ByteLevel fallback is scoped per %s, and the port assumes per token; "
+ "decoding the mixed token gave %r" % (scope, cases[0]["decoded"]))
+ return {"spec": spec, "separating_token": mixed, "scope": scope, "cases": cases}
+
+
+def _decoder_shape(tokenizer_json):
+ """The decoder section's type chain, flattened, so the fixture says what it was recorded on."""
+ import io
+ import json
+
+ document = json.load(io.open(tokenizer_json, encoding="utf-8"))
+ node = document.get("decoder")
+ if node is None:
+ return None
+
+ def shape(n):
+ kind = n.get("type")
+ if kind == "Sequence":
+ return {"type": "Sequence",
+ "decoders": [shape(step) for step in n.get("decoders", [])]}
+ return {"type": kind}
+
+ return shape(node)
+
+
FAMILIES = {
"lang_tables.json": lang_tables,
"lang_detect.json": lang_detect,
@@ -2095,6 +2438,7 @@ FAMILIES = {
"router.json": router,
"shortlist.json": shortlist,
"tokenizer_ids.json": tokenizer_ids,
+ "decode_text.json": decode_text,
"sequences.json": sequences,
"decode.json": decode_answers,
"python_json.json": python_json,
@@ -2139,8 +2483,41 @@ def unverifiable(payload):
# always present, so requiring every section to be skipped let a payload through whose two
# per-checkpoint halves were both skip markers -- and the write replaced a 107 KB fixture with
# a 9 KB one. Partial regeneration is still unsafe to write.
- return any("skipped" in value
- for value in payload.values() if isinstance(value, dict))
+ #
+ # At ANY depth, not just one level down. This looked only at the payload's immediate values,
+ # and `decode_text.json` groups its per-checkpoint sections under a `by` key -- so the skip
+ # markers sat two levels down, the payload was judged regenerable, and both halves of this
+ # docstring came true at once: on a machine without the checkpoints, `--check` reported the
+ # committed fixture as stale, and a plain run replaced 75 KB of parity expectations with 7 KB
+ # of skip markers. Measured, not reasoned about. A family should not have to keep its sections
+ # at the top level to be seen, so the search descends instead.
+ return _mentions_skip(payload)
+
+
+def _mentions_skip(node):
+ """True when a skip MARKER appears anywhere in the payload.
+
+ A marker is a dict of exactly one `skipped` key, which is what every builder above writes.
+ Merely containing a `skipped` key is not enough, and that is not a hypothetical distinction:
+ `decode_text.json` records, per text, what the reference returns with special tokens skipped
+ and without, and the first version of that family named the first of those two fields
+ `skipped`. The recursion below read it as a marker, so a fully recorded family reported itself
+ unregenerable -- with the checkpoints present it was left alone instead of checked, and
+ `--strict` in the parity lane would have failed on a family that had just been recorded
+ correctly.
+
+ That field is now `skipping_specials`, so the collision no longer exists in this script and
+ nothing here depends on the exact-key rule to avoid it. The rule stays because it is the
+ correct test for a marker, and because the next family to record a `skipped` field should not
+ have to discover this interaction the way that one did.
+ """
+ if isinstance(node, dict):
+ if set(node) == {"skipped"}:
+ return True
+ return any(_mentions_skip(value) for value in node.values())
+ if isinstance(node, list):
+ return any(_mentions_skip(item) for item in node)
+ return False
def main(argv=None):