Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(java): Tokenizer.decode, and the four rules the slices found
Adeshina Lasisi561c496508ed
diff --git a/.github/workflows/java.yml b/.github/workflows/java.ymlindex 2522b3fa..8219b2e3 100644--- a/.github/workflows/java.yml+++ b/.github/workflows/java.yml@@ -70,7 +70,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 585 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 625 --allow-aborted# The samples are single-file source programs, deliberately NOT a Gradle module: a# `samples` subproject would inherit the root build's MavenPublication and@@ -226,7 +226,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 585 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 625 --allow-aborted--require-passed TestJvmVersionTest- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2if: ${{ failure() }}@@ -263,13 +263,28 @@ jobs:- name: tokenizercheckpoints: english multilingualgraph: false- families: --only tokenizer_ids.json --only sequences.json --only decode.json+ families: >-+ --only tokenizer_ids.json --only sequences.json --only decode.json+ --only decode_text.jsontests: >---tests com.convaiinnovations.laya.tokenizer.TokenizerParityTest--tests com.convaiinnovations.laya.sequence.SequenceParityTest--tests com.convaiinnovations.laya.decode.DecoderParityTest--tests com.convaiinnovations.laya.tokenizer.PreTokenizerTablesTest- min-tests: 175+ --tests com.convaiinnovations.laya.tokenizer.TokenDecoderTest+ # Measured locally. The margin is deliberately smaller than the 8 the 183/175 pair+ # used, because the smallest class in this cell is PreTokenizerTablesTest at 5 tests+ # and a margin of 8 let that whole class be dropped with the floor still green+ # (measured: 369 of 374, exit 0). A floor below the total minus the smallest class+ # does not bind any single class.+ #+ # What this does NOT protect, despite an earlier version of this comment saying so:+ # the window SLICES. A count floor cannot see them -- the 403 slice assertions live+ # inside 46 of the ~195 dynamic tests, because the factory bundles all of one text's+ # slices into a single test, and a slice list that emptied would still register its+ # test. The slices are guarded inside the suite instead, by the `total >= 400`+ # assertion in TokenDecoderTest and the `cases.size() >= 23` corpus floor beside it.+ min-tests: 374timeout: 25# The one cell that needs the traced graph, and the only checkpoint the end-to-end# golden was recorded from.diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.ymlindex d2cf118f..6f1895f2 100644--- a/.github/workflows/release-java.yml+++ b/.github/workflows/release-java.yml@@ -58,7 +58,10 @@ jobs:- name: The suite was not vacuousrun: >python3 scripts/check_test_results.py laya-java/build/test-results/test- --min-tests 440 --allow-aborted+ # 631 of the 640 registered tests run without a checkpoint. 440 left 191 of+ # slack, so the whole decode family could have vanished from this lane with the+ # gate still green -- and a floor that does not bind is not a gate.+ --min-tests 625 --allow-aborted- name: Build the jarsrun: ./gradlew --no-daemon build -x test@@ -103,7 +106,11 @@ jobs:# jobs disagreed about what was being released.case "$GITHUB_REF" inrefs/tags/laya-java-v*) : ;;- *) echo "::error::refusing to publish from $GITHUB_REF; tag the release as laya-java-vX.Y.Z and run this on the tag" ; exit 1 ;;+ *)+ echo "::error::refusing to publish from $GITHUB_REF; tag the release as" \+ "laya-java-vX.Y.Z and run this on the tag"+ exit 1+ ;;esac- name: Version from the tagrun: |@@ -125,8 +132,20 @@ jobs:# All four, not two. Without the passphrase the build calls useInMemoryPgpKeys(key,# null) and uploads UNSIGNED artifacts, which Central rejects only after the upload --# a much worse place to find out.- test -n "$LAYA_SIGNING_KEY" || { echo "::error::LAYA_SIGNING_KEY is unset; see the header"; exit 1; }- test -n "$LAYA_SIGNING_PASSWORD" || { echo "::error::LAYA_SIGNING_PASSWORD is unset; an unsigned bundle is rejected by Central after upload"; exit 1; }- test -n "$MAVEN_CENTRAL_USERNAME" || { echo "::error::MAVEN_CENTRAL_USERNAME is unset; see the header"; exit 1; }- test -n "$MAVEN_CENTRAL_PASSWORD" || { echo "::error::MAVEN_CENTRAL_PASSWORD is unset; see the header"; exit 1; }+ #+ # A loop rather than four `test -n` lines. Written out, the four were 111 to 161 columns+ # and no longer read as the parallel checks they are, and each one exited on the FIRST+ # thing it found -- so a release missing two secrets cost two tagged runs to discover.+ # This names all of them at once.+ missing=""+ for secret in LAYA_SIGNING_KEY LAYA_SIGNING_PASSWORD \+ MAVEN_CENTRAL_USERNAME MAVEN_CENTRAL_PASSWORD; do+ [ -n "${!secret:-}" ] || missing="$missing $secret"+ done+ if [ -n "$missing" ]; then+ echo "::error::unset secret(s):$missing -- see this workflow's header for how each" \+ "one is produced. Note the signing passphrase in particular: without it the" \+ "bundle is built UNSIGNED and Central rejects it only after the upload."+ exit 1+ fi./gradlew --no-daemon publishAllPublicationsToCentralRepositorydiff --git a/laya-java/CHANGELOG.md b/laya-java/CHANGELOG.mdindex 17bcbc30..6f0f6b7a 100644--- a/laya-java/CHANGELOG.md+++ b/laya-java/CHANGELOG.md@@ -20,6 +20,13 @@ see [Installing](README.md#installing).`confidence` alongside `answerConfidence`.- Python-compatible tokenization (BPE and Metaspace, added tokens, normalizers), sequencebuilding, collation, answer decoding and calibration.+- `Tokenizer.decode(int[])` and `decode(int[], boolean skipSpecialTokens)`: ids back to text,+ matching the `tokenizers` crate's `ByteLevel`, `Replace`, `ByteFallback`, `Fuse`, `Strip` and+ `Sequence` decoders. Byte-exact against the reference over both shipped checkpoints, including+ 201,279 window slices — the shape that cuts a multi-byte character in half, where a decoder that+ agrees on every whole text can still be wrong. Decoding on the multilingual checkpoint is lossy+ by the reference's own behaviour: `Metaspace` prepends its marker and the round trip gains a+ leading space, which this port reproduces rather than corrects.- `lang.LanguageDetection`: script and language detection, the eight public fields and thefourteen intermediate steps the reference exposes.- `Presets`: the five preset question sets, compared against the reference word for word.diff --git a/laya-java/README.md b/laya-java/README.mdindex 886de5de..52404bb7 100644--- a/laya-java/README.md+++ b/laya-java/README.md@@ -296,9 +296,16 @@ import java.nio.file.Path;Tokenizer tok = Tokenizer.fromModelDirectory(Path.of("./checkpoint"));int[] ids = tok.encode("charged twice");int[] capped = tok.encode(longText, 48); // stops early; same prefix as the full encoding+String text = tok.decode(ids); // ids back to text; special tokens dropped+String withSpecials = tok.decode(ids, false); // ...keptAgentConfig cfg = AgentConfig.fromModelDirectory(Path.of("./checkpoint"));```+`decode` follows the reference rather than tidying after it, and on the multilingual checkpoint+that means it is **lossy**: `Metaspace` prepends its marker, so `decode(encode("Hello world"))` is+`" Hello world"`. The English checkpoint happens to round-trip. Correcting the space would make+every window of a long document tokenize differently from the reference, so the loss is kept.+`Agent.using(tokenizer, config, session)` assembles an agent from parts — for a caller that alreadyholds them, or to drive the batching and usage accounting through a stub`infer.InferenceSession` instead of a 1.2 GB graph.diff --git a/laya-java/fixtures/decode_text.json b/laya-java/fixtures/decode_text.jsonnew file mode 100644index 00000000..a3ae5981--- /dev/null+++ b/laya-java/fixtures/decode_text.json@@ -0,0 +1,3964 @@+{+ "texts": [+ "Hello world",+ "",+ " ",+ " ",+ "\t",+ "a\tb\nc",+ "café résumé",+ "注文をキャンセル",+ "مرحبا",+ "क्ष",+ "😀",+ "emoji 😀 here",+ "🇬🇧",+ "▁literal marker",+ "pre▁fix",+ " nbsp figure",+ "<0x41>",+ "a<0xFF>b",+ "\u0001\u0002",+ "",+ "",+ "I cannot log in and need a password reset.",+ "Esta mensagem e confidencial."+ ],+ "by": {+ "english": {+ "decoder": {+ "type": "ByteLevel"+ },+ "cases": [+ {+ "text": "Hello world",+ "ids": [+ 12092,+ 1533+ ],+ "decoded": "Hello world",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "Hello"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " world"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "Hello world"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "Hello world"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "Hello world"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 12092,+ 1533,+ 50282+ ],+ "skipping_specials": "Hello world",+ "kept": "[CLS]Hello world[SEP]"+ }+ },+ {+ "text": "",+ "ids": [],+ "decoded": "",+ "slices": [],+ "with_specials": {+ "ids": [+ 50281,+ 50282+ ],+ "skipping_specials": "",+ "kept": "[CLS][SEP]"+ }+ },+ {+ "text": " ",+ "ids": [+ 209+ ],+ "decoded": " ",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " "+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 209,+ 50282+ ],+ "skipping_specials": " ",+ "kept": "[CLS] [SEP]"+ }+ },+ {+ "text": " ",+ "ids": [+ 50275+ ],+ "decoded": " ",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " "+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 50275,+ 50282+ ],+ "skipping_specials": " ",+ "kept": "[CLS] [SEP]"+ }+ },+ {+ "text": "\t",+ "ids": [+ 186+ ],+ "decoded": "\t",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "\t"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "\t"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "\t"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "\t"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 186,+ 50282+ ],+ "skipping_specials": "\t",+ "kept": "[CLS]\t[SEP]"+ }+ },+ {+ "text": "a\tb\nc",+ "ids": [+ 66,+ 186,+ 67,+ 187,+ 68+ ],+ "decoded": "a\tb\nc",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "a"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "\t"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "b"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "\n"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "c"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "a\t"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "b\n"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "c"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "a\tb"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "\nc"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "a\tb\nc"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 66,+ 186,+ 67,+ 187,+ 68,+ 50282+ ],+ "skipping_specials": "a\tb\nc",+ "kept": "[CLS]a\tb\nc[SEP]"+ }+ },+ {+ "text": "café résumé",+ "ids": [+ 68,+ 2320,+ 860,+ 9568,+ 2204,+ 860+ ],+ "decoded": "café résumé",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "c"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "af"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "é"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": " ré"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "sum"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": "é"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "caf"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "é ré"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "sumé"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "café"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": " résumé"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "café résum"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": "é"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 68,+ 2320,+ 860,+ 9568,+ 2204,+ 860,+ 50282+ ],+ "skipping_specials": "café résumé",+ "kept": "[CLS]café résumé[SEP]"+ }+ },+ {+ "text": "注文をキャンセル",+ "ids": [+ 44712,+ 20855,+ 6449,+ 33548,+ 41760,+ 9467,+ 41662,+ 16509+ ],+ "decoded": "注文をキャンセル",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "注"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "文"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "を"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "キ"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "ャ"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": "ン"+ },+ {+ "start": 6,+ "size": 1,+ "decoded": "セ"+ },+ {+ "start": 7,+ "size": 1,+ "decoded": "ル"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "注文"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "をキ"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "ャン"+ },+ {+ "start": 6,+ "size": 2,+ "decoded": "セル"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "注文を"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "キャン"+ },+ {+ "start": 6,+ "size": 3,+ "decoded": "セル"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "注文をキャ"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": "ンセル"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 44712,+ 20855,+ 6449,+ 33548,+ 41760,+ 9467,+ 41662,+ 16509,+ 50282+ ],+ "skipping_specials": "注文をキャンセル",+ "kept": "[CLS]注文をキャンセル[SEP]"+ }+ },+ {+ "text": "مرحبا",+ "ids": [+ 5843,+ 6900,+ 21931,+ 13621,+ 3142+ ],+ "decoded": "مرحبا",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "م"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "ر"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "ح"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "ب"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "ا"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "مر"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "حب"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "ا"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "مرح"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "با"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "مرحبا"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 5843,+ 6900,+ 21931,+ 13621,+ 3142,+ 50282+ ],+ "skipping_specials": "مرحبا",+ "kept": "[CLS]مرحبا[SEP]"+ }+ },+ {+ "text": "क्ष",+ "ids": [+ 24381,+ 17665,+ 49003,+ 2483,+ 117+ ],+ "decoded": "क्ष",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "क"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "्"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": ""+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "क्"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "क्"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "ष"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "क्ष"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 24381,+ 17665,+ 49003,+ 2483,+ 117,+ 50282+ ],+ "skipping_specials": "क्ष",+ "kept": "[CLS]क्ष[SEP]"+ }+ },+ {+ "text": "😀",+ "ids": [+ 14931,+ 235,+ 211+ ],+ "decoded": "😀",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "😀"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "😀"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 14931,+ 235,+ 211,+ 50282+ ],+ "skipping_specials": "😀",+ "kept": "[CLS]😀[SEP]"+ }+ },+ {+ "text": "emoji 😀 here",+ "ids": [+ 43208,+ 8020,+ 49042,+ 211,+ 1060+ ],+ "decoded": "emoji 😀 here",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "emo"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "ji"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": " �"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": " here"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "emoji"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " 😀"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": " here"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "emoji �"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "� here"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "emoji 😀 here"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 43208,+ 8020,+ 49042,+ 211,+ 1060,+ 50282+ ],+ "skipping_specials": "emoji 😀 here",+ "kept": "[CLS]emoji 😀 here[SEP]"+ }+ },+ {+ "text": "🇬🇧",+ "ids": [+ 14931,+ 218,+ 107,+ 14931,+ 218,+ 102+ ],+ "decoded": "🇬🇧",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "��"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "��"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "🇬"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "🇧"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "🇬�"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": "�"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 14931,+ 218,+ 107,+ 14931,+ 218,+ 102,+ 50282+ ],+ "skipping_specials": "🇬🇧",+ "kept": "[CLS]🇬🇧[SEP]"+ }+ },+ {+ "text": "▁literal marker",+ "ids": [+ 10897,+ 212,+ 34943,+ 10705+ ],+ "decoded": "▁literal marker",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "literal"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": " marker"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "▁"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "literal marker"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "▁literal"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": " marker"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "▁literal marker"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 10897,+ 212,+ 34943,+ 10705,+ 50282+ ],+ "skipping_specials": "▁literal marker",+ "kept": "[CLS]▁literal marker[SEP]"+ }+ },+ {+ "text": "pre▁fix",+ "ids": [+ 3456,+ 10897,+ 212,+ 11097+ ],+ "decoded": "pre▁fix",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "pre"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "fix"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "pre�"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "�fix"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "pre▁"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "fix"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "pre▁fix"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 3456,+ 10897,+ 212,+ 11097,+ 50282+ ],+ "skipping_specials": "pre▁fix",+ "kept": "[CLS]pre▁fix[SEP]"+ }+ },+ {+ "text": " nbsp figure",+ "ids": [+ 575,+ 6732,+ 325,+ 218,+ 13206+ ],+ "decoded": " nbsp figure",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "nbsp"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "figure"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " nbsp"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " "+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "figure"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " nbsp�"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "�figure"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " nbsp figure"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 575,+ 6732,+ 325,+ 218,+ 13206,+ 50282+ ],+ "skipping_specials": " nbsp figure",+ "kept": "[CLS] nbsp figure[SEP]"+ }+ },+ {+ "text": "<0x41>",+ "ids": [+ 29,+ 17,+ 89,+ 3156,+ 31+ ],+ "decoded": "<0x41>",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "<"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "0"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "x"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "41"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": ">"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "<0"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "x41"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": ">"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "<0x"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "41>"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "<0x41>"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 29,+ 17,+ 89,+ 3156,+ 31,+ 50282+ ],+ "skipping_specials": "<0x41>",+ "kept": "[CLS]<0x41>[SEP]"+ }+ },+ {+ "text": "a<0xFF>b",+ "ids": [+ 66,+ 29,+ 17,+ 30962,+ 31,+ 67+ ],+ "decoded": "a<0xFF>b",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "a"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "<"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "0"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "xFF"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": ">"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": "b"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "a<"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "0xFF"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": ">b"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "a<0"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "xFF>b"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "a<0xFF>"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": "b"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 66,+ 29,+ 17,+ 30962,+ 31,+ 67,+ 50282+ ],+ "skipping_specials": "a<0xFF>b",+ "kept": "[CLS]a<0xFF>b[SEP]"+ }+ },+ {+ "text": "\u0001\u0002",+ "ids": [+ 178,+ 179+ ],+ "decoded": "\u0001\u0002",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "\u0001"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "\u0002"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "\u0001\u0002"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "\u0001\u0002"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "\u0001\u0002"+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 178,+ 179,+ 50282+ ],+ "skipping_specials": "\u0001\u0002",+ "kept": "[CLS]\u0001\u0002[SEP]"+ }+ },+ {+ "text": "",+ "ids": [+ 170,+ 211,+ 211+ ],+ "decoded": "",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": ""+ },+ {+ "start": 0,+ "size": 5,+ "decoded": ""+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 170,+ 211,+ 211,+ 50282+ ],+ "skipping_specials": "",+ "kept": "[CLS][SEP]"+ }+ },+ {+ "text": "",+ "ids": [+ 175,+ 110,+ 211,+ 211+ ],+ "decoded": "",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "��"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "�"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": ""+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 175,+ 110,+ 211,+ 211,+ 50282+ ],+ "skipping_specials": "",+ "kept": "[CLS][SEP]"+ }+ },+ {+ "text": "I cannot log in and need a password reset.",+ "ids": [+ 42,+ 2550,+ 2412,+ 275,+ 285,+ 878,+ 247,+ 9868,+ 14932,+ 15+ ],+ "decoded": "I cannot log in and need a password reset.",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "I"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " cannot"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": " log"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": " in"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": " and"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": " need"+ },+ {+ "start": 6,+ "size": 1,+ "decoded": " a"+ },+ {+ "start": 7,+ "size": 1,+ "decoded": " password"+ },+ {+ "start": 8,+ "size": 1,+ "decoded": " reset"+ },+ {+ "start": 9,+ "size": 1,+ "decoded": "."+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "I cannot"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " log in"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": " and need"+ },+ {+ "start": 6,+ "size": 2,+ "decoded": " a password"+ },+ {+ "start": 8,+ "size": 2,+ "decoded": " reset."+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "I cannot log"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": " in and need"+ },+ {+ "start": 6,+ "size": 3,+ "decoded": " a password reset"+ },+ {+ "start": 9,+ "size": 3,+ "decoded": "."+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "I cannot log in and"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": " need a password reset."+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 42,+ 2550,+ 2412,+ 275,+ 285,+ 878,+ 247,+ 9868,+ 14932,+ 15,+ 50282+ ],+ "skipping_specials": "I cannot log in and need a password reset.",+ "kept": "[CLS]I cannot log in and need a password reset.[SEP]"+ }+ },+ {+ "text": "Esta mensagem e confidencial.",+ "ids": [+ 38,+ 18233,+ 34939,+ 31646,+ 299,+ 1461,+ 301,+ 49016,+ 15+ ],+ "decoded": "Esta mensagem e confidencial.",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": "E"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "sta"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": " mens"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "agem"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": " e"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": " conf"+ },+ {+ "start": 6,+ "size": 1,+ "decoded": "id"+ },+ {+ "start": 7,+ "size": 1,+ "decoded": "encial"+ },+ {+ "start": 8,+ "size": 1,+ "decoded": "."+ },+ {+ "start": 0,+ "size": 2,+ "decoded": "Esta"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " mensagem"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": " e conf"+ },+ {+ "start": 6,+ "size": 2,+ "decoded": "idencial"+ },+ {+ "start": 8,+ "size": 2,+ "decoded": "."+ },+ {+ "start": 0,+ "size": 3,+ "decoded": "Esta mens"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "agem e conf"+ },+ {+ "start": 6,+ "size": 3,+ "decoded": "idencial."+ },+ {+ "start": 0,+ "size": 5,+ "decoded": "Esta mensagem e"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": " confidencial."+ }+ ],+ "with_specials": {+ "ids": [+ 50281,+ 38,+ 18233,+ 34939,+ 31646,+ 299,+ 1461,+ 301,+ 49016,+ 15,+ 50282+ ],+ "skipping_specials": "Esta mensagem e confidencial.",+ "kept": "[CLS]Esta mensagem e confidencial.[SEP]"+ }+ }+ ]+ },+ "multilingual": {+ "decoder": {+ "type": "Sequence",+ "decoders": [+ {+ "type": "Replace"+ },+ {+ "type": "ByteFallback"+ },+ {+ "type": "Fuse"+ }+ ]+ },+ "cases": [+ {+ "text": "Hello world",+ "ids": [+ 25957,+ 2134+ ],+ "decoded": " Hello world",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " Hello"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " world"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " Hello world"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " Hello world"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " Hello world"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 25957,+ 2134,+ 1+ ],+ "skipping_specials": " Hello world",+ "kept": "<bos> Hello world<eos>"+ }+ },+ {+ "text": "",+ "ids": [],+ "decoded": "",+ "slices": [],+ "with_specials": {+ "ids": [+ 2,+ 1+ ],+ "skipping_specials": "",+ "kept": "<bos><eos>"+ }+ },+ {+ "text": " ",+ "ids": [+ 235248+ ],+ "decoded": " ",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " "+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 235248,+ 1+ ],+ "skipping_specials": " ",+ "kept": "<bos> <eos>"+ }+ },+ {+ "text": " ",+ "ids": [+ 235248,+ 235248,+ 235248+ ],+ "decoded": " ",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 2,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " "+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " "+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 235248,+ 235248,+ 235248,+ 1+ ],+ "skipping_specials": " ",+ "kept": "<bos> <eos>"+ }+ },+ {+ "text": "\t",+ "ids": [+ 235248,+ 226+ ],+ "decoded": " \t",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "\t"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " \t"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " \t"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " \t"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 235248,+ 226,+ 1+ ],+ "skipping_specials": " \t",+ "kept": "<bos> \t<eos>"+ }+ },+ {+ "text": "a\tb\nc",+ "ids": [+ 476,+ 226,+ 235268,+ 108,+ 498+ ],+ "decoded": " a\tb\n c",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " a"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "\t"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "b"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "\n"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": " c"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " a\t"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "b\n"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": " c"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " a\tb"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "\n c"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " a\tb\n c"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 476,+ 226,+ 235268,+ 108,+ 498,+ 1+ ],+ "skipping_specials": " a\tb\n c",+ "kept": "<bos> a\tb\n c<eos>"+ }+ },+ {+ "text": "café résumé",+ "ids": [+ 18688,+ 96537+ ],+ "decoded": " café résumé",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " café"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " résumé"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " café résumé"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " café résumé"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " café résumé"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 18688,+ 96537,+ 1+ ],+ "skipping_specials": " café résumé",+ "kept": "<bos> café résumé<eos>"+ }+ },+ {+ "text": "注文をキャンセル",+ "ids": [+ 231293,+ 235432,+ 182129+ ],+ "decoded": " 注文をキャンセル",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " 注文"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "を"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "キャンセル"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " 注文を"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "キャンセル"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " 注文をキャンセル"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " 注文をキャンセル"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 231293,+ 235432,+ 182129,+ 1+ ],+ "skipping_specials": " 注文をキャンセル",+ "kept": "<bos> 注文をキャンセル<eos>"+ }+ },+ {+ "text": "مرحبا",+ "ids": [+ 89187,+ 19833+ ],+ "decoded": " مرحبا",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " مرح"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "با"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " مرحبا"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " مرحبا"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " مرحبا"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 89187,+ 19833,+ 1+ ],+ "skipping_specials": " مرحبا",+ "kept": "<bos> مرحبا<eos>"+ }+ },+ {+ "text": "क्ष",+ "ids": [+ 2280,+ 235485,+ 235762,+ 236355+ ],+ "decoded": " क्ष",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " क"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "्"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": ""+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "ष"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " क्"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "ष"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " क्"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "ष"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " क्ष"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 2280,+ 235485,+ 235762,+ 236355,+ 1+ ],+ "skipping_specials": " क्ष",+ "kept": "<bos> क्ष<eos>"+ }+ },+ {+ "text": "😀",+ "ids": [+ 79937+ ],+ "decoded": " 😀",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " 😀"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " 😀"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " 😀"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " 😀"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 79937,+ 1+ ],+ "skipping_specials": " 😀",+ "kept": "<bos> 😀<eos>"+ }+ },+ {+ "text": "emoji 😀 here",+ "ids": [+ 52810,+ 79937,+ 1517+ ],+ "decoded": " emoji 😀 here",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " emoji"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " 😀"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": " here"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " emoji 😀"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " here"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " emoji 😀 here"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " emoji 😀 here"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 52810,+ 79937,+ 1517,+ 1+ ],+ "skipping_specials": " emoji 😀 here",+ "kept": "<bos> emoji 😀 here<eos>"+ }+ },+ {+ "text": "🇬🇧",+ "ids": [+ 235248,+ 242330,+ 241461+ ],+ "decoded": " 🇬🇧",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "🇬"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "🇧"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " 🇬"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "🇧"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " 🇬🇧"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " 🇬🇧"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 235248,+ 242330,+ 241461,+ 1+ ],+ "skipping_specials": " 🇬🇧",+ "kept": "<bos> 🇬🇧<eos>"+ }+ },+ {+ "text": "▁literal marker",+ "ids": [+ 36414,+ 24416+ ],+ "decoded": " literal marker",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " literal"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " marker"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " literal marker"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " literal marker"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " literal marker"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 36414,+ 24416,+ 1+ ],+ "skipping_specials": " literal marker",+ "kept": "<bos> literal marker<eos>"+ }+ },+ {+ "text": "pre▁fix",+ "ids": [+ 953,+ 6530+ ],+ "decoded": " pre fix",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " pre"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " fix"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " pre fix"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " pre fix"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " pre fix"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 953,+ 6530,+ 1+ ],+ "skipping_specials": " pre fix",+ "kept": "<bos> pre fix<eos>"+ }+ },+ {+ "text": " nbsp figure",+ "ids": [+ 84306,+ 11450,+ 243499,+ 6270+ ],+ "decoded": " nbsp figure",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "nbsp"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "figure"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " nbsp"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " figure"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " nbsp "+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "figure"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " nbsp figure"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 84306,+ 11450,+ 243499,+ 6270,+ 1+ ],+ "skipping_specials": " nbsp figure",+ "kept": "<bos> nbsp figure<eos>"+ }+ },+ {+ "text": "<0x41>",+ "ids": [+ 968,+ 235276,+ 235297,+ 235310,+ 235274,+ 235313+ ],+ "decoded": " <0x41>",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " <"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "0"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "x"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "4"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "1"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": ">"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " <0"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "x4"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "1>"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " <0x"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "41>"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " <0x41"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": ">"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 968,+ 235276,+ 235297,+ 235310,+ 235274,+ 235313,+ 1+ ],+ "skipping_specials": " <0x41>",+ "kept": "<bos> <0x41><eos>"+ }+ },+ {+ "text": "a<0xFF>b",+ "ids": [+ 476,+ 235322,+ 235276,+ 16663,+ 235313,+ 235268+ ],+ "decoded": " a<0xFF>b",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " a"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "<"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "0"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "xFF"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": ">"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": "b"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " a<"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "0xFF"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": ">b"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " a<0"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "xFF>b"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " a<0xFF>"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": "b"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 476,+ 235322,+ 235276,+ 16663,+ 235313,+ 235268,+ 1+ ],+ "skipping_specials": " a<0xFF>b",+ "kept": "<bos> a<0xFF>b<eos>"+ }+ },+ {+ "text": "\u0001\u0002",+ "ids": [+ 235248,+ 238213,+ 247977+ ],+ "decoded": " \u0001\u0002",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "\u0001"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "\u0002"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " \u0001"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "\u0002"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " \u0001\u0002"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " \u0001\u0002"+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 235248,+ 238213,+ 247977,+ 1+ ],+ "skipping_specials": " \u0001\u0002",+ "kept": "<bos> \u0001\u0002<eos>"+ }+ },+ {+ "text": "",+ "ids": [+ 235248,+ 240539+ ],+ "decoded": " ",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 1,+ "size": 1,+ "decoded": ""+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " "+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " "+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 235248,+ 240539,+ 1+ ],+ "skipping_specials": " ",+ "kept": "<bos> <eos>"+ }+ },+ {+ "text": "",+ "ids": [+ 235248,+ 460,+ 393,+ 345,+ 345+ ],+ "decoded": " ",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " "+ },+ {+ "start": 1,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " �"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": "��"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "�"+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " ��"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": "��"+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " "+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 235248,+ 460,+ 393,+ 345,+ 345,+ 1+ ],+ "skipping_specials": " ",+ "kept": "<bos> <eos>"+ }+ },+ {+ "text": "I cannot log in and need a password reset.",+ "ids": [+ 590,+ 2952,+ 3846,+ 575,+ 578,+ 1476,+ 476,+ 8003,+ 13846,+ 235265+ ],+ "decoded": " I cannot log in and need a password reset.",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " I"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " cannot"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": " log"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": " in"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": " and"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": " need"+ },+ {+ "start": 6,+ "size": 1,+ "decoded": " a"+ },+ {+ "start": 7,+ "size": 1,+ "decoded": " password"+ },+ {+ "start": 8,+ "size": 1,+ "decoded": " reset"+ },+ {+ "start": 9,+ "size": 1,+ "decoded": "."+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " I cannot"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " log in"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": " and need"+ },+ {+ "start": 6,+ "size": 2,+ "decoded": " a password"+ },+ {+ "start": 8,+ "size": 2,+ "decoded": " reset."+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " I cannot log"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": " in and need"+ },+ {+ "start": 6,+ "size": 3,+ "decoded": " a password reset"+ },+ {+ "start": 9,+ "size": 3,+ "decoded": "."+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " I cannot log in and"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": " need a password reset."+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 590,+ 2952,+ 3846,+ 575,+ 578,+ 1476,+ 476,+ 8003,+ 13846,+ 235265,+ 1+ ],+ "skipping_specials": " I cannot log in and need a password reset.",+ "kept": "<bos> I cannot log in and need a password reset.<eos>"+ }+ },+ {+ "text": "Esta mensagem e confidencial.",+ "ids": [+ 22974,+ 70816,+ 526,+ 2103,+ 45972,+ 235265+ ],+ "decoded": " Esta mensagem e confidencial.",+ "slices": [+ {+ "start": 0,+ "size": 1,+ "decoded": " Esta"+ },+ {+ "start": 1,+ "size": 1,+ "decoded": " mensagem"+ },+ {+ "start": 2,+ "size": 1,+ "decoded": " e"+ },+ {+ "start": 3,+ "size": 1,+ "decoded": " conf"+ },+ {+ "start": 4,+ "size": 1,+ "decoded": "idencial"+ },+ {+ "start": 5,+ "size": 1,+ "decoded": "."+ },+ {+ "start": 0,+ "size": 2,+ "decoded": " Esta mensagem"+ },+ {+ "start": 2,+ "size": 2,+ "decoded": " e conf"+ },+ {+ "start": 4,+ "size": 2,+ "decoded": "idencial."+ },+ {+ "start": 0,+ "size": 3,+ "decoded": " Esta mensagem e"+ },+ {+ "start": 3,+ "size": 3,+ "decoded": " confidencial."+ },+ {+ "start": 0,+ "size": 5,+ "decoded": " Esta mensagem e confidencial"+ },+ {+ "start": 5,+ "size": 5,+ "decoded": "."+ }+ ],+ "with_specials": {+ "ids": [+ 2,+ 22974,+ 70816,+ 526,+ 2103,+ 45972,+ 235265,+ 1+ ],+ "skipping_specials": " Esta mensagem e confidencial.",+ "kept": "<bos> Esta mensagem e confidencial.<eos>"+ }+ }+ ]+ }+ },+ "mixed_token": {+ "spec": {+ "version": "1.0",+ "truncation": null,+ "padding": null,+ "added_tokens": [],+ "normalizer": null,+ "pre_tokenizer": {+ "type": "ByteLevel",+ "add_prefix_space": true,+ "trim_offsets": true,+ "use_regex": true+ },+ "post_processor": null,+ "decoder": {+ "type": "ByteLevel",+ "add_prefix_space": true,+ "trim_offsets": true,+ "use_regex": true+ },+ "model": {+ "type": "BPE",+ "dropout": null,+ "unk_token": null,+ "continuing_subword_prefix": null,+ "end_of_word_suffix": null,+ "fuse_unk": false,+ "byte_fallback": false,+ "ignore_merges": true,+ "vocab": {+ "!": 0,+ "\"": 1,+ "#": 2,+ "$": 3,+ "%": 4,+ "&": 5,+ "'": 6,+ "(": 7,+ ")": 8,+ "*": 9,+ "+": 10,+ ",": 11,+ "-": 12,+ ".": 13,+ "/": 14,+ "0": 15,+ "1": 16,+ "2": 17,+ "3": 18,+ "4": 19,+ "5": 20,+ "6": 21,+ "7": 22,+ "8": 23,+ "9": 24,+ ":": 25,+ ";": 26,+ "<": 27,+ "=": 28,+ ">": 29,+ "?": 30,+ "@": 31,+ "A": 32,+ "B": 33,+ "C": 34,+ "D": 35,+ "E": 36,+ "F": 37,+ "G": 38,+ "H": 39,+ "I": 40,+ "J": 41,+ "K": 42,+ "L": 43,+ "M": 44,+ "N": 45,+ "O": 46,+ "P": 47,+ "Q": 48,+ "R": 49,+ "S": 50,+ "T": 51,+ "U": 52,+ "V": 53,+ "W": 54,+ "X": 55,+ "Y": 56,+ "Z": 57,+ "[": 58,+ "\\": 59,+ "]": 60,+ "^": 61,+ "_": 62,+ "`": 63,+ "a": 64,+ "b": 65,+ "c": 66,+ "d": 67,+ "e": 68,+ "f": 69,+ "g": 70,+ "h": 71,+ "i": 72,+ "j": 73,+ "k": 74,+ "l": 75,+ "m": 76,+ "n": 77,+ "o": 78,+ "p": 79,+ "q": 80,+ "r": 81,+ "s": 82,+ "t": 83,+ "u": 84,+ "v": 85,+ "w": 86,+ "x": 87,+ "y": 88,+ "z": 89,+ "{": 90,+ "|": 91,+ "}": 92,+ "~": 93,+ "¡": 94,+ "¢": 95,+ "£": 96,+ "¤": 97,+ "¥": 98,+ "¦": 99,+ "§": 100,+ "¨": 101,+ "©": 102,+ "ª": 103,+ "«": 104,+ "¬": 105,+ "®": 106,+ "¯": 107,+ "°": 108,+ "±": 109,+ "²": 110,+ "³": 111,+ "´": 112,+ "µ": 113,+ "¶": 114,+ "·": 115,+ "¸": 116,+ "¹": 117,+ "º": 118,+ "»": 119,+ "¼": 120,+ "½": 121,+ "¾": 122,+ "¿": 123,+ "À": 124,+ "Á": 125,+ "Â": 126,+ "Ã": 127,+ "Ä": 128,+ "Å": 129,+ "Æ": 130,+ "Ç": 131,+ "È": 132,+ "É": 133,+ "Ê": 134,+ "Ë": 135,+ "Ì": 136,+ "Í": 137,+ "Î": 138,+ "Ï": 139,+ "Ð": 140,+ "Ñ": 141,+ "Ò": 142,+ "Ó": 143,+ "Ô": 144,+ "Õ": 145,+ "Ö": 146,+ "×": 147,+ "Ø": 148,+ "Ù": 149,+ "Ú": 150,+ "Û": 151,+ "Ü": 152,+ "Ý": 153,+ "Þ": 154,+ "ß": 155,+ "à": 156,+ "á": 157,+ "â": 158,+ "ã": 159,+ "ä": 160,+ "å": 161,+ "æ": 162,+ "ç": 163,+ "è": 164,+ "é": 165,+ "ê": 166,+ "ë": 167,+ "ì": 168,+ "í": 169,+ "î": 170,+ "ï": 171,+ "ð": 172,+ "ñ": 173,+ "ò": 174,+ "ó": 175,+ "ô": 176,+ "õ": 177,+ "ö": 178,+ "÷": 179,+ "ø": 180,+ "ù": 181,+ "ú": 182,+ "û": 183,+ "ü": 184,+ "ý": 185,+ "þ": 186,+ "ÿ": 187,+ "Ā": 188,+ "ā": 189,+ "Ă": 190,+ "ă": 191,+ "Ą": 192,+ "ą": 193,+ "Ć": 194,+ "ć": 195,+ "Ĉ": 196,+ "ĉ": 197,+ "Ċ": 198,+ "ċ": 199,+ "Č": 200,+ "č": 201,+ "Ď": 202,+ "ď": 203,+ "Đ": 204,+ "đ": 205,+ "Ē": 206,+ "ē": 207,+ "Ĕ": 208,+ "ĕ": 209,+ "Ė": 210,+ "ė": 211,+ "Ę": 212,+ "ę": 213,+ "Ě": 214,+ "ě": 215,+ "Ĝ": 216,+ "ĝ": 217,+ "Ğ": 218,+ "ğ": 219,+ "Ġ": 220,+ "ġ": 221,+ "Ģ": 222,+ "ģ": 223,+ "Ĥ": 224,+ "ĥ": 225,+ "Ħ": 226,+ "ħ": 227,+ "Ĩ": 228,+ "ĩ": 229,+ "Ī": 230,+ "ī": 231,+ "Ĭ": 232,+ "ĭ": 233,+ "Į": 234,+ "į": 235,+ "İ": 236,+ "ı": 237,+ "IJ": 238,+ "ij": 239,+ "Ĵ": 240,+ "ĵ": 241,+ "Ķ": 242,+ "ķ": 243,+ "ĸ": 244,+ "Ĺ": 245,+ "ĺ": 246,+ "Ļ": 247,+ "ļ": 248,+ "Ľ": 249,+ "ľ": 250,+ "Ŀ": 251,+ "ŀ": 252,+ "Ł": 253,+ "ł": 254,+ "Ń": 255,+ "Ġ ": 256,+ "Ġhi": 257,+ " ": 258+ },+ "merges": []+ }+ },+ "separating_token": "Ġ ",+ "scope": "token",+ "cases": [+ {+ "why": "the mixed token alone: the whole point",+ "ids": [+ 256+ ],+ "decoded": "Ġ "+ },+ {+ "why": "entirely in the alphabet",+ "ids": [+ 257+ ],+ "decoded": " hi"+ },+ {+ "why": "entirely literal",+ "ids": [+ 258+ ],+ "decoded": " "+ },+ {+ "why": "a mixed token beside a plain one",+ "ids": [+ 256,+ 257+ ],+ "decoded": "Ġ hi"+ }+ ]+ },+ "synthetic_decoders": {+ "strip": {+ "why": "Strip, which no shipped checkpoint declares",+ "spec": {+ "version": "1.0",+ "truncation": null,+ "padding": null,+ "added_tokens": [],+ "normalizer": null,+ "pre_tokenizer": null,+ "post_processor": null,+ "model": {+ "type": "BPE",+ "dropout": null,+ "unk_token": null,+ "continuing_subword_prefix": null,+ "end_of_word_suffix": null,+ "fuse_unk": false,+ "byte_fallback": true,+ "ignore_merges": true,+ "vocab": {+ " hello ": 0,+ " x": 1,+ " ": 2,+ "no": 3+ },+ "merges": []+ },+ "decoder": {+ "type": "Strip",+ "content": " ",+ "start": 2,+ "stop": 1+ }+ },+ "crate_panics": [+ [+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "cases": [+ {+ "ids": [+ 0+ ],+ "decoded": "hello"+ },+ {+ "ids": [+ 1+ ],+ "decoded": "x"+ },+ {+ "ids": [+ 3+ ],+ "decoded": "no"+ },+ {+ "ids": [+ 0,+ 3+ ],+ "decoded": "hellono"+ }+ ]+ },+ "bytefallback_hex_case": {+ "why": "which byte-token spellings the crate reads as a byte",+ "spec": {+ "version": "1.0",+ "truncation": null,+ "padding": null,+ "added_tokens": [],+ "normalizer": null,+ "pre_tokenizer": null,+ "post_processor": null,+ "model": {+ "type": "BPE",+ "dropout": null,+ "unk_token": null,+ "continuing_subword_prefix": null,+ "end_of_word_suffix": null,+ "fuse_unk": false,+ "byte_fallback": true,+ "ignore_merges": true,+ "vocab": {+ "<0xFF>": 0,+ "<0xff>": 1,+ "<0x41>": 2,+ "<0x61>": 3,+ "a": 4,+ "<0x+5>": 5,+ "<0x 5>": 6,+ "<0XFF>": 7,+ "<0xfF>": 8,+ "<0xF3>": 9,+ "<0xB0>": 10,+ "<0x80>": 11+ },+ "merges": []+ },+ "decoder": {+ "type": "ByteFallback"+ }+ },+ "cases": [+ {+ "ids": [+ 2+ ],+ "decoded": "A"+ },+ {+ "ids": [+ 3+ ],+ "decoded": "a"+ },+ {+ "ids": [+ 1+ ],+ "decoded": "�"+ },+ {+ "ids": [+ 0+ ],+ "decoded": "�"+ },+ {+ "ids": [+ 2,+ 3+ ],+ "decoded": "Aa"+ },+ {+ "ids": [+ 0,+ 0+ ],+ "decoded": "��"+ },+ {+ "ids": [+ 1,+ 4+ ],+ "decoded": "�a"+ },+ {+ "ids": [+ 5+ ],+ "decoded": "\u0005"+ },+ {+ "ids": [+ 6+ ],+ "decoded": "<0x 5>"+ },+ {+ "ids": [+ 7+ ],+ "decoded": "<0XFF>"+ },+ {+ "ids": [+ 8+ ],+ "decoded": "�"+ },+ {+ "ids": [+ 5,+ 4+ ],+ "decoded": "\u0005a"+ },+ {+ "ids": [+ 6,+ 4+ ],+ "decoded": "<0x 5>a"+ },+ {+ "ids": [+ 7,+ 4+ ],+ "decoded": "<0XFF>a"+ },+ {+ "ids": [+ 9,+ 10+ ],+ "decoded": "��"+ },+ {+ "ids": [+ 9,+ 10,+ 11+ ],+ "decoded": "���"+ },+ {+ "ids": [+ 9,+ 10,+ 11,+ 11+ ],+ "decoded": ""+ },+ {+ "ids": [+ 4,+ 9,+ 10,+ 4+ ],+ "decoded": "a��a"+ }+ ]+ },+ "no_decoder": {+ "why": "no decoder section: the separator the crate falls back to",+ "spec": {+ "version": "1.0",+ "truncation": null,+ "padding": null,+ "added_tokens": [],+ "normalizer": null,+ "pre_tokenizer": null,+ "post_processor": null,+ "model": {+ "type": "BPE",+ "dropout": null,+ "unk_token": null,+ "continuing_subword_prefix": null,+ "end_of_word_suffix": null,+ "fuse_unk": false,+ "byte_fallback": true,+ "ignore_merges": true,+ "vocab": {+ "ab": 0,+ "cd": 1,+ "ef": 2,+ " ": 3+ },+ "merges": []+ }+ },+ "cases": [+ {+ "ids": [+ 0+ ],+ "decoded": "ab"+ },+ {+ "ids": [+ 0,+ 1+ ],+ "decoded": "ab cd"+ },+ {+ "ids": [+ 0,+ 1,+ 2+ ],+ "decoded": "ab cd ef"+ },+ {+ "ids": [+ 0,+ 3,+ 1+ ],+ "decoded": "ab cd"+ },+ {+ "ids": [+ 3,+ 3+ ],+ "decoded": " "+ }+ ]+ },+ "nested_sequence": {+ "why": "a Sequence nested in a Sequence keeps its pieces",+ "spec": {+ "version": "1.0",+ "truncation": null,+ "padding": null,+ "added_tokens": [],+ "normalizer": null,+ "pre_tokenizer": null,+ "post_processor": null,+ "model": {+ "type": "BPE",+ "dropout": null,+ "unk_token": null,+ "continuing_subword_prefix": null,+ "end_of_word_suffix": null,+ "fuse_unk": false,+ "byte_fallback": true,+ "ignore_merges": true,+ "vocab": {+ "▁Hello": 0,+ "▁world": 1,+ "▁": 2,+ "▁a": 3+ },+ "merges": []+ },+ "decoder": {+ "type": "Sequence",+ "decoders": [+ {+ "type": "Sequence",+ "decoders": [+ {+ "type": "Replace",+ "pattern": {+ "String": "▁"+ },+ "content": " "+ },+ {+ "type": "ByteFallback"+ }+ ]+ },+ {+ "type": "Strip",+ "content": " ",+ "start": 1,+ "stop": 0+ }+ ]+ }+ },+ "cases": [+ {+ "ids": [+ 0,+ 1+ ],+ "decoded": "Helloworld"+ },+ {+ "ids": [+ 2,+ 2,+ 0+ ],+ "decoded": "Hello"+ },+ {+ "ids": [+ 2,+ 3,+ 2+ ],+ "decoded": "a"+ },+ {+ "ids": [+ 0+ ],+ "decoded": "Hello"+ },+ {+ "ids": [+ 3+ ],+ "decoded": "a"+ }+ ]+ },+ "bytefallback_piece_boundaries": {+ "why": "a failed byte run is one piece per byte, not one piece of N",+ "spec": {+ "version": "1.0",+ "truncation": null,+ "padding": null,+ "added_tokens": [],+ "normalizer": null,+ "pre_tokenizer": null,+ "post_processor": null,+ "model": {+ "type": "BPE",+ "dropout": null,+ "unk_token": null,+ "continuing_subword_prefix": null,+ "end_of_word_suffix": null,+ "fuse_unk": false,+ "byte_fallback": true,+ "ignore_merges": true,+ "vocab": {+ "<0xF3>": 0,+ "<0xB0>": 1,+ "<0x80>": 2,+ "hi": 3+ },+ "merges": []+ },+ "decoder": {+ "type": "Sequence",+ "decoders": [+ {+ "type": "ByteFallback"+ },+ {+ "type": "Replace",+ "pattern": {+ "String": "��"+ },+ "content": "X"+ }+ ]+ }+ },+ "cases": [+ {+ "ids": [+ 0,+ 1+ ],+ "decoded": "��"+ },+ {+ "ids": [+ 0,+ 1,+ 2+ ],+ "decoded": "���"+ },+ {+ "ids": [+ 3,+ 0,+ 1,+ 3+ ],+ "decoded": "hi��hi"+ },+ {+ "ids": [+ 0,+ 1,+ 2,+ 2+ ],+ "decoded": ""+ },+ {+ "ids": [+ 3+ ],+ "decoded": "hi"+ }+ ]+ }+ }+}diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/PreTokenizer.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/PreTokenizer.javaindex b77de0a2..3e134873 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/PreTokenizer.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/PreTokenizer.java@@ -32,6 +32,26 @@ abstract class PreTokenizer {/** Cuts one span of normalized text into pieces for the BPE stage. */abstract List<String> preTokenize(String text);+ /**+ * The byte a ByteLevel stand-in character represents, or -1 when it is not one of the 256.+ *+ * <p>Exposed on the enclosing class because {@link TokenDecoder} needs the inverse mapping and+ * {@code ByteLevel} is private here. One table, inverted once -- a decoder with its own copy+ * could disagree in one position and produce plausible text with one character wrong.+ */+ static int byteForChar(char c) {+ return ByteLevel.byteForChar(c);+ }++ /**+ * The stand-in character for a byte, which is what {@link #byteForChar} inverts.+ *+ * @param b a byte value, 0 to 255+ */+ static char charForByte(int b) {+ return ByteLevel.charForByte(b);+ }+/** Builds a pre-tokenizer from a checkpoint's {@code pre_tokenizer} node, which may be null. */static PreTokenizer from(Map<String, Object> node) {if (node == null) {@@ -127,6 +147,48 @@ abstract class PreTokenizer {this.useRegex = useRegex;}+ /**+ * The stand-in character for a byte.+ *+ * @param b a byte value, 0 to 255+ * @throws IllegalArgumentException for anything outside that range, rather than the+ * {@code ArrayIndexOutOfBoundsException} a bare index would raise+ */+ static char charForByte(int b) {+ if (b < 0 || b >= BYTE_TO_CHAR.length) {+ throw new IllegalArgumentException(+ "a byte is 0 to 255, got " + b);+ }+ return BYTE_TO_CHAR[b];+ }++ /**+ * The byte a stand-in character represents, or -1 when it is not one of the 256.+ *+ * <p>Built by inverting {@link #BYTE_TO_CHAR} rather than by a second construction, so the+ * two cannot disagree: a decode that mapped one character to the wrong byte would produce+ * plausible text with one character wrong, and the window it belongs to would re-tokenize+ * to different ids with nothing to show why.+ */+ static int byteForChar(char c) {+ return c < CHAR_TO_BYTE.length ? CHAR_TO_BYTE[c] : -1;+ }++ private static final int[] CHAR_TO_BYTE = buildCharToByte();++ private static int[] buildCharToByte() {+ int highest = 0;+ for (char c : BYTE_TO_CHAR) {+ highest = Math.max(highest, c);+ }+ int[] table = new int[highest + 1];+ java.util.Arrays.fill(table, -1);+ for (int b = 0; b < BYTE_TO_CHAR.length; b++) {+ table[BYTE_TO_CHAR[b]] = b;+ }+ return table;+ }+/*** The GPT-2 alphabet, built rather than transcribed.*diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/TokenDecoder.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/TokenDecoder.javanew file mode 100644index 00000000..2b9c8002--- /dev/null+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/TokenDecoder.java@@ -0,0 +1,514 @@+package com.convaiinnovations.laya.tokenizer;++import com.convaiinnovations.laya.json.Json;+import java.io.ByteArrayOutputStream;+import java.nio.charset.CharacterCodingException;+import java.nio.charset.CharsetDecoder;+import java.nio.charset.CodingErrorAction;+import java.nio.charset.StandardCharsets;+import java.nio.ByteBuffer;+import java.util.ArrayList;+import java.util.List;+import java.util.Map;++/**+ * The inverse of tokenization: a list of token strings back into text.+ *+ * <p>Needed because {@code predictLong} scans a long state in overlapping token windows and hands+ * each window back to {@code predictBatch} as TEXT, so every window has to be decoded. It is also+ * worth having on its own -- {@code laya-ts} and {@code laya-dotnet} both expose a decode -- but+ * the windowing is what made it necessary.+ *+ * <p>Unlike the pre-tokenizer, this has no Python reference to port: decoding lives in the Rust+ * {@code tokenizers} crate, so the rules below were probed out of it against the two shipped+ * checkpoints, and the recorded round-trips in {@code fixtures/decode_text.json} are the contract.+ *+ * <h2>Decoding is not lossless, and that is the contract</h2>+ *+ * <p>On the english checkpoint it happens to round-trip. On the multilingual one it does not, and+ * the port must reproduce the loss exactly rather than improve on it:+ *+ * <pre>+ * english "Hello world" -> "Hello world"+ * multilingual "Hello world" -> " Hello world" <- a leading space appears+ * multilingual "a\tb\nc" -> " a\tb\n c" <- and one after the newline+ * </pre>+ *+ * <p>That is Metaspace: its pre-tokenizer prepends U+2581 to the text ({@code prepend_scheme:+ * always}), and the decoder turns every U+2581 back into a space, including the prepended one. The+ * reference lives with it -- {@code predict_long}'s own comment says the 50% window overlap+ * absorbs the boundary drift -- so a port that trimmed the space would re-tokenize every window+ * differently from the reference and diverge on every answer downstream.+ */+abstract class TokenDecoder {++ /** The decoded text for these tokens, in order. */+ abstract String decode(List<String> tokens);++ /**+ * The decoder a {@code tokenizer.json} declares, or a plain join when it declares none.+ *+ * <p>A checkpoint with no {@code decoder} section is not an error: the crate joins the tokens+ * with a space, and refusing to load would reject a valid tokenizer over a decode path a+ * caller may never use. See {@link Join} for the separator, which this got wrong.+ */+ static TokenDecoder from(Map<String, Object> node) {+ if (node == null) {+ return new Join();+ }+ String type = Json.string(node, "type", null);+ if (type == null) {+ throw new Json.JsonException("a decoder node must declare a type");+ }+ switch (type) {+ case "ByteLevel":+ return new ByteLevel();+ case "Replace":+ return new Replace(node);+ case "ByteFallback":+ return new ByteFallback();+ case "Fuse":+ return new Fuse();+ case "Strip":+ return new Strip(node);+ case "Sequence": {+ Object steps = node.get("decoders");+ if (!(steps instanceof List)) {+ throw new Json.JsonException("a Sequence decoder needs a decoders array");+ }+ List<TokenDecoder> parts = new ArrayList<>();+ for (Object step : (List<?>) steps) {+ if (!(step instanceof Map)) {+ throw new Json.JsonException("every Sequence decoder step must be an object");+ }+ @SuppressWarnings("unchecked")+ Map<String, Object> stepNode = (Map<String, Object>) step;+ parts.add(from(stepNode));+ }+ return new Sequence(parts);+ }+ default:+ // Refusing beats guessing. A WordPiece or Metaspace decoder would produce+ // plausible-looking text with the wrong spaces, and the windows predictLong builds+ // from it would re-tokenize to different ids with nothing to show it had happened.+ throw new Json.JsonException(+ "laya-java implements the ByteLevel, Replace, ByteFallback, Fuse, Strip and"+ + " Sequence decoders; this checkpoint declares \"" + type + "\". Refusing"+ + " to load rather than decode it wrongly.");+ }+ }++ // ------------------------------------------------------------------ the steps++ /**+ * No decoder declared: the tokens joined with a SPACE.+ *+ * <p>A space, not nothing. The crate's fallback for an absent decoder is+ * {@code tokens.join(" ")}, and this joined with nothing -- wrong for every multi-token+ * decode, not for an edge case. Measured on the multilingual vocabulary with its+ * {@code decoder} key removed: ids {@code [25957, 2134]} give {@code "\u2581Hello+ * \u2581world"}, with a space between the pieces.+ *+ * <p>Unreachable from either shipped checkpoint, since both declare a decoder. That is the+ * reason it was wrong and stayed wrong: this is the one branch of {@code from} that is+ * permissive rather than refusing, so it is where a third-party checkpoint lands, and the+ * recorded fixtures can never reach it. It has its own test for that reason.+ */+ private static final class Join extends TokenDecoder {+ @Override+ String decode(List<String> tokens) {+ return String.join(" ", tokens);+ }+ }++ /**+ * The inverse of the ByteLevel pre-tokenizer: every stand-in character is one byte.+ *+ * <p>The bytes are then UTF-8, and a token boundary can fall INSIDE a multi-byte character --+ * CJK is three bytes and routinely splits across two tokens -- so the whole token list is+ * concatenated to bytes first and decoded once at the end. Decoding token by token would turn+ * every split character into replacement characters.+ */+ private static final class ByteLevel extends TokenDecoder {+ @Override+ String decode(List<String> tokens) {+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();+ for (String token : tokens) {+ // Whole-token fallback, not per character. The crate folds over a token's+ // characters and, if ANY of them is outside the stand-in alphabet, uses the+ // token's own UTF-8 bytes for the WHOLE token. Falling back per character+ // coincides on a token that is entirely literal -- which is the only kind this+ // vocabulary has -- and would differ on a mixed one, so the rule is written as+ // the crate writes it rather than as the corpus happens to need.+ byte[] mapped = mapAll(token);+ if (mapped == null) {+ byte[] raw = token.getBytes(StandardCharsets.UTF_8);+ bytes.write(raw, 0, raw.length);+ continue;+ }+ bytes.write(mapped, 0, mapped.length);+ }+ return lossyUtf8(bytes.toByteArray());+ }++ /** Every character as its byte, or null when one of them is not a stand-in. */+ private static byte[] mapAll(String token) {+ byte[] out = new byte[token.length()];+ for (int i = 0; i < token.length(); i++) {+ int b = PreTokenizer.byteForChar(token.charAt(i));+ if (b < 0) {+ return null;+ }+ out[i] = (byte) b;+ }+ return out;+ }+++ }++ /** {@code Replace}: a literal substring for another, applied to every token. */+ private static final class Replace extends TokenDecoder {++ private final String pattern;+ private final String content;++ Replace(Map<String, Object> node) {+ Object patternNode = node.get("pattern");+ String literal = null;+ if (patternNode instanceof Map) {+ @SuppressWarnings("unchecked")+ Map<String, Object> p = (Map<String, Object>) patternNode;+ literal = Json.string(p, "String", null);+ if (literal == null && p.containsKey("Regex")) {+ // A regex Replace would need the crate's regex dialect, and getting it subtly+ // wrong moves spaces in decoded text. None of the shipped checkpoints uses+ // one, so this refuses rather than approximating.+ throw new Json.JsonException(+ "laya-java implements only a literal String pattern in a Replace"+ + " decoder; this checkpoint declares a Regex one");+ }+ } else if (patternNode instanceof String) {+ literal = (String) patternNode;+ }+ if (literal == null) {+ throw new Json.JsonException("a Replace decoder needs a String pattern");+ }+ this.pattern = literal;+ this.content = Json.string(node, "content", "");+ }++ @Override+ String decode(List<String> tokens) {+ StringBuilder out = new StringBuilder();+ for (String token : tokens) {+ out.append(token.replace(pattern, content));+ }+ return out.toString();+ }++ @Override+ List<String> transform(List<String> tokens) {+ // One piece per token, NOT one joined string: ByteFallback runs after this in the+ // multilingual checkpoint's Sequence and needs the boundaries to find a byte run.+ List<String> out = new ArrayList<>(tokens.size());+ for (String token : tokens) {+ out.add(token.replace(pattern, content));+ }+ return out;+ }+ }++ /**+ * {@code ByteFallback}: a run of {@code <0xNN>} tokens is one UTF-8 sequence.+ *+ * <p>Reachable on the shipped multilingual checkpoint, which declares+ * {@code byte_fallback: true} and carries all 255 {@code <0xNN>} tokens. U+F0000 encodes as+ * four of them -- {@code <0xF3> <0xB0> <0x80> <0x80>} -- so a run has to be gathered and+ * decoded together; one at a time gives four replacement characters.+ */+ private static final class ByteFallback extends TokenDecoder {+ @Override+ String decode(List<String> tokens) {+ StringBuilder out = new StringBuilder();+ ByteArrayOutputStream run = new ByteArrayOutputStream();+ for (String token : tokens) {+ int b = byteToken(token);+ if (b >= 0) {+ run.write(b);+ continue;+ }+ flush(run, out);+ out.append(token);+ }+ flush(run, out);+ return out.toString();+ }++ @Override+ List<String> transform(List<String> tokens) {+ // A run that decodes is one piece; a run that does not becomes one piece PER BYTE.+ // The crate pushes U+FFFD into its token list once per byte it could not use, and a+ // later step in the Sequence sees those boundaries. Measured with+ // `Sequence[ByteFallback, Replace("\ufffd\ufffd" -> "X")]` over the two bytes of a+ // cut U+F0000: the crate leaves the pair alone because the pieces are separate, and+ // emitting them as one piece of two characters matched the pattern and gave "X".+ //+ // Invisible on the shipped multilingual checkpoint because Fuse is the next step and+ // re-joins everything, so the strings agree. The override exists to preserve+ // boundaries, and this was the one shape where it did not.+ List<String> out = new ArrayList<>();+ ByteArrayOutputStream run = new ByteArrayOutputStream();+ for (String token : tokens) {+ int b = byteToken(token);+ if (b >= 0) {+ run.write(b);+ continue;+ }+ flushPieces(run, out);+ out.add(token);+ }+ flushPieces(run, out);+ return out;+ }++ private static void flushPieces(ByteArrayOutputStream run, List<String> out) {+ if (run.size() == 0) {+ return;+ }+ byte[] bytes = run.toByteArray();+ run.reset();+ String text = strictUtf8OrNull(bytes);+ if (text != null) {+ out.add(text);+ return;+ }+ for (int i = 0; i < bytes.length; i++) {+ out.add("\ufffd");+ }+ }++ private static void flush(ByteArrayOutputStream run, StringBuilder out) {+ if (run.size() > 0) {+ out.append(strictOrReplacementPerByte(run.toByteArray()));+ run.reset();+ }+ }+ }++ /** {@code Fuse}: concatenate the pieces into one string. */+ private static final class Fuse extends TokenDecoder {+ @Override+ String decode(List<String> tokens) {+ StringBuilder out = new StringBuilder();+ for (String token : tokens) {+ out.append(token);+ }+ return out.toString();+ }+ }++ /** {@code Strip}: remove up to n copies of a character from each end of every token. */+ private static final class Strip extends TokenDecoder {++ private final char content;+ private final int start;+ private final int stop;++ Strip(Map<String, Object> node) {+ String c = Json.string(node, "content", " ");+ if (c.length() != 1) {+ throw new Json.JsonException(+ "a Strip decoder's content must be one character, got \"" + c + "\"");+ }+ this.content = c.charAt(0);+ this.start = Json.integer(node, "start", 0);+ this.stop = Json.integer(node, "stop", 0);+ }++ @Override+ String decode(List<String> tokens) {+ StringBuilder out = new StringBuilder();+ for (String token : tokens) {+ out.append(apply(token));+ }+ return out.toString();+ }++ @Override+ List<String> transform(List<String> tokens) {+ List<String> out = new ArrayList<>(tokens.size());+ for (String token : tokens) {+ out.add(apply(token));+ }+ return out;+ }++ String apply(String token) {+ int from = 0;+ int to = token.length();+ for (int i = 0; i < start && from < to && token.charAt(from) == content; i++) {+ from++;+ }+ for (int i = 0; i < stop && to > from && token.charAt(to - 1) == content; i++) {+ to--;+ }+ return token.substring(from, to);+ }+ }++ /**+ * {@code Sequence}: each step transforms the token LIST, not the joined string.+ *+ * <p>That distinction is the whole behaviour of the multilingual checkpoint's+ * {@code Sequence[Replace, ByteFallback, Fuse]}. Replace rewrites each token, ByteFallback+ * needs to see token boundaries to find a {@code <0xNN>} run, and only Fuse joins. Running the+ * steps over an already-joined string would make ByteFallback unable to tell a literal+ * {@code "<0x41>"} in the text from a byte token.+ */+ private static final class Sequence extends TokenDecoder {++ private final List<TokenDecoder> steps;++ Sequence(List<TokenDecoder> steps) {+ this.steps = List.copyOf(steps);+ }++ @Override+ String decode(List<String> tokens) {+ return String.join("", transform(tokens));+ }++ @Override+ List<String> transform(List<String> tokens) {+ // The chained list, NOT one joined piece. Without this override a Sequence inherits+ // the base default, which re-joins -- so a Sequence NESTED in a Sequence handed one+ // string to the outer sequence's next step where the crate hands it N pieces. The+ // crate's `decode_chain` chains `decode_chain` through its steps and returns a list.+ //+ // Measured with `Sequence[Sequence[Replace, ByteFallback], Strip(" ", start=1)]`: the+ // crate gives "Helloworld", because Strip sees two pieces and takes the leading space+ // off each; re-joining gave "Hello world". Neither shipped checkpoint nests deeper+ // than one level, which is why nothing caught it and why it has its own test.+ List<String> current = new ArrayList<>(tokens);+ for (TokenDecoder step : steps) {+ current = step.transform(current);+ }+ return current;+ }+ }++ /**+ * This step's effect on a token list, which is what a {@code Sequence} composes.+ *+ * <p>The default is "whatever this decoder produces, as one piece". The steps that have to+ * preserve boundaries for a later step override it.+ */+ List<String> transform(List<String> tokens) {+ return List.of(decode(tokens));+ }++ // ------------------------------------------------------------------ helpers++ /** The byte a {@code <0xNN>} token stands for, or -1 when the token is not one. */+ private static int byteToken(String token) {+ if (token == null || token.length() != 6+ || token.charAt(0) != '<' || token.charAt(1) != '0' || token.charAt(2) != 'x'+ || token.charAt(5) != '>') {+ return -1;+ }+ return parseByte(token.charAt(3), token.charAt(4));+ }++ /**+ * The two characters between {@code <0x} and {@code >} as a byte, following Rust's+ * {@code u8::from_str_radix(.., 16)} -- which is the function the crate calls.+ *+ * <p>Case-INSENSITIVE, and a leading {@code +} is accepted. Both were measured, after this+ * rejected lowercase on the stated grounds that "the crate writes these uppercase, and+ * accepting both would make a literal {@code <0xff>} in ordinary text decode as a byte". That+ * second clause describes what the crate actually does, rather than a hazard it avoids: asked+ * to decode a vocabulary whose only token is {@code <0xff>}, the crate returns U+FFFD -- it+ * read the byte -- and the same for {@code <0xfF>}. {@code <0x+5>} is byte 0x05.+ *+ * <p>{@code <0x 5>} and {@code <0XFF>} stay literal, because a space is not a digit and the+ * {@code <0x} prefix is matched case-sensitively. Both measured too, so the boundary is+ * recorded rather than assumed in the other direction this time.+ *+ * <p>No shipped vocabulary contains a lowercase byte token -- all 255 in the multilingual+ * checkpoint are uppercase -- so this was not a live bug. It was a wrong rule with a+ * confident comment, which is worse to leave in place than a wrong rule without one.+ */+ private static int parseByte(char high, char low) {+ if (high == '+') {+ return hex(low);+ }+ int h = hex(high);+ int l = hex(low);+ return h < 0 || l < 0 ? -1 : (h << 4) | l;+ }++ private static int hex(char c) {+ if (c >= '0' && c <= '9') {+ return c - '0';+ }+ if (c >= 'A' && c <= 'F') {+ return c - 'A' + 10;+ }+ if (c >= 'a' && c <= 'f') {+ return c - 'a' + 10;+ }+ return -1;+ }++ /**+ * UTF-8 with replacement, as the crate's lossy decode does.+ *+ * <p>Not {@code new String(bytes, UTF_8)}, which is also lossy but has differed from Rust's+ * {@code from_utf8_lossy} on how many replacement characters a truncated sequence produces.+ * A window boundary cuts multi-byte characters routinely, so this is a path that runs, not an+ * edge case: the decoder is configured explicitly to REPLACE rather than report, and one+ * replacement per malformed input sequence is what both produce.+ */+ private static String lossyUtf8(byte[] bytes) {+ CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder()+ .onMalformedInput(CodingErrorAction.REPLACE)+ .onUnmappableCharacter(CodingErrorAction.REPLACE);+ try {+ return decoder.decode(ByteBuffer.wrap(bytes)).toString();+ } catch (CharacterCodingException impossible) {+ throw new IllegalStateException("a REPLACE decoder cannot report an error", impossible);+ }+ }++ /**+ * {@code ByteFallback}'s rule, which is NOT lossy decoding: strict UTF-8, and on failure one+ * U+FFFD per BYTE of the run.+ *+ * <p>Measured, after assuming otherwise cost 629 of 201,279 window slices. A window boundary+ * cuts U+F0000 after two of its four bytes, leaving {@code F3 B0}; lossy decoding calls that+ * one maximal subpart and emits ONE replacement character, and the crate emits TWO -- one for+ * each byte it could not use. The two rules agree only when the run is valid, which is every+ * case that does not straddle a boundary, which is why this looked right until the slices were+ * compared.+ *+ * <p>The sibling {@code ByteLevel} decoder really does use lossy decoding, so the two paths in+ * this file deliberately differ. That is the crate's shape, not an oversight.+ */+ private static String strictOrReplacementPerByte(byte[] bytes) {+ String text = strictUtf8OrNull(bytes);+ return text != null ? text : "\ufffd".repeat(bytes.length);+ }++ /** The run as strict UTF-8, or null when it is not valid UTF-8. */+ private static String strictUtf8OrNull(byte[] bytes) {+ CharsetDecoder strict = StandardCharsets.UTF_8.newDecoder()+ .onMalformedInput(CodingErrorAction.REPORT)+ .onUnmappableCharacter(CodingErrorAction.REPORT);+ try {+ return strict.decode(ByteBuffer.wrap(bytes)).toString();+ } catch (CharacterCodingException invalid) {+ return null;+ }+ }+}diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/Tokenizer.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/Tokenizer.javaindex 4015fe35..548743c5 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/Tokenizer.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/tokenizer/Tokenizer.java@@ -42,6 +42,9 @@ public final class Tokenizer {private final Map<String, Integer> specials;private final Map<String, Integer> roles;private final String maskToken;+ private final TokenDecoder decoder;+ /** Ids the reference's decode drops unless asked not to; see {@link #decode(int[])}. */+ private final java.util.Set<Integer> specialIds;/** The roles a checkpoint names in {@code tokenizer_config.json}, in sequence-building order. */private static final String[] ROLE_KEYS =@@ -136,6 +139,7 @@ public final class Tokenizer {this.normalizer = Normalizer.from(objectOrNull(document, "normalizer"));this.preTokenizer = PreTokenizer.from(objectOrNull(document, "pre_tokenizer"));+ this.decoder = TokenDecoder.from(objectOrNull(document, "decoder"));// --- the id map: model.vocab UNION added_tokens, with added_tokens authoritative.//@@ -153,6 +157,17 @@ public final class Tokenizer {}List<AddedToken> added = readAddedTokens(document);+ // The ids a decode drops by default. Taken from each added token's own `special` flag+ // rather than from the laya config's four roles: the english checkpoint declares 116+ // added tokens and the config names only cls/sep/mask/pad, so the roles would leave+ // every other control token in the decoded text.+ java.util.Set<Integer> special = new java.util.LinkedHashSet<>();+ for (AddedToken token : added) {+ if (token.special()) {+ special.add(token.id());+ }+ }+ this.specialIds = java.util.Set.copyOf(special);for (AddedToken token : added) {Integer existing = vocab.get(token.content());if (existing != null && existing != token.id()) {@@ -340,6 +355,50 @@ public final class Tokenizer {// ------------------------------------------------------------------ lookups/** The number of distinct ids this tokenizer can emit, counting added tokens. */+ /**+ * These ids back into text, dropping special tokens -- the reference's default.+ *+ * <p>Used by {@code predictLong}, which splits a long state into overlapping token windows and+ * hands each one back as text so it is re-tokenized as an ordinary state.+ *+ * <p><b>This does not round-trip, and is not meant to.</b> On the english checkpoint it+ * happens to; on the multilingual one {@code "Hello world"} decodes to {@code " Hello world"},+ * because Metaspace prepends U+2581 when encoding and the decoder turns every U+2581 back+ * into a space. That is the reference's behaviour and the port reproduces it exactly -- a+ * decode that trimmed the space would re-tokenize each window differently and move every+ * answer downstream.+ */+ public String decode(int[] ids) {+ return decode(ids, true);+ }++ /**+ * These ids back into text.+ *+ * @param ids the token ids, in order+ * @param skipSpecialTokens drop ids flagged {@code special} in {@code added_tokens}, which is+ * what the reference does unless told otherwise+ */+ public String decode(int[] ids, boolean skipSpecialTokens) {+ if (ids == null) {+ throw new IllegalArgumentException("ids must not be null");+ }+ List<String> pieces = new ArrayList<>(ids.length);+ for (int id : ids) {+ if (skipSpecialTokens && specialIds.contains(id)) {+ continue;+ }+ String token = idToToken(id);+ if (token == null) {+ // An id outside the vocabulary. The crate skips it; failing here would turn a+ // caller's off-by-one into an exception from a method whose job is to be lossy.+ continue;+ }+ pieces.add(token);+ }+ return decoder.decode(pieces);+ }+public int vocabSize() {return ids.size();}diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/tokenizer/TokenDecoderTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/tokenizer/TokenDecoderTest.javanew file mode 100644index 00000000..815a927a--- /dev/null+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/tokenizer/TokenDecoderTest.java@@ -0,0 +1,448 @@+package com.convaiinnovations.laya.tokenizer;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.junit.jupiter.api.DynamicTest.dynamicTest;++import com.convaiinnovations.laya.Fixtures;+import com.convaiinnovations.laya.json.Json;+import com.convaiinnovations.laya.json.PythonJson;+import java.io.IOException;+import java.nio.charset.StandardCharsets;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import org.junit.jupiter.api.Assumptions;+import org.junit.jupiter.api.DisplayName;+import org.junit.jupiter.api.DynamicTest;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.TestFactory;++/**+ * {@code Tokenizer.decode}, against expectations recorded from the reference tokenizer.+ *+ * <p>Decoding has no Python reference to port -- it lives in the Rust {@code tokenizers} crate --+ * so {@code fixtures/decode_text.json} is the only specification this port has, and the three+ * rules it pins were each wrong on the first attempt. The counts in the comments below are what+ * each wrong guess cost against the reference, so a later simplification has a number to argue+ * with rather than a preference.+ *+ * <p>The SLICE cases carry the weight. {@code predictLong} cuts a tokenized state into windows at+ * arbitrary offsets and decodes each one, so a boundary lands inside a multi-byte character+ * routinely -- and every whole-text decode agreed while 629 of 201,279 slices did not. A fixture+ * of whole texts would have shipped the bug.+ */+final class TokenDecoderTest {++ @SuppressWarnings("unchecked")+ private static Map<String, Object> family() {+ return (Map<String, Object>) (Map<?, ?>) Fixtures.load("decode_text.json");+ }++ @SuppressWarnings("unchecked")+ private static Map<String, Object> forCheckpoint(String name) {+ Map<String, Object> by = (Map<String, Object>) family().get("by");+ Map<String, Object> entry = (Map<String, Object>) by.get(name);+ Assumptions.assumeTrue(entry != null && !entry.containsKey("skipped"),+ "the " + name + " checkpoint was not present when the fixture was recorded");+ return entry;+ }++ private static Tokenizer open(String name) {+ Path root = Fixtures.checkpoint(name);+ Assumptions.assumeTrue(root != null, "set LAYA_CHECKPOINTS to run the decode parity tests");+ try {+ return Tokenizer.fromFile(root.resolve("tokenizer").resolve("tokenizer.json"));+ } catch (java.io.IOException problem) {+ throw new IllegalStateException("cannot open the " + name + " tokenizer", problem);+ }+ }++ @TestFactory+ @DisplayName("every recorded decode matches, whole and sliced, on both checkpoints")+ @SuppressWarnings("unchecked")+ List<DynamicTest> recordedDecodesMatch() {+ List<DynamicTest> tests = new ArrayList<>();+ int sliceCases = 0;+ for (String name : List.of("english", "multilingual")) {+ Map<String, Object> entry = forCheckpoint(name);+ List<Object> cases = (List<Object>) entry.get("cases");+ assertTrue(cases.size() >= 23,+ () -> "the decode corpus lost cases: " + cases.size() + " of at least 23");+ Tokenizer tok = open(name);+ for (Object raw : cases) {+ Map<String, Object> row = (Map<String, Object>) raw;+ String text = (String) row.get("text");+ List<Object> wantIds = (List<Object>) row.get("ids");+ String label = name + " " + describe(text);++ tests.add(dynamicTest(label + " ids", () -> {+ int[] ids = tok.encode(text);+ assertEquals(wantIds.size(), ids.length, "id count for " + describe(text));+ for (int i = 0; i < ids.length; i++) {+ assertEquals(((Number) wantIds.get(i)).intValue(), ids[i],+ "id " + i + " for " + describe(text));+ }+ }));++ tests.add(dynamicTest(label + " whole", () ->+ assertEquals(row.get("decoded"), tok.decode(ids(wantIds)),+ "decode diverged on " + describe(text))));++ List<Object> slices = (List<Object>) row.get("slices");+ sliceCases += slices.size();+ if (slices.isEmpty()) {+ // No slices means no ids, which is true of exactly one corpus text. Asserting+ // THAT is worth a test; registering a loop over nothing is not -- it passed+ // unconditionally and still counted toward the suite's floor.+ tests.add(dynamicTest(label + " has no ids to slice", () -> {+ assertTrue(text.isEmpty(),+ () -> "only the empty text should tokenize to nothing, but "+ + describe(text) + " did");+ assertEquals(0, ids(wantIds).length, "and it must have no ids");+ }));+ continue;+ }+ tests.add(dynamicTest(label + " slices (" + slices.size() + ")", () -> {+ int[] ids = ids(wantIds);+ for (Object s : slices) {+ Map<String, Object> slice = (Map<String, Object>) s;+ int start = ((Number) slice.get("start")).intValue();+ int size = ((Number) slice.get("size")).intValue();+ int end = Math.min(start + size, ids.length);+ int[] chunk = new int[end - start];+ System.arraycopy(ids, start, chunk, 0, chunk.length);+ assertEquals(slice.get("decoded"), tok.decode(chunk),+ () -> String.format("slice [%d,%d) of %s", start, end,+ describe(text)));+ }+ }));++ Map<String, Object> specials = (Map<String, Object>) row.get("with_specials");+ tests.add(dynamicTest(label + " specials", () -> {+ int[] ids = ids((List<Object>) specials.get("ids"));+ assertEquals(specials.get("skipping_specials"), tok.decode(ids),+ "decode must skip special tokens by default");+ assertEquals(specials.get("kept"), tok.decode(ids, false),+ "decode(ids, false) must keep them");+ }));+ }+ }+ // A floor on the slices specifically, because they are the cases that caught the+ // ByteFallback rule and a corpus that lost them would still look healthy.+ final int total = sliceCases;+ assertTrue(total >= 400,+ () -> "the slice cases are the point of this family and there are only " + total);+ return tests;+ }++ @Test+ @DisplayName("the fixture was recorded against the decoders this port implements")+ @SuppressWarnings("unchecked")+ void fixtureNamesItsDecoders() {+ // Without this, a fixture regenerated against a checkpoint whose decoder changed would be+ // asserted against as though nothing had moved -- the same guard tokenizer_ids applies.+ Map<String, Object> en = (Map<String, Object>) forCheckpoint("english").get("decoder");+ assertEquals("ByteLevel", en.get("type"), "the english decoder changed");++ Map<String, Object> ml = (Map<String, Object>) forCheckpoint("multilingual").get("decoder");+ assertEquals("Sequence", ml.get("type"), "the multilingual decoder changed");+ List<Object> steps = (List<Object>) ml.get("decoders");+ List<String> types = new ArrayList<>();+ for (Object step : steps) {+ types.add((String) ((Map<String, Object>) step).get("type"));+ }+ assertEquals(List.of("Replace", "ByteFallback", "Fuse"), types,+ "the multilingual decoder chain changed");+ }++ // ------------------------------------------------- the three rules, each stated on its own++ @Test+ @DisplayName("a character outside the stand-in alphabet is kept, not dropped")+ void unmappedCharactersSurvive() {+ // Dropping them was the first guess and it got 414 of 5,312 texts wrong. This vocabulary+ // mixes byte-level tokens with 23 added tokens whose content is LITERAL whitespace -- id+ // 50275 is three real spaces, and 0x20 is lifted to U+0120 so a real space is not in the+ // alphabet at all.+ Tokenizer tok = open("english");+ assertEquals(" ", tok.decode(new int[] {50275}),+ "the literal-whitespace added token must survive decoding");+ assertEquals(" ", tok.decode(tok.encode(" ")), "and a byte-level space still works");+ assertEquals(" ", tok.decode(tok.encode(" ")));+ }++ /** Loads a synthetic spec recorded in the fixture, as the crate was asked about it. */+ private static Tokenizer fromSpec(Object spec) throws IOException {+ Path dir = Files.createTempDirectory("laya-synthetic");+ Path file = dir.resolve("tokenizer.json");+ Files.write(file, PythonJson.dumps(spec).getBytes(StandardCharsets.UTF_8));+ try {+ return Tokenizer.fromFile(file);+ } finally {+ Files.deleteIfExists(file);+ Files.deleteIfExists(dir);+ }+ }++ @TestFactory+ @DisplayName("the decoder shapes no shipped checkpoint declares")+ @SuppressWarnings("unchecked")+ List<DynamicTest> syntheticDecoderShapes() {+ // `TokenDecoder` implements six decoders, and the two shipped checkpoints between them use+ // four: english is ByteLevel, multilingual is Sequence[Replace, ByteFallback, Fuse]. So+ // the recorded corpus cannot reach Strip at all, cannot reach the no-decoder fallback, and+ // cannot tell a nested Sequence or a split byte run from a joined one. An adversarial+ // review gutted Strip entirely -- `apply` replaced by `return "MUTANT"` -- and all 191+ // cases passed.+ //+ // Four real divergences were hiding in that gap, each confirmed against the crate before+ // it was fixed: the no-decoder fallback joins with a SPACE (this joined with nothing), the+ // hex parse is case-insensitive and takes a leading "+" (this rejected both), a nested+ // Sequence keeps its pieces (this re-joined them), and a failed byte run is one piece PER+ // BYTE (this made it one piece of N). Every expectation below comes from the crate.+ Map<String, Object> shapes = (Map<String, Object>) family().get("synthetic_decoders");+ Assumptions.assumeTrue(shapes != null && !shapes.containsKey("skipped"),+ "the synthetic decoder shapes were not recorded");++ List<DynamicTest> tests = new ArrayList<>();+ for (Map.Entry<String, Object> entry : shapes.entrySet()) {+ String name = entry.getKey();+ Map<String, Object> shape = (Map<String, Object>) entry.getValue();+ tests.add(dynamicTest(name, () -> {+ Tokenizer tok = fromSpec(shape.get("spec"));+ List<Object> cases = (List<Object>) shape.get("cases");+ assertFalse(cases.isEmpty(), () -> name + " recorded no cases");+ for (Object raw : cases) {+ Map<String, Object> one = (Map<String, Object>) raw;+ int[] ids = ids((List<Object>) one.get("ids"));+ assertEquals(one.get("decoded"), tok.decode(ids, false),+ () -> name + " diverged on ids " + one.get("ids"));+ }++ // Where the crate PANICS there is no reference answer, so what is asserted is+ // that this port stays total: a Rust panic crossing the FFI boundary is not+ // behaviour worth reproducing, and silently returning something is not either.+ List<Object> panics = (List<Object>) shape.get("crate_panics");+ if (panics != null) {+ for (Object raw : panics) {+ int[] ids = ids((List<Object>) raw);+ assertEquals("", tok.decode(ids, false),+ () -> name + ": the crate panics on " + raw+ + " and this must return empty text, not throw");+ }+ }+ }));+ }+ assertTrue(tests.size() >= 5,+ () -> "five shapes were recorded and only " + tests.size() + " are here");+ return tests;+ }++ @Test+ @DisplayName("an unsupported or malformed decoder is refused, not guessed at")+ void refusalsFire() {+ // Every one of these threw nowhere in the suite before this test: an adversarial review+ // replaced all four refusal paths at once -- the unknown-type throw by `return new+ // Join()`, the Regex-pattern throw and the Strip content-length throw by `if (false)` --+ // and all 191 cases passed. They are the guards whose whole stated purpose is to stop a+ // future checkpoint being silently mis-decoded, so "nothing proves they fire" is the one+ // thing they cannot afford. None of this needs a checkpoint, so it runs in every lane.+ assertThrows(Json.JsonException.class,+ () -> TokenDecoder.from(Map.of("type", "Metaspace")),+ "a decoder this port does not implement must be refused, not approximated");+ assertThrows(Json.JsonException.class,+ () -> TokenDecoder.from(new LinkedHashMap<>()),+ "a decoder node with no type must be refused");+ // The MESSAGE, not just the type. Disabling the Regex check leaves `literal` null,+ // so the NEXT guard throws Json.JsonException anyway -- and a mutant that removed+ // the Regex refusal outright passed this assertion while it checked only the+ // exception class. The two refusals say different things ("this port does not do+ // regex" versus "your pattern is missing") and have to be told apart.+ Json.JsonException regex = assertThrows(Json.JsonException.class,+ () -> TokenDecoder.from(Map.of("type", "Replace",+ "pattern", Map.of("Regex", "\\s+"), "content", " ")),+ "a Regex Replace needs the crate's regex dialect and must be refused");+ assertTrue(regex.getMessage().contains("Regex"),+ () -> "the refusal must name the reason, got: " + regex.getMessage());+ assertThrows(Json.JsonException.class,+ () -> TokenDecoder.from(Map.of("type", "Replace", "content", " ")),+ "a Replace with no pattern must be refused");+ assertThrows(Json.JsonException.class,+ () -> TokenDecoder.from(Map.of("type", "Sequence")),+ "a Sequence with no decoders array must be refused");+ assertThrows(Json.JsonException.class,+ () -> TokenDecoder.from(Map.of("type", "Sequence",+ "decoders", List.of("ByteLevel"))),+ "a Sequence step that is not an object must be refused");+ assertThrows(Json.JsonException.class,+ () -> TokenDecoder.from(Map.of("type", "Strip", "content", "ab")),+ "a Strip whose content is not one character must be refused");++ // And the one branch that is deliberately permissive. Its separator is a space, which is+ // what the crate falls back to; the recorded `no_decoder` shape is the parity check and+ // this states that `from(null)` reaches the same code.+ assertEquals("ab cd", TokenDecoder.from(null).decode(List.of("ab", "cd")),+ "no decoder node means join with a space, as the crate does");++ assertThrows(IllegalArgumentException.class, () -> PreTokenizer.charForByte(256),+ "a byte is 0 to 255, and an out-of-range index must say so");+ assertThrows(IllegalArgumentException.class, () -> PreTokenizer.charForByte(-1),+ "including below the range");+ }++ @Test+ @DisplayName("the ByteLevel fallback is scoped to a token, not to a character")+ @SuppressWarnings("unchecked")+ void byteLevelFallsBackPerToken() throws IOException {+ // Neither real checkpoint can tell these rules apart. Their byte-level tokens are entirely+ // stand-in alphabet and their added tokens are entirely literal whitespace, so a mutant+ // that swapped per-token for per-character passed all 190 recorded cases -- the rule was+ // correct and unpinned, which is the same as unverified.+ //+ // The rules differ only for a token that MIXES the two, so the generator builds a+ // vocabulary containing one and records what the crate does. "Ġ " separates them:+ // U+0120 is the stand-in for byte 0x20 and the literal space is not in the alphabet.+ // per TOKEN -> every character's own UTF-8 bytes: C4 A0 20 -> "Ġ "+ // per CHARACTER -> the mapped one's byte: 20 20 -> " "+ Map<String, Object> scope = (Map<String, Object>) family().get("mixed_token");+ Assumptions.assumeTrue(scope != null && !scope.containsKey("skipped"),+ "the mixed-token scope was not recorded");+ // Documentation, not a gate: the generator RAISES when the probed scope is not "token",+ // so no other value can reach the fixture and this line can only fail on a hand-edited+ // one. The real guard is that `raise` -- verified by changing the separating token so the+ // crate answered differently, which failed the regeneration step rather than this test.+ // The parity assertions below are what pin the behaviour.+ assertEquals("token", scope.get("scope"),+ "the crate's fallback scope changed and this port assumes per token");++ // The spec travels in the fixture, so this loads the same tokenizer the crate was asked+ // about instead of a second hand-written copy of it.+ Path dir = Files.createTempDirectory("laya-mixed-token");+ Path file = dir.resolve("tokenizer.json");+ try {+ Files.write(file, PythonJson.dumps(scope.get("spec"))+ .getBytes(StandardCharsets.UTF_8));+ Tokenizer tok = Tokenizer.fromFile(file);+ for (Object raw : (List<Object>) scope.get("cases")) {+ Map<String, Object> one = (Map<String, Object>) raw;+ int[] ids = ids((List<Object>) one.get("ids"));+ assertEquals(one.get("decoded"), tok.decode(ids, false),+ (String) one.get("why"));+ }+ } finally {+ Files.deleteIfExists(file);+ Files.deleteIfExists(dir);+ }+ }++ @Test+ @DisplayName("ByteFallback emits one replacement character per byte, not per subpart")+ void byteFallbackReplacesPerByte() {+ // The rule that only the slices could see. U+F0000 is four ByteFallback tokens; cutting+ // after two leaves F3 B0, which lossy decoding calls one maximal subpart and reports as+ // ONE replacement character. The crate reports two -- one per byte it could not use.+ Tokenizer tok = open("multilingual");+ int[] ids = tok.encode(""); // U+F0000+ assertTrue(ids.length >= 3,+ () -> "expected a Metaspace marker plus a byte run, got " + ids.length + " ids");+ int[] firstThree = {ids[0], ids[1], ids[2]};+ assertEquals(" ��", tok.decode(firstThree),+ "two bytes of a four-byte character must give two replacement characters");+ // and the whole thing still decodes to the character itself+ assertEquals(" ", tok.decode(ids),+ "the complete byte run must decode to the character");+ }++ @Test+ @DisplayName("decoding is lossy on the multilingual checkpoint, and that is the contract")+ void metaspaceAddsALeadingSpace() {+ // A port that "fixed" this would re-tokenize every predictLong window differently from the+ // reference and move every answer downstream. The fixture records the loss; this states it+ // where a reader will see it.+ Tokenizer ml = open("multilingual");+ assertEquals(" Hello world", ml.decode(ml.encode("Hello world")),+ "Metaspace prepends U+2581 and the decoder turns it into a space");+ Tokenizer en = open("english");+ assertEquals("Hello world", en.decode(en.encode("Hello world")),+ "the english checkpoint happens to round-trip");+ }++ @Test+ @DisplayName("an unknown id and a null argument are handled, not thrown at")+ void edges() {+ Tokenizer tok = open("english");+ assertEquals("", tok.decode(new int[0]), "no ids is empty text");+ assertEquals("", tok.decode(new int[] {Integer.MAX_VALUE}),+ "an id outside the vocabulary is skipped, as the crate skips it");+ // A NEGATIVE id is not crate parity -- the crate's ids are u32, so the Python binding+ // raises OverflowError rather than skipping it. Treating it like any other id outside the+ // vocabulary is this port's own choice, made because `int[]` can hold one and a caller's+ // off-by-one should not become an exception from a method whose job is to be lossy. Stated+ // here rather than filed under "as the crate skips it", which is what it used to say.+ assertEquals("a", tok.decode(new int[] {Integer.MAX_VALUE, tok.encode("a")[0], -1}),+ "a negative id is skipped like any other unknown one, by this port's choice");+ assertTrue(assertThrowsIllegalArgument(() -> tok.decode(null)),+ "a null id array is a caller error, not a silent empty string");+ }++ private static boolean assertThrowsIllegalArgument(Runnable work) {+ try {+ work.run();+ return false;+ } catch (IllegalArgumentException expected) {+ return true;+ }+ }++ @Test+ @DisplayName("the byte table and its inverse agree on all 256 values")+ void byteTableRoundTrips() {+ // What this catches, precisely: a table that is not INJECTIVE. If two bytes were given+ // the same stand-in character, the inverse would lose one of them and the round trip+ // would fail here. Measured, because the comment used to claim more than that: a mutant+ // that SWAPS two entries (0x41 and 0x42) is still injective and passes this test and all+ // 191 recorded cases. The table's actual values are pinned on the encode side instead,+ // by the recorded tokenizer fixtures -- the same swap fails 10 cases in+ // SequenceParityTest -- and none of the 23 texts in this family contains an "A" or a "B",+ // which is why the decode side could not see it.+ for (int value = 0; value < 256; value++) {+ final int b = value;+ char stand = PreTokenizer.charForByte(b);+ assertEquals(b, PreTokenizer.byteForChar(stand),+ () -> String.format("byte 0x%02X does not survive the round trip", b));+ }+ assertFalse(PreTokenizer.byteForChar(' ') == 0x20,+ "a literal space must NOT be in the alphabet: 0x20 is lifted to U+0120, and the "+ + "whole literal-token rule depends on it being absent");+ }++ private static int[] ids(List<Object> recorded) {+ int[] out = new int[recorded.size()];+ for (int i = 0; i < out.length; i++) {+ out[i] = ((Number) recorded.get(i)).intValue();+ }+ return out;+ }++ /** A readable label that cannot contain a line break or a lone surrogate. */+ private static String describe(String text) {+ if (text.isEmpty()) {+ return "<empty>";+ }+ StringBuilder out = new StringBuilder();+ text.codePoints().limit(18).forEach(cp -> {+ if (cp < 0x20 || cp > 0x7E) {+ out.append(String.format("U+%04X", cp));+ } else {+ out.appendCodePoint(cp);+ }+ });+ return out.toString();+ }+}diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.pyindex e0cced77..dc247d06 100644--- a/laya-java/scripts/gen_fixtures.py+++ b/laya-java/scripts/gen_fixtures.py@@ -2088,6 +2088,349 @@ def email_clean():}+def decode_text():+ """What `Tokenizer.decode` must return, per checkpoint, recorded from the reference.++ Decoding has no Python reference to port -- it lives in the Rust `tokenizers` crate -- so this+ family is the only specification the Java side has, and the three rules it pins were each+ wrong on the first attempt:++ * a character outside the ByteLevel stand-in alphabet is KEPT as its own UTF-8 bytes, not+ dropped. The english vocabulary mixes byte-level tokens like "\\u0120world" with 23 added+ tokens whose content is literal whitespace -- id 50275 is three real spaces -- and dropping+ unmapped characters got 414 of 5,312 texts wrong.+ * the ByteLevel fallback is per TOKEN, not per character: one unmapped character and the+ whole token uses its raw bytes.+ * ByteFallback does NOT decode lossily. It tries strict UTF-8 and, on failure, emits one+ U+FFFD per BYTE of the run. Lossy decoding emits one per maximal subpart, which differs+ the moment a window boundary cuts a multi-byte character -- 629 of 201,279 window slices.++ The SLICES are the point of this family. `predictLong` cuts a tokenized state into windows at+ arbitrary offsets and decodes each one, so a boundary lands inside a multi-byte character+ routinely. Every whole-text decode agreed while 629 slices did not, so a fixture of whole+ texts would have shipped the bug.+ """+ import os++ rig = os.environ.get("LAYA_FIXTURE_CHECKPOINTS", DEFAULT_CHECKPOINT_ROOT)++ # Chosen for where a decoder breaks, not for coverage of prose.+ texts = [+ "Hello world",+ "",+ " ",+ " ", # an added token whose content is literal spaces+ "\t",+ "a\tb\nc", # multilingual inserts a space after the newline+ "café résumé",+ "注文をキャンセル", # 3-byte characters, split often+ "مرحبا",+ "क्ष", # ZWNJ, which is Cf and not a mark+ "\U0001f600", # 4-byte, the shortest ByteFallback run+ "emoji \U0001f600 here",+ "\U0001f1ec\U0001f1e7", # a flag: two regional indicators+ "▁literal marker", # the Metaspace marker as ORDINARY TEXT+ "pre▁fix",+ " nbsp figure",+ "<0x41>", # a byte token spelled out in ordinary text+ "a<0xFF>b",+ "",+ "", # private use+ "\U000f0000", # unassigned: four ByteFallback tokens+ "I cannot log in and need a password reset.",+ "Esta mensagem e confidencial.",+ ]++ out = {"texts": texts, "by": {}}+ for name in ("english", "multilingual"):+ root = os.path.join(rig, name)+ tokenizer_json = os.path.join(root, "tokenizer", "tokenizer.json")+ if not os.path.isfile(tokenizer_json):+ out["by"][name] = {"skipped": "checkpoint not present at %s" % root}+ continue+ from tokenizers import Tokenizer++ tok = Tokenizer.from_file(tokenizer_json)+ cases = []+ for text in texts:+ ids = tok.encode(text, add_special_tokens=False).ids+ # Every cut point at several widths: this is the predictLong shape, and the only+ # place the ByteFallback rule is observable.+ slices = []+ for size in (1, 2, 3, 5):+ for start in range(0, max(1, len(ids)), max(1, size)):+ chunk = ids[start:start + size]+ if chunk:+ slices.append({"start": start, "size": size,+ "decoded": tok.decode(chunk)})+ with_specials = tok.encode(text, add_special_tokens=True).ids+ cases.append({+ "text": text,+ "ids": list(ids),+ "decoded": tok.decode(ids),+ "slices": slices,+ "with_specials": {+ "ids": list(with_specials),+ # NOT "skipped": a lone `skipped` key is this script's marker for a family it+ # could not regenerate, and naming a data field that way once made this very+ # family look unregenerable.+ "skipping_specials": tok.decode(with_specials),+ "kept": tok.decode(with_specials, skip_special_tokens=False),+ },+ })+ out["by"][name] = {+ # What this tokenizer IS, so a fixture recorded against a different checkpoint cannot+ # be mistaken for a matching one -- the same guard tokenizer_ids applies.+ "decoder": _decoder_shape(tokenizer_json),+ "cases": cases,+ }+ out["mixed_token"] = _mixed_token_scope()+ out["synthetic_decoders"] = _synthetic_decoders()+ return out+++def _synthetic_decoders():+ """What the crate does with the decoder shapes no shipped checkpoint declares.++ `TokenDecoder` implements six decoders because a checkpoint may declare any of them, but the+ two shipped ones between them use four: english is `ByteLevel`, multilingual is+ `Sequence[Replace, ByteFallback, Fuse]`. So `Strip` shipped with no coverage at all -- a 44+ line decoder with a documented start/stop semantic that an adversarial mutant could gut while+ all 191 recorded cases passed -- and the `hex` helper's refusal of lowercase byte tokens had+ nothing exercising it either.++ Rather than assert what the implementation was written to do, each shape below is a real+ tokenizer the crate is asked to decode, exactly as `mixed_token` is. `byte_fallback` is set on+ the model so the port's loader guard, which refuses a vocabulary that could drop a character+ without trace, is satisfied by something other than a full byte-level alphabet.+ """+ import io+ import os+ import tempfile++ try:+ from tokenizers import Tokenizer+ except ImportError:+ return {"skipped": "the tokenizers package is not installed"}++ def spec_for(decoder, vocab):+ spec = {+ "version": "1.0", "truncation": None, "padding": None,+ "added_tokens": [], "normalizer": None, "pre_tokenizer": None,+ "post_processor": None,+ "model": {"type": "BPE", "dropout": None, "unk_token": None,+ "continuing_subword_prefix": None, "end_of_word_suffix": None,+ "fuse_unk": False, "byte_fallback": True, "ignore_merges": True,+ "vocab": vocab, "merges": []},+ }+ # `decoder=None` means the key is ABSENT, which is its own code path -- not a decoder+ # whose type is null.+ if decoder is not None:+ spec["decoder"] = decoder+ return spec++ shapes = {+ # Strip removes up to `start` leading and up to `stop` trailing copies of `content`, per+ # TOKEN. The crate serialises the Python `Strip(content, left, right)` as start/stop,+ # which is why those are the field names the port reads.+ "strip": {+ "why": "Strip, which no shipped checkpoint declares",+ "spec": spec_for(+ {"type": "Strip", "content": " ", "start": 2, "stop": 1},+ {" hello ": 0, " x": 1, " ": 2, "no": 3}),+ "ids": [[0], [1], [3], [0, 3]],+ # The crate PANICS on these, so there is no reference answer to record: stripping 2+ # leading spaces from the two-space token leaves an empty slice and `stop` then+ # indexes backwards through it -- "slice index starts at 2 but ends at 1", a Rust+ # panic that crosses the FFI boundary as pyo3_runtime.PanicException. The port guards+ # the same arithmetic (`to > from`) and returns "". That difference is deliberate and+ # is asserted on the Java side: a decoder that aborts the process on a token that is+ # all `content` is not behaviour worth reproducing.+ "crate_panics": [[2], [2, 2]],+ },+ # Which `<0x..>` spellings the crate reads as a BYTE. Its predicate is a length-6 token+ # starting "<0x" and ending ">", parsed by Rust's `u8::from_str_radix(.., 16)` -- which is+ # case-INSENSITIVE and accepts a leading "+". This port had it backwards, rejecting+ # lowercase and "+" with a comment asserting the crate wrote these uppercase only.+ #+ # The "+" case needs its own token: with only the upper/lower pair here, a mutant that+ # re-broke "+" passed all 195 cases. "<0x 5>" and "<0XFF>" record the other side of the+ # boundary -- a space is not a digit, and the "<0x" prefix IS case-sensitive -- so the+ # rule is pinned from both directions rather than only where it says yes.+ "bytefallback_hex_case": {+ "why": "which byte-token spellings the crate reads as a byte",+ "spec": spec_for(+ {"type": "ByteFallback"},+ {"<0xFF>": 0, "<0xff>": 1, "<0x41>": 2, "<0x61>": 3, "a": 4,+ "<0x+5>": 5, "<0x 5>": 6, "<0XFF>": 7, "<0xfF>": 8,+ "<0xF3>": 9, "<0xB0>": 10, "<0x80>": 11}),+ # The last four are a TRUNCATED multi-byte run, which is the only shape that can tell+ # lossy decoding from one replacement per byte: F3 B0 is a single maximal subpart+ # (lossy gives ONE U+FFFD) and two unusable bytes (the crate gives TWO). Every other+ # invalid run in this vocabulary is a single byte, where both rules agree -- so a+ # mutant that made this decoder lossy passed all 195 cases.+ "ids": [[2], [3], [1], [0], [2, 3], [0, 0], [1, 4],+ [5], [6], [7], [8], [5, 4], [6, 4], [7, 4],+ [9, 10], [9, 10, 11], [9, 10, 11, 11], [4, 9, 10, 4]],+ },+ # No `decoder` key at all. The crate's fallback is `tokens.join(" ")` -- a SPACE -- and+ # this port joined with nothing, which was wrong for every multi-token decode. Neither+ # shipped checkpoint can reach it, because both declare a decoder.+ "no_decoder": {+ "why": "no decoder section: the separator the crate falls back to",+ "spec": spec_for(None, {"ab": 0, "cd": 1, "ef": 2, " ": 3}),+ "ids": [[0], [0, 1], [0, 1, 2], [0, 3, 1], [3, 3]],+ },+ # A Sequence inside a Sequence. The crate chains `decode_chain` and returns a LIST, so the+ # outer sequence's next step sees the inner one's pieces; re-joining them collapsed the+ # boundaries. Strip is the step that can tell, because it acts per piece.+ "nested_sequence": {+ "why": "a Sequence nested in a Sequence keeps its pieces",+ "spec": spec_for(+ {"type": "Sequence", "decoders": [+ {"type": "Sequence", "decoders": [+ {"type": "Replace", "pattern": {"String": "\u2581"}, "content": " "},+ {"type": "ByteFallback"}]},+ {"type": "Strip", "content": " ", "start": 1, "stop": 0}]},+ {"\u2581Hello": 0, "\u2581world": 1, "\u2581": 2, "\u2581a": 3}),+ "ids": [[0, 1], [2, 2, 0], [2, 3, 2], [0], [3]],+ },+ # Whether a failed ByteFallback run is one piece of N replacement characters or N pieces+ # of one. Only a per-piece step after it can tell, so Replace looks for the PAIR: it+ # matches if the two characters ended up in the same piece.+ "bytefallback_piece_boundaries": {+ "why": "a failed byte run is one piece per byte, not one piece of N",+ "spec": spec_for(+ {"type": "Sequence", "decoders": [+ {"type": "ByteFallback"},+ {"type": "Replace", "pattern": {"String": "\ufffd\ufffd"},+ "content": "X"}]},+ {"<0xF3>": 0, "<0xB0>": 1, "<0x80>": 2, "hi": 3}),+ "ids": [[0, 1], [0, 1, 2], [3, 0, 1, 3], [0, 1, 2, 2], [3]],+ },+ }++ for name, shape in shapes.items():+ path = os.path.join(tempfile.mkdtemp(), "tokenizer.json")+ io.open(path, "w", encoding="utf-8").write(json.dumps(shape["spec"]))+ tok = Tokenizer.from_file(path)+ shape["cases"] = [{"ids": ids, "decoded": tok.decode(ids, skip_special_tokens=False)}+ for ids in shape.pop("ids")]+ # A shape whose cases all decoded to the same thing would pin nothing, and these are+ # hand-chosen rather than swept, so the check is worth its one line.+ distinct = {case["decoded"] for case in shape["cases"]}+ if len(distinct) < 2:+ raise AssertionError(+ "the %s shape's cases all decode to %r, so they cannot tell any rule apart"+ % (name, distinct.pop()))+ return shapes+++def _mixed_token_scope():+ """Whether the ByteLevel fallback is scoped to a TOKEN or to a CHARACTER, from the crate.++ Neither real checkpoint can answer this. Their byte-level tokens are entirely stand-in+ alphabet and their added tokens are entirely literal whitespace, so both rules produce the+ same bytes for every token in both vocabularies -- a mutant that swaps per-token for+ per-character passed all 190 recorded cases. The rules differ only for a token that MIXES+ the two, so this builds a vocabulary that has one and records what the crate does with it.++ The separating token is "Ġ ": U+0120 is the stand-in for byte 0x20, and the literal+ space after it is not in the alphabet at all.++ per TOKEN -> every character contributes its own UTF-8 bytes: C4 A0 20 -> "Ġ "+ per CHARACTER -> the mapped one contributes its byte: 20 20 -> " "++ The spec travels in the fixture rather than being written twice, so the Java test loads the+ same bytes the crate was asked about.+ """+ import io+ import os+ import tempfile++ try:+ from tokenizers import Tokenizer+ except ImportError:+ return {"skipped": "the tokenizers package is not installed"}++ mixed = "Ġ "++ # Every single-byte alphabet symbol, from the crate's own table rather than a transcription.+ # The port's loader refuses a vocabulary that cannot represent some text, and a vocabulary of+ # three tokens cannot -- so the three interesting tokens join a complete byte-level one. They+ # do not interact: decoding is driven by the ids this records, not by what else is present.+ from tokenizers.pre_tokenizers import ByteLevel as ByteLevelPre++ vocab = {}+ for symbol in sorted(ByteLevelPre.alphabet()):+ vocab[symbol] = len(vocab)+ interesting = {}+ for token in (mixed, "Ġhi", " "):+ if token not in vocab:+ vocab[token] = len(vocab)+ interesting[token] = vocab[token]++ spec = {+ "version": "1.0", "truncation": None, "padding": None,+ "added_tokens": [], "normalizer": None,+ # The pre-tokenizer matches the decoder. It plays no part in decoding, but the port's+ # loader refuses a vocabulary where a character could be dropped without trace -- no+ # unk_token, no byte_fallback, no byte-level pre-tokenizer -- and that guard is right.+ "pre_tokenizer": {"type": "ByteLevel", "add_prefix_space": True, "trim_offsets": True,+ "use_regex": True},+ "post_processor": None,+ "decoder": {"type": "ByteLevel", "add_prefix_space": True, "trim_offsets": True,+ "use_regex": True},+ "model": {"type": "BPE", "dropout": None, "unk_token": None,+ "continuing_subword_prefix": None, "end_of_word_suffix": None,+ "fuse_unk": False, "byte_fallback": False, "ignore_merges": True,+ "vocab": vocab, "merges": []},+ }++ path = os.path.join(tempfile.mkdtemp(), "tokenizer.json")+ io.open(path, "w", encoding="utf-8").write(json.dumps(spec))+ tok = Tokenizer.from_file(path)++ cases = [+ {"why": "the mixed token alone: the whole point",+ "ids": [interesting[mixed]]},+ {"why": "entirely in the alphabet", "ids": [interesting["Ġhi"]]},+ {"why": "entirely literal", "ids": [interesting[" "]]},+ {"why": "a mixed token beside a plain one",+ "ids": [interesting[mixed], interesting["Ġhi"]]},+ ]+ for case in cases:+ case["decoded"] = tok.decode(case["ids"], skip_special_tokens=False)++ scope = "token" if cases[0]["decoded"] == mixed else (+ "character" if cases[0]["decoded"] == " " else "neither")+ if scope != "token":+ raise AssertionError(+ "the crate's ByteLevel fallback is scoped per %s, and the port assumes per token; "+ "decoding the mixed token gave %r" % (scope, cases[0]["decoded"]))+ return {"spec": spec, "separating_token": mixed, "scope": scope, "cases": cases}+++def _decoder_shape(tokenizer_json):+ """The decoder section's type chain, flattened, so the fixture says what it was recorded on."""+ import io+ import json++ document = json.load(io.open(tokenizer_json, encoding="utf-8"))+ node = document.get("decoder")+ if node is None:+ return None++ def shape(n):+ kind = n.get("type")+ if kind == "Sequence":+ return {"type": "Sequence",+ "decoders": [shape(step) for step in n.get("decoders", [])]}+ return {"type": kind}++ return shape(node)++FAMILIES = {"lang_tables.json": lang_tables,"lang_detect.json": lang_detect,@@ -2095,6 +2438,7 @@ FAMILIES = {"router.json": router,"shortlist.json": shortlist,"tokenizer_ids.json": tokenizer_ids,+ "decode_text.json": decode_text,"sequences.json": sequences,"decode.json": decode_answers,"python_json.json": python_json,@@ -2139,8 +2483,41 @@ def unverifiable(payload):# always present, so requiring every section to be skipped let a payload through whose two# per-checkpoint halves were both skip markers -- and the write replaced a 107 KB fixture with# a 9 KB one. Partial regeneration is still unsafe to write.- return any("skipped" in value- for value in payload.values() if isinstance(value, dict))+ #+ # At ANY depth, not just one level down. This looked only at the payload's immediate values,+ # and `decode_text.json` groups its per-checkpoint sections under a `by` key -- so the skip+ # markers sat two levels down, the payload was judged regenerable, and both halves of this+ # docstring came true at once: on a machine without the checkpoints, `--check` reported the+ # committed fixture as stale, and a plain run replaced 75 KB of parity expectations with 7 KB+ # of skip markers. Measured, not reasoned about. A family should not have to keep its sections+ # at the top level to be seen, so the search descends instead.+ return _mentions_skip(payload)+++def _mentions_skip(node):+ """True when a skip MARKER appears anywhere in the payload.++ A marker is a dict of exactly one `skipped` key, which is what every builder above writes.+ Merely containing a `skipped` key is not enough, and that is not a hypothetical distinction:+ `decode_text.json` records, per text, what the reference returns with special tokens skipped+ and without, and the first version of that family named the first of those two fields+ `skipped`. The recursion below read it as a marker, so a fully recorded family reported itself+ unregenerable -- with the checkpoints present it was left alone instead of checked, and+ `--strict` in the parity lane would have failed on a family that had just been recorded+ correctly.++ That field is now `skipping_specials`, so the collision no longer exists in this script and+ nothing here depends on the exact-key rule to avoid it. The rule stays because it is the+ correct test for a marker, and because the next family to record a `skipped` field should not+ have to discover this interaction the way that one did.+ """+ if isinstance(node, dict):+ if set(node) == {"skipped"}:+ return True+ return any(_mentions_skip(value) for value in node.values())+ if isinstance(node, list):+ return any(_mentions_skip(item) for item in node)+ return Falsedef main(argv=None):