Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(java): a test asserted the JDK disagrees with CPython, and on JDK 21 it does not

Adeshina Lasisia3ad3d389bfb
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index 2d55712f..ca96e4c3 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -146,9 +146,17 @@ jobs:
strategy:
fail-fast: false
matrix:
- # 17 is covered by the build job above. 21 is the LTS most consumers are on; 24 is the
- # newest with a different Unicode version from 17, and is the one verified by hand.
- jdk: ["21", "24"]
+ # 17 as well as 21 and 24, even though the build job also tests on 17: that job runs
+ # WITHOUT `-PtestJavaVersion`, so `TestJvmVersionTest` is disabled there and "17 is
+ # tested" was true but unasserted. Here it is asserted.
+ #
+ # The three are not an arbitrary spread, they are three different Unicode versions --
+ # 17 carries 13.0, 21 carries 15.0 and 24 carries 16.0 -- and this port's whole
+ # character-handling strategy is that the answer must not depend on which. That is not
+ # theoretical: the JDK 21 cell failed on its first run, because `Character.isLetter` on
+ # Unicode 15.0 agrees with CPython 15.0 exactly, and a test had asserted the JDK
+ # DISAGREES. 17 and 24 passed. A two-version matrix would not have found it.
+ jdk: ["17", "21", "24"]
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
# For check_test_results.py only -- it is standard library, so there is nothing to install.
diff --git a/laya-java/CHANGELOG.md b/laya-java/CHANGELOG.md
index 1541144e..17bcbc30 100644
--- a/laya-java/CHANGELOG.md
+++ b/laya-java/CHANGELOG.md
@@ -35,17 +35,24 @@ see [Installing](README.md#installing).
### Testing
-- 768 tests, 594 of which need no model at all.
+- 769 tests, 626 of which need no model at all.
- Every expectation is **generated from the Python package**, never written by hand:
`laya-java/fixtures/*.json`, regenerated by `scripts/gen_fixtures.py` and gated by `--check` in
CI so a change to `laya/` that moves a recorded contract shows up here rather than as drift.
- The compiled-in Unicode tables are checked by a **sha256 over all 1,114,112 code points** per
property, so they are proven exhaustively rather than on the cases someone thought to write.
-- Five CI lanes: build plus model-free tests, the same tests on JDK 21 and JDK 24, and two parity
- cells split by whether they need a traced graph.
+- Six CI lanes: build plus model-free tests, the same tests on JDK 17, 21 and 24 — three
+ different Unicode versions, 13.0, 15.0 and 16.0 — and two parity cells split by whether they
+ need a traced graph.
- `-PtestJavaVersion` runs the tests on a different JDK from the one the classes are compiled
for, because the toolchain pins the compiler to 17 and a lane that merely installs another JDK
tests the same thing twice.
+- That matrix earned itself on its first run. The JDK 21 cell failed where 17 and 24 passed,
+ because `Character.isLetter` on Unicode 15.0 agrees with CPython 15.0 EXACTLY, and a test had
+ asserted that the JDK disagrees. Catching up is not a reason to drop the table -- it is the
+ reason the table exists, since the agreement is two version numbers lining up by accident and
+ un-happens on the next release in either direction. The assertion is now the version-independent
+ one, and whether a given JDK agrees is recorded rather than required.
### Known limitations
diff --git a/laya-java/README.md b/laya-java/README.md
index c5802821..886de5de 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -340,6 +340,10 @@ the bytecode stays at release 17:
./gradlew test -PtestJavaVersion=24 # compiled for 17, executed on 24
```
+CI runs the model-free suite on **17, 21 and 24** — three different Unicode versions (13.0, 15.0
+and 16.0). The artifact is compiled for 17, so it runs on 17 and anything newer; those three are
+the versions the suite is actually asserted against.
+
This matters more here than in most ports. `\p{L}` and `\p{N}` in `java.util.regex` follow the
JDK's own Unicode version, and `Character.isLetter` disagrees with itself across JDK 17 (Unicode
13.0) and JDK 24 (Unicode 16.0) on 751 of the code points this port has to classify — 0 of 751 on
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/lang/UnicodeTablesTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/lang/UnicodeTablesTest.java
index 5e44f03a..27db26d3 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/lang/UnicodeTablesTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/lang/UnicodeTablesTest.java
@@ -204,17 +204,22 @@ class UnicodeTablesTest {
}
@Test
- @DisplayName("the JDK's own predicates would NOT pass, which is why these tables exist")
- void theJdkDisagrees() {
- // Not a tautology: this is the measurement behind the comment on UnicodeTables, kept as a
- // test so the claim is checked rather than asserted. If a future JDK ever catches up, this
- // fails and the table can be reconsidered deliberately rather than kept out of habit.
- assertFalse(predicateDigest(Character::isLetter).equals(digests().get("alpha")),
- "Character.isLetter now agrees with CPython; the ALPHA table may be removable");
+ @DisplayName("two of these properties have no JDK equivalent, on any JDK")
+ void theJdkHasNoEquivalent() {
+ // These two disagree with CPython on EVERY JDK, because the JDK does not expose the
+ // property at all rather than exposing an older version of it:
+ //
+ // * `str.isspace()` takes U+00A0, U+0085, U+2007 and U+202F, and
+ // `Character.isWhitespace` rejects every one of them -- it is a different predicate,
+ // not a stale one.
+ // * the canonical combining CLASS is not exposed by the JDK. Mn/Mc is the nearest
+ // approximation, which is what the .NET port uses, and it differs on 1,528 code
+ // points that are category M with a class of zero.
+ //
+ // So these are the version-independent half of the justification for compiling the
+ // tables in, and they are safe to assert on a JDK matrix.
assertFalse(predicateDigest(Character::isWhitespace).equals(digests().get("space")),
"Character.isWhitespace now agrees with CPython's isspace");
- assertFalse(predicateDigest(Character::isUpperCase).equals(digests().get("upper")),
- "Character.isUpperCase now agrees with CPython's isupper");
assertFalse(predicateDigest(cp -> {
int type = Character.getType(cp);
return type == Character.NON_SPACING_MARK || type == Character.COMBINING_SPACING_MARK;
@@ -223,6 +228,41 @@ class UnicodeTablesTest {
+ " canonical combining class");
}
+ @Test
+ @DisplayName("whether the JDK's letter and case predicates agree is a property of the JDK")
+ void theJdkMayOrMayNotAgree() {
+ // This test used to assert that `Character.isLetter` DISAGREES with CPython, with a
+ // comment saying that if a JDK ever caught up the test should fail so the table could be
+ // reconsidered. A JDK caught up: 21 ships Unicode 15.0 and CPython is 15.0.0, so the two
+ // letter sets are identical there -- and the test failed on the JDK 21 cell while 17
+ // (Unicode 13.0) and 24 (16.0) passed.
+ //
+ // Catching up is not a reason to remove the table; it is the reason the table exists. The
+ // agreement is an accident of two version numbers lining up, and it un-happens on the
+ // next release in either direction. A port that read the predicate from the JDK would
+ // have been correct on 21 and wrong on 17 and 24, which is exactly the defect this port
+ // shipped in its pre-tokenizer.
+ //
+ // So the assertion is the JDK-independent one: whatever this JDK thinks, the TABLE is
+ // CPython's answer. Agreement is recorded rather than required.
+ boolean letters = predicateDigest(Character::isLetter).equals(digests().get("alpha"));
+ boolean upper = predicateDigest(Character::isUpperCase).equals(digests().get("upper"));
+ System.out.printf("JDK %s: isLetter agrees with CPython = %s, isUpperCase = %s%n",
+ Runtime.version().feature(), letters, upper);
+
+ // The part that must hold on every JDK: the tables are the reference's, not the JDK's.
+ assertEquals(digests().get("alpha"), predicateDigest(UnicodeTables::isAlpha),
+ "the ALPHA table no longer matches CPython");
+ assertEquals(digests().get("upper"), predicateDigest(UnicodeTables::isUpper),
+ "the UPPER table no longer matches CPython");
+ // And at least one of the four properties must still have no JDK equivalent, or there
+ // would be nothing left for these tables to do. `theJdkHasNoEquivalent` names which two.
+ assertFalse(letters && upper
+ && predicateDigest(Character::isWhitespace).equals(digests().get("space")),
+ "every JDK predicate now agrees with CPython; the tables can be reconsidered "
+ + "deliberately rather than kept out of habit");
+ }
+
@Test
@DisplayName("strip and splitOnWhitespace follow isSpace, not Character.isWhitespace")
void stripAndSplit() {