Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(java): ConfidenceGate, and the state a boolean cannot express

Adeshina Lasisi3e683fda211d
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index cc756551..b8a8ac0c 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -160,7 +160,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 625 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 680 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -319,7 +319,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 625 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 680 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml
index b00971a2..bb51d071 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -67,9 +67,10 @@ jobs:
# that lane already covered.
- name: Model-free tests
run: ./gradlew --no-daemon test
- # 631 of the 640 registered tests run without a checkpoint. The floor was 440, which
- # left 191 of slack -- the whole decode family could have vanished from this lane with
- # the gate still green, and a floor that does not bind is not a gate.
+ # 685 of the 699 registered tests run without a checkpoint. This floor has been raised
+ # as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor
+ # that does not bind is not a gate: under 625 the whole confidence-gate family of 26
+ # tests could have vanished from this lane with the gate still green.
#
# These lines sit ABOVE `run:`, not inside it. A `>` block folds to one line and `#` is
# not a comment inside a YAML scalar, so a comment written in there reaches the shell and
@@ -82,7 +83,7 @@ jobs:
- name: The suite was not vacuous
run: >
python3 scripts/check_test_results.py laya-java/build/test-results/test
- --min-tests 625 --allow-aborted
+ --min-tests 680 --allow-aborted
- name: Build the jars
run: ./gradlew --no-daemon build -x test
diff --git a/laya-java/CHANGELOG.md b/laya-java/CHANGELOG.md
index 1722b9bb..ef9dba65 100644
--- a/laya-java/CHANGELOG.md
+++ b/laya-java/CHANGELOG.md
@@ -20,6 +20,15 @@ see [Installing](README.md#installing).
`confidence` alongside `answerConfidence`.
- Python-compatible tokenization (BPE and Metaspace, added tokens, normalizers), sequence
building, collation, answer decoding and calibration.
+- `ConfidenceGate`: the opt-in abstention gate. Reports three states — `PASSED`, `ABSTAINED`
+ and `UNEVALUATED` — because an answer that carried no usable confidence is not a pass, and a
+ boolean cannot say so. Reads `answerConfidence` (what temperature scaling fits) and falls back
+ to `confidence` (normalised entropy, a different scale) only when the first is unusable.
+ Per-bucket thresholds are supported, since one threshold does not transfer across option
+ counts. Unlike the reference it does not mutate the answers — records cannot — so `apply`
+ returns a report, and an **ungated** call returns an empty `Optional` rather than a report of
+ nulls: the presence of the verdict is how a caller tells "no gate ran" from "everything
+ passed".
- `Agent.predictLong` and `LongPrediction`: questions answered over a state longer than the
context window. `predict` truncates such a state to one window and drops the rest silently;
this tokenizes it once, scans it in overlapping windows, scores them in shared graph calls and
diff --git a/laya-java/README.md b/laya-java/README.md
index 2ce22fb0..3337f767 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -301,6 +301,26 @@ String withSpecials = tok.decode(ids, false); // ...kept
AgentConfig cfg = AgentConfig.fromModelDirectory(Path.of("./checkpoint"));
```
+## Abstaining on a low-confidence answer
+
+```java
+var gated = ConfidenceGate.apply(prediction.answers(), 0.8); // null = no gate at all
+gated.ifPresent(report -> report.forEach((id, v) -> {
+ if (v.abstention() == ConfidenceGate.Abstention.ABSTAINED) {
+ escalate(id, v.answer(), v.threshold());
+ }
+}));
+```
+
+Three states, not a boolean: `UNEVALUATED` is an answer the gate could not read, and calling
+that a pass is the same mistake as calling it a flag. An **ungated** call returns an empty
+`Optional`, so the presence of a report is what tells you a gate ran.
+
+A threshold is not a claim that the number is calibrated — "about c of the answers returned at
+c are correct" holds only after temperatures have been fitted and validated for that checkpoint
+and question shape. Pass a `Map` instead of a scalar to gate each option-count bucket
+(`choice:2`, `score:6-10`, …) at the level its calibration actually earns.
+
## A state longer than the context window
```java
diff --git a/laya-java/fixtures/confidence_gate.json b/laya-java/fixtures/confidence_gate.json
new file mode 100644
index 00000000..89f0be8f
--- /dev/null
+++ b/laya-java/fixtures/confidence_gate.json
@@ -0,0 +1,829 @@
+{
+ "states": [
+ "passed",
+ "abstained",
+ "unevaluated"
+ ],
+ "cases": [
+ {
+ "case": "ungated-writes-nothing",
+ "min_confidence": null,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5
+ }
+ }
+ },
+ {
+ "case": "ungated-with-several",
+ "min_confidence": null,
+ "before": {
+ "a": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.333333,
+ "1": 0.333333,
+ "2": 0.333333
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.9,
+ "confidence": 0.8
+ },
+ "b": {
+ "type": "noul",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "noul": 0.6,
+ "answer_confidence": 0.4,
+ "confidence": 0.4
+ }
+ },
+ "after": {
+ "a": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.333333,
+ "1": 0.333333,
+ "2": 0.333333
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.9,
+ "confidence": 0.8
+ },
+ "b": {
+ "type": "noul",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "noul": 0.6,
+ "answer_confidence": 0.4,
+ "confidence": 0.4
+ }
+ }
+ },
+ {
+ "case": "scalar-abstains",
+ "min_confidence": 0.8,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5,
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ }
+ }
+ },
+ {
+ "case": "scalar-passes",
+ "min_confidence": 0.8,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.9,
+ "confidence": 0.5
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.9,
+ "confidence": 0.5,
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ }
+ }
+ },
+ {
+ "case": "scalar-exactly-at-threshold",
+ "min_confidence": 0.8,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.8,
+ "confidence": 0.5
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.8,
+ "confidence": 0.5,
+ "abstention": "passed",
+ "abstention_threshold": 0.8
+ }
+ }
+ },
+ {
+ "case": "scalar-zero-reports-passed",
+ "min_confidence": 0.0,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5,
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ },
+ {
+ "case": "scalar-one-abstains-everything",
+ "min_confidence": 1.0,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.99,
+ "confidence": 0.9
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.99,
+ "confidence": 0.9,
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 1.0
+ }
+ }
+ },
+ {
+ "case": "falls-back-to-confidence",
+ "min_confidence": 0.5,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "confidence": 0.3
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "confidence": 0.3,
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.5
+ }
+ }
+ },
+ {
+ "case": "prefers-answer-confidence",
+ "min_confidence": 0.5,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.9,
+ "confidence": 0.1
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.9,
+ "confidence": 0.1,
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ }
+ }
+ },
+ {
+ "case": "no-usable-confidence",
+ "min_confidence": 0.5,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0"
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.5
+ }
+ }
+ },
+ {
+ "case": "nan-is-not-a-confidence",
+ "min_confidence": 0.5,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": "__nan__",
+ "confidence": "__nan__"
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": "__nan__",
+ "confidence": "__nan__",
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.5
+ }
+ }
+ },
+ {
+ "case": "bool-is-not-a-confidence",
+ "min_confidence": 0.5,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": true
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": true,
+ "abstention": "unevaluated",
+ "abstention_threshold": 0.5
+ }
+ }
+ },
+ {
+ "case": "map-by-bucket",
+ "min_confidence": {
+ "choice:2": 0.9,
+ "choice:3-5": 0.1
+ },
+ "before": {
+ "two": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5
+ },
+ "five": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.2,
+ "1": 0.2,
+ "2": 0.2,
+ "3": 0.2,
+ "4": 0.2
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5
+ }
+ },
+ "after": {
+ "two": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5,
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.9
+ },
+ "five": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.2,
+ "1": 0.2,
+ "2": 0.2,
+ "3": 0.2,
+ "4": 0.2
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5,
+ "abstention": "passed",
+ "abstention_threshold": 0.1
+ }
+ }
+ },
+ {
+ "case": "map-default-for-unnamed-bucket",
+ "min_confidence": {
+ "choice:2": 0.9,
+ "default": 0.95
+ },
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.083333,
+ "1": 0.083333,
+ "2": 0.083333,
+ "3": 0.083333,
+ "4": 0.083333,
+ "5": 0.083333,
+ "6": 0.083333,
+ "7": 0.083333,
+ "8": 0.083333,
+ "9": 0.083333,
+ "10": 0.083333,
+ "11": 0.083333
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.083333,
+ "1": 0.083333,
+ "2": 0.083333,
+ "3": 0.083333,
+ "4": 0.083333,
+ "5": 0.083333,
+ "6": 0.083333,
+ "7": 0.083333,
+ "8": 0.083333,
+ "9": 0.083333,
+ "10": 0.083333,
+ "11": 0.083333
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5,
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.95
+ }
+ }
+ },
+ {
+ "case": "map-unnamed-bucket-no-default",
+ "min_confidence": {
+ "choice:2": 0.9
+ },
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.083333,
+ "1": 0.083333,
+ "2": 0.083333,
+ "3": 0.083333,
+ "4": 0.083333,
+ "5": 0.083333,
+ "6": 0.083333,
+ "7": 0.083333,
+ "8": 0.083333,
+ "9": 0.083333,
+ "10": 0.083333,
+ "11": 0.083333
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.083333,
+ "1": 0.083333,
+ "2": 0.083333,
+ "3": 0.083333,
+ "4": 0.083333,
+ "5": 0.083333,
+ "6": 0.083333,
+ "7": 0.083333,
+ "8": 0.083333,
+ "9": 0.083333,
+ "10": 0.083333,
+ "11": 0.083333
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.7,
+ "confidence": 0.5,
+ "abstention": "passed",
+ "abstention_threshold": 0.0
+ }
+ }
+ },
+ {
+ "case": "map-noul-bucket",
+ "min_confidence": {
+ "noul:2": 0.8
+ },
+ "before": {
+ "q": {
+ "type": "noul",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "noul": 0.6,
+ "answer_confidence": 0.6,
+ "confidence": 0.6
+ }
+ },
+ "after": {
+ "q": {
+ "type": "noul",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "noul": 0.6,
+ "answer_confidence": 0.6,
+ "confidence": 0.6,
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ }
+ }
+ },
+ {
+ "case": "map-score-bucket",
+ "min_confidence": {
+ "score:6-10": 0.8
+ },
+ "before": {
+ "q": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.142857,
+ "1": 0.142857,
+ "2": 0.142857,
+ "3": 0.142857,
+ "4": 0.142857,
+ "5": 0.142857,
+ "6": 0.142857
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "score": 1.0,
+ "legend": {
+ "0": "level 0",
+ "1": "level 1",
+ "2": "level 2",
+ "3": "level 3",
+ "4": "level 4",
+ "5": "level 5",
+ "6": "level 6"
+ },
+ "answer_confidence": 0.6,
+ "confidence": 0.6
+ }
+ },
+ "after": {
+ "q": {
+ "type": "score",
+ "probabilities": {
+ "0": 0.142857,
+ "1": 0.142857,
+ "2": 0.142857,
+ "3": 0.142857,
+ "4": 0.142857,
+ "5": 0.142857,
+ "6": 0.142857
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "score": 1.0,
+ "legend": {
+ "0": "level 0",
+ "1": "level 1",
+ "2": "level 2",
+ "3": "level 3",
+ "4": "level 4",
+ "5": "level 5",
+ "6": "level 6"
+ },
+ "answer_confidence": 0.6,
+ "confidence": 0.6,
+ "low_confidence": true,
+ "abstention": "abstained",
+ "abstention_threshold": 0.8
+ }
+ }
+ },
+ {
+ "case": "reflagging-clears-a-stale-flag",
+ "min_confidence": 0.5,
+ "before": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.9,
+ "confidence": 0.5,
+ "low_confidence": true
+ }
+ },
+ "after": {
+ "q": {
+ "type": "choice",
+ "probabilities": {
+ "0": 0.5,
+ "1": 0.5
+ },
+ "action": {
+ "act_probability": 0.5
+ },
+ "choice": "0",
+ "answer_confidence": 0.9,
+ "confidence": 0.5,
+ "abstention": "passed",
+ "abstention_threshold": 0.5
+ }
+ }
+ }
+ ],
+ "refusals": [
+ {
+ "input": "1.5",
+ "error": "min_confidence must be a float in [0.0, 1.0], got 1.5"
+ },
+ {
+ "input": "-0.1",
+ "error": "min_confidence must be a float in [0.0, 1.0], got -0.1"
+ },
+ {
+ "input": "True",
+ "error": "min_confidence must be a float in [0.0, 1.0], got True"
+ },
+ {
+ "input": "'0.5'",
+ "error": "min_confidence must be a float in [0.0, 1.0], got '0.5'"
+ },
+ {
+ "input": "nan",
+ "error": "min_confidence must be a float in [0.0, 1.0], got nan"
+ },
+ {
+ "input": "{}",
+ "error": "a min_confidence map must be a non-empty dict of bucket -> float, got {}"
+ },
+ {
+ "input": "{'choice:2': 2.0}",
+ "error": "min_confidence must be a float in [0.0, 1.0], got 2.0"
+ },
+ {
+ "input": "{1: 0.5}",
+ "error": "min_confidence map keys must be strings like 'choice:3-5', got 1"
+ }
+ ]
+}
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/ConfidenceGate.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/ConfidenceGate.java
new file mode 100644
index 00000000..635657a3
--- /dev/null
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/ConfidenceGate.java
@@ -0,0 +1,235 @@
+package com.convaiinnovations.laya;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalDouble;
+
+/**
+ * The opt-in abstention gate: which answers a caller should not act on without review.
+ *
+ * <p>A gate is a policy, and a policy whose application cannot be observed is not one. So this
+ * reports three states rather than a boolean — an answer that carried no usable confidence is
+ * {@link Abstention#UNEVALUATED}, which a boolean cannot express, and reporting it as a pass is
+ * the same lie as reporting it as a flag.
+ *
+ * <p><b>This does not mutate.</b> The reference writes {@code abstention},
+ * {@code abstention_threshold} and {@code low_confidence} into the answer dicts in place, which
+ * a record cannot do. {@link #apply} returns a report keyed the same way as the answers instead,
+ * and an UNGATED call returns {@link Optional#empty()} rather than a report full of nulls —
+ * because the reference's contract is that an ungated call writes <em>nothing at all</em>, and
+ * the presence of the field is what tells a caller the gate ran. A sentinel for the unconfigured
+ * case would put that information back into every caller's payload, which is what the contract
+ * avoids.
+ *
+ * <p>The gate reads {@code answerConfidence} — the probability mass on the answer being
+ * reported, which is what temperature scaling fits — and falls back to {@code confidence} only
+ * when the first is unusable. Those are different quantities on different scales, and
+ * {@code confidence} is normalised entropy that does not transfer across option counts, so the
+ * order matters and is pinned by the fixtures.
+ *
+ * <p>A threshold is <em>not</em> a claim that the number is calibrated. "About c of the answers
+ * returned at c are correct" holds only after temperatures have been fitted and validated for
+ * that checkpoint and question shape.
+ */
+public final class ConfidenceGate {
+
+ private ConfidenceGate() {
+ }
+
+ /** What the gate concluded about one answer. The three states, and only these three. */
+ public enum Abstention {
+ /** A gate ran and this answer's confidence cleared it. */
+ PASSED("passed"),
+ /** A gate ran and this answer's confidence fell below it. */
+ ABSTAINED("abstained"),
+ /** A gate ran and this answer carried no usable confidence, so it could not decide. */
+ UNEVALUATED("unevaluated");
+
+ private final String wireName;
+
+ Abstention(String wireName) {
+ this.wireName = wireName;
+ }
+
+ /** The name the reference writes, which is what a cross-language caller compares. */
+ public String wireName() {
+ return wireName;
+ }
+ }
+
+ /**
+ * One answer's verdict.
+ *
+ * @param lowConfidence the reference's {@code low_confidence} flag. Redundant with
+ * {@link Abstention#ABSTAINED} and kept because it is what the wire format carries and
+ * what an existing caller filters on
+ * @param threshold the threshold THIS answer was gated at, which differs per answer
+ * under a per-bucket map. Echoed because the flag consumes the threshold and drops it,
+ * so without this a batch gated per bucket cannot be re-split
+ */
+ public record Gated(Answer answer, boolean lowConfidence, Abstention abstention,
+ double threshold) {
+ }
+
+ /**
+ * The calibrated confidence, or empty when the answer did not report a usable one.
+ *
+ * <p>In the reference this also rejects a bool and a missing field. Neither is representable
+ * here — {@link Answer#answerConfidence()} is a primitive {@code double} that every answer
+ * has — so NaN and infinity are the only ways to be unusable on this side. The reference's
+ * other two paths are unreachable rather than unimplemented.
+ */
+ public static OptionalDouble answerConfidence(Answer answer) {
+ double value = answer.answerConfidence();
+ return Double.isFinite(value) ? OptionalDouble.of(value) : OptionalDouble.empty();
+ }
+
+ /**
+ * The number the gate compares against the threshold, or empty when there is none.
+ *
+ * <p>{@code answerConfidence} first, falling back to {@code confidence} so an answer
+ * carrying only the older field is still gated rather than silently passed. Empty means "no
+ * usable number", never an unusable one: a caller that reads the value must be able to tell
+ * "nothing to gate on" from "a gate ran on a NaN", and {@link #apply} reports those
+ * differently.
+ */
+ public static OptionalDouble gateConfidence(Answer answer) {
+ OptionalDouble calibrated = answerConfidence(answer);
+ if (calibrated.isPresent()) {
+ return calibrated;
+ }
+ double fallback = answer.confidence();
+ return Double.isFinite(fallback) ? OptionalDouble.of(fallback) : OptionalDouble.empty();
+ }
+
+ /**
+ * This answer's option-count bucket, in the reference's spelling, or empty when it has none.
+ *
+ * <p>{@code "choice:2"}, {@code "choice:3-5"}, {@code "score:6-10"}, {@code "noul:2"} and so
+ * on. One confidence threshold does not transfer across option counts, which is why a map
+ * can gate each bucket at the level its calibration actually earns.
+ */
+ public static Optional<String> optionBucket(Answer answer) {
+ int options;
+ if (answer instanceof Answer.Choice choice) {
+ options = choice.probabilities().size();
+ } else if (answer instanceof Answer.Score score) {
+ options = score.probabilities().size();
+ } else if (answer instanceof Answer.Noul) {
+ // A noul carries no probability map, and the reference falls back to 2 for it rather
+ // than refusing to bucket it.
+ options = 2;
+ } else {
+ return Optional.empty();
+ }
+ if (options == 0) {
+ return Optional.empty();
+ }
+ String size = options <= 2 ? "2" : options <= 5 ? "3-5" : options <= 10 ? "6-10" : "11+";
+ return Optional.of(answer.type() + ":" + size);
+ }
+
+ /**
+ * The threshold this answer's bucket is gated at under a per-bucket map.
+ *
+ * <p>Falls back to the map's {@code "default"} entry, then to 0.0 — gate nothing — so a
+ * bucket the map does not name never abstains by surprise.
+ */
+ public static double resolve(Answer answer, Map<String, Double> thresholds) {
+ Optional<String> bucket = optionBucket(answer);
+ if (bucket.isPresent()) {
+ Double exact = thresholds.get(bucket.get());
+ if (exact != null) {
+ return exact;
+ }
+ }
+ return thresholds.getOrDefault("default", 0.0);
+ }
+
+ /** Validates a scalar threshold, returning it. */
+ public static double checkMinConfidence(double value) {
+ if (!Double.isFinite(value) || value < 0.0 || value > 1.0) {
+ throw new IllegalArgumentException(String.format(
+ "min_confidence must be a float in [0.0, 1.0], got %s", value));
+ }
+ return value;
+ }
+
+ /** Validates a per-bucket threshold map, returning it. */
+ public static Map<String, Double> checkMinConfidenceMap(Map<String, ? extends Number> map) {
+ if (map == null || map.isEmpty()) {
+ throw new IllegalArgumentException(
+ "a min_confidence map must be a non-empty map of bucket -> float, got " + map);
+ }
+ Map<String, Double> out = new LinkedHashMap<>();
+ for (Map.Entry<String, ? extends Number> entry : map.entrySet()) {
+ if (entry.getKey() == null) {
+ throw new IllegalArgumentException(
+ "min_confidence map keys must be strings like 'choice:3-5', got null");
+ }
+ out.put(entry.getKey(), checkMinConfidence(entry.getValue().doubleValue()));
+ }
+ return Map.copyOf(out);
+ }
+
+ /**
+ * Gates every answer at one threshold.
+ *
+ * @param minConfidence the threshold, or null for an UNGATED call
+ * @return the report, or empty when no gate was configured — which is how a caller tells
+ * "this run had no gate" from "every answer cleared the gate"
+ */
+ public static Optional<Map<String, Gated>> apply(Map<String, Answer> answers,
+ Double minConfidence) {
+ if (minConfidence == null) {
+ return Optional.empty();
+ }
+ double threshold = checkMinConfidence(minConfidence);
+ return Optional.of(gate(answers, answer -> threshold, threshold == 0.0));
+ }
+
+ /**
+ * Gates each answer at its own bucket's threshold.
+ *
+ * @param minConfidence the map, or null for an UNGATED call
+ */
+ public static Optional<Map<String, Gated>> apply(Map<String, Answer> answers,
+ Map<String, ? extends Number> minConfidence) {
+ if (minConfidence == null) {
+ return Optional.empty();
+ }
+ Map<String, Double> thresholds = checkMinConfidenceMap(minConfidence);
+ // A map is never the "0.0 clears the flag" case: the reference applies that only to a
+ // scalar 0.0, because under a map each answer has its own threshold and one of them
+ // being zero says nothing about the others.
+ return Optional.of(gate(answers, answer -> resolve(answer, thresholds), false));
+ }
+
+ private interface ThresholdFor {
+ double of(Answer answer);
+ }
+
+ private static Map<String, Gated> gate(Map<String, Answer> answers, ThresholdFor thresholds,
+ boolean scalarZero) {
+ Map<String, Gated> out = new LinkedHashMap<>();
+ for (Map.Entry<String, Answer> entry : answers.entrySet()) {
+ Answer answer = entry.getValue();
+ double threshold = thresholds.of(answer);
+ OptionalDouble confidence = gateConfidence(answer);
+
+ boolean low = confidence.isPresent() && confidence.getAsDouble() < threshold;
+ if (scalarZero) {
+ // A scalar 0.0 was set, so states ARE reported -- but nothing can fall below it,
+ // and the reference clears any stale flag rather than leaving one from an
+ // earlier, stricter run.
+ low = false;
+ }
+ Abstention state = low ? Abstention.ABSTAINED
+ : confidence.isEmpty() ? Abstention.UNEVALUATED
+ : Abstention.PASSED;
+ out.put(entry.getKey(), new Gated(answer, low, state, threshold));
+ }
+ return Map.copyOf(out);
+ }
+}
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/ConfidenceGateTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/ConfidenceGateTest.java
new file mode 100644
index 00000000..e9020f48
--- /dev/null
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/ConfidenceGateTest.java
@@ -0,0 +1,265 @@
+package com.convaiinnovations.laya;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.DynamicTest.dynamicTest;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.DynamicTest;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestFactory;
+
+/**
+ * {@link ConfidenceGate} against {@code fixtures/confidence_gate.json}, recorded from
+ * {@code laya.confidence}.
+ *
+ * <p>The gate's whole contract is what it writes, so the fixture records each answer before and
+ * after and this asserts the difference. Three things it pins that a port gets wrong by default:
+ *
+ * <ul>
+ * <li>WHICH NUMBER is read. {@code answerConfidence} first, {@code confidence} only as a
+ * fallback — different quantities on different scales, so a port reading the wrong one
+ * gates at a number that looks plausible and is not the one the calibration fitted.</li>
+ * <li>That an UNGATED call writes NOTHING. The presence of the verdict is how a caller tells
+ * "no gate ran" from "everything passed", so a port that reports a fourth "unconfigured"
+ * state breaks that distinction for every caller.</li>
+ * <li>That a scalar {@code 0.0} IS a configured gate. Nothing can fall below it, so every
+ * answer with a usable number reads {@code passed} and any stale flag is cleared.</li>
+ * </ul>
+ *
+ * <p>Two of the reference's "unusable confidence" paths — a missing field and a bool — are
+ * unrepresentable here, because {@link Answer#answerConfidence()} is a primitive double. Those
+ * cases are asserted to reach {@code UNEVALUATED} through the one route that remains, NaN,
+ * rather than being skipped.
+ */
+final class ConfidenceGateTest {
+
+ @SuppressWarnings("unchecked")
+ private static Map<String, Object> family() {
+ return (Map<String, Object>) (Map<?, ?>) Fixtures.load("confidence_gate.json");
+ }
+
+ /** One recorded answer, as the typed record. {@code "__nan__"} is how the fixture spells NaN. */
+ @SuppressWarnings("unchecked")
+ private static Answer answerOf(Map<String, Object> recorded) {
+ double confidence = number(recorded.get("confidence"));
+ double answerConfidence = number(recorded.get("answer_confidence"));
+ double act = number(((Map<String, Object>) recorded.get("action")).get("act_probability"));
+ Map<String, Double> probabilities = new LinkedHashMap<>();
+ ((Map<String, Object>) recorded.get("probabilities"))
+ .forEach((k, v) -> probabilities.put(k, number(v)));
+ switch ((String) recorded.get("type")) {
+ case "noul":
+ return new Answer.Noul(number(recorded.get("noul")), confidence,
+ answerConfidence, act);
+ case "score": {
+ Map<String, String> legend = new LinkedHashMap<>();
+ ((Map<String, Object>) recorded.get("legend"))
+ .forEach((k, v) -> legend.put(k, (String) v));
+ return new Answer.Score(number(recorded.get("score")), legend, probabilities,
+ confidence, answerConfidence, act);
+ }
+ default:
+ return new Answer.Choice((String) recorded.get("choice"), probabilities,
+ confidence, answerConfidence, act);
+ }
+ }
+
+ /**
+ * A recorded number.
+ *
+ * <p>An ABSENT field becomes NaN, not zero. The reference treats "no usable confidence" and
+ * "a confidence of 0.0" as different outcomes — one is {@code unevaluated} and the other is
+ * {@code abstained} — and a port that defaulted the absent case to 0.0 would turn every
+ * answer the gate could not evaluate into one it rejected.
+ */
+ private static double number(Object value) {
+ if (value == null || "__nan__".equals(value) || value instanceof Boolean) {
+ // Absent, NaN and a BOOL all mean "no usable confidence". The reference rejects a
+ // bool explicitly, because `isinstance(True, int)` is true in Python and `True`
+ // would otherwise gate as 1.0 -- an answer that always clears every threshold. A
+ // primitive double cannot hold one, so NaN is the nearest thing Java can express
+ // and the outcome it must produce is the same: UNEVALUATED.
+ //
+ // Absent is NOT zero. "No usable confidence" and "a confidence of 0.0" are
+ // different outcomes -- unevaluated against abstained -- and defaulting the first
+ // to zero would turn every answer the gate could not read into one it rejected.
+ return Double.NaN;
+ }
+ return ((Number) value).doubleValue();
+ }
+
+ @TestFactory
+ @DisplayName("every recorded gate outcome matches, including the no-op")
+ @SuppressWarnings("unchecked")
+ List<DynamicTest> recordedOutcomesMatch() {
+ List<DynamicTest> tests = new ArrayList<>();
+ int ungated = 0;
+ int unevaluated = 0;
+ for (Object raw : (List<Object>) family().get("cases")) {
+ Map<String, Object> one = (Map<String, Object>) raw;
+ Object threshold = one.get("min_confidence");
+ if (threshold == null) {
+ ungated++;
+ }
+ Map<String, Object> after = (Map<String, Object>) one.get("after");
+ for (Object a : after.values()) {
+ if ("unevaluated".equals(((Map<String, Object>) a).get("abstention"))) {
+ unevaluated++;
+ }
+ }
+ tests.add(dynamicTest((String) one.get("case"), () -> {
+ Map<String, Answer> answers = new LinkedHashMap<>();
+ ((Map<String, Object>) one.get("before")).forEach(
+ (qid, value) -> answers.put(qid, answerOf((Map<String, Object>) value)));
+
+ Optional<Map<String, ConfidenceGate.Gated>> report;
+ if (threshold == null) {
+ report = ConfidenceGate.apply(answers, (Double) null);
+ } else if (threshold instanceof Map) {
+ report = ConfidenceGate.apply(answers,
+ (Map<String, ? extends Number>) threshold);
+ } else {
+ report = ConfidenceGate.apply(answers, ((Number) threshold).doubleValue());
+ }
+
+ if (threshold == null) {
+ // The whole no-op contract: not an empty report, NO report. The presence of
+ // the verdict is what tells a caller the gate ran.
+ assertTrue(report.isEmpty(),
+ "an ungated call must produce no verdict at all");
+ for (Object value : after.values()) {
+ Map<String, Object> a = (Map<String, Object>) value;
+ assertFalse(a.containsKey("abstention"),
+ "the reference wrote nothing, so nothing may be reported");
+ assertFalse(a.containsKey("abstention_threshold"), "nor a threshold");
+ }
+ return;
+ }
+
+ Map<String, ConfidenceGate.Gated> got = report.orElseThrow();
+ assertEquals(after.keySet(), got.keySet(), "one verdict per answer");
+ for (Map.Entry<String, Object> expected : after.entrySet()) {
+ Map<String, Object> a = (Map<String, Object>) expected.getValue();
+ ConfidenceGate.Gated verdict = got.get(expected.getKey());
+ String id = one.get("case") + "/" + expected.getKey();
+
+ assertEquals(a.get("abstention"), verdict.abstention().wireName(),
+ id + " abstention state");
+ assertEquals(number(a.get("abstention_threshold")), verdict.threshold(),
+ 1e-12, id + " echoed threshold");
+ // `low_confidence` is ABSENT in the reference when it does not apply, which
+ // is false here -- and must not be true merely because the answer abstained
+ // for want of a number.
+ assertEquals(Boolean.TRUE.equals(a.get("low_confidence")),
+ verdict.lowConfidence(), id + " low_confidence flag");
+ }
+ }));
+ }
+ final int noop = ungated;
+ final int unknown = unevaluated;
+ assertTrue(noop >= 2, () -> "the no-op contract needs cases and has " + noop);
+ assertTrue(unknown >= 3,
+ () -> "UNEVALUATED is the state a boolean cannot express and only " + unknown
+ + " case(s) reach it");
+ return tests;
+ }
+
+ @TestFactory
+ @DisplayName("every threshold the reference refuses is refused here")
+ @SuppressWarnings("unchecked")
+ List<DynamicTest> refusalsMatch() {
+ List<DynamicTest> tests = new ArrayList<>();
+ for (Object raw : (List<Object>) family().get("refusals")) {
+ Map<String, Object> one = (Map<String, Object>) raw;
+ String input = (String) one.get("input");
+ boolean refused = one.get("error") != null;
+ // Only the cases Java can express: a string threshold and a non-string map key are
+ // compile errors here, not runtime ones, which is a stronger guarantee than the
+ // reference's and is why they are skipped rather than asserted.
+ Double scalar = switch (input) {
+ case "1.5" -> 1.5;
+ case "-0.1" -> -0.1;
+ case "nan" -> Double.NaN;
+ default -> null;
+ };
+ if (scalar == null) {
+ continue;
+ }
+ tests.add(dynamicTest("scalar " + input, () -> {
+ assertTrue(refused, "the reference refuses " + input);
+ assertThrows(IllegalArgumentException.class,
+ () -> ConfidenceGate.checkMinConfidence(scalar),
+ input + " must be refused");
+ }));
+ }
+ tests.add(dynamicTest("an empty map", () ->
+ assertThrows(IllegalArgumentException.class,
+ () -> ConfidenceGate.checkMinConfidenceMap(Map.of()),
+ "an empty map is not a gate")));
+ tests.add(dynamicTest("a map with an out-of-range value", () ->
+ assertThrows(IllegalArgumentException.class,
+ () -> ConfidenceGate.checkMinConfidenceMap(Map.of("choice:2", 2.0)),
+ "a value outside [0,1] is refused wherever it appears")));
+ return tests;
+ }
+
+ @Test
+ @DisplayName("the option bucket is the reference's, including the noul fallback")
+ void buckets() {
+ assertEquals(Optional.of("choice:2"), ConfidenceGate.optionBucket(
+ choice(2)), "two options");
+ assertEquals(Optional.of("choice:3-5"), ConfidenceGate.optionBucket(choice(4)));
+ assertEquals(Optional.of("choice:3-5"), ConfidenceGate.optionBucket(choice(5)),
+ "5 is the top of the 3-5 band, not the bottom of the next");
+ assertEquals(Optional.of("choice:6-10"), ConfidenceGate.optionBucket(choice(6)));
+ assertEquals(Optional.of("choice:11+"), ConfidenceGate.optionBucket(choice(12)));
+ // A noul carries no probability map at all, and the reference buckets it as 2 rather
+ // than refusing -- so a per-bucket map can name `noul:2` and have it apply.
+ assertEquals(Optional.of("noul:2"),
+ ConfidenceGate.optionBucket(new Answer.Noul(0.6, 0.6, 0.6, 0.5)));
+ }
+
+ @Test
+ @DisplayName("an unnamed bucket with no default gates at zero, so it never abstains")
+ void unnamedBucketNeverAbstainsBySurprise() {
+ Answer twelve = choice(12);
+ assertEquals(0.0, ConfidenceGate.resolve(twelve, Map.of("choice:2", 0.9)), 1e-12,
+ "an unconfigured bucket must not inherit another bucket's threshold");
+ assertEquals(0.95, ConfidenceGate.resolve(twelve,
+ Map.of("choice:2", 0.9, "default", 0.95)), 1e-12, "...but a default applies");
+ }
+
+ @Test
+ @DisplayName("gateConfidence prefers the calibrated number and falls back to entropy")
+ void whichNumberIsRead() {
+ Answer both = new Answer.Choice("a", Map.of("a", 0.9, "b", 0.1), 0.1, 0.9, 0.5);
+ assertEquals(0.9, ConfidenceGate.gateConfidence(both).orElseThrow(), 1e-12,
+ "answerConfidence wins; the two differ here so a port reading the wrong one "
+ + "gates at 0.1 instead of 0.9");
+ Answer onlyEntropy = new Answer.Choice("a", Map.of("a", 0.9, "b", 0.1),
+ 0.3, Double.NaN, 0.5);
+ assertEquals(0.3, ConfidenceGate.gateConfidence(onlyEntropy).orElseThrow(), 1e-12,
+ "an answer carrying only the older field is still gated, not silently passed");
+ Answer neither = new Answer.Choice("a", Map.of("a", 0.9, "b", 0.1),
+ Double.NaN, Double.NaN, 0.5);
+ assertTrue(ConfidenceGate.gateConfidence(neither).isEmpty(),
+ "no usable number is empty, never zero -- zero would read as a real confidence "
+ + "that fails every threshold");
+ }
+
+ private static Answer choice(int options) {
+ Map<String, Double> probabilities = new LinkedHashMap<>();
+ for (int i = 0; i < options; i++) {
+ probabilities.put(Integer.toString(i), 1.0 / options);
+ }
+ return new Answer.Choice("0", probabilities, 0.5, 0.7, 0.5);
+ }
+}
diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.py
index 676cc668..1b335cb9 100644
--- a/laya-java/scripts/gen_fixtures.py
+++ b/laya-java/scripts/gen_fixtures.py
@@ -2935,6 +2935,140 @@ def predict_long_scan():
" batches; this family exists to pin those" % (name, multi, spans, capped))
return out
+#: (case id, answers, min_confidence). The gate's whole contract is observable from the payload
+#: it writes, so each case records the answer dicts before and after.
+CONFIDENCE_CASES = [
+ # The no-op contract. An ungated call must return EXACTLY the payload it returned before --
+ # no `abstention`, no `abstention_threshold`, no flag -- because the PRESENCE of the field is
+ # what tells a caller the gate ran.
+ ("ungated-writes-nothing", [("q", "choice", 0.7, 0.5, 2)], None),
+ ("ungated-with-several", [("a", "choice", 0.9, 0.8, 3), ("b", "noul", 0.4, 0.4, 2)], None),
+ # A scalar threshold.
+ ("scalar-abstains", [("q", "choice", 0.7, 0.5, 2)], 0.8),
+ ("scalar-passes", [("q", "choice", 0.9, 0.5, 2)], 0.8),
+ ("scalar-exactly-at-threshold", [("q", "choice", 0.8, 0.5, 2)], 0.8),
+ # 0.0 WAS set, so states are reported -- and nothing can fall below it, so every answer with
+ # a usable confidence reads `passed`. The threshold echo is what distinguishes that from a
+ # real pass at a real threshold.
+ ("scalar-zero-reports-passed", [("q", "choice", 0.7, 0.5, 2)], 0.0),
+ ("scalar-one-abstains-everything", [("q", "choice", 0.99, 0.9, 2)], 1.0),
+ # `answer_confidence` first, `confidence` only as a fallback -- a DIFFERENT quantity on a
+ # different scale, so a port that reads the wrong one gates at the wrong number.
+ ("falls-back-to-confidence", [("q", "choice", None, 0.3, 2)], 0.5),
+ ("prefers-answer-confidence", [("q", "choice", 0.9, 0.1, 2)], 0.5),
+ # No usable number is NOT a pass and NOT a flag: the gate ran and could not decide.
+ ("no-usable-confidence", [("q", "choice", None, None, 2)], 0.5),
+ ("nan-is-not-a-confidence", [("q", "choice", "nan", "nan", 2)], 0.5),
+ ("bool-is-not-a-confidence", [("q", "choice", True, None, 2)], 0.5),
+ # Per-bucket thresholds: one number does not transfer across option counts.
+ ("map-by-bucket", [("two", "choice", 0.7, 0.5, 2), ("five", "choice", 0.7, 0.5, 5)],
+ {"choice:2": 0.9, "choice:3-5": 0.1}),
+ ("map-default-for-unnamed-bucket", [("q", "choice", 0.7, 0.5, 12)],
+ {"choice:2": 0.9, "default": 0.95}),
+ ("map-unnamed-bucket-no-default", [("q", "choice", 0.7, 0.5, 12)], {"choice:2": 0.9}),
+ ("map-noul-bucket", [("q", "noul", 0.6, 0.6, 2)], {"noul:2": 0.8}),
+ ("map-score-bucket", [("q", "score", 0.6, 0.6, 7)], {"score:6-10": 0.8}),
+ # A previously-flagged answer that now clears the bar must have the flag REMOVED, not left.
+ ("reflagging-clears-a-stale-flag", [("q", "choice", 0.9, 0.5, 2)], 0.5),
+]
+
+
+def _conf_answer(qtype, answer_confidence, confidence, options):
+ """One answer dict in the shape `predict` returns."""
+ def number(v):
+ return float("nan") if v == "nan" else v
+ probabilities = {str(i): round(1.0 / options, 6) for i in range(options)}
+ answer = {"type": qtype, "probabilities": probabilities,
+ "action": {"act_probability": 0.5}}
+ if qtype == "choice":
+ answer["choice"] = "0"
+ elif qtype == "score":
+ answer["score"] = 1.0
+ answer["legend"] = {str(i): "level %d" % i for i in range(options)}
+ else:
+ answer["noul"] = 0.6
+ if answer_confidence is not None:
+ answer["answer_confidence"] = number(answer_confidence)
+ if confidence is not None:
+ answer["confidence"] = number(confidence)
+ return answer
+
+
+def confidence_gate():
+ """`laya.confidence`: which number the gate reads, and what it writes.
+
+ The gate is a policy, and a policy whose application cannot be observed is not one. Its whole
+ contract is the payload it writes, so each case records the answers BEFORE and AFTER, and the
+ cases are chosen so a port that reads the wrong field or writes the wrong state differs
+ somewhere:
+
+ * `answer_confidence` is the quantity temperature scaling fits; `confidence` is normalized
+ entropy, a different quantity on a different scale. The gate reads the first and falls
+ back to the second, so `prefers-answer-confidence` and `falls-back-to-confidence` differ
+ only in which field is present and must give different thresholds' worth of behaviour.
+ * an UNSET threshold writes nothing at all -- the presence of `abstention` is how a caller
+ tells "this run had no gate" from "this answer cleared the gate", so a port that writes
+ a sentinel for the unconfigured case breaks that distinction for every caller.
+ * `0.0` IS a threshold that was set. Nothing can fall below it, so every answer with a
+ usable number reads `passed`, and the echoed threshold is what distinguishes that from a
+ real pass at a real bar.
+ * a bool and a NaN are not confidences. Both give "no usable number", which the gate
+ reports as `unevaluated` -- not as a pass, which is the same lie as reporting it as a flag.
+ """
+ import copy
+ import math
+
+ from laya.confidence import GATE_STATES, apply_confidence_gate, check_min_confidence
+
+ cases = []
+ for cid, answers, threshold in CONFIDENCE_CASES:
+ payload = {qid: _conf_answer(t, ac, c, n) for qid, t, ac, c, n in answers}
+ if cid == "reflagging-clears-a-stale-flag":
+ # A result dict reused from an earlier, stricter run.
+ for a in payload.values():
+ a["low_confidence"] = True
+ before = copy.deepcopy(payload)
+ validated = None if threshold is None else check_min_confidence(threshold)
+ results = [{"model": "laya-rl-agent-onnx", "answers": payload, "usage": {}}]
+ apply_confidence_gate(results, validated)
+
+ def clean(d):
+ # NaN does not survive JSON, and recording it as null would lose the distinction
+ # between "absent" and "not a number" -- which is exactly what this family tests.
+ out = {}
+ for k, v in d.items():
+ if isinstance(v, float) and math.isnan(v):
+ out[k] = "__nan__"
+ elif isinstance(v, dict):
+ out[k] = clean(v)
+ else:
+ out[k] = v
+ return out
+
+ cases.append({
+ "case": cid,
+ "min_confidence": threshold,
+ "before": {q: clean(a) for q, a in before.items()},
+ "after": {q: clean(a) for q, a in results[0]["answers"].items()},
+ })
+
+ # Validation is part of the contract: a threshold outside [0,1], a bool, or an empty map is
+ # refused, and the message is what a caller sees.
+ refusals = []
+ for bad in (1.5, -0.1, True, "0.5", float("nan"), {}, {"choice:2": 2.0}, {1: 0.5}):
+ try:
+ check_min_confidence(bad)
+ refusals.append({"input": repr(bad), "error": None})
+ except (ValueError, TypeError) as problem:
+ refusals.append({"input": repr(bad), "error": str(problem)})
+
+ gated = sum(1 for c in cases if any("abstention" in a for a in c["after"].values()))
+ if gated < 12 or len(GATE_STATES) != 3:
+ raise AssertionError(
+ "only %d case(s) reach a gated answer across %d state(s); this family exists to pin"
+ " the three states and the no-op" % (gated, len(GATE_STATES)))
+ return {"states": list(GATE_STATES), "cases": cases, "refusals": refusals}
+
def _decoder_shape(tokenizer_json):
"""The decoder section's type chain, flattened, so the fixture says what it was recorded on."""
import io
@@ -2956,6 +3090,7 @@ def _decoder_shape(tokenizer_json):
FAMILIES = {
+ "confidence_gate.json": confidence_gate,
"predict_long.json": predict_long_scan,
"window_plan.json": window_plan,
"lang_tables.json": lang_tables,