Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(go): harden the predict path's public contract after review

Adeshina Lasisie0b510e86a4e
diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml
index 4cbe5182..712dc702 100644
--- a/.github/workflows/go.yml
+++ b/.github/workflows/go.yml
@@ -127,11 +127,11 @@ jobs:
LAYA_GO_FIXTURES: ${{ github.workspace }}/.go-goldens
LAYA_CHECKPOINTS: ${{ github.workspace }}/laya-java/.work/checkpoints
run: go test -race -count=1 -json ./... > "$RUNNER_TEMP/go-test.json"
- # Exactly the measured count (1254 tests and subtests, on both Go versions): dropping a
+ # Exactly the measured count (1307 tests and subtests, on both Go versions): dropping a
# case fails here, and adding one means raising this number.
- name: No test was skipped
if: ${{ !cancelled() }}
- run: python laya-go/scripts/check_test_json.py "$RUNNER_TEMP/go-test.json" --min-tests 1254
+ run: python laya-go/scripts/check_test_json.py "$RUNNER_TEMP/go-test.json" --min-tests 1307
# go.mod declares go 1.22, so that is what a consumer may build with.
- name: Run the tests on the oldest supported Go
@@ -143,7 +143,7 @@ jobs:
run: go test -count=1 -json ./... > "$RUNNER_TEMP/go-test-oldest.json"
- name: No test was skipped on the oldest Go either
if: ${{ !cancelled() }}
- run: python laya-go/scripts/check_test_json.py "$RUNNER_TEMP/go-test-oldest.json" --min-tests 1254
+ run: python laya-go/scripts/check_test_json.py "$RUNNER_TEMP/go-test-oldest.json" --min-tests 1307
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
if: ${{ failure() }}
diff --git a/laya-go/README.md b/laya-go/README.md
index 324b9ddb..43509b8f 100644
--- a/laya-go/README.md
+++ b/laya-go/README.md
@@ -2,8 +2,8 @@
A pure-Go port of [laya](../README.md). Standard library only: no cgo, no third-party modules.
-This first slice is the foundation the rest of the port stands on. The routing and model layers
-(ONNX, agent, router) come in later changes.
+The foundation (language detection, questions, sequences, the tokenizer) and the predict path
+(`infer`, `agent`) are here. The native ONNX session and the router come in later changes.
| Package | What it is |
| --- | --- |
@@ -11,7 +11,7 @@ This first slice is the foundation the rest of the port stands on. The routing a
| `schema` | Questions (`Choice`, `ChoiceOf`, `Score`, `Noul`, `NoulWith`, `Questions`) and results (`Answer`, `Prediction`, `Usage`) every other package shares. Questions are refused with laya's own wording and read or written in the Python wire form. |
| `sequence` | `build_sequence`, `build_head`, `state_room`, `window_budget` and `window_batch_cap`: `EncodeState`, `Build`, `StateRoom`, `PlanWindows`, `BatchCap`. |
| `infer` | The seam to a model runtime: `Batch` (the graph's five inputs), `Output` (its two), `Session`, and the shape checks `Batch.Validate` and `Output.Check`. |
-| `agent` | `ONNXAgent.predict` and `predict_batch` over an `infer.Session`: `Open`, `Predict`, `PredictBatch`, `LoadConfig`, with batch size, sort-by-length and context cancellation. |
+| `agent` | `ONNXAgent.predict` and `predict_batch` over an `infer.Session`: `Open`, `Predict`, `PredictBatch`, `LoadConfig`, with batch size, sort-by-length and context cancellation between runs. |
| `tokenizer` | A reader and runner for Hugging Face `tokenizer.json` (BPE): normalizer, pre-tokenizer, added tokens, byte fallback, decoder. |
| `internal/unicodedata` | The character predicates `laya/lang.py` leans on, recorded from CPython. |
| `internal/pyjson` | `json.dumps` and `repr` spelled as CPython spells them. |
@@ -39,11 +39,13 @@ order-sensitive state. `Analyze` reads every leaf, so its cost grows with the nu
## Layout
- No root package: the module path ends in `laya-go`, so tags are `laya-go/vX.Y.Z`.
-- Pure-Go packages are `lang` now and `router`, `presets` and `shortlist` later.
-- cgo, if ever needed, lives only in a future `onnx` package, with a `//go:build !cgo` stub.
- Importing `lang` never compiles cgo.
+- Every package in this module is pure Go: `lang`, `schema`, `sequence`, `tokenizer`, `infer` and
+ `agent` now, and `router`, `presets` and `shortlist` later.
+- cgo will live only in an `onnx` package, a nested module with its own `go.mod`, so `go test ./...`
+ here never compiles it. Importing `lang` or `agent` never compiles cgo.
- `tokenizer` is public (`LoadFS`, `Parse`); `internal/unicodedata`, `internal/pyjson`,
- `internal/decode` and `internal/collate` stay internal.
+ `internal/decode` and `internal/collate` stay internal. `infer.Batch` is the public name of
+ what `collate` builds.
## Parity
## Parity
diff --git a/laya-go/agent/agent.go b/laya-go/agent/agent.go
index 8a37a16e..48788796 100644
--- a/laya-go/agent/agent.go
+++ b/laya-go/agent/agent.go
@@ -5,19 +5,37 @@ import (
"errors"
"fmt"
"io/fs"
+ "reflect"
"sync"
+ "sync/atomic"
"github.com/NandhaKishorM/laya/laya-go/infer"
"github.com/NandhaKishorM/laya/laya-go/internal/decode"
"github.com/NandhaKishorM/laya/laya-go/schema"
+ "github.com/NandhaKishorM/laya/laya-go/sequence"
"github.com/NandhaKishorM/laya/laya-go/tokenizer"
)
-// ErrClosed is returned by an Agent that has been closed.
-var ErrClosed = errors.New("laya: agent is closed")
+// ErrClosed is returned by an Agent that has been closed. A session closed underneath an open
+// agent yields a wrapped infer.ErrClosed instead.
+var ErrClosed = errors.New("agent: agent is closed")
+
+// StateError identifies the state at fault in a PredictBatch call.
+type StateError struct {
+ // Index is the state's position in the states slice.
+ Index int
+ Err error
+}
+
+// Error names the state and the cause.
+func (e *StateError) Error() string { return fmt.Sprintf("agent: state %d: %v", e.Index, e.Err) }
+
+// Unwrap returns the cause.
+func (e *StateError) Unwrap() error { return e.Err }
// Agent answers typed questions about a state with a laya checkpoint, through an infer.Session.
-// It is the port of laya's ONNXAgent, and it is safe for concurrent use.
+// It is the port of laya's ONNXAgent, and it is safe for concurrent use. Use Open: the zero
+// Agent is closed.
type Agent struct {
tok *tokenizer.Tokenizer
padID int32
@@ -26,56 +44,81 @@ type Agent struct {
sess infer.Session
mu sync.RWMutex // held for reading by every call, for writing by Close
- closed bool
+ closed atomic.Bool // set when Close starts, so new and queued calls return at once
}
+// Option adjusts an Agent at Open. No options exist yet; the parameter is here so adding one is
+// not a breaking change.
+type Option func(*options)
+
+type options struct{}
+
// Open loads rl_agent_config.json and the tokenizer from checkpoint, a directory holding a
// laya checkpoint, and takes ownership of s: on error it closes s, and otherwise Close does.
-// The session must run the graph exported from that same checkpoint; nothing here can tell.
-func Open(checkpoint fs.FS, s infer.Session) (*Agent, error) {
+// The tokenizer must name the pad, CLS, SEP and mask tokens. The session must run the graph
+// exported from that same checkpoint; nothing here can tell.
+func Open(checkpoint fs.FS, s infer.Session, opts ...Option) (*Agent, error) {
if s == nil {
- return nil, errors.New("laya: Open needs a session")
+ return nil, errors.New("agent: Open needs a session")
+ }
+ var o options
+ for _, opt := range opts {
+ if opt != nil {
+ opt(&o)
+ }
}
a, err := open(checkpoint, s)
if err != nil {
- _ = s.Close()
- return nil, err
+ return nil, errors.Join(err, s.Close())
}
return a, nil
}
func open(checkpoint fs.FS, s infer.Session) (*Agent, error) {
+ if checkpoint == nil {
+ return nil, errors.New("agent: Open needs a checkpoint")
+ }
cfg, temps, err := loadConfig(checkpoint)
if err != nil {
return nil, err
}
tok, err := tokenizer.LoadFS(checkpoint, ".")
if err != nil {
- return nil, fmt.Errorf("laya: tokenizer: %w", err)
+ return nil, &ConfigError{Field: tokenizerField, Err: err}
}
- pad, ok := tok.Pad()
- if !ok {
- return nil, errors.New("laya: tokenizer names no pad token")
+ var pad int32
+ for _, t := range []struct {
+ name string
+ id func() (int32, bool)
+ }{{"pad", tok.Pad}, {"CLS", tok.CLS}, {"SEP", tok.SEP}, {"mask", tok.Mask}} {
+ id, ok := t.id()
+ if !ok {
+ return nil, fmt.Errorf("agent: %w: no %s token", sequence.ErrMissingSpecialToken, t.name)
+ }
+ if t.name == "pad" {
+ pad = id
+ }
}
return &Agent{tok: tok, padID: pad, cfg: cfg, temps: temps, sess: s}, nil
}
-// Config is the checkpoint configuration in use, after clamping.
+// Config returns a copy of the checkpoint configuration the agent runs with.
func (a *Agent) Config() Config {
cfg := a.cfg
cfg.Warnings = append([]string(nil), cfg.Warnings...)
return cfg
}
-// Close waits for calls in flight, then closes the session. It is idempotent; a later Predict
-// returns ErrClosed.
+// Close marks the agent closed, so new and queued calls return ErrClosed at once, then waits for
+// calls in flight and closes the session. Close blocks until calls in flight return; cancel
+// their contexts to bound it. It is idempotent.
func (a *Agent) Close() error {
+ first := a.closed.CompareAndSwap(false, true)
a.mu.Lock()
defer a.mu.Unlock()
- if a.closed {
+ if !first || a.sess == nil {
return nil
}
- a.closed = true
return a.sess.Close()
}
@@ -96,15 +139,21 @@ func WithLanguage(tag string) CallOption { return func(c *call) { c.lang = tag }
// holds states x questions rows.
func WithBatchSize(states int) CallOption { return func(c *call) { c.batch = states } }
-// WithSortByLength pads less: states are ordered by length inside a lookahead of eight batches
-// before they are chunked. It needs a batch size above 1 and below the number of states, and
-// results stay in input order. A different batch shape can move a probability by a rounding step.
-func WithSortByLength() CallOption { return func(c *call) { c.sort = true } }
+// WithSortByLength(true) pads less: states are ordered by length inside a lookahead of eight
+// batches before they are chunked. It needs a batch size above 1 and below the number of states,
+// and results stay in input order. A different batch shape can move a probability by a rounding
+// step. WithSortByLength(false) is the default.
+func WithSortByLength(on bool) CallOption { return func(c *call) { c.sort = on } }
-// Predict answers every question about one state. Its errors are those of PredictBatch.
+// Predict answers every question about one state. Its errors are those of PredictBatch, except
+// that a failing state's error is returned without the *StateError around it.
func (a *Agent) Predict(ctx context.Context, state any, qs schema.Questions, opts ...CallOption) (schema.Prediction, error) {
preds, err := a.PredictBatch(ctx, []any{state}, qs, opts...)
if err != nil {
+ var se *StateError
+ if errors.As(err, &se) {
+ err = se.Err
+ }
return schema.Prediction{}, err
}
return preds[0], nil
@@ -112,9 +161,17 @@ func (a *Agent) Predict(ctx context.Context, state any, qs schema.Questions, opt
// PredictBatch answers the questions for each state; results are in input order. A state is a
// string, or any value sequence.SerializeState accepts: a slice is a conversation whose end
-// survives truncation. Errors pass through unchanged: *schema.QuestionError,
-// *sequence.MarkersError (ID set), sequence.ErrUnsupportedState and ctx.Err(). With no questions
-// each state gets an empty answer and zero usage, with no model run.
+// survives truncation. A nil state or nil pointer is refused; a nil slice or map is an empty
+// list or dict. With no questions each state gets an empty answer and zero usage, with no model
+// run.
+//
+// Errors may be wrapped; match them with errors.Is and errors.As:
+// - ErrClosed, once Close has been called;
+// - ctx.Err(), checked on entry, before each state is encoded and between runs;
+// - *StateError, naming the state, around sequence.ErrUnsupportedState (also for a nil
+// state), sequence.ErrMissingSpecialToken or a *sequence.MarkersError with its ID set;
+// - *RunError, around whatever the session returned (so errors.Is finds context.Canceled and
+// infer.ErrClosed through it), a *infer.ShapeError, or a logit that cannot be decoded.
func (a *Agent) PredictBatch(ctx context.Context, states []any, qs schema.Questions, opts ...CallOption) ([]schema.Prediction, error) {
var c call
for _, opt := range opts {
@@ -122,14 +179,20 @@ func (a *Agent) PredictBatch(ctx context.Context, states []any, qs schema.Questi
opt(&c)
}
}
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ if a.sess == nil || a.closed.Load() {
+ return nil, ErrClosed
+ }
a.mu.RLock()
defer a.mu.RUnlock()
- if a.closed {
+ if a.closed.Load() {
return nil, ErrClosed
}
for i, state := range states {
- if state == nil {
- return nil, fmt.Errorf("laya: state %d is nil; pass a string, a map or a slice", i)
+ if isNil(state) {
+ return nil, &StateError{Index: i, Err: fmt.Errorf("is nil; pass a string, a map or a slice: %w", sequence.ErrUnsupportedState)}
}
}
if len(states) == 0 {
@@ -143,5 +206,14 @@ func (a *Agent) PredictBatch(ctx context.Context, states []any, qs schema.Questi
}
return preds, nil
}
- return a.infer(ctx, states, qs, c)
+ return a.predictChunks(ctx, states, qs, c)
+}
+
+// isNil reports an untyped nil or a nil pointer; a nil slice or map is an empty list or dict.
+func isNil(state any) bool {
+ if state == nil {
+ return true
+ }
+ rv := reflect.ValueOf(state)
+ return rv.Kind() == reflect.Pointer && rv.IsNil()
}
diff --git a/laya-go/agent/agent_test.go b/laya-go/agent/agent_test.go
index 4a4ed1f4..c09d6869 100644
--- a/laya-go/agent/agent_test.go
+++ b/laya-go/agent/agent_test.go
@@ -7,6 +7,7 @@ import (
"fmt"
"io/fs"
"reflect"
+ "slices"
"strings"
"sync"
"testing"
@@ -35,18 +36,25 @@ type run struct {
type fakeSession struct {
mu sync.Mutex
runs []run
+ pads [][]int64 // per run, the distinct ids under a zero mask
closes int
// before, when set, runs at the start of each Run with the number of runs so far.
before func(n int) error
// shape, when set, edits the output before it is returned.
shape func(*infer.Output)
+ // onCtx, when set, sees the context each Run was given.
+ onCtx func(context.Context)
}
-func (s *fakeSession) Run(_ context.Context, b infer.Batch) (infer.Output, error) {
+func (s *fakeSession) Run(ctx context.Context, b infer.Batch) (infer.Output, error) {
+ if s.onCtx != nil {
+ s.onCtx(ctx)
+ }
if err := b.Validate(); err != nil {
return infer.Output{}, err
}
r := run{Rows: b.Rows, Length: b.Length, Markers: b.Markers}
+ var pads []int64
out := infer.Output{Logits: make([]float32, b.Rows*b.Markers), ActLogits: make([]float32, b.Rows*2)}
for i := 0; i < b.Rows; i++ {
var sum, tokens, markers int64
@@ -54,6 +62,8 @@ func (s *fakeSession) Run(_ context.Context, b infer.Batch) (infer.Output, error
if b.AttentionMask[i*b.Length+j] == 1 {
sum += b.InputIDs[i*b.Length+j]
tokens++
+ } else if id := b.InputIDs[i*b.Length+j]; !slices.Contains(pads, id) {
+ pads = append(pads, id)
}
}
for j := 0; j < b.Markers; j++ {
@@ -73,6 +83,7 @@ func (s *fakeSession) Run(_ context.Context, b infer.Batch) (infer.Output, error
s.mu.Lock()
n := len(s.runs)
s.runs = append(s.runs, r)
+ s.pads = append(s.pads, pads)
before, shape := s.before, s.shape
s.mu.Unlock()
if before != nil {
@@ -346,7 +357,7 @@ func TestEmptyQuestionsAndStates(t *testing.T) {
func TestStateErrors(t *testing.T) {
a, s := open(t, `{}`)
ctx := context.Background()
- if _, err := a.PredictBatch(ctx, []any{"fine", nil}, questions(t)); err == nil || !strings.Contains(err.Error(), "state 1 is nil") {
+ if _, err := a.PredictBatch(ctx, []any{"fine", nil}, questions(t)); err == nil || !strings.Contains(err.Error(), "state 1: is nil") {
t.Errorf("nil state: %v", err)
}
if _, err := a.Predict(ctx, struct{ X int }{1}, questions(t)); !errors.Is(err, sequence.ErrUnsupportedState) {
@@ -458,7 +469,7 @@ func TestResultsAreTheSameWhateverTheBatching(t *testing.T) {
for _, sorted := range []bool{false, true} {
opts := []agent.CallOption{agent.WithBatchSize(size)}
if sorted {
- opts = append(opts, agent.WithSortByLength())
+ opts = append(opts, agent.WithSortByLength(true))
}
got, err := a.PredictBatch(ctx, states, qs, opts...)
if err != nil || len(got) != len(states) {
@@ -507,13 +518,13 @@ func TestChunkingAndSorting(t *testing.T) {
// Sorting reorders only when 1 < chunk < states: otherwise the runs are the unsorted ones.
for _, size := range []int{1, 11, 40} {
plain := record(agent.WithBatchSize(size))
- sorted := record(agent.WithBatchSize(size), agent.WithSortByLength())
+ sorted := record(agent.WithBatchSize(size), agent.WithSortByLength(true))
if !reflect.DeepEqual(plain, sorted) {
t.Errorf("batch %d: sorting changed the runs", size)
}
}
unsorted := lengths(record(agent.WithBatchSize(3)))
- sorted := lengths(record(agent.WithBatchSize(3), agent.WithSortByLength()))
+ sorted := lengths(record(agent.WithBatchSize(3), agent.WithSortByLength(true)))
for i := 1; i < len(sorted); i++ {
if sorted[i] < sorted[i-1] {
t.Errorf("sorted run lengths %v are not ascending", sorted)
@@ -650,7 +661,7 @@ func TestContextStopsTheBatchBetweenChunks(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
s.before = func(int) error { <-ctx.Done(); return nil }
- if _, err := a.PredictBatch(ctx, states, qs, agent.WithBatchSize(2), agent.WithSortByLength()); !errors.Is(err, context.DeadlineExceeded) {
+ if _, err := a.PredictBatch(ctx, states, qs, agent.WithBatchSize(2), agent.WithSortByLength(true)); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("PredictBatch = %v", err)
}
if n := len(s.seen()); n != 1 {
@@ -687,7 +698,7 @@ func TestConcurrentPredict(t *testing.T) {
}
continue
}
- got, err := a.PredictBatch(context.Background(), states, qs, agent.WithBatchSize(3+g), agent.WithSortByLength())
+ got, err := a.PredictBatch(context.Background(), states, qs, agent.WithBatchSize(3+g), agent.WithSortByLength(true))
if err != nil {
errs <- err
return
diff --git a/laya-go/agent/batch.go b/laya-go/agent/batch.go
index b889955a..ece9514f 100644
--- a/laya-go/agent/batch.go
+++ b/laya-go/agent/batch.go
@@ -12,13 +12,29 @@ import (
"github.com/NandhaKishorM/laya/laya-go/sequence"
)
+// RunError reports a failed model run: the session's error, an output that does not fit the
+// batch, or a logit that cannot be decoded.
+type RunError struct {
+ // Rows, Length and Markers are the dimensions of the batch that was run.
+ Rows, Length, Markers int
+ Err error
+}
+
+// Error names the batch and the cause.
+func (e *RunError) Error() string {
+ return fmt.Sprintf("agent: running %d rows of %d tokens with %d marker slots: %v", e.Rows, e.Length, e.Markers, e.Err)
+}
+
+// Unwrap returns the cause.
+func (e *RunError) Unwrap() error { return e.Err }
+
// lookaheadBatches is how many batches of states are encoded and sorted together.
const lookaheadBatches = 8
-// infer is ONNXAgent._infer_batch: encode, collate, run and decode in chunks of c.batch states.
+// predictChunks is ONNXAgent._infer_batch: encode, collate, run and decode in chunks of c.batch states.
// With c.sort the states are ordered by their longest question row inside a window of eight
// chunks, so a chunk pads to a shorter length; results stay in input order.
-func (a *Agent) infer(ctx context.Context, states []any, qs schema.Questions, c call) ([]schema.Prediction, error) {
+func (a *Agent) predictChunks(ctx context.Context, states []any, qs schema.Questions, c call) ([]schema.Prediction, error) {
chunk := len(states)
if c.batch > 0 {
chunk = min(c.batch, len(states))
@@ -33,9 +49,12 @@ func (a *Agent) infer(ctx context.Context, states []any, qs schema.Questions, c
part := states[start:min(start+window, len(states))]
encoded := make([][]sequence.Sequence, len(part))
for i, state := range part {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
var err error
if encoded[i], err = a.encode(state, qs); err != nil {
- return nil, err
+ return nil, &StateError{Index: start + i, Err: err}
}
}
order := make([]int, len(part))
@@ -55,6 +74,9 @@ func (a *Agent) infer(ctx context.Context, states []any, qs schema.Questions, c
if err := ctx.Err(); err != nil {
return nil, err
}
+ if a.closed.Load() {
+ return nil, ErrClosed
+ }
idx := order[off:min(off+chunk, len(order))]
preds, err := a.run(ctx, qs, encoded, idx, c.lang)
if err != nil {
@@ -100,14 +122,15 @@ func (a *Agent) run(ctx context.Context, qs schema.Questions, encoded [][]sequen
}
b, err := collate.Collate(rows, a.padID)
if err != nil {
- return nil, err
+ return nil, fmt.Errorf("agent: %w", err)
}
+ fail := func(err error) error { return &RunError{Rows: b.Rows, Length: b.Length, Markers: b.Markers, Err: err} }
out, err := a.sess.Run(ctx, b)
if err != nil {
- return nil, fmt.Errorf("laya: running %d rows of %d tokens: %w", b.Rows, b.Length, err)
+ return nil, fail(err)
}
if err := out.Check(b); err != nil {
- return nil, err
+ return nil, fail(err)
}
preds := make([]schema.Prediction, len(idx))
row := 0
@@ -118,13 +141,13 @@ func (a *Agent) run(ctx context.Context, qs schema.Questions, encoded [][]sequen
at, e := row+r, qs.At(r)
act, err := decode.ActionProbabilities(out.ActLogits[at*2 : at*2+2])
if err != nil {
- return nil, err
+ return nil, fail(fmt.Errorf("action logits of question %s: %w", e.ID, err))
}
ans, err := decode.Decode(e.Question, decode.Row{
Logits: out.Logits[at*b.Markers : (at+1)*b.Markers], K: len(s.Markers), Act: act,
}, lang, a.temps)
if err != nil {
- return nil, fmt.Errorf("laya: question %s: %w", e.ID, err)
+ return nil, fail(fmt.Errorf("question %s: %w", e.ID, err))
}
answers[r] = schema.Answered{ID: e.ID, Answer: ans}
}
diff --git a/laya-go/agent/config.go b/laya-go/agent/config.go
index 97858cee..300b6f62 100644
--- a/laya-go/agent/config.go
+++ b/laya-go/agent/config.go
@@ -11,6 +11,7 @@ import (
"github.com/NandhaKishorM/laya/laya-go/internal/decode"
"github.com/NandhaKishorM/laya/laya-go/internal/pyjson"
"github.com/NandhaKishorM/laya/laya-go/internal/pyvalue"
+ "github.com/NandhaKishorM/laya/laya-go/internal/unicodedata"
)
// configFile is the checkpoint file the configuration is read from.
@@ -25,7 +26,7 @@ const (
// ErrUnsupportedLayout is wrapped when a checkpoint's option_layout is not "sequential", the
// only layout the exported graph takes.
-var ErrUnsupportedLayout = errors.New("laya: option_layout is not supported by the ONNX runtime")
+var ErrUnsupportedLayout = errors.New("agent: option_layout is not supported by the ONNX runtime")
// ConfigError reports a missing or malformed rl_agent_config.json.
type ConfigError struct {
@@ -34,18 +35,29 @@ type ConfigError struct {
Err error
}
+// tokenizerField is the ConfigError.Field of a tokenizer that could not be loaded.
+const tokenizerField = "tokenizer"
+
// Error names the file, the field when there is one, and the cause.
func (e *ConfigError) Error() string {
- if e.Field == "" {
- return "laya: " + configFile + ": " + e.Err.Error()
+ cause := "<nil>"
+ if e.Err != nil {
+ cause = e.Err.Error()
+ }
+ switch e.Field {
+ case "":
+ return "agent: " + configFile + ": " + cause
+ case tokenizerField:
+ return "agent: tokenizer: " + cause // not a field of the config file
}
- return "laya: " + configFile + ": " + e.Field + ": " + e.Err.Error()
+ return "agent: " + configFile + ": " + e.Field + ": " + cause
}
// Unwrap returns the cause.
func (e *ConfigError) Unwrap() error { return e.Err }
-// Config is the checkpoint configuration an Agent runs with.
+// Config is the checkpoint configuration an Agent runs with. Temperatures are not in it; the
+// Agent holds them clamped.
type Config struct {
// MaxLen is the whole sequence budget: max_len, default 512.
MaxLen int
@@ -59,7 +71,7 @@ type Config struct {
}
// LoadConfig reads and checks rl_agent_config.json from the root of checkpoint, as ONNXAgent
-// does: max_len, head_max_len and option_layout, with the temperatures clamped.
+// does: max_len, head_max_len and option_layout, and a warning for every temperature clamped.
func LoadConfig(checkpoint fs.FS) (Config, error) {
cfg, _, err := loadConfig(checkpoint)
return cfg, err
@@ -103,6 +115,11 @@ func loadConfig(checkpoint fs.FS) (Config, *decode.Temperatures, error) {
}
n, isNumber := raw.(json.Number)
size, err := strconv.Atoi(string(n))
+ var numErr *strconv.NumError
+ if errors.As(err, &numErr) && numErr.Err == strconv.ErrRange && !strings.HasPrefix(string(n), "-") {
+ return Config{}, nil, &ConfigError{Field: f.key,
+ Err: fmt.Errorf("%s is too large for an int", n)}
+ }
if !isNumber || err != nil || size <= 0 {
return Config{}, nil, &ConfigError{Field: f.key,
Err: fmt.Errorf("want a positive integer, got %s", pyvalue.Repr(raw))}
@@ -160,9 +177,17 @@ func loadConfig(checkpoint fs.FS) (Config, *decode.Temperatures, error) {
return cfg, temps, nil
}
-// sameNumber is float(raw) == applied: false for anything that is not a number.
+// sameNumber is float(raw) == applied: false for anything float() refuses. A string is read as
+// decode.Clamp reads it: stripped, and without the hex and underscore spellings Python rejects.
func sameNumber(raw any, applied float64) bool {
switch x := raw.(type) {
+ case string:
+ s := unicodedata.Strip(x)
+ if strings.ContainsAny(s, "xXpP_") {
+ return false
+ }
+ f, err := strconv.ParseFloat(s, 64)
+ return err == nil && f == applied
case json.Number:
f, err := strconv.ParseFloat(string(x), 64)
return err == nil && f == applied
diff --git a/laya-go/agent/contract_test.go b/laya-go/agent/contract_test.go
new file mode 100644
index 00000000..c5ec0e14
--- /dev/null
+++ b/laya-go/agent/contract_test.go
@@ -0,0 +1,539 @@
+package agent_test
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "math"
+ "os"
+ "path/filepath"
+ "reflect"
+ "slices"
+ "strings"
+ "sync"
+ "testing"
+ "testing/fstest"
+ "time"
+
+ "github.com/NandhaKishorM/laya/laya-go/agent"
+ "github.com/NandhaKishorM/laya/laya-go/infer"
+ "github.com/NandhaKishorM/laya/laya-go/internal/fixtures"
+ "github.com/NandhaKishorM/laya/laya-go/schema"
+ "github.com/NandhaKishorM/laya/laya-go/sequence"
+)
+
+// within fails the test if f does not return in a second, and returns its error.
+func within(t *testing.T, what string, f func() error) error {
+ t.Helper()
+ done := make(chan error, 1)
+ go func() { done <- f() }()
+ select {
+ case err := <-done:
+ return err
+ case <-time.After(time.Second):
+ t.Fatalf("%s did not return within a second", what)
+ return nil
+ }
+}
+
+// blockedClose opens an agent whose first Run blocks until release is closed, starts a Predict
+// into that Run, then starts Close behind it and waits until the agent reports itself closed.
+func blockedClose(t *testing.T, s *fakeSession, states []any, opts ...agent.CallOption) (a *agent.Agent, release chan struct{}, predict chan error, closed chan struct{}) {
+ t.Helper()
+ a, err := agent.Open(toyCheckpoint(`{}`), s)
+ if err != nil {
+ t.Fatal(err)
+ }
+ started := make(chan struct{})
+ release = make(chan struct{})
+ s.before = func(n int) error {
+ if n == 0 {
+ close(started)
+ <-release
+ }
+ return nil
+ }
+ predict = make(chan error, 1)
+ go func() { _, err := a.PredictBatch(context.Background(), states, questions(t), opts...); predict <- err }()
+ <-started
+ closed = make(chan struct{})
+ go func() { a.Close(); close(closed) }()
+ for deadline := time.Now().Add(time.Second); ; time.Sleep(time.Millisecond) {
+ err := within(t, "a probe Predict", func() error {
+ _, err := a.Predict(context.Background(), "probe", questions(t))
+ return err
+ })
+ if errors.Is(err, agent.ErrClosed) {
+ return a, release, predict, closed
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("Close never marked the agent closed")
+ }
+ }
+}
+
+// TestCloseBehindABlockedRunDoesNotStallNewCalls is the reviewer's scenario: Close waits on a
+// call stuck in the session, and a new call, even one with a dead context, must not wait too.
+func TestCloseBehindABlockedRunDoesNotStallNewCalls(t *testing.T) {
+ s := &fakeSession{}
+ a, release, predict, closed := blockedClose(t, s, []any{"state"})
+ err := within(t, "Predict with a live context", func() error {
+ _, err := a.Predict(context.Background(), "x", questions(t))
+ return err
+ })
+ if !errors.Is(err, agent.ErrClosed) {
+ t.Errorf("new Predict = %v, want ErrClosed", err)
+ }
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ err = within(t, "Predict with a cancelled context", func() error {
+ _, err := a.Predict(ctx, "x", questions(t))
+ return err
+ })
+ if err != ctx.Err() {
+ t.Errorf("Predict with a cancelled context = %v, want ctx.Err()", err)
+ }
+ select {
+ case <-closed:
+ t.Fatal("Close returned while a call was in the session")
+ default:
+ }
+ close(release)
+ if err := <-predict; err != nil {
+ t.Errorf("the call in flight = %v", err)
+ }
+ <-closed
+}
+
+func TestCloseStopsAMultiChunkCallBetweenRuns(t *testing.T) {
+ s := &fakeSession{}
+ _, release, predict, closed := blockedClose(t, s, []any{"a", "b", "c"}, agent.WithBatchSize(1))
+ close(release)
+ if err := <-predict; !errors.Is(err, agent.ErrClosed) {
+ t.Errorf("the call in flight = %v, want ErrClosed after its first run", err)
+ }
+ <-closed
+ // The call's first run, and at most the probes that ran before Close began.
+ for i, r := range s.seen() {
+ if r.Rows == 9 {
+ t.Errorf("run %d held 3 states: the batch size was lost", i)
+ }
+ }
+ if got := len(s.seen()); got > 4 {
+ t.Errorf("%d runs: the call went on after Close", got)
+ }
+}
+
+func TestZeroAgentIsClosed(t *testing.T) {
+ var a agent.Agent
+ if err := a.Close(); err != nil {
+ t.Errorf("Close = %v", err)
+ }
+ if err := a.Close(); err != nil {
+ t.Errorf("second Close = %v", err)
+ }
+ if _, err := a.Predict(context.Background(), "x", questions(t)); !errors.Is(err, agent.ErrClosed) {
+ t.Errorf("Predict = %v", err)
+ }
+ if cfg := a.Config(); cfg.MaxLen != 0 {
+ t.Errorf("Config = %+v", cfg)
+ }
+}
+
+func TestConfigErrorWithoutACause(t *testing.T) {
+ for _, e := range []*agent.ConfigError{{}, {Field: "max_len"}} {
+ if msg := e.Error(); !strings.HasPrefix(msg, "agent: rl_agent_config.json") {
+ t.Errorf("Error = %q", msg)
+ }
+ }
+}
+
+type failingClose struct {
+ fakeSession
+ err error
+}
+
+func (f *failingClose) Close() error { f.fakeSession.Close(); return f.err }
+
+func TestOpenReportsTheSessionsCloseError(t *testing.T) {
+ boom := errors.New("close failed")
+ s := &failingClose{err: boom}
+ _, err := agent.Open(toyCheckpoint(`{"option_layout": "dual"}`), s)
+ if !errors.Is(err, agent.ErrUnsupportedLayout) || !errors.Is(err, boom) {
+ t.Fatalf("Open = %v, want both the load error and the close error", err)
+ }
+ if s.closes != 1 {
+ t.Errorf("closed %d times", s.closes)
+ }
+}
+
+func TestOpenRefusesWhatCannotRun(t *testing.T) {
+ t.Run("nil checkpoint", func(t *testing.T) {
+ s := &fakeSession{}
+ if a, err := agent.Open(nil, s); err == nil || a != nil || s.closes != 1 {
+ t.Fatalf("Open = %v, %v with %d closes", a, err, s.closes)
+ }
+ })
+ roles := map[string]string{"pad_token": "[PAD]", "cls_token": "[CLS]", "sep_token": "[SEP]", "mask_token": "[MASK]"}
+ for missing := range roles {
+ t.Run("no "+missing, func(t *testing.T) {
+ var parts []string
+ for role, content := range roles {
+ if role != missing {
+ parts = append(parts, fmt.Sprintf("%q:%q", role, content))
+ }
+ }
+ slices.Sort(parts)
+ fsys := toyCheckpoint(`{}`)
+ fsys["tokenizer/tokenizer_config.json"].Data = []byte("{" + strings.Join(parts, ",") + "}")
+ s := &fakeSession{}
+ _, err := agent.Open(fsys, s)
+ if !errors.Is(err, sequence.ErrMissingSpecialToken) || s.closes != 1 {
+ t.Fatalf("Open = %v with %d closes", err, s.closes)
+ }
+ })
+ }
+ t.Run("tokenizer load error", func(t *testing.T) {
+ fsys := toyCheckpoint(`{}`)
+ fsys["tokenizer/tokenizer.json"].Data = []byte(`{`)
+ var ce *agent.ConfigError
+ if _, err := agent.Open(fsys, &fakeSession{}); !errors.As(err, &ce) || ce.Field != "tokenizer" {
+ t.Fatalf("Open = %v, want a ConfigError for the tokenizer", err)
+ }
+ })
+}
+
+func TestNilOptionsAreSkipped(t *testing.T) {
+ s := &fakeSession{}
+ a, err := agent.Open(toyCheckpoint(`{}`), s, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer a.Close()
+ if _, err := a.PredictBatch(context.Background(), []any{"x", "y"}, questions(t), nil, agent.WithBatchSize(1), nil); err != nil {
+ t.Fatal(err)
+ }
+ if n := len(s.seen()); n != 2 {
+ t.Errorf("%d runs, want 2: the option after the nil one was lost", n)
+ }
+}
+
+func TestSortByLengthCanBeTurnedOff(t *testing.T) {
+ states := varied()[:11]
+ runs := func(opts ...agent.CallOption) []int {
+ a, s := open(t, `{}`)
+ if _, err := a.PredictBatch(context.Background(), states, questions(t), opts...); err != nil {
+ t.Fatal(err)
+ }
+ return lengths(s.seen())
+ }
+ plain := runs(agent.WithBatchSize(3))
+ if got := runs(agent.WithBatchSize(3), agent.WithSortByLength(true)); reflect.DeepEqual(got, plain) {
+ t.Fatal("sorting changed nothing; the test proves nothing")
+ }
+ if got := runs(agent.WithBatchSize(3), agent.WithSortByLength(true), agent.WithSortByLength(false)); !reflect.DeepEqual(got, plain) {
+ t.Errorf("WithSortByLength(false) left runs %v, want %v", got, plain)
+ }
+}
+
+func TestRunErrors(t *testing.T) {
+ boom := errors.New("boom")
+ cases := []struct {
+ name string
+ setup func(*fakeSession)
+ is error
+ as any // a pointer to the typed error that must also be found
+ }{
+ {"the session fails", func(s *fakeSession) { s.before = func(int) error { return boom } }, boom, nil},
+ {"the session reports a cancel", func(s *fakeSession) { s.before = func(int) error { return context.Canceled } }, context.Canceled, nil},
+ {"the session is closed under the agent", func(s *fakeSession) { s.before = func(int) error { return infer.ErrClosed } }, infer.ErrClosed, nil},
+ {"a wrong-shaped output", func(s *fakeSession) { s.shape = func(o *infer.Output) { o.Logits = o.Logits[1:] } }, nil, new(*infer.ShapeError)},
+ {"a NaN logit", func(s *fakeSession) { s.shape = func(o *infer.Output) { o.Logits[0] = float32(nan()) } }, nil, nil},
+ {"an infinite action logit", func(s *fakeSession) {
+ s.shape = func(o *infer.Output) { o.ActLogits[1] = float32(math.Inf(1)) }
+ }, nil, nil},
+ }
+ for _, c := range cases {
+ t.Run(c.name, func(t *testing.T) {
+ a, s := open(t, `{}`)
+ c.setup(s)
+ _, err := a.Predict(context.Background(), "state", questions(t))
+ var re *agent.RunError
+ if !errors.As(err, &re) {
+ t.Fatalf("Predict = %v, want a *RunError", err)
+ }
+ if r := s.seen()[0]; re.Rows != r.Rows || re.Length != r.Length || re.Markers != r.Markers || re.Rows != 3 {
+ t.Errorf("RunError = %+v, run was %+v", re, r)
+ }
+ if c.is != nil && !errors.Is(err, c.is) {
+ t.Errorf("Predict = %v, want it to wrap %v", err, c.is)
+ }
+ if c.as != nil && !errors.As(err, c.as) {
+ t.Errorf("Predict = %v, want %T", err, c.as)
+ }
+ if !strings.HasPrefix(err.Error(), "agent: ") {
+ t.Errorf("message %q lacks the package prefix", err)
+ }
+ })
+ }
+}
+
+// TestActionLogitFailureNamesTheQuestion pins the message of C8: the question id and the action
+// logits, not "logits" alone.
+func TestActionLogitFailureNamesTheQuestion(t *testing.T) {
+ a, s := open(t, `{}`)
+ s.shape = func(o *infer.Output) { o.ActLogits[2] = float32(nan()) } // row 1: urgency
+ _, err := a.Predict(context.Background(), "state", questions(t))
+ var re *agent.RunError
+ if !errors.As(err, &re) || !strings.Contains(err.Error(), "action logits of question urgency") {
+ t.Fatalf("Predict = %v", err)
+ }
+}
+
+func TestStateErrors_Batch(t *testing.T) {
+ a, _ := open(t, `{}`)
+ ctx := context.Background()
+ type S struct{ X int }
+ var nilPtr *string
+ cases := []struct {
+ name string
+ states []any
+ index int
+ opts []agent.CallOption
+ }{
+ {"unsupported", []any{"a", "b", S{1}, "d"}, 2, nil},
+ {"unsupported in a later window", []any{"a", "b", "c", S{1}, "e"}, 3, []agent.CallOption{agent.WithBatchSize(1)}},
+ {"untyped nil", []any{"a", nil}, 1, nil},
+ {"nil pointer", []any{"a", "b", nilPtr}, 2, nil},
+ }
+ for _, c := range cases {
+ t.Run(c.name, func(t *testing.T) {
+ _, err := a.PredictBatch(ctx, c.states, questions(t), c.opts...)
+ var se *agent.StateError
+ if !errors.As(err, &se) || se.Index != c.index || !errors.Is(err, sequence.ErrUnsupportedState) {
+ t.Fatalf("PredictBatch = %v, want a StateError at %d wrapping ErrUnsupportedState", err, c.index)
+ }
+ if !strings.Contains(err.Error(), fmt.Sprintf("state %d", c.index)) {
+ t.Errorf("message %q does not name the state", err)
+ }
+ })
+ }
+ // A nil slice and a nil map are an empty list and dict, as in the SDK's other packages.
+ if _, err := a.PredictBatch(ctx, []any{[]any(nil), map[string]any(nil)}, questions(t)); err != nil {
+ t.Errorf("nil slice and map: %v", err)
+ }
+ // A failing question names its state too; Predict returns the inner error alone.
+ wide := make([]string, 12)
+ for i := range wide {
+ wide[i] = fmt.Sprintf("option number %d", i)
+ }
+ qs, err := schema.NewQuestions(schema.Ask("wide", schema.MustQuestion(schema.ChoiceOf("pick", wide...))))
+ if err != nil {
+ t.Fatal(err)
+ }
+ narrow, _ := open(t, `{"max_len": 40, "head_max_len": 30}`)
+ var se *agent.StateError
+ var me *sequence.MarkersError
+ if _, err := narrow.PredictBatch(ctx, []any{"x", "y"}, qs); !errors.As(err, &se) || se.Index != 0 || !errors.As(err, &me) || me.ID != "wide" {
+ t.Errorf("PredictBatch = %v", err)
+ }
+ _, err = narrow.Predict(ctx, "x", qs)
+ if errors.As(err, &se) || !errors.As(err, &me) {
+ t.Errorf("Predict = %v, want the MarkersError without a StateError", err)
+ }
+ _, err = a.Predict(ctx, S{1}, questions(t))
+ if errors.As(err, &se) || !errors.Is(err, sequence.ErrUnsupportedState) {
+ t.Errorf("Predict(struct) = %v, want the bare error", err)
+ }
+}
+
+// TestNilStateIsRefusedWithoutQuestions pins that Go refuses the nil state Python raises a
+// TypeError for, even when there is nothing to ask and nothing would run.
+func TestNilStateIsRefusedWithoutQuestions(t *testing.T) {
+ a, _ := open(t, `{}`)
+ if _, err := a.PredictBatch(context.Background(), []any{"fine", nil}, schema.Questions{}); !errors.Is(err, sequence.ErrUnsupportedState) {
+ t.Fatalf("PredictBatch = %v", err)
+ }
+}
+
+func TestContextIsCheckedOnEntry(t *testing.T) {
+ a, s := open(t, `{}`)
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ if _, err := a.PredictBatch(ctx, nil, questions(t)); err != ctx.Err() {
+ t.Errorf("no states: %v, want ctx.Err()", err)
+ }
+ if _, err := a.PredictBatch(ctx, []any{"x"}, schema.Questions{}); err != ctx.Err() {
+ t.Errorf("no questions: %v, want ctx.Err()", err)
+ }
+ if len(s.seen()) != 0 {
+ t.Error("ran with a cancelled context")
+ }
+}
+
+type ctxKey struct{}
+
+// TestSessionSeesTheCallersContext pins that the context is forwarded to Run: a value on it,
+// and its cancellation, both reach the session.
+func TestSessionSeesTheCallersContext(t *testing.T) {
+ a, s := open(t, `{}`)
+ var mu sync.Mutex
+ var got []any
+ s.onCtx = func(ctx context.Context) {
+ mu.Lock()
+ got = append(got, ctx.Value(ctxKey{}))
+ mu.Unlock()
+ }
+ ctx := context.WithValue(context.Background(), ctxKey{}, "mine")
+ if _, err := a.Predict(ctx, "state", questions(t)); err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(got, []any{"mine"}) {
+ t.Fatalf("the session saw context values %v", got)
+ }
+ cctx, cancel := context.WithCancel(context.Background())
+ s.onCtx = func(c context.Context) { cancel(); <-c.Done() }
+ s.before = func(int) error { return cctx.Err() }
+ if _, err := within2(t, a, cctx); !errors.Is(err, context.Canceled) {
+ t.Errorf("Predict = %v, want the cancel to have reached the session", err)
+ }
+}
+
+// within2 runs one Predict that would hang forever if the session's context were not cancelled.
+func within2(t *testing.T, a *agent.Agent, ctx context.Context) (schema.Prediction, error) {
+ t.Helper()
+ var p schema.Prediction
+ err := within(t, "Predict", func() (err error) { p, err = a.Predict(ctx, "state", questions(t)); return })
+ return p, err
+}
+
+// TestMaskedPositionsHoldThePadID pins the pad id the collator is given, on the toy tokenizer
+// and on the shipped English one (pad id 50283), neither of which is 0.
+func TestMaskedPositionsHoldThePadID(t *testing.T) {
+ check := func(t *testing.T, a *agent.Agent, s *fakeSession, pad int64) {
+ t.Helper()
+ if _, err := a.PredictBatch(context.Background(), varied(), questions(t)); err != nil {
+ t.Fatal(err)
+ }
+ var padded bool
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ for _, pads := range s.pads {
+ if len(pads) > 0 {
+ padded = true
+ if !reflect.DeepEqual(pads, []int64{pad}) {
+ t.Errorf("masked positions hold %v, want only %d", pads, pad)
+ }
+ }
+ }
+ if !padded {
+ t.Fatal("no row was padded; the test proves nothing")
+ }
+ }
+ t.Run("toy", func(t *testing.T) {
+ a, s := open(t, `{}`)
+ check(t, a, s, fixtures.ToyPad)
+ })
+ t.Run("english", func(t *testing.T) {
+ root := fixtures.Checkpoints()
+ if root == "" {
+ t.Skip("set LAYA_CHECKPOINTS to a directory holding the english and multilingual checkpoints")
+ }
+ s := &fakeSession{}
+ a, err := agent.Open(os.DirFS(filepath.Join(root, "english")), s)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer a.Close()
+ check(t, a, s, 50283)
+ })
+}
+
+func TestEmptyQuestionsUsageIsAnEmptyList(t *testing.T) {
+ a, _ := open(t, `{}`)
+ preds, err := a.PredictBatch(context.Background(), []any{"x"}, schema.Questions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if tq := preds[0].Usage.TruncatedQuestions; tq == nil || len(tq) != 0 {
+ t.Fatalf("TruncatedQuestions = %#v, want an empty non-nil slice", tq)
+ }
+}
+
+// pyWarning is the text ONNXAgent.__init__ warns with, less its "laya ONNX: " prefix. The
+// expected strings below were produced by running laya.onnx_agent.ONNXAgent on each config with
+// onnxruntime, transformers and the tokenizer fix patched out, and reading the RuntimeWarning.
+const pyWarning = "this checkpoint ships temperatures outside [0.5, 5] which would distort confidence; clamping %s. " +
+ "Treat confidence from the affected buckets as uncalibrated."
+
+func TestClampWarningsMatchPython(t *testing.T) {
+ cases := []struct {
+ name, config, clamped string // clamped "" means no warning
+ }{
+ // A bucket and a type entry both clamped: Python lists buckets first.
+ {"buckets before types", `{"temperature": [1.0, 9, 1.0], "temperature_by_options": {"choice:11+": 0.1006}}`,
+ "choice:11+=0.1006 -> 0.5, temperature[1]=9 -> 5"},
+ // float(False) is 0.0, not the 1.0 applied; float(True) is 1.0 and is quiet.
+ {"a bool temperature", `{"temperature": [false, true, 1.0]}`, "temperature[0]=False -> 1"},
+ // float("2.0") is 2.0, applied unchanged; float("9") is 9.0, clamped to 5.
+ {"numeric strings", `{"temperature": ["2.0", "9", 1.0]}`, "temperature[1]='9' -> 5"},
+ }
+ for _, c := range cases {
+ t.Run(c.name, func(t *testing.T) {
+ cfg, err := agent.LoadConfig(toyCheckpoint(c.config))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if want := []string{fmt.Sprintf(pyWarning, c.clamped)}; !reflect.DeepEqual(cfg.Warnings, want) {
+ t.Fatalf("warnings = %q\nwant %q", cfg.Warnings, want)
+ }
+ })
+ }
+ for _, quiet := range []string{`{"temperature": ["2.0", " 1.5 ", 1]}`, `{"temperature": [true, 1, 1]}`} {
+ if cfg, err := agent.LoadConfig(toyCheckpoint(quiet)); err != nil || len(cfg.Warnings) != 0 {
+ t.Errorf("%s: %v, %q, want no warning", quiet, err, cfg.Warnings)
+ }
+ }
+}
+
+func TestMaxLenTooLarge(t *testing.T) {
+ _, err := agent.LoadConfig(toyCheckpoint(`{"max_len": 99999999999999999999}`))
+ var ce *agent.ConfigError
+ if !errors.As(err, &ce) || ce.Field != "max_len" || !strings.Contains(err.Error(), "99999999999999999999 is too large") {
+ t.Fatalf("LoadConfig = %v", err)
+ }
+ _, err = agent.LoadConfig(toyCheckpoint(`{"head_max_len": -99999999999999999999}`))
+ if !errors.As(err, &ce) || strings.Contains(err.Error(), "too large") || !strings.Contains(err.Error(), "positive integer") {
+ t.Fatalf("LoadConfig = %v, want the positive-integer message", err)
+ }
+}
+
+func TestMessagesCarryThePackageName(t *testing.T) {
+ for want, err := range map[string]error{
+ "agent: ": agent.ErrClosed, "infer: ": infer.ErrClosed,
+ } {
+ if !strings.HasPrefix(err.Error(), want) {
+ t.Errorf("%q lacks the prefix %q", err, want)
+ }
+ }
+ if !strings.HasPrefix(agent.ErrUnsupportedLayout.Error(), "agent: ") {
+ t.Errorf("%q", agent.ErrUnsupportedLayout)
+ }
+ if _, err := agent.LoadConfig(fstestEmpty()); !strings.HasPrefix(err.Error(), "agent: ") {
+ t.Errorf("%q", err)
+ }
+ if got := (&agent.StateError{Index: 3, Err: errors.New("x")}).Error(); got != "agent: state 3: x" {
+ t.Errorf("StateError = %q", got)
+ }
+}
+
+func fstestEmpty() fstest.MapFS { return fstest.MapFS{} }
+
+// TestTokenizerErrorDoesNotBlameTheConfigFile: a tokenizer that cannot be loaded is not a field of
+// rl_agent_config.json, so its message must not point there.
+func TestTokenizerErrorDoesNotBlameTheConfigFile(t *testing.T) {
+ err := &agent.ConfigError{Field: "tokenizer", Err: errors.New("no tokenizer.json")}
+ if got, want := err.Error(), "agent: tokenizer: no tokenizer.json"; got != want {
+ t.Errorf("got %q, want %q", got, want)
+ }
+}
diff --git a/laya-go/agent/doc.go b/laya-go/agent/doc.go
index 90a571d1..00b1b714 100644
--- a/laya-go/agent/doc.go
+++ b/laya-go/agent/doc.go
@@ -15,6 +15,23 @@
// or sorting. Batch shape does change padding, and with it a probability's last digits, so
// results are compared with a tolerance across shapes, not byte for byte.
//
+// # Differences from Python
+//
+// With no questions, Python writes a usage of two keys (input_tokens, output_tokens); schema.Usage
+// writes all seven, and so does this package.
+//
+// A nil state is refused, with or without questions, where Python raises a TypeError only when
+// there are questions. A nil slice or map is an empty list or dict, as in Python.
+//
+// max_len and head_max_len are checked at load and must be positive integers; Python reads them
+// with cfg.get and checks nothing until a call uses them.
+//
+// A temperature that is not a list of exactly 3 numbers is refused at load, as Python's torch
+// Agent refuses it; ONNXAgent loads it.
+//
+// A NaN or infinite logit is refused, including a -Inf for a real option, where Python would
+// answer probability 0 for that option. A model that emits one is broken.
+//
// # Scope
//
// Not here yet: per-language temperature overrides, the calibration file, min_confidence,
diff --git a/laya-go/agent/example_test.go b/laya-go/agent/example_test.go
index 74b79aff..ee288cbc 100644
--- a/laya-go/agent/example_test.go
+++ b/laya-go/agent/example_test.go
@@ -5,36 +5,16 @@ import (
"fmt"
"log"
"os"
- "testing/fstest"
"github.com/NandhaKishorM/laya/laya-go/agent"
"github.com/NandhaKishorM/laya/laya-go/infer"
- "github.com/NandhaKishorM/laya/laya-go/internal/fixtures"
"github.com/NandhaKishorM/laya/laya-go/schema"
)
-// scorer is a stand-in for a native session: it prefers the second option of every question.
-type scorer struct{}
-
-func (scorer) Run(_ context.Context, b infer.Batch) (infer.Output, error) {
- out := infer.Output{Logits: make([]float32, b.Rows*b.Markers), ActLogits: make([]float32, b.Rows*2)}
- for i := 0; i < b.Rows; i++ {
- out.Logits[i*b.Markers+1] = 2
- }
- return out, nil
-}
-
-func (scorer) Close() error { return nil }
-
+// Example is compiled and not run: it needs a checkpoint and a native session.
func Example() {
- // A checkpoint is a directory with rl_agent_config.json and tokenizer/. This one is a toy.
- spec, cfg := fixtures.ToyTokenizer()
- checkpoint := fstest.MapFS{
- "rl_agent_config.json": {Data: []byte(`{"max_len": 128, "head_max_len": 64}`)},
- "tokenizer/tokenizer.json": {Data: spec},
- "tokenizer/tokenizer_config.json": {Data: cfg},
- }
- a, err := agent.Open(checkpoint, scorer{}) // a real session comes from the onnx package
+ var session infer.Session // from the onnx package
+ a, err := agent.Open(os.DirFS("checkpoints/english"), session)
if err != nil {
log.Fatal(err)
}
@@ -48,7 +28,7 @@ func Example() {
log.Fatal(err)
}
preds, err := a.PredictBatch(context.Background(), []any{"I was charged twice", "Where is my order?"}, qs,
- agent.WithBatchSize(2), agent.WithSortByLength())
+ agent.WithBatchSize(2), agent.WithSortByLength(true), agent.WithLanguage("en"))
if err != nil {
log.Fatal(err)
}
@@ -56,23 +36,4 @@ func Example() {
intent, _ := p.Answers.Get("intent")
fmt.Println(intent.(*schema.ChoiceAnswer).Choice, p.Usage.InputTokens, p.Usage.Truncated)
}
- fmt.Println(a.Config().MaxLen)
- // Output:
- // refund 170 false
- // refund 168 false
- // 128
-}
-
-// ExampleOpen_checkpoint is compiled and not run: it needs a checkpoint and a native session.
-func ExampleOpen_checkpoint() {
- var session infer.Session // from the onnx package
- a, err := agent.Open(os.DirFS("checkpoints/english"), session)
- if err != nil {
- log.Fatal(err)
- }
- defer a.Close()
- qs := schema.Questions{}
- if _, err := a.Predict(context.Background(), "I was charged twice", qs, agent.WithLanguage("en")); err != nil {
- log.Fatal(err)
- }
}
diff --git a/laya-go/agent/plan_golden_test.go b/laya-go/agent/plan_golden_test.go
index 4fc18b87..b847560b 100644
--- a/laya-go/agent/plan_golden_test.go
+++ b/laya-go/agent/plan_golden_test.go
@@ -208,7 +208,7 @@ func TestPredictPlanGolden(t *testing.T) {
opts = append(opts, agent.WithBatchSize(fixtures.Int(t, n)))
}
if fixtures.Bool(t, rec.Must("sort")) {
- opts = append(opts, agent.WithSortByLength())
+ opts = append(opts, agent.WithSortByLength(true))
}
if tag := rec.Must("lang"); tag != nil {
opts = append(opts, agent.WithLanguage(fixtures.Str(t, tag)))
diff --git a/laya-go/agent/toy_end_to_end_test.go b/laya-go/agent/toy_end_to_end_test.go
new file mode 100644
index 00000000..da76b25f
--- /dev/null
+++ b/laya-go/agent/toy_end_to_end_test.go
@@ -0,0 +1,55 @@
+package agent_test
+
+import (
+ "context"
+ "fmt"
+ "testing"
+
+ "github.com/NandhaKishorM/laya/laya-go/agent"
+ "github.com/NandhaKishorM/laya/laya-go/infer"
+ "github.com/NandhaKishorM/laya/laya-go/schema"
+)
+
+// scorer is a stand-in for a native session: it prefers the second option of every question.
+type scorer struct{}
+
+func (scorer) Run(_ context.Context, b infer.Batch) (infer.Output, error) {
+ out := infer.Output{Logits: make([]float32, b.Rows*b.Markers), ActLogits: make([]float32, b.Rows*2)}
+ for i := 0; i < b.Rows; i++ {
+ out.Logits[i*b.Markers+1] = 2
+ }
+ return out, nil
+}
+
+func (scorer) Close() error { return nil }
+
+// TestToyCheckpointEndToEnd runs the documented example on the toy tokenizer.
+func TestToyCheckpointEndToEnd(t *testing.T) {
+ a, err := agent.Open(toyCheckpoint(`{"max_len": 128, "head_max_len": 64}`), scorer{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer a.Close()
+ qs, err := schema.NewQuestions(
+ schema.Ask("intent", schema.MustQuestion(schema.ChoiceOf("What does the customer want?", "status", "refund"))),
+ schema.Ask("angry", schema.MustQuestion(schema.Noul("Is the customer angry?"))),
+ )
+ if err != nil {
+ t.Fatal(err)
+ }
+ preds, err := a.PredictBatch(context.Background(), []any{"I was charged twice", "Where is my order?"}, qs,
+ agent.WithBatchSize(2), agent.WithSortByLength(true))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var got []string
+ for _, p := range preds {
+ intent, _ := p.Answers.Get("intent")
+ got = append(got, fmt.Sprint(intent.(*schema.ChoiceAnswer).Choice, " ", p.Usage.InputTokens, " ", p.Usage.Truncated))
+ }
+ got = append(got, fmt.Sprint(a.Config().MaxLen))
+ want := []string{"refund 170 false", "refund 168 false", "128"}
+ if fmt.Sprint(got) != fmt.Sprint(want) {
+ t.Fatalf("got %q, want %q", got, want)
+ }
+}
diff --git a/laya-go/infer/infer.go b/laya-go/infer/infer.go
index 88c5c38b..4d4e0b51 100644
--- a/laya-go/infer/infer.go
+++ b/laya-go/infer/infer.go
@@ -4,10 +4,22 @@ import (
"context"
"errors"
"fmt"
+ "math"
)
// ErrClosed is returned by a Session that has been closed.
-var ErrClosed = errors.New("laya: session is closed")
+var ErrClosed = errors.New("infer: session is closed")
+
+// The graph's tensor names, as ShapeError.Name carries them and as a native session binds them.
+const (
+ NameInputIDs = "input_ids"
+ NameAttentionMask = "attention_mask"
+ NameMarkerPos = "marker_pos"
+ NameMarkerMask = "marker_mask"
+ NameQType = "qtype"
+ NameLogits = "logits"
+ NameActLogits = "act_logits"
+)
// Batch is the graph's five inputs, each flattened row-major, as onnxruntime tensors take them.
type Batch struct {
@@ -24,9 +36,12 @@ type Output struct {
Logits, ActLogits []float32
}
-// Session runs one collated batch. Implementations must be safe for concurrent Run.
+// Session runs one collated batch. Implementations must be safe for concurrent Run. The agent
+// never calls Close concurrently with Run.
type Session interface {
// Run returns the logits for b. It should return ctx.Err() if ctx ends before the run does.
+ // Run must not modify b's slices or retain them after it returns. The returned Output's
+ // slices belong to the caller, and the session must not reuse them.
Run(ctx context.Context, b Batch) (Output, error)
// Close releases the runtime. It is idempotent; Run after Close returns ErrClosed.
Close() error
@@ -34,7 +49,7 @@ type Session interface {
// ShapeError is returned when a slice's length does not match the batch it belongs to.
type ShapeError struct {
- // Name is the tensor: "input_ids", "logits" and so on.
+ // Name is the tensor: NameInputIDs, NameLogits and so on.
Name string
// Got is the slice's length, Want the length the batch's dimensions call for.
Got, Want int
@@ -42,32 +57,52 @@ type ShapeError struct {
// Error names the tensor and both lengths.
func (e *ShapeError) Error() string {
- return fmt.Sprintf("laya: %s has %d values, want %d", e.Name, e.Got, e.Want)
+ return fmt.Sprintf("infer: %s has %d values, want %d", e.Name, e.Got, e.Want)
}
-// Validate reports a *ShapeError when a slice's length disagrees with Rows, Length and Markers,
-// and a plain error when a dimension is negative or Markers is below 2.
+// Validate checks, in order, that:
+// - no dimension is negative;
+// - a batch with rows has Length of at least 1 and Markers of at least 2, since the exported
+// head takes topk(2);
+// - Rows*Length and Rows*Markers do not overflow an int (Markers of at least 2 keeps Rows*2 safe too);
+// - every slice's length matches Rows, Length and Markers, else a *ShapeError;
+// - every MarkerPos lies in [0, Length).
+//
+// All but the ShapeError are plain errors.
func (b Batch) Validate() error {
if b.Rows < 0 || b.Length < 0 || b.Markers < 0 {
- return fmt.Errorf("laya: batch has negative dimensions %d x %d x %d", b.Rows, b.Length, b.Markers)
+ return fmt.Errorf("infer: batch has negative dimensions %d x %d x %d", b.Rows, b.Length, b.Markers)
}
- if b.Rows > 0 && b.Markers < 2 {
- return fmt.Errorf("laya: batch has %d marker slots, the graph needs at least 2", b.Markers)
+ if b.Rows > 0 {
+ if b.Length == 0 {
+ return fmt.Errorf("infer: batch has %d rows of 0 tokens", b.Rows)
+ }
+ if b.Markers < 2 {
+ return fmt.Errorf("infer: batch has %d marker slots, the graph needs at least 2", b.Markers)
+ }
+ if b.Length > math.MaxInt/b.Rows || b.Markers > math.MaxInt/b.Rows {
+ return fmt.Errorf("infer: batch dimensions %d x %d x %d overflow", b.Rows, b.Length, b.Markers)
+ }
}
for _, c := range []struct {
name string
got, want int
}{
- {"input_ids", len(b.InputIDs), b.Rows * b.Length},
- {"attention_mask", len(b.AttentionMask), b.Rows * b.Length},
- {"marker_pos", len(b.MarkerPos), b.Rows * b.Markers},
- {"marker_mask", len(b.MarkerMask), b.Rows * b.Markers},
- {"qtype", len(b.QType), b.Rows},
+ {NameInputIDs, len(b.InputIDs), b.Rows * b.Length},
+ {NameAttentionMask, len(b.AttentionMask), b.Rows * b.Length},
+ {NameMarkerPos, len(b.MarkerPos), b.Rows * b.Markers},
+ {NameMarkerMask, len(b.MarkerMask), b.Rows * b.Markers},
+ {NameQType, len(b.QType), b.Rows},
} {
if c.got != c.want {
return &ShapeError{Name: c.name, Got: c.got, Want: c.want}
}
}
+ for i, p := range b.MarkerPos {
+ if p < 0 || p >= int64(b.Length) {
+ return fmt.Errorf("infer: marker_pos[%d] is %d, outside [0, %d)", i, p, b.Length)
+ }
+ }
return nil
}
@@ -75,10 +110,10 @@ func (b Batch) Validate() error {
// for b.
func (o Output) Check(b Batch) error {
if len(o.Logits) != b.Rows*b.Markers {
- return &ShapeError{Name: "logits", Got: len(o.Logits), Want: b.Rows * b.Markers}
+ return &ShapeError{Name: NameLogits, Got: len(o.Logits), Want: b.Rows * b.Markers}
}
if len(o.ActLogits) != b.Rows*2 {
- return &ShapeError{Name: "act_logits", Got: len(o.ActLogits), Want: b.Rows * 2}
+ return &ShapeError{Name: NameActLogits, Got: len(o.ActLogits), Want: b.Rows * 2}
}
return nil
}
diff --git a/laya-go/infer/infer_test.go b/laya-go/infer/infer_test.go
index 34f54cd3..fd7d8c19 100644
--- a/laya-go/infer/infer_test.go
+++ b/laya-go/infer/infer_test.go
@@ -2,6 +2,8 @@ package infer_test
import (
"errors"
+ "math"
+ "strings"
"testing"
"github.com/NandhaKishorM/laya/laya-go/infer"
@@ -31,7 +33,18 @@ func TestBatchValidate(t *testing.T) {
{"short marker_mask", func(b *infer.Batch) { b.MarkerMask = b.MarkerMask[:1] }, "marker_mask"},
{"short qtype", func(b *infer.Batch) { b.QType = nil }, "qtype"},
{"one marker slot", func(b *infer.Batch) { *b = batch(3, 4, 1) }, "other"},
+ {"no marker slots", func(b *infer.Batch) { *b = batch(3, 4, 0) }, "other"},
{"negative rows", func(b *infer.Batch) { b.Rows = -1 }, "other"},
+ {"negative length", func(b *infer.Batch) { b.Length = -1 }, "other"},
+ {"negative markers", func(b *infer.Batch) { b.Markers = -2 }, "other"},
+ {"rows of no tokens", func(b *infer.Batch) { *b = batch(3, 0, 2) }, "other"},
+ {"marker position below zero", func(b *infer.Batch) { b.MarkerPos[2] = -1 }, "other"},
+ {"marker position at the length", func(b *infer.Batch) { b.MarkerPos[5] = 4 }, "other"},
+ {"marker position in range", func(b *infer.Batch) { b.MarkerPos[5] = 3 }, ""},
+ // 2 * 2^62 wraps to the most negative int, which no slice length equals.
+ {"rows x length overflows", func(b *infer.Batch) { *b = infer.Batch{Rows: 2, Length: math.MaxInt/2 + 1, Markers: 2} }, "overflow"},
+ {"rows x markers overflows", func(b *infer.Batch) { *b = infer.Batch{Rows: 4, Length: 1, Markers: math.MaxInt/2 + 1} }, "overflow"},
+ {"rows x length wraps to zero", func(b *infer.Batch) { *b = infer.Batch{Rows: math.MaxInt/2 + 1, Length: 2, Markers: 2} }, "overflow"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -42,9 +55,11 @@ func TestBatchValidate(t *testing.T) {
switch {
case c.field == "" && err != nil:
t.Fatalf("Validate = %v, want nil", err)
+ case c.field == "overflow" && (err == nil || errors.As(err, &se) || !strings.Contains(err.Error(), "overflow")):
+ t.Fatalf("Validate = %v, want an overflow error", err)
case c.field == "other" && (err == nil || errors.As(err, &se)):
t.Fatalf("Validate = %v, want a plain error", err)
- case c.field != "" && c.field != "other" && (!errors.As(err, &se) || se.Name != c.field):
+ case c.field != "" && c.field != "other" && c.field != "overflow" && (!errors.As(err, &se) || se.Name != c.field):
t.Fatalf("Validate = %v, want a ShapeError for %s", err, c.field)
}
})
@@ -70,6 +85,25 @@ func TestOutputCheck(t *testing.T) {
}
}
+func TestMessagesAndNamesCarryThePackage(t *testing.T) {
+ for _, err := range []error{
+ infer.ErrClosed, &infer.ShapeError{Name: infer.NameLogits, Got: 1, Want: 2},
+ infer.Batch{Rows: -1}.Validate(), batch(1, 1, 1).Validate(),
+ } {
+ if !strings.HasPrefix(err.Error(), "infer: ") {
+ t.Errorf("%q lacks the package prefix", err)
+ }
+ }
+ for got, want := range map[string]string{
+ infer.NameInputIDs: "input_ids", infer.NameAttentionMask: "attention_mask", infer.NameMarkerPos: "marker_pos",
+ infer.NameMarkerMask: "marker_mask", infer.NameQType: "qtype", infer.NameLogits: "logits", infer.NameActLogits: "act_logits",
+ } {
+ if got != want {
+ t.Errorf("tensor name %q, want %q", got, want)
+ }
+ }
+}
+
// TestCollateBatchIsInferBatch pins the alias: a Batch the collator builds is a valid infer.Batch.
func TestCollateBatchIsInferBatch(t *testing.T) {
b, err := collate.Collate([]collate.Row{
@@ -84,3 +118,12 @@ func TestCollateBatchIsInferBatch(t *testing.T) {
t.Fatalf("Validate = %v on %+v", err, ib)
}
}
+
+// TestValidateNamesTheEmptyRow pins that rows of no tokens are refused for that reason, not as a
+// side effect of the marker range check.
+func TestValidateNamesTheEmptyRow(t *testing.T) {
+ err := batch(3, 0, 2).Validate()
+ if err == nil || !strings.Contains(err.Error(), "0 tokens") {
+ t.Fatalf("Validate = %v", err)
+ }
+}
diff --git a/laya-go/internal/collate/collate.go b/laya-go/internal/collate/collate.go
index e33eabdc..baccac14 100644
--- a/laya-go/internal/collate/collate.go
+++ b/laya-go/internal/collate/collate.go
@@ -1,5 +1,5 @@
// Package collate pads built sequences into the rectangular tensors the graph takes. The
-// tensor layout is the ONNX graph's contract, not a caller's, so this stays internal.
+// tensor layout is the ONNX graph's contract; callers see the result as an infer.Batch.
//
// One row per question, not per state: four questions about one document are four rows that
// share a state. Rows are padded to the longest sequence and markers to the most options.
diff --git a/laya-go/scripts/gen_go_fixtures.py b/laya-go/scripts/gen_go_fixtures.py
index 600b8eb9..fc64125f 100644
--- a/laya-go/scripts/gen_go_fixtures.py
+++ b/laya-go/scripts/gen_go_fixtures.py
@@ -961,13 +961,37 @@ def build():
EXP_DEPENDENT = ("label_answers", "answers_fuzz")
+def _unnumber(value):
+ """value with every number replaced by 0, so only its structure is left."""
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, (int, float)):
+ return 0
+ if isinstance(value, dict):
+ return {k: _unnumber(v) for k, v in value.items()}
+ if isinstance(value, list):
+ return [_unnumber(v) for v in value]
+ return value
+
+
+def _answers_shape(answers):
+ """Each answer's type, key sets and order, legend and ids; no number, and no chosen value."""
+ out = {}
+ for question_id, answer in answers.items():
+ shaped = _unnumber(answer)
+ if "choice" in shaped:
+ shaped["choice"] = None
+ out[question_id] = shaped
+ return out
+
+
def portable(payload):
out = dict(payload)
for name in EXP_DEPENDENT:
out[name] = {"cases": [{k: v for k, v in case.items() if k != "answer_text"}
for case in payload[name]["cases"]]}
out["predict_plan"] = {
- name: ({"cases": [{**case, "results": [{k: v for k, v in r.items() if k != "answers"}
+ name: ({"cases": [{**case, "results": [{**r, "answers": _answers_shape(r["answers"])}
for r in case["results"]]}
for case in value["cases"]]} if name != "questions" else value)
for name, value in payload["predict_plan"].items()}