Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

router: model lifecycle (factory, leases, LRU, single-flight, on_load/on_evict)

Adeshina Lasisi7d5875d08d48
diff --git a/laya-go/README.md b/laya-go/README.md
index 7eab1e89..9f3d3082 100644
--- a/laya-go/README.md
+++ b/laya-go/README.md
@@ -18,11 +18,7 @@ The foundation (language detection, questions, sequences, the tokenizer) and the
| `sequence` | `build_sequence`, `build_head`, `state_room`, `window_budget` and `window_batch_cap`: `EncodeState`, `Build`, `StateRoom`, `PlanWindows`, `BatchCap`. |
| `infer` | The seam to a model runtime: `Batch` (the graph's five inputs), `Output` (its two), `Session`, and the shape checks `Batch.Validate` and `Output.Check`. |
| `agent` | `ONNXAgent.predict` and `predict_batch` over an `infer.Session`: `Open`, `Predict`, `PredictBatch`, `LoadConfig`, `WithCalibration` (installs a `calibrate` payload at `Open`) and `CalibrationPayload` (writes one), with batch size, sort-by-length and context cancellation between runs. |
-
-| `router` | Which checkpoint reads a request, the port of the routing half of `laya/router.py`: `New`, `Route`, `RouteBatch`, `NormaliseName`, `EnglishFromCode`, `MatchTypedDecisionsWorkflow`, `QuestionSchema`. Decisions carry laya's reason strings. Loads nothing. |
-
-| `router` | Which checkpoint reads a request, the port of the routing half of `laya/router.py`: `New`, `Route`, `RouteBatch`, `NormaliseName`, `EnglishFromCode`, `MatchTypedDecisionsWorkflow`, `QuestionSchema`. Decisions carry laya's reason strings. `OnRoute` hooks (`AddHook`, `WithInstalledHooks`, `ForHooks`) may observe or replace a decision, under the agent's hook dispatcher. Loads nothing. |
-| `agent` | `ONNXAgent.predict` and `predict_batch` over an `infer.Session`: `Open`, `Predict`, `PredictBatch`, `LoadConfig`, with batch size, sort-by-length and context cancellation between runs. |
+| `router` | Which checkpoint reads a request, the port of the routing half of `laya/router.py`: `New`, `Route`, `RouteBatch`, `NormaliseName`, `EnglishFromCode`, `MatchTypedDecisionsWorkflow`, `QuestionSchema`. Decisions carry laya's reason strings. `OnRoute` hooks (`AddHook`, `WithInstalledHooks`, `ForHooks`) may observe or replace a decision, under the agent's hook dispatcher. With `WithAgentFactory` it also loads: `Load`, `Lease` (an evicted agent closes after its last lease), `Attach`, `Preload`, `Unload`, `UnloadAll`, `Close`, LRU `WithMaxLoaded`, single-flight builds, `OnLoad`/`OnEvict` hooks. |
| `onnx` | A nested module (cgo, its own `go.mod`): `onnx.Open` runs a fused `laya.onnx` on ONNX Runtime and implements `infer.Session`, for `agent.Open`. Needs an ONNX Runtime 1.29+ shared library; `CGO_ENABLED=0` builds and `Open` returns `ErrNoCGO`. |
| `tokenizer` | A reader and runner for Hugging Face `tokenizer.json` (BPE): normalizer, pre-tokenizer, added tokens, byte fallback, decoder. |
| `internal/unicodedata` | The character predicates `laya/lang.py` leans on, recorded from CPython. |
diff --git a/laya-go/router/doc.go b/laya-go/router/doc.go
index 17a7351f..7bd1efe6 100644
--- a/laya-go/router/doc.go
+++ b/laya-go/router/doc.go
@@ -4,6 +4,16 @@
// hooks.go) may observe or replace the decision. Loading and routed inference are separate
// changes.
//
+// # Loading
+//
+// A Router built with WithAgentFactory also owns the models' lifecycle, as laya's Router does:
+// Load and Lease build a checkpoint on first use, concurrent callers share one build, a build that
+// fails is not cached, and at most WithMaxLoaded (default 2) stay resident, the least recently used
+// leaving first. Lease is the addition Go needs: an evicted agent closes only when its last Lease
+// is closed, so a prediction in flight never meets a closed session. Attach hands in an agent the
+// caller built, Preload builds checkpoints up front, Unload, UnloadAll and Close free them.
+// OnLoad and OnEvict hooks run after every internal lock is released.
+//
// The checkpoints are English, Multilingual and TypedDecisions. The order of the evidence is
// fixed, first match wins:
//
@@ -25,8 +35,10 @@
// A name is a string: Python's str(name) of a non-string has no counterpart. Router.register and
// the custom checkpoints are not ported, so the Router knows the three built-in names. A hint
// callable is a LangHint returning "" to abstain. explain=True is not ported: lang.Analysis has no
-// diagnostic field. The model lifecycle (on_load,
-// on_evict) belongs to a later change. Hooks run concurrently as in Python; WithSerialHooks
+// diagnostic field. Loading needs an AgentFactory: there is no hub download. WithMaxLoaded below 1
+// is an error, where Python clamps it to 1. Attach of a name the Router does not know is an error,
+// where Python registers it. Load returns an agent that closes on eviction; Lease keeps it open.
+// Attach and Preload raise MaxLoaded as in Python. Hooks run concurrently as in Python; WithSerialHooks
// runs one at a time (and then a hook must not call Route on its own Router). A hook failure
// names the hook's Go type.
package router
diff --git a/laya-go/router/example_test.go b/laya-go/router/example_test.go
index 42e652ab..97416cf6 100644
--- a/laya-go/router/example_test.go
+++ b/laya-go/router/example_test.go
@@ -1,8 +1,10 @@
package router_test
import (
+ "context"
"fmt"
+ "github.com/NandhaKishorM/laya/laya-go/agent"
"github.com/NandhaKishorM/laya/laya-go/router"
"github.com/NandhaKishorM/laya/laya-go/schema"
)
@@ -82,3 +84,20 @@ func ExampleRouter_AddHook() {
// saw english
// multilingual | pinned by policy
}
+
+func ExampleRouter_Lease() {
+ // The factory owns where a checkpoint comes from; a real one opens the weights with
+ // agent.Open and an onnx session. This one stands in.
+ var open router.AgentFactory = func(ctx context.Context, c router.Checkpoint) (*agent.Agent, error) {
+ return nil, fmt.Errorf("no weights for %s in this example", c)
+ }
+ r, _ := router.New(router.WithAgentFactory(open), router.WithMaxLoaded(1))
+ defer r.Close()
+
+ _, err := r.Lease(context.Background(), "english")
+ fmt.Println(err)
+ fmt.Println(r.Loaded(), r.MaxLoaded())
+ // Output:
+ // router: cannot load the english checkpoint: no weights for english in this example
+ // [] 1
+}
diff --git a/laya-go/router/hooks.go b/laya-go/router/hooks.go
index 13f89562..c54a36b9 100644
--- a/laya-go/router/hooks.go
+++ b/laya-go/router/hooks.go
@@ -1,8 +1,9 @@
package router
-// Hooks are laya/hooks.py for the one event a pure Router raises, on_route (on_load and on_evict
-// arrive with loading). A hook is any value with an OnRoute method; the same value may also be an
-// agent.Hook, as in Python, where one object implements several events. Each call of Route
+// Hooks are laya/hooks.py for the three events a Router raises: on_route, on_load and on_evict. A
+// hook is any value with an OnRoute, OnLoad or OnEvict method; the same value may also be an
+// agent.Hook, as in Python, where one object implements several events. OnLoad and OnEvict run
+// after the Router's lock is released (see lifecycle.go), so a hook may call the Router. Each call of Route
// dispatches over: the process-wide agent.DefaultHooks (which take only an agent.Hook, so a
// hook that is both embeds agent.BaseHook), the Router's installed hooks, then the
// call's ForHooks, and a hook without OnRoute is skipped.
@@ -54,16 +55,76 @@ type RouteContext struct {
Router *Router
}
-// BaseHook implements RouteHook with a no-op, for embedding.
+// LoadHook is a hook that sees a checkpoint being built.
+type LoadHook interface {
+ // OnLoad runs after the Router built and published a checkpoint, once per build, after the
+ // OnEvict calls of the checkpoints that build pushed out.
+ OnLoad(ctx *LoadContext)
+}
+
+// EvictHook is a hook that sees a checkpoint leave the Router.
+type EvictHook interface {
+ // OnEvict runs after a checkpoint left the Router, by LRU eviction, Unload, UnloadAll or
+ // Close. The agent may still be open: it closes when its last lease is released.
+ OnEvict(ctx *EvictContext)
+}
+
+// LoadContext is what an OnLoad hook sees. The agent is the one the Router now serves, which for
+// a build that lost a race with Attach is the attached one.
+type LoadContext struct {
+ // Model is the checkpoint that was built.
+ Model Checkpoint
+ // Agent is the agent serving Model. It can be evicted and closed at any moment; take a Lease
+ // to use it.
+ Agent *agent.Agent
+ // RunID is 32 hex characters, new for each event.
+ RunID string
+ // Router is the Router that loaded.
+ Router *Router
+}
+
+// EvictContext is what an OnEvict hook sees.
+type EvictContext struct {
+ // Model is the checkpoint that left.
+ Model Checkpoint
+ // RunID is 32 hex characters, new for each event.
+ RunID string
+ // Router is the Router it left.
+ Router *Router
+}
+
+// BaseHook implements RouteHook, LoadHook and EvictHook with no-ops, for embedding.
type BaseHook struct{}
// OnRoute does nothing.
func (BaseHook) OnRoute(*RouteContext) {}
-// HookFuncs is a RouteHook made of a function; nil is a no-op. Use it as *HookFuncs: a value
-// holds a func, which cannot be compared, so RemoveHook could never find it.
+// OnLoad does nothing.
+func (BaseHook) OnLoad(*LoadContext) {}
+
+// OnEvict does nothing.
+func (BaseHook) OnEvict(*EvictContext) {}
+
+// HookFuncs is a RouteHook, LoadHook and EvictHook made of functions; nil is a no-op. Use it as
+// *HookFuncs: a value holds funcs, which cannot be compared, so RemoveHook could never find it.
type HookFuncs struct {
OnRouteFunc func(*RouteContext)
+ OnLoadFunc func(*LoadContext)
+ OnEvictFunc func(*EvictContext)
+}
+
+// OnLoad calls OnLoadFunc.
+func (f *HookFuncs) OnLoad(c *LoadContext) {
+ if f.OnLoadFunc != nil {
+ f.OnLoadFunc(c)
+ }
+}
+
+// OnEvict calls OnEvictFunc.
+func (f *HookFuncs) OnEvict(c *EvictContext) {
+ if f.OnEvictFunc != nil {
+ f.OnEvictFunc(c)
+ }
}
// OnRoute calls OnRouteFunc.
@@ -121,9 +182,11 @@ func checkHooks(hooks []any) error {
return fmt.Errorf("router: hook %d (%T) cannot be compared, so it could never be removed; pass a pointer", i, h)
}
_, route := h.(RouteHook)
+ _, load := h.(LoadHook)
+ _, evict := h.(EvictHook)
_, predict := h.(agent.Hook)
- if !route && !predict {
- return fmt.Errorf("router: hook %d (%T) implements neither OnRoute nor agent.Hook", i, h)
+ if !route && !load && !evict && !predict {
+ return fmt.Errorf("router: hook %d (%T) implements none of OnRoute, OnLoad, OnEvict or agent.Hook", i, h)
}
}
return nil
@@ -331,3 +394,42 @@ func (r *Router) dispatchRoute(state any, qs schema.Questions, d Decision, c cal
}
return rc.Decision, nil
}
+
+// dispatchLifecycle runs one lifecycle event over the hooks that implement it, with the Router's
+// own policy; a call has no overrides. pick returns the hook's call, or nil when it lacks the
+// event. It must be called with no lock of the lifecycle held.
+func (r *Router) dispatchLifecycle(event string, pick func(h any) func()) error {
+ var calls []func()
+ var all []any
+ for _, h := range r.composeHooks(call{}) {
+ if f := pick(h); f != nil {
+ calls, all = append(calls, f), append(all, h)
+ }
+ }
+ if len(calls) == 0 {
+ return nil
+ }
+ run := hookrun.Run{Lock: r.hk.lock, Policy: hookrun.Policy{Timeout: r.hk.timeout, Raise: r.hk.raise, Warn: r.hk.warn}}
+ return run.Dispatch(context.Background(), event, len(calls),
+ func(i int) string { return hookrun.Name(all[i]) }, func(i int) { calls[i]() })
+}
+
+func (r *Router) dispatchLoad(key Checkpoint, a *agent.Agent) error {
+ ctx := &LoadContext{Model: key, Agent: a, RunID: newRunID(), Router: r}
+ return r.dispatchLifecycle("on_load", func(h any) func() {
+ if lh, ok := h.(LoadHook); ok {
+ return func() { lh.OnLoad(ctx) }
+ }
+ return nil
+ })
+}
+
+func (r *Router) dispatchEvict(key Checkpoint) error {
+ ctx := &EvictContext{Model: key, RunID: newRunID(), Router: r}
+ return r.dispatchLifecycle("on_evict", func(h any) func() {
+ if eh, ok := h.(EvictHook); ok {
+ return func() { eh.OnEvict(ctx) }
+ }
+ return nil
+ })
+}
diff --git a/laya-go/router/lifecycle.go b/laya-go/router/lifecycle.go
new file mode 100644
index 00000000..1587342a
--- /dev/null
+++ b/laya-go/router/lifecycle.go
@@ -0,0 +1,597 @@
+package router
+
+// The model lifecycle is laya/router.py's load, attach, preload, unload and eviction, with one
+// addition Python does not need: a Lease. Python frees an agent by dropping its reference and lets
+// the garbage collector decide when; an Agent here owns a native session that Close frees, so
+// eviction must not close an agent a caller is still predicting with. A slot counts its leases,
+// and an agent the Router built closes when it has left the Router and its last lease is
+// released.
+//
+// Locking, as in Python: mu guards the slots, the LRU order, the in-flight builds and the flags.
+// It is never held while a factory runs, while an agent closes, or while a hook runs. build
+// serialises factory calls across checkpoints, so two cold loads never hold two checkpoints in
+// flight at once. A load of a resident checkpoint never touches build.
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "sync"
+ "sync/atomic"
+
+ "github.com/NandhaKishorM/laya/laya-go/agent"
+)
+
+// DefaultMaxLoaded is how many checkpoints a Router keeps resident unless WithMaxLoaded says
+// otherwise.
+const DefaultMaxLoaded = 2
+
+// ErrClosed is returned by a Router that has been closed.
+var ErrClosed = errors.New("router: closed")
+
+// AgentFactory builds the agent for a checkpoint. It owns where the weights come from, the
+// device and the agent options. It is called at most once at a time per Router and never with a
+// lock held, and ctx is the context of the call that triggered the build. The Router closes the
+// agent it returns, so a factory must return a fresh agent on every call, never one it shares.
+type AgentFactory func(ctx context.Context, c Checkpoint) (*agent.Agent, error)
+
+// WithAgentFactory lets the Router load checkpoints. Without it Route works and every load fails.
+func WithAgentFactory(f AgentFactory) Option {
+ return func(c *config) error { c.factory = f; return nil }
+}
+
+// WithMaxLoaded sets how many checkpoints stay resident before the least recently used is
+// evicted; n must be at least 1. Attach and Preload raise it to fit what they make resident,
+// and never lower it.
+func WithMaxLoaded(n int) Option {
+ return func(c *config) error {
+ if n < 1 {
+ return fmt.Errorf("router: max loaded must be at least 1, got %d", n)
+ }
+ c.maxLoaded = n
+ return nil
+ }
+}
+
+type slot struct {
+ agent *agent.Agent
+ leases int
+ retired bool // left the Router
+ owns bool // the Router built it, so the Router closes it
+}
+
+type inflight struct {
+ done chan struct{}
+ err error // set before done closes
+}
+
+type lifecycle struct {
+ factory AgentFactory
+ build chan struct{} // one token: serialises factory calls
+
+ mu sync.Mutex
+ slots map[Checkpoint]*slot
+ order []Checkpoint // least recently used first
+ loading map[Checkpoint]*inflight
+ max int
+ closed bool
+}
+
+func (l *lifecycle) init(f AgentFactory, max int) {
+ l.factory, l.max = f, max
+ l.build = make(chan struct{}, 1)
+ l.slots = map[Checkpoint]*slot{}
+ l.loading = map[Checkpoint]*inflight{}
+}
+
+// outcome is what one acquire did, so its hooks fire after the locks are released.
+type outcome struct {
+ built bool
+ agent *agent.Agent
+ evicted []Checkpoint
+ closeErr error // closing what the build pushed out failed; the build still happened
+}
+
+// touch moves key to the most recently used end. mu is held.
+func (l *lifecycle) touch(key Checkpoint) {
+ for i, k := range l.order {
+ if k == key {
+ l.order = append(l.order[:i], l.order[i+1:]...)
+ break
+ }
+ }
+ l.order = append(l.order, key)
+}
+
+// retire marks s as gone from the Router and collects its agent for closing when nothing is using
+// it and the Router built it. mu is held.
+func (l *lifecycle) retire(s *slot, toClose *[]*agent.Agent) {
+ s.retired = true
+ if s.leases <= 0 && s.owns {
+ *toClose = append(*toClose, s.agent)
+ }
+}
+
+// evictLocked drops least recently used slots until max holds and returns who left. mu is held.
+func (l *lifecycle) evictLocked(toClose *[]*agent.Agent) []Checkpoint {
+ var evicted []Checkpoint
+ for len(l.order) > l.max {
+ victim := l.order[0]
+ l.order = l.order[1:]
+ if s := l.slots[victim]; s != nil {
+ delete(l.slots, victim)
+ evicted = append(evicted, victim)
+ l.retire(s, toClose)
+ }
+ }
+ return evicted
+}
+
+func closeAll(agents []*agent.Agent) error {
+ var errs []error
+ for _, a := range agents {
+ if err := a.Close(); err != nil {
+ errs = append(errs, err)
+ }
+ }
+ return errors.Join(errs...)
+}
+
+// release gives back one lease and closes the agent when it has left and this was the last.
+func (l *lifecycle) release(s *slot) error {
+ var toClose *agent.Agent
+ l.mu.Lock()
+ s.leases--
+ if s.leases <= 0 && s.retired && s.owns {
+ toClose = s.agent
+ }
+ l.mu.Unlock()
+ if toClose != nil {
+ return toClose.Close()
+ }
+ return nil
+}
+
+func isContextErr(err error) bool {
+ return errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)
+}
+
+// acquire returns the slot for key with one lease taken, building it if needed. Concurrent
+// callers for one key share a single build; a caller whose build failed returns the error and
+// caches nothing, and the others get the same error. A waiter does not inherit the builder's
+// cancellation: it builds for itself.
+func (l *lifecycle) acquire(ctx context.Context, key Checkpoint, out *outcome) (*slot, error) {
+ for {
+ l.mu.Lock()
+ if l.closed {
+ l.mu.Unlock()
+ return nil, ErrClosed
+ }
+ if s := l.slots[key]; s != nil { // not behind build: a resident load never waits for a cold one
+ l.touch(key)
+ s.leases++
+ l.mu.Unlock()
+ return s, nil
+ }
+ wait := l.loading[key]
+ var mine *inflight
+ if wait == nil {
+ mine = &inflight{done: make(chan struct{})}
+ l.loading[key] = mine
+ }
+ l.mu.Unlock()
+ if mine != nil {
+ return l.doBuild(ctx, key, mine, out)
+ }
+ select {
+ case <-wait.done:
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ }
+ if wait.err != nil {
+ if isContextErr(wait.err) && ctx.Err() == nil {
+ continue
+ }
+ return nil, fmt.Errorf("router: loading %s failed on the call that was building it: %w", key, wait.err)
+ }
+ }
+}
+
+func (l *lifecycle) create(ctx context.Context, key Checkpoint) (*agent.Agent, error) {
+ if l.factory == nil {
+ return nil, errors.New("router: this Router can route but not load: no agent factory was given; use WithAgentFactory")
+ }
+ a, err := l.factory(ctx, key)
+ if err != nil {
+ return nil, fmt.Errorf("router: cannot load the %s checkpoint: %w", key, err)
+ }
+ if a == nil {
+ return nil, fmt.Errorf("router: the agent factory returned no agent for %s", key)
+ }
+ return a, nil
+}
+
+// doBuild is the build this caller owns: mine is the handle the others wait on.
+func (l *lifecycle) doBuild(ctx context.Context, key Checkpoint, mine *inflight, out *outcome) (built *slot, err error) {
+ var toClose []*agent.Agent
+ finish := func() { // mu is held
+ if l.loading[key] == mine {
+ delete(l.loading, key)
+ }
+ close(mine.done)
+ }
+ defer func() {
+ rec := recover()
+ if rec != nil {
+ err = fmt.Errorf("router: building %s panicked: %v", key, rec)
+ }
+ if err != nil {
+ built = nil
+ l.mu.Lock()
+ mine.err = err
+ finish()
+ l.mu.Unlock()
+ }
+ if rec != nil {
+ panic(rec)
+ }
+ }()
+ select {
+ case l.build <- struct{}{}:
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ }
+ released := false
+ releaseBuild := func() {
+ if !released {
+ released = true
+ <-l.build
+ }
+ }
+ defer releaseBuild()
+
+ l.mu.Lock()
+ if s := l.slots[key]; s != nil { // built or attached while this caller waited for the build token
+ l.touch(key)
+ s.leases++
+ finish()
+ l.mu.Unlock()
+ return s, nil
+ }
+ l.mu.Unlock()
+
+ a, err := l.create(ctx, key)
+ if err != nil {
+ return nil, err
+ }
+
+ l.mu.Lock()
+ if s := l.slots[key]; s != nil { // attached while it was building: keep that one
+ l.touch(key)
+ s.leases++
+ built = s
+ toClose = append(toClose, a)
+ out.built, out.agent = true, s.agent
+ } else {
+ built = &slot{agent: a, owns: true, leases: 1}
+ l.slots[key] = built
+ l.order = append(l.order, key)
+ out.evicted = l.evictLocked(&toClose)
+ out.built, out.agent = true, a
+ }
+ finish()
+ l.mu.Unlock()
+ releaseBuild()
+ // Outside every lock: closing a native session can be slow.
+ out.closeErr = closeAll(toClose)
+ return built, nil
+}
+
+// take acquires a lease for key and settles what the build owes: a failed close of an evicted
+// agent fails the call, but the events are still delivered.
+func (r *Router) take(ctx context.Context, key Checkpoint) (*slot, *outcome, error) {
+ out := &outcome{}
+ s, err := r.lc.acquire(ctx, key, out)
+ if err != nil {
+ return nil, nil, err
+ }
+ if out.closeErr != nil {
+ err := out.closeErr
+ if rerr := r.lc.release(s); rerr != nil {
+ err = errors.Join(err, rerr)
+ }
+ if herr := r.dispatchOutcome(key, out); herr != nil {
+ err = errors.Join(err, herr)
+ }
+ return nil, nil, err
+ }
+ return s, out, nil
+}
+
+// dispatchOutcome fires the evictions a build caused, then its load, stopping at the first hook
+// error as Python does. It must be called with no lock held.
+func (r *Router) dispatchOutcome(key Checkpoint, out *outcome) error {
+ for _, ev := range out.evicted {
+ if err := r.dispatchEvict(ev); err != nil {
+ return err
+ }
+ }
+ if out.built {
+ return r.dispatchLoad(key, out.agent)
+ }
+ return nil
+}
+
+// Lease is a borrowed agent. The agent stays open until Close is called, however many times the
+// Router evicts or unloads it meanwhile. Close it when done; the zero Lease holds nothing.
+type Lease struct {
+ r *Router
+ s *slot
+ key Checkpoint
+ released atomic.Bool
+}
+
+// Checkpoint is the checkpoint the lease was taken for.
+func (l *Lease) Checkpoint() Checkpoint { return l.key }
+
+// Agent is the leased agent, or nil once the lease is closed.
+func (l *Lease) Agent() *agent.Agent {
+ if l == nil || l.s == nil || l.released.Load() {
+ return nil
+ }
+ return l.s.agent
+}
+
+// Close gives the lease back; when the agent has left the Router and this was its last lease, the
+// agent closes and Close returns its error. Closing twice is safe: only the first counts.
+func (l *Lease) Close() error {
+ if l == nil || l.s == nil || !l.released.CompareAndSwap(false, true) {
+ return nil
+ }
+ return l.r.lc.release(l.s)
+}
+
+// Lease borrows the agent for a checkpoint (a name or alias), building it on first use. OnEvict
+// and OnLoad hooks have run when it returns, with the lease already held, so the agent is never
+// closed under them. If a hook fails the lease is given back and the error returned, and the
+// checkpoint stays resident.
+func (r *Router) Lease(ctx context.Context, name string) (*Lease, error) {
+ key, err := NormaliseName(name)
+ if err != nil {
+ return nil, err
+ }
+ s, out, err := r.take(ctx, key)
+ if err != nil {
+ return nil, err
+ }
+ if err := r.dispatchOutcome(key, out); err != nil {
+ return nil, errors.Join(err, r.lc.release(s))
+ }
+ return &Lease{r: r, s: s, key: key}, nil
+}
+
+// Load returns the agent for a checkpoint, building it on first use. Concurrent callers share one
+// build. The agent is valid only while it stays resident: eviction and Unload close an agent the
+// Router built, so a reference held across either is a closed session. Use Lease, which holds the
+// agent open, unless the caller manages the lifetime itself. If a hook fails Load returns the
+// error and no agent, and the checkpoint stays resident.
+func (r *Router) Load(ctx context.Context, name string) (*agent.Agent, error) {
+ key, err := NormaliseName(name)
+ if err != nil {
+ return nil, err
+ }
+ s, out, err := r.take(ctx, key)
+ if err != nil {
+ return nil, err
+ }
+ // After release, so a hook that calls back in does not find this call holding a lease.
+ rerr := r.lc.release(s)
+ if herr := r.dispatchOutcome(key, out); herr != nil || rerr != nil {
+ return nil, errors.Join(herr, rerr)
+ }
+ return s.agent, nil
+}
+
+// Attach makes an agent the caller built resident under a checkpoint name, instead of loading a
+// second copy. The Router never closes an attached agent: the caller keeps ownership. It does not
+// evict; MaxLoaded is raised to fit everything resident. An agent already resident under the name
+// is replaced, and closes when its last lease is released if the Router built it. Attaching the
+// agent already there hands it to the caller: the Router stops owning it and never closes it. Python registers an unknown name on the spot; the
+// names here are fixed, so an unknown one is an *UnknownModelError. No hook runs.
+func (r *Router) Attach(name string, a *agent.Agent) error {
+ key, err := NormaliseName(name)
+ if err != nil {
+ return err
+ }
+ if a == nil {
+ return errors.New("router: Attach needs an agent")
+ }
+ l := &r.lc
+ var toClose []*agent.Agent
+ l.mu.Lock()
+ if l.closed {
+ l.mu.Unlock()
+ return ErrClosed
+ }
+ prev := l.slots[key]
+ l.slots[key] = &slot{agent: a}
+ if prev != nil && prev.agent != a { // the same agent changes hands, it is not closed
+ l.retire(prev, &toClose)
+ }
+ l.touch(key)
+ if len(l.slots) > l.max {
+ l.max = len(l.slots)
+ }
+ l.mu.Unlock()
+ return closeAll(toClose)
+}
+
+// Preload builds checkpoints up front so no request pays a cold load; with no names, all of them.
+// MaxLoaded is raised to fit both the checkpoints asked for and everything resident, so staged
+// preloading evicts neither. A resident checkpoint is left where it is in the LRU order. It stops
+// at the first failure.
+func (r *Router) Preload(ctx context.Context, names ...string) error {
+ keys := Checkpoints()
+ if len(names) > 0 {
+ keys = keys[:0:0]
+ for _, n := range names {
+ key, err := NormaliseName(n)
+ if err != nil {
+ return err
+ }
+ keys = append(keys, key)
+ }
+ }
+ l := &r.lc
+ l.mu.Lock()
+ if l.closed {
+ l.mu.Unlock()
+ return ErrClosed
+ }
+ union := map[Checkpoint]bool{}
+ for _, k := range keys {
+ union[k] = true
+ }
+ for k := range l.slots {
+ union[k] = true
+ }
+ if len(union) > l.max {
+ l.max = len(union)
+ }
+ l.mu.Unlock()
+ for _, key := range keys {
+ l.mu.Lock()
+ already := l.slots[key] != nil
+ l.mu.Unlock()
+ if already {
+ continue
+ }
+ s, out, err := r.take(ctx, key)
+ if err != nil {
+ return err
+ }
+ rerr := l.release(s)
+ if herr := r.dispatchOutcome(key, out); herr != nil || rerr != nil {
+ return errors.Join(herr, rerr)
+ }
+ }
+ return nil
+}
+
+// evictAll fires OnEvict for each freed checkpoint, stopping at the first hook error, after the
+// agents that could close have. The events are owed whether or not a close failed.
+func (r *Router) evictAll(freed []Checkpoint, toClose []*agent.Agent) error {
+ cerr := closeAll(toClose)
+ for _, k := range freed {
+ if err := r.dispatchEvict(k); err != nil {
+ return errors.Join(cerr, err)
+ }
+ }
+ return cerr
+}
+
+// Unload frees one checkpoint (a name or alias) and returns it if it was resident. It waits for
+// an in-flight build of that checkpoint only. The slot goes at once; its agent closes when the
+// last lease is released. OnEvict fires for a checkpoint freed.
+func (r *Router) Unload(ctx context.Context, name string) ([]Checkpoint, error) {
+ key, err := NormaliseName(name)
+ if err != nil {
+ return nil, err
+ }
+ l := &r.lc
+ for {
+ var toClose []*agent.Agent
+ var freed []Checkpoint
+ l.mu.Lock()
+ wait := l.loading[key]
+ if wait == nil {
+ if s := l.slots[key]; s != nil {
+ delete(l.slots, key)
+ freed = append(freed, key)
+ l.retire(s, &toClose)
+ }
+ l.removeOrder(key)
+ }
+ l.mu.Unlock()
+ if wait == nil {
+ return freed, r.evictAll(freed, toClose)
+ }
+ select {
+ case <-wait.done:
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ }
+ }
+}
+
+// removeOrder drops key from the LRU order. mu is held.
+func (l *lifecycle) removeOrder(key Checkpoint) {
+ for i, k := range l.order {
+ if k == key {
+ l.order = append(l.order[:i], l.order[i+1:]...)
+ return
+ }
+ }
+}
+
+// UnloadAll frees every checkpoint, least recently used first, after any build in flight has
+// finished, and returns who was freed.
+func (r *Router) UnloadAll(ctx context.Context) ([]Checkpoint, error) {
+ l := &r.lc
+ for {
+ var toClose []*agent.Agent
+ var freed []Checkpoint
+ var waits []*inflight
+ l.mu.Lock()
+ for _, w := range l.loading {
+ waits = append(waits, w)
+ }
+ if len(waits) == 0 {
+ freed = append(freed, l.order...)
+ for _, k := range l.order {
+ if s := l.slots[k]; s != nil {
+ l.retire(s, &toClose)
+ }
+ }
+ l.slots = map[Checkpoint]*slot{}
+ l.order = nil
+ }
+ l.mu.Unlock()
+ if len(waits) == 0 {
+ return freed, r.evictAll(freed, toClose)
+ }
+ for _, w := range waits {
+ select {
+ case <-w.done:
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ }
+ }
+ }
+}
+
+// Loaded lists the resident checkpoints, least recently used first.
+func (r *Router) Loaded() []Checkpoint {
+ r.lc.mu.Lock()
+ defer r.lc.mu.Unlock()
+ return append([]Checkpoint(nil), r.lc.order...)
+}
+
+// MaxLoaded is how many checkpoints the Router keeps before it evicts.
+func (r *Router) MaxLoaded() int {
+ r.lc.mu.Lock()
+ defer r.lc.mu.Unlock()
+ return r.lc.max
+}
+
+// Close seals the Router and frees every checkpoint. A build in flight finishes first, and its
+// caller gets the lease it asked for; that agent closes when the lease is released. An agent
+// under lease closes at its last release, an attached agent is left to its owner, and Close is
+// idempotent. Loads after Close return ErrClosed; Route still works.
+func (r *Router) Close() error {
+ r.lc.mu.Lock()
+ r.lc.closed = true
+ r.lc.mu.Unlock()
+ _, err := r.UnloadAll(context.Background())
+ return err
+}
diff --git a/laya-go/router/lifecycle_golden_test.go b/laya-go/router/lifecycle_golden_test.go
new file mode 100644
index 00000000..feccc7e8
--- /dev/null
+++ b/laya-go/router/lifecycle_golden_test.go
@@ -0,0 +1,294 @@
+package router_test
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "reflect"
+ "regexp"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/NandhaKishorM/laya/laya-go/agent"
+ "github.com/NandhaKishorM/laya/laya-go/internal/fixtures"
+ "github.com/NandhaKishorM/laya/laya-go/router"
+)
+
+// lifeLog is the golden's event log, shared by every hook of a case.
+type lifeLog struct {
+ mu sync.Mutex
+ events []string
+ labels map[*agent.Agent]string
+}
+
+func (l *lifeLog) add(s string) {
+ l.mu.Lock()
+ l.events = append(l.events, s)
+ l.mu.Unlock()
+}
+
+func (l *lifeLog) label(a *agent.Agent) string {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return l.labels[a]
+}
+
+// lifeAct is what a golden hook does after it logs: nothing, raise, or sleep.
+type lifeAct struct {
+ name, behaviour string
+ log *lifeLog
+}
+
+func (h *lifeAct) act() {
+ kind, arg, _ := strings.Cut(h.behaviour, ":")
+ switch kind {
+ case "raise":
+ panic(fmt.Errorf("boom from %s", h.name))
+ case "sleep":
+ var secs float64
+ fmt.Sscanf(arg, "%g", &secs)
+ time.Sleep(time.Duration(secs * float64(time.Second)))
+ }
+}
+
+func (h *lifeAct) onLoad(c *router.LoadContext) {
+ h.log.add(fmt.Sprintf("%s.on_load:%s:%s", h.name, c.Model, h.log.label(c.Agent)))
+ h.act()
+}
+
+func (h *lifeAct) onEvict(c *router.EvictContext) {
+ h.log.add(fmt.Sprintf("%s.on_evict:%s", h.name, c.Model))
+ h.act()
+}
+
+// The golden's hooks implement exactly the events it names, so each shape is its own type; the
+// failure text names the Go type, which lifeText maps back from Python's.
+type lifeLoad struct{ lifeAct }
+
+func (h *lifeLoad) OnLoad(c *router.LoadContext) { h.onLoad(c) }
+
+type lifeEvict struct{ lifeAct }
+
+func (h *lifeEvict) OnEvict(c *router.EvictContext) { h.onEvict(c) }
+
+type lifeBoth struct{ lifeAct }
+
+func (h *lifeBoth) OnLoad(c *router.LoadContext) { h.onLoad(c) }
+func (h *lifeBoth) OnEvict(c *router.EvictContext) { h.onEvict(c) }
+
+type lifeRoute struct{ router.BaseHook }
+
+func (lifeRoute) OnLoad(*router.LoadContext) {}
+func (lifeRoute) OnEvict(*router.EvictContext) {}
+
+type lifeOnlyRoute struct{}
+
+func (lifeOnlyRoute) OnRoute(*router.RouteContext) {}
+
+type lifePred struct{ agent.BaseHook }
+
+var lifeText = regexp.MustCompile(`laya: hook (\w+)\.(on_load|on_evict) (failed|exceeded)`)
+
+// goLifeText rewrites Python's "laya: hook Boom.on_load failed" to the Go type's name.
+func goLifeText(s string, types map[string]string) string {
+ return lifeText.ReplaceAllStringFunc(s, func(m string) string {
+ p := lifeText.FindStringSubmatch(m)
+ if p[3] == "exceeded" {
+ return "laya: hook " + types[p[1]] + " exceeded"
+ }
+ return "laya: hook " + types[p[1]] + "." + p[2] + " failed"
+ })
+}
+
+func TestLifecycleGolden(t *testing.T) {
+ root := fixtures.Load(t, "go_extra.json")
+ cases := fixtures.List(t, root.Must("lifecycle"))
+ if len(cases) < 250 {
+ t.Fatalf("only %d lifecycle cases", len(cases))
+ }
+ ops := 0
+ for _, cv := range cases {
+ c := fixtures.Obj(t, cv)
+ label := str(t, c, "case")
+ t.Run(label, func(t *testing.T) { ops += replayLifecycle(t, c) })
+ }
+ if ops < 1500 {
+ t.Fatalf("only %d ops replayed", ops)
+ }
+}
+
+func replayLifecycle(t *testing.T, c fixtures.Object) int {
+ t.Helper()
+ lg := &lifeLog{labels: map[*agent.Agent]string{}}
+ var mu sync.Mutex
+ calls := 0
+ fail := map[string]bool{}
+ factory := func(ctx context.Context, cp router.Checkpoint) (*agent.Agent, error) {
+ mu.Lock()
+ calls++
+ n := calls
+ bad := fail[string(cp)]
+ mu.Unlock()
+ if bad {
+ return nil, errors.New("boom building " + string(cp))
+ }
+ a := newProbe(t, nil)
+ lg.mu.Lock()
+ lg.labels[a] = fmt.Sprintf("built-%d", n)
+ lg.mu.Unlock()
+ return a, nil
+ }
+ var warns []string
+ var wmu sync.Mutex
+ opts := []router.Option{router.WithAgentFactory(factory), router.WithHookWarnings(func(m string) {
+ wmu.Lock()
+ warns = append(warns, m)
+ wmu.Unlock()
+ })}
+ for _, m := range fixtures.Obj(t, c.Must("router")) {
+ switch m.Key {
+ case "max_loaded":
+ opts = append(opts, router.WithMaxLoaded(fixtures.Int(t, m.Value)))
+ case "hooks_raise":
+ opts = append(opts, router.WithHookErrors(fixtures.Bool(t, m.Value)))
+ case "hooks_timeout":
+ opts = append(opts, router.WithHookTimeout(seconds(t, m.Value)))
+ case "hooks_concurrent":
+ if !fixtures.Bool(t, m.Value) {
+ opts = append(opts, router.WithSerialHooks())
+ }
+ case "standalone_repos":
+ opts = append(opts, router.WithStandaloneRepos(fixtures.Bool(t, m.Value)))
+ default:
+ t.Fatalf("unhandled router kwarg %q", m.Key)
+ }
+ }
+ types := map[string]string{}
+ var hooks []any
+ for _, hv := range fixtures.List(t, c.Must("hooks")) {
+ h := fixtures.Obj(t, hv)
+ name, beh := str(t, h, "name"), str(t, h, "behaviour")
+ act := lifeAct{name: name, behaviour: beh, log: lg}
+ ev := fixtures.Strings(t, h.Must("events"))
+ has := func(e string) bool {
+ for _, x := range ev {
+ if x == e {
+ return true
+ }
+ }
+ return false
+ }
+ switch {
+ case has("on_load") && has("on_evict"):
+ hooks, types[name] = append(hooks, &lifeBoth{act}), "lifeBoth"
+ case has("on_load"):
+ hooks, types[name] = append(hooks, &lifeLoad{act}), "lifeLoad"
+ case has("on_evict"):
+ hooks, types[name] = append(hooks, &lifeEvict{act}), "lifeEvict"
+ case has("on_route"):
+ hooks, types[name] = append(hooks, &lifeOnlyRoute{}), "lifeOnlyRoute"
+ default:
+ hooks, types[name] = append(hooks, &lifePred{}), "lifePred"
+ }
+ }
+ opts = append(opts, router.WithInstalledHooks(hooks...))
+ r, err := router.New(opts...)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { r.Close() })
+ if got, want := r.MaxLoaded(), fixtures.Int(t, c.Must("max_loaded")); got != want {
+ t.Fatalf("initial max_loaded = %d, want %d", got, want)
+ }
+ attached := map[string]*agent.Agent{}
+ ctx := context.Background()
+ steps := fixtures.List(t, c.Must("ops"))
+ for i, sv := range steps {
+ op := fixtures.Obj(t, sv)
+ lg.mu.Lock()
+ before := len(lg.events)
+ lg.mu.Unlock()
+ wmu.Lock()
+ warns = warns[:0]
+ wmu.Unlock()
+ var gotErr error
+ gotAgent := ""
+ name, _ := op.Get("name")
+ switch kind := str(t, op, "op"); kind {
+ case "load":
+ var a *agent.Agent
+ a, gotErr = r.Load(ctx, fixtures.Str(t, name))
+ gotAgent = lg.label(a)
+ case "preload":
+ var names []string
+ if v, _ := op.Get("names"); v != nil {
+ names = fixtures.Strings(t, v)
+ }
+ gotErr = r.Preload(ctx, names...)
+ case "attach":
+ label := str(t, op, "agent")
+ a := attached[label]
+ if a == nil {
+ a = newProbe(t, nil)
+ attached[label] = a
+ lg.mu.Lock()
+ lg.labels[a] = label
+ lg.mu.Unlock()
+ }
+ gotErr = r.Attach(fixtures.Str(t, name), a)
+ case "unload":
+ if name == nil {
+ _, gotErr = r.UnloadAll(ctx)
+ } else {
+ _, gotErr = r.Unload(ctx, fixtures.Str(t, name))
+ }
+ case "fail":
+ mu.Lock()
+ fail[fixtures.Str(t, name)] = true
+ mu.Unlock()
+ case "heal":
+ mu.Lock()
+ delete(fail, fixtures.Str(t, name))
+ mu.Unlock()
+ default:
+ t.Fatalf("unknown op %q", kind)
+ }
+ where := fmt.Sprintf("op %d (%s %v)", i, str(t, op, "op"), name)
+ if want, ok := op.Get("error"); ok {
+ w := goLifeText(fixtures.Str(t, want), types)
+ if gotErr == nil || !(gotErr.Error() == w || strings.HasSuffix(gotErr.Error(), ": "+w)) {
+ t.Fatalf("%s: error %v, want %q", where, gotErr, w)
+ }
+ } else if gotErr != nil {
+ t.Fatalf("%s: unexpected error %v", where, gotErr)
+ }
+ if want, ok := op.Get("agent"); ok && str(t, op, "op") == "load" && gotAgent != fixtures.Str(t, want) {
+ t.Fatalf("%s: agent %q, want %q", where, gotAgent, want)
+ }
+ lg.mu.Lock()
+ events := append([]string{}, lg.events[before:]...)
+ lg.mu.Unlock()
+ if want := fixtures.Strings(t, op.Must("events")); !reflect.DeepEqual(events, append([]string{}, want...)) {
+ t.Fatalf("%s: events %q, want %q", where, events, want)
+ }
+ wmu.Lock()
+ gotWarns := append([]string{}, warns...)
+ wmu.Unlock()
+ var wantWarns []string
+ for _, w := range fixtures.Strings(t, op.Must("warnings")) {
+ wantWarns = append(wantWarns, goLifeText(w, types))
+ }
+ if !reflect.DeepEqual(append([]string{}, gotWarns...), append([]string{}, wantWarns...)) {
+ t.Fatalf("%s: warnings %q, want %q", where, gotWarns, wantWarns)
+ }
+ if got, want := names(r.Loaded()), fixtures.Strings(t, op.Must("loaded")); !reflect.DeepEqual(append([]string{}, got...), append([]string{}, want...)) {
+ t.Fatalf("%s: loaded %v, want %v", where, got, want)
+ }
+ if got, want := r.MaxLoaded(), fixtures.Int(t, op.Must("max_loaded")); got != want {
+ t.Fatalf("%s: max_loaded %d, want %d", where, got, want)
+ }
+ }
+ return len(steps)
+}
diff --git a/laya-go/router/lifecycle_helpers_test.go b/laya-go/router/lifecycle_helpers_test.go
new file mode 100644
index 00000000..76859725
--- /dev/null
+++ b/laya-go/router/lifecycle_helpers_test.go
@@ -0,0 +1,187 @@
+package router_test
+
+import (
+ "context"
+ "errors"
+ "strconv"
+ "sync"
+ "sync/atomic"
+ "testing"
+ "testing/fstest"
+ "time"
+
+ "github.com/NandhaKishorM/laya/laya-go/agent"
+ "github.com/NandhaKishorM/laya/laya-go/infer"
+ "github.com/NandhaKishorM/laya/laya-go/internal/fixtures"
+ "github.com/NandhaKishorM/laya/laya-go/router"
+)
+
+// probeSession runs nothing and counts its Close calls.
+type probeSession struct {
+ closes atomic.Int32
+ closeErr error
+ // hold, when set, blocks Close until it is closed.
+ hold chan struct{}
+ // inClose is closed when Close is entered, if set.
+ inClose chan struct{}
+ once sync.Once
+}
+
+func (s *probeSession) Run(context.Context, infer.Batch) (infer.Output, error) {
+ return infer.Output{}, errors.New("probe session runs nothing")
+}
+
+func (s *probeSession) Close() error {
+ s.closes.Add(1)
+ if s.inClose != nil {
+ s.once.Do(func() { close(s.inClose) })
+ }
+ if s.hold != nil {
+ <-s.hold
+ }
+ return s.closeErr
+}
+
+func toyCheckpoint() fstest.MapFS {
+ spec, cfg := fixtures.ToyTokenizer()
+ return fstest.MapFS{
+ "rl_agent_config.json": {Data: []byte(`{}`)},
+ "tokenizer/tokenizer.json": {Data: spec},
+ "tokenizer/tokenizer_config.json": {Data: cfg},
+ }
+}
+
+func newProbe(tb testing.TB, s *probeSession) *agent.Agent {
+ tb.Helper()
+ if s == nil {
+ s = &probeSession{}
+ }
+ a, err := agent.Open(toyCheckpoint(), s)
+ if err != nil {
+ tb.Fatal(err)
+ }
+ return a
+}
+
+// built is one agent a fakeFactory made.
+type built struct {
+ label string
+ agent *agent.Agent
+ sess *probeSession
+}
+
+// fakeFactory is a deterministic AgentFactory: it counts calls, can fail or block per checkpoint,
+// and remembers every agent it made.
+type fakeFactory struct {
+ t testing.TB
+ mu sync.Mutex
+ n int
+ // fail maps a checkpoint to the error its builds return.
+ fail map[router.Checkpoint]error
+ // gate blocks a build of the checkpoint until the channel is closed; entered, if present, is
+ // sent to when a build starts.
+ gate map[router.Checkpoint]chan struct{}
+ entered chan router.Checkpoint
+ // panicOn makes a build of the checkpoint panic.
+ panicOn map[router.Checkpoint]bool
+ made []*built
+ byAgent map[*agent.Agent]*built
+}
+
+func newFactory(t testing.TB) *fakeFactory {
+ return &fakeFactory{t: t, fail: map[router.Checkpoint]error{}, gate: map[router.Checkpoint]chan struct{}{},
+ panicOn: map[router.Checkpoint]bool{}, byAgent: map[*agent.Agent]*built{}}
+}
+
+func (f *fakeFactory) build(ctx context.Context, c router.Checkpoint) (*agent.Agent, error) {
+ f.mu.Lock()
+ f.n++
+ label := "built-" + strconv.Itoa(f.n)
+ gate, entered, err, boom := f.gate[c], f.entered, f.fail[c], f.panicOn[c]
+ f.mu.Unlock()
+ if entered != nil {
+ entered <- c
+ }
+ if gate != nil {
+ select {
+ case <-gate:
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ }
+ }
+ if boom {
+ panic("factory exploded for " + string(c))
+ }
+ if err != nil {
+ return nil, err
+ }
+ s := &probeSession{}
+ a := newProbe(f.t, s)
+ b := &built{label: label, agent: a, sess: s}
+ f.mu.Lock()
+ f.made = append(f.made, b)
+ f.byAgent[a] = b
+ f.mu.Unlock()
+ return a, nil
+}
+
+func (f *fakeFactory) calls() int {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return f.n
+}
+
+func (f *fakeFactory) of(a *agent.Agent) *built {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return f.byAgent[a]
+}
+
+func (f *fakeFactory) setFail(c router.Checkpoint, err error) {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ if err == nil {
+ delete(f.fail, c)
+ } else {
+ f.fail[c] = err
+ }
+}
+
+func (f *fakeFactory) open(c router.Checkpoint) chan struct{} {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ g := make(chan struct{})
+ f.gate[c] = g
+ return g
+}
+
+func newRouter(t testing.TB, f *fakeFactory, opts ...router.Option) *router.Router {
+ t.Helper()
+ opts = append([]router.Option{router.WithAgentFactory(f.build)}, opts...)
+ r, err := router.New(opts...)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { r.Close() })
+ return r
+}
+
+// within fails the test if fn does not return in time, instead of hanging the run.
+func within(t testing.TB, d time.Duration, what string, fn func()) {
+ t.Helper()
+ done := make(chan struct{})
+ go func() { defer close(done); fn() }()
+ select {
+ case <-done:
+ case <-time.After(d):
+ t.Fatalf("%s did not finish within %v", what, d)
+ }
+}
+
+func names(cs []router.Checkpoint) []string {
+ out := make([]string, len(cs))
+ for i, c := range cs {
+ out[i] = string(c)
+ }
+ return out
+}
diff --git a/laya-go/router/lifecycle_test.go b/laya-go/router/lifecycle_test.go
new file mode 100644
index 00000000..9abbcc2c
--- /dev/null
+++ b/laya-go/router/lifecycle_test.go
@@ -0,0 +1,991 @@
+package router_test
+
+import (
+ "context"
+ "errors"
+ "math/rand"
+ "reflect"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/NandhaKishorM/laya/laya-go/agent"
+ "github.com/NandhaKishorM/laya/laya-go/router"
+ "github.com/NandhaKishorM/laya/laya-go/schema"
+)
+
+const (
+ en = router.English
+ ml = router.Multilingual
+ td = router.TypedDecisions
+)
+
+func loadedIs(t *testing.T, r *router.Router, want ...router.Checkpoint) {
+ t.Helper()
+ got := r.Loaded()
+ if len(got) == 0 && len(want) == 0 {
+ return
+ }
+ if !reflect.DeepEqual(got, want) {
+ t.Fatalf("loaded = %v, want %v", got, want)
+ }
+}
+
+func mustLoad(t *testing.T, r *router.Router, name string) *agent.Agent {
+ t.Helper()
+ a, err := r.Load(context.Background(), name)
+ if err != nil {
+ t.Fatalf("Load(%q): %v", name, err)
+ }
+ return a
+}
+
+func TestLoadIsCached(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ a, b := mustLoad(t, r, "english"), mustLoad(t, r, "en")
+ if a != b || f.calls() != 1 {
+ t.Fatalf("same agent %v, factory calls %d; want one build", a == b, f.calls())
+ }
+}
+
+func TestLoadedOrderIsLRU(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f, router.WithMaxLoaded(3))
+ for _, n := range []string{"english", "multilingual", "typed-decisions", "english"} {
+ mustLoad(t, r, n)
+ }
+ loadedIs(t, r, ml, td, en)
+}
+
+func TestEvictionClosesTheVictim(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f, router.WithMaxLoaded(1))
+ a := mustLoad(t, r, "english")
+ b := mustLoad(t, r, "multilingual")
+ loadedIs(t, r, ml)
+ if got := f.of(a).sess.closes.Load(); got != 1 {
+ t.Fatalf("evicted agent closed %d times, want 1", got)
+ }
+ if got := f.of(b).sess.closes.Load(); got != 0 {
+ t.Fatalf("resident agent closed %d times", got)
+ }
+}
+
+func TestEvictionOrderIsLeastRecentlyUsed(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ a := mustLoad(t, r, "english")
+ mustLoad(t, r, "multilingual")
+ mustLoad(t, r, "english") // touch: multilingual is now the oldest
+ mustLoad(t, r, "typed-decisions")
+ loadedIs(t, r, en, td)
+ if f.of(a).sess.closes.Load() != 0 {
+ t.Fatal("the recently used agent was closed")
+ }
+}
+
+func TestAttachedAgentsAreNotClosed(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f, router.WithMaxLoaded(1))
+ s := &probeSession{}
+ if err := r.Attach("english", newProbe(t, s)); err != nil {
+ t.Fatal(err)
+ }
+ mustLoad(t, r, "multilingual") // evicts nothing: Attach raised the limit to 1, and this makes 2
+ if _, err := r.Unload(context.Background(), "english"); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := r.UnloadAll(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if s.closes.Load() != 0 {
+ t.Fatal("the Router closed an agent it did not build")
+ }
+}
+
+func TestAttachRaisesMaxLoaded(t *testing.T) {
+ r := newRouter(t, newFactory(t))
+ for i, n := range []string{"english", "multilingual", "typed-decisions"} {
+ if err := r.Attach(n, newProbe(t, nil)); err != nil {
+ t.Fatal(err)
+ }
+ if got := r.MaxLoaded(); got != max(2, i+1) {
+ t.Fatalf("after %d attaches MaxLoaded = %d", i+1, got)
+ }
+ }
+ loadedIs(t, r, en, ml, td)
+}
+
+func TestAttachErrors(t *testing.T) {
+ r := newRouter(t, newFactory(t))
+ var unknown *router.UnknownModelError
+ if err := r.Attach("gpt", newProbe(t, nil)); !errors.As(err, &unknown) {
+ t.Fatalf("unknown name: %v", err)
+ }
+ if err := r.Attach("english", nil); err == nil {
+ t.Fatal("nil agent accepted")
+ }
+}
+
+func TestPreloadFitsEverythingAndStages(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f, router.WithMaxLoaded(1))
+ if err := r.Preload(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ loadedIs(t, r, en, ml, td)
+ if r.MaxLoaded() != 3 || f.calls() != 3 {
+ t.Fatalf("max %d calls %d", r.MaxLoaded(), f.calls())
+ }
+ r2 := newRouter(t, newFactory(t))
+ for _, n := range []string{"english", "multilingual", "typed-decisions"} {
+ if err := r2.Preload(context.Background(), n); err != nil {
+ t.Fatal(err)
+ }
+ }
+ loadedIs(t, r2, en, ml, td)
+}
+
+func TestPreloadResidentIsNotTouched(t *testing.T) {
+ r := newRouter(t, newFactory(t))
+ mustLoad(t, r, "english")
+ mustLoad(t, r, "multilingual")
+ if err := r.Preload(context.Background(), "english"); err != nil {
+ t.Fatal(err)
+ }
+ loadedIs(t, r, en, ml)
+}
+
+func TestPreloadUnknownNameChangesNothing(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ if err := r.Preload(context.Background(), "english", "gpt"); err == nil || f.calls() != 0 || r.MaxLoaded() != 2 {
+ t.Fatalf("err %v calls %d max %d", err, f.calls(), r.MaxLoaded())
+ }
+}
+
+func TestUnloadFrees(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ a := mustLoad(t, r, "english")
+ mustLoad(t, r, "multilingual")
+ freed, err := r.Unload(context.Background(), "en")
+ if err != nil || !reflect.DeepEqual(freed, []router.Checkpoint{en}) {
+ t.Fatalf("freed %v err %v", freed, err)
+ }
+ if f.of(a).sess.closes.Load() != 1 {
+ t.Fatal("unloaded agent not closed")
+ }
+ if freed, _ := r.Unload(context.Background(), "en"); len(freed) != 0 {
+ t.Fatalf("second unload freed %v", freed)
+ }
+ all, err := r.UnloadAll(context.Background())
+ if err != nil || !reflect.DeepEqual(all, []router.Checkpoint{ml}) {
+ t.Fatalf("UnloadAll = %v, %v", all, err)
+ }
+ loadedIs(t, r)
+}
+
+func TestLeaseOutlivesEviction(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f, router.WithMaxLoaded(1))
+ l1, err := r.Lease(context.Background(), "english")
+ if err != nil {
+ t.Fatal(err)
+ }
+ l2, err := r.Lease(context.Background(), "english")
+ if err != nil {
+ t.Fatal(err)
+ }
+ mustLoad(t, r, "multilingual") // evicts english while it is leased twice
+ loadedIs(t, r, ml)
+ sess := f.of(l1.Agent()).sess
+ if sess.closes.Load() != 0 {
+ t.Fatal("evicted agent closed under a lease")
+ }
+ if err := l1.Close(); err != nil || sess.closes.Load() != 0 {
+ t.Fatalf("closed with a lease outstanding: %v closes=%d", err, sess.closes.Load())
+ }
+ if err := l2.Close(); err != nil {
+ t.Fatal(err)
+ }
+ if sess.closes.Load() != 1 {
+ t.Fatalf("closes after the last release = %d", sess.closes.Load())
+ }
+ if l1.Agent() != nil || l1.Close() != nil {
+ t.Fatal("a closed lease must give no agent and close quietly")
+ }
+}
+
+func TestLeaseOnResidentIsReleasedWithoutClosing(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ l, err := r.Lease(context.Background(), "ml")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if l.Checkpoint() != ml {
+ t.Fatalf("checkpoint %v", l.Checkpoint())
+ }
+ sess := f.of(l.Agent()).sess
+ l.Close()
+ if sess.closes.Load() != 0 {
+ t.Fatal("a resident agent closed on release")
+ }
+}
+
+func TestUnloadWithLeaseClosesAtRelease(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ l, _ := r.Lease(context.Background(), "english")
+ sess := f.of(l.Agent()).sess
+ if _, err := r.Unload(context.Background(), "english"); err != nil {
+ t.Fatal(err)
+ }
+ if sess.closes.Load() != 0 {
+ t.Fatal("closed under the lease")
+ }
+ l.Close()
+ if sess.closes.Load() != 1 {
+ t.Fatal("not closed at release")
+ }
+}
+
+func TestClosingALeaseTwiceUnderARaceCountsOnce(t *testing.T) {
+ for i := 0; i < 50; i++ {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ l1, _ := r.Lease(context.Background(), "english")
+ l2, _ := r.Lease(context.Background(), "english")
+ sess := f.of(l1.Agent()).sess
+ if _, err := r.Unload(context.Background(), "english"); err != nil {
+ t.Fatal(err)
+ }
+ var wg sync.WaitGroup
+ for j := 0; j < 4; j++ {
+ wg.Add(1)
+ go func() { defer wg.Done(); l1.Close() }()
+ }
+ wg.Wait()
+ if sess.closes.Load() != 0 {
+ t.Fatal("one lease released several times closed the agent under the other")
+ }
+ l2.Close()
+ if sess.closes.Load() != 1 {
+ t.Fatal("not closed after the last release")
+ }
+ }
+}
+
+func TestConcurrentLoadsBuildOnce(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 64)
+ gate := f.open(en)
+ r := newRouter(t, f)
+ const n = 24
+ got := make([]*agent.Agent, n)
+ var wg sync.WaitGroup
+ for i := 0; i < n; i++ {
+ wg.Add(1)
+ go func(i int) {
+ defer wg.Done()
+ a, err := r.Load(context.Background(), "english")
+ if err != nil {
+ t.Error(err)
+ }
+ got[i] = a
+ }(i)
+ }
+ <-f.entered // the one build has started; the others are waiting on it
+ time.Sleep(50 * time.Millisecond)
+ close(gate)
+ wg.Wait()
+ for _, a := range got {
+ if a != got[0] || a == nil {
+ t.Fatal("callers did not share one agent")
+ }
+ }
+ if f.calls() != 1 {
+ t.Fatalf("factory called %d times", f.calls())
+ }
+}
+
+func TestResidentLoadDoesNotWaitForAColdBuild(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 4)
+ r := newRouter(t, f)
+ mustLoad(t, r, "english")
+ <-f.entered
+ gate := f.open(ml)
+ done := make(chan error, 1)
+ go func() { _, err := r.Load(context.Background(), "multilingual"); done <- err }()
+ <-f.entered // the cold build is inside the factory
+ within(t, 2*time.Second, "a resident Load behind a cold build", func() { mustLoad(t, r, "english") })
+ within(t, 2*time.Second, "Loaded behind a cold build", func() { r.Loaded() })
+ close(gate)
+ if err := <-done; err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestColdBuildsAreSerialised(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 4)
+ gate := f.open(en)
+ r := newRouter(t, f)
+ var wg sync.WaitGroup
+ wg.Add(2)
+ go func() { defer wg.Done(); r.Load(context.Background(), "english") }()
+ <-f.entered
+ go func() { defer wg.Done(); r.Load(context.Background(), "multilingual") }()
+ select {
+ case c := <-f.entered:
+ t.Fatalf("%s entered the factory while another build was in it", c)
+ case <-time.After(100 * time.Millisecond):
+ }
+ close(gate)
+ <-f.entered
+ wg.Wait()
+}
+
+var errBoom = errors.New("boom building")
+
+func TestFailedBuildIsNotCachedAndWakesWaiters(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 64)
+ gate := f.open(en)
+ f.setFail(en, errBoom)
+ r := newRouter(t, f)
+ const n = 8
+ errs := make([]error, n)
+ var wg sync.WaitGroup
+ for i := 0; i < n; i++ {
+ wg.Add(1)
+ go func(i int) { defer wg.Done(); _, errs[i] = r.Load(context.Background(), "english") }(i)
+ }
+ <-f.entered
+ time.Sleep(50 * time.Millisecond)
+ close(gate)
+ within(t, 5*time.Second, "waiters on a failed build", wg.Wait)
+ for i, err := range errs {
+ if !errors.Is(err, errBoom) {
+ t.Fatalf("caller %d: %v", i, err)
+ }
+ }
+ loadedIs(t, r)
+ if f.calls() != 1 {
+ t.Fatalf("waiters rebuilt: %d factory calls", f.calls())
+ }
+ f.setFail(en, nil)
+ f.mu.Lock()
+ delete(f.gate, en)
+ f.mu.Unlock()
+ mustLoad(t, r, "english") // not wedged, not cached
+ if f.calls() != 2 {
+ t.Fatalf("factory calls %d", f.calls())
+ }
+ loadedIs(t, r, en)
+}
+
+func TestFailedBuildEvictsNothing(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f, router.WithMaxLoaded(1))
+ mustLoad(t, r, "english")
+ f.setFail(ml, errBoom)
+ if _, err := r.Load(context.Background(), "multilingual"); !errors.Is(err, errBoom) {
+ t.Fatal(err)
+ }
+ loadedIs(t, r, en)
+}
+
+func TestPanickingFactoryWakesWaiters(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 8)
+ gate := f.open(en)
+ f.panicOn[en] = true
+ r := newRouter(t, f)
+ waiter := make(chan error, 1)
+ builder := make(chan any, 1)
+ go func() {
+ defer func() { builder <- recover() }()
+ r.Load(context.Background(), "english")
+ }()
+ <-f.entered
+ go func() { _, err := r.Load(context.Background(), "english"); waiter <- err }()
+ time.Sleep(50 * time.Millisecond)
+ close(gate)
+ select {
+ case err := <-waiter:
+ if err == nil {
+ t.Fatal("waiter got no error")
+ }
+ case <-time.After(5 * time.Second):
+ t.Fatal("waiter hung on a panicking build")
+ }
+ if <-builder == nil {
+ t.Fatal("the panic was swallowed")
+ }
+ f.panicOn[en] = false
+ mustLoad(t, r, "english")
+}
+
+func TestWaiterDoesNotInheritTheBuildersCancellation(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 8)
+ f.open(en) // never opened: the build ends only when its context does
+ r := newRouter(t, f)
+ bctx, cancel := context.WithCancel(context.Background())
+ berr := make(chan error, 1)
+ go func() { _, err := r.Load(bctx, "english"); berr <- err }()
+ <-f.entered
+ werr := make(chan error, 1)
+ go func() { _, err := r.Load(context.Background(), "english"); werr <- err }()
+ time.Sleep(50 * time.Millisecond)
+ f.mu.Lock()
+ delete(f.gate, en)
+ f.mu.Unlock()
+ cancel()
+ if err := <-berr; !errors.Is(err, context.Canceled) {
+ t.Fatalf("builder: %v", err)
+ }
+ select {
+ case err := <-werr:
+ if err != nil {
+ t.Fatalf("waiter inherited the cancellation: %v", err)
+ }
+ case <-time.After(5 * time.Second):
+ t.Fatal("waiter hung")
+ }
+}
+
+func TestWaiterHonoursItsOwnContext(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 8)
+ gate := f.open(en)
+ r := newRouter(t, f)
+ go r.Load(context.Background(), "english")
+ <-f.entered
+ ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
+ defer cancel()
+ if _, err := r.Load(ctx, "english"); !errors.Is(err, context.DeadlineExceeded) {
+ t.Fatalf("got %v", err)
+ }
+ close(gate)
+}
+
+func TestNoFactory(t *testing.T) {
+ r, err := router.New()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := r.Load(context.Background(), "english"); err == nil {
+ t.Fatal("a Router with no factory loaded")
+ }
+ if _, err := r.Route("hello", schema.Questions{}); err != nil {
+ t.Fatalf("routing must not need a factory: %v", err)
+ }
+ loadedIs(t, r)
+}
+
+func TestFactoryReturningNothing(t *testing.T) {
+ r, _ := router.New(router.WithAgentFactory(func(context.Context, router.Checkpoint) (*agent.Agent, error) { return nil, nil }))
+ if _, err := r.Load(context.Background(), "english"); err == nil {
+ t.Fatal("a nil agent was accepted")
+ }
+ loadedIs(t, r)
+}
+
+func TestMaxLoadedMustBePositive(t *testing.T) {
+ for _, n := range []int{0, -1} {
+ if _, err := router.New(router.WithMaxLoaded(n)); err == nil {
+ t.Fatalf("WithMaxLoaded(%d) accepted", n)
+ }
+ }
+ r, _ := router.New()
+ if r.MaxLoaded() != 2 {
+ t.Fatalf("default %d", r.MaxLoaded())
+ }
+}
+
+func TestAttachReplacesAndRetiresTheBuiltAgent(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ l, _ := r.Lease(context.Background(), "english")
+ sess := f.of(l.Agent()).sess
+ attached := newProbe(t, nil)
+ if err := r.Attach("english", attached); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustLoad(t, r, "english"); got != attached {
+ t.Fatal("Load did not return the attached agent")
+ }
+ if sess.closes.Load() != 0 {
+ t.Fatal("replaced agent closed under a lease")
+ }
+ l.Close()
+ if sess.closes.Load() != 1 {
+ t.Fatal("replaced agent never closed")
+ }
+ // and one nobody leased closes at once
+ r2 := newRouter(t, f)
+ a := mustLoad(t, r2, "english")
+ if err := r2.Attach("english", newProbe(t, nil)); err != nil {
+ t.Fatal(err)
+ }
+ if f.of(a).sess.closes.Load() != 1 {
+ t.Fatal("unleased replaced agent not closed")
+ }
+}
+
+func TestAttachingTheSameInstanceIsSafe(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ a := mustLoad(t, r, "english")
+ if err := r.Attach("english", a); err != nil {
+ t.Fatal(err)
+ }
+ if f.of(a).sess.closes.Load() != 0 {
+ t.Fatal("re-attaching the loaded agent closed it")
+ }
+ if got := mustLoad(t, r, "english"); got != a {
+ t.Fatal("agent changed")
+ }
+ // ownership moved to the caller: the Router no longer closes it
+ r.Unload(context.Background(), "english")
+ if f.of(a).sess.closes.Load() != 0 {
+ t.Fatal("the Router closed an agent that was handed back to the caller")
+ }
+}
+
+func TestAttachWhileBuildingKeepsTheAttachedAgent(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 4)
+ gate := f.open(en)
+ r := newRouter(t, f)
+ attached := newProbe(t, nil)
+ res := make(chan *agent.Agent, 1)
+ go func() { a, _ := r.Load(context.Background(), "english"); res <- a }()
+ <-f.entered
+ if err := r.Attach("english", attached); err != nil {
+ t.Fatal(err)
+ }
+ close(gate)
+ if got := <-res; got != attached {
+ t.Fatal("the build overwrote the attached agent")
+ }
+ f.mu.Lock()
+ dup := f.made[0]
+ f.mu.Unlock()
+ if dup.sess.closes.Load() != 1 {
+ t.Fatalf("the duplicate build closed %d times", dup.sess.closes.Load())
+ }
+}
+
+func TestCloseFreesAndSeals(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f)
+ a := mustLoad(t, r, "english")
+ s := &probeSession{}
+ r.Attach("multilingual", newProbe(t, s))
+ if err := r.Close(); err != nil {
+ t.Fatal(err)
+ }
+ if f.of(a).sess.closes.Load() != 1 || s.closes.Load() != 0 {
+ t.Fatal("Close must close what it built and leave what was attached")
+ }
+ loadedIs(t, r)
+ ctx := context.Background()
+ if _, err := r.Load(ctx, "english"); !errors.Is(err, router.ErrClosed) {
+ t.Fatalf("Load after Close: %v", err)
+ }
+ if _, err := r.Lease(ctx, "english"); !errors.Is(err, router.ErrClosed) {
+ t.Fatalf("Lease after Close: %v", err)
+ }
+ if err := r.Preload(ctx); !errors.Is(err, router.ErrClosed) {
+ t.Fatalf("Preload after Close: %v", err)
+ }
+ if err := r.Attach("english", newProbe(t, nil)); !errors.Is(err, router.ErrClosed) {
+ t.Fatalf("Attach after Close: %v", err)
+ }
+ if err := r.Close(); err != nil {
+ t.Fatalf("second Close: %v", err)
+ }
+}
+
+func TestCloseWaitsForAnInFlightBuild(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 4)
+ gate := f.open(en)
+ r := newRouter(t, f)
+ type res struct {
+ l *router.Lease
+ err error
+ }
+ got := make(chan res, 1)
+ go func() { l, err := r.Lease(context.Background(), "english"); got <- res{l, err} }()
+ <-f.entered
+ closed := make(chan error, 1)
+ go func() { closed <- r.Close() }()
+ select {
+ case <-closed:
+ t.Fatal("Close returned while a build was in flight")
+ case <-time.After(100 * time.Millisecond):
+ }
+ close(gate)
+ if err := <-closed; err != nil {
+ t.Fatal(err)
+ }
+ g := <-got
+ if g.err != nil {
+ t.Fatalf("the building caller must get its lease: %v", g.err)
+ }
+ loadedIs(t, r)
+ sess := f.of(g.l.Agent()).sess
+ if sess.closes.Load() != 0 {
+ t.Fatal("agent closed under the lease Close raced")
+ }
+ g.l.Close()
+ if sess.closes.Load() != 1 {
+ t.Fatal("the agent built during Close leaked")
+ }
+}
+
+func TestCloseWithAWaiterOnTheBuild(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 4)
+ gate := f.open(en)
+ r := newRouter(t, f)
+ go func() {
+ if l, err := r.Lease(context.Background(), "english"); err == nil {
+ l.Close()
+ }
+ }()
+ <-f.entered
+ werr := make(chan error, 1)
+ go func() { _, err := r.Load(context.Background(), "english"); werr <- err }()
+ time.Sleep(50 * time.Millisecond)
+ go r.Close()
+ time.Sleep(50 * time.Millisecond)
+ close(gate)
+ select {
+ case err := <-werr:
+ if err != nil && !errors.Is(err, router.ErrClosed) {
+ t.Fatalf("waiter: %v", err)
+ }
+ case <-time.After(5 * time.Second):
+ t.Fatal("waiter hung across Close")
+ }
+}
+
+func TestUnloadWaitsForItsOwnBuildOnly(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 4)
+ gate := f.open(en)
+ r := newRouter(t, f)
+ go r.Load(context.Background(), "english")
+ <-f.entered
+ within(t, 2*time.Second, "Unload of another checkpoint behind a build", func() {
+ r.Unload(context.Background(), "multilingual")
+ })
+ done := make(chan []router.Checkpoint, 1)
+ go func() { freed, _ := r.Unload(context.Background(), "english"); done <- freed }()
+ select {
+ case <-done:
+ t.Fatal("Unload returned while its checkpoint was building")
+ case <-time.After(100 * time.Millisecond):
+ }
+ close(gate)
+ if freed := <-done; len(freed) != 1 {
+ t.Fatalf("freed %v", freed)
+ }
+ loadedIs(t, r)
+}
+
+func TestUnloadAllWaitsForBuildsAndHonoursContext(t *testing.T) {
+ f := newFactory(t)
+ f.entered = make(chan router.Checkpoint, 4)
+ gate := f.open(en)
+ r := newRouter(t, f)
+ go r.Load(context.Background(), "english")
+ <-f.entered
+ ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
+ defer cancel()
+ if _, err := r.UnloadAll(ctx); !errors.Is(err, context.DeadlineExceeded) {
+ t.Fatalf("got %v", err)
+ }
+ if _, err := r.Unload(ctx, "english"); !errors.Is(err, context.DeadlineExceeded) {
+ t.Fatalf("got %v", err)
+ }
+ close(gate)
+}
+
+func TestASlowCloseHoldsNoLock(t *testing.T) {
+ f := newFactory(t)
+ slow := &probeSession{hold: make(chan struct{}), inClose: make(chan struct{})}
+ r := newRouter(t, f, router.WithMaxLoaded(1), router.WithAgentFactory(func(ctx context.Context, c router.Checkpoint) (*agent.Agent, error) {
+ if c == en {
+ return newProbe(t, slow), nil
+ }
+ return f.build(ctx, c)
+ }))
+ mustLoad(t, r, "english")
+ go r.Load(context.Background(), "multilingual") // evicts english; its Close blocks on slow.hold
+ <-slow.inClose
+ within(t, 2*time.Second, "Loaded during a slow close", func() { r.Loaded() })
+ within(t, 2*time.Second, "MaxLoaded during a slow close", func() { r.MaxLoaded() })
+ within(t, 2*time.Second, "Unload during a slow close", func() { r.Unload(context.Background(), "typed-decisions") })
+ close(slow.hold)
+}
+
+func TestCloseErrorFailsTheLoadButNotTheEvents(t *testing.T) {
+ f := newFactory(t)
+ bad := errors.New("close failed")
+ var events []string
+ var mu sync.Mutex
+ h := &router.HookFuncs{
+ OnLoadFunc: func(c *router.LoadContext) { mu.Lock(); events = append(events, "load:"+string(c.Model)); mu.Unlock() },
+ OnEvictFunc: func(c *router.EvictContext) {
+ mu.Lock()
+ events = append(events, "evict:"+string(c.Model))
+ mu.Unlock()
+ },
+ }
+ r := newRouter(t, f, router.WithMaxLoaded(1), router.WithInstalledHooks(h),
+ router.WithAgentFactory(func(ctx context.Context, c router.Checkpoint) (*agent.Agent, error) {
+ if c == en {
+ return newProbe(t, &probeSession{closeErr: bad}), nil
+ }
+ return f.build(ctx, c)
+ }))
+ mustLoad(t, r, "english")
+ l, err := r.Lease(context.Background(), "multilingual")
+ if !errors.Is(err, bad) || l != nil {
+ t.Fatalf("lease %v err %v", l, err)
+ }
+ loadedIs(t, r, ml)
+ mu.Lock()
+ got := append([]string(nil), events...)
+ mu.Unlock()
+ if want := []string{"load:english", "evict:english", "load:multilingual"}; !reflect.DeepEqual(got, want) {
+ t.Fatalf("events %v, want %v", got, want)
+ }
+ // the lease the failed call took was given back: eviction closes the new agent at once
+ b := f.made[0]
+ r.Unload(context.Background(), "multilingual")
+ if b.sess.closes.Load() != 1 {
+ t.Fatal("the failed call stranded its lease")
+ }
+}
+
+// ---- hooks
+
+func TestHooksRunOutsideTheLock(t *testing.T) {
+ f := newFactory(t)
+ var seen [][]router.Checkpoint
+ var mu sync.Mutex
+ var r *router.Router
+ h := &router.HookFuncs{
+ OnLoadFunc: func(c *router.LoadContext) {
+ mu.Lock()
+ seen = append(seen, c.Router.Loaded())
+ mu.Unlock()
+ c.Router.MaxLoaded()
+ if c.Model == en {
+ c.Router.Load(context.Background(), "multilingual") // re-enters the Router from a hook
+ }
+ },
+ OnEvictFunc: func(c *router.EvictContext) { c.Router.Loaded() },
+ }
+ r = newRouter(t, f, router.WithInstalledHooks(h), router.WithMaxLoaded(1))
+ within(t, 5*time.Second, "loads with hooks that call back in", func() {
+ mustLoad(t, r, "english")
+ mustLoad(t, r, "typed-decisions")
+ r.UnloadAll(context.Background())
+ })
+ mu.Lock()
+ defer mu.Unlock()
+ if len(seen) == 0 || len(seen[0]) == 0 {
+ t.Fatalf("on_load saw %v: the checkpoint was not resident yet", seen)
+ }
+}
+
+func TestLifecycleHookContext(t *testing.T) {
+ f := newFactory(t)
+ var load *router.LoadContext
+ var evict *router.EvictContext
+ h := &router.HookFuncs{
+ OnLoadFunc: func(c *router.LoadContext) { load = c },
+ OnEvictFunc: func(c *router.EvictContext) { evict = c },
+ }
+ r := newRouter(t, f, router.WithInstalledHooks(h), router.WithMaxLoaded(1))
+ a := mustLoad(t, r, "english")
+ if load == nil || load.Model != en || load.Agent != a || load.Router != r || len(load.RunID) != 32 {
+ t.Fatalf("load context %+v", load)
+ }
+ mustLoad(t, r, "multilingual")
+ if evict == nil || evict.Model != en || evict.Router != r || len(evict.RunID) != 32 || evict.RunID == load.RunID {
+ t.Fatalf("evict context %+v", evict)
+ }
+}
+
+func TestResidentLoadFiresNoHooks(t *testing.T) {
+ f := newFactory(t)
+ n := 0
+ h := &router.HookFuncs{OnLoadFunc: func(*router.LoadContext) { n++ }}
+ r := newRouter(t, f, router.WithInstalledHooks(h))
+ mustLoad(t, r, "english")
+ mustLoad(t, r, "english")
+ r.Lease(context.Background(), "english")
+ if n != 1 {
+ t.Fatalf("on_load fired %d times", n)
+ }
+}
+
+func TestLeaseHeldAcrossTheLoadHook(t *testing.T) {
+ f := newFactory(t)
+ h := &router.HookFuncs{OnLoadFunc: func(c *router.LoadContext) {
+ c.Router.Unload(context.Background(), string(c.Model)) // evicts the agent under the hook
+ }}
+ r := newRouter(t, f, router.WithInstalledHooks(h))
+ l, err := r.Lease(context.Background(), "english")
+ if err != nil {
+ t.Fatal(err)
+ }
+ sess := f.of(l.Agent()).sess
+ if sess.closes.Load() != 0 {
+ t.Fatal("the agent was closed under the lease while on_load ran")
+ }
+ l.Close()
+ if sess.closes.Load() != 1 {
+ t.Fatal("not closed at release")
+ }
+}
+
+func TestFailingHookKeepsTheCheckpointAndGivesBackTheLease(t *testing.T) {
+ f := newFactory(t)
+ fail := &router.HookFuncs{OnLoadFunc: func(*router.LoadContext) { panic(errors.New("hook says no")) }}
+ r := newRouter(t, f, router.WithMaxLoaded(1), router.WithInstalledHooks(fail))
+ l, err := r.Lease(context.Background(), "english")
+ if err == nil || l != nil {
+ t.Fatalf("lease %v err %v", l, err)
+ }
+ loadedIs(t, r, en)
+ r.RemoveHook(fail)
+ mustLoad(t, r, "multilingual") // evicts english
+ if got := f.made[0].sess.closes.Load(); got != 1 {
+ t.Fatalf("the failed Lease stranded its lease: closes=%d", got)
+ }
+ if a, err := r.Load(context.Background(), "multilingual"); err != nil || a == nil {
+ t.Fatal(err)
+ }
+}
+
+func TestSerialHooksDoNotOverlapAcrossLoads(t *testing.T) {
+ f := newFactory(t)
+ var in, peak int32
+ var mu sync.Mutex
+ h := &router.HookFuncs{OnLoadFunc: func(*router.LoadContext) {
+ mu.Lock()
+ in++
+ if in > peak {
+ peak = in
+ }
+ mu.Unlock()
+ time.Sleep(20 * time.Millisecond)
+ mu.Lock()
+ in--
+ mu.Unlock()
+ }}
+ r := newRouter(t, f, router.WithSerialHooks(), router.WithInstalledHooks(h), router.WithMaxLoaded(3))
+ var wg sync.WaitGroup
+ for _, n := range []string{"english", "multilingual", "typed-decisions"} {
+ wg.Add(1)
+ go func() { defer wg.Done(); r.Load(context.Background(), n) }()
+ }
+ wg.Wait()
+ if peak != 1 {
+ t.Fatalf("serial hooks overlapped: peak %d", peak)
+ }
+}
+
+func TestHookValueWithoutAnyEventIsRefused(t *testing.T) {
+ type nothing struct{}
+ if _, err := router.New(router.WithInstalledHooks(&nothing{})); err == nil {
+ t.Fatal("a hook with no event accepted")
+ }
+ if _, err := router.New(router.WithInstalledHooks(&router.HookFuncs{})); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// ---- stress
+
+func TestConcurrentStress(t *testing.T) {
+ f := newFactory(t)
+ r := newRouter(t, f, router.WithMaxLoaded(2))
+ attached := map[*agent.Agent]*probeSession{}
+ var amu sync.Mutex
+ var wg sync.WaitGroup
+ check := func(a *agent.Agent) {
+ if b := f.of(a); b != nil && b.sess.closes.Load() != 0 {
+ t.Error("an agent under lease was closed")
+ }
+ }
+ for g := 0; g < 8; g++ {
+ wg.Add(1)
+ go func(seed int64) {
+ defer wg.Done()
+ rng := rand.New(rand.NewSource(seed))
+ names := []string{"english", "multilingual", "typed-decisions", "en", "ml"}
+ ctx := context.Background()
+ for i := 0; i < 150; i++ {
+ n := names[rng.Intn(len(names))]
+ switch rng.Intn(8) {
+ case 0, 1, 2, 3:
+ l, err := r.Lease(ctx, n)
+ if err != nil {
+ if !errors.Is(err, router.ErrClosed) {
+ t.Error(err)
+ }
+ continue
+ }
+ check(l.Agent())
+ time.Sleep(time.Duration(rng.Intn(200)) * time.Microsecond)
+ check(l.Agent())
+ l.Close()
+ case 4:
+ r.Load(ctx, n)
+ case 5:
+ r.Unload(ctx, n)
+ case 6:
+ s := &probeSession{}
+ a := newProbe(t, s)
+ amu.Lock()
+ attached[a] = s
+ amu.Unlock()
+ r.Attach(n, a)
+ default:
+ r.Preload(ctx, n)
+ r.Loaded()
+ }
+ }
+ }(int64(g))
+ }
+ wg.Wait()
+ if err := r.Close(); err != nil {
+ t.Fatal(err)
+ }
+ for _, b := range f.made {
+ if got := b.sess.closes.Load(); got != 1 {
+ t.Errorf("%s closed %d times, want exactly 1", b.label, got)
+ }
+ }
+ amu.Lock()
+ for a, s := range attached {
+ if s.closes.Load() != 0 {
+ t.Errorf("attached agent %p was closed by the Router", a)
+ }
+ }
+ amu.Unlock()
+}
diff --git a/laya-go/router/router.go b/laya-go/router/router.go
index d2c0faa3..dd757546 100644
--- a/laya-go/router/router.go
+++ b/laya-go/router/router.go
@@ -51,6 +51,7 @@ type Router struct {
auto bool
hint LangHint
hk *hookState
+ lc lifecycle
}
// Option configures New.
@@ -63,6 +64,8 @@ type config struct {
auto bool
hint LangHint
sources []source
+ factory AgentFactory
+ maxLoaded int
}
type source struct {
@@ -116,7 +119,7 @@ func WithSourceIn(name, repo, subfolder string) Option {
// must not call Route on its own Router. A default or source name that is neither a checkpoint nor an alias is an
// *UnknownModelError. Sources are applied in the order given, the last one for a name winning.
func New(opts ...Option) (*Router, error) {
- c := config{def: string(Multilingual), hooks: hookConfig{raise: true}}
+ c := config{def: string(Multilingual), hooks: hookConfig{raise: true}, maxLoaded: DefaultMaxLoaded}
for _, o := range opts {
if err := o(&c); err != nil {
return nil, err
@@ -145,6 +148,7 @@ func New(opts ...Option) (*Router, error) {
return nil, err
}
r.def = def
+ r.lc.init(c.factory, c.maxLoaded)
return r, nil
}