Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fit: a CSV in, a calibration measured on rows the heads never saw, a floor on rows
Nandakishor9d3fcc8924ef
diff --git a/.gitignore b/.gitignoreindex 3affe912..fc5edb5f 100644--- a/.gitignore+++ b/.gitignore@@ -92,3 +92,4 @@ research/*_benchmark_results.json# local checkpoint weights, kept out of the tree# (fetch them with: python verify/checkpoints.py --fetch)models/+fit-routing/diff --git a/README.md b/README.mdindex 13607b42..628ade78 100644--- a/README.md+++ b/README.md@@ -106,16 +106,85 @@ agent.predict(video="call.mp4", questions=questions)```-**Fit and calibrate on your own labels.** Seconds, no gradient step, and the route line is placed for you:+**Fit and calibrate on your own labels.** Seconds, no gradient step, and the route is decided for you. Labels are a CSV with a `state` column and one column per question id; [`examples/data/routing_labels.csv`](https://github.com/NandhaKishorM/laya/blob/main/examples/data/routing_labels.csv) is a 60-row example that routes messages to billing, HR, PR, sales or technical and rates their urgency:++```csv+state,team,urgency+"We were charged twice for the March invoice, please refund the duplicate.",billing,2+"I would like to apply for the open backend engineer role. Where do I send my CV?",hr,0+"A journalist from the Economic Times is asking for a comment on today's outage.",pr,2+```++The whole loop, from import to the answer, is [`examples/42_fit_and_calibrate_vegaml.py`](https://github.com/NandhaKishorM/laya/blob/main/examples/42_fit_and_calibrate_vegaml.py). It holds out 4 rows per team (20), fits on the other 40, and measures the same 20 rows before and after. Condensed:```python-rows = [{"state": "...", "questions": questions, "gold": {"department": "billing"}}, ...]-report = agent.fit(rows) # heads per question, then temperature, conformal threshold and review band-print(report["serving"]) # the questions whose head beat the engine on your rows-agent.save_calibration("fit/") # reload later with agent.load_calibration("fit/")+import csv, laya++questions = {+ "team": {"type": "choice", "instructions": "Which team should handle this message?",+ "criteria": {"billing": "invoices, payments, refunds", "hr": "jobs, payroll, leave, policy",+ "pr": "press, media, public statements", "sales": "pricing, demos, quotes, trials",+ "technical": "bugs, outages, API and product questions"}},+ "urgency": {"type": "score", "instructions": "How soon does this need a reply?",+ "criteria": ["can wait", "this week", "today"]},+}+rows = list(csv.DictReader(open("examples/data/routing_labels.csv")))+held, fit_rows = rows[-20:], rows[:-20] # the script holds out 4 per team; same idea++agent = laya.Agent(vegaml=True)+print(laya.VegaAgent.min_rows(questions)) # 35: the fewest rows fit() accepts for these questions++before = agent.predict(held[0]["state"], questions)["answers"]["team"] # the cold engine++report = agent.fit([{"state": r["state"], "gold": {"team": r["team"], "urgency": r["urgency"]}}+ for r in fit_rows], questions=questions) # or agent.fit("examples/data/routing_labels.csv", questions=questions)+print(report["n_fit"], report["n_calibrate"]) # 30 fit the heads, 10 are held out inside fit() to calibrate on+print(report["serving"]) # the questions whose head beat the engine on the same rows+print(report["calibration"]["band"]) # the review line per question, or None where it could not hold++after = agent.predict(held[0]["state"], questions)["answers"]["team"]+agent.save_calibration("fit-routing/") # heads + calibration; agent.load_calibration("fit-routing/") later```-A head is kept only if it beats chance under cross-validation and served only if it also beats the engine on the same rows. The band's line sits where precision holds at 0.95 on your rows with a 95% Wilson bound to spare; `agent.thresholds()` prints what decides `route` right now.+What the script printed on a laptop GPU, every number measured in that run:++```+minimum rows fit() accepts for these questions: 35 (5 per option of the widest question for the heads, plus a quarter held out to calibrate on, never fewer than 10)++== before: the cold engine, 20 held-out rows+ team accuracy 0.600 ECE 0.221 mean confidence 0.629+ urgency accuracy 0.400 ECE 0.412 MAE 0.65 levels+ team set sizes {1: 13, 2: 7} abstain 12 route {'auto': 6, 'review': 14}++== fit + calibrate on 40 rows in 9s: 30 fit the heads, 10 held out to calibrate on+ team head cv 0.767 vs engine 0.633 on the same rows -> SERVED+ urgency head cv 0.400 vs engine 0.400 on the same rows -> left to the engine+ team temperature 0.83 conformal q-hat 0.987 ECE on the calibration rows 0.281 -> 0.241+ urgency temperature 6.44 conformal q-hat 0.741 ECE on the calibration rows 0.515 -> 0.142+ review line per question: {'team': None, 'urgency': None}+ ['team', 'urgency']: no line holds precision 0.95 with a 95% bound on 10 calibration rows, so the conformal set routes those answers++== after: heads + calibration, 20 held-out rows+ team accuracy 0.900 ECE 0.206 mean confidence 0.749+ urgency accuracy 0.400 ECE 0.059 MAE 0.65 levels+ team set sizes {2: 2, 3: 2, 4: 6, 5: 10} abstain 0 route {'review': 20}++== change on held-out rows: team accuracy +0.300, team ECE -0.015, urgency MAE +0.00++== one held-out message: "Why was a late fee added to this month's bill when we paid on the due " (gold: billing)+ before: {"choice": "billing", "probabilities": {"billing": 0.999995, "hr": 3e-06, "pr": 0.0, "sales": 1e-06, "technical": 1e-06}, "answer_confidence": 0.999995, "prediction_set": ["billing"], "abstain": false, "unbound": false, "readout": "engine", "route": "auto", "route_reason": "conformal singleton, not abstaining, in distribution"}+ after: {"choice": "billing", "probabilities": {"billing": 0.674496, "hr": 0.186172, "pr": 0.068743, "sales": 0.041369, "technical": 0.029221}, "answer_confidence": 0.674496, "prediction_set": ["billing", "hr", "pr", "sales", "technical"], "abstain": false, "unbound": null, "readout": "ttt", "route": "review", "route_reason": "band could not hold precision 0.95 on 10 calibration rows, so the conformal rule decides: conformal set at alpha 0.1 holds 5 options"}++== ablation: fitting rows per team, same 20 held-out rows+ per team rows team acc team ECE urg. acc urg. MAE route+ cold 0 0.600 0.221 0.400 0.65 {'auto': 6, 'review': 14}+ 7 35 0.800 0.111 0.400 0.65 {'auto': 14, 'review': 6}+ 8 40 0.900 0.206 0.400 0.65 {'review': 20}+```++Read it honestly. The head took routing from 0.600 to 0.900 on rows it never saw, and was refused for urgency because it did not beat the engine on the same rows, so that question stayed exactly where it was. The urgency ECE fell from 0.412 to 0.059 from a temperature alone, with the argmax untouched. Routing is where 10 calibration rows are too few: at alpha 0.1 the conformal quantile on 10 rows is the largest score, so the sets are wide, and no review line can certify 0.95 precision with a 95% bound on fewer than about 50 rows; the output says so in `route_reason` instead of pretending. The floor of 35 rows gets you a served head; a route you can act on wants a few hundred rows, and `report["calibration"]["band"]` tells you when you are there.++How this differs from fine-tuning: a fine-tune runs a gradient through the model until it agrees with your labels, which changes every answer the model gives, invalidates any calibration fitted before, and leaves you a new checkpoint to validate. `fit` never touches the reader. It fits a small logistic head per question over the frozen features, a convex problem with one solution, keeps the head only if it beats chance under cross-validation and serves it only if it also beats the zero-shot engine on the same rows, so a fit can lose and say so, as urgency did above. A quarter of the rows is held out inside `fit` and `calibrate` refits three numbers per question on them: a temperature, a conformal threshold so the prediction set covers at 90%, and the review line, placed where precision holds at 0.95 with a 95% Wilson bound to spare. Every other question is untouched by construction, nothing is forgotten, and `agent.thresholds()` prints what decides `route` right now.### The laya checkpoints, without vegaml@@ -175,6 +244,18 @@ The shipped checkpoints work zero-shot, but fine-tuning on decisions from your owith the frame times or the transcript. Decoding goes through the `ffmpeg` binary on yourPATH; laya takes on no Python dependency for it, and `laya.video` holds the helpers.`laya.vega.Router` sends a clip to vega like an image or a recording.+* **`fit` takes a CSV and refuses to calibrate on its own fitting rows.** `agent.fit("rows.csv",+ questions=questions)` reads a `state` column plus one column per question id+ (`laya.VegaAgent.load_rows`; `.jsonl` too). A quarter of the rows is now held out inside `fit`+ and the temperature, conformal threshold and review line are measured there, because a head is+ right on its own rows and calibrating on them put every confidence at 1.000 and the review line+ out of reach. `laya.VegaAgent.min_rows(questions)` is the floor `fit` accepts, and below it the+ error names the number. Where the calibration rows cannot certify the review line at+ `target_precision`, the band is reported `None` for that question and the conformal set routes+ its answers, with `route_reason` saying so, instead of every answer going to review.+ [`examples/42_fit_and_calibrate_vegaml.py`](https://github.com/NandhaKishorM/laya/blob/main/examples/42_fit_and_calibrate_vegaml.py)+ and [`examples/data/routing_labels.csv`](https://github.com/NandhaKishorM/laya/blob/main/examples/data/routing_labels.csv)+ are the measured loop; the quickstart carries its output.## What's new in 0.4.23diff --git a/docs/vegaml.md b/docs/vegaml.mdindex df887c0c..99a247a5 100644--- a/docs/vegaml.md+++ b/docs/vegaml.md@@ -50,7 +50,20 @@ agent.load_calibration("fit/") # on a fresh VegaAgent, identical answers an```Rows are the same `{"state", "questions", "gold" or "expected"}` shape laya's fine-tune reads,-so a corpus prepared for one fits the other unchanged. A head is kept only if it beats chance+so a corpus prepared for one fits the other unchanged. A path to a CSV works too: a `state`+column and one column per question id holding the gold label, as in+[`examples/data/routing_labels.csv`](https://github.com/NandhaKishorM/laya/blob/main/examples/data/routing_labels.csv),+60 messages routed to billing, HR, PR, sales or technical with an urgency level:+`agent.fit("examples/data/routing_labels.csv", questions=questions)`. The fewest rows `fit`+accepts is `laya.VegaAgent.min_rows(questions)`: five per option of the widest question for the+heads, plus a quarter held out inside `fit` to calibrate on, never fewer than ten, so 35 for a+five-way choice and 20 for a yes/no question. A quarter held out is the rule because a head is+right on its own rows: calibrating there rails the temperature to its floor and puts every+confidence at 1.000. When the calibration rows cannot certify the review line at+`target_precision`, the band is reported `None` for that question and the conformal set routes+its answers, with `route_reason` saying so. The full measured loop, before and after on held-out+rows with an ablation over the number of fitting rows, is `examples/42_fit_and_calibrate_vegaml.py`+and its output is in the README. A head is kept only if it beats chanceunder cross-validation and served only if it also beats the engine on the same rows:`report["heads"][qid]["used_by_auto"]` is the fact to read, not `fitted`.diff --git a/examples/42_fit_and_calibrate_vegaml.py b/examples/42_fit_and_calibrate_vegaml.pynew file mode 100644index 00000000..89ac195c--- /dev/null+++ b/examples/42_fit_and_calibrate_vegaml.py@@ -0,0 +1,161 @@+#!/usr/bin/env python3+"""Fit and calibrate on your own labels, end to end: the cold engine, then the fitted one,+on the same held-out rows, with every number measured here rather than typed.++ python examples/42_fit_and_calibrate_vegaml.py++Reads examples/data/routing_labels.csv (60 messages routed to billing, HR, PR, sales or+technical, each with an urgency level), holds out four rows per team, fits heads and the+calibration on the other forty, and prints: accuracy and ECE before and after, the fit report+(which heads earned serving, where the calibration was measured), the conformal set sizes and+abstentions, the auto/review split, one full answer before and after so the shape is visible,+and an ablation over how many fitting rows the same held-out set gets, from the minimum laya+accepts upward.++Needs laya 0.4.24 or newer (vegaml ships with it) and a few minutes on a laptop GPU, most of+it reading the fitting rows through the frozen reader.+"""+import csv+import json+import os+import sys+import time+from collections import Counter++import laya++HERE = os.path.dirname(os.path.abspath(__file__))+CSV = os.path.join(HERE, "data", "routing_labels.csv")++QUESTIONS = {+ "team": {"type": "choice", "instructions": "Which team should handle this message?",+ "criteria": {"billing": "invoices, payments, refunds",+ "hr": "jobs, payroll, leave, policy",+ "pr": "press, media, public statements",+ "sales": "pricing, demos, quotes, trials",+ "technical": "bugs, outages, API and product questions"}},+ "urgency": {"type": "score", "instructions": "How soon does this need a reply?",+ "criteria": ["can wait", "this week", "today"]},+}+HOLD_OUT_PER_TEAM = 4+++def ece(pairs, bins=10):+ """Expected calibration error of (confidence, was_correct) pairs, equal-width bins."""+ total = 0.0+ for b in range(bins):+ lo, hi = b / bins, (b + 1) / bins+ inside = [(c, ok) for c, ok in pairs if lo <= c < hi or (b == bins - 1 and c == 1.0)]+ if inside:+ conf = sum(c for c, _ in inside) / len(inside)+ acc = sum(ok for _, ok in inside) / len(inside)+ total += len(inside) / len(pairs) * abs(acc - conf)+ return total+++def measure(agent, rows):+ team_pairs, urg_pairs, set_sizes, abstains, routes, mae = [], [], [], 0, Counter(), 0.0+ for r in rows:+ out = agent.predict(r["state"], QUESTIONS)+ t, u = out["answers"]["team"], out["answers"]["urgency"]+ team_pairs.append((t["answer_confidence"], t["choice"] == r["team"]))+ urg_pairs.append((u["answer_confidence"], u["score"] == int(r["urgency"])))+ mae += abs(u["score"] - int(r["urgency"]))+ set_sizes.append(len(t["prediction_set"]))+ abstains += int(bool(t["abstain"]))+ routes[t["route"]] += 1+ n = len(rows)+ return {"n": n,+ "team_acc": sum(ok for _, ok in team_pairs) / n, "team_ece": ece(team_pairs),+ "team_conf": sum(c for c, _ in team_pairs) / n,+ "urg_acc": sum(ok for _, ok in urg_pairs) / n, "urg_ece": ece(urg_pairs), "urg_mae": mae / n,+ "set_sizes": dict(sorted(Counter(set_sizes).items())), "abstain": abstains,+ "routes": dict(routes)}+++def show(label, m):+ print("\n== %s, %d held-out rows" % (label, m["n"]))+ print(" team accuracy %.3f ECE %.3f mean confidence %.3f" % (m["team_acc"], m["team_ece"], m["team_conf"]))+ print(" urgency accuracy %.3f ECE %.3f MAE %.2f levels" % (m["urg_acc"], m["urg_ece"], m["urg_mae"]))+ print(" team set sizes %s abstain %d route %s" % (m["set_sizes"], m["abstain"], m["routes"]))+++def fit_on(agent, fit_rows):+ """Fresh heads and calibration from these rows; a quarter is held out inside fit()."""+ agent.vega.heads.clear()+ agent.calibration = None+ return agent.fit([{"state": r["state"], "gold": {"team": r["team"], "urgency": r["urgency"]}}+ for r in fit_rows], questions=QUESTIONS)+++def main():+ rows = list(csv.DictReader(open(CSV, encoding="utf-8")))+ by_team = {}+ for r in rows:+ by_team.setdefault(r["team"], []).append(r)+ held = [r for rs in by_team.values() for r in rs[-HOLD_OUT_PER_TEAM:]]+ fit_rows = [r for rs in by_team.values() for r in rs[:-HOLD_OUT_PER_TEAM]]+ print("%d rows: %d to fit, %d held out (%d per team)" % (len(rows), len(fit_rows), len(held), HOLD_OUT_PER_TEAM))+ print("minimum rows fit() accepts for these questions: %d (5 per option of the widest question "+ "for the heads, plus a quarter held out to calibrate on, never fewer than 10)"+ % laya.VegaAgent.min_rows(QUESTIONS))++ t0 = time.time()+ agent = laya.Agent(vegaml=True)+ print("loaded %s in %.0fs" % (agent, time.time() - t0))++ probe = held[0]+ before_answer = agent.predict(probe["state"], QUESTIONS)["answers"]["team"]+ before = measure(agent, held)+ show("before: the cold engine", before)++ t0 = time.time()+ report = fit_on(agent, fit_rows)+ print("\n== fit + calibrate on %d rows in %.0fs: %d fit the heads, %d held out to calibrate on"+ % (report["n_rows"], time.time() - t0, report["n_fit"], report["n_calibrate"]))+ for q, h in report["heads"].items():+ print(" %-8s head cv %.3f vs engine %.3f on the same rows -> %s"+ % (q, h["cv_accuracy"] or 0, h["engine_accuracy"] or 0,+ "SERVED" if h["used_by_auto"] else "left to the engine"))+ cal = report["calibration"]+ for q, c in cal["questions"].items():+ print(" %-8s temperature %.2f conformal q-hat %.3f ECE on the calibration rows %.3f -> %.3f"+ % (q, c["temperature"], c["qhat"], c["ece_before"], c["ece_after"]))+ print(" review line per question: %s" % cal["band"])+ if cal["band_unreachable"]:+ print(" %s: no line holds precision %.2f with a 95%% bound on %d calibration rows, so the "+ "conformal set routes those answers" % (cal["band_unreachable"], cal["target_precision"], cal["n_rows"]))++ after = measure(agent, held)+ show("after: heads + calibration", after)+ print("\n== change on held-out rows: team accuracy %+.3f, team ECE %+.3f, urgency MAE %+.2f"+ % (after["team_acc"] - before["team_acc"], after["team_ece"] - before["team_ece"],+ after["urg_mae"] - before["urg_mae"]))++ after_answer = agent.predict(probe["state"], QUESTIONS)["answers"]["team"]+ keep = ("choice", "probabilities", "answer_confidence", "prediction_set", "abstain", "unbound",+ "readout", "route", "route_reason")+ print("\n== one held-out message: %r (gold: %s)" % (probe["state"][:70], probe["team"]))+ for label, ans in (("before", before_answer), ("after", after_answer)):+ print(" %s: %s" % (label, json.dumps({k: ans[k] for k in keep if k in ans}, ensure_ascii=False)))++ agent.save_calibration(os.path.join(HERE, "..", "fit-routing"))+ print("\nsaved heads + calibration to fit-routing/; reload with agent.load_calibration(\"fit-routing/\")")++ # Ablation: the same 20 held-out rows, fewer fitting rows. 7 per team (35) is the floor+ # fit() accepts for these questions; the run above used 8 (40).+ print("\n== ablation: fitting rows per team, same %d held-out rows" % len(held))+ print(" %-9s %-5s %-9s %-9s %-9s %-9s %s" % ("per team", "rows", "team acc", "team ECE", "urg. acc", "urg. MAE", "route"))+ print(" %-9s %-5s %-9.3f %-9.3f %-9.3f %-9.2f %s" % ("cold", "0", before["team_acc"], before["team_ece"],+ before["urg_acc"], before["urg_mae"], before["routes"]))+ for per_team in (7, 8):+ subset = [r for rs in by_team.values() for r in rs[:per_team]]+ fit_on(agent, subset)+ m = measure(agent, held)+ print(" %-9d %-5d %-9.3f %-9.3f %-9.3f %-9.2f %s" % (per_team, len(subset), m["team_acc"], m["team_ece"],+ m["urg_acc"], m["urg_mae"], m["routes"]))+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/examples/README.md b/examples/README.mdindex f5796353..a3539d01 100644--- a/examples/README.md+++ b/examples/README.md@@ -28,6 +28,11 @@ python examples/01_first_call_minimal.py # any Python with torch + trans---+`data/routing_labels.csv` is a 60-row labelled set (five teams, three urgency levels) for+`laya.Agent(vegaml=True).fit(...)`; the main README's quickstart uses it.++---+## Stage 1 — first contact (01-03)| | what you'll learn |diff --git a/examples/data/routing_labels.csv b/examples/data/routing_labels.csvnew file mode 100644index 00000000..d2f2982e--- /dev/null+++ b/examples/data/routing_labels.csv@@ -0,0 +1,61 @@+state,team,urgency+"We were charged twice for the March invoice, please refund the duplicate.",billing,2+Can you send me a copy of last quarter's invoices for our records?,billing,0+The card on file expired and the renewal failed; how do I update payment details?,billing,1+Our invoice shows 12 seats but we only have 9 users. Please correct it.,billing,1+Is GST included in the amount on the invoice or added on top?,billing,0+Payment went through but the account still says overdue and access is blocked.,billing,2+Please change the billing address on our account to the new office.,billing,0+We want to move from monthly to annual billing. What is the process?,billing,0+I would like to apply for the open backend engineer role. Where do I send my CV?,hr,0+My payslip for October is missing the overtime hours I logged.,hr,1+How many leave days do I have left this year?,hr,0+I need to report a harassment incident involving a colleague. Who do I speak to?,hr,2+Can I get an employment verification letter for my visa application by Friday?,hr,1+What is the policy on working from another country for two weeks?,hr,0+My health insurance card has not arrived and I have a hospital visit tomorrow.,hr,2+I am resigning; please tell me the notice period and the exit steps.,hr,1+A journalist from the Economic Times is asking for a comment on today's outage.,pr,2+Could we get a quote from your CEO for our article on AI in banking?,pr,1+We would like to invite your founder to speak at our conference in January.,pr,0+Our podcast wants to feature your team. Who handles media requests?,pr,0+There is a viral post claiming your product leaks data. Do you have a statement?,pr,2+Can we use your logo in a partner announcement going out next week?,pr,1+Requesting a press kit and high-resolution photos of the leadership team.,pr,0+A TV channel wants a live interview about the funding round this evening.,pr,2+We are a 200-person company evaluating your product. Can we get a demo?,sales,1+What does the enterprise plan cost for 500 seats with SSO?,sales,1+Do you offer discounts for non-profits or universities?,sales,0+Our trial ends tomorrow and we want to buy before it lapses.,sales,2+Can you send a formal quote and the terms so procurement can review them?,sales,1+We need on-premises deployment. Is that available and at what price?,sales,1+Is there a reseller or partner programme we could join?,sales,0+Please extend our trial by a week; the decision meeting slipped.,sales,1+The API returns 500 errors on every request since this morning.,technical,2+How do I rotate an API key without downtime?,technical,1+Exports to CSV come out with the wrong encoding for Hindi text.,technical,1+"Is there a Python SDK, and does it support async calls?",technical,0+Webhooks stopped arriving an hour ago and our queue is backing up.,technical,2+The dashboard chart shows last week's numbers instead of today's.,technical,1+Which TLS versions does the endpoint accept?,technical,0+Login works on desktop but the mobile app says invalid token.,technical,1+Why was a late fee added to this month's bill when we paid on the due date?,billing,1+Please split the invoice between our two cost centres.,billing,0+The refund you promised three weeks ago has still not reached our account.,billing,2+Can we pay by bank transfer instead of card?,billing,0+"What documents do I need for my first day, and when does onboarding start?",hr,0+I was not paid for the public holiday last week.,hr,1+A new hire needs a laptop and accounts before Monday; who approves that?,hr,1+Can I switch to a four-day week under the flexible working policy?,hr,0+A blogger has published our internal roadmap and is asking for comment.,pr,2+We would like your founder on a panel about open-source AI next month.,pr,0+Can you confirm the headcount figure before our article goes to print at six?,pr,2+Requesting permission to film inside your office for a documentary.,pr,0+How does your pricing compare to Jev for a team of fifty?,sales,1+We need a signed NDA before we can share our evaluation data.,sales,1+Our board approved the purchase; send the contract for signature today.,sales,2+Do you have a startup plan for companies under ten people?,sales,0+Rate limits are hitting at 20 requests a second; the docs say 100.,technical,1+Production is down: every login returns a 502 from your gateway.,technical,2+Is the Python package compatible with Python 3.13?,technical,0+Batch predictions return answers in a different order from the inputs.,technical,1diff --git a/laya/vega.py b/laya/vega.pyindex 4eae0367..6e13a7ea 100644--- a/laya/vega.py+++ b/laya/vega.py@@ -349,7 +349,7 @@ class VegaAgent(Agent):answers = {}for qid, q in questions.items():a = _fold(raw["answers"][qid], q)- a.update(self._route(raw["answers"][qid], a, min_confidence))+ a.update(self._route(dict(raw["answers"][qid], _qid=qid), a, min_confidence))answers[qid] = aout = {"answers": answers,@@ -466,7 +466,12 @@ class VegaAgent(Agent):# ------------------------------------------------------------------ the routedef _route(self, raw_answer: Mapping, folded: Mapping, min_confidence: Optional[float]) -> Dict:"""``route`` and why, decided without a user-set number. See the module docstring."""- if "route" in raw_answer and raw_answer["route"] in ("auto", "review"):+ band_q = None+ cal = getattr(self, "calibration", None)+ if cal is not None and getattr(cal, "band", None) is not None:+ band_q = cal.band.questions.get(raw_answer.get("_qid", ""))+ unreachable = bool(band_q) and band_q.get("reachable") is False+ if "route" in raw_answer and raw_answer["route"] in ("auto", "review") and not unreachable:route, why = raw_answer["route"], "band: " + str(raw_answer.get("route_reason", ""))else:reasons = []@@ -483,6 +488,13 @@ class VegaAgent(Agent):route = "review" if reasons else "auto"why = "; ".join(reasons) if reasons else ("conformal singleton, not abstaining, in distribution")+ if unreachable:+ # The line was asked to hold target_precision with a 95% lower bound and the+ # calibration rows could not certify it (12 for 12 certifies nothing). Sending+ # every answer to review would make the fitted path useless, so the conformal+ # rule decides and the reason says what the band could not do.+ why = ("band could not hold precision %s on %s calibration rows, so the conformal "+ "rule decides: %s" % (band_q.get("target_precision"), band_q.get("examples"), why))if min_confidence is not None and route == "auto":c = folded.get("answer_confidence")if c is None or c < float(min_confidence):@@ -507,13 +519,49 @@ class VegaAgent(Agent):return out# ------------------------------------------------------------------ fitting- def _examples(self, rows: Sequence[Mapping], questions: Optional[Mapping]) -> Tuple[list, Dict]:+ @staticmethod+ def load_rows(path: Any, questions: Optional[Mapping] = None) -> List[Dict[str, Any]]:+ """Labelled rows from a file: ``.csv`` or ``.jsonl``.++ A CSV has a ``state`` column and one column per question id holding that row's gold+ label (an option id for ``choice``, a level index for ``score``, ``yes``/``no`` or+ ``true``/``false`` for ``noul``); an empty cell means the row carries no label for that+ question. ``examples/data/routing_labels.csv`` in the repository is the shape. JSONL+ rows are the ``{"state", "questions"?, "gold"}`` dicts :meth:`fit` takes directly.+ """+ import csv++ path = os.fspath(path)+ if path.lower().endswith(".jsonl"):+ with open(path, encoding="utf-8") as f:+ return [json.loads(line) for line in f if line.strip()]+ with open(path, encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ if not reader.fieldnames or "state" not in reader.fieldnames:+ raise ValueError("%s needs a `state` column; the other columns are question ids" % path)+ label_cols = [c for c in reader.fieldnames if c != "state"]+ if questions:+ unknown = [c for c in label_cols if c not in questions]+ if unknown:+ raise ValueError("column(s) %s in %s match no question id in %s"+ % (unknown, path, sorted(questions)))+ rows = []+ for rec in reader:+ gold = {c: rec[c].strip() for c in label_cols if (rec.get(c) or "").strip()}+ if rec.get("state", "").strip() and gold:+ rows.append({"state": rec["state"], "gold": gold})+ return rows++ def _examples(self, rows: Any, questions: Optional[Mapping]) -> Tuple[list, Dict]:"""laya training rows as vegaml fit examples: ``(state, {qid: label}, {qid: question})``.Rows are ``{"state", "questions"?, "gold"|"expected"}``: the shape ``items_from_rows``reads, so a corpus prepared for laya's fine-tune fits vega's heads unchanged. A row- without its own ``questions`` uses ``questions``.+ without its own ``questions`` uses ``questions``. A path to a ``.csv`` or ``.jsonl``+ is read with :meth:`load_rows` first."""+ if isinstance(rows, (str, os.PathLike)):+ rows = self.load_rows(rows, questions)examples, sheet = [], {}for r in rows:qs = r.get("questions") or questions@@ -529,9 +577,63 @@ class VegaAgent(Agent):examples.append((r["state"], labels, vq))return examples, sheet- def fit(self, rows: Sequence[Mapping], questions: Optional[Mapping] = None,+ #: Rows kept back from the heads so the calibration is measured on rows they never saw.+ CALIBRATION_SHARE = 0.25+ #: What vegaml needs to say anything: ten rows to calibrate, and for the heads at least+ #: ``folds`` rows of every option so stratified cross-validation can score each one.+ MIN_CALIBRATION_ROWS = 10++ @classmethod+ def min_rows(cls, questions: Mapping, folds: int = 5, calibrate: bool = True) -> int:+ """The fewest labelled rows :meth:`fit` accepts for these questions.++ Heads need ``folds`` rows per option (the widest question decides) and never fewer+ than ``2 * folds``; with ``calibrate`` another quarter is held out for the+ calibration, which needs ten rows of its own. A five-way choice therefore needs 35,+ a yes/no question 24. These are floors: the example in the README fits twelve rows+ per option, and more rows move every number in the right direction.+ """+ widest = max((len(_labels_for(q)) for q in questions.values()), default=2)+ heads = max(2 * folds, folds * widest)+ if not calibrate:+ return heads+ cal = max(cls.MIN_CALIBRATION_ROWS, math.ceil(heads * cls.CALIBRATION_SHARE / (1 - cls.CALIBRATION_SHARE)))+ return heads + cal++ @classmethod+ def _split(cls, examples: Sequence[Tuple], share: float) -> Tuple[list, list]:+ """A calibration share taken across the first question's labels in proportion, never+ fewer than ``MIN_CALIBRATION_ROWS`` in all and at least one row from every label that+ has two; deterministic, so a refit is the same split."""+ by_label: Dict[str, list] = {}+ for ex in examples:+ first = next(iter(ex[1].values()), "")+ by_label.setdefault(str(first), []).append(ex)+ target = min(len(examples) // 2, max(cls.MIN_CALIBRATION_ROWS, round(len(examples) * share)))+ take = {lab: (1 if len(m) >= 2 else 0) for lab, m in by_label.items()}+ # the rest of the target by largest remainder of each label's proportional share+ remaining = target - sum(take.values())+ order = sorted(by_label, key=lambda lab: -(len(by_label[lab]) * share - take[lab]))+ while remaining > 0:+ moved = False+ for lab in order:+ if remaining <= 0:+ break+ if take[lab] < len(by_label[lab]) // 2:+ take[lab] += 1; remaining -= 1; moved = True+ if not moved:+ break+ fit_part, cal_part = [], []+ for lab, members in by_label.items():+ k = take[lab]+ cal_part.extend(members[len(members) - k:])+ fit_part.extend(members[:len(members) - k])+ return fit_part, cal_part++ def fit(self, rows: Any, questions: Optional[Mapping] = None,folds: Optional[int] = None, calibrate: bool = True,- target_precision: Optional[float] = None) -> Dict[str, Any]:+ target_precision: Optional[float] = None,+ calibration_rows: Any = None) -> Dict[str, Any]:"""Heads on your labels, then the calibration and the band, in one call.What laya's fine-tune does with gradient steps, this does with a small head per@@ -542,21 +644,61 @@ class VegaAgent(Agent):default) the same rows then refit a temperature and a conformal threshold perquestion and place the band, which is where the automatic ``route`` comes from.Nothing is scored on a fitted row by this method; held-out evaluation is yours.++ ``rows`` may also be a path to a ``.csv`` (a ``state`` column plus one column per+ question id) or a ``.jsonl``; see :meth:`load_rows`.++ The calibration is measured on rows the heads never saw. By default a quarter of the+ rows, taken evenly across the first question's labels, is held out for it; pass+ ``calibration_rows`` to use a separate set instead and fit the heads on all of+ ``rows``. Calibrating on the fitting rows is the one thing this method refuses to do:+ a head is right on its own rows, so the temperature rails to its floor, every+ confidence becomes 1.000 and the review line cannot be placed. Fewer rows than+ :meth:`min_rows` is a ``ValueError`` naming the number."""examples, sheet = self._examples(rows, questions)if not examples:raise ValueError("no trainable rows")+ qs = questions or {}+ for ex in examples:+ qs = dict(qs, **{k: v for k, v in ex[2].items() if k not in qs}) if ex[2] else qs+ folds = folds or self.folds+ cal_examples = None+ if calibrate and calibration_rows is not None:+ cal_examples, cal_sheet = self._examples(calibration_rows, questions)+ sheet = dict(sheet, **cal_sheet)+ need = self.min_rows(qs, folds, calibrate=False)+ if len(examples) < need or len(cal_examples) < self.MIN_CALIBRATION_ROWS:+ raise ValueError("fit needs at least %d rows for the heads and %d to calibrate on for "+ "these questions; got %d and %d"+ % (need, self.MIN_CALIBRATION_ROWS, len(examples), len(cal_examples)))+ fit_examples = examples+ elif calibrate:+ need = self.min_rows(qs, folds, calibrate=True)+ if len(examples) < need:+ raise ValueError("fit needs at least %d labelled rows for these questions (%d for the "+ "heads, the rest held out to calibrate on); got %d. Pass "+ "calibrate=False to fit heads alone, or label more rows."+ % (need, self.min_rows(qs, folds, calibrate=False), len(examples)))+ fit_examples, cal_examples = self._split(examples, self.CALIBRATION_SHARE)+ else:+ need = self.min_rows(qs, folds, calibrate=False)+ if len(examples) < need:+ raise ValueError("fit needs at least %d labelled rows for these questions; got %d"+ % (need, len(examples)))+ fit_examples = exampleswith self._lock:- report = self.vega.fit(examples, sheet, folds=folds or self.folds)- self.fit_report = {"heads": report, "n_rows": len(examples),+ report = self.vega.fit(fit_examples, sheet, folds=folds)+ self.fit_report = {"heads": report, "n_rows": len(examples), "n_fit": len(fit_examples),+ "n_calibrate": len(cal_examples) if cal_examples else 0,"serving": sorted(q for q, r in report.items() if r.get("used_by_auto")),"kept": sorted(q for q, r in report.items() if r.get("fitted"))}if calibrate:self.fit_report["calibration"] = self.calibrate(- rows, questions, target_precision=target_precision, _examples=(examples, sheet))+ None, questions, target_precision=target_precision, _examples=(cal_examples, sheet))return self.fit_report- def calibrate(self, rows: Sequence[Mapping], questions: Optional[Mapping] = None,+ def calibrate(self, rows: Any, questions: Optional[Mapping] = None,target_precision: Optional[float] = None, alpha: Optional[float] = None,_examples=None) -> Dict[str, Any]:"""Refit the probabilities and place the band on your labelled rows.@@ -580,8 +722,12 @@ class VegaAgent(Agent):for q, r in self.calibration.report.items()}band = self.calibration.bandreturn {"questions": rep, "target_precision": tp, "alpha": a,+ "n_rows": len(examples),"band": None if band is None else- {q: round(float(v["threshold"]), 4) for q, v in band.questions.items()}}+ {q: (round(float(v["threshold"]), 4) if v.get("reachable", True) else None)+ for q, v in band.questions.items()},+ "band_unreachable": [] if band is None else+ sorted(q for q, v in band.questions.items() if v.get("reachable") is False)}fit_temperatures = calibratediff --git a/tests/test_vega.py b/tests/test_vega.pyindex bb764975..7cef8fab 100644--- a/tests/test_vega.py+++ b/tests/test_vega.py@@ -277,16 +277,16 @@ def test_batch_long_and_thresholds(agent):assert agent.thresholds()["policy"] == "conformal set + abstain + unbound"-def _synthetic_rows(n=24):+def _synthetic_rows(n=28):"""Labelled rows a head can learn: two departments, two kinds of complaint."""rows = []bills = ["Charged twice for %s, refund the duplicate." % m for m in("March", "April", "May", "June", "July", "August", "September", "October",- "November", "December", "January", "February")]+ "November", "December", "January", "February", "the first quarter", "the renewal")]bugs = ["The app crashes when I open %s." % s for s in("settings", "the inbox", "a report", "the dashboard", "search", "billing history","my profile", "notifications", "the export page", "the login screen", "the map",- "attachments")]+ "attachments", "the calendar", "the archive")]for i in range(n // 2):rows.append({"state": bills[i], "questions": {"dept": Q["dept"]}, "gold": {"dept": "billing"}})rows.append({"state": bugs[i], "questions": {"dept": Q["dept"]}, "expected": {"dept": "technical"}})@@ -303,7 +303,9 @@ def test_fit_calibrate_save_reload_roundtrip():assert a.calibration is not None and a.thresholds()["policy"] == "band"out = a.predict(rows[0]["state"], {"dept": Q["dept"]})assert out["calibrated"] is True- assert out["answers"]["dept"]["route_reason"].startswith("band:")+ # "band: ..." when the ten calibration rows could certify the line, "band could not hold ..."+ # with the conformal rule deciding when they could not; either way the band is what spoke+ assert out["answers"]["dept"]["route_reason"].startswith("band")with tempfile.TemporaryDirectory() as d:saved = a.save_calibration(d)assert saved["calibrated"] and os.path.exists(os.path.join(d, "heads.npz"))@@ -437,5 +439,108 @@ def test_video_with_speech_decides_on_the_transcript(agent, tmp_path):assert out["answers"]["dept"]["choice"] == "billing"+# ----------------------------------------------------------------------------- rows from a file+ROUTING_CSV = os.path.join(os.path.dirname(__file__), "..", "examples", "data", "routing_labels.csv")+ROUTING_Q = {+ "team": {"type": "choice", "instructions": "Which team should handle this message?",+ "criteria": {"billing": "invoices, payments, refunds", "hr": "jobs, payroll, leave, policy",+ "pr": "press, media, public statements", "sales": "pricing, demos, quotes, trials",+ "technical": "bugs, outages, API and product questions"}},+ "urgency": {"type": "score", "instructions": "How soon does this need a reply?",+ "criteria": ["can wait", "this week", "today"]},+}+++def test_the_example_csv_loads_as_fit_rows():+ rows = lv.VegaAgent.load_rows(ROUTING_CSV, ROUTING_Q)+ assert len(rows) == 60 and all(set(r) == {"state", "gold"} for r in rows)+ teams = {r["gold"]["team"] for r in rows}+ assert teams == set(ROUTING_Q["team"]["criteria"]), "every team in the sheet has rows"+ assert {r["gold"]["urgency"] for r in rows} == {"0", "1", "2"}+ # the rows become vegaml examples with the labels in the engine's own spelling+ a = object.__new__(lv.VegaAgent)+ examples, sheet = a._examples(ROUTING_CSV, ROUTING_Q)+ assert len(examples) == 60 and set(sheet) == {"team", "urgency"}+ state, labels, _ = examples[0]+ assert labels == {"team": "billing", "urgency": "2"} and "charged twice" in state+++def test_csv_columns_must_be_question_ids(tmp_path):+ p = tmp_path / "bad.csv"+ p.write_text("state,dept\nhello,billing\n", encoding="utf-8")+ with pytest.raises(ValueError, match="match no question id"):+ lv.VegaAgent.load_rows(str(p), ROUTING_Q)+ p.write_text("text,team\nhello,billing\n", encoding="utf-8")+ with pytest.raises(ValueError, match="needs a `state` column"):+ lv.VegaAgent.load_rows(str(p), ROUTING_Q)+++def test_jsonl_rows_load_as_given(tmp_path):+ p = tmp_path / "rows.jsonl"+ p.write_text('{"state": "a", "gold": {"team": "hr"}}\n\n{"state": "b", "gold": {"team": "pr"}}\n',+ encoding="utf-8")+ rows = lv.VegaAgent.load_rows(str(p))+ assert [r["gold"]["team"] for r in rows] == ["hr", "pr"]+++@needs_model+def test_fit_from_the_example_csv(agent):+ report = agent.fit(ROUTING_CSV, questions=ROUTING_Q, calibrate=True)+ assert report["n_rows"] == 60 and set(report["heads"]) == {"team", "urgency"}+ assert "calibration" in report and set(report["calibration"]["questions"]) <= {"team", "urgency"}+ out = agent.predict("Our trial ends tomorrow and we want to buy before it lapses.", ROUTING_Q)+ assert out["answers"]["team"]["choice"] in ROUTING_Q["team"]["criteria"]+ assert out["answers"]["team"]["route"] in ("auto", "review")+++# ----------------------------------------------------------------------------- fit's own rules+def test_min_rows_counts_the_widest_question_and_the_calibration_share():+ assert lv.VegaAgent.min_rows(ROUTING_Q) == 35 # 5 options x 5 folds, plus 10 to calibrate+ assert lv.VegaAgent.min_rows(ROUTING_Q, calibrate=False) == 25+ assert lv.VegaAgent.min_rows({"q": {"type": "noul", "instructions": "x"}}) == 20+++def test_split_holds_out_at_least_ten_rows_across_every_label():+ ex = [("s%d" % i, {"team": "abcde"[i % 5]}, {}) for i in range(40)]+ fit, cal = lv.VegaAgent._split(ex, lv.VegaAgent.CALIBRATION_SHARE)+ assert len(cal) == 10 and len(fit) == 30+ assert {e[1]["team"] for e in cal} == set("abcde"), "every label is on the calibration side"+ ex2 = [("s%d" % i, {"d": "ab"[i % 2]}, {}) for i in range(24)]+ fit2, cal2 = lv.VegaAgent._split(ex2, lv.VegaAgent.CALIBRATION_SHARE)+ assert len(cal2) == 10 and len(fit2) == 14, "a quarter of 24 is 6, but ten is the floor"+ assert fit2 + cal2 and lv.VegaAgent._split(ex2, 0.25) == (fit2, cal2), "deterministic"+++def test_fit_refuses_fewer_rows_than_the_minimum():+ a = object.__new__(lv.VegaAgent)+ a.folds = 5+ rows = [{"state": "s%d" % i, "gold": {"team": "abcde"[i % 5], "urgency": "1"}} for i in range(30)]+ with pytest.raises(ValueError, match="at least 35 labelled rows"):+ a.fit(rows, questions=ROUTING_Q)+ with pytest.raises(ValueError, match="at least 25 labelled rows"):+ a.fit(rows[:20], questions=ROUTING_Q, calibrate=False)+++def test_an_unreachable_band_hands_the_route_to_the_conformal_rule():+ class _Band:+ questions = {"team": {"threshold": 1.000001, "reachable": False, "examples": 10,+ "target_precision": 0.95}}++ class _Cal:+ band = _Band()++ a = object.__new__(lv.VegaAgent)+ a.alpha, a.risk, a.calibration = 0.1, "0.1", _Cal()+ raw = {"route": "review", "route_reason": "confidence 1.000 is below the 1.000 review line", "_qid": "team"}+ single = {"abstain": False, "unbound": False, "prediction_set": ["billing"], "answer_confidence": 0.9}+ r = a._route(raw, single, None)+ assert r["route"] == "auto" and r["route_reason"].startswith("band could not hold precision 0.95 on 10")+ wide = dict(single, prediction_set=["billing", "hr", "pr"])+ assert a._route(raw, wide, None)["route"] == "review"+ # a band that did hold keeps deciding+ _Band.questions["team"]["reachable"] = True+ assert a._route(raw, single, None) == {"route": "review", "route_reason": "band: confidence 1.000 is below the 1.000 review line"}++if __name__ == "__main__":sys.exit(pytest.main([__file__, "-q"]))