AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
dx(backend): say how to regenerate the expert style-eval baseline (#14690)
Reinier van der Leer2f07fa125a74
diff --git a/autogpt_platform/backend/AGENTS.md b/autogpt_platform/backend/AGENTS.mdindex 396a3400ec..f5a30392f4 100644--- a/autogpt_platform/backend/AGENTS.md+++ b/autogpt_platform/backend/AGENTS.md@@ -130,6 +130,10 @@ Key models (defined in `schema.prisma`):Model definitions, costs, and AutoPilot routing are catalog-as-code in `backend/data/llm_registry/catalog.py` — edit the file, open a PR (catalog-only diffs may ride `hotfix/*`→`master` for incident-speed changes). The catalog is the single source: metadata and billing dicts are derived from it at import. A block-selectable model additionally needs one `LLMModel` name line in `backend/data/llm_registry/llm_models.py` (an import-time check enforces the pairing); copilot-only models need just the catalog entry. Retire a model with a catalog PR (`is_enabled: False`) plus `python -m backend.data.llm_registry.retire <slug> --replacement <slug> --yes` to migrate existing graph nodes (dry-run by default, revertable). Full reference: [Managing LLM Models](@../../docs/platform/contributing/managing-llm-models.md).+### Adding or changing a roster expert++The seeded expert roster is `ROSTER` in `backend/api/features/experts/seed.py`. Adding, renaming or removing an entry turns the expert style eval red (`backend/copilot/eval/style/`), which scores what each expert would write against its own style spec and reads the result against a stored `baseline.json`. Two artefacts have to follow the roster edit. The first is a reference fixture, `backend/copilot/eval/style/fixtures/<name>.json`, holding 27 prompts across every kind — copy a sibling and rewrite it; this costs nothing. The second is a regenerated baseline: `poetry run expert-style-eval --write-baseline`, a paid run against the live models at roughly $1-2 per expert (the twenty-four-expert run on 2026-09-18 cost $26.89, recorded as `cost_usd` in the file). It takes the whole roster and refuses `--experts`/`--kinds`, because a filtered run would store part of the set under a whole-set fingerprint. Start with `poetry run expert-style-eval --dry-run`, which names the fingerprint components that moved and makes no calls. Regenerating also moves the per-expert scored counts — turns that hit the round cap are stored as errors — so the count `runner_test.py` pins for Max needs updating to match the new file.+### Adding a new blockFollow the comprehensive [Block SDK Guide](@../../docs/platform/block-sdk-guide.md) which covers:diff --git a/autogpt_platform/backend/backend/copilot/eval/style/assembly_test.py b/autogpt_platform/backend/backend/copilot/eval/style/assembly_test.pyindex 22fe0f958d..75942a2998 100644--- a/autogpt_platform/backend/backend/copilot/eval/style/assembly_test.py+++ b/autogpt_platform/backend/backend/copilot/eval/style/assembly_test.py@@ -46,7 +46,13 @@ def test_rubric_has_three_anchors_per_dimension():def test_the_baseline_covers_the_whole_roster_and_says_what_produced_it():baseline = load_baseline()- assert {b.expert for b in baseline.experts} == {e.name for e in roster_experts()}+ assert {b.expert for b in baseline.experts} == {e.name for e in roster_experts()}, (+ "baseline.json does not cover the current roster. Regenerate it with "+ "`poetry run expert-style-eval --write-baseline`: a paid run against the "+ "live models, roughly $1-2 per expert ($26.89 for the twenty-four-expert "+ "run on 2026-09-18). Adding, renaming or removing a roster expert needs "+ "one. Procedure: 'Adding or changing a roster expert' in backend/AGENTS.md."+ )for stored in baseline.experts:assert stored.scores.n == len(stored.by_prompt) > 0assert 0 <= stored.scores.mean <= 100@@ -67,7 +73,11 @@ def test_every_expert_has_thirty_prompts_across_every_kind(expert: str):def test_fixture_set_covers_the_whole_roster():- assert {f.expert for f in load_fixtures()} == {e.name for e in roster_experts()}+ assert {f.expert for f in load_fixtures()} == {e.name for e in roster_experts()}, (+ "every roster expert needs its own fixtures/<name>.json. Copy a sibling "+ "and rewrite its 27 reference prompts; this one costs nothing. Procedure: "+ "'Adding or changing a roster expert' in backend/AGENTS.md."+ )@pytest.mark.asynciodiff --git a/autogpt_platform/backend/backend/copilot/eval/style/runner_test.py b/autogpt_platform/backend/backend/copilot/eval/style/runner_test.pyindex 3a4f25f15e..6e2e2469dd 100644--- a/autogpt_platform/backend/backend/copilot/eval/style/runner_test.py+++ b/autogpt_platform/backend/backend/copilot/eval/style/runner_test.py@@ -362,7 +362,12 @@ async def test_run_scores_every_prompt_and_reads_it_against_the_baseline(assert result.cost_usd == pytest.approx(0.054)(comparison,) = result.comparisonassert comparison.expert == "Max"- assert comparison.shared_prompts == 23, "the baseline's four unscored prompts"+ assert comparison.shared_prompts == 23, (+ "Max's stored baseline scores 23 of its 27 prompts: four turns hit the "+ "round cap and are stored as errors. Regenerating baseline.json moves this "+ "number, so read Max's by_prompt length off the new file and update it "+ "here. No extra paid run."+ )save_baseline.assert_not_called()written = json.loads(out.read_text())assert written["fingerprint"] == result.fingerprint