AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).
fix(evals): point make eval and docs at the packaged scenarios (#240)
Rufusa0f31d907f6c
diff --git a/.claude/skills/anvil/SKILL.md b/.claude/skills/anvil/SKILL.mdindex 4d377867..3cde89f1 100644--- a/.claude/skills/anvil/SKILL.md+++ b/.claude/skills/anvil/SKILL.md@@ -757,7 +757,7 @@ Run these in 5b and INSERT each:(`check_name='eval-scenarios'`, FAIL if missing): if the diff adds a file under`packages/core/openexecutive/agents/` or modifies`prompts/domain_prompts.py`, it must also add/modify a file under- `evals/scenarios/`. Reference the `CLAUDE.md` "Adding a New Specialist Agent"+ `packages/core/openexecutive/evals/_scenarios/`. Reference the `CLAUDE.md` "Adding a New Specialist Agent"checklist in the failure message.- **Tests for new behavior** (`check_name='tests-present'`, WARN — Mediumconfidence, not a hard block): if non-trivial logic changed underdiff --git a/CLAUDE.md b/CLAUDE.mdindex 8853f4c0..21a15704 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -86,7 +86,7 @@ RAG context goes in the **user turn**, not the system prompt.4. Add knowledge docs to `knowledge/your_domain/`-5. Add `evals/scenarios/your_domain_001.yaml` and `your_domain_002.yaml`+5. Add `packages/core/openexecutive/evals/_scenarios/your_domain_001.yaml` and `your_domain_002.yaml`6. If the agent introduces a new pattern (new tool, new routing path, new memory contract), update `packages/core/openexecutive/architecture/architecture-facts.yaml`. Pure additions to `SPECIALIST_REGISTRY` are auto-reflected in the `agents` section without YAML edits.@@ -209,7 +209,7 @@ pytest packages/core/tests/unit/ -vpytest packages/core/tests/integration/ -v# Eval suite-cd evals && python run_evals.py --scenarios scenarios/ --output results/+make eval # runs packages/core/openexecutive/evals/_scenarios/*.yaml, writes evals/results/```## PR Requirementsdiff --git a/Makefile b/Makefileindex 12582121..5132547b 100644--- a/Makefile+++ b/Makefile@@ -54,7 +54,7 @@ lint:eval:cd packages/core && uv run python ../../evals/run_evals.py \- --scenarios ../../evals/scenarios/ \+ --scenarios openexecutive/evals/_scenarios/ \--output ../../evals/results/# --env-file makes ${VAR} interpolation in docker-compose.yml read thediff --git a/README.md b/README.mdindex 80c4d04d..01227c60 100644--- a/README.md+++ b/README.md@@ -461,7 +461,7 @@ data.3. Register in `packages/core/openexecutive/orchestrator/router.py` — add to `SPECIALIST_REGISTRY` and the `specialist` enum in `SPECIALIST_TOOLS`4. Add domain alias to `DOMAIN_ALIASES` in `packages/core/openexecutive/knowledge/retriever.py`5. Add knowledge docs to `knowledge/builtin/your_domain/`-6. Add at least 2 eval scenarios to `evals/scenarios/`+6. Add at least 2 eval scenarios to `packages/core/openexecutive/evals/_scenarios/`7. Submit a PR — CI requires all of the above## Developmentdiff --git a/docs/architecture.md b/docs/architecture.mdindex 53089413..8f6c63f2 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -636,7 +636,7 @@ class YourAgent(BaseAgent):5. Add knowledge docs to `knowledge/builtin/your_domain/`-6. Add at least 2 eval scenarios to `evals/scenarios/`+6. Add at least 2 eval scenarios to `packages/core/openexecutive/evals/_scenarios/`7. Submit PR — CI requires all of the above.diff --git a/evals/run_evals.py b/evals/run_evals.pyindex 6dc0692b..cad99d22 100644--- a/evals/run_evals.py+++ b/evals/run_evals.py@@ -1,8 +1,8 @@#!/usr/bin/env python3"""Eval runner for Open Executive.-Usage:- python run_evals.py --scenarios scenarios/ --output results/+Usage (or just `make eval`), from packages/core:+ uv run python ../../evals/run_evals.py --scenarios openexecutive/evals/_scenarios/ --output ../../evals/results/"""from __future__ import annotationsdiff --git a/packages/core/README.md b/packages/core/README.mdindex d0c28e53..550cc979 100644--- a/packages/core/README.md+++ b/packages/core/README.md@@ -91,7 +91,7 @@ See [docs/architecture.md](docs/architecture.md) for the full design.1. Create `packages/core/openexecutive/agents/your_agent.py` extending `BaseAgent`2. Add a system prompt in `packages/core/openexecutive/prompts/domain_prompts.py`3. Register the tool in `packages/core/openexecutive/orchestrator/router.py`-4. Add eval scenarios in `evals/scenarios/`+4. Add eval scenarios in `packages/core/openexecutive/evals/_scenarios/`5. Submit a PR with the agent, prompt, and at least 2 eval scenarios## Development