Skip to content

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

fix(evals): point make eval and docs at the packaged scenarios (#240)

Rufusa0f31d907f6c
diff --git a/.claude/skills/anvil/SKILL.md b/.claude/skills/anvil/SKILL.md
index 4d377867..3cde89f1 100644
--- a/.claude/skills/anvil/SKILL.md
+++ b/.claude/skills/anvil/SKILL.md
@@ -757,7 +757,7 @@ Run these in 5b and INSERT each:
(`check_name='eval-scenarios'`, FAIL if missing): if the diff adds a file under
`packages/core/openexecutive/agents/` or modifies
`prompts/domain_prompts.py`, it must also add/modify a file under
- `evals/scenarios/`. Reference the `CLAUDE.md` "Adding a New Specialist Agent"
+ `packages/core/openexecutive/evals/_scenarios/`. Reference the `CLAUDE.md` "Adding a New Specialist Agent"
checklist in the failure message.
- **Tests for new behavior** (`check_name='tests-present'`, WARN — Medium
confidence, not a hard block): if non-trivial logic changed under
diff --git a/CLAUDE.md b/CLAUDE.md
index 8853f4c0..21a15704 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -86,7 +86,7 @@ RAG context goes in the **user turn**, not the system prompt.
4. Add knowledge docs to `knowledge/your_domain/`
-5. Add `evals/scenarios/your_domain_001.yaml` and `your_domain_002.yaml`
+5. Add `packages/core/openexecutive/evals/_scenarios/your_domain_001.yaml` and `your_domain_002.yaml`
6. If the agent introduces a new pattern (new tool, new routing path, new memory contract), update `packages/core/openexecutive/architecture/architecture-facts.yaml`. Pure additions to `SPECIALIST_REGISTRY` are auto-reflected in the `agents` section without YAML edits.
@@ -209,7 +209,7 @@ pytest packages/core/tests/unit/ -v
pytest packages/core/tests/integration/ -v
# Eval suite
-cd evals && python run_evals.py --scenarios scenarios/ --output results/
+make eval # runs packages/core/openexecutive/evals/_scenarios/*.yaml, writes evals/results/
```
## PR Requirements
diff --git a/Makefile b/Makefile
index 12582121..5132547b 100644
--- a/Makefile
+++ b/Makefile
@@ -54,7 +54,7 @@ lint:
eval:
cd packages/core && uv run python ../../evals/run_evals.py \
- --scenarios ../../evals/scenarios/ \
+ --scenarios openexecutive/evals/_scenarios/ \
--output ../../evals/results/
# --env-file makes ${VAR} interpolation in docker-compose.yml read the
diff --git a/README.md b/README.md
index 80c4d04d..01227c60 100644
--- a/README.md
+++ b/README.md
@@ -461,7 +461,7 @@ data.
3. Register in `packages/core/openexecutive/orchestrator/router.py` — add to `SPECIALIST_REGISTRY` and the `specialist` enum in `SPECIALIST_TOOLS`
4. Add domain alias to `DOMAIN_ALIASES` in `packages/core/openexecutive/knowledge/retriever.py`
5. Add knowledge docs to `knowledge/builtin/your_domain/`
-6. Add at least 2 eval scenarios to `evals/scenarios/`
+6. Add at least 2 eval scenarios to `packages/core/openexecutive/evals/_scenarios/`
7. Submit a PR — CI requires all of the above
## Development
diff --git a/docs/architecture.md b/docs/architecture.md
index 53089413..8f6c63f2 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -636,7 +636,7 @@ class YourAgent(BaseAgent):
5. Add knowledge docs to `knowledge/builtin/your_domain/`
-6. Add at least 2 eval scenarios to `evals/scenarios/`
+6. Add at least 2 eval scenarios to `packages/core/openexecutive/evals/_scenarios/`
7. Submit PR — CI requires all of the above.
diff --git a/evals/run_evals.py b/evals/run_evals.py
index 6dc0692b..cad99d22 100644
--- a/evals/run_evals.py
+++ b/evals/run_evals.py
@@ -1,8 +1,8 @@
#!/usr/bin/env python3
"""Eval runner for Open Executive.
-Usage:
- python run_evals.py --scenarios scenarios/ --output results/
+Usage (or just `make eval`), from packages/core:
+ uv run python ../../evals/run_evals.py --scenarios openexecutive/evals/_scenarios/ --output ../../evals/results/
"""
from __future__ import annotations
diff --git a/packages/core/README.md b/packages/core/README.md
index d0c28e53..550cc979 100644
--- a/packages/core/README.md
+++ b/packages/core/README.md
@@ -91,7 +91,7 @@ See [docs/architecture.md](docs/architecture.md) for the full design.
1. Create `packages/core/openexecutive/agents/your_agent.py` extending `BaseAgent`
2. Add a system prompt in `packages/core/openexecutive/prompts/domain_prompts.py`
3. Register the tool in `packages/core/openexecutive/orchestrator/router.py`
-4. Add eval scenarios in `evals/scenarios/`
+4. Add eval scenarios in `packages/core/openexecutive/evals/_scenarios/`
5. Submit a PR with the agent, prompt, and at least 2 eval scenarios
## Development