Skip to content

Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.

feat(cli): crewai eval runs the crew itself, so a coding agent gets a verdict from one command (#8270)

João Moura3da49f872b15
diff --git a/docs/edge/ar/telemetry.mdx b/docs/edge/ar/telemetry.mdx
index b7e3967a0..0b0bce847 100644
--- a/docs/edge/ar/telemetry.mdx
+++ b/docs/edge/ar/telemetry.mdx
@@ -24,14 +24,14 @@ mode: "wide"
لتوفير رؤى أعمق. قد يتضمن جمع البيانات الموسع هذا معلومات شخصية إذا دمجها المستخدمون في طواقمهم أو مهامهم.
يجب على المستخدمين النظر بعناية في محتوى طواقمهم ومهامهم قبل تفعيل `share_crew`.
يمكن للمستخدمين تعطيل القياس عن بُعد في CrewAI عبر تعيين `CREWAI_DISABLE_TELEMETRY` إلى `true` أو `1` أو `yes` أو `on` (بغض النظر عن حالة الأحرف). `OTEL_SDK_DISABLED` بنفس القيم يعطّل أيضاً مُصدِّر CrewAI. مجموعة أدوات OpenTelemetry نفسها ما تزال تقبل `true` فقط لتعطيل بقية أدوات القياس في العملية.
-تتبع AMP مشمول بشكل منفصل في [التتبع](/ar/observability/tracing).
+تتبع AMP مشمول بشكل منفصل في [التتبع](/ar/observability/tracing). لا يُوقف `OTEL_SDK_DISABLED` تتبع AMP: عيّن `CREWAI_TRACING_ENABLED=false`، أو `tracing=False` على الطاقم أو التدفق.
### أمثلة:
```python
# Disable CrewAI telemetry only
os.environ['CREWAI_DISABLE_TELEMETRY'] = 'true'
-# Disable all OpenTelemetry (including CrewAI)
+# Disable all OpenTelemetry instrumentation, CrewAI's telemetry included (not AMP tracing)
os.environ['OTEL_SDK_DISABLED'] = 'true'
```
diff --git a/docs/edge/en/telemetry.mdx b/docs/edge/en/telemetry.mdx
index 9801c44c7..1fa5c7b81 100644
--- a/docs/edge/en/telemetry.mdx
+++ b/docs/edge/en/telemetry.mdx
@@ -24,14 +24,14 @@ When the `share_crew` feature is enabled, detailed data including task descripti
to provide deeper insights. This expanded data collection may include personal information if users have incorporated it into their crews or tasks.
Users should carefully consider the content of their crews and tasks before enabling `share_crew`.
Users can disable CrewAI telemetry by setting `CREWAI_DISABLE_TELEMETRY` to `true`, `1`, `yes`, or `on` (any case). `OTEL_SDK_DISABLED` with the same values also disables CrewAI's exporter. The OpenTelemetry SDK itself still only honors `true` for disabling other instrumentation in the process.
-AMP tracing is covered separately in [Tracing](/en/observability/tracing).
+AMP tracing is covered separately in [Tracing](/en/observability/tracing). `OTEL_SDK_DISABLED` does not turn it off: set `CREWAI_TRACING_ENABLED=false`, or `tracing=False` on the Crew or Flow.
### Examples:
```python
# Disable CrewAI telemetry only
os.environ['CREWAI_DISABLE_TELEMETRY'] = 'true'
-# Disable all OpenTelemetry (including CrewAI)
+# Disable all OpenTelemetry instrumentation, CrewAI's telemetry included (not AMP tracing)
os.environ['OTEL_SDK_DISABLED'] = 'true'
```
diff --git a/docs/edge/ko/telemetry.mdx b/docs/edge/ko/telemetry.mdx
index 223a3c6f4..0e73ade97 100644
--- a/docs/edge/ko/telemetry.mdx
+++ b/docs/edge/ko/telemetry.mdx
@@ -23,14 +23,14 @@ CrewAI는 익명 텔레메트리를 활용하여 사용 통계를 수집하며,
이 확대된 데이터 수집에는 사용자가 crew나 작업에 개인정보를 포함한 경우, 개인정보가 포함될 수 있습니다.
사용자는 `share_crew`를 활성화하기 전에 crew와 작업의 내용을 신중하게 검토해야 합니다.
사용자는 `CREWAI_DISABLE_TELEMETRY`를 `true`, `1`, `yes`, `on` 중 하나로 설정하여 CrewAI 텔레메트리를 비활성화할 수 있습니다(대소문자 무관). 같은 값의 `OTEL_SDK_DISABLED`도 CrewAI exporter를 끕니다. 프로세스 내 다른 OpenTelemetry 계측을 끄려면 OpenTelemetry SDK는 여전히 `true`만 인식합니다.
-AMP 추적은 [Tracing](/ko/observability/tracing)에서 별도로 다룹니다.
+AMP 추적은 [Tracing](/ko/observability/tracing)에서 별도로 다룹니다. `OTEL_SDK_DISABLED`는 AMP 추적을 끄지 않습니다. `CREWAI_TRACING_ENABLED=false`를 설정하거나 Crew 또는 Flow에 `tracing=False`를 지정하세요.
### 예시:
```python
# CrewAI telemetry만 비활성화
os.environ['CREWAI_DISABLE_TELEMETRY'] = 'true'
-# 모든 OpenTelemetry 비활성화(CrewAI 포함)
+# 모든 OpenTelemetry 계측 비활성화(CrewAI 텔레메트리 포함, AMP 추적은 제외)
os.environ['OTEL_SDK_DISABLED'] = 'true'
```
diff --git a/docs/edge/pt-BR/telemetry.mdx b/docs/edge/pt-BR/telemetry.mdx
index 21faa9223..52c49c53c 100644
--- a/docs/edge/pt-BR/telemetry.mdx
+++ b/docs/edge/pt-BR/telemetry.mdx
@@ -24,14 +24,14 @@ Quando o recurso `share_crew` está ativado, dados detalhados, incluindo descri
para fornecer insights mais detalhados. Essa coleta expandida pode incluir informações pessoais caso o usuário as tenha inserido em seus crews ou tarefas.
Usuários devem considerar cuidadosamente o conteúdo de seus crews e tarefas antes de habilitar o `share_crew`.
A telemetria do CrewAI pode ser desabilitada ao definir `CREWAI_DISABLE_TELEMETRY` como `true`, `1`, `yes` ou `on` (qualquer capitalização). `OTEL_SDK_DISABLED` com os mesmos valores também desabilita o exportador do CrewAI. O SDK do OpenTelemetry em si ainda só reconhece `true` para desabilitar as demais instrumentações do processo.
-O rastreamento AMP é tratado em [Tracing](/pt-BR/observability/tracing).
+O rastreamento AMP é tratado em [Tracing](/pt-BR/observability/tracing). `OTEL_SDK_DISABLED` não desliga o rastreamento AMP: defina `CREWAI_TRACING_ENABLED=false` ou `tracing=False` no Crew ou Flow.
### Exemplos:
```python
# Desabilitar apenas a telemetria do CrewAI
os.environ['CREWAI_DISABLE_TELEMETRY'] = 'true'
-# Desabilitar todo o OpenTelemetry (incluindo CrewAI)
+# Desabilitar toda a instrumentação OpenTelemetry, incluindo a telemetria do CrewAI (não o rastreamento AMP)
os.environ['OTEL_SDK_DISABLED'] = 'true'
```
diff --git a/lib/cli/src/crewai_cli/cli.py b/lib/cli/src/crewai_cli/cli.py
index d4a555b2f..35f493e31 100644
--- a/lib/cli/src/crewai_cli/cli.py
+++ b/lib/cli/src/crewai_cli/cli.py
@@ -741,14 +741,19 @@ def eval_command(
models: str | None,
deployment_id: str | None,
) -> None:
- """Evaluate the last traced run through CrewAI AMP, or compare models on the
- project's deployment (--models).
+ """Evaluate this crew through CrewAI AMP, or compare models on the project's
+ deployment (--models).
+
+ It grades the project's last traced run. When there is none, or the project
+ changed since it, it runs the crew first and grades that run: the run is
+ traced and its trace sent to CrewAI AMP, which grades it. Nothing is asked,
+ with or without a terminal, so `crewai eval` alone takes a coding agent from
+ a change to its verdict.
Run by a script or coding agent (no terminal), it prints a markdown brief
after the verdict — what failed and the change to make — so an agent can act
on it directly. That is when AMP provides the brief: otherwise its link, when
- AMP sends only that, else nothing more. It grades a run that already
- happened: after a change, `crewai run` again, then `crewai eval`.
+ AMP sends only that, else nothing more.
A run's evaluation exits 0 only when the goal gate PASSED, and 1 otherwise — a
failed gate, no verdict, or an evaluation that could not run — so a CI job can
diff --git a/lib/cli/src/crewai_cli/crew_run_tui.py b/lib/cli/src/crewai_cli/crew_run_tui.py
index aa8db5847..25474ae29 100644
--- a/lib/cli/src/crewai_cli/crew_run_tui.py
+++ b/lib/cli/src/crewai_cli/crew_run_tui.py
@@ -5,7 +5,7 @@ Two-column layout: left sidebar (tasks/agents/tokens) + main content
"""
import asyncio
-from collections.abc import Callable, Iterator
+from collections.abc import Callable, Iterable, Iterator
import contextlib
from contextlib import contextmanager
from contextvars import ContextVar
@@ -14,6 +14,7 @@ import os
from pathlib import Path
import re
import secrets
+import sys
import threading
import time
from typing import Any, ClassVar
@@ -263,13 +264,14 @@ class TraceConsentScreen(ModalScreen[bool]):
Binding("n", "consent_no", "No", show=False),
Binding("escape", "consent_no", "Cancel", show=False),
]
+ YES_LABEL = "Share Trace"
def compose(self) -> ComposeResult:
with Vertical(id="consent-dialog"):
yield Static(self._build_content(), id="consent-text")
with Horizontal(id="consent-buttons"):
yield Button(
- "Share Trace",
+ self.YES_LABEL,
id="btn-consent-yes",
classes="consent-btn",
)
@@ -296,6 +298,36 @@ class TraceConsentScreen(ModalScreen[bool]):
self.dismiss(False)
+# What an app says when the evaluation it was opened for turned tracing on.
+EVALUATION_NOTICE = (
+ "Tracing is now on for this project, so this run can be evaluated: "
+ "its trace is sent to CrewAI AMP, which grades it."
+)
+TRACING_OFF_IN_CODE = (
+ "This {noun} sets tracing=False, so its runs are not traced and cannot be "
+ "evaluated. Remove tracing=False to evaluate it."
+)
+
+
+class TurnTracingOnScreen(TraceConsentScreen):
+ """Evaluate, pressed on a run of a project that turned tracing off: the
+ run has to happen again, traced, and only a yes turns tracing on."""
+
+ YES_LABEL = "Turn On & Evaluate"
+
+ def _build_content(self) -> Text:
+ t = Text()
+ t.append(
+ " Tracing is turned off for this project.\n\n", style=f"bold {_C_TEXT}"
+ )
+ t.append(" Evaluating a run needs its trace. Turn tracing\n", style=_C_DIM)
+ t.append(" on and run it again?\n\n", style=_C_DIM)
+ t.append(
+ " The trace is sent to CrewAI AMP, which grades it.\n", style=_C_MUTED
+ )
+ return t
+
+
_AUTO_EVAL: ContextVar[dict[str, str | None] | None] = ContextVar(
"crewai_tui_auto_eval", default=None
)
@@ -313,31 +345,60 @@ _AUTO_EVAL: ContextVar[dict[str, str | None] | None] = ContextVar(
# random token, and it counts only while the file of that name exists in this
# user's crewAI data directory: the command writes it before the run and removes
# it after, and a `.env` cannot create a file.
+#
+# crewAI's tracing reads the same token to send the run's trace without asking
+# (`crewai_core.eval_handshake`, which must name the same variable and folder).
+# It is not imported here: this module also runs inside a project's environment
+# (the JSON crew runner), whose crewai-core may be older than this CLI.
_AWAITING_EVAL_ENV = "CREWAI_EVAL_AWAITING_RUN"
_AWAITING_TOKEN = re.compile(r"[0-9a-f]{32}")
def _awaiting_dir() -> Path:
- import appdirs
+ """This user's crewAI data directory, found from the account's home in the
+ account database, never from `HOME` or `XDG_DATA_HOME`: a project's `.env`
+ can set those, and a token file it planted there must not count."""
+ if os.name == "nt":
+ import appdirs
- return Path(appdirs.user_data_dir("crewai", "CrewAI")) / "eval-awaiting"
+ return Path(appdirs.user_data_dir("crewai", "CrewAI")) / "eval-awaiting"
+ try:
+ import pwd
+
+ home = Path(pwd.getpwuid(os.getuid()).pw_dir)
+ except (ImportError, KeyError, OSError):
+ home = Path.home()
+ base = (
+ home / "Library" / "Application Support"
+ if sys.platform == "darwin"
+ else home / ".local" / "share"
+ )
+ return base / "crewai" / "eval-awaiting"
@contextmanager
-def evaluating_after_run() -> Iterator[dict[str, str | None]]:
+def evaluating_after_run(
+ *, trusted_amp_origins: Iterable[str] = (), tracing_turned_on: bool = False
+) -> Iterator[dict[str, str | None]]:
"""Run the crew for an evaluation that is already under way.
- `crewai eval` with nothing traced offers to run the crew first. The app it
- opens would otherwise sit there until somebody quits it, with the command
- waiting behind — so inside this block the app closes itself when the run
- ends and leaves its execution id in the holder. It does NOT chain into an
- evaluation of its own: the command that opened it is the one evaluating.
+ `crewai eval` with nothing traced runs the crew first. The app it opens
+ would otherwise sit there until somebody quits it, with the command waiting
+ behind — so inside this block the app evaluates what it ran and then
+ leaves its execution id in the holder.
A run in the same process leaves its id here; a run in a child process
leaves it in the project's last-run record, which the caller reads when the
- holder comes back empty.
+ holder comes back empty. The token file carries what the command decided
+ before the project's `.env` was loaded: which AMP origins may receive the
+ saved login (TRUSTED_AMP_ORIGINS), and whether it just turned tracing on for
+ the project (TRACING_TURNED_ON) — the one case its app says where the trace
+ goes.
"""
- holder: dict[str, str | None] = {"execution_id": None}
+ holder: dict[str, str | None] = {
+ "execution_id": None,
+ "tracing_turned_on": "true" if tracing_turned_on else None,
+ }
token = _AUTO_EVAL.set(holder)
before = os.environ.get(_AWAITING_EVAL_ENV)
# Without the file a child app cannot be told, and the command grades the
@@ -346,8 +407,16 @@ def evaluating_after_run() -> Iterator[dict[str, str | None]]:
with contextlib.suppress(OSError):
nonce = secrets.token_hex(16)
_awaiting_dir().mkdir(parents=True, exist_ok=True)
- (_awaiting_dir() / nonce).touch(exist_ok=False)
- marker = _awaiting_dir() / nonce
+ path = _awaiting_dir() / nonce
+ with path.open("x", encoding="utf-8") as out:
+ _json.dump(
+ {
+ "trusted_amp_origins": sorted(set(trusted_amp_origins)),
+ "tracing_turned_on": tracing_turned_on,
+ },
+ out,
+ )
+ marker = path
os.environ[_AWAITING_EVAL_ENV] = nonce
try:
yield holder
@@ -366,9 +435,9 @@ def _an_evaluation_is_waiting() -> dict[str, str | None] | None:
"""The evaluation this run was started for, if there is one.
In this process the holder itself; in a child process the environment says
- an evaluation is waiting and the holder is a local one — the id travels
- back through the project's last-run record instead, which the child writes
- when its trace is exported.
+ an evaluation is waiting and the holder is a local one, built from the
+ token file — the id travels back through the project's last-run record
+ instead, which the child writes when its trace is exported.
"""
holder = _AUTO_EVAL.get()
if holder is not None:
@@ -378,10 +447,14 @@ def _an_evaluation_is_waiting() -> dict[str, str | None] | None:
if not _AWAITING_TOKEN.fullmatch(value):
return None
try:
- waiting = (_awaiting_dir() / value).is_file()
+ text = (_awaiting_dir() / value).read_text(encoding="utf-8")
except OSError:
return None
- return {"execution_id": None} if waiting else None
+ said: Any = {}
+ with contextlib.suppress(ValueError):
+ said = _json.loads(text)
+ turned_on = isinstance(said, dict) and said.get("tracing_turned_on") is True
+ return {"execution_id": None, "tracing_turned_on": "true" if turned_on else None}
def _recorded_run(execution_uuid: str) -> dict[str, Any] | None:
@@ -656,6 +729,8 @@ FooterKey .footer-key--key {
self._current_method: str | None = None
self._elapsed_frozen: float | None = None
self._want_deploy: bool = False
+ # Evaluate was pressed on a run that was not traced (action_evaluate_crew).
+ self._want_traced_rerun: bool = False
# This app's own execution, captured from the run while it runs. The
# project-wide record is the last run to FINISH anywhere, which another
# run in the same project can replace between this one ending and its
@@ -723,6 +798,24 @@ FooterKey .footer-key--key {
self._run_crew_worker()
elif self._crew_json_path:
self._load_and_run_worker()
+ self._say_this_run_will_be_evaluated()
+
+ def _say_this_run_will_be_evaluated(self) -> None:
+ """Opened to run the crew for an evaluation that just turned tracing on
+ for the project (the yes inferred from `crewai eval`, the first time):
+ say so, plainly, once. A run that cannot be traced says nothing."""
+ waiting = self._auto_eval or {}
+ if waiting.get("tracing_turned_on") and not self._tracing_off_in_code():
+ self.notify(EVALUATION_NOTICE, title="Evaluate", timeout=10)
+
+ def _tracing_off_in_code(self) -> bool:
+ """Does the crew or flow set `tracing=False`? Then no run of it is
+ traced, and none can be evaluated."""
+ return any(
+ getattr(target, "tracing", None) is False
+ for target in (self._crew, self._flow)
+ if target is not None
+ )
# ── Crew execution ──────────────────────────────────────
@@ -1283,6 +1376,10 @@ FooterKey .footer-key--key {
_is_test_environment,
)
+ # `crewai eval` started this run to grade it, and the grade is read off
+ # the trace: running the command was the yes.
+ if _an_evaluation_is_waiting() is not None:
+ return True
if _is_test_environment() or not _is_interactive_terminal():
return False
@@ -1355,19 +1452,65 @@ FooterKey .footer-key--key {
self._record_tui_button_click("evaluate")
traced = self._traced_execution_id()
if traced is None:
- from crewai_cli.experimental.eval_crew import record_stop
-
- record_stop("untraced")
- self.notify(
- "This run was not traced, so there is nothing to evaluate. "
- "Turn tracing on and run it again.",
- title="Evaluate",
- severity="warning",
+ # Nothing of this run reached AMP, so there is nothing to grade: the
+ # run happens again, traced, and is evaluated — once tracing is on.
+ # A project that turned it off, or never turned it on, is asked
+ # first; a no runs nothing.
+ from crewai_cli.experimental.eval_crew import (
+ _machine_amp_origins,
+ record_stop,
+ tracing_already_on,
+ tracing_turned_off,
+ untrusted_amp_stop,
)
+
+ if self._tracing_off_in_code():
+ # Running it again could not make a trace: nothing to pay for.
+ self.notify(
+ TRACING_OFF_IN_CODE.format(noun=self._run_noun),
+ title="Evaluate",
+ severity="warning",
+ )
+ return
+ # Before anything is asked or written: a run whose trace would go to
+ # an AMP this machine does not trust with its login is not run again
+ # (`crewai eval` refuses it too), so a yes here could not be honored.
+ # This process loaded the project's `.env` for the run, so only the
+ # AMPs this machine is configured for are trusted.
+ stop = untrusted_amp_stop(_machine_amp_origins())
+ if stop is not None:
+ record_stop("untrusted_amp")
+ self.notify(stop, title="Evaluate", severity="warning")
+ return
+ if tracing_turned_off():
+ self.push_screen(TurnTracingOnScreen(), self._tracing_on_answered)
+ return
+ if not tracing_already_on():
+ # Never turned on: asked first, on the same consent screen a
+ # run's trace is shared from.
+ self.push_screen(TraceConsentScreen(), self._tracing_on_answered)
+ return
+ self._run_again_traced()
return
self._start_evaluation(traced)
+ def _tracing_on_answered(self, yes: bool | None) -> None:
+ from crewai_cli.experimental.eval_crew import record_stop, turn_tracing_on
+
+ if yes is not True:
+ record_stop("declined")
+ return
+ turn_tracing_on()
+ self._run_again_traced()
+
+ def _run_again_traced(self) -> None:
+ """Close, and run the crew again traced through `crewai eval`'s own path
+ (`run_crew._chain_traced_rerun`)."""
+ self._want_traced_rerun = True
+ self._unsubscribe()
+ self.exit(self._crew_result)
+
def _start_evaluation(self, execution_id: str) -> None:
self._evaluation = {"state": "starting", "execution_id": execution_id}
# The run is over and the clock slowed to twice a second; something is
diff --git a/lib/cli/src/crewai_cli/experimental/eval_crew.py b/lib/cli/src/crewai_cli/experimental/eval_crew.py
index 847b1ad7d..6bd104379 100644
--- a/lib/cli/src/crewai_cli/experimental/eval_crew.py
+++ b/lib/cli/src/crewai_cli/experimental/eval_crew.py
@@ -3,8 +3,11 @@
crewAI records a traced run in `.crewai/last_run.json` when the run's spans
reach Wharf. This command reads that record (or takes `--run EXECUTION_ID`),
asks AMP to evaluate the run, prints and opens the URL AMP answers with,
-waits for the verdict and prints it. With no traced run recorded it offers
-to turn tracing on for the project and run the crew now.
+waits for the verdict and prints it. With no traced run recorded, or when the
+project changed since it, it runs the crew first and evaluates that run: the
+run is traced and its trace sent to AMP without asking (running `crewai eval`
+is the request to have it graded — `crewai_core.eval_handshake`), so a coding
+agent with no terminal gets a verdict from this one command.
Who may evaluate what is AMP's decision: an anonymous run once without an
account, then it needs one; a run traced while logged in for that
@@ -48,7 +51,7 @@ import webbrowser
import click
from crewai_core.constants import DEFAULT_CREWAI_ENTERPRISE_URL
from crewai_core.settings import Settings
-from dotenv import load_dotenv, set_key
+from dotenv import dotenv_values, set_key
import httpx
from rich.console import Console
from rich.table import Table
@@ -121,7 +124,9 @@ def _record_usage(*, logged_in: bool) -> None:
StopReason = Literal[
- "untraced",
+ "untraced_after_run",
+ "untrusted_amp",
+ "tracing_off",
"declined",
"no_project",
"login_unreadable",
@@ -145,21 +150,107 @@ def record_stop(reason: StopReason) -> None:
The reason is a word from a closed list and never the message, which can
carry AMP's own words or a local path.
"""
+ _count_usage(f"cli_usage:eval_stopped:{reason}")
+
+
+NOT_TRACED = (
+ "The crew ran, but its trace did not reach CrewAI AMP, so there is nothing to evaluate: "
+ "the run may have failed (its output is above), the crew or flow may set tracing=False, "
+ "or this project's crewai is older than the version that records runs for evaluation "
+ f"({LAST_RUN_FILE})."
+)
+
+# Why `crewai eval` ran the crew before evaluating, a word from a closed list:
+# nothing traced yet, the project changed since its last traced run, or the run
+# app's Evaluate pressed on a run that was not traced.
+RunReason = Literal["no_traced_run", "project_changed", "run_again"]
+RUNNING_THE_CREW: dict[RunReason, str] = {
+ "no_traced_run": "No traced run of this crew yet: running it now to evaluate it.",
+ "project_changed": (
+ "The project changed since its last traced run: running the crew again "
+ "to evaluate what it does now."
+ ),
+ "run_again": "That run was not traced: running the crew again to evaluate it.",
+}
+TRACING_NOW_ON = (
+ "Tracing is now on for this project: its runs' traces are sent to CrewAI AMP, "
+ "which grades them."
+)
+# A project that turned tracing off said no, and nothing here overrules it: at a
+# terminal the person is asked, and without one the command says how to turn it
+# on (a coding agent that does so has made that choice itself).
+TRACING_OFF_QUESTION = (
+ "Tracing is turned off for this project, and evaluating a run needs its trace. "
+ "Turn tracing on and run the crew now? Its trace is sent to CrewAI AMP."
+)
+TRACING_OFF_UNATTENDED = (
+ f"Tracing is turned off for this project ({TRACING_ENV_VAR}=false), and evaluating "
+ f"a run needs its trace. Set {TRACING_ENV_VAR}=true in the project's .env to turn it "
+ "on, then run `crewai eval` again."
+)
+TRACING_STAYS_OFF = "Nothing was run: tracing stays off for this project."
+TRACING_IS_ON = "Tracing is on for this project — its runs are traced to CrewAI AMP."
+NOT_SHARED = (
+ "Nothing was run: evaluating a run needs its trace, and no trace was shared."
+)
+UNTRUSTED_AMP = (
+ "The crew was not run: its trace would go to {url}, an AMP this machine is not "
+ "set up to trust with its CrewAI login. Set that AMP up (`crewai enterprise "
+ "configure {url}`) or remove CREWAI_PLUS_URL from the project's .env, then "
+ "evaluate again."
+)
+
+
+def tracing_turned_off() -> bool:
+ """Has this project said no to tracing — `CREWAI_TRACING_ENABLED` false or
+ 0, in its `.env` (which wins, as the run loads it) or else the environment?"""
+ value: str | None = None
+ with contextlib.suppress(Exception):
+ env_file = Path.cwd() / ".env"
+ if env_file.is_file():
+ value = dotenv_values(env_file).get(TRACING_ENV_VAR)
+ if value is None:
+ value = os.environ.get(TRACING_ENV_VAR)
+ return str(value or "").strip().lower() in ("false", "0")
+
+
+def turn_tracing_on() -> None:
+ """`CREWAI_TRACING_ENABLED=true` in the project's .env, and in this process:
+ what somebody asked whether to turn tracing on said yes to."""
+ env_file = Path.cwd() / ".env"
+ env_file.touch(exist_ok=True)
+ set_key(str(env_file), TRACING_ENV_VAR, "true", quote_mode="never")
+ os.environ[TRACING_ENV_VAR] = "true"
+
+
+def _count_usage(feature: str) -> None:
+ """One usage count, never more: a word from a closed list, nothing that
+ names the run, the account or its content."""
try:
from crewai_core.telemetry import Telemetry
telemetry = Telemetry()
telemetry.set_tracer()
- telemetry.feature_usage_span(f"cli_usage:eval_stopped:{reason}")
+ telemetry.feature_usage_span(feature)
except Exception: # noqa: S110 - telemetry must never break a command
pass
-NOT_TRACED = (
- "The run finished but no trace was recorded: the run may have failed, sharing the "
- "trace was declined, or this project's crewai is older than the version that records "
- f"the last run ({LAST_RUN_FILE}). Run the crew again and accept when asked, then `crewai eval`."
-)
+GATE_WORDS = frozenset({"passed", "failed", "inconclusive"})
+
+
+def record_finished(finished: dict[str, Any]) -> None:
+ """Count an evaluation that reached a verdict, by the gate's word.
+
+ `cli_usage:eval` counts the evaluations that started and `eval_stopped:*`
+ the ones that did not get there; this is the third end, so "reached a
+ result" is counted rather than read off a missing stop.
+ """
+ if finished.get("status") != "done":
+ return
+ verdict = finished.get("verdict")
+ gate = str(verdict.get("gate") if isinstance(verdict, dict) else "").lower()
+ _count_usage(f"cli_usage:eval_finished:{gate if gate in GATE_WORDS else 'unknown'}")
class EvaluationStoppedError(RuntimeError):
@@ -180,23 +271,57 @@ def _note(text: str, style: str = "dim") -> None:
console.print(Text(text), style=style)
-def eval_crew(run_id: str | None = None) -> None:
- """Evaluate the last traced run of this project, or the run RUN_ID."""
+def eval_crew(
+ run_id: str | None = None,
+ *,
+ run_again: bool = False,
+ run_args: dict[str, Any] | None = None,
+ trusted_amp_origins: set[str] | None = None,
+) -> None:
+ """Evaluate the last traced run of this project, or the run RUN_ID.
+
+ With no traced run, or when the project changed since its last one, the
+ crew runs first and that run is evaluated; RUN_AGAIN runs it whatever the
+ record says (the run app's Evaluate, pressed on a run that was not traced),
+ with RUN_ARGS — `run_crew`'s own arguments, so the run graded is the one the
+ person pressed Evaluate on (its definition, inputs and trained agents).
+ TRUSTED_AMP_ORIGINS: where the saved login may go, from a caller whose
+ process already loaded the project's `.env` (the run app); otherwise read
+ here, before this command loads it.
+ """
+ # Before anything here writes to the project (the project id), so this
+ # command's own writes never read as a change.
+ record = read_last_run() or {}
+ why = None if run_id else _why_run_the_crew(record, run_again=run_again)
project_id = get_or_create_project_id()
# Read before the project's .env is loaded, so a project cannot add itself.
- trusted = _trusted_amp_origins()
+ trusted = (
+ _trusted_amp_origins()
+ if trusted_amp_origins is None
+ else set(trusted_amp_origins)
+ )
_load_project_env()
- record = read_last_run() or {}
execution_id = run_id or record.get("execution_id")
- if execution_id is None:
- # Nothing traced here: the crew runs first, and the app that runs it
- # carries the evaluation on its own screen — link, progress, verdict.
- # It comes back with the run to grade when the app did NOT get to it: a
- # conversational session, or a flow that took the terminal instead.
- execution_id = _run_and_let_the_app_evaluate()
+ if why is not None:
+ # The crew runs first, and the app that runs it carries the evaluation
+ # on its own screen — link, progress, verdict. It comes back with the
+ # run to grade when no app got to it: a crew run without one (no
+ # terminal, or a classic crew), a conversational session, or a flow
+ # that took the terminal instead.
+ execution_id = _run_the_crew_to_evaluate(
+ why,
+ trusted=trusted,
+ run_args=run_args,
+ )
if execution_id is None:
return
record = read_last_run() or {}
+ if not execution_id:
+ # Unreachable: no run named and none recorded is a reason to run the crew.
+ return
+ execution_id = str(execution_id)
+ if not run_id:
+ remember_project_state(execution_id)
try:
client = _amp_client(trusted)
@@ -244,6 +369,7 @@ def eval_crew(run_id: str | None = None) -> None:
raise SystemExit(130) from None
if finished.get("status") == "failed":
record_stop("evaluation_failed")
+ record_finished(finished)
_print_verdict(finished, url)
_print_brief(finished)
_say_where_the_criteria_live(write_eval_config(finished))
@@ -424,10 +550,12 @@ def evaluate_run(
# count of evaluations that began, and an evaluation the app runs is one.
_record_usage(logged_in=client.api_key is not None)
record_evaluation_outcome(execution_id)
+ remember_project_state(execution_id)
on_started(started)
finished = _wait(client, started["id"], started.get("url"), on_status=on_status)
if finished.get("status") == "failed":
record_stop("evaluation_failed")
+ record_finished(finished)
written = write_eval_config(finished)
if written is not None:
finished = {**finished, "wrote_eval_config": written.name}
@@ -541,10 +669,13 @@ def _anonymous(client: PlusAPI) -> PlusAPI:
def _load_project_env() -> None:
- """The project's .env, as `crewai run` loads it — so CREWAI_PLUS_URL here is the one the run used."""
+ """The project's .env, as `crewai run` loads it — so CREWAI_PLUS_URL here is
+ the one the run used — with `crewai eval`'s token left as it was."""
env_file = Path.cwd() / ".env"
if env_file.is_file():
- load_dotenv(env_file, override=True)
+ from crewai_cli.run_crew import load_project_env
+
+ load_project_env(env_file)
def read_last_run(directory: Path | None = None) -> dict[str, Any] | None:
@@ -584,42 +715,259 @@ def saved_login() -> str | None:
) from error
-def _run_and_let_the_app_evaluate() -> str | None:
- """No traced run recorded here: offer to turn tracing on and run the crew.
+def _why_run_the_crew(
+ record: dict[str, Any], *, run_again: bool = False
+) -> RunReason | None:
+ """Why the crew has to run before there is a run to grade, or None."""
+ if run_again:
+ return "run_again"
+ if not record.get("execution_id"):
+ return "no_traced_run"
+ if _project_changed_since(record):
+ return "project_changed"
+ return None
+
- The run app evaluates what it ran, on its own screen, so this returns None
- when it did — the reader has already seen the link and the verdict. It
- returns the run to grade when no app got to it (a conversational session, a
- flow that took the terminal), and says so itself when nothing was traced.
+# What "the project" is when asking whether it changed since its last traced
+# run: its own files, not what tools and runs leave around them. Hidden folders
+# (.venv, .git, .crewai, caches) are skipped, and the `.env` is the one hidden
+# file that counts — it holds the keys and models a run uses.
+_NOT_THE_PROJECT_DIRS = frozenset(
+ {"__pycache__", "node_modules", "venv", "env", "build", "dist", "site-packages"}
+)
+_NOT_THE_PROJECT_SUFFIXES = (".pyc", ".log", ".egg-info")
+# Files a run writes once its crew has finished — a main.py that saves the result
+# after kickoff — land a moment after the record. Within this, a file is the
+# run's own; an edit after an evaluation (which takes minutes) is well past it.
+RUN_WRITES_FOR_SECONDS = 60.0
+# A project folder with more than this many entries is not walked to the end:
+# past it, the last traced run is taken as current.
+MAX_FILES_LOOKED_AT = 20_000
+
+
+def _project_changed_since(record: dict[str, Any]) -> bool:
+ """Did the project's own files change since RECORD's run?
+
+ For a run an evaluation already looked at, against the list of the
+ project's files taken then (`remember_project_state`): a file added,
+ removed or rewritten is a change, whoever made it. For any other run — one
+ `crewai run` recorded — a file modified after the record is one; a deleted
+ file is not seen there. A record without a readable time is taken as
+ current: there is nothing to compare against, and grading the run beats
+ running the crew for nothing.
"""
- if not Path("pyproject.toml").is_file():
+ remembered = _remembered_project_state(record.get("execution_id"))
+ if remembered is not None:
+ current = _project_state()
+ return current is not None and current != remembered
+ stamp = str(record.get("recorded_at") or record.get("finished_at") or "")
+ try:
+ when = datetime.fromisoformat(stamp)
+ except ValueError:
+ return False
+ if when.tzinfo is None:
+ return False
+ since = when.timestamp() + RUN_WRITES_FOR_SECONDS
+ files = _project_files()
+ return files is not None and any(info.st_mtime > since for info in files.values())
+
+
+def untrusted_amp_stop(trusted: set[str]) -> str | None:
+ """Why the crew must not run for an evaluation: its trace would go to an
+ AMP this command does not trust — the one the run traces to, read after
+ the project's `.env`. None when it may run.
+
+ Whether this machine holds a login right now does not matter: tracing
+ turned on toward that AMP would carry one there as soon as there is one."""
+ origin = _origin(PlusAPI().base_url)
+ if origin in trusted and _encrypted(origin):
+ return None
+ return UNTRUSTED_AMP.format(url=PlusAPI().base_url.rstrip("/"))
+
+
+def tracing_already_on() -> bool:
+ """Did somebody turn tracing on already — `CREWAI_TRACING_ENABLED=true`, or
+ the consent a person gave once? A crew that sets `tracing=True` in code is
+ not seen from here."""
+ try:
+ from crewai_core.user_data import is_tracing_enabled
+
+ return is_tracing_enabled()
+ except Exception:
+ return os.environ.get(TRACING_ENV_VAR, "").strip().lower() in ("true", "1")
+
+
+PROJECT_STATE_FILE = "eval_project_state.json"
+
+
+def _project_files() -> dict[str, os.stat_result] | None:
+ """The project's own files, by path relative to it: hidden folders, caches,
+ virtual environments, `crewai eval export` folders and `eval.jsonc` (the
+ evaluation's own writing) left out, the `.env` kept. None past
+ MAX_FILES_LOOKED_AT — too large to walk, so the last traced run is taken
+ as current."""
+ root = Path.cwd()
+ files: dict[str, os.stat_result] = {}
+ folders = [root]
+ while folders:
+ try:
+ entries = list(os.scandir(folders.pop()))
+ except OSError:
+ continue
+ for entry in entries:
+ if len(files) >= MAX_FILES_LOOKED_AT:
+ return None
+ name = entry.name
+ try:
+ if entry.is_dir(follow_symlinks=False):
+ # hidden, or a `crewai eval export` folder
+ if not (
+ name.startswith((".", "crewai-eval-"))
+ or name in _NOT_THE_PROJECT_DIRS
+ or name.endswith(_NOT_THE_PROJECT_SUFFIXES)
+ ):
+ folders.append(Path(entry.path))
+ continue
+ if (
+ name.startswith(".") and name != ".env"
+ ) or name == EVAL_CONFIG_FILE:
+ continue
+ if name.endswith(_NOT_THE_PROJECT_SUFFIXES):
+ continue
+ files[Path(entry.path).relative_to(root).as_posix()] = entry.stat(
+ follow_symlinks=False
+ )
+ except (OSError, ValueError):
+ continue
+ return files
+
+
+def _project_state() -> dict[str, list[int]] | None:
+ """Each project file with its modification time and size."""
+ files = _project_files()
+ if files is None:
+ return None
+ return {path: [info.st_mtime_ns, info.st_size] for path, info in files.items()}
+
+
+def remember_project_state(execution_id: str) -> None:
+ """Keep the list of the project's files as they are for the run an
+ evaluation is about to read, so the next `crewai eval` can tell whether the
+ project changed since — a deleted file included. Never raises."""
+ with contextlib.suppress(Exception):
+ state = _project_state()
+ path = Path.cwd() / LAST_RUN_FILE.parent / PROJECT_STATE_FILE
+ if state is None:
+ path.unlink(missing_ok=True)
+ return
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(
+ json.dumps({"execution_id": execution_id, "files": state}), encoding="utf-8"
+ )
+
+
+def _remembered_project_state(execution_id: Any) -> dict[str, list[int]] | None:
+ with contextlib.suppress(Exception):
+ path = Path.cwd() / LAST_RUN_FILE.parent / PROJECT_STATE_FILE
+ kept = json.loads(path.read_text(encoding="utf-8"))
+ if kept.get("execution_id") == execution_id and isinstance(
+ kept.get("files"), dict
+ ):
+ return {k: list(v) for k, v in kept["files"].items()}
+ return None
+
+
+def _at_a_terminal() -> bool:
+ return not is_dmn_mode_enabled() and sys.stdin.isatty()
+
+
+def _ask_before_the_first_enable() -> None:
+ """Nobody turned tracing on yet, and a person is here: ask on crewAI's
+ trace consent screen before turning it on. A no, or no answer before it
+ times out, runs nothing."""
+ from crewai.events.listeners.tracing.utils import prompt_user_for_trace_viewing
+
+ if not prompt_user_for_trace_viewing(sharing=True):
+ record_stop("declined")
+ console.print(NOT_SHARED, style="yellow")
+ raise SystemExit(0)
+ turn_tracing_on()
+ console.print(TRACING_IS_ON, style="green")
+
+
+def _ask_to_turn_tracing_on() -> None:
+ """The project turned tracing off: turn it on only on a yes.
+
+ With nobody to ask, the command stops and says how; a no runs nothing.
+ """
+ if is_dmn_mode_enabled() or not sys.stdin.isatty():
+ record_stop("tracing_off")
+ _fail(TRACING_OFF_UNATTENDED)
+ if not click.confirm(TRACING_OFF_QUESTION, default=True):
+ record_stop("declined")
+ console.print(TRACING_STAYS_OFF, style="yellow")
+ raise SystemExit(0)
+ turn_tracing_on()
+ console.print(TRACING_IS_ON, style="green")
+
+
+def _run_the_crew_to_evaluate(
+ why: RunReason,
+ *,
+ trusted: set[str] | None = None,
+ run_args: dict[str, Any] | None = None,
+) -> str | None:
+ """Run the crew, traced, to have a run to evaluate.
+
+ Running `crewai eval` asked for the run to be graded, and the grade is read
+ off its trace; the only question is the first enable of tracing, put to a
+ person at a terminal (`_ask_before_the_first_enable`). The run app
+ evaluates what it ran, on its own screen, so this returns None when it did
+ — the reader has already seen the link and the verdict. It returns the run
+ to grade when no app got to it, and says so itself when nothing was traced.
+ """
+ run_args = dict(run_args or {})
+ # A flow named by its definition runs outside a project too.
+ if not run_args.get("definition") and not Path("pyproject.toml").is_file():
record_stop("no_project")
_fail(
"No crewAI project here (no pyproject.toml). Run `crewai eval` from the project's "
"directory, or name a run: `crewai eval --run EXECUTION_ID`."
)
- steps = (
- "No traced run is recorded in this project. Turn tracing on — its runs are then "
- "traced to CrewAI AMP — and run the crew, then come back:\n"
- f" 1. add {TRACING_ENV_VAR}=true to .env\n 2. crewai run\n 3. crewai eval"
+ # This machine's login must never reach an AMP this command does not
+ # trust, and a project's crewai older than the handshake sends it to
+ # whatever AMP the project names whenever the run traces — by the
+ # project's .env, a consent given once, or `tracing=True` in code, which
+ # the CLI cannot see; nor can it tell which crewai the project runs. Nor
+ # may tracing be turned on toward it while there is no login yet: the next
+ # run after `crewai login` would carry it there. So toward such an AMP the
+ # crew is not run, nobody is asked, and nothing is written.
+ stop = untrusted_amp_stop(trusted or set())
+ if stop is not None:
+ record_stop("untrusted_amp")
+ _fail(stop)
+ if tracing_turned_off():
+ _ask_to_turn_tracing_on()
+ # Nobody turned tracing on. At a terminal the person is asked first, on
+ # crewAI's own consent screen; with nobody there (a coding agent, a
+ # script) the yes is inferred from this command and tracing is turned on
+ # for the project — said once, plainly, here and in the run app. From
+ # then on the project traces, and nothing more is asked or said.
+ turned_on_now = False
+ if not tracing_already_on():
+ if _at_a_terminal():
+ _ask_before_the_first_enable()
+ else:
+ turn_tracing_on()
+ turned_on_now = True
+ _count_usage(f"cli_usage:eval_ran_crew:{why}")
+ said = (
+ f"{RUNNING_THE_CREW[why]} {TRACING_NOW_ON}"
+ if turned_on_now
+ else RUNNING_THE_CREW[why]
)
- if is_dmn_mode_enabled() or not sys.stdin.isatty():
- # `Text`, never markup: the reason may be an OS error's own words, and
- # its `[Errno 13]` would be read as a style tag.
- reason, message = _nothing_traced_unattended() or ("untraced", steps)
- record_stop(reason)
- console.print(Text(message), style="yellow")
- raise SystemExit(1)
- if not click.confirm(
- "No traced run is recorded in this project. Turn tracing on and run the crew now? "
- "The run's trace is sent to CrewAI AMP.",
- default=True, # João, 2026-09-20: y/n with Y as the default — the prompt says what Enter does
- ):
- record_stop("declined")
- console.print(steps, style="yellow")
- raise SystemExit(0)
+ console.print(Text(said), style="bold")
- _enable_tracing()
from crewai_cli.crew_run_tui import evaluating_after_run
from crewai_cli.run_crew import run_crew
@@ -627,20 +975,25 @@ def _run_and_let_the_app_evaluate() -> str | None:
# ends. It runs in this process for some projects and in a child for others,
# so what says a run was traced is either the holder's id or the record the
# run wrote.
+ # The handshake (`crewai_core.eval_handshake`) carries what this command
+ # decided before the project's `.env` was loaded: the AMP origins its saved
+ # login may go to, and whether it just turned tracing on.
began = datetime.now(timezone.utc)
- with evaluating_after_run() as watched:
- run_crew()
+ with evaluating_after_run(
+ trusted_amp_origins=trusted or (), tracing_turned_on=turned_on_now
+ ) as watched:
+ run_crew(**run_args)
# The app says what it did with the run it watched, and its word settles
# this: it evaluated it (nothing left to do), or it had nothing to evaluate
# (say so once, in the terminal the screen has left). Only when no app got
- # here at all — a conversational session, a flow that took the terminal —
- # is there a run for this command to grade.
+ # here at all is there a run for this command to grade.
marker = evaluation_marker(after=began)
if marker is not None:
if marker.get("execution_id"):
+ remember_project_state(str(marker["execution_id"]))
return None
- record_stop("untraced")
+ record_stop("untraced_after_run")
console.print(NOT_TRACED, style="bold red")
raise SystemExit(1)
@@ -654,7 +1007,7 @@ def _run_and_let_the_app_evaluate() -> str | None:
if traced:
return str(traced)
- record_stop("untraced")
+ record_stop("untraced_after_run")
console.print(NOT_TRACED, style="bold red")
raise SystemExit(1)
@@ -672,44 +1025,6 @@ def _recorded_since(record: dict[str, Any], began: datetime) -> bool:
return when >= began - timedelta(seconds=1)
-def _nothing_traced_unattended() -> tuple[StopReason, str] | None:
- """Why a run with tracing on left nothing to evaluate, when nobody was there.
-
- An anonymous run asks before its trace leaves the machine, and a process with
- no terminal has nobody to ask — so its trace is kept local, tracing on or
- not. Telling that user to turn tracing on sends them round the same loop;
- logging in is what makes an unattended run traced. None when tracing is off
- or there is a login: the ordinary steps are the right ones then. A login
- that cannot be read says so instead, and is counted as that.
- """
- if os.environ.get(TRACING_ENV_VAR, "").strip().lower() not in ("true", "1"):
- return None
- try:
- if saved_login() is not None:
- return None
- except EvaluationStoppedError as unreadable:
- # A login that exists and cannot be read is the reason, and its
- # sentence says what to do about it.
- return unreadable.reason, str(unreadable)
- return "untraced", (
- "No traced run is recorded in this project. Tracing is on, but a run nobody is "
- "watching is only traced when you are logged in: run `crewai login`, then "
- "`crewai run` and `crewai eval` again."
- )
-
-
-def _enable_tracing() -> None:
- """`CREWAI_TRACING_ENABLED=true` in the project's .env, and in this process for the run about to start."""
- env_file = Path.cwd() / ".env"
- env_file.touch(exist_ok=True)
- set_key(str(env_file), TRACING_ENV_VAR, "true", quote_mode="never")
- os.environ[TRACING_ENV_VAR] = "true"
- console.print(
- "Tracing is on for this project — its runs are traced to CrewAI AMP.",
- style="green",
- )
-
-
def _start_evaluation(
client: PlusAPI,
execution_id: str,
diff --git a/lib/cli/src/crewai_cli/kickoff_flow.py b/lib/cli/src/crewai_cli/kickoff_flow.py
index 71b6079ef..92eddad20 100644
--- a/lib/cli/src/crewai_cli/kickoff_flow.py
+++ b/lib/cli/src/crewai_cli/kickoff_flow.py
@@ -64,7 +64,9 @@ def _load_conversational_flow_from_kickoff_script() -> Any | None:
return None
-def _run_conversational_flow_tui(flow: Any) -> Any:
+def _run_conversational_flow_tui(
+ flow: Any, definition: str | Path | None = None
+) -> Any:
from crewai.events.event_listener import EventListener
from crewai_cli.crew_run_tui import CrewRunApp
@@ -78,6 +80,13 @@ def _run_conversational_flow_tui(flow: Any) -> Any:
app._flow = flow
app.run()
+ if getattr(app, "_want_traced_rerun", False):
+ # This conversational flow again (by its definition, when it was named
+ # by one), traced, then evaluated.
+ from crewai_cli.run_crew import _chain_traced_rerun
+
+ _chain_traced_rerun({"definition": str(definition) if definition else None})
+
if app._status == "failed":
raise SystemExit(1)
diff --git a/lib/cli/src/crewai_cli/run_crew.py b/lib/cli/src/crewai_cli/run_crew.py
index e8ceb9677..af825ecda 100644
--- a/lib/cli/src/crewai_cli/run_crew.py
+++ b/lib/cli/src/crewai_cli/run_crew.py
@@ -1,6 +1,7 @@
from __future__ import annotations
from contextlib import AbstractContextManager, nullcontext
+import json
import os
from pathlib import Path
import re
@@ -48,6 +49,7 @@ The quotes are required in zsh so `crewai[tools]` is not treated as a glob.
"""
_JSON_CREW_RUNNER_CODE = """
import importlib.util
+import json
import os
from pathlib import Path
import sys
@@ -309,11 +311,9 @@ def _run_json_crew(
inputs: str | None = None,
) -> Any:
"""Load and run a JSON-defined crew."""
- from dotenv import load_dotenv
-
env_file = Path.cwd() / ".env"
if env_file.exists():
- load_dotenv(env_file, override=True)
+ load_project_env(env_file)
# JSON crews run in-process, so export the trained-agents file directly
# instead of forwarding it to a subprocess like classic crews do.
@@ -371,6 +371,14 @@ def _run_json_crew(
if getattr(app, "_want_deploy", False):
_chain_deploy()
+ if getattr(app, "_want_traced_rerun", False):
+ # The same crew with the inputs this run used, prompted ones included.
+ _chain_traced_rerun(
+ {
+ "trained_agents_file": trained_agents_file,
+ "inputs": json.dumps(runtime_inputs, default=str),
+ }
+ )
return app._crew_result
@@ -519,6 +527,46 @@ def _record_deploy_button(event: DeployButtonEvent) -> None:
pass
+# `crewai eval`'s token for the run it starts (`crew_run_tui._AWAITING_EVAL_ENV`,
+# `crewai_core.eval_handshake.AWAITING_ENV`; a test pins the three).
+EVAL_TOKEN_ENV = "CREWAI_EVAL_AWAITING_RUN" # noqa: S105 - a variable name, not a secret
+
+
+def load_project_env(env_file: Path) -> None:
+ """The project's `.env` over this process's environment, `crewai eval`'s
+ token excepted: the token this process was started with is kept, and a
+ `.env` never sets one. Otherwise a `.env` could unmark a run `crewai eval`
+ started, and its tracing would no longer apply the command's rules."""
+ from dotenv import load_dotenv
+
+ kept = os.environ.get(EVAL_TOKEN_ENV)
+ load_dotenv(env_file, override=True)
+ if kept is None:
+ os.environ.pop(EVAL_TOKEN_ENV, None)
+ else:
+ os.environ[EVAL_TOKEN_ENV] = kept
+
+
+def _chain_traced_rerun(run_args: dict[str, Any] | None = None) -> None:
+ """The run app's Evaluate, pressed on a run that was not traced: the run
+ happens again, traced, and that run is evaluated — `crewai eval`'s own
+ path, so the run app it opens shows the evaluation as it would there.
+
+ RUN_ARGS are `run_crew`'s arguments for the run the person watched (its
+ definition, inputs and trained agents), so the run graded is that one, not
+ the project's default."""
+ from crewai_cli.experimental.eval_crew import _machine_amp_origins, eval_crew
+
+ eval_crew(
+ run_again=True,
+ run_args={key: value for key, value in (run_args or {}).items() if value},
+ # This process loaded the project's `.env` for the run the person
+ # watched, so the `CREWAI_PLUS_URL` in it cannot be trusted with the
+ # saved login: the AMPs this machine is configured for, only.
+ trusted_amp_origins=_machine_amp_origins(),
+ )
+
+
def _chain_deploy() -> None:
from rich.console import Console
diff --git a/lib/cli/src/crewai_cli/run_declarative_flow.py b/lib/cli/src/crewai_cli/run_declarative_flow.py
index 4fbd2ca4f..6a1344274 100644
--- a/lib/cli/src/crewai_cli/run_declarative_flow.py
+++ b/lib/cli/src/crewai_cli/run_declarative_flow.py
@@ -67,18 +67,20 @@ def run_declarative_flow(definition: str | Path, inputs: str | None = None) -> N
# Load the project's .env before kickoff, mirroring the JSON-crew path
# (run_crew._run_json_crew) so flow projects pick up API keys/config the
# same way regardless of where crewai is installed.
- from dotenv import load_dotenv
+ from crewai_cli.run_crew import load_project_env
env_file = Path.cwd() / ".env"
if env_file.exists():
- load_dotenv(env_file, override=True)
+ load_project_env(env_file)
provided = parse_inputs_json(inputs) or {}
flow = load_declarative_flow(definition)
if _flow_is_conversational(flow):
- _run_conversational_declarative_flow(flow, inputs is not None)
+ _run_conversational_declarative_flow(
+ flow, inputs is not None, definition=definition
+ )
return
resolved_inputs = _resolve_flow_inputs(flow, provided)
@@ -91,7 +93,7 @@ def run_declarative_flow(definition: str | Path, inputs: str | None = None) -> N
# returns a pending marker rather than completing), neither of which the
# Textual TUI can handle correctly.
if is_interactive() and not _flow_uses_human_feedback(flow):
- _run_declarative_flow_tui(flow, resolved_inputs or None)
+ _run_declarative_flow_tui(flow, resolved_inputs or None, definition=definition)
return
try:
@@ -106,7 +108,7 @@ def run_declarative_flow(definition: str | Path, inputs: str | None = None) -> N
def _run_conversational_declarative_flow(
- flow: Flow[Any], inputs_supplied: bool
+ flow: Flow[Any], inputs_supplied: bool, *, definition: str | Path | None = None
) -> None:
"""Run a declarative chat flow on the conversational TUI.
@@ -158,11 +160,14 @@ def _run_conversational_declarative_flow(
from crewai_cli.kickoff_flow import _run_conversational_flow_tui
- _run_conversational_flow_tui(flow)
+ _run_conversational_flow_tui(flow, definition=definition)
def _run_declarative_flow_tui(
- flow: Flow[Any], resolved_inputs: dict[str, Any] | None
+ flow: Flow[Any],
+ resolved_inputs: dict[str, Any] | None,
+ *,
+ definition: str | Path | None = None,
) -> Any:
"""Run a declarative flow on the CrewAI TUI (the interactive default).
@@ -217,6 +222,18 @@ def _run_declarative_flow_tui(
from crewai_cli.run_crew import _chain_deploy
_chain_deploy()
+ if getattr(app, "_want_traced_rerun", False):
+ from crewai_cli.run_crew import _chain_traced_rerun
+
+ # This flow, by its definition, with the inputs this run used.
+ _chain_traced_rerun(
+ {
+ "definition": str(definition) if definition else None,
+ "inputs": json.dumps(resolved_inputs, default=str)
+ if resolved_inputs
+ else None,
+ }
+ )
return app._crew_result
diff --git a/lib/cli/tests/conftest.py b/lib/cli/tests/conftest.py
new file mode 100644
index 000000000..420fbdbb3
--- /dev/null
+++ b/lib/cli/tests/conftest.py
@@ -0,0 +1,13 @@
+"""Shared setup for the CLI's tests."""
+
+import pytest
+
+
[email protected](autouse=True)
+def _usage_counts_stay_on_this_machine(monkeypatch):
+ """The CLI counts what its commands do (`cli_usage:*`), and a suite that
+ exercises them would send those counts as if users had run them: every
+ `crewai eval` stop reason within the same second, from a laptop or from CI.
+ A test that checks the counting replaces `Telemetry` or sets the flag
+ itself."""
+ monkeypatch.setenv("CREWAI_DISABLE_TELEMETRY", "true")
diff --git a/lib/cli/tests/experimental/test_eval_crew.py b/lib/cli/tests/experimental/test_eval_crew.py
index 37a8ac803..971966d54 100644
--- a/lib/cli/tests/experimental/test_eval_crew.py
+++ b/lib/cli/tests/experimental/test_eval_crew.py
@@ -2,6 +2,7 @@
from __future__ import annotations
+import contextlib
from datetime import datetime, timedelta, timezone
import json
import os
@@ -21,9 +22,44 @@ from crewai_cli.cli import eval_command
@pytest.fixture(autouse=True)
def _awaiting_files_stay_in_tmp(tmp_path, monkeypatch):
"""The waiting-evaluation token file belongs to the user's crewAI data
- directory; a test keeps it in its own."""
+ directory; a test keeps it in its own (hidden, so it is never a change to
+ the project the test runs in)."""
+ # the run app's copy and crewAI's tracing read the same folder
+ for where in ("crewai_cli.crew_run_tui._awaiting_dir", "crewai_core.eval_handshake.awaiting_dir"):
+ monkeypatch.setattr(where, lambda: tmp_path / ".eval-awaiting")
+
+
[email protected](autouse=True)
+def _tracing_setting_put_back(monkeypatch):
+ """`crewai eval` loads the project's `.env` into this process; a test that
+ writes `CREWAI_TRACING_ENABLED` there must not leave it to the next."""
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+
+
[email protected](autouse=True)
+def consent_screen(monkeypatch):
+ """At a terminal, `crewai eval` asks on crewAI's trace consent screen
+ before it first turns tracing on; the person here says yes unless a test
+ says otherwise. Each question is recorded with how it was asked."""
+ asked: list[dict] = []
+ answer = {"yes": True}
+
+ def prompt(timeout_seconds=20, **kwargs):
+ asked.append(kwargs)
+ return answer["yes"]
+
monkeypatch.setattr(
- "crewai_cli.crew_run_tui._awaiting_dir", lambda: tmp_path / "eval-awaiting"
+ "crewai.events.listeners.tracing.utils.prompt_user_for_trace_viewing", prompt
+ )
+ return SimpleNamespace(asked=asked, answer=answer)
+
+
[email protected](autouse=True)
+def _no_crew_runs_unless_a_test_says_so(monkeypatch):
+ """`crewai eval` runs the crew when there is nothing current to grade; a
+ test that gets there without meaning to would start `uv run`."""
+ monkeypatch.setattr(
+ run_crew_module, "run_crew", lambda: pytest.fail("this test runs no crew")
)
@@ -323,7 +359,7 @@ def test_the_credential_goes_only_to_the_configured_amp_never_to_an_address_off_
# The project's .env is what `crewai run` traced with, so it is loaded first. This machine is
# logged in to that AMP, so the credential goes with the request and nothing is remarked on.
(directory / ".env").write_text("CREWAI_PLUS_URL=https://amp.test\n")
- record_last_run(directory, amp_base_url="https://amp.test/")
+ record_last_run(directory, "a-second-run", amp_base_url="https://amp.test/")
amp = install(monkeypatch, FakeAMP(statuses=[done()]))
eval_module.eval_crew()
assert eval_module.os.environ["CREWAI_PLUS_URL"] == "https://amp.test"
@@ -727,8 +763,9 @@ def test_dmn_mode_prints_the_url_but_opens_no_browser(project, monkeypatch, caps
assert opened == [] and URL in capsys.readouterr().out
-def test_run_skips_the_offer_when_nothing_is_recorded(project, monkeypatch, capsys):
- monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: pytest.fail("no offer with --run"))
+def test_a_named_run_never_runs_the_crew(project, monkeypatch, capsys):
+ """`--run <id>` grades that run: whatever the project's record says, the
+ crew does not run (the autouse guard fails the test if it does)."""
amp = install(monkeypatch, FakeAMP(statuses=[done()]))
eval_module.eval_crew(run_id="named-run")
@@ -739,7 +776,6 @@ def test_run_skips_the_offer_when_nothing_is_recorded(project, monkeypatch, caps
def test_outside_a_crewai_project_nothing_is_written_and_it_says_so(project, monkeypatch, capsys):
directory, _ = project
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
- monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: pytest.fail("no offer outside a project"))
amp = install(monkeypatch, FakeAMP())
with pytest.raises(SystemExit) as exit_:
@@ -750,54 +786,78 @@ def test_outside_a_crewai_project_nothing_is_written_and_it_says_so(project, mon
assert not (directory / ".env").exists() and amp.calls == []
-def test_without_a_traced_run_and_no_terminal_it_explains_and_exits(project, monkeypatch, capsys):
- (project[0] / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
[email protected]("logged_in", [True, False])
+def test_with_no_traced_run_a_coding_agent_gets_its_verdict_from_this_one_command(
+ project, monkeypatch, capsys, logged_in
+):
+ """No terminal, nothing traced, logged in or not: the crew runs, traced and
+ approved for upload by the command itself, and that run is graded. No
+ question: tracing is turned on for the project, said once, and the run
+ carries what the command decided before the project's `.env` was loaded."""
+ from crewai_core.eval_handshake import awaited_evaluation
+
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module, "is_dmn_mode_enabled", lambda: True)
- amp = install(monkeypatch, FakeAMP())
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
+ monkeypatch.setattr(eval_module, "saved_login", lambda: "tok" if logged_in else None)
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+ seen: dict[str, object] = {}
- with pytest.raises(SystemExit) as exit_:
- eval_module.eval_crew()
+ def fake_run_crew() -> None:
+ seen["tracing"] = os.environ.get("CREWAI_TRACING_ENABLED")
+ seen["said"] = awaited_evaluation() # what crewAI's tracing reads
+ record_last_run(directory, "fresh-run", recorded_at=_now())
- assert exit_.value.code == 1
- out = capsys.readouterr().out
- assert "No traced run is recorded" in out and "CREWAI_TRACING_ENABLED=true" in out and "crewai run" in out
- assert amp.calls == []
+ monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+ eval_module.eval_crew()
-def test_without_a_traced_run_it_offers_to_turn_tracing_on_and_run_the_crew(project, monkeypatch, capsys):
+ assert seen == {
+ "tracing": "true",
+ # this machine's AMP and crewAI's own, read before the project's .env
+ "said": {
+ "trusted_amp_origins": ["https://amp.test", "https://app.crewai.com"],
+ "tracing_turned_on": True,
+ },
+ }
+ assert amp.calls[0] == ("create", "fresh-run")
+ out = " ".join(capsys.readouterr().out.split())
+ assert (
+ "No traced run of this crew yet: running it now to evaluate it. Tracing is now on "
+ "for this project: its runs' traces are sent to CrewAI AMP, which grades them." in out
+ )
+ assert "Goal gate: PASSED" in out
+ assert "CREWAI_TRACING_ENABLED=true" in (directory / ".env").read_text()
+ assert "CREWAI_EVAL_AWAITING_RUN" not in os.environ
+
+
+def test_at_a_terminal_the_app_that_runs_the_crew_evaluates_it_without_a_question(
+ project, monkeypatch
+):
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
- prompts: list[tuple] = []
- monkeypatch.setattr(eval_module.click, "confirm", lambda text, **kwargs: prompts.append((text, kwargs)) or True)
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
ran: list[str] = []
def fake_run_crew() -> None:
ran.append("run")
- assert eval_module.os.environ.get("CREWAI_TRACING_ENABLED") == "true"
- record_last_run(directory, "fresh-run")
+ record_last_run(directory, "fresh-run", recorded_at=_now())
eval_module.record_evaluation_outcome("fresh-run") # what the app does
- import crewai_cli.run_crew as run_crew_module
-
monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
- monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
- monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
amp = install(monkeypatch, FakeAMP(statuses=[done()]))
eval_module.eval_crew()
assert ran == ["run"]
- text, kwargs = prompts[0]
- # the offer is about what happens, not about the variable that makes it happen
- assert "Turn tracing on and run the crew now?" in text and kwargs == {"default": True} # Enter is yes (João's call)
- assert "sent to CrewAI AMP" in text # what saying yes means, in the words that matter
- assert "CREWAI_TRACING_ENABLED" not in text # and not the variable that carries it
- assert "CREWAI_TRACING_ENABLED=true" in (directory / ".env").read_text()
- # the app that ran the crew evaluates it on its own screen; the command that
- # opened it does not ask AMP a second time
+ # the app that ran the crew evaluated it on its own screen; the command
+ # that opened it does not ask AMP a second time
assert amp.calls == []
- assert "Tracing is on for this project" in capsys.readouterr().out
+ # nobody had turned tracing on: it is now, for the project, said once
+ assert "CREWAI_TRACING_ENABLED=true" in (directory / ".env").read_text()
@pytest.mark.parametrize("logged_in", [True, False])
@@ -826,7 +886,8 @@ def test_the_command_counts_the_evaluation_and_nothing_that_names_it(
eval_module.eval_crew()
assert spans == [
- ("cli_usage:eval", {"authenticated": "true" if logged_in else "false"})
+ ("cli_usage:eval", {"authenticated": "true" if logged_in else "false"}),
+ ("cli_usage:eval_finished:passed", {}),
]
@@ -864,7 +925,6 @@ def test_the_fallback_grades_only_a_run_recorded_after_it_began(
began belongs to some other run."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.click, "confirm", lambda *a, **k: True)
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
monkeypatch.setattr(
run_crew_module,
@@ -881,7 +941,7 @@ def test_the_fallback_grades_only_a_run_recorded_after_it_began(
assert exited.value.code == 1
assert amp.calls == []
- assert "no trace was recorded" in capsys.readouterr().out
+ assert "did not reach CrewAI AMP" in capsys.readouterr().out
@pytest.mark.parametrize(
@@ -1045,7 +1105,6 @@ def test_the_app_it_opens_is_told_an_evaluation_is_waiting(project, monkeypatch)
told to evaluate what it ran, so nobody quits a screen to reach it."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.click, "confirm", lambda *a, **k: True)
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
def fake_run_crew() -> None:
@@ -1054,7 +1113,7 @@ def test_the_app_it_opens_is_told_an_evaluation_is_waiting(project, monkeypatch)
assert _AUTO_EVAL.get() is not None, "the app must be told in this process"
# and in a child process, where a ContextVar cannot reach: a token, and
# the file of that name that makes it count
- from crewai_cli.crew_run_tui import _awaiting_dir
+ from crewai_core.eval_handshake import awaiting_dir as _awaiting_dir
assert (_awaiting_dir() / os.environ["CREWAI_EVAL_AWAITING_RUN"]).is_file()
record_last_run(directory, "run-it-just-did")
@@ -1068,7 +1127,7 @@ def test_the_app_it_opens_is_told_an_evaluation_is_waiting(project, monkeypatch)
assert amp.calls == [] # the app did it, on its own screen
assert "CREWAI_EVAL_AWAITING_RUN" not in os.environ
- from crewai_cli.crew_run_tui import _awaiting_dir
+ from crewai_core.eval_handshake import awaiting_dir as _awaiting_dir
assert not any(_awaiting_dir().iterdir()) # and the token went with it
@@ -1080,7 +1139,6 @@ def test_a_run_no_app_evaluated_is_graded_by_the_command(project, monkeypatch):
rather than exiting as if somebody had."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.click, "confirm", lambda *a, **k: True)
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
monkeypatch.setattr(
run_crew_module,
@@ -1100,7 +1158,6 @@ def test_a_marker_from_an_earlier_run_never_counts_for_this_one(project, monkeyp
not about the run this command just watched — it is older than it."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.click, "confirm", lambda *a, **k: True)
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
_write_marker(directory, "some-older-run", _seconds_ago(3600))
@@ -1123,7 +1180,6 @@ def test_the_app_saying_it_had_nothing_to_grade_is_not_a_run_to_grade(
once — the screen that showed it is gone — and grades nothing."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.click, "confirm", lambda *a, **k: True)
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
def fake_run_crew() -> None:
@@ -1140,7 +1196,7 @@ def test_the_app_saying_it_had_nothing_to_grade_is_not_a_run_to_grade(
assert exit_code.value.code == 1
assert amp.calls == [] # never somebody else's run
- assert "no trace was recorded" in capsys.readouterr().out
+ assert "did not reach CrewAI AMP" in capsys.readouterr().out
def test_a_concurrent_run_cannot_send_the_command_after_the_wrong_one(
@@ -1150,7 +1206,6 @@ def test_a_concurrent_run_cannot_send_the_command_after_the_wrong_one(
while this app is open. The app's own marker is what settles it."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.click, "confirm", lambda *a, **k: True)
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
def fake_run_crew() -> None:
@@ -1171,7 +1226,6 @@ def test_a_run_that_recorded_nothing_says_so(project, monkeypatch, capsys):
show: this is the one thing the command still has to say."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.click, "confirm", lambda *a, **k: True)
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
monkeypatch.setattr(run_crew_module, "run_crew", lambda: None)
monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
@@ -1182,7 +1236,7 @@ def test_a_run_that_recorded_nothing_says_so(project, monkeypatch, capsys):
assert exit_code.value.code == 1
assert amp.calls == []
- assert "no trace was recorded" in capsys.readouterr().out
+ assert "did not reach CrewAI AMP" in capsys.readouterr().out
def test_a_run_from_hours_ago_is_not_waited_for(project, monkeypatch, capsys):
@@ -1216,23 +1270,9 @@ def test_an_id_somebody_typed_is_answered_at_once(project, monkeypatch):
]
-def test_declining_the_offer_exits_cleanly_with_the_steps(project, monkeypatch, capsys):
- (project[0] / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
- monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: False)
- amp = install(monkeypatch, FakeAMP())
-
- with pytest.raises(SystemExit) as exit_:
- eval_module.eval_crew()
-
- assert exit_.value.code == 0
- assert "crewai eval" in capsys.readouterr().out and amp.calls == []
-
-
def test_a_run_that_leaves_no_trace_behind_is_explained(project, monkeypatch, capsys):
(project[0] / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
- monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: True)
import crewai_cli.run_crew as run_crew_module
monkeypatch.setattr(run_crew_module, "run_crew", lambda: None)
@@ -1242,8 +1282,8 @@ def test_a_run_that_leaves_no_trace_behind_is_explained(project, monkeypatch, ca
eval_module.eval_crew()
assert exit_.value.code == 1
- out = capsys.readouterr().out
- assert "no trace was recorded" in out and "older than the version that records the last run" in out
+ out = " ".join(capsys.readouterr().out.split())
+ assert "did not reach CrewAI AMP" in out and "older than the version that records runs for evaluation" in out
def test_the_cli_command_maps_to_the_implementation(monkeypatch):
@@ -1254,7 +1294,9 @@ def test_the_cli_command_maps_to_the_implementation(monkeypatch):
assert runner.invoke(eval_command, []).exit_code == 0
assert runner.invoke(eval_command, ["--run", EXECUTION_ID]).exit_code == 0
assert calls == [{"run_id": None}, {"run_id": EXECUTION_ID}]
- assert "Evaluate the last traced run" in runner.invoke(eval_command, ["--help"]).output
+ help_text = " ".join(runner.invoke(eval_command, ["--help"]).output.split())
+ assert "Evaluate this crew through CrewAI AMP" in help_text
+ assert "it runs the crew first and grades that run" in help_text
def test_only_a_missing_login_reads_as_anonymous(monkeypatch, capsys):
@@ -1310,47 +1352,17 @@ def test_read_last_run_reads_the_record_crewai_writes(tmp_path):
assert record is not None and record["execution_id"] == EXECUTION_ID and record["amp_base_url"] == "https://amp.test"
- "tracing, login, says_login",
- [
- ("true", None, True), # tracing on, nobody logged in: log in
- ("true", "tok", False), # logged in: the ordinary steps
- (None, None, False), # tracing off: turn it on
- ],
-)
-def test_an_unattended_run_with_nothing_traced_says_what_would_trace_it(
- project, monkeypatch, capsys, tracing, login, says_login
-):
- """With no terminal, an anonymous run's trace stays on the machine even with
- tracing on. Telling that user to turn tracing on sends them round the same
- loop; logging in is what makes an unattended run traced."""
- directory, _ = project
- (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
- monkeypatch.setattr(eval_module, "saved_login", lambda: login)
- if tracing:
- monkeypatch.setenv("CREWAI_TRACING_ENABLED", tracing)
- else:
- monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
-
- with pytest.raises(SystemExit) as exited:
- eval_module.eval_crew()
-
- out = capsys.readouterr().out.replace("\n", " ")
- assert exited.value.code == 1
- assert ("run `crewai login`" in out) is says_login
- assert ("add CREWAI_TRACING_ENABLED=true" in out) is not says_login
-
-
-def test_an_unreadable_login_is_the_reason_given_when_nothing_was_traced(
+def test_an_unreadable_login_is_said_as_it_is_once_the_crew_has_run(
project, monkeypatch, capsys
):
- """A login that exists and cannot be read is why an unattended run was not
- traced, and its own sentence says what to do — not "turn tracing on"."""
+ """The crew runs whoever is logged in; a login that exists and cannot be
+ read stops the evaluation with its own sentence, printed as it is."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
- monkeypatch.setenv("CREWAI_TRACING_ENABLED", "true")
+ monkeypatch.setattr(
+ run_crew_module, "run_crew", lambda: record_last_run(directory, "fresh-run", recorded_at=_now())
+ )
def unreadable() -> None:
raise eval_module.EvaluationStoppedError(
@@ -1367,7 +1379,6 @@ def test_an_unreadable_login_is_the_reason_given_when_nothing_was_traced(
assert "Could not read the saved login" in out
# printed as it is: `[/Users/…]` read as markup is a closing tag, and a crash
assert "[/Users/me/.config/crewai]" in out
- assert "add CREWAI_TRACING_ENABLED=true" not in out
# ── Mode 1 files the evaluation under the project ───────────────────────────
@@ -2207,12 +2218,14 @@ def test_an_evaluation_amp_failed_in_the_run_app_is_counted_and_still_returned(
def test_an_unreadable_login_with_nobody_to_ask_counts_as_such(
project, monkeypatch, capsys, usage
):
- """Tracing on, no terminal, and a saved login that cannot be read: the login
- is the reason shown, so it is the reason counted — not "untraced"."""
+ """No terminal and a saved login that cannot be read: the crew runs, and the
+ login is the reason the evaluation stops, so it is the reason counted."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
- monkeypatch.setenv("CREWAI_TRACING_ENABLED", "true")
+ monkeypatch.setattr(
+ run_crew_module, "run_crew", lambda: record_last_run(directory, "fresh-run", recorded_at=_now())
+ )
monkeypatch.setattr(eval_module, "saved_login", _REAL_SAVED_LOGIN)
def broken_store():
@@ -2223,7 +2236,10 @@ def test_an_unreadable_login_with_nobody_to_ask_counts_as_such(
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
- assert usage == ["cli_usage:eval_stopped:login_unreadable"]
+ assert usage == [
+ "cli_usage:eval_ran_crew:no_traced_run",
+ "cli_usage:eval_stopped:login_unreadable",
+ ]
assert "crewai login" in capsys.readouterr().out
@@ -2240,8 +2256,6 @@ def test_a_start_the_run_app_was_refused_is_not_tried_again_by_the_command(
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
- monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: True)
- monkeypatch.setattr(eval_module, "_enable_tracing", lambda: None)
amp = install(monkeypatch, FakeAMP(create=httpx.Response(401, json={"detail": "Bad credentials"})))
monkeypatch.setattr(
CrewRunApp, "call_from_thread", lambda self, handler, *args: handler(*args)
@@ -2251,7 +2265,7 @@ def test_a_start_the_run_app_was_refused_is_not_tried_again_by_the_command(
holder: dict[str, str | None] = {"execution_id": None}
@contextlib.contextmanager
- def watched():
+ def watched(**_decided):
yield holder
def the_app_runs_and_evaluates() -> None:
@@ -2264,7 +2278,7 @@ def test_a_start_the_run_app_was_refused_is_not_tried_again_by_the_command(
eval_module.eval_crew()
assert [call for call in amp.calls if call[0] == "create"] == [("create", EXECUTION_ID)]
- assert usage == ["cli_usage:eval_stopped:amp_401"]
+ assert usage == ["cli_usage:eval_ran_crew:no_traced_run", "cli_usage:eval_stopped:amp_401"]
assert len(shown) == 1
@@ -2275,28 +2289,31 @@ def test_outside_a_project_counts_as_no_project(project, usage):
assert usage == ["cli_usage:eval_stopped:no_project"]
-def test_nothing_traced_with_nobody_to_ask_counts_as_untraced(project, monkeypatch, usage):
+ "gate, counted",
+ [("passed", "passed"), ("FAILED", "failed"), ("inconclusive", "inconclusive"), ("odd", "unknown")],
+)
+def test_a_run_with_nothing_traced_counts_why_the_crew_ran_and_how_it_ended(
+ project, monkeypatch, usage, gate, counted
+):
+ """Three counts: why the crew ran, that the evaluation began, and its gate —
+ so "reached a result" is counted rather than read off a missing stop."""
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
+ monkeypatch.setattr(
+ run_crew_module, "run_crew", lambda: record_last_run(directory, "fresh-run", recorded_at=_now())
+ )
+ install(monkeypatch, FakeAMP(statuses=[done(gate=gate)]))
- with pytest.raises(SystemExit, match="1"):
- eval_module.eval_crew()
-
- assert usage == ["cli_usage:eval_stopped:untraced"]
-
-
-def test_declining_to_run_with_tracing_on_counts_as_declined(project, monkeypatch, usage):
- directory, _ = project
- (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
- monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
- monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: False)
-
- with pytest.raises(SystemExit) as exited:
+ with contextlib.suppress(SystemExit): # a gate that did not pass exits 1
eval_module.eval_crew()
- assert exited.value.code == 0
- assert usage == ["cli_usage:eval_stopped:declined"]
+ assert usage == [
+ "cli_usage:eval_ran_crew:no_traced_run",
+ "cli_usage:eval",
+ f"cli_usage:eval_finished:{counted}",
+ ]
@pytest.mark.parametrize("app_saw_the_run", [True, False])
@@ -2310,12 +2327,10 @@ def test_a_run_that_left_no_trace_counts_as_untraced_once(
directory, _ = project
(directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
- monkeypatch.setattr(eval_module.click, "confirm", lambda *args, **kwargs: True)
- monkeypatch.setattr(eval_module, "_enable_tracing", lambda: None)
monkeypatch.setattr(run_crew_module, "run_crew", lambda: None)
@contextlib.contextmanager
- def watched():
+ def watched(**_decided):
yield {"execution_id": None}
monkeypatch.setattr("crewai_cli.crew_run_tui.evaluating_after_run", watched)
@@ -2328,7 +2343,10 @@ def test_a_run_that_left_no_trace_counts_as_untraced_once(
with pytest.raises(SystemExit, match="1"):
eval_module.eval_crew()
- assert usage == ["cli_usage:eval_stopped:untraced"]
+ assert usage == [
+ "cli_usage:eval_ran_crew:no_traced_run",
+ "cli_usage:eval_stopped:untraced_after_run",
+ ]
def test_a_failing_usage_count_never_changes_what_the_command_does(project, monkeypatch, capsys):
@@ -2889,3 +2907,545 @@ def test_the_written_line_stays_one_line_in_a_narrow_pipe(project, monkeypatch,
eval_module.export_evaluation(EVALUATION_ID)
assert capsys.readouterr().out == f"Wrote 5 files to {FOLDER}/ — start with report.md\n"
+
+
+# ── the crew runs again when the project changed since its last traced run ──
+
+
+def _write(path: Path, text: str = "x", seconds_ago: float = 0.0) -> Path:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(text)
+ when = datetime.now(timezone.utc).timestamp() - seconds_ago
+ os.utime(path, (when, when))
+ return path
+
+
+def _a_project_recorded(directory: Path, minutes_ago: float = 10) -> None:
+ """A crew project whose files predate its last traced run, recorded
+ MINUTES_AGO."""
+ _write(directory / "pyproject.toml", "[project]\nname = 'demo'\n", seconds_ago=3600)
+ _write(directory / "src" / "demo" / "crew.py", "crew = 1\n", seconds_ago=3600)
+ record_last_run(
+ directory, "last-traced-run", recorded_at=_seconds_ago(minutes_ago * 60)
+ )
+
+
+def test_an_unchanged_project_has_its_last_traced_run_graded_without_running(
+ project, monkeypatch, capsys
+):
+ directory, _ = project
+ _a_project_recorded(directory)
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ eval_module.eval_crew() # the autouse guard fails the test if the crew runs
+
+ assert amp.calls[0] == ("create", "last-traced-run")
+ assert "running" not in capsys.readouterr().out
+
+
+ "changed",
+ ["src/demo/crew.py", "src/demo/config/agents.yaml", "pyproject.toml", ".env", "knowledge/facts.md"],
+)
+def test_a_change_since_the_last_traced_run_runs_the_crew_again(
+ project, monkeypatch, capsys, usage, changed
+):
+ """What the evaluation would grade is no longer what the project does: a
+ coding agent that edits the crew and runs `crewai eval` gets its edit graded."""
+ directory, _ = project
+ _a_project_recorded(directory)
+ _write(directory / changed, "edited\n")
+ monkeypatch.setattr(
+ run_crew_module, "run_crew", lambda: record_last_run(directory, "the-new-run", recorded_at=_now())
+ )
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ eval_module.eval_crew()
+
+ assert amp.calls[0] == ("create", "the-new-run")
+ assert "The project changed since its last traced run: running the crew again" in capsys.readouterr().out
+ assert usage[0] == "cli_usage:eval_ran_crew:project_changed"
+
+
+ "written",
+ [
+ "eval.jsonc", # the evaluation writes it
+ ".crewai/last_eval.json",
+ ".venv/lib/site.py",
+ ".git/index",
+ ".DS_Store",
+ "src/demo/__pycache__/crew.cpython-313.pyc",
+ "node_modules/x/index.js",
+ "crewai-eval-6f31fe1a/report.md", # `crewai eval export`
+ "run.log",
+ "demo.egg-info/PKG-INFO",
+ ],
+)
+def test_what_a_run_and_its_tools_leave_around_the_project_is_no_change(
+ project, monkeypatch, written
+):
+ directory, _ = project
+ _a_project_recorded(directory)
+ _write(directory / written)
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ eval_module.eval_crew()
+
+ assert amp.calls[0] == ("create", "last-traced-run")
+
+
[email protected]("after_seconds, runs_again", [(30, False), (5 * 60, True)])
+def test_a_file_written_just_after_the_record_is_the_runs_own(
+ project, monkeypatch, after_seconds, runs_again
+):
+ """A main.py that saves the result once the crew returns writes a moment
+ after the record; an edit after an evaluation (minutes) is well past it."""
+ directory, _ = project
+ _a_project_recorded(directory, minutes_ago=10)
+ _write(directory / "report.md", seconds_ago=10 * 60 - after_seconds)
+ ran: list[str] = []
+
+ def fake_run_crew() -> None:
+ ran.append("run")
+ record_last_run(directory, "the-new-run", recorded_at=_now())
+
+ monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
+ install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ eval_module.eval_crew()
+
+ assert ran == (["run"] if runs_again else [])
+
+
+def test_a_record_without_a_time_is_graded_as_it_is(project, monkeypatch):
+ """Nothing to compare against: grading the run beats running the crew for
+ nothing."""
+ directory, _ = project
+ _write(directory / "src" / "demo" / "crew.py")
+ record_last_run(directory, "untimed-run") # no recorded_at, no finished_at
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ eval_module.eval_crew()
+
+ assert amp.calls[0] == ("create", "untimed-run")
+
+
+def test_run_again_runs_the_crew_whatever_the_record_says(project, monkeypatch, capsys, usage):
+ """The run app's Evaluate on a run that was not traced: the project's last
+ traced run is some other run, and grading it would answer the wrong press."""
+ directory, _ = project
+ _a_project_recorded(directory, minutes_ago=1)
+ monkeypatch.setattr(
+ run_crew_module, "run_crew", lambda: record_last_run(directory, "the-new-run", recorded_at=_now())
+ )
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ eval_module.eval_crew(run_again=True)
+
+ assert amp.calls[0] == ("create", "the-new-run")
+ assert "That run was not traced: running the crew again to evaluate it." in capsys.readouterr().out
+ assert usage[0] == "cli_usage:eval_ran_crew:run_again"
+
+
+ "opted_in", ["nobody", "the project's .env", "a consent given once"]
+)
+def test_tracing_is_turned_on_and_said_once_only_when_nobody_had(
+ project, monkeypatch, capsys, consent_screen, opted_in
+):
+ """With nobody at the terminal (a coding agent), the first evaluation
+ infers the yes, turns tracing on for the project and says so; from then on
+ the project traces, and nothing more is said. A project or a person who
+ turned it on already hears nothing new."""
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
+ if opted_in == "the project's .env":
+ (directory / ".env").write_text("CREWAI_TRACING_ENABLED=true\n")
+ if opted_in == "a consent given once":
+ monkeypatch.setattr("crewai_core.user_data.is_tracing_enabled", lambda: True)
+ told: list[object] = []
+
+ def fake_run_crew() -> None:
+ from crewai_core.eval_handshake import awaited_evaluation
+
+ told.append((awaited_evaluation() or {}).get("tracing_turned_on"))
+ record_last_run(directory, f"run-{len(told)}", recorded_at=_now())
+
+ monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
+ install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ eval_module.eval_crew()
+ first = " ".join(capsys.readouterr().out.split())
+ install(monkeypatch, FakeAMP(statuses=[done()]))
+ eval_module.eval_crew(run_again=True)
+ second = " ".join(capsys.readouterr().out.split())
+
+ turned_on = opted_in == "nobody"
+ assert ("Tracing is now on for this project" in first) is turned_on
+ assert "Tracing is now on for this project" not in second
+ assert told == [turned_on, False]
+ assert consent_screen.asked == [] # nobody there to ask
+ if opted_in != "a consent given once":
+ assert "CREWAI_TRACING_ENABLED=true" in (directory / ".env").read_text()
+
+
[email protected]("answer", ["yes", "no", "already on"])
+def test_at_a_terminal_tracing_is_turned_on_only_after_the_consent_screen(
+ project, monkeypatch, capsys, usage, consent_screen, answer
+):
+ """A person at the terminal is asked first, on the screen crewAI shows to
+ share a run's trace; a yes turns tracing on for the project and runs the
+ crew, a no (or no answer before it times out) runs nothing. A project that
+ traces already is not asked."""
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ if answer == "already on":
+ (directory / ".env").write_text("CREWAI_TRACING_ENABLED=true\n")
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
+ consent_screen.answer["yes"] = answer != "no"
+ ran: list[tuple[str, object]] = []
+
+ def fake_run_crew() -> None:
+ from crewai_core.eval_handshake import awaited_evaluation
+
+ said = awaited_evaluation() or {}
+ ran.append((os.environ.get("CREWAI_TRACING_ENABLED", ""), said.get("tracing_turned_on")))
+ record_last_run(directory, "the-new-run", recorded_at=_now())
+
+ monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ if answer == "no":
+ with pytest.raises(SystemExit) as exited:
+ eval_module.eval_crew()
+ assert exited.value.code == 0
+ else:
+ eval_module.eval_crew()
+ out = " ".join(capsys.readouterr().out.split())
+
+ assert consent_screen.asked == ([] if answer == "already on" else [{"sharing": True}])
+ assert "Tracing is now on" not in out # nothing inferred: they were asked, or had said
+ if answer == "no":
+ assert ran == [] and amp.calls == []
+ assert not (directory / ".env").exists()
+ assert "Nothing was run: evaluating a run needs its trace, and no trace was shared." in out
+ assert usage == ["cli_usage:eval_stopped:declined"]
+ else:
+ assert ran == [("true", False)]
+ assert "CREWAI_TRACING_ENABLED=true" in (directory / ".env").read_text()
+ assert ("Tracing is on for this project" in out) is (answer == "yes")
+ assert usage[0] == "cli_usage:eval_ran_crew:no_traced_run"
+
+
+# ── a project that turned tracing off is asked, never overruled ─────────────
+
+
[email protected]("answer", [True, False])
+def test_a_project_that_turned_tracing_off_is_asked_first_at_a_terminal(
+ project, monkeypatch, capsys, usage, answer
+):
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ (directory / ".env").write_text("CREWAI_TRACING_ENABLED=false\n")
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
+ asked: list[tuple[str, dict]] = []
+ monkeypatch.setattr(
+ eval_module.click, "confirm", lambda text, **kwargs: asked.append((text, kwargs)) or answer
+ )
+ ran: list[str] = []
+
+ def fake_run_crew() -> None:
+ ran.append(os.environ.get("CREWAI_TRACING_ENABLED", ""))
+ record_last_run(directory, "the-new-run", recorded_at=_now())
+
+ monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
+ install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ if answer:
+ eval_module.eval_crew()
+ else:
+ with pytest.raises(SystemExit) as exited:
+ eval_module.eval_crew()
+ assert exited.value.code == 0
+
+ text, kwargs = asked[0]
+ assert "Tracing is turned off for this project" in text and kwargs == {"default": True}
+ assert "sent to CrewAI AMP" in text and "CREWAI_TRACING_ENABLED" not in text
+ env = (directory / ".env").read_text()
+ if answer:
+ assert ran == ["true"]
+ assert "CREWAI_TRACING_ENABLED=true" in env # turned on, as asked
+ assert "Tracing is now on" not in capsys.readouterr().out # they just said yes
+ assert usage[0] == "cli_usage:eval_ran_crew:no_traced_run"
+ else:
+ assert ran == []
+ assert "CREWAI_TRACING_ENABLED=false" in env
+ assert "tracing stays off for this project" in capsys.readouterr().out
+ assert usage == ["cli_usage:eval_stopped:declined"]
+
+
+def test_with_tracing_turned_off_and_nobody_to_ask_it_says_how_to_turn_it_on(
+ project, monkeypatch, capsys, usage
+):
+ """A coding agent cannot answer a question; it can turn tracing on itself,
+ and the sentence says how."""
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ (directory / ".env").write_text("CREWAI_TRACING_ENABLED=0\n")
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
+ amp = install(monkeypatch, FakeAMP())
+
+ with pytest.raises(SystemExit) as exited:
+ eval_module.eval_crew() # the autouse guard fails the test if the crew runs
+
+ assert exited.value.code == 1
+ out = " ".join(capsys.readouterr().out.split())
+ assert "Tracing is turned off for this project" in out
+ assert "Set CREWAI_TRACING_ENABLED=true in the project's .env" in out
+ assert usage == ["cli_usage:eval_stopped:tracing_off"]
+ assert amp.calls == []
+
+
+ "dotenv, environment, off",
+ [
+ ("false", None, True),
+ ("0", "true", True), # the run loads the project's .env over the environment
+ ("true", "false", False),
+ (None, "false", True),
+ (None, "0", True),
+ (None, None, False),
+ (None, "true", False),
+ ],
+)
+def test_what_counts_as_tracing_turned_off(project, monkeypatch, dotenv, environment, off):
+ directory, _ = project
+ if dotenv is not None:
+ (directory / ".env").write_text(f"CREWAI_TRACING_ENABLED={dotenv}\n")
+ if environment is None:
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+ else:
+ monkeypatch.setenv("CREWAI_TRACING_ENABLED", environment)
+
+ assert eval_module.tracing_turned_off() is off
+
+
+# ── what the run graded is, and when the project changed ────────────────────
+
+
+def test_run_again_runs_the_run_the_person_pressed_evaluate_on(project, monkeypatch):
+ """Its definition and inputs travel with it; a flow named by its definition
+ needs no project around it."""
+ received: list[dict] = []
+ directory, _ = project
+
+ def fake_run_crew(**kwargs) -> None:
+ received.append(kwargs)
+ record_last_run(directory, "the-new-run", recorded_at=_now())
+
+ monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
+ install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ eval_module.eval_crew(
+ run_again=True,
+ run_args={"definition": "flows/other.yaml", "inputs": '{"topic": "bees"}'},
+ )
+
+ assert received == [{"definition": "flows/other.yaml", "inputs": '{"topic": "bees"}'}]
+
+
+def test_a_file_deleted_since_the_last_evaluation_is_a_change(project, monkeypatch, capsys):
+ """An mtime cannot say a file is gone; the list of files the last evaluation
+ kept can."""
+ directory, _ = project
+ _a_project_recorded(directory)
+ _write(directory / "src" / "demo" / "tools.py", "tool = 1\n", seconds_ago=3600)
+ install(monkeypatch, FakeAMP(statuses=[done()]))
+ eval_module.eval_crew() # grades the recorded run, and keeps the project's files
+
+ (directory / "src" / "demo" / "tools.py").unlink()
+ monkeypatch.setattr(
+ run_crew_module, "run_crew", lambda: record_last_run(directory, "the-new-run", recorded_at=_now())
+ )
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+ eval_module.eval_crew()
+
+ assert amp.calls[0] == ("create", "the-new-run")
+ assert "The project changed since its last traced run" in capsys.readouterr().out
+
+
+def test_this_commands_own_project_id_write_is_no_change(project, monkeypatch):
+ """A project without an id gets one written into pyproject.toml after the
+ check; the next `crewai eval` must not read that as an edit."""
+ directory, _ = project
+ _a_project_recorded(directory)
+
+ def writes_the_id() -> None:
+ pyproject = directory / "pyproject.toml"
+ if "project_id" not in pyproject.read_text():
+ pyproject.write_text(pyproject.read_text() + "[tool.crewai]\nproject_id = 'p-1'\n")
+
+ monkeypatch.setattr(eval_module, "get_or_create_project_id", writes_the_id)
+ install(monkeypatch, FakeAMP(statuses=[done()]))
+ eval_module.eval_crew()
+
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+ eval_module.eval_crew() # the autouse guard fails the test if the crew runs
+
+ assert amp.calls[0] == ("create", "last-traced-run")
+
+
+# ── the saved login and an AMP the project chose ─────────────────────────────
+
+
[email protected]("project_amp", ["https://amp.test", "https://other.example"])
+def test_a_rerun_from_the_run_app_trusts_only_what_the_app_says(
+ project, monkeypatch, usage, project_amp
+):
+ """The run app's process loaded the project's `.env` already, so its
+ CREWAI_PLUS_URL cannot vouch for itself: the app passes this machine's
+ AMPs, the run is told only those, and a run whose trace would carry this
+ machine's login anywhere else is not run."""
+ from crewai_core.eval_handshake import awaited_evaluation
+
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ monkeypatch.setenv("CREWAI_PLUS_URL", project_amp) # from the project's .env
+ told: list[object] = []
+
+ def fake_run_crew() -> None:
+ told.append((awaited_evaluation() or {}).get("trusted_amp_origins"))
+ record_last_run(directory, "the-new-run", recorded_at=_now())
+
+ monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
+ amp = install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ if project_amp == "https://amp.test":
+ eval_module.eval_crew(run_again=True, trusted_amp_origins={"https://amp.test"})
+ assert told == [["https://amp.test"]]
+ else:
+ with pytest.raises(SystemExit) as exited:
+ eval_module.eval_crew(run_again=True, trusted_amp_origins={"https://amp.test"})
+ assert exited.value.code == 1
+ assert told == [] and amp.calls == []
+ assert usage == ["cli_usage:eval_stopped:untrusted_amp"]
+
+
[email protected]("amp_url", ["https://evil.example", "https://amp.test"])
+def test_tracing_is_never_turned_on_toward_an_amp_the_command_does_not_trust(
+ project, monkeypatch, capsys, amp_url
+):
+ """A project's crewai older than the handshake sends the saved login to the
+ AMP its `.env` names whenever the run traces: toward an untrusted AMP the
+ command leaves tracing as it was and does not run the crew."""
+ from crewai_core.eval_handshake import awaited_evaluation
+
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ (directory / ".env").write_text(f"CREWAI_PLUS_URL={amp_url}\n")
+ seen: list[tuple] = []
+
+ def fake_run_crew() -> None:
+ said = awaited_evaluation() or {}
+ seen.append((os.environ.get("CREWAI_TRACING_ENABLED"), said.get("tracing_turned_on")))
+ record_last_run(directory, "the-new-run", recorded_at=_now())
+
+ monkeypatch.setattr(run_crew_module, "run_crew", fake_run_crew)
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: False)
+ install(monkeypatch, FakeAMP(statuses=[done()]))
+
+ trusted = amp_url == "https://amp.test"
+ if trusted:
+ eval_module.eval_crew()
+ else:
+ with pytest.raises(SystemExit):
+ eval_module.eval_crew()
+
+ assert seen == ([("true", True)] if trusted else [])
+ assert ("CREWAI_TRACING_ENABLED=true" in (directory / ".env").read_text()) is trusted
+ assert ("Tracing is now on" in capsys.readouterr().out) is trusted
+
+
+
[email protected]("at_a_terminal", [False, True])
+ "tracing_on_by",
+ ["the project's .env", "a consent given once", "nobody, or tracing=True in code"],
+)
+def test_a_crew_whose_trace_would_carry_the_login_to_an_untrusted_amp_is_not_run(
+ project, monkeypatch, capsys, usage, consent_screen, tracing_on_by, at_a_terminal
+):
+ """A project crewai older than the handshake would send this machine's
+ login to the AMP its `.env` names whenever the run traces, `tracing=True`
+ in code included, which the CLI cannot see; nor can it tell which crewai
+ the project runs. So the crew is not run, nobody is asked to turn tracing
+ on, and the sentence says how to go on."""
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ env = "CREWAI_PLUS_URL=https://evil.example\n"
+ if tracing_on_by == "the project's .env":
+ env += "CREWAI_TRACING_ENABLED=true\n"
+ elif tracing_on_by == "a consent given once":
+ monkeypatch.setattr("crewai_core.user_data.is_tracing_enabled", lambda: True)
+ (directory / ".env").write_text(env)
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: at_a_terminal)
+ amp = install(monkeypatch, FakeAMP())
+
+ with pytest.raises(SystemExit) as exited:
+ eval_module.eval_crew() # the autouse guard fails the test if the crew runs
+
+ assert exited.value.code == 1
+ out = " ".join(capsys.readouterr().out.split())
+ assert "The crew was not run: its trace would go to https://evil.example" in out
+ assert "crewai enterprise configure https://evil.example" in out
+ assert usage == ["cli_usage:eval_stopped:untrusted_amp"]
+ assert amp.calls == [] and consent_screen.asked == []
+ assert (directory / ".env").read_text() == env # nothing turned on
+
+
[email protected]("at_a_terminal", [False, True])
+def test_with_no_login_on_this_machine_yet_tracing_is_still_not_turned_on_toward_it(
+ project, monkeypatch, usage, consent_screen, at_a_terminal
+):
+ """No login to protect today is no reason to write CREWAI_TRACING_ENABLED=true
+ toward an AMP the project chose: the first run after `crewai login` would
+ carry the login there. Nothing is run, asked or written."""
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ env = "CREWAI_PLUS_URL=https://evil.example\n"
+ (directory / ".env").write_text(env)
+ monkeypatch.setattr(eval_module, "saved_login", lambda: None)
+ monkeypatch.delenv("CREWAI_USER_PAT", raising=False)
+ monkeypatch.delenv("CREWAI_ENTERPRISE_ACTION_AUTH_TOKEN", raising=False)
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: at_a_terminal)
+ amp = install(monkeypatch, FakeAMP())
+
+ with pytest.raises(SystemExit) as exited:
+ eval_module.eval_crew() # the autouse guard fails the test if the crew runs
+
+ assert exited.value.code == 1
+ assert (directory / ".env").read_text() == env
+ assert consent_screen.asked == [] and amp.calls == []
+ assert usage == ["cli_usage:eval_stopped:untrusted_amp"]
+
+
+
+def test_a_project_that_turned_tracing_off_is_not_asked_to_trace_toward_an_untrusted_amp(
+ project, monkeypatch, capsys, usage
+):
+ """A yes there could only end in the crew not being run: no question."""
+ directory, _ = project
+ (directory / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
+ (directory / ".env").write_text("CREWAI_PLUS_URL=https://evil.example\nCREWAI_TRACING_ENABLED=false\n")
+ monkeypatch.setattr(eval_module.sys.stdin, "isatty", lambda: True)
+ monkeypatch.setattr(eval_module.click, "confirm", lambda *a, **k: pytest.fail("no question"))
+ install(monkeypatch, FakeAMP())
+
+ with pytest.raises(SystemExit) as exited:
+ eval_module.eval_crew()
+
+ assert exited.value.code == 1
+ assert usage == ["cli_usage:eval_stopped:untrusted_amp"]
+ assert "CREWAI_TRACING_ENABLED=false" in (directory / ".env").read_text()
diff --git a/lib/cli/tests/test_crew_run_tui.py b/lib/cli/tests/test_crew_run_tui.py
index 4a5809817..b745d9eb5 100644
--- a/lib/cli/tests/test_crew_run_tui.py
+++ b/lib/cli/tests/test_crew_run_tui.py
@@ -312,10 +312,11 @@ def test_an_evaluation_that_stopped_can_be_tried_again(monkeypatch) -> None:
assert started == ["run-this-app"]
-def test_a_login_that_cannot_be_read_is_shown_on_screen(monkeypatch) -> None:
+def test_a_login_that_cannot_be_read_is_shown_on_screen(monkeypatch, tmp_path) -> None:
"""A credential that exists but cannot be read stops the evaluation, and
the sentence has to reach the screen: printing it would land under the
layout, and exiting would take it away."""
+ monkeypatch.chdir(tmp_path) # a stopped evaluation is noted in .crewai/
from crewai_cli.experimental import eval_crew
app = CrewRunApp()
@@ -336,9 +337,10 @@ def test_a_login_that_cannot_be_read_is_shown_on_screen(monkeypatch) -> None:
assert "crewai login" in app._evaluation["error"]
-def test_an_exit_with_no_reason_says_that_rather_than_its_code(monkeypatch) -> None:
+def test_an_exit_with_no_reason_says_that_rather_than_its_code(monkeypatch, tmp_path) -> None:
"""Nothing on this path should exit any more, but an exit carries a code
and `1` on screen is not an answer."""
+ monkeypatch.chdir(tmp_path) # a stopped evaluation is noted in .crewai/
app = CrewRunApp()
app._evaluation = {"state": "starting"}
monkeypatch.setattr(
@@ -396,23 +398,172 @@ def test_an_evaluation_that_stopped_says_so_where_it_was_running() -> None:
@pytest.mark.parametrize("why", ["not traced at all", "another run replaced the record"])
-def test_evaluate_says_so_when_this_run_was_not_traced(monkeypatch, why) -> None:
+def test_evaluate_on_a_run_that_was_not_traced_runs_it_again_traced(monkeypatch, why) -> None:
"""A run with no session of its own, or whose trace never reached AMP, has
nothing to grade — and a record naming a DIFFERENT run is not evidence that
- this one was traced."""
+ this one was traced. Tracing is on for the project, so nothing is asked:
+ the app closes and the run happens again, traced, and is evaluated
+ (`run_crew._chain_traced_rerun`)."""
+ # tracing not turned off (the suite's `.env.test` turns it off), and on
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+ # the run traces to this machine's own AMP (`.env.test` names a fake one)
+ monkeypatch.delenv("CREWAI_PLUS_URL", raising=False)
+ # A token the project's `.env` brought (a conversational flow loads it with
+ # `load_dotenv`) does not decide anything: only the AMP the run traces to.
+ monkeypatch.setenv("CREWAI_USER_PAT", "the-projects-own")
+ monkeypatch.setattr("crewai_core.user_data.is_tracing_enabled", lambda: True)
app = CrewRunApp()
app._status = "completed"
+ app._crew_result = object()
app._telemetry = Mock()
app._execution_uuid = None if why == "not traced at all" else "run-this-app"
- notices: list[str] = []
- app.notify = lambda message, **kwargs: notices.append(message) # type: ignore[method-assign]
- app.exit = lambda result=None: pytest.fail("the TUI must not leave for nothing") # type: ignore[method-assign]
+ app._unsubscribe = lambda: None # type: ignore[method-assign]
+ app.notify = lambda *a, **k: pytest.fail("no notice: it just runs again") # type: ignore[method-assign]
+ exits: list[object] = []
+ app.exit = lambda result=None: exits.append(result) # type: ignore[method-assign]
monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
app.action_evaluate_crew()
+ assert app._want_traced_rerun is True
+ assert exits == [app._crew_result]
assert app._evaluation is None
- assert "not traced" in notices[0]
+
+
+def test_a_run_crewai_eval_started_is_traced_with_nobody_at_the_screen(
+ monkeypatch, tmp_path
+) -> None:
+ """The run `crewai eval` started to grade: running the command was the yes,
+ whatever the terminal, the suite or the project's `.env` say."""
+ utils = "crewai.events.listeners.tracing.utils"
+ monkeypatch.setattr(f"{utils}._is_test_environment", lambda: True)
+ monkeypatch.setattr(f"{utils}._is_interactive_terminal", lambda: False)
+ monkeypatch.setenv("CREWAI_TRACING_ENABLED", "false")
+ app = CrewRunApp()
+
+ assert app._tracing_was_asked_for() is False
+ with crew_run_tui.evaluating_after_run():
+ assert app._tracing_was_asked_for() is True
+ assert app._tracing_was_asked_for() is False
+
+
+def test_the_token_this_app_sets_is_the_one_crewais_tracing_reads(
+ monkeypatch, tmp_path
+) -> None:
+ """The run app keeps its own copy of the handshake (it also runs against a
+ project's older crewai-core), so the two are pinned to one variable and one
+ folder: a run `crewai eval` started is approved for upload in the child."""
+ from crewai_core import eval_handshake
+
+ monkeypatch.undo() # the real folders, not the per-test ones
+ assert crew_run_tui._AWAITING_EVAL_ENV == eval_handshake.AWAITING_ENV
+ assert crew_run_tui._awaiting_dir() == eval_handshake.awaiting_dir()
+ # both pinned to the account's home: a `.env` moving HOME moves neither
+ monkeypatch.setenv("HOME", str(tmp_path / "planted-home"))
+ monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path / "planted-data"))
+ assert crew_run_tui._awaiting_dir() == eval_handshake.awaiting_dir()
+ assert not str(eval_handshake.awaiting_dir()).startswith(str(tmp_path))
+
+ folder = tmp_path / "eval-awaiting"
+ monkeypatch.setattr(crew_run_tui, "_awaiting_dir", lambda: folder)
+ monkeypatch.setattr(eval_handshake, "awaiting_dir", lambda: folder)
+ assert eval_handshake.run_awaited_by_evaluation() is False
+ with crew_run_tui.evaluating_after_run(
+ trusted_amp_origins=["https://app.crewai.com"], tracing_turned_on=True
+ ):
+ assert eval_handshake.awaited_evaluation() == {
+ "trusted_amp_origins": ["https://app.crewai.com"],
+ "tracing_turned_on": True,
+ }
+ assert eval_handshake.run_awaited_by_evaluation() is False
+
+
+ "opened_for", ["an evaluation that turned tracing on", "an evaluation", "nothing"]
+)
+def test_the_app_says_where_the_trace_goes_only_when_its_evaluation_turned_tracing_on(
+ monkeypatch, opened_for
+) -> None:
+ """Said once, plainly, the time the yes is inferred and tracing turned on
+ for the project; after that the project traces and nothing more is said."""
+ notices: list[tuple[str, dict[str, object]]] = []
+
+ def mount() -> None:
+ app = CrewRunApp()
+ app._subscribe = lambda: None # type: ignore[method-assign]
+ app.set_interval = lambda *a, **k: None # type: ignore[method-assign]
+ app.notify = lambda message, **kwargs: notices.append((message, kwargs)) # type: ignore[method-assign]
+ app.on_mount()
+
+ if opened_for == "nothing":
+ mount()
+ else:
+ with crew_run_tui.evaluating_after_run(
+ tracing_turned_on=opened_for == "an evaluation that turned tracing on"
+ ):
+ mount()
+
+ if opened_for == "an evaluation that turned tracing on":
+ assert notices == [
+ (
+ "Tracing is now on for this project, so this run can be evaluated: "
+ "its trace is sent to CrewAI AMP, which grades it.",
+ {"title": "Evaluate", "timeout": 10},
+ )
+ ]
+ else:
+ assert notices == []
+
+
+def test_a_child_app_reads_the_note_off_the_token_file(monkeypatch) -> None:
+ """In a child process there is no holder: the file the command wrote says it."""
+ notices: list[str] = []
+ with crew_run_tui.evaluating_after_run(tracing_turned_on=True):
+ token = crew_run_tui._AUTO_EVAL.set(None) # as a child process sees it
+ try:
+ app = CrewRunApp()
+ app.notify = lambda message, **kwargs: notices.append(message) # type: ignore[method-assign]
+ app._say_this_run_will_be_evaluated()
+ finally:
+ crew_run_tui._AUTO_EVAL.reset(token)
+
+ assert notices == [crew_run_tui.EVALUATION_NOTICE]
+
+
+def test_a_crew_that_sets_tracing_false_is_never_told_its_trace_is_sent() -> None:
+ notices: list[str] = []
+ with crew_run_tui.evaluating_after_run(tracing_turned_on=True):
+ app = CrewRunApp()
+ app._crew = SimpleNamespace(tracing=False)
+ app.notify = lambda message, **kwargs: notices.append(message) # type: ignore[method-assign]
+ app._say_this_run_will_be_evaluated()
+
+ assert notices == []
+
+
[email protected]("what", ["crew", "flow"])
+def test_evaluate_on_a_run_that_sets_tracing_false_says_why_and_runs_nothing(
+ monkeypatch, what
+) -> None:
+ """Running it again could not make a trace: nothing to pay for."""
+ app = CrewRunApp()
+ app._status = "completed"
+ app._telemetry = Mock()
+ app._execution_uuid = "run-this-app"
+ if what == "crew":
+ app._crew = SimpleNamespace(tracing=False)
+ else:
+ app._flow = SimpleNamespace(tracing=False)
+ app.exit = lambda result=None: pytest.fail("nothing to run again") # type: ignore[method-assign]
+ notices: list[tuple[str, dict[str, object]]] = []
+ app.notify = lambda message, **kwargs: notices.append((message, kwargs)) # type: ignore[method-assign]
+ monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
+
+ app.action_evaluate_crew()
+
+ message, kwargs = notices[0]
+ assert "sets tracing=False, so its runs are not traced and cannot be evaluated" in message
+ assert kwargs["severity"] == "warning" and app._want_traced_rerun is False
def test_tracing_asked_for_reads_the_declaration_but_keeps_the_other_guards(
@@ -2373,27 +2524,133 @@ def stops(monkeypatch) -> list[str]:
return features
-def test_pressing_evaluate_on_an_untraced_run_counts_the_press_and_why_it_stopped(
[email protected]("answer", [True, False])
+ ("tracing", "asked_on"),
+ [
+ ("turned off", crew_run_tui.TurnTracingOnScreen),
+ ("never turned on", crew_run_tui.TraceConsentScreen),
+ ],
+)
+def test_evaluate_on_an_untraced_run_asks_before_tracing_is_turned_on(
+ monkeypatch, tmp_path, stops, answer, tracing, asked_on
+) -> None:
+ """The project said no to tracing, or nobody ever turned it on: only a yes
+ turns it on (in its .env) and runs the crew again; a no leaves everything
+ as it was. A project that never turned it on is asked on the screen a
+ run's trace is shared from."""
+ monkeypatch.chdir(tmp_path)
+ if tracing == "turned off":
+ (tmp_path / ".env").write_text("CREWAI_TRACING_ENABLED=false\n")
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+ # the run traces to this machine's own AMP (`.env.test` names a fake one)
+ monkeypatch.delenv("CREWAI_PLUS_URL", raising=False)
+ monkeypatch.setattr(
+ "crewai_core.user_data.is_tracing_enabled",
+ lambda: os.environ.get("CREWAI_TRACING_ENABLED") == "true",
+ )
+ app = CrewRunApp()
+ app._status = "completed"
+ app._crew_result = object()
+ app._telemetry = Mock()
+ app._execution_uuid = "run-this-app"
+ app._unsubscribe = lambda: None # type: ignore[method-assign]
+ exits: list[object] = []
+ app.exit = lambda result=None: exits.append(result) # type: ignore[method-assign]
+ pushed: list[tuple[object, object]] = []
+ app.push_screen = lambda screen, callback=None: pushed.append((screen, callback)) # type: ignore[method-assign]
+ monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
+
+ app.action_evaluate_crew()
+
+ screen, answered = pushed[0]
+ assert type(screen) is asked_on and exits == []
+ answered(answer)
+ env = (tmp_path / ".env").read_text() if (tmp_path / ".env").exists() else ""
+ if answer:
+ assert "CREWAI_TRACING_ENABLED=true" in env
+ assert app._want_traced_rerun is True and exits == [app._crew_result]
+ assert stops == []
+ else:
+ assert env == ("CREWAI_TRACING_ENABLED=false\n" if tracing == "turned off" else "")
+ assert app._want_traced_rerun is False and exits == []
+ assert stops == ["cli_usage:eval_stopped:declined"]
+
+
[email protected]("tracing", ["turned off", "never turned on", "on"])
+def test_evaluate_toward_an_untrusted_amp_asks_nothing_and_writes_nothing(
+ monkeypatch, tmp_path, stops, tracing
+) -> None:
+ """The project's `.env` names an AMP this machine does not trust with its
+ login, logged in or not yet: `crewai eval` would refuse the re-run, so
+ nothing is asked, `.env` is left as it was, and the app says why instead
+ of closing."""
+ from crewai_cli.experimental import eval_crew
+
+ monkeypatch.chdir(tmp_path)
+ env = "CREWAI_PLUS_URL=https://evil.example\n"
+ if tracing == "turned off":
+ env += "CREWAI_TRACING_ENABLED=false\n"
+ if tracing == "on":
+ env += "CREWAI_TRACING_ENABLED=true\n"
+ (tmp_path / ".env").write_text(env)
+ monkeypatch.setenv("CREWAI_PLUS_URL", "https://evil.example") # the run's own load
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+ monkeypatch.setattr(
+ eval_crew,
+ "Settings",
+ lambda: SimpleNamespace(enterprise_base_url="https://app.crewai.com"),
+ )
+ app = CrewRunApp()
+ app._status = "completed"
+ app._crew_result = object()
+ app._telemetry = Mock()
+ app._execution_uuid = "run-this-app"
+ app._unsubscribe = lambda: None # type: ignore[method-assign]
+ exits: list[object] = []
+ app.exit = lambda result=None: exits.append(result) # type: ignore[method-assign]
+ app.push_screen = lambda *a, **k: pytest.fail("nothing is asked") # type: ignore[method-assign]
+ notices: list[str] = []
+ app.notify = lambda message, **k: notices.append(message) # type: ignore[method-assign]
+ monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
+
+ app.action_evaluate_crew()
+
+ assert exits == [] and app._want_traced_rerun is False
+ assert (tmp_path / ".env").read_text() == env
+ assert notices and "its trace would go to https://evil.example" in notices[0]
+ assert stops == ["cli_usage:eval_stopped:untrusted_amp"]
+
+
+def test_pressing_evaluate_on_an_untraced_run_counts_the_press_and_nothing_else(
monkeypatch, stops
) -> None:
+ """It is not a stop: the run happens again, and `crewai eval` counts why it
+ ran the crew (`eval_ran_crew:run_again`) and how that ended."""
+ # tracing not turned off (the suite's `.env.test` turns it off)
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+ # the run traces to this machine's own AMP (`.env.test` names a fake one)
+ monkeypatch.delenv("CREWAI_PLUS_URL", raising=False)
app = CrewRunApp()
app._status = "completed"
app._telemetry = Mock()
app._execution_uuid = "run-this-app"
+ app._unsubscribe = lambda: None # type: ignore[method-assign]
+ app.exit = lambda result=None: None # type: ignore[method-assign]
monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
- monkeypatch.setattr(CrewRunApp, "notify", lambda self, *a, **k: None)
app.action_evaluate_crew()
app._telemetry.feature_usage_span.assert_called_once_with("cli_usage:evaluate")
- assert stops == ["cli_usage:eval_stopped:untraced"]
+ assert stops == []
def test_a_run_started_by_crewai_eval_leaves_the_untraced_count_to_the_command(
- monkeypatch, stops
+ monkeypatch, tmp_path, stops
) -> None:
"""The command waiting behind this screen says "not traced" and counts it;
counting it here as well would count one stop twice."""
+ monkeypatch.chdir(tmp_path) # the app leaves its word in the project's .crewai/
monkeypatch.setattr(crew_run_tui, "_trace_was_recorded", lambda uuid: False)
monkeypatch.setattr(CrewRunApp, "notify", lambda self, *a, **k: None)
with crew_run_tui.evaluating_after_run():
diff --git a/lib/cli/tests/test_kickoff_flow.py b/lib/cli/tests/test_kickoff_flow.py
index b22201404..73c94beb2 100644
--- a/lib/cli/tests/test_kickoff_flow.py
+++ b/lib/cli/tests/test_kickoff_flow.py
@@ -98,3 +98,38 @@ def test_run_conversational_flow_tui_initializes_event_listener(monkeypatch) ->
assert result == "done"
assert calls == ["listener", "app", "run"]
+
+
+
+def test_evaluate_on_an_untraced_conversation_runs_it_again_for_its_evaluation(
+ monkeypatch,
+) -> None:
+ """The app closed because the session was not traced: the conversational
+ flow runs again through `crewai eval`'s own path rather than the request
+ being lost."""
+ from types import SimpleNamespace
+
+ from crewai_cli import kickoff_flow
+
+ class FakeApp:
+ def __init__(self, **kwargs):
+ self._status = "completed"
+ self._crew_result = "result"
+ self._want_traced_rerun = True
+
+ def run(self):
+ pass
+
+ monkeypatch.setattr("crewai_cli.crew_run_tui.CrewRunApp", FakeApp)
+ monkeypatch.setattr("crewai.events.event_listener.EventListener", lambda: None)
+ evaluated: list[dict[str, object]] = []
+ monkeypatch.setattr(
+ "crewai_cli.experimental.eval_crew.eval_crew",
+ lambda **kwargs: evaluated.append(kwargs),
+ )
+
+ chat = SimpleNamespace(name="Chat")
+ assert kickoff_flow._run_conversational_flow_tui(chat) == "result"
+ # named by its definition (`crewai run --definition chat.yaml`): that flow again
+ assert kickoff_flow._run_conversational_flow_tui(chat, definition="chat.yaml") == "result"
+ assert [e["run_args"] for e in evaluated] == [{}, {"definition": "chat.yaml"}]
diff --git a/lib/cli/tests/test_run_crew.py b/lib/cli/tests/test_run_crew.py
index f13104289..4d7587b0e 100644
--- a/lib/cli/tests/test_run_crew.py
+++ b/lib/cli/tests/test_run_crew.py
@@ -520,7 +520,7 @@ def test_missing_input_names_accepts_hyphenated_placeholders():
]
-def _patch_tui_run(monkeypatch, status: str):
+def _patch_tui_run(monkeypatch, status: str, want_traced_rerun: bool = False):
"""Stub the TUI pieces of _run_json_crew so only exit handling runs."""
monkeypatch.delenv("CREWAI_DMN", raising=False)
monkeypatch.setattr(sys.stdin, "isatty", lambda: True) # the TUI needs a terminal
@@ -530,6 +530,7 @@ def _patch_tui_run(monkeypatch, status: str):
self._status = status
self._crew_result = "result" if status == "completed" else None
self._want_deploy = False
+ self._want_traced_rerun = want_traced_rerun
def run(self):
pass
@@ -570,6 +571,36 @@ def test_run_json_crew_completed_status_returns_result(monkeypatch, tmp_path: Pa
assert run_crew_module._run_json_crew() == "result"
+def test_evaluate_on_an_untraced_run_runs_the_crew_again_for_its_evaluation(
+ monkeypatch, tmp_path: Path
+):
+ """The app closed because its run was not traced: what follows is
+ `crewai eval`'s own path, told to run the crew again whatever the record."""
+ monkeypatch.chdir(tmp_path)
+ # what the project's `.env` put in this process, as the run app's process has it
+ monkeypatch.setenv("CREWAI_PLUS_URL", "https://other.example")
+ _patch_tui_run(monkeypatch, status="completed", want_traced_rerun=True)
+ evaluated: list[dict[str, object]] = []
+ monkeypatch.setattr(
+ "crewai_cli.experimental.eval_crew.eval_crew",
+ lambda **kwargs: evaluated.append(kwargs),
+ )
+
+ assert run_crew_module._run_json_crew() == "result"
+ from crewai_cli.experimental.eval_crew import _machine_amp_origins
+
+ # the same crew, with the inputs this run used (none here), and the saved
+ # login trusted only to this machine's AMPs, never the `.env`'s
+ assert evaluated == [
+ {
+ "run_again": True,
+ "run_args": {"inputs": "{}"},
+ "trusted_amp_origins": _machine_amp_origins(),
+ }
+ ]
+ assert "https://other.example" not in evaluated[0]["trusted_amp_origins"]
+
+
def test_run_json_crew_dmn_mode_bypasses_tui(monkeypatch, tmp_path: Path, capsys):
from types import SimpleNamespace
@@ -1146,3 +1177,36 @@ def test_run_crew_runs_configured_declarative_flow_project(
assert capsys.readouterr().out == ""
assert calls == [(definition_path.resolve(), None)]
+
+
+
[email protected]("token_before", ["a" * 32, None])
+def test_a_projects_env_never_sets_or_replaces_crewai_evals_token(
+ monkeypatch, tmp_path: Path, token_before
+):
+ """A `.env` that set CREWAI_EVAL_AWAITING_RUN could unmark a run `crewai
+ eval` started (and its tracing would stop applying the command's rules)."""
+ env_file = tmp_path / ".env"
+ env_file.write_text("CREWAI_EVAL_AWAITING_RUN=" + "f" * 32 + "\nFROM_THE_PROJECT=yes\n")
+ if token_before is None:
+ monkeypatch.delenv("CREWAI_EVAL_AWAITING_RUN", raising=False)
+ else:
+ monkeypatch.setenv("CREWAI_EVAL_AWAITING_RUN", token_before)
+ monkeypatch.delenv("FROM_THE_PROJECT", raising=False)
+
+ run_crew_module.load_project_env(env_file)
+
+ assert os.environ.get("CREWAI_EVAL_AWAITING_RUN") == token_before
+ assert os.environ["FROM_THE_PROJECT"] == "yes"
+
+
+def test_the_three_names_for_the_eval_token_are_one() -> None:
+ from crewai_core import eval_handshake
+
+ from crewai_cli import crew_run_tui
+
+ assert (
+ run_crew_module.EVAL_TOKEN_ENV
+ == crew_run_tui._AWAITING_EVAL_ENV
+ == eval_handshake.AWAITING_ENV
+ )
diff --git a/lib/cli/tests/test_run_declarative_flow.py b/lib/cli/tests/test_run_declarative_flow.py
index 2fde34a34..1a3e88389 100644
--- a/lib/cli/tests/test_run_declarative_flow.py
+++ b/lib/cli/tests/test_run_declarative_flow.py
@@ -418,7 +418,9 @@ def test_id_restore_still_drops_unknown_keys(
# ── TUI vs terminal (headless/deploy) routing ──────────────────────
-def _install_fake_flow_app(monkeypatch, *, status, want_deploy=False, evaluation=None):
+def _install_fake_flow_app(
+ monkeypatch, *, status, want_deploy=False, evaluation=None, want_traced_rerun=False
+):
"""Replace CrewRunApp/EventListener/summary so _run_declarative_flow_tui is
driven by a controllable fake app."""
@@ -430,6 +432,7 @@ def _install_fake_flow_app(monkeypatch, *, status, want_deploy=False, evaluation
self._crew_name = crew_name
self._status = status
self._want_deploy = want_deploy
+ self._want_traced_rerun = want_traced_rerun
self._evaluation = evaluation
self._crew_result = "result"
@@ -471,7 +474,9 @@ def test_run_declarative_flow_interactive_uses_tui(
monkeypatch.setattr(
run_declarative_flow_module,
"_run_declarative_flow_tui",
- lambda flow, resolved: captured.update(flow=flow, inputs=resolved),
+ lambda flow, resolved, definition=None: captured.update(
+ flow=flow, inputs=resolved, definition=definition
+ ),
)
path = _write(tmp_path, REQUIRED_FLOW_YAML)
@@ -481,6 +486,7 @@ def test_run_declarative_flow_interactive_uses_tui(
assert captured["inputs"] == {"prospect_email": "[email protected]"}
assert captured["flow"] is not None
+ assert captured["definition"] == str(path) # what a traced re-run runs again
def test_run_declarative_flow_tui_failed_exits_nonzero(
@@ -526,6 +532,27 @@ def test_run_declarative_flow_tui_chains_deploy(
assert deploy_calls == [True]
+def test_evaluate_on_an_untraced_flow_run_runs_it_again_for_its_evaluation(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """The app closed because its run was not traced: what follows is
+ `crewai eval`'s own path, told to run the flow again whatever the record."""
+ _install_fake_flow_app(monkeypatch, status="completed", want_traced_rerun=True)
+ evaluated: list[dict[str, object]] = []
+ monkeypatch.setattr(
+ "crewai_cli.experimental.eval_crew.eval_crew",
+ lambda **kwargs: evaluated.append(kwargs),
+ )
+
+ run_declarative_flow_module._run_declarative_flow_tui(
+ SimpleNamespace(name="Flow"), None
+ )
+
+ assert [{k: v for k, v in e.items() if k != "trusted_amp_origins"} for e in evaluated] == [
+ {"run_again": True, "run_args": {}}
+ ]
+
+
def test_run_declarative_flow_tui_no_deploy_when_not_requested(
monkeypatch: pytest.MonkeyPatch,
) -> None:
@@ -667,7 +694,8 @@ def test_run_declarative_flow_opens_the_conversational_tui(
launched: list[Any] = []
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
- "crewai_cli.kickoff_flow._run_conversational_flow_tui", launched.append
+ "crewai_cli.kickoff_flow._run_conversational_flow_tui",
+ lambda flow, definition=None: launched.append(flow)
)
run_declarative_flow_module.run_declarative_flow(str(definition_path))
@@ -685,7 +713,7 @@ def test_conversational_flow_does_not_take_the_steps_tui(
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
- "crewai_cli.kickoff_flow._run_conversational_flow_tui", lambda flow: None
+ "crewai_cli.kickoff_flow._run_conversational_flow_tui", lambda flow, definition=None: None
)
monkeypatch.setattr(
run_declarative_flow_module,
@@ -706,7 +734,7 @@ def test_conversational_flow_rejects_inputs(
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui",
- lambda flow: pytest.fail("should have rejected --inputs before the TUI"),
+ lambda flow, definition=None: pytest.fail("should have rejected --inputs before the TUI"),
)
with pytest.raises(SystemExit) as exc_info:
@@ -730,7 +758,7 @@ def test_conversational_flow_rejects_an_empty_inputs_object(
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui",
- lambda flow: pytest.fail("an empty --inputs object still passed --inputs"),
+ lambda flow, definition=None: pytest.fail("an empty --inputs object still passed --inputs"),
)
with pytest.raises(SystemExit) as exc_info:
@@ -752,7 +780,8 @@ def test_conversational_flow_without_inputs_still_opens_the_tui(
launched: list[Any] = []
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
- "crewai_cli.kickoff_flow._run_conversational_flow_tui", launched.append
+ "crewai_cli.kickoff_flow._run_conversational_flow_tui",
+ lambda flow, definition=None: launched.append(flow)
)
run_declarative_flow_module.run_declarative_flow(str(definition_path), None)
@@ -781,7 +810,7 @@ def test_conversational_human_feedback_flow_avoids_the_tui(
monkeypatch.setattr(run_declarative_flow_module, "is_interactive", lambda: True)
monkeypatch.setattr(
"crewai_cli.kickoff_flow._run_conversational_flow_tui",
- lambda flow: pytest.fail("human-feedback flow reached the Textual TUI"),
+ lambda flow, definition=None: pytest.fail("human-feedback flow reached the Textual TUI"),
)
monkeypatch.setattr(
"crewai.flow.flow.Flow.chat", lambda self, **kw: chatted.append("repl")
diff --git a/lib/crewai-core/src/crewai_core/eval_handshake.py b/lib/crewai-core/src/crewai_core/eval_handshake.py
new file mode 100644
index 000000000..51c13a800
--- /dev/null
+++ b/lib/crewai-core/src/crewai_core/eval_handshake.py
@@ -0,0 +1,176 @@
+"""`crewai eval` and the run it starts to evaluate.
+
+When a project has no traced run to grade, or its code changed since the last
+one, `crewai eval` runs the crew itself and evaluates that run. The run has to
+know it was started for that: its trace is what the evaluation reads, so it is
+traced and sent to CrewAI AMP without asking again, whoever is at the keyboard
+(a person, or a coding agent with no terminal at all). Running `crewai eval` is
+the request to have the run graded, and the grade needs the trace.
+
+The run usually happens in a child process (`uv run …` in the project's own
+environment), which only the environment reaches. The child loads the project's
+`.env` over that environment, so a variable alone would be a switch any project
+could flip: every run approved for upload, with nobody having asked. Its value
+is therefore a random token, and it counts only while a file of that name
+exists in this user's crewAI data directory. The command writes the file before
+the run and removes it after, and a `.env` cannot create a file. Nor can it
+move the directory: it is found from this account's home in the account
+database, never from `HOME` or `XDG_DATA_HOME`, which a `.env` can set.
+
+The file also carries what the command decided before the project's `.env` was
+loaded, which the run must not take from that `.env`: the AMP origins the saved
+login may be sent to, and whether this command just turned tracing on.
+
+The CLI's run app (`crewai_cli.crew_run_tui`) keeps its own copy of the
+variable's name, the folder and the file's shape rather than importing them,
+because it also runs inside a project's environment whose crewai-core may be
+older; a test pins the two to the same values.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable, Iterator
+import contextlib
+from contextlib import contextmanager
+from ipaddress import ip_address
+import json
+import os
+from pathlib import Path
+import re
+import secrets
+import sys
+from typing import Any
+from urllib.parse import urlparse
+
+import appdirs
+
+
+AWAITING_ENV = "CREWAI_EVAL_AWAITING_RUN"
+_TOKEN = re.compile(r"[0-9a-f]{32}")
+
+
+def _account_home() -> Path:
+ """This account's home from the account database, never `$HOME`."""
+ if os.name == "posix":
+ with contextlib.suppress(ImportError, KeyError, OSError):
+ import pwd
+
+ return Path(pwd.getpwuid(os.getuid()).pw_dir)
+ return Path.home()
+
+
+def awaiting_dir() -> Path:
+ """Where the tokens live: this user's crewAI data directory, wherever the
+ environment says home is. Windows reads it from the known-folder API."""
+ if os.name == "nt":
+ return Path(appdirs.user_data_dir("crewai", "CrewAI")) / "eval-awaiting"
+ home = _account_home()
+ if sys.platform == "darwin":
+ base = home / "Library" / "Application Support"
+ else:
+ base = home / ".local" / "share" # XDG_DATA_HOME ignored on purpose
+ return base / "crewai" / "eval-awaiting"
+
+
+def awaited_evaluation() -> dict[str, Any] | None:
+ """What `crewai eval` said about this run, when it started it: None for a
+ run nobody is evaluating. Never raises: a directory or a file that cannot
+ be read means no evaluation is waiting; an older command's empty file
+ carries nothing but the request itself."""
+ value = os.environ.get(AWAITING_ENV) or ""
+ if not _TOKEN.fullmatch(value):
+ return None
+ try:
+ text = (awaiting_dir() / value).read_text(encoding="utf-8")
+ except OSError:
+ return None
+ with contextlib.suppress(ValueError):
+ said = json.loads(text)
+ if isinstance(said, dict):
+ return said
+ return {}
+
+
+def run_awaited_by_evaluation() -> bool:
+ """Was this run started by `crewai eval`, to be evaluated?"""
+ return awaited_evaluation() is not None
+
+
+def evaluation_trusts_amp(base_url: str | None) -> bool:
+ """May the saved login go to BASE_URL for this run? Only to an AMP origin
+ `crewai eval` trusted before the project's `.env` was loaded, over HTTPS or
+ plain HTTP to this machine — the rule the command applies to its own
+ requests. A run nobody is evaluating is not this rule's to decide."""
+ said = awaited_evaluation()
+ if said is None:
+ return True
+ origin = amp_origin(base_url)
+ trusted = said.get("trusted_amp_origins")
+ return (
+ origin is not None
+ and isinstance(trusted, list)
+ and origin in trusted
+ and _encrypted(origin)
+ )
+
+
+def amp_origin(url: str | None) -> str | None:
+ parsed = urlparse(str(url or ""))
+ return (
+ f"{parsed.scheme}://{parsed.netloc}".lower()
+ if parsed.scheme and parsed.netloc
+ else None
+ )
+
+
+def _encrypted(origin: str) -> bool:
+ parsed = urlparse(origin)
+ if parsed.scheme == "https":
+ return True
+ if parsed.scheme != "http":
+ return False
+ hostname = (parsed.hostname or "").rstrip(".")
+ if hostname == "localhost" or hostname.endswith(".localhost"):
+ return True
+ try:
+ return ip_address(hostname).is_loopback
+ except ValueError:
+ return False
+
+
+@contextmanager
+def awaiting_evaluation(
+ *, trusted_amp_origins: Iterable[str] = (), tracing_turned_on: bool = False
+) -> Iterator[bool]:
+ """Mark the runs started inside this block as awaited by `crewai eval`.
+
+ Yields whether the mark took: without its file a child run cannot be told,
+ and then it is traced only as the project would trace any run. The previous
+ value of the variable is put back after, and the file removed.
+ """
+ before = os.environ.get(AWAITING_ENV)
+ marker: Path | None = None
+ with contextlib.suppress(OSError):
+ nonce = secrets.token_hex(16)
+ awaiting_dir().mkdir(parents=True, exist_ok=True)
+ path = awaiting_dir() / nonce
+ with path.open("x", encoding="utf-8") as out:
+ json.dump(
+ {
+ "trusted_amp_origins": sorted(set(trusted_amp_origins)),
+ "tracing_turned_on": tracing_turned_on,
+ },
+ out,
+ )
+ marker = path
+ os.environ[AWAITING_ENV] = nonce
+ try:
+ yield marker is not None
+ finally:
+ if marker is not None:
+ with contextlib.suppress(OSError):
+ marker.unlink()
+ if before is None:
+ os.environ.pop(AWAITING_ENV, None)
+ else:
+ os.environ[AWAITING_ENV] = before
diff --git a/lib/crewai-core/tests/test_eval_handshake.py b/lib/crewai-core/tests/test_eval_handshake.py
new file mode 100644
index 000000000..de28fb1e7
--- /dev/null
+++ b/lib/crewai-core/tests/test_eval_handshake.py
@@ -0,0 +1,128 @@
+"""`crewai eval`'s word to the run it starts: a token that counts only while
+its file exists, in a folder a project's `.env` cannot move."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+from crewai_core import eval_handshake
+import pytest
+
+
[email protected](autouse=True)
+def _tokens_in_tmp(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
+ monkeypatch.setattr(
+ eval_handshake, "awaiting_dir", lambda: tmp_path / "eval-awaiting"
+ )
+ monkeypatch.delenv(eval_handshake.AWAITING_ENV, raising=False)
+
+
+def test_a_run_inside_the_block_is_awaited_and_nothing_is_left_after() -> None:
+ assert eval_handshake.run_awaited_by_evaluation() is False
+ with eval_handshake.awaiting_evaluation() as told:
+ assert told is True
+ assert eval_handshake.run_awaited_by_evaluation() is True
+ token = os.environ[eval_handshake.AWAITING_ENV]
+ assert (eval_handshake.awaiting_dir() / token).is_file()
+ assert eval_handshake.run_awaited_by_evaluation() is False
+ assert eval_handshake.AWAITING_ENV not in os.environ
+ assert list(eval_handshake.awaiting_dir().iterdir()) == []
+
+
[email protected]("value", ["1", "true", "0" * 32, "../etc/passwd", "a" * 31])
+def test_the_variable_alone_counts_for_nothing(
+ monkeypatch: pytest.MonkeyPatch, value: str
+) -> None:
+ """What a project's `.env` could set: no file stands behind it."""
+ monkeypatch.setenv(eval_handshake.AWAITING_ENV, value)
+ assert eval_handshake.run_awaited_by_evaluation() is False
+
+
+def test_the_value_before_the_block_is_put_back(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setenv(eval_handshake.AWAITING_ENV, "an-outer-value")
+ with eval_handshake.awaiting_evaluation():
+ assert os.environ[eval_handshake.AWAITING_ENV] != "an-outer-value"
+ assert os.environ[eval_handshake.AWAITING_ENV] == "an-outer-value"
+
+
+def test_a_folder_that_cannot_be_written_tells_no_run(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ blocked = tmp_path / "a-file"
+ blocked.write_text("not a folder")
+ monkeypatch.setattr(
+ eval_handshake, "awaiting_dir", lambda: blocked / "eval-awaiting"
+ )
+ with eval_handshake.awaiting_evaluation() as told:
+ assert told is False
+ assert eval_handshake.run_awaited_by_evaluation() is False
+
+
[email protected](os.name == "nt", reason="the account database is POSIX")
+def test_the_folder_is_the_accounts_whatever_the_environment_says(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ """A project's `.env` can set HOME and XDG_DATA_HOME; a token file it
+ planted under them must not count."""
+ monkeypatch.undo() # the real folder, not the per-test one
+ before = eval_handshake.awaiting_dir()
+ monkeypatch.setenv("HOME", str(tmp_path / "planted-home"))
+ monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path / "planted-data"))
+
+ assert eval_handshake.awaiting_dir() == before
+ assert not str(eval_handshake.awaiting_dir()).startswith(str(tmp_path))
+
+
+def test_the_file_carries_what_the_command_decided() -> None:
+ with eval_handshake.awaiting_evaluation(
+ trusted_amp_origins=["https://app.crewai.com", "https://app.crewai.com"],
+ tracing_turned_on=True,
+ ):
+ assert eval_handshake.awaited_evaluation() == {
+ "trusted_amp_origins": ["https://app.crewai.com"],
+ "tracing_turned_on": True,
+ }
+ assert eval_handshake.awaited_evaluation() is None
+
+
+ ("base_url", "trusted"),
+ [
+ ("https://app.crewai.com", True),
+ ("https://APP.crewai.com/", True),
+ ("https://evil.example", False), # a `.env`'s CREWAI_PLUS_URL
+ ("http://app.crewai.com", False), # trusted, but not over plain HTTP
+ ("http://localhost:3000", True),
+ (None, False),
+ ],
+)
+def test_the_saved_login_goes_only_to_an_amp_the_command_trusted(
+ base_url: str | None, trusted: bool
+) -> None:
+ with eval_handshake.awaiting_evaluation(
+ trusted_amp_origins=[
+ "https://app.crewai.com",
+ "http://app.crewai.com",
+ "http://localhost:3000",
+ ]
+ ):
+ assert eval_handshake.evaluation_trusts_amp(base_url) is trusted
+
+
+def test_a_run_nobody_is_evaluating_is_not_this_rules_to_decide() -> None:
+ assert eval_handshake.evaluation_trusts_amp("https://anywhere.example") is True
+
+
+def test_an_older_commands_empty_file_is_a_request_that_trusts_no_amp() -> None:
+ token = "a" * 32
+ eval_handshake.awaiting_dir().mkdir(parents=True)
+ (eval_handshake.awaiting_dir() / token).touch()
+ os.environ[eval_handshake.AWAITING_ENV] = token
+ try:
+ assert eval_handshake.awaited_evaluation() == {}
+ assert eval_handshake.evaluation_trusts_amp("https://app.crewai.com") is False
+ finally:
+ del os.environ[eval_handshake.AWAITING_ENV]
diff --git a/lib/crewai/src/crewai/events/listeners/tracing/utils.py b/lib/crewai/src/crewai/events/listeners/tracing/utils.py
index c58125441..c19c07042 100644
--- a/lib/crewai/src/crewai/events/listeners/tracing/utils.py
+++ b/lib/crewai/src/crewai/events/listeners/tracing/utils.py
@@ -15,6 +15,7 @@ from typing import Any, cast
import uuid
import click
+from crewai_core.eval_handshake import run_awaited_by_evaluation
from crewai_core.lock_store import lock as store_lock
from crewai_core.user_data import (
_load_user_data as _load_user_data,
@@ -122,7 +123,15 @@ def tracing_asked_for() -> bool:
failed closed in all three rather than upload what nobody approved in front
of a screen, and it still does — this turns a second question into an
answer, and changes nothing about where the first one is asked.
+
+ One run is asked for without a screen: the run `crewai eval` started to
+ evaluate. Running `crewai eval` is the request to have it graded, and the
+ grade is read off its trace, so a coding agent with no terminal approves the
+ upload by running the command. A copied `.env` cannot claim it
+ (`crewai_core.eval_handshake`).
"""
+ if run_awaited_by_evaluation():
+ return True
if not (
is_tracing_enabled_in_context()
or os.getenv("CREWAI_TRACING_ENABLED", "").lower() in ("true", "1")
@@ -138,8 +147,12 @@ def should_enable_tracing(*, override: bool | None = None) -> bool:
This is the single source of truth for tracing enablement.
Priority order:
1. Explicit override (e.g., Crew.tracing=True/False)
- 2. Environment variable CREWAI_TRACING_ENABLED
- 3. User consent from user_data
+ 2. CREWAI_TRACING_ENABLED set to false: an explicit no, which even a run
+ `crewai eval` started respects (the command asks to turn tracing on
+ first)
+ 3. A run `crewai eval` started to evaluate (its grade is read off the trace)
+ 4. CREWAI_TRACING_ENABLED set to true
+ 5. User consent from user_data
Args:
override: Explicit override for tracing (True=always enable, False=always disable, None=check other settings)
@@ -153,10 +166,12 @@ def should_enable_tracing(*, override: bool | None = None) -> bool:
return False
env_value = os.getenv("CREWAI_TRACING_ENABLED", "").lower()
- if env_value in ("true", "1"):
- return True
if env_value in ("false", "0"):
return False
+ if run_awaited_by_evaluation():
+ return True
+ if env_value in ("true", "1"):
+ return True
data = _load_user_data()
diff --git a/lib/crewai/src/crewai/execution.py b/lib/crewai/src/crewai/execution.py
index 3bf1835fa..a31b04a96 100644
--- a/lib/crewai/src/crewai/execution.py
+++ b/lib/crewai/src/crewai/execution.py
@@ -121,14 +121,33 @@ def begin_execution(
return token
+def _evaluation_trusts_this_amp() -> bool:
+ """For a run `crewai eval` started: may the saved login go to the AMP this
+ run would trace to? Any other run is not this rule's to decide (True)."""
+ from crewai_core.eval_handshake import (
+ evaluation_trusts_amp,
+ run_awaited_by_evaluation,
+ )
+ from crewai_core.plus_api import PlusAPI
+
+ if not run_awaited_by_evaluation():
+ return True
+ try:
+ base_url = PlusAPI(api_key=None).base_url
+ except Exception:
+ return False
+ return evaluation_trusts_amp(base_url)
+
+
def _tracing_disabled(tracing: bool | None) -> bool:
- return (
- os.getenv("OTEL_SDK_DISABLED", "").lower() == "true"
- or tracing is False
- or (
- tracing is None
- and os.getenv("CREWAI_TRACING_ENABLED", "").lower() in ("false", "0")
- )
+ """Tracing turned off for this run: `tracing=False`, or
+ `CREWAI_TRACING_ENABLED` false when the run says nothing.
+
+ `OTEL_SDK_DISABLED` is not one of them: it is the telemetry opt-out, a
+ separate setting, and tracing has its own."""
+ return tracing is False or (
+ tracing is None
+ and os.getenv("CREWAI_TRACING_ENABLED", "").lower() in ("false", "0")
)
@@ -156,6 +175,14 @@ def _start_tracing(execution_uuid: str, tracing: bool | None) -> None:
# The credential and its source are resolved ONCE: the pair that is sent is
# the pair a refusal names, whatever the environment says afterwards.
resolved = resolve_tracing_credential() if enabled else None
+ if resolved is not None and not _evaluation_trusts_this_amp():
+ # `crewai eval` started this run and decided, before the project's
+ # `.env` was loaded, which AMP may receive the saved login. One that
+ # `.env` chose instead gets an anonymous grant — never the login.
+ logger.info(
+ "Tracing this run anonymously: its AMP is not one crewai eval trusted"
+ )
+ resolved = None
if resolved is None:
from crewai.telemetry.tracing.ephemeral import ephemeral_tracing
diff --git a/lib/crewai/src/crewai/telemetry/tracing/session.py b/lib/crewai/src/crewai/telemetry/tracing/session.py
index 2062f8504..2546f4a55 100644
--- a/lib/crewai/src/crewai/telemetry/tracing/session.py
+++ b/lib/crewai/src/crewai/telemetry/tracing/session.py
@@ -89,6 +89,31 @@ def otlp_exporter(
)
+def _trace_provider(resource: Resource) -> TracerProvider:
+ """crewAI's own provider for a run's trace.
+
+ The OpenTelemetry SDK turns every provider into one that records nothing
+ when `OTEL_SDK_DISABLED` is set. That variable is the telemetry opt-out,
+ which tracing does not answer to — tracing has its own setting,
+ `CREWAI_TRACING_ENABLED` — so this provider records whatever it says.
+ """
+ provider = TracerProvider(resource=resource)
+ if getattr(provider, "_disabled", False):
+ # The SDK reads the variable once, in __init__, and offers no public way
+ # to say otherwise; this private attribute is its only switch. A test
+ # pins it, and a provider that still records nothing is said out loud.
+ provider._disabled = False
+ probe = provider.get_tracer(__name__).start_span("crewai.trace.probe")
+ if not probe.is_recording():
+ logger.warning(
+ "OTEL_SDK_DISABLED is set and this OpenTelemetry SDK version "
+ "ignores crewAI's override, so this run's trace records nothing. "
+ "Unset OTEL_SDK_DISABLED to trace it."
+ )
+ probe.end()
+ return provider
+
+
class TraceSession:
"""Own an event-driven trace and only its own event-bus subscriptions.
@@ -114,9 +139,7 @@ class TraceSession:
self.tracer_provider = (
providers.tracer_provider
if providers is not None
- else TracerProvider(
- resource=resource or Resource({"service.name": "crewai"})
- )
+ else _trace_provider(resource or Resource({"service.name": "crewai"}))
)
self.execution_attributes = ExecutionAttributes(execution_uuid, attributes)
self.tracer_provider.add_span_processor(self.execution_attributes)
diff --git a/lib/crewai/tests/telemetry/test_deferred_trace_transitions.py b/lib/crewai/tests/telemetry/test_deferred_trace_transitions.py
index df11bc281..808eb8677 100644
--- a/lib/crewai/tests/telemetry/test_deferred_trace_transitions.py
+++ b/lib/crewai/tests/telemetry/test_deferred_trace_transitions.py
@@ -72,7 +72,9 @@ def traces(monkeypatch):
return buffers, grants, recorders, prompt
[email protected]("disabled", ["flag", "env", "sdk"])
+# `OTEL_SDK_DISABLED` is not among them: it is the telemetry opt-out
+# (test_the_telemetry_opt_out_does_not_turn_tracing_off).
[email protected]("disabled", ["flag", "env"])
@pytest.mark.parametrize("finalize_only", [False, True])
def test_disabling_deferred_trace_discards_without_consent(
traces, monkeypatch, disabled, finalize_only
@@ -97,10 +99,8 @@ def test_disabling_deferred_trace_discards_without_consent(
assert old_lifetime is not None and buffers[0]._spans
if disabled == "flag":
flow.tracing = False
- elif disabled == "env":
- monkeypatch.setenv("CREWAI_TRACING_ENABLED", "false")
else:
- monkeypatch.setenv("OTEL_SDK_DISABLED", "true")
+ monkeypatch.setenv("CREWAI_TRACING_ENABLED", "false")
if not finalize_only:
assert flow.kickoff() == "private draft"
assert sessions[1] is None
@@ -178,6 +178,178 @@ def test_where_the_prompt_could_not_be_shown_nothing_is_uploaded(
assert not grants and not recorders # nothing left the machine
+ "closed", ["no terminal", "under test", "messages suppressed"]
+)
+def test_a_run_crewai_eval_started_is_uploaded_with_nobody_to_ask(
+ traces, monkeypatch, tmp_path, closed
+):
+ """`crewai eval` ran the crew to grade it, and the grade is read off the
+ trace: running the command was the yes. A coding agent with no terminal
+ gets its run uploaded."""
+ from crewai_core import eval_handshake
+
+ buffers, grants, recorders, prompt = traces
+ monkeypatch.setattr(eval_handshake, "awaiting_dir", lambda: tmp_path / "eval-awaiting")
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+ utils = "crewai.events.listeners.tracing.utils"
+ monkeypatch.setattr(f"{utils}._is_interactive_terminal", lambda: closed != "no terminal")
+ monkeypatch.setattr(f"{utils}._is_test_environment", lambda: closed == "under test")
+ monkeypatch.setattr(
+ f"{utils}.should_suppress_tracing_messages",
+ lambda: closed == "messages suppressed",
+ )
+
+ class Conversation(Flow):
+ @start()
+ def turn(self):
+ return "graded"
+
+ with eval_handshake.awaiting_evaluation() as told:
+ assert told
+ assert Conversation().kickoff() == "graded"
+ assert crewai_event_bus.flush()
+
+ prompt.assert_not_called()
+ assert grants and recorders # the trace reached AMP
+
+
[email protected]("off", ["false", "0"])
+def test_a_project_that_turned_tracing_off_is_not_traced_even_for_crewai_eval(
+ traces, monkeypatch, tmp_path, off
+):
+ """An explicit no is the project's; `crewai eval` asks to turn tracing on
+ before it runs anything, and nothing here overrules it."""
+ from crewai_core import eval_handshake
+
+ buffers, grants, recorders, prompt = traces
+ monkeypatch.setattr(eval_handshake, "awaiting_dir", lambda: tmp_path / "eval-awaiting")
+ monkeypatch.setenv("CREWAI_TRACING_ENABLED", off)
+ monkeypatch.setattr(
+ "crewai.events.listeners.tracing.utils._is_interactive_terminal", lambda: False
+ )
+
+ class Conversation(Flow):
+ @start()
+ def turn(self):
+ return "kept here"
+
+ with eval_handshake.awaiting_evaluation():
+ assert Conversation().kickoff() == "kept here"
+ assert crewai_event_bus.flush()
+
+ assert not buffers and not grants and not recorders # never even traced
+
+
+ ("amp", "tier"),
+ [("https://app.crewai.com", "authenticated"), ("https://evil.example", "ephemeral")],
+)
+def test_the_saved_login_goes_only_to_an_amp_crewai_eval_trusted(
+ traces, monkeypatch, tmp_path, amp, tier
+):
+ """`crewai eval` decides which AMP may receive the saved login before the
+ project's `.env` is loaded; an AMP that `.env` chose gets an anonymous
+ grant for the run the command started, never the login."""
+ from crewai.telemetry.tracing.grants import TraceGrantClient
+ from crewai_core import eval_handshake
+
+ # `traces` installs the scripted grants and exporters
+ monkeypatch.setattr(eval_handshake, "awaiting_dir", lambda: tmp_path / "eval-awaiting")
+ monkeypatch.delenv("CREWAI_TRACING_ENABLED", raising=False)
+ monkeypatch.setenv("CREWAI_PLUS_URL", amp)
+ monkeypatch.setattr(
+ "crewai.telemetry.tracing.grants.resolve_tracing_credential",
+ lambda: ("login", "the-saved-login"),
+ )
+ monkeypatch.setattr(
+ "crewai.events.listeners.tracing.utils._is_interactive_terminal", lambda: False
+ )
+ tiers: list[str] = []
+ real_create = TraceGrantClient.create
+ monkeypatch.setattr(
+ TraceGrantClient,
+ "create",
+ lambda client, execution_uuid: tiers.append(client._tier)
+ or real_create(client, execution_uuid),
+ )
+
+ class Conversation(Flow):
+ @start()
+ def turn(self):
+ return "graded"
+
+ with eval_handshake.awaiting_evaluation(
+ trusted_amp_origins=["https://app.crewai.com"]
+ ):
+ assert Conversation().kickoff() == "graded"
+ assert crewai_event_bus.flush()
+
+ assert tiers == [tier]
+
+
+def test_a_token_a_dotenv_could_set_approves_nothing(traces, monkeypatch, tmp_path):
+ """The variable alone is what a project's `.env` could carry; without the
+ file `crewai eval` writes, it counts for nothing."""
+ from crewai_core import eval_handshake
+
+ buffers, grants, recorders, prompt = traces
+ monkeypatch.setattr(eval_handshake, "awaiting_dir", lambda: tmp_path / "eval-awaiting")
+ monkeypatch.setenv(eval_handshake.AWAITING_ENV, "0" * 32)
+ monkeypatch.setenv("CREWAI_TRACING_ENABLED", "true")
+ utils = "crewai.events.listeners.tracing.utils"
+ monkeypatch.setattr(f"{utils}._is_interactive_terminal", lambda: False)
+ prompt.return_value = False
+
+ class Conversation(Flow):
+ @start()
+ def turn(self):
+ return "in a container"
+
+ assert Conversation(tracing=True).kickoff() == "in a container"
+ assert crewai_event_bus.flush()
+
+ assert not grants and not recorders # nothing left the machine
+
+
[email protected]("anonymous", [True, False])
+def test_the_telemetry_opt_out_does_not_turn_tracing_off(traces, monkeypatch, anonymous):
+ """`OTEL_SDK_DISABLED` is the telemetry opt-out, a separate setting: a run
+ with tracing on is still traced and uploaded, every span recorded."""
+ buffers, grants, recorders, prompt = traces
+ monkeypatch.setenv("OTEL_SDK_DISABLED", "true")
+ utils = "crewai.events.listeners.tracing.utils"
+ monkeypatch.setattr(f"{utils}._is_interactive_terminal", lambda: True)
+ monkeypatch.setattr(f"{utils}._is_test_environment", lambda: False)
+ if not anonymous:
+ monkeypatch.setattr(
+ "crewai.telemetry.tracing.grants.resolve_tracing_credential",
+ lambda: ("a-login", "login"),
+ )
+
+ class Conversation(Flow):
+ @start()
+ def turn(self):
+ return "traced"
+
+ assert Conversation(tracing=True).kickoff() == "traced"
+ assert crewai_event_bus.flush()
+
+ assert grants and recorders
+ assert any(recorder.get_finished_spans() for recorder in recorders)
+
+
+def test_crewais_trace_provider_records_with_the_sdk_switched_off(monkeypatch):
+ """Pins the SDK detail the provider relies on (its private `_disabled`)."""
+ from crewai.telemetry.tracing.session import _trace_provider
+ from opentelemetry.sdk.resources import Resource
+
+ monkeypatch.setenv("OTEL_SDK_DISABLED", "true")
+ span = _trace_provider(Resource({"service.name": "crewai"})).get_tracer("t").start_span("s")
+ assert span.is_recording()
+ span.end()
+
+
def test_enabling_deferred_trace_opens_a_new_flow_root(traces):
buffers, grants, recorders, prompt = traces
sessions = []
diff --git a/lib/crewai/tests/telemetry/test_session_trace_export.py b/lib/crewai/tests/telemetry/test_session_trace_export.py
index 1c5b989e1..48f10189c 100644
--- a/lib/crewai/tests/telemetry/test_session_trace_export.py
+++ b/lib/crewai/tests/telemetry/test_session_trace_export.py
@@ -746,7 +746,9 @@ def test_first_time_execution_uses_local_session_even_with_saved_credentials(
assert len(collector.grants) == int(approved)
[email protected]("disabled_by", ["override", "environment", "sdk"])
+# `OTEL_SDK_DISABLED` is not among them: it is the telemetry opt-out
+# (test_deferred_trace_transitions.py::test_the_telemetry_opt_out_does_not_turn_tracing_off).
[email protected]("disabled_by", ["override", "environment"])
def test_disabled_execution_does_not_collect_or_request_grants(
collector, monkeypatch, disabled_by
):
@@ -759,8 +761,6 @@ def test_disabled_execution_does_not_collect_or_request_grants(
)
if disabled_by == "environment":
monkeypatch.setenv("CREWAI_TRACING_ENABLED", "false")
- elif disabled_by == "sdk":
- monkeypatch.setenv("OTEL_SDK_DISABLED", "true")
consent = Mock(return_value=True)
with trace_consent(consent):
token = begin_execution(tracing=False if disabled_by == "override" else None)
diff --git a/lib/crewai/tests/telemetry/test_trace_lifecycle.py b/lib/crewai/tests/telemetry/test_trace_lifecycle.py
index 1cd75a426..ff1e010fa 100644
--- a/lib/crewai/tests/telemetry/test_trace_lifecycle.py
+++ b/lib/crewai/tests/telemetry/test_trace_lifecycle.py
@@ -305,7 +305,8 @@ def test_first_time_uses_local_session_even_with_saved_login(
assert get_trace_session() is None and get_execution_uuid() is None
[email protected]("disabled", ["flag", "env", "sdk"])
+# `OTEL_SDK_DISABLED` is not a tracing setting: it is the telemetry opt-out.
[email protected]("disabled", ["flag", "env"])
def test_explicit_disable_prevents_first_time_collection(monkeypatch, disabled):
monkeypatch.setattr(
"crewai.events.listeners.tracing.utils.should_auto_collect_first_time_traces",
@@ -316,8 +317,6 @@ def test_explicit_disable_prevents_first_time_collection(monkeypatch, disabled):
monkeypatch.setenv("CREWAI_USER_PAT", "synthetic-pat")
if disabled == "env":
monkeypatch.setenv("CREWAI_TRACING_ENABLED", "false")
- if disabled == "sdk":
- monkeypatch.setenv("OTEL_SDK_DISABLED", "true")
flow = ExampleFlow(tracing=False if disabled == "flag" else None)
assert flow.kickoff() == "hello world"
grant.assert_not_called()