Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
Python agent-evaluationaiai-alignmentai-evaluationai-governanceai-safetyclidiagnostic-tooleu-ai-acthallucination-detectioniso-42001llm-evaluationllm-securitynist-ai-rmfowasp-llmprompt-injectionpythonresponsible-airisk-assessmentrisk-management
chore: pro audit (#223)
stefyi-4355db7e2d49a8f4
diff --git a/docs/cli.md b/docs/cli.mdindex cf1d220..3f6cb5f 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -73,11 +73,32 @@ ifixai run -p openai -k "$OPENAI_API_KEY" -c DECEPTION # example: one category| `--concurrency`, `-j` | `5` | Max in-flight LLM requests (1-20). Overrides `IFIXAI_CONCURRENCY`. || `--no-parallel` | off | Alias for `--concurrency 1`. || `--dry-run` | off | Print inspection and judge-call estimates, then exit. |+| `--no-promo` | off | Skip the [Pro-version offer](#pro-version-offer) and start the run immediately. `IFIXAI_NO_PROMO=1` skips it on every run. || `--reliability-out` | `runs` | Directory for `manifest.json`, one subdir per run. || `--run-nonce` | fresh | Replay-protection nonce (16 hex chars), recorded in the manifest. || `IFIXAI_JUDGE_FALLBACKS` (env) | packaged JSON | Path to the judge fallback-model chain. See [Judge fallback models](#judge-fallback-models). || `--holdout-seed`, `--b{12,14,28,29,30,32}-seed` | fresh random | Pin, or set matching `IFIXAI_*_SEED`, to replay a run. See [reproducibility.md](reproducibility.md). |+## Pro-version offer++The first interactive `ifixai run` on a machine opens with one question: whether you'd like to claim your free fast audit at <https://www.ifixai.ai/>. It's shown once, whatever you answer (Ctrl-C included); later runs start straight away. The marker is a `pro-offer-shown` file in `$XDG_CONFIG_HOME/ifixai` (else `~/.config/ifixai`): delete it to see the offer again.++Move with the arrow keys and press Enter:++- **No** (preselected, just press Enter): the run starts.+- **Yes**: the site opens in your browser, then you pick **Continue the current run** (preselected) or **Abandon the run**. Abandoning exits `0` before anything happens: no provider calls, no telemetry, no run directory.++Ctrl-C on either question aborts without starting the run. Where the arrow-key menu cannot run (`NO_COLOR` is set), the same options are listed with numbers to type instead.++To start immediately, pass `--no-promo`, or set `IFIXAI_NO_PROMO=1` to skip it on every run:++```bash+ifixai run --provider openai --no-promo+export IFIXAI_NO_PROMO=1+```++The offer is never shown when stdin or stdout is not a terminal (pipes, scripts, agent-driven runs), in CI, or on `--dry-run`, so automated runs are never blocked by it. `--quiet` does not skip it; use `--no-promo`. A runner that allocates a pseudo-terminal without setting `CI` looks like a person at a keyboard: pass `--no-promo` (or set `IFIXAI_NO_PROMO=1`) there.+## How a run is judgedThe SUT (system under test) is the agent being graded via the SUT flags above; its key is never read from the environment. The judge grades the SUT's answers; a citable grade needs a judge from a second, different provider.diff --git a/ifixai/cli/_branding.py b/ifixai/cli/_branding.pyindex 381f752..f4a1d72 100644--- a/ifixai/cli/_branding.py+++ b/ifixai/cli/_branding.py@@ -109,7 +109,7 @@ def print_startup_banner(version: str, *, quiet: bool = False) -> None:for line in _LOGO_LINES:click.echo(" " + _truecolor(line, _ACCENT_RGB, bold=True))click.echo()- click.echo(_truecolor(f" ™ · v{version} · powered by iMe", _DIM_RGB))+ click.echo(_truecolor(f" ™ · v{version}", _DIM_RGB))click.echo()diff --git a/ifixai/cli/pro_offer.py b/ifixai/cli/pro_offer.pynew file mode 100644index 0000000..9373fde--- /dev/null+++ b/ifixai/cli/pro_offer.py@@ -0,0 +1,147 @@+"""Pro-version offer shown at the start of an interactive ``ifixai run``.++The offer is made once per machine, before telemetry and before any provider+call, so a run the user abandons for the website leaves no trace beyond the+``pro-offer-shown`` marker. It is never shown when stdin or stdout is not a+terminal, or in CI, so scripted and agent-driven runs cannot block on it. Skip+it with ``--no-promo`` or ``IFIXAI_NO_PROMO=1``.+"""++import sys+import webbrowser+from enum import Enum+from typing import TextIO, TypedDict++import click++from ifixai import telemetry+from ifixai.cli import ui++PRO_URL = "https://www.ifixai.ai/"+NO_PROMO_ENV_VAR = "IFIXAI_NO_PROMO"+OFFER_MARKER_NAME = "pro-offer-shown"++YES_OPTION = "Yes"+NO_OPTION = "No"+CONTINUE_OPTION = "Continue the current run"+ABANDON_OPTION = "Abandon the run"+++class RunDecision(str, Enum):+ """What the user wants done with the current run after seeing the offer."""++ CONTINUE = "continue"+ ABANDON = "abandon"+++class ChoiceQuestion(TypedDict):+ """One question with a fixed set of answers; Enter alone picks ``default``."""++ message: str+ options: tuple[str, ...]+ default: str+++PRO_INTEREST_QUESTION = ChoiceQuestion(+ message="Claim your free fast audit now?",+ options=(YES_OPTION, NO_OPTION),+ default=NO_OPTION,+)+RUN_DECISION_QUESTION = ChoiceQuestion(+ message="What would you like to do with this run?",+ options=(CONTINUE_OPTION, ABANDON_OPTION),+ default=CONTINUE_OPTION,+)+++def offer_pro_version() -> RunDecision:+ """Offer the Pro version and report whether the current run should go ahead.++ Opens the Pro website when the user accepts. Returns ``CONTINUE`` without+ asking anything when nobody is at a terminal to answer or it was already shown.+ """+ if not can_prompt_user() or not mark_offer_shown():+ return RunDecision.CONTINUE+ print_offer()+ if ask_choice(PRO_INTEREST_QUESTION) != YES_OPTION:+ return RunDecision.CONTINUE+ open_pro_site()+ if ask_choice(RUN_DECISION_QUESTION) == ABANDON_OPTION:+ return RunDecision.ABANDON+ return RunDecision.CONTINUE+++def can_prompt_user() -> bool:+ """True only when a person is at a terminal to answer the offer."""+ return (+ is_terminal(sys.stdin) and is_terminal(sys.stdout) and not telemetry.in_ci()+ )+++def mark_offer_shown() -> bool:+ """Create the shown-once marker; False if it exists or can't be written.++ Marked before asking, so any answer (Ctrl-C included) counts. When it can't+ be written the offer is skipped, so a looping script never stops on it twice.+ """+ try:+ (telemetry._ensure_config_dir() / OFFER_MARKER_NAME).touch(exist_ok=False)+ except OSError:+ return False+ return True+++def is_terminal(stream: TextIO | None) -> bool:+ """True when the stream exists and is attached to a terminal.++ Python sets a standard stream to ``None`` when the process has no handle+ for it (a closed descriptor, ``pythonw``, a detached Windows process).+ """+ return stream is not None and stream.isatty()+++def print_offer() -> None:+ """Print the Pro-version pitch and its link."""+ click.echo(click.style(" iFixAi Pro", bold=True))+ click.echo(" Claim your free fast audit")+ click.echo(click.style(f" {PRO_URL}", fg="cyan"))+ click.echo()+++def open_pro_site() -> None:+ """Open the Pro website in the default browser; print the link when that fails."""+ if webbrowser.open(PRO_URL):+ click.echo(click.style(f"Opened {PRO_URL} in your browser.", fg="green"))+ return+ click.echo(+ click.style(+ f"Could not open a browser. Visit {PRO_URL} to claim your free fast audit.",+ fg="yellow",+ )+ )+++def ask_choice(question: ChoiceQuestion) -> str:+ """Ask with an arrow-key menu, or a numbered list where menus are unavailable."""+ if ui.is_interactive():+ return ui.select_or_abort(+ question["message"], question["options"], question["default"]+ )+ return ask_numbered_choice(question)+++def ask_numbered_choice(question: ChoiceQuestion) -> str:+ """List the options with numbers and read the number the user types.++ The plain path for terminals where the arrow-key menu cannot run: ``NO_COLOR``+ is set, or ``questionary`` is not installed.+ """+ options = question["options"]+ for number, option in enumerate(options, start=1):+ click.echo(f" [{number}] {option}")+ typed_number = click.prompt(+ question["message"],+ type=click.IntRange(1, len(options)),+ default=options.index(question["default"]) + 1,+ )+ return options[typed_number - 1]diff --git a/ifixai/cli/run.py b/ifixai/cli/run.pyindex b93d063..a72dc2d 100644--- a/ifixai/cli/run.py+++ b/ifixai/cli/run.py@@ -35,6 +35,7 @@ from ifixai.cli.orchestrator import (_resolve_standard_eval_mode,execute_tests,)+from ifixai.cli.pro_offer import NO_PROMO_ENV_VAR, RunDecision, offer_pro_versionfrom ifixai.cli.reports import save_reportsfrom ifixai.cli.schemas import InteractiveConfigfrom ifixai.core.concurrency import (@@ -734,6 +735,16 @@ def _validate_min_score(ctx: click.Context, param: click.Parameter, value: floathelp="Disable pseudonymous run telemetry for this run. Set IFIXAI_TELEMETRY=0 or ""DO_NOT_TRACK=1 to disable it permanently.",)+ "--no-promo",+ "no_promo",+ is_flag=True,+ default=False,+ envvar=NO_PROMO_ENV_VAR,+ help="Skip the Pro-version offer and start the run immediately. Set "+ f"{NO_PROMO_ENV_VAR}=1 to skip it on every run. The offer is never shown "+ "in CI or when input/output is not a terminal.",+)@click.option("--print-telemetry","print_telemetry",@@ -796,6 +807,7 @@ def run(grounding: str,quiet: bool,no_telemetry: bool,+ no_promo: bool,print_telemetry: bool,show_install_id: bool,) -> None:@@ -938,6 +950,10 @@ def run(holdout_seed = resume_manifest.holdout_seedprint_startup_banner(IFIXAI_VERSION, quiet=quiet)+ # Asked before telemetry and any provider call, so abandoning leaves no trace.+ if not (no_promo or dry_run) and offer_pro_version() is RunDecision.ABANDON:+ click.echo("Run abandoned -- no inspections were run.")+ returnif no_telemetry:telemetry.disable()telemetry.show_disclosure()diff --git a/ifixai/cli/ui.py b/ifixai/cli/ui.pyindex b4ad310..f94878d 100644--- a/ifixai/cli/ui.py+++ b/ifixai/cli/ui.py@@ -16,6 +16,7 @@ from ifixai.cli._branding import supports_color_ACCENT = "rgb(232,99,42)"_DIM = "grey62"+SELECT_INSTRUCTION = "(↑/↓ to move, enter to select)"@lru_cache(maxsize=1)@@ -109,7 +110,7 @@ def select(default=default or choices[0],style=_qstyle(),use_shortcuts=False,- instruction="(↑/↓ to move, enter to select)",+ instruction=SELECT_INSTRUCTION,)if descriptions:kwargs["show_description"] = True@@ -117,6 +118,25 @@ def select(return answer if answer is not None else (default or choices[0])+def select_or_abort(message: str, choices: Sequence[str], default: str) -> str:+ """Arrow-key single choice where Ctrl-C aborts instead of picking the default.++ ``select`` answers a cancelled prompt with its default, which is wrong when+ the default starts work the user just tried to stop. Raises+ ``KeyboardInterrupt`` on Ctrl-C. Callers must check ``is_interactive()`` first.+ """+ import questionary++ return questionary.select(+ message,+ choices=_choices(list(choices), None),+ default=default,+ style=_qstyle(),+ use_shortcuts=False,+ instruction=SELECT_INSTRUCTION,+ ).unsafe_ask()++def multiselect(message: str,choices: Sequence[str],diff --git a/ifixai/telemetry.py b/ifixai/telemetry.pyindex 617b98e..bd169fe 100644--- a/ifixai/telemetry.py+++ b/ifixai/telemetry.py@@ -111,7 +111,8 @@ def disable() -> None:_force_disabled = True-def _in_ci() -> bool:+def in_ci() -> bool:+ """True when the process is running under an automated CI system."""ci = os.environ.get("CI")if ci is not None and ci.strip().lower() not in {"", "0", "false", "no"}:return True@@ -127,7 +128,7 @@ def is_enabled() -> bool:return Falseif "DO_NOT_TRACK" in os.environ: # presence-based, any value incl. "0"return False- if _in_ci():+ if in_ci():return Falsetry:if _optout_path().exists():diff --git a/ifixai/tests/test_cli_arrow_menu.py b/ifixai/tests/test_cli_arrow_menu.pynew file mode 100644index 0000000..78d8c55--- /dev/null+++ b/ifixai/tests/test_cli_arrow_menu.py@@ -0,0 +1,102 @@+"""Real keystrokes through the arrow-key menu (needs ``questionary`` installed)."""++from collections.abc import Iterator+from pathlib import Path+from typing import Any++import pytest++from ifixai.cli import pro_offer, ui+from ifixai.cli.pro_offer import PRO_URL, RunDecision++pytest.importorskip("questionary")+application = pytest.importorskip("prompt_toolkit.application")+keyboard = pytest.importorskip("prompt_toolkit.input")+screen = pytest.importorskip("prompt_toolkit.output")++UP_ARROW = "\x1b[A"+DOWN_ARROW = "\x1b[B"+ENTER = "\r"+CTRL_C = "\x03"+++class BrowserRecorder:+ """Stands in for ``webbrowser.open`` and records the URLs it was asked for."""++ def __init__(self) -> None:+ self.opened_urls: list[str] = []++ def __call__(self, url: str) -> bool:+ self.opened_urls.append(url)+ return True+++def report_terminal_attached() -> bool:+ return True++[email protected](autouse=True)+def config_home(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ """Keeps the offer's shown-once marker in ``tmp_path``, never the real home."""+ monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path))+++def keys() -> Iterator[Any]:+ """A keyboard the test types on; the menus read from it instead of a console."""+ with keyboard.create_pipe_input() as pipe:+ with application.create_app_session(input=pipe, output=screen.DummyOutput()):+ yield pipe+++def test_enter_alone_picks_the_default(keys: Any) -> None:+ keys.send_text(ENTER)++ assert ui.select_or_abort("Pick one", ["first", "second"], "second") == "second"+++def test_arrows_move_the_pointer_before_enter(keys: Any) -> None:+ keys.send_text(UP_ARROW + ENTER)++ assert ui.select_or_abort("Pick one", ["first", "second"], "second") == "first"+++def test_ctrl_c_raises_instead_of_answering_with_the_default(keys: Any) -> None:+ keys.send_text(CTRL_C)++ with pytest.raises(KeyboardInterrupt):+ ui.select_or_abort("Pick one", ["first", "second"], "second")+++def test_the_existing_select_still_answers_a_cancel_with_its_default(+ keys: Any, monkeypatch: pytest.MonkeyPatch+) -> None:+ monkeypatch.setattr(ui, "is_interactive", report_terminal_attached)+ keys.send_text(CTRL_C)++ assert ui.select("Pick one", ["first", "second"], default="second") == "second"+++ "typed,expected,expected_urls",+ [+ (ENTER, RunDecision.CONTINUE, []),+ (UP_ARROW + ENTER + ENTER, RunDecision.CONTINUE, [PRO_URL]),+ (UP_ARROW + ENTER + DOWN_ARROW + ENTER, RunDecision.ABANDON, [PRO_URL]),+ ],+)+def test_the_offer_end_to_end_with_real_keystrokes(+ keys: Any,+ monkeypatch: pytest.MonkeyPatch,+ typed: str,+ expected: RunDecision,+ expected_urls: list[str],+) -> None:+ browser = BrowserRecorder()+ monkeypatch.setattr(pro_offer.webbrowser, "open", browser)+ monkeypatch.setattr(pro_offer, "can_prompt_user", report_terminal_attached)+ monkeypatch.setattr(ui, "is_interactive", report_terminal_attached)+ keys.send_text(typed)++ assert pro_offer.offer_pro_version() is expected+ assert browser.opened_urls == expected_urlsdiff --git a/ifixai/tests/test_cli_pro_offer.py b/ifixai/tests/test_cli_pro_offer.pynew file mode 100644index 0000000..41c2ae8--- /dev/null+++ b/ifixai/tests/test_cli_pro_offer.py@@ -0,0 +1,536 @@+"""The Pro-version offer shown at the start of an interactive ``ifixai run``."""++import io+import sys+from collections.abc import Sequence+from pathlib import Path++import click+import pytest+from click.testing import CliRunner++import ifixai.cli.run as run_module+from ifixai import telemetry+from ifixai.cli import pro_offer, ui+from ifixai.cli.main import ifixai_cli+from ifixai.cli.pro_offer import (+ ABANDON_OPTION,+ CONTINUE_OPTION,+ NO_OPTION,+ NO_PROMO_ENV_VAR,+ PRO_INTEREST_QUESTION,+ PRO_URL,+ YES_OPTION,+ RunDecision,+)++UNKNOWN_TEST_ID = "ZZ99"+UNKNOWN_TEST_ERROR = "unknown test ID"+ABANDONED_MESSAGE = "Run abandoned"+OFFER_LINE = " Claim your free fast audit\n"+OFFER_QUESTION = "Claim your free fast audit now?"+RETIRED_COPY = ("free sample", "Try the Pro version")+SKIPPED_RUNS = [+ (["--no-promo"], {}),+ ([], {NO_PROMO_ENV_VAR: "1"}),+ (["--dry-run"], {}),+]+++class FakeStream(io.StringIO):+ """A text stream that reports whether it is attached to a terminal.++ ``typed_text`` is what a user would have typed, for a stream used as stdin.+ """++ def __init__(self, is_terminal: bool, typed_text: str = "") -> None:+ super().__init__(typed_text)+ self.is_terminal = is_terminal++ def isatty(self) -> bool:+ return self.is_terminal+++class BrowserRecorder:+ """Stands in for ``webbrowser.open`` and records the URLs it was asked for."""++ def __init__(self, is_launch_successful: bool) -> None:+ self.is_launch_successful = is_launch_successful+ self.opened_urls: list[str] = []++ def __call__(self, url: str) -> bool:+ self.opened_urls.append(url)+ return self.is_launch_successful+++class OfferRecorder:+ """Stands in for ``offer_pro_version`` and counts how often the run asked."""++ def __init__(self, decision: RunDecision) -> None:+ self.decision = decision+ self.call_count = 0++ def __call__(self) -> RunDecision:+ self.call_count += 1+ return self.decision+++class CallCounter:+ """Stands in for a no-argument side effect and counts its calls."""++ def __init__(self) -> None:+ self.call_count = 0++ def __call__(self) -> None:+ self.call_count += 1+++class ScriptedMenu:+ """Stands in for the arrow-key menu and answers each question in order."""++ def __init__(self, answers: list[str]) -> None:+ self.remaining_answers = iter(answers)+ self.defaults: list[str] = []++ def __call__(self, message: str, choices: Sequence[str], default: str) -> str:+ self.defaults.append(default)+ return next(self.remaining_answers)+++class CancelledMenu:+ """Stands in for the arrow-key menu when the user presses Ctrl-C on it."""++ def __call__(self, message: str, choices: Sequence[str], default: str) -> str:+ raise KeyboardInterrupt+++class UnexpectedMenu:+ """Stands in for the arrow-key menu on a terminal where it must not be used."""++ def __call__(self, message: str, choices: Sequence[str], default: str) -> str:+ raise AssertionError("the arrow-key menu was used where it cannot run")+++def report_terminal_attached() -> bool:+ return True+++def report_outside_ci() -> bool:+ return False+++def show_offer_decision() -> None:+ """Minimal host command so CliRunner can answer the offer's prompts."""+ click.echo(f"decision={pro_offer.offer_pro_version().value}")++[email protected](autouse=True)+def offer_marker(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:+ """Where the shown-once marker lands; kept in ``tmp_path``, never the real home."""+ monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "config"))+ return tmp_path / "config" / "ifixai" / "pro-offer-shown"+++def browser(monkeypatch: pytest.MonkeyPatch) -> BrowserRecorder:+ recorder = BrowserRecorder(is_launch_successful=True)+ monkeypatch.setattr(pro_offer.webbrowser, "open", recorder)+ return recorder+++def attached_terminal(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(pro_offer, "can_prompt_user", report_terminal_attached)+++def run_args(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> list[str]:+ """A ``run`` invocation that stops at validation, before any inspection."""+ monkeypatch.chdir(tmp_path)+ monkeypatch.delenv(NO_PROMO_ENV_VAR, raising=False)+ return [+ "run", "--provider", "mock", "--eval-mode", "self",+ "--test", UNKNOWN_TEST_ID, "--no-telemetry",+ ]+++def mock_run_args(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> list[str]:+ """A ``run`` invocation that executes one offline inspection into ``tmp_path``."""+ monkeypatch.chdir(tmp_path)+ monkeypatch.delenv(NO_PROMO_ENV_VAR, raising=False)+ return [+ "run", "--provider", "mock", "--fixture", "customer_support",+ "--test", "B01", "--eval-mode", "self", "--no-telemetry", "--no-parallel",+ "--output", str(tmp_path / "reports"),+ "--reliability-out", str(tmp_path / "runs"),+ ]++[email protected]("answers", ["2\n", "\n"])+def test_declining_continues_the_run_without_opening_the_site(+ attached_terminal: None, browser: BrowserRecorder, answers: str+) -> None:+ result = CliRunner().invoke(show_offer_decision, input=answers)++ assert result.exit_code == 0, result.output+ assert PRO_URL in result.output+ assert "decision=continue" in result.output+ assert browser.opened_urls == []+++ "answers,expected",+ [("1\n1\n", "continue"), ("1\n\n", "continue"), ("1\n2\n", "abandon")],+)+def test_accepting_opens_the_site_then_asks_what_to_do_with_the_run(+ attached_terminal: None, browser: BrowserRecorder, answers: str, expected: str+) -> None:+ result = CliRunner().invoke(show_offer_decision, input=answers)++ assert result.exit_code == 0, result.output+ assert browser.opened_urls == [PRO_URL]+ assert "Continue the current run" in result.output+ assert "Abandon the run" in result.output+ assert f"decision={expected}" in result.output+++def test_a_browser_that_will_not_open_still_leaves_the_link_and_the_choice(+ attached_terminal: None, monkeypatch: pytest.MonkeyPatch+) -> None:+ failing_browser = BrowserRecorder(is_launch_successful=False)+ monkeypatch.setattr(pro_offer.webbrowser, "open", failing_browser)++ result = CliRunner().invoke(show_offer_decision, input="1\n2\n")++ assert result.exit_code == 0, result.output+ assert "Could not open a browser" in result.output+ assert f"Visit {PRO_URL}" in result.output+ assert "decision=abandon" in result.output+++def test_a_number_outside_the_list_is_refused_and_asked_again(+ attached_terminal: None, browser: BrowserRecorder+) -> None:+ result = CliRunner().invoke(show_offer_decision, input="7\n0\nyes\n1\n2\n")++ assert result.exit_code == 0, result.output+ assert result.output.count("Error:") == 3+ assert browser.opened_urls == [PRO_URL]+ assert "decision=abandon" in result.output+++def test_no_color_terminals_get_the_numbered_list_not_the_arrow_menu(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setenv("NO_COLOR", "1")+ monkeypatch.setattr(sys, "stdin", FakeStream(is_terminal=True, typed_text="1\n"))+ monkeypatch.setattr(sys, "stdout", FakeStream(is_terminal=True))+ monkeypatch.setattr(ui, "select_or_abort", UnexpectedMenu())++ assert ui.is_interactive() is False+ assert pro_offer.ask_choice(PRO_INTEREST_QUESTION) == YES_OPTION+++def test_no_terminal_means_no_prompt_and_the_run_continues(+ browser: BrowserRecorder, offer_marker: Path+) -> None:+ # CliRunner's streams are not terminals, so the real gate is exercised here.+ result = CliRunner().invoke(show_offer_decision, input="1\n2\n")++ assert result.output.strip() == "decision=continue"+ assert browser.opened_urls == []+ assert not offer_marker.exists()+++ "is_stdin_terminal,is_stdout_terminal,expected",+ [(True, True, True), (False, True, False), (True, False, False)],+)+def test_prompting_needs_a_terminal_on_both_streams(+ monkeypatch: pytest.MonkeyPatch,+ is_stdin_terminal: bool,+ is_stdout_terminal: bool,+ expected: bool,+) -> None:+ monkeypatch.setattr(telemetry, "in_ci", report_outside_ci)+ monkeypatch.setattr(sys, "stdin", FakeStream(is_stdin_terminal))+ monkeypatch.setattr(sys, "stdout", FakeStream(is_stdout_terminal))++ assert pro_offer.can_prompt_user() is expected++[email protected]("missing_stream", ["stdin", "stdout"])+def test_a_missing_standard_stream_is_not_a_terminal(+ monkeypatch: pytest.MonkeyPatch, missing_stream: str+) -> None:+ # Python sets a standard stream to None when the process has no handle for+ # it, e.g. `ifixai run ... <&-` or a detached Windows process.+ monkeypatch.setattr(telemetry, "in_ci", report_outside_ci)+ monkeypatch.setattr(sys, "stdin", FakeStream(is_terminal=True))+ monkeypatch.setattr(sys, "stdout", FakeStream(is_terminal=True))+ monkeypatch.setattr(sys, missing_stream, None)++ assert pro_offer.can_prompt_user() is False+++def test_ci_is_never_prompted_even_with_a_terminal(+ monkeypatch: pytest.MonkeyPatch, offer_marker: Path+) -> None:+ monkeypatch.setenv("CI", "true")+ monkeypatch.setattr(sys, "stdin", FakeStream(is_terminal=True))+ monkeypatch.setattr(sys, "stdout", FakeStream(is_terminal=True))++ assert pro_offer.can_prompt_user() is False+ assert pro_offer.offer_pro_version() is RunDecision.CONTINUE+ assert not offer_marker.exists()++[email protected]("extra_args,env", SKIPPED_RUNS)+def test_run_skips_the_offer_when_told_to_or_when_nothing_will_run(+ run_args: list[str],+ monkeypatch: pytest.MonkeyPatch,+ extra_args: list[str],+ env: dict[str, str],+) -> None:+ offer = OfferRecorder(RunDecision.ABANDON)+ monkeypatch.setattr(run_module, "offer_pro_version", offer)++ result = CliRunner().invoke(ifixai_cli, [*run_args, *extra_args], env=env)++ assert offer.call_count == 0+ assert ABANDONED_MESSAGE not in result.output+ assert UNKNOWN_TEST_ERROR in result.output+++def test_run_makes_the_offer_once_and_continues_when_the_user_keeps_the_run(+ run_args: list[str], monkeypatch: pytest.MonkeyPatch+) -> None:+ offer = OfferRecorder(RunDecision.CONTINUE)+ monkeypatch.setattr(run_module, "offer_pro_version", offer)++ result = CliRunner().invoke(ifixai_cli, run_args)++ assert offer.call_count == 1+ assert result.exit_code == 1, result.output+ assert UNKNOWN_TEST_ERROR in result.output+++def test_abandoning_exits_cleanly_before_the_run_leaves_any_trace(+ tmp_path: Path,+ mock_run_args: list[str],+ attached_terminal: None,+ browser: BrowserRecorder,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ started = CallCounter()+ monkeypatch.setattr(telemetry, "emit_started", started)++ result = CliRunner().invoke(ifixai_cli, mock_run_args, input="1\n2\n")++ assert result.exit_code == 0, result.output+ assert ABANDONED_MESSAGE in result.output+ assert browser.opened_urls == [PRO_URL]+ assert started.call_count == 0+ assert not (tmp_path / "runs").exists()+ assert not (tmp_path / "reports").exists()+++def test_continuing_after_visiting_the_site_runs_the_inspections(+ tmp_path: Path,+ mock_run_args: list[str],+ attached_terminal: None,+ browser: BrowserRecorder,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ started = CallCounter()+ monkeypatch.setattr(telemetry, "emit_started", started)++ result = CliRunner().invoke(ifixai_cli, mock_run_args, input="1\n1\n")++ assert ABANDONED_MESSAGE not in result.output+ assert browser.opened_urls == [PRO_URL]+ assert started.call_count == 1+ assert (tmp_path / "runs").exists()+ assert list((tmp_path / "reports").glob("*.json")), result.output+++def test_input_ending_at_the_numbered_prompt_aborts_instead_of_starting_the_run(+ tmp_path: Path,+ mock_run_args: list[str],+ attached_terminal: None,+ browser: BrowserRecorder,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ started = CallCounter()+ monkeypatch.setattr(telemetry, "emit_started", started)++ result = CliRunner().invoke(ifixai_cli, mock_run_args, input="")++ assert result.exit_code == 1, result.output+ assert "Aborted!" in result.output+ assert started.call_count == 0+ assert not (tmp_path / "runs").exists()+++def test_declining_at_the_real_prompt_lets_the_run_proceed(+ run_args: list[str], attached_terminal: None, browser: BrowserRecorder+) -> None:+ result = CliRunner().invoke(ifixai_cli, run_args, input="2\n")++ assert result.exit_code == 1, result.output+ assert PRO_URL in result.output+ assert UNKNOWN_TEST_ERROR in result.output+ assert browser.opened_urls == []+++ "answers,expected,expected_urls",+ [+ ([NO_OPTION], RunDecision.CONTINUE, []),+ ([YES_OPTION, CONTINUE_OPTION], RunDecision.CONTINUE, [PRO_URL]),+ ([YES_OPTION, ABANDON_OPTION], RunDecision.ABANDON, [PRO_URL]),+ ],+)+def test_arrow_menu_answers_drive_the_same_decisions(+ attached_terminal: None,+ browser: BrowserRecorder,+ monkeypatch: pytest.MonkeyPatch,+ answers: list[str],+ expected: RunDecision,+ expected_urls: list[str],+) -> None:+ monkeypatch.setattr(ui, "is_interactive", report_terminal_attached)+ monkeypatch.setattr(ui, "select_or_abort", ScriptedMenu(answers))++ assert pro_offer.offer_pro_version() is expected+ assert browser.opened_urls == expected_urls+++def test_arrow_menus_start_on_the_answer_that_keeps_the_run_going(+ attached_terminal: None, browser: BrowserRecorder, monkeypatch: pytest.MonkeyPatch+) -> None:+ menu = ScriptedMenu([YES_OPTION, CONTINUE_OPTION])+ monkeypatch.setattr(ui, "is_interactive", report_terminal_attached)+ monkeypatch.setattr(ui, "select_or_abort", menu)++ pro_offer.offer_pro_version()++ assert menu.defaults == [NO_OPTION, CONTINUE_OPTION]+++def test_ctrl_c_on_the_arrow_menu_aborts_instead_of_starting_the_run(+ tmp_path: Path,+ mock_run_args: list[str],+ attached_terminal: None,+ browser: BrowserRecorder,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ started = CallCounter()+ monkeypatch.setattr(telemetry, "emit_started", started)+ monkeypatch.setattr(ui, "is_interactive", report_terminal_attached)+ monkeypatch.setattr(ui, "select_or_abort", CancelledMenu())++ result = CliRunner().invoke(ifixai_cli, mock_run_args)++ assert result.exit_code == 1, result.output+ assert "Aborted!" in result.output+ assert started.call_count == 0+ assert not (tmp_path / "runs").exists()+++def test_the_offer_invites_a_free_fast_audit(+ attached_terminal: None, monkeypatch: pytest.MonkeyPatch+) -> None:+ failing_browser = BrowserRecorder(is_launch_successful=False)+ monkeypatch.setattr(pro_offer.webbrowser, "open", failing_browser)++ result = CliRunner().invoke(show_offer_decision, input="1\n1\n")++ assert OFFER_LINE in result.output+ assert OFFER_QUESTION in result.output+ assert [copy for copy in RETIRED_COPY if copy in result.output] == []+++def test_a_second_run_starts_straight_away_without_the_offer(+ run_args: list[str],+ attached_terminal: None,+ browser: BrowserRecorder,+ offer_marker: Path,+) -> None:+ first = CliRunner().invoke(ifixai_cli, run_args, input="2\n")+ # No input: a second offer would stop at the prompt and abort.+ second = CliRunner().invoke(ifixai_cli, run_args, input="")++ assert OFFER_LINE in first.output+ assert offer_marker.exists()+ assert OFFER_LINE not in second.output+ assert "Aborted!" not in second.output+ assert UNKNOWN_TEST_ERROR in second.output+++ "answers",+ [[NO_OPTION], [YES_OPTION, CONTINUE_OPTION], [YES_OPTION, ABANDON_OPTION]],+)+def test_any_answer_counts_as_shown(+ attached_terminal: None,+ browser: BrowserRecorder,+ monkeypatch: pytest.MonkeyPatch,+ answers: list[str],+) -> None:+ monkeypatch.setattr(ui, "is_interactive", report_terminal_attached)+ monkeypatch.setattr(ui, "select_or_abort", ScriptedMenu(answers))+ pro_offer.offer_pro_version()+ monkeypatch.setattr(ui, "select_or_abort", UnexpectedMenu())++ assert pro_offer.offer_pro_version() is RunDecision.CONTINUE+++def test_ctrl_c_on_the_offer_counts_as_shown(+ attached_terminal: None, browser: BrowserRecorder, monkeypatch: pytest.MonkeyPatch+) -> None:+ monkeypatch.setattr(ui, "is_interactive", report_terminal_attached)+ monkeypatch.setattr(ui, "select_or_abort", CancelledMenu())+ with pytest.raises(KeyboardInterrupt):+ pro_offer.offer_pro_version()+ monkeypatch.setattr(ui, "select_or_abort", UnexpectedMenu())++ assert pro_offer.offer_pro_version() is RunDecision.CONTINUE++[email protected]("extra_args,env", SKIPPED_RUNS)+def test_a_skipped_offer_is_not_marked_shown(+ run_args: list[str],+ attached_terminal: None,+ browser: BrowserRecorder,+ offer_marker: Path,+ extra_args: list[str],+ env: dict[str, str],+) -> None:+ result = CliRunner().invoke(ifixai_cli, [*run_args, *extra_args], env=env)++ assert OFFER_LINE not in result.output+ assert UNKNOWN_TEST_ERROR in result.output+ assert not offer_marker.exists()++[email protected]("blocking_file", ["config", "config/ifixai"])+def test_a_marker_that_cannot_be_written_skips_the_offer_and_runs(+ tmp_path: Path,+ run_args: list[str],+ attached_terminal: None,+ browser: BrowserRecorder,+ blocking_file: str,+) -> None:+ # A file where the config dir should be: the marker can't be created.+ blocker = tmp_path / blocking_file+ blocker.parent.mkdir(parents=True, exist_ok=True)+ blocker.write_text("not a directory", encoding="utf-8")++ result = CliRunner().invoke(ifixai_cli, run_args, input="")++ assert OFFER_LINE not in result.output+ assert "Aborted!" not in result.output+ assert UNKNOWN_TEST_ERROR in result.output