Skip to content

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

7.0 KB
Raw Blame

Get started

Clean machine to a citable scorecard in four steps. Flags, judges, and modes: cli.md.

1. Install

python3 -m venv .venv && source .venv/bin/activate   # Windows: py -m venv .venv, then .venv\Scripts\activate
pip install ifixai

Add your provider's extra when you test a real model, e.g. pip install "ifixai[openai]" (provider reference). ifixai init shows which keys are set.

Windows: if PowerShell can't find ifixai after pip install, add Python's Scripts\ folder to your PATH, or run it as python -m ifixai.cli.main.

2. Prove the pipeline runs

No keys, no network:

ifixai run --provider mock --eval-mode self

Runs the default 10 inspections on a built-in demo agent and writes a report to ./ifixai-results/. A plumbing check, not a diagnosis: the demo agent gives unsafe replies planted in its setup, so it fails 4 of the 10 (B01, B12, B16 and P01), gets grade D and exits 2. With --full-run it runs 59 (B11 and M06 need a custom adapter, so they are set aside) and fails 14 (what the demo shows). Every ifixai run on this page runs those 10; add --full-run for all 61 (what the default covers).

3. Run a real model

First write my-fixture.yaml, your agent's roles, tools and rules (how). Every run below passes it: without --fixture, iFixAi tests the demo agent's made-up setup, not yours.

Prefer your real deployed agent, reached over its own HTTP endpoint. Its token goes in AGENT_API_KEY, passed as --api-key (no auth: --auth-method none instead). A second provider grades the replies, here OpenAI:

pip install "ifixai[openai]"
export OPENAI_API_KEY=sk-...
export AGENT_API_KEY=...        # your agent's own token
ifixai run --provider http --endpoint <agent-url> --api-key "$AGENT_API_KEY" --fixture ./my-fixture.yaml --grounding sut --dry-run

--dry-run prints the estimate and stops; drop it to run. --grounding sut (the default) keeps the agent's own system prompt instead of injecting one built from the fixture.

No endpoint? Test the bare model. Its key is read from its environment variable:

pip install "ifixai[anthropic]"
export ANTHROPIC_API_KEY=sk-ant-api03-...
ifixai run --provider anthropic --fixture ./my-fixture.yaml --eval-mode self

Every run writes JSON and Markdown to ./ifixai-results/ (override with --output). The report shows:

  • the A–F grade and the three mandatory-minimum gates (B01, B08, P01; scoring.md);
  • the five core pillars that set the grade, plus 20 extended categories (reported, never graded);
  • warnings[]: inspections left INCONCLUSIVE (with why) instead of an invented score; not_applicable[]: checks set aside before the run.

--eval-mode self is flagged as self-judged in the output: a smoke test, not a citable result.

4. Get a real, citable grade

Add a second provider's SDK extra and key, drop --eval-mode self; the judge auto-pairs from the environment:

pip install "ifixai[anthropic,openai]"   # or ifixai[all]
export ANTHROPIC_API_KEY=sk-ant-api03-...   # SUT: graded
export OPENAI_API_KEY=sk-...                # judge: any second, different provider
ifixai run --provider anthropic --fixture ./my-fixture.yaml

The SUT's own provider is excluded from judge selection. With no second key the run refuses rather than self-judging. Recommended judge models and costs: cli.md.

Guided wizard

Prefer menus to flags? ifixai setup is an arrow-key wizard: pick provider, model, judge and suite, and it writes ifixai.yaml. After that, ifixai run needs only your fixture (the wizard offers the built-in ones).

ifixai setup
ifixai run --fixture ./my-fixture.yaml

The wizard detects API keys already in your environment and shows them at the top of each prompt. No key found? It tells you which env var to export; if it's still missing when you run, you're prompted for it before the first API call. What ifixai.yaml holds: cli.md.

Run it from your coding agent

Your coding agent can be the operator: it finds your agent, builds the fixture, shows the cost via a free --dry-run, runs only after you say yes, and explains the scorecard. Claude Code and Codex: install the plugin (README, plugin/README.md). Any agent: one command writes a native /ifixai-skill slash command. It needs only uv and Python 3.10+, no API key:

uvx ifixai install --agents cursor   # any slug: claude, vscode, windsurf, cline, continue, gemini, zed
uvx ifixai install --agents all      # every agent at once
uvx ifixai install --list            # every supported agent and where its file lands

Supported: Claude Code, Cursor, VS Code / Copilot, Windsurf, Cline, Continue, Gemini and Zed, plus an AGENTS.md bridge (on Codex, use the plugin). Then run /ifixai-skill in that agent; its runs are zero-install too (uvx --from "ifixai[<provider>]" ifixai run). On a new project, name the agent with --agents (auto-detect only finds agents whose folder already exists). Already have the CLI on your PATH? Drop the uvx prefix. The command is named ifixai-skill so it never collides with the Claude Code plugin's /ifixai; pass --name ifixai for the bare name.

Three ways to run

All three run the same engine. The difference is how you drive it.

Guided wizard Explicit flags Plugin or skill
How you drive it ifixai setup once, then ifixai run with no flags; config saved to ifixai.yaml every option as a flag; fully scriptable your coding agent is the operator: finds your setup, builds the fixture, runs it, explains the scorecard
Best for first-time users, repeatable runs, team onboarding CI, automation, scripted batches a guided, explained run inside the agent you already use
Keys detected by the wizard; ifixai.yaml stores the env-var name, never the secret each provider's env var, or --api-key same
What you test any provider, or your agent's real endpoint same same
Who grades it the model itself, one independent judge, or a panel of judges same same
Output JSON + Markdown reports + terminal scorecard same same, plus an interactive results page
Suite picked in the wizard 10 inspections by default; --full-run for all 61, or --suite (options) the agent picks, same engine
Works in any terminal any terminal or CI Claude Code, Codex, Cursor, VS Code, Windsurf, Cline, Continue, Gemini, Zed

Next steps