Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
Test your own agent
iFixAi tests the system you deploy: model plus tools, retrieval, and governance. Its chat
endpoint is enough: every inspection with a chat probe asks your agent through it, a bare
model API (--provider openai / anthropic / ...) included. A check that can't apply is
listed as not applicable before the run, with why. Flags and extras: provider reference.
Path 1: HTTP endpoint
Your agent serves POST /v1/chat/completions:
ifixai run --provider http --endpoint http://localhost:8000/v1 \
--api-key "$YOUR_TOKEN" --model your-agent --fixture ./my-fixture.yaml --eval-mode self
To score B28 (RAG context integrity), return a sources array next to choices:
{ "choices": [ { "message": { "role": "assistant", "content": "..." } } ],
"sources": [ { "document_uri": "kb://policy/42", "text": "...", "relevance_score": 0.91 } ] }
Custom auth: set IFIXAI_EXTRA_HEADERS to a JSON object; pick auth_method
(bearer / api_key / basic / none) via the Python API.
Path 2: ChatProvider adapter
For an in-process agent, subclass ChatProvider. Only
send_message is required: the inspections ask your agent through it. Hooks are optional
and default to None. Pass your instance through the Python API and run
as usual.
from ifixai.providers.base import ChatProvider
from ifixai.core.types import ChatMessage, ProviderConfig
class MyAgentProvider(ChatProvider):
async def send_message(self, messages: list[ChatMessage], config: ProviderConfig) -> str:
reply = await my_agent.ainvoke([{"role": m.role, "content": m.content} for m in messages])
return reply.text
B01, B02, B04, B23, C02, C05, C11, M02, M03, M07, P01, P08, V08, X04 and X11 no longer
need an adapter: each sends its probe pack over chat and the replies decide it. Hooks
those inspections read (authorize_tool, get_confidence, route_to_human, ...) show as
notes beside the probe results and never change the verdict.
| Capability hook | What it adds |
|---|---|
attest_response_identity |
M06, which has no chat probe: without the hook it is not applicable |
invoke_tool |
B11, whose chat probe can't score it alone: without the agent's own hook it is not applicable |
retrieve_sources |
B05 source checks, when the fixture lists 10 or more data sources. The http provider's /retrieve route is optional, so it counts only in a run with a judge |
list_tools / invoke_tool / get_audit_trail / apply_override / get_configuration_version |
extra checks in B03, B08, B13, B14, B25, scored only when the fixture's governance: answers none of that inspection's hooks |
| any hook of the 15 above | a note beside the probe results |
IDs and structural requirements: inspections.md, fixture_authoring.md.
No code: declare governance in the fixture
A governance: block sets the rules the probes test; it never earns a pass. Its hook
answers are shown as declared and not scored. Ways to supply it, most trustworthy first:
- Inline
governance:block on the diagnostic fixture. governance: { synthesize: true }(last resort): derived fromtools/permissions/roles.
A fixture that ships with iFixAi (default and the examples) has a sample org's governance:
block. ifixai run puts it on the mock only; on any other agent it stays off, and the run says
so in one line. Copy the fixture to your own file to use its block on your agent.
Fields, and the abilities that set checks aside: fixture_authoring.md.
Which judge?
Atomic-claim verdicts accept JSON true/false and the strings "true"/"false"
(case-insensitive, with surrounding whitespace ignored). This applies to
supported and ground-truth response_correct. Other values, including numeric
0/1, "yes"/"no", null and missing fields, are discarded as invalid judge
output. A string "false" always
scores as false; a nonempty string never becomes true merely because it exists.
Analytic rubric dimensions accept JSON booleans, integer 0/1, and recognized
verdict words such as "false"/"true", "fail"/"pass", and "no"/"yes".
Anything else (a missing or null value, other numbers, unknown words) is read as if
the judge left that dimension out: a must-pass one leaves the item unscored unless a
retry gets a readable answer; a soft one counts as not met. This applies to every
supported rubric envelope.
--eval-mode self is a smoke test, flagged self-judge bias. By default a second,
different provider grades the SUT, so nothing scores itself:
- Default judge: any non-SUT provider key in your env, on that provider's default model.
- Multiple keys: tiebreaker order
anthropic → openai → atlascloud → minimax → gemini → openrouter → azure → bedrock → huggingface. - No non-SUT key: the run refuses unless you pass
--eval-mode self. - Override:
--judge-provider/--judge-api-key/--judge-model. Pin these foropenrouterandazure: auto-routing can land SUT and judge on the same vendor.
For a real verdict:
# Standard: one cross-provider judge (export a second provider key, or pin --judge-provider).
ifixai run --provider http --endpoint http://localhost:8000/v1 --api-key "$YOUR_TOKEN" --fixture ./my-fixture.yaml
# Full: hand-built fixture + two or more independent judges.
ifixai run --mode full \
--provider http --endpoint http://localhost:8000/v1 --api-key "$YOUR_TOKEN" \
--fixture ./my-fixture.yaml \
--judge-provider anthropic --judge-api-key "$ANTHROPIC_API_KEY" \
--judge-provider openai --judge-api-key "$OPENAI_API_KEY"
Modes: cli.md. Judges: methodology.md.
Coverage
Counts are for a --full-run. A default run executes 10 of these inspections.
| SUT shape | Inspections graded |
|---|---|
| Chat endpoint only (HTTP agent or bare model API), any fixture | every inspection with a chat probe; M06 and B11 are listed as not applicable * |
--provider mock (zero credentials) |
the same; the mock's hooks answer from the fixture, so M06 and B11 are not applicable * |
Adapter with attest_response_identity / invoke_tool |
M06 / B11 as well; B11 only when the fixture's governance: answers none of its hooks |
The fixture can set more checks aside, each listed with why before the run: its abilities,
or no tool that can delete or overwrite that some role may not use (P01). With a judge, a stated purpose that leaves out
training work sets M03 and V08 aside too; those are listed once the run starts.
* The bundled default fixture (NimbusForge, managed_it_infrastructure) declares a
specific domain so B32 scores, and it carries seeded defects: a mock --full-run on it
fails 14 inspections by design. They are properties of the fixture, not your
system. Defect map: ifixai/fixtures/default/README.md.
On a custom fixture, set a specific metadata.domain (or metadata.on_topic_examples)
to score B32.
Provider reference
Install the extra first, e.g. pip install -e ".[gemini]". The SUT key is read from the
provider's env var below; --api-key / -k overrides it, and http always needs it (or --auth-method none).
--provider |
Extra | SUT credential | Example flags |
|---|---|---|---|
http (recommended) |
none | your server token | --provider http --endpoint http://localhost:8000/v1 --grounding sut -k TOKEN --model your-model-id |
anthropic |
.[anthropic] |
ANTHROPIC_API_KEY |
--provider anthropic -k "$ANTHROPIC_API_KEY" --model claude-sonnet-4-6 |
openai |
.[openai] |
OPENAI_API_KEY |
--provider openai -k "$OPENAI_API_KEY" --model gpt-4o |
atlascloud |
.[atlascloud] |
ATLASCLOUD_API_KEY or ATLAS_CLOUD_API_KEY |
--provider atlascloud -k "$ATLASCLOUD_API_KEY" --model qwen/qwen3.5-flash |
minimax |
none | MINIMAX_API_KEY |
--provider minimax -k "$MINIMAX_API_KEY" --model MiniMax-M3.1-Flash |
openrouter |
.[openrouter] |
OPENROUTER_API_KEY |
--provider openrouter -k "$OPENROUTER_API_KEY" --model openai/gpt-4o plus explicit judge |
orcarouter |
.[orcarouter] |
ORCAROUTER_API_KEY |
--provider orcarouter -k "$ORCAROUTER_API_KEY" --model openai/gpt-4o plus explicit judge |
requesty |
.[requesty] |
REQUESTY_API_KEY |
--provider requesty -k "$REQUESTY_API_KEY" --model openai/gpt-4o-mini plus explicit judge; --endpoint https://router.eu.requesty.ai/v1 for EU data residency |
cloudflare |
.[cloudflare] |
CLOUDFLARE_API_TOKEN plus CLOUDFLARE_ACCOUNT_ID |
--provider cloudflare -k "$CLOUDFLARE_API_TOKEN" --model workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast plus explicit judge |
vercel |
.[vercel] |
AI_GATEWAY_API_KEY |
--provider vercel -k "$AI_GATEWAY_API_KEY" --model openai/gpt-4o-mini plus explicit judge |
gemini |
.[gemini] |
GEMINI_API_KEY or GOOGLE_API_KEY |
--provider gemini -k "$GEMINI_API_KEY" |
azure |
.[azure] |
AZURE_OPENAI_API_KEY |
--provider azure --endpoint https://YOUR_RESOURCE.openai.azure.com/ -k "$AZURE_OPENAI_API_KEY" --model YOUR_DEPLOYMENT_NAME plus explicit judge |
bedrock |
.[bedrock] |
AWS credential chain | --provider bedrock -k not-used --model anthropic.claude-sonnet-4-6 |
huggingface |
.[huggingface] |
HF_TOKEN or HUGGINGFACE_API_TOKEN |
--provider huggingface -k "$HF_TOKEN" --model meta-llama/Llama-3.1-8B-Instruct |
langchain |
none (pip install langchain on your server only) |
key for the model behind your server | --provider langchain --endpoint http://localhost:8000 -k "$OPENAI_API_KEY" --eval-mode self |
mock |
none | none | --provider mock --eval-mode self |
Notes:
.[all]installs every SDK above;.[dev]is for contributing only.httptests your deployed agent as-shipped;--grounding sutgrades the governance it already enforces.bedrockuses the standard AWS credential chain;--api-keyis a required placeholder, never sent.litellmextra is Python API only (provider="litellm"), not a CLI choice.- An
openrouterjudge falls back to other models when the gateway errors, so a 502 costs one probe, not the run. Chain:ifixai/judge/judge_fallbacks.json, see cli.md. cloudflareis the Cloudflare AI Gateway OpenAI-compatible endpoint,gateway.ai.cloudflare.com/v1/<account>/<gateway>/compat. It needs two values: an API token allowed to run the gateway (CLOUDFLARE_API_TOKEN; Cloudflare documents the AI Gateway > Run permission for this) and the account ID, which is part of the URL (CLOUDFLARE_ACCOUNT_ID, or pass the full base URL with--endpoint). A judge has no--endpoint, so acloudflarejudge always reads the account ID from the environment. Requests go through the gateway nameddefaultunlessCLOUDFLARE_GATEWAY_IDnames another (the ID is case-sensitive), and every request asks the gateway to bypass its response cache (cf-aig-skip-cache), so that a repeated probe is answered by the model and not by a stored reply. Models are Workers AI ids such asworkers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast(the default; the Workers free plan's daily allocation covered about one 61-probe inspection in testing, so a full run needs a paid Workers plan, and a run that exhausts the allocation stops with Cloudflare's message), orprovider/modelslugs such asopenai/gpt-4.1-miniandgoogle-ai-studio/gemini-2.5-flash, which need Unified Billing credits or a provider key stored in the gateway. Cloudflare answers a bad token and a wrong account ID with the same401 Unauthorized; iFixAi lists what to check when it sees one. Cloudflare also has a newer REST route onapi.cloudflare.comand steers new integrations to it, but that route rejects a gateway token (it wants a token with the Workers AI permission), so this provider uses the gateway URL.vercelis the Vercel AI Gateway. Models arecreator/modelslugs from its catalog, e.g.anthropic/claude-haiku-4.5. Until the Vercel team has a payment method on file the gateway answers every request with403 customer_verification_required, free tier included; iFixAi reports Vercel's message, which links to the page for adding one.
LangServe wire contract
--provider langchain is a thin HTTP client for a LangServe
endpoint; iFixAi never imports LangChain. Serve any Runnable that accepts
{"messages": [...]} via add_routes(app, agent_runnable). iFixAi POSTs to
{endpoint}/invoke (default endpoint http://localhost:8000):
// request
{ "input": { "messages": [ { "role": "user", "content": "…" } ] },
"config": { "configurable": { "temperature": 0.0 } } } // seed / max_tokens added when set
// response: `output` as a string, or `output.content` if it is an object
{ "output": "the agent's reply" }
Source: ifixai/providers/langchain.py. To expose
structural surfaces (tools, audit trail, authorization), use a ChatProvider adapter (Path 2).