Skip to content

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

2.3 KB
Raw Blame

Python API

ifixai.api runs the same engine as the CLI. Run functions are async; helpers are sync.

import asyncio
from ifixai.api import run_inspections

result = asyncio.run(run_inspections(
    provider="openai",
    api_key="sk-...",
    model="gpt-4o",
    fixture="default",
    system_name="my-agent",
))
print(result.overall_score, result.grade)

Functions

Function Purpose
run_inspections(...) Run all registered inspections (async) → TestRunResult. Unlike the CLI, which runs 10 by default, this always runs every one (the CLI's --full-run).
run_strategic(...) Run the top 8 strategic tests (async)
run_selected(test_ids, ...) Run a chosen list of test IDs (async)
run_single(test_id, ...) Run a single test by ID (async); raises ValueError with why when it is not applicable
compare_scorecards(baseline, enhanced) Vendor-neutral comparison report (sync)
list_tests() Return all InspectionSpec definitions (sync)
list_fixtures() Return built-in fixture names (sync)

Common parameters

Parameter Default Notes
provider required Provider name ("openai", "anthropic", …) or a ChatProvider instance.
api_key "" SUT key.
fixture "default" Registered name or a Fixture object.
model, endpoint, system_prompt None SUT overrides.
system_name, system_version "", "1.0" Report labels.
timeout, max_retries 30, 3 Per-request controls.
judge_config, pipeline_config None JudgeConfig / EvaluationPipelineConfig for judges, ensemble, budgets.
sut_temperature, sut_seed, run_nonce, holdout_ids 0.0, None, None, None Determinism / replay controls (mirror CLI seeds).

To test a custom agent, implement ChatProvider (../ifixai/providers/base.py) and pass the instance as provider. Only send_message is needed: the inspections ask your agent through it. Hooks you add (authorize_tool, get_confidence, ...) show as notes next to the probe results and never decide the checks the probes decide. TestRunResult.not_applicable lists the checks set aside before the run, each with why. Walkthrough: testing-your-agent.md.