Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
Contributing to ifixai
Thanks for considering a contribution. This guide covers the mechanics of adding inspections, fixtures, and providers.
Environment setup
git clone <your-fork-url> ifixai
cd ifixai
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pre-commit install
The pre-commit install step wires up local hooks (gitleaks, ruff, and a .env guard). Run them on demand with pre-commit run --all-files.
Verify:
ruff check ifixai
bandit -r ifixai -ll
The package ships ifixai/ plus the per-inspection bundles under ifixai/inspections/b<NN>_<slug>/.
Continuous integration
GitHub Actions (.github/workflows/ci.yml) runs on every push to any branch that contains the workflow file.
For pull requests, the same checks also run when the PR base branch is main, fix/**, or feat/**. A PR targeting another branch (for example develop or release/1.0) does not receive a pull_request workflow run; you may still see a recent run from pushes to the PR head branch—open the Actions tab and filter by branch if needed.
Adding a test (inspection)
Each inspection lives in its own folder under ifixai/inspections/bNN_short_name/. Required contents:
runner.py— declaresSPECand theBaseTestsubclassdefinition.yaml— the conversation plan (steps, prompts, evaluation hints)rubric.yaml— analytic-judge dimensions and weights (default; loaded when no outcome-type-specific file exists)rubric_{outcome_type}.yaml— optional outcome-type-specific rubric (e.g.rubric_comply.yaml); loaded in preference torubric.yamlwhen the inspection requests that outcome type. Use this when a single inspection covers bothrefuseandcomplycases that require different scoring dimensions.references.yaml— reference responses used by atomic-claims groundingcorpus.yaml— adversarial seeds (only B12, B14, B28, B30)
The minimum contract:
- Declare the
SPEC, anInspectionSpecinstance withtest_id,name,category(one of the fiveInspectionCategoryvalues),description,threshold,weight,scoring_method, and optionalis_strategic/is_mandatory_minimumflags. The canonical test → pillar list isdocs/inspections.md; update that table when you add or recategorise an inspection. - Implement a subclass of
BaseTest(fromifixai.harness.base). Overriderun()to produce a list ofEvidenceItems. Useself.pipeline.evaluate(...)to get a pass/fail from the configured judge. - Declare
required_fixture_keys: frozenset[str]on the subclass listing every fixture key the inspection's templates reference. The fixture loader validates this at load time; inspections that reference keys the fixture doesn't provide fail fast with an actionable error. - Render every prompt through
ifixai.utils.template_renderer.render(template, context). Directstr.format(...)or f-string interpolation on fixture values is forbidden — it silently leaks{placeholder}literals to the model when a key is missing. - Register the inspection in
ifixai/harness/registry.py(import the class andSPECfromifixai.inspections.bNN_short_name.runner+ add toALL_SPECS+create_inspectionswitch). - Update
ifixai/scoring/category_weights.pyonly if the inspection belongs to the strategic set. - Run
ifixai validate(no args) — the layout validator will fail loudly if any required artifact is missing or the folder name disagrees with the YAMLtest_id.
Inspection self-validation
Run ifixai validate (no args) after authoring or editing an inspection. The layout validator checks:
- the per-test folder contains every required artifact (
runner.py,definition.yaml,rubric.yaml,references.yaml, pluscorpus.yamlfor B12/B14/B28/B30); optionalrubric_{outcome_type}.yamlfiles are not validated for presence but must pass schema checks if present; - the folder name agrees with the YAML
test_id; SPECinvariants (id, category, threshold, weight, strategic flags) match the registry;- the runner registers cleanly with
harness/registry.py.
Authoring a fixture
Fixtures are YAML files under ifixai/fixtures/. Validate against ifixai/fixtures/schema.json. A fixture MUST supply every key listed in the union of every registered inspection's required_fixture_keys.
The x-placeholders section of the schema (lint-only) enumerates the placeholder keys any inspection may reference. Keep it in sync when you add an inspection that references a new key.
Three example fixtures live under ifixai/fixtures/examples/. Copy one as a starting point.
Registering a new provider
Providers implement the ChatProvider protocol from ifixai/providers/base.py. Steps:
- Create
ifixai/providers/<your_provider>.pyimplementing at minimumsend_message— other capability methods may raiseNotImplementedErrorif the provider does not expose them. - Register the provider string in
ifixai/providers/resolver.py. - Add an optional dependency extra in
pyproject.tomlunder[project.optional-dependencies]so users install only what they need. - Do not swallow exceptions silently. If the provider has idiomatic error types, translate them into
ProviderError(or a subclass). - Exercise the provider end-to-end against a fixture before opening a PR (
ifixai run --provider <your_provider> ...); attach the resulting scorecard snippet to the PR body.
Running checks locally
# lint
ruff check ifixai
# type check
mypy ifixai
# security scan
bandit -r ifixai -ll
# layout / inspection validation
ifixai validate
Commit conventions
Follow the Conventional Commits-style prefixes used by this project:
feat:new user-visible featurefix:bug fixrefactor:behaviour-preserving changedocs:documentation onlychore:tooling / housekeepingperf:performance improvementci:CI configuration
Keep commits small and atomic.
Pull requests
- Target branch:
main. - Include a test plan in the PR body.
- Confirm
ruff,bandit, andifixai validateall pass locally before requesting review. CI runsifixai validate(layout) plusifixai validateon each file underifixai/fixtures/examples/.mypyis advisory (run it locally if you touched typed surfaces, but it is not a CI gate). - For any inspection / fixture / provider change, paste one worked example scorecard snippet (JSON or Markdown) into the PR body.
Where to ask
Open a GitHub issue on the repository for questions, bug reports, or feature proposals. For security-sensitive reports see SECURITY.md.