Skip to content

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

12.1 KB
Raw Blame

Recorded trajectory assurance

iFixAi can audit a retained sequence of agent actions against a business policy, then freeze and replay the findings without a model or service connection. Use it to investigate an approval used twice, a worker acting outside delegated scope, a shared budget exhausted across actors, or an output citing the wrong source. Each finding links to the recorded events that support it. This is a separate workflow from inspections and scorecards; it does not change existing grades.

The auditor is passive. It evaluates already-recorded actions and never executes a tool named in imported data. TraceRecorder.invoke executes only the Python callable supplied by your application, records its observed outcome and propagates its exceptions. It does not enforce policy or reverse side effects.

Observe an application action

from ifixai.trajectory import (
    ActorPolicy, AuditPolicy, BudgetRule, PermissionRule, TraceRecorder,
    create_audit_bundle, write_capture_jsonl,
)
from ifixai.reporting.trajectory import write_trajectory_reports

store = {"refunded_units": 0}
def refund(amount):
    store["refunded_units"] += amount
    return store["refunded_units"]

policy = AuditPolicy(
    name="local refund policy", tenant_id="shop",
    trusted_collectors=("local-recorder",),
    actors={"worker": ActorPolicy(roles=("operator",))},
    permissions=(PermissionRule(role="operator", action="refund",
                                resource_scope="orders:*"),),
    budgets=(BudgetRule(scope="trace", limit_units=100),),
    require_citations=False,
)
recorder = TraceRecorder("shop", "local-recorder")
recorder.invoke(refund, 25, actor_id="worker", action="refund",
                resource="orders:42", arguments={"amount": 25}, cost_units=25)
capture = recorder.seal()
write_capture_jsonl(capture, "capture.jsonl")  # native captures only; new file
bundle = create_audit_bundle(capture, policy)
assert bundle.report.decision == "CLEAR"
write_trajectory_reports(bundle, "refund-evidence")  # must be a new directory

This example mutates a local dictionary, not money. The application supplies the semantic arguments and integer cost; the recorder does not prove that they match the callable or billing. Export to JSONL accepts native captures without import warnings; it cannot erase incomplete imported coverage. Raw argument dictionaries and response text are not serialized. Treat actor identifiers, resource names and labels as retained data. The recorder serializes event emission under a lock, refuses sealing while a call is pending, and marks a recording incomplete after a failed call. Automatic parent links represent local program order, not distributed physical causality. Synchronous callables are supported; instrument asynchronous execution explicitly.

The larger runnable example constructs compliant and adversarial local refund trajectories with approvals, delegations, sources and outputs, then exercises canonical JSONL and instrumented OTLP.

CLI and portable evidence

Save policy.model_dump_json(indent=2) as policy.json. All paths below are local.

ifixai trajectory audit policy.json capture.jsonl --output evidence
ifixai trajectory replay evidence/bundle.json --output replay-evidence
ifixai trajectory audit policy.json spans.json --format otlp --trace-id <32-hex-trace-id> --output otlp-evidence
ifixai trajectory what-if evidence/bundle.json alternate-policy.json --output scenario-evidence

Every command creates a new owner-only directory with bundle.json, report.json, report.html and report.md. Existing destinations are refused. HTML uses no remote resources and includes a searchable timeline, recorded parent links, violation witnesses, family coverage and budget usage. Open report.html locally. what-if also writes comparison.json, identifying resolved, new and persistent findings by (rule, code, event_id). It evaluates the same retained facts under another policy; it does not execute a remediation or demonstrate its effectiveness. The original bundle is read only.

Exit Decision Meaning
0 CLEAR No observed violation or coverage gap in the evaluated policy and retained capture
1 BLOCK At least one observed policy violation; evidence gaps remain visible
2 INCONCLUSIVE No observed violation, but missing evidence or unevaluated required coverage prevents a conclusion

Malformed input, replay integrity failure and output-directory errors are command errors (exit 1) without a report decision. CLI errors do not echo raw imported payloads. A trace with no checked completed action cannot be CLEAR. A configured approval family without an applicable completed action, required citations with no output, or configured budgets without completed actions remain inconclusive. Disabled or unused families are shown as NOT_EVALUATED; this is not a pass.

Replay input is bounded to 256 MiB. The bundle stores the complete normalized policy, capture, deterministic report and SHA-256 checksum. Replay validates the checksum and recomputes the report; editing a report and rehashing it does not bypass recomputation. A checksum proves neither collector authenticity nor capture truth. Replays require a compatible schema/evaluator; a changed evaluation is refused rather than silently accepted.

Policy and evidence semantics

Family Evaluated invariant
Graph Unique IDs; contiguous sequence; same tenant/trace; monotonic time; preceding parents; start/end, count and connected lineage
Permission Completed actions require a known actor and a direct role grant or valid delegated authority
Approval Every matching rule requires the bound subject, action, exact resource and argument digest; authorized approver, causal precedence, expiry, revocation and shared use count
Delegation Known issuer/recipient; issuer permission; parent recipient owns child issuance; action and scope attenuation; causal lineage; every ancestor remains unexpired and unrevoked
Budget Charge every completed observed action once, even unauthorized actions; exact integer trace and actor pools
Citation Output source IDs, digests, known source actors and preceding causal lineage

Resources on actions and approvals are exact. Policy/delegation scopes permit an exact resource or one terminal * prefix wildcard (orders:*). Matching is literal, not regex or shell execution. Multiple matching approval rules all apply: an issuer must satisfy each rule's approver roles and human requirements. Expiry equality is expired. Completed attempts consume a referenced approval use even if the binding is invalid. Invalid recorded revocation blocks the report and conservatively denies subsequent use; it never confers authority.

Delegation chains deeper than 128 grants are coverage gaps and cannot produce CLEAR. The cumulative ancestry traversal limit is 2,000,000 steps; exhaustion halts authority evaluation with a coverage gap while retaining the completed-action budget ledger. Failed or unknown actions are coverage gaps because a failure can follow a side effect. Their unknown costs cannot be assumed zero. Missing referenced grants and sources are missing evidence; the auditor does not invent their content. An invalid graph prevents positive authority evaluation. Observed violations take precedence over non-graph evidence gaps. Tenant mismatch prevents policy evaluation.

Citation lineage does not establish semantic support, truth or licensing. Human identity, collector trust, completion and cost fields are caller declarations. Collect and secure them in your application; this tool is not a signed collector, runtime authorization service or deployment certificate.

Canonical JSONL schema

The first nonblank line is a strict capture header; each remaining nonblank line is a TraceEvent. Unknown fields, duplicate JSON keys, nonfinite constants and invalid UTF-8 are rejected. Input is bounded to 64 MiB and 100,000 events. Canonical rows are never silently sorted. Event IDs are bounded identifiers; trace IDs are 32 hex characters. Timestamps and amounts are nonnegative strict integers bounded to signed 64-bit range, rejecting booleans and floats. Exact models and defaults are exposed in ifixai.trajectory.models.

{"schema_version":"1","trace_id":"1234567890abcdef1234567890abcdef","tenant_id":"shop","collector_id":"local-recorder","capture_complete":true,"source_format":"jsonl"}

Every event has event_id, trace_id, tenant_id, sequence, time_ns, parents and a payload discriminated by kind. Supported kinds are start, approval, revocation, delegation, action, source, output and end. Seal with an end whose declared_events includes itself and whose complete declares coverage. Use canonical_digest / arguments_digest for normalized JSON SHA-256 digests.

Explicit OTLP JSON instrumentation

The importer accepts an OTLP ExportTraceServiceRequest JSON envelope with resourceSpans[].scopeSpans[].spans[]. Ordinary GenAI span names do not encode the business-policy authority needed for this audit. Every retained semantic span must carry an ifixai.event string attribute containing its full canonical TraceEvent JSON. A normal uninstrumented export is incomplete evidence.

Resource attributes are ifixai.tenant_id and ifixai.collector_id as stringValue, and ifixai.capture_complete as boolValue. Bind the embedded event's ID to the span's nonzero 16-hex spanId, its trace to nonzero 32-hex traceId, and its time to endTimeUnixNano (or startTimeUnixNano if end is absent). IDs accept hex case and normalize lowercase. Timestamps accept decimal strings or exact integers. If there are explicit parents, parentSpanId must identify a retained canonical parent; additional parents remain in event.parents.

{"key":"ifixai.event","value":{"stringValue":"{...full canonical event JSON...}"}}

Unknown ordinary OTLP envelope fields are ignored. Unknown reserved ifixai.* attributes, missing/inconsistent resource metadata, uninstrumented spans, missing canonical parent bindings, or reported dropped attributes/events/links create coverage warnings. Duplicate reserved attributes and ambiguous typed values are rejected. Span kind/status enums, when supplied, must use numeric values; missing start/end timestamp metadata is a coverage warning. Multi-trace exports require explicit trace selection; unselected traces are intentionally excluded. OTLP events are normalized by their explicit sequence, then the same graph audit checks duplicates, gaps and time. Timestamps alone never prove approval or delegation authority.

The envelope follows the official OTLP JSON encoding specification. The ifixai.* instrumentation contract is specific to this feature and is not an OpenTelemetry semantic convention or an automatically installed integration.

What it doesn't do

  • It only knows what your application tells it. You add the instrumentation and choose the policy. Arguments, actors, human identity and costs are your declarations, and the recorder only watches synchronous callables you pass it.
  • It doesn't assume a failed call did nothing. A call that raises leaves coverage incomplete, because it may have had side effects first.
  • Citation checks only follow the recorded links from an output back to its sources.
  • The checksum catches changes to a bundle. It can't show who made it.
  • Delegation chains deeper than 128 grants, and audits that hit the 2,000,000-step ancestry limit, can't come back CLEAR.
  • A report covers the evidence it was given. It says nothing about activity nobody recorded, and it isn't a production certificate.

Verification

Run python -m pytest ifixai/tests/trajectory -q for independent policy oracles, strict import/replay, adversarial grants, recorder outcomes and offline workflows. The example and benchmark use synthetic local facts, not production deployment evidence. Full repository checks remain ruff check ifixai, bandit -r ifixai -ll, ifixai validate and validation of every bundled example fixture. No API key, provider call or new runtime dependency is required by this workflow.