Skip to content

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

feat(telemetry): pseudonymous run telemetry via PostHog (#51)

n-papaioannouee7c7d75bac8
diff --git a/README.md b/README.md
index fb1c22e..137b6a8 100644
--- a/README.md
+++ b/README.md
@@ -227,6 +227,21 @@ Docs are sorted by what you came to do. Start in **[docs/](docs/)**:
- 📖 **Looking it up** → [CLI](docs/cli.md) · [Python API](docs/python-api.md) · [Scoring](docs/scoring.md) · [Inspections](docs/inspections.md)
- 💡 **Why it works this way** → [Methodology](docs/methodology.md)
+## Telemetry
+
+iFixAi sends pseudonymous run telemetry — a random local install id plus
+started/completed, the tool version, your OS name, which interface you used (CLI or
+plugin), and a timestamp — so we can see how many people use it and whether they return. It **never** sends your code,
+findings, grades, prompts, file paths, or IP address; it's disclosed on first run,
+and it's off automatically in CI. See exactly what would be sent:
+
+```bash
+ifixai run --print-telemetry
+```
+
+Opt out anytime with `--no-telemetry`, `IFIXAI_TELEMETRY=0`, or `DO_NOT_TRACK=1`.
+Full details, retention, and how to erase your data: **[SECURITY.md](SECURITY.md#telemetry)**.
+
## Contributing
Issues and PRs welcome. See **[CONTRIBUTING.md](CONTRIBUTING.md)**. Good first issues are
diff --git a/SECURITY.md b/SECURITY.md
index f9095f3..d37f07d 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -51,6 +51,33 @@ The scrubber (`ifixai/providers/secrets.py::scrub_secrets`) is parametrically ve
Generic `Authorization: Bearer <token>` and `X-API-Key: <value>` headers are scrubbed case-insensitively regardless of provider. Providers using custom auth schemes that do not match one of the shapes above fall back to the bearer-token regex when the value appears in a standard HTTP header.
+## Telemetry
+
+The CLI sends pseudonymous run telemetry so we can count how many people use iFixAi and whether they return. It is consent-first: disclosed in plain language on first run, off automatically in CI, and every opt-out is honored *before* any identifier is created or any event is sent.
+
+**What is sent** — one `ifixai_started` event per run, plus `ifixai_completed` when a run produces a report:
+
+| Field | Example | Why |
+|---|---|---|
+| install id | random `uuid4`, stored at `${XDG_CONFIG_HOME:-~/.config}/ifixai/install-id` (`0600`) | unique installs + retention |
+| event | `ifixai_started` / `ifixai_completed` | run count + completion rate |
+| version | `3.0.2` | adoption per release |
+| os | `Darwin` / `Linux` / `Windows` (from `platform.system()`) | platform mix |
+| surface | `cli` / `plugin` | separate the two run interfaces |
+| timestamp | ISO-8601 UTC | retention across days |
+
+**Never sent:** file contents, findings, grades, prompts, file/repo paths, hostnames, usernames, environment values, or your IP address. The collector (PostHog, US region) is configured to discard the request IP at ingestion, and every event also carries `$ip: null` and `$geoip_disable: true`. Inspect the exact payload anytime with `ifixai run --print-telemetry`.
+
+**Opt out** — any one of these disables telemetry (checked before anything is created or sent):
+
+- `--no-telemetry` (this run only)
+- `IFIXAI_TELEMETRY=0` (also `false` / `no` / `off`)
+- `DO_NOT_TRACK=1` (presence-based — any value)
+- a file at `${XDG_CONFIG_HOME:-~/.config}/ifixai/telemetry-opt-out`
+- automatically in CI (`CI`, or a known vendor variable such as `GITHUB_ACTIONS`)
+
+**Data protection (GDPR / UK GDPR).** The install id is a persistent identifier, so this telemetry is *pseudonymous personal data*, not anonymous. Lawful basis: legitimate interest (Art 6(1)(f)) in understanding adoption and retention; the processing is minimal (five fields, no model I/O), IP-stripped, opt-out-respecting, and never used for profiling or advertising. **Retention: events are kept indefinitely** — we do not auto-delete. **Erasure:** run `ifixai run --show-id` to read your install id, then email **[email protected]** to have its events deleted; deleting the local `install-id` file stops future linkage but does not remove rows already collected. Data controller: iMe (**[email protected]**).
+
## Coordinated disclosure
We prefer coordinated disclosure. If you give us at least the window described under *Reporting a vulnerability* above before publishing, we will credit you in the release notes of the fixing version — unless you request otherwise.
diff --git a/conftest.py b/conftest.py
index 40c9a8e..2b20062 100644
--- a/conftest.py
+++ b/conftest.py
@@ -1,6 +1,13 @@
import pytest
[email protected](autouse=True)
+def _disable_telemetry(monkeypatch):
+ """Never emit real telemetry from the test suite. Tests that exercise telemetry
+ re-enable it in their own fixtures (via monkeypatch.delenv)."""
+ monkeypatch.setenv("IFIXAI_TELEMETRY", "0")
+
+
def pytest_collection_modifyitems(
config: pytest.Config, items: list[pytest.Item]
) -> None:
diff --git a/ifixai/cli/run.py b/ifixai/cli/run.py
index c5cea21..c84acc7 100644
--- a/ifixai/cli/run.py
+++ b/ifixai/cli/run.py
@@ -15,6 +15,7 @@ from pathlib import Path
import click
from ifixai import __version__ as IFIXAI_VERSION
+from ifixai import telemetry
from ifixai.cli import ui
from ifixai.cli._branding import (
print_startup_banner,
@@ -582,6 +583,28 @@ def _print_concurrency_banner(resolved: int) -> None:
help="Suppress the startup banner and the post-run iMe Core conclusion. "
"Stdout still contains scores so CI gates keep working.",
)
+ "--no-telemetry",
+ "no_telemetry",
+ is_flag=True,
+ default=False,
+ help="Disable pseudonymous run telemetry for this run. Set IFIXAI_TELEMETRY=0 or "
+ "DO_NOT_TRACK=1 to disable it permanently.",
+)
+ "--print-telemetry",
+ "print_telemetry",
+ is_flag=True,
+ default=False,
+ help="Print the exact pseudonymous telemetry payload that would be sent, then exit.",
+)
+ "--show-id",
+ "show_install_id",
+ is_flag=True,
+ default=False,
+ help="Print this install's telemetry id (for data-erasure requests), then exit.",
+)
@click.pass_context
def run(
ctx: click.Context,
@@ -625,8 +648,20 @@ def run(
run_nonce: str | None,
grounding: str,
quiet: bool,
+ no_telemetry: bool,
+ print_telemetry: bool,
+ show_install_id: bool,
) -> None:
"""Run ifixai against a target AI assistant."""
+ if print_telemetry:
+ telemetry.print_payload()
+ return
+ if show_install_id:
+ click.echo(
+ telemetry.show_id()
+ or "No telemetry id yet (telemetry hasn't run or is opted out)."
+ )
+ return
run_start_monotonic = time.monotonic()
try:
@@ -698,6 +733,10 @@ def run(
b32_seed_pinned = b32_seed is not None
print_startup_banner(IFIXAI_VERSION, quiet=quiet)
+ if no_telemetry:
+ telemetry.disable()
+ telemetry.show_disclosure()
+ telemetry.emit_started("cli")
resolved_concurrency = _resolve_concurrency(concurrency, no_parallel)
_print_concurrency_banner(resolved_concurrency)
concurrency_governor = ConcurrencyGovernor(resolved_concurrency)
@@ -1404,6 +1443,8 @@ def run(
click.echo(f" Run nonce: {manifest.run_nonce}")
click.echo(f" Manifest: {manifest_path}")
+ telemetry.emit_completed("cli")
+
if result.overall_score is None:
click.echo(
click.style(
diff --git a/ifixai/plugin/orchestrator.py b/ifixai/plugin/orchestrator.py
index 2af71ed..f42b7ae 100644
--- a/ifixai/plugin/orchestrator.py
+++ b/ifixai/plugin/orchestrator.py
@@ -30,6 +30,7 @@ import sys
from pathlib import Path
from typing import NamedTuple, cast
+from ifixai import telemetry
from ifixai.api import run_selected
from ifixai.core.fixture_loader import load_fixture
from ifixai.evaluation.manifest import generate_run_nonce
@@ -1045,8 +1046,16 @@ def main() -> None:
"bad model id before billing (escape hatch for a provider that rejects "
"tiny probes)",
)
+ parser.add_argument(
+ "--no-telemetry", action="store_true",
+ help="disable pseudonymous run telemetry for this run (IFIXAI_TELEMETRY=0 "
+ "or DO_NOT_TRACK=1 disables it permanently)",
+ )
args = parser.parse_args()
+ if args.no_telemetry:
+ telemetry.disable()
+
live = args.mode in LIVE_MODES
sut_provider = args.provider.lower()
@@ -1262,6 +1271,12 @@ def main() -> None:
print("\nRe-run with --yes to start.")
return
+ # Offline rehearsals (stub/record/replay) are not real runs — only live runs
+ # emit, so the metric counts diagnostics, not demos.
+ if live:
+ telemetry.show_disclosure()
+ telemetry.emit_started("plugin")
+
# The profile-derived fixture was already written to a visible file on the
# confirm screen (above); run from exactly that file so what the user saw and
# could edit is what gets graded. No throwaway temp, nothing deleted.
@@ -1381,6 +1396,8 @@ def main() -> None:
"This rehearses the pipeline; it is NOT a diagnostic of a real agent. ***\n"
)
print(generate_markdown_report(result))
+ if live:
+ telemetry.emit_completed("plugin")
print("\n--- coverage ---")
for pillar, score, cov in _coverage_rows(result):
print(f" {pillar:18} {score:>5} {cov}")
diff --git a/ifixai/telemetry.py b/ifixai/telemetry.py
new file mode 100644
index 0000000..9555810
--- /dev/null
+++ b/ifixai/telemetry.py
@@ -0,0 +1,315 @@
+"""Pseudonymous run telemetry for iFixAi.
+
+We count how often iFixAi is run, by how many distinct installs, and whether the
+same install returns across days. One ``ifixai_started`` event per run plus an
+``ifixai_completed`` event on a finished run, keyed by a random per-install id
+stored locally. Stdlib only: a synchronous ``urllib`` POST on a daemon thread,
+deliberately not aiohttp (the run paths close their asyncio loop before exit, so
+the existing session can't be reused at flush time).
+
+Telemetry must never block, slow, or break a run: every path is wrapped, the send
+is fire-and-forget with a hard timeout, and failures are swallowed. See the
+Telemetry section of ``SECURITY.md`` for what's sent, the opt-outs, and the
+privacy posture.
+"""
+
+from __future__ import annotations
+
+import atexit
+import json
+import os
+import platform
+import queue
+import ssl
+import sys
+import threading
+import urllib.request
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+
+from ifixai._version import VERSION
+
+# PostHog (US region) capture endpoint + project key. The project key is public
+# by design (write-only: it can send events, never read them), so it ships as a
+# constant. Pinned literals, no env/flag override (an overridable URL would be an
+# SSRF/redirect primitive).
+POSTHOG_ENDPOINT = "https://us.i.posthog.com/i/v0/e/"
+POSTHOG_PROJECT_KEY = "phc_sSAKNHJwLY748x2BtZvDPR5xdnMr7PndS2gmnb2LsD2H"
+
+# The only user-data property keys that may ever leave the machine.
+_ALLOWED_PROPS = frozenset({"version", "os", "surface"})
+
+_REQUEST_TIMEOUT = 1.0 # seconds, hard cap per send
+_FLUSH_BUDGET = 1.0 # seconds, total atexit join budget
+
+# Env vars that indicate an automated CI run (telemetry off).
+_CI_VENDOR_VARS = (
+ "GITHUB_ACTIONS", "GITLAB_CI", "BUILDKITE", "CIRCLECI",
+ "JENKINS_URL", "TF_BUILD", "TEAMCITY_VERSION", "TRAVIS", "APPVEYOR",
+)
+
+_DISCLOSURE_TEXT = (
+ "iFixAi sends pseudonymous run telemetry: a random local install id (a "
+ "persistent identifier, so it counts as personal data under GDPR), whether a "
+ "run started and completed, the tool version, your OS name, whether you ran "
+ "the CLI or the plugin, and a UTC timestamp. It never sends your code, "
+ "findings, grades, prompts, file paths, "
+ "or IP address — run with --print-telemetry to see exactly what goes out. "
+ "Opt out anytime with --no-telemetry, IFIXAI_TELEMETRY=0, or DO_NOT_TRACK=1 "
+ "(and it's off automatically in CI); opting out sends nothing."
+)
+
+_started_emitted = False
+_completed_emitted = False
+_force_disabled = False
+_sender: "_Sender | None" = None
+_sender_lock = threading.Lock()
+
+
+# --------------------------------------------------------------------------- #
+# Storage paths
+# --------------------------------------------------------------------------- #
+def _config_dir() -> Path:
+ base = os.environ.get("XDG_CONFIG_HOME") or str(Path.home() / ".config")
+ return Path(base) / "ifixai"
+
+
+def _id_path() -> Path:
+ return _config_dir() / "install-id"
+
+
+def _disclosure_path() -> Path:
+ return _config_dir() / "disclosure-shown"
+
+
+def _optout_path() -> Path:
+ return _config_dir() / "telemetry-opt-out"
+
+
+def _ensure_config_dir() -> Path:
+ """Create the config dir (best-effort ``0700``) and return it."""
+ directory = _config_dir()
+ directory.mkdir(parents=True, exist_ok=True)
+ try:
+ os.chmod(directory, 0o700)
+ except OSError:
+ pass
+ return directory
+
+
+# --------------------------------------------------------------------------- #
+# Opt-out resolution
+# --------------------------------------------------------------------------- #
+def disable() -> None:
+ """Disable telemetry for this process (wired to the ``--no-telemetry`` flag)."""
+ global _force_disabled
+ _force_disabled = True
+
+
+def _in_ci() -> bool:
+ ci = os.environ.get("CI")
+ if ci is not None and ci.strip().lower() not in {"", "0", "false", "no"}:
+ return True
+ return any(v in os.environ for v in _CI_VENDOR_VARS)
+
+
+def is_enabled() -> bool:
+ """Resolve every opt-out. Off if any matches; a filesystem error reads as off."""
+ if _force_disabled:
+ return False
+ val = os.environ.get("IFIXAI_TELEMETRY")
+ if val is not None and val.strip().lower() in {"0", "false", "no", "off"}:
+ return False
+ if "DO_NOT_TRACK" in os.environ: # presence-based, any value incl. "0"
+ return False
+ if _in_ci():
+ return False
+ try:
+ if _optout_path().exists():
+ return False
+ except OSError:
+ return False
+ return True
+
+
+# --------------------------------------------------------------------------- #
+# Install id
+# --------------------------------------------------------------------------- #
+def _read_id() -> str | None:
+ try:
+ value = _id_path().read_text(encoding="utf-8").strip()
+ return value or None
+ except OSError:
+ return None
+
+
+def install_id() -> str | None:
+ """Return the persistent install id, creating it once. ``None`` on fs failure.
+
+ Creation is an exclusive create (``O_EXCL``): if two fresh runs race, only the
+ first writer makes the file and everyone re-reads the same winning id.
+ """
+ existing = _read_id()
+ if existing is not None:
+ return existing
+ path = _id_path()
+ try:
+ _ensure_config_dir()
+ new_id = str(uuid.uuid4())
+ try:
+ fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ try:
+ os.write(fd, new_id.encode("utf-8"))
+ finally:
+ os.close(fd)
+ except FileExistsError:
+ pass # another process won the race; read its id below
+ return _read_id()
+ except OSError:
+ return None
+
+
+# --------------------------------------------------------------------------- #
+# Disclosure
+# --------------------------------------------------------------------------- #
+def show_disclosure() -> None:
+ """Print the one-time disclosure (to stderr) and mark it shown. Idempotent."""
+ if not is_enabled():
+ return
+ marker = _disclosure_path()
+ try:
+ if marker.exists():
+ return
+ except OSError:
+ return
+ print(_DISCLOSURE_TEXT, file=sys.stderr)
+ try:
+ _ensure_config_dir()
+ marker.write_text("", encoding="utf-8")
+ except OSError:
+ pass
+
+
+# --------------------------------------------------------------------------- #
+# Event emission
+# --------------------------------------------------------------------------- #
+def emit_started(surface: str) -> None:
+ """Enqueue one ``ifixai_started`` event (idempotent per process).
+
+ ``surface`` is ``"cli"`` or ``"plugin"`` so the two run paths stay separable.
+ """
+ global _started_emitted
+ if _started_emitted:
+ return
+ _started_emitted = True
+ _enqueue("ifixai_started", surface)
+
+
+def emit_completed(surface: str) -> None:
+ """Enqueue an ``ifixai_completed`` event for a finished run (idempotent per process)."""
+ global _completed_emitted
+ if _completed_emitted:
+ return
+ _completed_emitted = True
+ _enqueue("ifixai_completed", surface)
+
+
+def _build_payload(event: str, install: str, surface: str) -> bytes:
+ # Build strictly FROM the allowlist so nothing outside it can ever be sent —
+ # an `assert` would be stripped under `python -O`, silently killing the guard.
+ # sorted() keeps the --print-telemetry output deterministic.
+ source = {"version": VERSION, "os": platform.system(), "surface": surface}
+ props: dict[str, object] = {key: source[key] for key in sorted(_ALLOWED_PROPS)}
+ props["$ip"] = None # PostHog: don't capture/derive geo from the request IP
+ props["$geoip_disable"] = True
+ envelope = {
+ "api_key": POSTHOG_PROJECT_KEY,
+ "event": event,
+ "distinct_id": install,
+ "properties": props,
+ "timestamp": datetime.now(timezone.utc).isoformat(),
+ }
+ return json.dumps(envelope).encode("utf-8")
+
+
+def _enqueue(event: str, surface: str) -> None:
+ try:
+ if not is_enabled():
+ return
+ install = install_id()
+ if install is None:
+ return
+ _get_sender().submit(_build_payload(event, install, surface))
+ except Exception:
+ pass # telemetry must never break a run
+
+
+# --------------------------------------------------------------------------- #
+# Background sender (daemon thread + atexit flush)
+# --------------------------------------------------------------------------- #
+class _Sender:
+ def __init__(self) -> None:
+ self._queue: "queue.Queue[bytes | None]" = queue.Queue()
+ self._opener = urllib.request.build_opener(
+ urllib.request.ProxyHandler({}), # bypass ambient http(s)_proxy
+ urllib.request.HTTPSHandler(context=ssl.create_default_context()),
+ )
+ self._thread = threading.Thread(target=self._run, daemon=True)
+ self._thread.start()
+
+ def submit(self, body: bytes) -> None:
+ self._queue.put(body)
+
+ def flush(self, budget: float) -> None:
+ self._queue.put(None) # sentinel: drain remaining items, then stop
+ self._thread.join(timeout=budget)
+
+ def _run(self) -> None:
+ while True:
+ body = self._queue.get()
+ if body is None:
+ return
+ try:
+ req = urllib.request.Request(
+ POSTHOG_ENDPOINT,
+ data=body,
+ headers={"Content-Type": "application/json"},
+ method="POST",
+ )
+ with self._opener.open(req, timeout=_REQUEST_TIMEOUT) as resp:
+ resp.read()
+ except Exception:
+ pass # dead/blocked endpoint must never raise
+
+
+def _get_sender() -> "_Sender":
+ global _sender
+ with _sender_lock:
+ if _sender is None:
+ _sender = _Sender()
+ atexit.register(_flush)
+ return _sender
+
+
+def _flush() -> None:
+ if _sender is not None:
+ try:
+ _sender.flush(_FLUSH_BUDGET)
+ except Exception:
+ pass
+
+
+# --------------------------------------------------------------------------- #
+# CLI affordances
+# --------------------------------------------------------------------------- #
+def print_payload() -> None:
+ """``--print-telemetry``: print the exact ``started`` JSON without sending or creating an id."""
+ install = _read_id() or "<install-id created on first real run>"
+ body = json.loads(_build_payload("ifixai_started", install, "cli"))
+ print(json.dumps(body, indent=2))
+
+
+def show_id() -> str | None:
+ """``--show-id``: return the stored install id (for erasure requests), or None."""
+ return _read_id()