Skip to content

AI powered OSINT engine to discover and map 840+ social media identities across the web

feat: OSINT investigation suite expansion (13 features)

arxhr0075a4b9e353a54
diff --git a/README.md b/README.md
index 9083c0f..3cd5322 100644
--- a/README.md
+++ b/README.md
@@ -25,13 +25,20 @@
## Highlights
- **840+ platforms** scanned asynchronously in seconds
-- **ML + heuristic detection** — a trained model blended with 25 structural signals (HTTP status, DOM shape, keywords, fingerprints) instead of naive status-code checks
-- **Modern terminal UI** — live progress, sorted result tables, summary panels (powered by [rich](https://github.com/Textualize/rich))
+- **ML + heuristic detection** — a trained model blended with 30 structural signals (HTTP status, DOM shape, keywords, fingerprints) instead of naive status-code checks
+- **Profile extraction** — display name, bio, and avatar pulled from each hit (OpenGraph / JSON-LD / per-site CSS)
+- **Cross-site correlation** — cluster profiles that look like the same person by avatar hash, bio, shared links, and name (`--correlate`)
+- **Recursive expansion** — follow linked usernames out of bios and re-scan them (`--recurse-depth N`)
+- **Domain check** — is `<username>.{com,io,net,…}` registered and live? (`--domains`)
+- **Watch mode** — re-scan on an interval and alert on changes, optionally to a webhook (`--watch 6h --notify <url>`)
+- **Resumable scans** — checkpoint progress and continue after an interruption (`--resume file.jsonl`)
+- **Modern terminal UI** — live progress, sorted result tables, summary panels (powered by [rich](https://github.com/Textualize/rich)); plus an interactive browser (`aliens_eye tui`, optional extra)
+- **MCP server** — expose scanning to LLM agents (`aliens_eye serve`, optional extra)
- **Proxy & Tor support** — `--proxy socks5://...` or just `--tor`
-- **Site filtering** — `--site github,reddit`, `--exclude-site`, `--no-nsfw`
-- **Self-check** — `aliens_eye selfcheck` validates detection accuracy against accounts known to exist
-- **Retrainable** — collect your own labeled dataset and retrain the model with `aliens_eye train`
-- **Reports** in JSON, CSV, HTML, and Markdown
+- **Site filtering** — `--site github,reddit`, `--exclude-site`, `--no-nsfw`, plus drop-in `sites.d/` plugin site maps
+- **Calibrated self-check** — `aliens_eye selfcheck` reports precision / recall / F1 / FPR per site
+- **Retrainable + active learning** — retrain with `aliens_eye train`, or hand-label uncertain hits with `aliens_eye label`
+- **Reports** in JSON, CSV, HTML, Markdown, PDF, and graph formats (GEXF, Mermaid, Maltego CSV)
- **Playwright fallback** for JavaScript-heavy pages (optional extra)
## Install
@@ -47,6 +54,10 @@ pip install "aliens-eye[browser]" # Playwright fallback for hard pages
python -m playwright install chromium
pip install "aliens-eye[train]" # scikit-learn, for retraining the ML model
+pip install "aliens-eye[correlate]" # Pillow, for avatar-image matching in --correlate
+pip install "aliens-eye[pdf]" # reportlab, for --format pdf
+pip install "aliens-eye[tui]" # textual, for the interactive `tui` browser
+pip install "aliens-eye[serve]" # mcp, for the `serve` MCP server
```
Or with Docker:
@@ -106,13 +117,47 @@ aliens_eye username --plain
# View results from a previous scan
aliens_eye -r results/username_advanced_20260611_120000.json
-# Validate detection accuracy against known accounts
-aliens_eye selfcheck
+# Correlate hits into "likely same person" clusters + check domains
+aliens_eye username --correlate --domains
+
+# Follow linked usernames out of found bios and re-scan them
+aliens_eye username --recurse-depth 1
+
+# Export a graph of the results (import into Gephi / Maltego / Mermaid)
+aliens_eye username --correlate --format gexf,mermaid,maltego
+
+# Investigator PDF with embedded avatars
+aliens_eye username --format pdf
+
+# Watch for changes every 6 hours and POST them to a webhook
+aliens_eye username --watch 6h --notify https://hooks.example/aliens
+
+# Resume an interrupted scan
+aliens_eye username --resume scan.jsonl
+
+# Compare two saved reports
+aliens_eye diff results/old.json results/new.json
+
+# Validate detection accuracy (precision / recall / F1 per site)
+aliens_eye selfcheck --negatives 2 --report json
+
+# Interactively label uncertain hits into a training set
+aliens_eye label results/username_basic_20260611_120000.json --out labeled.csv
+
+# Interactive terminal browser (needs [tui])
+aliens_eye tui username
+
+# Run the MCP server for LLM agents (needs [serve])
+aliens_eye serve
```
+Custom platforms: drop a `{ "site_name": "https://site/{}" }` JSON file into
+`./sites.d/` (or the user config dir's `sites.d/`) and it is merged automatically;
+`--sites-dir DIR` adds another location.
+
## How detection works
-Every response is converted into a 25-dimensional feature vector: HTTP status buckets, username placement (path/title/meta), error and profile keywords, DOM structure (images, forms, profile/error CSS classes), response timing, redirect counts, and per-site fingerprint matches learned from previous scans.
+Every response is converted into a 30-dimensional feature vector: HTTP status buckets, username placement (path/title/meta/canonical), error and profile keywords, DOM structure (images, forms, profile/error CSS classes), structured-data signals (og:type, JSON-LD Person), response timing, redirect counts, and per-site fingerprint matches learned from previous scans.
Two judges then vote:
diff --git a/pyproject.toml b/pyproject.toml
index 8354a1c..1b33d09 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -35,6 +35,10 @@ dependencies = [
[project.optional-dependencies]
browser = ["playwright>=1.42"]
train = ["scikit-learn>=1.3", "numpy>=1.24"]
+correlate = ["Pillow>=10.0"]
+tui = ["textual>=0.60"]
+pdf = ["reportlab>=4.0"]
+serve = ["mcp>=1.0"]
dev = [
"pytest>=8",
"pytest-asyncio>=0.23",
diff --git a/src/aliens_eye/cli.py b/src/aliens_eye/cli.py
index 255705c..0980dd8 100644
--- a/src/aliens_eye/cli.py
+++ b/src/aliens_eye/cli.py
@@ -23,6 +23,7 @@ from aliens_eye.core.scanner import (
UsernameScanner,
filter_sites,
load_nsfw_sites,
+ load_site_plugins,
load_sites_data,
)
from aliens_eye.utils.console import get_console, set_plain
@@ -123,6 +124,39 @@ def build_parser() -> ArgumentParser:
choices=["quick", "full", "aggressive"],
help="Scan profile preset (skips interactive prompt)",
)
+ parser.add_argument(
+ "--watch",
+ type=str,
+ help="Re-scan on an interval and alert on changes (e.g. 30m, 6h, 1d)",
+ )
+ parser.add_argument("--notify", type=str, help="Webhook URL to POST watch-mode changes to")
+ parser.add_argument(
+ "--correlate",
+ action="store_true",
+ help="Cluster Found/Maybe profiles that look like the same person (avatar/bio/links)",
+ )
+ parser.add_argument(
+ "--domains",
+ action="store_true",
+ help="Check whether <username>.{com,io,net,...} domains are registered/live",
+ )
+ parser.add_argument(
+ "--recurse-depth",
+ type=int,
+ default=0,
+ help="Extract linked usernames from found profiles and re-scan, up to N hops",
+ )
+ parser.add_argument("--sites-dir", type=str, help="Directory of extra *.json site maps to merge")
+ parser.add_argument("--resume", type=str, help="Checkpoint file to resume an interrupted scan from")
+ parser.add_argument("--from-email", type=str, help="Seed usernames from an email address")
+ parser.add_argument("--from-name", type=str, help="Seed usernames from a real name")
+ parser.add_argument("--from-phone", type=str, help="Seed usernames from a phone number")
+ parser.add_argument(
+ "--only-found", action="store_true", help="Report only Found/Maybe results"
+ )
+ parser.add_argument(
+ "--json-stdout", action="store_true", help="Print results JSON to stdout (implies --plain)"
+ )
return parser
@@ -136,6 +170,13 @@ def build_selfcheck_parser() -> ArgumentParser:
parser.add_argument("--model", type=str, help="Path to a custom ML model JSON")
parser.add_argument("--sites", type=str, help="Path to a custom sites JSON")
parser.add_argument("--plain", action="store_true", help="Plain output")
+ parser.add_argument(
+ "--negatives", type=int, default=1, help="Random non-existent usernames to test per site"
+ )
+ parser.add_argument(
+ "--report", choices=["table", "json"], default="table",
+ help="Output format: rich table (default) or machine-readable JSON",
+ )
return parser
@@ -160,6 +201,58 @@ def build_train_parser() -> ArgumentParser:
return parser
+def build_diff_parser() -> ArgumentParser:
+ parser = ArgumentParser(
+ prog="aliens_eye diff",
+ description="Compare two saved scan reports and show what changed",
+ )
+ parser.add_argument("old", help="Path to the older report JSON")
+ parser.add_argument("new", help="Path to the newer report JSON")
+ parser.add_argument("--plain", action="store_true", help="Plain output")
+ parser.add_argument(
+ "--json-stdout", action="store_true", help="Print the diff as JSON to stdout"
+ )
+ return parser
+
+
+def build_serve_parser() -> ArgumentParser:
+ parser = ArgumentParser(
+ prog="aliens_eye serve",
+ description="Run an MCP server exposing scan tools over stdio",
+ )
+ parser.add_argument("--sites", type=str, help="Path to a custom sites JSON")
+ parser.add_argument("--no-ml", action="store_true", help="Disable ML detection")
+ parser.add_argument("-v", "--verbose", action="store_true")
+ return parser
+
+
+def build_tui_parser() -> ArgumentParser:
+ parser = ArgumentParser(
+ prog="aliens_eye tui",
+ description="Interactive terminal UI for scanning",
+ )
+ parser.add_argument("username", nargs="?", help="Username to scan")
+ parser.add_argument("-l", "--level", choices=["basic", "intermediate", "advanced"], default="basic")
+ parser.add_argument("--sites", type=str, help="Path to a custom sites JSON")
+ parser.add_argument("--no-ml", action="store_true", help="Disable ML detection")
+ return parser
+
+
+def build_label_parser() -> ArgumentParser:
+ parser = ArgumentParser(
+ prog="aliens_eye label",
+ description="Interactively label low-confidence results and append them to a dataset",
+ )
+ parser.add_argument("report", help="Path to a saved report JSON to label")
+ parser.add_argument("--out", type=str, default="labeled.csv", help="Dataset CSV to append to")
+ parser.add_argument(
+ "--band", type=str, default="Maybe",
+ help="Which statuses to review: 'Maybe' (default) or 'all'",
+ )
+ parser.add_argument("-v", "--verbose", action="store_true")
+ return parser
+
+
def parse_formats(value: str) -> list[str]:
formats = [item.strip().lower() for item in value.split(",") if item.strip()]
if not formats:
@@ -184,6 +277,22 @@ def parse_site_list(value: str | None) -> list[str] | None:
return items or None
+def seed_usernames_from_lookups(args) -> list[str]:
+ """Expand --from-email/--from-name/--from-phone into candidate usernames."""
+ from aliens_eye.core.variations import usernames_from_email, usernames_from_name
+
+ seeds: list[str] = []
+ if getattr(args, "from_email", None):
+ seeds.extend(usernames_from_email(args.from_email))
+ if getattr(args, "from_name", None):
+ seeds.extend(usernames_from_name(args.from_name))
+ if getattr(args, "from_phone", None):
+ digits = "".join(ch for ch in args.from_phone if ch.isdigit())
+ if len(digits) >= 4:
+ seeds.append(digits)
+ return seeds
+
+
def prompt_scan_profile() -> str:
console = get_console()
console.print("\n[green]Select scan profile:[/green]")
@@ -345,16 +454,27 @@ def apply_cli_overrides(config: ScannerConfig, args) -> None:
config.exclude_nsfw = True
config.include_sites = parse_site_list(args.site)
config.exclude_sites = parse_site_list(args.exclude_site)
+ if args.only_found:
+ config.only_found = True
+ if args.json_stdout:
+ config.json_stdout = True
+ config.plain_output = True
if args.plain:
config.plain_output = True
-def prepare_sites(config: ScannerConfig, logger) -> dict:
+def prepare_sites(config: ScannerConfig, logger, sites_dir: str | None = None) -> dict:
sites_data = load_sites_data(config.sites_path)
if not sites_data:
logger.error("Could not load sites.json.")
return {}
+ extra_dirs = [Path(sites_dir)] if sites_dir else None
+ plugins = load_site_plugins(extra_dirs)
+ if plugins:
+ logger.info("Merged %d plugin site(s) from sites.d directories", len(plugins))
+ sites_data = {**sites_data, **plugins}
+
exclude = list(config.exclude_sites or [])
if config.exclude_nsfw:
exclude.extend(load_nsfw_sites())
@@ -417,7 +537,9 @@ async def run_scan(args) -> None:
config = ScannerConfig()
defaults = apply_config(config, config_data)
apply_cli_overrides(config, args)
- if config.plain_output:
+ if config.json_stdout:
+ set_plain(True, stderr=True)
+ elif config.plain_output:
set_plain(True)
console = get_console()
@@ -426,15 +548,20 @@ async def run_scan(args) -> None:
ResultsExporter(exporter_dir).display_results_from_file(args.read)
return
- sites_data = prepare_sites(config, logger)
+ sites_data = prepare_sites(config, logger, sites_dir=args.sites_dir)
if not sites_data:
return
- display_banner(len(sites_data))
+ if not config.json_stdout:
+ display_banner(len(sites_data))
- if args.username:
- usernames = args.username
- else:
+ usernames = list(args.username)
+ usernames.extend(seed_usernames_from_lookups(args))
+ usernames = list(dict.fromkeys(usernames))
+ if not usernames:
+ if config.json_stdout:
+ logger.error("No username provided.")
+ return
username = Prompt.ask("[green]Enter username to scan[/green]", console=console)
usernames = [username]
@@ -447,7 +574,7 @@ async def run_scan(args) -> None:
fingerprints = FingerprintStore(config.fingerprints_path, config.max_fingerprints_per_label)
fingerprints.load(logger)
- interactive = sys.stdin.isatty() and not args.read
+ interactive = sys.stdin.isatty() and not args.read and not config.json_stdout
scan_level = args.level or defaults.get("level")
if scan_level not in {"basic", "intermediate", "advanced"}:
@@ -473,6 +600,12 @@ async def run_scan(args) -> None:
if config.use_playwright:
browser_fallback = BrowserFallback(config.timeout, DEFAULT_USER_AGENT)
+ checkpoint = None
+ if args.resume:
+ from aliens_eye.core.checkpoint import Checkpoint
+
+ checkpoint = Checkpoint(args.resume)
+
exporter = ResultsExporter(config.output_dir)
scanner = UsernameScanner(
sites_data=sites_data,
@@ -482,8 +615,36 @@ async def run_scan(args) -> None:
fingerprints=fingerprints,
logger=logger,
browser_fallback=browser_fallback,
+ checkpoint=checkpoint,
)
+ if args.watch:
+ from aliens_eye.core import watch as watch_mod
+
+ try:
+ interval = watch_mod.parse_duration(args.watch)
+ except ValueError as exc:
+ logger.error("Invalid --watch interval: %s", exc)
+ return
+ try:
+ await watch_mod.run_watch(
+ scanner=scanner,
+ exporter=exporter,
+ usernames=usernames,
+ scan_level=scan_level,
+ formats=formats,
+ interval=interval,
+ logger=logger,
+ console=console,
+ output_dir=config.output_dir,
+ notify_url=args.notify,
+ )
+ finally:
+ fingerprints.save()
+ if browser_fallback:
+ await browser_fallback.close()
+ return
+
try:
for username in usernames:
if not username or len(username) < 2:
@@ -491,7 +652,31 @@ async def run_scan(args) -> None:
continue
all_results = await scanner.scan_with_variations(username, scan_level)
- written = exporter.save_results(username, scan_level, all_results, formats)
+
+ if args.recurse_depth and args.recurse_depth > 0:
+ await _recurse_scan(
+ scanner, all_results, username, scan_level, args.recurse_depth, console
+ )
+
+ correlation = None
+ if args.correlate:
+ from aliens_eye.core.correlate import correlate_report
+
+ report = exporter.report_dict(username, scan_level, all_results)
+ correlation = await correlate_report(report, proxy=config.proxy, timeout=config.timeout)
+ _print_clusters(console, correlation)
+
+ domains = None
+ if args.domains:
+ from aliens_eye.core.domains import check_domains
+
+ domains = await check_domains(username, proxy=config.proxy)
+ _print_domains(console, domains)
+
+ written = exporter.save_results(
+ username, scan_level, all_results, formats,
+ correlation=correlation, domains=domains,
+ )
files_list = ", ".join(str(path) for path in written)
console.print(f"\n[green]Results saved to:[/green] [blue]{files_list}[/blue]")
@@ -503,16 +688,82 @@ async def run_scan(args) -> None:
f"[green]Summary: Found {total_found} profiles across "
f"{len(all_results)} username variations[/green]"
)
+ if checkpoint is not None:
+ checkpoint.finalize()
finally:
fingerprints.save()
if browser_fallback:
await browser_fallback.close()
+async def _recurse_scan(scanner, all_results, base_username, scan_level, depth, console) -> None:
+ """Extract linked usernames from found profiles and scan them, up to `depth` hops."""
+ from aliens_eye.core.expand import candidate_usernames_from_results
+
+ seen = {base_username.lower()}
+ frontier = candidate_usernames_from_results(all_results, exclude=seen)
+ for hop in range(depth):
+ frontier = [u for u in frontier if u.lower() not in seen]
+ if not frontier:
+ break
+ console.print(
+ f"[blue]Recursion hop {hop + 1}:[/blue] {len(frontier)} linked username(s): "
+ f"[dim]{', '.join(frontier[:8])}{'...' if len(frontier) > 8 else ''}[/dim]"
+ )
+ next_frontier: list[str] = []
+ for uname in frontier:
+ seen.add(uname.lower())
+ sub = await scanner.scan_all_sites(uname)
+ all_results[uname] = sub
+ next_frontier.extend(
+ candidate_usernames_from_results({uname: sub}, exclude=seen)
+ )
+ frontier = list(dict.fromkeys(next_frontier))
+
+
+def _print_clusters(console, correlation: dict) -> None:
+ clusters = correlation.get("clusters", [])
+ if not clusters:
+ console.print("[dim]Correlation: no cross-site clusters found.[/dim]")
+ return
+ from rich.table import Table
+
+ table = Table(title="Correlation: likely same person", header_style="bold blue")
+ table.add_column("#", justify="right")
+ table.add_column("Sites", style="yellow")
+ table.add_column("Linked by")
+ for i, cluster in enumerate(clusters, 1):
+ sites = ", ".join(f"{m['site']}" for m in cluster["members"])
+ table.add_row(str(i), sites, ", ".join(cluster.get("reasons", [])))
+ console.print(table)
+ if not correlation.get("avatar_hashing"):
+ console.print(
+ "[dim]Tip: install \"aliens-eye\\[correlate]\" for avatar-image matching.[/dim]"
+ )
+
+
+def _print_domains(console, domains: dict) -> None:
+ from rich.table import Table
+
+ table = Table(title="Domain check", header_style="bold blue")
+ table.add_column("Domain", style="yellow")
+ table.add_column("Registered")
+ table.add_column("Live")
+ for entry in domains.get("results", []):
+ table.add_row(
+ entry["domain"],
+ "[green]yes[/green]" if entry["registered"] else "[dim]no[/dim]",
+ "[green]yes[/green]" if entry["live"] else "[dim]no[/dim]",
+ )
+ console.print(table)
+
+
async def run_selfcheck_command(args) -> None:
from aliens_eye.selfcheck import run_selfcheck
- if args.plain:
+ if args.report == "json":
+ set_plain(True, stderr=True)
+ elif args.plain:
set_plain(True)
logger = setup_logger(args.verbose)
config = ScannerConfig(retries=1)
@@ -520,7 +771,10 @@ async def run_selfcheck_command(args) -> None:
detector = Detector()
if not args.no_ml:
detector.load_model(logger, Path(args.model) if args.model else None)
- await run_selfcheck(sites_data, detector, config, logger)
+ await run_selfcheck(
+ sites_data, detector, config, logger,
+ negatives=args.negatives, report_format=args.report,
+ )
async def run_train_command(args) -> None:
@@ -550,6 +804,68 @@ async def run_train_command(args) -> None:
)
+def run_diff_command(args) -> None:
+ from aliens_eye.core.report import ReportError, diff_reports, load_report
+
+ if args.json_stdout:
+ set_plain(True, stderr=True)
+ elif args.plain:
+ set_plain(True)
+ console = get_console()
+ try:
+ old = load_report(args.old)
+ new = load_report(args.new)
+ except ReportError as exc:
+ console.print(f"[red]{exc}[/red]")
+ sys.exit(1)
+
+ delta = diff_reports(old, new)
+ if args.json_stdout:
+ print(json.dumps(delta, indent=2))
+ return
+
+ from rich.table import Table
+
+ if not (delta["new"] or delta["gone"] or delta["changed"]):
+ console.print("[dim]No differences between the two reports.[/dim]")
+ return
+
+ table = Table(title="Report diff", header_style="bold blue")
+ table.add_column("Change")
+ table.add_column("Account (variation:site)", style="yellow")
+ table.add_column("Detail")
+ for item in delta["new"]:
+ table.add_row("[green]+ NEW[/green]", item["key"], item.get("url", ""))
+ for item in delta["gone"]:
+ table.add_row("[red]- GONE[/red]", item["key"], item.get("url", ""))
+ for item in delta["changed"]:
+ before, after = item["before"], item["after"]
+ detail = (
+ f"{before['status']} ({before['confidence']}%) -> "
+ f"{after['status']} ({after['confidence']}%)"
+ )
+ table.add_row("[yellow]~ CHANGED[/yellow]", item["key"], detail)
+ console.print(table)
+
+
+async def run_serve_command(args) -> None:
+ from aliens_eye.mcp_server import serve
+
+ await serve(args)
+
+
+def run_tui_command(args) -> None:
+ from aliens_eye.tui.app import run_tui
+
+ run_tui(args)
+
+
+async def run_label_command(args) -> None:
+ from aliens_eye.ml.label import label_report
+
+ await label_report(args)
+
+
def main() -> None:
if sys.platform == "win32":
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[union-attr]
@@ -563,6 +879,18 @@ def main() -> None:
elif argv and argv[0] == "train":
args = build_train_parser().parse_args(argv[1:])
asyncio.run(run_train_command(args))
+ elif argv and argv[0] == "diff":
+ args = build_diff_parser().parse_args(argv[1:])
+ run_diff_command(args)
+ elif argv and argv[0] == "serve":
+ args = build_serve_parser().parse_args(argv[1:])
+ asyncio.run(run_serve_command(args))
+ elif argv and argv[0] == "tui":
+ args = build_tui_parser().parse_args(argv[1:])
+ run_tui_command(args)
+ elif argv and argv[0] == "label":
+ args = build_label_parser().parse_args(argv[1:])
+ asyncio.run(run_label_command(args))
else:
args = build_parser().parse_args(argv)
asyncio.run(run_scan(args))
diff --git a/src/aliens_eye/core/analyzer.py b/src/aliens_eye/core/analyzer.py
index af5dcd5..6742190 100644
--- a/src/aliens_eye/core/analyzer.py
+++ b/src/aliens_eye/core/analyzer.py
@@ -1,7 +1,9 @@
+import json
import re
from dataclasses import dataclass
+from importlib import resources
from typing import Any
-from urllib.parse import urlparse
+from urllib.parse import urljoin, urlparse
from selectolax.parser import HTMLParser
@@ -15,6 +17,29 @@ from .config import (
)
_JSON_LD_PERSON_RE = re.compile(r'"@type"\s*:\s*"Person"', re.IGNORECASE)
+_JSON_LD_STR_FIELD_RE = {
+ "name": re.compile(r'"name"\s*:\s*"([^"]+)"'),
+ "image": re.compile(r'"image"\s*:\s*"([^"]+)"'),
+}
+
+_site_profiles_cache: dict[str, Any] | None = None
+
+
+def _load_site_profiles() -> dict[str, Any]:
+ """Optional per-site CSS-selector overrides for profile fields.
+
+ Maps a site name to ``{"name": sel, "bio": sel, "avatar": sel}``. Missing or
+ invalid file yields an empty map (generic extraction is used instead).
+ """
+ global _site_profiles_cache
+ if _site_profiles_cache is None:
+ try:
+ text = (resources.files("aliens_eye.data") / "site_profiles.json").read_text("utf-8")
+ data = json.loads(text)
+ _site_profiles_cache = data if isinstance(data, dict) else {}
+ except (FileNotFoundError, json.JSONDecodeError, OSError):
+ _site_profiles_cache = {}
+ return _site_profiles_cache
@dataclass
@@ -129,6 +154,8 @@ class FeatureExtractor:
body_node = tree.css_first("body")
text_length = float(len(body_node.text(deep=True))) if body_node else 0.0
+ profile = self._extract_profile(tree, url, site_name, title_text)
+
features = {
"http_200": 1.0 if http_code == 200 else 0.0,
"http_3xx": 1.0 if 300 <= http_code < 400 else 0.0,
@@ -166,6 +193,7 @@ class FeatureExtractor:
signals = {
"site": site_name,
"title": title_text,
+ "profile": profile,
"meta_samples": meta_contents[:5],
"url_analysis": {
"domain": parsed_url.netloc,
@@ -195,3 +223,86 @@ class FeatureExtractor:
}
return FeatureBundle(features=features, signals=signals, fingerprint=fingerprint)
+
+ def _extract_profile(
+ self, tree: HTMLParser, url: str, site_name: str, title_text: str
+ ) -> dict[str, str]:
+ """Best-effort display name / bio / avatar for a profile page.
+
+ Generic and provider-agnostic: OpenGraph tags, then JSON-LD Person,
+ then plain <title>/<meta>. Per-site CSS overrides in
+ data/site_profiles.json take precedence when present.
+ """
+ name = self._meta_content(tree, 'meta[property="og:title"]')
+ bio = self._meta_content(tree, 'meta[property="og:description"]')
+ avatar = self._meta_content(tree, 'meta[property="og:image"]')
+
+ if not (name and avatar):
+ ld_name, ld_image = self._json_ld_person_fields(tree)
+ name = name or ld_name
+ avatar = avatar or ld_image
+
+ if not bio:
+ bio = self._meta_content(tree, 'meta[name="description"]')
+ if not name:
+ name = title_text
+ if not avatar:
+ icon = tree.css_first('link[rel="apple-touch-icon"]')
+ if icon:
+ avatar = icon.attributes.get("href") or ""
+
+ overrides = _load_site_profiles().get(site_name)
+ if isinstance(overrides, dict):
+ name = self._select_text(tree, overrides.get("name")) or name
+ bio = self._select_text(tree, overrides.get("bio")) or bio
+ avatar = self._select_attr(tree, overrides.get("avatar")) or avatar
+
+ if avatar:
+ avatar = urljoin(url, avatar)
+
+ return {
+ "name": (name or "").strip()[:200],
+ "bio": (bio or "").strip()[:500],
+ "avatar": (avatar or "").strip(),
+ }
+
+ @staticmethod
+ def _meta_content(tree: HTMLParser, selector: str) -> str:
+ node = tree.css_first(selector)
+ if node:
+ return (node.attributes.get("content") or "").strip()
+ return ""
+
+ @staticmethod
+ def _json_ld_person_fields(tree: HTMLParser) -> tuple[str, str]:
+ for node in tree.css('script[type="application/ld+json"]'):
+ script_text = node.text() or ""
+ if not _JSON_LD_PERSON_RE.search(script_text):
+ continue
+ name_match = _JSON_LD_STR_FIELD_RE["name"].search(script_text)
+ image_match = _JSON_LD_STR_FIELD_RE["image"].search(script_text)
+ return (
+ name_match.group(1) if name_match else "",
+ image_match.group(1) if image_match else "",
+ )
+ return "", ""
+
+ @staticmethod
+ def _select_text(tree: HTMLParser, selector: str | None) -> str:
+ if not selector:
+ return ""
+ node = tree.css_first(selector)
+ return node.text().strip() if node else ""
+
+ @staticmethod
+ def _select_attr(tree: HTMLParser, selector: str | None) -> str:
+ if not selector:
+ return ""
+ node = tree.css_first(selector)
+ if not node:
+ return ""
+ for attr in ("src", "href", "content", "data-src"):
+ value = node.attributes.get(attr)
+ if value:
+ return value
+ return ""
diff --git a/src/aliens_eye/core/checkpoint.py b/src/aliens_eye/core/checkpoint.py
new file mode 100644
index 0000000..4840b57
--- /dev/null
+++ b/src/aliens_eye/core/checkpoint.py
@@ -0,0 +1,65 @@
+"""Resumable scan checkpoints.
+
+Each completed ``(variation, site)`` result is appended as one JSON line to a
+checkpoint file. On resume, previously completed sites are skipped and their
+results are reused, so a scan interrupted by Ctrl-C or a crash continues where it
+left off instead of re-hitting every site.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from pathlib import Path
+from typing import Any
+
+
+class Checkpoint:
+ def __init__(self, path: str | Path) -> None:
+ self.path = Path(path)
+ self._lock = asyncio.Lock()
+ # (variation, site) -> stored result dict
+ self._done: dict[tuple[str, str], dict[str, Any]] = {}
+ self._load()
+
+ def _load(self) -> None:
+ if not self.path.exists():
+ return
+ try:
+ for line in self.path.read_text(encoding="utf-8").splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ entry = json.loads(line)
+ self._done[(entry["variation"], entry["site"])] = entry["result"]
+ except (OSError, json.JSONDecodeError, KeyError):
+ # A corrupt/partial checkpoint is treated as empty rather than fatal.
+ self._done = {}
+
+ def is_done(self, variation: str, site: str) -> bool:
+ return (variation, site) in self._done
+
+ def results_for(self, variation: str) -> dict[str, dict[str, Any]]:
+ """site -> result for every completed site of this variation."""
+ return {
+ site: result
+ for (var, site), result in self._done.items()
+ if var == variation
+ }
+
+ async def record(self, variation: str, site: str, result: dict[str, Any]) -> None:
+ key = (variation, site)
+ async with self._lock:
+ if key in self._done:
+ return
+ self._done[key] = result
+ self.path.parent.mkdir(parents=True, exist_ok=True)
+ with self.path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps({"variation": variation, "site": site, "result": result}) + "\n")
+
+ def finalize(self) -> None:
+ """Remove the checkpoint file once the scan has completed successfully."""
+ try:
+ self.path.unlink(missing_ok=True)
+ except OSError:
+ pass
diff --git a/src/aliens_eye/core/config.py b/src/aliens_eye/core/config.py
index d397029..03e1ea0 100644
--- a/src/aliens_eye/core/config.py
+++ b/src/aliens_eye/core/config.py
@@ -1,4 +1,6 @@
+import json
from dataclasses import dataclass, field
+from importlib import resources
from pathlib import Path
from platformdirs import user_cache_dir
@@ -18,7 +20,7 @@ DEFAULT_HEADERS = {
"Upgrade-Insecure-Requests": "1",
}
-ERROR_KEYWORDS = [
+_EN_ERROR_KEYWORDS = [
"not found",
"doesn't exist",
"didn't find",
@@ -55,7 +57,7 @@ ERROR_KEYWORDS = [
"page was not found",
]
-POSITIVE_KEYWORDS = [
+_EN_POSITIVE_KEYWORDS = [
"follow",
"subscribe",
"like",
@@ -83,7 +85,7 @@ POSITIVE_KEYWORDS = [
"timeline",
]
-META_KEYWORDS = [
+_EN_META_KEYWORDS = [
"profile picture",
"profile image",
"avatar",
@@ -95,6 +97,38 @@ META_KEYWORDS = [
"user information",
]
+def _merge_keywords(group: str, base: list[str]) -> list[str]:
+ """Union the packaged multilingual keywords with the English base list.
+
+ Falls back to the base list if data/keywords.json is missing or invalid.
+ Matching is substring-on-lowercased-content, so keywords are lowercased and
+ deduplicated while preserving order (English first).
+ """
+ merged: list[str] = []
+ seen: set[str] = set()
+
+ def _add(words) -> None:
+ for word in words:
+ key = str(word).strip().lower()
+ if key and key not in seen:
+ seen.add(key)
+ merged.append(key)
+
+ _add(base)
+ try:
+ text = (resources.files("aliens_eye.data") / "keywords.json").read_text("utf-8")
+ data = json.loads(text)
+ for lang_words in (data.get(group) or {}).values():
+ _add(lang_words)
+ except (FileNotFoundError, json.JSONDecodeError, OSError, AttributeError):
+ pass
+ return merged
+
+
+ERROR_KEYWORDS = _merge_keywords("error", _EN_ERROR_KEYWORDS)
+POSITIVE_KEYWORDS = _merge_keywords("positive", _EN_POSITIVE_KEYWORDS)
+META_KEYWORDS = _merge_keywords("meta", _EN_META_KEYWORDS)
+
AUTH_PATTERNS = [
"/login",
"/signin",
@@ -139,3 +173,5 @@ class ScannerConfig:
model_path: Path | None = None
use_ml: bool = True
plain_output: bool = False
+ only_found: bool = False
+ json_stdout: bool = False
diff --git a/src/aliens_eye/core/correlate.py b/src/aliens_eye/core/correlate.py
new file mode 100644
index 0000000..b7dccdf
--- /dev/null
+++ b/src/aliens_eye/core/correlate.py
@@ -0,0 +1,265 @@
+"""Cross-site correlation: cluster Found/Maybe profiles that look like the same person.
+
+Signals used per profile: avatar perceptual hash (dhash, requires Pillow via the
+``[correlate]`` extra), normalized display name, tokenized bio, and external links
+found in the bio. Profiles are linked by a union-find over pairwise similarity and
+grouped into clusters ("likely the same person").
+
+Avatar hashing is best-effort: if Pillow is missing or an image fails to download,
+correlation still runs on name/bio/link signals alone.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import re
+from dataclasses import dataclass, field
+from typing import Any
+
+import aiohttp
+
+from .config import DEFAULT_HEADERS, ScannerConfig
+from .scanner import build_connector
+
+_URL_RE = re.compile(r"https?://[^\s\"'<>)]+", re.IGNORECASE)
+_HANDLE_RE = re.compile(r"@([A-Za-z0-9_.]{2,30})")
+_WORD_RE = re.compile(r"[a-z0-9]+")
+
+# Similarity thresholds.
+AVATAR_MAX_HAMMING = 8 # <= this many differing bits => same image
+BIO_MIN_JACCARD = 0.5 # token overlap ratio to link on bio alone
+LINK_MATCH_WEIGHT = True # a shared external link links two profiles
+
+
+@dataclass
+class Profile:
+ variation: str
+ site: str
+ name: str
+ bio: str
+ avatar: str
+ url: str
+ status: str
+ avatar_hash: int | None = None
+ links: set[str] = field(default_factory=set)
+ handles: set[str] = field(default_factory=set)
+
+ def __post_init__(self) -> None:
+ # Derive links/handles from the bio when not supplied explicitly.
+ if not self.links:
+ self.links = set(_URL_RE.findall(self.bio or ""))
+ if not self.handles:
+ self.handles = {h.lower() for h in _HANDLE_RE.findall(self.bio or "")}
+
+ @property
+ def key(self) -> str:
+ return f"{self.variation}:{self.site}"
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "variation": self.variation,
+ "site": self.site,
+ "name": self.name,
+ "bio": self.bio,
+ "avatar": self.avatar,
+ "url": self.url,
+ "status": self.status,
+ "avatar_hash": format(self.avatar_hash, "016x") if self.avatar_hash is not None else None,
+ }
+
+
+def _norm_name(name: str) -> str:
+ return " ".join(_WORD_RE.findall((name or "").lower()))
+
+
+def _tokens(text: str) -> set[str]:
+ return {w for w in _WORD_RE.findall((text or "").lower()) if len(w) > 2}
+
+
+def _jaccard(a: set[str], b: set[str]) -> float:
+ if not a or not b:
+ return 0.0
+ inter = len(a & b)
+ union = len(a | b)
+ return inter / union if union else 0.0
+
+
+def profiles_from_report(report: dict[str, Any]) -> list[Profile]:
+ """Extract Found/Maybe profiles (with any profile signals) from a report dict."""
+ profiles: list[Profile] = []
+ for variation, block in report.get("variations", {}).items():
+ for site, info in (block.get("sites", {}) or {}).items():
+ if info.get("status") not in {"Found", "Maybe"}:
+ continue
+ prof = info.get("ai_analysis", {}).get("signals", {}).get("profile", {}) or {}
+ bio = prof.get("bio", "") or ""
+ p = Profile(
+ variation=variation,
+ site=site,
+ name=prof.get("name", "") or "",
+ bio=bio,
+ avatar=prof.get("avatar", "") or "",
+ url=info.get("url", "") or "",
+ status=info.get("status", ""),
+ links=set(_URL_RE.findall(bio)),
+ handles={h.lower() for h in _HANDLE_RE.findall(bio)},
+ )
+ profiles.append(p)
+ return profiles
+
+
+# --- avatar hashing -------------------------------------------------------
+
+def _dhash(image_bytes: bytes) -> int | None:
+ """8x8 difference hash of an image -> 64-bit int. None if Pillow/decoding fails."""
+ try:
+ import io
+
+ from PIL import Image
+ except ImportError:
+ return None
+ try:
+ img = Image.open(io.BytesIO(image_bytes)).convert("L").resize((9, 8))
+ except Exception:
+ return None
+ bits = 0
+ for row in range(8):
+ for col in range(8):
+ left = img.getpixel((col, row))
+ right = img.getpixel((col + 1, row))
+ bits = (bits << 1) | (1 if left > right else 0)
+ return bits
+
+
+def _hamming(a: int, b: int) -> int:
+ return bin(a ^ b).count("1")
+
+
+def pillow_available() -> bool:
+ try:
+ import PIL # noqa: F401
+
+ return True
+ except ImportError:
+ return False
+
+
+async def _download_and_hash(
+ session: aiohttp.ClientSession, profile: Profile, timeout: float
+) -> None:
+ if not profile.avatar:
+ return
+ try:
+ async with session.get(profile.avatar, timeout=aiohttp.ClientTimeout(total=timeout)) as resp:
+ if resp.status != 200:
+ return
+ data = await resp.content.read(2_000_000) # cap at 2 MB
+ except Exception:
+ return
+ profile.avatar_hash = _dhash(data)
+
+
+async def hash_avatars(profiles: list[Profile], proxy: str | None = None, timeout: float = 10.0) -> None:
+ """Download and dhash every profile avatar in place. No-op without Pillow."""
+ if not pillow_available():
+ return
+ targets = [p for p in profiles if p.avatar]
+ if not targets:
+ return
+
+ limit = min(20, len(targets))
+ connector = build_connector(ScannerConfig(proxy=proxy), limit)
+ sem = asyncio.Semaphore(limit)
+ async with aiohttp.ClientSession(headers=DEFAULT_HEADERS, connector=connector) as session:
+
+ async def worker(p: Profile) -> None:
+ async with sem:
+ await _download_and_hash(session, p, timeout)
+
+ await asyncio.gather(*(worker(p) for p in targets))
+
+
+# --- clustering -----------------------------------------------------------
+
+class _UnionFind:
+ def __init__(self, n: int) -> None:
+ self.parent = list(range(n))
+
+ def find(self, x: int) -> int:
+ while self.parent[x] != x:
+ self.parent[x] = self.parent[self.parent[x]]
+ x = self.parent[x]
+ return x
+
+ def union(self, a: int, b: int) -> None:
+ ra, rb = self.find(a), self.find(b)
+ if ra != rb:
+ self.parent[rb] = ra
+
+
+def _pair_reason(a: Profile, b: Profile) -> str | None:
+ """Return the reason two profiles are the same person, or None."""
+ if (
+ a.avatar_hash is not None
+ and b.avatar_hash is not None
+ and _hamming(a.avatar_hash, b.avatar_hash) <= AVATAR_MAX_HAMMING
+ ):
+ return "avatar"
+ if a.links & b.links:
+ return "shared-link"
+ if a.handles & b.handles or (a.variation and a.variation in b.handles) or (b.variation and b.variation in a.handles):
+ return "handle"
+ na, nb = _norm_name(a.name), _norm_name(b.name)
+ if na and na == nb and len(na) > 3:
+ return "name"
+ if _jaccard(_tokens(a.bio), _tokens(b.bio)) >= BIO_MIN_JACCARD:
+ return "bio"
+ return None
+
+
+def cluster_profiles(profiles: list[Profile]) -> list[dict[str, Any]]:
+ """Union-find clustering. Returns clusters as dicts (only multi-site clusters)."""
+ n = len(profiles)
+ uf = _UnionFind(n)
+ reasons: dict[tuple[int, int], str] = {}
+ for i in range(n):
+ for j in range(i + 1, n):
+ reason = _pair_reason(profiles[i], profiles[j])
+ if reason:
+ uf.union(i, j)
+ reasons[(i, j)] = reason
+
+ groups: dict[int, list[int]] = {}
+ for idx in range(n):
+ groups.setdefault(uf.find(idx), []).append(idx)
+
+ clusters: list[dict[str, Any]] = []
+ for members in groups.values():
+ if len(members) < 2:
+ continue
+ member_reasons = sorted(
+ {reasons[(i, j)] for i in members for j in members if (i, j) in reasons}
+ )
+ clusters.append(
+ {
+ "size": len(members),
+ "reasons": member_reasons,
+ "members": [profiles[i].to_dict() for i in members],
+ }
+ )
+ clusters.sort(key=lambda c: c["size"], reverse=True)
+ return clusters
+
+
+async def correlate_report(
+ report: dict[str, Any], proxy: str | None = None, timeout: float = 10.0
+) -> dict[str, Any]:
+ """Full correlation pass over a report dict. Returns a ``correlation`` block."""
+ profiles = profiles_from_report(report)
+ await hash_avatars(profiles, proxy=proxy, timeout=timeout)
+ clusters = cluster_profiles(profiles)
+ return {
+ "avatar_hashing": pillow_available(),
+ "profiles_considered": len(profiles),
+ "clusters": clusters,
+ }
diff --git a/src/aliens_eye/core/domains.py b/src/aliens_eye/core/domains.py
new file mode 100644
index 0000000..f3ae3be
--- /dev/null
+++ b/src/aliens_eye/core/domains.py
@@ -0,0 +1,89 @@
+"""Domain availability check for a username.
+
+For a username, test ``<username>.<tld>`` across a handful of common TLDs:
+DNS resolution says whether the domain is registered/points somewhere, and an
+HTTP request says whether it serves a live site. No third-party whois service or
+API key is used — just DNS + HTTP.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import re
+import socket
+from typing import Any
+
+import aiohttp
+
+from .config import DEFAULT_HEADERS, ScannerConfig
+from .scanner import build_connector
+
+DEFAULT_TLDS = ["com", "io", "net", "org", "dev", "me", "co", "app"]
+
+_LABEL_RE = re.compile(r"[^a-z0-9-]")
+
+
+def domain_label(username: str) -> str:
+ """Reduce a username to a valid DNS label (lowercase alnum + hyphen)."""
+ label = _LABEL_RE.sub("", username.lower().replace("_", "-").replace(".", "-"))
+ return label.strip("-")
+
+
+async def _resolves(host: str) -> bool:
+ loop = asyncio.get_event_loop()
+ try:
+ await loop.getaddrinfo(host, None, type=socket.SOCK_STREAM)
+ return True
+ except (socket.gaierror, OSError):
+ return False
+
+
+async def _is_live(session: aiohttp.ClientSession, host: str, timeout: float) -> bool:
+ for scheme in ("https", "http"):
+ try:
+ async with session.get(
+ f"{scheme}://{host}",
+ timeout=aiohttp.ClientTimeout(total=timeout),
+ allow_redirects=True,
+ ) as resp:
+ if resp.status < 500:
+ return True
+ except Exception:
+ continue
+ return False
+
+
+async def check_domains(
+ username: str,
+ tlds: list[str] | None = None,
+ proxy: str | None = None,
+ timeout: float = 8.0,
+) -> dict[str, Any]:
+ """Check ``<username>.<tld>`` registration + liveness for each TLD."""
+ label = domain_label(username)
+ tlds = tlds or DEFAULT_TLDS
+ results: list[dict[str, Any]] = []
+ if not label:
+ return {"username": username, "label": label, "results": results}
+
+ hosts = [f"{label}.{tld}" for tld in tlds]
+ resolved = dict(zip(hosts, await asyncio.gather(*(_resolves(h) for h in hosts)), strict=False))
+
+ connector = build_connector(ScannerConfig(proxy=proxy), len(hosts))
+ async with aiohttp.ClientSession(headers=DEFAULT_HEADERS, connector=connector) as session:
+ live_flags = await asyncio.gather(
+ *(
+ _is_live(session, host, timeout) if resolved[host] else _false()
+ for host in hosts
+ )
+ )
+
+ for host, live in zip(hosts, live_flags, strict=False):
+ results.append(
+ {"domain": host, "registered": resolved[host], "live": bool(live)}
+ )
+ return {"username": username, "label": label, "results": results}
+
+
+async def _false() -> bool:
+ return False
diff --git a/src/aliens_eye/core/expand.py b/src/aliens_eye/core/expand.py
new file mode 100644
index 0000000..9ec7034
--- /dev/null
+++ b/src/aliens_eye/core/expand.py
@@ -0,0 +1,85 @@
+"""Recursive expansion: mine linked usernames out of found profiles.
+
+Given scan results, pull candidate usernames from each Found/Maybe profile's bio
+(``@handle`` mentions) and external links that follow a known ``domain/<user>``
+shape (github.com/<user>, twitter.com/<user>, ...). These feed another scan hop.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any
+from urllib.parse import urlparse
+
+_HANDLE_RE = re.compile(r"@([A-Za-z0-9_.]{2,30})")
+_URL_RE = re.compile(r"https?://[^\s\"'<>)]+", re.IGNORECASE)
+
+# Domains where the first path segment is the username.
+_USER_PATH_DOMAINS = {
+ "github.com", "gitlab.com", "twitter.com", "x.com", "instagram.com",
+ "t.me", "telegram.me", "medium.com", "dev.to", "reddit.com",
+ "tiktok.com", "youtube.com", "twitch.tv", "keybase.io", "about.me",
+ "codepen.io", "replit.com", "soundcloud.com", "pinterest.com",
+}
+
+# Path segments that are never usernames.
+_PATH_STOPWORDS = {
+ "in", "u", "user", "users", "profile", "channel", "c", "watch", "p",
+ "about", "home", "login", "signup", "help", "settings", "explore",
+}
+
+
+def _username_from_url(url: str) -> str | None:
+ try:
+ parsed = urlparse(url)
+ except ValueError:
+ return None
+ host = parsed.netloc.lower()
+ if host.startswith("www."):
+ host = host[4:]
+ if host not in _USER_PATH_DOMAINS:
+ return None
+ segments = [s for s in parsed.path.split("/") if s]
+ if not segments:
+ return None
+ candidate = segments[0].lstrip("@")
+ if candidate.lower() in _PATH_STOPWORDS:
+ candidate = segments[1] if len(segments) > 1 else ""
+ candidate = candidate.strip()
+ if len(candidate) < 2 or "." in candidate and not candidate.replace(".", "").isalnum():
+ return None
+ return candidate if candidate else None
+
+
+def candidate_usernames_from_results(
+ all_results: dict[str, list[dict[str, Any]]],
+ exclude: set[str] | None = None,
+ limit: int = 25,
+) -> list[str]:
+ """Collect deduped candidate usernames linked from Found/Maybe profiles."""
+ exclude = {e.lower() for e in (exclude or set())}
+ found: list[str] = []
+ seen: set[str] = set()
+
+ def _add(name: str) -> None:
+ name = name.strip()
+ low = name.lower()
+ if len(name) < 2 or low in exclude or low in seen:
+ return
+ seen.add(low)
+ found.append(name)
+
+ for results in all_results.values():
+ for item in results:
+ if item.get("status") not in {"Found", "Maybe"}:
+ continue
+ profile = item.get("ai_analysis", {}).get("signals", {}).get("profile", {}) or {}
+ bio = profile.get("bio", "") or ""
+ for handle in _HANDLE_RE.findall(bio):
+ _add(handle)
+ for url in _URL_RE.findall(bio):
+ uname = _username_from_url(url)
+ if uname:
+ _add(uname)
+
+ return found[:limit]
diff --git a/src/aliens_eye/core/exporter.py b/src/aliens_eye/core/exporter.py
index 2bb98b1..38e752b 100644
--- a/src/aliens_eye/core/exporter.py
+++ b/src/aliens_eye/core/exporter.py
@@ -1,6 +1,7 @@
import csv
import json
from datetime import datetime
+from html import escape as _esc
from pathlib import Path
from typing import Any
@@ -9,7 +10,18 @@ from rich.prompt import Prompt
from aliens_eye.utils.console import build_results_table, get_console
-KNOWN_FORMATS = {"json", "csv", "html", "md", "markdown", "all"}
+KNOWN_FORMATS = {
+ "json", "csv", "html", "md", "markdown",
+ "gexf", "maltego", "mermaid", "pdf", "all",
+}
+
+# Formats that need the correlation graph rather than the flat report.
+GRAPH_FORMATS = {"gexf", "maltego", "mermaid"}
+
+
+def _md_cell(value: Any) -> str:
+ """Sanitize a value for use inside a Markdown table cell."""
+ return str(value or "").replace("|", "\\|").replace("\n", " ").replace("\r", " ").strip()
class ResultsExporter:
@@ -26,11 +38,17 @@ class ResultsExporter:
level: str,
all_results: dict[str, list[dict[str, Any]]],
formats: list[str],
+ correlation: dict[str, Any] | None = None,
+ domains: dict[str, Any] | None = None,
) -> list[Path]:
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
results_data = self._build_results_data(
base_username, level, all_results, timestamp
)
+ if correlation is not None:
+ results_data["correlation"] = correlation
+ if domains is not None:
+ results_data["domains"] = domains
written: list[Path] = []
stem = f"{base_username}_{level}_{timestamp}"
@@ -55,8 +73,37 @@ class ResultsExporter:
self._write_markdown(md_path, results_data)
written.append(md_path)
+ # Graph and PDF formats are opt-in only (not bundled into "all"), since
+ # they are analysis artifacts rather than the standard document report.
+ if "gexf" in formats:
+ p = self.output_dir / f"{stem}.gexf"
+ self._write_gexf(p, results_data)
+ written.append(p)
+ if "maltego" in formats:
+ p = self.output_dir / f"{stem}_maltego.csv"
+ self._write_maltego(p, results_data)
+ written.append(p)
+ if "mermaid" in formats:
+ p = self.output_dir / f"{stem}.mmd"
+ self._write_mermaid(p, results_data)
+ written.append(p)
+ if "pdf" in formats:
+ p = self.output_dir / f"{stem}.pdf"
+ if self._write_pdf(p, results_data):
+ written.append(p)
+
return written
+ def report_dict(
+ self,
+ base_username: str,
+ level: str,
+ all_results: dict[str, list[dict[str, Any]]],
+ ) -> dict[str, Any]:
+ """Public accessor for the structured report (used for --json-stdout)."""
+ timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+ return self._build_results_data(base_username, level, all_results, timestamp)
+
def _build_results_data(
self,
base_username: str,
@@ -165,10 +212,24 @@ class ResultsExporter:
rows = []
for variation, data in variations.items():
for site, info in data.get("sites", {}).items():
+ profile = info.get("ai_analysis", {}).get("signals", {}).get("profile", {})
+ is_hit = info.get("status") in {"Found", "Maybe"}
+ avatar_url = profile.get("avatar", "") if is_hit else ""
+ avatar_cell = (
+ f"<img src='{_esc(avatar_url, quote=True)}' alt='' style='height:40px;border-radius:4px'>"
+ if avatar_url
+ else ""
+ )
+ name = _esc(profile.get("name", "")) if is_hit else ""
+ bio = _esc(profile.get("bio", "")) if is_hit else ""
+ url = info.get("url", "")
rows.append(
- f"<tr><td>{variation}</td><td>{site}</td><td>{info.get('status','')}</td>"
- f"<td>{info.get('confidence',0)}</td><td>{info.get('code',0)}</td>"
- f"<td><a href='{info.get('url','')}'>{info.get('url','')}</a></td></tr>"
+ f"<tr><td>{_esc(variation)}</td><td>{_esc(site)}</td>"
+ f"<td>{_esc(info.get('status',''))}</td>"
+ f"<td>{_esc(str(info.get('confidence',0)))}</td>"
+ f"<td>{_esc(str(info.get('code',0)))}</td>"
+ f"<td><a href='{_esc(url, quote=True)}'>{_esc(url)}</a></td>"
+ f"<td>{avatar_cell}</td><td>{name}</td><td>{bio}</td></tr>"
)
html = f"""
@@ -188,8 +249,8 @@ th {{ background-color: #f2f2f2; text-align: left; }}
</head>
<body>
<h1>Aliens Eye Report</h1>
-<p><strong>Base Username:</strong> {summary.get('base_username','')}</p>
-<p><strong>Scan Level:</strong> {summary.get('scan_level','')}</p>
+<p><strong>Base Username:</strong> {_esc(str(summary.get('base_username','')))}</p>
+<p><strong>Scan Level:</strong> {_esc(str(summary.get('scan_level','')))}</p>
<p><strong>Total Variations:</strong> {summary.get('total_variations',0)}</p>
<p><strong>Total Sites Scanned:</strong> {summary.get('total_sites_scanned',0)}</p>
<p><strong>Total Found:</strong> {summary.get('total_found',0)}</p>
@@ -204,6 +265,9 @@ th {{ background-color: #f2f2f2; text-align: left; }}
<th>Confidence</th>
<th>HTTP Code</th>
<th>URL</th>
+<th>Avatar</th>
+<th>Name</th>
+<th>Bio</th>
</tr>
</thead>
<tbody>
@@ -242,8 +306,8 @@ th {{ background-color: #f2f2f2; text-align: left; }}
f"Found: {info.get('found', 0)} | Maybe: {info.get('maybe', 0)} | "
f"Not Found: {info.get('not_found', 0)} | Errors: {info.get('errors', 0)}",
"",
- "| Site | Status | Confidence | HTTP | URL |",
- "| --- | --- | --- | --- | --- |",
+ "| Site | Status | Confidence | HTTP | Name | Bio | URL |",
+ "| --- | --- | --- | --- | --- | --- | --- |",
]
)
sites = sorted(
@@ -257,14 +321,114 @@ th {{ background-color: #f2f2f2; text-align: left; }}
status = site_info.get("status", "")
if status not in {"Found", "Maybe"}:
continue
+ profile = (
+ site_info.get("ai_analysis", {})
+ .get("signals", {})
+ .get("profile", {})
+ )
+ name = _md_cell(profile.get("name", ""))
+ bio = _md_cell(profile.get("bio", ""))
lines.append(
f"| {site} | {status} | {site_info.get('confidence', 0)}% | "
- f"{site_info.get('code', 0)} | {site_info.get('url', '')} |"
+ f"{site_info.get('code', 0)} | {name} | {bio} | {site_info.get('url', '')} |"
)
lines.append("")
path.write_text("\n".join(lines), encoding="utf-8")
+ # --- graph exports ----------------------------------------------------
+
+ @staticmethod
+ def _graph_model(results_data: dict[str, Any]) -> dict[str, Any]:
+ """Build (nodes, edges) from Found/Maybe accounts + correlation clusters."""
+ nodes: dict[str, dict[str, Any]] = {}
+ edges: list[dict[str, str]] = []
+ variations = results_data.get("variations", {})
+ for variation, block in variations.items():
+ has_hit = False
+ for site, info in (block.get("sites", {}) or {}).items():
+ if info.get("status") not in {"Found", "Maybe"}:
+ continue
+ has_hit = True
+ acct = f"{variation}:{site}"
+ nodes[acct] = {"label": site, "type": "account", "url": info.get("url", "")}
+ edges.append({"source": variation, "target": acct, "label": "has-account"})
+ if has_hit:
+ nodes[variation] = {"label": variation, "type": "username", "url": ""}
+
+ for cluster in results_data.get("correlation", {}).get("clusters", []):
+ members = [m["variation"] + ":" + m["site"] for m in cluster.get("members", [])]
+ reason = ",".join(cluster.get("reasons", [])) or "same-person"
+ for i in range(len(members)):
+ for j in range(i + 1, len(members)):
+ if members[i] in nodes and members[j] in nodes:
+ edges.append({"source": members[i], "target": members[j], "label": reason})
+ return {"nodes": nodes, "edges": edges}
+
+ def _write_gexf(self, path: Path, results_data: dict[str, Any]) -> None:
+ model = self._graph_model(results_data)
+ ids = {name: str(i) for i, name in enumerate(model["nodes"])}
+ node_xml = "".join(
+ f'<node id="{ids[name]}" label="{_esc(data["label"], quote=True)}"/>'
+ for name, data in model["nodes"].items()
+ )
+ edge_xml = "".join(
+ f'<edge id="{i}" source="{ids[e["source"]]}" target="{ids[e["target"]]}" '
+ f'label="{_esc(e["label"], quote=True)}"/>'
+ for i, e in enumerate(model["edges"])
+ if e["source"] in ids and e["target"] in ids
+ )
+ xml = (
+ '<?xml version="1.0" encoding="UTF-8"?>\n'
+ '<gexf xmlns="http://gexf.net/1.3" version="1.3">\n'
+ '<graph defaultedgetype="undirected">\n'
+ f"<nodes>{node_xml}</nodes>\n<edges>{edge_xml}</edges>\n"
+ "</graph>\n</gexf>\n"
+ )
+ path.write_text(xml, encoding="utf-8")
+
+ def _write_maltego(self, path: Path, results_data: dict[str, Any]) -> None:
+ model = self._graph_model(results_data)
+ with path.open("w", newline="", encoding="utf-8") as handle:
+ writer = csv.writer(handle)
+ writer.writerow(["SourceEntity", "SourceType", "TargetEntity", "TargetType", "Relationship"])
+ for e in model["edges"]:
+ src = model["nodes"].get(e["source"], {})
+ tgt = model["nodes"].get(e["target"], {})
+ if not src or not tgt:
+ continue
+ writer.writerow([
+ e["source"], "maltego.Affiliation" if src.get("type") == "account" else "maltego.Alias",
+ e["target"], "maltego.Affiliation" if tgt.get("type") == "account" else "maltego.Alias",
+ e["label"],
+ ])
+
+ def _write_mermaid(self, path: Path, results_data: dict[str, Any]) -> None:
+ model = self._graph_model(results_data)
+ ids = {name: f"n{i}" for i, name in enumerate(model["nodes"])}
+ lines = ["```mermaid", "graph TD"]
+ for name, data in model["nodes"].items():
+ label = data["label"].replace('"', "'")
+ shape = f'(["{label}"])' if data["type"] == "username" else f'["{label}"]'
+ lines.append(f" {ids[name]}{shape}")
+ for e in model["edges"]:
+ if e["source"] in ids and e["target"] in ids:
+ lines.append(f" {ids[e['source']]} -- {e['label']} --- {ids[e['target']]}")
+ lines.append("```")
+ path.write_text("\n".join(lines) + "\n", encoding="utf-8")
+
+ def _write_pdf(self, path: Path, results_data: dict[str, Any]) -> bool:
+ """Investigator PDF report. Returns False (with a note) if reportlab is missing."""
+ try:
+ from aliens_eye.core.pdf_report import write_pdf
+ except ImportError:
+ self.console.print(
+ "[yellow]PDF export needs reportlab: pip install \"aliens-eye\\[pdf]\"[/yellow]"
+ )
+ return False
+ write_pdf(path, results_data)
+ return True
+
def display_results_from_file(self, file_path: str) -> None:
try:
with open(file_path, encoding="utf-8") as handle:
diff --git a/src/aliens_eye/core/pdf_report.py b/src/aliens_eye/core/pdf_report.py
new file mode 100644
index 0000000..c8c7a0d
--- /dev/null
+++ b/src/aliens_eye/core/pdf_report.py
@@ -0,0 +1,126 @@
+"""Investigator-style PDF report (optional, requires the ``[pdf]`` extra: reportlab).
+
+Renders the scan summary, per-variation Found/Maybe tables with embedded avatars,
+and any correlation clusters. Importing this module raises ImportError if reportlab
+is not installed; the exporter catches that and prints an install hint.
+"""
+
+from __future__ import annotations
+
+import io
+import urllib.request
+from pathlib import Path
+from typing import Any
+
+from reportlab.lib import colors
+from reportlab.lib.pagesizes import A4
+from reportlab.lib.styles import getSampleStyleSheet
+from reportlab.lib.units import mm
+from reportlab.platypus import (
+ Image,
+ Paragraph,
+ SimpleDocTemplate,
+ Spacer,
+ Table,
+ TableStyle,
+)
+
+_MAX_AVATARS = 40
+
+
+def _fetch_avatar(url: str, timeout: float = 6.0) -> io.BytesIO | None:
+ if not url:
+ return None
+ try:
+ req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 aliens-eye"})
+ with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 - user-provided profile URL
+ if resp.status != 200:
+ return None
+ data = resp.read(2_000_000)
+ return io.BytesIO(data)
+ except Exception:
+ return None
+
+
+def write_pdf(path: str | Path, results_data: dict[str, Any]) -> None:
+ styles = getSampleStyleSheet()
+ doc = SimpleDocTemplate(str(path), pagesize=A4, title="Aliens Eye Report")
+ story: list[Any] = []
+
+ summary = results_data.get("scan_summary", {})
+ story.append(Paragraph("Aliens Eye — OSINT Report", styles["Title"]))
+ story.append(Paragraph(f"Base username: <b>{summary.get('base_username', '')}</b>", styles["Normal"]))
+ story.append(Paragraph(f"Scan level: {summary.get('scan_level', '')}", styles["Normal"]))
+ story.append(Paragraph(f"Timestamp: {summary.get('timestamp', '')}", styles["Normal"]))
+ story.append(Paragraph(
+ f"Found: {summary.get('total_found', 0)} | "
+ f"High confidence: {summary.get('total_high_confidence', 0)} | "
+ f"Sites scanned: {summary.get('total_sites_scanned', 0)}",
+ styles["Normal"],
+ ))
+ story.append(Spacer(1, 6 * mm))
+
+ avatars_used = 0
+ for variation, block in results_data.get("variations", {}).items():
+ hits = [
+ (site, info)
+ for site, info in (block.get("sites", {}) or {}).items()
+ if info.get("status") in {"Found", "Maybe"}
+ ]
+ if not hits:
+ continue
+ hits.sort(key=lambda kv: (kv[1].get("status") != "Found", -kv[1].get("confidence", 0)))
+ story.append(Paragraph(f"@{variation}", styles["Heading2"]))
+
+ data = [["", "Site", "Status", "Conf", "Name / Bio", "URL"]]
+ for site, info in hits:
+ profile = info.get("ai_analysis", {}).get("signals", {}).get("profile", {}) or {}
+ img_cell: Any = ""
+ if avatars_used < _MAX_AVATARS and profile.get("avatar"):
+ buf = _fetch_avatar(profile["avatar"])
+ if buf is not None:
+ try:
+ img_cell = Image(buf, width=12 * mm, height=12 * mm)
+ avatars_used += 1
+ except Exception:
+ img_cell = ""
+ name = profile.get("name", "")
+ bio = (profile.get("bio", "") or "")[:120]
+ name_bio = Paragraph(f"<b>{_x(name)}</b><br/>{_x(bio)}", styles["BodyText"])
+ url = Paragraph(_x(info.get("url", "")), styles["BodyText"])
+ data.append([
+ img_cell, site, info.get("status", ""),
+ f"{info.get('confidence', 0)}%", name_bio, url,
+ ])
+
+ table = Table(data, colWidths=[14 * mm, 26 * mm, 16 * mm, 12 * mm, 52 * mm, 50 * mm])
+ table.setStyle(TableStyle([
+ ("BACKGROUND", (0, 0), (-1, 0), colors.HexColor("#222222")),
+ ("TEXTCOLOR", (0, 0), (-1, 0), colors.white),
+ ("FONTSIZE", (0, 0), (-1, -1), 7),
+ ("VALIGN", (0, 0), (-1, -1), "MIDDLE"),
+ ("GRID", (0, 0), (-1, -1), 0.25, colors.grey),
+ ("ROWBACKGROUNDS", (0, 1), (-1, -1), [colors.white, colors.HexColor("#f4f4f4")]),
+ ]))
+ story.append(table)
+ story.append(Spacer(1, 4 * mm))
+
+ clusters = results_data.get("correlation", {}).get("clusters", [])
+ if clusters:
+ story.append(Paragraph("Correlation — likely same person", styles["Heading2"]))
+ for i, cluster in enumerate(clusters, 1):
+ sites = ", ".join(m["site"] for m in cluster["members"])
+ reasons = ", ".join(cluster.get("reasons", []))
+ story.append(Paragraph(f"<b>Cluster {i}</b> ({reasons}): {_x(sites)}", styles["BodyText"]))
+
+ doc.build(story)
+
+
+def _x(text: Any) -> str:
+ """Escape the small subset of markup reportlab Paragraph treats as XML."""
+ return (
+ str(text or "")
+ .replace("&", "&amp;")
+ .replace("<", "&lt;")
+ .replace(">", "&gt;")
+ )
diff --git a/src/aliens_eye/core/report.py b/src/aliens_eye/core/report.py
new file mode 100644
index 0000000..ee0a536
--- /dev/null
+++ b/src/aliens_eye/core/report.py
@@ -0,0 +1,83 @@
+"""Shared helpers for loading and comparing saved scan reports.
+
+A report is the JSON structure written by :class:`ResultsExporter`
+(``{"scan_summary": ..., "variations": {v: {"sites": {s: {...}}}}}``). These
+helpers are the single place that knows that shape, reused by watch mode, the
+``diff`` command, and anything else that consumes prior results from disk.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+class ReportError(ValueError):
+ """Raised when a file is not a recognizable Aliens Eye report."""
+
+
+def load_report(path: str | Path) -> dict[str, Any]:
+ """Load and validate a saved report JSON. Raises ReportError on bad input."""
+ p = Path(path)
+ try:
+ data = json.loads(p.read_text(encoding="utf-8"))
+ except FileNotFoundError as exc:
+ raise ReportError(f"File not found: {p}") from exc
+ except (OSError, json.JSONDecodeError) as exc:
+ raise ReportError(f"Invalid JSON file: {p}") from exc
+ if not isinstance(data, dict) or "variations" not in data:
+ raise ReportError(f"Unrecognized results file: {p}")
+ return data
+
+
+def iter_sites(report: dict[str, Any]):
+ """Yield ``(variation, site, site_info)`` triples from a report dict."""
+ for variation, block in report.get("variations", {}).items():
+ for site, info in (block.get("sites", {}) or {}).items():
+ yield variation, site, info
+
+
+def found_keys(report: dict[str, Any]) -> set[str]:
+ """Set of ``variation:site`` keys with status ``Found`` in a report."""
+ return {
+ f"{variation}:{site}"
+ for variation, site, info in iter_sites(report)
+ if info.get("status") == "Found"
+ }
+
+
+def status_map(report: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ """Map ``variation:site`` -> ``{status, confidence, url}`` for every site."""
+ result: dict[str, dict[str, Any]] = {}
+ for variation, site, info in iter_sites(report):
+ result[f"{variation}:{site}"] = {
+ "status": info.get("status", ""),
+ "confidence": info.get("confidence", 0),
+ "url": info.get("url", ""),
+ }
+ return result
+
+
+def diff_reports(old: dict[str, Any], new: dict[str, Any]) -> dict[str, list[dict[str, Any]]]:
+ """Compare two reports.
+
+ Returns ``new`` / ``gone`` (Found membership changes) and ``changed``
+ (status or confidence moved for a key present in both).
+ """
+ old_found = found_keys(old)
+ new_found = found_keys(new)
+ old_map = status_map(old)
+ new_map = status_map(new)
+
+ changed: list[dict[str, Any]] = []
+ for key in sorted(set(old_map) & set(new_map)):
+ before, after = old_map[key], new_map[key]
+ if before["status"] != after["status"] or before["confidence"] != after["confidence"]:
+ changed.append({"key": key, "before": before, "after": after})
+
+ return {
+ "new": [{"key": k, "url": new_map.get(k, {}).get("url", "")} for k in sorted(new_found - old_found)],
+ "gone": [{"key": k, "url": old_map.get(k, {}).get("url", "")} for k in sorted(old_found - new_found)],
+ "changed": changed,
+ }
diff --git a/src/aliens_eye/core/scanner.py b/src/aliens_eye/core/scanner.py
index 8cb32d4..ee98ba8 100644
--- a/src/aliens_eye/core/scanner.py
+++ b/src/aliens_eye/core/scanner.py
@@ -41,6 +41,31 @@ def load_sites_data(path: Path | None = None) -> dict[str, str]:
return json.loads(text)
+def load_site_plugins(extra_dirs: list[Path] | None = None) -> dict[str, str]:
+ """Merge every ``*.json`` site map found in the plugin directories.
+
+ Looks in ``./sites.d/`` and the user config dir's ``sites.d/`` by default,
+ plus any ``extra_dirs``. Each file must be a ``{site_name: url_template}``
+ map; later files override earlier ones on name conflicts.
+ """
+ from platformdirs import user_config_dir
+
+ dirs = [Path("sites.d"), Path(user_config_dir("aliens_eye")) / "sites.d"]
+ dirs.extend(extra_dirs or [])
+ merged: dict[str, str] = {}
+ for directory in dirs:
+ if not directory.is_dir():
+ continue
+ for path in sorted(directory.glob("*.json")):
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError):
+ continue
+ if isinstance(data, dict):
+ merged.update({str(k): str(v) for k, v in data.items()})
+ return merged
+
+
def load_nsfw_sites() -> list[str]:
"""Names of sites flagged as NSFW, shipped as package data."""
try:
@@ -95,6 +120,7 @@ class UsernameScanner:
fingerprints: FingerprintStore,
logger,
browser_fallback: BrowserFallback | None = None,
+ checkpoint=None,
) -> None:
self.sites_data = sites_data
self.config = config
@@ -103,6 +129,7 @@ class UsernameScanner:
self.fingerprints = fingerprints
self.logger = logger
self.browser_fallback = browser_fallback
+ self.checkpoint = checkpoint
self.console = get_console()
self.results_dir = config.output_dir
@@ -134,14 +161,27 @@ class UsernameScanner:
if not self.sites_data:
return results
- conn_limit = max(1, min(self.config.concurrent, len(self.sites_data)))
+ done: dict[str, dict[str, Any]] = (
+ self.checkpoint.results_for(username) if self.checkpoint else {}
+ )
+ pending = {site: tmpl for site, tmpl in self.sites_data.items() if site not in done}
+ results.extend(done.values())
+ if done:
+ self.console.print(
+ f"[dim]Resuming {username}: {len(done)} site(s) from checkpoint, "
+ f"{len(pending)} remaining.[/dim]"
+ )
+ if not pending:
+ return results
+
+ conn_limit = max(1, min(self.config.concurrent, len(pending)))
rate_limiter = DomainRateLimiter()
queue: asyncio.Queue[tuple[str, str]] = asyncio.Queue()
- for site, tmpl in self.sites_data.items():
+ for site, tmpl in pending.items():
queue.put_nowait((site, tmpl))
view = ScanView(self.console)
- view.start(username, len(self.sites_data))
+ view.start(username, len(pending))
connector = build_connector(self.config, conn_limit)
start_time = time.monotonic()
@@ -189,22 +229,25 @@ class UsernameScanner:
except Exception as exc:
url = self._format_url(site, tmpl, username)
self.logger.debug("Worker failed for %s: %s", site, exc)
- results.append(
- {
- "site": site,
- "url": url,
- "final_url": url,
- "status": "Error",
- "code": 0,
- "response_time": 0.0,
- "confidence": 0,
- "ai_analysis": {"error": str(exc)},
- }
- )
- finally:
- found = sum(1 for r in results if r["status"] == "Found")
- view.advance(found)
- queue.task_done()
+ result = {
+ "site": site,
+ "url": url,
+ "final_url": url,
+ "status": "Error",
+ "code": 0,
+ "response_time": 0.0,
+ "confidence": 0,
+ "ai_analysis": {"error": str(exc)},
+ }
+ results.append(result)
+ if self.checkpoint is not None:
+ try:
+ await self.checkpoint.record(username, site, result)
+ except Exception as exc: # noqa: BLE001 - checkpoint is best-effort
+ self.logger.debug("Checkpoint write failed for %s: %s", site, exc)
+ found = sum(1 for r in results if r["status"] == "Found")
+ view.advance(found)
+ queue.task_done()
async def _scan_site(
self,
diff --git a/src/aliens_eye/core/variations.py b/src/aliens_eye/core/variations.py
index 71b18f1..3d1cb47 100644
--- a/src/aliens_eye/core/variations.py
+++ b/src/aliens_eye/core/variations.py
@@ -1,3 +1,45 @@
+import re
+
+
+def usernames_from_email(email: str) -> list[str]:
+ """Derive candidate usernames from an email address.
+
+ Uses the local part plus common separator splits, e.g.
+ ``[email protected]`` -> ``john.doe``, ``johndoe``, ``john_doe``, ``john``.
+ """
+ local = str(email).strip().split("@", 1)[0].lower()
+ if not local:
+ return []
+ parts = [p for p in re.split(r"[._\-+]", local) if p]
+ candidates = [local]
+ if len(parts) > 1:
+ joined = "".join(parts)
+ candidates.extend([joined, "_".join(parts), ".".join(parts), parts[0]])
+ return list(dict.fromkeys(c for c in candidates if len(c) >= 2))
+
+
+def usernames_from_name(name: str) -> list[str]:
+ """Derive candidate usernames from a real name.
+
+ ``John Doe`` -> ``john``, ``doe``, ``johndoe``, ``john.doe``,
+ ``john_doe``, ``jdoe``, ``doej``.
+ """
+ parts = [p for p in re.split(r"\s+", str(name).strip().lower()) if p.isalnum()]
+ if not parts:
+ return []
+ if len(parts) == 1:
+ return [parts[0]] if len(parts[0]) >= 2 else []
+ first, last = parts[0], parts[-1]
+ candidates = [
+ first,
+ last,
+ f"{first}{last}",
+ f"{first}.{last}",
+ f"{first}_{last}",
+ f"{first[0]}{last}",
+ f"{last}{first[0]}",
+ ]
+ return list(dict.fromkeys(c for c in candidates if len(c) >= 2))
def generate_username_variations(username: str, level: str) -> list[str]:
diff --git a/src/aliens_eye/core/watch.py b/src/aliens_eye/core/watch.py
new file mode 100644
index 0000000..e82b971
--- /dev/null
+++ b/src/aliens_eye/core/watch.py
@@ -0,0 +1,127 @@
+"""Watch / monitor mode: re-scan on an interval and alert on changes.
+
+Reuses the existing scanner and exporter. Each cycle runs a full scan, saves a
+timestamped report, and diffs the set of Found accounts against the previous
+cycle (seeded from the most recent report on disk, if any).
+"""
+
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+from typing import Any
+
+import aiohttp
+
+from .report import ReportError, found_keys, load_report
+
+_UNITS = {"s": 1.0, "m": 60.0, "h": 3600.0, "d": 86400.0}
+
+
+def parse_duration(value: str) -> float:
+ """Parse a duration like ``30s``/``15m``/``6h``/``1d`` into seconds.
+
+ A bare number is treated as seconds. Raises ValueError on bad input.
+ """
+ text = str(value).strip().lower()
+ if not text:
+ raise ValueError("empty duration")
+ unit = text[-1]
+ if unit in _UNITS:
+ number = text[:-1]
+ factor = _UNITS[unit]
+ else:
+ number = text
+ factor = 1.0
+ seconds = float(number) * factor
+ if seconds <= 0:
+ raise ValueError(f"duration must be positive: {value!r}")
+ return seconds
+
+
+def found_set(all_results: dict[str, list[dict[str, Any]]]) -> set[str]:
+ """Set of ``variation:site`` keys currently detected as Found."""
+ keys: set[str] = set()
+ for variation, results in all_results.items():
+ for item in results:
+ if item.get("status") == "Found":
+ keys.add(f"{variation}:{item.get('site', '')}")
+ return keys
+
+
+def diff_found(prev: set[str], curr: set[str]) -> dict[str, list[str]]:
+ """New and disappeared Found accounts between two cycles."""
+ return {
+ "new": sorted(curr - prev),
+ "gone": sorted(prev - curr),
+ }
+
+
+def _baseline(output_dir: Path, username: str, level: str) -> set[str]:
+ reports = sorted(output_dir.glob(f"{username}_{level}_*.json"))
+ if not reports:
+ return set()
+ try:
+ return found_keys(load_report(reports[-1]))
+ except ReportError:
+ return set()
+
+
+async def _notify(webhook_url: str, payload: dict[str, Any], logger) -> None:
+ try:
+ async with aiohttp.ClientSession() as session:
+ await session.post(webhook_url, json=payload, timeout=15)
+ except Exception as exc: # noqa: BLE001 - notification is best-effort
+ logger.warning("Webhook notify failed: %s", exc)
+
+
+async def run_watch(
+ scanner,
+ exporter,
+ usernames: list[str],
+ scan_level: str,
+ formats: list[str],
+ interval: float,
+ logger,
+ console,
+ output_dir: Path,
+ notify_url: str | None = None,
+) -> None:
+ """Re-scan the given usernames every ``interval`` seconds until interrupted."""
+ previous: dict[str, set[str]] = {
+ username: _baseline(output_dir, username, scan_level) for username in usernames
+ }
+ console.print(
+ f"[blue]Watch mode:[/blue] re-scanning every [yellow]{interval:.0f}s[/yellow] "
+ f"(Ctrl-C to stop)"
+ )
+
+ cycle = 0
+ while True:
+ cycle += 1
+ console.print(f"\n[dim]--- watch cycle {cycle} ---[/dim]")
+ for username in usernames:
+ if not username or len(username) < 2:
+ continue
+ all_results = await scanner.scan_with_variations(username, scan_level)
+ exporter.save_results(username, scan_level, all_results, formats)
+
+ current = found_set(all_results)
+ delta = diff_found(previous.get(username, set()), current)
+ previous[username] = current
+
+ if delta["new"] or delta["gone"]:
+ for key in delta["new"]:
+ console.print(f"[bold green]+ NEW:[/bold green] {key}")
+ for key in delta["gone"]:
+ console.print(f"[bold red]- GONE:[/bold red] {key}")
+ if notify_url:
+ await _notify(
+ notify_url,
+ {"username": username, "level": scan_level, **delta},
+ logger,
+ )
+ else:
+ console.print(f"[dim]No change for {username} ({len(current)} found).[/dim]")
+
+ await asyncio.sleep(interval)
diff --git a/src/aliens_eye/data/keywords.json b/src/aliens_eye/data/keywords.json
new file mode 100644
index 0000000..2b3271d
--- /dev/null
+++ b/src/aliens_eye/data/keywords.json
@@ -0,0 +1,232 @@
+{
+ "error": {
+ "en": [
+ "not found",
+ "doesn't exist",
+ "didn't find",
+ "does not exist",
+ "something went wrong",
+ "no such user",
+ "user not found",
+ "cannot find",
+ "can't find",
+ "not exist",
+ "profile not found",
+ "account does not exist",
+ "username not found",
+ "no user found",
+ "no results found",
+ "no such username",
+ "isn't available",
+ "that content is unavailable",
+ "page not found",
+ "404",
+ "404 error",
+ "404 not found",
+ "error",
+ "sorry",
+ "oops",
+ "unavailable",
+ "account suspended",
+ "invalid username",
+ "account not found",
+ "account terminated",
+ "account disabled",
+ "user doesn't have an account",
+ "this account doesn't exist",
+ "page was not found"
+ ],
+ "es": [
+ "no encontrado",
+ "no encontrada",
+ "no existe",
+ "no se encontró",
+ "usuario no encontrado",
+ "página no encontrada",
+ "no disponible",
+ "cuenta suspendida",
+ "nombre de usuario no válido"
+ ],
+ "fr": [
+ "introuvable",
+ "n'existe pas",
+ "page introuvable",
+ "utilisateur introuvable",
+ "non trouvé",
+ "compte suspendu",
+ "indisponible"
+ ],
+ "de": [
+ "nicht gefunden",
+ "existiert nicht",
+ "seite nicht gefunden",
+ "benutzer nicht gefunden",
+ "nicht verfügbar",
+ "konto gesperrt"
+ ],
+ "pt": [
+ "não encontrado",
+ "não encontrada",
+ "não existe",
+ "página não encontrada",
+ "usuário não encontrado",
+ "indisponível",
+ "conta suspensa"
+ ],
+ "ru": [
+ "не найдено",
+ "не найден",
+ "не существует",
+ "страница не найдена",
+ "пользователь не найден",
+ "недоступно"
+ ],
+ "ja": [
+ "見つかりません",
+ "存在しません",
+ "ページが見つかりません",
+ "ユーザーが見つかりません"
+ ],
+ "zh": [
+ "未找到",
+ "不存在",
+ "页面不存在",
+ "用户不存在",
+ "找不到"
+ ],
+ "ar": [
+ "غير موجود",
+ "لم يتم العثور",
+ "الصفحة غير موجودة",
+ "المستخدم غير موجود"
+ ],
+ "hi": [
+ "नहीं मिला",
+ "मौजूद नहीं",
+ "पेज नहीं मिला",
+ "उपयोगकर्ता नहीं मिला"
+ ]
+ },
+ "positive": {
+ "en": [
+ "follow",
+ "subscribe",
+ "like",
+ "share",
+ "following",
+ "followers",
+ "profile",
+ "user",
+ "posts",
+ "photos",
+ "bio",
+ "status",
+ "tweets",
+ "joined",
+ "member since",
+ "online",
+ "verified",
+ "active",
+ "comments",
+ "uploads",
+ "reviews",
+ "friends",
+ "connections",
+ "activity",
+ "timeline"
+ ],
+ "es": [
+ "seguidores",
+ "siguiendo",
+ "perfil",
+ "publicaciones",
+ "miembro desde",
+ "verificado"
+ ],
+ "fr": [
+ "abonnés",
+ "abonnements",
+ "profil",
+ "publications",
+ "membre depuis",
+ "vérifié"
+ ],
+ "de": [
+ "abonnenten",
+ "profil",
+ "beiträge",
+ "mitglied seit",
+ "verifiziert"
+ ],
+ "pt": [
+ "seguidores",
+ "seguindo",
+ "perfil",
+ "publicações",
+ "membro desde",
+ "verificado"
+ ],
+ "ru": [
+ "подписчики",
+ "подписки",
+ "профиль",
+ "публикации",
+ "участник с"
+ ],
+ "ja": [
+ "フォロワー",
+ "フォロー",
+ "プロフィール",
+ "投稿"
+ ],
+ "zh": [
+ "粉丝",
+ "关注",
+ "个人资料",
+ "帖子",
+ "已认证"
+ ],
+ "ar": [
+ "متابعون",
+ "الملف الشخصي",
+ "منشورات"
+ ],
+ "hi": [
+ "फ़ॉलोअर्स",
+ "प्रोफ़ाइल",
+ "पोस्ट"
+ ]
+ },
+ "meta": {
+ "en": [
+ "profile picture",
+ "profile image",
+ "avatar",
+ "user page",
+ "username",
+ "user profile",
+ "account info",
+ "account information",
+ "user information"
+ ],
+ "es": [
+ "foto de perfil",
+ "imagen de perfil",
+ "página de usuario"
+ ],
+ "fr": [
+ "photo de profil",
+ "image de profil",
+ "page utilisateur"
+ ],
+ "de": [
+ "profilbild",
+ "benutzerseite"
+ ],
+ "pt": [
+ "foto de perfil",
+ "imagem de perfil",
+ "página do usuário"
+ ]
+ }
+}
diff --git a/src/aliens_eye/data/site_profiles.json b/src/aliens_eye/data/site_profiles.json
new file mode 100644
index 0000000..6cba59d
--- /dev/null
+++ b/src/aliens_eye/data/site_profiles.json
@@ -0,0 +1,63 @@
+{
+ "_comment": "site_name -> {name: <css>, bio: <css>, avatar: <css>}. CSS overrides the generic OpenGraph/JSON-LD extraction; an override that matches nothing is ignored (the generic value is kept). Only add selectors for server-rendered profile pages.",
+ "github": {
+ "name": "span.p-name",
+ "bio": "div.user-profile-bio",
+ "avatar": "img.avatar-user"
+ },
+ "gitlab": {
+ "name": "div.user-profile .cover-title",
+ "bio": "div.user-profile .cover-desc",
+ "avatar": "div.avatar-holder img.avatar"
+ },
+ "dev.to": {
+ "name": "h1.crayons-title",
+ "bio": "div.profile-header__details .crayons-subtitle-3",
+ "avatar": "img.profile-header__avatar"
+ },
+ "keybase": {
+ "name": "span.full-name",
+ "bio": "p.bio",
+ "avatar": "img.avatar"
+ },
+ "gravatar": {
+ "name": "h1.profile-name",
+ "bio": "p.profile-description",
+ "avatar": "img.avatar"
+ },
+ "about.me": {
+ "name": "h1",
+ "bio": "div.bio",
+ "avatar": "img.avatar"
+ },
+ "codepen": {
+ "name": "h1.profile-name",
+ "bio": "div.profile-bio",
+ "avatar": "img.profile-avatar"
+ },
+ "steam": {
+ "name": "span.actual_persona_name",
+ "bio": "div.profile_summary",
+ "avatar": "div.playerAvatarAutoSizeInner img"
+ },
+ "wordpress": {
+ "name": "h2.author-name",
+ "bio": "p.author-description",
+ "avatar": "img.avatar"
+ },
+ "telegram": {
+ "name": "div.tgme_page_title span",
+ "bio": "div.tgme_page_description",
+ "avatar": "img.tgme_page_photo_image"
+ },
+ "lobsters": {
+ "name": "span.user_tag",
+ "bio": "div.about",
+ "avatar": "img.avatar"
+ },
+ "replit": {
+ "name": "h1",
+ "bio": "div.bio",
+ "avatar": "img.profile-image"
+ }
+}
diff --git a/src/aliens_eye/mcp_server.py b/src/aliens_eye/mcp_server.py
new file mode 100644
index 0000000..103397a
--- /dev/null
+++ b/src/aliens_eye/mcp_server.py
@@ -0,0 +1,90 @@
+"""MCP server exposing Aliens Eye as tools for LLM agents (optional ``[serve]`` extra: mcp).
+
+Runs over stdio and exposes:
+- ``scan_username`` — scan a username across all sites, return the structured report
+- ``correlate`` — run cross-site correlation over a report and return clusters
+- ``read_report`` — load a previously saved report JSON from disk
+
+Start it with ``aliens_eye serve`` and point an MCP client (e.g. Claude) at the
+command.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+
+async def _run_scan(username: str, level: str, sites: str | None, no_ml: bool) -> dict[str, Any]:
+ from aliens_eye.core.analyzer import FeatureExtractor
+ from aliens_eye.core.config import ScannerConfig
+ from aliens_eye.core.detector import Detector
+ from aliens_eye.core.exporter import ResultsExporter
+ from aliens_eye.core.fingerprints import FingerprintStore
+ from aliens_eye.core.scanner import UsernameScanner, load_sites_data
+ from aliens_eye.utils.console import set_plain
+ from aliens_eye.utils.logger import setup_logger
+
+ set_plain(True, stderr=True)
+ logger = setup_logger(False)
+ config = ScannerConfig()
+ sites_data = load_sites_data(Path(sites) if sites else None)
+ detector = Detector()
+ if not no_ml:
+ detector.load_model(logger)
+ extractor = FeatureExtractor()
+ fingerprints = FingerprintStore(config.fingerprints_path, config.max_fingerprints_per_label)
+ fingerprints.load(logger)
+ scanner = UsernameScanner(
+ sites_data=sites_data, config=config, extractor=extractor,
+ detector=detector, fingerprints=fingerprints, logger=logger,
+ )
+ try:
+ all_results = await scanner.scan_with_variations(username, level)
+ finally:
+ fingerprints.save()
+ return ResultsExporter(config.output_dir).report_dict(username, level, all_results)
+
+
+async def serve(args) -> None:
+ try:
+ from mcp.server.fastmcp import FastMCP
+ except ImportError:
+ from aliens_eye.utils.console import get_console
+
+ get_console().print(
+ "[yellow]The MCP server needs the mcp package: pip install \"aliens-eye\\[serve]\"[/yellow]"
+ )
+ return
+
+ sites = getattr(args, "sites", None)
+ no_ml = getattr(args, "no_ml", False)
+ mcp = FastMCP("aliens-eye")
+
+ @mcp.tool()
+ async def scan_username(username: str, level: str = "basic") -> dict[str, Any]:
+ """Scan a username across all known sites. level: basic|intermediate|advanced."""
+ if not username or len(username) < 2:
+ return {"error": "username must be at least 2 characters"}
+ if level not in {"basic", "intermediate", "advanced"}:
+ level = "basic"
+ return await _run_scan(username, level, sites, no_ml)
+
+ @mcp.tool()
+ async def correlate(report: dict[str, Any]) -> dict[str, Any]:
+ """Cluster the profiles in a scan report that look like the same person."""
+ from aliens_eye.core.correlate import correlate_report
+
+ return await correlate_report(report)
+
+ @mcp.tool()
+ def read_report(path: str) -> dict[str, Any]:
+ """Load a previously saved Aliens Eye report JSON from disk."""
+ from aliens_eye.core.report import ReportError, load_report
+
+ try:
+ return load_report(path)
+ except ReportError as exc:
+ return {"error": str(exc)}
+
+ await mcp.run_stdio_async()
diff --git a/src/aliens_eye/ml/collect.py b/src/aliens_eye/ml/collect.py
index 64ddf79..cec2feb 100644
--- a/src/aliens_eye/ml/collect.py
+++ b/src/aliens_eye/ml/collect.py
@@ -36,6 +36,19 @@ def load_selfcheck_data() -> dict[str, list[str]]:
}
+def write_rows(output_path: Path, rows: list[list[float]], append: bool = False) -> None:
+ """Write labeled rows to a CSV. Writes the header only for a new/overwritten file."""
+ output_path = Path(output_path)
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ write_header = not (append and output_path.exists())
+ mode = "a" if append else "w"
+ with output_path.open(mode, encoding="utf-8", newline="") as handle:
+ writer = csv.writer(handle)
+ if write_header:
+ writer.writerow(FEATURE_SCHEMA + ["label"])
+ writer.writerows(rows)
+
+
def random_username(rng: random.Random) -> str:
length = rng.randint(14, 20)
return "".join(rng.choices(string.ascii_lowercase + string.digits, k=length))
@@ -121,12 +134,6 @@ async def collect_dataset(
await asyncio.gather(*(run_job(job) for job in jobs))
- output_path = Path(output_path)
- output_path.parent.mkdir(parents=True, exist_ok=True)
- with output_path.open("w", encoding="utf-8", newline="") as handle:
- writer = csv.writer(handle)
- writer.writerow(FEATURE_SCHEMA + ["label"])
- writer.writerows(rows)
-
+ write_rows(output_path, rows, append=False)
logger.info("Wrote %d labeled samples to %s", len(rows), output_path)
return len(rows)
diff --git a/src/aliens_eye/ml/label.py b/src/aliens_eye/ml/label.py
new file mode 100644
index 0000000..e84dfec
--- /dev/null
+++ b/src/aliens_eye/ml/label.py
@@ -0,0 +1,75 @@
+"""Active-learning loop: hand-label uncertain results and grow the dataset.
+
+Reads a saved report, walks the low-confidence band (Maybe by default), and asks
+you to confirm each hit. Confirmed labels are vectorized from the features already
+stored in the report and appended to a training CSV, so a later ``train fit`` can
+learn from your corrections.
+"""
+
+from __future__ import annotations
+
+from aliens_eye.core.features import vectorize_features
+from aliens_eye.core.report import ReportError, iter_sites, load_report
+from aliens_eye.ml.collect import write_rows
+from aliens_eye.utils.console import get_console
+from aliens_eye.utils.logger import setup_logger
+
+
+async def label_report(args) -> None:
+ from pathlib import Path
+
+ from rich.prompt import Prompt
+
+ console = get_console()
+ logger = setup_logger(getattr(args, "verbose", False))
+ try:
+ report = load_report(args.report)
+ except ReportError as exc:
+ console.print(f"[red]{exc}[/red]")
+ return
+
+ band = (args.band or "Maybe").lower()
+ statuses = {"found", "maybe", "not found"} if band == "all" else {"maybe"}
+
+ candidates = [
+ (variation, site, info)
+ for variation, site, info in iter_sites(report)
+ if info.get("status", "").lower() in statuses
+ ]
+ if not candidates:
+ console.print(f"[yellow]No '{band}' results to label in {args.report}.[/yellow]")
+ return
+
+ console.print(
+ f"[blue]Labeling {len(candidates)} result(s).[/blue] "
+ "[dim]y = exists, n = does not exist, s = skip, q = quit[/dim]\n"
+ )
+
+ rows: list[list[float]] = []
+ for variation, site, info in candidates:
+ features = info.get("ai_analysis", {}).get("features")
+ if not isinstance(features, dict) or not features:
+ continue
+ profile = info.get("ai_analysis", {}).get("signals", {}).get("profile", {})
+ name = profile.get("name", "")
+ console.print(
+ f"[yellow]{site}[/yellow] / [bold]{variation}[/bold] "
+ f"status={info.get('status')} conf={info.get('confidence', 0)}% "
+ f"{info.get('url', '')}"
+ + (f"\n [dim]name: {name}[/dim]" if name else "")
+ )
+ choice = Prompt.ask(" label", choices=["y", "n", "s", "q"], default="s", console=console)
+ if choice == "q":
+ break
+ if choice == "s":
+ continue
+ label = 1 if choice == "y" else 0
+ rows.append(vectorize_features(features) + [float(label)])
+
+ if not rows:
+ console.print("[dim]No labels recorded.[/dim]")
+ return
+
+ write_rows(Path(args.out), rows, append=True)
+ console.print(f"[green]Appended {len(rows)} labeled sample(s) -> {args.out}[/green]")
+ logger.info("Appended %d samples to %s", len(rows), args.out)
diff --git a/src/aliens_eye/selfcheck.py b/src/aliens_eye/selfcheck.py
index 777d981..fd7b537 100644
--- a/src/aliens_eye/selfcheck.py
+++ b/src/aliens_eye/selfcheck.py
@@ -1,7 +1,8 @@
-"""Validate detection accuracy against accounts known to exist.
+"""Validate detection accuracy against accounts known to exist (and to not exist).
-Scans each (site, username) pair from data/selfcheck.json and checks that the
-detector reports "Found". Doubles as a rot detector for sites.json and as a
+Scans each known ``(site, username)`` positive from data/selfcheck.json plus a few
+random negatives per site, then reports precision / recall / F1 / false-positive
+rate overall and per site. Doubles as a rot detector for sites.json and a
calibration baseline for the ML model.
"""
@@ -9,7 +10,10 @@ from __future__ import annotations
import asyncio
import json
+import random
+import string
from importlib import resources
+from typing import Any
import aiohttp
from rich.panel import Panel
@@ -35,45 +39,74 @@ def load_selfcheck_data() -> dict[str, str]:
}
+def _random_username(rng: random.Random) -> str:
+ length = rng.randint(14, 20)
+ return "".join(rng.choices(string.ascii_lowercase + string.digits, k=length))
+
+
+def _metrics(tp: int, fp: int, fn: int, tn: int) -> dict[str, float]:
+ precision = tp / (tp + fp) if (tp + fp) else 0.0
+ recall = tp / (tp + fn) if (tp + fn) else 0.0
+ f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
+ fpr = fp / (fp + tn) if (fp + tn) else 0.0
+ return {
+ "precision": round(precision, 4),
+ "recall": round(recall, 4),
+ "f1": round(f1, 4),
+ "false_positive_rate": round(fpr, 4),
+ "tp": tp, "fp": fp, "fn": fn, "tn": tn,
+ }
+
+
async def run_selfcheck(
sites_data: dict[str, str],
detector: Detector,
config: ScannerConfig,
logger,
-) -> float:
- """Run the self-check and return accuracy in [0, 1]."""
+ negatives: int = 1,
+ report_format: str | None = None,
+ seed: int | None = 1234,
+) -> dict[str, Any]:
+ """Run the self-check. Returns a metrics dict; renders a report unless report_format='json'."""
console = get_console()
selfcheck = load_selfcheck_data()
extractor = FeatureExtractor()
rate_limiter = DomainRateLimiter()
+ rng = random.Random(seed)
+
+ # Build (site, username, expected_label) jobs. label 1 = should be Found.
+ jobs: list[tuple[str, str, str, int]] = []
+ for site, username in selfcheck.items():
+ if site not in sites_data:
+ continue
+ jobs.append((site, sites_data[site], username, 1))
+ for _ in range(max(0, negatives)):
+ jobs.append((site, sites_data[site], _random_username(rng), 0))
- checks = [
- (site, sites_data[site], username)
- for site, username in selfcheck.items()
- if site in sites_data
- ]
skipped = [site for site in selfcheck if site not in sites_data]
- if skipped:
+ if skipped and report_format != "json":
console.print(f"[yellow]Skipping sites missing from sites.json: {', '.join(skipped)}[/yellow]")
- if not checks:
- console.print("[red]No self-check sites available.[/red]")
- return 0.0
+ if not jobs:
+ if report_format != "json":
+ console.print("[red]No self-check sites available.[/red]")
+ return {"overall": _metrics(0, 0, 0, 0), "per_site": {}, "errors": 0}
rows: list[dict] = []
- semaphore = asyncio.Semaphore(min(10, len(checks)))
-
+ semaphore = asyncio.Semaphore(min(10, len(jobs)))
connector = aiohttp.TCPConnector(limit=10)
async with aiohttp.ClientSession(headers=DEFAULT_HEADERS, connector=connector) as session:
- async def check(site: str, template: str, username: str) -> None:
- url = template.format(username)
+ async def check(site: str, template: str, username: str, label: int) -> None:
+ try:
+ url = template.format(username)
+ except Exception:
+ url = template.replace("{}", username)
async with semaphore:
fetch = await fetch_url(session, url, config, rate_limiter, logger)
if fetch.error:
- rows.append(
- {"site": site, "username": username, "status": "Error",
- "confidence": 0, "ok": False, "note": fetch.error[:60]}
- )
+ rows.append({"site": site, "username": username, "label": label,
+ "status": "Error", "confidence": 0, "predicted": None,
+ "note": fetch.error[:60]})
return
bundle = extractor.extract(
fetch.content, fetch.final_url, username, site,
@@ -82,42 +115,96 @@ async def run_selfcheck(
bundle.features["fingerprint_match_found"] = 0.0
bundle.features["fingerprint_match_not_found"] = 0.0
detection = detector.predict(bundle.features)
- rows.append(
- {"site": site, "username": username, "status": detection.status,
- "confidence": detection.confidence, "ok": detection.status == "Found",
- "note": detection.method}
- )
+ rows.append({"site": site, "username": username, "label": label,
+ "status": detection.status, "confidence": detection.confidence,
+ "predicted": 1 if detection.status == "Found" else 0,
+ "note": detection.method})
- await asyncio.gather(*(check(*c) for c in checks))
+ await asyncio.gather(*(check(*j) for j in jobs))
- rows.sort(key=lambda r: (not r["ok"], r["site"]))
- table = Table(title="Self-check: accounts known to exist", header_style="bold blue")
+ # Aggregate confusion matrix overall + per site (ignoring network errors).
+ per_site: dict[str, dict[str, int]] = {}
+ tp = fp = fn = tn = 0
+ for r in rows:
+ if r["predicted"] is None:
+ continue
+ cell = per_site.setdefault(r["site"], {"tp": 0, "fp": 0, "fn": 0, "tn": 0})
+ if r["label"] == 1 and r["predicted"] == 1:
+ key = "tp"
+ elif r["label"] == 1 and r["predicted"] == 0:
+ key = "fn"
+ elif r["label"] == 0 and r["predicted"] == 1:
+ key = "fp"
+ else:
+ key = "tn"
+ cell[key] += 1
+ tp += key == "tp"
+ fp += key == "fp"
+ fn += key == "fn"
+ tn += key == "tn"
+
+ overall = _metrics(tp, fp, fn, tn)
+ errors = sum(1 for r in rows if r["status"] == "Error")
+ result = {
+ "overall": overall,
+ "per_site": {s: _metrics(c["tp"], c["fp"], c["fn"], c["tn"]) for s, c in per_site.items()},
+ "errors": errors,
+ "samples": len([r for r in rows if r["predicted"] is not None]),
+ }
+
+ if report_format == "json":
+ print(json.dumps(result, indent=2))
+ return result
+
+ _render(console, rows, result)
+ return result
+
+
+def _render(console, rows: list[dict], result: dict[str, Any]) -> None:
+ rows.sort(key=lambda r: (r["label"] == r.get("predicted"), r["site"]))
+ table = Table(title="Self-check: known accounts + random negatives", header_style="bold blue")
table.add_column("Site", style="yellow")
table.add_column("Username")
+ table.add_column("Expect")
table.add_column("Detected")
table.add_column("Confidence", justify="right")
table.add_column("Result")
- table.add_column("Method", style="dim")
for r in rows:
+ expect = "exists" if r["label"] == 1 else "absent"
+ if r["predicted"] is None:
+ outcome = Text("ERROR", style="dim red")
+ else:
+ correct = r["label"] == r["predicted"]
+ outcome = Text("PASS" if correct else "FAIL", style="bold green" if correct else "bold red")
table.add_row(
- r["site"],
- r["username"],
+ r["site"], r["username"], expect,
Text(r["status"], style="green" if r["status"] == "Found" else "red"),
- f"{r['confidence']}%",
- Text("PASS", style="bold green") if r["ok"] else Text("FAIL", style="bold red"),
- r["note"],
+ f"{r['confidence']}%", outcome,
)
console.print(table)
- passed = sum(1 for r in rows if r["ok"])
- errors = sum(1 for r in rows if r["status"] == "Error")
- accuracy = passed / len(rows)
- style = "green" if accuracy >= 0.8 else "yellow" if accuracy >= 0.6 else "red"
+ # Per-site metrics table.
+ site_table = Table(title="Per-site metrics", header_style="bold blue")
+ site_table.add_column("Site", style="yellow")
+ for col in ("Precision", "Recall", "F1", "FPR"):
+ site_table.add_column(col, justify="right")
+ for site, m in sorted(result["per_site"].items()):
+ site_table.add_row(
+ site, f"{m['precision']:.0%}", f"{m['recall']:.0%}",
+ f"{m['f1']:.2f}", f"{m['false_positive_rate']:.0%}",
+ )
+ console.print(site_table)
+
+ o = result["overall"]
+ f1 = o["f1"]
+ style = "green" if f1 >= 0.8 else "yellow" if f1 >= 0.6 else "red"
console.print(
Panel(
- f"[{style}]Accuracy: {passed}/{len(rows)} ({accuracy:.0%})[/{style}]"
- + (f" [dim]{errors} network errors[/dim]" if errors else ""),
+ f"[{style}]Precision {o['precision']:.0%} Recall {o['recall']:.0%} "
+ f"F1 {o['f1']:.2f} FPR {o['false_positive_rate']:.0%}[/{style}] "
+ f"[dim]({result['samples']} samples"
+ + (f", {result['errors']} network errors" if result["errors"] else "")
+ + ")[/dim]",
border_style=style,
)
)
- return accuracy
diff --git a/src/aliens_eye/tui/__init__.py b/src/aliens_eye/tui/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/src/aliens_eye/tui/app.py b/src/aliens_eye/tui/app.py
new file mode 100644
index 0000000..21af0ca
--- /dev/null
+++ b/src/aliens_eye/tui/app.py
@@ -0,0 +1,162 @@
+"""Interactive terminal UI for browsing scan results (optional ``[tui]`` extra: textual).
+
+The scan runs first (in plain mode so its progress does not fight the TUI), then a
+Textual app presents a filterable, searchable table of every result. Keybindings:
+``f`` toggle Found/Maybe only, ``/`` search, ``o`` open the selected URL in a
+browser, ``q`` quit.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import webbrowser
+from pathlib import Path
+from typing import Any
+
+
+def _install_hint() -> None:
+ from aliens_eye.utils.console import get_console
+
+ get_console().print(
+ "[yellow]The TUI needs Textual: pip install \"aliens-eye\\[tui]\"[/yellow]"
+ )
+
+
+def _flatten(all_results: dict[str, list[dict[str, Any]]]) -> list[dict[str, Any]]:
+ rows: list[dict[str, Any]] = []
+ for variation, results in all_results.items():
+ for r in results:
+ profile = r.get("ai_analysis", {}).get("signals", {}).get("profile", {}) or {}
+ rows.append({
+ "variation": variation,
+ "site": r.get("site", ""),
+ "status": r.get("status", ""),
+ "confidence": r.get("confidence", 0),
+ "name": profile.get("name", ""),
+ "url": r.get("url", ""),
+ })
+ rows.sort(key=lambda x: (x["status"] != "Found", x["status"] != "Maybe", -x["confidence"]))
+ return rows
+
+
+async def _scan(args) -> dict[str, list[dict[str, Any]]]:
+ from aliens_eye.core.analyzer import FeatureExtractor
+ from aliens_eye.core.config import ScannerConfig
+ from aliens_eye.core.detector import Detector
+ from aliens_eye.core.fingerprints import FingerprintStore
+ from aliens_eye.core.scanner import UsernameScanner, load_sites_data
+ from aliens_eye.utils.console import set_plain
+ from aliens_eye.utils.logger import setup_logger
+
+ set_plain(True)
+ logger = setup_logger(False)
+ config = ScannerConfig()
+ sites_data = load_sites_data(Path(args.sites) if args.sites else None)
+ detector = Detector()
+ if not args.no_ml:
+ detector.load_model(logger)
+ extractor = FeatureExtractor()
+ fingerprints = FingerprintStore(config.fingerprints_path, config.max_fingerprints_per_label)
+ fingerprints.load(logger)
+ scanner = UsernameScanner(
+ sites_data=sites_data, config=config, extractor=extractor,
+ detector=detector, fingerprints=fingerprints, logger=logger,
+ )
+ try:
+ return await scanner.scan_with_variations(args.username, args.level)
+ finally:
+ fingerprints.save()
+
+
+def run_tui(args) -> None:
+ try:
+ from textual.app import App, ComposeResult
+ from textual.containers import Vertical
+ from textual.widgets import DataTable, Footer, Header, Input
+ except ImportError:
+ _install_hint()
+ return
+
+ from aliens_eye.utils.console import get_console
+
+ username = args.username or get_console().input("Enter username to scan: ").strip()
+ if not username or len(username) < 2:
+ get_console().print("[red]Invalid username.[/red]")
+ return
+ args.username = username
+
+ all_results = asyncio.run(_scan(args))
+ rows = _flatten(all_results)
+
+ class ResultsApp(App):
+ BINDINGS = [
+ ("q", "quit", "Quit"),
+ ("f", "toggle_found", "Found/Maybe only"),
+ ("o", "open_url", "Open URL"),
+ ("slash", "focus_search", "Search"),
+ ]
+ CSS = "Input { dock: top; }"
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.found_only = False
+ self.query_text = ""
+
+ def compose(self) -> ComposeResult:
+ yield Header()
+ yield Vertical(
+ Input(placeholder="Filter by site / name / url…", id="search"),
+ DataTable(id="table"),
+ )
+ yield Footer()
+
+ def on_mount(self) -> None:
+ table = self.query_one("#table", DataTable)
+ table.add_columns("Site", "Variation", "Status", "Conf", "Name", "URL")
+ table.cursor_type = "row"
+ self._refresh()
+
+ def _visible(self) -> list[dict[str, Any]]:
+ out = rows
+ if self.found_only:
+ out = [r for r in out if r["status"] in {"Found", "Maybe"}]
+ if self.query_text:
+ q = self.query_text.lower()
+ out = [
+ r for r in out
+ if q in r["site"].lower() or q in str(r["name"]).lower() or q in r["url"].lower()
+ ]
+ return out
+
+ def _refresh(self) -> None:
+ table = self.query_one("#table", DataTable)
+ table.clear()
+ for r in self._visible():
+ table.add_row(
+ r["site"], r["variation"], r["status"], f"{r['confidence']}%",
+ str(r["name"])[:30], r["url"],
+ )
+ found = sum(1 for r in rows if r["status"] == "Found")
+ self.title = f"Aliens Eye — @{username}"
+ self.sub_title = f"{found} found / {len(rows)} results" + (
+ " [Found/Maybe only]" if self.found_only else ""
+ )
+
+ def on_input_changed(self, event: Input.Changed) -> None:
+ self.query_text = event.value
+ self._refresh()
+
+ def action_toggle_found(self) -> None:
+ self.found_only = not self.found_only
+ self._refresh()
+
+ def action_focus_search(self) -> None:
+ self.query_one("#search", Input).focus()
+
+ def action_open_url(self) -> None:
+ table = self.query_one("#table", DataTable)
+ visible = self._visible()
+ if 0 <= table.cursor_row < len(visible):
+ webbrowser.open(visible[table.cursor_row]["url"])
+
+ ResultsApp().run()
diff --git a/src/aliens_eye/utils/console.py b/src/aliens_eye/utils/console.py
index b9a43a8..10992d1 100644
--- a/src/aliens_eye/utils/console.py
+++ b/src/aliens_eye/utils/console.py
@@ -39,10 +39,11 @@ def get_console(plain: bool = False) -> Console:
return _console
-def set_plain(plain: bool) -> None:
+def set_plain(plain: bool, stderr: bool = False) -> None:
global _console
no_color = plain or not sys.stdout.isatty()
- _console = Console(no_color=no_color, highlight=False)
+ # Route rich output to stderr so stdout can carry machine-readable JSON.
+ _console = Console(no_color=no_color, highlight=False, stderr=stderr)
def status_text(status: str, confidence: int = 0) -> Text:
@@ -121,14 +122,22 @@ def build_results_table(results: list[dict[str, Any]], title: str | None = None)
table.add_column("Site", style="yellow", no_wrap=True)
table.add_column("Status")
table.add_column("HTTP", justify="center")
+ table.add_column("Name", overflow="ellipsis", max_width=24)
table.add_column("URL", overflow="fold")
table.add_column("Time", justify="right")
for r in results:
code = r.get("code", 0)
+ name = (
+ r.get("ai_analysis", {})
+ .get("signals", {})
+ .get("profile", {})
+ .get("name", "")
+ )
table.add_row(
str(r.get("site", ""))[:30],
status_text(r.get("status", "Unknown"), r.get("confidence", 0)),
Text(str(code), style="green" if code == 200 else "red"),
+ str(name)[:24],
r.get("url", ""),
f"{r.get('response_time', 0):.2f}s",
)
diff --git a/tests/test_new_features.py b/tests/test_new_features.py
new file mode 100644
index 0000000..86c9d6e
--- /dev/null
+++ b/tests/test_new_features.py
@@ -0,0 +1,210 @@
+"""Tests for the feature-expansion batch: report diff, correlation, expansion,
+domains, checkpoint, graph export, dataset append, and selfcheck metrics."""
+
+import json
+
+from aliens_eye.core import report as report_mod
+from aliens_eye.core.checkpoint import Checkpoint
+from aliens_eye.core.correlate import (
+ Profile,
+ cluster_profiles,
+ profiles_from_report,
+)
+from aliens_eye.core.domains import domain_label
+from aliens_eye.core.expand import candidate_usernames_from_results
+from aliens_eye.core.exporter import ResultsExporter
+from aliens_eye.ml.collect import write_rows
+from aliens_eye.selfcheck import _metrics
+
+
+def _profile_result(site, variation, status="Found", name="", bio="", avatar="", url=""):
+ return {
+ "site": site,
+ "url": url or f"https://{site}.com/{variation}",
+ "final_url": url,
+ "status": status,
+ "code": 200,
+ "response_time": 0.1,
+ "confidence": 90,
+ "ai_analysis": {
+ "features": {"http_200": 1.0},
+ "signals": {"profile": {"name": name, "bio": bio, "avatar": avatar}},
+ },
+ }
+
+
+def _report(results):
+ return ResultsExporter.__new__(ResultsExporter)._build_results_data(
+ "alice", "basic", results, "20260101_000000"
+ )
+
+
+# --- report loader + diff -------------------------------------------------
+
+def test_load_report_roundtrip_and_diff(tmp_path):
+ old_results = {"alice": [_profile_result("github", "alice")]}
+ new_results = {
+ "alice": [
+ _profile_result("github", "alice"),
+ _profile_result("reddit", "alice"),
+ ]
+ }
+ old_path = tmp_path / "old.json"
+ new_path = tmp_path / "new.json"
+ old_path.write_text(json.dumps(_report(old_results)), encoding="utf-8")
+ new_path.write_text(json.dumps(_report(new_results)), encoding="utf-8")
+
+ old = report_mod.load_report(old_path)
+ new = report_mod.load_report(new_path)
+ delta = report_mod.diff_reports(old, new)
+ new_keys = [item["key"] for item in delta["new"]]
+ assert "alice:reddit" in new_keys
+ assert delta["gone"] == []
+
+
+def test_load_report_rejects_garbage(tmp_path):
+ bad = tmp_path / "bad.json"
+ bad.write_text("{}", encoding="utf-8")
+ try:
+ report_mod.load_report(bad)
+ except report_mod.ReportError:
+ pass
+ else:
+ raise AssertionError("expected ReportError")
+
+
+def test_diff_detects_status_change():
+ old = _report({"alice": [_profile_result("github", "alice", status="Maybe")]})
+ new = _report({"alice": [_profile_result("github", "alice", status="Found")]})
+ changed = report_mod.diff_reports(old, new)["changed"]
+ assert changed and changed[0]["key"] == "alice:github"
+
+
+# --- correlation ----------------------------------------------------------
+
+def test_cluster_by_shared_name():
+ profiles = [
+ Profile("alice", "github", name="Alice Smith", bio="", avatar="", url="", status="Found"),
+ Profile("alice", "gitlab", name="Alice Smith", bio="", avatar="", url="", status="Found"),
+ Profile("bob", "reddit", name="Bob Jones", bio="", avatar="", url="", status="Found"),
+ ]
+ clusters = cluster_profiles(profiles)
+ assert len(clusters) == 1
+ assert clusters[0]["size"] == 2
+ assert "name" in clusters[0]["reasons"]
+
+
+def test_cluster_by_shared_link():
+ profiles = [
+ Profile("a", "s1", name="", bio="see https://me.example/x", avatar="", url="", status="Found"),
+ Profile("a", "s2", name="", bio="also https://me.example/x here", avatar="", url="", status="Maybe"),
+ ]
+ clusters = cluster_profiles(profiles)
+ assert len(clusters) == 1
+ assert "shared-link" in clusters[0]["reasons"]
+
+
+def test_no_cluster_when_unrelated():
+ profiles = [
+ Profile("a", "s1", name="Zoe", bio="cats", avatar="", url="", status="Found"),
+ Profile("b", "s2", name="Max", bio="dogs", avatar="", url="", status="Found"),
+ ]
+ assert cluster_profiles(profiles) == []
+
+
+def test_profiles_from_report_skips_not_found():
+ rep = _report({
+ "alice": [
+ _profile_result("github", "alice", status="Found", name="A"),
+ _profile_result("reddit", "alice", status="Not Found"),
+ ]
+ })
+ profiles = profiles_from_report(rep)
+ assert [p.site for p in profiles] == ["github"]
+
+
+# --- expansion ------------------------------------------------------------
+
+def test_candidate_usernames_from_bio():
+ results = {
+ "alice": [
+ _profile_result(
+ "github", "alice", status="Found",
+ bio="I'm @bob_dev, also https://twitter.com/charlie",
+ )
+ ]
+ }
+ candidates = candidate_usernames_from_results(results, exclude={"alice"})
+ assert "bob_dev" in candidates
+ assert "charlie" in candidates
+ assert "alice" not in [c.lower() for c in candidates]
+
+
+# --- domains --------------------------------------------------------------
+
+def test_domain_label_sanitizes():
+ assert domain_label("John_Doe.99") == "john-doe-99"
+ assert domain_label("!!!") == ""
+
+
+# --- checkpoint -----------------------------------------------------------
+
+async def test_checkpoint_roundtrip(tmp_path):
+ path = tmp_path / "ck.jsonl"
+ ck = Checkpoint(path)
+ result = _profile_result("github", "alice")
+ await ck.record("alice", "github", result)
+ assert ck.is_done("alice", "github")
+
+ reloaded = Checkpoint(path)
+ assert reloaded.is_done("alice", "github")
+ assert reloaded.results_for("alice")["github"]["site"] == "github"
+
+ reloaded.finalize()
+ assert not path.exists()
+
+
+# --- graph export ---------------------------------------------------------
+
+def test_graph_exports_written(tmp_path):
+ exporter = ResultsExporter(tmp_path)
+ results = {"alice": [_profile_result("github", "alice", name="Alice")]}
+ written = exporter.save_results("alice", "basic", results, ["gexf", "mermaid", "maltego"])
+ suffixes = {p.suffix for p in written}
+ assert ".gexf" in suffixes
+ assert ".mmd" in suffixes
+ assert any(p.name.endswith("_maltego.csv") for p in written)
+ gexf = next(p for p in written if p.suffix == ".gexf")
+ assert "github" in gexf.read_text(encoding="utf-8")
+
+
+def test_all_format_excludes_graph_and_pdf(tmp_path):
+ exporter = ResultsExporter(tmp_path)
+ results = {"alice": [_profile_result("github", "alice")]}
+ written = exporter.save_results("alice", "basic", results, ["all"])
+ suffixes = {p.suffix for p in written}
+ assert suffixes == {".json", ".csv", ".html", ".md"}
+
+
+# --- dataset append -------------------------------------------------------
+
+def test_write_rows_append(tmp_path):
+ from aliens_eye.core.features import FEATURE_SCHEMA
+
+ path = tmp_path / "ds.csv"
+ row = [0.0] * len(FEATURE_SCHEMA) + [1.0]
+ write_rows(path, [row], append=True)
+ write_rows(path, [row], append=True)
+ lines = path.read_text(encoding="utf-8").strip().splitlines()
+ assert lines[0].startswith("http_200") # single header
+ assert len(lines) == 3 # header + 2 rows
+
+
+# --- selfcheck metrics ----------------------------------------------------
+
+def test_metrics_math():
+ m = _metrics(tp=8, fp=2, fn=2, tn=8)
+ assert m["precision"] == 0.8
+ assert m["recall"] == 0.8
+ assert round(m["f1"], 2) == 0.8
+ assert m["false_positive_rate"] == 0.2