AI powered OSINT engine to discover and map 840+ social media identities across the web
Python aiartificial-intelligencedebainhackhackerhackinghackingtoolsinformationinformation-gatheringlinuxlinux-desktoposintpythonpython-scriptpython3social-mediasocialmediahackingtermuxtermux-tool
feat: OSINT investigation suite expansion (13 features)
arxhr0075a4b9e353a54
diff --git a/README.md b/README.mdindex 9083c0f..3cd5322 100644--- a/README.md+++ b/README.md@@ -25,13 +25,20 @@## Highlights- **840+ platforms** scanned asynchronously in seconds-- **ML + heuristic detection** — a trained model blended with 25 structural signals (HTTP status, DOM shape, keywords, fingerprints) instead of naive status-code checks-- **Modern terminal UI** — live progress, sorted result tables, summary panels (powered by [rich](https://github.com/Textualize/rich))+- **ML + heuristic detection** — a trained model blended with 30 structural signals (HTTP status, DOM shape, keywords, fingerprints) instead of naive status-code checks+- **Profile extraction** — display name, bio, and avatar pulled from each hit (OpenGraph / JSON-LD / per-site CSS)+- **Cross-site correlation** — cluster profiles that look like the same person by avatar hash, bio, shared links, and name (`--correlate`)+- **Recursive expansion** — follow linked usernames out of bios and re-scan them (`--recurse-depth N`)+- **Domain check** — is `<username>.{com,io,net,…}` registered and live? (`--domains`)+- **Watch mode** — re-scan on an interval and alert on changes, optionally to a webhook (`--watch 6h --notify <url>`)+- **Resumable scans** — checkpoint progress and continue after an interruption (`--resume file.jsonl`)+- **Modern terminal UI** — live progress, sorted result tables, summary panels (powered by [rich](https://github.com/Textualize/rich)); plus an interactive browser (`aliens_eye tui`, optional extra)+- **MCP server** — expose scanning to LLM agents (`aliens_eye serve`, optional extra)- **Proxy & Tor support** — `--proxy socks5://...` or just `--tor`-- **Site filtering** — `--site github,reddit`, `--exclude-site`, `--no-nsfw`-- **Self-check** — `aliens_eye selfcheck` validates detection accuracy against accounts known to exist-- **Retrainable** — collect your own labeled dataset and retrain the model with `aliens_eye train`-- **Reports** in JSON, CSV, HTML, and Markdown+- **Site filtering** — `--site github,reddit`, `--exclude-site`, `--no-nsfw`, plus drop-in `sites.d/` plugin site maps+- **Calibrated self-check** — `aliens_eye selfcheck` reports precision / recall / F1 / FPR per site+- **Retrainable + active learning** — retrain with `aliens_eye train`, or hand-label uncertain hits with `aliens_eye label`+- **Reports** in JSON, CSV, HTML, Markdown, PDF, and graph formats (GEXF, Mermaid, Maltego CSV)- **Playwright fallback** for JavaScript-heavy pages (optional extra)## Install@@ -47,6 +54,10 @@ pip install "aliens-eye[browser]" # Playwright fallback for hard pagespython -m playwright install chromiumpip install "aliens-eye[train]" # scikit-learn, for retraining the ML model+pip install "aliens-eye[correlate]" # Pillow, for avatar-image matching in --correlate+pip install "aliens-eye[pdf]" # reportlab, for --format pdf+pip install "aliens-eye[tui]" # textual, for the interactive `tui` browser+pip install "aliens-eye[serve]" # mcp, for the `serve` MCP server```Or with Docker:@@ -106,13 +117,47 @@ aliens_eye username --plain# View results from a previous scanaliens_eye -r results/username_advanced_20260611_120000.json-# Validate detection accuracy against known accounts-aliens_eye selfcheck+# Correlate hits into "likely same person" clusters + check domains+aliens_eye username --correlate --domains++# Follow linked usernames out of found bios and re-scan them+aliens_eye username --recurse-depth 1++# Export a graph of the results (import into Gephi / Maltego / Mermaid)+aliens_eye username --correlate --format gexf,mermaid,maltego++# Investigator PDF with embedded avatars+aliens_eye username --format pdf++# Watch for changes every 6 hours and POST them to a webhook+aliens_eye username --watch 6h --notify https://hooks.example/aliens++# Resume an interrupted scan+aliens_eye username --resume scan.jsonl++# Compare two saved reports+aliens_eye diff results/old.json results/new.json++# Validate detection accuracy (precision / recall / F1 per site)+aliens_eye selfcheck --negatives 2 --report json++# Interactively label uncertain hits into a training set+aliens_eye label results/username_basic_20260611_120000.json --out labeled.csv++# Interactive terminal browser (needs [tui])+aliens_eye tui username++# Run the MCP server for LLM agents (needs [serve])+aliens_eye serve```+Custom platforms: drop a `{ "site_name": "https://site/{}" }` JSON file into+`./sites.d/` (or the user config dir's `sites.d/`) and it is merged automatically;+`--sites-dir DIR` adds another location.+## How detection works-Every response is converted into a 25-dimensional feature vector: HTTP status buckets, username placement (path/title/meta), error and profile keywords, DOM structure (images, forms, profile/error CSS classes), response timing, redirect counts, and per-site fingerprint matches learned from previous scans.+Every response is converted into a 30-dimensional feature vector: HTTP status buckets, username placement (path/title/meta/canonical), error and profile keywords, DOM structure (images, forms, profile/error CSS classes), structured-data signals (og:type, JSON-LD Person), response timing, redirect counts, and per-site fingerprint matches learned from previous scans.Two judges then vote:diff --git a/pyproject.toml b/pyproject.tomlindex 8354a1c..1b33d09 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -35,6 +35,10 @@ dependencies = [[project.optional-dependencies]browser = ["playwright>=1.42"]train = ["scikit-learn>=1.3", "numpy>=1.24"]+correlate = ["Pillow>=10.0"]+tui = ["textual>=0.60"]+pdf = ["reportlab>=4.0"]+serve = ["mcp>=1.0"]dev = ["pytest>=8","pytest-asyncio>=0.23",diff --git a/src/aliens_eye/cli.py b/src/aliens_eye/cli.pyindex 255705c..0980dd8 100644--- a/src/aliens_eye/cli.py+++ b/src/aliens_eye/cli.py@@ -23,6 +23,7 @@ from aliens_eye.core.scanner import (UsernameScanner,filter_sites,load_nsfw_sites,+ load_site_plugins,load_sites_data,)from aliens_eye.utils.console import get_console, set_plain@@ -123,6 +124,39 @@ def build_parser() -> ArgumentParser:choices=["quick", "full", "aggressive"],help="Scan profile preset (skips interactive prompt)",)+ parser.add_argument(+ "--watch",+ type=str,+ help="Re-scan on an interval and alert on changes (e.g. 30m, 6h, 1d)",+ )+ parser.add_argument("--notify", type=str, help="Webhook URL to POST watch-mode changes to")+ parser.add_argument(+ "--correlate",+ action="store_true",+ help="Cluster Found/Maybe profiles that look like the same person (avatar/bio/links)",+ )+ parser.add_argument(+ "--domains",+ action="store_true",+ help="Check whether <username>.{com,io,net,...} domains are registered/live",+ )+ parser.add_argument(+ "--recurse-depth",+ type=int,+ default=0,+ help="Extract linked usernames from found profiles and re-scan, up to N hops",+ )+ parser.add_argument("--sites-dir", type=str, help="Directory of extra *.json site maps to merge")+ parser.add_argument("--resume", type=str, help="Checkpoint file to resume an interrupted scan from")+ parser.add_argument("--from-email", type=str, help="Seed usernames from an email address")+ parser.add_argument("--from-name", type=str, help="Seed usernames from a real name")+ parser.add_argument("--from-phone", type=str, help="Seed usernames from a phone number")+ parser.add_argument(+ "--only-found", action="store_true", help="Report only Found/Maybe results"+ )+ parser.add_argument(+ "--json-stdout", action="store_true", help="Print results JSON to stdout (implies --plain)"+ )return parser@@ -136,6 +170,13 @@ def build_selfcheck_parser() -> ArgumentParser:parser.add_argument("--model", type=str, help="Path to a custom ML model JSON")parser.add_argument("--sites", type=str, help="Path to a custom sites JSON")parser.add_argument("--plain", action="store_true", help="Plain output")+ parser.add_argument(+ "--negatives", type=int, default=1, help="Random non-existent usernames to test per site"+ )+ parser.add_argument(+ "--report", choices=["table", "json"], default="table",+ help="Output format: rich table (default) or machine-readable JSON",+ )return parser@@ -160,6 +201,58 @@ def build_train_parser() -> ArgumentParser:return parser+def build_diff_parser() -> ArgumentParser:+ parser = ArgumentParser(+ prog="aliens_eye diff",+ description="Compare two saved scan reports and show what changed",+ )+ parser.add_argument("old", help="Path to the older report JSON")+ parser.add_argument("new", help="Path to the newer report JSON")+ parser.add_argument("--plain", action="store_true", help="Plain output")+ parser.add_argument(+ "--json-stdout", action="store_true", help="Print the diff as JSON to stdout"+ )+ return parser+++def build_serve_parser() -> ArgumentParser:+ parser = ArgumentParser(+ prog="aliens_eye serve",+ description="Run an MCP server exposing scan tools over stdio",+ )+ parser.add_argument("--sites", type=str, help="Path to a custom sites JSON")+ parser.add_argument("--no-ml", action="store_true", help="Disable ML detection")+ parser.add_argument("-v", "--verbose", action="store_true")+ return parser+++def build_tui_parser() -> ArgumentParser:+ parser = ArgumentParser(+ prog="aliens_eye tui",+ description="Interactive terminal UI for scanning",+ )+ parser.add_argument("username", nargs="?", help="Username to scan")+ parser.add_argument("-l", "--level", choices=["basic", "intermediate", "advanced"], default="basic")+ parser.add_argument("--sites", type=str, help="Path to a custom sites JSON")+ parser.add_argument("--no-ml", action="store_true", help="Disable ML detection")+ return parser+++def build_label_parser() -> ArgumentParser:+ parser = ArgumentParser(+ prog="aliens_eye label",+ description="Interactively label low-confidence results and append them to a dataset",+ )+ parser.add_argument("report", help="Path to a saved report JSON to label")+ parser.add_argument("--out", type=str, default="labeled.csv", help="Dataset CSV to append to")+ parser.add_argument(+ "--band", type=str, default="Maybe",+ help="Which statuses to review: 'Maybe' (default) or 'all'",+ )+ parser.add_argument("-v", "--verbose", action="store_true")+ return parser++def parse_formats(value: str) -> list[str]:formats = [item.strip().lower() for item in value.split(",") if item.strip()]if not formats:@@ -184,6 +277,22 @@ def parse_site_list(value: str | None) -> list[str] | None:return items or None+def seed_usernames_from_lookups(args) -> list[str]:+ """Expand --from-email/--from-name/--from-phone into candidate usernames."""+ from aliens_eye.core.variations import usernames_from_email, usernames_from_name++ seeds: list[str] = []+ if getattr(args, "from_email", None):+ seeds.extend(usernames_from_email(args.from_email))+ if getattr(args, "from_name", None):+ seeds.extend(usernames_from_name(args.from_name))+ if getattr(args, "from_phone", None):+ digits = "".join(ch for ch in args.from_phone if ch.isdigit())+ if len(digits) >= 4:+ seeds.append(digits)+ return seeds++def prompt_scan_profile() -> str:console = get_console()console.print("\n[green]Select scan profile:[/green]")@@ -345,16 +454,27 @@ def apply_cli_overrides(config: ScannerConfig, args) -> None:config.exclude_nsfw = Trueconfig.include_sites = parse_site_list(args.site)config.exclude_sites = parse_site_list(args.exclude_site)+ if args.only_found:+ config.only_found = True+ if args.json_stdout:+ config.json_stdout = True+ config.plain_output = Trueif args.plain:config.plain_output = True-def prepare_sites(config: ScannerConfig, logger) -> dict:+def prepare_sites(config: ScannerConfig, logger, sites_dir: str | None = None) -> dict:sites_data = load_sites_data(config.sites_path)if not sites_data:logger.error("Could not load sites.json.")return {}+ extra_dirs = [Path(sites_dir)] if sites_dir else None+ plugins = load_site_plugins(extra_dirs)+ if plugins:+ logger.info("Merged %d plugin site(s) from sites.d directories", len(plugins))+ sites_data = {**sites_data, **plugins}+exclude = list(config.exclude_sites or [])if config.exclude_nsfw:exclude.extend(load_nsfw_sites())@@ -417,7 +537,9 @@ async def run_scan(args) -> None:config = ScannerConfig()defaults = apply_config(config, config_data)apply_cli_overrides(config, args)- if config.plain_output:+ if config.json_stdout:+ set_plain(True, stderr=True)+ elif config.plain_output:set_plain(True)console = get_console()@@ -426,15 +548,20 @@ async def run_scan(args) -> None:ResultsExporter(exporter_dir).display_results_from_file(args.read)return- sites_data = prepare_sites(config, logger)+ sites_data = prepare_sites(config, logger, sites_dir=args.sites_dir)if not sites_data:return- display_banner(len(sites_data))+ if not config.json_stdout:+ display_banner(len(sites_data))- if args.username:- usernames = args.username- else:+ usernames = list(args.username)+ usernames.extend(seed_usernames_from_lookups(args))+ usernames = list(dict.fromkeys(usernames))+ if not usernames:+ if config.json_stdout:+ logger.error("No username provided.")+ returnusername = Prompt.ask("[green]Enter username to scan[/green]", console=console)usernames = [username]@@ -447,7 +574,7 @@ async def run_scan(args) -> None:fingerprints = FingerprintStore(config.fingerprints_path, config.max_fingerprints_per_label)fingerprints.load(logger)- interactive = sys.stdin.isatty() and not args.read+ interactive = sys.stdin.isatty() and not args.read and not config.json_stdoutscan_level = args.level or defaults.get("level")if scan_level not in {"basic", "intermediate", "advanced"}:@@ -473,6 +600,12 @@ async def run_scan(args) -> None:if config.use_playwright:browser_fallback = BrowserFallback(config.timeout, DEFAULT_USER_AGENT)+ checkpoint = None+ if args.resume:+ from aliens_eye.core.checkpoint import Checkpoint++ checkpoint = Checkpoint(args.resume)+exporter = ResultsExporter(config.output_dir)scanner = UsernameScanner(sites_data=sites_data,@@ -482,8 +615,36 @@ async def run_scan(args) -> None:fingerprints=fingerprints,logger=logger,browser_fallback=browser_fallback,+ checkpoint=checkpoint,)+ if args.watch:+ from aliens_eye.core import watch as watch_mod++ try:+ interval = watch_mod.parse_duration(args.watch)+ except ValueError as exc:+ logger.error("Invalid --watch interval: %s", exc)+ return+ try:+ await watch_mod.run_watch(+ scanner=scanner,+ exporter=exporter,+ usernames=usernames,+ scan_level=scan_level,+ formats=formats,+ interval=interval,+ logger=logger,+ console=console,+ output_dir=config.output_dir,+ notify_url=args.notify,+ )+ finally:+ fingerprints.save()+ if browser_fallback:+ await browser_fallback.close()+ return+try:for username in usernames:if not username or len(username) < 2:@@ -491,7 +652,31 @@ async def run_scan(args) -> None:continueall_results = await scanner.scan_with_variations(username, scan_level)- written = exporter.save_results(username, scan_level, all_results, formats)++ if args.recurse_depth and args.recurse_depth > 0:+ await _recurse_scan(+ scanner, all_results, username, scan_level, args.recurse_depth, console+ )++ correlation = None+ if args.correlate:+ from aliens_eye.core.correlate import correlate_report++ report = exporter.report_dict(username, scan_level, all_results)+ correlation = await correlate_report(report, proxy=config.proxy, timeout=config.timeout)+ _print_clusters(console, correlation)++ domains = None+ if args.domains:+ from aliens_eye.core.domains import check_domains++ domains = await check_domains(username, proxy=config.proxy)+ _print_domains(console, domains)++ written = exporter.save_results(+ username, scan_level, all_results, formats,+ correlation=correlation, domains=domains,+ )files_list = ", ".join(str(path) for path in written)console.print(f"\n[green]Results saved to:[/green] [blue]{files_list}[/blue]")@@ -503,16 +688,82 @@ async def run_scan(args) -> None:f"[green]Summary: Found {total_found} profiles across "f"{len(all_results)} username variations[/green]")+ if checkpoint is not None:+ checkpoint.finalize()finally:fingerprints.save()if browser_fallback:await browser_fallback.close()+async def _recurse_scan(scanner, all_results, base_username, scan_level, depth, console) -> None:+ """Extract linked usernames from found profiles and scan them, up to `depth` hops."""+ from aliens_eye.core.expand import candidate_usernames_from_results++ seen = {base_username.lower()}+ frontier = candidate_usernames_from_results(all_results, exclude=seen)+ for hop in range(depth):+ frontier = [u for u in frontier if u.lower() not in seen]+ if not frontier:+ break+ console.print(+ f"[blue]Recursion hop {hop + 1}:[/blue] {len(frontier)} linked username(s): "+ f"[dim]{', '.join(frontier[:8])}{'...' if len(frontier) > 8 else ''}[/dim]"+ )+ next_frontier: list[str] = []+ for uname in frontier:+ seen.add(uname.lower())+ sub = await scanner.scan_all_sites(uname)+ all_results[uname] = sub+ next_frontier.extend(+ candidate_usernames_from_results({uname: sub}, exclude=seen)+ )+ frontier = list(dict.fromkeys(next_frontier))+++def _print_clusters(console, correlation: dict) -> None:+ clusters = correlation.get("clusters", [])+ if not clusters:+ console.print("[dim]Correlation: no cross-site clusters found.[/dim]")+ return+ from rich.table import Table++ table = Table(title="Correlation: likely same person", header_style="bold blue")+ table.add_column("#", justify="right")+ table.add_column("Sites", style="yellow")+ table.add_column("Linked by")+ for i, cluster in enumerate(clusters, 1):+ sites = ", ".join(f"{m['site']}" for m in cluster["members"])+ table.add_row(str(i), sites, ", ".join(cluster.get("reasons", [])))+ console.print(table)+ if not correlation.get("avatar_hashing"):+ console.print(+ "[dim]Tip: install \"aliens-eye\\[correlate]\" for avatar-image matching.[/dim]"+ )+++def _print_domains(console, domains: dict) -> None:+ from rich.table import Table++ table = Table(title="Domain check", header_style="bold blue")+ table.add_column("Domain", style="yellow")+ table.add_column("Registered")+ table.add_column("Live")+ for entry in domains.get("results", []):+ table.add_row(+ entry["domain"],+ "[green]yes[/green]" if entry["registered"] else "[dim]no[/dim]",+ "[green]yes[/green]" if entry["live"] else "[dim]no[/dim]",+ )+ console.print(table)++async def run_selfcheck_command(args) -> None:from aliens_eye.selfcheck import run_selfcheck- if args.plain:+ if args.report == "json":+ set_plain(True, stderr=True)+ elif args.plain:set_plain(True)logger = setup_logger(args.verbose)config = ScannerConfig(retries=1)@@ -520,7 +771,10 @@ async def run_selfcheck_command(args) -> None:detector = Detector()if not args.no_ml:detector.load_model(logger, Path(args.model) if args.model else None)- await run_selfcheck(sites_data, detector, config, logger)+ await run_selfcheck(+ sites_data, detector, config, logger,+ negatives=args.negatives, report_format=args.report,+ )async def run_train_command(args) -> None:@@ -550,6 +804,68 @@ async def run_train_command(args) -> None:)+def run_diff_command(args) -> None:+ from aliens_eye.core.report import ReportError, diff_reports, load_report++ if args.json_stdout:+ set_plain(True, stderr=True)+ elif args.plain:+ set_plain(True)+ console = get_console()+ try:+ old = load_report(args.old)+ new = load_report(args.new)+ except ReportError as exc:+ console.print(f"[red]{exc}[/red]")+ sys.exit(1)++ delta = diff_reports(old, new)+ if args.json_stdout:+ print(json.dumps(delta, indent=2))+ return++ from rich.table import Table++ if not (delta["new"] or delta["gone"] or delta["changed"]):+ console.print("[dim]No differences between the two reports.[/dim]")+ return++ table = Table(title="Report diff", header_style="bold blue")+ table.add_column("Change")+ table.add_column("Account (variation:site)", style="yellow")+ table.add_column("Detail")+ for item in delta["new"]:+ table.add_row("[green]+ NEW[/green]", item["key"], item.get("url", ""))+ for item in delta["gone"]:+ table.add_row("[red]- GONE[/red]", item["key"], item.get("url", ""))+ for item in delta["changed"]:+ before, after = item["before"], item["after"]+ detail = (+ f"{before['status']} ({before['confidence']}%) -> "+ f"{after['status']} ({after['confidence']}%)"+ )+ table.add_row("[yellow]~ CHANGED[/yellow]", item["key"], detail)+ console.print(table)+++async def run_serve_command(args) -> None:+ from aliens_eye.mcp_server import serve++ await serve(args)+++def run_tui_command(args) -> None:+ from aliens_eye.tui.app import run_tui++ run_tui(args)+++async def run_label_command(args) -> None:+ from aliens_eye.ml.label import label_report++ await label_report(args)++def main() -> None:if sys.platform == "win32":sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[union-attr]@@ -563,6 +879,18 @@ def main() -> None:elif argv and argv[0] == "train":args = build_train_parser().parse_args(argv[1:])asyncio.run(run_train_command(args))+ elif argv and argv[0] == "diff":+ args = build_diff_parser().parse_args(argv[1:])+ run_diff_command(args)+ elif argv and argv[0] == "serve":+ args = build_serve_parser().parse_args(argv[1:])+ asyncio.run(run_serve_command(args))+ elif argv and argv[0] == "tui":+ args = build_tui_parser().parse_args(argv[1:])+ run_tui_command(args)+ elif argv and argv[0] == "label":+ args = build_label_parser().parse_args(argv[1:])+ asyncio.run(run_label_command(args))else:args = build_parser().parse_args(argv)asyncio.run(run_scan(args))diff --git a/src/aliens_eye/core/analyzer.py b/src/aliens_eye/core/analyzer.pyindex af5dcd5..6742190 100644--- a/src/aliens_eye/core/analyzer.py+++ b/src/aliens_eye/core/analyzer.py@@ -1,7 +1,9 @@+import jsonimport refrom dataclasses import dataclass+from importlib import resourcesfrom typing import Any-from urllib.parse import urlparse+from urllib.parse import urljoin, urlparsefrom selectolax.parser import HTMLParser@@ -15,6 +17,29 @@ from .config import ()_JSON_LD_PERSON_RE = re.compile(r'"@type"\s*:\s*"Person"', re.IGNORECASE)+_JSON_LD_STR_FIELD_RE = {+ "name": re.compile(r'"name"\s*:\s*"([^"]+)"'),+ "image": re.compile(r'"image"\s*:\s*"([^"]+)"'),+}++_site_profiles_cache: dict[str, Any] | None = None+++def _load_site_profiles() -> dict[str, Any]:+ """Optional per-site CSS-selector overrides for profile fields.++ Maps a site name to ``{"name": sel, "bio": sel, "avatar": sel}``. Missing or+ invalid file yields an empty map (generic extraction is used instead).+ """+ global _site_profiles_cache+ if _site_profiles_cache is None:+ try:+ text = (resources.files("aliens_eye.data") / "site_profiles.json").read_text("utf-8")+ data = json.loads(text)+ _site_profiles_cache = data if isinstance(data, dict) else {}+ except (FileNotFoundError, json.JSONDecodeError, OSError):+ _site_profiles_cache = {}+ return _site_profiles_cache@dataclass@@ -129,6 +154,8 @@ class FeatureExtractor:body_node = tree.css_first("body")text_length = float(len(body_node.text(deep=True))) if body_node else 0.0+ profile = self._extract_profile(tree, url, site_name, title_text)+features = {"http_200": 1.0 if http_code == 200 else 0.0,"http_3xx": 1.0 if 300 <= http_code < 400 else 0.0,@@ -166,6 +193,7 @@ class FeatureExtractor:signals = {"site": site_name,"title": title_text,+ "profile": profile,"meta_samples": meta_contents[:5],"url_analysis": {"domain": parsed_url.netloc,@@ -195,3 +223,86 @@ class FeatureExtractor:}return FeatureBundle(features=features, signals=signals, fingerprint=fingerprint)++ def _extract_profile(+ self, tree: HTMLParser, url: str, site_name: str, title_text: str+ ) -> dict[str, str]:+ """Best-effort display name / bio / avatar for a profile page.++ Generic and provider-agnostic: OpenGraph tags, then JSON-LD Person,+ then plain <title>/<meta>. Per-site CSS overrides in+ data/site_profiles.json take precedence when present.+ """+ name = self._meta_content(tree, 'meta[property="og:title"]')+ bio = self._meta_content(tree, 'meta[property="og:description"]')+ avatar = self._meta_content(tree, 'meta[property="og:image"]')++ if not (name and avatar):+ ld_name, ld_image = self._json_ld_person_fields(tree)+ name = name or ld_name+ avatar = avatar or ld_image++ if not bio:+ bio = self._meta_content(tree, 'meta[name="description"]')+ if not name:+ name = title_text+ if not avatar:+ icon = tree.css_first('link[rel="apple-touch-icon"]')+ if icon:+ avatar = icon.attributes.get("href") or ""++ overrides = _load_site_profiles().get(site_name)+ if isinstance(overrides, dict):+ name = self._select_text(tree, overrides.get("name")) or name+ bio = self._select_text(tree, overrides.get("bio")) or bio+ avatar = self._select_attr(tree, overrides.get("avatar")) or avatar++ if avatar:+ avatar = urljoin(url, avatar)++ return {+ "name": (name or "").strip()[:200],+ "bio": (bio or "").strip()[:500],+ "avatar": (avatar or "").strip(),+ }++ @staticmethod+ def _meta_content(tree: HTMLParser, selector: str) -> str:+ node = tree.css_first(selector)+ if node:+ return (node.attributes.get("content") or "").strip()+ return ""++ @staticmethod+ def _json_ld_person_fields(tree: HTMLParser) -> tuple[str, str]:+ for node in tree.css('script[type="application/ld+json"]'):+ script_text = node.text() or ""+ if not _JSON_LD_PERSON_RE.search(script_text):+ continue+ name_match = _JSON_LD_STR_FIELD_RE["name"].search(script_text)+ image_match = _JSON_LD_STR_FIELD_RE["image"].search(script_text)+ return (+ name_match.group(1) if name_match else "",+ image_match.group(1) if image_match else "",+ )+ return "", ""++ @staticmethod+ def _select_text(tree: HTMLParser, selector: str | None) -> str:+ if not selector:+ return ""+ node = tree.css_first(selector)+ return node.text().strip() if node else ""++ @staticmethod+ def _select_attr(tree: HTMLParser, selector: str | None) -> str:+ if not selector:+ return ""+ node = tree.css_first(selector)+ if not node:+ return ""+ for attr in ("src", "href", "content", "data-src"):+ value = node.attributes.get(attr)+ if value:+ return value+ return ""diff --git a/src/aliens_eye/core/checkpoint.py b/src/aliens_eye/core/checkpoint.pynew file mode 100644index 0000000..4840b57--- /dev/null+++ b/src/aliens_eye/core/checkpoint.py@@ -0,0 +1,65 @@+"""Resumable scan checkpoints.++Each completed ``(variation, site)`` result is appended as one JSON line to a+checkpoint file. On resume, previously completed sites are skipped and their+results are reused, so a scan interrupted by Ctrl-C or a crash continues where it+left off instead of re-hitting every site.+"""++from __future__ import annotations++import asyncio+import json+from pathlib import Path+from typing import Any+++class Checkpoint:+ def __init__(self, path: str | Path) -> None:+ self.path = Path(path)+ self._lock = asyncio.Lock()+ # (variation, site) -> stored result dict+ self._done: dict[tuple[str, str], dict[str, Any]] = {}+ self._load()++ def _load(self) -> None:+ if not self.path.exists():+ return+ try:+ for line in self.path.read_text(encoding="utf-8").splitlines():+ line = line.strip()+ if not line:+ continue+ entry = json.loads(line)+ self._done[(entry["variation"], entry["site"])] = entry["result"]+ except (OSError, json.JSONDecodeError, KeyError):+ # A corrupt/partial checkpoint is treated as empty rather than fatal.+ self._done = {}++ def is_done(self, variation: str, site: str) -> bool:+ return (variation, site) in self._done++ def results_for(self, variation: str) -> dict[str, dict[str, Any]]:+ """site -> result for every completed site of this variation."""+ return {+ site: result+ for (var, site), result in self._done.items()+ if var == variation+ }++ async def record(self, variation: str, site: str, result: dict[str, Any]) -> None:+ key = (variation, site)+ async with self._lock:+ if key in self._done:+ return+ self._done[key] = result+ self.path.parent.mkdir(parents=True, exist_ok=True)+ with self.path.open("a", encoding="utf-8") as handle:+ handle.write(json.dumps({"variation": variation, "site": site, "result": result}) + "\n")++ def finalize(self) -> None:+ """Remove the checkpoint file once the scan has completed successfully."""+ try:+ self.path.unlink(missing_ok=True)+ except OSError:+ passdiff --git a/src/aliens_eye/core/config.py b/src/aliens_eye/core/config.pyindex d397029..03e1ea0 100644--- a/src/aliens_eye/core/config.py+++ b/src/aliens_eye/core/config.py@@ -1,4 +1,6 @@+import jsonfrom dataclasses import dataclass, field+from importlib import resourcesfrom pathlib import Pathfrom platformdirs import user_cache_dir@@ -18,7 +20,7 @@ DEFAULT_HEADERS = {"Upgrade-Insecure-Requests": "1",}-ERROR_KEYWORDS = [+_EN_ERROR_KEYWORDS = ["not found","doesn't exist","didn't find",@@ -55,7 +57,7 @@ ERROR_KEYWORDS = ["page was not found",]-POSITIVE_KEYWORDS = [+_EN_POSITIVE_KEYWORDS = ["follow","subscribe","like",@@ -83,7 +85,7 @@ POSITIVE_KEYWORDS = ["timeline",]-META_KEYWORDS = [+_EN_META_KEYWORDS = ["profile picture","profile image","avatar",@@ -95,6 +97,38 @@ META_KEYWORDS = ["user information",]+def _merge_keywords(group: str, base: list[str]) -> list[str]:+ """Union the packaged multilingual keywords with the English base list.++ Falls back to the base list if data/keywords.json is missing or invalid.+ Matching is substring-on-lowercased-content, so keywords are lowercased and+ deduplicated while preserving order (English first).+ """+ merged: list[str] = []+ seen: set[str] = set()++ def _add(words) -> None:+ for word in words:+ key = str(word).strip().lower()+ if key and key not in seen:+ seen.add(key)+ merged.append(key)++ _add(base)+ try:+ text = (resources.files("aliens_eye.data") / "keywords.json").read_text("utf-8")+ data = json.loads(text)+ for lang_words in (data.get(group) or {}).values():+ _add(lang_words)+ except (FileNotFoundError, json.JSONDecodeError, OSError, AttributeError):+ pass+ return merged+++ERROR_KEYWORDS = _merge_keywords("error", _EN_ERROR_KEYWORDS)+POSITIVE_KEYWORDS = _merge_keywords("positive", _EN_POSITIVE_KEYWORDS)+META_KEYWORDS = _merge_keywords("meta", _EN_META_KEYWORDS)+AUTH_PATTERNS = ["/login","/signin",@@ -139,3 +173,5 @@ class ScannerConfig:model_path: Path | None = Noneuse_ml: bool = Trueplain_output: bool = False+ only_found: bool = False+ json_stdout: bool = Falsediff --git a/src/aliens_eye/core/correlate.py b/src/aliens_eye/core/correlate.pynew file mode 100644index 0000000..b7dccdf--- /dev/null+++ b/src/aliens_eye/core/correlate.py@@ -0,0 +1,265 @@+"""Cross-site correlation: cluster Found/Maybe profiles that look like the same person.++Signals used per profile: avatar perceptual hash (dhash, requires Pillow via the+``[correlate]`` extra), normalized display name, tokenized bio, and external links+found in the bio. Profiles are linked by a union-find over pairwise similarity and+grouped into clusters ("likely the same person").++Avatar hashing is best-effort: if Pillow is missing or an image fails to download,+correlation still runs on name/bio/link signals alone.+"""++from __future__ import annotations++import asyncio+import re+from dataclasses import dataclass, field+from typing import Any++import aiohttp++from .config import DEFAULT_HEADERS, ScannerConfig+from .scanner import build_connector++_URL_RE = re.compile(r"https?://[^\s\"'<>)]+", re.IGNORECASE)+_HANDLE_RE = re.compile(r"@([A-Za-z0-9_.]{2,30})")+_WORD_RE = re.compile(r"[a-z0-9]+")++# Similarity thresholds.+AVATAR_MAX_HAMMING = 8 # <= this many differing bits => same image+BIO_MIN_JACCARD = 0.5 # token overlap ratio to link on bio alone+LINK_MATCH_WEIGHT = True # a shared external link links two profiles+++@dataclass+class Profile:+ variation: str+ site: str+ name: str+ bio: str+ avatar: str+ url: str+ status: str+ avatar_hash: int | None = None+ links: set[str] = field(default_factory=set)+ handles: set[str] = field(default_factory=set)++ def __post_init__(self) -> None:+ # Derive links/handles from the bio when not supplied explicitly.+ if not self.links:+ self.links = set(_URL_RE.findall(self.bio or ""))+ if not self.handles:+ self.handles = {h.lower() for h in _HANDLE_RE.findall(self.bio or "")}++ @property+ def key(self) -> str:+ return f"{self.variation}:{self.site}"++ def to_dict(self) -> dict[str, Any]:+ return {+ "variation": self.variation,+ "site": self.site,+ "name": self.name,+ "bio": self.bio,+ "avatar": self.avatar,+ "url": self.url,+ "status": self.status,+ "avatar_hash": format(self.avatar_hash, "016x") if self.avatar_hash is not None else None,+ }+++def _norm_name(name: str) -> str:+ return " ".join(_WORD_RE.findall((name or "").lower()))+++def _tokens(text: str) -> set[str]:+ return {w for w in _WORD_RE.findall((text or "").lower()) if len(w) > 2}+++def _jaccard(a: set[str], b: set[str]) -> float:+ if not a or not b:+ return 0.0+ inter = len(a & b)+ union = len(a | b)+ return inter / union if union else 0.0+++def profiles_from_report(report: dict[str, Any]) -> list[Profile]:+ """Extract Found/Maybe profiles (with any profile signals) from a report dict."""+ profiles: list[Profile] = []+ for variation, block in report.get("variations", {}).items():+ for site, info in (block.get("sites", {}) or {}).items():+ if info.get("status") not in {"Found", "Maybe"}:+ continue+ prof = info.get("ai_analysis", {}).get("signals", {}).get("profile", {}) or {}+ bio = prof.get("bio", "") or ""+ p = Profile(+ variation=variation,+ site=site,+ name=prof.get("name", "") or "",+ bio=bio,+ avatar=prof.get("avatar", "") or "",+ url=info.get("url", "") or "",+ status=info.get("status", ""),+ links=set(_URL_RE.findall(bio)),+ handles={h.lower() for h in _HANDLE_RE.findall(bio)},+ )+ profiles.append(p)+ return profiles+++# --- avatar hashing -------------------------------------------------------++def _dhash(image_bytes: bytes) -> int | None:+ """8x8 difference hash of an image -> 64-bit int. None if Pillow/decoding fails."""+ try:+ import io++ from PIL import Image+ except ImportError:+ return None+ try:+ img = Image.open(io.BytesIO(image_bytes)).convert("L").resize((9, 8))+ except Exception:+ return None+ bits = 0+ for row in range(8):+ for col in range(8):+ left = img.getpixel((col, row))+ right = img.getpixel((col + 1, row))+ bits = (bits << 1) | (1 if left > right else 0)+ return bits+++def _hamming(a: int, b: int) -> int:+ return bin(a ^ b).count("1")+++def pillow_available() -> bool:+ try:+ import PIL # noqa: F401++ return True+ except ImportError:+ return False+++async def _download_and_hash(+ session: aiohttp.ClientSession, profile: Profile, timeout: float+) -> None:+ if not profile.avatar:+ return+ try:+ async with session.get(profile.avatar, timeout=aiohttp.ClientTimeout(total=timeout)) as resp:+ if resp.status != 200:+ return+ data = await resp.content.read(2_000_000) # cap at 2 MB+ except Exception:+ return+ profile.avatar_hash = _dhash(data)+++async def hash_avatars(profiles: list[Profile], proxy: str | None = None, timeout: float = 10.0) -> None:+ """Download and dhash every profile avatar in place. No-op without Pillow."""+ if not pillow_available():+ return+ targets = [p for p in profiles if p.avatar]+ if not targets:+ return++ limit = min(20, len(targets))+ connector = build_connector(ScannerConfig(proxy=proxy), limit)+ sem = asyncio.Semaphore(limit)+ async with aiohttp.ClientSession(headers=DEFAULT_HEADERS, connector=connector) as session:++ async def worker(p: Profile) -> None:+ async with sem:+ await _download_and_hash(session, p, timeout)++ await asyncio.gather(*(worker(p) for p in targets))+++# --- clustering -----------------------------------------------------------++class _UnionFind:+ def __init__(self, n: int) -> None:+ self.parent = list(range(n))++ def find(self, x: int) -> int:+ while self.parent[x] != x:+ self.parent[x] = self.parent[self.parent[x]]+ x = self.parent[x]+ return x++ def union(self, a: int, b: int) -> None:+ ra, rb = self.find(a), self.find(b)+ if ra != rb:+ self.parent[rb] = ra+++def _pair_reason(a: Profile, b: Profile) -> str | None:+ """Return the reason two profiles are the same person, or None."""+ if (+ a.avatar_hash is not None+ and b.avatar_hash is not None+ and _hamming(a.avatar_hash, b.avatar_hash) <= AVATAR_MAX_HAMMING+ ):+ return "avatar"+ if a.links & b.links:+ return "shared-link"+ if a.handles & b.handles or (a.variation and a.variation in b.handles) or (b.variation and b.variation in a.handles):+ return "handle"+ na, nb = _norm_name(a.name), _norm_name(b.name)+ if na and na == nb and len(na) > 3:+ return "name"+ if _jaccard(_tokens(a.bio), _tokens(b.bio)) >= BIO_MIN_JACCARD:+ return "bio"+ return None+++def cluster_profiles(profiles: list[Profile]) -> list[dict[str, Any]]:+ """Union-find clustering. Returns clusters as dicts (only multi-site clusters)."""+ n = len(profiles)+ uf = _UnionFind(n)+ reasons: dict[tuple[int, int], str] = {}+ for i in range(n):+ for j in range(i + 1, n):+ reason = _pair_reason(profiles[i], profiles[j])+ if reason:+ uf.union(i, j)+ reasons[(i, j)] = reason++ groups: dict[int, list[int]] = {}+ for idx in range(n):+ groups.setdefault(uf.find(idx), []).append(idx)++ clusters: list[dict[str, Any]] = []+ for members in groups.values():+ if len(members) < 2:+ continue+ member_reasons = sorted(+ {reasons[(i, j)] for i in members for j in members if (i, j) in reasons}+ )+ clusters.append(+ {+ "size": len(members),+ "reasons": member_reasons,+ "members": [profiles[i].to_dict() for i in members],+ }+ )+ clusters.sort(key=lambda c: c["size"], reverse=True)+ return clusters+++async def correlate_report(+ report: dict[str, Any], proxy: str | None = None, timeout: float = 10.0+) -> dict[str, Any]:+ """Full correlation pass over a report dict. Returns a ``correlation`` block."""+ profiles = profiles_from_report(report)+ await hash_avatars(profiles, proxy=proxy, timeout=timeout)+ clusters = cluster_profiles(profiles)+ return {+ "avatar_hashing": pillow_available(),+ "profiles_considered": len(profiles),+ "clusters": clusters,+ }diff --git a/src/aliens_eye/core/domains.py b/src/aliens_eye/core/domains.pynew file mode 100644index 0000000..f3ae3be--- /dev/null+++ b/src/aliens_eye/core/domains.py@@ -0,0 +1,89 @@+"""Domain availability check for a username.++For a username, test ``<username>.<tld>`` across a handful of common TLDs:+DNS resolution says whether the domain is registered/points somewhere, and an+HTTP request says whether it serves a live site. No third-party whois service or+API key is used — just DNS + HTTP.+"""++from __future__ import annotations++import asyncio+import re+import socket+from typing import Any++import aiohttp++from .config import DEFAULT_HEADERS, ScannerConfig+from .scanner import build_connector++DEFAULT_TLDS = ["com", "io", "net", "org", "dev", "me", "co", "app"]++_LABEL_RE = re.compile(r"[^a-z0-9-]")+++def domain_label(username: str) -> str:+ """Reduce a username to a valid DNS label (lowercase alnum + hyphen)."""+ label = _LABEL_RE.sub("", username.lower().replace("_", "-").replace(".", "-"))+ return label.strip("-")+++async def _resolves(host: str) -> bool:+ loop = asyncio.get_event_loop()+ try:+ await loop.getaddrinfo(host, None, type=socket.SOCK_STREAM)+ return True+ except (socket.gaierror, OSError):+ return False+++async def _is_live(session: aiohttp.ClientSession, host: str, timeout: float) -> bool:+ for scheme in ("https", "http"):+ try:+ async with session.get(+ f"{scheme}://{host}",+ timeout=aiohttp.ClientTimeout(total=timeout),+ allow_redirects=True,+ ) as resp:+ if resp.status < 500:+ return True+ except Exception:+ continue+ return False+++async def check_domains(+ username: str,+ tlds: list[str] | None = None,+ proxy: str | None = None,+ timeout: float = 8.0,+) -> dict[str, Any]:+ """Check ``<username>.<tld>`` registration + liveness for each TLD."""+ label = domain_label(username)+ tlds = tlds or DEFAULT_TLDS+ results: list[dict[str, Any]] = []+ if not label:+ return {"username": username, "label": label, "results": results}++ hosts = [f"{label}.{tld}" for tld in tlds]+ resolved = dict(zip(hosts, await asyncio.gather(*(_resolves(h) for h in hosts)), strict=False))++ connector = build_connector(ScannerConfig(proxy=proxy), len(hosts))+ async with aiohttp.ClientSession(headers=DEFAULT_HEADERS, connector=connector) as session:+ live_flags = await asyncio.gather(+ *(+ _is_live(session, host, timeout) if resolved[host] else _false()+ for host in hosts+ )+ )++ for host, live in zip(hosts, live_flags, strict=False):+ results.append(+ {"domain": host, "registered": resolved[host], "live": bool(live)}+ )+ return {"username": username, "label": label, "results": results}+++async def _false() -> bool:+ return Falsediff --git a/src/aliens_eye/core/expand.py b/src/aliens_eye/core/expand.pynew file mode 100644index 0000000..9ec7034--- /dev/null+++ b/src/aliens_eye/core/expand.py@@ -0,0 +1,85 @@+"""Recursive expansion: mine linked usernames out of found profiles.++Given scan results, pull candidate usernames from each Found/Maybe profile's bio+(``@handle`` mentions) and external links that follow a known ``domain/<user>``+shape (github.com/<user>, twitter.com/<user>, ...). These feed another scan hop.+"""++from __future__ import annotations++import re+from typing import Any+from urllib.parse import urlparse++_HANDLE_RE = re.compile(r"@([A-Za-z0-9_.]{2,30})")+_URL_RE = re.compile(r"https?://[^\s\"'<>)]+", re.IGNORECASE)++# Domains where the first path segment is the username.+_USER_PATH_DOMAINS = {+ "github.com", "gitlab.com", "twitter.com", "x.com", "instagram.com",+ "t.me", "telegram.me", "medium.com", "dev.to", "reddit.com",+ "tiktok.com", "youtube.com", "twitch.tv", "keybase.io", "about.me",+ "codepen.io", "replit.com", "soundcloud.com", "pinterest.com",+}++# Path segments that are never usernames.+_PATH_STOPWORDS = {+ "in", "u", "user", "users", "profile", "channel", "c", "watch", "p",+ "about", "home", "login", "signup", "help", "settings", "explore",+}+++def _username_from_url(url: str) -> str | None:+ try:+ parsed = urlparse(url)+ except ValueError:+ return None+ host = parsed.netloc.lower()+ if host.startswith("www."):+ host = host[4:]+ if host not in _USER_PATH_DOMAINS:+ return None+ segments = [s for s in parsed.path.split("/") if s]+ if not segments:+ return None+ candidate = segments[0].lstrip("@")+ if candidate.lower() in _PATH_STOPWORDS:+ candidate = segments[1] if len(segments) > 1 else ""+ candidate = candidate.strip()+ if len(candidate) < 2 or "." in candidate and not candidate.replace(".", "").isalnum():+ return None+ return candidate if candidate else None+++def candidate_usernames_from_results(+ all_results: dict[str, list[dict[str, Any]]],+ exclude: set[str] | None = None,+ limit: int = 25,+) -> list[str]:+ """Collect deduped candidate usernames linked from Found/Maybe profiles."""+ exclude = {e.lower() for e in (exclude or set())}+ found: list[str] = []+ seen: set[str] = set()++ def _add(name: str) -> None:+ name = name.strip()+ low = name.lower()+ if len(name) < 2 or low in exclude or low in seen:+ return+ seen.add(low)+ found.append(name)++ for results in all_results.values():+ for item in results:+ if item.get("status") not in {"Found", "Maybe"}:+ continue+ profile = item.get("ai_analysis", {}).get("signals", {}).get("profile", {}) or {}+ bio = profile.get("bio", "") or ""+ for handle in _HANDLE_RE.findall(bio):+ _add(handle)+ for url in _URL_RE.findall(bio):+ uname = _username_from_url(url)+ if uname:+ _add(uname)++ return found[:limit]diff --git a/src/aliens_eye/core/exporter.py b/src/aliens_eye/core/exporter.pyindex 2bb98b1..38e752b 100644--- a/src/aliens_eye/core/exporter.py+++ b/src/aliens_eye/core/exporter.py@@ -1,6 +1,7 @@import csvimport jsonfrom datetime import datetime+from html import escape as _escfrom pathlib import Pathfrom typing import Any@@ -9,7 +10,18 @@ from rich.prompt import Promptfrom aliens_eye.utils.console import build_results_table, get_console-KNOWN_FORMATS = {"json", "csv", "html", "md", "markdown", "all"}+KNOWN_FORMATS = {+ "json", "csv", "html", "md", "markdown",+ "gexf", "maltego", "mermaid", "pdf", "all",+}++# Formats that need the correlation graph rather than the flat report.+GRAPH_FORMATS = {"gexf", "maltego", "mermaid"}+++def _md_cell(value: Any) -> str:+ """Sanitize a value for use inside a Markdown table cell."""+ return str(value or "").replace("|", "\\|").replace("\n", " ").replace("\r", " ").strip()class ResultsExporter:@@ -26,11 +38,17 @@ class ResultsExporter:level: str,all_results: dict[str, list[dict[str, Any]]],formats: list[str],+ correlation: dict[str, Any] | None = None,+ domains: dict[str, Any] | None = None,) -> list[Path]:timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")results_data = self._build_results_data(base_username, level, all_results, timestamp)+ if correlation is not None:+ results_data["correlation"] = correlation+ if domains is not None:+ results_data["domains"] = domainswritten: list[Path] = []stem = f"{base_username}_{level}_{timestamp}"@@ -55,8 +73,37 @@ class ResultsExporter:self._write_markdown(md_path, results_data)written.append(md_path)+ # Graph and PDF formats are opt-in only (not bundled into "all"), since+ # they are analysis artifacts rather than the standard document report.+ if "gexf" in formats:+ p = self.output_dir / f"{stem}.gexf"+ self._write_gexf(p, results_data)+ written.append(p)+ if "maltego" in formats:+ p = self.output_dir / f"{stem}_maltego.csv"+ self._write_maltego(p, results_data)+ written.append(p)+ if "mermaid" in formats:+ p = self.output_dir / f"{stem}.mmd"+ self._write_mermaid(p, results_data)+ written.append(p)+ if "pdf" in formats:+ p = self.output_dir / f"{stem}.pdf"+ if self._write_pdf(p, results_data):+ written.append(p)+return written+ def report_dict(+ self,+ base_username: str,+ level: str,+ all_results: dict[str, list[dict[str, Any]]],+ ) -> dict[str, Any]:+ """Public accessor for the structured report (used for --json-stdout)."""+ timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")+ return self._build_results_data(base_username, level, all_results, timestamp)+def _build_results_data(self,base_username: str,@@ -165,10 +212,24 @@ class ResultsExporter:rows = []for variation, data in variations.items():for site, info in data.get("sites", {}).items():+ profile = info.get("ai_analysis", {}).get("signals", {}).get("profile", {})+ is_hit = info.get("status") in {"Found", "Maybe"}+ avatar_url = profile.get("avatar", "") if is_hit else ""+ avatar_cell = (+ f"<img src='{_esc(avatar_url, quote=True)}' alt='' style='height:40px;border-radius:4px'>"+ if avatar_url+ else ""+ )+ name = _esc(profile.get("name", "")) if is_hit else ""+ bio = _esc(profile.get("bio", "")) if is_hit else ""+ url = info.get("url", "")rows.append(- f"<tr><td>{variation}</td><td>{site}</td><td>{info.get('status','')}</td>"- f"<td>{info.get('confidence',0)}</td><td>{info.get('code',0)}</td>"- f"<td><a href='{info.get('url','')}'>{info.get('url','')}</a></td></tr>"+ f"<tr><td>{_esc(variation)}</td><td>{_esc(site)}</td>"+ f"<td>{_esc(info.get('status',''))}</td>"+ f"<td>{_esc(str(info.get('confidence',0)))}</td>"+ f"<td>{_esc(str(info.get('code',0)))}</td>"+ f"<td><a href='{_esc(url, quote=True)}'>{_esc(url)}</a></td>"+ f"<td>{avatar_cell}</td><td>{name}</td><td>{bio}</td></tr>")html = f"""@@ -188,8 +249,8 @@ th {{ background-color: #f2f2f2; text-align: left; }}</head><body><h1>Aliens Eye Report</h1>-<p><strong>Base Username:</strong> {summary.get('base_username','')}</p>-<p><strong>Scan Level:</strong> {summary.get('scan_level','')}</p>+<p><strong>Base Username:</strong> {_esc(str(summary.get('base_username','')))}</p>+<p><strong>Scan Level:</strong> {_esc(str(summary.get('scan_level','')))}</p><p><strong>Total Variations:</strong> {summary.get('total_variations',0)}</p><p><strong>Total Sites Scanned:</strong> {summary.get('total_sites_scanned',0)}</p><p><strong>Total Found:</strong> {summary.get('total_found',0)}</p>@@ -204,6 +265,9 @@ th {{ background-color: #f2f2f2; text-align: left; }}<th>Confidence</th><th>HTTP Code</th><th>URL</th>+<th>Avatar</th>+<th>Name</th>+<th>Bio</th></tr></thead><tbody>@@ -242,8 +306,8 @@ th {{ background-color: #f2f2f2; text-align: left; }}f"Found: {info.get('found', 0)} | Maybe: {info.get('maybe', 0)} | "f"Not Found: {info.get('not_found', 0)} | Errors: {info.get('errors', 0)}","",- "| Site | Status | Confidence | HTTP | URL |",- "| --- | --- | --- | --- | --- |",+ "| Site | Status | Confidence | HTTP | Name | Bio | URL |",+ "| --- | --- | --- | --- | --- | --- | --- |",])sites = sorted(@@ -257,14 +321,114 @@ th {{ background-color: #f2f2f2; text-align: left; }}status = site_info.get("status", "")if status not in {"Found", "Maybe"}:continue+ profile = (+ site_info.get("ai_analysis", {})+ .get("signals", {})+ .get("profile", {})+ )+ name = _md_cell(profile.get("name", ""))+ bio = _md_cell(profile.get("bio", ""))lines.append(f"| {site} | {status} | {site_info.get('confidence', 0)}% | "- f"{site_info.get('code', 0)} | {site_info.get('url', '')} |"+ f"{site_info.get('code', 0)} | {name} | {bio} | {site_info.get('url', '')} |")lines.append("")path.write_text("\n".join(lines), encoding="utf-8")+ # --- graph exports ----------------------------------------------------++ @staticmethod+ def _graph_model(results_data: dict[str, Any]) -> dict[str, Any]:+ """Build (nodes, edges) from Found/Maybe accounts + correlation clusters."""+ nodes: dict[str, dict[str, Any]] = {}+ edges: list[dict[str, str]] = []+ variations = results_data.get("variations", {})+ for variation, block in variations.items():+ has_hit = False+ for site, info in (block.get("sites", {}) or {}).items():+ if info.get("status") not in {"Found", "Maybe"}:+ continue+ has_hit = True+ acct = f"{variation}:{site}"+ nodes[acct] = {"label": site, "type": "account", "url": info.get("url", "")}+ edges.append({"source": variation, "target": acct, "label": "has-account"})+ if has_hit:+ nodes[variation] = {"label": variation, "type": "username", "url": ""}++ for cluster in results_data.get("correlation", {}).get("clusters", []):+ members = [m["variation"] + ":" + m["site"] for m in cluster.get("members", [])]+ reason = ",".join(cluster.get("reasons", [])) or "same-person"+ for i in range(len(members)):+ for j in range(i + 1, len(members)):+ if members[i] in nodes and members[j] in nodes:+ edges.append({"source": members[i], "target": members[j], "label": reason})+ return {"nodes": nodes, "edges": edges}++ def _write_gexf(self, path: Path, results_data: dict[str, Any]) -> None:+ model = self._graph_model(results_data)+ ids = {name: str(i) for i, name in enumerate(model["nodes"])}+ node_xml = "".join(+ f'<node id="{ids[name]}" label="{_esc(data["label"], quote=True)}"/>'+ for name, data in model["nodes"].items()+ )+ edge_xml = "".join(+ f'<edge id="{i}" source="{ids[e["source"]]}" target="{ids[e["target"]]}" '+ f'label="{_esc(e["label"], quote=True)}"/>'+ for i, e in enumerate(model["edges"])+ if e["source"] in ids and e["target"] in ids+ )+ xml = (+ '<?xml version="1.0" encoding="UTF-8"?>\n'+ '<gexf xmlns="http://gexf.net/1.3" version="1.3">\n'+ '<graph defaultedgetype="undirected">\n'+ f"<nodes>{node_xml}</nodes>\n<edges>{edge_xml}</edges>\n"+ "</graph>\n</gexf>\n"+ )+ path.write_text(xml, encoding="utf-8")++ def _write_maltego(self, path: Path, results_data: dict[str, Any]) -> None:+ model = self._graph_model(results_data)+ with path.open("w", newline="", encoding="utf-8") as handle:+ writer = csv.writer(handle)+ writer.writerow(["SourceEntity", "SourceType", "TargetEntity", "TargetType", "Relationship"])+ for e in model["edges"]:+ src = model["nodes"].get(e["source"], {})+ tgt = model["nodes"].get(e["target"], {})+ if not src or not tgt:+ continue+ writer.writerow([+ e["source"], "maltego.Affiliation" if src.get("type") == "account" else "maltego.Alias",+ e["target"], "maltego.Affiliation" if tgt.get("type") == "account" else "maltego.Alias",+ e["label"],+ ])++ def _write_mermaid(self, path: Path, results_data: dict[str, Any]) -> None:+ model = self._graph_model(results_data)+ ids = {name: f"n{i}" for i, name in enumerate(model["nodes"])}+ lines = ["```mermaid", "graph TD"]+ for name, data in model["nodes"].items():+ label = data["label"].replace('"', "'")+ shape = f'(["{label}"])' if data["type"] == "username" else f'["{label}"]'+ lines.append(f" {ids[name]}{shape}")+ for e in model["edges"]:+ if e["source"] in ids and e["target"] in ids:+ lines.append(f" {ids[e['source']]} -- {e['label']} --- {ids[e['target']]}")+ lines.append("```")+ path.write_text("\n".join(lines) + "\n", encoding="utf-8")++ def _write_pdf(self, path: Path, results_data: dict[str, Any]) -> bool:+ """Investigator PDF report. Returns False (with a note) if reportlab is missing."""+ try:+ from aliens_eye.core.pdf_report import write_pdf+ except ImportError:+ self.console.print(+ "[yellow]PDF export needs reportlab: pip install \"aliens-eye\\[pdf]\"[/yellow]"+ )+ return False+ write_pdf(path, results_data)+ return True+def display_results_from_file(self, file_path: str) -> None:try:with open(file_path, encoding="utf-8") as handle:diff --git a/src/aliens_eye/core/pdf_report.py b/src/aliens_eye/core/pdf_report.pynew file mode 100644index 0000000..c8c7a0d--- /dev/null+++ b/src/aliens_eye/core/pdf_report.py@@ -0,0 +1,126 @@+"""Investigator-style PDF report (optional, requires the ``[pdf]`` extra: reportlab).++Renders the scan summary, per-variation Found/Maybe tables with embedded avatars,+and any correlation clusters. Importing this module raises ImportError if reportlab+is not installed; the exporter catches that and prints an install hint.+"""++from __future__ import annotations++import io+import urllib.request+from pathlib import Path+from typing import Any++from reportlab.lib import colors+from reportlab.lib.pagesizes import A4+from reportlab.lib.styles import getSampleStyleSheet+from reportlab.lib.units import mm+from reportlab.platypus import (+ Image,+ Paragraph,+ SimpleDocTemplate,+ Spacer,+ Table,+ TableStyle,+)++_MAX_AVATARS = 40+++def _fetch_avatar(url: str, timeout: float = 6.0) -> io.BytesIO | None:+ if not url:+ return None+ try:+ req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 aliens-eye"})+ with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 - user-provided profile URL+ if resp.status != 200:+ return None+ data = resp.read(2_000_000)+ return io.BytesIO(data)+ except Exception:+ return None+++def write_pdf(path: str | Path, results_data: dict[str, Any]) -> None:+ styles = getSampleStyleSheet()+ doc = SimpleDocTemplate(str(path), pagesize=A4, title="Aliens Eye Report")+ story: list[Any] = []++ summary = results_data.get("scan_summary", {})+ story.append(Paragraph("Aliens Eye — OSINT Report", styles["Title"]))+ story.append(Paragraph(f"Base username: <b>{summary.get('base_username', '')}</b>", styles["Normal"]))+ story.append(Paragraph(f"Scan level: {summary.get('scan_level', '')}", styles["Normal"]))+ story.append(Paragraph(f"Timestamp: {summary.get('timestamp', '')}", styles["Normal"]))+ story.append(Paragraph(+ f"Found: {summary.get('total_found', 0)} | "+ f"High confidence: {summary.get('total_high_confidence', 0)} | "+ f"Sites scanned: {summary.get('total_sites_scanned', 0)}",+ styles["Normal"],+ ))+ story.append(Spacer(1, 6 * mm))++ avatars_used = 0+ for variation, block in results_data.get("variations", {}).items():+ hits = [+ (site, info)+ for site, info in (block.get("sites", {}) or {}).items()+ if info.get("status") in {"Found", "Maybe"}+ ]+ if not hits:+ continue+ hits.sort(key=lambda kv: (kv[1].get("status") != "Found", -kv[1].get("confidence", 0)))+ story.append(Paragraph(f"@{variation}", styles["Heading2"]))++ data = [["", "Site", "Status", "Conf", "Name / Bio", "URL"]]+ for site, info in hits:+ profile = info.get("ai_analysis", {}).get("signals", {}).get("profile", {}) or {}+ img_cell: Any = ""+ if avatars_used < _MAX_AVATARS and profile.get("avatar"):+ buf = _fetch_avatar(profile["avatar"])+ if buf is not None:+ try:+ img_cell = Image(buf, width=12 * mm, height=12 * mm)+ avatars_used += 1+ except Exception:+ img_cell = ""+ name = profile.get("name", "")+ bio = (profile.get("bio", "") or "")[:120]+ name_bio = Paragraph(f"<b>{_x(name)}</b><br/>{_x(bio)}", styles["BodyText"])+ url = Paragraph(_x(info.get("url", "")), styles["BodyText"])+ data.append([+ img_cell, site, info.get("status", ""),+ f"{info.get('confidence', 0)}%", name_bio, url,+ ])++ table = Table(data, colWidths=[14 * mm, 26 * mm, 16 * mm, 12 * mm, 52 * mm, 50 * mm])+ table.setStyle(TableStyle([+ ("BACKGROUND", (0, 0), (-1, 0), colors.HexColor("#222222")),+ ("TEXTCOLOR", (0, 0), (-1, 0), colors.white),+ ("FONTSIZE", (0, 0), (-1, -1), 7),+ ("VALIGN", (0, 0), (-1, -1), "MIDDLE"),+ ("GRID", (0, 0), (-1, -1), 0.25, colors.grey),+ ("ROWBACKGROUNDS", (0, 1), (-1, -1), [colors.white, colors.HexColor("#f4f4f4")]),+ ]))+ story.append(table)+ story.append(Spacer(1, 4 * mm))++ clusters = results_data.get("correlation", {}).get("clusters", [])+ if clusters:+ story.append(Paragraph("Correlation — likely same person", styles["Heading2"]))+ for i, cluster in enumerate(clusters, 1):+ sites = ", ".join(m["site"] for m in cluster["members"])+ reasons = ", ".join(cluster.get("reasons", []))+ story.append(Paragraph(f"<b>Cluster {i}</b> ({reasons}): {_x(sites)}", styles["BodyText"]))++ doc.build(story)+++def _x(text: Any) -> str:+ """Escape the small subset of markup reportlab Paragraph treats as XML."""+ return (+ str(text or "")+ .replace("&", "&")+ .replace("<", "<")+ .replace(">", ">")+ )diff --git a/src/aliens_eye/core/report.py b/src/aliens_eye/core/report.pynew file mode 100644index 0000000..ee0a536--- /dev/null+++ b/src/aliens_eye/core/report.py@@ -0,0 +1,83 @@+"""Shared helpers for loading and comparing saved scan reports.++A report is the JSON structure written by :class:`ResultsExporter`+(``{"scan_summary": ..., "variations": {v: {"sites": {s: {...}}}}}``). These+helpers are the single place that knows that shape, reused by watch mode, the+``diff`` command, and anything else that consumes prior results from disk.+"""++from __future__ import annotations++import json+from pathlib import Path+from typing import Any+++class ReportError(ValueError):+ """Raised when a file is not a recognizable Aliens Eye report."""+++def load_report(path: str | Path) -> dict[str, Any]:+ """Load and validate a saved report JSON. Raises ReportError on bad input."""+ p = Path(path)+ try:+ data = json.loads(p.read_text(encoding="utf-8"))+ except FileNotFoundError as exc:+ raise ReportError(f"File not found: {p}") from exc+ except (OSError, json.JSONDecodeError) as exc:+ raise ReportError(f"Invalid JSON file: {p}") from exc+ if not isinstance(data, dict) or "variations" not in data:+ raise ReportError(f"Unrecognized results file: {p}")+ return data+++def iter_sites(report: dict[str, Any]):+ """Yield ``(variation, site, site_info)`` triples from a report dict."""+ for variation, block in report.get("variations", {}).items():+ for site, info in (block.get("sites", {}) or {}).items():+ yield variation, site, info+++def found_keys(report: dict[str, Any]) -> set[str]:+ """Set of ``variation:site`` keys with status ``Found`` in a report."""+ return {+ f"{variation}:{site}"+ for variation, site, info in iter_sites(report)+ if info.get("status") == "Found"+ }+++def status_map(report: dict[str, Any]) -> dict[str, dict[str, Any]]:+ """Map ``variation:site`` -> ``{status, confidence, url}`` for every site."""+ result: dict[str, dict[str, Any]] = {}+ for variation, site, info in iter_sites(report):+ result[f"{variation}:{site}"] = {+ "status": info.get("status", ""),+ "confidence": info.get("confidence", 0),+ "url": info.get("url", ""),+ }+ return result+++def diff_reports(old: dict[str, Any], new: dict[str, Any]) -> dict[str, list[dict[str, Any]]]:+ """Compare two reports.++ Returns ``new`` / ``gone`` (Found membership changes) and ``changed``+ (status or confidence moved for a key present in both).+ """+ old_found = found_keys(old)+ new_found = found_keys(new)+ old_map = status_map(old)+ new_map = status_map(new)++ changed: list[dict[str, Any]] = []+ for key in sorted(set(old_map) & set(new_map)):+ before, after = old_map[key], new_map[key]+ if before["status"] != after["status"] or before["confidence"] != after["confidence"]:+ changed.append({"key": key, "before": before, "after": after})++ return {+ "new": [{"key": k, "url": new_map.get(k, {}).get("url", "")} for k in sorted(new_found - old_found)],+ "gone": [{"key": k, "url": old_map.get(k, {}).get("url", "")} for k in sorted(old_found - new_found)],+ "changed": changed,+ }diff --git a/src/aliens_eye/core/scanner.py b/src/aliens_eye/core/scanner.pyindex 8cb32d4..ee98ba8 100644--- a/src/aliens_eye/core/scanner.py+++ b/src/aliens_eye/core/scanner.py@@ -41,6 +41,31 @@ def load_sites_data(path: Path | None = None) -> dict[str, str]:return json.loads(text)+def load_site_plugins(extra_dirs: list[Path] | None = None) -> dict[str, str]:+ """Merge every ``*.json`` site map found in the plugin directories.++ Looks in ``./sites.d/`` and the user config dir's ``sites.d/`` by default,+ plus any ``extra_dirs``. Each file must be a ``{site_name: url_template}``+ map; later files override earlier ones on name conflicts.+ """+ from platformdirs import user_config_dir++ dirs = [Path("sites.d"), Path(user_config_dir("aliens_eye")) / "sites.d"]+ dirs.extend(extra_dirs or [])+ merged: dict[str, str] = {}+ for directory in dirs:+ if not directory.is_dir():+ continue+ for path in sorted(directory.glob("*.json")):+ try:+ data = json.loads(path.read_text(encoding="utf-8"))+ except (OSError, json.JSONDecodeError):+ continue+ if isinstance(data, dict):+ merged.update({str(k): str(v) for k, v in data.items()})+ return merged++def load_nsfw_sites() -> list[str]:"""Names of sites flagged as NSFW, shipped as package data."""try:@@ -95,6 +120,7 @@ class UsernameScanner:fingerprints: FingerprintStore,logger,browser_fallback: BrowserFallback | None = None,+ checkpoint=None,) -> None:self.sites_data = sites_dataself.config = config@@ -103,6 +129,7 @@ class UsernameScanner:self.fingerprints = fingerprintsself.logger = loggerself.browser_fallback = browser_fallback+ self.checkpoint = checkpointself.console = get_console()self.results_dir = config.output_dir@@ -134,14 +161,27 @@ class UsernameScanner:if not self.sites_data:return results- conn_limit = max(1, min(self.config.concurrent, len(self.sites_data)))+ done: dict[str, dict[str, Any]] = (+ self.checkpoint.results_for(username) if self.checkpoint else {}+ )+ pending = {site: tmpl for site, tmpl in self.sites_data.items() if site not in done}+ results.extend(done.values())+ if done:+ self.console.print(+ f"[dim]Resuming {username}: {len(done)} site(s) from checkpoint, "+ f"{len(pending)} remaining.[/dim]"+ )+ if not pending:+ return results++ conn_limit = max(1, min(self.config.concurrent, len(pending)))rate_limiter = DomainRateLimiter()queue: asyncio.Queue[tuple[str, str]] = asyncio.Queue()- for site, tmpl in self.sites_data.items():+ for site, tmpl in pending.items():queue.put_nowait((site, tmpl))view = ScanView(self.console)- view.start(username, len(self.sites_data))+ view.start(username, len(pending))connector = build_connector(self.config, conn_limit)start_time = time.monotonic()@@ -189,22 +229,25 @@ class UsernameScanner:except Exception as exc:url = self._format_url(site, tmpl, username)self.logger.debug("Worker failed for %s: %s", site, exc)- results.append(- {- "site": site,- "url": url,- "final_url": url,- "status": "Error",- "code": 0,- "response_time": 0.0,- "confidence": 0,- "ai_analysis": {"error": str(exc)},- }- )- finally:- found = sum(1 for r in results if r["status"] == "Found")- view.advance(found)- queue.task_done()+ result = {+ "site": site,+ "url": url,+ "final_url": url,+ "status": "Error",+ "code": 0,+ "response_time": 0.0,+ "confidence": 0,+ "ai_analysis": {"error": str(exc)},+ }+ results.append(result)+ if self.checkpoint is not None:+ try:+ await self.checkpoint.record(username, site, result)+ except Exception as exc: # noqa: BLE001 - checkpoint is best-effort+ self.logger.debug("Checkpoint write failed for %s: %s", site, exc)+ found = sum(1 for r in results if r["status"] == "Found")+ view.advance(found)+ queue.task_done()async def _scan_site(self,diff --git a/src/aliens_eye/core/variations.py b/src/aliens_eye/core/variations.pyindex 71b18f1..3d1cb47 100644--- a/src/aliens_eye/core/variations.py+++ b/src/aliens_eye/core/variations.py@@ -1,3 +1,45 @@+import re+++def usernames_from_email(email: str) -> list[str]:+ """Derive candidate usernames from an email address.++ Uses the local part plus common separator splits, e.g.+ ``[email protected]`` -> ``john.doe``, ``johndoe``, ``john_doe``, ``john``.+ """+ local = str(email).strip().split("@", 1)[0].lower()+ if not local:+ return []+ parts = [p for p in re.split(r"[._\-+]", local) if p]+ candidates = [local]+ if len(parts) > 1:+ joined = "".join(parts)+ candidates.extend([joined, "_".join(parts), ".".join(parts), parts[0]])+ return list(dict.fromkeys(c for c in candidates if len(c) >= 2))+++def usernames_from_name(name: str) -> list[str]:+ """Derive candidate usernames from a real name.++ ``John Doe`` -> ``john``, ``doe``, ``johndoe``, ``john.doe``,+ ``john_doe``, ``jdoe``, ``doej``.+ """+ parts = [p for p in re.split(r"\s+", str(name).strip().lower()) if p.isalnum()]+ if not parts:+ return []+ if len(parts) == 1:+ return [parts[0]] if len(parts[0]) >= 2 else []+ first, last = parts[0], parts[-1]+ candidates = [+ first,+ last,+ f"{first}{last}",+ f"{first}.{last}",+ f"{first}_{last}",+ f"{first[0]}{last}",+ f"{last}{first[0]}",+ ]+ return list(dict.fromkeys(c for c in candidates if len(c) >= 2))def generate_username_variations(username: str, level: str) -> list[str]:diff --git a/src/aliens_eye/core/watch.py b/src/aliens_eye/core/watch.pynew file mode 100644index 0000000..e82b971--- /dev/null+++ b/src/aliens_eye/core/watch.py@@ -0,0 +1,127 @@+"""Watch / monitor mode: re-scan on an interval and alert on changes.++Reuses the existing scanner and exporter. Each cycle runs a full scan, saves a+timestamped report, and diffs the set of Found accounts against the previous+cycle (seeded from the most recent report on disk, if any).+"""++from __future__ import annotations++import asyncio+from pathlib import Path+from typing import Any++import aiohttp++from .report import ReportError, found_keys, load_report++_UNITS = {"s": 1.0, "m": 60.0, "h": 3600.0, "d": 86400.0}+++def parse_duration(value: str) -> float:+ """Parse a duration like ``30s``/``15m``/``6h``/``1d`` into seconds.++ A bare number is treated as seconds. Raises ValueError on bad input.+ """+ text = str(value).strip().lower()+ if not text:+ raise ValueError("empty duration")+ unit = text[-1]+ if unit in _UNITS:+ number = text[:-1]+ factor = _UNITS[unit]+ else:+ number = text+ factor = 1.0+ seconds = float(number) * factor+ if seconds <= 0:+ raise ValueError(f"duration must be positive: {value!r}")+ return seconds+++def found_set(all_results: dict[str, list[dict[str, Any]]]) -> set[str]:+ """Set of ``variation:site`` keys currently detected as Found."""+ keys: set[str] = set()+ for variation, results in all_results.items():+ for item in results:+ if item.get("status") == "Found":+ keys.add(f"{variation}:{item.get('site', '')}")+ return keys+++def diff_found(prev: set[str], curr: set[str]) -> dict[str, list[str]]:+ """New and disappeared Found accounts between two cycles."""+ return {+ "new": sorted(curr - prev),+ "gone": sorted(prev - curr),+ }+++def _baseline(output_dir: Path, username: str, level: str) -> set[str]:+ reports = sorted(output_dir.glob(f"{username}_{level}_*.json"))+ if not reports:+ return set()+ try:+ return found_keys(load_report(reports[-1]))+ except ReportError:+ return set()+++async def _notify(webhook_url: str, payload: dict[str, Any], logger) -> None:+ try:+ async with aiohttp.ClientSession() as session:+ await session.post(webhook_url, json=payload, timeout=15)+ except Exception as exc: # noqa: BLE001 - notification is best-effort+ logger.warning("Webhook notify failed: %s", exc)+++async def run_watch(+ scanner,+ exporter,+ usernames: list[str],+ scan_level: str,+ formats: list[str],+ interval: float,+ logger,+ console,+ output_dir: Path,+ notify_url: str | None = None,+) -> None:+ """Re-scan the given usernames every ``interval`` seconds until interrupted."""+ previous: dict[str, set[str]] = {+ username: _baseline(output_dir, username, scan_level) for username in usernames+ }+ console.print(+ f"[blue]Watch mode:[/blue] re-scanning every [yellow]{interval:.0f}s[/yellow] "+ f"(Ctrl-C to stop)"+ )++ cycle = 0+ while True:+ cycle += 1+ console.print(f"\n[dim]--- watch cycle {cycle} ---[/dim]")+ for username in usernames:+ if not username or len(username) < 2:+ continue+ all_results = await scanner.scan_with_variations(username, scan_level)+ exporter.save_results(username, scan_level, all_results, formats)++ current = found_set(all_results)+ delta = diff_found(previous.get(username, set()), current)+ previous[username] = current++ if delta["new"] or delta["gone"]:+ for key in delta["new"]:+ console.print(f"[bold green]+ NEW:[/bold green] {key}")+ for key in delta["gone"]:+ console.print(f"[bold red]- GONE:[/bold red] {key}")+ if notify_url:+ await _notify(+ notify_url,+ {"username": username, "level": scan_level, **delta},+ logger,+ )+ else:+ console.print(f"[dim]No change for {username} ({len(current)} found).[/dim]")++ await asyncio.sleep(interval)diff --git a/src/aliens_eye/data/keywords.json b/src/aliens_eye/data/keywords.jsonnew file mode 100644index 0000000..2b3271d--- /dev/null+++ b/src/aliens_eye/data/keywords.json@@ -0,0 +1,232 @@+{+ "error": {+ "en": [+ "not found",+ "doesn't exist",+ "didn't find",+ "does not exist",+ "something went wrong",+ "no such user",+ "user not found",+ "cannot find",+ "can't find",+ "not exist",+ "profile not found",+ "account does not exist",+ "username not found",+ "no user found",+ "no results found",+ "no such username",+ "isn't available",+ "that content is unavailable",+ "page not found",+ "404",+ "404 error",+ "404 not found",+ "error",+ "sorry",+ "oops",+ "unavailable",+ "account suspended",+ "invalid username",+ "account not found",+ "account terminated",+ "account disabled",+ "user doesn't have an account",+ "this account doesn't exist",+ "page was not found"+ ],+ "es": [+ "no encontrado",+ "no encontrada",+ "no existe",+ "no se encontró",+ "usuario no encontrado",+ "página no encontrada",+ "no disponible",+ "cuenta suspendida",+ "nombre de usuario no válido"+ ],+ "fr": [+ "introuvable",+ "n'existe pas",+ "page introuvable",+ "utilisateur introuvable",+ "non trouvé",+ "compte suspendu",+ "indisponible"+ ],+ "de": [+ "nicht gefunden",+ "existiert nicht",+ "seite nicht gefunden",+ "benutzer nicht gefunden",+ "nicht verfügbar",+ "konto gesperrt"+ ],+ "pt": [+ "não encontrado",+ "não encontrada",+ "não existe",+ "página não encontrada",+ "usuário não encontrado",+ "indisponível",+ "conta suspensa"+ ],+ "ru": [+ "не найдено",+ "не найден",+ "не существует",+ "страница не найдена",+ "пользователь не найден",+ "недоступно"+ ],+ "ja": [+ "見つかりません",+ "存在しません",+ "ページが見つかりません",+ "ユーザーが見つかりません"+ ],+ "zh": [+ "未找到",+ "不存在",+ "页面不存在",+ "用户不存在",+ "找不到"+ ],+ "ar": [+ "غير موجود",+ "لم يتم العثور",+ "الصفحة غير موجودة",+ "المستخدم غير موجود"+ ],+ "hi": [+ "नहीं मिला",+ "मौजूद नहीं",+ "पेज नहीं मिला",+ "उपयोगकर्ता नहीं मिला"+ ]+ },+ "positive": {+ "en": [+ "follow",+ "subscribe",+ "like",+ "share",+ "following",+ "followers",+ "profile",+ "user",+ "posts",+ "photos",+ "bio",+ "status",+ "tweets",+ "joined",+ "member since",+ "online",+ "verified",+ "active",+ "comments",+ "uploads",+ "reviews",+ "friends",+ "connections",+ "activity",+ "timeline"+ ],+ "es": [+ "seguidores",+ "siguiendo",+ "perfil",+ "publicaciones",+ "miembro desde",+ "verificado"+ ],+ "fr": [+ "abonnés",+ "abonnements",+ "profil",+ "publications",+ "membre depuis",+ "vérifié"+ ],+ "de": [+ "abonnenten",+ "profil",+ "beiträge",+ "mitglied seit",+ "verifiziert"+ ],+ "pt": [+ "seguidores",+ "seguindo",+ "perfil",+ "publicações",+ "membro desde",+ "verificado"+ ],+ "ru": [+ "подписчики",+ "подписки",+ "профиль",+ "публикации",+ "участник с"+ ],+ "ja": [+ "フォロワー",+ "フォロー",+ "プロフィール",+ "投稿"+ ],+ "zh": [+ "粉丝",+ "关注",+ "个人资料",+ "帖子",+ "已认证"+ ],+ "ar": [+ "متابعون",+ "الملف الشخصي",+ "منشورات"+ ],+ "hi": [+ "फ़ॉलोअर्स",+ "प्रोफ़ाइल",+ "पोस्ट"+ ]+ },+ "meta": {+ "en": [+ "profile picture",+ "profile image",+ "avatar",+ "user page",+ "username",+ "user profile",+ "account info",+ "account information",+ "user information"+ ],+ "es": [+ "foto de perfil",+ "imagen de perfil",+ "página de usuario"+ ],+ "fr": [+ "photo de profil",+ "image de profil",+ "page utilisateur"+ ],+ "de": [+ "profilbild",+ "benutzerseite"+ ],+ "pt": [+ "foto de perfil",+ "imagem de perfil",+ "página do usuário"+ ]+ }+}diff --git a/src/aliens_eye/data/site_profiles.json b/src/aliens_eye/data/site_profiles.jsonnew file mode 100644index 0000000..6cba59d--- /dev/null+++ b/src/aliens_eye/data/site_profiles.json@@ -0,0 +1,63 @@+{+ "_comment": "site_name -> {name: <css>, bio: <css>, avatar: <css>}. CSS overrides the generic OpenGraph/JSON-LD extraction; an override that matches nothing is ignored (the generic value is kept). Only add selectors for server-rendered profile pages.",+ "github": {+ "name": "span.p-name",+ "bio": "div.user-profile-bio",+ "avatar": "img.avatar-user"+ },+ "gitlab": {+ "name": "div.user-profile .cover-title",+ "bio": "div.user-profile .cover-desc",+ "avatar": "div.avatar-holder img.avatar"+ },+ "dev.to": {+ "name": "h1.crayons-title",+ "bio": "div.profile-header__details .crayons-subtitle-3",+ "avatar": "img.profile-header__avatar"+ },+ "keybase": {+ "name": "span.full-name",+ "bio": "p.bio",+ "avatar": "img.avatar"+ },+ "gravatar": {+ "name": "h1.profile-name",+ "bio": "p.profile-description",+ "avatar": "img.avatar"+ },+ "about.me": {+ "name": "h1",+ "bio": "div.bio",+ "avatar": "img.avatar"+ },+ "codepen": {+ "name": "h1.profile-name",+ "bio": "div.profile-bio",+ "avatar": "img.profile-avatar"+ },+ "steam": {+ "name": "span.actual_persona_name",+ "bio": "div.profile_summary",+ "avatar": "div.playerAvatarAutoSizeInner img"+ },+ "wordpress": {+ "name": "h2.author-name",+ "bio": "p.author-description",+ "avatar": "img.avatar"+ },+ "telegram": {+ "name": "div.tgme_page_title span",+ "bio": "div.tgme_page_description",+ "avatar": "img.tgme_page_photo_image"+ },+ "lobsters": {+ "name": "span.user_tag",+ "bio": "div.about",+ "avatar": "img.avatar"+ },+ "replit": {+ "name": "h1",+ "bio": "div.bio",+ "avatar": "img.profile-image"+ }+}diff --git a/src/aliens_eye/mcp_server.py b/src/aliens_eye/mcp_server.pynew file mode 100644index 0000000..103397a--- /dev/null+++ b/src/aliens_eye/mcp_server.py@@ -0,0 +1,90 @@+"""MCP server exposing Aliens Eye as tools for LLM agents (optional ``[serve]`` extra: mcp).++Runs over stdio and exposes:+- ``scan_username`` — scan a username across all sites, return the structured report+- ``correlate`` — run cross-site correlation over a report and return clusters+- ``read_report`` — load a previously saved report JSON from disk++Start it with ``aliens_eye serve`` and point an MCP client (e.g. Claude) at the+command.+"""++from __future__ import annotations++from pathlib import Path+from typing import Any+++async def _run_scan(username: str, level: str, sites: str | None, no_ml: bool) -> dict[str, Any]:+ from aliens_eye.core.analyzer import FeatureExtractor+ from aliens_eye.core.config import ScannerConfig+ from aliens_eye.core.detector import Detector+ from aliens_eye.core.exporter import ResultsExporter+ from aliens_eye.core.fingerprints import FingerprintStore+ from aliens_eye.core.scanner import UsernameScanner, load_sites_data+ from aliens_eye.utils.console import set_plain+ from aliens_eye.utils.logger import setup_logger++ set_plain(True, stderr=True)+ logger = setup_logger(False)+ config = ScannerConfig()+ sites_data = load_sites_data(Path(sites) if sites else None)+ detector = Detector()+ if not no_ml:+ detector.load_model(logger)+ extractor = FeatureExtractor()+ fingerprints = FingerprintStore(config.fingerprints_path, config.max_fingerprints_per_label)+ fingerprints.load(logger)+ scanner = UsernameScanner(+ sites_data=sites_data, config=config, extractor=extractor,+ detector=detector, fingerprints=fingerprints, logger=logger,+ )+ try:+ all_results = await scanner.scan_with_variations(username, level)+ finally:+ fingerprints.save()+ return ResultsExporter(config.output_dir).report_dict(username, level, all_results)+++async def serve(args) -> None:+ try:+ from mcp.server.fastmcp import FastMCP+ except ImportError:+ from aliens_eye.utils.console import get_console++ get_console().print(+ "[yellow]The MCP server needs the mcp package: pip install \"aliens-eye\\[serve]\"[/yellow]"+ )+ return++ sites = getattr(args, "sites", None)+ no_ml = getattr(args, "no_ml", False)+ mcp = FastMCP("aliens-eye")++ @mcp.tool()+ async def scan_username(username: str, level: str = "basic") -> dict[str, Any]:+ """Scan a username across all known sites. level: basic|intermediate|advanced."""+ if not username or len(username) < 2:+ return {"error": "username must be at least 2 characters"}+ if level not in {"basic", "intermediate", "advanced"}:+ level = "basic"+ return await _run_scan(username, level, sites, no_ml)++ @mcp.tool()+ async def correlate(report: dict[str, Any]) -> dict[str, Any]:+ """Cluster the profiles in a scan report that look like the same person."""+ from aliens_eye.core.correlate import correlate_report++ return await correlate_report(report)++ @mcp.tool()+ def read_report(path: str) -> dict[str, Any]:+ """Load a previously saved Aliens Eye report JSON from disk."""+ from aliens_eye.core.report import ReportError, load_report++ try:+ return load_report(path)+ except ReportError as exc:+ return {"error": str(exc)}++ await mcp.run_stdio_async()diff --git a/src/aliens_eye/ml/collect.py b/src/aliens_eye/ml/collect.pyindex 64ddf79..cec2feb 100644--- a/src/aliens_eye/ml/collect.py+++ b/src/aliens_eye/ml/collect.py@@ -36,6 +36,19 @@ def load_selfcheck_data() -> dict[str, list[str]]:}+def write_rows(output_path: Path, rows: list[list[float]], append: bool = False) -> None:+ """Write labeled rows to a CSV. Writes the header only for a new/overwritten file."""+ output_path = Path(output_path)+ output_path.parent.mkdir(parents=True, exist_ok=True)+ write_header = not (append and output_path.exists())+ mode = "a" if append else "w"+ with output_path.open(mode, encoding="utf-8", newline="") as handle:+ writer = csv.writer(handle)+ if write_header:+ writer.writerow(FEATURE_SCHEMA + ["label"])+ writer.writerows(rows)++def random_username(rng: random.Random) -> str:length = rng.randint(14, 20)return "".join(rng.choices(string.ascii_lowercase + string.digits, k=length))@@ -121,12 +134,6 @@ async def collect_dataset(await asyncio.gather(*(run_job(job) for job in jobs))- output_path = Path(output_path)- output_path.parent.mkdir(parents=True, exist_ok=True)- with output_path.open("w", encoding="utf-8", newline="") as handle:- writer = csv.writer(handle)- writer.writerow(FEATURE_SCHEMA + ["label"])- writer.writerows(rows)-+ write_rows(output_path, rows, append=False)logger.info("Wrote %d labeled samples to %s", len(rows), output_path)return len(rows)diff --git a/src/aliens_eye/ml/label.py b/src/aliens_eye/ml/label.pynew file mode 100644index 0000000..e84dfec--- /dev/null+++ b/src/aliens_eye/ml/label.py@@ -0,0 +1,75 @@+"""Active-learning loop: hand-label uncertain results and grow the dataset.++Reads a saved report, walks the low-confidence band (Maybe by default), and asks+you to confirm each hit. Confirmed labels are vectorized from the features already+stored in the report and appended to a training CSV, so a later ``train fit`` can+learn from your corrections.+"""++from __future__ import annotations++from aliens_eye.core.features import vectorize_features+from aliens_eye.core.report import ReportError, iter_sites, load_report+from aliens_eye.ml.collect import write_rows+from aliens_eye.utils.console import get_console+from aliens_eye.utils.logger import setup_logger+++async def label_report(args) -> None:+ from pathlib import Path++ from rich.prompt import Prompt++ console = get_console()+ logger = setup_logger(getattr(args, "verbose", False))+ try:+ report = load_report(args.report)+ except ReportError as exc:+ console.print(f"[red]{exc}[/red]")+ return++ band = (args.band or "Maybe").lower()+ statuses = {"found", "maybe", "not found"} if band == "all" else {"maybe"}++ candidates = [+ (variation, site, info)+ for variation, site, info in iter_sites(report)+ if info.get("status", "").lower() in statuses+ ]+ if not candidates:+ console.print(f"[yellow]No '{band}' results to label in {args.report}.[/yellow]")+ return++ console.print(+ f"[blue]Labeling {len(candidates)} result(s).[/blue] "+ "[dim]y = exists, n = does not exist, s = skip, q = quit[/dim]\n"+ )++ rows: list[list[float]] = []+ for variation, site, info in candidates:+ features = info.get("ai_analysis", {}).get("features")+ if not isinstance(features, dict) or not features:+ continue+ profile = info.get("ai_analysis", {}).get("signals", {}).get("profile", {})+ name = profile.get("name", "")+ console.print(+ f"[yellow]{site}[/yellow] / [bold]{variation}[/bold] "+ f"status={info.get('status')} conf={info.get('confidence', 0)}% "+ f"{info.get('url', '')}"+ + (f"\n [dim]name: {name}[/dim]" if name else "")+ )+ choice = Prompt.ask(" label", choices=["y", "n", "s", "q"], default="s", console=console)+ if choice == "q":+ break+ if choice == "s":+ continue+ label = 1 if choice == "y" else 0+ rows.append(vectorize_features(features) + [float(label)])++ if not rows:+ console.print("[dim]No labels recorded.[/dim]")+ return++ write_rows(Path(args.out), rows, append=True)+ console.print(f"[green]Appended {len(rows)} labeled sample(s) -> {args.out}[/green]")+ logger.info("Appended %d samples to %s", len(rows), args.out)diff --git a/src/aliens_eye/selfcheck.py b/src/aliens_eye/selfcheck.pyindex 777d981..fd7b537 100644--- a/src/aliens_eye/selfcheck.py+++ b/src/aliens_eye/selfcheck.py@@ -1,7 +1,8 @@-"""Validate detection accuracy against accounts known to exist.+"""Validate detection accuracy against accounts known to exist (and to not exist).-Scans each (site, username) pair from data/selfcheck.json and checks that the-detector reports "Found". Doubles as a rot detector for sites.json and as a+Scans each known ``(site, username)`` positive from data/selfcheck.json plus a few+random negatives per site, then reports precision / recall / F1 / false-positive+rate overall and per site. Doubles as a rot detector for sites.json and acalibration baseline for the ML model."""@@ -9,7 +10,10 @@ from __future__ import annotationsimport asyncioimport json+import random+import stringfrom importlib import resources+from typing import Anyimport aiohttpfrom rich.panel import Panel@@ -35,45 +39,74 @@ def load_selfcheck_data() -> dict[str, str]:}+def _random_username(rng: random.Random) -> str:+ length = rng.randint(14, 20)+ return "".join(rng.choices(string.ascii_lowercase + string.digits, k=length))+++def _metrics(tp: int, fp: int, fn: int, tn: int) -> dict[str, float]:+ precision = tp / (tp + fp) if (tp + fp) else 0.0+ recall = tp / (tp + fn) if (tp + fn) else 0.0+ f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0+ fpr = fp / (fp + tn) if (fp + tn) else 0.0+ return {+ "precision": round(precision, 4),+ "recall": round(recall, 4),+ "f1": round(f1, 4),+ "false_positive_rate": round(fpr, 4),+ "tp": tp, "fp": fp, "fn": fn, "tn": tn,+ }++async def run_selfcheck(sites_data: dict[str, str],detector: Detector,config: ScannerConfig,logger,-) -> float:- """Run the self-check and return accuracy in [0, 1]."""+ negatives: int = 1,+ report_format: str | None = None,+ seed: int | None = 1234,+) -> dict[str, Any]:+ """Run the self-check. Returns a metrics dict; renders a report unless report_format='json'."""console = get_console()selfcheck = load_selfcheck_data()extractor = FeatureExtractor()rate_limiter = DomainRateLimiter()+ rng = random.Random(seed)++ # Build (site, username, expected_label) jobs. label 1 = should be Found.+ jobs: list[tuple[str, str, str, int]] = []+ for site, username in selfcheck.items():+ if site not in sites_data:+ continue+ jobs.append((site, sites_data[site], username, 1))+ for _ in range(max(0, negatives)):+ jobs.append((site, sites_data[site], _random_username(rng), 0))- checks = [- (site, sites_data[site], username)- for site, username in selfcheck.items()- if site in sites_data- ]skipped = [site for site in selfcheck if site not in sites_data]- if skipped:+ if skipped and report_format != "json":console.print(f"[yellow]Skipping sites missing from sites.json: {', '.join(skipped)}[/yellow]")- if not checks:- console.print("[red]No self-check sites available.[/red]")- return 0.0+ if not jobs:+ if report_format != "json":+ console.print("[red]No self-check sites available.[/red]")+ return {"overall": _metrics(0, 0, 0, 0), "per_site": {}, "errors": 0}rows: list[dict] = []- semaphore = asyncio.Semaphore(min(10, len(checks)))-+ semaphore = asyncio.Semaphore(min(10, len(jobs)))connector = aiohttp.TCPConnector(limit=10)async with aiohttp.ClientSession(headers=DEFAULT_HEADERS, connector=connector) as session:- async def check(site: str, template: str, username: str) -> None:- url = template.format(username)+ async def check(site: str, template: str, username: str, label: int) -> None:+ try:+ url = template.format(username)+ except Exception:+ url = template.replace("{}", username)async with semaphore:fetch = await fetch_url(session, url, config, rate_limiter, logger)if fetch.error:- rows.append(- {"site": site, "username": username, "status": "Error",- "confidence": 0, "ok": False, "note": fetch.error[:60]}- )+ rows.append({"site": site, "username": username, "label": label,+ "status": "Error", "confidence": 0, "predicted": None,+ "note": fetch.error[:60]})returnbundle = extractor.extract(fetch.content, fetch.final_url, username, site,@@ -82,42 +115,96 @@ async def run_selfcheck(bundle.features["fingerprint_match_found"] = 0.0bundle.features["fingerprint_match_not_found"] = 0.0detection = detector.predict(bundle.features)- rows.append(- {"site": site, "username": username, "status": detection.status,- "confidence": detection.confidence, "ok": detection.status == "Found",- "note": detection.method}- )+ rows.append({"site": site, "username": username, "label": label,+ "status": detection.status, "confidence": detection.confidence,+ "predicted": 1 if detection.status == "Found" else 0,+ "note": detection.method})- await asyncio.gather(*(check(*c) for c in checks))+ await asyncio.gather(*(check(*j) for j in jobs))- rows.sort(key=lambda r: (not r["ok"], r["site"]))- table = Table(title="Self-check: accounts known to exist", header_style="bold blue")+ # Aggregate confusion matrix overall + per site (ignoring network errors).+ per_site: dict[str, dict[str, int]] = {}+ tp = fp = fn = tn = 0+ for r in rows:+ if r["predicted"] is None:+ continue+ cell = per_site.setdefault(r["site"], {"tp": 0, "fp": 0, "fn": 0, "tn": 0})+ if r["label"] == 1 and r["predicted"] == 1:+ key = "tp"+ elif r["label"] == 1 and r["predicted"] == 0:+ key = "fn"+ elif r["label"] == 0 and r["predicted"] == 1:+ key = "fp"+ else:+ key = "tn"+ cell[key] += 1+ tp += key == "tp"+ fp += key == "fp"+ fn += key == "fn"+ tn += key == "tn"++ overall = _metrics(tp, fp, fn, tn)+ errors = sum(1 for r in rows if r["status"] == "Error")+ result = {+ "overall": overall,+ "per_site": {s: _metrics(c["tp"], c["fp"], c["fn"], c["tn"]) for s, c in per_site.items()},+ "errors": errors,+ "samples": len([r for r in rows if r["predicted"] is not None]),+ }++ if report_format == "json":+ print(json.dumps(result, indent=2))+ return result++ _render(console, rows, result)+ return result+++def _render(console, rows: list[dict], result: dict[str, Any]) -> None:+ rows.sort(key=lambda r: (r["label"] == r.get("predicted"), r["site"]))+ table = Table(title="Self-check: known accounts + random negatives", header_style="bold blue")table.add_column("Site", style="yellow")table.add_column("Username")+ table.add_column("Expect")table.add_column("Detected")table.add_column("Confidence", justify="right")table.add_column("Result")- table.add_column("Method", style="dim")for r in rows:+ expect = "exists" if r["label"] == 1 else "absent"+ if r["predicted"] is None:+ outcome = Text("ERROR", style="dim red")+ else:+ correct = r["label"] == r["predicted"]+ outcome = Text("PASS" if correct else "FAIL", style="bold green" if correct else "bold red")table.add_row(- r["site"],- r["username"],+ r["site"], r["username"], expect,Text(r["status"], style="green" if r["status"] == "Found" else "red"),- f"{r['confidence']}%",- Text("PASS", style="bold green") if r["ok"] else Text("FAIL", style="bold red"),- r["note"],+ f"{r['confidence']}%", outcome,)console.print(table)- passed = sum(1 for r in rows if r["ok"])- errors = sum(1 for r in rows if r["status"] == "Error")- accuracy = passed / len(rows)- style = "green" if accuracy >= 0.8 else "yellow" if accuracy >= 0.6 else "red"+ # Per-site metrics table.+ site_table = Table(title="Per-site metrics", header_style="bold blue")+ site_table.add_column("Site", style="yellow")+ for col in ("Precision", "Recall", "F1", "FPR"):+ site_table.add_column(col, justify="right")+ for site, m in sorted(result["per_site"].items()):+ site_table.add_row(+ site, f"{m['precision']:.0%}", f"{m['recall']:.0%}",+ f"{m['f1']:.2f}", f"{m['false_positive_rate']:.0%}",+ )+ console.print(site_table)++ o = result["overall"]+ f1 = o["f1"]+ style = "green" if f1 >= 0.8 else "yellow" if f1 >= 0.6 else "red"console.print(Panel(- f"[{style}]Accuracy: {passed}/{len(rows)} ({accuracy:.0%})[/{style}]"- + (f" [dim]{errors} network errors[/dim]" if errors else ""),+ f"[{style}]Precision {o['precision']:.0%} Recall {o['recall']:.0%} "+ f"F1 {o['f1']:.2f} FPR {o['false_positive_rate']:.0%}[/{style}] "+ f"[dim]({result['samples']} samples"+ + (f", {result['errors']} network errors" if result["errors"] else "")+ + ")[/dim]",border_style=style,))- return accuracydiff --git a/src/aliens_eye/tui/__init__.py b/src/aliens_eye/tui/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/src/aliens_eye/tui/app.py b/src/aliens_eye/tui/app.pynew file mode 100644index 0000000..21af0ca--- /dev/null+++ b/src/aliens_eye/tui/app.py@@ -0,0 +1,162 @@+"""Interactive terminal UI for browsing scan results (optional ``[tui]`` extra: textual).++The scan runs first (in plain mode so its progress does not fight the TUI), then a+Textual app presents a filterable, searchable table of every result. Keybindings:+``f`` toggle Found/Maybe only, ``/`` search, ``o`` open the selected URL in a+browser, ``q`` quit.+"""++from __future__ import annotations++import asyncio+import webbrowser+from pathlib import Path+from typing import Any+++def _install_hint() -> None:+ from aliens_eye.utils.console import get_console++ get_console().print(+ "[yellow]The TUI needs Textual: pip install \"aliens-eye\\[tui]\"[/yellow]"+ )+++def _flatten(all_results: dict[str, list[dict[str, Any]]]) -> list[dict[str, Any]]:+ rows: list[dict[str, Any]] = []+ for variation, results in all_results.items():+ for r in results:+ profile = r.get("ai_analysis", {}).get("signals", {}).get("profile", {}) or {}+ rows.append({+ "variation": variation,+ "site": r.get("site", ""),+ "status": r.get("status", ""),+ "confidence": r.get("confidence", 0),+ "name": profile.get("name", ""),+ "url": r.get("url", ""),+ })+ rows.sort(key=lambda x: (x["status"] != "Found", x["status"] != "Maybe", -x["confidence"]))+ return rows+++async def _scan(args) -> dict[str, list[dict[str, Any]]]:+ from aliens_eye.core.analyzer import FeatureExtractor+ from aliens_eye.core.config import ScannerConfig+ from aliens_eye.core.detector import Detector+ from aliens_eye.core.fingerprints import FingerprintStore+ from aliens_eye.core.scanner import UsernameScanner, load_sites_data+ from aliens_eye.utils.console import set_plain+ from aliens_eye.utils.logger import setup_logger++ set_plain(True)+ logger = setup_logger(False)+ config = ScannerConfig()+ sites_data = load_sites_data(Path(args.sites) if args.sites else None)+ detector = Detector()+ if not args.no_ml:+ detector.load_model(logger)+ extractor = FeatureExtractor()+ fingerprints = FingerprintStore(config.fingerprints_path, config.max_fingerprints_per_label)+ fingerprints.load(logger)+ scanner = UsernameScanner(+ sites_data=sites_data, config=config, extractor=extractor,+ detector=detector, fingerprints=fingerprints, logger=logger,+ )+ try:+ return await scanner.scan_with_variations(args.username, args.level)+ finally:+ fingerprints.save()+++def run_tui(args) -> None:+ try:+ from textual.app import App, ComposeResult+ from textual.containers import Vertical+ from textual.widgets import DataTable, Footer, Header, Input+ except ImportError:+ _install_hint()+ return++ from aliens_eye.utils.console import get_console++ username = args.username or get_console().input("Enter username to scan: ").strip()+ if not username or len(username) < 2:+ get_console().print("[red]Invalid username.[/red]")+ return+ args.username = username++ all_results = asyncio.run(_scan(args))+ rows = _flatten(all_results)++ class ResultsApp(App):+ BINDINGS = [+ ("q", "quit", "Quit"),+ ("f", "toggle_found", "Found/Maybe only"),+ ("o", "open_url", "Open URL"),+ ("slash", "focus_search", "Search"),+ ]+ CSS = "Input { dock: top; }"++ def __init__(self) -> None:+ super().__init__()+ self.found_only = False+ self.query_text = ""++ def compose(self) -> ComposeResult:+ yield Header()+ yield Vertical(+ Input(placeholder="Filter by site / name / url…", id="search"),+ DataTable(id="table"),+ )+ yield Footer()++ def on_mount(self) -> None:+ table = self.query_one("#table", DataTable)+ table.add_columns("Site", "Variation", "Status", "Conf", "Name", "URL")+ table.cursor_type = "row"+ self._refresh()++ def _visible(self) -> list[dict[str, Any]]:+ out = rows+ if self.found_only:+ out = [r for r in out if r["status"] in {"Found", "Maybe"}]+ if self.query_text:+ q = self.query_text.lower()+ out = [+ r for r in out+ if q in r["site"].lower() or q in str(r["name"]).lower() or q in r["url"].lower()+ ]+ return out++ def _refresh(self) -> None:+ table = self.query_one("#table", DataTable)+ table.clear()+ for r in self._visible():+ table.add_row(+ r["site"], r["variation"], r["status"], f"{r['confidence']}%",+ str(r["name"])[:30], r["url"],+ )+ found = sum(1 for r in rows if r["status"] == "Found")+ self.title = f"Aliens Eye — @{username}"+ self.sub_title = f"{found} found / {len(rows)} results" + (+ " [Found/Maybe only]" if self.found_only else ""+ )++ def on_input_changed(self, event: Input.Changed) -> None:+ self.query_text = event.value+ self._refresh()++ def action_toggle_found(self) -> None:+ self.found_only = not self.found_only+ self._refresh()++ def action_focus_search(self) -> None:+ self.query_one("#search", Input).focus()++ def action_open_url(self) -> None:+ table = self.query_one("#table", DataTable)+ visible = self._visible()+ if 0 <= table.cursor_row < len(visible):+ webbrowser.open(visible[table.cursor_row]["url"])++ ResultsApp().run()diff --git a/src/aliens_eye/utils/console.py b/src/aliens_eye/utils/console.pyindex b9a43a8..10992d1 100644--- a/src/aliens_eye/utils/console.py+++ b/src/aliens_eye/utils/console.py@@ -39,10 +39,11 @@ def get_console(plain: bool = False) -> Console:return _console-def set_plain(plain: bool) -> None:+def set_plain(plain: bool, stderr: bool = False) -> None:global _consoleno_color = plain or not sys.stdout.isatty()- _console = Console(no_color=no_color, highlight=False)+ # Route rich output to stderr so stdout can carry machine-readable JSON.+ _console = Console(no_color=no_color, highlight=False, stderr=stderr)def status_text(status: str, confidence: int = 0) -> Text:@@ -121,14 +122,22 @@ def build_results_table(results: list[dict[str, Any]], title: str | None = None)table.add_column("Site", style="yellow", no_wrap=True)table.add_column("Status")table.add_column("HTTP", justify="center")+ table.add_column("Name", overflow="ellipsis", max_width=24)table.add_column("URL", overflow="fold")table.add_column("Time", justify="right")for r in results:code = r.get("code", 0)+ name = (+ r.get("ai_analysis", {})+ .get("signals", {})+ .get("profile", {})+ .get("name", "")+ )table.add_row(str(r.get("site", ""))[:30],status_text(r.get("status", "Unknown"), r.get("confidence", 0)),Text(str(code), style="green" if code == 200 else "red"),+ str(name)[:24],r.get("url", ""),f"{r.get('response_time', 0):.2f}s",)diff --git a/tests/test_new_features.py b/tests/test_new_features.pynew file mode 100644index 0000000..86c9d6e--- /dev/null+++ b/tests/test_new_features.py@@ -0,0 +1,210 @@+"""Tests for the feature-expansion batch: report diff, correlation, expansion,+domains, checkpoint, graph export, dataset append, and selfcheck metrics."""++import json++from aliens_eye.core import report as report_mod+from aliens_eye.core.checkpoint import Checkpoint+from aliens_eye.core.correlate import (+ Profile,+ cluster_profiles,+ profiles_from_report,+)+from aliens_eye.core.domains import domain_label+from aliens_eye.core.expand import candidate_usernames_from_results+from aliens_eye.core.exporter import ResultsExporter+from aliens_eye.ml.collect import write_rows+from aliens_eye.selfcheck import _metrics+++def _profile_result(site, variation, status="Found", name="", bio="", avatar="", url=""):+ return {+ "site": site,+ "url": url or f"https://{site}.com/{variation}",+ "final_url": url,+ "status": status,+ "code": 200,+ "response_time": 0.1,+ "confidence": 90,+ "ai_analysis": {+ "features": {"http_200": 1.0},+ "signals": {"profile": {"name": name, "bio": bio, "avatar": avatar}},+ },+ }+++def _report(results):+ return ResultsExporter.__new__(ResultsExporter)._build_results_data(+ "alice", "basic", results, "20260101_000000"+ )+++# --- report loader + diff -------------------------------------------------++def test_load_report_roundtrip_and_diff(tmp_path):+ old_results = {"alice": [_profile_result("github", "alice")]}+ new_results = {+ "alice": [+ _profile_result("github", "alice"),+ _profile_result("reddit", "alice"),+ ]+ }+ old_path = tmp_path / "old.json"+ new_path = tmp_path / "new.json"+ old_path.write_text(json.dumps(_report(old_results)), encoding="utf-8")+ new_path.write_text(json.dumps(_report(new_results)), encoding="utf-8")++ old = report_mod.load_report(old_path)+ new = report_mod.load_report(new_path)+ delta = report_mod.diff_reports(old, new)+ new_keys = [item["key"] for item in delta["new"]]+ assert "alice:reddit" in new_keys+ assert delta["gone"] == []+++def test_load_report_rejects_garbage(tmp_path):+ bad = tmp_path / "bad.json"+ bad.write_text("{}", encoding="utf-8")+ try:+ report_mod.load_report(bad)+ except report_mod.ReportError:+ pass+ else:+ raise AssertionError("expected ReportError")+++def test_diff_detects_status_change():+ old = _report({"alice": [_profile_result("github", "alice", status="Maybe")]})+ new = _report({"alice": [_profile_result("github", "alice", status="Found")]})+ changed = report_mod.diff_reports(old, new)["changed"]+ assert changed and changed[0]["key"] == "alice:github"+++# --- correlation ----------------------------------------------------------++def test_cluster_by_shared_name():+ profiles = [+ Profile("alice", "github", name="Alice Smith", bio="", avatar="", url="", status="Found"),+ Profile("alice", "gitlab", name="Alice Smith", bio="", avatar="", url="", status="Found"),+ Profile("bob", "reddit", name="Bob Jones", bio="", avatar="", url="", status="Found"),+ ]+ clusters = cluster_profiles(profiles)+ assert len(clusters) == 1+ assert clusters[0]["size"] == 2+ assert "name" in clusters[0]["reasons"]+++def test_cluster_by_shared_link():+ profiles = [+ Profile("a", "s1", name="", bio="see https://me.example/x", avatar="", url="", status="Found"),+ Profile("a", "s2", name="", bio="also https://me.example/x here", avatar="", url="", status="Maybe"),+ ]+ clusters = cluster_profiles(profiles)+ assert len(clusters) == 1+ assert "shared-link" in clusters[0]["reasons"]+++def test_no_cluster_when_unrelated():+ profiles = [+ Profile("a", "s1", name="Zoe", bio="cats", avatar="", url="", status="Found"),+ Profile("b", "s2", name="Max", bio="dogs", avatar="", url="", status="Found"),+ ]+ assert cluster_profiles(profiles) == []+++def test_profiles_from_report_skips_not_found():+ rep = _report({+ "alice": [+ _profile_result("github", "alice", status="Found", name="A"),+ _profile_result("reddit", "alice", status="Not Found"),+ ]+ })+ profiles = profiles_from_report(rep)+ assert [p.site for p in profiles] == ["github"]+++# --- expansion ------------------------------------------------------------++def test_candidate_usernames_from_bio():+ results = {+ "alice": [+ _profile_result(+ "github", "alice", status="Found",+ bio="I'm @bob_dev, also https://twitter.com/charlie",+ )+ ]+ }+ candidates = candidate_usernames_from_results(results, exclude={"alice"})+ assert "bob_dev" in candidates+ assert "charlie" in candidates+ assert "alice" not in [c.lower() for c in candidates]+++# --- domains --------------------------------------------------------------++def test_domain_label_sanitizes():+ assert domain_label("John_Doe.99") == "john-doe-99"+ assert domain_label("!!!") == ""+++# --- checkpoint -----------------------------------------------------------++async def test_checkpoint_roundtrip(tmp_path):+ path = tmp_path / "ck.jsonl"+ ck = Checkpoint(path)+ result = _profile_result("github", "alice")+ await ck.record("alice", "github", result)+ assert ck.is_done("alice", "github")++ reloaded = Checkpoint(path)+ assert reloaded.is_done("alice", "github")+ assert reloaded.results_for("alice")["github"]["site"] == "github"++ reloaded.finalize()+ assert not path.exists()+++# --- graph export ---------------------------------------------------------++def test_graph_exports_written(tmp_path):+ exporter = ResultsExporter(tmp_path)+ results = {"alice": [_profile_result("github", "alice", name="Alice")]}+ written = exporter.save_results("alice", "basic", results, ["gexf", "mermaid", "maltego"])+ suffixes = {p.suffix for p in written}+ assert ".gexf" in suffixes+ assert ".mmd" in suffixes+ assert any(p.name.endswith("_maltego.csv") for p in written)+ gexf = next(p for p in written if p.suffix == ".gexf")+ assert "github" in gexf.read_text(encoding="utf-8")+++def test_all_format_excludes_graph_and_pdf(tmp_path):+ exporter = ResultsExporter(tmp_path)+ results = {"alice": [_profile_result("github", "alice")]}+ written = exporter.save_results("alice", "basic", results, ["all"])+ suffixes = {p.suffix for p in written}+ assert suffixes == {".json", ".csv", ".html", ".md"}+++# --- dataset append -------------------------------------------------------++def test_write_rows_append(tmp_path):+ from aliens_eye.core.features import FEATURE_SCHEMA++ path = tmp_path / "ds.csv"+ row = [0.0] * len(FEATURE_SCHEMA) + [1.0]+ write_rows(path, [row], append=True)+ write_rows(path, [row], append=True)+ lines = path.read_text(encoding="utf-8").strip().splitlines()+ assert lines[0].startswith("http_200") # single header+ assert len(lines) == 3 # header + 2 rows+++# --- selfcheck metrics ----------------------------------------------------++def test_metrics_math():+ m = _metrics(tp=8, fp=2, fn=2, tn=8)+ assert m["precision"] == 0.8+ assert m["recall"] == 0.8+ assert round(m["f1"], 2) == 0.8+ assert m["false_positive_rate"] == 0.2