AI powered OSINT engine to discover and map 840+ social media identities across the web
Aliens Eye Working Notes
This document explains the runtime flow, data pipeline, and internal modules.
High-level flow
flowchart TD
A[Start] --> B[Load config and CLI args]
B --> C{Read results file?}
C -->|Yes| D[Load JSON report]
C -->|No| E[Load packaged sites.json]
E --> E2[Apply site filters: --site / --exclude-site / --no-nsfw]
E2 --> F[Generate username variations]
F --> G[Queue workers]
G --> H[Fetch URL with retries, optional proxy/Tor]
H --> I[Extract features with selectolax]
I --> J[Fingerprint scoring]
J --> K[Detect: ML model + heuristics blended]
K --> L{Maybe + Playwright enabled?}
L -->|Yes| M[Render page and re-extract]
L -->|No| N[Record result]
M --> N
N --> O[Update fingerprints]
O --> P[Write JSON/CSV/HTML/Markdown reports]
P --> Q[Done]
Module map
src/aliens_eye/cli.py: argparse CLI, subcommands (selfcheck,train), interactive promptssrc/aliens_eye/core/analyzer.py: HTML parsing and feature extraction (selectolax)src/aliens_eye/core/http.py: fetch with retries, backoff, response size caps, HTTP proxysrc/aliens_eye/core/rate_limit.py: per-domain delay controlsrc/aliens_eye/core/detector.py: ML + heuristic blended scoring and status selectionsrc/aliens_eye/core/fingerprints.py: persisted match fingerprints by sitesrc/aliens_eye/core/scanner.py: async queue workers, site filtering, SOCKS connectorsrc/aliens_eye/core/exporter.py: JSON/CSV/HTML/Markdown report generationsrc/aliens_eye/corpus/store.py: on-disk corpus layout (records + content-addressed bodies)src/aliens_eye/corpus/record.py: capture live responses into a corpussrc/aliens_eye/corpus/replay.py: serve a corpus back through the fetch interfacesrc/aliens_eye/ml/inference.py: pure-python model inference (no sklearn at runtime)src/aliens_eye/ml/collect.py: labeled dataset builder from ground-truth accountssrc/aliens_eye/ml/train.py: sklearn training, exports coefficients to model.jsonsrc/aliens_eye/selfcheck.py: accuracy validation against known accountssrc/aliens_eye/utils/console.py: rich-based progress, tables, panelssrc/aliens_eye/data/: sites.json, model.json, selfcheck.json, nsfw_sites.json, seed_dataset.csv
Feature extraction
Signals include:
- HTTP status buckets (200, 3xx, 4xx, 5xx)
- Presence of username in URL path
- Auth-related path patterns
- Positive and error keyword counts (content and meta)
- DOM counts (img, form, input, profile/error class hints)
- Response time and content length
- Redirect count
- Fingerprint match counts
- Heuristic score (fed to the ML model as a feature)
These are stored as a consistent feature schema (core/features.py) shared by the
heuristic engine, training, and inference.
Detection logic
Two judges vote on every response:
- Heuristic engine — weighted score over the features, squashed to a probability with a sigmoid.
- ML model — logistic regression trained offline with sklearn and exported
to
data/model.json(scaler stats + coefficients). Inference is a pure-python dot product, so the installed package has no ML dependencies.
The final probability is w * ml + (1 - w) * heuristic. The loaded model
supplies w and both thresholds; the constants in core/detector.py
(ML_WEIGHT = 0.4, FOUND_THRESHOLD = 0.6, NOT_FOUND_THRESHOLD = 0.35) are
only fallbacks for when no model loads or a model file omits the fields.
| ml weight | Found > | Not Found < | |
|---|---|---|---|
Shipped data/model.json (default runtime) |
0.9 | 0.6202 | 0.3597 |
| Fallback constants (heuristic-only / fields absent) | 0.4 | 0.6 | 0.35 |
Confidence is scaled by distance from the threshold. If the model file is
missing or invalid, detection falls back to heuristics alone.
tests/test_detector.py pins the shipped model's values against this table, so
the two cannot drift apart silently.
Frozen response corpus
Evaluating against the live web is not reproducible: platforms change markup,
rate-limit differently day to day, and block some networks outright, so a change
in measured accuracy cannot be attributed to the detector. corpus/ removes the
network from the measurement loop.
flowchart LR
A[ground truth] --> B[corpus record]
B -->|live fetch| C[(records.jsonl<br/>bodies/*.gz)]
C --> D[corpus replay]
D --> E[analyzer -> detector -> metrics]
F[live fetch_url] --> E
fetch_url is the only seam between the scanner and the network, so replay
substitutes just that callable. Feature extraction, scoring and vectorisation
run byte-identical code on live and replayed responses, which means an evaluation
measures the detector rather than the harness.
Determinism needs three things beyond stored bytes:
- The rate limiter is not consulted, so replay never sleeps and wall-clock timing cannot leak into results.
response_timeis the recorded value, not a fresh measurement.- The fingerprint store must be read-only.
FingerprintStoreaccumulates signatures during a scan and scores each response against whatever earlier responses happened to finish first — live, its contribution depends on async worker completion order.FingerprintStore(path, read_only=True)freezes it.
When --corpus is given, the corpus also supplies the evaluation set: rows
come from the recorded labels rather than from freshly generated usernames.
Regenerating them would produce URLs the corpus never captured, so every negative
would miss and the run would report a false-positive rate over an empty negative
class.
aliens_eye corpus record --out corpus/v1 --split all --negatives 4 --seed 1234
aliens_eye corpus stats corpus/v1
aliens_eye selfcheck --split holdout --corpus corpus/v1 --report json
Captured corpora are local artifacts and are not committed: they hold third-party page content, and a full capture is tens of megabytes.
Negatives are generated in two kinds, tagged per record: random (14-20 random
alphanumerics) and plausible (ordinary word morphology, e.g. quietfalcon84).
A pool of only random strings is an artificially easy negative class — separable
on length and character distribution alone — and inflates every metric.
Plausible negatives are not verified absent by construction: on a large platform
one may collide with a real account, so they carry label 0 on assumption.
Ablation harness
aliens_eye eval ablate --corpus DIR scores several detector configurations over
the same frozen corpus: naive status baselines, each judge in isolation, the
shipped blend, and one run per zeroed feature group. Features are extracted once
per response and shared across configurations, so a difference in the table is a
difference in the configuration and not in the harness.
Every row carries a percentile-bootstrap 95% CI on F1, and every non-reference
row a paired-bootstrap CI on the F1 difference against the shipped blend.
Paired, because all configurations score the same rows. A difference whose CI
spans zero is reported as ns — on corpora this small most differences are not
resolvable, and the table says so rather than inviting a reader to rank noise.
The Maybe rate is reported alongside precision. Detection is tri-state, so a configuration can inflate precision by abstaining on everything hard; quoting precision without the abstention rate beside it would be misleading.
What this measured, on a site-disjoint holdout of 142 platforms: no
configuration is distinguishable from if status == 200 on F1. Every judge --
ML alone, heuristics alone, the shipped blend -- lands in the same band with
overlapping confidence intervals, and so do Sherlock's hand-curated per-site
rules.
What does differ is the error profile. The 2.5.0 model, trained on full pages (see below), runs at FPR 0.089 against 0.251 for the model it replaced -- 47 false "Found" results on 530 holdout negatives instead of 133 -- at the cost of recall (0.51 against 0.67). Both differences are significant under a paired bootstrap; F1 is not.
Everything measured before 2.5.0 was on truncated pages. The HTTP reader parsed only the first network chunk of each page -- a random-length prefix -- so pre-2.5.0 corpora, results and the old model all reflect that. In particular, an earlier finding that ~45% of pages "changed within hours" compared one random truncation with another and should not be relied on.
A caution worth keeping, because this repo learned it the hard way: an earlier run on a 43-site corpus showed ML alone beating the shipped blend by a statistically significant +0.126 F1. That corpus included sites the model had trained on. On a properly site-disjoint holdout the effect vanished (+0.013, not significant).
External baselines
aliens_eye eval external --corpus DIR --sherlock PATH --whatsmyname PATH runs
those projects' own per-site rules over the stored responses, so the comparison
isolates detection logic from HTTP behaviour. Rule data is not vendored —
fetch it from upstream and mind the licence (Sherlock MIT; WhatsMyName CC BY-SA
4.0). Each run records the rule file's sha256.
Rows a tool has no rule for, or that its rules leave undecided, are excluded from its score rather than counted against it; the covered-row count is reported alongside. Charging a tool for sites it never claimed to support would understate it — and WhatsMyName in particular decides only a small fraction of rows.
Ground-truth splits
Ground truth is site-disjoint across two files:
data/selfcheck.json— train split (30 sites, 45 positives). The only splittrain collectreads.data/eval_holdout.json— holdout (13 sites, 15 positives). Never seen by training.
The split is by site, not by account: the detector learns per-site page
structure, so holding out only accounts from an already-trained site would
overstate generalization. aliens_eye selfcheck --split holdout measures
generalization to unseen platforms; --split train measures fit and reads
optimistically high. train collect --split holdout is refused unless
--allow-leakage is passed.
Training pipeline
aliens_eye train collectscans ground-truth accounts from the train split (label 1) and randomized non-existent usernames (label 0), writing one feature row per scan.aliens_eye train fittrains LogisticRegression + StandardScaler and exports the model JSON, including the blend weight and thresholds calibrated by out-of-fold grid search.aliens_eye selfcheck --split holdoutmeasures live accuracy on sites the model never trained on.
Accuracy, measured honestly. The shipped model was trained on 1,455 samples from the train split (286 platforms) and evaluated on the site-disjoint holdout (142 platforms, 700 rows): precision 0.65, recall 0.51, FPR 0.089, F1 0.57. Before 2.5.0 it was trained on truncated pages -- see the 2.5.0 changelog -- and ran at FPR 0.25. F1 is not distinguishable from an HTTP-status baseline on held-out platforms; the gain is in precision, i.e. far fewer false leads.
Fingerprints
Fingerprints store compact signatures of known Found and Not Found pages per site:
- title hash
- meta hash
- DOM signature
- server header
During detection, similarity scores are added as features to reduce false positives.
The cache lives in the platform cache dir (via platformdirs), not in the package.
Retry and rate limiting
- Retryable statuses: 408, 429, 500, 502, 503, 504
- Backoff: exponential with jitter and optional Retry-After handling
- Per-domain delay: prevents hammering a single host
Proxies
http:///https://proxies are passed per-request to aiohttpsocks4:///socks5://proxies replace the connector withaiohttp_socks.ProxyConnector(--toris shorthand forsocks5://127.0.0.1:9050)
Playwright fallback (optional)
If enabled, Playwright is used only when a result is Maybe. The rendered DOM is
re-parsed and can upgrade confidence. Install with pip install aliens-eye[browser].
Output pipeline
Reports are timestamped and written to:
- JSON (full detail)
- CSV (flat table)
- HTML (shareable summary)
- Markdown (Found/Maybe digest)
Config precedence
flowchart TD
A[Defaults] --> B[config.json or platform config dir]
B --> C[CLI flags]
C --> D[Final runtime config]