Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

36.7 KB
Raw Blame
#!/usr/bin/env python3
"""Fetch GitHub repo stats and append a daily snapshot to state/stats-data.jsonl.

Runs from .github/workflows/repo-stats.yml on a 24h cron. The GitHub Action
default token has push access to the repo it runs in, which is exactly what
the Traffic API (/repos/:owner/:repo/traffic/*) requires.

Why a custom script (not the off-the-shelf `github-repo-stats` Action):
- Mr Bizarro asked for "stable, has historic data, basic stuff" — keep the
  surface tiny and readable top-to-bottom.
- No build step, no node, no docker. Just stdlib urllib.
- Schema lives next to us; we can add fields later without renegotiating
  with an upstream marketplace action.

------------------------------------------------------------------------
RUNNING LOCALLY
------------------------------------------------------------------------

    GITHUB_TOKEN=$(gh auth token) python3 scripts/fetch_repo_stats.py
    GITHUB_TOKEN=$(gh auth token) python3 scripts/fetch_repo_stats.py --dry-run
    GITHUB_TOKEN=$(gh auth token) python3 scripts/fetch_repo_stats.py \
        --repo owner/name

Env vars:
- GITHUB_TOKEN     default token (auto-injected in GitHub Actions)
- GH_STATS_TOKEN   explicit override; wins over GITHUB_TOKEN
- PHOSPHENE_REPO   default repo override (CLI --repo wins over env)

Required token scopes for the default repo:
- public_repo / repo  (Traffic API requires push access)
- read:org            (only if querying a private org repo)

The classic `gh auth token` for the repo owner is sufficient.

------------------------------------------------------------------------
OUTPUT SCHEMA  (one JSON object per UTC day, newline-delimited)
------------------------------------------------------------------------

    {
      "date":   "2026-05-22",          # UTC YYYY-MM-DD (the day fetched)
      "fetched_at": "...Z",            # full ISO timestamp
      "stars":         N,                         # total today
      "forks":         N,
      "open_issues":   N,                         # excludes PRs
      "watchers":      N,                         # alias for stars (GH legacy)
      "subscribers":   N,                         # actual GitHub "watch"
      "open_prs":      N,
      # The live "what needs attention" board: the ACTUAL open issues +
      # PRs (not just counts), newest-first, capped at OPEN_ITEMS_CAP.
      # Non-sensitive fields only — this stays local-only (gitignored).
      "open_items": [
        {"number": N, "title": "...", "author": "login",
         "created_at": "...Z", "html_url": "...", "is_pr": false,
         "labels": [{"name": "bug", "color": "d73a4a"}],
         "comments": N, "draft": false},
        ...
      ],
      "open_items_truncated": false,   # true if list hit OPEN_ITEMS_CAP
      # Back-compat: most-recent-day slice. Same numbers as the last
      # element of clones_window/views_window. Kept so older dashboards
      # don't break.
      "clones": {"count": N, "uniques": N},
      "views":  {"count": N, "uniques": N},
      # Full 15-day window from the Traffic API. The API window rotates
      # daily — snapshotting this gives us a continuous record once we
      # accumulate a few days.
      "clones_window": [{"date": "2026-05-08", "count": N, "uniques": N}, ...],
      "views_window":  [{"date": "2026-05-08", "count": N, "uniques": N}, ...],
      # Cumulative star count per day, derived from /stargazers. Lets a
      # fresh dashboard show a stars curve from day 1 instead of
      # waiting for daily snapshots to accumulate.
      "stars_timeline": [{"date": "2024-09-12", "count": 1}, ...],
      "referrers": [{"referrer": "...", "count": N, "uniques": N}, ...],
      "paths":     [{"path": "...", "count": N, "uniques": N, "title": "..."}, ...],
      "releases":  {"total": N, "by_release": {"v1.0": N, ...}}
    }

Schema invariants:
- `date` is UTC YYYY-MM-DD, lexicographic-sortable.
- New fields may be added; existing field names/shapes must not change.
- stars_timeline is cumulative (not per-day). The dashboard derives
  deltas itself. Cumulative is monotonic, which is friendlier for
  partial / capped fetches.
- All counts are non-negative ints.

Idempotent: if today's date already has an entry, the new fetch REPLACES
it (so re-runs in the same UTC day don't duplicate rows).

APPEND-ONLY on disk (2026-10-14). The archive is the only copy of every
clone/view day older than GitHub's 14-day window, so this script never
re-serialises history: a new day is a plain O_APPEND of one line, and a
same-day re-run swaps only the LAST line, keeping every earlier line
byte-for-byte. A row for today that is NOT the last line (a hand-edited
file) is left alone and the new row is appended after it — the dashboard
keeps the later snapshot for a date. See _write_row().

The clones/views fields are duplicated for back-compat: they're the
SUM of the latest 24-hour entry. clones_window/views_window have the
full 15-day curve from the same API call.
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path

# ---- config --------------------------------------------------------------

DEFAULT_REPO = os.environ.get("PHOSPHENE_REPO", "mrbizarro/phosphene")
TOKEN = (
    os.environ.get("GH_STATS_TOKEN")  # explicit override
    or os.environ.get("GITHUB_TOKEN")  # default in GH Actions
    or ""
).strip()
# Writes into the panel's gitignored state/ dir (NEVER in the public
# repo — Mr Bizarro 2026-05-22). PHOSPHENE_STATS_OUT overrides for
# debugging against a sandbox path. Pre-2026-05-22 this was
# docs/stats-data.jsonl which got committed publicly; that was rolled
# back and the location moved here.
OUTPUT = Path(
    os.environ.get(
        "PHOSPHENE_STATS_OUT",
        str(Path(__file__).resolve().parent.parent / "state" / "stats-data.jsonl"),
    )
)

API = "https://api.github.com"
UA = "phosphene-repo-stats/2"

# Star scan ceiling. /stargazers paginates 100/page so this is 50 pages.
# A real repo crossing 5k stars deserves a deliberate revisit, not a
# silent runaway.
STARS_CAP = 5000
# Pagination ceilings on the other historical pulls. Each cap maps to ~50
# API pages at per_page=100. For a fresh project these never trip; they're
# bounded-runtime insurance for forks/issues/PRs/commits on a viral repo.
FORKS_CAP   = 2000
ISSUES_CAP  = 5000   # combined issues + PRs in one stream
COMMITS_CAP = 3000   # trailing 365-day window
# The live "open items" board only needs the current backlog head, not the
# whole history. 200 open issues+PRs is already an unusable wall on a
# glance-board; beyond that we truncate and flag it.
OPEN_ITEMS_CAP = 200
# All releases, paginated (see fetch_releases). 2,000 is ~20 pages.
RELEASES_CAP = 2000

# Retry policy for transient failures.
RETRY_BACKOFFS = (2, 8, 30)

# Rate-limit floor — sleep until reset if we drop below this. Scaled to the
# bucket that answered (see _handle_rate_limit): the Search API's bucket is
# 30 requests a MINUTE, so a fixed floor of 100 put every run to sleep for a
# full minute after its one search call ("rate-limit low: 29/30 — sleeping
# 62s") with 4,900 core requests still unspent.
RATE_LIMIT_FLOOR = 100

# Tracked across the run so we can print a single-line summary at exit.
_last_rate_limit: dict = {"remaining": None, "limit": None, "reset": None}

if not TOKEN:
    sys.stderr.write(
        "ERROR: no GitHub token. Set GH_STATS_TOKEN or run inside GitHub "
        "Actions (which injects GITHUB_TOKEN automatically).\n"
    )
    sys.exit(1)


# ---- http helpers --------------------------------------------------------


def _handle_rate_limit(headers) -> None:
    """Record current rate-limit state. If we're below the floor, sleep
    until the documented reset time."""
    remaining_raw = headers.get("X-RateLimit-Remaining")
    limit_raw = headers.get("X-RateLimit-Limit")
    reset_raw = headers.get("X-RateLimit-Reset")
    if remaining_raw is None:
        return
    try:
        remaining = int(remaining_raw)
    except (TypeError, ValueError):
        return
    try:
        limit = int(limit_raw) if limit_raw is not None else None
    except (TypeError, ValueError):
        limit = None
    try:
        reset = int(reset_raw) if reset_raw is not None else None
    except (TypeError, ValueError):
        reset = None
    _last_rate_limit["remaining"] = remaining
    _last_rate_limit["limit"] = limit
    _last_rate_limit["reset"] = reset

    floor = RATE_LIMIT_FLOOR
    if limit:
        floor = min(RATE_LIMIT_FLOOR, max(1, limit // 10))
    if remaining < floor and reset:
        now = int(time.time())
        wait = max(0, reset - now) + 2  # small cushion
        # Cap absurd waits so a misconfigured token doesn't hang CI.
        wait = min(wait, 3600)
        sys.stderr.write(
            f"  rate-limit low: {remaining}/{limit} — sleeping {wait}s "
            f"until reset\n"
        )
        time.sleep(wait)


def _do_request(url: str, accept: str):
    """Single HTTP request. Returns (parsed_body, headers).
    Raises HTTPError on non-2xx and URLError on network failure."""
    req = urllib.request.Request(
        url,
        headers={
            "Authorization": f"Bearer {TOKEN}",
            "Accept": accept,
            "User-Agent": UA,
            "X-GitHub-Api-Version": "2022-11-28",
        },
    )
    with urllib.request.urlopen(req, timeout=20) as resp:
        body = resp.read()
        headers = resp.headers
    _handle_rate_limit(headers)
    parsed = json.loads(body) if body else None
    return parsed, headers


def _get(path: str, accept: str = "application/vnd.github+json", *,
         with_headers: bool = False):
    """GET an API path with retries. Permanent errors (4xx) fail fast;
    transient ones (5xx / network) retry per RETRY_BACKOFFS.

    If with_headers=True, returns (body, headers); else just body.
    """
    url = path if path.startswith("http") else f"{API}{path}"
    last_exc: Exception | None = None
    for attempt, backoff in enumerate((0,) + RETRY_BACKOFFS):
        if backoff:
            sys.stderr.write(
                f"  retry {attempt}/{len(RETRY_BACKOFFS)} after {backoff}s: {url}\n"
            )
            time.sleep(backoff)
        try:
            body, headers = _do_request(url, accept)
            return (body, headers) if with_headers else body
        except urllib.error.HTTPError as exc:
            err_body = b""
            try:
                err_body = exc.read()
            except Exception:
                pass
            # Try to record rate-limit headers even on errors.
            _handle_rate_limit(exc.headers or {})
            text = err_body.decode("utf-8", errors="replace")
            if exc.code in (401, 403, 404, 422):
                sys.stderr.write(
                    f"HTTP {exc.code} on {url}: {text[:300]}\n"
                )
                # 403 with rate-limit body is technically transient, but
                # without a token-permission diff we can't distinguish a
                # "you can't see this" 403 from a "you hit the limit"
                # 403. Fail fast; the operator can re-run later.
                raise
            sys.stderr.write(
                f"HTTP {exc.code} on {url} (transient): {text[:200]}\n"
            )
            last_exc = exc
        except (urllib.error.URLError, TimeoutError, OSError) as exc:
            sys.stderr.write(f"network error on {url}: {exc}\n")
            last_exc = exc
    assert last_exc is not None
    raise last_exc


def _get_paginated(path: str, accept: str = "application/vnd.github+json",
                   *, max_items: int | None = None) -> list:
    """Drain ?per_page=100 pages. Used for stargazers timeline. Stops
    when a page is short or when max_items is hit."""
    out: list = []
    page = 1
    sep = "&" if "?" in path else "?"
    while True:
        chunk = _get(f"{path}{sep}per_page=100&page={page}", accept=accept)
        if not chunk:
            break
        if not isinstance(chunk, list):
            break
        out.extend(chunk)
        if max_items is not None and len(out) >= max_items:
            out = out[:max_items]
            break
        if len(chunk) < 100:
            break
        page += 1
        if page > 200:  # absolute safety
            break
    return out


# ---- metric fetchers -----------------------------------------------------


def fetch_repo(repo: str) -> dict:
    """The repo object — covers stars/forks/issues/watchers."""
    r = _get(f"/repos/{repo}")
    return {
        "stars":       int(r.get("stargazers_count", 0)),
        "forks":       int(r.get("forks_count", 0)),
        "open_issues": int(r.get("open_issues_count", 0)),  # includes PRs
        "watchers":    int(r.get("watchers_count", 0)),     # alias for stars
        "subscribers": int(r.get("subscribers_count", 0)),  # actual "watch"
    }


def fetch_open_prs(repo: str) -> int:
    """The repo's open_issues_count includes PRs. Disambiguate so the
    dashboard can show issues and PRs separately."""
    # /search counts are cheap and don't paginate full result lists.
    q = f"repo:{repo}+is:pr+is:open"
    r = _get(f"/search/issues?q={q}")
    return int(r.get("total_count", 0))


def fetch_open_items(repo: str) -> list[dict]:
    """The actual list of currently-open issues AND pull requests, so the
    dashboard can render a live "what needs attention" board instead of a
    bare count.

    GitHub's /issues endpoint returns PRs too (a PR is a subclass of
    Issue), discriminated by the presence of a `pull_request` key. We keep
    only the small, non-sensitive fields the board renders — number, title,
    author login, created_at, is_pr, labels (name only), and comment count.
    We deliberately DROP body text, reactions, assignees, etc.: the board
    is a glanceable attention list, and this file is local-only anyway
    (Mr Bizarro 2026-07-10).

    Newest-first (GitHub's default sort=created&direction=desc), capped at
    OPEN_ITEMS_CAP so a viral repo's backlog can't bloat the row. The count
    fields (open_issues / open_prs) remain the source of truth for totals;
    a `truncated` flag tells the dashboard when the list is a head slice."""
    entries = _get_paginated(
        f"/repos/{repo}/issues?state=open&sort=created&direction=desc",
        max_items=OPEN_ITEMS_CAP,
    )
    out: list[dict] = []
    for e in entries:
        if not isinstance(e, dict):
            continue
        is_pr = "pull_request" in e and bool(e.get("pull_request"))
        user = e.get("user") or {}
        labels = []
        for lb in e.get("labels") or []:
            if isinstance(lb, dict):
                name = lb.get("name")
                if name:
                    labels.append({
                        "name": str(name)[:60],
                        "color": str(lb.get("color") or "")[:6],
                    })
            elif isinstance(lb, str):
                labels.append({"name": lb[:60], "color": ""})
        out.append({
            "number":     int(e.get("number", 0)),
            "title":      (e.get("title") or "")[:280],
            "author":     str(user.get("login") or "") or "ghost",
            "created_at": e.get("created_at") or "",
            "html_url":   e.get("html_url") or "",
            "is_pr":      is_pr,
            "labels":     labels[:6],
            "comments":   int(e.get("comments", 0)),
            "draft":      bool(e.get("draft", False)) if is_pr else False,
        })
    return out


def _normalize_traffic_day(entry: dict) -> dict:
    """Map a Traffic-API entry to our schema. `timestamp` arrives as an
    ISO-8601 datetime (e.g. '2026-05-08T00:00:00Z'); we keep only the
    date portion to match our row-level `date` field."""
    ts = entry.get("timestamp", "")
    date = ts[:10] if isinstance(ts, str) and len(ts) >= 10 else ""
    return {
        "date":    date,
        "count":   int(entry.get("count", 0)),
        "uniques": int(entry.get("uniques", 0)),
    }


def fetch_clones_window(repo: str) -> list[dict]:
    """Full 15-day clones array. Per-day, normalized to {date,count,uniques}.
    The Traffic API rotates this window daily, so the only way to retain
    older days is to snapshot them into JSONL rows."""
    r = _get(f"/repos/{repo}/traffic/clones")
    days = r.get("clones") or []
    return [_normalize_traffic_day(d) for d in days]


def fetch_views_window(repo: str) -> list[dict]:
    """Full 15-day views array — same shape as clones."""
    r = _get(f"/repos/{repo}/traffic/views")
    days = r.get("views") or []
    return [_normalize_traffic_day(d) for d in days]


def fetch_stars_timeline(repo: str, total_stars: int) -> tuple[list[dict], bool]:
    """Cumulative star count per UTC day.

    Walks /stargazers with the star+json accept header (which adds
    `starred_at`). We DO NOT keep `user.login` — only the timestamp.
    Aggregates into a cumulative count: for each day where ≥1 star
    landed, emit {date, count}. Days with no stars are omitted so
    sparse history stays small; the dashboard can ffill itself.

    Returns (timeline, partial).
    `partial` is True if we hit STARS_CAP and bailed out — the dashboard
    can show a "showing first N stars" note.
    """
    # Skip the heavy pagination if the repo is empty.
    if total_stars <= 0:
        return [], False

    capped = total_stars > STARS_CAP
    max_items = STARS_CAP if capped else None
    entries = _get_paginated(
        f"/repos/{repo}/stargazers",
        accept="application/vnd.github.v3.star+json",
        max_items=max_items,
    )

    # Buckets keyed by YYYY-MM-DD. Use timestamps only — no login.
    counts_by_day: dict[str, int] = {}
    for e in entries:
        if not isinstance(e, dict):
            continue
        ts = e.get("starred_at", "")
        if not isinstance(ts, str) or len(ts) < 10:
            continue
        day = ts[:10]
        counts_by_day[day] = counts_by_day.get(day, 0) + 1

    # Convert to cumulative timeline.
    timeline: list[dict] = []
    running = 0
    for day in sorted(counts_by_day.keys()):
        running += counts_by_day[day]
        timeline.append({"date": day, "count": running})

    return timeline, capped


def fetch_forks_timeline(repo: str, total_forks: int) -> tuple[list[dict], bool]:
    """Cumulative fork count per UTC day. Same shape as stars_timeline,
    derived from /forks created_at. Cheap — most repos have few forks.
    Capped at FORKS_CAP for safety on viral repos."""
    if total_forks <= 0:
        return [], False
    capped = total_forks > FORKS_CAP
    max_items = FORKS_CAP if capped else None
    entries = _get_paginated(f"/repos/{repo}/forks?sort=oldest",
                             max_items=max_items)
    counts_by_day: dict[str, int] = {}
    for e in entries:
        if not isinstance(e, dict):
            continue
        ts = e.get("created_at", "")
        if not isinstance(ts, str) or len(ts) < 10:
            continue
        day = ts[:10]
        counts_by_day[day] = counts_by_day.get(day, 0) + 1
    timeline: list[dict] = []
    running = 0
    for day in sorted(counts_by_day.keys()):
        running += counts_by_day[day]
        timeline.append({"date": day, "count": running})
    return timeline, capped


def fetch_issue_pr_timelines(repo: str) -> dict:
    """Daily opened/closed counts for both Issues and PRs.

    Walks /issues with state=all (GitHub's "issues" endpoint returns
    PRs too — they're a subclass of Issue — discriminated by the
    presence of a `pull_request` key). Splits into 4 separate
    timelines: issues_opened / issues_closed / prs_opened / prs_closed.

    Each timeline: list of {date, count} for DAILY counts (not
    cumulative — the dashboard wants "velocity" which is a daily
    rate). Capped at ISSUES_CAP entries total."""
    entries = _get_paginated(
        f"/repos/{repo}/issues?state=all&sort=created&direction=asc",
        max_items=ISSUES_CAP,
    )
    issues_opened: dict[str, int] = {}
    issues_closed: dict[str, int] = {}
    prs_opened: dict[str, int]    = {}
    prs_closed: dict[str, int]    = {}
    for e in entries:
        if not isinstance(e, dict):
            continue
        is_pr = "pull_request" in e and bool(e.get("pull_request"))
        opened_at = (e.get("created_at") or "")[:10]
        closed_at = (e.get("closed_at") or "")[:10] if e.get("closed_at") else ""
        if len(opened_at) == 10:
            d = prs_opened if is_pr else issues_opened
            d[opened_at] = d.get(opened_at, 0) + 1
        if len(closed_at) == 10:
            d = prs_closed if is_pr else issues_closed
            d[closed_at] = d.get(closed_at, 0) + 1
    def _to_list(bucket: dict[str, int]) -> list[dict]:
        return [{"date": d, "count": bucket[d]} for d in sorted(bucket)]
    return {
        "issues_opened": _to_list(issues_opened),
        "issues_closed": _to_list(issues_closed),
        "prs_opened":    _to_list(prs_opened),
        "prs_closed":    _to_list(prs_closed),
    }


def fetch_commits_timeline(repo: str) -> list[dict]:
    """Daily commit count over the trailing window. Walks /commits with
    ?since=ISO-365-days-ago paginated. Capped at COMMITS_CAP to bound
    runtime on a long-lived repo. Daily counts (not cumulative — the
    dashboard wants activity, not totals)."""
    from datetime import timedelta
    since = (datetime.now(timezone.utc) - timedelta(days=365)).strftime(
        "%Y-%m-%dT%H:%M:%SZ")
    entries = _get_paginated(
        f"/repos/{repo}/commits?since={since}",
        max_items=COMMITS_CAP,
    )
    counts_by_day: dict[str, int] = {}
    for e in entries:
        if not isinstance(e, dict):
            continue
        # Commit author date (the developer's clock). /commits.commit.author.date
        # is the canonical timestamp; .committer.date can differ on rebases.
        commit = e.get("commit", {}) or {}
        author = commit.get("author", {}) or {}
        ts = (author.get("date") or "")[:10]
        if len(ts) != 10:
            continue
        counts_by_day[ts] = counts_by_day.get(ts, 0) + 1
    return [{"date": d, "count": counts_by_day[d]} for d in sorted(counts_by_day)]


_RELEASES_CACHE: dict[str, list] = {}


def fetch_releases(repo: str) -> list:
    """Every release, all pages. One fetch per run, shared by the timeline
    and the download totals.

    This used to be ONE request with per_page=100, i.e. silently the newest
    100 releases. The repo had 99 on 2026-10-14; the next tag would have
    started dropping the oldest release off the table and out of the
    download total without a word. RELEASES_CAP bounds a runaway."""
    if repo in _RELEASES_CACHE:
        return _RELEASES_CACHE[repo]
    try:
        rs = _get_paginated(f"/repos/{repo}/releases", max_items=RELEASES_CAP)
    except urllib.error.HTTPError:
        rs = []
    rs = [r for r in rs if isinstance(r, dict)]
    _RELEASES_CACHE[repo] = rs
    return rs


def fetch_release_timeline(repo: str) -> list[dict]:
    """Release publish dates + cumulative download counts. Lets the
    dashboard show "release X shipped on Y, has Z downloads" — much
    more useful than the bare {total, by_release: {...}} blob."""
    rs = fetch_releases(repo)
    out: list[dict] = []
    for rel in rs:
        if not isinstance(rel, dict):
            continue
        tag = rel.get("tag_name", "?")
        published = (rel.get("published_at") or rel.get("created_at") or "")
        total = 0
        counts: list[int] = []
        by_pack: dict[str, list[int]] = {}
        for a in rel.get("assets") or []:
            try:
                n = int(a.get("download_count", 0))
            except (TypeError, ValueError):
                continue
            total += n
            counts.append(n)
            # Pack files are named <pack>__<file>; a release can carry
            # several packs (q4_25, q8_25, gemma4_25) with different reach.
            name = str(a.get("name") or "")
            if "__" in name:
                by_pack.setdefault(name.split("__", 1)[0], []).append(n)
        # A weight pack is ~40 files per install, so its "downloads" is
        # ~40× the number of installs that fetched it (84.4K read as a crowd
        # using the weights without the app — it was ~1,000 installs). The
        # median per-file count within ONE pack is that pack's complete
        # downloads (big shards run higher: interrupted fetches resume them).
        packs = {}
        for pk, ns in by_pack.items():
            ns.sort()
            packs[pk] = ns[len(ns) // 2]
        pack_downloads = max(packs.values()) if packs else (
            sorted(counts)[len(counts) // 2] if counts else 0)
        out.append({
            "tag": tag,
            "published_at": published,
            "total_downloads": total,
            "asset_count": len(counts),
            "pack_downloads": pack_downloads,
            "packs": packs,
        })
    # GitHub returns newest-first; sort oldest-first for charting.
    out.sort(key=lambda r: r.get("published_at") or "")
    return out


def fetch_referrers(repo: str) -> list[dict]:
    """Top 10 referring domains over the rolling 14-day window. We snapshot
    the 14d aggregate every day rather than try to derive a per-day
    breakdown (the API doesn't give one)."""
    r = _get(f"/repos/{repo}/traffic/popular/referrers")
    if not isinstance(r, list):
        return []
    return [
        {
            "referrer": x.get("referrer", "?"),
            "count":    int(x.get("count", 0)),
            "uniques":  int(x.get("uniques", 0)),
        }
        for x in r[:10]
    ]


def fetch_paths(repo: str) -> list[dict]:
    """Top 10 paths viewed over the rolling 14d. Tells us which docs are
    actually getting read (README vs STATE vs ROADMAP, etc)."""
    r = _get(f"/repos/{repo}/traffic/popular/paths")
    if not isinstance(r, list):
        return []
    return [
        {
            "path":    x.get("path", "?"),
            "title":   (x.get("title", "") or "")[:120],
            "count":   int(x.get("count", 0)),
            "uniques": int(x.get("uniques", 0)),
        }
        for x in r[:10]
    ]


def fetch_release_downloads(repo: str) -> dict:
    """Cumulative downloads per release asset. GitHub provides no time
    series here, so we just take a snapshot — the dashboard can derive
    deltas across snapshots itself. Empty when the repo has no releases
    (Phosphene currently ships via Pinokio clone, not release binaries)."""
    rs = fetch_releases(repo)
    total = 0
    per_release: dict = {}
    for rel in rs:
        tag = rel.get("tag_name", "?")
        rel_total = 0
        for a in rel.get("assets") or []:
            n = int(a.get("download_count", 0))
            rel_total += n
            total += n
        per_release[tag] = rel_total
    return {"total": total, "by_release": per_release}


# ---- summary helpers -----------------------------------------------------


def _window_total(window: list[dict]) -> int:
    return sum(int(d.get("count", 0)) for d in window)


def _wow_delta(window: list[dict]) -> int:
    """Week-over-week count delta inside the same 15-day window. Compares
    the last 7 entries to the 7 before them. Returns 0 if we don't have
    14 days of data yet."""
    if len(window) < 14:
        return 0
    recent = sum(int(d.get("count", 0)) for d in window[-7:])
    prior = sum(int(d.get("count", 0)) for d in window[-14:-7])
    return recent - prior


def _stars_wow_delta(timeline: list[dict]) -> int:
    """Stars gained in the last 7 UTC days, vs the 7 before. Cumulative
    timeline → derive deltas."""
    if len(timeline) < 2:
        return 0
    now = datetime.now(timezone.utc).date()
    today_iso = now.isoformat()
    # Find cumulative star count as of T, T-7d, T-14d. timeline is
    # cumulative-by-day with gaps; carry-forward by using the last entry
    # whose date <= target.
    def cum_at(target: str) -> int:
        last = 0
        for entry in timeline:
            if entry.get("date", "") <= target:
                last = int(entry.get("count", 0))
            else:
                break
        return last
    seven = (now.toordinal() - 7)
    fourteen = (now.toordinal() - 14)
    seven_iso = datetime.fromordinal(seven).strftime("%Y-%m-%d")
    fourteen_iso = datetime.fromordinal(fourteen).strftime("%Y-%m-%d")
    recent = cum_at(today_iso) - cum_at(seven_iso)
    prior = cum_at(seven_iso) - cum_at(fourteen_iso)
    return recent - prior


# ---- archive write -------------------------------------------------------


def _write_row(path: Path, row: dict) -> None:
    """Add today's row to the archive WITHOUT rewriting history.

    - No row for this date at the end of the file: append one line
      (O_APPEND), after a newline if the file does not end in one.
    - The LAST line is already this date (a same-day re-run): replace just
      that line. Every earlier line is copied through as raw bytes, never
      parsed and re-dumped, then the file is swapped atomically (temp +
      fsync + os.replace) so a crash leaves the old or the new file, never a
      torn one.

    History used to be rewritten on every run: the whole file was parsed,
    re-serialised and written back. It happened to round-trip byte-for-byte,
    but one unparseable line was silently DROPPED (a `continue` on
    JSONDecodeError), and those lines are the only copy of days GitHub no
    longer serves."""
    line = json.dumps(row, ensure_ascii=False).encode("utf-8") + b"\n"
    path.parent.mkdir(parents=True, exist_ok=True)
    try:
        raw = path.read_bytes()
    except FileNotFoundError:
        raw = b""
    body = raw[:-1] if raw.endswith(b"\n") else raw
    cut = body.rfind(b"\n") + 1          # start of the last line (0 if one line)
    last = body[cut:]
    same_day = False
    if last.strip():
        try:
            same_day = json.loads(last).get("date") == row.get("date")
        except (json.JSONDecodeError, AttributeError, UnicodeDecodeError):
            same_day = False
    if not same_day:
        with open(path, "ab") as fh:
            if raw and not raw.endswith(b"\n"):
                fh.write(b"\n")
            fh.write(line)
            fh.flush()
            try:
                os.fsync(fh.fileno())
            except OSError:
                pass
        print(f"  appended {row.get('date')} -> {path}", flush=True)
        return
    payload = raw[:cut] + line
    tmp = path.with_name(f".{path.name}.{os.getpid()}.tmp")
    with open(tmp, "wb") as fh:
        fh.write(payload)
        fh.flush()
        try:
            os.fsync(fh.fileno())
        except OSError:
            pass
    os.replace(tmp, path)
    print(f"  replaced today's row ({row.get('date')}) -> {path}", flush=True)


# ---- main ----------------------------------------------------------------


def _parse_args(argv: list[str]) -> argparse.Namespace:
    p = argparse.ArgumentParser(
        description="Fetch GitHub repo stats and append a daily JSONL row.",
    )
    p.add_argument(
        "--repo",
        default=DEFAULT_REPO,
        help=f"OWNER/NAME (default: {DEFAULT_REPO}, or $PHOSPHENE_REPO)",
    )
    p.add_argument(
        "--dry-run",
        action="store_true",
        help="Fetch and print, but don't write the JSONL file.",
    )
    return p.parse_args(argv)


def main(argv: list[str] | None = None) -> int:
    args = _parse_args(argv or sys.argv[1:])
    repo = args.repo

    now = datetime.now(timezone.utc)
    today = now.strftime("%Y-%m-%d")
    iso_now = now.strftime("%Y-%m-%dT%H:%M:%SZ")
    print(f"[{iso_now}] fetching stats for {repo}"
          + ("  (dry-run)" if args.dry_run else ""), flush=True)

    repo_obj = fetch_repo(repo)
    print(f"  repo: {repo_obj}", flush=True)
    open_prs = fetch_open_prs(repo)
    repo_obj["open_prs"] = open_prs
    # open_issues from the repo API includes PRs — subtract to disambiguate.
    repo_obj["open_issues"] = max(0, repo_obj["open_issues"] - open_prs)
    print(f"  prs: {open_prs} (open_issues adjusted to {repo_obj['open_issues']})",
          flush=True)

    # The actual open issues + PRs, so the dashboard renders a live
    # attention board (not just the count). Cheap on a normal repo; capped.
    open_items = fetch_open_items(repo)
    open_items_truncated = len(open_items) >= OPEN_ITEMS_CAP
    _n_pr = sum(1 for it in open_items if it.get("is_pr"))
    print(f"  open items: {len(open_items)} listed "
          f"({len(open_items) - _n_pr} issues · {_n_pr} prs)"
          + ("  (TRUNCATED)" if open_items_truncated else ""), flush=True)

    clones_window = fetch_clones_window(repo)
    views_window = fetch_views_window(repo)
    # Back-compat: most-recent-day slice as a dict.
    if clones_window:
        last = clones_window[-1]
        clones_today = {"count": last["count"], "uniques": last["uniques"]}
    else:
        clones_today = {"count": 0, "uniques": 0}
    if views_window:
        last = views_window[-1]
        views_today = {"count": last["count"], "uniques": last["uniques"]}
    else:
        views_today = {"count": 0, "uniques": 0}
    print(f"  clones window: {len(clones_window)} days, "
          f"total={_window_total(clones_window)}, "
          f"today={clones_today['count']} ({clones_today['uniques']} unique)",
          flush=True)
    print(f"  views  window: {len(views_window)} days, "
          f"total={_window_total(views_window)}, "
          f"today={views_today['count']} ({views_today['uniques']} unique)",
          flush=True)

    timeline, partial = fetch_stars_timeline(repo, repo_obj["stars"])
    if partial:
        print(f"  stars timeline: PARTIAL — capped at {STARS_CAP} "
              f"(repo has {repo_obj['stars']})", flush=True)
    else:
        print(f"  stars timeline: {len(timeline)} day-buckets, "
              f"final cum={(timeline[-1]['count'] if timeline else 0)}",
              flush=True)

    # Full historical backfill — captured fresh each run so old days
    # never need to be re-derived. These are the signals the GitHub
    # Traffic API can't ever give us beyond 14 days (clones/views), but
    # everything else has full history available retroactively. Mr Bizarro
    # 2026-05-22: "Can you retroactively pull the data from the first
    # days of the project?" — yes, for these.
    forks_timeline, forks_partial = fetch_forks_timeline(repo, repo_obj["forks"])
    if forks_partial:
        print(f"  forks timeline: PARTIAL — capped at {FORKS_CAP}", flush=True)
    else:
        print(f"  forks timeline: {len(forks_timeline)} day-buckets, "
              f"final cum={(forks_timeline[-1]['count'] if forks_timeline else 0)}",
              flush=True)

    issue_pr = fetch_issue_pr_timelines(repo)
    print(f"  issues/PRs: opened {len(issue_pr['issues_opened'])}/"
          f"{len(issue_pr['prs_opened'])} buckets, "
          f"closed {len(issue_pr['issues_closed'])}/"
          f"{len(issue_pr['prs_closed'])} buckets", flush=True)

    commits_timeline = fetch_commits_timeline(repo)
    commits_total_365 = sum(d["count"] for d in commits_timeline)
    print(f"  commits timeline (365d): {len(commits_timeline)} active days, "
          f"{commits_total_365} commits total", flush=True)

    release_timeline = fetch_release_timeline(repo)
    print(f"  release timeline: {len(release_timeline)} releases", flush=True)

    referrers = fetch_referrers(repo)
    paths = fetch_paths(repo)
    releases = fetch_release_downloads(repo)
    print(f"  referrers: {len(referrers)}  paths: {len(paths)}  "
          f"release downloads total: {releases.get('total', 0)}", flush=True)

    row = {
        "date": today,
        "fetched_at": iso_now,
        **repo_obj,
        "clones": clones_today,
        "views":  views_today,
        "clones_window": clones_window,
        "views_window":  views_window,
        "stars_timeline":   timeline,
        "forks_timeline":   forks_timeline,
        "issues_opened":    issue_pr["issues_opened"],
        "issues_closed":    issue_pr["issues_closed"],
        "prs_opened":       issue_pr["prs_opened"],
        "prs_closed":       issue_pr["prs_closed"],
        "open_items":            open_items,
        "open_items_truncated":  open_items_truncated,
        "commits_timeline": commits_timeline,
        "release_timeline": release_timeline,
        "referrers": referrers,
        "paths":     paths,
        "releases":  releases,
    }

    if args.dry_run:
        print("  dry-run: not writing JSONL", flush=True)
    else:
        _write_row(OUTPUT, row)

    # Summary line for GitHub Actions step output.
    clones_wow = _wow_delta(clones_window)
    views_wow = _wow_delta(views_window)
    stars_wow = _stars_wow_delta(timeline)
    summary = (
        f"summary: stars={repo_obj['stars']} ({stars_wow:+d} w/w)  "
        f"clones={_window_total(clones_window)} ({clones_wow:+d} w/w)  "
        f"views={_window_total(views_window)} ({views_wow:+d} w/w)  "
        f"referrers={len(referrers)}  paths={len(paths)}"
    )
    print(summary, flush=True)

    rl = _last_rate_limit
    if rl["remaining"] is not None:
        print(f"  rate-limit remaining: {rl['remaining']}/{rl['limit']}",
              flush=True)

    return 0


if __name__ == "__main__":
    sys.exit(main())