Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

Track privacy-safe website traffic (#3392)

zhifu gao93e675d51d1b
diff --git a/docs/community_growth_20k.md b/docs/community_growth_20k.md
index be6229cc0..e2993c618 100644
--- a/docs/community_growth_20k.md
+++ b/docs/community_growth_20k.md
@@ -371,7 +371,7 @@ Paste the Markdown output into launch notes, weekly community updates, or releas
- [x] Pin a GitHub discussion for the release. On 2026-07-23, [`modelscope/FunASR#3376`](https://github.com/modelscope/FunASR/discussions/3376) was pinned through the repository's authenticated Discussions admin UI. GitHub returned `Discussion has successfully been pinned`, and the admin controls now expose `Edit pinned discussion` and `Unpin discussion`; the `v1.3.26` OpenAI-compatible ASR, vLLM, and llama.cpp release story is therefore visible at the top of the public Discussions page.
- [x] Triage all new issues within 48 hours. On 2026-07-23, all four repositories were patrolled with the GitHub token on `ind-gpu8`: `modelscope/FunASR`, `FunAudioLLM/Fun-ASR`, `FunAudioLLM/SenseVoice`, and `modelscope/FunClip` all had `0` open pull requests; Fun-ASR, SenseVoice, and FunClip had `0` open issues. The only open FunASR issues were already triaged: #3302 is labeled `question` + `needs feedback` and is waiting for the reporter to retest the merged hotword/postprocess fix, while #3104 is labeled `enhancement` + `needs maintainer decision` and is waiting on release-scope confirmation for RL training code. No unlabelled or unowned open issue remained in the 48-hour window.
- [x] Convert top 3 support questions into docs. On 2026-07-23, the latest 80 FunASR issues were triaged into three repeat support blockers: install / hub / model-id selection (#3321, #3045, #3042, #2973, #2976), llama.cpp / GGUF / CUDA-Vulkan runtime packages (#3298, #3297, #3296, #3289, #3243), and realtime / VAD / vLLM / server output behavior (#3101, #3109, #3038, #3031, #2968, #2965). `docs/troubleshooting.md` and `docs/troubleshooting_zh.md` now start with a Top support questions section that maps those three issue clusters to the existing install, model-download, OpenAI-compatible server, WebSocket, and llama.cpp / GGUF FAQ anchors.
-- [ ] Track stars, PyPI downloads, issue volume, and docs traffic. `collect_growth_metrics.py` now tracks GitHub stars, open issues, and open pull requests for the four core repositories and adds PyPIStats download windows for `funasr` as `downloads_last_7_days` and `downloads_last_30_days`. On 2026-07-23, the live PyPIStats window was refreshed at `116,407` downloads in 7 days and `442,556` downloads in 30 days through `2026-07-21`; successful runs now write `/cpfs_speech/user/zhifu.gzf/.cache/funasr-ops/pypi-downloads-funasr.json`, and failed live runs fall back to that cache with status `stale_available` instead of dropping the download metric during SSL/429 outages. GitHub/PyPI tracking is now covered; docs traffic remains unavailable until an analytics credential or export is connected.
+- [x] Track stars, PyPI downloads, issue volume, and docs traffic. `collect_growth_metrics.py` tracks GitHub stars, open issues, and open pull requests for the four core repositories and adds PyPIStats download windows for `funasr` as `downloads_last_7_days` and `downloads_last_30_days`. On 2026-07-23, the live PyPIStats window was refreshed at `117,850` downloads in 7 days and `445,755` downloads in 30 days through `2026-07-22`; successful runs write `/cpfs_speech/user/zhifu.gzf/.cache/funasr-ops/pypi-downloads-funasr.json`, and failed live runs fall back to that cache with status `stale_available` instead of dropping the metric during SSL/429 outages. `scripts/collect_website_traffic.py` completes the docs-traffic side from plain or gzip Nginx logs with `python scripts/collect_website_traffic.py --log '/var/log/nginx/access.log*' --days 30 --format json`; it filters bots, static assets, unsuccessful responses, and out-of-window requests, and emits aggregate counts only so raw client IPs and user agents are never emitted. A production run on `funasr-web` over 15 rotated logs at `2026-07-23T04:52:08Z` measured 30-day traffic at `88,239` page views / `47,084` approximate unique visitors, including docs at `15,501` / `11,843` and blog at `27,901` / `22,716`. The matching 7-day run measured `21,915` / `12,668`, including docs at `4,638` / `3,612` and blog at `6,901` / `5,590`. Stars, PyPI downloads, issue volume, and owned-site traffic now have repeatable baselines for patrols.
## Release-note template
diff --git a/scripts/collect_website_traffic.py b/scripts/collect_website_traffic.py
new file mode 100644
index 000000000..f64dcd550
--- /dev/null
+++ b/scripts/collect_website_traffic.py
@@ -0,0 +1,330 @@
+#!/usr/bin/env python3
+"""Aggregate privacy-safe funasr.com traffic metrics from Nginx access logs."""
+
+from __future__ import annotations
+
+import argparse
+import datetime as dt
+import glob
+import gzip
+import hashlib
+import json
+import os
+import re
+from collections import Counter, defaultdict
+from pathlib import Path
+from typing import Iterable, Iterator, NamedTuple, Optional, Sequence
+from urllib.parse import unquote, urlsplit
+
+
+LOG_PATTERN = re.compile(
+ r'^(?P<client_ip>\S+) \S+ \S+ \[(?P<timestamp>[^]]+)] '
+ r'"(?P<method>[A-Z]+) (?P<target>\S+) [^"]+" '
+ r'(?P<status>\d{3}) \S+ "(?P<referrer>[^"]*)" "(?P<user_agent>[^"]*)"$'
+)
+BOT_PATTERN = re.compile(
+ r"bot|spider|crawler|slurp|headless|lighthouse|pagespeed|curl/|wget/|"
+ r"python-requests|aiohttp|scrapy|go-http-client|uptime|monitoring|zgrab|masscan",
+ re.IGNORECASE,
+)
+STATIC_EXTENSIONS = {
+ ".avif",
+ ".css",
+ ".eot",
+ ".gif",
+ ".ico",
+ ".jpeg",
+ ".jpg",
+ ".js",
+ ".json",
+ ".map",
+ ".mp3",
+ ".mp4",
+ ".ogg",
+ ".pdf",
+ ".png",
+ ".svg",
+ ".txt",
+ ".wav",
+ ".webm",
+ ".webp",
+ ".woff",
+ ".woff2",
+ ".xml",
+ ".zip",
+}
+IGNORED_PREFIXES = ("/.well-known/", "/assets/", "/fonts/", "/images/", "/static/")
+DOC_PREFIXES = ("/docs/", "/en/docs/", "/guide/", "/en/guide/")
+DOC_FILENAMES = {
+ "agent.html",
+ "benchmark.html",
+ "cli-tutorial.html",
+ "download.html",
+ "ecosystem.html",
+ "llama-cpp.html",
+ "models.html",
+ "quickstart.html",
+ "vs-whisper.html",
+}
+SECTIONS = ("homepage", "docs", "blog", "donors", "other")
+
+
+class AccessRecord(NamedTuple):
+ client_ip: str
+ timestamp: dt.datetime
+ method: str
+ path: str
+ status: int
+ referrer_host: str
+ user_agent: str
+
+
+def normalize_path(target: str) -> str:
+ path = unquote(urlsplit(target).path or "/")
+ if not path.startswith("/"):
+ path = f"/{path}"
+ return path
+
+
+def referrer_host(referrer: str) -> str:
+ if not referrer or referrer == "-":
+ return ""
+ return (urlsplit(referrer).hostname or "").lower()
+
+
+def parse_access_log_line(line: str) -> Optional[AccessRecord]:
+ match = LOG_PATTERN.match(line.rstrip("\n"))
+ if match is None:
+ return None
+ try:
+ timestamp = dt.datetime.strptime(match.group("timestamp"), "%d/%b/%Y:%H:%M:%S %z")
+ status = int(match.group("status"))
+ except ValueError:
+ return None
+ return AccessRecord(
+ client_ip=match.group("client_ip"),
+ timestamp=timestamp,
+ method=match.group("method"),
+ path=normalize_path(match.group("target")),
+ status=status,
+ referrer_host=referrer_host(match.group("referrer")),
+ user_agent=match.group("user_agent"),
+ )
+
+
+def is_page_path(path: str) -> bool:
+ lowered = path.lower()
+ if lowered in {"/favicon.ico", "/robots.txt", "/sitemap.xml"}:
+ return False
+ if lowered.startswith(IGNORED_PREFIXES):
+ return False
+ return Path(lowered).suffix not in STATIC_EXTENSIONS
+
+
+def classify_page(path: str) -> str:
+ lowered = path.lower()
+ if lowered in {"/", "/index.html", "/en", "/en/", "/en/index.html"}:
+ return "homepage"
+ if lowered.startswith(("/blog/", "/en/blog/")):
+ return "blog"
+ if lowered in {"/donors.html", "/en/donors.html"}:
+ return "donors"
+ if lowered.startswith(DOC_PREFIXES) or Path(lowered).name in DOC_FILENAMES:
+ return "docs"
+ return "other"
+
+
+def anonymized_visitor(record: AccessRecord, salt: bytes) -> bytes:
+ value = f"{record.client_ip}\0{record.user_agent}".encode("utf-8", errors="replace")
+ return hashlib.blake2b(value, key=salt, digest_size=16).digest()
+
+
+def aggregate_records(
+ records: Iterable[Optional[AccessRecord]],
+ *,
+ now: dt.datetime,
+ days: int,
+ top_limit: int,
+ visitor_salt: Optional[bytes] = None,
+) -> dict:
+ if now.tzinfo is None:
+ raise ValueError("now must include a timezone")
+ if days <= 0:
+ raise ValueError("days must be positive")
+ if top_limit <= 0:
+ raise ValueError("top_limit must be positive")
+
+ cutoff = now - dt.timedelta(days=days)
+ salt = visitor_salt or os.urandom(16)
+ page_views = 0
+ visitors = set()
+ section_views = Counter()
+ section_visitors = defaultdict(set)
+ path_views = Counter()
+ path_visitors = defaultdict(set)
+ referrer_views = Counter()
+ filtered = Counter()
+
+ for record in records:
+ if record is None:
+ filtered["malformed"] += 1
+ continue
+ record_time = record.timestamp.astimezone(now.tzinfo)
+ if record_time < cutoff or record_time > now:
+ filtered["outside_window"] += 1
+ continue
+ if record.method not in {"GET", "HEAD"}:
+ filtered["non_page_methods"] += 1
+ continue
+ if record.status != 200:
+ filtered["non_success"] += 1
+ continue
+ if BOT_PATTERN.search(record.user_agent):
+ filtered["bots"] += 1
+ continue
+ if not is_page_path(record.path):
+ filtered["static_assets"] += 1
+ continue
+
+ visitor = anonymized_visitor(record, salt)
+ section = classify_page(record.path)
+ page_views += 1
+ visitors.add(visitor)
+ section_views[section] += 1
+ section_visitors[section].add(visitor)
+ path_views[record.path] += 1
+ path_visitors[record.path].add(visitor)
+ if record.referrer_host and not (
+ record.referrer_host == "funasr.com" or record.referrer_host.endswith(".funasr.com")
+ ):
+ referrer_views[record.referrer_host] += 1
+
+ sections = {
+ section: {
+ "page_views": section_views[section],
+ "unique_visitors": len(section_visitors[section]),
+ }
+ for section in SECTIONS
+ }
+ top_paths = [
+ {
+ "path": path,
+ "page_views": views,
+ "unique_visitors": len(path_visitors[path]),
+ }
+ for path, views in sorted(path_views.items(), key=lambda item: (-item[1], item[0]))[:top_limit]
+ ]
+ top_referrers = [
+ {"host": host, "page_views": views}
+ for host, views in sorted(referrer_views.items(), key=lambda item: (-item[1], item[0]))[:top_limit]
+ ]
+ filter_names = (
+ "malformed",
+ "outside_window",
+ "non_page_methods",
+ "non_success",
+ "bots",
+ "static_assets",
+ )
+ return {
+ "collected_at_utc": now.astimezone(dt.timezone.utc).isoformat(),
+ "window_start_utc": cutoff.astimezone(dt.timezone.utc).isoformat(),
+ "days": days,
+ "page_views": page_views,
+ "unique_visitors": len(visitors),
+ "sections": sections,
+ "top_paths": top_paths,
+ "top_referrers": top_referrers,
+ "filtered": {name: filtered[name] for name in filter_names},
+ "privacy": "Aggregate counts only; raw client IPs and user agents are never emitted.",
+ }
+
+
+def iter_log_lines(paths: Iterable[Path]) -> Iterator[str]:
+ for path in paths:
+ opener = gzip.open if path.suffix == ".gz" else open
+ with opener(path, "rt", encoding="utf-8", errors="replace") as handle:
+ yield from handle
+
+
+def resolve_log_paths(patterns: Sequence[str]) -> list[Path]:
+ paths = []
+ seen = set()
+ for pattern in patterns:
+ for match in sorted(glob.glob(pattern)):
+ path = Path(match)
+ if path.is_file() and path not in seen:
+ paths.append(path)
+ seen.add(path)
+ return paths
+
+
+def collect_traffic(
+ paths: Sequence[Path],
+ *,
+ now: dt.datetime,
+ days: int,
+ top_limit: int,
+) -> dict:
+ records = (parse_access_log_line(line) for line in iter_log_lines(paths))
+ result = aggregate_records(records, now=now, days=days, top_limit=top_limit)
+ result["source_file_count"] = len(paths)
+ return result
+
+
+def parse_now(value: Optional[str]) -> dt.datetime:
+ if value is None:
+ return dt.datetime.now(dt.timezone.utc)
+ parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
+ if parsed.tzinfo is None:
+ raise argparse.ArgumentTypeError("--now must include a timezone")
+ return parsed
+
+
+def render_text(metrics: dict) -> str:
+ lines = [
+ f"funasr.com traffic ({metrics['days']} days)",
+ f"Page views: {metrics['page_views']:,}",
+ f"Approximate unique visitors: {metrics['unique_visitors']:,}",
+ ]
+ for section in SECTIONS:
+ values = metrics["sections"][section]
+ lines.append(
+ f"{section}: {values['page_views']:,} views / "
+ f"{values['unique_visitors']:,} approximate visitors"
+ )
+ return "\n".join(lines)
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--log",
+ action="append",
+ required=True,
+ help="Access-log path or glob. Repeat for multiple files.",
+ )
+ parser.add_argument("--days", type=int, default=30, help="Trailing window length (default: 30).")
+ parser.add_argument("--top", type=int, default=10, help="Number of top paths/referrers (default: 10).")
+ parser.add_argument("--now", help="Timezone-aware ISO timestamp for reproducible backfills.")
+ parser.add_argument("--format", choices=("text", "json"), default="text")
+ args = parser.parse_args(argv)
+
+ try:
+ now = parse_now(args.now)
+ paths = resolve_log_paths(args.log)
+ if not paths:
+ parser.error("no access-log files matched --log")
+ metrics = collect_traffic(paths, now=now, days=args.days, top_limit=args.top)
+ except (OSError, ValueError) as error:
+ parser.error(str(error))
+
+ if args.format == "json":
+ print(json.dumps(metrics, ensure_ascii=False, indent=2))
+ else:
+ print(render_text(metrics))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/tests/test_collect_website_traffic.py b/tests/test_collect_website_traffic.py
new file mode 100644
index 000000000..5f9593ab5
--- /dev/null
+++ b/tests/test_collect_website_traffic.py
@@ -0,0 +1,161 @@
+import gzip
+import importlib.util
+import json
+import subprocess
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+
+def load_module():
+ script = Path(__file__).resolve().parents[1] / "scripts" / "collect_website_traffic.py"
+ spec = importlib.util.spec_from_file_location("collect_website_traffic", script)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def access_line(
+ ip,
+ timestamp,
+ path,
+ *,
+ status=200,
+ referrer="-",
+ user_agent="Mozilla/5.0 Chrome/142.0",
+):
+ return (
+ f'{ip} - - [{timestamp}] "GET {path} HTTP/2.0" {status} 123 '
+ f'"{referrer}" "{user_agent}"\n'
+ )
+
+
+def test_parse_access_log_line_normalizes_path_and_referrer():
+ module = load_module()
+ record = module.parse_access_log_line(
+ access_line(
+ "203.0.113.5",
+ "23/Jul/2026:11:00:00 +0800",
+ "/docs/guide.html?source=release",
+ referrer="https://www.google.com/search?q=funasr",
+ )
+ )
+
+ assert record.client_ip == "203.0.113.5"
+ assert record.timestamp.isoformat() == "2026-07-23T11:00:00+08:00"
+ assert record.method == "GET"
+ assert record.path == "/docs/guide.html"
+ assert record.status == 200
+ assert record.referrer_host == "www.google.com"
+
+
+def test_aggregate_filters_bots_assets_failures_and_old_requests():
+ module = load_module()
+ lines = [
+ access_line("203.0.113.10", "23/Jul/2026:09:00:00 +0800", "/"),
+ access_line("203.0.113.10", "23/Jul/2026:09:01:00 +0800", "/docs/quickstart.html"),
+ access_line(
+ "203.0.113.20",
+ "23/Jul/2026:09:02:00 +0800",
+ "/en/blog/local-asr.html",
+ referrer="https://news.ycombinator.com/item?id=1",
+ ),
+ access_line("203.0.113.20", "23/Jul/2026:09:03:00 +0800", "/donors.html"),
+ access_line(
+ "198.51.100.1",
+ "23/Jul/2026:09:04:00 +0800",
+ "/docs/quickstart.html",
+ user_agent="Googlebot/2.1",
+ ),
+ access_line("198.51.100.2", "23/Jul/2026:09:05:00 +0800", "/assets/app.js"),
+ access_line("198.51.100.3", "23/Jul/2026:09:06:00 +0800", "/docs/missing.html", status=404),
+ access_line("198.51.100.4", "20/Jun/2026:09:07:00 +0800", "/docs/old.html"),
+ ]
+ records = [module.parse_access_log_line(line) for line in lines]
+
+ metrics = module.aggregate_records(
+ records,
+ now=datetime(2026, 7, 23, 12, 0, tzinfo=timezone.utc),
+ days=30,
+ top_limit=10,
+ )
+
+ assert metrics["page_views"] == 4
+ assert metrics["unique_visitors"] == 2
+ assert metrics["sections"] == {
+ "homepage": {"page_views": 1, "unique_visitors": 1},
+ "docs": {"page_views": 1, "unique_visitors": 1},
+ "blog": {"page_views": 1, "unique_visitors": 1},
+ "donors": {"page_views": 1, "unique_visitors": 1},
+ "other": {"page_views": 0, "unique_visitors": 0},
+ }
+ assert metrics["filtered"]["bots"] == 1
+ assert metrics["filtered"]["static_assets"] == 1
+ assert metrics["filtered"]["non_success"] == 1
+ assert metrics["filtered"]["outside_window"] == 1
+ assert metrics["top_referrers"] == [{"host": "news.ycombinator.com", "page_views": 1}]
+ serialized = json.dumps(metrics)
+ assert "203.0.113" not in serialized
+ assert "Mozilla" not in serialized
+
+
+def test_iter_log_lines_supports_plain_and_gzip_files(tmp_path):
+ module = load_module()
+ plain = tmp_path / "access.log"
+ compressed = tmp_path / "access.log.1.gz"
+ plain.write_text("plain\n")
+ with gzip.open(compressed, "wt") as handle:
+ handle.write("compressed\n")
+
+ assert list(module.iter_log_lines([plain, compressed])) == ["plain\n", "compressed\n"]
+
+
+def test_classify_owned_product_guides_as_docs():
+ module = load_module()
+
+ for path in (
+ "/quickstart.html",
+ "/en/quickstart.html",
+ "/models.html",
+ "/en/models.html",
+ "/vs-whisper.html",
+ "/en/vs-whisper.html",
+ "/ecosystem.html",
+ "/en/llama-cpp.html",
+ ):
+ assert module.classify_page(path) == "docs"
+
+
+def test_cli_outputs_aggregate_json_without_identifiers(tmp_path):
+ script = Path(__file__).resolve().parents[1] / "scripts" / "collect_website_traffic.py"
+ log = tmp_path / "access.log"
+ log.write_text(
+ access_line(
+ "203.0.113.55",
+ "23/Jul/2026:09:00:00 +0800",
+ "/docs/deployment.html",
+ )
+ )
+
+ result = subprocess.run(
+ [
+ sys.executable,
+ str(script),
+ "--log",
+ str(log),
+ "--now",
+ "2026-07-23T12:00:00+00:00",
+ "--days",
+ "30",
+ "--format",
+ "json",
+ ],
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ payload = json.loads(result.stdout)
+
+ assert payload["page_views"] == 1
+ assert payload["sections"]["docs"]["page_views"] == 1
+ assert "203.0.113.55" not in result.stdout
diff --git a/tests/test_growth_plan_doc.py b/tests/test_growth_plan_doc.py
index 7113b0074..c6cd49bb1 100644
--- a/tests/test_growth_plan_doc.py
+++ b/tests/test_growth_plan_doc.py
@@ -363,23 +363,36 @@ def test_growth_plan_records_homepage_entrypoint_completion():
assert "[ ] Update homepage hero and docs entry points" not in text
-def test_growth_plan_records_metrics_tracking_progress():
+def test_growth_plan_records_metrics_tracking_completion():
text = PLAN.read_text()
required_markers = [
- "[ ] Track stars, PyPI downloads, issue volume, and docs traffic.",
+ "[x] Track stars, PyPI downloads, issue volume, and docs traffic.",
"PyPIStats",
"`downloads_last_7_days`",
"`downloads_last_30_days`",
"`collect_growth_metrics.py`",
"GitHub stars, open issues, and open pull requests",
- "`116,407` downloads in 7 days",
- "`442,556` downloads in 30 days",
- "`2026-07-21`",
+ "`117,850` downloads in 7 days",
+ "`445,755` downloads in 30 days",
+ "`2026-07-22`",
"pypi-downloads-funasr.json",
"`stale_available`",
"SSL/429 outages",
- "docs traffic remains unavailable until an analytics credential or export is connected",
+ "`scripts/collect_website_traffic.py`",
+ "`python scripts/collect_website_traffic.py --log '/var/log/nginx/access.log*' --days 30 --format json`",
+ "plain or gzip Nginx logs",
+ "raw client IPs and user agents are never emitted",
+ "over 15 rotated logs",
+ "`88,239` page views / `47,084` approximate unique visitors",
+ "docs at `15,501` / `11,843`",
+ "blog at `27,901` / `22,716`",
+ "`21,915` / `12,668`",
+ "docs at `4,638` / `3,612`",
+ "blog at `6,901` / `5,590`",
]
for marker in required_markers:
assert marker in text
+
+ assert "[ ] Track stars, PyPI downloads, issue volume, and docs traffic." not in text
+ assert "docs traffic remains unavailable" not in text