You're reading this. The skill predicted it. A workflow that turns every post into a calibrated experiment—score, blind-predict, retro, evolve. The future doesn't reward effort, it rewards those who see the pattern first. 1M followers in a month — not luck, system.
Python
feat(xhs-explore): public-page fallback, archive/summarize, image download (#46)
SingularGuyLeBorn22952821ca3f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9e7b8fc..fa93931 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,20 @@ All notable changes to cheat-on-content will be documented here.## [Unreleased]+### Added — xhs-explore adapter: 公开页兜底、正文/图片归档、账号汇总++- 融合 [xhs-analytics](https://github.com/SingularGuyLeBorn/xhs-analytics) 的公开页解析能力:+ - `crawler.fetch_public_note()` 无登录解析 `window.__INITIAL_STATE__`,补全正文、图片 URL、标签。+ - `crawler.fetch_public_comments()` 当前台 API 拿不到评论时,用公开页 top 评论兜底(通常 ~10 条)。+ - `crawler.download_image()` 异步下载笔记图片。+- `review.py` 新增子命令:+ - `archive <notes.json> [output_root] [limit]`:批量归档已发布笔记正文与图片。+ - `summarize [out_dir]`:基于创作者中心最近 50 条 + 公开页标签生成账号级汇总。+- `review.py note` 支持环境变量 `XHS_DOWNLOAD_IMAGES=1` 下载图片到 `videos/<...>/images/`。+- `renderer.py` 在 `report.md` 中新增「正文 + 标签」与「图片」块。+- `requirements.txt` 增加 `requests>=2.28`。+- `Session.open()` 在未下载 Playwright Chromium 时自动 fallback 到系统 Chrome(`channel="chrome"`)。+### Added — 受众画像 persona(cheat-persona skill + audience.md)**动机**:用户选题 / 写稿时缺一个清晰的"谁在看"的镜子。新增受众画像功能——从复盘评论数据聚类出账号真实受众。diff --git a/adapters/perf-data/xhs-explore/README.md b/adapters/perf-data/xhs-explore/README.mdindex 6f60cd8..7a46c71 100644--- a/adapters/perf-data/xhs-explore/README.md+++ b/adapters/perf-data/xhs-explore/README.md@@ -4,6 +4,8 @@> **来源**:照搬 `douyin-session` adapter 的架构(Playwright 持久化登录态 + 被动拦截 XHR),> 接口路径与字段参考 NanmiCoder/MediaCrawler 与 ReaJason/xhs。已在真实创作者账号端到端验证(2026-05)。+> 2026-06 融合 [xhs-analytics](https://github.com/SingularGuyLeBorn/xhs-analytics) 的公开页解析能力:+> 无登录抓取正文 / 图片 / 标签 / top 评论兜底,以及批量归档与账号汇总命令。---@@ -12,13 +14,14 @@小红书反爬靠 `x-s`/`x-t`/`x-s-common` 签名 + `xsec_token`——纯 HTTP / requests 拿不到数据。xhs-explore 用 **Playwright + 持久化 Chromium context** 模拟真实浏览器,**不逆向签名、不伪造请求**,-让页面自己发带签名的请求,我们只被动拦截返回的 JSON:+让页面自己发带签名的请求,我们只被动拦截返回的 JSON;同时用公开页 `__INITIAL_STATE__` 做正文/图片/评论兜底:- 首次扫码登录创作者中心,cookie 存在**你的内容项目根目录** `.auth-xhs/`- 之后每次抓取复用 cookie-- 抓两路数据:+- 抓三路数据:1. **创作者中心 galaxy 接口**(`/api/galaxy/v2/creator/note/user/posted`)— 你**自己**笔记的运营数据:观看/点赞/收藏/评论/分享,**不需要 xsec_token**,最稳。列表每条自带 `xsec_token`,抓自己的笔记无需手动粘带 token 的链接2. **前台 web API**(`/api/sns/web/v1/feed` + `/api/sns/web/v2/comment/page`)— 拿确认字段的点赞/收藏数 + 评论文本+ 3. **公开页兜底**(`www.xiaohongshu.com/explore/<note_id>` 的 `window.__INITIAL_STATE__`)— **无需登录**即可补全正文、图片 URL、标签;当前台 API 拿不到评论时,用公开页里的 top 评论兜底(通常 ~10 条)输出写到**你的内容项目** `videos/<...>/report.md`。调试产物(URL dump / 截图 / galaxy 原始 JSON)写到 `.cheat-cache/xhs-explore-debug/`。@@ -42,8 +45,8 @@ cd ~/Documents/my-channelpython3 -m venv .venvsource .venv/bin/activate # Windows: .venv\Scripts\activate-# 3. 装 Playwright + Chromium-pip install playwright>=1.44+# 3. 装依赖(Playwright + Chromium + requests)+pip install -r "$ADAPTER/requirements.txt"playwright install chromium# 4. 首次扫码登录小红书创作者中心@@ -70,9 +73,18 @@ python "$ADAPTER/review.py" list# 抓特定笔记python "$ADAPTER/review.py" note <note_id> [script.txt]-# 输出在 当前目录/videos/<日期>_<标题>/report.md+# 抓特定笔记并下载图片到 videos/<...>/images/+XHS_DOWNLOAD_IMAGES=1 python "$ADAPTER/review.py" note <note_id>++# 批量归档已发布笔记的正文+图片(公开页解析,无登录)+python "$ADAPTER/review.py" archive my_notes.json data/raw/notes/archive 50++# 账号级汇总(基于创作者中心最近 50 条 + 公开页标签)+python "$ADAPTER/review.py" summarize reports```+输出在 当前目录/videos/<日期>_<标题>/report.md;archive 输出在 `<output_root>/<note_id>/`。+## 怎么拿到 note_id小红书笔记 URL:@@ -86,7 +98,10 @@ cheat-publish 登记发布时把 note_id 存到 prediction header,cheat-retro由 `renderer.py` 生成:- 笔记元信息(标题、发布时间、链接、IP 归属)- 数据快照(曝光/浏览、点赞、收藏、评论、分享 + 派生比率:赞曝比 / 藏曝比 / 评曝比 / 分曝比 + 涨粉)+- 正文 + 标签(公开页兜底时才有)+- 图片(本地路径优先,否则 URL)- galaxy 原始字段 JSON(debug / 接口改版时核对字段用)+- 原始稿子(cheat-retro 传入)- Top 评论(按赞数排序,含文本 + 赞数 + IP)## 失败模式(按概率从高到低)@@ -96,16 +111,17 @@ cheat-publish 登记发布时把 note_id 存到 prediction header,cheat-retro| 曝光显示 0 但 JSON 有数 | galaxy 字段 key 不在候选列表 | 看 report.md 里 galaxy JSON,把真实 key 加进 `_normalize_note` || `ensure_login` 超时 | cookie 过期 | 重跑 `python crawler.py login` || 笔记列表为空 | 创作者中心改了 galaxy 接口路径 | 看 `.cheat-cache/xhs-explore-debug/creator_urls.txt`,更新 `GALAXY_*_KEYS` |-| 评论抓不到 | xsec_token 缺失 / 评论被关 | 看 `frontend_urls.txt`;拿不到就降级 manual 粘评论(cheat-retro 会提示) |+| 评论抓不到 | xsec_token 缺失 / 评论被关 / 登录过期 | 先看 `frontend_urls.txt`;再用公开页兜底 top 评论;最后 manual 粘评论 |+| 正文/图片/标签缺失 | 公开页也解析失败 | 检查 note_id 与 xsec_token;网页结构改版时更新 `_extract_initial_state` || Chromium 崩溃 | 内存不足 | 关其他 Chromium;`playwright install chromium --force` |**关键现实**:小红书接口 path(`/feed`、`/comment/page`、galaxy)相对稳定,但签名和 xsec_token 机制常变。-本 adapter 用被动拦截规避签名风险——最易抖动处是 **galaxy 创作者接口**和**评论回复**。+本 adapter 用被动拦截 + 公开页兜底规避签名风险——最易抖动处是 **galaxy 创作者接口**和**评论回复**。## 稳定性等级★★ — Playwright + 登录态能扛比纯 HTTP 强得多的反爬,但仍受小红书前端改版影响。-评论路径(依赖 xsec_token)比创作者数据路径脆,必要时降级 manual。+评论路径(依赖 xsec_token)比创作者数据路径脆,必要时降级 manual 或公开页兜底。## 风险提示@@ -119,9 +135,9 @@ cheat-publish 登记发布时把 note_id 存到 prediction header,cheat-retro```adapters/perf-data/xhs-explore/├── README.md # 本文件-├── requirements.txt # playwright>=1.44-├── crawler.py # 抓取核心(galaxy 笔记列表+数据 / 前台 interact+评论)-├── review.py # CLI 入口(login / list / note <note_id>)+├── requirements.txt # playwright>=1.44, requests>=2.28+├── crawler.py # 抓取核心(galaxy / 前台 API / 公开页兜底 / 图片下载)+├── review.py # CLI 入口(login / list / note / archive / summarize)├── renderer.py # 把抓回的 JSON 渲染成 report.md├── paths.py # 项目根 / .auth-xhs / debug 路径解析└── run.sh # cheat-retro 调用的 wrapperdiff --git a/adapters/perf-data/xhs-explore/crawler.py b/adapters/perf-data/xhs-explore/crawler.pyindex e87473c..9e3f838 100644--- a/adapters/perf-data/xhs-explore/crawler.py+++ b/adapters/perf-data/xhs-explore/crawler.py@@ -9,14 +9,22 @@- 创作者**自己的**笔记数据走 galaxy 接口(不需要 xsec_token),是最稳的主路。- 评论走前台 web API(需要 xsec_token),让页面自己导航触发带 token 的请求;拿不到就优雅降级(report.md 标 comments_unavailable,cheat-retro 回落到 manual 粘评论)。++本文件新增的能力(来自 xhs-analytics 的公开页解析):+- fetch_public_note: 无登录解析 explore 页面 __INITIAL_STATE__,拿正文 / 图片 / 标签。+- fetch_public_comments: 公开页兜底 top 评论。+- download_image: 下载笔记图片到本地。"""from __future__ import annotationsimport asyncioimport json+import re+import urllib.parsefrom pathlib import Pathfrom typing import Any+import requestsfrom playwright.async_api import BrowserContext, Page, Response, async_playwrightfrom paths import auth_dir, debug_dir@@ -35,6 +43,11 @@ GALAXY_NOTE_STATS_KEYS = (FEED_KEY = "/api/sns/web/v1/feed"COMMENT_KEY = "/api/sns/web/v2/comment/page"+DEFAULT_USER_AGENT = (+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "+ "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"+)+class Session:"""单浏览器会话,按顺序跑多步抓取。"""@@ -48,12 +61,22 @@ class Session:pw = await async_playwright().start()auth_path = auth_dir()auth_path.mkdir(parents=True, exist_ok=True)- ctx = await pw.chromium.launch_persistent_context(- user_data_dir=str(auth_path),- headless=headless,- viewport={"width": 1440, "height": 900},- args=["--disable-blink-features=AutomationControlled"],- )+ common_kwargs = {+ "user_data_dir": str(auth_path),+ "headless": headless,+ "viewport": {"width": 1440, "height": 900},+ "args": ["--disable-blink-features=AutomationControlled"],+ }+ try:+ ctx = await pw.chromium.launch_persistent_context(**common_kwargs)+ except Exception as exc:+ # 部分环境只装了系统 Chrome,没下载 Playwright Chromium,fallback 到 channel=chrome+ try:+ ctx = await pw.chromium.launch_persistent_context(+ **common_kwargs, channel="chrome"+ )+ except Exception:+ raise excreturn cls(ctx, pw)async def close(self) -> None:@@ -108,24 +131,43 @@ async def _acquire_web_session(page: Page) -> None:pass-async def ensure_login(timeout_s: int = 300) -> bool:- """扫码登录创作者中心;检测到创作者登录态后顺便换取 web_session,然后自动关闭。"""+async def ensure_login(timeout_s: int = 180, max_refresh: int = 5) -> bool:+ """扫码登录创作者中心;检测到创作者登录态后顺便换取 web_session,然后自动关闭。++ 二维码本身会过期,所以用 timeout_s 作为单次等待上限,超时后自动刷新页面重新出码,+ 最多刷新 max_refresh 次。用户有充足时间扫码。+ """sess = await Session.open()try:page = await sess.ctx.new_page()await page.goto(CREATOR_HOME)- print(f"[登录] 在弹出的 Chromium 窗口里扫码登录小红书创作者中心。最多等 {timeout_s} 秒……")- for i in range(timeout_s):- try:- if await _creator_logged_in(sess.ctx) and "login" not in page.url:- print(f"[登录] ✓ 创作者中心登录态已确认(用时 {i}s)")- await _acquire_web_session(page)- await asyncio.sleep(1)- return True- except Exception:- pass- await asyncio.sleep(1)- print("[登录] 超时未检测到登录态。")+ print(f"[登录] 在弹出的 Chromium 窗口里扫码登录小红书创作者中心。每次二维码有效期约 {timeout_s} 秒,超时自动刷新。")++ for refresh in range(max_refresh + 1):+ for i in range(timeout_s):+ try:+ if await _creator_logged_in(sess.ctx) and "login" not in page.url:+ print(f"[登录] ✓ 创作者中心登录态已确认(总用时 {refresh * timeout_s + i}s)")+ await _acquire_web_session(page)+ await asyncio.sleep(1)+ return True+ except Exception:+ pass++ # 每 30 秒提醒一次,避免用户以为卡死+ if i > 0 and i % 30 == 0:+ print(f"[登录] 已等待 {i} 秒,请用小红书 App 扫码(或等待自动刷新二维码)……")++ await asyncio.sleep(1)++ if refresh < max_refresh:+ print(f"[登录] 本次二维码未扫码或已过期,正在刷新页面重新出码(第 {refresh + 1}/{max_refresh} 次刷新)……")+ try:+ await page.reload(wait_until="domcontentloaded", timeout=30000)+ except Exception:+ await page.goto(CREATOR_HOME)++ print("[登录] 超过最大刷新次数仍未检测到登录态,已停止。如需继续请重新运行本命令。")return Falsefinally:await sess.close()@@ -153,6 +195,10 @@ async def fetch_recent_notes(sess: Session, limit: int = 50) -> list[dict]:try:await page.goto(CREATOR_NOTE_MANAGER, wait_until="domcontentloaded", timeout=60000)await asyncio.sleep(8)+ # cookie 过期时会被 302 到登录页+ if "login" in page.url or "redirectReason=401" in page.url:+ print("[登录] 创作者中心已跳转登录页,cookie 可能已过期。请运行:python crawler.py login")+ return []for _ in range(4):await page.evaluate("window.scrollBy(0, 1200)")await asyncio.sleep(1.5)@@ -247,7 +293,213 @@ def _to_int(x: Any) -> int:return 0-async def fetch_note_frontend(sess: Session, note_id: str, note_url: str | None = None) -> dict:+# ---------------------------------------------------------------------------+# 公开页解析(来自 xhs-analytics 的核心能力):无登录拿正文、图片、标签、评论兜底+# ---------------------------------------------------------------------------++def _extract_initial_state(html: str) -> dict | None:+ """从小红书 explore 页面 HTML 中解析 window.__INITIAL_STATE__。"""+ marker = "window.__INITIAL_STATE__="+ start = html.find(marker)+ if start == -1:+ return None+ script_start = html.rfind("<script", 0, start)+ script_end = html.find("</script>", start)+ if script_start == -1 or script_end == -1:+ return None+ script = html[script_start:script_end]+ assign_start = script.find(marker) + len(marker)+ json_str = script[assign_start:]+ json_str = re.sub(r":\s*undefined\s*([,}\]])", r":null\1", json_str)+ try:+ return json.loads(json_str)+ except Exception:+ return None+++def _image_url(img: dict) -> str:+ """从 imageList 元素中提取可用的图片 URL。"""+ if not isinstance(img, dict):+ return ""+ for key in ("urlDefault", "url"):+ if img.get(key):+ return img[key]+ for info in img.get("infoList", []) or []:+ if isinstance(info, dict) and info.get("imageScene") == "WB_DFT" and info.get("url"):+ return info["url"]+ for info in img.get("infoList", []) or []:+ if isinstance(info, dict) and info.get("url"):+ return info["url"]+ return ""+++def _xsec_token_from_url(url: str | None) -> str:+ """从笔记 URL 的 query 中拆出 xsec_token(保留原编码)。"""+ if not url:+ return ""+ parsed = urllib.parse.urlparse(url)+ return urllib.parse.unquote(urllib.parse.parse_qs(parsed.query).get("xsec_token", [""])[0])+++def fetch_public_note(note_id: str, xsec_token: str) -> dict:+ """无登录抓取 explore 公开页,解析 __INITIAL_STATE__。++ 返回 dict:success, note_id, title, desc/body, images, tags, time, counts, raw。+ """+ token = urllib.parse.unquote(xsec_token)+ url = (+ f"https://www.xiaohongshu.com/explore/{note_id}"+ f"?xsec_token={urllib.parse.quote(token)}"+ f"&xsec_source=pc_creatormng"+ )+ headers = {+ "User-Agent": DEFAULT_USER_AGENT,+ "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",+ "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",+ }+ r = requests.get(url, headers=headers, timeout=20)+ r.raise_for_status()++ state = _extract_initial_state(r.text)+ if not state:+ raise ValueError("无法解析页面初始状态")++ note_detail_map = state.get("note", {}).get("noteDetailMap", {})+ if note_id not in note_detail_map:+ raise ValueError("页面未返回该笔记数据")++ raw = note_detail_map[note_id]+ note = raw.get("note", {})+ interact = note.get("interactInfo", {})++ return {+ "success": True,+ "note_id": note_id,+ "title": note.get("title", ""),+ "desc": note.get("desc", ""),+ "body": note.get("desc", ""),+ "type": note.get("type", ""),+ "time": note.get("time", 0),+ "images": [_image_url(img) for img in note.get("imageList", [])],+ "tags": [tag.get("name", "") for tag in note.get("tagList", []) if tag.get("name")],+ "counts": {+ "liked": _to_int(interact.get("likedCount")),+ "collected": _to_int(interact.get("collectedCount")),+ "comment": _to_int(interact.get("commentCount")),+ "shared": _to_int(interact.get("shareCount")),+ },+ "raw": raw,+ }+++def _normalize_public_comment(c: dict) -> dict:+ """把 __INITIAL_STATE__ / edith 接口里的评论字段统一成 adapter 格式。"""+ user = c.get("user_info") or c.get("userInfo") or {}+ like_count = c.get("like_count") or c.get("likeCount") or 0+ sub_count = c.get("sub_comment_count") or c.get("subCommentCount") or 0+ return {+ "cid": str(c.get("id") or c.get("comment_id") or ""),+ "text": c.get("content") or "",+ "like_count": _to_int(like_count),+ "sub_comment_count": _to_int(sub_count),+ "create_time": c.get("create_time") or c.get("createTime") or 0,+ "user_name": user.get("nickname") or "",+ "ip_label": c.get("ip_location") or c.get("ipLocation") or "",+ }+++def fetch_public_comments(note_id: str, xsec_token: str, max_comments: int = 20) -> list[dict]:+ """公开页 __INITIAL_STATE__ 兜底 top 评论(通常 ~10 条)。"""+ token = urllib.parse.unquote(xsec_token)+ url = (+ f"https://www.xiaohongshu.com/explore/{note_id}"+ f"?xsec_token={urllib.parse.quote(token)}"+ f"&xsec_source=pc_creatormng"+ )+ headers = {+ "User-Agent": DEFAULT_USER_AGENT,+ "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",+ "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",+ }+ r = requests.get(url, headers=headers, timeout=20)+ r.raise_for_status()+ state = _extract_initial_state(r.text)+ if not state:+ return []++ raw = state.get("note", {}).get("noteDetailMap", {}).get(note_id, {})+ comments = []+ # __INITIAL_STATE__ 里的评论路径可能是 comments.list 或 commentsList+ for key in ("comments", "commentsList"):+ container = raw.get(key) if isinstance(raw, dict) else None+ if isinstance(container, dict):+ arr = container.get("list") or container.get("comments") or []+ if isinstance(arr, list):+ comments.extend([_normalize_public_comment(c) for c in arr])+ elif isinstance(container, list):+ comments.extend([_normalize_public_comment(c) for c in container])++ seen = set()+ dedup = []+ for c in comments:+ if not c["cid"] or c["cid"] in seen:+ continue+ seen.add(c["cid"])+ dedup.append(c)+ dedup.sort(key=lambda x: x["like_count"], reverse=True)+ return dedup[:max_comments]+++async def download_image(url: str, dest: Path, timeout: int = 30) -> bool:+ """下载单张图片到 dest(会根据 Content-Type 修正扩展名)。异步封装。"""+ if not url:+ return False+ dest = Path(dest)+ dest.parent.mkdir(parents=True, exist_ok=True)++ def _download() -> bool:+ headers = {+ "User-Agent": DEFAULT_USER_AGENT,+ "Accept": "image/webp,image/apng,image/*,*/*;q=0.8",+ "Referer": "https://www.xiaohongshu.com/",+ }+ r = requests.get(url, headers=headers, timeout=timeout, stream=True)+ r.raise_for_status()++ parsed = urllib.parse.urlparse(url)+ path = urllib.parse.unquote(parsed.path)+ ext = Path(path).suffix.lower()+ if ext not in {".jpg", ".jpeg", ".png", ".webp", ".gif", ".avif"}:+ ct = r.headers.get("Content-Type", "").lower()+ if "webp" in ct:+ ext = ".webp"+ elif "png" in ct:+ ext = ".png"+ elif "gif" in ct:+ ext = ".gif"+ else:+ ext = ".jpg"+ dest_final = dest.with_suffix(ext)++ with open(dest_final, "wb") as f:+ for chunk in r.iter_content(chunk_size=8192):+ if chunk:+ f.write(chunk)+ return True++ try:+ return await asyncio.to_thread(_download)+ except Exception as exc:+ print(f"[下载图片] 失败 {url}: {exc}")+ return False+++async def fetch_note_frontend(+ sess: Session,+ note_id: str,+ note_url: str | None = None,+ xsec_token: str | None = None,+) -> dict:"""打开前台笔记页 → 拦截 feed(interact_info 确认字段)+ comment/page。前台需要 xsec_token + 登录态(web_session)。@@ -255,6 +507,8 @@ async def fetch_note_frontend(sess: Session, note_id: str, note_url: str | None→ 直接用它导航,最稳。- 否则退回裸 explore URL(仅对已登录账号访问自己笔记可能可行)。token 缺失 / 未登录 → dump 并降级(评论留给 manual)。++ 拦截不到评论时,会用公开页 __INITIAL_STATE__ 里的 top 评论兜底。"""feed: dict = {}comments: list[dict] = []@@ -306,6 +560,20 @@ async def fetch_note_frontend(sess: Session, note_id: str, note_url: str | Nonestagnant = 0last = cur+ # 兜底:公开页 __INITIAL_STATE__ 里的 top 评论+ if not comments:+ xsec = xsec_token or _xsec_token_from_url(note_url)+ if xsec:+ try:+ public_comments = await asyncio.to_thread(+ fetch_public_comments, note_id, xsec, max_comments=20+ )+ if public_comments:+ comments = public_comments+ print(f" 公开页兜底 {len(comments)} 条评论")+ except Exception as exc:+ print(f"[诊断] 公开页评论兜底失败:{exc}")+if not comments:dbg = debug_dir()dbg.mkdir(parents=True, exist_ok=True)@@ -389,8 +657,29 @@ def _dump(urls: list[str], url_file: str, captured: list[dict], cap_file: str) -pass+def _merge_public_note(note: dict, public: dict) -> None:+ """用公开页数据补全 note(不覆盖已有的 galaxy 运营数据)。"""+ if not public.get("success"):+ return+ for key in ("title", "desc", "body", "type", "images", "tags"):+ if public.get(key) and not note.get(key):+ note[key] = public[key]+ if public.get("time") and not note.get("create_time"):+ note["create_time"] = _to_int(public["time"])+ counts = public.get("counts") or {}+ mapping = {+ "like_count": counts.get("liked"),+ "collect_count": counts.get("collected"),+ "comment_count": counts.get("comment"),+ "share_count": counts.get("shared"),+ }+ for k, v in mapping.items():+ if v and not note.get(k):+ note[k] = _to_int(v)++async def fetch_all(note_id: str, note_url: str | None = None) -> dict:- """一个会话跑完笔记列表(含 galaxy 指标)+ 前台 interact + 评论。"""+ """一个会话跑完笔记列表(含 galaxy 指标)+ 前台 interact + 评论 + 公开页正文/图片兜底。"""sess = await Session.open()try:print(" → 打开创作者中心,拉笔记列表 + 运营数据")@@ -408,8 +697,19 @@ async def fetch_all(note_id: str, note_url: str | None = None) -> dict:front_url = (f"https://www.xiaohongshu.com/explore/{note_id}"f"?xsec_token={note['xsec_token']}&xsec_source=pc_creatormng")+ xsec = _xsec_token_from_url(front_url) or note.get("xsec_token", "")++ # 先用公开页补正文/图片/标签(无登录、低成本),不覆盖 galaxy 已确认的计数+ if xsec:+ try:+ public = await asyncio.to_thread(fetch_public_note, note_id, xsec)+ _merge_public_note(note, public)+ print(" ✓ 公开页正文/图片/标签已补全")+ except Exception as exc:+ print(f"[诊断] 公开页兜底失败:{exc}")+print(" → 打开前台笔记页抓 interact + 评论")- front = await fetch_note_frontend(sess, note_id, note_url=front_url)+ front = await fetch_note_frontend(sess, note_id, note_url=front_url, xsec_token=xsec)# 前台 interact 字段是确认的——用它补全/覆盖 galaxy 里可能缺的计数for k in ("like_count", "collect_count", "comment_count", "share_count"):if front["interact"].get(k):diff --git a/adapters/perf-data/xhs-explore/renderer.py b/adapters/perf-data/xhs-explore/renderer.pyindex 243aa8b..f7ff7a9 100644--- a/adapters/perf-data/xhs-explore/renderer.py+++ b/adapters/perf-data/xhs-explore/renderer.py@@ -41,6 +41,11 @@ def _ratio(num, denom) -> str:return f"{num/denom*100:.2f}%"+def _escape_md(text: str) -> str:+ """轻度转义,避免正文里的 # 被误认为标题。"""+ return text.replace("\n", " \n").replace("#", "\\#")++def render_report(note: dict, script: str, comments: list[dict]) -> str:lines: list[str] = []title = note.get("title") or "(无标题)"@@ -68,6 +73,29 @@ def render_report(note: dict, script: str, comments: list[dict]) -> str:lines.append(f"- 涨粉:{_fmt_num(note.get('fans_inc'))}")lines.append("")+ # 正文 + 标签(公开页兜底时才有)+ body = note.get("body") or note.get("desc") or ""+ tags = note.get("tags") or []+ if body or tags:+ lines.append("## 正文")+ lines.append("")+ if tags:+ lines.append(" ".join(f"#{t}" for t in tags))+ lines.append("")+ lines.append(_escape_md(body) if body else "(无)")+ lines.append("")++ # 图片(本地优先,否则用 URL)+ image_paths = note.get("image_paths") or []+ image_urls = note.get("images") or []+ images = image_paths or image_urls+ if images:+ lines.append("## 图片")+ lines.append("")+ for i, img in enumerate(images, 1):+ lines.append(f"")+ lines.append("")+# galaxy 原始 JSON——首跑时用来确认曝光字段的真实 key 名raw = note.get("raw")if raw:diff --git a/adapters/perf-data/xhs-explore/requirements.txt b/adapters/perf-data/xhs-explore/requirements.txtindex 86a8fae..99fd92d 100644--- a/adapters/perf-data/xhs-explore/requirements.txt+++ b/adapters/perf-data/xhs-explore/requirements.txt@@ -1 +1,2 @@playwright>=1.44+requests>=2.28diff --git a/adapters/perf-data/xhs-explore/review.py b/adapters/perf-data/xhs-explore/review.pyindex a364459..0305480 100644--- a/adapters/perf-data/xhs-explore/review.py+++ b/adapters/perf-data/xhs-explore/review.py@@ -5,17 +5,48 @@python review.py login # 仅登录(首次扫码)python review.py list # 列最近笔记(验证登录态)python review.py note <note_id> [script.txt] # 直接指定笔记+ python review.py archive <notes.json> [output_root] [limit] # 批量归档正文+图片+ python review.py summarize [out_dir] # 账号级汇总(基于最近 50 条)++archive / summarize 能力来自 xhs-analytics 的公开页解析,可无登录抓取正文与图片。"""from __future__ import annotationsimport asyncio+import io+import json+import osimport reimport sys+import time+from collections import defaultdictfrom pathlib import Pathimport crawlerimport renderer-from paths import videos_dir+from paths import runtime_project_root, videos_dir++# Windows Git Bash 等默认 GBK 控制台,emoji/中文 print 会崩溃,强制 UTF-8 输出+sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")+sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")+++LOG_FILE: io.TextIOWrapper | None = None+++def _log(msg: str) -> None:+ ts = time.strftime("%Y-%m-%d %H:%M:%S")+ line = f"[{ts}] {msg}"+ try:+ print(line)+ except Exception:+ pass+ if LOG_FILE is not None:+ try:+ LOG_FILE.write(line + "\n")+ LOG_FILE.flush()+ except Exception:+ passdef _parse_note_arg(arg: str) -> tuple[str, str | None]:@@ -104,12 +135,233 @@ async def run_with_id(note_arg: str, script_path: str | None) -> None:out_dir.mkdir(parents=True, exist_ok=True)if script:(out_dir / "script.txt").write_text(script, encoding="utf-8")++ # 可选:下载图片到本地(默认关闭;通过环境变量 XHS_DOWNLOAD_IMAGES=1 开启)+ if os.environ.get("XHS_DOWNLOAD_IMAGES", "").lower() in ("1", "true", "yes"):+ img_dir = out_dir / "images"+ img_dir.mkdir(parents=True, exist_ok=True)+ paths: list[str] = []+ for i, url in enumerate(note.get("images", []), 1):+ dest = img_dir / f"{i:02d}"+ if await crawler.download_image(url, dest):+ # download_image 会根据 Content-Type 修正扩展名,这里找回真实文件名+ actual = next(dest.parent.glob(f"{dest.name}.*"), dest)+ paths.append(str(actual.relative_to(out_dir)).replace("\\", "/"))+ if paths:+ note["image_paths"] = paths+md = renderer.render_report(note, script, comments)report = out_dir / "report.md"report.write_text(md, encoding="utf-8")print(f"\n✓ {report}")+# ---------------------------------------------------------------------------+# archive: 批量归档已发布笔记的正文、图片、标签(公开页解析,无登录)+# ---------------------------------------------------------------------------++async def run_archive(input_path: Path, output_root: Path, limit: int | None = None,+ max_comments: int = 0) -> None:+ global LOG_FILE+ output_root.mkdir(parents=True, exist_ok=True)+ log_path = output_root / f"archive_{time.strftime('%Y%m%d_%H%M%S')}.log"+ LOG_FILE = open(log_path, "w", encoding="utf-8", errors="replace")+ _log(f"启动归档: input={input_path}, output={output_root}")++ notes = json.loads(input_path.read_text(encoding="utf-8"))+ if not isinstance(notes, list):+ raise ValueError("输入 JSON 必须是笔记列表")++ notes = sorted(notes, key=lambda x: x.get("time", ""), reverse=True)+ if limit:+ notes = notes[:limit]++ results: list[dict] = []+ for i, note in enumerate(notes, 1):+ note_id = note.get("id") or note.get("note_id")+ xsec = note.get("xsecToken") or note.get("xsec_token", "")+ title = note.get("title", "")+ if not note_id or not xsec:+ results.append({+ "success": False,+ "note_id": note_id,+ "error": "缺少 note_id 或 xsec_token",+ })+ continue++ out_dir = output_root / note_id+ _log(f"({i}/{len(notes)}) 归档 {note_id} {title[:30]}...")+ try:+ public = await asyncio.to_thread(crawler.fetch_public_note, note_id, xsec)+ if not public.get("success"):+ err = public.get("error", "公开页解析失败")+ _log(f" -> 失败: {err}")+ results.append({"success": False, "note_id": note_id, "error": err})+ continue++ if max_comments > 0:+ comments = await asyncio.to_thread(+ crawler.fetch_public_comments, note_id, xsec, max_comments+ )+ public["comments"] = comments+ public["comments_fetched"] = len(comments)++ out_dir.mkdir(parents=True, exist_ok=True)+ (out_dir / "note_detail.json").write_text(+ json.dumps(public, ensure_ascii=False, indent=2), encoding="utf-8"+ )++ img_dir = out_dir / "images"+ img_dir.mkdir(parents=True, exist_ok=True)+ img_count = 0+ for idx, url in enumerate(public.get("images", []), 1):+ if await crawler.download_image(url, img_dir / f"{idx:02d}"):+ img_count += 1++ summary = {+ "success": True,+ "note_id": note_id,+ "output_dir": str(out_dir),+ "image_count": img_count,+ "comment_count": public.get("comments_fetched", 0),+ }+ _log(f" -> 完成: 图片 {img_count} 张")+ results.append(summary)+ except Exception as e:+ _log(f" -> 异常: {e}")+ results.append({"success": False, "note_id": note_id, "error": str(e)})++ if i < len(notes):+ await asyncio.sleep(0.5)++ summary_path = output_root / f"archive_summary_{time.strftime('%Y%m%d_%H%M%S')}.json"+ summary_path.write_text(json.dumps(results, ensure_ascii=False, indent=2), encoding="utf-8")+ success = sum(1 for r in results if r.get("success"))+ _log(f"完成: 成功 {success}, 失败 {len(results) - success}, 总计 {len(results)}")+ _log(f"汇总: {summary_path}")+ if LOG_FILE is not None:+ LOG_FILE.close()+ LOG_FILE = None+++# ---------------------------------------------------------------------------+# summarize: 账号级汇总(基于创作者中心最近笔记 + 公开页标签)+# ---------------------------------------------------------------------------++def _series_of(note: dict) -> str:+ """按标签/标题关键字给笔记归系列,简单兜底。"""+ tags = note.get("tags") or []+ if tags:+ return tags[0]+ title = note.get("title", "")+ # 常见系列关键字(可扩展)+ for kw in ("Agent", "RAG", "CS336", "Lec", "MCP", "LLM"):+ if kw.lower() in title.lower():+ return kw+ return "其他"+++def _summarize(notes: list[dict]) -> str:+ lines: list[str] = []+ lines.append("# xhs-explore 账号汇总\n")+ lines.append(f"生成时间:{time.strftime('%Y-%m-%d %H:%M')}")+ lines.append(f"样本数:{len(notes)} 篇\n")++ total_view = sum(n.get("view_count", 0) or 0 for n in notes)+ total_like = sum(n.get("like_count", 0) or 0 for n in notes)+ total_collect = sum(n.get("collect_count", 0) or 0 for n in notes)+ total_comment = sum(n.get("comment_count", 0) or 0 for n in notes)+ lines.append("## 整体数据\n")+ lines.append(f"- 总浏览:{total_view:,}")+ lines.append(f"- 总点赞:{total_like:,}")+ lines.append(f"- 总收藏:{total_collect:,}")+ lines.append(f"- 总评论:{total_comment:,}")+ if total_view:+ lines.append(f"- 平均赞阅比:{total_like / total_view * 100:.2f}%")+ lines.append(f"- 平均藏阅比:{total_collect / total_view * 100:.2f}%")+ lines.append("")++ # 按浏览 Top 10+ sorted_notes = sorted(notes, key=lambda x: x.get("view_count", 0) or 0, reverse=True)[:10]+ lines.append("## 浏览量 Top 10\n")+ lines.append("| 排名 | 标题 | 浏览 | 点赞 | 收藏 | 评论 |")+ lines.append("|------|------|------|------|------|------|")+ for idx, n in enumerate(sorted_notes, 1):+ title = (n.get("title") or "")[:30]+ lines.append(+ f"| {idx} | {title} | {n.get('view_count', 0):,} | "+ f"{n.get('like_count', 0)} | {n.get('collect_count', 0)} | {n.get('comment_count', 0)} |"+ )+ lines.append("")++ # 系列汇总+ series: dict[str, list[dict]] = defaultdict(list)+ for n in notes:+ series[_series_of(n)].append(n)+ lines.append("## 系列汇总(按首标签/标题关键字)\n")+ lines.append("| 系列 | 篇数 | 总浏览 | 均浏览 | 赞阅比 | 藏阅比 | 评阅比 |")+ lines.append("|------|------|--------|--------|--------|--------|--------|")+ for name, items in sorted(series.items(), key=lambda x: sum(i.get("view_count", 0) or 0 for i in x[1]), reverse=True):+ views = sum(i.get("view_count", 0) or 0 for i in items)+ likes = sum(i.get("like_count", 0) or 0 for i in items)+ collects = sum(i.get("collect_count", 0) or 0 for i in items)+ comments = sum(i.get("comment_count", 0) or 0 for i in items)+ avg_view = views / len(items) if items else 0+ like_rate = likes / views * 100 if views else 0+ collect_rate = collects / views * 100 if views else 0+ comment_rate = comments / views * 100 if views else 0+ lines.append(+ f"| {name} | {len(items)} | {views:,} | {avg_view:.0f} | "+ f"{like_rate:.2f}% | {collect_rate:.2f}% | {comment_rate:.2f}% |"+ )+ lines.append("")++ lines.append(+ "> 提示:系列判定优先取公开页标签,无标签时按标题关键字兜底。"+ "可通过 `review.py archive` 批量归档后人工校准。\n"+ )+ return "\n".join(lines)+++async def run_summarize(out_dir: Path) -> None:+ out_dir.mkdir(parents=True, exist_ok=True)+ print("[summarize] 拉取创作者中心最近 50 条笔记……")+ # summarize 是批量后台命令,用 headless 避免弹窗打扰+ sess = await crawler.Session.open(headless=True)+ try:+ notes = await crawler.fetch_recent_notes(sess, limit=50)+ finally:+ await sess.close()++ # 用公开页补全标签/正文/图片(失败也不阻塞)+ for n in notes:+ xsec = n.get("xsec_token") or ""+ if not xsec:+ continue+ try:+ public = await asyncio.to_thread(crawler.fetch_public_note, n["note_id"], xsec)+ crawler._merge_public_note(n, public)+ except Exception:+ pass++ md = _summarize(notes)+ out_path = out_dir / f"xhs_summary_{time.strftime('%Y%m%d_%H%M%S')}.md"+ out_path.write_text(md, encoding="utf-8")+ print(f"\n✓ {out_path}")+++def _usage() -> str:+ return """+用法:+ python review.py 交互式选笔记+ python review.py login 扫码登录+ python review.py list 列最近笔记+ python review.py note <note_id> [script.txt]+ python review.py archive <notes.json> [output_root] [limit]+ python review.py summarize [out_dir]+""".strip()++def main() -> None:if len(sys.argv) > 1 and sys.argv[1] == "login":asyncio.run(crawler.ensure_login())@@ -132,6 +384,22 @@ def main() -> None:print(f"[{i}] {n['note_id']} {t} 曝光{renderer._fmt_num(n.get('view_count'))} {title}")asyncio.run(_list())return+ if len(sys.argv) > 1 and sys.argv[1] == "archive":+ if len(sys.argv) < 3:+ print(_usage())+ sys.exit(2)+ input_path = Path(sys.argv[2].strip('"').strip("'")).expanduser().resolve()+ output_root = Path(sys.argv[3]).expanduser().resolve() if len(sys.argv) > 3 else runtime_project_root() / "data" / "raw" / "notes" / "archive"+ limit = int(sys.argv[4]) if len(sys.argv) > 4 else None+ asyncio.run(run_archive(input_path, output_root, limit=limit))+ return+ if len(sys.argv) > 1 and sys.argv[1] == "summarize":+ out_dir = Path(sys.argv[2]).expanduser().resolve() if len(sys.argv) > 2 else runtime_project_root() / "reports"+ asyncio.run(run_summarize(out_dir))+ return+ if len(sys.argv) > 1 and sys.argv[1] in ("-h", "--help"):+ print(_usage())+ returnasyncio.run(run())