Skip to content

You're reading this. The skill predicted it. A workflow that turns every post into a calibrated experiment—score, blind-predict, retro, evolve. The future doesn't reward effort, it rewards those who see the pattern first. 1M followers in a month — not luck, system.

Python

feat(xhs-explore): public-page fallback, archive/summarize, image download (#46)

SingularGuyLeBorn22952821ca3f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9e7b8fc..fa93931 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,20 @@ All notable changes to cheat-on-content will be documented here.
## [Unreleased]
+### Added — xhs-explore adapter: 公开页兜底、正文/图片归档、账号汇总
+
+- 融合 [xhs-analytics](https://github.com/SingularGuyLeBorn/xhs-analytics) 的公开页解析能力:
+ - `crawler.fetch_public_note()` 无登录解析 `window.__INITIAL_STATE__`,补全正文、图片 URL、标签。
+ - `crawler.fetch_public_comments()` 当前台 API 拿不到评论时,用公开页 top 评论兜底(通常 ~10 条)。
+ - `crawler.download_image()` 异步下载笔记图片。
+- `review.py` 新增子命令:
+ - `archive <notes.json> [output_root] [limit]`:批量归档已发布笔记正文与图片。
+ - `summarize [out_dir]`:基于创作者中心最近 50 条 + 公开页标签生成账号级汇总。
+- `review.py note` 支持环境变量 `XHS_DOWNLOAD_IMAGES=1` 下载图片到 `videos/<...>/images/`。
+- `renderer.py` 在 `report.md` 中新增「正文 + 标签」与「图片」块。
+- `requirements.txt` 增加 `requests>=2.28`。
+- `Session.open()` 在未下载 Playwright Chromium 时自动 fallback 到系统 Chrome(`channel="chrome"`)。
+
### Added — 受众画像 persona(cheat-persona skill + audience.md)
**动机**:用户选题 / 写稿时缺一个清晰的"谁在看"的镜子。新增受众画像功能——从复盘评论数据聚类出账号真实受众。
diff --git a/adapters/perf-data/xhs-explore/README.md b/adapters/perf-data/xhs-explore/README.md
index 6f60cd8..7a46c71 100644
--- a/adapters/perf-data/xhs-explore/README.md
+++ b/adapters/perf-data/xhs-explore/README.md
@@ -4,6 +4,8 @@
> **来源**:照搬 `douyin-session` adapter 的架构(Playwright 持久化登录态 + 被动拦截 XHR),
> 接口路径与字段参考 NanmiCoder/MediaCrawler 与 ReaJason/xhs。已在真实创作者账号端到端验证(2026-05)。
+> 2026-06 融合 [xhs-analytics](https://github.com/SingularGuyLeBorn/xhs-analytics) 的公开页解析能力:
+> 无登录抓取正文 / 图片 / 标签 / top 评论兜底,以及批量归档与账号汇总命令。
---
@@ -12,13 +14,14 @@
小红书反爬靠 `x-s`/`x-t`/`x-s-common` 签名 + `xsec_token`——纯 HTTP / requests 拿不到数据。
xhs-explore 用 **Playwright + 持久化 Chromium context** 模拟真实浏览器,**不逆向签名、不伪造请求**,
-让页面自己发带签名的请求,我们只被动拦截返回的 JSON:
+让页面自己发带签名的请求,我们只被动拦截返回的 JSON;同时用公开页 `__INITIAL_STATE__` 做正文/图片/评论兜底:
- 首次扫码登录创作者中心,cookie 存在**你的内容项目根目录** `.auth-xhs/`
- 之后每次抓取复用 cookie
-- 抓两路数据:
+- 抓三路数据:
1. **创作者中心 galaxy 接口**(`/api/galaxy/v2/creator/note/user/posted`)— 你**自己**笔记的运营数据:观看/点赞/收藏/评论/分享,**不需要 xsec_token**,最稳。列表每条自带 `xsec_token`,抓自己的笔记无需手动粘带 token 的链接
2. **前台 web API**(`/api/sns/web/v1/feed` + `/api/sns/web/v2/comment/page`)— 拿确认字段的点赞/收藏数 + 评论文本
+ 3. **公开页兜底**(`www.xiaohongshu.com/explore/<note_id>` 的 `window.__INITIAL_STATE__`)— **无需登录**即可补全正文、图片 URL、标签;当前台 API 拿不到评论时,用公开页里的 top 评论兜底(通常 ~10 条)
输出写到**你的内容项目** `videos/<...>/report.md`。
调试产物(URL dump / 截图 / galaxy 原始 JSON)写到 `.cheat-cache/xhs-explore-debug/`。
@@ -42,8 +45,8 @@ cd ~/Documents/my-channel
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
-# 3. 装 Playwright + Chromium
-pip install playwright>=1.44
+# 3. 装依赖(Playwright + Chromium + requests)
+pip install -r "$ADAPTER/requirements.txt"
playwright install chromium
# 4. 首次扫码登录小红书创作者中心
@@ -70,9 +73,18 @@ python "$ADAPTER/review.py" list
# 抓特定笔记
python "$ADAPTER/review.py" note <note_id> [script.txt]
-# 输出在 当前目录/videos/<日期>_<标题>/report.md
+# 抓特定笔记并下载图片到 videos/<...>/images/
+XHS_DOWNLOAD_IMAGES=1 python "$ADAPTER/review.py" note <note_id>
+
+# 批量归档已发布笔记的正文+图片(公开页解析,无登录)
+python "$ADAPTER/review.py" archive my_notes.json data/raw/notes/archive 50
+
+# 账号级汇总(基于创作者中心最近 50 条 + 公开页标签)
+python "$ADAPTER/review.py" summarize reports
```
+输出在 当前目录/videos/<日期>_<标题>/report.md;archive 输出在 `<output_root>/<note_id>/`。
+
## 怎么拿到 note_id
小红书笔记 URL:
@@ -86,7 +98,10 @@ cheat-publish 登记发布时把 note_id 存到 prediction header,cheat-retro
由 `renderer.py` 生成:
- 笔记元信息(标题、发布时间、链接、IP 归属)
- 数据快照(曝光/浏览、点赞、收藏、评论、分享 + 派生比率:赞曝比 / 藏曝比 / 评曝比 / 分曝比 + 涨粉)
+- 正文 + 标签(公开页兜底时才有)
+- 图片(本地路径优先,否则 URL)
- galaxy 原始字段 JSON(debug / 接口改版时核对字段用)
+- 原始稿子(cheat-retro 传入)
- Top 评论(按赞数排序,含文本 + 赞数 + IP)
## 失败模式(按概率从高到低)
@@ -96,16 +111,17 @@ cheat-publish 登记发布时把 note_id 存到 prediction header,cheat-retro
| 曝光显示 0 但 JSON 有数 | galaxy 字段 key 不在候选列表 | 看 report.md 里 galaxy JSON,把真实 key 加进 `_normalize_note` |
| `ensure_login` 超时 | cookie 过期 | 重跑 `python crawler.py login` |
| 笔记列表为空 | 创作者中心改了 galaxy 接口路径 | 看 `.cheat-cache/xhs-explore-debug/creator_urls.txt`,更新 `GALAXY_*_KEYS` |
-| 评论抓不到 | xsec_token 缺失 / 评论被关 | 看 `frontend_urls.txt`;拿不到就降级 manual 粘评论(cheat-retro 会提示) |
+| 评论抓不到 | xsec_token 缺失 / 评论被关 / 登录过期 | 先看 `frontend_urls.txt`;再用公开页兜底 top 评论;最后 manual 粘评论 |
+| 正文/图片/标签缺失 | 公开页也解析失败 | 检查 note_id 与 xsec_token;网页结构改版时更新 `_extract_initial_state` |
| Chromium 崩溃 | 内存不足 | 关其他 Chromium;`playwright install chromium --force` |
**关键现实**:小红书接口 path(`/feed`、`/comment/page`、galaxy)相对稳定,但签名和 xsec_token 机制常变。
-本 adapter 用被动拦截规避签名风险——最易抖动处是 **galaxy 创作者接口**和**评论回复**。
+本 adapter 用被动拦截 + 公开页兜底规避签名风险——最易抖动处是 **galaxy 创作者接口**和**评论回复**。
## 稳定性等级
★★ — Playwright + 登录态能扛比纯 HTTP 强得多的反爬,但仍受小红书前端改版影响。
-评论路径(依赖 xsec_token)比创作者数据路径脆,必要时降级 manual。
+评论路径(依赖 xsec_token)比创作者数据路径脆,必要时降级 manual 或公开页兜底。
## 风险提示
@@ -119,9 +135,9 @@ cheat-publish 登记发布时把 note_id 存到 prediction header,cheat-retro
```
adapters/perf-data/xhs-explore/
├── README.md # 本文件
-├── requirements.txt # playwright>=1.44
-├── crawler.py # 抓取核心(galaxy 笔记列表+数据 / 前台 interact+评论)
-├── review.py # CLI 入口(login / list / note <note_id>)
+├── requirements.txt # playwright>=1.44, requests>=2.28
+├── crawler.py # 抓取核心(galaxy / 前台 API / 公开页兜底 / 图片下载)
+├── review.py # CLI 入口(login / list / note / archive / summarize)
├── renderer.py # 把抓回的 JSON 渲染成 report.md
├── paths.py # 项目根 / .auth-xhs / debug 路径解析
└── run.sh # cheat-retro 调用的 wrapper
diff --git a/adapters/perf-data/xhs-explore/crawler.py b/adapters/perf-data/xhs-explore/crawler.py
index e87473c..9e3f838 100644
--- a/adapters/perf-data/xhs-explore/crawler.py
+++ b/adapters/perf-data/xhs-explore/crawler.py
@@ -9,14 +9,22 @@
- 创作者**自己的**笔记数据走 galaxy 接口(不需要 xsec_token),是最稳的主路。
- 评论走前台 web API(需要 xsec_token),让页面自己导航触发带 token 的请求;
拿不到就优雅降级(report.md 标 comments_unavailable,cheat-retro 回落到 manual 粘评论)。
+
+本文件新增的能力(来自 xhs-analytics 的公开页解析):
+- fetch_public_note: 无登录解析 explore 页面 __INITIAL_STATE__,拿正文 / 图片 / 标签。
+- fetch_public_comments: 公开页兜底 top 评论。
+- download_image: 下载笔记图片到本地。
"""
from __future__ import annotations
import asyncio
import json
+import re
+import urllib.parse
from pathlib import Path
from typing import Any
+import requests
from playwright.async_api import BrowserContext, Page, Response, async_playwright
from paths import auth_dir, debug_dir
@@ -35,6 +43,11 @@ GALAXY_NOTE_STATS_KEYS = (
FEED_KEY = "/api/sns/web/v1/feed"
COMMENT_KEY = "/api/sns/web/v2/comment/page"
+DEFAULT_USER_AGENT = (
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+ "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
+)
+
class Session:
"""单浏览器会话,按顺序跑多步抓取。"""
@@ -48,12 +61,22 @@ class Session:
pw = await async_playwright().start()
auth_path = auth_dir()
auth_path.mkdir(parents=True, exist_ok=True)
- ctx = await pw.chromium.launch_persistent_context(
- user_data_dir=str(auth_path),
- headless=headless,
- viewport={"width": 1440, "height": 900},
- args=["--disable-blink-features=AutomationControlled"],
- )
+ common_kwargs = {
+ "user_data_dir": str(auth_path),
+ "headless": headless,
+ "viewport": {"width": 1440, "height": 900},
+ "args": ["--disable-blink-features=AutomationControlled"],
+ }
+ try:
+ ctx = await pw.chromium.launch_persistent_context(**common_kwargs)
+ except Exception as exc:
+ # 部分环境只装了系统 Chrome,没下载 Playwright Chromium,fallback 到 channel=chrome
+ try:
+ ctx = await pw.chromium.launch_persistent_context(
+ **common_kwargs, channel="chrome"
+ )
+ except Exception:
+ raise exc
return cls(ctx, pw)
async def close(self) -> None:
@@ -108,24 +131,43 @@ async def _acquire_web_session(page: Page) -> None:
pass
-async def ensure_login(timeout_s: int = 300) -> bool:
- """扫码登录创作者中心;检测到创作者登录态后顺便换取 web_session,然后自动关闭。"""
+async def ensure_login(timeout_s: int = 180, max_refresh: int = 5) -> bool:
+ """扫码登录创作者中心;检测到创作者登录态后顺便换取 web_session,然后自动关闭。
+
+ 二维码本身会过期,所以用 timeout_s 作为单次等待上限,超时后自动刷新页面重新出码,
+ 最多刷新 max_refresh 次。用户有充足时间扫码。
+ """
sess = await Session.open()
try:
page = await sess.ctx.new_page()
await page.goto(CREATOR_HOME)
- print(f"[登录] 在弹出的 Chromium 窗口里扫码登录小红书创作者中心。最多等 {timeout_s} 秒……")
- for i in range(timeout_s):
- try:
- if await _creator_logged_in(sess.ctx) and "login" not in page.url:
- print(f"[登录] ✓ 创作者中心登录态已确认(用时 {i}s)")
- await _acquire_web_session(page)
- await asyncio.sleep(1)
- return True
- except Exception:
- pass
- await asyncio.sleep(1)
- print("[登录] 超时未检测到登录态。")
+ print(f"[登录] 在弹出的 Chromium 窗口里扫码登录小红书创作者中心。每次二维码有效期约 {timeout_s} 秒,超时自动刷新。")
+
+ for refresh in range(max_refresh + 1):
+ for i in range(timeout_s):
+ try:
+ if await _creator_logged_in(sess.ctx) and "login" not in page.url:
+ print(f"[登录] ✓ 创作者中心登录态已确认(总用时 {refresh * timeout_s + i}s)")
+ await _acquire_web_session(page)
+ await asyncio.sleep(1)
+ return True
+ except Exception:
+ pass
+
+ # 每 30 秒提醒一次,避免用户以为卡死
+ if i > 0 and i % 30 == 0:
+ print(f"[登录] 已等待 {i} 秒,请用小红书 App 扫码(或等待自动刷新二维码)……")
+
+ await asyncio.sleep(1)
+
+ if refresh < max_refresh:
+ print(f"[登录] 本次二维码未扫码或已过期,正在刷新页面重新出码(第 {refresh + 1}/{max_refresh} 次刷新)……")
+ try:
+ await page.reload(wait_until="domcontentloaded", timeout=30000)
+ except Exception:
+ await page.goto(CREATOR_HOME)
+
+ print("[登录] 超过最大刷新次数仍未检测到登录态,已停止。如需继续请重新运行本命令。")
return False
finally:
await sess.close()
@@ -153,6 +195,10 @@ async def fetch_recent_notes(sess: Session, limit: int = 50) -> list[dict]:
try:
await page.goto(CREATOR_NOTE_MANAGER, wait_until="domcontentloaded", timeout=60000)
await asyncio.sleep(8)
+ # cookie 过期时会被 302 到登录页
+ if "login" in page.url or "redirectReason=401" in page.url:
+ print("[登录] 创作者中心已跳转登录页,cookie 可能已过期。请运行:python crawler.py login")
+ return []
for _ in range(4):
await page.evaluate("window.scrollBy(0, 1200)")
await asyncio.sleep(1.5)
@@ -247,7 +293,213 @@ def _to_int(x: Any) -> int:
return 0
-async def fetch_note_frontend(sess: Session, note_id: str, note_url: str | None = None) -> dict:
+# ---------------------------------------------------------------------------
+# 公开页解析(来自 xhs-analytics 的核心能力):无登录拿正文、图片、标签、评论兜底
+# ---------------------------------------------------------------------------
+
+def _extract_initial_state(html: str) -> dict | None:
+ """从小红书 explore 页面 HTML 中解析 window.__INITIAL_STATE__。"""
+ marker = "window.__INITIAL_STATE__="
+ start = html.find(marker)
+ if start == -1:
+ return None
+ script_start = html.rfind("<script", 0, start)
+ script_end = html.find("</script>", start)
+ if script_start == -1 or script_end == -1:
+ return None
+ script = html[script_start:script_end]
+ assign_start = script.find(marker) + len(marker)
+ json_str = script[assign_start:]
+ json_str = re.sub(r":\s*undefined\s*([,}\]])", r":null\1", json_str)
+ try:
+ return json.loads(json_str)
+ except Exception:
+ return None
+
+
+def _image_url(img: dict) -> str:
+ """从 imageList 元素中提取可用的图片 URL。"""
+ if not isinstance(img, dict):
+ return ""
+ for key in ("urlDefault", "url"):
+ if img.get(key):
+ return img[key]
+ for info in img.get("infoList", []) or []:
+ if isinstance(info, dict) and info.get("imageScene") == "WB_DFT" and info.get("url"):
+ return info["url"]
+ for info in img.get("infoList", []) or []:
+ if isinstance(info, dict) and info.get("url"):
+ return info["url"]
+ return ""
+
+
+def _xsec_token_from_url(url: str | None) -> str:
+ """从笔记 URL 的 query 中拆出 xsec_token(保留原编码)。"""
+ if not url:
+ return ""
+ parsed = urllib.parse.urlparse(url)
+ return urllib.parse.unquote(urllib.parse.parse_qs(parsed.query).get("xsec_token", [""])[0])
+
+
+def fetch_public_note(note_id: str, xsec_token: str) -> dict:
+ """无登录抓取 explore 公开页,解析 __INITIAL_STATE__。
+
+ 返回 dict:success, note_id, title, desc/body, images, tags, time, counts, raw。
+ """
+ token = urllib.parse.unquote(xsec_token)
+ url = (
+ f"https://www.xiaohongshu.com/explore/{note_id}"
+ f"?xsec_token={urllib.parse.quote(token)}"
+ f"&xsec_source=pc_creatormng"
+ )
+ headers = {
+ "User-Agent": DEFAULT_USER_AGENT,
+ "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
+ "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
+ }
+ r = requests.get(url, headers=headers, timeout=20)
+ r.raise_for_status()
+
+ state = _extract_initial_state(r.text)
+ if not state:
+ raise ValueError("无法解析页面初始状态")
+
+ note_detail_map = state.get("note", {}).get("noteDetailMap", {})
+ if note_id not in note_detail_map:
+ raise ValueError("页面未返回该笔记数据")
+
+ raw = note_detail_map[note_id]
+ note = raw.get("note", {})
+ interact = note.get("interactInfo", {})
+
+ return {
+ "success": True,
+ "note_id": note_id,
+ "title": note.get("title", ""),
+ "desc": note.get("desc", ""),
+ "body": note.get("desc", ""),
+ "type": note.get("type", ""),
+ "time": note.get("time", 0),
+ "images": [_image_url(img) for img in note.get("imageList", [])],
+ "tags": [tag.get("name", "") for tag in note.get("tagList", []) if tag.get("name")],
+ "counts": {
+ "liked": _to_int(interact.get("likedCount")),
+ "collected": _to_int(interact.get("collectedCount")),
+ "comment": _to_int(interact.get("commentCount")),
+ "shared": _to_int(interact.get("shareCount")),
+ },
+ "raw": raw,
+ }
+
+
+def _normalize_public_comment(c: dict) -> dict:
+ """把 __INITIAL_STATE__ / edith 接口里的评论字段统一成 adapter 格式。"""
+ user = c.get("user_info") or c.get("userInfo") or {}
+ like_count = c.get("like_count") or c.get("likeCount") or 0
+ sub_count = c.get("sub_comment_count") or c.get("subCommentCount") or 0
+ return {
+ "cid": str(c.get("id") or c.get("comment_id") or ""),
+ "text": c.get("content") or "",
+ "like_count": _to_int(like_count),
+ "sub_comment_count": _to_int(sub_count),
+ "create_time": c.get("create_time") or c.get("createTime") or 0,
+ "user_name": user.get("nickname") or "",
+ "ip_label": c.get("ip_location") or c.get("ipLocation") or "",
+ }
+
+
+def fetch_public_comments(note_id: str, xsec_token: str, max_comments: int = 20) -> list[dict]:
+ """公开页 __INITIAL_STATE__ 兜底 top 评论(通常 ~10 条)。"""
+ token = urllib.parse.unquote(xsec_token)
+ url = (
+ f"https://www.xiaohongshu.com/explore/{note_id}"
+ f"?xsec_token={urllib.parse.quote(token)}"
+ f"&xsec_source=pc_creatormng"
+ )
+ headers = {
+ "User-Agent": DEFAULT_USER_AGENT,
+ "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
+ "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
+ }
+ r = requests.get(url, headers=headers, timeout=20)
+ r.raise_for_status()
+ state = _extract_initial_state(r.text)
+ if not state:
+ return []
+
+ raw = state.get("note", {}).get("noteDetailMap", {}).get(note_id, {})
+ comments = []
+ # __INITIAL_STATE__ 里的评论路径可能是 comments.list 或 commentsList
+ for key in ("comments", "commentsList"):
+ container = raw.get(key) if isinstance(raw, dict) else None
+ if isinstance(container, dict):
+ arr = container.get("list") or container.get("comments") or []
+ if isinstance(arr, list):
+ comments.extend([_normalize_public_comment(c) for c in arr])
+ elif isinstance(container, list):
+ comments.extend([_normalize_public_comment(c) for c in container])
+
+ seen = set()
+ dedup = []
+ for c in comments:
+ if not c["cid"] or c["cid"] in seen:
+ continue
+ seen.add(c["cid"])
+ dedup.append(c)
+ dedup.sort(key=lambda x: x["like_count"], reverse=True)
+ return dedup[:max_comments]
+
+
+async def download_image(url: str, dest: Path, timeout: int = 30) -> bool:
+ """下载单张图片到 dest(会根据 Content-Type 修正扩展名)。异步封装。"""
+ if not url:
+ return False
+ dest = Path(dest)
+ dest.parent.mkdir(parents=True, exist_ok=True)
+
+ def _download() -> bool:
+ headers = {
+ "User-Agent": DEFAULT_USER_AGENT,
+ "Accept": "image/webp,image/apng,image/*,*/*;q=0.8",
+ "Referer": "https://www.xiaohongshu.com/",
+ }
+ r = requests.get(url, headers=headers, timeout=timeout, stream=True)
+ r.raise_for_status()
+
+ parsed = urllib.parse.urlparse(url)
+ path = urllib.parse.unquote(parsed.path)
+ ext = Path(path).suffix.lower()
+ if ext not in {".jpg", ".jpeg", ".png", ".webp", ".gif", ".avif"}:
+ ct = r.headers.get("Content-Type", "").lower()
+ if "webp" in ct:
+ ext = ".webp"
+ elif "png" in ct:
+ ext = ".png"
+ elif "gif" in ct:
+ ext = ".gif"
+ else:
+ ext = ".jpg"
+ dest_final = dest.with_suffix(ext)
+
+ with open(dest_final, "wb") as f:
+ for chunk in r.iter_content(chunk_size=8192):
+ if chunk:
+ f.write(chunk)
+ return True
+
+ try:
+ return await asyncio.to_thread(_download)
+ except Exception as exc:
+ print(f"[下载图片] 失败 {url}: {exc}")
+ return False
+
+
+async def fetch_note_frontend(
+ sess: Session,
+ note_id: str,
+ note_url: str | None = None,
+ xsec_token: str | None = None,
+) -> dict:
"""打开前台笔记页 → 拦截 feed(interact_info 确认字段)+ comment/page。
前台需要 xsec_token + 登录态(web_session)。
@@ -255,6 +507,8 @@ async def fetch_note_frontend(sess: Session, note_id: str, note_url: str | None
→ 直接用它导航,最稳。
- 否则退回裸 explore URL(仅对已登录账号访问自己笔记可能可行)。
token 缺失 / 未登录 → dump 并降级(评论留给 manual)。
+
+ 拦截不到评论时,会用公开页 __INITIAL_STATE__ 里的 top 评论兜底。
"""
feed: dict = {}
comments: list[dict] = []
@@ -306,6 +560,20 @@ async def fetch_note_frontend(sess: Session, note_id: str, note_url: str | None
stagnant = 0
last = cur
+ # 兜底:公开页 __INITIAL_STATE__ 里的 top 评论
+ if not comments:
+ xsec = xsec_token or _xsec_token_from_url(note_url)
+ if xsec:
+ try:
+ public_comments = await asyncio.to_thread(
+ fetch_public_comments, note_id, xsec, max_comments=20
+ )
+ if public_comments:
+ comments = public_comments
+ print(f" 公开页兜底 {len(comments)} 条评论")
+ except Exception as exc:
+ print(f"[诊断] 公开页评论兜底失败:{exc}")
+
if not comments:
dbg = debug_dir()
dbg.mkdir(parents=True, exist_ok=True)
@@ -389,8 +657,29 @@ def _dump(urls: list[str], url_file: str, captured: list[dict], cap_file: str) -
pass
+def _merge_public_note(note: dict, public: dict) -> None:
+ """用公开页数据补全 note(不覆盖已有的 galaxy 运营数据)。"""
+ if not public.get("success"):
+ return
+ for key in ("title", "desc", "body", "type", "images", "tags"):
+ if public.get(key) and not note.get(key):
+ note[key] = public[key]
+ if public.get("time") and not note.get("create_time"):
+ note["create_time"] = _to_int(public["time"])
+ counts = public.get("counts") or {}
+ mapping = {
+ "like_count": counts.get("liked"),
+ "collect_count": counts.get("collected"),
+ "comment_count": counts.get("comment"),
+ "share_count": counts.get("shared"),
+ }
+ for k, v in mapping.items():
+ if v and not note.get(k):
+ note[k] = _to_int(v)
+
+
async def fetch_all(note_id: str, note_url: str | None = None) -> dict:
- """一个会话跑完笔记列表(含 galaxy 指标)+ 前台 interact + 评论。"""
+ """一个会话跑完笔记列表(含 galaxy 指标)+ 前台 interact + 评论 + 公开页正文/图片兜底。"""
sess = await Session.open()
try:
print(" → 打开创作者中心,拉笔记列表 + 运营数据")
@@ -408,8 +697,19 @@ async def fetch_all(note_id: str, note_url: str | None = None) -> dict:
front_url = (f"https://www.xiaohongshu.com/explore/{note_id}"
f"?xsec_token={note['xsec_token']}&xsec_source=pc_creatormng")
+ xsec = _xsec_token_from_url(front_url) or note.get("xsec_token", "")
+
+ # 先用公开页补正文/图片/标签(无登录、低成本),不覆盖 galaxy 已确认的计数
+ if xsec:
+ try:
+ public = await asyncio.to_thread(fetch_public_note, note_id, xsec)
+ _merge_public_note(note, public)
+ print(" ✓ 公开页正文/图片/标签已补全")
+ except Exception as exc:
+ print(f"[诊断] 公开页兜底失败:{exc}")
+
print(" → 打开前台笔记页抓 interact + 评论")
- front = await fetch_note_frontend(sess, note_id, note_url=front_url)
+ front = await fetch_note_frontend(sess, note_id, note_url=front_url, xsec_token=xsec)
# 前台 interact 字段是确认的——用它补全/覆盖 galaxy 里可能缺的计数
for k in ("like_count", "collect_count", "comment_count", "share_count"):
if front["interact"].get(k):
diff --git a/adapters/perf-data/xhs-explore/renderer.py b/adapters/perf-data/xhs-explore/renderer.py
index 243aa8b..f7ff7a9 100644
--- a/adapters/perf-data/xhs-explore/renderer.py
+++ b/adapters/perf-data/xhs-explore/renderer.py
@@ -41,6 +41,11 @@ def _ratio(num, denom) -> str:
return f"{num/denom*100:.2f}%"
+def _escape_md(text: str) -> str:
+ """轻度转义,避免正文里的 # 被误认为标题。"""
+ return text.replace("\n", " \n").replace("#", "\\#")
+
+
def render_report(note: dict, script: str, comments: list[dict]) -> str:
lines: list[str] = []
title = note.get("title") or "(无标题)"
@@ -68,6 +73,29 @@ def render_report(note: dict, script: str, comments: list[dict]) -> str:
lines.append(f"- 涨粉:{_fmt_num(note.get('fans_inc'))}")
lines.append("")
+ # 正文 + 标签(公开页兜底时才有)
+ body = note.get("body") or note.get("desc") or ""
+ tags = note.get("tags") or []
+ if body or tags:
+ lines.append("## 正文")
+ lines.append("")
+ if tags:
+ lines.append(" ".join(f"#{t}" for t in tags))
+ lines.append("")
+ lines.append(_escape_md(body) if body else "(无)")
+ lines.append("")
+
+ # 图片(本地优先,否则用 URL)
+ image_paths = note.get("image_paths") or []
+ image_urls = note.get("images") or []
+ images = image_paths or image_urls
+ if images:
+ lines.append("## 图片")
+ lines.append("")
+ for i, img in enumerate(images, 1):
+ lines.append(f"![图片{i}]({img})")
+ lines.append("")
+
# galaxy 原始 JSON——首跑时用来确认曝光字段的真实 key 名
raw = note.get("raw")
if raw:
diff --git a/adapters/perf-data/xhs-explore/requirements.txt b/adapters/perf-data/xhs-explore/requirements.txt
index 86a8fae..99fd92d 100644
--- a/adapters/perf-data/xhs-explore/requirements.txt
+++ b/adapters/perf-data/xhs-explore/requirements.txt
@@ -1 +1,2 @@
playwright>=1.44
+requests>=2.28
diff --git a/adapters/perf-data/xhs-explore/review.py b/adapters/perf-data/xhs-explore/review.py
index a364459..0305480 100644
--- a/adapters/perf-data/xhs-explore/review.py
+++ b/adapters/perf-data/xhs-explore/review.py
@@ -5,17 +5,48 @@
python review.py login # 仅登录(首次扫码)
python review.py list # 列最近笔记(验证登录态)
python review.py note <note_id> [script.txt] # 直接指定笔记
+ python review.py archive <notes.json> [output_root] [limit] # 批量归档正文+图片
+ python review.py summarize [out_dir] # 账号级汇总(基于最近 50 条)
+
+archive / summarize 能力来自 xhs-analytics 的公开页解析,可无登录抓取正文与图片。
"""
from __future__ import annotations
import asyncio
+import io
+import json
+import os
import re
import sys
+import time
+from collections import defaultdict
from pathlib import Path
import crawler
import renderer
-from paths import videos_dir
+from paths import runtime_project_root, videos_dir
+
+# Windows Git Bash 等默认 GBK 控制台,emoji/中文 print 会崩溃,强制 UTF-8 输出
+sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
+sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")
+
+
+LOG_FILE: io.TextIOWrapper | None = None
+
+
+def _log(msg: str) -> None:
+ ts = time.strftime("%Y-%m-%d %H:%M:%S")
+ line = f"[{ts}] {msg}"
+ try:
+ print(line)
+ except Exception:
+ pass
+ if LOG_FILE is not None:
+ try:
+ LOG_FILE.write(line + "\n")
+ LOG_FILE.flush()
+ except Exception:
+ pass
def _parse_note_arg(arg: str) -> tuple[str, str | None]:
@@ -104,12 +135,233 @@ async def run_with_id(note_arg: str, script_path: str | None) -> None:
out_dir.mkdir(parents=True, exist_ok=True)
if script:
(out_dir / "script.txt").write_text(script, encoding="utf-8")
+
+ # 可选:下载图片到本地(默认关闭;通过环境变量 XHS_DOWNLOAD_IMAGES=1 开启)
+ if os.environ.get("XHS_DOWNLOAD_IMAGES", "").lower() in ("1", "true", "yes"):
+ img_dir = out_dir / "images"
+ img_dir.mkdir(parents=True, exist_ok=True)
+ paths: list[str] = []
+ for i, url in enumerate(note.get("images", []), 1):
+ dest = img_dir / f"{i:02d}"
+ if await crawler.download_image(url, dest):
+ # download_image 会根据 Content-Type 修正扩展名,这里找回真实文件名
+ actual = next(dest.parent.glob(f"{dest.name}.*"), dest)
+ paths.append(str(actual.relative_to(out_dir)).replace("\\", "/"))
+ if paths:
+ note["image_paths"] = paths
+
md = renderer.render_report(note, script, comments)
report = out_dir / "report.md"
report.write_text(md, encoding="utf-8")
print(f"\n✓ {report}")
+# ---------------------------------------------------------------------------
+# archive: 批量归档已发布笔记的正文、图片、标签(公开页解析,无登录)
+# ---------------------------------------------------------------------------
+
+async def run_archive(input_path: Path, output_root: Path, limit: int | None = None,
+ max_comments: int = 0) -> None:
+ global LOG_FILE
+ output_root.mkdir(parents=True, exist_ok=True)
+ log_path = output_root / f"archive_{time.strftime('%Y%m%d_%H%M%S')}.log"
+ LOG_FILE = open(log_path, "w", encoding="utf-8", errors="replace")
+ _log(f"启动归档: input={input_path}, output={output_root}")
+
+ notes = json.loads(input_path.read_text(encoding="utf-8"))
+ if not isinstance(notes, list):
+ raise ValueError("输入 JSON 必须是笔记列表")
+
+ notes = sorted(notes, key=lambda x: x.get("time", ""), reverse=True)
+ if limit:
+ notes = notes[:limit]
+
+ results: list[dict] = []
+ for i, note in enumerate(notes, 1):
+ note_id = note.get("id") or note.get("note_id")
+ xsec = note.get("xsecToken") or note.get("xsec_token", "")
+ title = note.get("title", "")
+ if not note_id or not xsec:
+ results.append({
+ "success": False,
+ "note_id": note_id,
+ "error": "缺少 note_id 或 xsec_token",
+ })
+ continue
+
+ out_dir = output_root / note_id
+ _log(f"({i}/{len(notes)}) 归档 {note_id} {title[:30]}...")
+ try:
+ public = await asyncio.to_thread(crawler.fetch_public_note, note_id, xsec)
+ if not public.get("success"):
+ err = public.get("error", "公开页解析失败")
+ _log(f" -> 失败: {err}")
+ results.append({"success": False, "note_id": note_id, "error": err})
+ continue
+
+ if max_comments > 0:
+ comments = await asyncio.to_thread(
+ crawler.fetch_public_comments, note_id, xsec, max_comments
+ )
+ public["comments"] = comments
+ public["comments_fetched"] = len(comments)
+
+ out_dir.mkdir(parents=True, exist_ok=True)
+ (out_dir / "note_detail.json").write_text(
+ json.dumps(public, ensure_ascii=False, indent=2), encoding="utf-8"
+ )
+
+ img_dir = out_dir / "images"
+ img_dir.mkdir(parents=True, exist_ok=True)
+ img_count = 0
+ for idx, url in enumerate(public.get("images", []), 1):
+ if await crawler.download_image(url, img_dir / f"{idx:02d}"):
+ img_count += 1
+
+ summary = {
+ "success": True,
+ "note_id": note_id,
+ "output_dir": str(out_dir),
+ "image_count": img_count,
+ "comment_count": public.get("comments_fetched", 0),
+ }
+ _log(f" -> 完成: 图片 {img_count} 张")
+ results.append(summary)
+ except Exception as e:
+ _log(f" -> 异常: {e}")
+ results.append({"success": False, "note_id": note_id, "error": str(e)})
+
+ if i < len(notes):
+ await asyncio.sleep(0.5)
+
+ summary_path = output_root / f"archive_summary_{time.strftime('%Y%m%d_%H%M%S')}.json"
+ summary_path.write_text(json.dumps(results, ensure_ascii=False, indent=2), encoding="utf-8")
+ success = sum(1 for r in results if r.get("success"))
+ _log(f"完成: 成功 {success}, 失败 {len(results) - success}, 总计 {len(results)}")
+ _log(f"汇总: {summary_path}")
+ if LOG_FILE is not None:
+ LOG_FILE.close()
+ LOG_FILE = None
+
+
+# ---------------------------------------------------------------------------
+# summarize: 账号级汇总(基于创作者中心最近笔记 + 公开页标签)
+# ---------------------------------------------------------------------------
+
+def _series_of(note: dict) -> str:
+ """按标签/标题关键字给笔记归系列,简单兜底。"""
+ tags = note.get("tags") or []
+ if tags:
+ return tags[0]
+ title = note.get("title", "")
+ # 常见系列关键字(可扩展)
+ for kw in ("Agent", "RAG", "CS336", "Lec", "MCP", "LLM"):
+ if kw.lower() in title.lower():
+ return kw
+ return "其他"
+
+
+def _summarize(notes: list[dict]) -> str:
+ lines: list[str] = []
+ lines.append("# xhs-explore 账号汇总\n")
+ lines.append(f"生成时间:{time.strftime('%Y-%m-%d %H:%M')}")
+ lines.append(f"样本数:{len(notes)} 篇\n")
+
+ total_view = sum(n.get("view_count", 0) or 0 for n in notes)
+ total_like = sum(n.get("like_count", 0) or 0 for n in notes)
+ total_collect = sum(n.get("collect_count", 0) or 0 for n in notes)
+ total_comment = sum(n.get("comment_count", 0) or 0 for n in notes)
+ lines.append("## 整体数据\n")
+ lines.append(f"- 总浏览:{total_view:,}")
+ lines.append(f"- 总点赞:{total_like:,}")
+ lines.append(f"- 总收藏:{total_collect:,}")
+ lines.append(f"- 总评论:{total_comment:,}")
+ if total_view:
+ lines.append(f"- 平均赞阅比:{total_like / total_view * 100:.2f}%")
+ lines.append(f"- 平均藏阅比:{total_collect / total_view * 100:.2f}%")
+ lines.append("")
+
+ # 按浏览 Top 10
+ sorted_notes = sorted(notes, key=lambda x: x.get("view_count", 0) or 0, reverse=True)[:10]
+ lines.append("## 浏览量 Top 10\n")
+ lines.append("| 排名 | 标题 | 浏览 | 点赞 | 收藏 | 评论 |")
+ lines.append("|------|------|------|------|------|------|")
+ for idx, n in enumerate(sorted_notes, 1):
+ title = (n.get("title") or "")[:30]
+ lines.append(
+ f"| {idx} | {title} | {n.get('view_count', 0):,} | "
+ f"{n.get('like_count', 0)} | {n.get('collect_count', 0)} | {n.get('comment_count', 0)} |"
+ )
+ lines.append("")
+
+ # 系列汇总
+ series: dict[str, list[dict]] = defaultdict(list)
+ for n in notes:
+ series[_series_of(n)].append(n)
+ lines.append("## 系列汇总(按首标签/标题关键字)\n")
+ lines.append("| 系列 | 篇数 | 总浏览 | 均浏览 | 赞阅比 | 藏阅比 | 评阅比 |")
+ lines.append("|------|------|--------|--------|--------|--------|--------|")
+ for name, items in sorted(series.items(), key=lambda x: sum(i.get("view_count", 0) or 0 for i in x[1]), reverse=True):
+ views = sum(i.get("view_count", 0) or 0 for i in items)
+ likes = sum(i.get("like_count", 0) or 0 for i in items)
+ collects = sum(i.get("collect_count", 0) or 0 for i in items)
+ comments = sum(i.get("comment_count", 0) or 0 for i in items)
+ avg_view = views / len(items) if items else 0
+ like_rate = likes / views * 100 if views else 0
+ collect_rate = collects / views * 100 if views else 0
+ comment_rate = comments / views * 100 if views else 0
+ lines.append(
+ f"| {name} | {len(items)} | {views:,} | {avg_view:.0f} | "
+ f"{like_rate:.2f}% | {collect_rate:.2f}% | {comment_rate:.2f}% |"
+ )
+ lines.append("")
+
+ lines.append(
+ "> 提示:系列判定优先取公开页标签,无标签时按标题关键字兜底。"
+ "可通过 `review.py archive` 批量归档后人工校准。\n"
+ )
+ return "\n".join(lines)
+
+
+async def run_summarize(out_dir: Path) -> None:
+ out_dir.mkdir(parents=True, exist_ok=True)
+ print("[summarize] 拉取创作者中心最近 50 条笔记……")
+ # summarize 是批量后台命令,用 headless 避免弹窗打扰
+ sess = await crawler.Session.open(headless=True)
+ try:
+ notes = await crawler.fetch_recent_notes(sess, limit=50)
+ finally:
+ await sess.close()
+
+ # 用公开页补全标签/正文/图片(失败也不阻塞)
+ for n in notes:
+ xsec = n.get("xsec_token") or ""
+ if not xsec:
+ continue
+ try:
+ public = await asyncio.to_thread(crawler.fetch_public_note, n["note_id"], xsec)
+ crawler._merge_public_note(n, public)
+ except Exception:
+ pass
+
+ md = _summarize(notes)
+ out_path = out_dir / f"xhs_summary_{time.strftime('%Y%m%d_%H%M%S')}.md"
+ out_path.write_text(md, encoding="utf-8")
+ print(f"\n✓ {out_path}")
+
+
+def _usage() -> str:
+ return """
+用法:
+ python review.py 交互式选笔记
+ python review.py login 扫码登录
+ python review.py list 列最近笔记
+ python review.py note <note_id> [script.txt]
+ python review.py archive <notes.json> [output_root] [limit]
+ python review.py summarize [out_dir]
+""".strip()
+
+
def main() -> None:
if len(sys.argv) > 1 and sys.argv[1] == "login":
asyncio.run(crawler.ensure_login())
@@ -132,6 +384,22 @@ def main() -> None:
print(f"[{i}] {n['note_id']} {t} 曝光{renderer._fmt_num(n.get('view_count'))} {title}")
asyncio.run(_list())
return
+ if len(sys.argv) > 1 and sys.argv[1] == "archive":
+ if len(sys.argv) < 3:
+ print(_usage())
+ sys.exit(2)
+ input_path = Path(sys.argv[2].strip('"').strip("'")).expanduser().resolve()
+ output_root = Path(sys.argv[3]).expanduser().resolve() if len(sys.argv) > 3 else runtime_project_root() / "data" / "raw" / "notes" / "archive"
+ limit = int(sys.argv[4]) if len(sys.argv) > 4 else None
+ asyncio.run(run_archive(input_path, output_root, limit=limit))
+ return
+ if len(sys.argv) > 1 and sys.argv[1] == "summarize":
+ out_dir = Path(sys.argv[2]).expanduser().resolve() if len(sys.argv) > 2 else runtime_project_root() / "reports"
+ asyncio.run(run_summarize(out_dir))
+ return
+ if len(sys.argv) > 1 and sys.argv[1] in ("-h", "--help"):
+ print(_usage())
+ return
asyncio.run(run())