Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Python agent-infrastructureai-agentai-searchautomationbilibiliclaude-codeclicursorfree-apillm-toolsmcppythonreddit-scrapertwitter-scraperweb-scraperxiaohongshuyoutube-transcript
Initial: forked from runesleo/x-reader (MIT License) - thank you @runes_leo!
Panniantongee2ad83b1206
diff --git a/.env.example b/.env.examplenew file mode 100644index 00000000..248671e0--- /dev/null+++ b/.env.example@@ -0,0 +1,14 @@+# === Telegram (optional, for TG channel fetching) ===+TG_API_ID=+TG_API_HASH=++# === Groq Whisper (optional, for video/audio transcription) ===+# Free API key: https://console.groq.com/keys+GROQ_API_KEY=++# === Gemini (optional, for AI-powered analysis) ===+GEMINI_API_KEY=++# === Output ===+OUTPUT_DIR=./output+INBOX_FILE=./unified_inbox.jsondiff --git a/.gitignore b/.gitignorenew file mode 100644index 00000000..45357b9b--- /dev/null+++ b/.gitignore@@ -0,0 +1,32 @@+# Python+__pycache__/+*.py[cod]+*.egg-info/+dist/+build/+.venv/+venv/++# Environment+.env++# Telethon session+*.session+tg_session*++# Data+*.db+*.log+data/+unified_inbox.json+output/++# Audio/video temp files+/tmp/media_*++# OS+.DS_Store++# IDE+.vscode/+.idea/diff --git a/LICENSE b/LICENSEnew file mode 100644index 00000000..5fa59ae4--- /dev/null+++ b/LICENSE@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2026 Leo (@runes_leo)++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.diff --git a/README.md b/README.mdnew file mode 100644index 00000000..d28fc577--- /dev/null+++ b/README.md@@ -0,0 +1,247 @@+# x-reader++[](https://www.python.org/downloads/)+[](LICENSE)++Universal content reader — fetch, transcribe, and digest content from any platform.++Give it a URL (article, video, podcast, tweet), get back structured content. Works as CLI, Python library, MCP server, or Claude Code skills.++## What It Does++```+Any URL → Platform Detection → Fetch Content → Unified Output+ ↓ ↓+ auto-detect text: Jina Reader+ 7+ platforms video: yt-dlp subtitles+ audio: Whisper transcription+ API: Bilibili / RSS / Telegram+```++The Python layer handles text fetching and YouTube subtitle extraction. The **Claude Code skills** (optional) add full Whisper transcription for video/podcast and AI-powered content analysis.++## Three Layers++x-reader is composable. Use the layers you need:++| Layer | What | Format | Install |+|-------|------|--------|---------|+| **Python CLI/Library** | Basic content fetching + unified schema | See [Install](#install) | Required |+| **Claude Code Skills** | Video transcription + AI analysis | Copy `skills/` to `~/.claude/skills/` | Optional |+| **MCP Server** | Expose reading as MCP tools | `python mcp_server.py` | Optional |++### Layer 1: Python CLI++```bash+# Fetch any URL+x-reader https://mp.weixin.qq.com/s/abc123++# Fetch a tweet+x-reader https://x.com/elonmusk/status/123456++# Fetch multiple URLs+x-reader https://url1.com https://url2.com++# Login to a platform (one-time, for browser fallback)+x-reader login xhs++# View inbox+x-reader list+```++### Layer 2: Claude Code Skills++> Requires cloning the repo (not included in pip install).++For video/podcast transcription and content analysis:++```+skills/+├── video/ # YouTube/Bilibili/podcast → full transcript via Whisper+└── analyzer/ # Any content → structured analysis report+```++Install:+```bash+cp -r skills/video ~/.claude/skills/video+cp -r skills/analyzer ~/.claude/skills/analyzer+```++Then in Claude Code, just send a YouTube/Bilibili/podcast link — the video skill auto-triggers and produces a full transcript + summary.++### Layer 3: MCP Server++> Requires cloning the repo (mcp_server.py is not included in pip install).++```bash+git clone https://github.com/runesleo/x-reader.git+cd x-reader+pip install -e ".[mcp]"+python mcp_server.py+```++Tools exposed:+- `read_url(url)` — fetch any URL+- `read_batch(urls)` — fetch multiple URLs concurrently+- `list_inbox()` — view previously fetched content+- `detect_platform(url)` — identify platform from URL++Claude Code config (`~/.claude/claude_desktop_config.json`):+```json+{+ "mcpServers": {+ "x-reader": {+ "command": "python",+ "args": ["/path/to/x-reader/mcp_server.py"]+ }+ }+}+```++## Supported Platforms++| Platform | Text Fetch | Video/Audio Transcript |+|----------|-----------|----------------------|+| YouTube | ✅ Jina | ✅ yt-dlp subtitles → Groq Whisper fallback |+| Bilibili (B站) | ✅ API | ✅ via Claude Code skill |+| X / Twitter | ✅ Jina → Playwright | — |+| WeChat (微信公众号) | ✅ Jina → Playwright | — |+| Xiaohongshu (小红书) | ✅ Jina → Playwright* | — |+| Telegram | ✅ Telethon | — |+| RSS | ✅ feedparser | — |+| 小宇宙 (Xiaoyuzhou) | — | ✅ via Claude Code skill |+| Apple Podcasts | — | ✅ via Claude Code skill |+| Any web page | ✅ Jina fallback | — |++> \*XHS requires a one-time login: `x-reader login xhs` (saves session for Playwright fallback)+>+> YouTube Whisper transcription requires `GROQ_API_KEY` — get a free key from [Groq](https://console.groq.com/keys)++## Install++```bash+# From GitHub (recommended)+pip install git+https://github.com/runesleo/x-reader.git++# With Telegram support+pip install "x-reader[telegram] @ git+https://github.com/runesleo/x-reader.git"++# With browser fallback (Playwright — for XHS/WeChat anti-scraping)+pip install "x-reader[browser] @ git+https://github.com/runesleo/x-reader.git"+playwright install chromium++# With all optional dependencies+pip install "x-reader[all] @ git+https://github.com/runesleo/x-reader.git"+playwright install chromium+```++Or clone and install locally:+```bash+git clone https://github.com/runesleo/x-reader.git+cd x-reader+pip install -e ".[all]"+playwright install chromium+```++### Dependencies for video/audio (optional)++```bash+# macOS+brew install yt-dlp ffmpeg++# Linux+pip install yt-dlp+apt install ffmpeg+```++For Whisper transcription, get a free API key from [Groq](https://console.groq.com/keys) and set:+```bash+export GROQ_API_KEY=your_key_here+```++## Use as Library++```python+import asyncio+from x_reader.reader import UniversalReader++async def main():+ reader = UniversalReader()+ content = await reader.read("https://mp.weixin.qq.com/s/abc123")+ print(content.title)+ print(content.content[:200])++asyncio.run(main())+```++## Configuration++Copy `.env.example` to `.env`:++```bash+cp .env.example .env+```++| Variable | Required | Description |+|----------|----------|-------------|+| `TG_API_ID` | Telegram only | From https://my.telegram.org |+| `TG_API_HASH` | Telegram only | From https://my.telegram.org |+| `GROQ_API_KEY` | Whisper only | From https://console.groq.com/keys (free) |+| `INBOX_FILE` | No | Path to inbox JSON (default: `./unified_inbox.json`) |+| `OUTPUT_DIR` | No | Directory for Markdown output (default: disabled) |+| `OBSIDIAN_VAULT` | No | Path to Obsidian vault (writes to `01-收集箱/x-reader-inbox.md`) |++## Architecture++```+x-reader/+├── x_reader/ # Python package+│ ├── cli.py # CLI entry point+│ ├── reader.py # URL dispatcher (UniversalReader)+│ ├── schema.py # Unified data model (UnifiedContent + Inbox)+│ ├── login.py # Browser login manager (saves sessions)+│ ├── fetchers/+│ │ ├── jina.py # Jina Reader (universal fallback)+│ │ ├── browser.py # Playwright headless (anti-scraping fallback)+│ │ ├── bilibili.py # Bilibili API+│ │ ├── youtube.py # yt-dlp subtitle extraction+│ │ ├── rss.py # feedparser+│ │ ├── telegram.py # Telethon+│ │ ├── twitter.py # Jina-based+│ │ ├── wechat.py # Jina → Playwright fallback+│ │ └── xhs.py # Jina → Playwright + session fallback+│ └── utils/+│ └── storage.py # JSON + Markdown dual output+├── skills/ # Claude Code skills+│ ├── video/ # Video/podcast → transcript + summary+│ └── analyzer/ # Content → structured analysis+├── mcp_server.py # MCP server entry point+└── pyproject.toml+```++## How the Layers Work Together++```+User sends URL+ │+ ├─ Text content (article, tweet, WeChat)+ │ └─ Python fetcher → UnifiedContent → inbox+ │+ ├─ Video (YouTube, Bilibili, X video)+ │ ├─ Python fetcher → metadata (title, description)+ │ └─ Video skill → full transcript via subtitles/Whisper+ │+ ├─ Podcast (小宇宙, Apple Podcasts)+ │ └─ Video skill → full transcript via Whisper+ │+ └─ Analysis requested+ └─ Analyzer skill → structured report + action items+```++## Author++Built by [@runes_leo](https://x.com/runes_leo) — more AI tools at [leolabs.me](https://leolabs.me)++## License++MITdiff --git a/mcp_server.py b/mcp_server.pynew file mode 100644index 00000000..1cde552c--- /dev/null+++ b/mcp_server.py@@ -0,0 +1,102 @@+# -*- coding: utf-8 -*-+"""+x-reader MCP Server — expose content reading as MCP tools.++Usage:+ python mcp_server.py # stdio transport (for Claude Code)+ python mcp_server.py --transport sse # SSE transport (for web clients)++Claude Code config (~/.claude/claude_desktop_config.json):+ {+ "mcpServers": {+ "x-reader": {+ "command": "python",+ "args": ["/path/to/x-reader/mcp_server.py"]+ }+ }+ }+"""++import asyncio+from dotenv import load_dotenv+from mcp.server.fastmcp import FastMCP++load_dotenv()++from x_reader.reader import UniversalReader+from x_reader.schema import UnifiedInbox++mcp = FastMCP(+ "x-reader",+ instructions="Universal content reader — give it any URL, get structured content back.",+)++reader = UniversalReader(inbox=UnifiedInbox())+++async def read_url(url: str) -> str:+ """+ Read content from any URL and return structured result.++ Supports: YouTube, Bilibili, X/Twitter, WeChat, Xiaohongshu,+ Telegram, RSS, and any generic web page.++ Returns JSON with: title, content, url, source_type, platform metadata.+ """+ import json++ content = await reader.read(url)+ result = content.to_dict()+ # Keep it readable+ return json.dumps(result, ensure_ascii=False, indent=2)+++async def read_batch(urls: list[str]) -> str:+ """+ Read multiple URLs concurrently. Returns JSON array of results.++ Failed URLs are logged but don't block other results.+ """+ import json++ contents = await reader.read_batch(urls)+ results = [c.to_dict() for c in contents]+ return json.dumps(results, ensure_ascii=False, indent=2)+++async def list_inbox() -> str:+ """+ List all items in the content inbox.++ Returns JSON array of previously fetched content.+ """+ import json++ items = [item.to_dict() for item in reader.inbox.items]+ return json.dumps(items, ensure_ascii=False, indent=2)+++async def detect_platform(url: str) -> str:+ """+ Detect which platform a URL belongs to.++ Returns the platform name: youtube, bilibili, twitter, wechat,+ xhs, telegram, rss, or generic.+ """+ return reader._detect_platform(url)+++if __name__ == "__main__":+ import sys++ transport = "stdio"+ if "--transport" in sys.argv:+ idx = sys.argv.index("--transport")+ if idx + 1 < len(sys.argv):+ transport = sys.argv[idx + 1]++ mcp.run(transport=transport)diff --git a/pyproject.toml b/pyproject.tomlnew file mode 100644index 00000000..3155637c--- /dev/null+++ b/pyproject.toml@@ -0,0 +1,38 @@+[project]+name = "x-reader"+version = "0.2.0"+description = "Universal content reader — fetch, normalize, and digest content from 7+ platforms"+readme = "README.md"+license = {text = "MIT"}+requires-python = ">=3.10"+authors = [+ {name = "Leo", email = "[email protected]"}+]+keywords = ["content-reader", "rss", "telegram", "bilibili", "xiaohongshu", "digest"]+dependencies = [+ "requests>=2.28",+ "feedparser>=6.0",+ "python-dotenv>=1.0",+ "loguru>=0.7",+]++[project.urls]+Homepage = "https://github.com/runesleo/x-reader"+Repository = "https://github.com/runesleo/x-reader"+Issues = "https://github.com/runesleo/x-reader/issues"++[project.optional-dependencies]+telegram = ["telethon>=1.34"]+mcp = ["mcp[cli]>=1.0"]+browser = ["playwright>=1.40"]+all = ["telethon>=1.34", "mcp[cli]>=1.0", "playwright>=1.40"]++[project.scripts]+x-reader = "x_reader.cli:main"++[build-system]+requires = ["hatchling"]+build-backend = "hatchling.build"++[tool.hatch.build.targets.wheel]+packages = ["x_reader"]diff --git a/skills/analyzer/skill.md b/skills/analyzer/skill.mdnew file mode 100644index 00000000..ed1953bb--- /dev/null+++ b/skills/analyzer/skill.md@@ -0,0 +1,131 @@+# Content Analyzer Skill++> Any content → structured analysis report with actionable insights++## Trigger++When user sends content (URL, text, or transcript) with analysis intent:+- `/analyze [URL]`+- "Analyze this article"+- "What are the key takeaways?"+- Auto-triggered after video/podcast transcription (from video skill)++## Pipeline++### Step 1: Get Content++Choose tool based on input type:++| Input | Tool |+|-------|------|+| Tweet URL | `fetch_tweet` or Jina Reader |+| Web URL | `WebFetch` or Jina Reader |+| Local file | Read file directly |+| Transcript from video skill | Use directly |++### Step 2: Multi-Dimensional Analysis++Scan content across these dimensions. Only output dimensions with actual content — skip empty ones.++```markdown+## 📖 Summary++[1-3 sentence core thesis]++**Source**: [author/publisher] · [date]+**Type**: [tweet/article/video/podcast/report]++---++## 💡 Key Insights++### 🎯 Core Arguments+- **Thesis**: [Main argument or finding]+- **Evidence**: [Supporting data or reasoning]+- **Strength**: [How convincing? What's missing?]++### 🤖 Tools & Methods+- **What**: [Tools, frameworks, or techniques mentioned]+- **How**: [How they're used or applied]+- **Relevance**: [Could you use this?]++### ⚙️ Workflow Ideas+- **Optimization**: [Process improvements mentioned]+- **Automation**: [What could be automated]+- **Integration**: [How to fit into existing workflow]++### 📊 Data & Numbers+- **Key metrics**: [Important numbers mentioned]+- **Trends**: [Patterns in the data]+- **Gaps**: [What data is missing]++### ⚠️ Risks & Warnings+- **Pitfalls**: [Explicitly mentioned risks]+- **Blind spots**: [What the author might be missing]+- **Counter-arguments**: [Alternative perspectives]++### 🔗 Resources+- **Tools/APIs**: [Mentioned tools or data sources]+- **People**: [Worth following or referencing]+- **Further reading**: [Related content]++### 🧠 Mental Model Shifts+- **Before**: [Common assumption]+- **After**: [New understanding from this content]+- **Impact**: [How this changes decisions]++---++## ✅ Action Items++### Quick Wins (< 30 min)+- [ ] [Action 1] — Impact: ★★★★ | Effort: Easy+- [ ] [Action 2] — Impact: ★★★ | Effort: Easy++### Deeper Work (1-3 hours)+- [ ] [Action 3] — Impact: ★★★ | Effort: Medium+- [ ] [Action 4] — Impact: ★★ | Effort: Medium++### Exploration (needs validation)+- [ ] [Action 5] — Impact: ★★★ | Effort: Hard | Nature: Exploratory+```++### Step 3: Personalized Relevance (Customizable)++Map insights to YOUR context. Edit the dimensions below to match your own projects, interests, and systems.++```markdown+## 🔄 How This Applies to Me++### My Projects+- **[Project A]**: [How this insight connects]+- **[Project B]**: [What I could apply]++### My Knowledge Base+- **Update**: [Which notes/docs to update]+- **New entry**: [What to add to my knowledge system]++### My Decision Log+- **Changed my mind about**: [what and why]+- **Confirmed my belief that**: [what]+```++> **Customization**: Edit the dimensions in Step 2 and Step 3 to match your own+> domain. A trader might add "Market Impact" and "Risk Assessment". A developer+> might add "Architecture Patterns" and "Tech Debt". Make it yours.++## Output Modes++| Mode | Trigger | Output |+|------|---------|--------|+| **Full** (default) | `/analyze [URL]` | All dimensions |+| **Sparse** | `/analyze [URL] --sparse` | Only hit dimensions, skip empty |+| **Brief** | `/analyze [URL] --brief` | Action items only |++## Best Practices++1. **Scan all dimensions, but don't force-fill** — skip empty dimensions cleanly+2. **Actions must be specific** — not "learn about X" but "read X docs chapter Y"+3. **Distinguish fact from opinion** — mark the author's claims vs verified facts+4. **Source everything** — tag where each insight comes from in the original content+5. **ROI awareness** — not every action is worth doing, assess effort vs impactdiff --git a/skills/video/skill.md b/skills/video/skill.mdnew file mode 100644index 00000000..97298027--- /dev/null+++ b/skills/video/skill.md@@ -0,0 +1,292 @@+# Video & Podcast Digest Skill++> Send a video/podcast link → get full transcript + structured summary++## Supported Platforms++| Platform | Type | Subtitles | Whisper Transcription |+|----------|------|-----------|----------------------|+| YouTube | Video | ✅ | ✅ |+| Bilibili | Video | ✅ | ✅ |+| X/Twitter | Video | ❌ | ✅ |+| Xiaoyuzhou (小宇宙) | Podcast | ❌ | ✅ |+| Apple Podcasts | Podcast | ❌ | ✅ |+| Direct links (mp3/mp4/m3u8) | Any | ❌ | ✅ |++## Trigger++Auto-triggered when a media URL is detected:+- YouTube: `youtube.com`, `youtu.be`+- Bilibili: `bilibili.com`, `b23.tv`+- X/Twitter: `x.com`, `twitter.com` (tweets with video)+- Xiaoyuzhou: `xiaoyuzhoufm.com`+- Apple Podcasts: `podcasts.apple.com`+- Direct: `.mp3`, `.mp4`, `.m3u8`, `.m4a`, `.webm`++## Pipeline++### Step 0: Detect Media Type++| URL Pattern | Type | Pipeline |+|-------------|------|----------|+| `xiaoyuzhoufm.com/episode/` | Podcast | → Step 1b (Xiaoyuzhou) |+| `podcasts.apple.com` | Podcast | → Step 1c (Apple) |+| `bilibili.com`, `b23.tv` | Video | → Step 1d (Bilibili API) |+| `.mp3`, `.m4a` direct link | Audio | → Step 2b (direct download) |+| Other | Video | → Step 1a (subtitle extraction) |++### Step 1a: Video — Extract Subtitles++```bash+# Clean up temp files+rm -f /tmp/media_sub*.vtt /tmp/media_audio.mp3 /tmp/media_transcript*.json /tmp/media_segment_*.mp3 2>/dev/null || true++# YouTube (prefer English, fallback Chinese)+yt-dlp --skip-download --write-auto-sub --sub-lang "en,zh-Hans" -o "/tmp/media_sub" "VIDEO_URL"++# Bilibili+yt-dlp --skip-download --write-auto-sub --sub-lang "zh-Hans,zh" -o "/tmp/media_sub" "VIDEO_URL"+```++Check for subtitles:+```bash+ls /tmp/media_sub*.vtt 2>/dev/null+```+- **Has subtitles** → Read VTT content, skip to Step 3+- **No subtitles** → Step 2a (download audio)++### Step 1b: Xiaoyuzhou (小宇宙) — Extract Audio URL++```bash+# Extract CDN direct link from __NEXT_DATA__+# Xiaoyuzhou is a Next.js SPA, but initial HTML contains audio URL in __NEXT_DATA__+AUDIO_URL=$(curl -sL -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \+ "EPISODE_URL" \+ | grep -oE 'https://media\.xyzcdn\.net/[^"]+\.(m4a|mp3)' \+ | head -1)++echo "Audio URL: $AUDIO_URL"++# Download audio+curl -L -o /tmp/media_audio.mp3 "$AUDIO_URL"+```++> If curl extraction is empty (rare), fallback: use Puppeteer/browser to get rendered page and extract.++→ Step 2b (check size & transcribe)++### Step 1c: Apple Podcasts — via yt-dlp++```bash+yt-dlp -f "ba[ext=m4a]/ba/b" --extract-audio --audio-format mp3 --audio-quality 5 \+ -o "/tmp/media_audio.%(ext)s" "APPLE_PODCAST_URL"+```++→ Step 2b (check size & transcribe)++### Step 1d: Bilibili — API Direct Audio Stream++yt-dlp returns 412 for Bilibili even with cookies. Use Bilibili's API instead:++```bash+# 1. Extract BV number from URL+BV="BV1xxxxx" # Replace with actual BV number++# 2. Get video info (title, duration, CID)+curl -s "https://api.bilibili.com/x/web-interface/view?bvid=$BV" \+ -H "User-Agent: Mozilla/5.0" -H "Referer: https://www.bilibili.com/" \+ | python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(f\"Title: {d['title']}\nDuration: {d['duration']}s\nCID: {d['cid']}\")"++# 3. Get audio stream URL+CID=<CID from previous step>+AUDIO_URL=$(curl -s "https://api.bilibili.com/x/player/playurl?bvid=$BV&cid=$CID&fnval=16&qn=64" \+ -H "User-Agent: Mozilla/5.0" -H "Referer: https://www.bilibili.com/" \+ | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['dash']['audio'][0]['baseUrl'])")++# 4. Download audio (Referer header required, otherwise 403)+curl -L -o /tmp/media_audio.m4s \+ -H "User-Agent: Mozilla/5.0" -H "Referer: https://www.bilibili.com/" "$AUDIO_URL"++# 5. Convert to mp3+ffmpeg -y -i /tmp/media_audio.m4s -acodec libmp3lame -q:a 5 /tmp/media_audio.mp3+```++→ Step 2b (check size & transcribe)++### Step 2a: Video — Download Audio (when no subtitles)++```bash+# YouTube may need --cookies-from-browser chrome to bypass bot detection+yt-dlp --cookies-from-browser chrome -f "ba[ext=m4a]/ba/b" --extract-audio --audio-format mp3 --audio-quality 5 \+ -o "/tmp/media_audio.%(ext)s" "VIDEO_URL"+```++### Step 2b: Check Audio Size & Segment++```bash+FILE_SIZE=$(stat -f%z /tmp/media_audio.* 2>/dev/null || stat -c%s /tmp/media_audio.* 2>/dev/null)+echo "File size: $FILE_SIZE bytes"+```++- **≤ 25MB (25000000)** → Step 2c (transcribe directly)+- **> 25MB** → Split first, then transcribe each segment++**Splitting large audio (>25MB)**:+```bash+# Get total duration+DURATION=$(ffprobe -v error -show_entries format=duration -of csv=p=0 /tmp/media_audio.* | head -1)++# Split into 10-minute segments (keeps each under 25MB)+SEGMENT_SEC=600+SEGMENTS=$(python3 -c "import math; print(math.ceil(float('$DURATION')/$SEGMENT_SEC))")++# Cut segments+for i in $(seq 0 $((SEGMENTS-1))); do+ START=$((i * SEGMENT_SEC))+ ffmpeg -y -i /tmp/media_audio.* -ss $START -t $SEGMENT_SEC -acodec libmp3lame -q:a 5 \+ "/tmp/media_segment_${i}.mp3" 2>/dev/null+done+```++→ Call Step 2c for each segment **sequentially** (parallel triggers Groq 524 timeout), concatenate results++### Step 2c: Whisper Transcription++**Prerequisite**: `GROQ_API_KEY` environment variable++```bash+# Check API key+if [ -z "$GROQ_API_KEY" ]; then+ echo "❌ GROQ_API_KEY not set. Get one at: https://console.groq.com/keys"+ exit 1+fi++# Transcribe single file (replace AUDIO_FILE with actual path)+curl -s -X POST "https://api.groq.com/openai/v1/audio/transcriptions" \+ -H "Authorization: Bearer $GROQ_API_KEY" \+ -H "Content-Type: multipart/form-data" \+ -F "file=@AUDIO_FILE" \+ -F "model=whisper-large-v3-turbo" \+ -F "response_format=verbose_json" \+ -F "language=zh" \+ > /tmp/media_transcript.json++# Extract plain text+python3 -c "import json; print(json.load(open('/tmp/media_transcript.json'))['text'])"+```++**Whisper model options**:+| Model | Speed | Accuracy | Use Case |+|-------|-------|----------|----------|+| `whisper-large-v3-turbo` | 10x realtime | High | Default choice |+| `whisper-large-v3` | 5x realtime | Highest | Professional/noisy content |++**Language parameter**:+- Chinese: `language=zh`+- English: `language=en`+- Auto-detect: omit language parameter++### Step 3: Structured Summary++Choose output format based on media type:++**Video (≤20 min)**:+1. **Overview** (1-2 sentences)+2. **Key Points** (3-5 bullet points)+3. **Notable Quotes** (if any)+4. **Action Items** (if applicable)++**Podcast (>20 min)**:+1. **Overview** (2-3 sentences: who discussed what)+2. **Chapter Summary** (segmented by topic, 2-3 sentences each)+3. **Key Points** (5-8 bullet points)+4. **Notable Quotes**+5. **Action Items** (if applicable)++## Output Format++### Video++```+## 📺 Video Digest++**Title**: [Video Title]+**Duration**: [x minutes]+**Language**: [Chinese/English]++### Overview+[1-2 sentence summary]++### Key Points+1. [Point 1]+2. [Point 2]+...++### Notable Quotes+> "xxx" — [timestamp]++### Action Items+- [if applicable]+```++### Podcast++```+## 🎙️ Podcast Digest++**Show**: [Podcast Name]+**Episode**: [Episode Title]+**Duration**: [x minutes]+**Guests**: [if any]++### Overview+[2-3 sentences: who discussed what, core conclusions]++### Chapter Summary+#### 1. [Topic] (~xx:xx-xx:xx)+[2-3 sentences of core content]++#### 2. [Topic] (~xx:xx-xx:xx)+[2-3 sentences of core content]+...++### Key Points+1. [Point 1]+2. [Point 2]+...++### Notable Quotes+> "xxx"++### Action Items+- [if applicable]+```++## Error Handling++| Situation | Action |+|-----------|--------|+| No subtitles + no GROQ_API_KEY | Prompt user to set API key |+| No subtitles + has API key | Auto Whisper transcription |+| Xiaoyuzhou curl extraction empty | Use Puppeteer/browser to get rendered HTML |+| Audio >25MB | ffmpeg segment (10min/segment), transcribe sequentially |+| Podcast >2 hours | Warn user about duration, confirm before proceeding |+| Groq 524 timeout | Do NOT parallelize — transcribe sequentially, sleep 5-8s between segments |+| Groq 429 rate limit | 7200s/hour limit, wait for retry-after header, then retry |+| yt-dlp Bilibili 412 | Use Bilibili API instead (Step 1d) |+| yt-dlp YouTube bot detection | Add `--cookies-from-browser chrome` |+| Network timeout | Retry once |+| Spotify links | Inform user: not supported (DRM protected) |++## Groq Whisper Limits++- Max 25MB per request+- Free tier: 7200 seconds of audio/hour (rolling window), ~20 hours/day+- Supported formats: mp3, mp4, mpeg, mpga, m4a, wav, webm++## Dependencies++- `yt-dlp`: video download + subtitle extraction+- `ffmpeg`: audio conversion + segmentation+- `curl`: Xiaoyuzhou audio download, Bilibili API+- `GROQ_API_KEY`: Whisper transcription API (free at https://console.groq.com/keys)diff --git a/x_reader/__init__.py b/x_reader/__init__.pynew file mode 100644index 00000000..8ab0f3ef--- /dev/null+++ b/x_reader/__init__.py@@ -0,0 +1,3 @@+"""x-reader: Universal content reader for 7+ platforms."""++__version__ = "0.1.0"diff --git a/x_reader/cli.py b/x_reader/cli.pynew file mode 100644index 00000000..c0b28d43--- /dev/null+++ b/x_reader/cli.py@@ -0,0 +1,139 @@+# -*- coding: utf-8 -*-+"""+x-reader CLI — fetch content from any platform.++Usage:+ x-reader <url> # Fetch a single URL+ x-reader <url1> <url2> ... # Fetch multiple URLs+ x-reader list # Show inbox contents+ x-reader clear # Clear inbox+"""++import sys+import asyncio+import json+from pathlib import Path++from dotenv import load_dotenv+load_dotenv()++from x_reader.reader import UniversalReader+from x_reader.schema import UnifiedInbox, SourceType+++def get_inbox_path() -> str:+ import os+ return os.getenv("INBOX_FILE", "unified_inbox.json")+++def cmd_fetch(urls: list[str]):+ """Fetch one or more URLs."""+ inbox = UnifiedInbox(get_inbox_path())+ reader = UniversalReader(inbox=inbox)++ async def run():+ if len(urls) == 1:+ item = await reader.read(urls[0])+ print(f"✅ [{item.source_type.value}] {item.title[:60]}")+ print(f" {item.url}")+ print(f" {item.content[:200]}...")+ else:+ items = await reader.read_batch(urls)+ for item in items:+ print(f"✅ [{item.source_type.value}] {item.title[:60]}")+ print(f"\n📦 Fetched {len(items)}/{len(urls)} URLs")++ try:+ asyncio.run(run())+ except KeyboardInterrupt:+ print("\n⏹ Cancelled")+ except Exception as e:+ print(f"❌ {e}")+ sys.exit(1)+++def cmd_list():+ """Show inbox contents."""+ inbox = UnifiedInbox(get_inbox_path())+ if not inbox.items:+ print("📦 Inbox is empty")+ return++ print(f"📦 Inbox: {len(inbox.items)} items\n")++ emoji_map = {+ SourceType.TELEGRAM: "📢", SourceType.RSS: "📰",+ SourceType.BILIBILI: "🎬", SourceType.XIAOHONGSHU: "📕",+ SourceType.TWITTER: "🐦", SourceType.WECHAT: "💬",+ SourceType.YOUTUBE: "▶️", SourceType.MANUAL: "✏️",+ }++ for i, item in enumerate(inbox.items[-20:], 1):+ emoji = emoji_map.get(item.source_type, "📄")+ print(f" {i:2d}. {emoji} [{item.source_type.value:8s}] {item.title[:50]}")+++def cmd_clear():+ """Clear inbox."""+ path = Path(get_inbox_path())+ if path.exists():+ confirm = input("Clear inbox? (y/N) ")+ if confirm.lower() == 'y':+ path.write_text("[]")+ print("✅ Inbox cleared")+ else:+ print("📦 Inbox is already empty")+++def cmd_login(platform: str):+ """Open browser for manual login to a platform."""+ from x_reader.login import login+ login(platform)+++def main():+ if len(sys.argv) < 2:+ print("""+📖 x-reader — Universal content reader++Usage:+ x-reader <url> Fetch content from any URL+ x-reader <url1> <url2> Fetch multiple URLs+ x-reader login <platform> Login to a platform (saves session for browser fallback)+ x-reader list Show inbox contents+ x-reader clear Clear inbox++Supported platforms:+ WeChat, Telegram, X/Twitter, YouTube,+ Bilibili, Xiaohongshu, RSS, and any web page++Examples:+ x-reader https://mp.weixin.qq.com/s/abc123+ x-reader https://x.com/elonmusk/status/123456+ x-reader https://www.xiaohongshu.com/explore/abc123+ x-reader login xhs+""")+ return++ cmd = sys.argv[1].lower()++ if cmd == "login":+ if len(sys.argv) < 3:+ print("❌ Usage: x-reader login <platform>")+ print(" Supported: xhs, wechat")+ sys.exit(1)+ cmd_login(sys.argv[2])+ elif cmd == "list":+ cmd_list()+ elif cmd == "clear":+ cmd_clear()+ elif cmd.startswith("http") or cmd.startswith("www.") or "." in cmd:+ urls = [arg for arg in sys.argv[1:] if arg.startswith(("http", "www.")) or "." in arg]+ cmd_fetch(urls)+ else:+ print(f"❌ Unknown command: {cmd}")+ print(" Run 'x-reader' with no args for help")+++if __name__ == "__main__":+ main()diff --git a/x_reader/fetchers/__init__.py b/x_reader/fetchers/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/x_reader/fetchers/bilibili.py b/x_reader/fetchers/bilibili.pynew file mode 100644index 00000000..34305ae2--- /dev/null+++ b/x_reader/fetchers/bilibili.py@@ -0,0 +1,46 @@+# -*- coding: utf-8 -*-+"""Bilibili video fetcher — uses official web API."""++import re+import requests+from loguru import logger+from typing import Dict, Any+++API_URL = "https://api.bilibili.com/x/web-interface/view"+HEADERS = {+ "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"+}+++async def fetch_bilibili(url_or_bv: str) -> Dict[str, Any]:+ """Fetch Bilibili video metadata via official API."""+ logger.info(f"Fetching Bilibili: {url_or_bv}")++ bv_id = url_or_bv+ if "bilibili.com" in url_or_bv or "b23.tv" in url_or_bv:+ match = re.search(r'BV\w+', url_or_bv)+ if match:+ bv_id = match.group()+ else:+ raise ValueError(f"Cannot extract BV ID from: {url_or_bv}")++ resp = requests.get(API_URL, params={"bvid": bv_id}, headers=HEADERS, timeout=10)+ resp.raise_for_status()+ data = resp.json()++ if data.get("code") != 0:+ raise ValueError(f"Bilibili API error: {data.get('message')}")++ video = data["data"]+ return {+ "title": video.get("title", ""),+ "description": video.get("desc", ""),+ "author": video.get("owner", {}).get("name", ""),+ "url": f"https://www.bilibili.com/video/{bv_id}",+ "cover": video.get("pic", ""),+ "bvid": bv_id,+ "duration": video.get("duration", 0),+ "view_count": video.get("stat", {}).get("view", 0),+ "platform": "bilibili",+ }diff --git a/x_reader/fetchers/browser.py b/x_reader/fetchers/browser.pynew file mode 100644index 00000000..10ac3479--- /dev/null+++ b/x_reader/fetchers/browser.py@@ -0,0 +1,88 @@+# -*- coding: utf-8 -*-+"""+Playwright browser fetcher — headless Chromium fallback for anti-scraping sites.++Used when Jina Reader fails (403/451/timeout). Supports persistent login+sessions via Playwright's storage_state for platforms requiring authentication.++Install: pip install "x-reader[browser]" && playwright install chromium+"""++from loguru import logger+from pathlib import Path++SESSION_DIR = Path.home() / ".x-reader" / "sessions"+TIMEOUT_MS = 30_000+++async def fetch_via_browser(url: str, storage_state: str = None) -> dict:+ """+ Fetch a URL using headless Chromium via Playwright.++ Args:+ url: Target URL to fetch.+ storage_state: Path to a Playwright storage state JSON file (cookies/localStorage).+ If provided, the browser context will load this session.++ Returns:+ dict with keys: title, content, url, author+ """+ try:+ from playwright.async_api import async_playwright+ except ImportError:+ raise RuntimeError(+ "Playwright is not installed. Run:\n"+ ' pip install "x-reader[browser]"\n'+ " playwright install chromium"+ )++ logger.info(f"Browser fetch: {url}")++ async with async_playwright() as p:+ browser = await p.chromium.launch(headless=True)++ context_kwargs = {}+ if storage_state and Path(storage_state).exists():+ context_kwargs["storage_state"] = storage_state+ logger.info(f"Using session: {storage_state}")++ context = await browser.new_context(+ user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "+ "AppleWebKit/537.36 (KHTML, like Gecko) "+ "Chrome/120.0.0.0 Safari/537.36",+ **context_kwargs,+ )+ page = await context.new_page()++ try:+ await page.goto(url, wait_until="domcontentloaded", timeout=TIMEOUT_MS)+ # Extra wait for JS-heavy pages+ await page.wait_for_timeout(2000)++ title = await page.title()+ # Extract main text content, stripping scripts/styles+ content = await page.evaluate("""() => {+ const el = document.querySelector('article')+ || document.querySelector('main')+ || document.querySelector('.content')+ || document.body;+ return el ? el.innerText : '';+ }""")++ result = {+ "title": (title or "").strip()[:200],+ "content": (content or "").strip(),+ "url": url,+ "author": "",+ }+ logger.info(f"Browser fetch OK: {title[:60]}")+ return result++ finally:+ await context.close()+ await browser.close()+++def get_session_path(platform: str) -> str:+ """Get the session file path for a platform."""+ return str(SESSION_DIR / f"{platform}.json")diff --git a/x_reader/fetchers/jina.py b/x_reader/fetchers/jina.pynew file mode 100644index 00000000..3ee68822--- /dev/null+++ b/x_reader/fetchers/jina.py@@ -0,0 +1,63 @@+# -*- coding: utf-8 -*-+"""+Jina Reader — universal fallback for content extraction.++Uses https://r.jina.ai/{url} to extract markdown from any web page.+Free, no API key required, handles JS rendering and anti-scraping.+"""++import requests+from loguru import logger+++JINA_BASE = "https://r.jina.ai"+TIMEOUT = 30++HEADERS = {+ "Accept": "text/markdown",+ "User-Agent": "x-reader/0.1",+}+++def fetch_via_jina(url: str) -> dict:+ """+ Fetch any URL via Jina Reader and return structured data.++ Returns:+ dict with keys: title, content, url, author (best-effort)+ """+ jina_url = f"{JINA_BASE}/{url}"+ logger.info(f"Jina fetch: {url}")++ try:+ resp = requests.get(jina_url, headers=HEADERS, timeout=TIMEOUT)+ resp.raise_for_status()+ text = resp.text++ # Jina returns markdown; first line is usually the title+ lines = text.strip().split("\n")+ title = ""+ content_lines = []++ for line in lines:+ if not title and line.strip():+ # First non-empty line as title, strip markdown heading+ title = line.lstrip("#").strip()+ else:+ content_lines.append(line)++ content = "\n".join(content_lines).strip()++ return {+ "title": title[:200],+ "content": content,+ "url": url,+ "author": "",+ }++ except requests.Timeout:+ logger.error(f"Jina timeout: {url}")+ raise+ except requests.RequestException as e:+ logger.error(f"Jina fetch failed: {url} — {e}")+ raisediff --git a/x_reader/fetchers/rss.py b/x_reader/fetchers/rss.pynew file mode 100644index 00000000..666155db--- /dev/null+++ b/x_reader/fetchers/rss.py@@ -0,0 +1,47 @@+# -*- coding: utf-8 -*-+"""RSS feed fetcher — uses feedparser."""++import feedparser+from loguru import logger+from typing import Dict, Any, List+++async def fetch_rss(url: str, limit: int = 20) -> List[Dict[str, Any]]:+ """+ Fetch and parse an RSS/Atom feed.++ Args:+ url: RSS feed URL+ limit: Max number of entries to return++ Returns:+ List of article dicts with: title, summary, url, source, published+ """+ logger.info(f"Fetching RSS: {url}")++ feed = feedparser.parse(url)++ if feed.bozo and not feed.entries:+ raise ValueError(f"Failed to parse RSS feed: {feed.bozo_exception}")++ source_name = feed.feed.get("title", url)+ articles = []++ for entry in feed.entries[:limit]:+ summary = ""+ if hasattr(entry, "summary"):+ summary = entry.summary+ elif hasattr(entry, "content"):+ summary = entry.content[0].get("value", "")++ articles.append({+ "title": entry.get("title", ""),+ "summary": summary,+ "url": entry.get("link", ""),+ "source": source_name,+ "published": entry.get("published", ""),+ "platform": "rss",+ })++ logger.info(f"RSS: {len(articles)} articles from {source_name}")+ return articlesdiff --git a/x_reader/fetchers/telegram.py b/x_reader/fetchers/telegram.pynew file mode 100644index 00000000..803339dc--- /dev/null+++ b/x_reader/fetchers/telegram.py@@ -0,0 +1,71 @@+# -*- coding: utf-8 -*-+"""+Telegram channel fetcher — uses Telethon.++Requires: pip install x-reader[telegram]+Requires: TG_API_ID + TG_API_HASH in .env+"""++import os+from datetime import datetime, timedelta, timezone+from loguru import logger+from typing import Dict, Any, List+++async def fetch_telegram(+ channel: str,+ limit: int = 20,+ hours: int = 24,+ session_path: str = None,+) -> List[Dict[str, Any]]:+ """+ Fetch recent messages from a Telegram channel.++ Args:+ channel: Channel username (e.g. 'predictionmkt')+ limit: Max messages per channel+ hours: Only fetch messages from the last N hours+ session_path: Path to Telethon session file++ Returns:+ List of message dicts+ """+ try:+ from telethon import TelegramClient+ from telethon.tl.types import Message+ except ImportError:+ raise ImportError(+ "Telethon is required for Telegram fetching. "+ "Install with: pip install x-reader[telegram]"+ )++ api_id = os.getenv("TG_API_ID", "")+ api_hash = os.getenv("TG_API_HASH", "")++ if not api_id or not api_hash:+ raise ValueError("TG_API_ID and TG_API_HASH must be set in .env")++ session = session_path or os.getenv("TG_SESSION_PATH", "./tg_session")+ cutoff = datetime.now(timezone.utc) - timedelta(hours=hours)++ messages = []+ async with TelegramClient(session, int(api_id), api_hash) as client:+ logger.info(f"Fetching TG channel: {channel}")+ entity = await client.get_entity(channel)++ async for msg in client.iter_messages(entity, limit=limit):+ if not isinstance(msg, Message) or not msg.text:+ continue+ if msg.date < cutoff:+ break++ messages.append({+ "text": msg.text,+ "views": msg.views or 0,+ "date": msg.date.isoformat(),+ "url": f"https://t.me/{channel}/{msg.id}",+ "platform": "telegram",+ })++ logger.info(f"TG {channel}: {len(messages)} messages")+ return messagesdiff --git a/x_reader/fetchers/twitter.py b/x_reader/fetchers/twitter.pynew file mode 100644index 00000000..362881ba--- /dev/null+++ b/x_reader/fetchers/twitter.py@@ -0,0 +1,217 @@+# -*- coding: utf-8 -*-+"""+X/Twitter fetcher — three-tier fallback:++1. X oEmbed API (fast, reliable for individual tweets, no login needed)+2. Jina Reader (handles non-tweet X pages like profiles)+3. Playwright + saved session (handles login-required content)++Install browser tier: pip install "x-reader[browser]" && playwright install chromium+Save X session: x-reader login twitter+"""++import re+import requests+from loguru import logger+from typing import Dict, Any++from x_reader.fetchers.jina import fetch_via_jina+++OEMBED_URL = "https://publish.twitter.com/oembed"+++def _extract_author(url: str) -> str:+ """Extract @username from tweet URL."""+ match = re.search(r'x\.com/(\w+)/status', url)+ return f"@{match.group(1)}" if match else ""+++def _is_tweet_url(url: str) -> bool:+ """Check if this is a direct tweet/status URL (vs profile or other X page)."""+ return bool(re.search(r'x\.com/\w+/status/\d+', url))+++def _fetch_via_oembed(url: str) -> Dict[str, Any]:+ """+ Fetch tweet text via X's oEmbed API.+ Free, reliable, no auth needed. Works for public tweets.+ Note: oEmbed requires twitter.com URLs (not x.com).+ """+ # oEmbed API requires twitter.com format+ oembed_query_url = url.replace("x.com", "twitter.com")+ resp = requests.get(+ OEMBED_URL,+ params={"url": oembed_query_url, "omit_script": "true"},+ timeout=10,+ )+ resp.raise_for_status()+ data = resp.json()++ # Strip HTML tags from the embedded HTML to get clean text+ html = data.get("html", "")+ text = re.sub(r'<[^>]+>', ' ', html)+ text = re.sub(r'\s+', ' ', text).strip()++ return {+ "text": text,+ "author": data.get("author_name", ""),+ "author_url": data.get("author_url", ""),+ "title": text[:100] if text else "",+ }+++async def _fetch_via_playwright(url: str) -> Dict[str, Any]:+ """+ Fetch tweet via Playwright with X-specific DOM selectors.+ Uses saved login session if available (~/.x-reader/sessions/twitter.json).+ """+ try:+ from playwright.async_api import async_playwright+ except ImportError:+ raise RuntimeError(+ "Playwright not installed. Run:\n"+ ' pip install "x-reader[browser]"\n'+ " playwright install chromium"+ )++ from x_reader.fetchers.browser import get_session_path+ from pathlib import Path++ session_path = get_session_path("twitter")+ has_session = Path(session_path).exists()+ if has_session:+ logger.info(f"Using saved X session: {session_path}")++ async with async_playwright() as p:+ browser = await p.chromium.launch(headless=True)++ context_kwargs = {}+ if has_session:+ context_kwargs["storage_state"] = session_path++ context = await browser.new_context(+ user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "+ "AppleWebKit/537.36 (KHTML, like Gecko) "+ "Chrome/120.0.0.0 Safari/537.36",+ **context_kwargs,+ )+ page = await context.new_page()++ try:+ await page.goto(url, wait_until="domcontentloaded", timeout=30_000)++ # Wait for tweet text to render (X is a SPA, needs JS execution)+ try:+ await page.wait_for_selector(+ '[data-testid="tweetText"]', timeout=10_000+ )+ except Exception:+ pass # May not appear if login required++ # Extract tweet content with X-specific selectors+ tweet_text = await page.evaluate("""() => {+ // Priority 1: tweet text element+ const tweetEl = document.querySelector('[data-testid="tweetText"]');+ if (tweetEl) return tweetEl.innerText;++ // Priority 2: article element (thread view)+ const article = document.querySelector('article');+ if (article) return article.innerText;++ // Priority 3: main content area+ const main = document.querySelector('main');+ if (main) return main.innerText;++ return '';+ }""")++ title = await page.title()++ return {+ "text": (tweet_text or "").strip(),+ "title": (title or "").strip()[:200],+ }+ finally:+ await context.close()+ await browser.close()+++async def fetch_twitter(url: str) -> Dict[str, Any]:+ """+ Fetch a tweet or X post with three-tier fallback.++ Args:+ url: Tweet URL (x.com or twitter.com)++ Returns:+ Dict with: text, author, url, title, platform+ """+ url = url.replace("twitter.com", "x.com")+ author = _extract_author(url)++ # Tier 1: oEmbed API (best for individual tweets)+ if _is_tweet_url(url):+ try:+ logger.info(f"[Twitter] Tier 1 — oEmbed: {url}")+ data = _fetch_via_oembed(url)+ if data.get("text") and len(data["text"].strip()) > 20:+ return {+ "text": data["text"],+ "author": author or data.get("author", ""),+ "url": url,+ "title": data.get("title", ""),+ "platform": "twitter",+ }+ logger.warning("[Twitter] oEmbed returned thin content")+ except Exception as e:+ logger.warning(f"[Twitter] oEmbed failed ({e})")++ # Tier 2: Jina Reader (handles profiles, threads, non-tweet pages)+ try:+ logger.info(f"[Twitter] Tier 2 — Jina: {url}")+ data = fetch_via_jina(url)+ content = data.get("content", "")+ title = data.get("title", "")+ jina_ok = (+ content+ and len(content.strip()) > 100+ and "not yet fully loaded" not in content.lower()+ and title.lower() not in ("x", "title: x", "")+ )+ if jina_ok:+ return {+ "text": content,+ "author": author,+ "url": url,+ "title": title,+ "platform": "twitter",+ }+ logger.warning("[Twitter] Jina returned unusable content")+ except Exception as e:+ logger.warning(f"[Twitter] Jina failed ({e})")++ # Tier 3: Playwright + session with X-specific extraction+ try:+ logger.info(f"[Twitter] Tier 3 — Playwright: {url}")+ data = await _fetch_via_playwright(url)+ content = data.get("text", "")+ if content and len(content.strip()) > 20:+ return {+ "text": content,+ "author": author,+ "url": url,+ "title": data.get("title", ""),+ "platform": "twitter",+ }+ logger.warning("[Twitter] Playwright returned empty content")+ except RuntimeError:+ raise+ except Exception as e:+ logger.error(f"[Twitter] All methods failed: {e}")++ raise RuntimeError(+ f"❌ All Twitter fetch methods failed for: {url}\n"+ f" Try: x-reader login twitter (to save session for browser fallback)\n"+ f" Then retry: x-reader {url}"+ )diff --git a/x_reader/fetchers/wechat.py b/x_reader/fetchers/wechat.pynew file mode 100644index 00000000..9e237984--- /dev/null+++ b/x_reader/fetchers/wechat.py@@ -0,0 +1,62 @@+# -*- coding: utf-8 -*-+"""+WeChat article fetcher — two-tier fallback:++1. Jina Reader (fast, no deps)+2. Playwright headless (no login needed for public articles)+"""++from loguru import logger+from typing import Dict, Any+++async def fetch_wechat(url: str) -> Dict[str, Any]:+ """+ Fetch a WeChat public account article with fallback.++ Args:+ url: mp.weixin.qq.com article URL++ Returns:+ Dict with: title, content, author, url, platform+ """+ # Tier 1: Jina Reader+ try:+ logger.info(f"[WeChat] Tier 1 — Jina: {url}")+ from x_reader.fetchers.jina import fetch_via_jina++ data = fetch_via_jina(url)+ if data.get("content"):+ return {+ "title": data["title"],+ "content": data["content"],+ "author": data.get("author", ""),+ "url": url,+ "platform": "wechat",+ }+ logger.warning("[WeChat] Jina returned empty content, falling back to browser")+ except Exception as e:+ logger.warning(f"[WeChat] Jina failed ({e}), falling back to browser")++ # Tier 2: Playwright headless (no session needed)+ try:+ logger.info(f"[WeChat] Tier 2 — Playwright headless: {url}")+ from x_reader.fetchers.browser import fetch_via_browser++ data = await fetch_via_browser(url)+ return {+ "title": data["title"],+ "content": data["content"],+ "author": data.get("author", ""),+ "url": url,+ "platform": "wechat",+ }+ except RuntimeError:+ # Playwright not installed — re-raise with original Jina error context+ raise+ except Exception as e:+ logger.error(f"[WeChat] Browser fetch also failed: {e}")+ raise RuntimeError(+ f"❌ All WeChat fetch methods failed.\n"+ f" Last error: {e}"+ )diff --git a/x_reader/fetchers/xhs.py b/x_reader/fetchers/xhs.pynew file mode 100644index 00000000..215a679e--- /dev/null+++ b/x_reader/fetchers/xhs.py@@ -0,0 +1,78 @@+# -*- coding: utf-8 -*-+"""+Xiaohongshu (RED) note fetcher — three-tier fallback:++1. Jina Reader (fast, no deps)+2. Playwright + saved session (handles 451/403)+3. Error with login instructions++Install browser tier: pip install "x-reader[browser]" && playwright install chromium+"""++from loguru import logger+from typing import Dict, Any+from pathlib import Path++from x_reader.fetchers.jina import fetch_via_jina+++async def fetch_xhs(url: str) -> Dict[str, Any]:+ """+ Fetch a Xiaohongshu note with three-tier fallback.++ Args:+ url: xiaohongshu.com or xhslink.com URL++ Returns:+ Dict with: title, content, author, url, platform+ """+ # Tier 1: Jina Reader+ try:+ logger.info(f"[XHS] Tier 1 — Jina: {url}")+ data = fetch_via_jina(url)+ if data.get("content"):+ return {+ "title": data["title"],+ "content": data["content"],+ "author": data.get("author", ""),+ "url": url,+ "platform": "xhs",+ }+ logger.warning("[XHS] Jina returned empty content, falling back to browser")+ except Exception as e:+ logger.warning(f"[XHS] Jina failed ({e}), falling back to browser")++ # Tier 2: Playwright with session+ from x_reader.fetchers.browser import get_session_path, SESSION_DIR++ session_path = get_session_path("xhs")+ if not Path(session_path).exists():+ # Tier 3: No session — guide user+ raise RuntimeError(+ f"❌ XHS blocked Jina and no saved session found.\n"+ f" Run: x-reader login xhs\n"+ f" Then retry this URL."+ )++ try:+ logger.info(f"[XHS] Tier 2 — Playwright with session: {url}")+ from x_reader.fetchers.browser import fetch_via_browser++ data = await fetch_via_browser(url, storage_state=session_path)+ return {+ "title": data["title"],+ "content": data["content"],+ "author": data.get("author", ""),+ "url": url,+ "platform": "xhs",+ }+ except RuntimeError:+ # Playwright not installed+ raise+ except Exception as e:+ logger.error(f"[XHS] Browser fetch also failed: {e}")+ raise RuntimeError(+ f"❌ All XHS fetch methods failed.\n"+ f" Last error: {e}\n"+ f" Try: x-reader login xhs (to refresh session)"+ )diff --git a/x_reader/fetchers/youtube.py b/x_reader/fetchers/youtube.pynew file mode 100644index 00000000..12827607--- /dev/null+++ b/x_reader/fetchers/youtube.py@@ -0,0 +1,211 @@+# -*- coding: utf-8 -*-+"""+YouTube video fetcher — three-tier content extraction:++1. yt-dlp auto-subtitles (fastest, best quality for subtitled videos)+2. yt-dlp audio download → Groq Whisper API transcription (for non-subtitled videos)+3. Jina Reader fallback (page description only)++Requires: yt-dlp installed (brew install yt-dlp / pip install yt-dlp)+Optional: GROQ_API_KEY env var for Whisper transcription+"""++import re+import os+import subprocess+import tempfile+from loguru import logger+from typing import Dict, Any++from x_reader.fetchers.jina import fetch_via_jina+++def _extract_video_id(url: str) -> str:+ """Extract video ID from YouTube URL."""+ match = re.search(r'(?:v=|youtu\.be/)([a-zA-Z0-9_-]{11})', url)+ return match.group(1) if match else ""+++def _get_subtitles_via_ytdlp(url: str, lang: str = "en") -> str:+ """+ Download auto-generated subtitles using yt-dlp.+ Returns subtitle text, or empty string if unavailable.+ """+ with tempfile.TemporaryDirectory() as tmpdir:+ output_path = os.path.join(tmpdir, "sub")++ cmd = [+ "yt-dlp",+ "--write-auto-sub",+ "--write-sub",+ "--sub-lang", lang,+ "--sub-format", "srt",+ "--skip-download",+ "-o", output_path,+ url,+ ]++ try:+ subprocess.run(cmd, capture_output=True, text=True, timeout=60)+ except FileNotFoundError:+ logger.warning("yt-dlp not found. Install with: brew install yt-dlp")+ return ""+ except subprocess.TimeoutExpired:+ logger.warning("yt-dlp subtitle download timed out")+ return ""++ for ext in [f".{lang}.srt", f".{lang}.vtt"]:+ sub_file = output_path + ext+ if os.path.exists(sub_file):+ return _parse_srt(sub_file)++ return ""+++def _parse_srt(filepath: str) -> str:+ """Parse SRT file into clean text (strip timestamps and sequence numbers)."""+ with open(filepath, 'r', encoding='utf-8') as f:+ lines = f.readlines()++ text_lines = []+ seen = set()++ for line in lines:+ line = line.strip()+ if not line or line.isdigit() or '-->' in line:+ continue+ if line.startswith('[') and line.endswith(']'):+ continue+ if line not in seen:+ seen.add(line)+ text_lines.append(line)++ return " ".join(text_lines)+++def _transcribe_via_whisper(url: str) -> str:+ """+ Download audio with yt-dlp and transcribe via Groq Whisper API.++ Requires: GROQ_API_KEY env var + yt-dlp + ffmpeg installed.+ Groq Whisper limit: 25MB audio file.+ Returns transcript text, or empty string if unavailable.+ """+ api_key = os.getenv("GROQ_API_KEY")+ if not api_key:+ logger.info("GROQ_API_KEY not set, skipping Whisper transcription")+ return ""++ with tempfile.TemporaryDirectory() as tmpdir:+ output_template = os.path.join(tmpdir, "audio.%(ext)s")++ cmd = [+ "yt-dlp",+ "-x",+ "--audio-format", "m4a",+ "--audio-quality", "5",+ "-o", output_template,+ "--no-playlist",+ url,+ ]++ try:+ subprocess.run(cmd, capture_output=True, text=True, timeout=180)+ except FileNotFoundError:+ logger.warning("yt-dlp not found for audio download")+ return ""+ except subprocess.TimeoutExpired:+ logger.warning("yt-dlp audio download timed out")+ return ""++ # Find the downloaded audio file+ audio_path = os.path.join(tmpdir, "audio.m4a")+ if not os.path.exists(audio_path):+ for f in os.listdir(tmpdir):+ if f.startswith("audio."):+ audio_path = os.path.join(tmpdir, f)+ break+ else:+ logger.warning("No audio file downloaded")+ return ""++ file_size = os.path.getsize(audio_path)+ if file_size > 25 * 1024 * 1024:+ logger.warning(f"Audio file too large ({file_size // 1024 // 1024}MB > 25MB limit)")+ return ""++ logger.info(f"Transcribing {file_size // 1024}KB audio via Groq Whisper...")++ import requests+ try:+ with open(audio_path, "rb") as f:+ response = requests.post(+ "https://api.groq.com/openai/v1/audio/transcriptions",+ headers={"Authorization": f"Bearer {api_key}"},+ files={"file": (os.path.basename(audio_path), f, "audio/mp4")},+ data={"model": "whisper-large-v3", "response_format": "text"},+ timeout=120,+ )++ if response.status_code == 200:+ transcript = response.text.strip()+ logger.info(f"Whisper transcript: {len(transcript)} chars")+ return transcript+ else:+ logger.warning(f"Groq Whisper API error: {response.status_code} {response.text[:200]}")+ return ""+ except Exception as e:+ logger.warning(f"Whisper transcription failed: {e}")+ return ""+++async def fetch_youtube(url: str, sub_lang: str = "en") -> Dict[str, Any]:+ """+ Fetch YouTube video content with three-tier extraction.++ Strategy:+ 1. yt-dlp auto-subtitles (full transcript, fastest)+ 2. yt-dlp audio + Groq Whisper API (for non-subtitled videos)+ 3. Jina Reader fallback (page description only)++ Args:+ url: YouTube video URL+ sub_lang: Subtitle language code (default: "en")++ Returns:+ Dict with: title, description, author, url, video_id, has_transcript, platform+ """+ logger.info(f"Fetching YouTube: {url}")+ video_id = _extract_video_id(url)++ # Step 1: Get metadata via Jina (fast, always works)+ jina_data = fetch_via_jina(url)+ title = jina_data["title"]++ # Step 2: Try yt-dlp auto-subtitles+ logger.info(f"Extracting subtitles ({sub_lang})...")+ transcript = _get_subtitles_via_ytdlp(url, lang=sub_lang)++ # Step 3: No subtitles? Try Whisper transcription+ if not transcript:+ logger.info("No subtitles available, trying Whisper transcription...")+ transcript = _transcribe_via_whisper(url)++ if transcript:+ logger.info(f"Got transcript: {len(transcript)} chars")+ content = transcript+ has_transcript = True+ else:+ logger.info("No transcript available, using page description")+ content = jina_data["content"]+ has_transcript = False++ return {+ "title": title,+ "description": content,+ "author": jina_data.get("author", ""),+ "url": url,+ "video_id": video_id,+ "has_transcript": has_transcript,+ "platform": "youtube",+ }diff --git a/x_reader/login.py b/x_reader/login.pynew file mode 100644index 00000000..1d1d8773--- /dev/null+++ b/x_reader/login.py@@ -0,0 +1,91 @@+# -*- coding: utf-8 -*-+"""+Login manager — opens a visible browser for manual login, saves session.++Usage:+ x-reader login xhs # Login to Xiaohongshu+ x-reader login wechat # Login to WeChat (if needed)++Sessions are saved as Playwright storage_state JSON files.+"""++from pathlib import Path+from loguru import logger++SESSION_DIR = Path.home() / ".x-reader" / "sessions"++PLATFORM_URLS = {+ "xhs": "https://www.xiaohongshu.com/explore",+ "xiaohongshu": "https://www.xiaohongshu.com/explore",+ "wechat": "https://mp.weixin.qq.com",+ "twitter": "https://x.com/login",+ "x": "https://x.com/login",+}+++def login(platform: str) -> None:+ """+ Open a visible browser for the user to log in manually.+ After login, saves cookies/localStorage to a session file.++ Args:+ platform: Platform key (e.g. 'xhs', 'wechat')+ """+ try:+ from playwright.sync_api import sync_playwright+ except ImportError:+ print(+ "❌ Playwright is not installed. Run:\n"+ ' pip install "x-reader[browser]"\n'+ " playwright install chromium"+ )+ return++ platform = platform.lower()+ login_url = PLATFORM_URLS.get(platform)+ if not login_url:+ supported = ", ".join(sorted(PLATFORM_URLS.keys()))+ print(f"❌ Unknown platform: {platform}")+ print(f" Supported: {supported}")+ return++ SESSION_DIR.mkdir(parents=True, exist_ok=True)+ session_path = SESSION_DIR / f"{platform}.json"+ # Normalize alias to canonical name+ if platform in ("xhs", "xiaohongshu"):+ canonical = "xhs"+ elif platform in ("twitter", "x"):+ canonical = "twitter"+ else:+ canonical = platform+ session_path = SESSION_DIR / f"{canonical}.json"++ print(f"🌐 Opening {platform} login page: {login_url}")+ print(" Please log in manually in the browser window.")+ print(" When done, close the browser or press Ctrl+C.\n")++ with sync_playwright() as p:+ browser = p.chromium.launch(headless=False)+ context = browser.new_context(+ user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "+ "AppleWebKit/537.36 (KHTML, like Gecko) "+ "Chrome/120.0.0.0 Safari/537.36",+ )+ page = context.new_page()+ page.goto(login_url)++ try:+ # Wait for user to log in — blocks until browser is closed+ page.wait_for_event("close", timeout=300_000) # 5 min max+ except KeyboardInterrupt:+ pass+ except Exception:+ pass # Browser closed by user++ # Save session regardless of how we got here+ context.storage_state(path=str(session_path))+ logger.info(f"Session saved: {session_path}")+ print(f"\n✅ Session saved to {session_path}")++ context.close()+ browser.close()diff --git a/x_reader/reader.py b/x_reader/reader.pynew file mode 100644index 00000000..150f3df5--- /dev/null+++ b/x_reader/reader.py@@ -0,0 +1,154 @@+# -*- coding: utf-8 -*-+"""+Universal Reader — routes any URL to the right fetcher.++The core dispatcher: give it a URL, get back structured content.+"""++import asyncio+from urllib.parse import urlparse+from loguru import logger+from typing import Dict, Any, Optional++from x_reader.schema import (+ UnifiedContent, UnifiedInbox, SourceType,+ from_bilibili, from_twitter, from_wechat,+ from_xiaohongshu, from_youtube, from_rss, from_telegram,+)+from x_reader.fetchers.jina import fetch_via_jina+++class UniversalReader:+ """+ Routes URLs to platform-specific fetchers.+ Falls back to Jina Reader for unknown platforms.+ """++ def __init__(self, inbox: Optional[UnifiedInbox] = None):+ self.inbox = inbox++ def _detect_platform(self, url: str) -> str:+ """Detect platform from URL."""+ domain = urlparse(url).netloc.lower()++ if "mp.weixin.qq.com" in domain:+ return "wechat"+ if "x.com" in domain or "twitter.com" in domain:+ return "twitter"+ if "youtube.com" in domain or "youtu.be" in domain:+ return "youtube"+ if "xiaohongshu.com" in domain or "xhslink.com" in domain:+ return "xhs"+ if "bilibili.com" in domain or "b23.tv" in domain:+ return "bilibili"+ if "xiaoyuzhoufm.com" in domain:+ return "podcast"+ if "podcasts.apple.com" in domain:+ return "podcast"+ if "t.me" in domain or "telegram.org" in domain:+ return "telegram"+ if url.endswith(".xml") or "/rss" in url or "/feed" in url or "/atom" in url:+ return "rss"+ return "generic"++ async def read(self, url: str) -> UnifiedContent:+ """+ Fetch content from any URL and return as UnifiedContent.++ The main entry point — give it a URL, get back structured content.+ """+ # Ensure URL has scheme+ if not url.startswith(("http://", "https://")):+ url = f"https://{url}"++ platform = self._detect_platform(url)+ logger.info(f"[{platform}] {url[:60]}...")++ try:+ content = await self._fetch(platform, url)++ # Save to inbox if configured+ if self.inbox:+ if self.inbox.add(content):+ self.inbox.save()+ logger.info(f"Saved to inbox: {content.title[:50]}")++ # Save to markdown output if configured+ from x_reader.utils.storage import save_to_markdown+ save_to_markdown(content)++ return content++ except Exception as e:+ logger.error(f"[{platform}] Failed: {e}")+ raise++ async def _fetch(self, platform: str, url: str) -> UnifiedContent:+ """Dispatch to platform-specific fetcher."""++ if platform == "bilibili":+ from x_reader.fetchers.bilibili import fetch_bilibili+ data = await fetch_bilibili(url)+ return from_bilibili(data)++ if platform == "twitter":+ from x_reader.fetchers.twitter import fetch_twitter+ data = await fetch_twitter(url)+ return from_twitter(data)++ if platform == "wechat":+ from x_reader.fetchers.wechat import fetch_wechat+ data = await fetch_wechat(url)+ return from_wechat(data)++ if platform == "xhs":+ from x_reader.fetchers.xhs import fetch_xhs+ data = await fetch_xhs(url)+ return from_xiaohongshu(data)++ if platform == "youtube":+ from x_reader.fetchers.youtube import fetch_youtube+ data = await fetch_youtube(url)+ return from_youtube(data)++ if platform == "rss":+ from x_reader.fetchers.rss import fetch_rss+ articles = await fetch_rss(url, limit=1)+ if articles:+ return from_rss(articles[0])+ raise ValueError(f"No articles found in RSS feed: {url}")++ if platform == "telegram":+ from x_reader.fetchers.telegram import fetch_telegram+ # Extract channel username from t.me URL+ path = urlparse(url).path.strip("/").split("/")[0]+ channel = path if path else url+ messages = await fetch_telegram(channel, limit=1)+ if messages:+ return from_telegram(messages[0], channel, channel)+ raise ValueError(f"No messages from Telegram channel: {url}")++ # Fallback: Jina Reader for any unknown URL+ logger.info(f"Using Jina fallback for: {url}")+ data = fetch_via_jina(url)+ return UnifiedContent(+ source_type=SourceType.MANUAL,+ source_name=urlparse(url).netloc,+ title=data["title"],+ content=data["content"],+ url=url,+ )++ async def read_batch(self, urls: list[str]) -> list[UnifiedContent]:+ """Fetch multiple URLs concurrently."""+ tasks = [self.read(url) for url in urls]+ results = await asyncio.gather(*tasks, return_exceptions=True)++ contents = []+ for url, result in zip(urls, results):+ if isinstance(result, Exception):+ logger.error(f"Batch failed for {url}: {result}")+ else:+ contents.append(result)++ return contentsdiff --git a/x_reader/schema.py b/x_reader/schema.pynew file mode 100644index 00000000..4885d187--- /dev/null+++ b/x_reader/schema.py@@ -0,0 +1,277 @@+# -*- coding: utf-8 -*-+"""+Unified content schema for x-reader.++Defines the standard data format for all content sources:+- Telegram channels+- RSS feeds+- Bilibili videos+- Xiaohongshu (RED) notes+- WeChat articles+- X/Twitter posts+- YouTube videos+- Manual input+"""++from dataclasses import dataclass, field, asdict+from datetime import datetime, timedelta+from typing import Optional, List+from enum import Enum+import hashlib+import json+++class SourceType(str, Enum):+ """Content source types."""+ TELEGRAM = "telegram"+ RSS = "rss"+ BILIBILI = "bilibili"+ XIAOHONGSHU = "xhs"+ TWITTER = "twitter"+ WECHAT = "wechat"+ YOUTUBE = "youtube"+ MANUAL = "manual"+++class MediaType(str, Enum):+ """Media types."""+ TEXT = "text"+ VIDEO = "video"+ AUDIO = "audio"+ IMAGE = "image"+++class Priority(str, Enum):+ """Content priority levels."""+ HOT = "hot"+ QUALITY = "quality"+ DEEP = "deep"+ NORMAL = "normal"+ LOW = "low"+++@dataclass+class UnifiedContent:+ """Unified content format across all platforms."""++ # === Required ===+ source_type: SourceType+ source_name: str+ title: str+ content: str+ url: str++ # === Auto-generated ===+ id: str = ""+ fetched_at: str = ""++ # === Media ===+ media_type: MediaType = MediaType.TEXT+ media_url: Optional[str] = None++ # === Scoring ===+ score: int = 0+ priority: Priority = Priority.NORMAL+ category: str = ""+ tags: List[str] = field(default_factory=list)++ # === Processing state ===+ processed: bool = False+ digest_date: Optional[str] = None++ # === Translation ===+ title_cn: Optional[str] = None+ content_cn: Optional[str] = None++ # === Metadata ===+ extra: dict = field(default_factory=dict)++ def __post_init__(self):+ if not self.id:+ self.id = hashlib.md5(self.url.encode()).hexdigest()[:12]+ if not self.fetched_at:+ self.fetched_at = datetime.now().isoformat()++ def to_dict(self) -> dict:+ d = asdict(self)+ d['source_type'] = self.source_type.value+ d['media_type'] = self.media_type.value+ d['priority'] = self.priority.value+ return d++ @classmethod+ def from_dict(cls, data: dict) -> 'UnifiedContent':+ if isinstance(data.get('source_type'), str):+ data['source_type'] = SourceType(data['source_type'])+ if isinstance(data.get('media_type'), str):+ data['media_type'] = MediaType(data['media_type'])+ if isinstance(data.get('priority'), str):+ data['priority'] = Priority(data['priority'])+ known = {f.name for f in cls.__dataclass_fields__.values()}+ data = {k: v for k, v in data.items() if k in known}+ return cls(**data)+++# =============================================================================+# Converters: platform-specific dict → UnifiedContent+# =============================================================================++def from_telegram(msg: dict, channel_name: str, channel_username: str) -> UnifiedContent:+ return UnifiedContent(+ source_type=SourceType.TELEGRAM,+ source_name=channel_name,+ title=msg.get('text', '')[:100],+ content=msg.get('text', ''),+ url=msg.get('url', f"https://t.me/{channel_username}"),+ extra={"views": msg.get('views', 0), "channel_username": channel_username},+ )+++def from_rss(article: dict) -> UnifiedContent:+ return UnifiedContent(+ source_type=SourceType.RSS,+ source_name=article.get('source', ''),+ title=article.get('title', ''),+ content=article.get('summary', ''),+ url=article.get('url', article.get('link', '')),+ score=article.get('score', 0),+ category=article.get('category', ''),+ title_cn=article.get('title_cn'),+ content_cn=article.get('summary_cn'),+ )+++def from_bilibili(video: dict) -> UnifiedContent:+ return UnifiedContent(+ source_type=SourceType.BILIBILI,+ source_name=video.get('author', ''),+ title=video.get('title', ''),+ content=video.get('description', ''),+ url=video.get('url', ''),+ media_type=MediaType.VIDEO,+ media_url=video.get('cover', ''),+ extra={+ "bvid": video.get('bvid', ''),+ "duration": video.get('duration', 0),+ "view_count": video.get('view_count', 0),+ },+ )+++def from_twitter(data: dict) -> UnifiedContent:+ return UnifiedContent(+ source_type=SourceType.TWITTER,+ source_name=data.get('author', ''),+ title=data.get('text', '')[:100],+ content=data.get('text', ''),+ url=data.get('url', ''),+ extra={+ "likes": data.get('likes', 0),+ "retweets": data.get('retweets', 0),+ },+ )+++def from_wechat(article: dict) -> UnifiedContent:+ return UnifiedContent(+ source_type=SourceType.WECHAT,+ source_name=article.get('author', ''),+ title=article.get('title', ''),+ content=article.get('content', ''),+ url=article.get('url', ''),+ )+++def from_xiaohongshu(note: dict) -> UnifiedContent:+ return UnifiedContent(+ source_type=SourceType.XIAOHONGSHU,+ source_name=note.get('author', ''),+ title=note.get('title', ''),+ content=note.get('content', ''),+ url=note.get('url', ''),+ media_type=MediaType.IMAGE if note.get('images') else MediaType.TEXT,+ extra={+ "likes": note.get('likes', 0),+ "collects": note.get('collects', 0),+ },+ )+++def from_youtube(video: dict) -> UnifiedContent:+ return UnifiedContent(+ source_type=SourceType.YOUTUBE,+ source_name=video.get('author', ''),+ title=video.get('title', ''),+ content=video.get('description', ''),+ url=video.get('url', ''),+ media_type=MediaType.VIDEO,+ extra={+ "duration": video.get('duration', ''),+ "view_count": video.get('view_count', 0),+ },+ )+++def from_manual(title: str, content: str, url: str = "") -> UnifiedContent:+ return UnifiedContent(+ source_type=SourceType.MANUAL,+ source_name="manual",+ title=title,+ content=content,+ url=url or f"manual://{hashlib.md5(title.encode()).hexdigest()[:8]}",+ )+++# =============================================================================+# Unified Inbox+# =============================================================================++class UnifiedInbox:+ """JSON-based content inbox with dedup."""++ def __init__(self, filepath: str = "unified_inbox.json"):+ self.filepath = filepath+ self.items: List[UnifiedContent] = []+ self.load()++ def load(self):+ import os+ if os.path.exists(self.filepath):+ try:+ with open(self.filepath, 'r', encoding='utf-8') as f:+ data = json.load(f)+ self.items = [UnifiedContent.from_dict(d) for d in data]+ except (json.JSONDecodeError, IOError):+ self.items = []++ def save(self):+ with open(self.filepath, 'w', encoding='utf-8') as f:+ json.dump([item.to_dict() for item in self.items], f,+ ensure_ascii=False, indent=2)++ def add(self, item: UnifiedContent) -> bool:+ if any(i.id == item.id for i in self.items):+ return False+ self.items.append(item)+ return True++ def add_batch(self, items: List[UnifiedContent]) -> int:+ return sum(1 for item in items if self.add(item))++ def get_unprocessed(self) -> List[UnifiedContent]:+ return [i for i in self.items if not i.processed]++ def get_by_source(self, source_type: SourceType) -> List[UnifiedContent]:+ return [i for i in self.items if i.source_type == source_type]++ def mark_processed(self, item_id: str, digest_date: str = None):+ for item in self.items:+ if item.id == item_id:+ item.processed = True+ if digest_date:+ item.digest_date = digest_date+ break++ def clear_old(self, days: int = 7):+ cutoff = (datetime.now() - timedelta(days=days)).isoformat()+ self.items = [i for i in self.items if i.fetched_at > cutoff]diff --git a/x_reader/utils/__init__.py b/x_reader/utils/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/x_reader/utils/storage.py b/x_reader/utils/storage.pynew file mode 100644index 00000000..9de0bf94--- /dev/null+++ b/x_reader/utils/storage.py@@ -0,0 +1,88 @@+# -*- coding: utf-8 -*-+"""+Storage utilities — save content to JSON inbox and optional Markdown file.++Implements the "atomic archiving" from the tweet:+- unified_inbox.json (for AI/programmatic use)+- markdown file (for human reading, e.g. Obsidian)+"""++import json+import os+from datetime import datetime+from pathlib import Path+from loguru import logger++from x_reader.schema import UnifiedContent+++def save_to_json(item: UnifiedContent, filepath: str = "unified_inbox.json"):+ """Append content to JSON inbox file."""+ path = Path(filepath)+ data = []++ if path.exists():+ try:+ with open(path, 'r', encoding='utf-8') as f:+ data = json.load(f)+ except (json.JSONDecodeError, IOError):+ data = []++ data.append(item.to_dict())++ # Keep last 500 entries to prevent unbounded growth+ data = data[-500:]++ with open(path, 'w', encoding='utf-8') as f:+ json.dump(data, f, ensure_ascii=False, indent=2)++ logger.info(f"Saved to JSON: {path}")+++def save_to_markdown(item: UnifiedContent, filepath: str = None):+ """+ Append content to a Markdown file (e.g. Obsidian vault).++ Supports two output modes:+ - OUTPUT_DIR: Write to {OUTPUT_DIR}/content_hub.md+ - OBSIDIAN_VAULT: Write to {OBSIDIAN_VAULT}/01-收集箱/x-reader-inbox.md++ If neither is set, skips markdown output.+ """+ if not filepath:+ # Priority 1: Obsidian vault+ vault_path = os.getenv("OBSIDIAN_VAULT", "")+ if vault_path:+ filepath = os.path.join(vault_path, "01-收集箱", "x-reader-inbox.md")+ else:+ # Priority 2: generic output dir+ output_dir = os.getenv("OUTPUT_DIR", "")+ if not output_dir:+ return+ filepath = os.path.join(output_dir, "content_hub.md")++ path = Path(filepath)+ path.parent.mkdir(parents=True, exist_ok=True)++ emoji = {+ "telegram": "📢", "rss": "📰", "bilibili": "🎬",+ "xhs": "📕", "twitter": "🐦", "wechat": "💬",+ "youtube": "▶️", "manual": "✏️",+ }.get(item.source_type.value, "📄")++ with open(path, 'a', encoding='utf-8') as f:+ f.write(f"\n## {emoji} {item.title}\n")+ f.write(f"- Source: {item.source_name} ({item.source_type.value})\n")+ f.write(f"- URL: {item.url}\n")+ f.write(f"- Fetched: {item.fetched_at[:16]}\n\n")+ f.write(f"{item.content[:2000]}\n")+ f.write("\n---\n")++ logger.info(f"Saved to Markdown: {path}")+++def save_content(item: UnifiedContent, json_path: str = None, md_path: str = None):+ """Save content to both JSON and Markdown."""+ inbox_file = json_path or os.getenv("INBOX_FILE", "unified_inbox.json")+ save_to_json(item, inbox_file)+ save_to_markdown(item, md_path)