Skip to content

⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.🎯 告别信息过载,你的 AI 舆情监控助手与热点筛选工具!聚合多平台热点 + RSS 订阅,支持关键词精准筛选。AI 智能筛选新闻 + AI 翻译 + AI 分析简报直推手机,也支持接入 MCP 架构,赋能 AI 自然语言对话分析、情感洞察与趋势预测等。支持 Docker ,数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/bark/slack 等渠道智能推送。

feat: v2.0.0 重大重构

sansan3fbbfeff10d4
diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 0000000000..905b0b0b89
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,35 @@
+.git/
+.gitignore
+*.md
+README.md
+
+output/
+
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+*.so
+.pytest_cache/
+
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+.DS_Store
+Thumbs.db
+
+docker/.env
+
+_image/
+
+.github/
+
+*.log
+.env.local
+.env.*.local
+version
+index.html
\ No newline at end of file
diff --git a/.github/ISSUE_TEMPLATE/01-bug-report.yml b/.github/ISSUE_TEMPLATE/01-bug-report.yml
new file mode 100644
index 0000000000..46f3d4e715
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/01-bug-report.yml
@@ -0,0 +1,123 @@
+# yaml-language-server: $schema=https://json.schemastore.org/github-issue-forms.json
+
+name: 🐛 Bug 报告
+description: 报告程序错误或异常行为
+title: "[Bug] "
+labels: ["bug"]
+assignees:
+ - sansan0
+body:
+ - type: markdown
+ attributes:
+ value: |
+ 感谢你的反馈!详细的 Bug 报告有助于快速定位和修复问题。
+
+ - type: dropdown
+ id: bug-category
+ attributes:
+ label: 🏷️ 问题类型
+ options:
+ - 数据抓取失败
+ - 频率词匹配异常
+ - 通知发送失败
+ - 配置解析错误
+ - Docker 部署问题
+ - 报告生成错误
+ - 性能问题
+ - 其他
+ validations:
+ required: true
+
+ - type: textarea
+ id: bug-description
+ attributes:
+ label: 📝 问题描述
+ description: 详细描述遇到的问题
+ placeholder: |
+ 请清楚地描述:
+ - 发生了什么问题
+ - 问题的具体表现
+ - 影响范围
+ validations:
+ required: true
+
+ - type: textarea
+ id: reproduction-steps
+ attributes:
+ label: 🔄 复现步骤
+ description: 如何重现这个问题?
+ placeholder: |
+ 1. 运行命令:python main.py
+ 2. 配置设置:...
+ 3. 观察到的现象:...
+ validations:
+ required: true
+
+ - type: textarea
+ id: expected-behavior
+ attributes:
+ label: ✅ 期望行为
+ description: 正常情况下应该发生什么?
+ placeholder: 描述期望的正常行为...
+ validations:
+ required: true
+
+ - type: dropdown
+ id: environment
+ attributes:
+ label: 🖥️ 运行环境
+ options:
+ - 本地 Python 环境
+ - Docker 容器
+ - GitHub Actions
+ - 其他
+ validations:
+ required: true
+
+ - type: input
+ id: python-version
+ attributes:
+ label: 🐍 Python 版本
+ description: 使用的 Python 版本
+ placeholder: 如:3.10.0
+ validations:
+ required: true
+
+ - type: textarea
+ id: error-logs
+ attributes:
+ label: 📋 错误日志
+ description: 相关的错误信息或日志
+ placeholder: |
+ 请粘贴完整的错误堆栈信息或相关日志:
+
+ ```
+ 错误信息...
+ ```
+ validations:
+ required: true
+
+ - type: textarea
+ id: config-info
+ attributes:
+ label: ⚙️ 配置信息
+ description: 相关的配置设置(请隐藏敏感信息)
+ placeholder: |
+ 相关的配置片段(请移除敏感信息如 webhook URL):
+
+ ```yaml
+ crawler:
+ request_interval: 1000
+ enable_crawler: true
+ ```
+
+ - type: textarea
+ id: additional-context
+ attributes:
+ label: 📎 其他信息
+ description: 其他可能有用的信息
+ placeholder: |
+ - 网络环境
+ - 使用的平台配置
+ - 频率词配置
+ - 其他相关信息
\ No newline at end of file
diff --git a/.github/ISSUE_TEMPLATE/02-feature-request.yml b/.github/ISSUE_TEMPLATE/02-feature-request.yml
new file mode 100644
index 0000000000..db4c375c95
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/02-feature-request.yml
@@ -0,0 +1,98 @@
+# yaml-language-server: $schema=https://json.schemastore.org/github-issue-forms.json
+
+name: 💡 功能建议
+description: 提出新功能想法或改进建议
+title: "[功能] "
+labels: ["enhancement"]
+assignees:
+ - sansan0
+body:
+ - type: markdown
+ attributes:
+ value: |
+ 感谢你的创意!好的想法让这个项目变得更加实用。
+
+ - type: dropdown
+ id: feature-category
+ attributes:
+ label: 🏷️ 功能类别
+ options:
+ - 数据抓取增强
+ - 分析算法改进
+ - 通知方式扩展
+ - 配置管理优化
+ - 部署便利性
+ - 数据可视化
+ - 监控与告警
+ - 其他
+ validations:
+ required: true
+
+ - type: textarea
+ id: feature-description
+ attributes:
+ label: 💭 功能描述
+ description: 详细描述你希望添加的功能
+ placeholder: |
+ 请详细描述:
+ - 功能的具体内容
+ - 如何使用这个功能
+ - 期望的效果
+ validations:
+ required: true
+
+ - type: textarea
+ id: use-case
+ attributes:
+ label: 🎯 使用场景
+ description: 这个功能会在什么情况下使用?
+ placeholder: |
+ 例如:
+ - 当用户需要...的时候
+ - 在...场景下会很有帮助
+ - 可以解决...问题
+ validations:
+ required: true
+
+ - type: textarea
+ id: current-workaround
+ attributes:
+ label: 🔄 现有解决方案
+ description: 目前是如何处理这个需求的?
+ placeholder: |
+ - 目前的替代方案
+ - 现有方案的不足
+ - 为什么需要新功能
+
+ - type: dropdown
+ id: feature-priority
+ attributes:
+ label: ⭐ 优先级
+ description: 你认为这个功能的重要程度
+ options:
+ - 高 - 非常重要,强烈建议
+ - 中 - 会很有用,建议考虑
+ - 低 - 锦上添花,有时间可做
+ validations:
+ required: true
+
+ - type: textarea
+ id: implementation-ideas
+ attributes:
+ label: 🛠️ 实现想法(可选)
+ description: 如果你有实现思路,欢迎分享
+ placeholder: |
+ - 技术实现方案
+ - 配置设计建议
+ - 用户界面设计
+ - 参考项目或工具
+
+ - type: textarea
+ id: additional-context
+ attributes:
+ label: 📎 其他信息
+ description: 其他想要补充的内容
+ placeholder: |
+ - 相关截图或示例
+ - 类似功能的参考
+ - 其他相关想法
\ No newline at end of file
diff --git a/.github/ISSUE_TEMPLATE/03-config-help.yml b/.github/ISSUE_TEMPLATE/03-config-help.yml
new file mode 100644
index 0000000000..3d54a1a934
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/03-config-help.yml
@@ -0,0 +1,108 @@
+# yaml-language-server: $schema=https://json.schemastore.org/github-issue-forms.json
+
+name: ⚙️ 配置帮助
+description: 配置相关的问题或优化建议
+title: "[配置] "
+labels: ["配置", "帮助"]
+assignees:
+ - sansan0
+body:
+ - type: markdown
+ attributes:
+ value: |
+ 遇到配置问题?我们来帮你解决!
+
+ 📖 **建议先查看 README 文档和配置文件注释**
+
+ - type: dropdown
+ id: config-type
+ attributes:
+ label: 🏷️ 配置类型
+ options:
+ - 基础配置问题
+ - Webhook 配置
+ - Docker 部署配置
+ - 频率词配置
+ - 平台配置
+ - 定时任务配置
+ - 其他
+ validations:
+ required: true
+
+ - type: dropdown
+ id: problem-type
+ attributes:
+ label: 🔍 问题类型
+ options:
+ - 配置不生效
+ - 不知道如何配置
+ - 配置优化建议
+ - 配置文档改进
+ - 配置验证问题
+ validations:
+ required: true
+
+ - type: textarea
+ id: config-content
+ attributes:
+ label: 📄 相关配置
+ description: 请提供相关的配置内容(请隐藏敏感信息)
+ placeholder: |
+ 请贴出相关的配置片段(记得隐藏敏感信息):
+
+ ```yaml
+ notification:
+ enable_notification: true
+ webhooks:
+ feishu_url: "***"
+ ```
+ validations:
+ required: true
+
+ - type: textarea
+ id: problem-description
+ attributes:
+ label: 📝 问题描述
+ description: 详细描述遇到的配置问题
+ placeholder: |
+ 请详细描述:
+ - 遇到的具体问题
+ - 期望的效果
+ - 已经尝试的方法
+ validations:
+ required: true
+
+ - type: textarea
+ id: error-messages
+ attributes:
+ label: ❌ 错误信息(如果有)
+ description: 相关的错误信息或日志
+ placeholder: |
+ 如果有错误信息,请贴出来:
+
+ ```
+ 错误信息...
+ ```
+
+ - type: dropdown
+ id: environment
+ attributes:
+ label: 🖥️ 运行环境
+ options:
+ - 本地 Python 环境
+ - Docker 容器
+ - GitHub Actions
+ - 其他
+ validations:
+ required: true
+
+ - type: textarea
+ id: additional-info
+ attributes:
+ label: 📎 其他信息
+ description: 其他可能有用的信息
+ placeholder: |
+ - 操作系统版本
+ - 网络环境
+ - 使用场景
+ - 其他相关信息
\ No newline at end of file
diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml
new file mode 100644
index 0000000000..0443cb2b73
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -0,0 +1,3 @@
+# yaml-language-server: $schema=https://json.schemastore.org/github-issue-config.json
+
+blank_issues_enabled: false
\ No newline at end of file
diff --git a/.github/workflows/crawler.yml b/.github/workflows/crawler.yml
index 147359f2de..c4f6d4f8d8 100644
--- a/.github/workflows/crawler.yml
+++ b/.github/workflows/crawler.yml
@@ -2,7 +2,7 @@ name: Hot News Crawler
on:
schedule:
- - cron: '*/50 * * * *' # 每50分钟运行一次
+ - cron: "*/30 * * * *" # 每 30 分钟运行一次
workflow_dispatch:
# 添加权限设置
@@ -12,41 +12,52 @@ permissions:
jobs:
crawl:
runs-on: ubuntu-latest
-
+
steps:
- - name: Checkout repository
- uses: actions/checkout@v3
-
- - name: Set up Python
- uses: actions/setup-python@v4
- with:
- python-version: '3.9'
-
- - name: Install dependencies
- run: |
- python -m pip install --upgrade pip
- pip install requests pytz
-
- - name: Create frequency_words.txt if not exists
- run: |
- if [ ! -f frequency_words.txt ]; then
- echo "Creating empty frequency_words.txt file"
- touch frequency_words.txt
- fi
-
- - name: Run crawler
- env:
- FEISHU_WEBHOOK_URL: ${{ secrets.FEISHU_WEBHOOK_URL }}
- TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
- TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }}
- DINGTALK_WEBHOOK_URL: ${{ secrets.DINGTALK_WEBHOOK_URL }}
- WEWORK_WEBHOOK_URL: ${{ secrets.WEWORK_WEBHOOK_URL }}
- GITHUB_ACTIONS: true
- run: python main.py
-
- - name: Commit and push if changes
- run: |
- git config --global user.name 'GitHub Actions'
- git config --global user.email '[email protected]'
- git add -A
- git diff --quiet && git diff --staged --quiet || (git commit -m "Auto update by GitHub Actions at $(TZ=Asia/Shanghai date)" && git push)
+ - name: Checkout repository
+ uses: actions/checkout@v3
+
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: "3.9"
+
+ - name: Install dependencies
+ run: |
+ python -m pip install --upgrade pip
+ pip install -r requirements.txt
+
+ - name: Verify required files
+ run: |
+ echo "🔍 检查必需的配置文件..."
+
+ if [ ! -f config/config.yaml ]; then
+ echo "❌ 错误: config/config.yaml 文件不存在"
+ echo "请参考项目文档创建配置文件"
+ exit 1
+ fi
+
+ if [ ! -f config/frequency_words.txt ]; then
+ echo "❌ 错误: config/frequency_words.txt 文件不存在"
+ echo "请参考项目文档创建频率词配置文件"
+ exit 1
+ fi
+
+ echo "✅ 配置文件检查通过"
+
+ - name: Run crawler
+ env:
+ FEISHU_WEBHOOK_URL: ${{ secrets.FEISHU_WEBHOOK_URL }}
+ TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
+ TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }}
+ DINGTALK_WEBHOOK_URL: ${{ secrets.DINGTALK_WEBHOOK_URL }}
+ WEWORK_WEBHOOK_URL: ${{ secrets.WEWORK_WEBHOOK_URL }}
+ GITHUB_ACTIONS: true
+ run: python main.py
+
+ - name: Commit and push if changes
+ run: |
+ git config --global user.name 'GitHub Actions'
+ git config --global user.email '[email protected]'
+ git add -A
+ git diff --quiet && git diff --staged --quiet || (git commit -m "Auto update by GitHub Actions at $(TZ=Asia/Shanghai date)" && git push)
diff --git a/config/config.yaml b/config/config.yaml
new file mode 100644
index 0000000000..cb52abf3fd
--- /dev/null
+++ b/config/config.yaml
@@ -0,0 +1,72 @@
+app:
+ version: "2.0.0"
+ version_check_url: "https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/version"
+ show_version_update: true # 控制显示版本更新提示,改成 false 将不接受新版本提示
+
+crawler:
+ request_interval: 1000 # 请求间隔(毫秒)
+ enable_crawler: true # 是否启用爬取新闻功能,false 时直接停止程序
+ use_proxy: false # 是否启用代理,false 时为关闭
+ default_proxy: "http://127.0.0.1:10086"
+
+# 🔸 daily(当日汇总模式)
+# • 推送时机:按时推送
+# • 显示内容:当日所有匹配新闻 + 新增新闻区域
+# • 适用场景:日报总结、全面了解当日热点趋势
+#
+# 🔸 current(当前榜单模式)
+# • 推送时机:按时推送
+# • 显示内容:当前榜单匹配新闻 + 新增新闻区域
+# • 适用场景:实时热点追踪、了解当前最火的内容
+#
+# 🔸 incremental(增量监控模式)
+# • 推送时机:有新增才推送
+# • 显示内容:新出现的匹配频率词新闻
+# • 适用场景:避免重复信息干扰
+
+report:
+ mode: "daily" # 可选: "daily"|"incremental"|"current"
+ rank_threshold: 5 # 排名高亮阈值
+
+notification:
+ enable_notification: true # 是否启用通知功能,false 时不发送手机通知
+ message_batch_size: 4000 # 消息分批大小(字节)(这个配置别动)
+ batch_send_interval: 1 # 批次发送间隔(秒)
+ feishu_message_separator: "━━━━━━━━━━━━━━━━━━━" # feishu 消息分割线
+
+ webhooks:
+ feishu_url: "" # 飞书机器人的 webhook URL
+ dingtalk_url: "" # 钉钉机器人的 webhook URL
+ wework_url: "" # 企业微信机器人的 webhook URL
+ telegram_bot_token: "" # Telegram Bot Token
+ telegram_chat_id: "" # Telegram Chat ID
+
+# 用于让关注度更高的新闻在更前面显示,合起来是 1 就行
+weight:
+ rank_weight: 0.6 # 排名权重
+ frequency_weight: 0.3 # 频次权重
+ hotness_weight: 0.1 # 热度权重
+
+platforms:
+ - id: "toutiao"
+ name: "今日头条"
+ - id: "baidu"
+ name: "百度热搜"
+ - id: "wallstreetcn-hot"
+ name: "华尔街见闻"
+ - id: "thepaper"
+ name: "澎湃新闻"
+ - id: "bilibili-hot-search"
+ name: "bilibili 热搜"
+ - id: "cls-hot"
+ name: "财联社热门"
+ - id: "ifeng"
+ name: "凤凰网"
+ - id: "tieba"
+ name: "贴吧"
+ - id: "weibo"
+ name: "微博"
+ - id: "douyin"
+ name: "抖音"
+ - id: "zhihu"
+ name: "知乎"
diff --git a/frequency_words.txt b/config/frequency_words.txt
similarity index 85%
rename from frequency_words.txt
rename to config/frequency_words.txt
index ac145a480c..3cb77aea06 100644
--- a/frequency_words.txt
+++ b/config/frequency_words.txt
@@ -41,6 +41,10 @@ HarmonyOS
字节
张一鸣
+小米
+雷军
+xiaomi
+
马斯克
特斯拉
@@ -59,11 +63,14 @@ gemini
deepmind
chatgpt
+Sam Altman
openai
claude
Anthropic
+苹果
+库克
iphone
ipad
mac
@@ -75,18 +82,25 @@ ai
汽车
自动驾驶
-l3
+
+手机
机器人
+国产
+中国
+
+美国
+
芯片
-半导体
光刻机
科技
核能
+新质生产力
+
月球
登月
火星
diff --git a/docker/.env b/docker/.env
new file mode 100644
index 0000000000..c9cc6b6105
--- /dev/null
+++ b/docker/.env
@@ -0,0 +1,11 @@
+# Webhook 配置
+FEISHU_WEBHOOK_URL=
+TELEGRAM_BOT_TOKEN=
+TELEGRAM_CHAT_ID=
+DINGTALK_WEBHOOK_URL=
+WEWORK_WEBHOOK_URL=
+
+# 运行配置
+CRON_SCHEDULE=*/30 * * * * # 定时任务表达式,每 30 分钟执行一次(比如 8点,8点半,9点,9点半这种时间规律执行)
+RUN_MODE=cron # 运行模式:cron/once
+IMMEDIATE_RUN=true # 启动时立即执行一次
\ No newline at end of file
diff --git a/docker/Dockerfile b/docker/Dockerfile
new file mode 100644
index 0000000000..55591f8470
--- /dev/null
+++ b/docker/Dockerfile
@@ -0,0 +1,36 @@
+FROM python:3.10-slim
+
+WORKDIR /app
+
+# https://github.com/aptible/supercronic/releases
+ENV SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/v0.2.34/supercronic-linux-amd64 \
+ SUPERCRONIC_SHA1SUM=e8631edc1775000d119b70fd40339a7238eece14 \
+ SUPERCRONIC=supercronic-linux-amd64
+
+RUN apt-get update && \
+ apt-get install -y --no-install-recommends curl && \
+ curl -fsSLO "$SUPERCRONIC_URL" && \
+ echo "${SUPERCRONIC_SHA1SUM} ${SUPERCRONIC}" | sha1sum -c - && \
+ chmod +x "$SUPERCRONIC" && \
+ mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" && \
+ ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic && \
+ apt-get remove -y curl && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY main.py .
+COPY docker/manage.py .
+COPY docker/entrypoint.sh /entrypoint.sh
+
+RUN chmod +x /entrypoint.sh && \
+ chmod +x manage.py && \
+ mkdir -p /app/config /app/output
+
+ENV PYTHONUNBUFFERED=1 \
+ CONFIG_PATH=/app/config/config.yaml \
+ FREQUENCY_WORDS_PATH=/app/config/frequency_words.txt
+
+ENTRYPOINT ["/entrypoint.sh"]
\ No newline at end of file
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
new file mode 100644
index 0000000000..263bd2c814
--- /dev/null
+++ b/docker/docker-compose.yml
@@ -0,0 +1,22 @@
+services:
+ trend-radar:
+ build:
+ context: ..
+ dockerfile: docker/Dockerfile
+ container_name: trend-radar
+ restart: unless-stopped
+
+ volumes:
+ - ../config:/app/config:ro
+ - ../output:/app/output
+
+ environment:
+ - TZ=Asia/Shanghai
+ - FEISHU_WEBHOOK_URL=${FEISHU_WEBHOOK_URL:-}
+ - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN:-}
+ - TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID:-}
+ - DINGTALK_WEBHOOK_URL=${DINGTALK_WEBHOOK_URL:-}
+ - WEWORK_WEBHOOK_URL=${WEWORK_WEBHOOK_URL:-}
+ - CRON_SCHEDULE=${CRON_SCHEDULE:-*/5 * * * *}
+ - RUN_MODE=${RUN_MODE:-cron}
+ - IMMEDIATE_RUN=${IMMEDIATE_RUN:-true}
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
new file mode 100644
index 0000000000..08a8e81e62
--- /dev/null
+++ b/docker/entrypoint.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+set -e
+
+# 检查配置文件
+if [ ! -f "/app/config/config.yaml" ] || [ ! -f "/app/config/frequency_words.txt" ]; then
+ echo "❌ 配置文件缺失"
+ exit 1
+fi
+
+# 保存环境变量
+env >> /etc/environment
+
+case "${RUN_MODE:-cron}" in
+"once")
+ echo "🔄 单次执行"
+ exec /usr/local/bin/python main.py
+ ;;
+"cron")
+ # 生成 crontab
+ echo "${CRON_SCHEDULE:-*/30 * * * *} cd /app && /usr/local/bin/python main.py" > /tmp/crontab
+
+ echo "📅 生成的crontab内容:"
+ cat /tmp/crontab
+
+ if ! /usr/local/bin/supercronic-linux-amd64 -test /tmp/crontab; then
+ echo "❌ crontab格式验证失败"
+ exit 1
+ fi
+
+ # 立即执行一次(如果配置了)
+ if [ "${IMMEDIATE_RUN:-false}" = "true" ]; then
+ echo "▶️ 立即执行一次"
+ /usr/local/bin/python main.py
+ fi
+
+ echo "⏰ 启动supercronic: ${CRON_SCHEDULE:-*/30 * * * *}"
+ echo "🎯 supercronic 将作为 PID 1 运行"
+
+ exec /usr/local/bin/supercronic-linux-amd64 -passthrough-logs /tmp/crontab
+ ;;
+*)
+ exec "$@"
+ ;;
+esac
\ No newline at end of file
diff --git a/docker/manage.py b/docker/manage.py
new file mode 100644
index 0000000000..e72d553e04
--- /dev/null
+++ b/docker/manage.py
@@ -0,0 +1,472 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+新闻爬虫容器管理工具 - supercronic
+"""
+
+import os
+import sys
+import subprocess
+import time
+from pathlib import Path
+
+
+def run_command(cmd, shell=True, capture_output=True):
+ """执行系统命令"""
+ try:
+ result = subprocess.run(
+ cmd, shell=shell, capture_output=capture_output, text=True
+ )
+ return result.returncode == 0, result.stdout, result.stderr
+ except Exception as e:
+ return False, "", str(e)
+
+
+def manual_run():
+ """手动执行一次爬虫"""
+ print("🔄 手动执行爬虫...")
+ try:
+ result = subprocess.run(
+ ["python", "main.py"], cwd="/app", capture_output=False, text=True
+ )
+ if result.returncode == 0:
+ print("✅ 执行完成")
+ else:
+ print(f"❌ 执行失败,退出码: {result.returncode}")
+ except Exception as e:
+ print(f"❌ 执行出错: {e}")
+
+
+def parse_cron_schedule(cron_expr):
+ """解析cron表达式并返回人类可读的描述"""
+ if not cron_expr or cron_expr == "未设置":
+ return "未设置"
+
+ try:
+ parts = cron_expr.strip().split()
+ if len(parts) != 5:
+ return f"原始表达式: {cron_expr}"
+
+ minute, hour, day, month, weekday = parts
+
+ # 分析分钟
+ if minute == "*":
+ minute_desc = "每分钟"
+ elif minute.startswith("*/"):
+ interval = minute[2:]
+ minute_desc = f"每{interval}分钟"
+ elif "," in minute:
+ minute_desc = f"在第{minute}分钟"
+ else:
+ minute_desc = f"在第{minute}分钟"
+
+ # 分析小时
+ if hour == "*":
+ hour_desc = "每小时"
+ elif hour.startswith("*/"):
+ interval = hour[2:]
+ hour_desc = f"每{interval}小时"
+ elif "," in hour:
+ hour_desc = f"在{hour}点"
+ else:
+ hour_desc = f"在{hour}点"
+
+ # 分析日期
+ if day == "*":
+ day_desc = "每天"
+ elif day.startswith("*/"):
+ interval = day[2:]
+ day_desc = f"每{interval}天"
+ else:
+ day_desc = f"每月{day}号"
+
+ # 分析月份
+ if month == "*":
+ month_desc = "每月"
+ else:
+ month_desc = f"在{month}月"
+
+ # 分析星期
+ weekday_names = {
+ "0": "周日", "1": "周一", "2": "周二", "3": "周三",
+ "4": "周四", "5": "周五", "6": "周六", "7": "周日"
+ }
+ if weekday == "*":
+ weekday_desc = ""
+ else:
+ weekday_desc = f"在{weekday_names.get(weekday, weekday)}"
+
+ # 组合描述
+ if minute.startswith("*/") and hour == "*" and day == "*" and month == "*" and weekday == "*":
+ # 简单的间隔模式,如 */30 * * * *
+ return f"每{minute[2:]}分钟执行一次"
+ elif hour != "*" and minute != "*" and day == "*" and month == "*" and weekday == "*":
+ # 每天特定时间,如 0 9 * * *
+ return f"每天{hour}:{minute.zfill(2)}执行"
+ elif weekday != "*" and day == "*":
+ # 每周特定时间
+ return f"{weekday_desc}{hour}:{minute.zfill(2)}执行"
+ else:
+ # 复杂模式,显示详细信息
+ desc_parts = [part for part in [month_desc, day_desc, weekday_desc, hour_desc, minute_desc] if part and part != "每月" and part != "每天" and part != "每小时"]
+ if desc_parts:
+ return " ".join(desc_parts) + "执行"
+ else:
+ return f"复杂表达式: {cron_expr}"
+
+ except Exception as e:
+ return f"解析失败: {cron_expr}"
+
+
+def show_status():
+ """显示容器状态"""
+ print("📊 容器状态:")
+
+ # 检查 PID 1 状态
+ supercronic_is_pid1 = False
+ pid1_cmdline = ""
+ try:
+ with open('/proc/1/cmdline', 'r') as f:
+ pid1_cmdline = f.read().replace('\x00', ' ').strip()
+ print(f" 🔍 PID 1 进程: {pid1_cmdline}")
+
+ if "supercronic" in pid1_cmdline.lower():
+ print(" ✅ supercronic 正确运行为 PID 1")
+ supercronic_is_pid1 = True
+ else:
+ print(" ❌ PID 1 不是 supercronic")
+ print(f" 📋 实际的 PID 1: {pid1_cmdline}")
+ except Exception as e:
+ print(f" ❌ 无法读取 PID 1 信息: {e}")
+
+ # 检查环境变量
+ cron_schedule = os.environ.get("CRON_SCHEDULE", "未设置")
+ run_mode = os.environ.get("RUN_MODE", "未设置")
+ immediate_run = os.environ.get("IMMEDIATE_RUN", "未设置")
+
+ print(f" ⚙️ 运行配置:")
+ print(f" CRON_SCHEDULE: {cron_schedule}")
+
+ # 解析并显示cron表达式的含义
+ cron_description = parse_cron_schedule(cron_schedule)
+ print(f" ⏰ 执行频率: {cron_description}")
+
+ print(f" RUN_MODE: {run_mode}")
+ print(f" IMMEDIATE_RUN: {immediate_run}")
+
+ # 检查配置文件
+ config_files = ["/app/config/config.yaml", "/app/config/frequency_words.txt"]
+ print(" 📁 配置文件:")
+ for file_path in config_files:
+ if Path(file_path).exists():
+ print(f" ✅ {Path(file_path).name}")
+ else:
+ print(f" ❌ {Path(file_path).name} 缺失")
+
+ # 检查关键文件
+ key_files = [
+ ("/usr/local/bin/supercronic-linux-amd64", "supercronic二进制文件"),
+ ("/usr/local/bin/supercronic", "supercronic软链接"),
+ ("/tmp/crontab", "crontab文件"),
+ ("/entrypoint.sh", "启动脚本")
+ ]
+
+ print(" 📂 关键文件检查:")
+ for file_path, description in key_files:
+ if Path(file_path).exists():
+ print(f" ✅ {description}: 存在")
+ # 对于crontab文件,显示内容
+ if file_path == "/tmp/crontab":
+ try:
+ with open(file_path, 'r') as f:
+ crontab_content = f.read().strip()
+ print(f" 内容: {crontab_content}")
+ except:
+ pass
+ else:
+ print(f" ❌ {description}: 不存在")
+
+ # 检查容器运行时间
+ print(" ⏱️ 容器时间信息:")
+ try:
+ # 检查 PID 1 的启动时间
+ with open('/proc/1/stat', 'r') as f:
+ stat_content = f.read().strip().split()
+ if len(stat_content) >= 22:
+ # starttime 是第22个字段(索引21)
+ starttime_ticks = int(stat_content[21])
+
+ # 读取系统启动时间
+ with open('/proc/stat', 'r') as stat_f:
+ for line in stat_f:
+ if line.startswith('btime'):
+ boot_time = int(line.split()[1])
+ break
+ else:
+ boot_time = 0
+
+ # 读取系统时钟频率
+ clock_ticks = os.sysconf(os.sysconf_names['SC_CLK_TCK'])
+
+ if boot_time > 0:
+ pid1_start_time = boot_time + (starttime_ticks / clock_ticks)
+ current_time = time.time()
+ uptime_seconds = int(current_time - pid1_start_time)
+ uptime_minutes = uptime_seconds // 60
+ uptime_hours = uptime_minutes // 60
+
+ if uptime_hours > 0:
+ print(f" PID 1 运行时间: {uptime_hours} 小时 {uptime_minutes % 60} 分钟")
+ else:
+ print(f" PID 1 运行时间: {uptime_minutes} 分钟 ({uptime_seconds} 秒)")
+ else:
+ print(f" PID 1 运行时间: 无法精确计算")
+ else:
+ print(" ❌ 无法解析 PID 1 统计信息")
+ except Exception as e:
+ print(f" ❌ 时间检查失败: {e}")
+
+ # 状态总结和建议
+ print(" 📊 状态总结:")
+ if supercronic_is_pid1:
+ print(" ✅ supercronic 正确运行为 PID 1")
+ print(" ✅ 定时任务应该正常工作")
+
+ # 显示当前的调度信息
+ if cron_schedule != "未设置":
+ print(f" ⏰ 当前调度: {cron_description}")
+
+ # 提供一些常见的调度建议
+ if "分钟" in cron_description and "每30分钟" not in cron_description and "每60分钟" not in cron_description:
+ print(" 💡 频繁执行模式,适合实时监控")
+ elif "小时" in cron_description:
+ print(" 💡 按小时执行模式,适合定期汇总")
+ elif "天" in cron_description:
+ print(" 💡 每日执行模式,适合日报生成")
+
+ print(" 💡 如果定时任务不执行,检查:")
+ print(" • crontab 格式是否正确")
+ print(" • 时区设置是否正确")
+ print(" • 应用程序是否有错误")
+ else:
+ print(" ❌ supercronic 状态异常")
+ if pid1_cmdline:
+ print(f" 📋 当前 PID 1: {pid1_cmdline}")
+ print(" 💡 建议操作:")
+ print(" • 重启容器: docker restart trend-radar")
+ print(" • 检查容器日志: docker logs trend-radar")
+
+ # 显示日志检查建议
+ print(" 📋 运行状态检查:")
+ print(" • 查看完整容器日志: docker logs trend-radar")
+ print(" • 查看实时日志: docker logs -f trend-radar")
+ print(" • 手动执行测试: python manage.py run")
+ print(" • 重启容器服务: docker restart trend-radar")
+
+
+def show_config():
+ """显示当前配置"""
+ print("⚙️ 当前配置:")
+
+ env_vars = [
+ "CRON_SCHEDULE",
+ "RUN_MODE",
+ "IMMEDIATE_RUN",
+ "FEISHU_WEBHOOK_URL",
+ "DINGTALK_WEBHOOK_URL",
+ "WEWORK_WEBHOOK_URL",
+ "TELEGRAM_BOT_TOKEN",
+ "TELEGRAM_CHAT_ID",
+ "CONFIG_PATH",
+ "FREQUENCY_WORDS_PATH",
+ ]
+
+ for var in env_vars:
+ value = os.environ.get(var, "未设置")
+ # 隐藏敏感信息
+ if any(sensitive in var for sensitive in ["WEBHOOK", "TOKEN", "KEY"]):
+ if value and value != "未设置":
+ masked_value = value[:10] + "***" if len(value) > 10 else "***"
+ print(f" {var}: {masked_value}")
+ else:
+ print(f" {var}: {value}")
+ else:
+ print(f" {var}: {value}")
+
+ crontab_file = "/tmp/crontab"
+ if Path(crontab_file).exists():
+ print(" 📅 Crontab内容:")
+ try:
+ with open(crontab_file, "r") as f:
+ content = f.read().strip()
+ print(f" {content}")
+ except Exception as e:
+ print(f" 读取失败: {e}")
+ else:
+ print(" 📅 Crontab文件不存在")
+
+
+def show_files():
+ """显示输出文件"""
+ print("📁 输出文件:")
+
+ output_dir = Path("/app/output")
+ if not output_dir.exists():
+ print(" 📭 输出目录不存在")
+ return
+
+ # 显示最近的文件
+ date_dirs = sorted([d for d in output_dir.iterdir() if d.is_dir()], reverse=True)
+
+ if not date_dirs:
+ print(" 📭 输出目录为空")
+ return
+
+ # 显示最近2天的文件
+ for date_dir in date_dirs[:2]:
+ print(f" 📅 {date_dir.name}:")
+ for subdir in ["html", "txt"]:
+ sub_path = date_dir / subdir
+ if sub_path.exists():
+ files = list(sub_path.glob("*"))
+ if files:
+ recent_files = sorted(
+ files, key=lambda x: x.stat().st_mtime, reverse=True
+ )[:3]
+ print(f" 📂 {subdir}: {len(files)} 个文件")
+ for file in recent_files:
+ mtime = time.ctime(file.stat().st_mtime)
+ size_kb = file.stat().st_size // 1024
+ print(
+ f" 📄 {file.name} ({size_kb}KB, {mtime.split()[3][:5]})"
+ )
+ else:
+ print(f" 📂 {subdir}: 空")
+
+
+def show_logs():
+ """显示实时日志"""
+ print("📋 实时日志 (按 Ctrl+C 退出):")
+ print("💡 提示: 这将显示 PID 1 进程的输出")
+ try:
+ # 尝试多种方法查看日志
+ log_files = [
+ "/proc/1/fd/1", # PID 1 的标准输出
+ "/proc/1/fd/2", # PID 1 的标准错误
+ ]
+
+ for log_file in log_files:
+ if Path(log_file).exists():
+ print(f"📄 尝试读取: {log_file}")
+ subprocess.run(["tail", "-f", log_file], check=True)
+ break
+ else:
+ print("📋 无法找到标准日志文件,建议使用: docker logs trend-radar")
+
+ except KeyboardInterrupt:
+ print("\n👋 退出日志查看")
+ except Exception as e:
+ print(f"❌ 查看日志失败: {e}")
+ print("💡 建议使用: docker logs trend-radar")
+
+
+def restart_supercronic():
+ """重启supercronic进程"""
+ print("🔄 重启supercronic...")
+ print("⚠️ 注意: supercronic 是 PID 1,无法直接重启")
+
+ # 检查当前 PID 1
+ try:
+ with open('/proc/1/cmdline', 'r') as f:
+ pid1_cmdline = f.read().replace('\x00', ' ').strip()
+ print(f" 🔍 当前 PID 1: {pid1_cmdline}")
+
+ if "supercronic" in pid1_cmdline.lower():
+ print(" ✅ PID 1 是 supercronic")
+ print(" 💡 要重启 supercronic,需要重启整个容器:")
+ print(" docker restart trend-radar")
+ else:
+ print(" ❌ PID 1 不是 supercronic,这是异常状态")
+ print(" 💡 建议重启容器以修复问题:")
+ print(" docker restart trend-radar")
+ except Exception as e:
+ print(f" ❌ 无法检查 PID 1: {e}")
+ print(" 💡 建议重启容器: docker restart trend-radar")
+
+
+def show_help():
+ """显示帮助信息"""
+ help_text = """
+🐳 TrendRadar 容器管理工具
+
+📋 命令列表:
+ run - 手动执行一次爬虫
+ status - 显示容器运行状态
+ config - 显示当前配置
+ files - 显示输出文件
+ logs - 实时查看日志
+ restart - 重启说明
+ help - 显示此帮助
+
+📖 使用示例:
+ # 在容器中执行
+ python manage.py run
+ python manage.py status
+ python manage.py logs
+
+ # 在宿主机执行
+ docker exec -it trend-radar python manage.py run
+ docker exec -it trend-radar python manage.py status
+ docker logs trend-radar
+
+💡 常用操作指南:
+ 1. 检查运行状态: status
+ - 查看 supercronic 是否为 PID 1
+ - 检查配置文件和关键文件
+ - 查看 cron 调度设置
+
+ 2. 手动执行测试: run
+ - 立即执行一次新闻爬取
+ - 测试程序是否正常工作
+
+ 3. 查看日志: logs
+ - 实时监控运行情况
+ - 也可使用: docker logs trend-radar
+
+ 4. 重启服务: restart
+ - 由于 supercronic 是 PID 1,需要重启整个容器
+ - 使用: docker restart trend-radar
+"""
+ print(help_text)
+
+
+def main():
+ if len(sys.argv) < 2:
+ show_help()
+ return
+
+ command = sys.argv[1]
+ commands = {
+ "run": manual_run,
+ "status": show_status,
+ "config": show_config,
+ "files": show_files,
+ "logs": show_logs,
+ "restart": restart_supercronic,
+ "help": show_help,
+ }
+
+ if command in commands:
+ try:
+ commands[command]()
+ except KeyboardInterrupt:
+ print("\n👋 操作已取消")
+ except Exception as e:
+ print(f"❌ 执行出错: {e}")
+ else:
+ print(f"❌ 未知命令: {command}")
+ print("运行 'python manage.py help' 查看可用命令")
+
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
diff --git a/main.py b/main.py
index 82dfd4efe5..dfdd87af3e 100644
--- a/main.py
+++ b/main.py
@@ -1,52 +1,151 @@
# coding=utf-8
import json
-import time
+import os
import random
import re
-from datetime import datetime
+import time
import webbrowser
-from typing import Dict, List, Tuple, Optional, Union
+from dataclasses import dataclass
+from datetime import datetime
from pathlib import Path
-import os
+from typing import Dict, List, Tuple, Optional, Union
-import requests
import pytz
+import requests
+import yaml
+
+
+class ConfigManager:
+ """配置管理器"""
+
+ @staticmethod
+ def _load_config_file() -> Dict:
+ """加载配置文件"""
+ config_path = os.environ.get("CONFIG_PATH", "config/config.yaml")
+
+ if not Path(config_path).exists():
+ raise FileNotFoundError(f"配置文件 {config_path} 不存在")
+
+ try:
+ with open(config_path, "r", encoding="utf-8") as f:
+ config_data = yaml.safe_load(f)
+
+ print(f"配置文件加载成功: {config_path}")
+ return config_data
+
+ except Exception as e:
+ raise RuntimeError(f"配置文件解析失败: {e}")
-CONFIG = {
- "VERSION": "1.4.1",
- "VERSION_CHECK_URL": "https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/version",
- "SHOW_VERSION_UPDATE": True, # 控制显示版本更新提示,改成 False 将不接受新版本提示
- "FEISHU_MESSAGE_SEPARATOR": "━━━━━━━━━━━━━━━━━━━", # feishu消息分割线
- "REQUEST_INTERVAL": 1000, # 请求间隔(毫秒)
- "REPORT_TYPE": "daily", # 报告类型: "current"|"daily"|"both"
- "RANK_THRESHOLD": 5, # 排名高亮阈值
- "USE_PROXY": True, # 是否启用代理
- "DEFAULT_PROXY": "http://127.0.0.1:10086",
- "ENABLE_CRAWLER": True, # 是否启用爬取新闻功能,False时直接停止程序
- "ENABLE_NOTIFICATION": True, # 是否启用通知功能,False时不发送手机通知
- "FOCUS_NEW_ONLY": False, # 是否只关注新增新闻,True时只统计和推送新增的新闻(增量推送)
- # FOCUS_NEW_ONLY 增量推送开关:避免重复推送相同内容,只在有新内容时才发通知
- # 优点:1.减少重复推送噪音 2.专注最新动态 3.避免通知疲劳
- # 适用场景:1.高频监控(≤30分钟间隔) 2.实时热点追踪 3.只关心新话题而非持续热度
- "MESSAGE_BATCH_SIZE": 4000, # 消息分批大小(字节)
- "BATCH_SEND_INTERVAL": 1, # 批次发送间隔(秒)
- # 飞书机器人的 webhook URL
- "FEISHU_WEBHOOK_URL": "",
- # 钉钉机器人的 webhook URL
- "DINGTALK_WEBHOOK_URL": "",
- # 企业微信机器人的 webhook URL
- "WEWORK_WEBHOOK_URL": "",
- # Telegram 要填两个
- "TELEGRAM_BOT_TOKEN": "",
- "TELEGRAM_CHAT_ID": "",
- # 用于让关注度更高的新闻在更前面显示,这里是权重排序配置,合起来是 1 就行
- "WEIGHT_CONFIG": {
- "RANK_WEIGHT": 0.6, # 排名
- "FREQUENCY_WEIGHT": 0.3, # 频次
- "HOTNESS_WEIGHT": 0.1, # 热度
- },
-}
+ def __init__(self):
+ self.config_data = self._load_config_file()
+ self.config = self._build_config()
+ self.platforms = self.config_data["platforms"]
+
+ def _get_webhook_config(self, config_key: str, env_key: str) -> str:
+ """获取 Webhook 配置"""
+ env_value = os.environ.get(env_key, "").strip()
+ if env_value:
+ return env_value
+
+ return (
+ self.config_data.get("notification", {})
+ .get("webhooks", {})
+ .get(config_key, "")
+ )
+
+ def _build_config(self) -> Dict:
+ """构建配置字典,环境变量优先级高于配置文件"""
+
+ feishu_url = self._get_webhook_config("feishu_url", "FEISHU_WEBHOOK_URL")
+ dingtalk_url = self._get_webhook_config("dingtalk_url", "DINGTALK_WEBHOOK_URL")
+ wework_url = self._get_webhook_config("wework_url", "WEWORK_WEBHOOK_URL")
+ telegram_token = self._get_webhook_config(
+ "telegram_bot_token", "TELEGRAM_BOT_TOKEN"
+ )
+ telegram_chat_id = self._get_webhook_config(
+ "telegram_chat_id", "TELEGRAM_CHAT_ID"
+ )
+
+ # 输出配置来源信息
+ webhook_sources = []
+ if feishu_url:
+ source = "环境变量" if os.environ.get("FEISHU_WEBHOOK_URL") else "配置文件"
+ webhook_sources.append(f"飞书({source})")
+ if dingtalk_url:
+ source = (
+ "环境变量" if os.environ.get("DINGTALK_WEBHOOK_URL") else "配置文件"
+ )
+ webhook_sources.append(f"钉钉({source})")
+ if wework_url:
+ source = "环境变量" if os.environ.get("WEWORK_WEBHOOK_URL") else "配置文件"
+ webhook_sources.append(f"企业微信({source})")
+ if telegram_token and telegram_chat_id:
+ token_source = (
+ "环境变量" if os.environ.get("TELEGRAM_BOT_TOKEN") else "配置文件"
+ )
+ chat_source = (
+ "环境变量" if os.environ.get("TELEGRAM_CHAT_ID") else "配置文件"
+ )
+ webhook_sources.append(f"Telegram({token_source}/{chat_source})")
+
+ if webhook_sources:
+ print(f"Webhook 配置来源: {', '.join(webhook_sources)}")
+ else:
+ print("未配置任何 Webhook")
+
+ config = {
+ "VERSION": self.config_data["app"]["version"],
+ "VERSION_CHECK_URL": self.config_data["app"]["version_check_url"],
+ "SHOW_VERSION_UPDATE": self.config_data["app"]["show_version_update"],
+ "FEISHU_MESSAGE_SEPARATOR": self.config_data["notification"][
+ "feishu_message_separator"
+ ],
+ "REQUEST_INTERVAL": self.config_data["crawler"]["request_interval"],
+ "REPORT_MODE": self.config_data["report"]["mode"],
+ "RANK_THRESHOLD": self.config_data["report"]["rank_threshold"],
+ "USE_PROXY": self.config_data["crawler"]["use_proxy"],
+ "DEFAULT_PROXY": self.config_data["crawler"]["default_proxy"],
+ "ENABLE_CRAWLER": self.config_data["crawler"]["enable_crawler"],
+ "ENABLE_NOTIFICATION": self.config_data["notification"][
+ "enable_notification"
+ ],
+ "MESSAGE_BATCH_SIZE": self.config_data["notification"][
+ "message_batch_size"
+ ],
+ "BATCH_SEND_INTERVAL": self.config_data["notification"][
+ "batch_send_interval"
+ ],
+ "FEISHU_WEBHOOK_URL": feishu_url,
+ "DINGTALK_WEBHOOK_URL": dingtalk_url,
+ "WEWORK_WEBHOOK_URL": wework_url,
+ "TELEGRAM_BOT_TOKEN": telegram_token,
+ "TELEGRAM_CHAT_ID": telegram_chat_id,
+ "WEIGHT_CONFIG": {
+ "RANK_WEIGHT": self.config_data["weight"]["rank_weight"],
+ "FREQUENCY_WEIGHT": self.config_data["weight"]["frequency_weight"],
+ "HOTNESS_WEIGHT": self.config_data["weight"]["hotness_weight"],
+ },
+ }
+
+ return config
+
+ def get_config(self) -> Dict:
+ """获取配置字典"""
+ return self.config
+
+ def get_platforms(self) -> List:
+ """获取平台列表"""
+ return self.platforms
+
+
+print("正在加载配置...")
+config_manager = ConfigManager()
+CONFIG = config_manager.get_config()
+PLATFORMS = config_manager.get_platforms()
+
+print(f"TrendRadar v{CONFIG['VERSION']} 配置加载完成")
+print(f"监控平台数量: {len(PLATFORMS)}")
class TimeHelper:
@@ -75,10 +174,10 @@ class VersionChecker:
parts = version_str.strip().split(".")
if len(parts) != 3:
raise ValueError("版本号格式不正确")
- return tuple(int(part) for part in parts)
+ return int(parts[0]), int(parts[1]), int(parts[2])
except (ValueError, AttributeError):
print(f"无法解析版本号: {version_str}")
- return (0, 0, 0)
+ return 0, 0, 0
@staticmethod
def compare_versions(current: str, remote: str) -> int:
@@ -220,17 +319,17 @@ class DataFetcher:
) -> Tuple[Dict, Dict, List]:
"""爬取多个网站数据"""
results = {}
- id_to_alias = {}
+ id_to_name = {}
failed_ids = []
for i, id_info in enumerate(ids_list):
if isinstance(id_info, tuple):
- id_value, alias = id_info
+ id_value, name = id_info
else:
id_value = id_info
- alias = id_value
+ name = id_value
- id_to_alias[id_value] = alias
+ id_to_name[id_value] = name
response, _, _ = self.fetch_data(id_info)
if response:
@@ -265,7 +364,7 @@ class DataFetcher:
time.sleep(actual_interval / 1000)
print(f"成功: {list(results.keys())}, 失败: {failed_ids}")
- return results, id_to_alias, failed_ids
+ return results, id_to_name, failed_ids
class DataProcessor:
@@ -277,10 +376,7 @@ class DataProcessor:
if not isinstance(title, str):
title = str(title)
- # 移除或替换常见的特殊字符
- cleaned_title = title.replace("\n", " ").replace( # 换行符替换为空格
- "\r", " "
- ) # 回车符替换为空格
+ cleaned_title = title.replace("\n", " ").replace("\r", " ")
cleaned_title = re.sub(r"\s+", " ", cleaned_title)
@@ -301,8 +397,10 @@ class DataProcessor:
return len(files) <= 1 # 0个文件或1个文件都算第一次
@staticmethod
- def detect_latest_new_titles(id_to_alias: Dict) -> Dict:
- """检测当日最新批次的新增标题"""
+ def detect_latest_new_titles(
+ current_platform_ids: Optional[List[str]] = None,
+ ) -> Dict:
+ """检测当日最新批次的新增标题,支持按当前监控平台过滤"""
date_folder = TimeHelper.format_date_folder()
txt_dir = Path("output") / date_folder / "txt"
@@ -311,26 +409,43 @@ class DataProcessor:
files = sorted([f for f in txt_dir.iterdir() if f.suffix == ".txt"])
if len(files) < 2:
- # 如果只有一个文件(第一次爬取),没有"新增"的概念,返回空字典
return {}
+ # 解析最新文件
latest_file = files[-1]
- latest_titles = DataProcessor._parse_file_titles(latest_file)
+ latest_titles, _ = DataProcessor._parse_file_titles(latest_file)
- # 汇总历史标题
+ # 如果指定了当前平台列表,过滤最新文件数据
+ if current_platform_ids is not None:
+ filtered_latest_titles = {}
+ for source_id, title_data in latest_titles.items():
+ if source_id in current_platform_ids:
+ filtered_latest_titles[source_id] = title_data
+ latest_titles = filtered_latest_titles
+
+ # 汇总历史标题(按平台过滤)
historical_titles = {}
for file_path in files[:-1]:
- historical_data = DataProcessor._parse_file_titles(file_path)
- for source_name, titles_data in historical_data.items():
- if source_name not in historical_titles:
- historical_titles[source_name] = set()
+ historical_data, _ = DataProcessor._parse_file_titles(file_path)
+
+ # 过滤历史数据
+ if current_platform_ids is not None:
+ filtered_historical_data = {}
+ for source_id, title_data in historical_data.items():
+ if source_id in current_platform_ids:
+ filtered_historical_data[source_id] = title_data
+ historical_data = filtered_historical_data
+
+ for source_id, titles_data in historical_data.items():
+ if source_id not in historical_titles:
+ historical_titles[source_id] = set()
for title in titles_data.keys():
- historical_titles[source_name].add(title)
+ historical_titles[source_id].add(title)
# 找出新增标题
new_titles = {}
- for source_name, latest_source_titles in latest_titles.items():
- historical_set = historical_titles.get(source_name, set())
+ for source_id, latest_source_titles in latest_titles.items():
+ historical_set = historical_titles.get(source_id, set())
source_new_titles = {}
for title, title_data in latest_source_titles.items():
@@ -338,21 +453,15 @@ class DataProcessor:
source_new_titles[title] = title_data
if source_new_titles:
- source_id = None
- for id_val, alias in id_to_alias.items():
- if alias == source_name:
- source_id = id_val
- break
-
- if source_id:
- new_titles[source_id] = source_new_titles
+ new_titles[source_id] = source_new_titles
return new_titles
@staticmethod
- def _parse_file_titles(file_path: Path) -> Dict:
- """解析单个txt文件的标题数据"""
- titles_by_source = {}
+ def _parse_file_titles(file_path: Path) -> Tuple[Dict, Dict]:
+ """解析单个txt文件的标题数据,返回(titles_by_id, id_to_name)"""
+ titles_by_id = {}
+ id_to_name = {}
with open(file_path, "r", encoding="utf-8") as f:
content = f.read()
@@ -366,8 +475,18 @@ class DataProcessor:
if len(lines) < 2:
continue
- source_name = lines[0].strip()
- titles_by_source[source_name] = {}
+ # id | name 或 id
+ header_line = lines[0].strip()
+ if " | " in header_line:
+ parts = header_line.split(" | ", 1)
+ source_id = parts[0].strip()
+ name = parts[1].strip()
+ id_to_name[source_id] = name
+ else:
+ source_id = header_line
+ id_to_name[source_id] = source_id
+
+ titles_by_id[source_id] = {}
for line in lines[1:]:
if line.strip():
@@ -383,7 +502,7 @@ class DataProcessor:
rank_str, title_part = title_part.split(". ", 1)
rank = int(rank_str)
- # 提取MOBILE URL
+ # 提取 MOBILE URL
mobile_url = ""
if " [MOBILE:" in title_part:
title_part, mobile_part = title_part.rsplit(
@@ -392,7 +511,7 @@ class DataProcessor:
if mobile_part.endswith("]"):
mobile_url = mobile_part[:-1]
- # 提取URL
+ # 提取 URL
url = ""
if " [URL:" in title_part:
title_part, url_part = title_part.rsplit(" [URL:", 1)
@@ -402,7 +521,7 @@ class DataProcessor:
title = DataProcessor.clean_title(title_part.strip())
ranks = [rank] if rank is not None else [1]
- titles_by_source[source_name][title] = {
+ titles_by_id[source_id][title] = {
"ranks": ranks,
"url": url,
"mobileUrl": mobile_url,
@@ -411,10 +530,10 @@ class DataProcessor:
except Exception as e:
print(f"解析标题行出错: {line}, 错误: {e}")
- return titles_by_source
+ return titles_by_id, id_to_name
@staticmethod
- def save_titles_to_file(results: Dict, id_to_alias: Dict, failed_ids: List) -> str:
+ def save_titles_to_file(results: Dict, id_to_name: Dict, failed_ids: List) -> str:
"""保存标题到文件"""
file_path = FileHelper.get_output_path(
"txt", f"{TimeHelper.format_time_filename()}.txt"
@@ -422,8 +541,12 @@ class DataProcessor:
with open(file_path, "w", encoding="utf-8") as f:
for id_value, title_data in results.items():
- display_name = id_to_alias.get(id_value, id_value)
- f.write(f"{display_name}\n")
+ # id | name 或 id
+ name = id_to_name.get(id_value)
+ if name and name != id_value:
+ f.write(f"{id_value} | {name}\n")
+ else:
+ f.write(f"{id_value}\n")
# 按排名排序标题
sorted_titles = []
@@ -457,20 +580,23 @@ class DataProcessor:
if failed_ids:
f.write("==== 以下ID请求失败 ====\n")
for id_value in failed_ids:
- display_name = id_to_alias.get(id_value, id_value)
- f.write(f"{display_name} (ID: {id_value})\n")
+ f.write(f"{id_value}\n")
return file_path
@staticmethod
def load_frequency_words(
- frequency_file: str = "frequency_words.txt",
+ frequency_file: Optional[str] = None,
) -> Tuple[List[Dict], List[str]]:
"""加载频率词配置"""
+ if frequency_file is None:
+ frequency_file = os.environ.get(
+ "FREQUENCY_WORDS_PATH", "config/frequency_words.txt"
+ )
+
frequency_path = Path(frequency_file)
if not frequency_path.exists():
- print(f"频率词文件 {frequency_file} 不存在")
- return [], []
+ raise FileNotFoundError(f"频率词文件 {frequency_file} 不存在")
with open(frequency_path, "r", encoding="utf-8") as f:
content = f.read()
@@ -515,8 +641,10 @@ class DataProcessor:
return processed_groups, filter_words
@staticmethod
- def read_all_today_titles() -> Tuple[Dict, Dict, Dict]:
- """读取当天所有标题文件"""
+ def read_all_today_titles(
+ current_platform_ids: Optional[List[str]] = None,
+ ) -> Tuple[Dict, Dict, Dict]:
+ """读取当天所有标题文件,支持按当前监控平台过滤"""
date_folder = TimeHelper.format_date_folder()
txt_dir = Path("output") / date_folder / "txt"
@@ -524,7 +652,7 @@ class DataProcessor:
return {}, {}, {}
all_results = {}
- id_to_alias = {}
+ final_id_to_name = {}
title_info = {}
files = sorted([f for f in txt_dir.iterdir() if f.suffix == ".txt"])
@@ -532,109 +660,55 @@ class DataProcessor:
for file_path in files:
time_info = file_path.stem
- with open(file_path, "r", encoding="utf-8") as f:
- content = f.read()
-
- sections = content.split("\n\n")
- for section in sections:
- if not section.strip() or "==== 以下ID请求失败 ====" in section:
- continue
-
- lines = section.strip().split("\n")
- if len(lines) < 2:
- continue
-
- source_name = lines[0].strip()
- title_data = {}
-
- for line in lines[1:]:
- if line.strip():
- try:
- rank = None
- title_part = line.strip()
-
- # 提取行首的排名数字
- if (
- ". " in title_part
- and title_part.split(". ")[0].isdigit()
- ):
- parts = title_part.split(". ", 1)
- rank = int(parts[0])
- title_part = parts[1]
-
- # 提取 MOBILE URL
- mobile_url = ""
- if " [MOBILE:" in title_part:
- title_part, mobile_part = title_part.rsplit(
- " [MOBILE:", 1
- )
- if mobile_part.endswith("]"):
- mobile_url = mobile_part[:-1]
-
- # 提取 URL
- url = ""
- if " [URL:" in title_part:
- title_part, url_part = title_part.rsplit(
- " [URL:", 1
- )
- if url_part.endswith("]"):
- url = url_part[:-1]
-
- title = title_part.strip()
- ranks = [rank] if rank is not None else [1]
-
- title_data[title] = {
- "ranks": ranks,
- "url": url,
- "mobileUrl": mobile_url,
- }
-
- except Exception as e:
- print(f"解析标题行出错: {line}, 错误: {e}")
-
- DataProcessor._process_source_data(
- source_name,
- title_data,
- time_info,
- all_results,
- title_info,
- id_to_alias,
- )
+ titles_by_id, file_id_to_name = DataProcessor._parse_file_titles(file_path)
- # 转换为ID格式
- id_results = {}
- id_title_info = {}
- for name, titles in all_results.items():
- for id_value, alias in id_to_alias.items():
- if alias == name:
- id_results[id_value] = titles
- id_title_info[id_value] = title_info[name]
- break
+ if current_platform_ids is not None:
+ filtered_titles_by_id = {}
+ filtered_id_to_name = {}
+
+ for source_id, title_data in titles_by_id.items():
+ if source_id in current_platform_ids:
+ filtered_titles_by_id[source_id] = title_data
+ if source_id in file_id_to_name:
+ filtered_id_to_name[source_id] = file_id_to_name[source_id]
+
+ titles_by_id = filtered_titles_by_id
+ file_id_to_name = filtered_id_to_name
- return id_results, id_to_alias, id_title_info
+ final_id_to_name.update(file_id_to_name)
+
+ for source_id, title_data in titles_by_id.items():
+ DataProcessor._process_source_data(
+ source_id,
+ title_data,
+ time_info,
+ all_results,
+ title_info,
+ )
+
+ return all_results, final_id_to_name, title_info
@staticmethod
def _process_source_data(
- source_name: str,
+ source_id: str,
title_data: Dict,
time_info: str,
all_results: Dict,
title_info: Dict,
- id_to_alias: Dict,
) -> None:
"""处理来源数据,合并重复标题"""
- if source_name not in all_results:
- all_results[source_name] = title_data
+ if source_id not in all_results:
+ all_results[source_id] = title_data
- if source_name not in title_info:
- title_info[source_name] = {}
+ if source_id not in title_info:
+ title_info[source_id] = {}
for title, data in title_data.items():
ranks = data.get("ranks", [])
url = data.get("url", "")
mobile_url = data.get("mobileUrl", "")
- title_info[source_name][title] = {
+ title_info[source_id][title] = {
"first_time": time_info,
"last_time": time_info,
"count": 1,
@@ -642,22 +716,19 @@ class DataProcessor:
"url": url,
"mobileUrl": mobile_url,
}
-
- reversed_id = source_name.lower().replace(" ", "-")
- id_to_alias[reversed_id] = source_name
else:
for title, data in title_data.items():
ranks = data.get("ranks", [])
url = data.get("url", "")
mobile_url = data.get("mobileUrl", "")
- if title not in all_results[source_name]:
- all_results[source_name][title] = {
+ if title not in all_results[source_id]:
+ all_results[source_id][title] = {
"ranks": ranks,
"url": url,
"mobileUrl": mobile_url,
}
- title_info[source_name][title] = {
+ title_info[source_id][title] = {
"first_time": time_info,
"last_time": time_info,
"count": 1,
@@ -666,7 +737,7 @@ class DataProcessor:
"mobileUrl": mobile_url,
}
else:
- existing_data = all_results[source_name][title]
+ existing_data = all_results[source_id][title]
existing_ranks = existing_data.get("ranks", [])
existing_url = existing_data.get("url", "")
existing_mobile_url = existing_data.get("mobileUrl", "")
@@ -676,19 +747,19 @@ class DataProcessor:
if rank not in merged_ranks:
merged_ranks.append(rank)
- all_results[source_name][title] = {
+ all_results[source_id][title] = {
"ranks": merged_ranks,
"url": existing_url or url,
"mobileUrl": existing_mobile_url or mobile_url,
}
- title_info[source_name][title]["last_time"] = time_info
- title_info[source_name][title]["ranks"] = merged_ranks
- title_info[source_name][title]["count"] += 1
- if not title_info[source_name][title].get("url"):
- title_info[source_name][title]["url"] = url
- if not title_info[source_name][title].get("mobileUrl"):
- title_info[source_name][title]["mobileUrl"] = mobile_url
+ title_info[source_id][title]["last_time"] = time_info
+ title_info[source_id][title]["ranks"] = merged_ranks
+ title_info[source_id][title]["count"] += 1
+ if not title_info[source_id][title].get("url"):
+ title_info[source_id][title]["url"] = url
+ if not title_info[source_id][title].get("mobileUrl"):
+ title_info[source_id][title]["mobileUrl"] = mobile_url
class StatisticsCalculator:
@@ -722,7 +793,6 @@ class StatisticsCalculator:
hotness_ratio = high_rank_count / len(ranks) if ranks else 0
hotness_weight = hotness_ratio * 100
- # 综合权重计算
total_weight = (
rank_weight * weight_config["RANK_WEIGHT"]
+ frequency_weight * weight_config["FREQUENCY_WEIGHT"]
@@ -746,12 +816,12 @@ class StatisticsCalculator:
# 主要按权重排序,权重相同时按最高排名排序,再相同时按出现次数排序
min_rank = min(ranks) if ranks else 999
- return (-weight, min_rank, -count)
+ return -weight, min_rank, -count
return sorted(titles_list, key=get_sort_key)
@staticmethod
- def _matches_word_groups(
+ def matches_word_groups(
title: str, word_groups: List[Dict], filter_words: List[str]
) -> bool:
"""检查标题是否匹配词组规则"""
@@ -791,44 +861,65 @@ class StatisticsCalculator:
results: Dict,
word_groups: List[Dict],
filter_words: List[str],
- id_to_alias: Dict,
+ id_to_name: Dict,
title_info: Optional[Dict] = None,
rank_threshold: int = CONFIG["RANK_THRESHOLD"],
new_titles: Optional[Dict] = None,
- focus_new_only: bool = False,
+ mode: str = "daily",
) -> Tuple[List[Dict], int]:
"""统计词频,支持必须词、频率词、过滤词,并标记新增标题"""
- # 检测是否是当天第一次爬取
is_first_today = DataProcessor.is_first_crawl_today()
# 确定处理的数据源和新增标记逻辑
- if focus_new_only:
+ if mode == "incremental":
if is_first_today:
- # 新增模式 + 当天第一次:处理所有新闻,都标记为新增
+ # 增量模式 + 当天第一次:处理所有新闻,都标记为新增
results_to_process = results
all_news_are_new = True
- total_input_news = sum(len(titles) for titles in results.values())
- print(
- f"新增模式:当天第一次爬取,处理 {total_input_news} 条新闻(所有匹配的新闻都视为新增)"
- )
else:
- # 新增模式 + 当天非第一次:只处理新增的新闻
+ # 增量模式 + 当天非第一次:只处理新增的新闻
results_to_process = new_titles if new_titles else {}
- all_news_are_new = True # 处理的都是新增新闻
- if new_titles:
- total_new_count = sum(len(titles) for titles in new_titles.values())
+ all_news_are_new = True
+ elif mode == "current":
+ # current 模式:只处理当前时间批次的新闻,但统计信息来自全部历史
+ if title_info:
+ latest_time = None
+ for source_titles in title_info.values():
+ for title_data in source_titles.values():
+ last_time = title_data.get("last_time", "")
+ if last_time:
+ if latest_time is None or last_time > latest_time:
+ latest_time = last_time
+
+ # 只处理 last_time 等于最新时间的新闻
+ if latest_time:
+ results_to_process = {}
+ for source_id, source_titles in results.items():
+ if source_id in title_info:
+ filtered_titles = {}
+ for title, title_data in source_titles.items():
+ if title in title_info[source_id]:
+ info = title_info[source_id][title]
+ if info.get("last_time") == latest_time:
+ filtered_titles[title] = title_data
+ if filtered_titles:
+ results_to_process[source_id] = filtered_titles
+
print(
- f"新增模式:检测到 {total_new_count} 条新增新闻,开始进行频率词匹配..."
+ f"当前榜单模式:最新时间 {latest_time},筛选出 {sum(len(titles) for titles in results_to_process.values())} 条当前榜单新闻"
)
else:
- print("新增模式:未检测到新增新闻")
+ results_to_process = results
+ else:
+ results_to_process = results
+ all_news_are_new = False
else:
- # 正常模式:处理所有新闻
+ # 当日汇总模式:处理所有新闻
results_to_process = results
all_news_are_new = False
total_input_news = sum(len(titles) for titles in results.values())
- print(f"正常模式:处理 {total_input_news} 条新闻")
+ print(f"当日汇总模式:处理 {total_input_news} 条新闻")
word_stats = {}
total_titles = 0
@@ -855,15 +946,17 @@ class StatisticsCalculator:
continue
# 使用统一的匹配逻辑
- matches_frequency_words = StatisticsCalculator._matches_word_groups(
+ matches_frequency_words = StatisticsCalculator.matches_word_groups(
title, word_groups, filter_words
)
if not matches_frequency_words:
continue
- # 如果是新增模式,统计匹配的新增新闻数量
- if focus_new_only and all_news_are_new:
+ # 如果是增量模式或 current 模式第一次,统计匹配的新增新闻数量
+ if (mode == "incremental" and all_news_are_new) or (
+ mode == "current" and is_first_today
+ ):
matched_new_count += 1
source_ranks = title_data.get("ranks", [])
@@ -905,7 +998,22 @@ class StatisticsCalculator:
url = source_url
mobile_url = source_mobile_url
+ # 对于 current 模式,从历史统计信息中获取完整数据
if (
+ mode == "current"
+ and title_info
+ and source_id in title_info
+ and title in title_info[source_id]
+ ):
+ info = title_info[source_id][title]
+ first_time = info.get("first_time", "")
+ last_time = info.get("last_time", "")
+ count_info = info.get("count", 1)
+ if "ranks" in info and info["ranks"]:
+ ranks = info["ranks"]
+ url = info.get("url", source_url)
+ mobile_url = info.get("mobileUrl", source_mobile_url)
+ elif (
title_info
and source_id in title_info
and title in title_info[source_id]
@@ -926,30 +1034,22 @@ class StatisticsCalculator:
first_time, last_time
)
- source_alias = id_to_alias.get(source_id, source_id)
+ source_name = id_to_name.get(source_id, source_id)
# 判断是否为新增
is_new = False
if all_news_are_new:
- # 新增模式下所有处理的新闻都是新增,或者当天第一次的所有新闻都是新增
+ # 增量模式下所有处理的新闻都是新增,或者当天第一次的所有新闻都是新增
is_new = True
elif new_titles and source_id in new_titles:
- # 正常模式下,检查是否在新增列表中
+ # 检查是否在新增列表中
new_titles_for_source = new_titles[source_id]
- if title in new_titles_for_source:
- is_new = True
- else:
- # 如果直接匹配失败,尝试去除首尾空格后匹配
- title_stripped = title.strip()
- for new_title in new_titles_for_source.keys():
- if title_stripped == new_title.strip():
- is_new = True
- break
+ is_new = title in new_titles_for_source
word_stats[group_key]["titles"][source_id].append(
{
"title": title,
- "source_alias": source_alias,
+ "source_name": source_name,
"first_time": first_time,
"last_time": last_time,
"time_display": time_display,
@@ -967,20 +1067,36 @@ class StatisticsCalculator:
processed_titles[source_id][title] = True
break
- if focus_new_only and not is_first_today:
- if new_titles:
- total_new_count = sum(len(titles) for titles in new_titles.values())
+ # 最后统一打印汇总信息
+ if mode == "incremental":
+ if is_first_today:
+ total_input_news = sum(len(titles) for titles in results.values())
print(
- f"新增模式:{total_new_count} 条新增新闻中,有 {matched_new_count} 条匹配频率词"
+ f"增量模式:当天第一次爬取,{total_input_news} 条新闻中有 {matched_new_count} 条匹配频率词"
)
- if matched_new_count == 0:
- print("新增模式:没有新增新闻匹配频率词,将不会发送通知")
else:
- print("新增模式:未检测到新增新闻")
- elif focus_new_only and is_first_today:
- print(
- f"新增模式:当天第一次爬取,{matched_new_count} 条新闻匹配频率词并将推送"
+ if new_titles:
+ total_new_count = sum(len(titles) for titles in new_titles.values())
+ print(
+ f"增量模式:{total_new_count} 条新增新闻中,有 {matched_new_count} 条匹配频率词"
+ )
+ if matched_new_count == 0:
+ print("增量模式:没有新增新闻匹配频率词,将不会发送通知")
+ else:
+ print("增量模式:未检测到新增新闻")
+ elif mode == "current":
+ total_input_news = sum(
+ len(titles) for titles in results_to_process.values()
)
+ if is_first_today:
+ print(
+ f"当前榜单模式:当天第一次爬取,{total_input_news} 条当前榜单新闻中有 {matched_new_count} 条匹配频率词"
+ )
+ else:
+ matched_count = sum(stat["count"] for stat in word_stats.values())
+ print(
+ f"当前榜单模式:{total_input_news} 条当前榜单新闻中有 {matched_count} 条匹配频率词"
+ )
stats = []
for group_key, data in word_stats.items():
@@ -988,7 +1104,6 @@ class StatisticsCalculator:
for source_id, title_list in data["titles"].items():
all_titles.extend(title_list)
- # 按权重排序标题
sorted_titles = StatisticsCalculator.sort_titles_by_weight(
all_titles, rank_threshold
)
@@ -1021,7 +1136,6 @@ class StatisticsCalculator:
min_rank = unique_ranks[0]
max_rank = unique_ranks[-1]
- # 根据格式类型选择不同的标记方式
if format_type == "html":
highlight_start = "<font color='red'><strong>"
highlight_end = "</strong></font>"
@@ -1041,7 +1155,6 @@ class StatisticsCalculator:
highlight_start = "**"
highlight_end = "**"
- # 格式化排名显示
if min_rank <= rank_threshold:
if min_rank == max_rank:
return f"{highlight_start}[{min_rank}]{highlight_end}"
@@ -1054,27 +1167,27 @@ class StatisticsCalculator:
return f"[{min_rank} - {max_rank}]"
@staticmethod
- def _format_rank_for_html(ranks: List[int], rank_threshold: int = 5) -> str:
+ def format_rank_for_html(ranks: List[int], rank_threshold: int = 5) -> str:
"""格式化HTML排名显示"""
return StatisticsCalculator._format_rank_base(ranks, rank_threshold, "html")
@staticmethod
- def _format_rank_for_feishu(ranks: List[int], rank_threshold: int = 5) -> str:
+ def format_rank_for_feishu(ranks: List[int], rank_threshold: int = 5) -> str:
"""格式化飞书排名显示"""
return StatisticsCalculator._format_rank_base(ranks, rank_threshold, "feishu")
@staticmethod
- def _format_rank_for_dingtalk(ranks: List[int], rank_threshold: int = 5) -> str:
+ def format_rank_for_dingtalk(ranks: List[int], rank_threshold: int = 5) -> str:
"""格式化钉钉排名显示"""
return StatisticsCalculator._format_rank_base(ranks, rank_threshold, "dingtalk")
@staticmethod
- def _format_rank_for_wework(ranks: List[int], rank_threshold: int = 5) -> str:
+ def format_rank_for_wework(ranks: List[int], rank_threshold: int = 5) -> str:
"""格式化企业微信排名显示"""
return StatisticsCalculator._format_rank_base(ranks, rank_threshold, "wework")
@staticmethod
- def _format_rank_for_telegram(ranks: List[int], rank_threshold: int = 5) -> str:
+ def format_rank_for_telegram(ranks: List[int], rank_threshold: int = 5) -> str:
"""格式化Telegram排名显示"""
return StatisticsCalculator._format_rank_base(ranks, rank_threshold, "telegram")
@@ -1097,32 +1210,36 @@ class ReportGenerator:
stats: List[Dict],
total_titles: int,
failed_ids: Optional[List] = None,
- is_daily: bool = False,
new_titles: Optional[Dict] = None,
- id_to_alias: Optional[Dict] = None,
+ id_to_name: Optional[Dict] = None,
+ mode: str = "daily",
+ is_daily_summary: bool = False,
) -> str:
"""生成HTML报告"""
- if is_daily:
- filename = "当日统计.html"
+ if is_daily_summary:
+ if mode == "current":
+ filename = "当前榜单汇总.html"
+ elif mode == "incremental":
+ filename = "当日增量.html"
+ else:
+ filename = "当日汇总.html"
else:
filename = f"{TimeHelper.format_time_filename()}.html"
file_path = FileHelper.get_output_path("html", filename)
- # 数据处理层
report_data = ReportGenerator._prepare_report_data(
- stats, failed_ids, new_titles, id_to_alias
+ stats, failed_ids, new_titles, id_to_name, mode
)
- # 渲染层
html_content = ReportGenerator._render_html_content(
- report_data, total_titles, is_daily
+ report_data, total_titles, is_daily_summary, mode
)
with open(file_path, "w", encoding="utf-8") as f:
f.write(html_content)
- if is_daily:
+ if is_daily_summary:
root_file_path = Path("index.html")
with open(root_file_path, "w", encoding="utf-8") as f:
f.write(html_content)
@@ -1134,16 +1251,19 @@ class ReportGenerator:
stats: List[Dict],
failed_ids: Optional[List] = None,
new_titles: Optional[Dict] = None,
- id_to_alias: Optional[Dict] = None,
- hide_new_section: bool = False,
+ id_to_name: Optional[Dict] = None,
+ mode: str = "daily",
) -> Dict:
"""准备报告数据"""
processed_new_titles = []
+ # 在增量模式下隐藏新增新闻区域
+ hide_new_section = mode == "incremental"
+
# 只有在非隐藏模式下才处理新增新闻部分
if not hide_new_section:
filtered_new_titles = {}
- if new_titles and id_to_alias:
+ if new_titles and id_to_name:
word_groups, filter_words = DataProcessor.load_frequency_words()
for source_id, titles_data in new_titles.items():
filtered_titles = ReportGenerator._apply_frequency_filter(
@@ -1152,9 +1272,9 @@ class ReportGenerator:
if filtered_titles:
filtered_new_titles[source_id] = filtered_titles
- if filtered_new_titles and id_to_alias:
+ if filtered_new_titles and id_to_name:
for source_id, titles_data in filtered_new_titles.items():
- source_alias = id_to_alias.get(source_id, source_id)
+ source_name = id_to_name.get(source_id, source_id)
source_titles = []
for title, title_data in titles_data.items():
@@ -1164,7 +1284,7 @@ class ReportGenerator:
processed_title = {
"title": title,
- "source_alias": source_alias,
+ "source_name": source_name,
"time_display": "",
"count": 1,
"ranks": ranks,
@@ -1179,7 +1299,7 @@ class ReportGenerator:
processed_new_titles.append(
{
"source_id": source_id,
- "source_alias": source_alias,
+ "source_name": source_name,
"titles": source_titles,
}
)
@@ -1193,7 +1313,7 @@ class ReportGenerator:
for title_data in stat["titles"]:
processed_title = {
"title": title_data["title"],
- "source_alias": title_data["source_alias"],
+ "source_name": title_data["source_name"],
"time_display": title_data["time_display"],
"count": title_data["count"],
"ranks": title_data["ranks"],
@@ -1239,7 +1359,7 @@ class ReportGenerator:
filtered_titles = {}
for title, title_data in titles_data.items():
- if StatisticsCalculator._matches_word_groups(
+ if StatisticsCalculator.matches_word_groups(
title, word_groups, filter_words
):
filtered_titles[title] = title_data
@@ -1263,7 +1383,7 @@ class ReportGenerator:
@staticmethod
def _format_title_html(title_data: Dict) -> str:
"""格式化HTML标题显示"""
- rank_display = StatisticsCalculator._format_rank_for_html(
+ rank_display = StatisticsCalculator.format_rank_for_html(
title_data["ranks"], title_data["rank_threshold"]
)
@@ -1271,14 +1391,14 @@ class ReportGenerator:
cleaned_title = DataProcessor.clean_title(title_data["title"])
escaped_title = ReportGenerator._html_escape(cleaned_title)
- escaped_source_alias = ReportGenerator._html_escape(title_data["source_alias"])
+ escaped_source_name = ReportGenerator._html_escape(title_data["source_name"])
if link_url:
escaped_url = ReportGenerator._html_escape(link_url)
- formatted_title = f'[{escaped_source_alias}] <a href="{escaped_url}" target="_blank" class="news-link">{escaped_title}</a>'
+ formatted_title = f'[{escaped_source_name}] <a href="{escaped_url}" target="_blank" class="news-link">{escaped_title}</a>'
else:
formatted_title = (
- f'[{escaped_source_alias}] <span class="no-link">{escaped_title}</span>'
+ f'[{escaped_source_name}] <span class="no-link">{escaped_title}</span>'
)
if rank_display:
@@ -1296,7 +1416,10 @@ class ReportGenerator:
@staticmethod
def _render_html_content(
- report_data: Dict, total_titles: int, is_daily: bool = False
+ report_data: Dict,
+ total_titles: int,
+ is_daily_summary: bool = False,
+ mode: str = "daily",
) -> str:
"""渲染HTML内容"""
html = """
@@ -1357,14 +1480,20 @@ class ReportGenerator:
<h1>频率词统计报告</h1>
"""
- if is_daily:
- html += "<p>报告类型: 当日汇总</p>"
+ if is_daily_summary:
+ if mode == "current":
+ html += "<p>报告类型: 当前榜单模式</p>"
+ elif mode == "incremental":
+ html += "<p>报告类型: 增量模式</p>"
+ else:
+ html += "<p>报告类型: 当日汇总</p>"
+ else:
+ html += "<p>报告类型: 实时分析</p>"
now = TimeHelper.get_beijing_time()
html += f"<p>总标题数: {total_titles}</p>"
html += f"<p>生成时间: {now.strftime('%Y-%m-%d %H:%M:%S')}</p>"
- # 渲染失败平台
if report_data["failed_ids"]:
html += """
<div class="error">
@@ -1378,7 +1507,6 @@ class ReportGenerator:
</div>
"""
- # 渲染统计表格
html += """
<table>
<tr>
@@ -1412,7 +1540,6 @@ class ReportGenerator:
</table>
"""
- # 渲染新增新闻部分
if report_data["new_titles"]:
html += f"""
<div class="new-section">
@@ -1421,7 +1548,7 @@ class ReportGenerator:
for source_data in report_data["new_titles"]:
escaped_source = ReportGenerator._html_escape(
- source_data["source_alias"]
+ source_data["source_name"]
)
html += (
f"<h4>{escaped_source} ({len(source_data['titles'])} 条)</h4><ul>"
@@ -1433,7 +1560,6 @@ class ReportGenerator:
formatted_title = ReportGenerator._format_title_html(
title_data_copy
)
- # 移除来源标签
if "] " in formatted_title:
formatted_title = formatted_title.split("] ", 1)[1]
html += f"<li>{formatted_title}</li>"
@@ -1452,7 +1578,7 @@ class ReportGenerator:
@staticmethod
def _format_title_feishu(title_data: Dict, show_source: bool = True) -> str:
"""格式化飞书标题显示"""
- rank_display = StatisticsCalculator._format_rank_for_feishu(
+ rank_display = StatisticsCalculator.format_rank_for_feishu(
title_data["ranks"], title_data["rank_threshold"]
)
@@ -1468,7 +1594,7 @@ class ReportGenerator:
title_prefix = "🆕 " if title_data["is_new"] else ""
if show_source:
- result = f"<font color='grey'>[{title_data['source_alias']}]</font> {title_prefix}{formatted_title}"
+ result = f"<font color='grey'>[{title_data['source_name']}]</font> {title_prefix}{formatted_title}"
else:
result = f"{title_prefix}{formatted_title}"
@@ -1484,7 +1610,7 @@ class ReportGenerator:
@staticmethod
def _format_title_dingtalk(title_data: Dict, show_source: bool = True) -> str:
"""格式化钉钉标题显示"""
- rank_display = StatisticsCalculator._format_rank_for_dingtalk(
+ rank_display = StatisticsCalculator.format_rank_for_dingtalk(
title_data["ranks"], title_data["rank_threshold"]
)
@@ -1500,7 +1626,7 @@ class ReportGenerator:
title_prefix = "🆕 " if title_data["is_new"] else ""
if show_source:
- result = f"[{title_data['source_alias']}] {title_prefix}{formatted_title}"
+ result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}"
else:
result = f"{title_prefix}{formatted_title}"
@@ -1516,7 +1642,7 @@ class ReportGenerator:
@staticmethod
def _format_title_wework(title_data: Dict, show_source: bool = True) -> str:
"""格式化企业微信标题显示"""
- rank_display = StatisticsCalculator._format_rank_for_wework(
+ rank_display = StatisticsCalculator.format_rank_for_wework(
title_data["ranks"], title_data["rank_threshold"]
)
@@ -1532,7 +1658,7 @@ class ReportGenerator:
title_prefix = "🆕 " if title_data["is_new"] else ""
if show_source:
- result = f"[{title_data['source_alias']}] {title_prefix}{formatted_title}"
+ result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}"
else:
result = f"{title_prefix}{formatted_title}"
@@ -1548,7 +1674,7 @@ class ReportGenerator:
@staticmethod
def _format_title_telegram(title_data: Dict, show_source: bool = True) -> str:
"""格式化Telegram标题显示"""
- rank_display = StatisticsCalculator._format_rank_for_telegram(
+ rank_display = StatisticsCalculator.format_rank_for_telegram(
title_data["ranks"], title_data["rank_threshold"]
)
@@ -1564,7 +1690,7 @@ class ReportGenerator:
title_prefix = "🆕 " if title_data["is_new"] else ""
if show_source:
- result = f"[{title_data['source_alias']}] {title_prefix}{formatted_title}"
+ result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}"
else:
result = f"{title_prefix}{formatted_title}"
@@ -1579,14 +1705,13 @@ class ReportGenerator:
@staticmethod
def _render_feishu_content(
- report_data: Dict, update_info: Optional[Dict] = None
+ report_data: Dict, update_info: Optional[Dict] = None, mode: str = "daily"
) -> str:
"""渲染飞书内容"""
text_content = ""
- # 渲染热点词汇统计
if report_data["stats"]:
- text_content += "📊 **热点词汇统计**\n\n"
+ text_content += f"📊 **热点词汇统计**\n\n"
total_count = len(report_data["stats"])
@@ -1616,9 +1741,14 @@ class ReportGenerator:
text_content += f"\n{CONFIG['FEISHU_MESSAGE_SEPARATOR']}\n\n"
if not text_content:
- text_content = "📭 暂无匹配的热点词汇\n\n"
+ if mode == "incremental":
+ mode_text = "增量模式下暂无新增匹配的热点词汇"
+ elif mode == "current":
+ mode_text = "当前榜单模式下暂无匹配的热点词汇"
+ else:
+ mode_text = "暂无匹配的热点词汇"
+ text_content = f"📭 {mode_text}\n\n"
- # 渲染新增新闻部分
if report_data["new_titles"]:
if text_content and "暂无匹配" not in text_content:
text_content += f"\n{CONFIG['FEISHU_MESSAGE_SEPARATOR']}\n\n"
@@ -1628,7 +1758,7 @@ class ReportGenerator:
)
for source_data in report_data["new_titles"]:
- text_content += f"**{source_data['source_alias']}** ({len(source_data['titles'])} 条):\n"
+ text_content += f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n"
for j, title_data in enumerate(source_data["titles"], 1):
title_data_copy = title_data.copy()
@@ -1640,7 +1770,6 @@ class ReportGenerator:
text_content += "\n"
- # 渲染失败平台
if report_data["failed_ids"]:
if text_content and "暂无匹配" not in text_content:
text_content += f"\n{CONFIG['FEISHU_MESSAGE_SEPARATOR']}\n\n"
@@ -1649,11 +1778,9 @@ class ReportGenerator:
for i, id_value in enumerate(report_data["failed_ids"], 1):
text_content += f" • <font color='red'>{id_value}</font>\n"
- # 添加时间戳
now = TimeHelper.get_beijing_time()
text_content += f"\n\n<font color='grey'>更新时间:{now.strftime('%Y-%m-%d %H:%M:%S')}</font>"
- # 版本更新提示
if update_info:
text_content += f"\n<font color='grey'>TrendRadar 发现新版本 {update_info['remote_version']},当前 {update_info['current_version']}</font>"
@@ -1661,27 +1788,24 @@ class ReportGenerator:
@staticmethod
def _render_dingtalk_content(
- report_data: Dict, update_info: Optional[Dict] = None
+ report_data: Dict, update_info: Optional[Dict] = None, mode: str = "daily"
) -> str:
"""渲染钉钉内容"""
text_content = ""
- # 计算总标题数
total_titles = sum(
len(stat["titles"]) for stat in report_data["stats"] if stat["count"] > 0
)
now = TimeHelper.get_beijing_time()
- # 顶部统计信息
text_content += f"**总新闻数:** {total_titles}\n\n"
text_content += f"**时间:** {now.strftime('%Y-%m-%d %H:%M:%S')}\n\n"
text_content += f"**类型:** 热点分析报告\n\n"
text_content += "---\n\n"
- # 渲染热点词汇统计
if report_data["stats"]:
- text_content += "📊 **热点词汇统计**\n\n"
+ text_content += f"📊 **热点词汇统计**\n\n"
total_count = len(report_data["stats"])
@@ -1715,9 +1839,14 @@ class ReportGenerator:
text_content += f"\n---\n\n"
if not report_data["stats"]:
- text_content += "📭 暂无匹配的热点词汇\n\n"
+ if mode == "incremental":
+ mode_text = "增量模式下暂无新增匹配的热点词汇"
+ elif mode == "current":
+ mode_text = "当前榜单模式下暂无匹配的热点词汇"
+ else:
+ mode_text = "暂无匹配的热点词汇"
+ text_content += f"📭 {mode_text}\n\n"
- # 渲染新增新闻部分
if report_data["new_titles"]:
if text_content and "暂无匹配" not in text_content:
text_content += f"\n---\n\n"
@@ -1727,7 +1856,7 @@ class ReportGenerator:
)
for source_data in report_data["new_titles"]:
- text_content += f"**{source_data['source_alias']}** ({len(source_data['titles'])} 条):\n\n"
+ text_content += f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n\n"
for j, title_data in enumerate(source_data["titles"], 1):
title_data_copy = title_data.copy()
@@ -1739,7 +1868,6 @@ class ReportGenerator:
text_content += "\n"
- # 渲染失败平台
if report_data["failed_ids"]:
if text_content and "暂无匹配" not in text_content:
text_content += f"\n---\n\n"
@@ -1748,10 +1876,8 @@ class ReportGenerator:
for i, id_value in enumerate(report_data["failed_ids"], 1):
text_content += f" • **{id_value}**\n"
- # 添加时间戳
text_content += f"\n\n> 更新时间:{now.strftime('%Y-%m-%d %H:%M:%S')}"
- # 版本更新提示
if update_info:
text_content += f"\n> TrendRadar 发现新版本 **{update_info['remote_version']}**,当前 **{update_info['current_version']}**"
@@ -1763,11 +1889,11 @@ class ReportGenerator:
format_type: str,
update_info: Optional[Dict] = None,
max_bytes: int = CONFIG["MESSAGE_BATCH_SIZE"],
+ mode: str = "daily",
) -> List[str]:
"""分批处理消息内容,确保词组标题+至少第一条新闻的完整性"""
batches = []
- # 基础信息构建
total_titles = sum(
len(stat["titles"]) for stat in report_data["stats"] if stat["count"] > 0
)
@@ -1792,20 +1918,25 @@ class ReportGenerator:
stats_header = ""
if report_data["stats"]:
if format_type == "wework":
- stats_header = "📊 **热点词汇统计**\n\n"
+ stats_header = f"📊 **热点词汇统计**\n\n"
elif format_type == "telegram":
- stats_header = "📊 热点词汇统计\n\n"
+ stats_header = f"📊 热点词汇统计\n\n"
current_batch = base_header
current_batch_has_content = False
- # 空内容处理
if (
not report_data["stats"]
and not report_data["new_titles"]
and not report_data["failed_ids"]
):
- simple_content = "📭 暂无匹配的热点词汇\n\n"
+ if mode == "incremental":
+ mode_text = "增量模式下暂无新增匹配的热点词汇"
+ elif mode == "current":
+ mode_text = "当前榜单模式下暂无匹配的热点词汇"
+ else:
+ mode_text = "暂无匹配的热点词汇"
+ simple_content = f"📭 {mode_text}\n\n"
final_content = base_header + simple_content + base_footer
batches.append(final_content)
return batches
@@ -1970,9 +2101,9 @@ class ReportGenerator:
for source_data in report_data["new_titles"]:
source_header = ""
if format_type == "wework":
- source_header = f"**{source_data['source_alias']}** ({len(source_data['titles'])} 条):\n\n"
+ source_header = f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n\n"
elif format_type == "telegram":
- source_header = f"{source_data['source_alias']} ({len(source_data['titles'])} 条):\n\n"
+ source_header = f"{source_data['source_name']} ({len(source_data['titles'])} 条):\n\n"
# 构建第一条新增新闻
first_news_line = ""
@@ -2049,7 +2180,6 @@ class ReportGenerator:
current_batch += "\n"
- # 处理失败平台
if report_data["failed_ids"]:
failed_header = ""
if format_type == "wework":
@@ -2095,55 +2225,62 @@ class ReportGenerator:
def send_to_webhooks(
stats: List[Dict],
failed_ids: Optional[List] = None,
- report_type: str = "单次爬取",
+ report_type: str = "当日汇总",
new_titles: Optional[Dict] = None,
- id_to_alias: Optional[Dict] = None,
+ id_to_name: Optional[Dict] = None,
update_info: Optional[Dict] = None,
proxy_url: Optional[str] = None,
- hide_new_section: bool = False,
+ mode: str = "daily",
) -> Dict[str, bool]:
"""发送数据到多个webhook平台"""
results = {}
- # 数据处理层
report_data = ReportGenerator._prepare_report_data(
- stats, failed_ids, new_titles, id_to_alias, hide_new_section
+ stats, failed_ids, new_titles, id_to_name, mode
)
- # 获取环境变量中的webhook配置
- feishu_url = os.environ.get("FEISHU_WEBHOOK_URL", CONFIG["FEISHU_WEBHOOK_URL"])
- dingtalk_url = os.environ.get(
- "DINGTALK_WEBHOOK_URL", CONFIG["DINGTALK_WEBHOOK_URL"]
- )
- wework_url = os.environ.get("WEWORK_WEBHOOK_URL", CONFIG["WEWORK_WEBHOOK_URL"])
- telegram_token = os.environ.get(
- "TELEGRAM_BOT_TOKEN", CONFIG["TELEGRAM_BOT_TOKEN"]
- )
- telegram_chat_id = os.environ.get(
- "TELEGRAM_CHAT_ID", CONFIG["TELEGRAM_CHAT_ID"]
- )
+ feishu_url = CONFIG["FEISHU_WEBHOOK_URL"]
+ dingtalk_url = CONFIG["DINGTALK_WEBHOOK_URL"]
+ wework_url = CONFIG["WEWORK_WEBHOOK_URL"]
+ telegram_token = CONFIG["TELEGRAM_BOT_TOKEN"]
+ telegram_chat_id = CONFIG["TELEGRAM_CHAT_ID"]
update_info_to_send = update_info if CONFIG["SHOW_VERSION_UPDATE"] else None
# 发送到飞书
if feishu_url:
results["feishu"] = ReportGenerator._send_to_feishu(
- feishu_url, report_data, report_type, update_info_to_send, proxy_url
+ feishu_url,
+ report_data,
+ report_type,
+ update_info_to_send,
+ proxy_url,
+ mode,
)
# 发送到钉钉
if dingtalk_url:
results["dingtalk"] = ReportGenerator._send_to_dingtalk(
- dingtalk_url, report_data, report_type, update_info_to_send, proxy_url
+ dingtalk_url,
+ report_data,
+ report_type,
+ update_info_to_send,
+ proxy_url,
+ mode,
)
# 发送到企业微信
if wework_url:
results["wework"] = ReportGenerator._send_to_wework(
- wework_url, report_data, report_type, update_info_to_send, proxy_url
+ wework_url,
+ report_data,
+ report_type,
+ update_info_to_send,
+ proxy_url,
+ mode,
)
- # 发送到Telegram
+ # 发送到 Telegram
if telegram_token and telegram_chat_id:
results["telegram"] = ReportGenerator._send_to_telegram(
telegram_token,
@@ -2152,6 +2289,7 @@ class ReportGenerator:
report_type,
update_info_to_send,
proxy_url,
+ mode,
)
if not results:
@@ -2166,11 +2304,14 @@ class ReportGenerator:
report_type: str,
update_info: Optional[Dict] = None,
proxy_url: Optional[str] = None,
+ mode: str = "daily",
) -> bool:
"""发送到飞书"""
headers = {"Content-Type": "application/json"}
- text_content = ReportGenerator._render_feishu_content(report_data, update_info)
+ text_content = ReportGenerator._render_feishu_content(
+ report_data, update_info, mode
+ )
total_titles = sum(
len(stat["titles"]) for stat in report_data["stats"] if stat["count"] > 0
)
@@ -2213,12 +2354,13 @@ class ReportGenerator:
report_type: str,
update_info: Optional[Dict] = None,
proxy_url: Optional[str] = None,
+ mode: str = "daily",
) -> bool:
"""发送到钉钉"""
headers = {"Content-Type": "application/json"}
text_content = ReportGenerator._render_dingtalk_content(
- report_data, update_info
+ report_data, update_info, mode
)
payload = {
@@ -2263,6 +2405,7 @@ class ReportGenerator:
report_type: str,
update_info: Optional[Dict] = None,
proxy_url: Optional[str] = None,
+ mode: str = "daily",
) -> bool:
"""发送到企业微信(支持分批发送)"""
headers = {"Content-Type": "application/json"}
@@ -2272,7 +2415,7 @@ class ReportGenerator:
# 获取分批内容
batches = ReportGenerator._split_content_into_batches(
- report_data, "wework", update_info
+ report_data, "wework", update_info, mode=mode
)
print(f"企业微信消息分为 {len(batches)} 批次发送 [{report_type}]")
@@ -2335,6 +2478,7 @@ class ReportGenerator:
report_type: str,
update_info: Optional[Dict] = None,
proxy_url: Optional[str] = None,
+ mode: str = "daily",
) -> bool:
"""发送到Telegram(支持分批发送)"""
headers = {"Content-Type": "application/json"}
@@ -2346,7 +2490,7 @@ class ReportGenerator:
# 获取分批内容
batches = ReportGenerator._split_content_into_batches(
- report_data, "telegram", update_info
+ report_data, "telegram", update_info, mode=mode
)
print(f"Telegram消息分为 {len(batches)} 批次发送 [{report_type}]")
@@ -2403,21 +2547,98 @@ class ReportGenerator:
return True
+@dataclass
+class ModeStrategy:
+ """模式策略配置"""
+
+ mode_name: str
+ description: str
+ realtime_report_type: str
+ summary_report_type: str
+ should_send_realtime: bool
+ should_generate_summary: bool
+ summary_mode: str
+
+ def get_log_messages(self) -> Dict[str, str]:
+ """返回该模式的各种日志消息"""
+ return {
+ "mode_description": self.description,
+ "realtime_skip": f"跳过实时推送通知:{self.mode_name}下未检测到匹配的新闻",
+ "summary_skip": f"跳过{self.summary_report_type}通知:未匹配到有效的新闻内容",
+ }
+
+
class NewsAnalyzer:
"""新闻分析器"""
+ MODE_STRATEGIES = {
+ "incremental": ModeStrategy(
+ mode_name="增量模式",
+ description="增量模式(只关注新增新闻,无新增时不推送)",
+ realtime_report_type="实时增量",
+ summary_report_type="当日汇总",
+ should_send_realtime=True,
+ should_generate_summary=True,
+ summary_mode="daily",
+ ),
+ "current": ModeStrategy(
+ mode_name="当前榜单模式",
+ description="当前榜单模式(当前榜单匹配新闻 + 新增新闻区域 + 按时推送)",
+ realtime_report_type="实时当前榜单",
+ summary_report_type="当前榜单汇总",
+ should_send_realtime=True,
+ should_generate_summary=True,
+ summary_mode="current",
+ ),
+ "daily": ModeStrategy(
+ mode_name="当日汇总模式",
+ description="当日汇总模式(所有匹配新闻 + 新增新闻区域 + 按时推送)",
+ realtime_report_type="",
+ summary_report_type="当日汇总",
+ should_send_realtime=False,
+ should_generate_summary=True,
+ summary_mode="daily",
+ ),
+ }
+
def __init__(
self,
request_interval: int = CONFIG["REQUEST_INTERVAL"],
- report_type: str = CONFIG["REPORT_TYPE"],
+ report_mode: str = CONFIG["REPORT_MODE"],
rank_threshold: int = CONFIG["RANK_THRESHOLD"],
):
self.request_interval = request_interval
- self.report_type = report_type
+ self.report_mode = report_mode
self.rank_threshold = rank_threshold
self.is_github_actions = os.environ.get("GITHUB_ACTIONS") == "true"
+ self.is_docker_container = self._detect_docker_environment()
self.update_info = None
self.proxy_url = None
+ self._setup_proxy()
+ self.data_fetcher = DataFetcher(self.proxy_url)
+
+ if self.is_github_actions:
+ self._check_version_update()
+
+ def _detect_docker_environment(self) -> bool:
+ """检测是否运行在 Docker 容器中"""
+ try:
+ if os.environ.get("DOCKER_CONTAINER") == "true":
+ return True
+
+ if os.path.exists("/.dockerenv"):
+ return True
+
+ return False
+ except Exception:
+ return False
+
+ def _should_open_browser(self) -> bool:
+ """判断是否应该打开浏览器"""
+ return not self.is_github_actions and not self.is_docker_container
+
+ def _setup_proxy(self) -> None:
+ """设置代理配置"""
if not self.is_github_actions and CONFIG["USE_PROXY"]:
self.proxy_url = CONFIG["DEFAULT_PROXY"]
print("本地环境,使用代理")
@@ -2426,11 +2647,6 @@ class NewsAnalyzer:
else:
print("GitHub Actions环境,不使用代理")
- self.data_fetcher = DataFetcher(self.proxy_url)
-
- if self.is_github_actions:
- self._check_version_update()
-
def _check_version_update(self) -> None:
"""检查版本更新"""
try:
@@ -2449,106 +2665,251 @@ class NewsAnalyzer:
except Exception as e:
print(f"版本检查出错: {e}")
+ def _get_mode_strategy(self) -> ModeStrategy:
+ """获取当前模式的策略配置"""
+ return self.MODE_STRATEGIES.get(self.report_mode, self.MODE_STRATEGIES["daily"])
+
+ def _has_webhook_configured(self) -> bool:
+ """检查是否配置了webhook"""
+ return any(
+ [
+ CONFIG["FEISHU_WEBHOOK_URL"],
+ CONFIG["DINGTALK_WEBHOOK_URL"],
+ CONFIG["WEWORK_WEBHOOK_URL"],
+ (CONFIG["TELEGRAM_BOT_TOKEN"] and CONFIG["TELEGRAM_CHAT_ID"]),
+ ]
+ )
+
def _has_valid_content(
self, stats: List[Dict], new_titles: Optional[Dict] = None
) -> bool:
"""检查是否有有效的新闻内容"""
- if CONFIG["FOCUS_NEW_ONLY"]:
- # 新增模式下,只要stats有内容就说明有新增的匹配新闻
+ if self.report_mode in ["incremental", "current"]:
+ # 增量模式和current模式下,只要stats有内容就说明有匹配的新闻
return any(stat["count"] > 0 for stat in stats)
else:
- # 正常模式下,检查是否有匹配的频率词新闻或新增新闻
+ # 当日汇总模式下,检查是否有匹配的频率词新闻或新增新闻
has_matched_news = any(stat["count"] > 0 for stat in stats)
has_new_news = bool(
new_titles and any(len(titles) > 0 for titles in new_titles.values())
)
return has_matched_news or has_new_news
- def generate_daily_summary(self) -> Optional[str]:
- """生成当日统计报告"""
- print("生成当日统计报告...")
+ def _load_analysis_data(
+ self,
+ ) -> Optional[Tuple[Dict, Dict, Dict, Dict, List, List]]:
+ """统一的数据加载和预处理,使用当前监控平台列表过滤历史数据"""
+ try:
+ # 获取当前配置的监控平台ID列表
+ current_platform_ids = []
+ for platform in PLATFORMS:
+ current_platform_ids.append(platform["id"])
- all_results, id_to_alias, title_info = DataProcessor.read_all_today_titles()
+ print(f"当前监控平台: {current_platform_ids}")
- if not all_results:
- print("没有找到当天的数据")
+ all_results, id_to_name, title_info = DataProcessor.read_all_today_titles(
+ current_platform_ids
+ )
+
+ if not all_results:
+ print("没有找到当天的数据")
+ return None
+
+ total_titles = sum(len(titles) for titles in all_results.values())
+ print(f"读取到 {total_titles} 个标题(已按当前监控平台过滤)")
+
+ new_titles = DataProcessor.detect_latest_new_titles(current_platform_ids)
+ word_groups, filter_words = DataProcessor.load_frequency_words()
+
+ return (
+ all_results,
+ id_to_name,
+ title_info,
+ new_titles,
+ word_groups,
+ filter_words,
+ )
+ except Exception as e:
+ print(f"数据加载失败: {e}")
return None
- total_titles = sum(len(titles) for titles in all_results.values())
- print(f"读取到 {total_titles} 个标题")
+ def _prepare_current_title_info(self, results: Dict, time_info: str) -> Dict:
+ """从当前抓取结果构建标题信息"""
+ title_info = {}
+ for source_id, titles_data in results.items():
+ title_info[source_id] = {}
+ for title, title_data in titles_data.items():
+ ranks = title_data.get("ranks", [])
+ url = title_data.get("url", "")
+ mobile_url = title_data.get("mobileUrl", "")
- new_titles = DataProcessor.detect_latest_new_titles(id_to_alias)
- word_groups, filter_words = DataProcessor.load_frequency_words()
+ title_info[source_id][title] = {
+ "first_time": time_info,
+ "last_time": time_info,
+ "count": 1,
+ "ranks": ranks,
+ "url": url,
+ "mobileUrl": mobile_url,
+ }
+ return title_info
+
+ def _run_analysis_pipeline(
+ self,
+ data_source: Dict,
+ mode: str,
+ title_info: Dict,
+ new_titles: Dict,
+ word_groups: List[Dict],
+ filter_words: List[str],
+ id_to_name: Dict,
+ failed_ids: Optional[List] = None,
+ is_daily_summary: bool = False,
+ ) -> Tuple[List[Dict], str]:
+ """统一的分析流水线:数据处理 → 统计计算 → HTML生成"""
+ # 统计计算
stats, total_titles = StatisticsCalculator.count_word_frequency(
- all_results,
+ data_source,
word_groups,
filter_words,
- id_to_alias,
+ id_to_name,
title_info,
self.rank_threshold,
new_titles,
- focus_new_only=CONFIG["FOCUS_NEW_ONLY"],
+ mode=mode,
)
+ # HTML生成
html_file = ReportGenerator.generate_html_report(
stats,
total_titles,
- is_daily=True,
+ failed_ids=failed_ids,
new_titles=new_titles,
- id_to_alias=id_to_alias,
+ id_to_name=id_to_name,
+ mode=mode,
+ is_daily_summary=is_daily_summary,
)
- print(f"当日HTML统计报告已生成: {html_file}")
- # 检查通知配置
- has_webhook = any(
- [
- os.environ.get("FEISHU_WEBHOOK_URL", CONFIG["FEISHU_WEBHOOK_URL"]),
- os.environ.get("DINGTALK_WEBHOOK_URL", CONFIG["DINGTALK_WEBHOOK_URL"]),
- os.environ.get("WEWORK_WEBHOOK_URL", CONFIG["WEWORK_WEBHOOK_URL"]),
- (
- os.environ.get("TELEGRAM_BOT_TOKEN", CONFIG["TELEGRAM_BOT_TOKEN"])
- and os.environ.get("TELEGRAM_CHAT_ID", CONFIG["TELEGRAM_CHAT_ID"])
- ),
- ]
- )
+ return stats, html_file
+
+ def _send_notification_if_needed(
+ self,
+ stats: List[Dict],
+ report_type: str,
+ mode: str,
+ failed_ids: Optional[List] = None,
+ new_titles: Optional[Dict] = None,
+ id_to_name: Optional[Dict] = None,
+ ) -> bool:
+ """统一的通知发送逻辑,包含所有判断条件"""
+ has_webhook = self._has_webhook_configured()
if (
CONFIG["ENABLE_NOTIFICATION"]
and has_webhook
- and self.report_type in ["daily", "both"]
and self._has_valid_content(stats, new_titles)
):
- hide_new_section = CONFIG["FOCUS_NEW_ONLY"]
-
ReportGenerator.send_to_webhooks(
stats,
- [],
- "当日汇总",
+ failed_ids or [],
+ report_type,
new_titles,
- id_to_alias,
+ id_to_name,
self.update_info,
self.proxy_url,
- hide_new_section=hide_new_section,
+ mode=mode,
)
+ return True
elif CONFIG["ENABLE_NOTIFICATION"] and not has_webhook:
print("⚠️ 警告:通知功能已启用但未配置webhook URL,将跳过通知发送")
elif not CONFIG["ENABLE_NOTIFICATION"]:
- print("跳过当日汇总通知:通知功能已禁用")
+ print(f"跳过{report_type}通知:通知功能已禁用")
elif (
CONFIG["ENABLE_NOTIFICATION"]
and has_webhook
and not self._has_valid_content(stats, new_titles)
):
- if CONFIG["FOCUS_NEW_ONLY"]:
- print("跳过当日汇总通知:新增模式下未检测到匹配的新增新闻")
+ mode_strategy = self._get_mode_strategy()
+ log_messages = mode_strategy.get_log_messages()
+ if "实时" in report_type:
+ print(log_messages["realtime_skip"])
else:
- print("跳过当日汇总通知:未匹配到有效的新闻内容")
+ print(log_messages["summary_skip"])
+
+ return False
+
+ def _generate_summary_report(self, mode_strategy: ModeStrategy) -> Optional[str]:
+ """生成汇总报告(带通知)"""
+ summary_type = (
+ "当前榜单汇总" if mode_strategy.summary_mode == "current" else "当日汇总"
+ )
+ print(f"生成{summary_type}报告...")
+
+ # 加载分析数据
+ analysis_data = self._load_analysis_data()
+ if not analysis_data:
+ return None
+
+ all_results, id_to_name, title_info, new_titles, word_groups, filter_words = (
+ analysis_data
+ )
+
+ # 运行分析流水线
+ stats, html_file = self._run_analysis_pipeline(
+ all_results,
+ mode_strategy.summary_mode,
+ title_info,
+ new_titles,
+ word_groups,
+ filter_words,
+ id_to_name,
+ is_daily_summary=True,
+ )
+
+ print(f"{summary_type}报告已生成: {html_file}")
+
+ # 发送通知
+ self._send_notification_if_needed(
+ stats,
+ mode_strategy.summary_report_type,
+ mode_strategy.summary_mode,
+ new_titles=new_titles,
+ id_to_name=id_to_name,
+ )
return html_file
- def run(self) -> None:
- """执行分析流程"""
+ def _generate_summary_html(self, mode: str = "daily") -> Optional[str]:
+ """生成汇总HTML"""
+ summary_type = "当前榜单汇总" if mode == "current" else "当日汇总"
+ print(f"生成{summary_type}HTML...")
+
+ # 加载分析数据
+ analysis_data = self._load_analysis_data()
+ if not analysis_data:
+ return None
+
+ all_results, id_to_name, title_info, new_titles, word_groups, filter_words = (
+ analysis_data
+ )
+
+ # 运行分析流水线
+ _, html_file = self._run_analysis_pipeline(
+ all_results,
+ mode,
+ title_info,
+ new_titles,
+ word_groups,
+ filter_words,
+ id_to_name,
+ is_daily_summary=True,
+ )
+
+ print(f"{summary_type}HTML已生成: {html_file}")
+ return html_file
+
+ def _initialize_and_check_config(self) -> None:
+ """通用初始化和配置检查"""
now = TimeHelper.get_beijing_time()
print(f"当前北京时间: {now.strftime('%Y-%m-%d %H:%M:%S')}")
@@ -2556,20 +2917,7 @@ class NewsAnalyzer:
print("爬虫功能已禁用(ENABLE_CRAWLER=False),程序退出")
return
- # 检查是否配置了任何webhook URL
- has_webhook = any(
- [
- os.environ.get("FEISHU_WEBHOOK_URL", CONFIG["FEISHU_WEBHOOK_URL"]),
- os.environ.get("DINGTALK_WEBHOOK_URL", CONFIG["DINGTALK_WEBHOOK_URL"]),
- os.environ.get("WEWORK_WEBHOOK_URL", CONFIG["WEWORK_WEBHOOK_URL"]),
- (
- os.environ.get("TELEGRAM_BOT_TOKEN", CONFIG["TELEGRAM_BOT_TOKEN"])
- and os.environ.get("TELEGRAM_CHAT_ID", CONFIG["TELEGRAM_CHAT_ID"])
- ),
- ]
- )
-
- # 通知功能状态检查和提示
+ has_webhook = self._has_webhook_configured()
if not CONFIG["ENABLE_NOTIFICATION"]:
print("通知功能已禁用(ENABLE_NOTIFICATION=False),将只进行数据抓取")
elif not has_webhook:
@@ -2577,129 +2925,183 @@ class NewsAnalyzer:
else:
print("通知功能已启用,将发送webhook通知")
- print(f"报告类型: {self.report_type}")
+ mode_strategy = self._get_mode_strategy()
+ print(f"报告模式: {self.report_mode}")
+ print(f"运行模式: {mode_strategy.description}")
- if CONFIG["FOCUS_NEW_ONLY"]:
- print("运行模式: 新增检测模式(只关注新增新闻)")
- else:
- print("运行模式: 正常模式(频率词统计 + 新增检测)")
-
- ids = [
- ("toutiao", "今日头条"),
- ("baidu", "百度热搜"),
- ("wallstreetcn-hot", "华尔街见闻"),
- ("thepaper", "澎湃新闻"),
- ("bilibili-hot-search", "bilibili 热搜"),
- ("cls-hot", "财联社热门"),
- ("ifeng", "凤凰网"),
- "tieba",
- "weibo",
- "douyin",
- "zhihu",
- ]
+ def _crawl_data(self) -> Tuple[Dict, Dict, List]:
+ """执行数据爬取"""
+ ids = []
+ for platform in PLATFORMS:
+ if "name" in platform:
+ ids.append((platform["id"], platform["name"]))
+ else:
+ ids.append(platform["id"])
+ print(f"配置的监控平台: {[p.get('name', p['id']) for p in PLATFORMS]}")
print(f"开始爬取数据,请求间隔 {self.request_interval} 毫秒")
FileHelper.ensure_directory_exists("output")
- results, id_to_alias, failed_ids = self.data_fetcher.crawl_websites(
+ results, id_to_name, failed_ids = self.data_fetcher.crawl_websites(
ids, self.request_interval
)
- title_file = DataProcessor.save_titles_to_file(results, id_to_alias, failed_ids)
+ title_file = DataProcessor.save_titles_to_file(results, id_to_name, failed_ids)
print(f"标题已保存到: {title_file}")
- new_titles = DataProcessor.detect_latest_new_titles(id_to_alias)
+ return results, id_to_name, failed_ids
- # 构建标题信息
- time_info = Path(title_file).stem
- title_info = {}
- for source_id, titles_data in results.items():
- title_info[source_id] = {}
- for title, title_data in titles_data.items():
- ranks = title_data.get("ranks", [])
- url = title_data.get("url", "")
- mobile_url = title_data.get("mobileUrl", "")
+ def _execute_mode_strategy(
+ self,
+ mode_strategy: ModeStrategy,
+ results: Dict,
+ id_to_name: Dict,
+ failed_ids: List,
+ ) -> Optional[str]:
+ """执行模式特定逻辑"""
+ # 获取当前监控平台ID列表
+ current_platform_ids = [platform["id"] for platform in PLATFORMS]
+
+ new_titles = DataProcessor.detect_latest_new_titles(current_platform_ids)
+ time_info = Path(
+ DataProcessor.save_titles_to_file(results, id_to_name, failed_ids)
+ ).stem
+ word_groups, filter_words = DataProcessor.load_frequency_words()
- title_info[source_id][title] = {
- "first_time": time_info,
- "last_time": time_info,
- "count": 1,
- "ranks": ranks,
- "url": url,
- "mobileUrl": mobile_url,
- }
+ # current模式下,实时推送需要使用完整的历史数据来保证统计信息的完整性
+ if self.report_mode == "current":
+ # 加载完整的历史数据(已按当前平台过滤)
+ analysis_data = self._load_analysis_data()
+ if analysis_data:
+ (
+ all_results,
+ historical_id_to_name,
+ historical_title_info,
+ historical_new_titles,
+ _,
+ _,
+ ) = analysis_data
- word_groups, filter_words = DataProcessor.load_frequency_words()
+ print(
+ f"current模式:使用过滤后的历史数据,包含平台:{list(all_results.keys())}"
+ )
- stats, total_titles = StatisticsCalculator.count_word_frequency(
- results,
- word_groups,
- filter_words,
- id_to_alias,
- title_info,
- self.rank_threshold,
- new_titles,
- focus_new_only=CONFIG["FOCUS_NEW_ONLY"],
- )
+ stats, html_file = self._run_analysis_pipeline(
+ all_results,
+ self.report_mode,
+ historical_title_info,
+ historical_new_titles,
+ word_groups,
+ filter_words,
+ historical_id_to_name,
+ failed_ids=failed_ids,
+ )
- # 只有启用通知且配置了webhook且有有效内容时才发送通知
- if (
- CONFIG["ENABLE_NOTIFICATION"]
- and has_webhook
- and self.report_type in ["current", "both"]
- and self._has_valid_content(stats, new_titles)
- ):
- hide_new_section = CONFIG["FOCUS_NEW_ONLY"]
+ combined_id_to_name = {**historical_id_to_name, **id_to_name}
- ReportGenerator.send_to_webhooks(
- stats,
- failed_ids,
- "单次爬取",
+ print(f"HTML报告已生成: {html_file}")
+
+ # 发送实时通知(使用完整历史数据的统计结果)
+ summary_html = None
+ if mode_strategy.should_send_realtime:
+ self._send_notification_if_needed(
+ stats,
+ mode_strategy.realtime_report_type,
+ self.report_mode,
+ failed_ids=failed_ids,
+ new_titles=historical_new_titles,
+ id_to_name=combined_id_to_name,
+ )
+ else:
+ print("❌ 严重错误:无法读取刚保存的数据文件")
+ raise RuntimeError("数据一致性检查失败:保存后立即读取失败")
+ else:
+ title_info = self._prepare_current_title_info(results, time_info)
+ stats, html_file = self._run_analysis_pipeline(
+ results,
+ self.report_mode,
+ title_info,
new_titles,
- id_to_alias,
- self.update_info,
- self.proxy_url,
- hide_new_section=hide_new_section,
+ word_groups,
+ filter_words,
+ id_to_name,
+ failed_ids=failed_ids,
)
- elif CONFIG["ENABLE_NOTIFICATION"] and not has_webhook:
- print("⚠️ 警告:通知功能已启用但未配置webhook URL,将跳过通知发送")
- elif not CONFIG["ENABLE_NOTIFICATION"]:
- print("跳过单次爬取通知:通知功能已禁用")
- elif (
- CONFIG["ENABLE_NOTIFICATION"]
- and has_webhook
- and not self._has_valid_content(stats, new_titles)
- ):
- if CONFIG["FOCUS_NEW_ONLY"]:
- print("跳过单次爬取通知:新增模式下未检测到匹配的新增新闻")
+ print(f"HTML报告已生成: {html_file}")
+
+ # 发送实时通知(如果需要)
+ summary_html = None
+ if mode_strategy.should_send_realtime:
+ self._send_notification_if_needed(
+ stats,
+ mode_strategy.realtime_report_type,
+ self.report_mode,
+ failed_ids=failed_ids,
+ new_titles=new_titles,
+ id_to_name=id_to_name,
+ )
+
+ # 生成汇总报告(如果需要)
+ summary_html = None
+ if mode_strategy.should_generate_summary:
+ if mode_strategy.should_send_realtime:
+ # 如果已经发送了实时通知,汇总只生成HTML不发送通知
+ summary_html = self._generate_summary_html(mode_strategy.summary_mode)
+ else:
+ # daily模式:直接生成汇总报告并发送通知
+ summary_html = self._generate_summary_report(mode_strategy)
+
+ # 打开浏览器(仅在非容器环境)
+ if self._should_open_browser() and html_file:
+ if summary_html:
+ summary_url = "file://" + str(Path(summary_html).resolve())
+ print(f"正在打开汇总报告: {summary_url}")
+ webbrowser.open(summary_url)
+ else:
+ file_url = "file://" + str(Path(html_file).resolve())
+ print(f"正在打开HTML报告: {file_url}")
+ webbrowser.open(file_url)
+ elif self.is_docker_container and html_file:
+ if summary_html:
+ print(f"汇总报告已生成(Docker环境): {summary_html}")
else:
- print("跳过单次爬取通知:未匹配到有效的新闻内容")
+ print(f"HTML报告已生成(Docker环境): {html_file}")
- html_file = ReportGenerator.generate_html_report(
- stats, total_titles, failed_ids, False, new_titles, id_to_alias
- )
- print(f"HTML报告已生成: {html_file}")
+ return summary_html
- daily_html = self.generate_daily_summary()
+ def run(self) -> None:
+ """执行分析流程"""
+ try:
+ self._initialize_and_check_config()
- if not self.is_github_actions and html_file:
- file_url = "file://" + str(Path(html_file).resolve())
- print(f"正在打开HTML报告: {file_url}")
- webbrowser.open(file_url)
+ mode_strategy = self._get_mode_strategy()
- if daily_html:
- daily_url = "file://" + str(Path(daily_html).resolve())
- print(f"正在打开当日统计报告: {daily_url}")
- webbrowser.open(daily_url)
+ results, id_to_name, failed_ids = self._crawl_data()
+
+ self._execute_mode_strategy(mode_strategy, results, id_to_name, failed_ids)
+
+ except Exception as e:
+ print(f"分析流程执行出错: {e}")
+ raise
def main():
- analyzer = NewsAnalyzer(
- request_interval=CONFIG["REQUEST_INTERVAL"],
- report_type=CONFIG["REPORT_TYPE"],
- rank_threshold=CONFIG["RANK_THRESHOLD"],
- )
- analyzer.run()
+ try:
+ analyzer = NewsAnalyzer(
+ request_interval=CONFIG["REQUEST_INTERVAL"],
+ report_mode=CONFIG["REPORT_MODE"],
+ rank_threshold=CONFIG["RANK_THRESHOLD"],
+ )
+ analyzer.run()
+ except FileNotFoundError as e:
+ print(f"❌ 配置文件错误: {e}")
+ print("\n请确保以下文件存在:")
+ print(" • config/config.yaml")
+ print(" • config/frequency_words.txt")
+ print("\n参考项目文档进行正确配置")
+ except Exception as e:
+ print(f"❌ 程序运行错误: {e}")
+ raise
if __name__ == "__main__":
diff --git a/readme.md b/readme.md
index 4d6e965e1e..9d6ab25010 100644
--- a/readme.md
+++ b/readme.md
@@ -1,6 +1,6 @@
<div align="center">
-# 🎯 TrendRadar
+# 🎯TrendRadar
**你的专属热点助手 —— 让手机只推送你真正关心的新闻**
@@ -10,7 +10,7 @@
[![GitHub Forks](https://img.shields.io/github/forks/sansan0/TrendRadar?style=flat-square&logo=github&color=blue)](https://github.com/sansan0/TrendRadar/network/members)
[![License](https://img.shields.io/badge/license-GPL--3.0-blue.svg?style=flat-square)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.7%2B-3776AB?style=flat-square&logo=python&logoColor=ffdd54)](https://www.python.org/)
-[![Version](https://img.shields.io/badge/version-v1.4.1-green.svg?style=flat-square)](https://github.com/sansan0/TrendRadar)
+[![Version](https://img.shields.io/badge/version-v2.0.0-green.svg?style=flat-square)](https://github.com/sansan0/TrendRadar)
[![企业微信通知](https://img.shields.io/badge/企业微信-通知支持-00D4AA?style=flat-square)](https://work.weixin.qq.com/)
[![Telegram通知](https://img.shields.io/badge/Telegram-通知支持-00D4AA?style=flat-square)](https://telegram.org/)
@@ -18,34 +18,50 @@
[![飞书通知](https://img.shields.io/badge/飞书-通知支持-00D4AA?style=flat-square)](https://www.feishu.cn/)
[![GitHub Actions](https://img.shields.io/badge/GitHub_Actions-自动化-2088FF?style=flat-square&logo=github-actions&logoColor=white)](https://github.com/sansan0/TrendRadar)
[![GitHub Pages](https://img.shields.io/badge/GitHub_Pages-部署-4285F4?style=flat-square&logo=github&logoColor=white)](https://sansan0.github.io/TrendRadar)
+[![Docker](https://img.shields.io/badge/Docker-部署-2496ED?style=flat-square&logo=docker&logoColor=white)](https://hub.docker.com/)
</div>
----
> 如果本项目对你有所帮助,**点个 Star ⭐** 就是对我最大的支持。
>
-> 遇到问题可以直接提 issues,我会认真回复每一个。也可以关注【硅基茶水间】公众号,那里我会分享开源项目的思路和经验。
->
-> 我从 2025 年 4 月末开始经常开源项目,之前一直各种使用其他大佬的开源项目,虽然也勤于 star, 但确实更多的是在"白嫖"。现在想通过自己的项目回馈开源社区,同时记录这个过程中的收获和思考。
->
-> 本账号专门用来维护开源项目。如果你也在做开源,或者对开源感兴趣,**欢迎一起交流**。
+> 遇到问题可以直接提 issues。也可以【硅基茶水间】公众号留言问题。
+
----
## ✨ 核心功能
### **全网热点聚合**
-一次监控 11+ 个主流平台(微博、知乎、抖音、百度等),再也不用逐个 APP 刷新闻)
+默认监控 11 个主流平台
+
+- 今日头条
+- 百度热搜
+- 华尔街见闻
+- 澎湃新闻
+- bilibili 热搜
+- 财联社热门
+- 凤凰网
+- 贴吧
+- 微博
+- 抖音
+- 知乎
-> _理论上支持 35 个左右的平台,我默认只放了 11 个,想增加什么股票等金融相关资讯的,需要你自己手动添加,有需求的看最下方的**高级用法**_
+> _理论上支持 35 个左右,如果你想增加额外的金融类等相关资讯推送,可看最下方的**自定义监控平台**_
### **智能推送策略**
-提供两种工作模式满足不同需求:**默认模式**持续追踪热点排名变化、出现频次和时间跨度,定时推送完整分析报告(也包括新增热点);**增量模式**仅检测新增热点,无新内容时不推送,避免信息噪音
+提供三种工作模式满足不同需求:
-> _默认模式适合全面了解热点趋势,增量模式适合高频监控(≤30 分钟)场景,专注最新动态而非持续热度_
+| 模式 | 推送时机 | 显示内容 | 适用场景 |
+|------|----------|----------|----------|
+| **当日汇总模式**<br/>`daily` | 按时推送 | 当日所有匹配新闻<br/>+ 新增新闻区域 | 日报总结<br/>全面了解当日热点趋势 |
+| **当前榜单模式**<br/>`current` | 按时推送 | 当前榜单匹配新闻<br/>+ 新增新闻区域 | 实时热点追踪<br/>了解当前最火的内容 |
+| **增量监控模式**<br/>`incremental` | 有新增才推送 | 新出现的匹配频率词新闻 | 避免重复信息干扰<br/>高频监控场景 |
+
+- **📈 投资者/交易员** → 选择 `incremental`,及时获取新增资讯(建议自建服务器进行 docker 部署,提高检测频率)
+- **📰 自媒体人/内容创作者** → 选择 `current`,掌握实时热点趋势
+- **📋 企业管理者/普通用户** → 选择 `daily`,定时获取完整日报
### **精准内容筛选**
@@ -59,9 +75,9 @@
### **零技术门槛部署**
-GitHub 一键 Fork 即可使用,无需服务器,无需编程基础。如果要手机接收通知, 要达到 1 分钟部署完毕的效果,请选择**企业微信**
+GitHub 一键 Fork 即可使用,无需编程基础。
-> _就,很快_
+> _如果要手机接收通知, 要达到 1 分钟部署完毕的效果,请选择**企业微信**_
### **减少 APP 依赖**
@@ -73,7 +89,7 @@ GitHub 一键 Fork 即可使用,无需服务器,无需编程基础。如果
**典型场景:** 股市投资监控、品牌舆情追踪、行业动态关注、生活资讯获取
-**💡 提示:** 想要网页版?启用 GitHub Pages 即可获得自动更新的新闻页面。进入你的仓库 Settings → Pages,启用后就能获得专属链接,比如我这里[TrendRadar](https://sansan0.github.io/TrendRadar/)。
+**💡 提示:** 想要网页版?进入你的仓库 Settings → Pages,启用 GitHub Pages 即可获得自动更新的新闻页面链接。比如我这里[TrendRadar](https://sansan0.github.io/TrendRadar/)。
<p align="center">
<img src="_image/2025-06-14.jpg" alt="最新推送效果" width="400"/>
@@ -81,7 +97,24 @@ GitHub 一键 Fork 即可使用,无需服务器,无需编程基础。如果
## 📝 更新日志
-**升级说明:** 已**fork**的同学,如果想获得最新功能:只需把本项目中 main.py 的所有代码替换掉你的旧代码即可(直接 GitHub 网页上编辑)。
+**升级说明:**
+- **小版本更新**:直接在 GitHub 网页编辑器中,用本项目的 `main.py` 代码替换你 fork 仓库中的对应文件
+- **大版本升级**:从 v1.x 升级到 v2.0 建议删除现有 fork 后重新 fork,这样更省力且避免配置冲突
+
+### 2025/07/17 - v2.0.0
+
+**重大重构**:
+- 配置管理重构:所有配置现在通过 `config/config.yaml` 文件管理(main.py 我依旧没拆分,方便你们复制升级)
+- 运行模式升级:支持三种模式 - `daily`(当日汇总)、`current`(当前榜单)、`incremental`(增量监控)
+- Docker 支持:完整的 Docker 部署方案,支持容器化运行
+
+**配置文件说明**:
+- `config/config.yaml` - 主配置文件(应用设置、爬虫配置、通知配置、平台配置等)
+- `config/frequency_words.txt` - 关键词配置(监控词汇设置)
+
+
+<details>
+<summary><strong>📝 点击查看历史更新</strong></summary>
### 2025/07/09 - v1.4.1
@@ -89,9 +122,6 @@ GitHub 一键 Fork 即可使用,无需服务器,无需编程基础。如果
**修复问题**: 某些情况下,由于新闻本身含有特殊符号导致的偶发性排版异常。
-<details>
-<summary><strong>📝 查看历史更新</strong></summary>
-
### 2025/06/23 - v1.3.0
企业微信 和 Telegram 的推送消息有长度限制,对此我采用将消息拆分推送的方式。开发文档详见[企业微信](https://developer.work.weixin.qq.com/document/path/91770) 和 [Telegram](https://core.telegram.org/bots/api)
@@ -168,21 +198,6 @@ frequency_words.txt 文件增加了一个【必须词】功能,使用 + 号
</details>
-## 🔍 支持的平台
-
-目前已支持以下 11 个热门平台:
-
-- 今日头条
-- 百度热搜
-- 华尔街见闻
-- 澎湃新闻
-- bilibili 热搜
-- 财联社热门
-- 凤凰网
-- 贴吧
-- 微博
-- 抖音
-- 知乎
## 🚀 使用方式
@@ -286,7 +301,7 @@ frequency_words.txt 文件增加了一个【必须词】功能,使用 + 号
</details>
<details>
- <summary>🟣 <strong>Telegram Bot</strong>(配置最复杂)</summary>
+ <summary>🟣 <strong>Telegram Bot</strong></summary>
**GitHub Secret 配置:**
- 名称:`TELEGRAM_BOT_TOKEN` - 你的 Telegram Bot Token
@@ -316,12 +331,16 @@ frequency_words.txt 文件增加了一个【必须词】功能,使用 + 号
- `TELEGRAM_CHAT_ID`:填入第 2 步获得的 Chat ID
</details>
-3. **自定义关键词**:
+3. **配置关键词和设置**:
- - 修改`frequency_words.txt`文件,添加你关心的关键词
+ - **关键词配置**: 修改 `config/frequency_words.txt` 文件,添加你关心的关键词
+ - **运行模式配置**: 在 `config/config.yaml` 中修改 `report.mode` 设置:
+ - `"daily"` - 当日汇总模式(默认)
+ - `"current"` - 当前榜单模式
+ - `"incremental"` - 增量监控模式
<details>
- <summary><strong>⚙️ 点击查看 frequency_words.txt 完整配置教程</strong></summary>
+ <summary><strong>⚙️ 点击查看 frequency_words.txt 配置教程</strong></summary>
在 `frequency_words.txt` 文件中配置监控的关键词,支持三种语法和词组功能。
@@ -472,7 +491,7 @@ frequency_words.txt 文件增加了一个【必须词】功能,使用 + 号
4. **自动运行**:
- - 项目已包含`.github/workflows/crawler.yml`配置文件,默认每 50 分钟自动运行一次
+ - 项目已包含`.github/workflows/crawler.yml`配置文件,默认每 30 分钟自动运行一次
- 你也可以在 GitHub 仓库的 Actions 页面手动触发运行
5. **查看结果**:
@@ -508,64 +527,128 @@ frequency_words.txt 文件增加了一个【必须词】功能,使用 + 号
| [时间 ~ 时间] | [12 时 30 分 ~ 14 时 00 分] | 时间范围 | 标题出现的时间范围(首次~最后) |
| (N 次) | (4 次) | 出现次数 | 标题在监控期间出现的总次数 |
-## 🔧 高级用法
-
-### 自定义监控平台
-
-本项目的数据来源于 [newsnow](https://github.com/ourongxing/newsnow) 项目,你可以点击[这个网站](https://newsnow.busiyi.world/),点击里面的【更多】,可以看到所有支持的平台。如果你想支持更多平台或者不想看某些平台,可以访问 https://github.com/ourongxing/newsnow/tree/main/server/sources ,根据里面的文件名来调整 main.py 中的下面代码(如果不懂怎么看这些代码,也可以去我公众号留言):
-
-```python
-ids = [
- ("toutiao", "今日头条"),
- ("baidu", "百度热搜"),
- ("wallstreetcn-hot", "华尔街见闻"),
- ("thepaper", "澎湃新闻"),
- ("bilibili-hot-search", "bilibili 热搜"),
- ("cls-hot", "财联社热门"),
- ("ifeng", "凤凰网"),
- "tieba",
- "weibo",
- "douyin",
- "zhihu",
-]
+
+### 🔧 自定义监控平台
+
+本项目的资讯数据来源于 [newsnow](https://github.com/ourongxing/newsnow) ,你可以点击[网站](https://newsnow.busiyi.world/),点击[更多],查看是否有你想要的平台。
+
+具体添加可访问 [项目源代码](https://github.com/ourongxing/newsnow/tree/main/server/sources),根据里面的文件名,在 `config/config.yaml` 文件中修改 `platforms` 配置:
+
+```yaml
+platforms:
+ - id: "toutiao"
+ name: "今日头条"
+ - id: "baidu"
+ name: "百度热搜"
+ - id: "wallstreetcn-hot"
+ name: "华尔街见闻"
+ # 添加更多平台...
```
-## ❓ 常见问题
+### 🐳 Docker 部署
-1. **GitHub Actions 不执行怎么办?**
+1. **克隆项目并进入目录**:
+ ```bash
+ git clone https://github.com/sansan0/TrendRadar.git
+ cd TrendRadar
+ ```
- - 检查`.github/workflows/crawler.yml`文件是否存在
- - 在 Actions 页面手动触发一次 workflow
- - 确认你有足够的 GitHub Actions 免费分钟数
+2. **配置文件**:
+ - 修改 `config/config.yaml` 和 `config/frequency_words.txt`
+ - **推送链接填写**,**设置推送定时**可通过 .env 进行配置
-2. **没有收到飞书通知怎么办?**
+3. **启动服务**:
+ ```bash
+ cd docker
+ docker-compose up -d
+ ```
- - 检查`FEISHU_WEBHOOK_URL`是否正确设置(环境变量或 CONFIG 中)
- - 检查飞书机器人是否仍在群内且启用
- - 查看程序输出中是否有发送失败的错误信息
- - 确认飞书流程配置中的参数结构正确
+4. **管理服务**:
+ ```bash
+ # 查看运行状态
+ docker exec -it trend-radar python manage.py status
+
+ # 手动执行一次爬虫
+ docker exec -it trend-radar python manage.py run
+
+ # 查看实时日志
+ docker exec -it trend-radar python manage.py logs
+
+ # 显示当前配置
+ docker exec -it trend-radar python manage.py config
+
+ # 显示输出文件
+ docker exec -it trend-radar python manage.py files
+
+ # 重启定时服务
+ docker exec -it trend-radar python manage.py restart
+
+ # 显示帮助信息
+ docker exec -it trend-radar python manage.py help
+ ```
-3. **想要停止手机接收新闻通知消息?**
- - 将`ENABLE_NOTIFICATION`设置为`False`
+## ☕ 赞助让代码更香.jpg
-4. **想要停止在本地继续生成新闻数据?**
+<div align="center">
- - 将`ENABLE_CRAWLER`设置为`False`
+| Wechat | Alipay |
+|:---:|:---:|
+| <img src="https://cdn-1258574687.cos.ap-shanghai.myqcloud.com/img/%2F2025%2F07%2F17%2F2ae0a88d98079f7e876c2b4dc85233c6-9e8025.JPG" width="300" title="微信支付"/> | <img src="https://cdn-1258574687.cos.ap-shanghai.myqcloud.com/img/%2F2025%2F07%2F17%2Fed4f20ab8e35be51f8e84c94e6e239b4-fe4947.JPG" width="300" title="支付宝支付"/> |
-## 📧 学习交流
+</div>
-扫码关注微信公众号,里面有文章是讲我写的这些项目的,咳如果对你有了点帮助,献上【点赞,转发,推荐】三连,就算支持了俺这个作者的开发了,顺便也可以反馈使用问题:
+
+## 📧 学习交流
<div align="center">
-![微信底部留言](_image/support.jpg)
+<img src="_image/support.jpg" title="硅基茶水间留言"/>
+<br>
+<img src="_image/weixin.png" width="600" title="硅基茶水间"/>
</div>
-![微信公众号](_image/weixin.png)
-## Star History
+```mermaid
+flowchart TD
+ A[👤 用户开始] --> B[🍴 Fork 项目]
+ B --> C[⚙️ 选择通知方式]
+
+ C --> D1[📱 企业微信群机器人<br/>最简单快速]
+ C --> D2[💬 飞书机器人<br/>显示效果最佳]
+ C --> D3[🔔 钉钉机器人<br/>]
+ C --> D4[📟 Telegram Bot<br/>]
+
+ D1 --> E[🔑 配置 GitHub Secrets<br/>填入机器人 Webhook 地址]
+ D2 --> E
+ D3 --> E
+ D4 --> E
+
+ E --> F[📝 编辑关键词配置<br/>config/frequency_words.txt<br/>添加你关心的词汇]
+ F --> G[🎯 选择运行模式<br/>config/config.yaml<br/>daily/current/incremental]
+
+ G --> H[✅ 配置完成]
+ H --> I[🤖 系统每30分钟自动运行]
+
+ I --> J[📊 爬取各大平台热点]
+ J --> K[🔍 根据关键词筛选]
+ K --> L[📱 推送到你的手机]
+
+ L --> M[📈 查看推送结果]
+ M --> N{满意效果?}
+ N -->|不满意| F
+ N -->|满意| O[🎉 持续接收精准推送]
+
+ style A fill:#e1f5fe
+ style B fill:#f3e5f5
+ style E fill:#fff3e0
+ style F fill:#e8f5e8
+ style G fill:#e8f5e8
+ style L fill:#ffebee
+ style O fill:#e8f5e8
+```
+
[![Star History Chart](https://api.star-history.com/svg?repos=sansan0/TrendRadar&type=Date)](https://www.star-history.com/#sansan0/TrendRadar&Date)
@@ -583,6 +666,6 @@ GPL-3.0 License
**⭐ 如果这个工具对你有帮助,请给项目点个 Star 支持开发!**
-[🔝 回到顶部](#TrendRadar-多平台热点资讯监控分析系统)
+[🔝 回到顶部](#trendradar)
</div>
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000000..bef336ff69
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,3 @@
+requests==2.32.4
+pytz==2025.2
+PyYAML==6.0.2
\ No newline at end of file
diff --git a/version b/version
index 347f5833ee..227cea2156 100644
--- a/version
+++ b/version
@@ -1 +1 @@
-1.4.1
+2.0.0