Skip to content

⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.🎯 告别信息过载,你的 AI 舆情监控助手与热点筛选工具!聚合多平台热点 + RSS 订阅,支持关键词精准筛选。AI 智能筛选新闻 + AI 翻译 + AI 分析简报直推手机,也支持接入 MCP 架构,赋能 AI 自然语言对话分析、情感洞察与趋势预测等。支持 Docker ,数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/bark/slack 等渠道智能推送。

feat: AI 翻译分批处理 & 模块拆分重构,升级至 v6.10.0

sansan4df231891fc3
diff --git a/README-EN.md b/README-EN.md
index adfe562a7f..ffb1baf932 100644
--- a/README-EN.md
+++ b/README-EN.md
@@ -11,7 +11,7 @@ Deploy in <strong>30 seconds</strong> — Say goodbye to endless scrolling, only
[![GitHub Stars](https://img.shields.io/github/stars/sansan0/TrendRadar?style=flat-square&logo=github&color=yellow)](https://github.com/sansan0/TrendRadar/stargazers)
[![GitHub Forks](https://img.shields.io/github/forks/sansan0/TrendRadar?style=flat-square&logo=github&color=blue)](https://github.com/sansan0/TrendRadar/network/members)
[![License](https://img.shields.io/badge/license-GPL--3.0-blue.svg?style=flat-square)](LICENSE)
-[![Version](https://img.shields.io/badge/version-v6.9.1-blue.svg)](https://github.com/sansan0/TrendRadar)
+[![Version](https://img.shields.io/badge/version-v6.10.0-blue.svg)](https://github.com/sansan0/TrendRadar)
[![MCP](https://img.shields.io/badge/MCP-v4.1.0-green.svg)](https://github.com/sansan0/TrendRadar)
[![Docker Pulls](https://img.shields.io/docker/pulls/wantcat/trendradar?style=flat-square&logo=docker&logoColor=white&label=TrendRadar%20Pulls&color=2496ED)](https://hub.docker.com/r/wantcat/trendradar)
[![Docker Pulls](https://img.shields.io/docker/pulls/wantcat/trendradar-mcp?style=flat-square&logo=docker&logoColor=white&label=MCP%20Pulls&color=2496ED)](https://hub.docker.com/r/wantcat/trendradar-mcp)
@@ -197,10 +197,11 @@ This contributes to the sustainable maintenance of the project and the growth of
- **Tip**: Check [Changelog] to understand specific [Features]
-### 2026/06/02 - v6.9.0
+### 2026/06/19 - v6.10.0
-- **Platform Domain Safety Validation**: New `expected_domain` config option validates returned data link domains — mismatched data is automatically discarded with a warning, effectively preventing link hijacking or data tampering
-- **Custom Platform API URL**: Support self-hosting newsnow and configuring `api_url` to use your own data source
+- **AI Translation Batch Processing**: Automatically splits large translation requests into batches, preventing single-request overflow failures
+- **Module Refactoring**: Split context.py and \_\_main\_\_.py, extracted AI filter pipeline into a standalone module for clearer responsibilities and easier maintenance
+- **Fixed Feishu Source Label Display**: Fixed source labels and AI standalone source overview being swallowed by CommonMark rendering in Feishu cards
### 2026/02/09 - mcp-v4.0.0
@@ -214,6 +215,11 @@ This contributes to the sustainable maintenance of the project and the growth of
<details>
<summary>👉 Click to expand: <strong>Historical Updates</strong></summary>
+### 2026/06/02 - v6.9.0
+
+- **Platform Domain Safety Validation**: New `expected_domain` config option validates returned data link domains — mismatched data is automatically discarded with a warning, effectively preventing link hijacking or data tampering
+- **Custom Platform API URL**: Support self-hosting newsnow and configuring `api_url` to use your own data source
+
### 2026/05/23 - v6.8.0
- **HTML Report Major Enhancement**: Added report metadata display (generation time, data sources, version), dark mode auto-adaptation, improved Tab bar interaction, and trend arrow visualization for a significantly better browser reading experience
diff --git a/README.md b/README.md
index 1f101edd63..9928136387 100644
--- a/README.md
+++ b/README.md
@@ -12,7 +12,7 @@
[![GitHub Stars](https://img.shields.io/github/stars/sansan0/TrendRadar?style=flat-square&logo=github&color=yellow)](https://github.com/sansan0/TrendRadar/stargazers)
[![GitHub Forks](https://img.shields.io/github/forks/sansan0/TrendRadar?style=flat-square&logo=github&color=blue)](https://github.com/sansan0/TrendRadar/network/members)
[![License](https://img.shields.io/badge/license-GPL--3.0-blue.svg?style=flat-square)](LICENSE)
-[![Version](https://img.shields.io/badge/version-v6.9.1-blue.svg)](https://github.com/sansan0/TrendRadar)
+[![Version](https://img.shields.io/badge/version-v6.10.0-blue.svg)](https://github.com/sansan0/TrendRadar)
[![MCP](https://img.shields.io/badge/MCP-v4.1.0-green.svg)](https://github.com/sansan0/TrendRadar)
[![Docker Pulls](https://img.shields.io/docker/pulls/wantcat/trendradar?style=flat-square&logo=docker&logoColor=white&label=TrendRadar%20Pulls&color=2496ED)](https://hub.docker.com/r/wantcat/trendradar)
[![Docker Pulls](https://img.shields.io/docker/pulls/wantcat/trendradar-mcp?style=flat-square&logo=docker&logoColor=white&label=MCP%20Pulls&color=2496ED)](https://hub.docker.com/r/wantcat/trendradar-mcp)
@@ -245,10 +245,11 @@
- **提示**:建议查看【历史更新】,明确具体的【功能内容】
-### 2026/06/02 - v6.9.0
+### 2026/06/19 - v6.10.0
-- **热榜域名安全校验**:新增 `expected_domain` 配置项,校验返回数据链接的域名合法性,不匹配时自动丢弃数据并警告,有效防范链接劫持或数据篡改
-- **自定义热榜 API 地址**:支持自部署 newsnow 并配置 `api_url` 使用自己的数据源
+- **AI 翻译分批处理**:大量标题翻译时自动分批请求,避免单次请求超限导致翻译失败
+- **模块拆分重构**:拆分 context.py 和 \_\_main\_\_.py,AI 筛选流水线独立为 filter_pipeline 模块,职责更清晰、维护更方便
+- **修复飞书来源标签显示**:修复飞书卡片中来源标签和 AI 独立源点速览被 CommonMark 吞掉不显示的问题
### 2026/02/09 - mcp-v4.0.0
@@ -262,6 +263,11 @@
<details>
<summary>👉 点击展开:<strong>历史更新</strong></summary>
+### 2026/06/02 - v6.9.0
+
+- **热榜域名安全校验**:新增 `expected_domain` 配置项,校验返回数据链接的域名合法性,不匹配时自动丢弃数据并警告,有效防范链接劫持或数据篡改
+- **自定义热榜 API 地址**:支持自部署 newsnow 并配置 `api_url` 使用自己的数据源
+
### 2026/05/23 - v6.8.0
- **HTML 报告全面增强**:新增报告元数据展示(生成时间、数据来源、版本号)、暗色模式自动适配、Tab 栏交互优化、趋势箭头可视化,浏览器阅读体验大幅提升
diff --git a/config/config.yaml b/config/config.yaml
index 8b538ba173..b743c7c76a 100644
--- a/config/config.yaml
+++ b/config/config.yaml
@@ -1,6 +1,6 @@
# ═══════════════════════════════════════════════════════════════
# TrendRadar 配置文件
-# Version: 2.3.0
+# Version: 2.4.0
# ═══════════════════════════════════════════════════════════════
@@ -444,6 +444,9 @@ ai_translation:
language: "中文" # 翻译目标语言(如 "English", "Korean", "法语")
prompt_file: "ai_translation_prompt.txt" # 提示词文件(相对于 config 目录)
+ batch_size: 100 # 每批发送给 AI 的标题数,超过此数量自动分批
+ batch_interval: 2 # 分批间隔(秒),避免触发 API 限流,0=不等待
+
# 翻译范围(display.regions 关闭的区域即使开启也不会翻译)
scope:
hotlist: false # 翻译热榜标题(true=翻译, false=不翻译)
diff --git a/docs/assets/script.js b/docs/assets/script.js
index 6e62e2ec44..9baca30f83 100644
--- a/docs/assets/script.js
+++ b/docs/assets/script.js
@@ -1018,6 +1018,8 @@ function renderControls(mod) {
html = createToggleControl(mod.key, "enabled", "开启 AI 自动翻译");
html += createInputControl(mod.key, "language", "目标语言");
html += createInputControl(mod.key, "prompt_file", "提示词配置文件");
+ html += createNumberControl(mod.key, "batch_size", "每批翻译标题数");
+ html += createNumberControl(mod.key, "batch_interval", "分批间隔 (秒)");
break;
}
diff --git a/pyproject.toml b/pyproject.toml
index 28face9e7d..b66934537c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "trendradar"
-version = "6.9.1"
+version = "6.10.0"
description = "TrendRadar - 热点新闻聚合与分析工具"
requires-python = ">=3.12"
dependencies = [
diff --git a/trendradar/__init__.py b/trendradar/__init__.py
index 966054085b..cab573c3a6 100644
--- a/trendradar/__init__.py
+++ b/trendradar/__init__.py
@@ -9,5 +9,5 @@ TrendRadar - 热点新闻聚合与分析工具
from trendradar.context import AppContext
-__version__ = "6.9.1"
+__version__ = "6.10.0"
__all__ = ["AppContext", "__version__"]
diff --git a/trendradar/__main__.py b/trendradar/__main__.py
index d34a8866f1..3cf9d235c8 100644
--- a/trendradar/__main__.py
+++ b/trendradar/__main__.py
@@ -7,170 +7,23 @@ TrendRadar 主程序
"""
import argparse
-import copy
-import json
import os
-import re
-import sys
import webbrowser
-from datetime import datetime, timezone
from pathlib import Path
-from typing import Dict, List, Tuple, Optional
-
-import requests
+from typing import Dict, List, Optional, Tuple
from trendradar.context import AppContext
from trendradar import __version__
-from trendradar.core import load_config, parse_multi_account_config, validate_paired_configs
+from trendradar.core import load_config
from trendradar.core.analyzer import convert_keyword_stats_to_platform_stats
from trendradar.crawler import DataFetcher
from trendradar.storage import convert_crawl_results_to_news_data
from trendradar.utils.time import DEFAULT_TIMEZONE, is_within_days, calculate_days_old
from trendradar.ai import AIAnalyzer, AIAnalysisResult
from trendradar.core.scheduler import ResolvedSchedule
-from trendradar.core.cdn import fetch_with_fallback
-
-
-def _parse_version(version_str: str) -> Tuple[int, int, int]:
- """解析版本号字符串为元组"""
- try:
- parts = version_str.strip().split(".")
- if len(parts) >= 3:
- return int(parts[0]), int(parts[1]), int(parts[2])
- return 0, 0, 0
- except (ValueError, AttributeError, TypeError):
- return 0, 0, 0
-
-
-def _compare_version(local: str, remote: str) -> str:
- """比较版本号,返回状态文字"""
- local_tuple = _parse_version(local)
- remote_tuple = _parse_version(remote)
-
- if local_tuple < remote_tuple:
- return "⚠️ 需要更新"
- elif local_tuple > remote_tuple:
- return "🔮 超前版本"
- else:
- return "✅ 已是最新"
-
-
-def _fetch_remote_version(version_url: str, proxy_url: Optional[str] = None) -> Optional[str]:
- """获取远程版本号(支持 CDN 多源回退)"""
- return fetch_with_fallback(version_url, proxy_url)
-
-
-def _parse_config_versions(content: str) -> Dict[str, str]:
- """解析配置文件版本内容为字典"""
- versions = {}
- try:
- if not content:
- return versions
- for line in content.splitlines():
- line = line.strip()
- if not line or "=" not in line:
- continue
- name, version = line.split("=", 1)
- versions[name.strip()] = version.strip()
- except Exception as e:
- print(f"[版本检查] 解析配置版本失败: {e}")
- return versions
-
-
-def check_all_versions(
- version_url: str,
- configs_version_url: Optional[str] = None,
- proxy_url: Optional[str] = None
-) -> Tuple[bool, Optional[str]]:
- """
- 统一版本检查:程序版本 + 配置文件版本
-
- Args:
- version_url: 远程程序版本检查 URL
- configs_version_url: 远程配置文件版本检查 URL (返回格式: filename=version)
- proxy_url: 代理 URL
-
- Returns:
- (need_update, remote_version): 程序是否需要更新及远程版本号
- """
- # 获取远程版本
- remote_version = _fetch_remote_version(version_url, proxy_url)
-
- # 获取远程配置版本(如果有提供 URL)
- remote_config_versions = {}
- if configs_version_url:
- content = _fetch_remote_version(configs_version_url, proxy_url)
- if content:
- remote_config_versions = _parse_config_versions(content)
-
- print("=" * 60)
- print("版本检查")
- print("=" * 60)
-
- if remote_version:
- print(f"远程程序版本: {remote_version}")
- else:
- print("远程程序版本: 获取失败")
-
- if configs_version_url:
- if remote_config_versions:
- print(f"远程配置清单: 获取成功 ({len(remote_config_versions)} 个文件)")
- else:
- print("远程配置清单: 获取失败或为空")
-
- print("-" * 60)
+from trendradar.commands import check_all_versions, run_doctor, run_test_notification, handle_status_commands
+from trendradar.commands.version import _fetch_remote_version, _parse_version
- program_status = _compare_version(__version__, remote_version) if remote_version else "(无法比较)"
- print(f" 主程序版本: {__version__} {program_status}")
-
- config_files = [
- Path("config/config.yaml"),
- Path("config/timeline.yaml"),
- Path("config/frequency_words.txt"),
- Path("config/ai_interests.txt"),
- Path("config/ai_analysis_prompt.txt"),
- Path("config/ai_translation_prompt.txt"),
- ]
-
- version_pattern = re.compile(r"Version:\s*(\d+\.\d+\.\d+)", re.IGNORECASE)
-
- for config_file in config_files:
- if not config_file.exists():
- print(f" {config_file.name}: 文件不存在")
- continue
-
- try:
- with open(config_file, "r", encoding="utf-8") as f:
- local_version = None
- for i, line in enumerate(f):
- if i >= 20:
- break
- match = version_pattern.search(line)
- if match:
- local_version = match.group(1)
- break
-
- # 获取该文件的远程版本
- target_remote_version = remote_config_versions.get(config_file.name)
-
- if local_version:
- if target_remote_version:
- status = _compare_version(local_version, target_remote_version)
- print(f" {config_file.name}: {local_version} {status}")
- else:
- print(f" {config_file.name}: {local_version} (未找到远程版本)")
- else:
- print(f" {config_file.name}: 未找到本地版本号")
- except Exception as e:
- print(f" {config_file.name}: 读取失败 - {e}")
-
- print("=" * 60)
-
- # 返回程序版本的更新状态
- if remote_version:
- need_update = _parse_version(__version__) < _parse_version(remote_version)
- return need_update, remote_version if need_update else None
- return False, None
# === 主分析器 ===
@@ -1783,412 +1636,8 @@ class NewsAnalyzer:
self.ctx.cleanup()
-def _record_doctor_result(results: List[Tuple[str, str, str]], status: str, item: str, detail: str) -> None:
- """记录并打印 doctor 检查结果"""
- icon_map = {
- "pass": "✅",
- "warn": "⚠️",
- "fail": "❌",
- }
- icon = icon_map.get(status, "•")
- results.append((status, item, detail))
- print(f"{icon} {item}: {detail}")
-
-
-def _save_doctor_report(
- results: List[Tuple[str, str, str]],
- pass_count: int,
- warn_count: int,
- fail_count: int,
- config_path: Optional[str],
-) -> None:
- """保存 doctor 体检报告到 JSON 文件"""
- report = {
- "version": __version__,
- "generated_at": datetime.now(timezone.utc).isoformat(),
- "config_path": config_path or os.environ.get("CONFIG_PATH", "config/config.yaml"),
- "summary": {
- "pass": pass_count,
- "warn": warn_count,
- "fail": fail_count,
- "ok": fail_count == 0,
- },
- "checks": [
- {"status": status, "item": item, "detail": detail}
- for status, item, detail in results
- ],
- }
-
- try:
- output_dir = Path("output") / "meta"
- output_dir.mkdir(parents=True, exist_ok=True)
- output_path = output_dir / "doctor_report.json"
- output_path.write_text(
- json.dumps(report, ensure_ascii=False, indent=2),
- encoding="utf-8",
- )
- print(f"体检报告已保存: {output_path}")
- except Exception as e:
- print(f"⚠️ 体检报告保存失败: {e}")
-
-
-def _run_doctor(config_path: Optional[str] = None) -> bool:
- """运行环境体检"""
- print("=" * 60)
- print(f"TrendRadar v{__version__} 环境体检")
- print("=" * 60)
-
- results: List[Tuple[str, str, str]] = []
- config = None
-
- # 1) Python 版本检查
- py_ok = sys.version_info >= (3, 10)
- py_version = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}"
- if py_ok:
- _record_doctor_result(results, "pass", "Python版本", f"{py_version} (满足 >= 3.10)")
- else:
- _record_doctor_result(results, "fail", "Python版本", f"{py_version} (不满足 >= 3.10)")
-
- # 2) 关键文件检查
- if config_path is None:
- config_path = os.environ.get("CONFIG_PATH", "config/config.yaml")
-
- required_files = [
- (config_path, "主配置文件"),
- ("config/frequency_words.txt", "关键词文件"),
- ]
- optional_files = [
- ("config/timeline.yaml", "调度文件"),
- ]
-
- for path_str, desc in required_files:
- if Path(path_str).exists():
- _record_doctor_result(results, "pass", desc, f"已找到: {path_str}")
- else:
- _record_doctor_result(results, "fail", desc, f"缺失: {path_str}")
-
- for path_str, desc in optional_files:
- if Path(path_str).exists():
- _record_doctor_result(results, "pass", desc, f"已找到: {path_str}")
- else:
- _record_doctor_result(results, "warn", desc, f"未找到: {path_str}(将使用默认调度模板)")
-
- # 3) 配置加载检查
- try:
- config = load_config(config_path)
- _record_doctor_result(results, "pass", "配置加载", f"加载成功: {config_path}")
- except Exception as e:
- _record_doctor_result(results, "fail", "配置加载", f"加载失败: {e}")
-
- # 后续检查依赖配置对象
- if config:
- # 4) 调度配置检查
- try:
- ctx = AppContext(config)
- schedule = ctx.create_scheduler().resolve()
- detail = f"调度解析成功(report_mode={schedule.report_mode}, ai_mode={schedule.ai_mode})"
- _record_doctor_result(results, "pass", "调度配置", detail)
- except Exception as e:
- _record_doctor_result(results, "fail", "调度配置", f"解析失败: {e}")
-
- # 5) AI 配置检查(按功能场景区分严重级别)
- ai_analysis_enabled = config.get("AI_ANALYSIS", {}).get("ENABLED", False)
- ai_translation_enabled = config.get("AI_TRANSLATION", {}).get("ENABLED", False)
- ai_filter_enabled = config.get("FILTER", {}).get("METHOD", "keyword") == "ai"
- ai_enabled = ai_analysis_enabled or ai_translation_enabled or ai_filter_enabled
-
- if ai_enabled:
- try:
- from trendradar.ai.client import AIClient
- valid, message = AIClient(config.get("AI", {})).validate_config()
- if valid:
- _record_doctor_result(results, "pass", "AI配置", f"模型: {config.get('AI', {}).get('MODEL', '')}")
- else:
- # AI 分析/翻译是硬依赖;AI 筛选缺失时会自动回退关键词匹配
- if ai_analysis_enabled or ai_translation_enabled:
- _record_doctor_result(results, "fail", "AI配置", message)
- else:
- _record_doctor_result(results, "warn", "AI配置", f"{message}(AI 筛选将回退关键词模式)")
- except Exception as e:
- _record_doctor_result(results, "fail", "AI配置", f"校验异常: {e}")
- else:
- _record_doctor_result(results, "warn", "AI配置", "未启用 AI 功能,跳过校验")
-
- # 6) 存储配置检查
- try:
- storage_cfg = config.get("STORAGE", {})
- backend = storage_cfg.get("BACKEND", "auto")
- remote = storage_cfg.get("REMOTE", {})
- missing_remote_keys = [
- k for k in ("BUCKET_NAME", "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "ENDPOINT_URL")
- if not remote.get(k)
- ]
-
- if backend == "remote" and missing_remote_keys:
- _record_doctor_result(
- results, "fail", "存储配置",
- f"remote 模式缺少配置: {', '.join(missing_remote_keys)}"
- )
- elif backend == "auto" and os.environ.get("GITHUB_ACTIONS") == "true" and missing_remote_keys:
- _record_doctor_result(
- results, "warn", "存储配置",
- "GitHub Actions + auto 模式未完整配置远程存储,可能导致数据丢失"
- )
- else:
- sm = AppContext(config).get_storage_manager()
- _record_doctor_result(results, "pass", "存储配置", f"当前后端: {sm.backend_name}")
- except Exception as e:
- _record_doctor_result(results, "fail", "存储配置", f"检查失败: {e}")
-
- # 7) 通知渠道配置检查
- channel_details = []
- channel_issues = []
- max_accounts = config.get("MAX_ACCOUNTS_PER_CHANNEL", 3)
-
- # 普通单值/多值渠道
- for key, name in [
- ("FEISHU_WEBHOOK_URL", "飞书"),
- ("DINGTALK_WEBHOOK_URL", "钉钉"),
- ("WEWORK_WEBHOOK_URL", "企业微信"),
- ("BARK_URL", "Bark"),
- ("SLACK_WEBHOOK_URL", "Slack"),
- ("GENERIC_WEBHOOK_URL", "通用Webhook"),
- ]:
- values = parse_multi_account_config(config.get(key, ""))
- if values:
- channel_details.append(f"{name}({min(len(values), max_accounts)}个)")
-
- # Telegram 配对校验
- tg_tokens = parse_multi_account_config(config.get("TELEGRAM_BOT_TOKEN", ""))
- tg_chats = parse_multi_account_config(config.get("TELEGRAM_CHAT_ID", ""))
- if tg_tokens or tg_chats:
- valid, count = validate_paired_configs(
- {"bot_token": tg_tokens, "chat_id": tg_chats},
- "Telegram",
- required_keys=["bot_token", "chat_id"],
- )
- if valid and count > 0:
- channel_details.append(f"Telegram({min(count, max_accounts)}个)")
- else:
- channel_issues.append("Telegram bot_token/chat_id 配置不完整或数量不一致")
-
- # ntfy 配对校验(token 可选)
- ntfy_server = config.get("NTFY_SERVER_URL", "")
- ntfy_topics = parse_multi_account_config(config.get("NTFY_TOPIC", ""))
- ntfy_tokens = parse_multi_account_config(config.get("NTFY_TOKEN", ""))
- if ntfy_server and ntfy_topics:
- if ntfy_tokens:
- valid, count = validate_paired_configs(
- {"topic": ntfy_topics, "token": ntfy_tokens},
- "ntfy",
- )
- if valid and count > 0:
- channel_details.append(f"ntfy({min(count, max_accounts)}个)")
- else:
- channel_issues.append("ntfy topic/token 数量不一致")
- else:
- channel_details.append(f"ntfy({min(len(ntfy_topics), max_accounts)}个)")
-
- # 邮件配置完整性
- email_ready = all(
- [
- config.get("EMAIL_FROM"),
- config.get("EMAIL_PASSWORD"),
- config.get("EMAIL_TO"),
- ]
- )
- if email_ready:
- channel_details.append("邮件")
- elif any([config.get("EMAIL_FROM"), config.get("EMAIL_PASSWORD"), config.get("EMAIL_TO")]):
- channel_issues.append("邮件配置不完整(需要 from/password/to 同时配置)")
-
- if channel_issues and not channel_details:
- _record_doctor_result(results, "fail", "通知配置", ";".join(channel_issues))
- elif channel_issues and channel_details:
- detail = f"可用渠道: {', '.join(channel_details)};问题: {';'.join(channel_issues)}"
- _record_doctor_result(results, "warn", "通知配置", detail)
- elif channel_details:
- _record_doctor_result(results, "pass", "通知配置", f"可用渠道: {', '.join(channel_details)}")
- else:
- _record_doctor_result(results, "warn", "通知配置", "未配置任何通知渠道")
-
- # 8) 输出目录可写检查
- try:
- output_dir = Path("output")
- output_dir.mkdir(parents=True, exist_ok=True)
- probe_file = output_dir / ".doctor_write_probe"
- probe_file.write_text("ok", encoding="utf-8")
- probe_file.unlink(missing_ok=True)
- _record_doctor_result(results, "pass", "输出目录", f"可写: {output_dir}")
- except Exception as e:
- _record_doctor_result(results, "fail", "输出目录", f"不可写: {e}")
-
- pass_count = sum(1 for status, _, _ in results if status == "pass")
- warn_count = sum(1 for status, _, _ in results if status == "warn")
- fail_count = sum(1 for status, _, _ in results if status == "fail")
-
- _save_doctor_report(results, pass_count, warn_count, fail_count, config_path)
-
- print("-" * 60)
- print(f"体检结果: ✅ {pass_count} 项通过 ⚠️ {warn_count} 项警告 ❌ {fail_count} 项失败")
- print("=" * 60)
-
- if fail_count == 0:
- print("体检通过。")
- return True
-
- print("体检未通过,请先修复失败项。")
- return False
-
-
-def _build_test_report_data(ctx: AppContext) -> Dict:
- """构造通知测试用报告数据"""
- now = ctx.get_time()
- time_display = now.strftime("%H:%M")
- title = f"TrendRadar 通知测试消息({now.strftime('%Y-%m-%d %H:%M:%S')})"
-
- return {
- "stats": [
- {
- "word": "连通性测试",
- "count": 1,
- "titles": [
- {
- "title": title,
- "source_name": "TrendRadar",
- "url": "https://github.com/sansan0/TrendRadar",
- "mobile_url": "",
- "ranks": [1],
- "rank_threshold": ctx.rank_threshold,
- "count": 1,
- "is_new": True,
- "time_display": time_display,
- "matched_keyword": "连通性测试",
- }
- ],
- }
- ],
- "failed_ids": [],
- "new_titles": [],
- "id_to_name": {},
- }
-
-
-def _create_test_html_file(ctx: AppContext) -> Optional[str]:
- """创建邮件测试用 HTML 文件"""
- try:
- now = ctx.get_time()
- output_dir = Path("output") / "html" / ctx.format_date()
- output_dir.mkdir(parents=True, exist_ok=True)
- html_path = output_dir / f"notification_test_{ctx.format_time()}.html"
- html_content = f"""<!DOCTYPE html>
-<html lang="zh-CN">
-<head><meta charset="UTF-8"><title>TrendRadar 通知测试</title></head>
-<body>
-<h2>TrendRadar 通知连通性测试</h2>
-<p>测试时间:{now.strftime('%Y-%m-%d %H:%M:%S')} ({ctx.timezone})</p>
-<p>这是一条测试消息,用于验证邮件渠道是否可达。</p>
-</body>
-</html>"""
- html_path.write_text(html_content, encoding="utf-8")
- return str(html_path)
- except Exception as e:
- print(f"[测试通知] 创建测试 HTML 失败: {e}")
- return None
-
-
-def _run_test_notification(config: Dict) -> bool:
- """发送测试通知到已配置渠道"""
- from trendradar.notification import NotificationDispatcher
-
- ctx = AppContext(config)
-
- try:
- # 检查是否配置了通知渠道
- has_notification = any(
- [
- config.get("FEISHU_WEBHOOK_URL"),
- config.get("DINGTALK_WEBHOOK_URL"),
- config.get("WEWORK_WEBHOOK_URL"),
- (config.get("TELEGRAM_BOT_TOKEN") and config.get("TELEGRAM_CHAT_ID")),
- (config.get("EMAIL_FROM") and config.get("EMAIL_PASSWORD") and config.get("EMAIL_TO")),
- (config.get("NTFY_SERVER_URL") and config.get("NTFY_TOPIC")),
- config.get("BARK_URL"),
- config.get("SLACK_WEBHOOK_URL"),
- config.get("GENERIC_WEBHOOK_URL"),
- ]
- )
- if not has_notification:
- print("未检测到可用通知渠道,请先在 config.yaml 或环境变量中配置。")
- return False
-
- # 测试时固定展示区域,避免用户关闭 HOTLIST 导致测试内容为空
- test_config = copy.deepcopy(config)
- test_display = test_config.setdefault("DISPLAY", {})
- test_regions = test_display.setdefault("REGIONS", {})
- test_regions.update(
- {
- "HOTLIST": True,
- "NEW_ITEMS": False,
- "RSS": False,
- "STANDALONE": False,
- "AI_ANALYSIS": False,
- }
- )
-
- # 测试时禁用翻译,避免触发额外 AI 调用
- if "AI_TRANSLATION" in test_config:
- test_config["AI_TRANSLATION"]["ENABLED"] = False
-
- proxy_url = test_config.get("DEFAULT_PROXY", "") if test_config.get("USE_PROXY") else None
- if proxy_url:
- print("[测试通知] 检测到代理配置,将使用代理发送")
-
- dispatcher = NotificationDispatcher(
- config=test_config,
- get_time_func=ctx.get_time,
- split_content_func=ctx.split_content,
- translator=None,
- )
-
- report_data = _build_test_report_data(ctx)
- html_file_path = _create_test_html_file(ctx)
-
- print("=" * 60)
- print("通知连通性测试")
- print("=" * 60)
-
- results = dispatcher.dispatch_all(
- report_data=report_data,
- report_type="通知连通性测试",
- proxy_url=proxy_url,
- mode="daily",
- html_file_path=html_file_path,
- )
-
- if not results:
- print("没有可测试的有效通知渠道(可能配置不完整)。")
- return False
-
- print("-" * 60)
- success_count = 0
- for channel, ok in results.items():
- if ok:
- success_count += 1
- print(f"✅ {channel}: 测试成功")
- else:
- print(f"❌ {channel}: 测试失败")
-
- print("-" * 60)
- print(f"测试结果: {success_count}/{len(results)} 个渠道成功")
- return success_count > 0
- finally:
- ctx.cleanup()
-
-
def main():
"""主程序入口"""
- # 解析命令行参数
parser = argparse.ArgumentParser(
description="TrendRadar - 热点新闻聚合与分析工具",
formatter_class=argparse.RawDescriptionHelpFormatter,
@@ -2206,44 +1655,28 @@ def main():
python -m trendradar --test-notification # 测试通知渠道连通性
"""
)
- parser.add_argument(
- "--show-schedule",
- action="store_true",
- help="显示当前调度状态"
- )
- parser.add_argument(
- "--doctor",
- action="store_true",
- help="运行环境与配置体检"
- )
- parser.add_argument(
- "--test-notification",
- action="store_true",
- help="发送测试通知到已配置渠道"
- )
+ parser.add_argument("--show-schedule", action="store_true", help="显示当前调度状态")
+ parser.add_argument("--doctor", action="store_true", help="运行环境与配置体检")
+ parser.add_argument("--test-notification", action="store_true", help="发送测试通知到已配置渠道")
args = parser.parse_args()
debug_mode = False
try:
- # 处理 doctor 命令(不依赖完整运行流程)
if args.doctor:
- ok = _run_doctor()
+ ok = run_doctor()
if not ok:
raise SystemExit(1)
return
- # 先加载配置
config = load_config()
- # 处理状态查看命令
if args.show_schedule:
- _handle_status_commands(config)
+ handle_status_commands(config)
return
- # 处理通知测试命令
if args.test_notification:
- ok = _run_test_notification(config)
+ ok = run_test_notification(config)
if not ok:
raise SystemExit(1)
return
@@ -2251,23 +1684,19 @@ def main():
version_url = config.get("VERSION_CHECK_URL", "")
configs_version_url = config.get("CONFIGS_VERSION_CHECK_URL", "")
- # 统一版本检查(程序版本 + 配置文件版本,只请求一次远程)
need_update = False
remote_version = None
if version_url:
need_update, remote_version = check_all_versions(version_url, configs_version_url)
- # 复用已加载的配置,避免重复加载
analyzer = NewsAnalyzer(config=config)
- # 设置更新信息(复用已获取的远程版本,不再重复请求)
if analyzer.is_github_actions and need_update and remote_version:
analyzer.update_info = {
"current_version": __version__,
"remote_version": remote_version,
}
- # 获取 debug 配置
debug_mode = analyzer.ctx.config.get("DEBUG", False)
analyzer.run()
except FileNotFoundError as e:
@@ -2282,61 +1711,5 @@ def main():
raise
-def _handle_status_commands(config: Dict) -> None:
- """处理状态查看命令 - 显示当前调度状态"""
- from trendradar.context import AppContext
-
- ctx = AppContext(config)
-
- print("=" * 60)
- print(f"TrendRadar v{__version__} 调度状态")
- print("=" * 60)
-
- try:
- scheduler = ctx.create_scheduler()
- schedule = scheduler.resolve()
-
- now = ctx.get_time()
- date_str = ctx.format_date()
-
- print(f"\n⏰ 当前时间: {now.strftime('%Y-%m-%d %H:%M:%S')} ({ctx.timezone})")
- print(f"📅 当前日期: {date_str}")
-
- print(f"\n📋 调度信息:")
- print(f" 日计划: {schedule.day_plan}")
- if schedule.period_key:
- print(f" 当前时间段: {schedule.period_name or schedule.period_key} ({schedule.period_key})")
- else:
- print(f" 当前时间段: 无(使用默认配置)")
-
- print(f"\n🔧 行为开关:")
- print(f" 采集数据: {'✅ 是' if schedule.collect else '❌ 否'}")
- print(f" AI 分析: {'✅ 是' if schedule.analyze else '❌ 否'}")
- print(f" 推送通知: {'✅ 是' if schedule.push else '❌ 否'}")
- print(f" 报告模式: {schedule.report_mode}")
- print(f" AI 模式: {schedule.ai_mode}")
-
- if schedule.period_key:
- print(f"\n🔁 一次性控制:")
- if schedule.once_analyze:
- already_analyzed = scheduler.already_executed(schedule.period_key, "analyze", date_str)
- print(f" AI 分析: 仅一次 {'(今日已执行 ⚠️)' if already_analyzed else '(今日未执行 ✅)'}")
- else:
- print(f" AI 分析: 不限次数")
- if schedule.once_push:
- already_pushed = scheduler.already_executed(schedule.period_key, "push", date_str)
- print(f" 推送通知: 仅一次 {'(今日已执行 ⚠️)' if already_pushed else '(今日未执行 ✅)'}")
- else:
- print(f" 推送通知: 不限次数")
-
- except Exception as e:
- print(f"\n❌ 获取调度状态失败: {e}")
-
- print("\n" + "=" * 60)
-
- # 清理资源
- ctx.cleanup()
-
-
if __name__ == "__main__":
main()
diff --git a/trendradar/ai/__init__.py b/trendradar/ai/__init__.py
index ff28d7cff2..845936c1c7 100644
--- a/trendradar/ai/__init__.py
+++ b/trendradar/ai/__init__.py
@@ -7,6 +7,7 @@ TrendRadar AI 模块
from .analyzer import AIAnalyzer, AIAnalysisResult
from .filter import AIFilter, AIFilterResult
+from .filter_pipeline import AIFilterPipeline
from .translator import AITranslator, TranslationResult, BatchTranslationResult
from .formatter import (
get_ai_analysis_renderer,
@@ -24,6 +25,7 @@ __all__ = [
# 智能筛选
"AIFilter",
"AIFilterResult",
+ "AIFilterPipeline",
# 翻译器
"AITranslator",
"TranslationResult",
diff --git a/trendradar/ai/analyzer.py b/trendradar/ai/analyzer.py
index 99ad268a99..7408afd1b3 100644
--- a/trendradar/ai/analyzer.py
+++ b/trendradar/ai/analyzer.py
@@ -8,7 +8,7 @@ AI 分析器模块
import json
from dataclasses import dataclass, field
-from typing import Any, Callable, Dict, List, Optional
+from typing import Any, Callable, Dict, List, NamedTuple, Optional
from trendradar.ai.client import AIClient
from trendradar.ai.prompt_loader import load_prompt_template
@@ -45,6 +45,16 @@ class AIAnalysisResult:
include_standalone: bool = False # 是否启用独立展示区分析
+class PreparedNewsContent(NamedTuple):
+ news_content: str
+ rss_content: str
+ hotlist_total: int
+ rss_total: int
+ analyzed_count: int
+ hotlist_analyzed: int
+ rss_analyzed: int
+
+
class AIAnalyzer:
"""AI 分析器"""
@@ -139,17 +149,17 @@ class AIAnalyzer:
)
# 准备新闻内容并获取统计数据
- news_content, rss_content, hotlist_total, rss_total, analyzed_count, hotlist_analyzed, rss_analyzed = self._prepare_news_content(stats, rss_stats)
- total_news = hotlist_total + rss_total
+ prepared = self._prepare_news_content(stats, rss_stats)
+ total_news = prepared.hotlist_total + prepared.rss_total
- if not news_content and not rss_content:
+ if not prepared.news_content and not prepared.rss_content:
return AIAnalysisResult(
success=False,
skipped=True,
error="本轮无新增热点内容,跳过 AI 分析",
total_news=total_news,
- hotlist_count=hotlist_total,
- rss_count=rss_total,
+ hotlist_count=prepared.hotlist_total,
+ rss_count=prepared.rss_total,
analyzed_news=0,
max_news_limit=self.max_news
)
@@ -166,12 +176,12 @@ class AIAnalyzer:
user_prompt = user_prompt.replace("{report_mode}", report_mode)
user_prompt = user_prompt.replace("{report_type}", report_type)
user_prompt = user_prompt.replace("{current_time}", current_time)
- user_prompt = user_prompt.replace("{news_count}", str(hotlist_total))
- user_prompt = user_prompt.replace("{rss_count}", str(rss_total))
+ user_prompt = user_prompt.replace("{news_count}", str(prepared.hotlist_total))
+ user_prompt = user_prompt.replace("{rss_count}", str(prepared.rss_total))
user_prompt = user_prompt.replace("{platforms}", ", ".join(platforms) if platforms else "多平台")
user_prompt = user_prompt.replace("{keywords}", ", ".join(keywords[:20]) if keywords else "无")
- user_prompt = user_prompt.replace("{news_content}", news_content)
- user_prompt = user_prompt.replace("{rss_content}", rss_content)
+ user_prompt = user_prompt.replace("{news_content}", prepared.news_content)
+ user_prompt = user_prompt.replace("{rss_content}", prepared.rss_content)
user_prompt = user_prompt.replace("{language}", self.language)
# 构建独立展示区内容
@@ -218,11 +228,11 @@ class AIAnalyzer:
# 填充统计数据
result.total_news = total_news
- result.hotlist_count = hotlist_total
- result.rss_count = rss_total
- result.analyzed_news = analyzed_count
- result.hotlist_analyzed = hotlist_analyzed
- result.rss_analyzed = rss_analyzed
+ result.hotlist_count = prepared.hotlist_total
+ result.rss_count = prepared.rss_total
+ result.analyzed_news = prepared.analyzed_count
+ result.hotlist_analyzed = prepared.hotlist_analyzed
+ result.rss_analyzed = prepared.rss_analyzed
result.standalone_analyzed = standalone_count
result.max_news_limit = self.max_news
result.include_rss = self.include_rss
@@ -246,16 +256,7 @@ class AIAnalyzer:
self,
stats: List[Dict],
rss_stats: Optional[List[Dict]] = None,
- ) -> tuple:
- """
- 准备新闻内容文本(增强版)
-
- 热榜新闻包含:来源、标题、排名范围、时间范围、出现次数
- RSS 包含:来源、标题、发布时间
-
- Returns:
- tuple: (news_content, rss_content, hotlist_total, rss_total, analyzed_count, hotlist_analyzed, rss_analyzed)
- """
+ ) -> PreparedNewsContent:
news_lines = []
rss_lines = []
news_count = 0
@@ -359,7 +360,15 @@ class AIAnalyzer:
rss_content = "\n".join(rss_lines) if rss_lines else ""
total_count = news_count + rss_count
- return news_content, rss_content, hotlist_total, rss_total, total_count, news_count, rss_count
+ return PreparedNewsContent(
+ news_content=news_content,
+ rss_content=rss_content,
+ hotlist_total=hotlist_total,
+ rss_total=rss_total,
+ analyzed_count=total_count,
+ hotlist_analyzed=news_count,
+ rss_analyzed=rss_count,
+ )
def _call_ai(self, user_prompt: str) -> str:
"""调用 AI API(使用 LiteLLM)"""
diff --git a/trendradar/ai/filter_pipeline.py b/trendradar/ai/filter_pipeline.py
new file mode 100644
index 0000000000..f6a2047c58
--- /dev/null
+++ b/trendradar/ai/filter_pipeline.py
@@ -0,0 +1,647 @@
+# coding=utf-8
+"""
+AI 筛选流水线
+
+从 context.py 提取的完整 AI 筛选业务流程:
+标签管理 → 待分类新闻收集 → 批量 AI 分类 → 结果保存 → 报告数据转换
+"""
+
+from typing import Any, Callable, Dict, List, Optional
+
+from trendradar.ai.filter import AIFilter, AIFilterResult
+from trendradar.utils.time import (
+ DEFAULT_TIMEZONE,
+ convert_time_for_display,
+ format_iso_time_friendly,
+ is_within_days,
+)
+
+
+class AIFilterPipeline:
+ """AI 筛选流水线,编排标签提取、批量分类、结果存储的完整流程"""
+
+ def __init__(
+ self,
+ config: Dict[str, Any],
+ storage_manager: Any,
+ get_time_func: Callable,
+ ):
+ self.config = config
+ self.storage = storage_manager
+ self.get_time = get_time_func
+
+ self._ai_config = config.get("AI", {})
+ self._filter_config = config.get("AI_FILTER", {})
+ self._debug = config.get("DEBUG", False)
+
+ rss_config = config.get("RSS", {})
+ self._rss_enabled = rss_config.get("ENABLED", False)
+ self._rss_feeds = rss_config.get("FEEDS", [])
+
+ freshness_config = rss_config.get("FRESHNESS_FILTER", {})
+ self._freshness_enabled = freshness_config.get("ENABLED", True)
+ self._default_max_age_days = freshness_config.get("MAX_AGE_DAYS", 3)
+ self._timezone = config.get("TIMEZONE", DEFAULT_TIMEZONE)
+
+ self._priority_sort_enabled = config.get("FILTER", {}).get("PRIORITY_SORT_ENABLED", False)
+ self._rank_threshold = config.get("RANK_THRESHOLD", 50)
+ self._max_news = config.get("MAX_NEWS_PER_KEYWORD", 0)
+
+ self._feed_max_age_map = self._build_feed_max_age_map()
+
+ def _build_feed_max_age_map(self) -> Dict[str, int]:
+ result = {}
+ for feed_cfg in self._rss_feeds:
+ feed_id = feed_cfg.get("id", "")
+ max_age = feed_cfg.get("max_age_days")
+ if max_age is not None:
+ try:
+ result[feed_id] = int(max_age)
+ except (ValueError, TypeError):
+ pass
+ return result
+
+ def run(self, interests_file: Optional[str] = None) -> Optional[AIFilterResult]:
+ """
+ 执行 AI 智能筛选完整流程
+
+ 1. 读取兴趣描述文件,计算 hash
+ 2. 对比数据库 prompt_hash,决定是否重新提取标签
+ 3. 收集待分类新闻(去重)
+ 4. 按 batch_size 分组调用 AI 分类
+ 5. 保存结果
+ 6. 查询 active 结果,按标签分组返回
+ """
+ filter_config = self._filter_config
+
+ ai_filter = AIFilter(self._ai_config, filter_config, self.get_time, self._debug)
+
+ configured_interests = interests_file or filter_config.get("INTERESTS_FILE")
+ effective_interests_file = configured_interests or "ai_interests.txt"
+
+ if self._debug:
+ print(f"[AI筛选][DEBUG] === 配置信息 ===")
+ print(f"[AI筛选][DEBUG] 存储后端: {self.storage.backend_name}")
+ print(f"[AI筛选][DEBUG] batch_size={filter_config.get('BATCH_SIZE', 200)}, "
+ f"batch_interval={filter_config.get('BATCH_INTERVAL', 5)}")
+ print(f"[AI筛选][DEBUG] interests_file={effective_interests_file}")
+ print(f"[AI筛选][DEBUG] prompt_file={filter_config.get('PROMPT_FILE', 'prompt.txt')}")
+ print(f"[AI筛选][DEBUG] extract_prompt_file={filter_config.get('EXTRACT_PROMPT_FILE', 'extract_prompt.txt')}")
+
+ # 1. 读取兴趣描述
+ interests_content = ai_filter.load_interests_content(configured_interests)
+ if not interests_content:
+ return AIFilterResult(success=False, error="兴趣描述文件为空或不存在")
+
+ current_hash = ai_filter.compute_interests_hash(interests_content, effective_interests_file)
+
+ if self._debug:
+ print(f"[AI筛选][DEBUG] 兴趣描述 hash: {current_hash}")
+ print(f"[AI筛选][DEBUG] 兴趣描述内容 ({len(interests_content)} 字符):\n{interests_content}")
+
+ # 2. 开启批量模式
+ self.storage.begin_batch()
+
+ # 3. 检查提示词是否变更
+ stored_hash = self.storage.get_latest_prompt_hash(interests_file=effective_interests_file)
+
+ if self._debug:
+ print(f"[AI筛选][DEBUG] 数据库存储 hash: {stored_hash}")
+ print(f"[AI筛选][DEBUG] hash 对比: stored={stored_hash} vs current={current_hash} → {'匹配' if stored_hash == current_hash else '不匹配'}")
+
+ if stored_hash != current_hash:
+ self._handle_tag_update(
+ ai_filter, interests_content, current_hash, stored_hash,
+ effective_interests_file, filter_config,
+ )
+
+ # 获取当前 active 标签
+ active_tags = self.storage.get_active_ai_filter_tags(interests_file=effective_interests_file)
+ if self._debug:
+ print(f"[AI筛选][DEBUG] 从数据库获取 active 标签: {len(active_tags)} 个")
+ for t in active_tags:
+ print(f"[AI筛选][DEBUG] id={t['id']} tag={t['tag']} priority={t.get('priority', 9999)} version={t.get('version')} hash={t.get('prompt_hash', '')[:8]}...")
+
+ if not active_tags:
+ self.storage.end_batch()
+ return AIFilterResult(success=False, error="没有可用的标签")
+
+ print(f"[AI筛选] 使用 {len(active_tags)} 个标签")
+
+ # 4. 收集待分类新闻
+ pending_news, pending_rss, all_news, analyzed_hotlist, all_rss, analyzed_rss, freshness_filtered_rss = self._collect_pending_news(effective_interests_file)
+
+ self._print_pending_stats(
+ all_news, analyzed_hotlist, pending_news,
+ all_rss, analyzed_rss, pending_rss, freshness_filtered_rss,
+ )
+
+ total_pending = len(pending_news) + len(pending_rss)
+ if total_pending == 0:
+ print("[AI筛选] 没有新增新闻需要分类")
+
+ # 5. 批量分类
+ total_results, succeeded_news_ids, succeeded_rss_ids = self._classify_batches(
+ ai_filter, pending_news, pending_rss, active_tags, interests_content, filter_config,
+ )
+
+ # 6. 保存结果
+ self._save_results(
+ total_results, succeeded_news_ids, succeeded_rss_ids,
+ effective_interests_file, current_hash,
+ )
+
+ # 7. 结束批量模式
+ self.storage.end_batch()
+
+ # 8. 查询并组装返回结果
+ all_results = self.storage.get_active_ai_filter_results(interests_file=effective_interests_file)
+
+ if self._debug:
+ print(f"[AI筛选][DEBUG] === 最终汇总 ===")
+ print(f"[AI筛选][DEBUG] 数据库 active 分类结果: {len(all_results)} 条")
+ tag_counts: dict = {}
+ for r in all_results:
+ tag_name = r.get("tag", "?")
+ src_type = r.get("source_type", "?")
+ key = f"{tag_name}({src_type})"
+ tag_counts[key] = tag_counts.get(key, 0) + 1
+ for key, count in sorted(tag_counts.items()):
+ print(f"[AI筛选][DEBUG] {key}: {count} 条")
+
+ return self._build_filter_result(all_results, active_tags, total_pending)
+
+ def _handle_tag_update(
+ self,
+ ai_filter: AIFilter,
+ interests_content: str,
+ current_hash: str,
+ stored_hash: Optional[str],
+ effective_interests_file: str,
+ filter_config: Dict,
+ ) -> None:
+ new_version = self.storage.get_latest_ai_filter_tag_version() + 1
+ threshold = filter_config.get("RECLASSIFY_THRESHOLD", 0.6)
+
+ if stored_hash is None:
+ print(f"[AI筛选] 首次运行 ({effective_interests_file}),提取标签...")
+ tags_data = ai_filter.extract_tags(interests_content)
+ if not tags_data:
+ self.storage.end_batch()
+ raise _TagExtractionError()
+ tags_data = _with_ordered_priorities(tags_data, start_priority=1)
+ saved_count = self.storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file)
+ print(f"[AI筛选] 已保存 {saved_count} 个标签 (版本 {new_version})")
+ return
+
+ old_tags = self.storage.get_active_ai_filter_tags(interests_file=effective_interests_file)
+ update_result = ai_filter.update_tags(old_tags, interests_content)
+
+ if update_result is None:
+ print(f"[AI筛选] AI 标签更新失败,回退到重新提取")
+ tags_data = ai_filter.extract_tags(interests_content)
+ if not tags_data:
+ self.storage.end_batch()
+ raise _TagExtractionError()
+ tags_data = _with_ordered_priorities(tags_data, start_priority=1)
+ deprecated_count = self.storage.deprecate_all_ai_filter_tags(interests_file=effective_interests_file)
+ self.storage.clear_analyzed_news(interests_file=effective_interests_file)
+ saved_count = self.storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file)
+ print(f"[AI筛选] 废弃 {deprecated_count} 个旧标签, 保存 {saved_count} 个新标签 (版本 {new_version})")
+ return
+
+ change_ratio = update_result["change_ratio"]
+ keep_tags = update_result["keep"]
+ add_tags = update_result["add"]
+ remove_tags = update_result["remove"]
+
+ if self._debug:
+ print(f"[AI筛选][DEBUG] AI 标签更新: keep={len(keep_tags)}, add={len(add_tags)}, remove={len(remove_tags)}, change_ratio={change_ratio:.2f}, threshold={threshold:.2f}")
+
+ if change_ratio >= threshold:
+ print(f"[AI筛选] 兴趣文件变更: {effective_interests_file} (AI change_ratio={change_ratio:.2f} >= threshold={threshold:.2f} → 全量重分类)")
+ tags_data = ai_filter.extract_tags(interests_content)
+ if not tags_data:
+ self.storage.end_batch()
+ raise _TagExtractionError()
+ tags_data = _with_ordered_priorities(tags_data, start_priority=1)
+ deprecated_count = self.storage.deprecate_all_ai_filter_tags(interests_file=effective_interests_file)
+ self.storage.clear_analyzed_news(interests_file=effective_interests_file)
+ saved_count = self.storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file)
+ print(f"[AI筛选] 废弃 {deprecated_count} 个旧标签, 保存 {saved_count} 个新标签 (版本 {new_version})")
+ else:
+ self._apply_incremental_update(
+ old_tags, keep_tags, add_tags, remove_tags,
+ change_ratio, threshold, new_version, current_hash,
+ effective_interests_file,
+ )
+
+ def _apply_incremental_update(
+ self,
+ old_tags, keep_tags, add_tags, remove_tags,
+ change_ratio, threshold, new_version, current_hash,
+ effective_interests_file,
+ ) -> None:
+ print(f"[AI筛选] 兴趣文件变更: {effective_interests_file} (AI change_ratio={change_ratio:.2f} < threshold={threshold:.2f} → 增量更新)")
+ print(f"[AI筛选] 保留 {len(keep_tags)} 个标签, 新增 {len(add_tags)} 个, 废弃 {len(remove_tags)} 个")
+
+ if remove_tags:
+ remove_set = set(remove_tags)
+ removed_ids = [t["id"] for t in old_tags if t["tag"] in remove_set]
+ if removed_ids:
+ self.storage.deprecate_specific_ai_filter_tags(removed_ids)
+ if self._debug:
+ print(f"[AI筛选][DEBUG] 废弃标签 IDs: {removed_ids}")
+
+ keep_with_priority = []
+ if keep_tags:
+ self.storage.update_ai_filter_tag_descriptions(keep_tags, interests_file=effective_interests_file)
+ keep_with_priority = _with_ordered_priorities(keep_tags, start_priority=1)
+ self.storage.update_ai_filter_tag_priorities(keep_with_priority, interests_file=effective_interests_file)
+
+ if add_tags:
+ add_start = keep_with_priority[-1]["priority"] + 1 if keep_with_priority else 1
+ add_with_priority = _with_ordered_priorities(add_tags, start_priority=add_start)
+ saved_count = self.storage.save_ai_filter_tags(add_with_priority, new_version, current_hash, interests_file=effective_interests_file)
+ if self._debug:
+ print(f"[AI筛选][DEBUG] 新增保存 {saved_count} 个标签")
+
+ self.storage.update_ai_filter_tags_hash(effective_interests_file, current_hash)
+
+ if add_tags:
+ cleared = self.storage.clear_unmatched_analyzed_news(interests_file=effective_interests_file)
+ if cleared > 0:
+ print(f"[AI筛选] 清除 {cleared} 条不匹配记录,将在新标签下重新分析")
+
+ def _collect_pending_news(self, effective_interests_file: str):
+ all_news = self.storage.get_all_news_ids()
+ analyzed_hotlist = self.storage.get_analyzed_news_ids("hotlist", interests_file=effective_interests_file)
+ pending_news = [n for n in all_news if n["id"] not in analyzed_hotlist]
+
+ pending_rss = []
+ freshness_filtered_rss = 0
+ all_rss = []
+ analyzed_rss = set()
+
+ if self._rss_enabled:
+ all_rss = self.storage.get_all_rss_ids()
+
+ fresh_rss = []
+ for n in all_rss:
+ published_at = n.get("published_at", "")
+ feed_id = n.get("source_id", "")
+ max_days = self._feed_max_age_map.get(feed_id, self._default_max_age_days)
+ if self._freshness_enabled and max_days > 0 and published_at:
+ if not is_within_days(published_at, max_days, self._timezone):
+ freshness_filtered_rss += 1
+ continue
+ fresh_rss.append(n)
+
+ analyzed_rss = self.storage.get_analyzed_news_ids("rss", interests_file=effective_interests_file)
+ pending_rss = [n for n in fresh_rss if n["id"] not in analyzed_rss]
+
+ return pending_news, pending_rss, all_news, analyzed_hotlist, all_rss, analyzed_rss, freshness_filtered_rss
+
+ def _print_pending_stats(self, all_news, analyzed_hotlist, pending_news, all_rss, analyzed_rss, pending_rss, freshness_filtered_rss):
+ hotlist_total = len(all_news)
+ hotlist_skipped = len(analyzed_hotlist)
+ hotlist_pending = len(pending_news)
+ print(f"[AI筛选] 热榜: 总计 {hotlist_total} 条, 已分析跳过 {hotlist_skipped} 条, 本次发送AI分析 {hotlist_pending} 条")
+ if self._rss_enabled:
+ rss_total = len(all_rss)
+ rss_skipped = len(analyzed_rss)
+ rss_pending = len(pending_rss)
+ freshness_info = f", 新鲜度过滤 {freshness_filtered_rss} 条" if freshness_filtered_rss > 0 else ""
+ print(f"[AI筛选] RSS: 总计 {rss_total} 条{freshness_info}, 已分析跳过 {rss_skipped} 条, 本次发送AI分析 {rss_pending} 条")
+
+ def _classify_batches(self, ai_filter, pending_news, pending_rss, active_tags, interests_content, filter_config):
+ batch_size = filter_config.get("BATCH_SIZE", 200)
+ batch_interval = filter_config.get("BATCH_INTERVAL", 5)
+ total_results = []
+ batch_count = 0
+
+ succeeded_news_ids = []
+ for i in range(0, len(pending_news), batch_size):
+ if batch_count > 0 and batch_interval > 0:
+ import time
+ print(f"[AI筛选] 批次间隔等待 {batch_interval} 秒...")
+ time.sleep(batch_interval)
+ batch = pending_news[i:i + batch_size]
+ titles_for_ai = [
+ {"id": n["id"], "title": n["title"], "source": n.get("source_name", "")}
+ for n in batch
+ ]
+ batch_results = ai_filter.classify_batch(titles_for_ai, active_tags, interests_content)
+ batch_count += 1
+ if batch_results is None:
+ print(f"[AI筛选] 热榜批次 {i // batch_size + 1}: {len(batch)} 条 → 分类失败,将在下次运行重试")
+ continue
+ for r in batch_results:
+ r["source_type"] = "hotlist"
+ total_results.extend(batch_results)
+ succeeded_news_ids.extend(n["id"] for n in batch)
+ print(f"[AI筛选] 热榜批次 {i // batch_size + 1}: {len(batch)} 条 → {len(batch_results)} 条匹配")
+
+ succeeded_rss_ids = []
+ for i in range(0, len(pending_rss), batch_size):
+ if batch_count > 0 and batch_interval > 0:
+ import time
+ print(f"[AI筛选] 批次间隔等待 {batch_interval} 秒...")
+ time.sleep(batch_interval)
+ batch = pending_rss[i:i + batch_size]
+ titles_for_ai = [
+ {"id": n["id"], "title": n["title"], "source": n.get("source_name", "")}
+ for n in batch
+ ]
+ batch_results = ai_filter.classify_batch(titles_for_ai, active_tags, interests_content)
+ batch_count += 1
+ if batch_results is None:
+ print(f"[AI筛选] RSS 批次 {i // batch_size + 1}: {len(batch)} 条 → 分类失败,将在下次运行重试")
+ continue
+ for r in batch_results:
+ r["source_type"] = "rss"
+ total_results.extend(batch_results)
+ succeeded_rss_ids.extend(n["id"] for n in batch)
+ print(f"[AI筛选] RSS 批次 {i // batch_size + 1}: {len(batch)} 条 → {len(batch_results)} 条匹配")
+
+ return total_results, succeeded_news_ids, succeeded_rss_ids
+
+ def _save_results(self, total_results, succeeded_news_ids, succeeded_rss_ids, effective_interests_file, current_hash):
+ if total_results:
+ saved = self.storage.save_ai_filter_results(total_results)
+ print(f"[AI筛选] 保存 {saved} 条分类结果")
+ if self._debug and saved != len(total_results):
+ print(f"[AI筛选][DEBUG] !! 保存数量不一致: 期望 {len(total_results)}, 实际 {saved}(可能有重复记录被跳过)")
+
+ matched_hotlist_ids = {r["news_item_id"] for r in total_results if r.get("source_type") == "hotlist"}
+ matched_rss_ids = {r["news_item_id"] for r in total_results if r.get("source_type") == "rss"}
+
+ if succeeded_news_ids:
+ self.storage.save_analyzed_news(
+ succeeded_news_ids, "hotlist", effective_interests_file,
+ current_hash, matched_hotlist_ids
+ )
+
+ if succeeded_rss_ids:
+ self.storage.save_analyzed_news(
+ succeeded_rss_ids, "rss", effective_interests_file,
+ current_hash, matched_rss_ids
+ )
+
+ if succeeded_news_ids or succeeded_rss_ids:
+ total_analyzed = len(succeeded_news_ids) + len(succeeded_rss_ids)
+ total_matched = len(matched_hotlist_ids) + len(matched_rss_ids)
+ print(f"[AI筛选] 已记录 {total_analyzed} 条新闻分析状态 (匹配 {total_matched}, 不匹配 {total_analyzed - total_matched})")
+
+ def _build_filter_result(
+ self,
+ raw_results: List[Dict],
+ tags: List[Dict],
+ total_processed: int,
+ ) -> AIFilterResult:
+ tag_priority_map = {}
+ for idx, t in enumerate(tags, start=1):
+ tag_name = str(t.get("tag", "")).strip() if isinstance(t, dict) else ""
+ if not tag_name:
+ continue
+ try:
+ tag_priority_map[tag_name] = int(t.get("priority", idx))
+ except (TypeError, ValueError):
+ tag_priority_map[tag_name] = idx
+
+ tag_groups: Dict[str, Dict] = {}
+ seen_titles: Dict[str, set] = {}
+
+ for r in raw_results:
+ tag_name = r["tag"]
+ if tag_name not in tag_groups:
+ raw_priority = r.get("tag_priority", tag_priority_map.get(tag_name, 9999))
+ try:
+ tag_position = int(raw_priority)
+ except (TypeError, ValueError):
+ tag_position = 9999
+ tag_groups[tag_name] = {
+ "tag": tag_name,
+ "description": r.get("tag_description", ""),
+ "position": tag_position,
+ "count": 0,
+ "items": [],
+ }
+ seen_titles[tag_name] = set()
+
+ title = r["title"]
+ if title in seen_titles[tag_name]:
+ continue
+ seen_titles[tag_name].add(title)
+
+ tag_groups[tag_name]["items"].append({
+ "title": title,
+ "source_id": r.get("source_id", ""),
+ "source_name": r.get("source_name", ""),
+ "url": r.get("url", ""),
+ "mobile_url": r.get("mobile_url", ""),
+ "rank": r.get("rank", 0),
+ "ranks": r.get("ranks", []),
+ "first_time": r.get("first_time", ""),
+ "last_time": r.get("last_time", ""),
+ "count": r.get("count", 1),
+ "relevance_score": r.get("relevance_score", 0),
+ "source_type": r.get("source_type", "hotlist"),
+ })
+ tag_groups[tag_name]["count"] += 1
+
+ if self._priority_sort_enabled:
+ sorted_tags = sorted(
+ tag_groups.values(),
+ key=lambda x: (x.get("position", 9999), -x["count"], x["tag"]),
+ )
+ else:
+ sorted_tags = sorted(
+ tag_groups.values(),
+ key=lambda x: (-x["count"], x.get("position", 9999), x["tag"]),
+ )
+
+ total_matched = sum(t["count"] for t in sorted_tags)
+
+ return AIFilterResult(
+ tags=sorted_tags,
+ total_matched=total_matched,
+ total_processed=total_processed,
+ success=True,
+ )
+
+ def convert_to_report_data(
+ self,
+ ai_filter_result: AIFilterResult,
+ mode: str = "daily",
+ new_titles: Optional[Dict] = None,
+ rss_new_urls: Optional[set] = None,
+ ) -> tuple:
+ """
+ 将 AI 筛选结果转换为与关键词匹配相同的数据结构
+
+ Returns:
+ (hotlist_stats, rss_stats, rss_new_stats)
+ """
+ hotlist_stats = []
+ rss_stats = []
+ rss_new_stats = []
+ min_score = self._filter_config.get("MIN_SCORE", 0)
+
+ latest_time = None
+ if mode == "current":
+ for tag_data in ai_filter_result.tags:
+ for item in tag_data.get("items", []):
+ if item.get("source_type", "hotlist") == "hotlist":
+ last_time = item.get("last_time", "")
+ if last_time and (latest_time is None or last_time > latest_time):
+ latest_time = last_time
+ if latest_time:
+ print(f"[AI筛选] current 模式:最新时间 {latest_time},过滤已下榜新闻")
+
+ filtered_count = 0
+ for tag_data in ai_filter_result.tags:
+ tag_name = tag_data.get("tag", "")
+ items = tag_data.get("items", [])
+ if not items:
+ continue
+
+ hotlist_titles = []
+ rss_titles = []
+
+ for item in items:
+ source_type = item.get("source_type", "hotlist")
+
+ if mode == "current" and latest_time and source_type == "hotlist":
+ if item.get("last_time", "") != latest_time:
+ filtered_count += 1
+ continue
+
+ if min_score > 0:
+ score = item.get("relevance_score", 0)
+ if score < min_score:
+ continue
+
+ first_time = item.get("first_time", "")
+ last_time = item.get("last_time", "")
+ if source_type == "rss":
+ if self._freshness_enabled and first_time:
+ feed_id = item.get("source_id", "")
+ max_days = self._feed_max_age_map.get(feed_id, self._default_max_age_days)
+ if max_days > 0 and not is_within_days(first_time, max_days, self._timezone):
+ continue
+ time_display = format_iso_time_friendly(first_time, self._timezone, include_date=True) if first_time else ""
+ else:
+ if first_time and last_time and first_time != last_time:
+ first_display = convert_time_for_display(first_time)
+ last_display = convert_time_for_display(last_time)
+ time_display = f"[{first_display} ~ {last_display}]"
+ elif first_time:
+ time_display = convert_time_for_display(first_time)
+ else:
+ time_display = ""
+
+ if source_type == "rss":
+ is_new = False
+ if rss_new_urls:
+ item_url = item.get("url", "")
+ is_new = item_url in rss_new_urls if item_url else False
+ else:
+ is_new = False
+ if new_titles:
+ item_source_id = item.get("source_id", "")
+ item_title = item.get("title", "")
+ if item_source_id in new_titles:
+ is_new = item_title in new_titles[item_source_id]
+
+ if mode == "incremental" and not is_new:
+ continue
+
+ title_entry = {
+ "title": item.get("title", ""),
+ "source_name": item.get("source_name", ""),
+ "url": item.get("url", ""),
+ "mobile_url": item.get("mobile_url", ""),
+ "ranks": item.get("ranks", []),
+ "rank_threshold": self._rank_threshold,
+ "count": item.get("count", 1),
+ "is_new": is_new,
+ "time_display": time_display,
+ "matched_keyword": tag_name,
+ }
+
+ if source_type == "rss":
+ rss_titles.append(title_entry)
+ else:
+ hotlist_titles.append(title_entry)
+
+ if hotlist_titles:
+ if self._max_news > 0:
+ hotlist_titles = hotlist_titles[:self._max_news]
+ hotlist_stats.append({
+ "word": tag_name,
+ "count": len(hotlist_titles),
+ "position": tag_data.get("position", 9999),
+ "titles": hotlist_titles,
+ })
+
+ if rss_titles:
+ if self._max_news > 0:
+ rss_titles = rss_titles[:self._max_news]
+ rss_stats.append({
+ "word": tag_name,
+ "count": len(rss_titles),
+ "position": tag_data.get("position", 9999),
+ "titles": rss_titles,
+ })
+ new_rss_titles = [t for t in rss_titles if t.get("is_new")]
+ if new_rss_titles:
+ rss_new_stats.append({
+ "word": tag_name,
+ "count": len(new_rss_titles),
+ "position": tag_data.get("position", 9999),
+ "titles": new_rss_titles,
+ })
+
+ if mode == "current" and filtered_count > 0:
+ total_kept = sum(s["count"] for s in hotlist_stats)
+ print(f"[AI筛选] current 模式:过滤 {filtered_count} 条已下榜新闻,保留 {total_kept} 条当前在榜")
+
+ if min_score > 0:
+ hotlist_kept = sum(s["count"] for s in hotlist_stats)
+ rss_kept = sum(s["count"] for s in rss_stats)
+ total_kept = hotlist_kept + rss_kept
+ parts = [f"热榜 {hotlist_kept} 条"]
+ if rss_kept > 0:
+ parts.append(f"RSS {rss_kept} 条")
+ print(f"[AI筛选] 分数过滤:min_score={min_score},保留 {total_kept} 条 score≥{min_score} ({', '.join(parts)})")
+
+ sort_key_priority = lambda x: (x.get("position", 9999), -x["count"], x["word"])
+ sort_key_count = lambda x: (-x["count"], x.get("position", 9999), x["word"])
+ sort_key = sort_key_priority if self._priority_sort_enabled else sort_key_count
+ hotlist_stats.sort(key=sort_key)
+ rss_stats.sort(key=sort_key)
+ rss_new_stats.sort(key=sort_key)
+
+ return hotlist_stats, rss_stats, rss_new_stats
+
+
+class _TagExtractionError(Exception):
+ pass
+
+
+def _with_ordered_priorities(tags: List[Dict], start_priority: int = 1) -> List[Dict]:
+ normalized: List[Dict] = []
+ priority = start_priority
+ for tag_data in tags:
+ if not isinstance(tag_data, dict):
+ continue
+ tag_name = str(tag_data.get("tag", "")).strip()
+ if not tag_name:
+ continue
+ item = dict(tag_data)
+ item["tag"] = tag_name
+ item["priority"] = priority
+ normalized.append(item)
+ priority += 1
+ return normalized
diff --git a/trendradar/ai/formatter.py b/trendradar/ai/formatter.py
index bf24eb1892..1b15927536 100644
--- a/trendradar/ai/formatter.py
+++ b/trendradar/ai/formatter.py
@@ -65,19 +65,36 @@ def _format_list_content(text: str) -> str:
return result
-def _format_standalone_summaries(summaries: dict) -> str:
- """格式化独立展示区概括为纯文本行,每个源名称单独一行"""
+def _format_standalone_summaries(
+ summaries: dict, bracket_left: str = "[", bracket_right: str = "]"
+) -> str:
+ """格式化独立展示区概括为纯文本行,每个源名称单独一行
+
+ Args:
+ summaries: 源名称 -> 概括文本 的字典
+ bracket_left/bracket_right: 源名称两侧的方括号字符。飞书卡片 2.0 markdown
+ 基于 CommonMark,裸 ``[源名]:`` 会被当作「链接引用定义」整段吞掉,
+ 故飞书须传入 HTML 实体 ``&#91;`` ``&#93;``(其余渠道用默认裸方括号)。
+ """
if not summaries:
return ""
lines = []
for source_name, summary in summaries.items():
if summary:
- lines.append(f"[{source_name}]:\n{summary}")
+ lines.append(f"{bracket_left}{source_name}{bracket_right}:\n{summary}")
return "\n\n".join(lines)
-def render_ai_analysis_markdown(result: AIAnalysisResult) -> str:
- """渲染为通用 Markdown 格式(Telegram、企业微信、ntfy、Bark、Slack)"""
+def _render_ai_analysis_markdown_like(
+ result: AIAnalysisResult, standalone_brackets=("[", "]")
+) -> str:
+ """Markdown 系渠道的通用渲染骨架(飞书 / 企业微信 / ntfy / Slack 共用)
+
+ Args:
+ standalone_brackets: 独立源点速览中源名两侧的括号字符。飞书卡片 markdown
+ 须用 HTML 实体 ``("&#91;", "&#93;")`` 避免源名被「链接引用定义」吞掉,
+ 其余渠道沿用默认裸方括号。
+ """
if not result.success:
if result.skipped:
return f"ℹ️ {result.error}"
@@ -107,49 +124,30 @@ def render_ai_analysis_markdown(result: AIAnalysisResult) -> str:
)
if result.standalone_summaries:
- summaries_text = _format_standalone_summaries(result.standalone_summaries)
+ summaries_text = _format_standalone_summaries(
+ result.standalone_summaries, *standalone_brackets
+ )
if summaries_text:
lines.extend(["**独立源点速览**", summaries_text])
return "\n".join(lines)
-def render_ai_analysis_feishu(result: AIAnalysisResult) -> str:
- """渲染为飞书卡片 Markdown 格式"""
- if not result.success:
- if result.skipped:
- return f"ℹ️ {result.error}"
- return f"⚠️ AI 分析失败: {result.error}"
-
- lines = ["**✨ AI 热点分析**", ""]
-
- if result.core_trends:
- lines.extend(["**核心热点态势**", _format_list_content(result.core_trends), ""])
-
- if result.sentiment_controversy:
- lines.extend(
- ["**舆论风向争议**", _format_list_content(result.sentiment_controversy), ""]
- )
-
- if result.signals:
- lines.extend(["**异动与弱信号**", _format_list_content(result.signals), ""])
-
- if result.rss_insights:
- lines.extend(
- ["**RSS 深度洞察**", _format_list_content(result.rss_insights), ""]
- )
+def render_ai_analysis_markdown(result: AIAnalysisResult) -> str:
+ """渲染为通用 Markdown 格式(企业微信、ntfy、Slack)"""
+ return _render_ai_analysis_markdown_like(result)
- if result.outlook_strategy:
- lines.extend(
- ["**研判策略建议**", _format_list_content(result.outlook_strategy), ""]
- )
- if result.standalone_summaries:
- summaries_text = _format_standalone_summaries(result.standalone_summaries)
- if summaries_text:
- lines.extend(["**独立源点速览**", summaries_text])
+def render_ai_analysis_feishu(result: AIAnalysisResult) -> str:
+ """渲染为飞书卡片 2.0 markdown 格式
- return "\n".join(lines)
+ 飞书卡片 markdown 基于 CommonMark,裸 ``[源名]:`` 会被解析为「链接引用定义」
+ (link reference definition) 而整段不显示,故独立源点速览的源名改用 HTML 实体
+ 方括号 ``&#91;`` ``&#93;``(与 report/formatter.py 标题来源标签的处理一致)。
+ """
+ return _render_ai_analysis_markdown_like(
+ result, standalone_brackets=("&#91;", "&#93;")
+ )
def render_ai_analysis_dingtalk(result: AIAnalysisResult) -> str:
diff --git a/trendradar/commands/__init__.py b/trendradar/commands/__init__.py
new file mode 100644
index 0000000000..e3cc7d6a75
--- /dev/null
+++ b/trendradar/commands/__init__.py
@@ -0,0 +1,18 @@
+# coding=utf-8
+"""
+CLI 命令模块
+
+独立的 CLI 子命令:doctor、test-notification、show-schedule、version-check
+"""
+
+from .doctor import run_doctor
+from .test_notification import run_test_notification
+from .status import handle_status_commands
+from .version import check_all_versions
+
+__all__ = [
+ "run_doctor",
+ "run_test_notification",
+ "handle_status_commands",
+ "check_all_versions",
+]
diff --git a/trendradar/commands/doctor.py b/trendradar/commands/doctor.py
new file mode 100644
index 0000000000..62f0eb05a2
--- /dev/null
+++ b/trendradar/commands/doctor.py
@@ -0,0 +1,263 @@
+# coding=utf-8
+"""环境体检命令"""
+
+import json
+import os
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+
+from trendradar import __version__
+from trendradar.context import AppContext
+from trendradar.core import load_config, parse_multi_account_config, validate_paired_configs
+
+
+def _record_result(results: List[Tuple[str, str, str]], status: str, item: str, detail: str) -> None:
+ icon_map = {"pass": "✅", "warn": "⚠️", "fail": "❌"}
+ icon = icon_map.get(status, "•")
+ results.append((status, item, detail))
+ print(f"{icon} {item}: {detail}")
+
+
+def _save_report(
+ results: List[Tuple[str, str, str]],
+ pass_count: int,
+ warn_count: int,
+ fail_count: int,
+ config_path: Optional[str],
+) -> None:
+ report = {
+ "version": __version__,
+ "generated_at": datetime.now(timezone.utc).isoformat(),
+ "config_path": config_path or os.environ.get("CONFIG_PATH", "config/config.yaml"),
+ "summary": {
+ "pass": pass_count,
+ "warn": warn_count,
+ "fail": fail_count,
+ "ok": fail_count == 0,
+ },
+ "checks": [
+ {"status": status, "item": item, "detail": detail}
+ for status, item, detail in results
+ ],
+ }
+
+ try:
+ output_dir = Path("output") / "meta"
+ output_dir.mkdir(parents=True, exist_ok=True)
+ output_path = output_dir / "doctor_report.json"
+ output_path.write_text(
+ json.dumps(report, ensure_ascii=False, indent=2),
+ encoding="utf-8",
+ )
+ print(f"体检报告已保存: {output_path}")
+ except Exception as e:
+ print(f"⚠️ 体检报告保存失败: {e}")
+
+
+def run_doctor(config_path: Optional[str] = None) -> bool:
+ """运行环境体检"""
+ print("=" * 60)
+ print(f"TrendRadar v{__version__} 环境体检")
+ print("=" * 60)
+
+ results: List[Tuple[str, str, str]] = []
+ config = None
+
+ # 1) Python 版本检查
+ py_ok = sys.version_info >= (3, 10)
+ py_version = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}"
+ if py_ok:
+ _record_result(results, "pass", "Python版本", f"{py_version} (满足 >= 3.10)")
+ else:
+ _record_result(results, "fail", "Python版本", f"{py_version} (不满足 >= 3.10)")
+
+ # 2) 关键文件检查
+ if config_path is None:
+ config_path = os.environ.get("CONFIG_PATH", "config/config.yaml")
+
+ required_files = [
+ (config_path, "主配置文件"),
+ ("config/frequency_words.txt", "关键词文件"),
+ ]
+ optional_files = [
+ ("config/timeline.yaml", "调度文件"),
+ ]
+
+ for path_str, desc in required_files:
+ if Path(path_str).exists():
+ _record_result(results, "pass", desc, f"已找到: {path_str}")
+ else:
+ _record_result(results, "fail", desc, f"缺失: {path_str}")
+
+ for path_str, desc in optional_files:
+ if Path(path_str).exists():
+ _record_result(results, "pass", desc, f"已找到: {path_str}")
+ else:
+ _record_result(results, "warn", desc, f"未找到: {path_str}(将使用默认调度模板)")
+
+ # 3) 配置加载检查
+ try:
+ config = load_config(config_path)
+ _record_result(results, "pass", "配置加载", f"加载成功: {config_path}")
+ except Exception as e:
+ _record_result(results, "fail", "配置加载", f"加载失败: {e}")
+
+ if config:
+ _check_with_config(results, config)
+
+ pass_count = sum(1 for status, _, _ in results if status == "pass")
+ warn_count = sum(1 for status, _, _ in results if status == "warn")
+ fail_count = sum(1 for status, _, _ in results if status == "fail")
+
+ _save_report(results, pass_count, warn_count, fail_count, config_path)
+
+ print("-" * 60)
+ print(f"体检结果: ✅ {pass_count} 项通过 ⚠️ {warn_count} 项警告 ❌ {fail_count} 项失败")
+ print("=" * 60)
+
+ if fail_count == 0:
+ print("体检通过。")
+ return True
+
+ print("体检未通过,请先修复失败项。")
+ return False
+
+
+def _check_with_config(results: List[Tuple[str, str, str]], config: Dict) -> None:
+ # 4) 调度配置检查
+ try:
+ ctx = AppContext(config)
+ schedule = ctx.create_scheduler().resolve()
+ detail = f"调度解析成功(report_mode={schedule.report_mode}, ai_mode={schedule.ai_mode})"
+ _record_result(results, "pass", "调度配置", detail)
+ except Exception as e:
+ _record_result(results, "fail", "调度配置", f"解析失败: {e}")
+
+ # 5) AI 配置检查
+ ai_analysis_enabled = config.get("AI_ANALYSIS", {}).get("ENABLED", False)
+ ai_translation_enabled = config.get("AI_TRANSLATION", {}).get("ENABLED", False)
+ ai_filter_enabled = config.get("FILTER", {}).get("METHOD", "keyword") == "ai"
+ ai_enabled = ai_analysis_enabled or ai_translation_enabled or ai_filter_enabled
+
+ if ai_enabled:
+ try:
+ from trendradar.ai.client import AIClient
+ valid, message = AIClient(config.get("AI", {})).validate_config()
+ if valid:
+ _record_result(results, "pass", "AI配置", f"模型: {config.get('AI', {}).get('MODEL', '')}")
+ else:
+ if ai_analysis_enabled or ai_translation_enabled:
+ _record_result(results, "fail", "AI配置", message)
+ else:
+ _record_result(results, "warn", "AI配置", f"{message}(AI 筛选将回退关键词模式)")
+ except Exception as e:
+ _record_result(results, "fail", "AI配置", f"校验异常: {e}")
+ else:
+ _record_result(results, "warn", "AI配置", "未启用 AI 功能,跳过校验")
+
+ # 6) 存储配置检查
+ try:
+ storage_cfg = config.get("STORAGE", {})
+ backend = storage_cfg.get("BACKEND", "auto")
+ remote = storage_cfg.get("REMOTE", {})
+ missing_remote_keys = [
+ k for k in ("BUCKET_NAME", "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "ENDPOINT_URL")
+ if not remote.get(k)
+ ]
+
+ if backend == "remote" and missing_remote_keys:
+ _record_result(
+ results, "fail", "存储配置",
+ f"remote 模式缺少配置: {', '.join(missing_remote_keys)}"
+ )
+ elif backend == "auto" and os.environ.get("GITHUB_ACTIONS") == "true" and missing_remote_keys:
+ _record_result(
+ results, "warn", "存储配置",
+ "GitHub Actions + auto 模式未完整配置远程存储,可能导致数据丢失"
+ )
+ else:
+ sm = AppContext(config).get_storage_manager()
+ _record_result(results, "pass", "存储配置", f"当前后端: {sm.backend_name}")
+ except Exception as e:
+ _record_result(results, "fail", "存储配置", f"检查失败: {e}")
+
+ # 7) 通知渠道配置检查
+ _check_notification_channels(results, config)
+
+ # 8) 输出目录可写检查
+ try:
+ output_dir = Path("output")
+ output_dir.mkdir(parents=True, exist_ok=True)
+ probe_file = output_dir / ".doctor_write_probe"
+ probe_file.write_text("ok", encoding="utf-8")
+ probe_file.unlink(missing_ok=True)
+ _record_result(results, "pass", "输出目录", f"可写: {output_dir}")
+ except Exception as e:
+ _record_result(results, "fail", "输出目录", f"不可写: {e}")
+
+
+def _check_notification_channels(results: List[Tuple[str, str, str]], config: Dict) -> None:
+ channel_details = []
+ channel_issues = []
+ max_accounts = config.get("MAX_ACCOUNTS_PER_CHANNEL", 3)
+
+ for key, name in [
+ ("FEISHU_WEBHOOK_URL", "飞书"),
+ ("DINGTALK_WEBHOOK_URL", "钉钉"),
+ ("WEWORK_WEBHOOK_URL", "企业微信"),
+ ("BARK_URL", "Bark"),
+ ("SLACK_WEBHOOK_URL", "Slack"),
+ ("GENERIC_WEBHOOK_URL", "通用Webhook"),
+ ]:
+ values = parse_multi_account_config(config.get(key, ""))
+ if values:
+ channel_details.append(f"{name}({min(len(values), max_accounts)}个)")
+
+ tg_tokens = parse_multi_account_config(config.get("TELEGRAM_BOT_TOKEN", ""))
+ tg_chats = parse_multi_account_config(config.get("TELEGRAM_CHAT_ID", ""))
+ if tg_tokens or tg_chats:
+ valid, count = validate_paired_configs(
+ {"bot_token": tg_tokens, "chat_id": tg_chats},
+ "Telegram",
+ required_keys=["bot_token", "chat_id"],
+ )
+ if valid and count > 0:
+ channel_details.append(f"Telegram({min(count, max_accounts)}个)")
+ else:
+ channel_issues.append("Telegram bot_token/chat_id 配置不完整或数量不一致")
+
+ ntfy_server = config.get("NTFY_SERVER_URL", "")
+ ntfy_topics = parse_multi_account_config(config.get("NTFY_TOPIC", ""))
+ ntfy_tokens = parse_multi_account_config(config.get("NTFY_TOKEN", ""))
+ if ntfy_server and ntfy_topics:
+ if ntfy_tokens:
+ valid, count = validate_paired_configs(
+ {"topic": ntfy_topics, "token": ntfy_tokens},
+ "ntfy",
+ )
+ if valid and count > 0:
+ channel_details.append(f"ntfy({min(count, max_accounts)}个)")
+ else:
+ channel_issues.append("ntfy topic/token 数量不一致")
+ else:
+ channel_details.append(f"ntfy({min(len(ntfy_topics), max_accounts)}个)")
+
+ email_ready = all(
+ [config.get("EMAIL_FROM"), config.get("EMAIL_PASSWORD"), config.get("EMAIL_TO")]
+ )
+ if email_ready:
+ channel_details.append("邮件")
+ elif any([config.get("EMAIL_FROM"), config.get("EMAIL_PASSWORD"), config.get("EMAIL_TO")]):
+ channel_issues.append("邮件配置不完整(需要 from/password/to 同时配置)")
+
+ if channel_issues and not channel_details:
+ _record_result(results, "fail", "通知配置", ";".join(channel_issues))
+ elif channel_issues and channel_details:
+ detail = f"可用渠道: {', '.join(channel_details)};问题: {';'.join(channel_issues)}"
+ _record_result(results, "warn", "通知配置", detail)
+ elif channel_details:
+ _record_result(results, "pass", "通知配置", f"可用渠道: {', '.join(channel_details)}")
+ else:
+ _record_result(results, "warn", "通知配置", "未配置任何通知渠道")
diff --git a/trendradar/commands/status.py b/trendradar/commands/status.py
new file mode 100644
index 0000000000..5604bab92d
--- /dev/null
+++ b/trendradar/commands/status.py
@@ -0,0 +1,60 @@
+# coding=utf-8
+"""调度状态查看命令"""
+
+from typing import Dict
+
+from trendradar import __version__
+from trendradar.context import AppContext
+
+
+def handle_status_commands(config: Dict) -> None:
+ """显示当前调度状态"""
+ ctx = AppContext(config)
+
+ print("=" * 60)
+ print(f"TrendRadar v{__version__} 调度状态")
+ print("=" * 60)
+
+ try:
+ scheduler = ctx.create_scheduler()
+ schedule = scheduler.resolve()
+
+ now = ctx.get_time()
+ date_str = ctx.format_date()
+
+ print(f"\n⏰ 当前时间: {now.strftime('%Y-%m-%d %H:%M:%S')} ({ctx.timezone})")
+ print(f"📅 当前日期: {date_str}")
+
+ print(f"\n📋 调度信息:")
+ print(f" 日计划: {schedule.day_plan}")
+ if schedule.period_key:
+ print(f" 当前时间段: {schedule.period_name or schedule.period_key} ({schedule.period_key})")
+ else:
+ print(f" 当前时间段: 无(使用默认配置)")
+
+ print(f"\n🔧 行为开关:")
+ print(f" 采集数据: {'✅ 是' if schedule.collect else '❌ 否'}")
+ print(f" AI 分析: {'✅ 是' if schedule.analyze else '❌ 否'}")
+ print(f" 推送通知: {'✅ 是' if schedule.push else '❌ 否'}")
+ print(f" 报告模式: {schedule.report_mode}")
+ print(f" AI 模式: {schedule.ai_mode}")
+
+ if schedule.period_key:
+ print(f"\n🔁 一次性控制:")
+ if schedule.once_analyze:
+ already_analyzed = scheduler.already_executed(schedule.period_key, "analyze", date_str)
+ print(f" AI 分析: 仅一次 {'(今日已执行 ⚠️)' if already_analyzed else '(今日未执行 ✅)'}")
+ else:
+ print(f" AI 分析: 不限次数")
+ if schedule.once_push:
+ already_pushed = scheduler.already_executed(schedule.period_key, "push", date_str)
+ print(f" 推送通知: 仅一次 {'(今日已执行 ⚠️)' if already_pushed else '(今日未执行 ✅)'}")
+ else:
+ print(f" 推送通知: 不限次数")
+
+ except Exception as e:
+ print(f"\n❌ 获取调度状态失败: {e}")
+
+ print("\n" + "=" * 60)
+
+ ctx.cleanup()
diff --git a/trendradar/commands/test_notification.py b/trendradar/commands/test_notification.py
new file mode 100644
index 0000000000..6464ce80bb
--- /dev/null
+++ b/trendradar/commands/test_notification.py
@@ -0,0 +1,148 @@
+# coding=utf-8
+"""通知测试命令"""
+
+import copy
+from pathlib import Path
+from typing import Dict, Optional
+
+from trendradar.context import AppContext
+
+
+def _build_test_report_data(ctx: AppContext) -> Dict:
+ now = ctx.get_time()
+ time_display = now.strftime("%H:%M")
+ title = f"TrendRadar 通知测试消息({now.strftime('%Y-%m-%d %H:%M:%S')})"
+
+ return {
+ "stats": [
+ {
+ "word": "连通性测试",
+ "count": 1,
+ "titles": [
+ {
+ "title": title,
+ "source_name": "TrendRadar",
+ "url": "https://github.com/sansan0/TrendRadar",
+ "mobile_url": "",
+ "ranks": [1],
+ "rank_threshold": ctx.rank_threshold,
+ "count": 1,
+ "is_new": True,
+ "time_display": time_display,
+ "matched_keyword": "连通性测试",
+ }
+ ],
+ }
+ ],
+ "failed_ids": [],
+ "new_titles": [],
+ "id_to_name": {},
+ }
+
+
+def _create_test_html_file(ctx: AppContext) -> Optional[str]:
+ try:
+ now = ctx.get_time()
+ output_dir = Path("output") / "html" / ctx.format_date()
+ output_dir.mkdir(parents=True, exist_ok=True)
+ html_path = output_dir / f"notification_test_{ctx.format_time()}.html"
+ html_content = f"""<!DOCTYPE html>
+<html lang="zh-CN">
+<head><meta charset="UTF-8"><title>TrendRadar 通知测试</title></head>
+<body>
+<h2>TrendRadar 通知连通性测试</h2>
+<p>测试时间:{now.strftime('%Y-%m-%d %H:%M:%S')} ({ctx.timezone})</p>
+<p>这是一条测试消息,用于验证邮件渠道是否可达。</p>
+</body>
+</html>"""
+ html_path.write_text(html_content, encoding="utf-8")
+ return str(html_path)
+ except Exception as e:
+ print(f"[测试通知] 创建测试 HTML 失败: {e}")
+ return None
+
+
+def run_test_notification(config: Dict) -> bool:
+ """发送测试通知到已配置渠道"""
+ from trendradar.notification import NotificationDispatcher
+
+ ctx = AppContext(config)
+
+ try:
+ has_notification = any(
+ [
+ config.get("FEISHU_WEBHOOK_URL"),
+ config.get("DINGTALK_WEBHOOK_URL"),
+ config.get("WEWORK_WEBHOOK_URL"),
+ (config.get("TELEGRAM_BOT_TOKEN") and config.get("TELEGRAM_CHAT_ID")),
+ (config.get("EMAIL_FROM") and config.get("EMAIL_PASSWORD") and config.get("EMAIL_TO")),
+ (config.get("NTFY_SERVER_URL") and config.get("NTFY_TOPIC")),
+ config.get("BARK_URL"),
+ config.get("SLACK_WEBHOOK_URL"),
+ config.get("GENERIC_WEBHOOK_URL"),
+ ]
+ )
+ if not has_notification:
+ print("未检测到可用通知渠道,请先在 config.yaml 或环境变量中配置。")
+ return False
+
+ test_config = copy.deepcopy(config)
+ test_display = test_config.setdefault("DISPLAY", {})
+ test_regions = test_display.setdefault("REGIONS", {})
+ test_regions.update(
+ {
+ "HOTLIST": True,
+ "NEW_ITEMS": False,
+ "RSS": False,
+ "STANDALONE": False,
+ "AI_ANALYSIS": False,
+ }
+ )
+
+ if "AI_TRANSLATION" in test_config:
+ test_config["AI_TRANSLATION"]["ENABLED"] = False
+
+ proxy_url = test_config.get("DEFAULT_PROXY", "") if test_config.get("USE_PROXY") else None
+ if proxy_url:
+ print("[测试通知] 检测到代理配置,将使用代理发送")
+
+ dispatcher = NotificationDispatcher(
+ config=test_config,
+ get_time_func=ctx.get_time,
+ split_content_func=ctx.split_content,
+ translator=None,
+ )
+
+ report_data = _build_test_report_data(ctx)
+ html_file_path = _create_test_html_file(ctx)
+
+ print("=" * 60)
+ print("通知连通性测试")
+ print("=" * 60)
+
+ results = dispatcher.dispatch_all(
+ report_data=report_data,
+ report_type="通知连通性测试",
+ proxy_url=proxy_url,
+ mode="daily",
+ html_file_path=html_file_path,
+ )
+
+ if not results:
+ print("没有可测试的有效通知渠道(可能配置不完整)。")
+ return False
+
+ print("-" * 60)
+ success_count = 0
+ for channel, ok in results.items():
+ if ok:
+ success_count += 1
+ print(f"✅ {channel}: 测试成功")
+ else:
+ print(f"❌ {channel}: 测试失败")
+
+ print("-" * 60)
+ print(f"测试结果: {success_count}/{len(results)} 个渠道成功")
+ return success_count > 0
+ finally:
+ ctx.cleanup()
diff --git a/trendradar/commands/version.py b/trendradar/commands/version.py
new file mode 100644
index 0000000000..ed60aa226f
--- /dev/null
+++ b/trendradar/commands/version.py
@@ -0,0 +1,138 @@
+# coding=utf-8
+"""版本检查命令"""
+
+import re
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+
+from trendradar import __version__
+from trendradar.core.cdn import fetch_with_fallback
+
+
+def _parse_version(version_str: str) -> Tuple[int, int, int]:
+ try:
+ parts = version_str.strip().split(".")
+ if len(parts) >= 3:
+ return int(parts[0]), int(parts[1]), int(parts[2])
+ return 0, 0, 0
+ except (ValueError, AttributeError, TypeError):
+ return 0, 0, 0
+
+
+def _compare_version(local: str, remote: str) -> str:
+ local_tuple = _parse_version(local)
+ remote_tuple = _parse_version(remote)
+
+ if local_tuple < remote_tuple:
+ return "⚠️ 需要更新"
+ elif local_tuple > remote_tuple:
+ return "🔮 超前版本"
+ else:
+ return "✅ 已是最新"
+
+
+def _fetch_remote_version(version_url: str, proxy_url: Optional[str] = None) -> Optional[str]:
+ return fetch_with_fallback(version_url, proxy_url)
+
+
+def _parse_config_versions(content: str) -> Dict[str, str]:
+ versions = {}
+ try:
+ if not content:
+ return versions
+ for line in content.splitlines():
+ line = line.strip()
+ if not line or "=" not in line:
+ continue
+ name, version = line.split("=", 1)
+ versions[name.strip()] = version.strip()
+ except Exception as e:
+ print(f"[版本检查] 解析配置版本失败: {e}")
+ return versions
+
+
+def check_all_versions(
+ version_url: str,
+ configs_version_url: Optional[str] = None,
+ proxy_url: Optional[str] = None
+) -> Tuple[bool, Optional[str]]:
+ """
+ 统一版本检查:程序版本 + 配置文件版本
+
+ Returns:
+ (need_update, remote_version): 程序是否需要更新及远程版本号
+ """
+ remote_version = _fetch_remote_version(version_url, proxy_url)
+
+ remote_config_versions = {}
+ if configs_version_url:
+ content = _fetch_remote_version(configs_version_url, proxy_url)
+ if content:
+ remote_config_versions = _parse_config_versions(content)
+
+ print("=" * 60)
+ print("版本检查")
+ print("=" * 60)
+
+ if remote_version:
+ print(f"远程程序版本: {remote_version}")
+ else:
+ print("远程程序版本: 获取失败")
+
+ if configs_version_url:
+ if remote_config_versions:
+ print(f"远程配置清单: 获取成功 ({len(remote_config_versions)} 个文件)")
+ else:
+ print("远程配置清单: 获取失败或为空")
+
+ print("-" * 60)
+
+ program_status = _compare_version(__version__, remote_version) if remote_version else "(无法比较)"
+ print(f" 主程序版本: {__version__} {program_status}")
+
+ config_files = [
+ Path("config/config.yaml"),
+ Path("config/timeline.yaml"),
+ Path("config/frequency_words.txt"),
+ Path("config/ai_interests.txt"),
+ Path("config/ai_analysis_prompt.txt"),
+ Path("config/ai_translation_prompt.txt"),
+ ]
+
+ version_pattern = re.compile(r"Version:\s*(\d+\.\d+\.\d+)", re.IGNORECASE)
+
+ for config_file in config_files:
+ if not config_file.exists():
+ print(f" {config_file.name}: 文件不存在")
+ continue
+
+ try:
+ with open(config_file, "r", encoding="utf-8") as f:
+ local_version = None
+ for i, line in enumerate(f):
+ if i >= 20:
+ break
+ match = version_pattern.search(line)
+ if match:
+ local_version = match.group(1)
+ break
+
+ target_remote_version = remote_config_versions.get(config_file.name)
+
+ if local_version:
+ if target_remote_version:
+ status = _compare_version(local_version, target_remote_version)
+ print(f" {config_file.name}: {local_version} {status}")
+ else:
+ print(f" {config_file.name}: {local_version} (未找到远程版本)")
+ else:
+ print(f" {config_file.name}: 未找到本地版本号")
+ except Exception as e:
+ print(f" {config_file.name}: 读取失败 - {e}")
+
+ print("=" * 60)
+
+ if remote_version:
+ need_update = _parse_version(__version__) < _parse_version(remote_version)
+ return need_update, remote_version if need_update else None
+ return False, None
diff --git a/trendradar/context.py b/trendradar/context.py
index d9c43d1ad5..79bc2a9819 100644
--- a/trendradar/context.py
+++ b/trendradar/context.py
@@ -39,7 +39,8 @@ from trendradar.notification import (
NotificationDispatcher,
)
from trendradar.ai import AITranslator
-from trendradar.ai.filter import AIFilter, AIFilterResult
+from trendradar.ai.filter import AIFilterResult
+from trendradar.ai.filter_pipeline import AIFilterPipeline, _TagExtractionError
from trendradar.storage import get_storage_manager
@@ -498,436 +499,21 @@ class AppContext:
# === AI 智能筛选 ===
- @staticmethod
- def _with_ordered_priorities(tags: List[Dict], start_priority: int = 1) -> List[Dict]:
- """按当前列表顺序补齐优先级(值越小优先级越高)"""
- normalized: List[Dict] = []
- priority = start_priority
- for tag_data in tags:
- if not isinstance(tag_data, dict):
- continue
- tag_name = str(tag_data.get("tag", "")).strip()
- if not tag_name:
- continue
- item = dict(tag_data)
- item["tag"] = tag_name
- item["priority"] = priority
- normalized.append(item)
- priority += 1
- return normalized
+ def _get_ai_filter_pipeline(self) -> "AIFilterPipeline":
+ return AIFilterPipeline(
+ config=self.config,
+ storage_manager=self.get_storage_manager(),
+ get_time_func=self.get_time,
+ )
def run_ai_filter(self, interests_file: Optional[str] = None) -> Optional[AIFilterResult]:
- """
- 执行 AI 智能筛选完整流程
-
- Args:
- interests_file: 兴趣描述文件名(位于 config/custom/ai/),None=使用默认 config/ai_interests.txt
-
- 1. 读取兴趣描述文件,计算 hash
- 2. 对比数据库 prompt_hash,决定是否重新提取标签
- 3. 收集待分类新闻(去重)
- 4. 按 batch_size 分组调用 AI 分类
- 5. 保存结果
- 6. 查询 active 结果,按标签分组返回
-
- Returns:
- AIFilterResult 或 None(未启用或出错)
- """
+ """执行 AI 智能筛选完整流程"""
if not self.ai_filter_enabled:
return None
-
- filter_config = self.ai_filter_config
- ai_config = self.config.get("AI", {})
- debug = self.config.get("DEBUG", False)
-
- # 创建 AIFilter 实例
- ai_filter = AIFilter(ai_config, filter_config, self.get_time, debug)
-
- # 确定实际使用的兴趣文件名
- # None = 使用默认 config/ai_interests.txt,指定文件名 = config/custom/ai/{name}
- configured_interests = interests_file or filter_config.get("INTERESTS_FILE")
- effective_interests_file = configured_interests or "ai_interests.txt"
-
- if debug:
- print(f"[AI筛选][DEBUG] === 配置信息 ===")
- print(f"[AI筛选][DEBUG] 存储后端: {self.get_storage_manager().backend_name}")
- print(f"[AI筛选][DEBUG] batch_size={filter_config.get('BATCH_SIZE', 200)}, "
- f"batch_interval={filter_config.get('BATCH_INTERVAL', 5)}")
- print(f"[AI筛选][DEBUG] interests_file={effective_interests_file}")
- print(f"[AI筛选][DEBUG] prompt_file={filter_config.get('PROMPT_FILE', 'prompt.txt')}")
- print(f"[AI筛选][DEBUG] extract_prompt_file={filter_config.get('EXTRACT_PROMPT_FILE', 'extract_prompt.txt')}")
-
- # 1. 读取兴趣描述
- # 传 configured_interests(可能为 None)给 load_interests_content,
- # 让它区分"默认文件(config/ai_interests.txt)"和"自定义文件(config/custom/ai/)"
- interests_content = ai_filter.load_interests_content(configured_interests)
- if not interests_content:
- return AIFilterResult(success=False, error="兴趣描述文件为空或不存在")
-
- current_hash = ai_filter.compute_interests_hash(interests_content, effective_interests_file)
- storage = self.get_storage_manager()
-
- if debug:
- print(f"[AI筛选][DEBUG] 兴趣描述 hash: {current_hash}")
- print(f"[AI筛选][DEBUG] 兴趣描述内容 ({len(interests_content)} 字符):\n{interests_content}")
-
- # 2. 开启批量模式(远程后端延迟上传,所有写操作完成后统一上传)
- storage.begin_batch()
-
- # 3. 检查提示词是否变更
- stored_hash = storage.get_latest_prompt_hash(interests_file=effective_interests_file)
-
- if debug:
- print(f"[AI筛选][DEBUG] 数据库存储 hash: {stored_hash}")
- print(f"[AI筛选][DEBUG] hash 对比: stored={stored_hash} vs current={current_hash} → {'匹配' if stored_hash == current_hash else '不匹配'}")
-
- if stored_hash != current_hash:
- new_version = storage.get_latest_ai_filter_tag_version() + 1
- threshold = filter_config.get("RECLASSIFY_THRESHOLD", 0.6)
-
- if stored_hash is None:
- # 首次运行,直接提取并保存全部标签
- print(f"[AI筛选] 首次运行 ({effective_interests_file}),提取标签...")
- tags_data = ai_filter.extract_tags(interests_content)
- if not tags_data:
- storage.end_batch()
- return AIFilterResult(success=False, error="标签提取失败")
- tags_data = self._with_ordered_priorities(tags_data, start_priority=1)
- saved_count = storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file)
- print(f"[AI筛选] 已保存 {saved_count} 个标签 (版本 {new_version})")
- else:
- # 兴趣描述已变更,让 AI 对比旧标签和新兴趣,给出更新方案
- old_tags = storage.get_active_ai_filter_tags(interests_file=effective_interests_file)
- update_result = ai_filter.update_tags(old_tags, interests_content)
-
- if update_result is None:
- # AI 标签更新失败,回退到重新提取全部标签
- print(f"[AI筛选] AI 标签更新失败,回退到重新提取")
- tags_data = ai_filter.extract_tags(interests_content)
- if not tags_data:
- storage.end_batch()
- return AIFilterResult(success=False, error="标签提取失败")
- tags_data = self._with_ordered_priorities(tags_data, start_priority=1)
- deprecated_count = storage.deprecate_all_ai_filter_tags(interests_file=effective_interests_file)
- storage.clear_analyzed_news(interests_file=effective_interests_file)
- saved_count = storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file)
- print(f"[AI筛选] 废弃 {deprecated_count} 个旧标签, 保存 {saved_count} 个新标签 (版本 {new_version})")
- else:
- change_ratio = update_result["change_ratio"]
- keep_tags = update_result["keep"]
- add_tags = update_result["add"]
- remove_tags = update_result["remove"]
-
- if debug:
- print(f"[AI筛选][DEBUG] AI 标签更新: keep={len(keep_tags)}, add={len(add_tags)}, remove={len(remove_tags)}, change_ratio={change_ratio:.2f}, threshold={threshold:.2f}")
-
- if change_ratio >= threshold:
- # 全量重分类:废弃所有旧标签,用 extract_tags 重新提取
- print(f"[AI筛选] 兴趣文件变更: {effective_interests_file} (AI change_ratio={change_ratio:.2f} >= threshold={threshold:.2f} → 全量重分类)")
- tags_data = ai_filter.extract_tags(interests_content)
- if not tags_data:
- storage.end_batch()
- return AIFilterResult(success=False, error="标签提取失败")
- tags_data = self._with_ordered_priorities(tags_data, start_priority=1)
- deprecated_count = storage.deprecate_all_ai_filter_tags(interests_file=effective_interests_file)
- storage.clear_analyzed_news(interests_file=effective_interests_file)
- saved_count = storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file)
- print(f"[AI筛选] 废弃 {deprecated_count} 个旧标签, 保存 {saved_count} 个新标签 (版本 {new_version})")
- else:
- # 增量更新:按 AI 指示操作
- print(f"[AI筛选] 兴趣文件变更: {effective_interests_file} (AI change_ratio={change_ratio:.2f} < threshold={threshold:.2f} → 增量更新)")
- print(f"[AI筛选] 保留 {len(keep_tags)} 个标签, 新增 {len(add_tags)} 个, 废弃 {len(remove_tags)} 个")
-
- # 废弃 AI 标记移除的标签
- if remove_tags:
- remove_set = set(remove_tags)
- removed_ids = [t["id"] for t in old_tags if t["tag"] in remove_set]
- if removed_ids:
- storage.deprecate_specific_ai_filter_tags(removed_ids)
- if debug:
- print(f"[AI筛选][DEBUG] 废弃标签 IDs: {removed_ids}")
-
- # 更新保留标签的描述
- keep_with_priority = []
- if keep_tags:
- storage.update_ai_filter_tag_descriptions(keep_tags, interests_file=effective_interests_file)
- keep_with_priority = self._with_ordered_priorities(keep_tags, start_priority=1)
- storage.update_ai_filter_tag_priorities(keep_with_priority, interests_file=effective_interests_file)
-
- # 保存新增标签
- if add_tags:
- add_start = keep_with_priority[-1]["priority"] + 1 if keep_with_priority else 1
- add_with_priority = self._with_ordered_priorities(add_tags, start_priority=add_start)
- saved_count = storage.save_ai_filter_tags(add_with_priority, new_version, current_hash, interests_file=effective_interests_file)
- if debug:
- print(f"[AI筛选][DEBUG] 新增保存 {saved_count} 个标签")
-
- # 更新保留标签的 hash(标记为已处理)
- storage.update_ai_filter_tags_hash(effective_interests_file, current_hash)
-
- # 增量更新:清除不匹配新闻的分析记录,让它们有机会被新标签集重新分析
- if add_tags:
- cleared = storage.clear_unmatched_analyzed_news(interests_file=effective_interests_file)
- if cleared > 0:
- print(f"[AI筛选] 清除 {cleared} 条不匹配记录,将在新标签下重新分析")
-
- # 3. 获取当前 active 标签
- active_tags = storage.get_active_ai_filter_tags(interests_file=effective_interests_file)
- if debug:
- print(f"[AI筛选][DEBUG] 从数据库获取 active 标签: {len(active_tags)} 个")
- for t in active_tags:
- print(f"[AI筛选][DEBUG] id={t['id']} tag={t['tag']} priority={t.get('priority', 9999)} version={t.get('version')} hash={t.get('prompt_hash', '')[:8]}...")
-
- if not active_tags:
- storage.end_batch()
- return AIFilterResult(success=False, error="没有可用的标签")
-
- print(f"[AI筛选] 使用 {len(active_tags)} 个标签")
-
- # 4. 收集待分类新闻
- # 热榜
- all_news = storage.get_all_news_ids()
- analyzed_hotlist = storage.get_analyzed_news_ids("hotlist", interests_file=effective_interests_file)
- pending_news = [n for n in all_news if n["id"] not in analyzed_hotlist]
-
- # RSS(先做新鲜度过滤,再去除已分类的)
- pending_rss = []
- freshness_filtered_rss = 0
- if self.rss_enabled:
- all_rss = storage.get_all_rss_ids()
-
- # 应用新鲜度过滤(与推送阶段一致)
- rss_config = self.rss_config
- freshness_config = rss_config.get("FRESHNESS_FILTER", {})
- freshness_enabled = freshness_config.get("ENABLED", True)
- default_max_age_days = freshness_config.get("MAX_AGE_DAYS", 3)
- timezone = self.config.get("TIMEZONE", DEFAULT_TIMEZONE)
-
- # 构建 feed_id -> max_age_days 的映射
- feed_max_age_map = {}
- for feed_cfg in self.rss_feeds:
- feed_id = feed_cfg.get("id", "")
- max_age = feed_cfg.get("max_age_days")
- if max_age is not None:
- try:
- feed_max_age_map[feed_id] = int(max_age)
- except (ValueError, TypeError):
- pass
-
- fresh_rss = []
- for n in all_rss:
- published_at = n.get("published_at", "")
- feed_id = n.get("source_id", "")
- max_days = feed_max_age_map.get(feed_id, default_max_age_days)
- if freshness_enabled and max_days > 0 and published_at:
- if not is_within_days(published_at, max_days, timezone):
- freshness_filtered_rss += 1
- continue
- fresh_rss.append(n)
-
- analyzed_rss = storage.get_analyzed_news_ids("rss", interests_file=effective_interests_file)
- pending_rss = [n for n in fresh_rss if n["id"] not in analyzed_rss]
-
- # 始终打印总量/已分析/待分析 的详细数据
- hotlist_total = len(all_news)
- hotlist_skipped = len(analyzed_hotlist)
- hotlist_pending = len(pending_news)
- print(f"[AI筛选] 热榜: 总计 {hotlist_total} 条, 已分析跳过 {hotlist_skipped} 条, 本次发送AI分析 {hotlist_pending} 条")
- if self.rss_enabled:
- rss_total = len(all_rss)
- rss_skipped = len(analyzed_rss)
- rss_pending = len(pending_rss)
- freshness_info = f", 新鲜度过滤 {freshness_filtered_rss} 条" if freshness_filtered_rss > 0 else ""
- print(f"[AI筛选] RSS: 总计 {rss_total} 条{freshness_info}, 已分析跳过 {rss_skipped} 条, 本次发送AI分析 {rss_pending} 条")
-
- total_pending = len(pending_news) + len(pending_rss)
- if total_pending == 0:
- print("[AI筛选] 没有新增新闻需要分类")
-
- # 5. 批量分类
- batch_size = filter_config.get("BATCH_SIZE", 200)
- batch_interval = filter_config.get("BATCH_INTERVAL", 5)
- total_results = []
- batch_count = 0 # 跨热榜和 RSS 的全局批次计数
-
- # 处理热榜
- succeeded_news_ids = [] # 成功分类(含无匹配)的热榜 id;仅这些标记已分析,失败批次留待重试
- for i in range(0, len(pending_news), batch_size):
- if batch_count > 0 and batch_interval > 0:
- import time
- print(f"[AI筛选] 批次间隔等待 {batch_interval} 秒...")
- time.sleep(batch_interval)
- batch = pending_news[i:i + batch_size]
- titles_for_ai = [
- {"id": n["id"], "title": n["title"], "source": n.get("source_name", "")}
- for n in batch
- ]
- batch_results = ai_filter.classify_batch(titles_for_ai, active_tags, interests_content)
- batch_count += 1
- if batch_results is None:
- # 调用失败:不标记该批次已分析,留待下次运行重试,避免新闻静默丢失
- print(f"[AI筛选] 热榜批次 {i // batch_size + 1}: {len(batch)} 条 → 分类失败,将在下次运行重试")
- continue
- for r in batch_results:
- r["source_type"] = "hotlist"
- total_results.extend(batch_results)
- succeeded_news_ids.extend(n["id"] for n in batch)
- print(f"[AI筛选] 热榜批次 {i // batch_size + 1}: {len(batch)} 条 → {len(batch_results)} 条匹配")
-
- # 处理 RSS
- succeeded_rss_ids = [] # 成功分类(含无匹配)的 RSS id;仅这些标记已分析,失败批次留待重试
- for i in range(0, len(pending_rss), batch_size):
- if batch_count > 0 and batch_interval > 0:
- import time
- print(f"[AI筛选] 批次间隔等待 {batch_interval} 秒...")
- time.sleep(batch_interval)
- batch = pending_rss[i:i + batch_size]
- titles_for_ai = [
- {"id": n["id"], "title": n["title"], "source": n.get("source_name", "")}
- for n in batch
- ]
- batch_results = ai_filter.classify_batch(titles_for_ai, active_tags, interests_content)
- batch_count += 1
- if batch_results is None:
- # 调用失败:不标记该批次已分析,留待下次运行重试
- print(f"[AI筛选] RSS 批次 {i // batch_size + 1}: {len(batch)} 条 → 分类失败,将在下次运行重试")
- continue
- for r in batch_results:
- r["source_type"] = "rss"
- total_results.extend(batch_results)
- succeeded_rss_ids.extend(n["id"] for n in batch)
- print(f"[AI筛选] RSS 批次 {i // batch_size + 1}: {len(batch)} 条 → {len(batch_results)} 条匹配")
-
- # 6. 保存结果
- if total_results:
- saved = storage.save_ai_filter_results(total_results)
- print(f"[AI筛选] 保存 {saved} 条分类结果")
- if debug and saved != len(total_results):
- print(f"[AI筛选][DEBUG] !! 保存数量不一致: 期望 {len(total_results)}, 实际 {saved}(可能有重复记录被跳过)")
-
- # 6.5 记录已分析的新闻(匹配+不匹配,用于去重)。仅记录成功分类的批次;
- # 失败批次的 id 不写入,使其下次运行重新分类,避免 AI 抖动导致新闻静默丢失
- matched_hotlist_ids = {r["news_item_id"] for r in total_results if r.get("source_type") == "hotlist"}
- matched_rss_ids = {r["news_item_id"] for r in total_results if r.get("source_type") == "rss"}
-
- if succeeded_news_ids:
- storage.save_analyzed_news(
- succeeded_news_ids, "hotlist", effective_interests_file,
- current_hash, matched_hotlist_ids
- )
-
- if succeeded_rss_ids:
- storage.save_analyzed_news(
- succeeded_rss_ids, "rss", effective_interests_file,
- current_hash, matched_rss_ids
- )
-
- if succeeded_news_ids or succeeded_rss_ids:
- total_analyzed = len(succeeded_news_ids) + len(succeeded_rss_ids)
- total_matched = len(matched_hotlist_ids) + len(matched_rss_ids)
- print(f"[AI筛选] 已记录 {total_analyzed} 条新闻分析状态 (匹配 {total_matched}, 不匹配 {total_analyzed - total_matched})")
-
- # 7. 结束批量模式(统一上传数据库到远程存储)
- storage.end_batch()
-
- # 8. 查询并组装返回结果
- all_results = storage.get_active_ai_filter_results(interests_file=effective_interests_file)
-
- if debug:
- print(f"[AI筛选][DEBUG] === 最终汇总 ===")
- print(f"[AI筛选][DEBUG] 数据库 active 分类结果: {len(all_results)} 条")
- # 按标签统计
- tag_counts: dict = {}
- for r in all_results:
- tag_name = r.get("tag", "?")
- src_type = r.get("source_type", "?")
- key = f"{tag_name}({src_type})"
- tag_counts[key] = tag_counts.get(key, 0) + 1
- for key, count in sorted(tag_counts.items()):
- print(f"[AI筛选][DEBUG] {key}: {count} 条")
-
- return self._build_filter_result(all_results, active_tags, total_pending)
-
- def _build_filter_result(
- self,
- raw_results: List[Dict],
- tags: List[Dict],
- total_processed: int,
- ) -> AIFilterResult:
- """将数据库查询结果组装为 AIFilterResult"""
- priority_sort_enabled = self.ai_priority_sort_enabled
- tag_priority_map = {}
- for idx, t in enumerate(tags, start=1):
- tag_name = str(t.get("tag", "")).strip() if isinstance(t, dict) else ""
- if not tag_name:
- continue
- try:
- tag_priority_map[tag_name] = int(t.get("priority", idx))
- except (TypeError, ValueError):
- tag_priority_map[tag_name] = idx
-
- # 按标签分组
- tag_groups: Dict[str, Dict] = {}
- seen_titles: Dict[str, set] = {} # 每个标签下去重
-
- for r in raw_results:
- tag_name = r["tag"]
- if tag_name not in tag_groups:
- raw_priority = r.get("tag_priority", tag_priority_map.get(tag_name, 9999))
- try:
- tag_position = int(raw_priority)
- except (TypeError, ValueError):
- tag_position = 9999
- tag_groups[tag_name] = {
- "tag": tag_name,
- "description": r.get("tag_description", ""),
- "position": tag_position,
- "count": 0,
- "items": [],
- }
- seen_titles[tag_name] = set()
-
- title = r["title"]
- if title in seen_titles[tag_name]:
- continue
- seen_titles[tag_name].add(title)
-
- tag_groups[tag_name]["items"].append({
- "title": title,
- "source_id": r.get("source_id", ""),
- "source_name": r.get("source_name", ""),
- "url": r.get("url", ""),
- "mobile_url": r.get("mobile_url", ""),
- "rank": r.get("rank", 0),
- "ranks": r.get("ranks", []),
- "first_time": r.get("first_time", ""),
- "last_time": r.get("last_time", ""),
- "count": r.get("count", 1),
- "relevance_score": r.get("relevance_score", 0),
- "source_type": r.get("source_type", "hotlist"),
- })
- tag_groups[tag_name]["count"] += 1
-
- # 根据配置排序:位置优先 / 数量优先
- if priority_sort_enabled:
- sorted_tags = sorted(
- tag_groups.values(),
- key=lambda x: (x.get("position", 9999), -x["count"], x["tag"]),
- )
- else:
- sorted_tags = sorted(
- tag_groups.values(),
- key=lambda x: (-x["count"], x.get("position", 9999), x["tag"]),
- )
-
- total_matched = sum(t["count"] for t in sorted_tags)
-
- return AIFilterResult(
- tags=sorted_tags,
- total_matched=total_matched,
- total_processed=total_processed,
- success=True,
- )
+ try:
+ return self._get_ai_filter_pipeline().run(interests_file)
+ except _TagExtractionError:
+ return AIFilterResult(success=False, error="标签提取失败")
def convert_ai_filter_to_report_data(
self,
@@ -936,204 +522,10 @@ class AppContext:
new_titles: Optional[Dict] = None,
rss_new_urls: Optional[set] = None,
) -> tuple:
- """
- 将 AI 筛选结果转换为与关键词匹配相同的数据结构
-
- AIFilterResult.tags 中每个 tag 对应一个 "word"(关键词组)。
- tag.items 中 source_type="hotlist" 的条目进入热榜 stats,
- source_type="rss" 的条目进入 rss_items stats。
-
- Args:
- ai_filter_result: AI 筛选结果
- mode: 报告模式 ("daily" | "current" | "incremental")
- new_titles: 热榜新增标题 {source_id: {title: data}},用于 is_new 检测
- rss_new_urls: 新增 RSS 条目的 URL 集合,用于 is_new 检测
-
- Returns:
- (hotlist_stats, rss_stats, rss_new_stats):
- - hotlist_stats: 与 count_word_frequency() 产出格式一致
- - rss_stats: 与 rss_items 格式一致
- - rss_new_stats: RSS 新增区(rss_stats 中 is_new 的子集),与 rss_new_items 格式一致
- """
- hotlist_stats = []
- rss_stats = []
- rss_new_stats = [] # AI 筛选的 RSS 新增区(is_new 子集),与关键词路径 rss_new_stats 对齐
- max_news = self.config.get("MAX_NEWS_PER_KEYWORD", 0)
- min_score = self.ai_filter_config.get("MIN_SCORE", 0)
-
- # current 模式:计算最新时间,只保留当前在榜的热榜新闻
- # 与 count_word_frequency(mode="current") 的过滤逻辑对齐
- latest_time = None
- if mode == "current":
- for tag_data in ai_filter_result.tags:
- for item in tag_data.get("items", []):
- if item.get("source_type", "hotlist") == "hotlist":
- last_time = item.get("last_time", "")
- if last_time and (latest_time is None or last_time > latest_time):
- latest_time = last_time
- if latest_time:
- print(f"[AI筛选] current 模式:最新时间 {latest_time},过滤已下榜新闻")
-
- # RSS 新鲜度过滤配置(与推送阶段一致)
- rss_config = self.rss_config
- freshness_config = rss_config.get("FRESHNESS_FILTER", {})
- freshness_enabled = freshness_config.get("ENABLED", True)
- default_max_age_days = freshness_config.get("MAX_AGE_DAYS", 3)
- timezone = self.config.get("TIMEZONE", DEFAULT_TIMEZONE)
-
- feed_max_age_map = {}
- for feed_cfg in self.rss_feeds:
- feed_id = feed_cfg.get("id", "")
- max_age = feed_cfg.get("max_age_days")
- if max_age is not None:
- try:
- feed_max_age_map[feed_id] = int(max_age)
- except (ValueError, TypeError):
- pass
-
- filtered_count = 0
- for tag_data in ai_filter_result.tags:
- tag_name = tag_data.get("tag", "")
- items = tag_data.get("items", [])
- if not items:
- continue
-
- hotlist_titles = []
- rss_titles = []
-
- for item in items:
- source_type = item.get("source_type", "hotlist")
-
- # current 模式:跳过已下榜的热榜新闻
- if mode == "current" and latest_time and source_type == "hotlist":
- if item.get("last_time", "") != latest_time:
- filtered_count += 1
- continue
-
- # 分数阈值过滤:跳过相关度低于 min_score 的新闻
- if min_score > 0:
- score = item.get("relevance_score", 0)
- if score < min_score:
- continue
-
- # 构建时间显示
- first_time = item.get("first_time", "")
- last_time = item.get("last_time", "")
- if source_type == "rss":
- # RSS 新鲜度过滤:跳过超过 max_age_days 的旧文章
- if freshness_enabled and first_time:
- feed_id = item.get("source_id", "")
- max_days = feed_max_age_map.get(feed_id, default_max_age_days)
- if max_days > 0 and not is_within_days(first_time, max_days, timezone):
- continue
-
- # RSS 条目:first_time 是 ISO 格式,用友好格式显示
- if first_time:
- time_display = format_iso_time_friendly(first_time, timezone, include_date=True)
- else:
- time_display = ""
- else:
- # 热榜条目:使用 [HH:MM ~ HH:MM] 格式(与 keyword 模式一致)
- if first_time and last_time and first_time != last_time:
- first_display = convert_time_for_display(first_time)
- last_display = convert_time_for_display(last_time)
- time_display = f"[{first_display} ~ {last_display}]"
- elif first_time:
- time_display = convert_time_for_display(first_time)
- else:
- time_display = ""
-
- # 计算 is_new(与 keyword 模式 core/analyzer.py:335-342 对齐)
- if source_type == "rss":
- is_new = False
- if rss_new_urls:
- item_url = item.get("url", "")
- is_new = item_url in rss_new_urls if item_url else False
- else:
- is_new = False
- if new_titles:
- item_source_id = item.get("source_id", "")
- item_title = item.get("title", "")
- if item_source_id in new_titles:
- is_new = item_title in new_titles[item_source_id]
-
- # incremental 模式下仅保留本轮新增命中的条目。
- # run_ai_filter() 返回的是 active 结果集合,因此这里需要
- # 显式过滤掉历史已命中的旧条目,才能与 keyword 模式行为对齐。
- if mode == "incremental" and not is_new:
- continue
-
- title_entry = {
- "title": item.get("title", ""),
- "source_name": item.get("source_name", ""),
- "url": item.get("url", ""),
- "mobile_url": item.get("mobile_url", ""),
- "ranks": item.get("ranks", []),
- "rank_threshold": self.rank_threshold,
- "count": item.get("count", 1),
- "is_new": is_new,
- "time_display": time_display,
- "matched_keyword": tag_name,
- }
-
- if source_type == "rss":
- rss_titles.append(title_entry)
- else:
- hotlist_titles.append(title_entry)
-
- if hotlist_titles:
- if max_news > 0:
- hotlist_titles = hotlist_titles[:max_news]
- hotlist_stats.append({
- "word": tag_name,
- "count": len(hotlist_titles),
- "position": tag_data.get("position", 9999),
- "titles": hotlist_titles,
- })
-
- if rss_titles:
- if max_news > 0:
- rss_titles = rss_titles[:max_news]
- rss_stats.append({
- "word": tag_name,
- "count": len(rss_titles),
- "position": tag_data.get("position", 9999),
- "titles": rss_titles,
- })
- # 新增 RSS 区:仅保留本轮新增(is_new)的条目,供推送/HTML 的"RSS 新增"区块使用
- new_rss_titles = [t for t in rss_titles if t.get("is_new")]
- if new_rss_titles:
- rss_new_stats.append({
- "word": tag_name,
- "count": len(new_rss_titles),
- "position": tag_data.get("position", 9999),
- "titles": new_rss_titles,
- })
-
- if mode == "current" and filtered_count > 0:
- total_kept = sum(s["count"] for s in hotlist_stats)
- print(f"[AI筛选] current 模式:过滤 {filtered_count} 条已下榜新闻,保留 {total_kept} 条当前在榜")
-
- if min_score > 0:
- hotlist_kept = sum(s["count"] for s in hotlist_stats)
- rss_kept = sum(s["count"] for s in rss_stats)
- total_kept = hotlist_kept + rss_kept
- parts = [f"热榜 {hotlist_kept} 条"]
- if rss_kept > 0:
- parts.append(f"RSS {rss_kept} 条")
- print(f"[AI筛选] 分数过滤:min_score={min_score},保留 {total_kept} 条 score≥{min_score} ({', '.join(parts)})")
-
- priority_sort_enabled = self.ai_priority_sort_enabled
- if priority_sort_enabled:
- hotlist_stats.sort(key=lambda x: (x.get("position", 9999), -x["count"], x["word"]))
- rss_stats.sort(key=lambda x: (x.get("position", 9999), -x["count"], x["word"]))
- rss_new_stats.sort(key=lambda x: (x.get("position", 9999), -x["count"], x["word"]))
- else:
- hotlist_stats.sort(key=lambda x: (-x["count"], x.get("position", 9999), x["word"]))
- rss_stats.sort(key=lambda x: (-x["count"], x.get("position", 9999), x["word"]))
- rss_new_stats.sort(key=lambda x: (-x["count"], x.get("position", 9999), x["word"]))
-
- return hotlist_stats, rss_stats, rss_new_stats
+ """将 AI 筛选结果转换为与关键词匹配相同的数据结构"""
+ return self._get_ai_filter_pipeline().convert_to_report_data(
+ ai_filter_result, mode, new_titles, rss_new_urls,
+ )
# === 资源清理 ===
diff --git a/trendradar/notification/dispatcher.py b/trendradar/notification/dispatcher.py
index f573efdda5..4074e0efce 100644
--- a/trendradar/notification/dispatcher.py
+++ b/trendradar/notification/dispatcher.py
@@ -12,6 +12,7 @@
from __future__ import annotations
+import time
from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional
from trendradar.core.config import (
@@ -158,10 +159,64 @@ class NotificationDispatcher:
print("[翻译] 没有需要翻译的内容")
return report_data, rss_items, rss_new_items, standalone_data
- print(f"[翻译] 共 {len(titles_to_translate)} 条标题待翻译")
+ total_count = len(titles_to_translate)
+ trans_config = self.config.get("AI_TRANSLATION", {})
+ batch_size = trans_config.get("BATCH_SIZE", 100)
+ batch_interval = trans_config.get("BATCH_INTERVAL", 2)
+ num_batches = (total_count + batch_size - 1) // batch_size
- # 批量翻译
- result = self.translator.translate_batch(titles_to_translate)
+ if num_batches > 1:
+ print(f"[翻译] 共 {total_count} 条标题待翻译,分 {num_batches} 批(每批 {batch_size} 条,间隔 {batch_interval}s)")
+ else:
+ print(f"[翻译] 共 {total_count} 条标题待翻译")
+
+ # 分批翻译
+ from trendradar.ai.translator import BatchTranslationResult
+
+ merged_result = BatchTranslationResult(total_count=total_count)
+ batch_count = 0
+
+ for i in range(0, total_count, batch_size):
+ if batch_count > 0 and batch_interval > 0:
+ time.sleep(batch_interval)
+ batch_texts = titles_to_translate[i:i + batch_size]
+ batch_num = batch_count + 1
+ if num_batches > 1:
+ print(f"[翻译] 第 {batch_num}/{num_batches} 批({len(batch_texts)} 条)...")
+ result = self.translator.translate_batch(batch_texts)
+ merged_result.results.extend(result.results)
+ merged_result.success_count += result.success_count
+ merged_result.fail_count += result.fail_count
+
+ # debug 模式:输出每批的详细信息
+ if self.config.get("DEBUG", False):
+ batch_label = f"[翻译][DEBUG][批次 {batch_num}]" if num_batches > 1 else "[翻译][DEBUG]"
+ if result.prompt:
+ print(f"{batch_label} === 发送给 AI 的 Prompt ===")
+ print(result.prompt)
+ print(f"{batch_label} === Prompt 结束 ===")
+ if result.raw_response:
+ print(f"{batch_label} === AI 原始响应 ===")
+ print(result.raw_response)
+ print(f"{batch_label} === 响应结束 ===")
+ expected = len(batch_texts)
+ if result.parsed_count != expected:
+ print(f"{batch_label} ⚠️ 行数不匹配:期望 {expected} 条,AI 返回 {result.parsed_count} 条")
+ unchanged_count = 0
+ for j, res in enumerate(result.results):
+ global_idx = i + j + 1
+ if not res.success and res.error:
+ print(f"{batch_label} [{global_idx}] !! 失败: {res.error}")
+ elif res.original_text == res.translated_text:
+ unchanged_count += 1
+ else:
+ print(f"{batch_label} [{global_idx}] {res.original_text} => {res.translated_text}")
+ if unchanged_count > 0:
+ print(f"{batch_label} (另有 {unchanged_count} 条未变化,已省略)")
+
+ batch_count += 1
+
+ result = merged_result
if result.success_count == 0:
print(f"[翻译] 翻译失败: {result.results[0].error if result.results else '未知错误'}")
@@ -169,32 +224,6 @@ class NotificationDispatcher:
print(f"[翻译] 翻译完成: {result.success_count}/{result.total_count} 成功")
- # debug 模式:输出完整 prompt、AI 原始响应、逐条对照
- if self.config.get("DEBUG", False):
- if result.prompt:
- print(f"[翻译][DEBUG] === 发送给 AI 的 Prompt ===")
- print(result.prompt)
- print(f"[翻译][DEBUG] === Prompt 结束 ===")
- if result.raw_response:
- print(f"[翻译][DEBUG] === AI 原始响应 ===")
- print(result.raw_response)
- print(f"[翻译][DEBUG] === 响应结束 ===")
- # 行数不匹配警告
- expected = len(titles_to_translate)
- if result.parsed_count != expected:
- print(f"[翻译][DEBUG] ⚠️ 行数不匹配:期望 {expected} 条,AI 返回 {result.parsed_count} 条")
- # 逐条对照
- unchanged_count = 0
- for i, res in enumerate(result.results):
- if not res.success and res.error:
- print(f"[翻译][DEBUG] [{i+1}] !! 失败: {res.error}")
- elif res.original_text == res.translated_text:
- unchanged_count += 1
- else:
- print(f"[翻译][DEBUG] [{i+1}] {res.original_text} => {res.translated_text}")
- if unchanged_count > 0:
- print(f"[翻译][DEBUG] (另有 {unchanged_count} 条未变化,已省略)")
-
# 回填翻译结果(仅在翻译文本非空时替换,防止空翻译覆盖原始标题)
for i, (loc_type, idx1, idx2) in enumerate(title_locations):
if i < len(result.results) and result.results[i].success:
diff --git a/trendradar/report/formatter.py b/trendradar/report/formatter.py
index a79b718212..e84a31afb4 100644
--- a/trendradar/report/formatter.py
+++ b/trendradar/report/formatter.py
@@ -66,9 +66,9 @@ def format_title_for_platform(
title_prefix = "🆕 " if title_data.get("is_new") else ""
if show_source:
- result = f"<font color='grey'>[{title_data['source_name']}]</font> {title_prefix}{formatted_title}"
+ result = f"<font color='grey'>&#91;{title_data['source_name']}&#93;</font> {title_prefix}{formatted_title}"
elif show_keyword and keyword:
- result = f"<font color='blue'>[{keyword}]</font> {title_prefix}{formatted_title}"
+ result = f"<font color='blue'>&#91;{keyword}&#93;</font> {title_prefix}{formatted_title}"
else:
result = f"{title_prefix}{formatted_title}"
diff --git a/uv.lock b/uv.lock
index 8bc2feebac..d2bff152c9 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1996,7 +1996,7 @@ wheels = [
[[package]]
name = "trendradar"
-version = "6.9.1"
+version = "6.10.0"
source = { editable = "." }
dependencies = [
{ name = "boto3" },
diff --git a/version b/version
index b81ba511f6..11946a9fd7 100644
--- a/version
+++ b/version
@@ -1 +1 @@
-6.9.1
\ No newline at end of file
+6.10.0
\ No newline at end of file
diff --git a/version_configs b/version_configs
index 61819ef957..a917892f00 100644
--- a/version_configs
+++ b/version_configs
@@ -1,4 +1,4 @@
-config.yaml=2.3.0
+config.yaml=2.4.0
timeline.yaml=1.2.0
frequency_words.txt=1.1.0
ai_interests.txt=1.1.0