⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.🎯 告别信息过载,你的 AI 舆情监控助手与热点筛选工具!聚合多平台热点 + RSS 订阅,支持关键词精准筛选。AI 智能筛选新闻 + AI 翻译 + AI 分析简报直推手机,也支持接入 MCP 架构,赋能 AI 自然语言对话分析、情感洞察与趋势预测等。支持 Docker ,数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/bark/slack 等渠道智能推送。
Python aibarkdata-analysisdockerhot-newsllmmailmcpmcp-servernewsntfypythonrsstrending-topicswechatwework
fix: 修复增量模式下识别问题
sansan7468f8adcced
diff --git a/README-EN.md b/README-EN.mdindex 1038e4c4aa..eed7d85af7 100644--- a/README-EN.md+++ b/README-EN.md@@ -13,7 +13,7 @@[](https://github.com/sansan0/TrendRadar/stargazers)[](https://github.com/sansan0/TrendRadar/network/members)[](LICENSE)-[](https://github.com/sansan0/TrendRadar)+[](https://github.com/sansan0/TrendRadar)[](https://github.com/sansan0/TrendRadar)[](https://work.weixin.qq.com/)@@ -330,10 +330,11 @@ Transform from "algorithm recommendation captivity" to "actively getting the inf- **Major Version Upgrade**: Upgrading from v1.x to v2.y, recommend deleting existing fork and re-forking to save effort and avoid config conflicts-### 2025/12/17 - v4.0.1+### 2025/12/20 - v4.0.3++- Added URL normalization to fix duplicate push issues caused by dynamic parameters (e.g., Weibo's `band_rank`)+- Fixed incremental mode detection logic to correctly identify historical titles-- StorageManager adds push record proxy methods-- S3 client switches to virtual-hosted style for better compatibility (supports Tencent Cloud COS and more services)### 2025/12/13 - mcp-v1.1.0@@ -349,6 +350,12 @@ Transform from "algorithm recommendation captivity" to "actively getting the inf<summary>👉 Click to expand: <strong>Historical Updates</strong></summary>+### 2025/12/17 - v4.0.1++- StorageManager adds push record proxy methods+- S3 client switches to virtual-hosted style for better compatibility (supports Tencent Cloud COS and more services)++### 2025/12/13 - v4.0.0**🎉 Major Update: Comprehensive Refactoring of Storage and Core Architecture**diff --git a/README.md b/README.mdindex 9efbcd50d1..335b1c4103 100644--- a/README.md+++ b/README.md@@ -13,7 +13,7 @@[](https://github.com/sansan0/TrendRadar/stargazers)[](https://github.com/sansan0/TrendRadar/network/members)[](LICENSE)-[](https://github.com/sansan0/TrendRadar)+[](https://github.com/sansan0/TrendRadar)[](https://github.com/sansan0/TrendRadar)[](https://work.weixin.qq.com/)@@ -377,10 +377,10 @@ GitHub 一键 Fork 即可使用,无需编程基础。-### 2025/12/17 - v4.0.1+### 2025/12/20 - v4.0.3-- StorageManager 添加推送记录代理方法-- S3 客户端切换至 virtual-hosted style 以提升兼容性(支持腾讯云 COS 等更多服务)+- 新增 URL 标准化功能,解决微博等平台因动态参数(如 `band_rank`)导致的重复推送问题+- 修复增量模式检测逻辑,正确识别历史标题### 2025/12/13 - mcp-v1.1.0@@ -397,6 +397,13 @@ GitHub 一键 Fork 即可使用,无需编程基础。<summary>👉 点击展开:<strong>历史更新</strong></summary>++### 2025/12/17 - v4.0.1++- StorageManager 添加推送记录代理方法+- S3 客户端切换至 virtual-hosted style 以提升兼容性(支持腾讯云 COS 等更多服务)++### 2025/12/13 - v4.0.0**🎉 重大更新:全面重构存储和核心架构**diff --git a/config/config.yaml b/config/config.yamlindex 1a5adb5a31..7db74d58ef 100644--- a/config/config.yaml+++ b/config/config.yaml@@ -73,7 +73,7 @@ crawler:# 推送模式选择report:- mode: "daily" # 可选: "daily"|"incremental"|"current"+ mode: "current" # 可选: "daily"|"incremental"|"current"rank_threshold: 5 # 排名高亮阈值sort_by_position_first: false # 排序优先级:true=先按配置位置排序,false=先按热点条数排序max_news_per_keyword: 0 # 每个关键词最大显示数量,0=不限制diff --git a/trendradar/__init__.py b/trendradar/__init__.pyindex a36deb45df..1ac1129d82 100644--- a/trendradar/__init__.py+++ b/trendradar/__init__.py@@ -9,5 +9,5 @@ TrendRadar - 热点新闻聚合与分析工具from trendradar.context import AppContext-__version__ = "4.0.2"+__version__ = "4.0.3"__all__ = ["AppContext", "__version__"]diff --git a/trendradar/__main__.py b/trendradar/__main__.pyindex 51af38d5cb..372626329c 100644--- a/trendradar/__main__.py+++ b/trendradar/__main__.py@@ -214,8 +214,14 @@ class NewsAnalyzer:self, stats: List[Dict], new_titles: Optional[Dict] = None) -> bool:"""检查是否有有效的新闻内容"""- if self.report_mode in ["incremental", "current"]:- # 增量模式和current模式下,只要stats有内容就说明有匹配的新闻+ if self.report_mode == "incremental":+ # 增量模式:必须有新增标题才推送+ has_new_titles = bool(+ new_titles and any(len(titles) > 0 for titles in new_titles.values())+ )+ return has_new_titles+ elif self.report_mode == "current":+ # current模式:只要stats有内容就说明有匹配的新闻return any(stat["count"] > 0 for stat in stats)else:# 当日汇总模式下,检查是否有匹配的频率词新闻或新增新闻@@ -227,15 +233,17 @@ class NewsAnalyzer:def _load_analysis_data(self,+ quiet: bool = False,) -> Optional[Tuple[Dict, Dict, Dict, Dict, List, List]]:"""统一的数据加载和预处理,使用当前监控平台列表过滤历史数据"""try:# 获取当前配置的监控平台ID列表current_platform_ids = self.ctx.platform_ids- print(f"当前监控平台: {current_platform_ids}")+ if not quiet:+ print(f"当前监控平台: {current_platform_ids}")all_results, id_to_name, title_info = self.ctx.read_today_titles(- current_platform_ids+ current_platform_ids, quiet=quiet)if not all_results:@@ -243,9 +251,10 @@ class NewsAnalyzer:return Nonetotal_titles = sum(len(titles) for titles in all_results.values())- print(f"读取到 {total_titles} 个标题(已按当前监控平台过滤)")+ if not quiet:+ print(f"读取到 {total_titles} 个标题(已按当前监控平台过滤)")- new_titles = self.ctx.detect_new_titles(current_platform_ids)+ new_titles = self.ctx.detect_new_titles(current_platform_ids, quiet=quiet)word_groups, filter_words, global_filters = self.ctx.load_frequency_words()return (@@ -293,6 +302,7 @@ class NewsAnalyzer:failed_ids: Optional[List] = None,is_daily_summary: bool = False,global_filters: Optional[List[str]] = None,+ quiet: bool = False,) -> Tuple[List[Dict], Optional[str]]:"""统一的分析流水线:数据处理 → 统计计算 → HTML生成"""@@ -306,6 +316,7 @@ class NewsAnalyzer:new_titles,mode=mode,global_filters=global_filters,+ quiet=quiet,)# HTML生成(如果启用)@@ -406,9 +417,12 @@ class NewsAnalyzer:):mode_strategy = self._get_mode_strategy()if "实时" in report_type:- print(- f"跳过实时推送通知:{mode_strategy['mode_name']}下未检测到匹配的新闻"- )+ if self.report_mode == "incremental":+ print("跳过实时推送通知:增量模式下未检测到新增的新闻")+ else:+ print(+ f"跳过实时推送通知:{mode_strategy['mode_name']}下未检测到匹配的新闻"+ )else:print(f"跳过{mode_strategy['summary_report_type']}通知:未匹配到有效的新闻内容"@@ -466,8 +480,8 @@ class NewsAnalyzer:summary_type = "当前榜单汇总" if mode == "current" else "当日汇总"print(f"生成{summary_type}HTML...")- # 加载分析数据- analysis_data = self._load_analysis_data()+ # 加载分析数据(静默模式,避免重复输出日志)+ analysis_data = self._load_analysis_data(quiet=True)if not analysis_data:return None@@ -475,7 +489,7 @@ class NewsAnalyzer:analysis_data)- # 运行分析流水线+ # 运行分析流水线(静默模式,避免重复输出日志)_, html_file = self._run_analysis_pipeline(all_results,mode,@@ -486,6 +500,7 @@ class NewsAnalyzer:id_to_name,is_daily_summary=True,global_filters=global_filters,+ quiet=True,)if html_file:diff --git a/trendradar/context.py b/trendradar/context.pyindex 3fc7d153a4..045821ab87 100644--- a/trendradar/context.py+++ b/trendradar/context.py@@ -167,20 +167,20 @@ class AppContext:return save_titles_to_file(results, id_to_name, failed_ids, output_path, clean_title)def read_today_titles(- self, platform_ids: Optional[List[str]] = None+ self, platform_ids: Optional[List[str]] = None, quiet: bool = False) -> Tuple[Dict, Dict, Dict]:"""读取当天所有标题"""- return read_all_today_titles(self.get_storage_manager(), platform_ids)+ return read_all_today_titles(self.get_storage_manager(), platform_ids, quiet=quiet)def detect_new_titles(- self, platform_ids: Optional[List[str]] = None+ self, platform_ids: Optional[List[str]] = None, quiet: bool = False) -> Dict:"""检测最新批次的新增标题"""- return detect_latest_new_titles(self.get_storage_manager(), platform_ids)+ return detect_latest_new_titles(self.get_storage_manager(), platform_ids, quiet=quiet)def is_first_crawl(self) -> bool:"""检测是否是当天第一次爬取"""- return is_first_crawl_today("output", self.format_date())+ return self.get_storage_manager().is_first_crawl_today()# === 频率词处理 ===@@ -212,6 +212,7 @@ class AppContext:new_titles: Optional[Dict] = None,mode: str = "daily",global_filters: Optional[List[str]] = None,+ quiet: bool = False,) -> Tuple[List[Dict], int]:"""统计词频"""return count_word_frequency(@@ -229,6 +230,7 @@ class AppContext:sort_by_position_first=self.config.get("SORT_BY_POSITION_FIRST", False),is_first_crawl_func=self.is_first_crawl,convert_time_func=self.convert_time_display,+ quiet=quiet,)# === 报告生成 ===diff --git a/trendradar/core/analyzer.py b/trendradar/core/analyzer.pyindex 7cb1913e69..2294103853 100644--- a/trendradar/core/analyzer.py+++ b/trendradar/core/analyzer.py@@ -102,6 +102,7 @@ def count_word_frequency(sort_by_position_first: bool = False,is_first_crawl_func: Optional[Callable[[], bool]] = None,convert_time_func: Optional[Callable[[str], str]] = None,+ quiet: bool = False,) -> Tuple[List[Dict], int]:"""统计词频,支持必须词、频率词、过滤词、全局过滤词,并标记新增标题@@ -121,6 +122,7 @@ def count_word_frequency(sort_by_position_first: 是否优先按配置位置排序is_first_crawl_func: 检测是否是当天第一次爬取的函数convert_time_func: 时间格式转换函数+ quiet: 是否静默模式(不打印日志)Returns:Tuple[List[Dict], int]: (统计结果列表, 总标题数)@@ -461,9 +463,10 @@ def count_word_frequency(# 先按热点条数,再按配置位置(原逻辑)stats.sort(key=lambda x: (-x["count"], x["position"]))- # 打印过滤后的匹配新闻数(与推送显示一致)+ # 打印过滤后的匹配新闻数matched_news_count = sum(len(stat["titles"]) for stat in stats if stat["count"] > 0)- if mode == "daily":- print(f"频率词过滤后:{matched_news_count} 条新闻匹配(将显示在推送中)")+ if not quiet and mode == "daily":+ print(f"当日汇总模式:处理 {total_titles} 条新闻,模式:频率词过滤")+ print(f"频率词过滤后:{matched_news_count} 条新闻匹配")return stats, total_titlesdiff --git a/trendradar/core/data.py b/trendradar/core/data.pyindex f56eabec16..7297e8897d 100644--- a/trendradar/core/data.py+++ b/trendradar/core/data.py@@ -152,6 +152,7 @@ def read_all_today_titles_from_storage(def read_all_today_titles(storage_manager,current_platform_ids: Optional[List[str]] = None,+ quiet: bool = False,) -> Tuple[Dict, Dict, Dict]:"""读取当天所有标题(从存储后端)@@ -159,6 +160,7 @@ def read_all_today_titles(Args:storage_manager: 存储管理器实例current_platform_ids: 当前监控的平台 ID 列表(用于过滤)+ quiet: 是否静默模式(不打印日志)Returns:Tuple[Dict, Dict, Dict]: (all_results, id_to_name, title_info)@@ -167,11 +169,12 @@ def read_all_today_titles(storage_manager, current_platform_ids)- if all_results:- total_count = sum(len(titles) for titles in all_results.values())- print(f"[存储] 已从存储后端读取 {total_count} 条标题")- else:- print("[存储] 当天暂无数据")+ if not quiet:+ if all_results:+ total_count = sum(len(titles) for titles in all_results.values())+ print(f"[存储] 已从存储后端读取 {total_count} 条标题")+ else:+ print("[存储] 当天暂无数据")return all_results, final_id_to_name, title_info@@ -202,19 +205,35 @@ def detect_latest_new_titles_from_storage(# 没有历史数据(第一次抓取),不应该有"新增"标题return {}- # 收集历史标题(不包括最新批次的时间)+ # 获取最新批次时间latest_time = latest_data.crawl_time- historical_titles = {}+ # 步骤1:收集最新批次的标题(last_crawl_time = latest_time 的标题)+ latest_titles = {}+ for source_id, news_list in latest_data.items.items():+ if current_platform_ids is not None and source_id not in current_platform_ids:+ continue+ latest_titles[source_id] = {}+ for item in news_list:+ latest_titles[source_id][item.title] = {+ "ranks": [item.rank],+ "url": item.url or "",+ "mobileUrl": item.mobile_url or "",+ }++ # 步骤2:收集历史标题+ # 关键逻辑:一个标题只要其 first_crawl_time < latest_time,就是历史标题+ # 这样即使同一标题有多条记录(URL 不同),只要任何一条是历史的,该标题就算历史+ historical_titles = {}for source_id, news_list in all_data.items.items():if current_platform_ids is not None and source_id not in current_platform_ids:continuehistorical_titles[source_id] = set()for item in news_list:- # 只统计非最新批次的标题first_time = getattr(item, 'first_time', item.crawl_time)- if first_time != latest_time:+ # 如果该记录的首次出现时间早于最新批次,则该标题是历史标题+ if first_time < latest_time:historical_titles[source_id].add(item.title)# 检查是否是当天第一次抓取(没有任何历史标题)@@ -223,22 +242,15 @@ def detect_latest_new_titles_from_storage(if not has_historical_data:return {}- # 找出新增标题+ # 步骤3:找出新增标题 = 最新批次标题 - 历史标题new_titles = {}- for source_id, news_list in latest_data.items.items():- if current_platform_ids is not None and source_id not in current_platform_ids:- continue-+ for source_id, source_latest_titles in latest_titles.items():historical_set = historical_titles.get(source_id, set())source_new_titles = {}- for item in news_list:- if item.title not in historical_set:- source_new_titles[item.title] = {- "ranks": [item.rank],- "url": item.url or "",- "mobileUrl": item.mobile_url or "",- }+ for title, title_data in source_latest_titles.items():+ if title not in historical_set:+ source_new_titles[title] = title_dataif source_new_titles:new_titles[source_id] = source_new_titles@@ -253,6 +265,7 @@ def detect_latest_new_titles_from_storage(def detect_latest_new_titles(storage_manager,current_platform_ids: Optional[List[str]] = None,+ quiet: bool = False,) -> Dict:"""检测当日最新批次的新增标题(从存储后端)@@ -260,12 +273,13 @@ def detect_latest_new_titles(Args:storage_manager: 存储管理器实例current_platform_ids: 当前监控的平台 ID 列表(用于过滤)+ quiet: 是否静默模式(不打印日志)Returns:Dict: 新增标题 {source_id: {title: title_data}}"""new_titles = detect_latest_new_titles_from_storage(storage_manager, current_platform_ids)- if new_titles:+ if new_titles and not quiet:total_new = sum(len(titles) for titles in new_titles.values())print(f"[存储] 从存储后端检测到 {total_new} 条新增标题")return new_titlesdiff --git a/trendradar/notification/renderer.py b/trendradar/notification/renderer.pyindex dcee02a04c..9cb4bc2769 100644--- a/trendradar/notification/renderer.py+++ b/trendradar/notification/renderer.py@@ -6,7 +6,7 @@"""from datetime import datetime-from typing import Dict, List, Optional, Callable+from typing import Dict, Optional, Callablefrom trendradar.report.formatter import format_title_for_platformdiff --git a/trendradar/storage/base.py b/trendradar/storage/base.pyindex 94f7a1cee5..2164379980 100644--- a/trendradar/storage/base.py+++ b/trendradar/storage/base.py@@ -7,9 +7,7 @@from abc import ABC, abstractmethodfrom dataclasses import dataclass, field-from datetime import datetimefrom typing import Dict, List, Optional, Any-import json@dataclassdiff --git a/trendradar/storage/local.py b/trendradar/storage/local.pyindex f0fa3ebbc6..7d49a4deb6 100644--- a/trendradar/storage/local.py+++ b/trendradar/storage/local.py@@ -6,13 +6,12 @@"""import sqlite3-import osimport shutilimport pytzimport refrom datetime import datetime, timedeltafrom pathlib import Path-from typing import Dict, List, Optional, Any+from typing import Dict, List, Optionalfrom trendradar.storage.base import StorageBackend, NewsItem, NewsDatafrom trendradar.utils.time import (@@ -20,6 +19,7 @@ from trendradar.utils.time import (format_date_folder,format_time_filename,)+from trendradar.utils.url import normalize_urlclass LocalStorageBackend(StorageBackend):@@ -148,12 +148,15 @@ class LocalStorageBackend(StorageBackend):for item in news_list:try:- # 检查是否已存在(通过 URL + platform_id)- if item.url:+ # 标准化 URL(去除动态参数,如微博的 band_rank)+ normalized_url = normalize_url(item.url, source_id) if item.url else ""++ # 检查是否已存在(通过标准化 URL + platform_id)+ if normalized_url:cursor.execute("""SELECT id, title FROM news_itemsWHERE url = ? AND platform_id = ?- """, (item.url, source_id))+ """, (normalized_url, source_id))existing = cursor.fetchone()if existing:@@ -191,14 +194,14 @@ class LocalStorageBackend(StorageBackend):data.crawl_time, now_str, existing_id))updated_count += 1else:- # 不存在,插入新记录+ # 不存在,插入新记录(存储标准化后的 URL)cursor.execute("""INSERT INTO news_items(title, platform_id, rank, url, mobile_url,first_crawl_time, last_crawl_time, crawl_count,created_at, updated_at)VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?)- """, (item.title, source_id, item.rank, item.url,+ """, (item.title, source_id, item.rank, normalized_url,item.mobile_url, data.crawl_time, data.crawl_time,now_str, now_str))new_id = cursor.lastrowid@@ -217,7 +220,7 @@ class LocalStorageBackend(StorageBackend):first_crawl_time, last_crawl_time, crawl_count,created_at, updated_at)VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?)- """, (item.title, source_id, item.rank, item.url,+ """, (item.title, source_id, item.rank, "",item.mobile_url, data.crawl_time, data.crawl_time,now_str, now_str))new_id = cursor.lastrowid@@ -524,6 +527,9 @@ class LocalStorageBackend(StorageBackend):"""检测新增的标题+ 该方法比较当前抓取数据与历史数据,找出新增的标题。+ 关键逻辑:只有在历史批次中从未出现过的标题才算新增。+Args:current_data: 当前抓取的数据@@ -541,10 +547,24 @@ class LocalStorageBackend(StorageBackend):new_titles[source_id] = {item.title: item for item in news_list}return new_titles- # 收集历史标题+ # 获取当前批次时间+ current_time = current_data.crawl_time++ # 收集历史标题(first_time < current_time 的标题)+ # 这样可以正确处理同一标题因 URL 变化而产生多条记录的情况historical_titles: Dict[str, set] = {}for source_id, news_list in historical_data.items.items():- historical_titles[source_id] = {item.title for item in news_list}+ historical_titles[source_id] = set()+ for item in news_list:+ first_time = getattr(item, 'first_time', item.crawl_time)+ if first_time < current_time:+ historical_titles[source_id].add(item.title)++ # 检查是否有历史数据+ has_historical_data = any(len(titles) > 0 for titles in historical_titles.values())+ if not has_historical_data:+ # 第一次抓取,没有"新增"概念+ return {}# 检测新增new_titles = {}diff --git a/trendradar/storage/remote.py b/trendradar/storage/remote.pyindex cf0bb0c919..0041f22033 100644--- a/trendradar/storage/remote.py+++ b/trendradar/storage/remote.py@@ -34,6 +34,7 @@ from trendradar.utils.time import (format_date_folder,format_time_filename,)+from trendradar.utils.url import normalize_urlclass RemoteStorageBackend(StorageBackend):@@ -355,12 +356,15 @@ class RemoteStorageBackend(StorageBackend):for item in news_list:try:- # 检查是否已存在(通过 URL + platform_id)- if item.url:+ # 标准化 URL(去除动态参数,如微博的 band_rank)+ normalized_url = normalize_url(item.url, source_id) if item.url else ""++ # 检查是否已存在(通过标准化 URL + platform_id)+ if normalized_url:cursor.execute("""SELECT id, title FROM news_itemsWHERE url = ? AND platform_id = ?- """, (item.url, source_id))+ """, (normalized_url, source_id))existing = cursor.fetchone()if existing:@@ -398,14 +402,14 @@ class RemoteStorageBackend(StorageBackend):data.crawl_time, now_str, existing_id))updated_count += 1else:- # 不存在,插入新记录+ # 不存在,插入新记录(存储标准化后的 URL)cursor.execute("""INSERT INTO news_items(title, platform_id, rank, url, mobile_url,first_crawl_time, last_crawl_time, crawl_count,created_at, updated_at)VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?)- """, (item.title, source_id, item.rank, item.url,+ """, (item.title, source_id, item.rank, normalized_url,item.mobile_url, data.crawl_time, data.crawl_time,now_str, now_str))new_id = cursor.lastrowid@@ -424,7 +428,7 @@ class RemoteStorageBackend(StorageBackend):first_crawl_time, last_crawl_time, crawl_count,created_at, updated_at)VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?)- """, (item.title, source_id, item.rank, item.url,+ """, (item.title, source_id, item.rank, "",item.mobile_url, data.crawl_time, data.crawl_time,now_str, now_str))new_id = cursor.lastrowid@@ -693,7 +697,12 @@ class RemoteStorageBackend(StorageBackend):return Nonedef detect_new_titles(self, current_data: NewsData) -> Dict[str, Dict]:- """检测新增的标题"""+ """+ 检测新增的标题++ 该方法比较当前抓取数据与历史数据,找出新增的标题。+ 关键逻辑:只有在历史批次中从未出现过的标题才算新增。+ """try:historical_data = self.get_today_all_data(current_data.date)@@ -703,9 +712,24 @@ class RemoteStorageBackend(StorageBackend):new_titles[source_id] = {item.title: item for item in news_list}return new_titles+ # 获取当前批次时间+ current_time = current_data.crawl_time++ # 收集历史标题(first_time < current_time 的标题)+ # 这样可以正确处理同一标题因 URL 变化而产生多条记录的情况historical_titles: Dict[str, set] = {}for source_id, news_list in historical_data.items.items():- historical_titles[source_id] = {item.title for item in news_list}+ historical_titles[source_id] = set()+ for item in news_list:+ first_time = getattr(item, 'first_time', item.crawl_time)+ if first_time < current_time:+ historical_titles[source_id].add(item.title)++ # 检查是否有历史数据+ has_historical_data = any(len(titles) > 0 for titles in historical_titles.values())+ if not has_historical_data:+ # 第一次抓取,没有"新增"概念+ return {}new_titles = {}for source_id, news_list in current_data.items.items():diff --git a/trendradar/utils/__init__.py b/trendradar/utils/__init__.pyindex 0b6a46d277..a3873201fd 100644--- a/trendradar/utils/__init__.py+++ b/trendradar/utils/__init__.py@@ -10,6 +10,7 @@ from trendradar.utils.time import (get_current_time_display,convert_time_for_display,)+from trendradar.utils.url import normalize_url, get_url_signature__all__ = ["get_configured_time",@@ -17,4 +18,6 @@ __all__ = ["format_time_filename","get_current_time_display","convert_time_for_display",+ "normalize_url",+ "get_url_signature",]diff --git a/trendradar/utils/url.py b/trendradar/utils/url.pynew file mode 100644index 0000000000..9d78f7c227--- /dev/null+++ b/trendradar/utils/url.py@@ -0,0 +1,146 @@+# coding=utf-8+"""+URL 处理工具模块++提供 URL 标准化功能,用于去重时消除动态参数的影响:+- normalize_url: 标准化 URL,去除动态参数+"""++from urllib.parse import urlparse, urlunparse, parse_qs, urlencode+from typing import Dict, Set+++# 各平台需要移除的特定参数+# - weibo: 有 band_rank(排名)和 Refer(来源)动态参数+# - 其他平台: URL 为路径格式或简单关键词查询,无需处理+PLATFORM_PARAMS_TO_REMOVE: Dict[str, Set[str]] = {+ # 微博:band_rank 是动态排名参数,Refer 是来源参数,t 是时间范围参数+ # 示例:https://s.weibo.com/weibo?q=xxx&t=31&band_rank=1&Refer=top+ # 保留:q(关键词)+ # 移除:band_rank, Refer, t+ "weibo": {"band_rank", "Refer", "t"},+}++# 通用追踪参数(适用于所有平台)+# 这些参数通常由分享链接或广告追踪添加,不影响内容识别+COMMON_TRACKING_PARAMS: Set[str] = {+ # UTM 追踪参数+ "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",+ # 常见追踪参数+ "ref", "referrer", "source", "channel",+ # 时间戳和随机参数+ "_t", "timestamp", "_", "random",+ # 分享相关+ "share_token", "share_id", "share_from",+}+++def normalize_url(url: str, platform_id: str = "") -> str:+ """+ 标准化 URL,去除动态参数++ 用于数据库去重,确保同一条新闻的不同 URL 变体能被正确识别为同一条。++ 处理规则:+ 1. 去除平台特定的动态参数(如微博的 band_rank)+ 2. 去除通用追踪参数(如 utm_*)+ 3. 保留核心查询参数(如搜索关键词 q=, wd=, keyword=)+ 4. 对查询参数按字母序排序(确保一致性)++ Args:+ url: 原始 URL+ platform_id: 平台 ID,用于应用平台特定规则++ Returns:+ 标准化后的 URL++ Examples:+ >>> normalize_url("https://s.weibo.com/weibo?q=test&band_rank=6&Refer=top", "weibo")+ 'https://s.weibo.com/weibo?q=test'++ >>> normalize_url("https://example.com/page?id=1&utm_source=twitter", "")+ 'https://example.com/page?id=1'+ """+ if not url:+ return url++ try:+ # 解析 URL+ parsed = urlparse(url)++ # 如果没有查询参数,直接返回+ if not parsed.query:+ return url++ # 解析查询参数+ params = parse_qs(parsed.query, keep_blank_values=True)++ # 收集需要移除的参数(使用小写进行比较)+ params_to_remove: Set[str] = set()++ # 添加通用追踪参数+ params_to_remove.update(COMMON_TRACKING_PARAMS)++ # 添加平台特定参数+ if platform_id and platform_id in PLATFORM_PARAMS_TO_REMOVE:+ params_to_remove.update(PLATFORM_PARAMS_TO_REMOVE[platform_id])++ # 过滤参数(参数名转小写进行比较)+ filtered_params = {+ key: values+ for key, values in params.items()+ if key.lower() not in {p.lower() for p in params_to_remove}+ }++ # 如果过滤后没有参数了,返回不带查询字符串的 URL+ if not filtered_params:+ return urlunparse((+ parsed.scheme,+ parsed.netloc,+ parsed.path,+ parsed.params,+ "", # 空查询字符串+ "" # 移除 fragment+ ))++ # 重建查询字符串(按字母序排序以确保一致性)+ sorted_params = []+ for key in sorted(filtered_params.keys()):+ for value in filtered_params[key]:+ sorted_params.append((key, value))++ new_query = urlencode(sorted_params)++ # 重建 URL(移除 fragment)+ normalized = urlunparse((+ parsed.scheme,+ parsed.netloc,+ parsed.path,+ parsed.params,+ new_query,+ "" # 移除 fragment+ ))++ return normalized++ except Exception:+ # 解析失败时返回原始 URL+ return url+++def get_url_signature(url: str, platform_id: str = "") -> str:+ """+ 获取 URL 的签名(用于快速比较)++ 基于标准化 URL 生成签名,可用于:+ - 快速判断两个 URL 是否指向同一内容+ - 作为缓存键++ Args:+ url: 原始 URL+ platform_id: 平台 ID++ Returns:+ URL 签名字符串+ """+ return normalize_url(url, platform_id)diff --git a/version b/versionindex 4bac418e4a..aa31e71b03 100644--- a/version+++ b/version@@ -1 +1 @@-4.0.2\ No newline at end of file+4.0.3\ No newline at end of file