Skip to content

armgit / peach

Public Imported from GitHub
0

Peach: Local-first video metadata scraper and media manager.

fix(ledger): 整理前的账本备份只留最近 5 份

longmeidaof1c79ac7a4c5
diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md
index ffdb70a18..80702adc4 100644
--- a/docs/OPERATIONS.md
+++ b/docs/OPERATIONS.md
@@ -647,6 +647,7 @@ Windows 托盘在取得单实例锁后启动独立监视器。监视器用 Win32
- 真实迁移前依次执行:SQLite 备份、asset/tag 计数、`PRAGMA integrity_check`、迁移版本检查、服务冒烟测试。已应用与待应用的迁移只认 `peach migrate status` 的输出。
- 已应用的迁移文件不能修改,任何后续变更都要新增一个版本。已应用的迁移被改写会造成校验和漂移,只能用迁移前备份重放、逐条比对差异为 0 后再校正 `schema_migration`。
- 数据库备份自动清退:`ledger.pre-*.db` 由 `peach.ledger_backups` 管理,「最近 5 份」「24 小时内」「比 `ledger.db` 新」三类全部保留,其余连同 `-wal`/`-shm` 删除。Windows 托盘每次启动自动执行;`scripts/prune_ledger_backups.py` 手动运行时默认只列计划,`--apply` 才删除;数据库 `integrity_check` 不是 ok 时拒绝清退,退出码 2。每份备份一百多 MB,5 天不清就是 7 GB。
+- 整理备份自动清退:下载后整理与手动整理写入前各留一份整库备份,放在 `<Peach 数据目录>/generated/organize-backups/`,文件名是 `ledger.pre-download-<任务>-<时间>.db` 与 `ledger.pre-organize-<时间>-<任务>.db`。每次写备份后按同一套规则清退这个目录,只留最近 5 份(`ORGANIZE_KEEP_RECENT`),不按年龄保留(`ORGANIZE_MIN_AGE` 为零):一波下载一小时能整理四十个文件,24 小时窗口会攒下几十 GB。正在被整理引用的那份由 `ledger_backups.holding` 登记,Web 服务进程里的清退都跳过;目录里的其他文件名不动。手动查看或清理用 `scripts/prune_ledger_backups.py --organize`,同样默认只列计划,`--apply` 才删除;登记不跨进程,整理进行中不要给 `--keep 0`。
- `<Peach 数据目录>/logs` 里的 `*.log` 统一保留半年,不限大小(`peach.log_retention.sweep`,托盘在启动任何子进程之前运行):半年没写过的文件整份删除;还在写的文件按自然月分段,上次写入在更早月份的改名成 `<名字>.until-<最后写入日期>.log`,再过半年删除。子进程直接追加 stdout,不经过 `logging`,所以按文件处理,不按行处理。
- 更新与打包产物自动清退:托盘每次启动只保留最近 2 份 `dist/Peach/Peach.pre-source-sync-*.exe` 备份,并删掉 `<Peach 数据目录>/state/source-sync-build/` 里不属于待应用记录的暂存构建(`WindowsUpdateInstaller.sweep_artifacts`);`build_windows.ps1` 成功后删掉 PyInstaller 工作目录 `build/windows/app`;单文件托盘每次启动删掉 `%TEMP%` 里超过一天、不属于自己的 `_MEI*` 解压残留(`sweep_onefile_extractions`,托盘被结束进程时 PyInstaller 不会自己清理,每份 40–80 MB)。自动清理只认这几种命名:`Peach.exe` 本体、手工放进 `dist/` 的目录、`build/release-*`、`attic/` 都不碰,手工构建的残留要自己删。
- 可重建缓存的删除范围由当前数据库路径决定:生产库只能清理同一个 `peach-data` 下的缓存,临时库只能清理它的临时目录,范围之外一律跳过。测试里的清空回收站必须用临时 Peach 数据目录,漏配会删到真实封面。
diff --git a/docs/REUSE.md b/docs/REUSE.md
index f73ab1295..73fd459c0 100644
--- a/docs/REUSE.md
+++ b/docs/REUSE.md
@@ -619,6 +619,7 @@ A 盘垃圾隔离使用短名称 `peach-purge-<32 位 UUID>.peach-quarantine`,
- MISSAV 入口优先读取 `state/missav-actresses.json` 的实际名录 href,保留原字形、数字前缀与括号艺名;`directory_name_key` 仅对匹配折叠字形及纯汉字、假名名称的空白,英文名称空白保留。同名对应不同路径时不显示入口;明确 `verified=false` 的来源项阻止该名称入口及模板回退,页面重新核验后可解除。未采全名录的缺项保留模板候选,不能判为站点未收录。七沢みあ、逢沢みゆ的 h1 与 canonical 已在用户 Chrome 验证,证据在 `<私有验证记录>`;名录原始分页和全库匹配 CSV 在 `review/missav-directory/`。API 与全库审计共用 `entry_links.build`,派生入口不写 ledger,403 验证页记「未取得」。艺名选择沿用现名、读音排除与来源冲突复核。无新增依赖。
- FC2 作品计数复用 `ordered_multipart_items`,`web_catalog.video_work_key` 按账本代次缓存已确认的合集键;资料、名册、同台艺人、标签与补全共用。`/api/items.work_total` 供作品读数,`total` 仍用于文件分页;不确定分段不归组。出演名单展开复用 BoardUI `Button`。跨角色人工合并走 `merge_entity`,`user:identity-merge:<kind>` 的别名与墓碑共同证明旧资料页地址,单纯同名不跨角色跳转。
- 馆藏分组缓存声明 `asset`、`asset_entity`、`entity` 依赖;下载进度和复核留痕不使分组重算,同键计算串行共用。作品计数通过 SQLite `create_function` 注册资产到作品的只读字典查找,沿用既有版次与分卷判据。下载后处理复用已绑定资产、整理备份及逐项外键基线比较(`performer_alias_followup`);单部采集只结算本资产候选,进度在 `state/download-processing/`,共享 CSV 仍受全库处理锁保护。资源查询缓存拒收缺字段和来源失败的旧结果。无新增依赖。
+- 整理备份的保留复用 `peach.ledger_backups` 的清退规则、完整性拒绝与边车清扫,只多给目录、文件名模式与份数;`organize_backup` 统一写备份、持有在用的那份并清退,下载后整理与手动整理共用,不另写一套清理。
- Application 管理路由与资源,域控制器适配原生宿主。文件检查含本地与网盘,来源等分、共用确认弹层;CloudDrive 分档建议首启与配置页共用。
- 名字里的括号都走 `split_composite_aliases.py`:自动那拨只认罗马字复合人名,`--from-review` 那拨按人工判定清掉不承载名字的尾巴,旧写法留作别名;读音、厂牌消歧和角色出处不拆。`<私有验证记录>` 中的 28 条 creator 注音、575 条 tag 角色出处和 10 条 series 厂牌或载体消歧均保留,不属于待执行批次。
- 全库创作者归属复核走 `scripts/audit_creator_attributions.py` 与 `metadata_creator_attributions`:复用 `code_creators` 的文件番号核对、DMM `matching_cids`、`western_release_identity`、厂牌规范名和别名、`studio_sites.is_platform`、字段归属、SQLite 备份及 `review_csv`。发行站作品号同时核对目录名与媒体文件名;来源目录见 [Legs Japan](https://www.legsjapan.com/en/)、[Fellatio Japan](https://www.fellatiojapan.com/en/),平台身份见 [myfans 官方说明](https://creators.myfans.jp/)。预览覆盖全部来源关系与无作品实体;执行带冻结计划、备份和回滚记录,在事务内重算。只撤下有文件或发行来源证据的 `legacy:asset` 关系,保留用户判断、外部身份、规范实体和个人记录。月份、画质、份数与合集注记只回归已存在的完整账号;结构目录按整名切分判定,词汇取 `catalog_rules` 标签、`genre_taxonomy` 来源词与 `resources/naming/structural_directory_words.txt`,不登记具体馆藏的目录名;`creator_collection_base` 同时用于审计与摄取。公开厂牌断言可补空厂牌投影及关系,缺少规范厂牌时记录新实体完整快照;`upsert_asset_entity(update_entity_metadata=False)` 保留既有资料。结构目录、已知转载站及可信厂牌不进入自动 creator 摄取。外键检查逐项比较既有悬空;回滚核对字段版本、关系和新实体引用,后续改动拒绝整批恢复。
diff --git a/scripts/prune_ledger_backups.py b/scripts/prune_ledger_backups.py
index 0cfae49a7..8541f306f 100644
--- a/scripts/prune_ledger_backups.py
+++ b/scripts/prune_ledger_backups.py
@@ -3,6 +3,10 @@
保留规则在 `peach.ledger_backups`:最近几份、还没满 24 小时、比当前账本更新的都留;
当前账本 `integrity_check` 不是 ok 时拒绝清退。主文件已经不在的 `-wal`/`-shm` 一并扫掉。
托盘每次启动按同一规则自动跑一遍,这个脚本给手动与排障用。
+
+`--organize` 改看 `<generated>/organize-backups/` 里的整理备份,只认
+`ledger.pre-download-*` 与 `ledger.pre-organize-*`,缺省只留最近几份、不按年龄保留;
+每次整理写备份时也按这套规则清一遍。
"""
from __future__ import annotations
@@ -15,24 +19,35 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from peach import ledger_backups # noqa: E402
-from peach.config import DATABASE_PATH # noqa: E402
+from peach.config import DATABASE_PATH, GENERATED_DIR # noqa: E402
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--db", type=Path, default=DATABASE_PATH, help="当前账本路径")
- parser.add_argument("--keep", type=int, default=ledger_backups.KEEP_RECENT,
+ parser.add_argument("--organize", type=Path, nargs="?", const=GENERATED_DIR, default=None,
+ metavar="GENERATED",
+ help="改清整理备份;可给 generated 目录,缺省取当前配置")
+ parser.add_argument("--keep", type=int, default=None,
help="无论多旧都保留的最近份数")
- parser.add_argument("--min-age-hours", type=float,
- default=ledger_backups.MIN_AGE.total_seconds() / 3600,
+ parser.add_argument("--min-age-hours", type=float, default=None,
help="不满这个小时数的备份一律保留")
parser.add_argument("--apply", action="store_true", help="真的删除;缺省只打印计划")
parser.add_argument("--json", action="store_true", help="机器可读输出")
args = parser.parse_args(argv)
+ if args.organize is not None:
+ scope = {"directory": ledger_backups.organize_dir(args.organize.resolve()),
+ "pattern": ledger_backups.ORGANIZE_PATTERN}
+ keep, min_age = ledger_backups.ORGANIZE_KEEP_RECENT, ledger_backups.ORGANIZE_MIN_AGE
+ else:
+ scope = {}
+ keep, min_age = ledger_backups.KEEP_RECENT, ledger_backups.MIN_AGE
decided = ledger_backups.prune(
- args.db.resolve(), apply=args.apply, keep_recent=args.keep,
- min_age=timedelta(hours=args.min_age_hours),
+ args.db.resolve(), apply=args.apply,
+ keep_recent=keep if args.keep is None else args.keep,
+ min_age=min_age if args.min_age_hours is None else timedelta(hours=args.min_age_hours),
+ **scope,
)
if args.json:
print(json.dumps({
diff --git a/src/peach/download_postprocess.py b/src/peach/download_postprocess.py
index bc6095568..e1b4c1c74 100644
--- a/src/peach/download_postprocess.py
+++ b/src/peach/download_postprocess.py
@@ -5,7 +5,7 @@ import time
import json
from pathlib import Path
-from . import junk, organize, settings_file
+from . import junk, ledger_backups, organize, settings_file
from .catalog_rules import same_release_code
from .downloads import (INGESTED, POSTPROCESS_COMPLETE, POSTPROCESS_FAILED,
POSTPROCESS_PENDING, POSTPROCESS_QUEUED, POSTPROCESS_RUNNING,
@@ -14,7 +14,6 @@ from .field_owners import SCAN_FILENAME, write_owned_fields
from .followups import FollowupType, register
from .library_processing import COLLECT_STAGE, _ffprobe_path, process_library
from .media_probe import probe_unmeasured
-from .migrations import sqlite_backup
from .task_runs import declared_rows
TASK_KEY = "download-postprocess"
@@ -138,11 +137,10 @@ def _organize(contract, task, main: int) -> dict:
raise RuntimeError('文件整理未完成:' + ';'.join(str(row['reason']) for row in blocked))
if not movable:
return {"moved": 0, "skipped": next(iter(plan["reasons"]), "无需整理")}
- backup = Path(contract.candidate_root) / "organize-backups" / (
- f"ledger.pre-download-{task.id}-{time.strftime('%Y%m%d-%H%M%S')}.db")
- backup.parent.mkdir(parents=True, exist_ok=True)
- sqlite_backup(Path(contract.db_path), backup)
- with contract.database.write_transaction() as connection:
+ name = f"ledger.pre-download-{task.id}-{time.strftime('%Y%m%d-%H%M%S')}.db"
+ with ledger_backups.organize_backup(Path(contract.db_path), contract.candidate_root,
+ name) as backup, \
+ contract.database.write_transaction() as connection:
integrity = connection.execute('PRAGMA integrity_check').fetchone()[0]
if integrity != 'ok':
raise RuntimeError(f'整理前 ledger 校验失败:integrity={integrity}')
diff --git a/src/peach/ledger_backups.py b/src/peach/ledger_backups.py
index 11e1ae0b8..84b2f6799 100644
--- a/src/peach/ledger_backups.py
+++ b/src/peach/ledger_backups.py
@@ -9,24 +9,44 @@
主文件已经不在的 `-wal`/`-shm` 另外扫一遍。清退按 `.db` 列举,边车只跟着主文件走,所以
主文件删掉而边车没删掉的那一刻,这几个文件就再没有人会看它们一眼:下一轮列不到,只会
一直留在账本目录里。2026-09-13 在真实数据目录数到 20 个,最早的来自 08-21。
+
+整理(下载后整理与手动整理)的备份放在 `<generated>/organize-backups/`,由
+`organize_backup` 写出并在同一处清退,只认 `ORGANIZE_PATTERN` 两种文件名。下载一波能在
+一小时里整理四十个文件,按年龄保留在这里等于不设上限,所以这一类只按份数留
+(`ORGANIZE_KEEP_RECENT`,`ORGANIZE_MIN_AGE` 为零);整理进行中的那份由 `holding`
+登记,同一进程里的清退都不碰它。登记不跨进程:手动脚本给 `--keep 0` 时只靠份数保护。
"""
from __future__ import annotations
import logging
import re
import sqlite3
+import threading
+from collections.abc import Iterator
+from contextlib import contextmanager
from dataclasses import dataclass
from datetime import datetime, timedelta
from pathlib import Path
+from .migrations import sqlite_backup
+
log = logging.getLogger(__name__)
BACKUP_PATTERN = re.compile(r"^ledger\.pre-.+\.db$")
SIDECAR_SUFFIXES = ("-wal", "-shm")
-SIDECAR_PATTERN = re.compile(r"^ledger\.pre-.+\.db(?:" + "|".join(SIDECAR_SUFFIXES) + r")$")
KEEP_RECENT = 5
MIN_AGE = timedelta(hours=24)
+ORGANIZE_DIRNAME = "organize-backups"
+ORGANIZE_PATTERN = re.compile(r"^ledger\.pre-(?:download|organize)-.+\.db$")
+ORGANIZE_KEEP_RECENT = 5
+ORGANIZE_MIN_AGE = timedelta(0)
+
+#: 本进程里正在被一轮写入引用的备份。下载后整理与手动整理都在 Web 服务进程的线程里跑,
+#: 一方清退时要看得见另一方手上的那份。
+_held: set[Path] = set()
+_held_lock = threading.Lock()
+
@dataclass(frozen=True)
class BackupPlan:
@@ -37,6 +57,47 @@ class BackupPlan:
orphans: tuple[Path, ...] = () # 主文件已不在的 `-wal`/`-shm`,和 remove 一起删
+@contextmanager
+def holding(path: Path) -> Iterator[Path]:
+ """登记一份正在使用的备份;登记期间本进程里的清退都把它留下。"""
+ key = path.resolve()
+ with _held_lock:
+ _held.add(key)
+ try:
+ yield path
+ finally:
+ with _held_lock:
+ _held.discard(key)
+
+
+def _is_held(path: Path) -> bool:
+ with _held_lock:
+ return path.resolve() in _held
+
+
+def organize_dir(generated_root: Path | str) -> Path:
+ return Path(generated_root) / ORGANIZE_DIRNAME
+
+
+@contextmanager
+def organize_backup(live: Path, generated_root: Path | str, name: str) -> Iterator[Path]:
+ """整理前写一份整库备份,整理期间持有它,并清退同目录里超出保留份数的旧备份。
+
+ 清退失败只记日志:少删几份旧备份不能让这次整理失败。
+ """
+ if not ORGANIZE_PATTERN.match(name):
+ raise ValueError(f"整理备份文件名不合规则:{name}")
+ backup = organize_dir(generated_root) / name
+ sqlite_backup(live, backup)
+ with holding(backup):
+ try:
+ prune(live, apply=True, directory=backup.parent, pattern=ORGANIZE_PATTERN,
+ keep_recent=ORGANIZE_KEEP_RECENT, min_age=ORGANIZE_MIN_AGE)
+ except Exception:
+ log.warning("整理备份清退失败", exc_info=True)
+ yield backup
+
+
def _size_with_sidecars(path: Path) -> int:
total = 0
for candidate in (path, *(path.with_name(path.name + s) for s in SIDECAR_SUFFIXES)):
@@ -47,28 +108,29 @@ def _size_with_sidecars(path: Path) -> int:
return total
-def list_backups(live: Path) -> list[Path]:
- """同目录下所有备份,按修改时间从旧到新。"""
- if not live.parent.is_dir():
+def list_backups(directory: Path, pattern: re.Pattern[str] = BACKUP_PATTERN) -> list[Path]:
+ """目录下所有备份,按修改时间从旧到新。"""
+ if not directory.is_dir():
return []
found = [
- path for path in live.parent.iterdir()
- if path.is_file() and BACKUP_PATTERN.match(path.name)
+ path for path in directory.iterdir()
+ if path.is_file() and pattern.match(path.name)
]
return sorted(found, key=lambda path: path.stat().st_mtime)
-def orphan_sidecars(live: Path) -> list[Path]:
+def orphan_sidecars(directory: Path, pattern: re.Pattern[str] = BACKUP_PATTERN) -> list[Path]:
"""主文件已经不在的备份边车,按名字排序。"""
- if not live.parent.is_dir():
+ if not directory.is_dir():
return []
found = []
- for path in live.parent.iterdir():
- if not path.is_file() or not SIDECAR_PATTERN.match(path.name):
+ for path in directory.iterdir():
+ if not path.is_file():
continue
- base = path.with_name(path.name.rsplit("-", 1)[0])
- if not base.is_file():
- found.append(path)
+ for suffix in SIDECAR_SUFFIXES:
+ base = path.name.removesuffix(suffix)
+ if base != path.name and pattern.match(base) and not (directory / base).is_file():
+ found.append(path)
return sorted(found)
@@ -97,12 +159,16 @@ def _live_mtime(live: Path) -> float:
def plan(
live: Path,
*,
+ directory: Path | None = None,
+ pattern: re.Pattern[str] = BACKUP_PATTERN,
keep_recent: int = KEEP_RECENT,
min_age: timedelta = MIN_AGE,
now: datetime | None = None,
) -> BackupPlan:
- backups = list_backups(live)
- orphans = tuple(orphan_sidecars(live))
+ """`directory` 缺省是账本所在目录;完整性检查总是针对 `live`。"""
+ folder = directory if directory is not None else live.parent
+ backups = list_backups(folder, pattern)
+ orphans = tuple(orphan_sidecars(folder, pattern))
if not backups and not orphans:
return BackupPlan((), ())
if not live.is_file():
@@ -118,7 +184,7 @@ def plan(
for backup in backups:
mtime = backup.stat().st_mtime
young = moment - datetime.fromtimestamp(mtime) < min_age
- if backup in recent or young or mtime > live_mtime:
+ if backup in recent or young or mtime > live_mtime or _is_held(backup):
keep.append(backup)
else:
remove.append(backup)
@@ -132,12 +198,15 @@ def prune(
live: Path,
*,
apply: bool,
+ directory: Path | None = None,
+ pattern: re.Pattern[str] = BACKUP_PATTERN,
keep_recent: int = KEEP_RECENT,
min_age: timedelta = MIN_AGE,
now: datetime | None = None,
) -> BackupPlan:
"""算出计划;`apply` 为真且没有拒绝理由时真的删。删不掉的记日志、继续。"""
- decided = plan(live, keep_recent=keep_recent, min_age=min_age, now=now)
+ decided = plan(live, directory=directory, pattern=pattern,
+ keep_recent=keep_recent, min_age=min_age, now=now)
if not apply or decided.refused:
return decided
doomed = [target for backup in decided.remove
diff --git a/src/peach/web_organize.py b/src/peach/web_organize.py
index b4b2c3ca1..55f1de0f6 100644
--- a/src/peach/web_organize.py
+++ b/src/peach/web_organize.py
@@ -12,8 +12,7 @@ import sqlite3
import time
from pathlib import Path
-from . import junk, organize
-from .migrations import sqlite_backup
+from . import junk, ledger_backups, organize
from .media_sources import SCANNABLE_LOCATIONS
from .organize_templates import PLACEHOLDERS, TemplateError, example_path, validate_template
from .platform import location_roots
@@ -86,22 +85,21 @@ def _run_write(contract, job_id: str, work) -> dict:
中心的进度也要拿同一把锁,同一个线程里就是死锁。逐条提交让每个事务都很短,
并发的普通写入照常进行。
"""
- backup = Path(contract.candidate_root) / "organize-backups" / (
- f"ledger.pre-organize-{time.strftime('%Y%m%d-%H%M%S')}-{job_id}.db")
- backup.parent.mkdir(parents=True, exist_ok=True)
- sqlite_backup(Path(contract.db_path), backup)
- connection = sqlite3.connect(str(contract.db_path), timeout=30)
- connection.row_factory = sqlite3.Row
- try:
- before, _ = verify_after_write(connection)
- if before != 'ok':
- raise RuntimeError(f'写入前 ledger 校验失败:integrity={before}')
- baseline = {tuple(row) for row in connection.execute('PRAGMA foreign_key_check')}
- result = work(connection)
- integrity, _ = verify_after_write(connection)
- introduced = {tuple(row) for row in connection.execute('PRAGMA foreign_key_check')} - baseline
- finally:
- connection.close()
+ name = f"ledger.pre-organize-{time.strftime('%Y%m%d-%H%M%S')}-{job_id}.db"
+ with ledger_backups.organize_backup(Path(contract.db_path), contract.candidate_root,
+ name) as backup:
+ connection = sqlite3.connect(str(contract.db_path), timeout=30)
+ connection.row_factory = sqlite3.Row
+ try:
+ before, _ = verify_after_write(connection)
+ if before != 'ok':
+ raise RuntimeError(f'写入前 ledger 校验失败:integrity={before}')
+ baseline = {tuple(row) for row in connection.execute('PRAGMA foreign_key_check')}
+ result = work(connection)
+ integrity, _ = verify_after_write(connection)
+ introduced = {tuple(row) for row in connection.execute('PRAGMA foreign_key_check')} - baseline
+ finally:
+ connection.close()
contract.cache_bust()
if integrity != "ok" or introduced:
raise RuntimeError(
diff --git a/tests/test_downloads_postprocess.py b/tests/test_downloads_postprocess.py
index 38562a819..8f92b768b 100644
--- a/tests/test_downloads_postprocess.py
+++ b/tests/test_downloads_postprocess.py
@@ -1,11 +1,14 @@
"""下载后处理在独立账本和临时媒体上的失败与重试。"""
+import os
import tempfile
+import time
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import Mock, patch
from peach import download_postprocess as post
+from peach import ledger_backups
from peach.downloads import DownloadStore, DownloadTask, INGESTED, POSTPROCESS_PENDING
from peach.repository import LedgerDatabase
from support.ledger import fresh_ledger
@@ -68,6 +71,22 @@ class DownloadPostprocessTests(unittest.TestCase):
self.assertEqual(self.store.get(self.task.id).postprocess_state, 'complete')
self.assertTrue(self.target.exists())
+ def test_organize_backup_lands_in_the_organize_folder_and_old_ones_are_pruned(self):
+ folder = ledger_backups.organize_dir(self.contract.candidate_root)
+ folder.mkdir(parents=True)
+ old = []
+ for index in range(ledger_backups.ORGANIZE_KEEP_RECENT + 2):
+ path = folder / f'ledger.pre-download-{index}-20260801-000000.db'
+ path.write_bytes(b'backup')
+ moment = time.time() - 30 * 86400 + index
+ os.utime(path, (moment, moment))
+ old.append(path)
+ result = post._organize(self.contract, self.task, self.asset_id)
+ backup = Path(result['backup'])
+ self.assertEqual(backup.parent, folder)
+ self.assertEqual(ledger_backups.list_backups(folder, ledger_backups.ORGANIZE_PATTERN),
+ [*old[3:], backup])
+
def test_retry_uses_the_bound_asset_after_a_move(self):
post._organize(self.contract, self.task, self.asset_id)
main, path, _ = post._bind_main(self.contract, self.store.get(self.task.id))
diff --git a/tests/test_ledger_backups.py b/tests/test_ledger_backups.py
index 5157c171c..7a596a7ba 100644
--- a/tests/test_ledger_backups.py
+++ b/tests/test_ledger_backups.py
@@ -1,4 +1,6 @@
+import contextlib
import importlib.util
+import io
import json
import os
import sqlite3
@@ -6,8 +8,9 @@ import tempfile
import unittest
from datetime import datetime, timedelta
from pathlib import Path
+from types import SimpleNamespace
-from peach import ledger_backups
+from peach import ledger_backups, web_organize
ROOT = Path(__file__).resolve().parents[1]
@@ -197,5 +200,138 @@ class LedgerBackupRetentionTests(unittest.TestCase):
self.assertFalse(old.exists())
+class OrganizeBackupRetentionTests(unittest.TestCase):
+ """整理备份放在 generated 下,只按份数留,进行中的那份任何清退都不碰。"""
+
+ make_ledger = LedgerBackupRetentionTests.make_ledger
+ make_backup = LedgerBackupRetentionTests.make_backup
+
+ def layout(self, root: Path) -> tuple[Path, Path, Path]:
+ database = root / "database"
+ database.mkdir()
+ generated = root / "generated"
+ folder = ledger_backups.organize_dir(generated)
+ folder.mkdir(parents=True)
+ return self.make_ledger(database, when=datetime.now()), generated, folder
+
+ def aged(self, folder: Path, names: list[str]) -> list[Path]:
+ """一个月前写的一批备份,越靠后越新。"""
+ return [self.make_backup(folder, name, datetime.now() - timedelta(days=30, hours=-index))
+ for index, name in enumerate(names)]
+
+ def test_keeps_only_the_recent_count_and_touches_nothing_but_the_two_names(self):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory).resolve()
+ live, _generated, folder = self.layout(root)
+ backups = [
+ self.make_backup(
+ folder,
+ f"ledger.pre-download-{index}-20260905-10{index:02d}00.db" if index % 2
+ else f"ledger.pre-organize-20260905-10{index:02d}00-job{index}.db",
+ NOW - timedelta(hours=2, minutes=-index))
+ for index in range(8)
+ ]
+ stamp(live, NOW)
+ foreign = self.make_backup(folder, "ledger.pre-migrate-20260801.db",
+ NOW - timedelta(days=30))
+ beside = self.make_backup(live.parent, "ledger.pre-old.db", NOW - timedelta(days=30))
+
+ decided = ledger_backups.prune(
+ live, apply=True, directory=folder, pattern=ledger_backups.ORGANIZE_PATTERN,
+ keep_recent=ledger_backups.ORGANIZE_KEEP_RECENT,
+ min_age=ledger_backups.ORGANIZE_MIN_AGE, now=NOW)
+
+ self.assertEqual(ledger_backups.ORGANIZE_KEEP_RECENT, 5)
+ self.assertEqual(set(decided.remove), set(backups[:3]),
+ "两小时内的也只留最近 5 份:下载一波一小时能写四十份")
+ self.assertEqual(set(decided.keep), set(backups[3:]))
+ for path in backups[:3]:
+ self.assertFalse(path.exists())
+ for path in (*backups[3:], foreign, beside, live):
+ self.assertTrue(path.exists(), "只认两种整理备份名,也不碰账本目录")
+
+ def test_a_held_backup_survives_any_prune_until_released(self):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory).resolve()
+ live, _generated, folder = self.layout(root)
+ backup = self.make_backup(folder, "ledger.pre-organize-20260801-000000-a.db",
+ NOW - timedelta(days=30))
+ scope = {"directory": folder, "pattern": ledger_backups.ORGANIZE_PATTERN,
+ "keep_recent": 0, "min_age": timedelta(0), "now": NOW}
+
+ with ledger_backups.holding(backup):
+ decided = ledger_backups.prune(live, apply=True, **scope)
+ self.assertEqual(decided.remove, ())
+ self.assertTrue(backup.exists(), "整理还在用的那份不能删")
+
+ ledger_backups.prune(live, apply=True, **scope)
+ self.assertFalse(backup.exists())
+
+ def test_organize_backup_writes_a_copy_and_prunes_down_to_the_recent_count(self):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory).resolve()
+ live, generated, folder = self.layout(root)
+ old = self.aged(folder, [f"ledger.pre-download-{index}-20260801-000000.db"
+ for index in range(6)])
+ name = "ledger.pre-download-99-20260905-120000.db"
+
+ with ledger_backups.organize_backup(live, generated, name) as backup:
+ self.assertEqual(backup, folder / name)
+ self.assertEqual(ledger_backups.integrity(backup), "ok", "是一份完整的账本副本")
+ remaining = ledger_backups.list_backups(folder, ledger_backups.ORGANIZE_PATTERN)
+ self.assertEqual(remaining, [*old[2:], backup])
+ ledger_backups.prune(live, apply=True, directory=folder,
+ pattern=ledger_backups.ORGANIZE_PATTERN,
+ keep_recent=0, min_age=timedelta(0))
+ self.assertTrue(backup.exists(), "同进程另一处清退也看得见这份在用")
+
+ with self.assertRaises(ValueError):
+ with ledger_backups.organize_backup(live, generated, "ledger.pre-migrate-x.db"):
+ pass
+ self.assertFalse((folder / "ledger.pre-migrate-x.db").exists())
+
+ def test_manual_organize_write_backs_up_into_the_organize_folder(self):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory).resolve()
+ live, generated, folder = self.layout(root)
+ old = self.aged(folder, [f"ledger.pre-organize-20260801-00000{index}-j{index}.db"
+ for index in range(6)])
+ contract = SimpleNamespace(db_path=live, candidate_root=generated,
+ cache_bust=lambda: None)
+ seen = []
+
+ def work(_connection):
+ seen.extend(ledger_backups.list_backups(folder, ledger_backups.ORGANIZE_PATTERN))
+ return {"moved": 0}
+
+ result = web_organize._run_write(contract, "job1", work)
+
+ backup = Path(result["backup"])
+ self.assertEqual(backup.parent, folder)
+ self.assertEqual(seen, [*old[2:], backup], "写入开始前已清到最近 5 份")
+ self.assertTrue(backup.exists())
+
+ def test_script_organize_mode_plans_the_organize_folder_only(self):
+ script = load_script()
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory).resolve()
+ live, generated, folder = self.layout(root)
+ backups = self.aged(folder, [f"ledger.pre-download-{index}-20260801-000000.db"
+ for index in range(7)])
+ beside = self.make_backup(live.parent, "ledger.pre-old.db",
+ datetime.now() - timedelta(days=30))
+
+ buffer = io.StringIO()
+ with contextlib.redirect_stdout(buffer):
+ code = script.main(["--db", str(live), "--organize", str(generated), "--json"])
+
+ self.assertEqual(code, 0)
+ report = json.loads(buffer.getvalue())
+ self.assertFalse(report["applied"])
+ self.assertEqual(report["remove"], [path.name for path in backups[:2]])
+ for path in (*backups, beside):
+ self.assertTrue(path.exists(), "缺省是 dry-run")
+
+
if __name__ == "__main__":
unittest.main()