Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
fix: sync renames a note's row in place when its id changes (#159)
Francois du Plessis214ef6658211
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3133d71..0d6d2e9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,8 @@## [Unreleased]+- **Sync renames a note's row in place when its `id:` changes (#159, from @fduple).** An edited `id:` went through the upsert as an insert of the new id, which `UNIQUE(notes.path)` refused on every sync; the stale row kept the old id and collided with the file that really owned it. Sync now moves the row to the new id, keeps the rows only the database holds (claims, embeddings, assets, and the `sources` and `escalations` rows for its URL), rebuilds the rest from the markdown, and frees the old id in the same pass. Updates run before adds so a new file named after the freed id indexes in one run. A rename is refused, with an error naming the file, when the frontmatter declares no id or does not parse, or when a new file named after the target id claims it. If SQLite rolls the whole sync transaction back mid-pass, sync stops and reports that nothing from the pass was kept.+- **`note rm` removes the asset files its own rows name (#159, from @fduple).** After an id rename a note's asset files stay in `research/assets/<old id>/`, so removing `research/assets/<id>/` wholesale could delete another note's files and leave the removed note's own behind. `note rm` now removes the files the note's `assets` rows name anywhere under `research/assets/`, plus what `research/assets/<id>/` holds that no other note's row names, never a file another note's row names, and nothing outside `research/assets/`. Symlinks are unlinked as links, a symlinked `research/assets/<id>` is left alone, and a file that cannot be removed is listed under `assets_not_removed` without stopping the delete.- **BEHAVIOR CHANGE — `install <path>` refuses a path that does not exist unless `--create` (#155, contributed by @fduple).** `hyperresearch install` and `install --steps-only` create missing parents, and with no TTY nothing stood in between, so a stale or mistyped path came back as a new empty vault with exit 0 while the real vault kept its old skills. Both now check the path before writing anything: a missing path fails with `PATH_NOT_FOUND`, and a file, as the path itself or as one of its parents, fails with `NOT_A_DIRECTORY`. `--create` makes the directory and any missing parents first (a failure there is `CREATE_FAILED`); it cannot help when a file is in the way. The check runs before the interactive setup route too, so from a terminal `install <new-dir>`, which used to open setup and create the directory, is now refused the same way; `hyperresearch setup <path>` still creates it, and setup's non-TTY / `--json` handoff to install passes `--create`. An existing directory without a vault is the normal first-time install and is unchanged, as is `--global`. Scripts that install into a directory they have not made yet need `--create`.- **The Codex Stop hook stays in the project.** A vault at the home directory (where `install` or the first-run bootstrap was run from `~`) made `<root>/.codex/hooks.json` Codex's user-level hooks file, so the stop gate ran in every Codex session. It finds a vault by walking up from the session's directory, so a home-vault run that was mid-pipeline would force-continue unrelated sessions anywhere under `~`. The hook is now skipped, with a "NOT installed" note, whenever the project's `.codex/` is `$CODEX_HOME` (default `~/.codex`). Separately, the first-run bootstrap inside a Codex session crashed with `OSError: Read-only file system`, after writing the step files, because the workspace-write sandbox keeps `.codex/` read-only. It now reports the hook as NOT installed together with the command to run outside the sandbox, and the entry skill tells the orchestrator to re-run that command once with escalated permissions.diff --git a/src/hyperresearch/cli/note.py b/src/hyperresearch/cli/note.pyindex bca0cbe..76fbc77 100644--- a/src/hyperresearch/cli/note.py+++ b/src/hyperresearch/cli/note.py@@ -2,7 +2,10 @@from __future__ import annotations+import os+import statfrom datetime import UTC+from pathlib import Pathimport typer@@ -630,6 +633,32 @@ def note_mv(console.print(f"[green]Moved:[/] {row['path']} → {rel_new}")+def _file_identity(path: Path) -> tuple | None:+ """What makes `path` the same file as another spelling of it, or None+ when no regular file is there.++ The device and inode, so case, symlinks and `..` do not matter.+ """+ try:+ st = os.stat(path)+ except OSError:+ return None+ if not stat.S_ISREG(st.st_mode):+ return None+ return (st.st_dev, st.st_ino)+++def _located(path: Path) -> Path:+ """`path` with its directory resolved; a symlink stays where it sits."""+ return path.parent.resolve() / path.name+++def _is_below(path: Path, root: Path) -> bool:+ """Whether `path`, symlinks resolved, lies strictly below the directory `root`."""+ where, root = path.resolve(), root.resolve()+ return where != root and where.is_relative_to(root)++@app.command("rm")def note_rm(note_id: str = typer.Argument(..., help="Note ID to delete"),@@ -643,10 +672,13 @@ def note_rm(Every fetch-then-delete cycle leaked disk. The current implementationalso removes:- the raw file referenced in the note's `raw_file` frontmatter field- - any entries in the `assets` table and their files on disk+ - the files under `research/assets/` that the note's rows in the+ `assets` table name, and the rest of `research/assets/<id>/` unless+ that is a symlink, except a file another note's row names; nothing+ outside `research/assets/` is removed+ A file that cannot be removed does not stop the delete; it is reported+ under `assets_not_removed`."""- import shutil-from hyperresearch.core.vault import Vaultvault = Vault.discover()@@ -692,13 +724,109 @@ def note_rm(except Exception:pass- # Assets directory- assets_dir = vault.root / "research" / "assets" / note_id- if assets_dir.exists() and assets_dir.is_dir():- for asset_file in assets_dir.iterdir():- if asset_file.is_file():- removed_assets.append(asset_file.name)- shutil.rmtree(assets_dir, ignore_errors=True)+ # Assets. A row names its file by full path, and the file need not sit+ # under research/assets/<note_id>/: fetch saves a note's files under the+ # stem of its file at the time, and sync renames a row in place when the+ # file's `id:` is rewritten, leaving the files where they are. So the+ # files to remove are the ones this note's own rows name anywhere under+ # research/assets/, plus whatever research/assets/<note_id>/ holds that+ # no other note's row names (fetch can write a file it records no row+ # for). A file another note's row names stays, in whichever directory it+ # is. Files are compared by identity, not by spelling. Nothing is removed+ # whose path, symlinks resolved, is not below this vault's+ # research/assets/: a row naming a file elsewhere (a copy of the vault+ # still holds the original's paths) is reported and its file left alone.+ assets_root = vault.root / "research" / "assets"+ assets_dir = assets_root / note_id+ own_files: list[Path] = []+ others: set[tuple] = set()+ # Where other notes' rows point, as spelled: a symlink a row names is+ # that note's, whatever it points at.+ other_paths: set[Path] = set()+ for r in vault.db.execute("SELECT note_id, filename FROM assets"):+ if r["note_id"] == note_id:+ own_files.append(Path(r["filename"]))+ else:+ other_paths.add(_located(Path(r["filename"])))+ ident = _file_identity(Path(r["filename"]))+ if ident is not None:+ others.add(ident)+ seen: set[tuple] = set()+ seen_links: set[Path] = set()+ assets_not_removed: list[str] = []+ # The directories a removal may have emptied.+ dirs: set[Path] = set()++ def _label(path: Path) -> str:+ # Vault-relative; a symlink is named where it sits, not by its target.+ try:+ return _located(path).relative_to(vault.root.resolve()).as_posix()+ except ValueError:+ return str(path)++ def _remove_link(path: Path) -> None:+ # A symlink is removed as itself, dangling or not: unlinking it never+ # touches its target, wherever that is. Only a link that sits below+ # research/assets/ goes, and not one another note's row names.+ where = _located(path)+ if where in other_paths or where in seen_links:+ return+ seen_links.add(where)+ if not where.parent.is_relative_to(assets_root.resolve()):+ assets_not_removed.append(f"{path}: outside research/assets/, left in place")+ return+ label = _label(path)+ try:+ path.unlink()+ except OSError as exc:+ assets_not_removed.append(f"{label}: {exc.strerror or exc}")+ return+ removed_assets.append(label)+ dirs.add(path.parent)++ def _remove_file(path: Path) -> None:+ if path.is_symlink():+ _remove_link(path)+ return+ ident = _file_identity(path)+ if ident is None or ident in others or ident in seen:+ return+ seen.add(ident)+ if not _is_below(path, assets_root):+ assets_not_removed.append(f"{path}: outside research/assets/, left in place")+ return+ label = _label(path)+ try:+ path.unlink()+ except OSError as exc:+ assets_not_removed.append(f"{label}: {exc.strerror or exc}")+ return+ removed_assets.append(label)+ dirs.add(path.parent)++ for path in own_files:+ _remove_file(path)+ # A symlinked research/assets/<note_id> points at a directory that is not+ # this note's to empty or remove, inside the vault or not; the link stays,+ # as before.+ if assets_dir.is_dir() and not assets_dir.is_symlink():+ dirs.add(assets_dir)+ for path in sorted(assets_dir.rglob("*")):+ if path.is_dir() and not path.is_symlink():+ dirs.add(path)+ else:+ _remove_file(path)+ # An empty one goes, deepest first: never a symlink, and only below+ # research/assets/.+ for d in sorted({_located(d) for d in dirs}, key=lambda d: len(d.parts), reverse=True):+ if d.is_symlink() or not _is_below(d, assets_root):+ continue+ try:+ if not d.is_dir() or any(d.iterdir()):+ continue+ d.rmdir()+ except OSError as exc:+ assets_not_removed.append(f"{_label(d)}/: {exc.strerror or exc}")# Finally, unlink the .md fileif file_path.exists():@@ -715,6 +843,8 @@ def note_rm(payload["removed_raw"] = removed_rawif removed_assets:payload["removed_assets"] = removed_assets+ if assets_not_removed:+ payload["assets_not_removed"] = assets_not_removedif json_output:output(success(payload, vault=str(vault.root)), json_mode=True)@@ -724,6 +854,8 @@ def note_rm(msg += f"\n raw: {removed_raw}"if removed_assets:msg += f"\n assets: {len(removed_assets)} file(s)"+ if assets_not_removed:+ msg += "\n assets not removed: " + "; ".join(assets_not_removed)console.print(msg)diff --git a/src/hyperresearch/core/note.py b/src/hyperresearch/core/note.pyindex 03cd802..ff5b6ae 100644--- a/src/hyperresearch/core/note.py+++ b/src/hyperresearch/core/note.py@@ -37,11 +37,16 @@ def read_note(file_path: Path, vault_root: Path) -> Note:content_hash = hashlib.sha256(raw_bytes).hexdigest()content = raw_bytes.decode("utf-8-sig") # Handles BOMmeta, body = parse_frontmatter(content)+ # parse_frontmatter hands the whole content back as the body when it found+ # no frontmatter it could parse; a file that opens with a delimiter all the+ # same has one, broken or half-written. A file without one is not broken.+ frontmatter_broken = body == content and re.match(r"---[ \t]*\r?\n", content) is not Nonerel_path = file_path.relative_to(vault_root).as_posix()# Derive ID from filename if not set in frontmatter- if not meta.id:+ id_declared = bool(meta.id)+ if not id_declared:meta.id = slugify(file_path.stem)# Extract outgoing wiki links, filtering citation footnotes and URLs@@ -58,6 +63,8 @@ def read_note(file_path: Path, vault_root: Path) -> Note:meta=meta,body=body,path=rel_path,+ frontmatter_broken=frontmatter_broken,+ id_declared=id_declared,content_hash=content_hash,word_count=word_count,outgoing_links=outgoing,diff --git a/src/hyperresearch/core/sync.py b/src/hyperresearch/core/sync.pyindex fbadb74..dd2fc5b 100644--- a/src/hyperresearch/core/sync.py+++ b/src/hyperresearch/core/sync.py@@ -36,6 +36,20 @@ class SyncResult:duration_ms: float = 0+class _TransactionLostError(Exception):+ """SQLite rolled the whole sync transaction back in the middle of a pass."""+++class _UnreadableNewFileError(Exception):+ """A new file named after an id in play has no id sync can read, so+ whether it claims that id is unknown."""++ def __init__(self, rel_path: str, reason: str) -> None:+ super().__init__(f"{rel_path}: {reason}")+ self.rel_path = rel_path+ self.reason = reason++def _should_exclude(rel_path: str, exclude_parts: list[str]) -> bool:"""Fast exclusion check — match on first path component."""first = rel_path.split("/", 1)[0]@@ -141,18 +155,52 @@ def execute_sync(vault, plan: SyncPlan) -> SyncResult:conn.execute("BEGIN IMMEDIATE")changed_ids: set[str] = set()+ assets_root = vault.root / "research" / "assets"# Defense-in-depth (#25): refuse to silently smash an existing row's path# field when a second file in the same pass derives the same id, or when# a new file claims an id that's already in the DB at a different path.# The collision would otherwise be order-dependent and lose data on the# next `note update`.+ #+ # The reverse map gives the id the row at a path carries, so a file whose+ # `id:` changed is renamed in place; inserted as a new id, it was refused+ # by UNIQUE(notes.path) on every run.deleted_paths = set(plan.to_delete)id_to_path: dict[str, str] = {}+ path_to_id: dict[str, str] = {}for row in conn.execute("SELECT id, path FROM notes"):if row["path"] in deleted_paths:continueid_to_path[row["id"]] = row["path"]+ path_to_id[row["path"]] = row["id"]+ # Every new file by stem: a stem can be new in more than one directory+ # (research/notes/ and research/temp/).+ new_files_by_stem: dict[str, list[Path]] = {}+ for p in plan.to_add:+ new_files_by_stem.setdefault(p.stem, []).append(p)+ renamed_from: set[str] = set()++ def _new_files_claiming(note_id: str) -> list:+ """The new files named after `note_id` whose id is `note_id`.++ Fetch writes <id>.md and records its source and assets under the+ stem, so a new file named after an id is that id's owner unless it+ declares another. One whose frontmatter is there but does not parse+ (a file caught mid-write) raises: its id is unknown.+ """+ claimants = []+ for candidate in new_files_by_stem.get(note_id, ()):+ rel = candidate.relative_to(vault.root).as_posix()+ try:+ claimant = read_note(candidate, vault.root)+ except Exception as exc:+ raise _UnreadableNewFileError(rel, str(exc)) from exc+ if claimant.frontmatter_broken:+ raise _UnreadableNewFileError(rel, "its frontmatter does not parse")+ if claimant.meta.id == note_id:+ claimants.append(claimant)+ return claimantsdef _upsert_with_collision_check(file_path: Path) -> str | None:note = read_note(file_path, vault.root)@@ -167,11 +215,94 @@ def execute_sync(vault, plan: SyncPlan) -> SyncResult:),})return None- _upsert_note_to_db(conn, note, now_iso, file_mtime=file_path.stat().st_mtime)+ old_id = path_to_id.get(rel)+ rename = old_id is not None and old_id != note.meta.id+ if rename and not note.id_declared:+ # A broken, half-written or id-less frontmatter is no reason to+ # move the note's rows and free its id for another file.+ result.errors.append({+ "path": rel,+ "error": (+ f"the indexed id is '{old_id}' but the frontmatter declares no id "+ "(missing, or the frontmatter does not parse). Not re-indexed: "+ "fix the frontmatter and run sync again."+ ),+ })+ return None+ keep_urls: set[str] = set()+ keep_assets_dir: Path | None = None+ if rename:+ try:+ new_id_claimants = _new_files_claiming(note.meta.id)+ # Rows under the old id were recorded for the file with that+ # stem: this one, or a new file named after the old id.+ old_id_claimants = [] if file_path.stem == old_id else _new_files_claiming(old_id)+ except _UnreadableNewFileError as exc:+ # A guess could take that file's id or move its URL rows.+ result.errors.append({+ "path": rel,+ "error": (+ f"the id changed from '{old_id}' to '{note.meta.id}', but new "+ f"file '{exc.rel_path}' is named after one of them and its id "+ f"cannot be read ({exc.reason}). Not re-indexed: fix that file "+ "and run sync again."+ ),+ })+ return None+ if new_id_claimants:+ # Fetch recorded that file's URL under its stem, so the id is that file's.+ result.errors.append({+ "path": rel,+ "error": (+ f"id collision: '{note.meta.id}' is claimed by new file "+ f"'{new_id_claimants[0].path}'. Skipped to avoid silent overwrite."+ ),+ })+ return None+ # Their URLs, and their assets in research/assets/<old_id>/, stay.+ keep_urls = {c.meta.source for c in old_id_claimants if c.meta.source}+ if old_id_claimants:+ keep_assets_dir = assets_root / old_id+ # A file's rename and upsert succeed or fail together, and the maps+ # change only after both: a rename left behind by a failed upsert+ # would park the row under an id a later file could take.+ conn.execute("SAVEPOINT sync_note")+ try:+ if rename:+ _rename_note_id(conn, old_id, note.meta.id, keep_urls, keep_assets_dir)+ _upsert_note_to_db(conn, note, now_iso, file_mtime=file_path.stat().st_mtime)+ except Exception:+ # On some errors (SQLITE_FULL among them) SQLite has already+ # rolled back the whole transaction, savepoints included, and a+ # ROLLBACK TO or RELEASE would raise "no such savepoint" in place+ # of the real error.+ if conn.in_transaction:+ conn.execute("ROLLBACK TO sync_note")+ conn.execute("RELEASE sync_note")+ raise+ conn.execute("RELEASE sync_note")+ if rename:+ # Free the old id for a later file in this pass.+ id_to_path.pop(old_id, None)+ changed_ids.add(old_id)+ renamed_from.add(old_id)id_to_path[note.meta.id] = rel+ path_to_id[rel] = note.meta.idchanged_ids.add(note.meta.id)return note.meta.id+ def _record_error(path: str, exc: Exception) -> None:+ result.errors.append({"path": path, "error": str(exc)})+ if not conn.in_transaction:+ # SQLite rolled back the whole transaction (SQLITE_FULL among+ # others), but the maps still free the ids of undone deletes and+ # renames: a later file would land on a restored row and take its+ # children. Stop the pass, and say so in the error.+ result.errors[-1]["error"] += (+ "; nothing from this sync pass was kept: fix that error and run sync again"+ )+ raise _TransactionLostError from exc+try:# Process deletesfor rel_path in plan.to_delete:@@ -182,23 +313,34 @@ def execute_sync(vault, plan: SyncPlan) -> SyncResult:changed_ids.add(row["id"])result.deleted += 1except Exception as e:- result.errors.append({"path": rel_path, "error": str(e)})+ _record_error(rel_path, e)- # Process adds- for file_path in plan.to_add:+ # Updates before adds: a rename frees an id that a new file in the+ # same pass may own, so this converges in one run. It also means that+ # when an id edit and a new file claim the same id, the edit wins and+ # the new file is refused as a collision, unless the new file is named+ # after that id (see _new_files_claiming).+ for file_path in plan.to_update:try:if _upsert_with_collision_check(file_path) is not None:- result.added += 1+ result.updated += 1except Exception as e:- result.errors.append({"path": str(file_path), "error": str(e)})+ _record_error(str(file_path), e)- # Process updates- for file_path in plan.to_update:+ # Process adds+ for file_path in plan.to_add:try:if _upsert_with_collision_check(file_path) is not None:- result.updated += 1+ result.added += 1except Exception as e:- result.errors.append({"path": str(file_path), "error": str(e)})+ _record_error(str(file_path), e)++ # Rows a rename kept for a claimant that then failed its own upsert+ # would fail the deferred foreign-key check: orphan them as a delete+ # of that id would (sources to NULL, assets rows gone, files kept).+ for old_id in renamed_from - set(id_to_path):+ conn.execute("UPDATE sources SET note_id = NULL WHERE note_id = ?", (old_id,))+ conn.execute("DELETE FROM assets WHERE note_id = ?", (old_id,))# Resolve links — only for changed notes' outgoing + incoming_resolve_links_incremental(conn, changed_ids)@@ -209,6 +351,11 @@ def execute_sync(vault, plan: SyncPlan) -> SyncResult:(now_iso,),)conn.commit()+ except _TransactionLostError:+ # Nothing was kept, so nothing is counted; also discard whatever a+ # statement after the rollback opened on this connection.+ conn.rollback()+ result.added = result.updated = result.deleted = 0except Exception:conn.rollback()raise@@ -217,6 +364,68 @@ def execute_sync(vault, plan: SyncPlan) -> SyncResult:return result+# Child tables keyed by note_id. A rename drops the rows the upsert rebuilds+# from the markdown, and moves the rows only the DB holds, which a delete and+# re-insert would cascade away, null out (`sources`) or leave pointing at a+# dead id (`escalations` has no foreign key).+_NOTE_ID_REBUILT_TABLES = ("note_content", "tags", "aliases")+_NOTE_ID_MOVED_TABLES = ("embeddings", "claims", "assets")+_NOTE_ID_URL_TABLES = ("sources", "escalations")+++def _rename_note_id(+ conn,+ old_id: str,+ new_id: str,+ keep_urls: set[str],+ keep_assets_dir: Path | None = None,+) -> None:+ """Move a note row to a new id in place, with the rows only the DB holds.++ No foreign key declares ON UPDATE, so the parent-key change and the+ child re-pointing cannot both pass an immediate check: the check is+ deferred to commit. The pragma lasts until the transaction ends, not+ the savepoint; no other statement in a sync pass can violate a foreign+ key (each inserts a child after its parent or deletes through the+ cascade), so that changes nothing for them. `sources` and+ `escalations` rows whose URL is in `keep_urls`, and `assets` rows whose+ file lies in `keep_assets_dir`, stay under the old id for the new file+ that owns it; the caller orphans them if no row holds that id at commit.+ Asset files never move: their rows name them by full path.++ Caller guarantees `new_id` is not in use.+ """+ from hyperresearch.core.claims import _under++ conn.execute("PRAGMA defer_foreign_keys = ON")+ for table in _NOTE_ID_REBUILT_TABLES:+ conn.execute(f"DELETE FROM {table} WHERE note_id = ?", (old_id,))+ conn.execute("DELETE FROM links WHERE source_id = ?", (old_id,))+ conn.execute("DELETE FROM notes_fts WHERE id = ?", (old_id,))+ conn.execute("UPDATE notes SET id = ? WHERE id = ?", (new_id, old_id))+ kept_assets: list[int] = []+ if keep_assets_dir is not None:+ kept_assets = [+ row["id"]+ for row in conn.execute("SELECT id, filename FROM assets WHERE note_id = ?", (old_id,))+ if _under(Path(row["filename"]), keep_assets_dir)+ ]+ for table in _NOTE_ID_MOVED_TABLES:+ not_kept = ""+ params: tuple = (new_id, old_id)+ if table == "assets" and kept_assets:+ not_kept = f" AND id NOT IN ({','.join('?' for _ in kept_assets)})"+ params += tuple(kept_assets)+ conn.execute(f"UPDATE {table} SET note_id = ? WHERE note_id = ?{not_kept}", params)+ keep = sorted(keep_urls)+ not_kept = f" AND url NOT IN ({','.join('?' for _ in keep)})" if keep else ""+ for table in _NOTE_ID_URL_TABLES:+ conn.execute(+ f"UPDATE {table} SET note_id = ? WHERE note_id = ?{not_kept}",+ (new_id, old_id, *keep),+ )++def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> None:"""Insert or update a note and all related tables using proper UPSERT."""meta = note.metadiff --git a/src/hyperresearch/models/note.py b/src/hyperresearch/models/note.pyindex b9c2b53..33aae80 100644--- a/src/hyperresearch/models/note.py+++ b/src/hyperresearch/models/note.py@@ -145,6 +145,11 @@ class Note(BaseModel):meta: NoteMetabody: strpath: str # Relative path from vault root+ # Set by read_note: whether the file opens with a frontmatter that did not+ # parse, and whether the frontmatter gave `meta.id` (otherwise the id is+ # the filename stem).+ frontmatter_broken: bool = False+ id_declared: bool = Falsecontent_hash: str = ""word_count: int = 0outgoing_links: list[str] = Field(default_factory=list)diff --git a/tests/test_cli/test_note_rm_assets.py b/tests/test_cli/test_note_rm_assets.pynew file mode 100644index 0000000..bcf7b49--- /dev/null+++ b/tests/test_cli/test_note_rm_assets.py@@ -0,0 +1,293 @@+"""Which asset files `note rm` removes, and which it keeps.++Fetch names an assets directory after the stem of the note's file at the+time, and the id under that stem can change: sync renames a note's row in+place when its `id:` is rewritten and leaves the files where fetch wrote+them. So the rows decide, not the directory name.+"""++from __future__ import annotations++import json+import os+import re+import sys+from pathlib import Path++import pytest+from typer.testing import CliRunner++from hyperresearch.cli import app+from hyperresearch.core.vault import Vault++runner = CliRunner()+++def vault_with_notes(tmp_path: Path) -> Path:+ vault_dir = tmp_path / "kb"+ runner.invoke(app, ["init", str(vault_dir)])+ os.chdir(vault_dir)+ runner.invoke(app, ["note", "new", "Alpha Note", "--tag", "test"])+ runner.invoke(app, ["note", "new", "Beta Note", "--tag", "test"])+ runner.invoke(app, ["sync"])+ return vault_dir+++def _row(vault_dir: Path, note_id: str, path: Path) -> None:+ """Record `path` as an asset of `note_id`, spelled as given."""+ vault = Vault(vault_dir)+ vault.db.execute(+ "INSERT INTO assets (note_id, type, filename, created_at) VALUES (?, 'image', ?, 't')",+ (note_id, str(path)),+ )+ vault.db.commit()+ vault.close()+++def _asset(vault_dir: Path, note_id: str, rel: str, data: bytes = b"x") -> Path:+ """Write research/assets/<rel> and record it as an asset of `note_id`."""+ path = vault_dir / "research" / "assets" / rel+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_bytes(data)+ _row(vault_dir, note_id, path)+ return path+++def _symlink(link: Path, target: Path) -> None:+ link.parent.mkdir(parents=True, exist_ok=True)+ try:+ link.symlink_to(target, target_is_directory=target.is_dir())+ except OSError as exc:+ pytest.skip(f"cannot create a symlink here: {exc}")+++def _rm(note_id: str) -> dict:+ result = runner.invoke(app, ["note", "rm", note_id, "--force", "--json"])+ assert result.exit_code == 0, result.output+ return json.loads(result.output)["data"]+++def test_note_rm_keeps_asset_files_another_note_owns(vault_with_notes):+ """A file another note's assets row names survives the delete, even when+ the note's own row names it too, and so does the directory holding it.+ The row's spelling of the path does not matter: files are matched by+ identity."""+ assets = vault_with_notes / "research" / "assets"+ own = _asset(vault_with_notes, "alpha-note", "alpha-note/own.png", b"a")+ other = assets / "alpha-note" / "other.png"+ other.write_bytes(b"b")+ (assets / "beta-note").mkdir()+ _row(vault_with_notes, "beta-note", assets / "beta-note" / ".." / "alpha-note" / "other.png")+ shared = _asset(vault_with_notes, "alpha-note", "alpha-note/shared.png", b"s")+ _row(vault_with_notes, "beta-note", shared)++ data = _rm("alpha-note")++ assert data.get("removed_assets") == ["research/assets/alpha-note/own.png"]+ assert "assets_not_removed" not in data+ assert not own.exists()+ assert other.read_bytes() == b"b"+ assert shared.read_bytes() == b"s"+ assert (assets / "alpha-note").is_dir()+++def test_note_rm_removes_the_files_its_own_rows_name_outside_its_directory(vault_with_notes):+ """After an id rewrite the note's asset rows still point into the+ directory fetch named after the file's stem, `old-id`. Those files go+ with the note, that directory goes once it is empty, another note's+ directory is left alone, and a row naming a file that is already gone+ is not an error. The note's own directory goes too, with a file no row+ names in a subdirectory (fetch can write a file it records no row for)+ and an empty subdirectory."""+ assets = vault_with_notes / "research" / "assets"+ own = _asset(vault_with_notes, "alpha-note", "old-id/figure.png", b"a")+ missing = _asset(vault_with_notes, "alpha-note", "old-id/missing.png")+ missing.unlink()+ other = _asset(vault_with_notes, "beta-note", "beta-note/shot.png", b"b")+ stray = assets / "alpha-note" / "sub" / "stray.png"+ stray.parent.mkdir(parents=True)+ stray.write_bytes(b"s")+ (assets / "alpha-note" / "empty").mkdir()++ data = _rm("alpha-note")++ assert data.get("removed_assets") == [+ "research/assets/old-id/figure.png",+ "research/assets/alpha-note/sub/stray.png",+ ]+ assert "assets_not_removed" not in data+ assert not own.exists()+ assert not (assets / "old-id").exists()+ assert other.read_bytes() == b"b"+ assert not (assets / "alpha-note").exists()+++def test_note_rm_leaves_a_file_its_row_names_outside_research_assets(vault_with_notes, tmp_path):+ """A copy of a vault keeps the original's paths in its rows. Deleting a+ note in the copy must not reach into the original: such a file is left+ alone and reported, and the copy's own directory still goes. Nothing+ above it goes either: research/assets/ stays even once empty."""+ original = tmp_path / "original" / "research" / "assets" / "alpha-note" / "shot.png"+ original.parent.mkdir(parents=True)+ original.write_bytes(b"o")+ _row(vault_with_notes, "alpha-note", original)+ copy = _asset(vault_with_notes, "alpha-note", "alpha-note/shot.png", b"c")+ loose = _asset(vault_with_notes, "alpha-note", "loose.png", b"l")++ data = _rm("alpha-note")++ assert data.get("removed_assets") == [+ "research/assets/alpha-note/shot.png",+ "research/assets/loose.png",+ ]+ assert data.get("assets_not_removed") == [f"{original}: outside research/assets/, left in place"]+ assert original.read_bytes() == b"o"+ assert not copy.exists() and not loose.exists()+ assert not copy.parent.exists()+ assert loose.parent.is_dir()+++def test_note_rm_does_not_walk_a_symlinked_assets_directory(vault_with_notes, tmp_path):+ """research/assets/<id> as a symlink to a directory outside the vault:+ nothing there is removed or listed, and the link stays."""+ outside = tmp_path / "outside"+ files = [outside / "victim.txt", outside / "deep" / "victim2.txt"]+ for path in files:+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_bytes(b"keep")+ link = vault_with_notes / "research" / "assets" / "alpha-note"+ _symlink(link, outside)++ data = _rm("alpha-note")++ assert "removed_assets" not in data and "assets_not_removed" not in data+ assert all(path.read_bytes() == b"keep" for path in files)+ assert link.is_symlink()+++def test_note_rm_keeps_the_directory_a_symlinked_assets_directory_points_to(vault_with_notes):+ """research/assets/<id> as a symlink to another note's directory. A file+ the note's own row names through the link goes, but the directory it+ sat in stays even once empty, and so does the link: neither is this+ note's to remove."""+ assets = vault_with_notes / "research" / "assets"+ target = assets / "beta-note"+ own = target / "own.png"+ own.parent.mkdir(parents=True)+ own.write_bytes(b"o")+ link = assets / "alpha-note"+ _symlink(link, target)+ _row(vault_with_notes, "alpha-note", link / "own.png")++ data = _rm("alpha-note")++ assert data.get("removed_assets") == ["research/assets/beta-note/own.png"]+ assert "assets_not_removed" not in data+ assert not own.exists()+ assert target.is_dir() and link.is_symlink()+++def test_note_rm_names_a_removed_symlink_by_its_own_path(vault_with_notes):+ """A symlink in the note's directory goes as a link: its target stays,+ and the payload names the link, not the target."""+ assets = vault_with_notes / "research" / "assets"+ target = assets / "beta-note" / "real.png"+ target.parent.mkdir(parents=True)+ target.write_bytes(b"r")+ link = assets / "alpha-note" / "link.png"+ _symlink(link, target)++ data = _rm("alpha-note")++ assert data.get("removed_assets") == ["research/assets/alpha-note/link.png"]+ assert not link.is_symlink() and target.read_bytes() == b"r"+++ sys.platform == "win32" or (hasattr(os, "geteuid") and os.geteuid() == 0),+ reason="a read-only directory does not stop unlink here",+)+def test_note_rm_reports_a_file_it_cannot_remove_and_finishes_the_delete(vault_with_notes):+ """A file that cannot be unlinked (a viewer holding it open on Windows;+ a read-only directory stands in for that here) does not abort the+ delete: the note still goes, and the file is reported as not removed,+ never as removed."""+ locked = _asset(vault_with_notes, "alpha-note", "alpha-note/locked.png", b"l")+ os.chmod(locked.parent, 0o555)+ try:+ data = _rm("alpha-note")+ finally:+ os.chmod(locked.parent, 0o755)++ assert "removed_assets" not in data+ not_removed = data.get("assets_not_removed", [])+ assert len(not_removed) == 1+ assert not_removed[0].startswith("research/assets/alpha-note/locked.png: ")+ assert locked.read_bytes() == b"l"+ assert not (vault_with_notes / "research" / "notes" / "alpha-note.md").exists()+ vault = Vault(vault_with_notes)+ assert vault.db.execute("SELECT COUNT(*) FROM notes WHERE id = 'alpha-note'").fetchone()[0] == 0+ vault.close()+++def test_note_rm_removes_symlinks_in_the_notes_directory_not_their_targets(+ vault_with_notes, tmp_path+):+ """A dangling link and a link to a file outside the vault, in the note's+ own directory: each link goes, as `shutil.rmtree` removed it, the target+ stays, and the emptied directory goes with them."""+ outside = tmp_path / "outside.txt"+ outside.write_bytes(b"keep")+ assets_dir = vault_with_notes / "research" / "assets" / "alpha-note"+ assets_dir.mkdir(parents=True)+ _symlink(assets_dir / "dangling.png", tmp_path / "missing.png")+ _symlink(assets_dir / "outside.png", outside)++ data = _rm("alpha-note")++ assert sorted(data["removed_assets"]) == [+ "research/assets/alpha-note/dangling.png",+ "research/assets/alpha-note/outside.png",+ ]+ assert "assets_not_removed" not in data+ assert outside.read_bytes() == b"keep"+ assert not assets_dir.exists()+++def test_note_rm_keeps_a_symlink_another_notes_row_names(vault_with_notes, tmp_path):+ outside = tmp_path / "outside.txt"+ outside.write_bytes(b"keep")+ assets_dir = vault_with_notes / "research" / "assets" / "alpha-note"+ assets_dir.mkdir(parents=True)+ link = assets_dir / "shared.png"+ _symlink(link, outside)+ _row(vault_with_notes, "beta-note", link)++ data = _rm("alpha-note")++ assert "removed_assets" not in data+ assert link.is_symlink()+++def test_note_rm_after_an_id_swap_keeps_the_other_notes_files(vault_with_notes):+ """End to end through sync: alpha-note's id is edited to alpha-2, then+ beta-note's to the freed alpha-note. Its files stay in+ research/assets/beta-note/, and research/assets/alpha-note/ now holds+ alpha-2's. Removing the directory named after the id would delete+ alpha-2's file and leave the removed note's own."""+ notes = vault_with_notes / "research" / "notes"+ alpha_shot = _asset(vault_with_notes, "alpha-note", "alpha-note/screenshot.png")+ beta_shot = _asset(vault_with_notes, "beta-note", "beta-note/screenshot.png")+ for name, new_id in (("alpha-note.md", "alpha-2"), ("beta-note.md", "alpha-note")):+ path = notes / name+ text = path.read_text(encoding="utf-8")+ path.write_text(re.sub(r"(?m)^id: .*$", f"id: {new_id}", text, count=1), encoding="utf-8")+ result = runner.invoke(app, ["sync", "--json"])+ assert result.exit_code == 0, result.output+ assert json.loads(result.output)["data"]["errors"] == []++ _rm("alpha-note")++ assert alpha_shot.exists()+ assert not beta_shot.exists()diff --git a/tests/test_core/test_sync.py b/tests/test_core/test_sync.pyindex 597bff0..4a45c9f 100644--- a/tests/test_core/test_sync.py+++ b/tests/test_core/test_sync.py@@ -1,5 +1,10 @@"""Tests for the sync engine."""+import os+import sqlite3++import pytest+from hyperresearch.core.sync import compute_sync_plan, execute_sync@@ -308,6 +313,337 @@ def test_sync_cleans_up_stale_collided_row(tmp_vault):row = tmp_vault.db.execute("SELECT path FROM notes WHERE id = ?", ("foo",)).fetchone()assert row["path"] == "research/notes/foo.md"++# ---------------------------------------------------------------------------+# An id rewrite on an existing path. A file whose path already has a row and+# whose `id:` now differs used to go through the upsert as an INSERT of the new+# id: UNIQUE(notes.path) refused it, the stale row kept the old id, and on the+# next run the OTHER file that legitimately owns the old id was reported as an+# id collision against the stale path, on every sync, until the rewritten file+# was renamed or moved away and back, which drops the rows only the database+# holds (its claims, embeddings and assets).+# ---------------------------------------------------------------------------+++def _note_with_id(path, note_id: str, title: str, body: str, extra: str = ""):+ """Write a note with an explicit id; `extra` is more frontmatter lines."""+ rewrite = path.exists()+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(f"---\nid: {note_id}\ntitle: {title}\n{extra}---\n\n{body}\n", encoding="utf-8")+ if rewrite:+ # A rewrite within one filesystem timestamp tick keeps the old mtime,+ # and the sync plan skips a file whose mtime did not move. Step it+ # forward so the plan sees the edit, as it would for a real one.+ st = path.stat()+ os.utime(path, (st.st_atime, st.st_mtime + 5))+++SOURCE_B = "source: https://ex.org/b\n"+++def test_sync_rewrites_a_note_id_in_place_and_frees_the_old_one(tmp_vault):+ notes = tmp_vault.notes_dir+ loser = notes / "topic-2.md"+ winner = notes / "topic.md"++ # The shadowed-id state: the `-2` file won the row for id `topic`, so the+ # real owner collides on every sync.+ _note_with_id(loser, "topic", "Topic (dup)", "Second body [[other]].", extra=SOURCE_B)+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ tmp_vault.db.execute(+ "INSERT INTO sources (url, note_id, domain, fetched_at, provider) VALUES (?, ?, ?, ?, ?)",+ ("https://ex.org/b", "topic", "ex.org", "2026-09-15T00:00:00", "builtin"),+ )+ tmp_vault.db.execute(+ "INSERT INTO claims (note_id, claim, claim_hash, ingested_at) VALUES (?, ?, ?, ?)",+ ("topic", "a claim", "h1", "2026-09-15T00:00:00"),+ )+ tmp_vault.db.commit()+ _note_with_id(winner, "topic", "Topic", "First body.")+ r = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))+ assert len(r.errors) == 1 and "id collision" in r.errors[0]["error"]++ # The repair a user would make: rewrite only the loser's `id:` line.+ _note_with_id(loser, "topic-2", "Topic (dup)", "Second body [[other]].", extra=SOURCE_B)++ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))+ assert result.errors == []+ rows = {+ r["id"]: r["path"]+ for r in tmp_vault.db.execute("SELECT id, path FROM notes ORDER BY id")+ }+ assert rows == {"topic": "research/notes/topic.md", "topic-2": "research/notes/topic-2.md"}++ # Children the markdown cannot rebuild moved with the row: the fetch dedup+ # map still knows the URL, under the new id, and the extracted claim survived.+ src = tmp_vault.db.execute("SELECT note_id FROM sources WHERE url = ?", ("https://ex.org/b",)).fetchone()+ assert src["note_id"] == "topic-2"+ claim = tmp_vault.db.execute("SELECT note_id FROM claims WHERE claim_hash = ?", ("h1",)).fetchone()+ assert claim["note_id"] == "topic-2"+ # Rebuilt children carry the right id on each side: `topic` is now the+ # winner's row, `topic-2` the loser's, and no FTS or link row dangles.+ body = tmp_vault.db.execute("SELECT body FROM note_content WHERE note_id = ?", ("topic",)).fetchone()+ assert "First body" in body["body"]+ body2 = tmp_vault.db.execute("SELECT body FROM note_content WHERE note_id = ?", ("topic-2",)).fetchone()+ assert "Second body" in body2["body"]+ links = {+ (r["source_id"], r["target_ref"])+ for r in tmp_vault.db.execute("SELECT source_id, target_ref FROM links")+ }+ assert links == {("topic-2", "other")}+ fts_ids = sorted(r["id"] for r in tmp_vault.db.execute("SELECT id FROM notes_fts"))+ assert fts_ids == ["topic", "topic-2"]++ # And the second pass is a no-op with no errors.+ plan2 = compute_sync_plan(tmp_vault)+ assert plan2.to_add == [] and plan2.to_update == [] and plan2.to_delete == []+ assert execute_sync(tmp_vault, plan2).errors == []+++def test_sync_id_rewrite_moves_every_child_row_when_nothing_takes_the_old_id(tmp_vault):+ """A plain id rewrite with no other file claiming the old id. Nothing+ later in the pass touches rows under the old id, so the rename itself+ has to leave none behind: it moves the rows only the database holds and+ drops the ones the upsert rebuilds under the new id, the two without a+ foreign key (notes_fts and links) included. The asset file stays where+ it is: its row names it by full path and keeps resolving."""+ notes = tmp_vault.notes_dir+ extra = "tags: [t1]\naliases: [Dup]\n"+ _note_with_id(notes / "other.md", "other", "Other", "Points at [[topic]].")+ _note_with_id(notes / "topic-2.md", "topic", "Topic (dup)", "Second body [[other]].", extra=extra)+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ # Rows only the database holds. `escalations.note_id` has no foreign key,+ # so nothing but the rename would notice it staying behind.+ now = "2026-09-15T00:00:00"+ tmp_vault.db.execute(+ "INSERT INTO embeddings (note_id, model, dimensions, vector, created_at) VALUES (?, ?, ?, ?, ?)",+ ("topic", "m", 1, b"\x00", now),+ )+ shot = tmp_vault.root / "research" / "assets" / "topic" / "screenshot.png"+ shot.parent.mkdir(parents=True)+ shot.write_bytes(b"png")+ tmp_vault.db.execute(+ "INSERT INTO assets (note_id, type, filename, created_at) VALUES (?, ?, ?, ?)",+ ("topic", "screenshot", str(shot), now),+ )+ tmp_vault.db.execute(+ "INSERT INTO escalations (url, reason, note_id, created_at, updated_at) VALUES (?, ?, ?, ?, ?)",+ ("https://ex.org/e", "login_wall", "topic", now, now),+ )+ tmp_vault.db.commit()++ _note_with_id(notes / "topic-2.md", "topic-2", "Topic (dup)", "Second body [[other]].", extra=extra)+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))+ assert result.errors == [] and result.updated == 1++ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"other": "research/notes/other.md", "topic-2": "research/notes/topic-2.md"}+ for table, column in (+ ("notes_fts", "id"), ("note_content", "note_id"), ("tags", "note_id"),+ ("aliases", "note_id"), ("links", "source_id"), ("embeddings", "note_id"),+ ("assets", "note_id"), ("escalations", "note_id"),+ ):+ ids = {r[0] for r in tmp_vault.db.execute(f"SELECT {column} FROM {table}")}+ assert "topic" not in ids and "topic-2" in ids, table+ # The FTS row was rebuilt under the new id: the body is searchable under+ # it, and the incoming [[topic]] link no longer resolves to anything.+ hit = tmp_vault.db.execute("SELECT id FROM notes_fts WHERE notes_fts MATCH 'second'").fetchall()+ assert [r["id"] for r in hit] == ["topic-2"]+ link = tmp_vault.db.execute("SELECT target_id FROM links WHERE source_id = 'other'").fetchone()+ assert link["target_id"] is None+ assets = [tuple(r) for r in tmp_vault.db.execute("SELECT note_id, filename FROM assets")]+ assert assets == [("topic-2", str(shot))] and shot.read_bytes() == b"png"+++def test_sync_id_rewrite_covers_every_table_with_a_foreign_key_to_notes(tmp_vault):+ """A child table added to the schema later has to be added to the rename+ too: rows it left behind would fail the commit of every pass that holds+ an id rewrite."""+ from hyperresearch.core.sync import (+ _NOTE_ID_MOVED_TABLES,+ _NOTE_ID_REBUILT_TABLES,+ _NOTE_ID_URL_TABLES,+ )++ conn = tmp_vault.db+ tables = [r[0] for r in conn.execute("SELECT name FROM sqlite_master WHERE type = 'table'")]+ referencing = {+ table for table in tables+ for fk in conn.execute(f"PRAGMA foreign_key_list('{table}')")+ if fk["table"] == "notes"+ }+ handled = {*_NOTE_ID_MOVED_TABLES, *_NOTE_ID_REBUILT_TABLES, *_NOTE_ID_URL_TABLES}+ assert referencing and referencing <= handled+++def test_sync_id_rewrite_whose_upsert_fails_leaves_the_row_and_its_children_alone(tmp_vault):+ """The rename and the upsert succeed or fail together. `a.md` rewrites+ its id from `xid` to `yid` and, in the same edit, carries a duplicate+ alias that the aliases primary key refuses inside the upsert. Its row+ stays under `xid` with its claims and sources rows, a new file that+ claims `xid` in the same pass is still a collision, and a new file that+ claims `yid`, which nothing owns, indexes. `d.md`, an ordinary edit with+ the same bad alias, fails the same way. Both failed files keep their old+ content hash, so the next run reports them again rather than counting+ them as synced."""+ notes = tmp_vault.notes_dir+ _note_with_id(notes / "a.md", "xid", "A", "A body.")+ _note_with_id(notes / "d.md", "dee", "D", "D body.")+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ tmp_vault.db.execute(+ "INSERT INTO sources (url, note_id, domain, fetched_at, provider) VALUES (?, ?, ?, ?, ?)",+ ("https://ex.org/a", "xid", "ex.org", "2026-09-15T00:00:00", "builtin"),+ )+ tmp_vault.db.execute(+ "INSERT INTO claims (note_id, claim, claim_hash, ingested_at) VALUES (?, ?, ?, ?)",+ ("xid", "a claim", "h1", "2026-09-15T00:00:00"),+ )+ tmp_vault.db.commit()++ _note_with_id(notes / "a.md", "yid", "A", "A body.", extra="aliases: [dup, dup]\n")+ _note_with_id(notes / "b.md", "yid", "B", "B body.")+ _note_with_id(notes / "c.md", "xid", "C", "C body.")+ _note_with_id(notes / "d.md", "dee", "D", "D body, edited.", extra="aliases: [dup, dup]\n")+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))++ errors = {os.path.basename(e["path"]): e["error"] for e in result.errors}+ assert set(errors) == {"a.md", "c.md", "d.md"}+ assert "aliases" in errors["a.md"] and "aliases" in errors["d.md"]+ assert "id collision: 'xid' already belongs to 'research/notes/a.md'" in errors["c.md"]+ assert result.added == 1 and result.updated == 0+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"xid": "research/notes/a.md", "yid": "research/notes/b.md", "dee": "research/notes/d.md"}+ assert [r["note_id"] for r in tmp_vault.db.execute("SELECT note_id FROM claims")] == ["xid"]+ assert [r["note_id"] for r in tmp_vault.db.execute("SELECT note_id FROM sources")] == ["xid"]+ assert tmp_vault.db.execute("SELECT COUNT(*) FROM aliases").fetchone()[0] == 0+ fts_ids = sorted(r["id"] for r in tmp_vault.db.execute("SELECT id FROM notes_fts"))+ assert fts_ids == ["dee", "xid", "yid"]+ body = tmp_vault.db.execute("SELECT body FROM note_content WHERE note_id = 'dee'").fetchone()+ assert "edited" not in body["body"]+ assert tmp_vault.db.execute("PRAGMA foreign_key_check").fetchall() == []++ # The next run reports both again (`a.md` now as the ordinary collision+ # with `b.md`); nothing moves.+ result2 = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))+ errors2 = {os.path.basename(e["path"]): e["error"] for e in result2.errors}+ assert set(errors2) == {"a.md", "c.md", "d.md"} and "aliases" in errors2["d.md"]+ rows2 = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows2 == rows+++def test_sync_stops_the_pass_once_the_transaction_is_lost(tmp_vault):+ """A whole-transaction rollback takes the pass's earlier writes with it,+ a delete and a rename included, while the collision maps still describe+ them. `z.md` is gone from disk, `a.md` rewrites its id from `xid` to+ `yid`, `b.md` then fills the capped database, `d.md` is an ordinary+ edit, and new `c.md` and `e.md` claim `xid` and `zed`, which the maps+ call free. Neither may land on a restored row and take its claims and+ sources rows: the pass stops at the rollback and keeps nothing, and the+ next sync does all six files."""+ notes = tmp_vault.notes_dir+ _note_with_id(notes / "a.md", "xid", "A", "A body.")+ _note_with_id(notes / "b.md", "bee", "B", "B body.")+ _note_with_id(notes / "d.md", "dee", "D", "D body.")+ _note_with_id(notes / "z.md", "zed", "Z", "Z body.")+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ tmp_vault.db.execute(+ "INSERT INTO sources (url, note_id, domain, fetched_at, provider) VALUES (?, ?, ?, ?, ?)",+ ("https://ex.org/a", "xid", "ex.org", "2026-09-15T00:00:00", "builtin"),+ )+ for note_id, claim_hash in (("xid", "h1"), ("zed", "h2")):+ tmp_vault.db.execute(+ "INSERT INTO claims (note_id, claim, claim_hash, ingested_at) VALUES (?, ?, ?, ?)",+ (note_id, "a claim", claim_hash, "2026-09-15T00:00:00"),+ )+ tmp_vault.db.commit()+ before = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ pages = tmp_vault.db.execute("PRAGMA page_count").fetchone()[0]+ tmp_vault.db.execute(f"PRAGMA max_page_count = {pages}")++ (notes / "z.md").unlink()+ _note_with_id(notes / "a.md", "yid", "A", "A body.")+ _note_with_id(notes / "b.md", "bee", "B", "B body, grown. " + "lorem ipsum " * 10000)+ _note_with_id(notes / "c.md", "xid", "C", "C body.")+ _note_with_id(notes / "d.md", "dee", "D", "D body, edited.")+ _note_with_id(notes / "e.md", "zed", "E", "E body.")+ plan = compute_sync_plan(tmp_vault)+ # The scan order is the filesystem's. Fix it, so the rename comes first.+ plan.to_update.sort(key=lambda p: p.name)+ plan.to_add.sort(key=lambda p: p.name)+ assert [p.name for p in plan.to_update] == ["a.md", "b.md", "d.md"]+ assert [p.name for p in plan.to_add] == ["c.md", "e.md"]+ assert plan.to_delete == ["research/notes/z.md"]+ result = execute_sync(tmp_vault, plan)++ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == before+ assert rows["xid"] == "research/notes/a.md"+ claims = sorted(r["note_id"] for r in tmp_vault.db.execute("SELECT note_id FROM claims"))+ assert claims == ["xid", "zed"]+ assert [r["note_id"] for r in tmp_vault.db.execute("SELECT note_id FROM sources")] == ["xid"]+ body = tmp_vault.db.execute("SELECT body FROM note_content WHERE note_id = 'dee'").fetchone()+ assert "edited" not in body["body"]+ assert len(result.errors) == 1+ assert result.errors[0]["path"].endswith("b.md")+ assert "disk is full" in result.errors[0]["error"]+ # Nothing was kept, and the summary says so.+ assert (result.added, result.updated, result.deleted) == (0, 0, 0)+ assert not tmp_vault.db.in_transaction++ # With room again, the next sync does the whole plan: the delete goes+ # first and takes `zed`'s claim with it, the rename moves the row with+ # its children, and `c.md` and `e.md` get rows of their own.+ tmp_vault.db.execute("PRAGMA max_page_count = 1073741823")+ plan2 = compute_sync_plan(tmp_vault)+ assert sorted(p.name for p in plan2.to_update) == ["a.md", "b.md", "d.md"]+ assert sorted(p.name for p in plan2.to_add) == ["c.md", "e.md"]+ result2 = execute_sync(tmp_vault, plan2)+ assert result2.errors == []+ assert (result2.added, result2.updated, result2.deleted) == (2, 3, 1)+ rows2 = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows2 == {+ "yid": "research/notes/a.md",+ "bee": "research/notes/b.md",+ "xid": "research/notes/c.md",+ "dee": "research/notes/d.md",+ "zed": "research/notes/e.md",+ }+ assert [r["note_id"] for r in tmp_vault.db.execute("SELECT note_id FROM claims")] == ["yid"]+ assert [r["note_id"] for r in tmp_vault.db.execute("SELECT note_id FROM sources")] == ["yid"]+ assert tmp_vault.db.execute("PRAGMA foreign_key_check").fetchall() == []+++def test_sync_stops_the_pass_when_a_delete_loses_the_transaction(tmp_vault, monkeypatch):+ """The same rule in the delete loop. SQLite's own rollback is stood in+ for by a delete that rolls the transaction back and raises, because a+ delete cannot be made to fill the database."""+ notes = tmp_vault.notes_dir+ for name in ("gone-1", "gone-2", "kept"):+ _note_with_id(notes / f"{name}.md", name, name, f"{name} body.")+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ (notes / "gone-1.md").unlink()+ (notes / "gone-2.md").unlink()+ _note_with_id(notes / "kept.md", "kept", "kept", "kept body, edited.")+ _note_with_id(notes / "new.md", "new", "new", "new body.")+ calls = []++ def losing_delete(conn, note_id):+ calls.append(note_id)+ conn.rollback()+ raise sqlite3.OperationalError("database or disk is full")++ monkeypatch.setattr("hyperresearch.core.sync._delete_note_from_db", losing_delete)+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))++ assert len(calls) == 1+ assert len(result.errors) == 1 and "disk is full" in result.errors[0]["error"]+ assert (result.added, result.updated, result.deleted) == (0, 0, 0)+ ids = sorted(r["id"] for r in tmp_vault.db.execute("SELECT id FROM notes"))+ assert ids == ["gone-1", "gone-2", "kept"]+ body = tmp_vault.db.execute("SELECT body FROM note_content WHERE note_id = 'kept'").fetchone()+ assert "edited" not in body["body"]++def test_sync_registers_capped_collision_notes_separately(tmp_vault):"""Two notes whose titles collide on the slug length cap must land as twoDB rows — the orphan-note bug collapsed the second into the first id and@@ -357,3 +693,285 @@ def test_sync_skips_bracketed_markdown_link_labels(tmp_vault):).fetchall())assert refs == ["real-target"]+++def test_sync_does_not_rekey_a_note_whose_frontmatter_lost_its_id(tmp_vault):+ """A file whose frontmatter stops parsing (a lost closing `---`, or a+ watch read mid-write) reads with the filename as its id. That is not an+ id edit: the row, its claims and the links to it stay as they were."""+ notes = tmp_vault.notes_dir+ _note_with_id(notes / "renamed.md", "topic", "Topic", "Body.")+ _note_with_id(notes / "other.md", "other", "Other", "See [[topic]].")+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ tmp_vault.db.execute(+ "INSERT INTO claims (note_id, claim, claim_hash, ingested_at) VALUES (?, ?, ?, ?)",+ ("topic", "a claim", "h1", "2026-09-15T00:00:00"),+ )+ tmp_vault.db.commit()++ path = notes / "renamed.md"+ path.write_text("---\nid: topic\ntitle: Topic\n\nBody, edited.\n", encoding="utf-8")+ st = path.stat()+ os.utime(path, (st.st_atime, st.st_mtime + 5))+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))++ assert len(result.errors) == 1 and "declares no id" in result.errors[0]["error"]+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"topic": "research/notes/renamed.md", "other": "research/notes/other.md"}+ assert [r["note_id"] for r in tmp_vault.db.execute("SELECT note_id FROM claims")] == ["topic"]+ link = tmp_vault.db.execute("SELECT target_id FROM links WHERE source_id = 'other'").fetchone()+ assert link["target_id"] == "topic"++[email protected]("id_line", ["id: topic\n", ""], ids=["declared", "from-the-file-name"])+def test_sync_id_rewrite_leaves_the_claimants_url_and_asset_rows_under_the_old_id(tmp_vault, id_line):+ """While `topic-2.md` held the id `topic`, the fetch that wrote the new+ `topic.md` recorded that file's URL under `topic`, the file's stem, and+ saved its assets in research/assets/topic/ under it too; sync refused+ `topic.md` as a collision. Those rows stay with the id and its new+ owner, whether `topic.md` declares the id or takes it from its file+ name; the renamed note's own URL and asset rows, which point into its+ own directory, move with it. No file moves."""+ notes = tmp_vault.notes_dir+ assets = tmp_vault.root / "research" / "assets"+ _note_with_id(notes / "topic-2.md", "topic", "Topic dup", "Second.", SOURCE_B)+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ own = assets / "topic-2" / "figure.png"+ theirs = assets / "topic" / "screenshot.png"+ for path in (own, theirs):+ path.parent.mkdir(parents=True)+ path.write_bytes(path.name.encode())+ tmp_vault.db.execute(+ "INSERT INTO assets (note_id, type, filename, created_at) VALUES ('topic', 'image', ?, 't')",+ (str(path),),+ )+ for url in ("https://ex.org/a", "https://ex.org/b"):+ tmp_vault.db.execute("INSERT INTO sources (url, note_id) VALUES (?, 'topic')", (url,))+ tmp_vault.db.commit()+ (notes / "topic.md").write_text(+ f"---\n{id_line}title: Topic\nsource: https://ex.org/a\n---\n\nFirst.\n", encoding="utf-8"+ )+ assert len(execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors) == 1++ _note_with_id(notes / "topic-2.md", "topic-2", "Topic dup", "Second.", SOURCE_B)+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))++ assert result.errors == [] and (result.added, result.updated) == (1, 1)+ rows = sorted((r["note_id"], r["filename"]) for r in tmp_vault.db.execute("SELECT note_id, filename FROM assets"))+ assert rows == [("topic", str(theirs)), ("topic-2", str(own))]+ owners = dict(tmp_vault.db.execute("SELECT url, note_id FROM sources").fetchall())+ assert owners == {"https://ex.org/a": "topic", "https://ex.org/b": "topic-2"}+ assert own.read_bytes() == b"figure.png" and theirs.read_bytes() == b"screenshot.png"+ files = sorted(p.relative_to(assets).as_posix() for p in assets.rglob("*") if p.is_file())+ assert files == ["topic-2/figure.png", "topic/screenshot.png"]+++def test_sync_id_rewrite_of_the_file_named_after_the_old_id_takes_all_its_rows(tmp_vault):+ """`topic.md` is the file fetch recorded `topic`'s rows for, so when its+ own `id:` changes, every row under `topic` is its own, even while a new+ file elsewhere named `topic.md` claims the id: the URL and the asset row+ in research/assets/topic/ move with the renamed note."""+ notes = tmp_vault.notes_dir+ shot = tmp_vault.root / "research" / "assets" / "topic" / "screenshot.png"+ source_t = "source: https://ex.org/t\n"+ _note_with_id(notes / "topic.md", "topic", "Topic", "Fetched.", source_t)+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ shot.parent.mkdir(parents=True)+ shot.write_bytes(b"png")+ tmp_vault.db.execute(+ "INSERT INTO assets (note_id, type, filename, created_at) VALUES ('topic', 'screenshot', ?, 't')",+ (str(shot),),+ )+ tmp_vault.db.execute("INSERT INTO sources (url, note_id) VALUES ('https://ex.org/t', 'topic')")+ tmp_vault.db.commit()++ _note_with_id(notes / "topic.md", "topic-x", "Topic", "Fetched.", source_t)+ _note_with_id(tmp_vault.temp_dir / "topic.md", "topic", "Topic stub", "Stub.", source_t)+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))++ assert result.errors == [] and (result.added, result.updated) == (1, 1)+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"topic-x": "research/notes/topic.md", "topic": "research/temp/topic.md"}+ assert [tuple(r) for r in tmp_vault.db.execute("SELECT note_id FROM sources")] == [("topic-x",)]+ assets = [tuple(r) for r in tmp_vault.db.execute("SELECT note_id, filename FROM assets")]+ assert assets == [("topic-x", str(shot))]+++def test_sync_orphans_the_rows_a_failed_claimant_left_under_the_freed_id(tmp_vault):+ """`topic-2.md` renames itself away from `topic` while new `topic.md`+ claims that id, so the rename leaves the claimant's URL and asset rows+ under `topic`. The claimant then fails its own upsert (a duplicate+ alias) and no row holds `topic` when the pass commits. The rows kept+ for it must be orphaned as a delete of `topic` would orphan them, or the+ deferred foreign-key check fails the commit and the whole pass with it."""+ notes = tmp_vault.notes_dir+ assets = tmp_vault.root / "research" / "assets"+ _note_with_id(notes / "topic-2.md", "topic", "Topic dup", "Second.", SOURCE_B)+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ shot = assets / "topic" / "screenshot.png"+ shot.parent.mkdir(parents=True)+ shot.write_bytes(b"png")+ for url in ("https://ex.org/a", "https://ex.org/b"):+ tmp_vault.db.execute("INSERT INTO sources (url, note_id) VALUES (?, 'topic')", (url,))+ tmp_vault.db.execute(+ "INSERT INTO assets (note_id, type, filename, created_at) VALUES ('topic', 'screenshot', ?, 't')",+ (str(shot),),+ )+ tmp_vault.db.commit()++ _note_with_id(+ notes / "topic.md", "topic", "Topic", "First.",+ "source: https://ex.org/a\naliases: [dup, dup]\n",+ )+ _note_with_id(notes / "topic-2.md", "topic-2", "Topic dup", "Second.", SOURCE_B)+ raised = result = None+ try:+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))+ except sqlite3.IntegrityError as exc:+ raised = exc+ assert raised is None, f"the pass raised instead of orphaning the kept rows: {raised}"++ assert len(result.errors) == 1 and result.errors[0]["path"].endswith("topic.md")+ assert "aliases" in result.errors[0]["error"]+ assert (result.added, result.updated) == (0, 1)+ assert not tmp_vault.db.in_transaction+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"topic-2": "research/notes/topic-2.md"}+ owners = dict(tmp_vault.db.execute("SELECT url, note_id FROM sources").fetchall())+ assert owners == {"https://ex.org/a": None, "https://ex.org/b": "topic-2"}+ assert tmp_vault.db.execute("SELECT COUNT(*) FROM assets").fetchone()[0] == 0+ assert shot.exists()+ assert tmp_vault.db.execute("PRAGMA foreign_key_check").fetchall() == []+++def test_sync_id_edit_does_not_take_the_id_of_a_new_file_named_after_it(tmp_vault):+ """`foo.md` (id `foo`) is new in this pass, and fetch records its URL+ under `foo`. An id edit of `foo-2.md` to `foo` in the same pass must not+ win the id just because updates run before adds."""+ notes = tmp_vault.notes_dir+ _note_with_id(notes / "foo-2.md", "foo-2", "Foo dup", "Second.")+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []++ _note_with_id(notes / "foo-2.md", "foo", "Foo dup", "Second.")+ _note_with_id(notes / "foo.md", "foo", "Foo", "Fetched.")+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))++ assert len(result.errors) == 1+ assert result.errors[0]["path"] == "research/notes/foo-2.md"+ assert "claimed by new file 'research/notes/foo.md'" in result.errors[0]["error"]+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"foo": "research/notes/foo.md", "foo-2": "research/notes/foo-2.md"}+++def test_sync_id_edit_proceeds_when_the_file_named_after_the_id_declares_another(tmp_vault):+ """A new `foo.md` whose frontmatter says `id: bar` is the note `bar`,+ not a claim on `foo`: the rename of `foo-2.md` to `foo` goes through in+ the same pass, and `foo.md` indexes as `bar`."""+ notes = tmp_vault.notes_dir+ _note_with_id(notes / "foo-2.md", "foo-2", "Foo dup", "Second.")+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []++ _note_with_id(notes / "foo-2.md", "foo", "Foo dup", "Second.")+ _note_with_id(notes / "foo.md", "bar", "Bar", "Not foo.")+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))++ assert result.errors == []+ assert (result.added, result.updated) == (1, 1)+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"foo": "research/notes/foo-2.md", "bar": "research/notes/foo.md"}++[email protected]("temp_first", [False, True])+def test_sync_id_rewrite_keeps_the_urls_of_every_new_file_named_after_the_old_id(tmp_vault, temp_first):+ """Two new files named `topic.md`, one in research/notes/ and one in+ research/temp/, both claim `topic` while `topic-2.md` renames away from+ it. Whichever is listed first gets the id and the other collides, but+ the URL rows fetch recorded for both stay under `topic` either way."""+ notes = tmp_vault.notes_dir+ _note_with_id(notes / "topic-2.md", "topic", "Topic dup", "Second.", SOURCE_B)+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ _note_with_id(notes / "topic.md", "topic", "Topic", "First.", "source: https://ex.org/a\n")+ _note_with_id(tmp_vault.temp_dir / "topic.md", "topic", "Topic stub", "Stub.", "source: https://ex.org/c\n")+ for url in ("https://ex.org/a", "https://ex.org/b", "https://ex.org/c"):+ tmp_vault.db.execute("INSERT INTO sources (url, note_id) VALUES (?, 'topic')", (url,))+ tmp_vault.db.commit()++ _note_with_id(notes / "topic-2.md", "topic-2", "Topic dup", "Second.", SOURCE_B)+ plan = compute_sync_plan(tmp_vault)+ # The scan order is the filesystem's. Fix it, both ways round.+ plan.to_add.sort(key=lambda p: (p.parent.name == "temp") != temp_first)+ assert [p.name for p in plan.to_add] == ["topic.md", "topic.md"]+ first, second = (p.relative_to(tmp_vault.root).as_posix() for p in plan.to_add)+ result = execute_sync(tmp_vault, plan)++ assert [e["path"] for e in result.errors] == [second]+ assert "id collision" in result.errors[0]["error"]+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"topic": first, "topic-2": "research/notes/topic-2.md"}+ owners = dict(tmp_vault.db.execute("SELECT url, note_id FROM sources").fetchall())+ assert owners == {"https://ex.org/a": "topic", "https://ex.org/b": "topic-2", "https://ex.org/c": "topic"}+++ ("rel", "text"),+ [+ (+ "notes/topic.md",+ "---\nid: topic\ntitle: Topic\nsource: https://ex.org/a\nstatus: [bad\n---\n\nFirst.\n",+ ),+ ("notes/topic.md", "---\nid: topic\ntitle: Topic\nsource: https://ex.org/a\n\nFirst.\n"),+ ("temp/topic-2.md", "---\nid: topic-2\ntitle: Stub\nstatus: [bad\n---\n\nStub.\n"),+ ],+ ids=["old-id-does-not-parse", "old-id-caught-mid-write", "new-id-does-not-parse"],+)+def test_sync_id_rewrite_waits_while_a_new_file_named_after_either_id_does_not_parse(+ tmp_vault, rel, text+):+ """`topic-2.md` rewrites its id from `topic` to `topic-2` while a new+ file named after one of the two ids has frontmatter that does not parse,+ or no closing `---` yet (a file caught mid-write). Whether it claims+ that id is unknown, so the rename waits: the+ row and the URL rows under `topic` stay as they are, and the error+ names the file. A wrong guess would move the new file's URL to the+ renamed note for good, or take an id that is that file's."""+ notes = tmp_vault.notes_dir+ _note_with_id(notes / "topic-2.md", "topic", "Topic dup", "Second.", SOURCE_B)+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []+ for url in ("https://ex.org/a", "https://ex.org/b"):+ tmp_vault.db.execute("INSERT INTO sources (url, note_id) VALUES (?, 'topic')", (url,))+ tmp_vault.db.commit()++ (tmp_vault.root / "research" / rel).write_text(text, encoding="utf-8")+ _note_with_id(notes / "topic-2.md", "topic-2", "Topic dup", "Second.", SOURCE_B)+ result = execute_sync(tmp_vault, compute_sync_plan(tmp_vault))++ errors = [e["error"] for e in result.errors if e["path"] == "research/notes/topic-2.md"]+ assert len(errors) == 1+ assert f"new file 'research/{rel}'" in errors[0] and "run sync again" in errors[0]+ assert (result.added, result.updated) == (0, 0)+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"topic": "research/notes/topic-2.md"}+ owners = dict(tmp_vault.db.execute("SELECT url, note_id FROM sources").fetchall())+ assert owners == {"https://ex.org/a": "topic", "https://ex.org/b": "topic"}+++def test_sync_id_rewrite_takes_a_new_file_without_frontmatter_as_a_claimant(tmp_vault):+ """A new `topic.md` with no frontmatter at all has the id of its file+ name, like any note without an `id:` line, and is not a broken one: the+ rename of `topic-2.md` away from `topic` goes through in the same pass.+ Sync's own plan skips such a file, so the test admits it by hand, as a+ plan that indexes header-less files would."""+ notes = tmp_vault.notes_dir+ _note_with_id(notes / "topic-2.md", "topic", "Topic dup", "Second.", SOURCE_B)+ assert execute_sync(tmp_vault, compute_sync_plan(tmp_vault)).errors == []++ (notes / "topic.md").write_text("# Topic\n\nNo header.\n", encoding="utf-8")+ _note_with_id(notes / "topic-2.md", "topic-2", "Topic dup", "Second.", SOURCE_B)+ plan = compute_sync_plan(tmp_vault)+ assert plan.to_add == []+ plan.to_add.append(notes / "topic.md")+ result = execute_sync(tmp_vault, plan)++ assert result.errors == [] and (result.added, result.updated) == (1, 1)+ rows = {r["id"]: r["path"] for r in tmp_vault.db.execute("SELECT id, path FROM notes")}+ assert rows == {"topic": "research/notes/topic.md", "topic-2": "research/notes/topic-2.md"}