Skip to content

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

fix(assets): recover records when a drive comes back with hashing off (#16646)

Simon Pinfolde6beab52ef70
diff --git a/app/assets/docs/README.md b/app/assets/docs/README.md
index 4cd131f8b..613c74c4b 100644
--- a/app/assets/docs/README.md
+++ b/app/assets/docs/README.md
@@ -50,15 +50,19 @@ stateDiagram-v2
MissingHashed --> LiveHashed: hash match at path (hashing on)
MissingUnhashed --> LiveHashed: size and mtime match at path (hashing on)
+ MissingHashed --> LiveHashed: size and mtime match at path (hashing off)
+ MissingUnhashed --> LiveUnhashed: size and mtime match at path (hashing off)
```
-A reappeared file that matches no missing candidate, or more than one, does not recover any of them. The scanner creates a new content row for the file instead, and the missing candidates stay missing. The same old-missing-plus-new-content shape applies to a same-path edit or reuse: the old row is marked missing and a separate new row and record are created for the new bytes, never transformed in place.
+A reappeared file that matches no missing candidate does not recover any of them; with hashing on, neither does one that matches more than one (with hashing off, the newest match recovers, as described below). The scanner creates a new content row for the file instead, and the missing candidates stay missing. The same old-missing-plus-new-content shape applies to a same-path edit or reuse that change detection classifies as new content (see File edited in place): the old row is marked missing and a separate new row and record are created for the new bytes, never transformed in place.
-Only a hash can recover a missing content row, with one narrow exception for rows that were never hashed. The scanner hashes the file now present at the missing path and recovers the row only when exactly one missing candidate for that path has the same hash. If no candidate matches, the scanner creates new content. If multiple candidates match, none recover. Recovery also never fires for a path a live content row already occupies: the file there belongs to that row, and ordinary change detection owns it.
+With hashing on, only a hash can recover a missing content row, with one narrow exception for rows that were never hashed. The scanner hashes the file now present at the missing path and recovers the row only when exactly one missing candidate for that path has the same hash. If no candidate matches, the scanner creates new content. If multiple candidates match, none recover. Recovery also never fires for a path a live content row already occupies: the file there belongs to that row, and ordinary change detection owns it.
A missing row can have a null hash, for example a file deleted before it was ever hashed while every server was down during an off-to-on hashing transition. Such a row can never satisfy the hash comparison, so it has its own narrower recovery condition: it recovers only when it is the single missing null-hash candidate at that path, and its recorded byte size and modification time exactly match the reappeared file's verified stat. The freshly computed hash is set on the row as part of recovery. The stat facts are the only recorded identity a never-hashed row has, so a same-path reappearance that does not preserve them creates new content instead, and the old row stays missing rather than risk recovering the wrong record.
-Hash-based recovery does not compare modification times; only the null-hash recovery described above requires the recorded byte size and modification time to match exactly. With hashing disabled, recovery is unavailable.
+Hash-based recovery does not compare modification times; only the null-hash recovery described above requires the recorded byte size and modification time to match exactly.
+
+With hashing off, byte size and modification time are the identity the scanner checks on every live row, so they also decide recovery. A file that reappears at a path no live row occupies recovers the missing row at that path whose recorded byte size and modification time exactly match the file's stat, keeping any stored hash. This is what brings a library back, with its record ids, names, tags, metadata and job links, after its drive was offline for a scan. Only a row that still has an asset record is a candidate: a row whose records were all deleted while it was missing would otherwise come back with nothing to show it. If more than one candidate matches, the newest one recovers: that is the row that was live most recently, and the older ones stay missing. A file whose size or modification time changed while it was away (for example, copied back without preserving timestamps) creates new content, as with the null-hash rule. A different file with the same size and modification time recovers the old row, the same blind spot hashing-off change detection has for a live row.
Missing rows and their asset records persist until explicitly deleted. Routine scans do not remove them.
@@ -87,7 +91,7 @@ From-hash lookup is disabled while hashing is off.
### File removed outside the API
-Mark the content missing and keep every asset record visible. The usual hash-based recovery applies if a file later appears at the same path.
+Mark the content missing and keep every asset record visible. The usual recovery (by hash, or by size and modification time with hashing off) applies if a file later appears at the same path.
### File edited in place
@@ -319,7 +323,7 @@ A file lock prevents more than one server process from opening the same database
### Ambiguous recovery
-If more than one missing content row matches a recreated file's hash, recover none of them.
+If more than one missing content row matches a recreated file's hash, recover none of them. With hashing off, the newest of several rows matching the file's size and modification time recovers (see Missing content).
### Database replacement while running
diff --git a/app/assets/event_log.py b/app/assets/event_log.py
index c6ef2796f..1cbeea6ac 100644
--- a/app/assets/event_log.py
+++ b/app/assets/event_log.py
@@ -89,6 +89,8 @@ ALLOWED_FIELDS: dict[str, Callable[[Any], bool]] = {
"hash_failed": _is_count,
"enrich_failed": _is_count,
"permission_denied": _is_count,
+ "missing_marked_count": _is_count,
+ "recovered_count": _is_count,
"count": _is_count,
"error_type": _is_safe_string,
"hashing_enabled": _is_flag,
diff --git a/app/assets/scanner.py b/app/assets/scanner.py
index 39b1fa6c8..152caae25 100644
--- a/app/assets/scanner.py
+++ b/app/assets/scanner.py
@@ -35,8 +35,10 @@ from app.assets.scanner_changes import (
detect_content_change,
drain_pending_verifications,
live_contents_under_prefixes,
+ missing_content_ids_by_path,
pending_recovery_count,
recover_missing_content,
+ recover_missing_content_by_stat,
)
from app.assets.scanner_admission import (
PARTIAL_DOWNLOAD_EXTENSIONS as PARTIAL_DOWNLOAD_EXTENSIONS,
@@ -81,6 +83,8 @@ class _ScanProgress(Protocol):
hash_failed: int
enrich_failed: int
permission_denied: int
+ missing_marked: int
+ recovered: int
def mark_emitted(self, key: str) -> bool: ...
@@ -200,7 +204,7 @@ def observe_references_on_filesystem(
for content_id, path, size_bytes, mtime_ns in contents:
try:
stat_result = os.stat(path, follow_symlinks=True)
- except FileNotFoundError:
+ except (FileNotFoundError, NotADirectoryError):
observations.append(_ReferenceObservation(content_id, size_bytes, mtime_ns, None))
except PermissionError as e:
_log_scan_error("reference_stat", e)
@@ -208,9 +212,10 @@ def observe_references_on_filesystem(
progress.permission_denied += 1
logging.debug("Permission denied accessing %s", path)
except OSError as e:
+ # An I/O error (a flaky network share, a stale handle) says nothing about
+ # whether the file still exists, so the row stays live, as _is_gone leaves it.
_log_scan_error("reference_stat", e)
logging.debug("OSError checking %s: %s", path, e)
- observations.append(_ReferenceObservation(content_id, size_bytes, mtime_ns, None))
else:
survivors.add(os.path.abspath(path))
if stat_result.st_mtime_ns != mtime_ns:
@@ -222,7 +227,9 @@ def observe_references_on_filesystem(
def apply_reference_observations(
session: Session, observations: list[_ReferenceObservation]
-) -> None:
+) -> int:
+ """Apply the observations; returns how many rows were marked missing."""
+ marked = 0
for observation in observations:
content = session.get(AssetContent, observation.content_id)
# Skip a row another writer changed since it was observed; the next scan sees it afresh.
@@ -235,6 +242,7 @@ def apply_reference_observations(
continue
if observation.stat_result is None:
mark_content_missing(session, content.id)
+ marked += 1
continue
detect_content_change(
session,
@@ -242,20 +250,23 @@ def apply_reference_observations(
observation.stat_result,
hashing_is_enabled=mode.hashing_enabled(),
)
+ return marked
def _sync_prefixes_in_write_txn(
prefixes: list[str], progress: _ScanProgress | None
-) -> set[str]:
+) -> tuple[set[str], int]:
+ """Returns the surviving paths and how many rows were marked missing."""
with create_session() as session:
observations, survivors = observe_references_on_filesystem(
session, prefixes, progress
)
+ marked = 0
if observations:
with create_write_session() as session:
- apply_reference_observations(session, observations)
+ marked = apply_reference_observations(session, observations)
session.commit()
- return survivors
+ return survivors, marked
def sync_root_safely(
@@ -266,7 +277,9 @@ def sync_root_safely(
Returns survivors (existing paths) or empty set on failure.
"""
try:
- return _sync_prefixes_in_write_txn(get_scan_prefixes_for_root(root), progress)
+ survivors, marked = _sync_prefixes_in_write_txn(
+ get_scan_prefixes_for_root(root), progress
+ )
except Exception as exc:
logging.exception("fast DB scan failed for %s: %s", root, exc)
emit(
@@ -275,6 +288,9 @@ def sync_root_safely(
error_type=error_type(exc),
)
return set()
+ if progress is not None:
+ progress.missing_marked += marked
+ return survivors
def sync_temp_references_safely(
@@ -457,7 +473,9 @@ def _listing_verdict(
def mark_unlisted_references_missing_safely(
- root: RootType, observations: list[_ReferenceObservation]
+ root: RootType,
+ observations: list[_ReferenceObservation],
+ progress: _ScanProgress | None = None,
) -> None:
"""Retire rows whose file the listing lacks, through the same guarded write
sync_root applies to a row whose file has vanished."""
@@ -465,11 +483,14 @@ def mark_unlisted_references_missing_safely(
return
try:
with create_write_session() as session:
- apply_reference_observations(session, observations)
+ marked = apply_reference_observations(session, observations)
session.commit()
except Exception as exc:
logging.exception("fast DB scan failed for %s: %s", root, exc)
emit("scanner.fast_scan_failed", root=root, error_type=error_type(exc))
+ return
+ if progress is not None:
+ progress.missing_marked += marked
def list_output_for_rescan() -> ListingWalk:
@@ -556,6 +577,13 @@ def build_asset_specs(
return specs, tag_pool, skipped
+@dataclass
+class SeedCounts:
+ """What one seed_asset_specs call did besides creating records."""
+
+ recovered: int = 0
+
+
class _SpecObservation(NamedTuple):
"""A spec's file as seen before the write transaction opens.
@@ -596,9 +624,16 @@ def seed_asset_specs(
session: Session,
specs: list[SeedAssetSpec],
observed: dict[str, _SpecObservation | None] | None = None,
+ counts: SeedCounts | None = None,
+ missing_ids_by_path: dict[str, list[str]] | None = None,
) -> tuple[int, Exception | None]:
+ """``missing_ids_by_path`` is only used with hashing off. insert_asset_specs reads it
+ before its write transaction opens; when omitted, it is read here."""
if observed is None:
observed = observe_asset_specs(specs)
+ hashing_is_enabled = mode.hashing_enabled()
+ if not hashing_is_enabled and missing_ids_by_path is None:
+ missing_ids_by_path = missing_content_ids_by_path(session, _observed_paths(observed))
created = 0
first_error: Exception | None = None
# Counted, not gated through _ScanProgress.mark_emitted like its neighbours, because this
@@ -619,12 +654,16 @@ def seed_asset_specs(
)
invalid_mtimes += 1
continue
- recovery = recover_missing_content(
- session,
- path,
- observation.snapshot,
- hashing_is_enabled=mode.hashing_enabled(),
- )
+ if hashing_is_enabled:
+ recovery = recover_missing_content(
+ session, path, observation.snapshot, hashing_is_enabled=True
+ )
+ else:
+ recovery = recover_missing_content_by_stat(
+ session, path, stat_result, (missing_ids_by_path or {}).get(path, [])
+ )
+ if recovery == "recovered" and counts is not None:
+ counts.recovered += 1
if recovery != "no_match":
continue
content, _inserted = create_content_reporting_insert(
@@ -669,14 +708,27 @@ def seed_asset_specs(
return created, first_error
+def _observed_paths(observed: dict[str, _SpecObservation | None]) -> list[str]:
+ return [path for path, observation in observed.items() if observation is not None]
+
+
def insert_asset_specs(
- specs: list[SeedAssetSpec], _tag_pool: set[str]
+ specs: list[SeedAssetSpec],
+ _tag_pool: set[str],
+ progress: _ScanProgress | None = None,
) -> tuple[int, Exception | None]:
if not specs:
return 0, None
observed = observe_asset_specs(specs)
+ missing_ids_by_path = None
+ if not mode.hashing_enabled():
+ with create_session() as sess:
+ missing_ids_by_path = missing_content_ids_by_path(sess, _observed_paths(observed))
+ counts = SeedCounts()
with create_write_session() as sess:
- created, first_error = seed_asset_specs(sess, specs, observed)
+ created, first_error = seed_asset_specs(
+ sess, specs, observed, counts, missing_ids_by_path
+ )
try:
sess.commit()
except Exception:
@@ -688,6 +740,8 @@ def insert_asset_specs(
except Exception:
logging.exception("Failed to roll back asset batch after commit failure")
return 0, first_error
+ if progress is not None:
+ progress.recovered += counts.recovered
return created, first_error
diff --git a/app/assets/scanner_changes.py b/app/assets/scanner_changes.py
index df257b82c..bed169d4a 100644
--- a/app/assets/scanner_changes.py
+++ b/app/assets/scanner_changes.py
@@ -1,8 +1,9 @@
"""Reconciles catalogued content against what is actually on disk: retiring rows
whose file is gone, splitting a row whose bytes changed, and recovering one
-whose file came back. Recovery fires only when the returning file's hash
-identifies exactly one missing row and no live row already occupies that path,
-so a restored file can never leave two live rows describing one location.
+whose file came back. Recovery never fires for a path a live row already
+occupies, so a restored file can never leave two live rows describing one
+location. With hashing on, the returning file's hash must identify exactly one
+missing row; with hashing off, its size and modification time must match.
"""
from __future__ import annotations
@@ -14,7 +15,7 @@ from typing import Literal
import sqlalchemy as sa
from sqlalchemy.orm import Session
-from app.assets.database.models import AssetContent
+from app.assets.database.models import Asset, AssetContent
from app.assets.database.queries.records import (
create_content,
create_record,
@@ -22,6 +23,7 @@ from app.assets.database.queries.records import (
unset_content_missing,
)
from app.assets.helpers import path_prefix_matcher, sql_path_under_prefix, to_stored_hash
+from app.assets.services.file_utils import get_mtime_ns
from app.assets.services.path_utils import compute_loader_path, get_name_and_tags_from_asset_path
from app.assets.services.snapshot_hash import snapshot_hash
@@ -107,6 +109,60 @@ def recover_missing_content(
return "recovered"
+def missing_content_ids_by_path(session: Session, paths: list[str]) -> dict[str, list[str]]:
+ """The ids of the missing rows at ``paths``. No index serves a missing row's path, so
+ callers run this before their write transaction opens."""
+ by_path: dict[str, list[str]] = {}
+ for start in range(0, len(paths), 500):
+ chunk = paths[start : start + 500]
+ for content_id, path in session.execute(
+ sa.select(AssetContent.id, AssetContent.path).where(
+ AssetContent.is_missing.is_(True), AssetContent.path.in_(chunk)
+ )
+ ):
+ by_path.setdefault(path, []).append(content_id)
+ return by_path
+
+
+def recover_missing_content_by_stat(
+ session: Session,
+ path: str,
+ stat_result: os.stat_result,
+ candidate_ids: list[str],
+) -> Literal["recovered", "no_match"]:
+ """Hashing-off recovery: size and modification time are the identity a hashing-off
+ scan checks on a live row, so a returning file that matches them restores its row."""
+ mtime_ns = get_mtime_ns(stat_result)
+ candidates = [session.get(AssetContent, content_id) for content_id in candidate_ids]
+ matches = [
+ candidate
+ for candidate in candidates
+ if candidate is not None
+ and candidate.is_missing
+ and candidate.path == path
+ and (candidate.size_bytes, candidate.mtime_ns) == (stat_result.st_size, mtime_ns)
+ # A row whose records were all deleted while it was missing would come back
+ # live with nothing to show it, and hold the path so no scan ever lists it.
+ and session.scalar(sa.select(Asset.id).where(Asset.content_id == candidate.id).limit(1))
+ is not None
+ ]
+ if not matches:
+ return "no_match"
+ # "= 0", not "IS 0", so the partial live-path index serves it instead of a table scan.
+ occupied = session.scalar(
+ sa.select(AssetContent.id)
+ .where(AssetContent.path == path, AssetContent.is_missing == sa.false())
+ .limit(1)
+ )
+ if occupied is not None:
+ return "no_match"
+ # Several match only when earlier offline cycles left copies of one file behind;
+ # the newest is the one that was live last.
+ recovered = max(matches, key=lambda candidate: (candidate.created_at, candidate.id))
+ unset_content_missing(session, recovered.id)
+ return "recovered"
+
+
def is_path_under_prefixes(path: str, prefixes: list[str]) -> bool:
return path_prefix_matcher(prefixes)(path)
diff --git a/app/assets/seeder.py b/app/assets/seeder.py
index a538f4ba4..5b7f68235 100644
--- a/app/assets/seeder.py
+++ b/app/assets/seeder.py
@@ -115,6 +115,10 @@ class _ScanState:
hash_failed: int = 0
enrich_failed: int = 0
permission_denied: int = 0
+ # Rows the fast scan marked missing because their file was not found. Pruning reports
+ # its own count, and temp retirement is routine, so neither is included.
+ missing_marked: int = 0
+ recovered: int = 0
cancel_stage: str | None = None
_emitted_keys: set[str] = field(default_factory=set)
@@ -727,6 +731,8 @@ class _AssetSeeder:
hash_failed=scan_state.hash_failed,
enrich_failed=scan_state.enrich_failed,
permission_denied=scan_state.permission_denied,
+ missing_marked_count=scan_state.missing_marked,
+ recovered_count=scan_state.recovered,
root=root,
)
@@ -791,6 +797,17 @@ class _AssetSeeder:
pending["phase"].value,
)
+ @staticmethod
+ def _emit_marked_missing(root: RootType, marked: int) -> None:
+ """Report the rows a scan marked missing because their file was not found."""
+ if marked > 0:
+ emit(
+ "seeder.marked_missing",
+ count=marked,
+ stage=_ScanStage.FAST_SCAN.value,
+ root=root,
+ )
+
def _run_fast_phase(self, roots: tuple[RootType, ...]) -> tuple[int, int, int]:
"""Run phase 1: fast scan to create stub records.
@@ -814,7 +831,9 @@ class _AssetSeeder:
live_references = live_references_safely(r)
existing_paths.update(live_references)
else:
+ marked_before = scan_state.missing_marked
existing_paths.update(sync_root_safely(r, scan_state))
+ self._emit_marked_missing(r, scan_state.missing_marked - marked_before)
logging.debug(
"Fast scan: sync_root phase took %.3fs (%d existing paths)",
time.perf_counter() - t_sync,
@@ -834,7 +853,9 @@ class _AssetSeeder:
)
if walk is not None:
vanished, unlisted = unlisted_references(live_references, walk.listings)
- mark_unlisted_references_missing_safely("output", vanished)
+ marked_before = scan_state.missing_marked
+ mark_unlisted_references_missing_safely("output", vanished, scan_state)
+ self._emit_marked_missing("output", scan_state.missing_marked - marked_before)
logging.debug(
"Fast scan: output listing: %d dirs listed, %d rows retired, "
"%d rows skipped (not listed, still on disk)",
@@ -887,7 +908,7 @@ class _AssetSeeder:
batch_tags = {t for spec in batch for t in spec["tags"]}
created = 0
try:
- created, batch_error = insert_asset_specs(batch, batch_tags)
+ created, batch_error = insert_asset_specs(batch, batch_tags, scan_state)
total_created += created
if batch_error is not None:
raise batch_error
diff --git a/tests-unit/assets_test/services/test_admission_gate.py b/tests-unit/assets_test/services/test_admission_gate.py
index 1673295eb..03804af4e 100644
--- a/tests-unit/assets_test/services/test_admission_gate.py
+++ b/tests-unit/assets_test/services/test_admission_gate.py
@@ -120,6 +120,7 @@ def test_stat_error_drops_entry_and_allows_other_watch_entries_to_commit(
monkeypatch.setattr("folder_paths.get_input_directory", lambda: str(temp_dir))
monkeypatch.setattr(scanner_admission, "os", SimpleNamespace(stat=_stat))
monkeypatch.setattr("app.database.db.WriteSession", sessionmaker(bind=db_engine))
+ monkeypatch.setattr("app.database.db.Session", sessionmaker(bind=db_engine))
with caplog.at_level(logging.INFO):
tick_watch_list()
@@ -207,6 +208,7 @@ def test_spec_construction_failure_drops_the_entry_without_wedging_the_watch_lis
scanner_admission, "get_name_and_tags_from_asset_path", _name_and_tags
)
monkeypatch.setattr("app.database.db.WriteSession", sessionmaker(bind=db_engine))
+ monkeypatch.setattr("app.database.db.Session", sessionmaker(bind=db_engine))
with caplog.at_level(logging.INFO):
tick_watch_list()
diff --git a/tests-unit/assets_test/services/test_offline_drive_recovery.py b/tests-unit/assets_test/services/test_offline_drive_recovery.py
new file mode 100644
index 000000000..10b9f6db1
--- /dev/null
+++ b/tests-unit/assets_test/services/test_offline_drive_recovery.py
@@ -0,0 +1,454 @@
+"""A library on a drive that is offline for a scan comes back as the same records
+once the drive returns. Run through the seeder's real fast phase on an in-memory
+catalog, with hashing off unless a test says otherwise."""
+
+import errno
+import logging
+import os
+import re
+import shutil
+from contextlib import contextmanager
+from datetime import timedelta
+from pathlib import Path
+from unittest.mock import patch
+
+import folder_paths
+import pytest
+import sqlalchemy as sa
+from sqlalchemy.orm import Session as SASession, sessionmaker
+
+from app.assets import mode, scanner, seeder as seeder_module
+from app.assets.database.models import Asset, AssetContent, AssetTag
+from app.assets.database.queries.records import (
+ create_content,
+ create_record,
+ ensure_tag,
+ ensure_tag_link,
+ mark_content_missing,
+)
+from app.assets.scanner import SeedAssetSpec, insert_asset_specs, seed_asset_specs
+from app.assets.scanner_changes import recover_missing_content_by_stat
+from app.assets.scanner_admission import _WATCH_LIST
+
+ROOTS = ("input", "output")
+EVENT_LINE = re.compile(r"^\[assets-event\] (?P<event>\S+)(?P<fields>(?: \S+)*)$")
+
+
+def drive(temp_dir: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
+ """A removable drive holding both the input and the output directory."""
+ drive = temp_dir / "drive"
+ for name in ROOTS:
+ (drive / name).mkdir(parents=True)
+ monkeypatch.setattr("folder_paths.get_output_directory", lambda: str(drive / "output"))
+ monkeypatch.setattr("folder_paths.get_input_directory", lambda: str(drive / "input"))
+ monkeypatch.setattr(scanner, "get_comfy_models_folders", lambda: [])
+ return drive
+
+
[email protected](autouse=True)
+def isolated_state(db_engine):
+ @contextmanager
+ def _create_session():
+ with SASession(db_engine) as sess:
+ yield sess
+
+ _WATCH_LIST.clear()
+ with patch("app.assets.scanner.create_session", _create_session), \
+ patch("app.assets.seeder.create_session", _create_session), \
+ patch("app.database.db.WriteSession", sessionmaker(bind=db_engine)):
+ yield
+ _WATCH_LIST.clear()
+
+
+def _scan(roots=ROOTS) -> seeder_module._ScanState:
+ seeder = seeder_module._AssetSeeder()
+ seeder._scan_state = seeder_module._ScanState()
+ seeder._phase = seeder_module.ScanPhase.FAST
+ seeder._run_gate.set()
+ seeder._cancel_event.clear()
+ seeder._run_fast_phase(roots)
+ return seeder._scan_state
+
+
+def _events(caplog: pytest.LogCaptureFixture, name: str) -> list[dict[str, str]]:
+ found = []
+ for record in caplog.records:
+ match = EVENT_LINE.match(record.getMessage())
+ if match is not None and match.group("event") == name:
+ found.append(dict(pair.split("=", 1) for pair in match.group("fields").split()))
+ return found
+
+
+def _populate(drive: Path) -> list[Path]:
+ files = []
+ for name in ROOTS:
+ for i in range(5):
+ path = drive / name / f"{name}_{i}.png"
+ path.write_bytes(f"{name}-{i}".encode() * (i + 1))
+ files.append(path)
+ return files
+
+
+def _customise(session) -> dict[str, tuple[str, dict, str, set[str]]]:
+ """Give every record the edits a user makes; returns them keyed by record id."""
+ session.expire_all()
+ edits = {}
+ for i, record in enumerate(session.scalars(sa.select(Asset).order_by(Asset.name))):
+ record.name = f"renamed {i}"
+ record.user_metadata = {"note": i}
+ record.job_id = f"job-{i}"
+ ensure_tag(session, "favourite")
+ ensure_tag_link(session, asset_id=record.id, tag_name="favourite", origin="manual")
+ session.commit()
+ for record in session.scalars(sa.select(Asset)):
+ edits[record.id] = (record.name, record.user_metadata, record.job_id, _tags(session, record.id))
+ return edits
+
+
+def _tags(session, record_id: str) -> set[str]:
+ return set(session.scalars(sa.select(AssetTag.tag_name).where(AssetTag.asset_id == record_id)))
+
+
+def _records(session) -> dict[str, tuple[str, dict, str, set[str]]]:
+ session.expire_all()
+ return {
+ record.id: (record.name, record.user_metadata, record.job_id, _tags(session, record.id))
+ for record in session.scalars(sa.select(Asset))
+ }
+
+
+def _missing_count(session) -> int:
+ session.expire_all()
+ return session.scalar(
+ sa.select(sa.func.count()).select_from(AssetContent).where(AssetContent.is_missing.is_(True))
+ )
+
+
+def _take_offline(drive: Path, variant: str) -> Path:
+ parked = drive.with_name("drive.offline")
+ drive.rename(parked)
+ if variant == "empty":
+ drive.mkdir() # an unmounted mount point
+ return parked
+
+
+def _bring_back(drive: Path, parked: Path) -> None:
+ shutil.rmtree(drive, ignore_errors=True)
+ parked.rename(drive)
+
+
[email protected]("variant", ["absent", "empty"])
+def test_offline_drive_round_trip_keeps_every_record(drive, session, caplog, variant):
+ files = _populate(drive)
+ _scan()
+ edits = _customise(session)
+ assert len(edits) == len(files)
+
+ parked = _take_offline(drive, variant)
+ with caplog.at_level(logging.INFO):
+ offline = _scan()
+ assert offline.missing_marked == len(files)
+ assert _missing_count(session) == len(files)
+ assert sorted(e["root"] for e in _events(caplog, "seeder.marked_missing")) == ["input", "output"]
+ assert {e["stage"] for e in _events(caplog, "seeder.marked_missing")} == {"fast_scan"}
+
+ _bring_back(drive, parked)
+ back = _scan()
+
+ assert back.recovered == len(files)
+ assert _missing_count(session) == 0
+ assert _records(session) == edits
+ assert session.scalar(sa.select(sa.func.count()).select_from(AssetContent)) == len(files)
+
+
+def test_output_listing_rescan_recovers_a_returning_output_drive(drive, session, caplog):
+ files = [path for path in _populate(drive) if path.parent.name == "output"]
+ _scan(("output",))
+ edits = _customise(session)
+
+ parked = _take_offline(drive, "absent")
+ with caplog.at_level(logging.INFO):
+ offline = _scan(("output",))
+ assert offline.missing_marked == len(files)
+ assert _events(caplog, "seeder.marked_missing") == [
+ {"count": str(len(files)), "root": "output", "stage": "fast_scan"}
+ ]
+
+ _bring_back(drive, parked)
+ back = _scan(("output",))
+
+ assert back.recovered == len(files)
+ assert _records(session) == edits
+ assert _missing_count(session) == 0
+
+
+def test_an_io_error_leaves_rows_live(drive, session, monkeypatch):
+ files = _populate(drive)
+ _scan()
+ real_stat = os.stat
+
+ def flaky_stat(path, *args, **kwargs):
+ if str(path).startswith(str(drive)):
+ raise OSError(errno.EIO, "Input/output error")
+ return real_stat(path, *args, **kwargs)
+
+ monkeypatch.setattr(os, "stat", flaky_stat)
+ state = _scan()
+ monkeypatch.setattr(os, "stat", real_stat)
+
+ assert state.missing_marked == 0
+ assert _missing_count(session) == 0
+ assert session.scalar(sa.select(sa.func.count()).select_from(AssetContent)) == len(files)
+
+
+def test_a_genuinely_deleted_file_is_still_marked_missing(drive, session, caplog):
+ files = _populate(drive)
+ _scan()
+ for path in files[:3]:
+ path.unlink()
+
+ with caplog.at_level(logging.INFO):
+ state = _scan()
+
+ assert state.missing_marked == 3
+ assert state.recovered == 0
+ assert _missing_count(session) == 3
+ assert _events(caplog, "seeder.marked_missing") == [
+ {"count": "3", "root": "input", "stage": "fast_scan"}
+ ]
+
+
+def test_a_returning_file_with_a_changed_mtime_gets_a_new_record(drive, session):
+ files = _populate(drive)
+ _scan()
+ target = files[0]
+ parked = _take_offline(drive, "absent")
+ _scan()
+ _bring_back(drive, parked)
+ stat_result = target.stat()
+ os.utime(target, ns=(stat_result.st_atime_ns, stat_result.st_mtime_ns + 1_000_000_000))
+
+ state = _scan()
+
+ assert state.recovered == len(files) - 1
+ session.expire_all()
+ rows = list(session.scalars(sa.select(AssetContent).where(AssetContent.path == str(target))))
+ assert sorted(row.is_missing for row in rows) == [False, True]
+
+
+def test_hashing_on_recovers_through_the_hash_path(drive, session):
+ class _HashingOn:
+ enable_asset_hashing = True
+
+ mode.init(_HashingOn())
+ files = _populate(drive)
+ _scan()
+ edits = _customise(session)
+ parked = _take_offline(drive, "absent")
+ _scan()
+ _bring_back(drive, parked)
+
+ with patch(
+ "app.assets.scanner.recover_missing_content_by_stat"
+ ) as by_stat, patch(
+ "app.assets.scanner.missing_content_ids_by_path"
+ ) as prefetch:
+ back = _scan()
+
+ by_stat.assert_not_called()
+ prefetch.assert_not_called()
+ assert back.recovered == len(files)
+ assert _records(session) == edits
+
+
+def _spec(path: Path) -> SeedAssetSpec:
+ stat_result = path.stat()
+ return {
+ "abs_path": str(path),
+ "size_bytes": stat_result.st_size,
+ "mtime_ns": stat_result.st_mtime_ns,
+ "info_name": path.name,
+ "tags": ["input"],
+ "fname": path.name,
+ "metadata": None,
+ "mime_type": None,
+ "job_id": None,
+ }
+
+
+def _missing_row(session, path: Path) -> AssetContent:
+ stat_result = path.stat()
+ content = create_content(
+ session, path=str(path), size_bytes=stat_result.st_size, mtime_ns=stat_result.st_mtime_ns
+ )
+ create_record(session, content_id=content.id, name=path.name)
+ mark_content_missing(session, content.id)
+ return content
+
+
+def test_the_newest_of_several_matching_missing_rows_recovers(session, temp_dir):
+ path = temp_dir / "left-behind.png"
+ path.write_bytes(b"one file, catalogued twice by earlier offline cycles")
+ older = _missing_row(session, path)
+ newer = _missing_row(session, path)
+ older.created_at = newer.created_at - timedelta(days=1)
+ session.commit()
+
+ created, error = seed_asset_specs(session, [_spec(path)])
+ session.commit()
+
+ assert (created, error) == (0, None)
+ assert session.get(AssetContent, newer.id).is_missing is False
+ assert session.get(AssetContent, older.id).is_missing is True
+ assert session.get(AssetTag, {"asset_id": session.scalar(
+ sa.select(Asset.id).where(Asset.content_id == newer.id)
+ ), "tag_name": "missing"}) is None
+
+
+def test_stat_recovery_skips_a_path_a_live_row_already_occupies(session, temp_dir):
+ path = temp_dir / "contested.png"
+ path.write_bytes(b"bytes")
+ missing = _missing_row(session, path)
+ stat_result = path.stat()
+ live = create_content(
+ session, path=str(path), size_bytes=stat_result.st_size, mtime_ns=stat_result.st_mtime_ns
+ )
+ create_record(session, content_id=live.id, name=path.name)
+ session.commit()
+
+ created, error = seed_asset_specs(session, [_spec(path)])
+ session.commit()
+
+ assert (created, error) == (0, None)
+ assert session.get(AssetContent, missing.id).is_missing is True
+ assert session.get(AssetContent, live.id).is_missing is False
+
+
+def test_a_recovery_rolled_back_by_a_failed_commit_is_not_counted(session, temp_dir, db_engine):
+ path = temp_dir / "rolled-back.png"
+ path.write_bytes(b"bytes")
+ _missing_row(session, path)
+ session.commit()
+ progress = seeder_module._ScanState()
+
+ @contextmanager
+ def failing_write_session():
+ with SASession(db_engine) as sess:
+ def fail():
+ raise RuntimeError("disk full")
+
+ sess.commit = fail
+ yield sess
+
+ with patch("app.assets.scanner.create_write_session", failing_write_session):
+ with pytest.raises(RuntimeError):
+ insert_asset_specs([_spec(path)], set(), progress)
+
+ assert progress.recovered == 0
+
+
[email protected]("while_away", ["unregistered", "base_path_unresolved"])
+def test_a_model_folder_pruned_while_unregistered_recovers_when_it_returns(
+ temp_dir, session, monkeypatch, while_away
+):
+ """A folder a custom node registers late, or an extra_model_paths entry whose base
+ path did not resolve for one launch, is pruned at startup and seen again later."""
+ models = temp_dir / "lazy_models" / "checkpoints"
+ models.mkdir(parents=True)
+ for i in range(3):
+ (models / f"ckpt_{i}.safetensors").write_bytes(b"weights" * (i + 1))
+ for name in ("input", "output", "temp"):
+ (temp_dir / name).mkdir()
+ monkeypatch.setattr("folder_paths.get_input_directory", lambda: str(temp_dir / "input"))
+ monkeypatch.setattr("folder_paths.get_output_directory", lambda: str(temp_dir / "output"))
+ monkeypatch.setattr("folder_paths.get_temp_directory", lambda: str(temp_dir / "temp"))
+
+ def register(path: Path | None) -> None:
+ paths = {} if path is None else {"checkpoints": ([str(path)], {".safetensors"})}
+ monkeypatch.setattr(folder_paths, "folder_names_and_paths", paths)
+ monkeypatch.setattr(folder_paths, "filename_list_cache", {})
+
+ register(models)
+ _scan(("models",))
+ edits = _customise(session)
+ assert len(edits) == 3
+
+ register(None if while_away == "unregistered" else temp_dir / "$UNSET_VAR" / "checkpoints")
+ assert scanner.mark_missing_outside_prefixes_safely(scanner.get_owned_prefixes()) == 3
+ _scan(("models",))
+ assert _missing_count(session) == 3
+
+ register(models)
+ back = _scan(("models",))
+
+ assert back.recovered == 3
+ assert _missing_count(session) == 0
+ assert _records(session) == edits
+
+
+def test_a_row_whose_records_were_deleted_while_missing_does_not_recover(session, temp_dir):
+ path = temp_dir / "deleted-while-offline.png"
+ path.write_bytes(b"bytes")
+ orphan = _missing_row(session, path)
+ session.execute(sa.delete(AssetTag))
+ session.execute(sa.delete(Asset).where(Asset.content_id == orphan.id))
+ session.commit()
+
+ created, error = seed_asset_specs(session, [_spec(path)])
+ session.commit()
+
+ assert (created, error) == (1, None)
+ assert session.get(AssetContent, orphan.id).is_missing is True
+ live = session.scalar(
+ sa.select(AssetContent).where(AssetContent.path == str(path), AssetContent.is_missing.is_(False))
+ )
+ assert session.scalar(sa.select(Asset.id).where(Asset.content_id == live.id)) is not None
+
+
+def test_a_newer_recordless_row_does_not_win_over_the_users_record(session, temp_dir):
+ path = temp_dir / "deduplicated.png"
+ path.write_bytes(b"bytes")
+ original = _missing_row(session, path)
+ duplicate = _missing_row(session, path)
+ original.created_at = duplicate.created_at - timedelta(days=1)
+ session.execute(sa.delete(AssetTag).where(AssetTag.asset_id.in_(
+ sa.select(Asset.id).where(Asset.content_id == duplicate.id)
+ )))
+ session.execute(sa.delete(Asset).where(Asset.content_id == duplicate.id))
+ session.commit()
+
+ created, error = seed_asset_specs(session, [_spec(path)])
+ session.commit()
+
+ assert (created, error) == (0, None)
+ assert session.get(AssetContent, original.id).is_missing is False
+ assert session.get(AssetContent, duplicate.id).is_missing is True
+
+
+def test_stat_recovery_runs_no_table_scan_in_the_write_transaction(session, temp_dir, db_engine):
+ """Recovery runs once per returning file inside the batch's write transaction, so a
+ table scan there makes the lock window grow with the catalog."""
+ path = temp_dir / "returning.png"
+ path.write_bytes(b"bytes")
+ _missing_row(session, path)
+ session.commit()
+ candidates = {str(path): [row.id for row in session.scalars(sa.select(AssetContent))]}
+ statements: list[tuple[str, object]] = []
+
+ def capture(conn, cursor, statement, parameters, context, executemany):
+ if statement.lstrip().upper().startswith(("SELECT", "UPDATE", "DELETE")):
+ statements.append((statement, parameters))
+
+ sa.event.listen(db_engine, "before_cursor_execute", capture)
+ try:
+ assert recover_missing_content_by_stat(session, str(path), path.stat(), candidates[str(path)]) == "recovered"
+ session.flush()
+ finally:
+ sa.event.remove(db_engine, "before_cursor_execute", capture)
+
+ assert statements
+ with db_engine.connect() as conn:
+ for statement, parameters in statements:
+ plan = conn.exec_driver_sql(f"EXPLAIN QUERY PLAN {statement}", parameters).all()
+ assert not any("SCAN asset_contents" in row[-1] for row in plan), (statement, plan)
diff --git a/tests-unit/assets_test/services/test_recovery_gate.py b/tests-unit/assets_test/services/test_recovery_gate.py
index 59b365570..ed2bb064f 100644
--- a/tests-unit/assets_test/services/test_recovery_gate.py
+++ b/tests-unit/assets_test/services/test_recovery_gate.py
@@ -107,10 +107,19 @@ def test_no_hash_match_creates_fresh_rows(session, temp_dir: Path):
assert len(session.scalars(select(Asset)).all()) == 2
-def test_off_mode_no_recovery(session, temp_dir: Path):
+def _stat_content(session, path: Path, hash_value: str | None) -> tuple[AssetContent, Asset]:
+ content, record = _missing_content(session, path, hash_value)
+ stat_result = path.stat()
+ content.size_bytes = stat_result.st_size
+ content.mtime_ns = stat_result.st_mtime_ns
+ session.commit()
+ return content, record
+
+
+def test_off_mode_recovers_by_exact_stat_without_hashing(session, temp_dir: Path):
path = temp_dir / "off.bin"
path.write_bytes(b"bytes")
- missing, _ = _missing_content(session, path, _stored_hash(path))
+ missing, record = _stat_content(session, path, _stored_hash(path))
with (
patch("app.assets.scanner.mode.hashing_enabled", return_value=False),
@@ -120,6 +129,26 @@ def test_off_mode_no_recovery(session, temp_dir: Path):
session.commit()
hash_mock.assert_not_called()
+ assert error is None
+ assert created == 0
+ recovered = session.get(AssetContent, missing.id)
+ assert recovered.is_missing is False
+ assert recovered.hash == _stored_hash(path)
+ assert session.get(AssetTag, {"asset_id": record.id, "tag_name": "missing"}) is None
+
+
[email protected]("field", ["size_bytes", "mtime_ns"])
+def test_off_mode_stat_mismatch_creates_fresh_rows(session, temp_dir: Path, field: str):
+ path = temp_dir / "off-changed.bin"
+ path.write_bytes(b"bytes")
+ missing, _ = _stat_content(session, path, None)
+ setattr(missing, field, getattr(missing, field) + 1)
+ session.commit()
+
+ with patch("app.assets.scanner.mode.hashing_enabled", return_value=False):
+ created, error = seed_asset_specs(session, [_spec(path)])
+ session.commit()
+
assert error is None
assert created == 1
assert session.get(AssetContent, missing.id).is_missing is True
diff --git a/tests-unit/assets_test/test_event_log.py b/tests-unit/assets_test/test_event_log.py
index dccdfadc0..410430327 100644
--- a/tests-unit/assets_test/test_event_log.py
+++ b/tests-unit/assets_test/test_event_log.py
@@ -34,6 +34,8 @@ VALID_VALUES: dict[str, list[object]] = {
"hash_failed": [2],
"enrich_failed": [0],
"permission_denied": [0],
+ "missing_marked_count": [0, 10],
+ "recovered_count": [10],
"count": [1],
"error_type": ["ValueError", "FileNotFoundError"],
"hashing_enabled": [True, False],
diff --git a/tests-unit/seeder_test/test_seeder.py b/tests-unit/seeder_test/test_seeder.py
index ea49b9e0b..8cda28845 100644
--- a/tests-unit/seeder_test/test_seeder.py
+++ b/tests-unit/seeder_test/test_seeder.py
@@ -215,8 +215,10 @@ def test_multi_root_scan_emits_one_started_and_completed_without_root(
"enrich_failed": 0,
"enriched": 4,
"hash_failed": 0,
+ "missing_marked_count": 0,
"permission_denied": 0,
"phase": "full",
+ "recovered_count": 0,
"skipped": 2,
}
@@ -230,6 +232,8 @@ def test_scan_completed_reports_per_scan_failure_counts(
hash_failed=2,
enrich_failed=3,
permission_denied=1,
+ missing_marked=10,
+ recovered=7,
)
clock = iter((10.0, 10.5))
monkeypatch.setattr(seeder_module.time, "perf_counter", lambda: next(clock))
@@ -244,6 +248,8 @@ def test_scan_completed_reports_per_scan_failure_counts(
assert completed[0]["hash_failed"] == 2
assert completed[0]["enrich_failed"] == 3
assert completed[0]["permission_denied"] == 1
+ assert completed[0]["missing_marked_count"] == 10
+ assert completed[0]["recovered_count"] == 7
def test_enrich_phase_does_not_count_returned_ids_as_failures(
@@ -604,7 +610,7 @@ def test_batch_insert_failure_emits_only_the_exception_type(
),
)
- def fail_insert(batch, batch_tags) -> int:
+ def fail_insert(batch, batch_tags, progress=None) -> int:
raise PermissionError("/private/models/asset.safetensors")
monkeypatch.setattr(seeder_module, "insert_asset_specs", fail_insert)
diff --git a/tests-unit/test_assets_event_log_static.py b/tests-unit/test_assets_event_log_static.py
index d92018ded..fd76692e2 100644
--- a/tests-unit/test_assets_event_log_static.py
+++ b/tests-unit/test_assets_event_log_static.py
@@ -51,6 +51,7 @@ EXPECTED_CALL_SITES: frozenset[CallSite] = frozenset(
CallSite("app/assets/seeder.py", "_run_scan", "seeder.marked_missing"),
CallSite("app/assets/seeder.py", "mark_missing_outside_prefixes", "seeder.marked_missing"),
CallSite("app/assets/seeder.py", "_run_fast_phase", "seeder.batch_insert_failed"),
+ CallSite("app/assets/seeder.py", "_emit_marked_missing", "seeder.marked_missing"),
# todo 11 - scanner failure paths
CallSite("app/assets/scanner.py", "sync_root_safely", "scanner.fast_scan_failed"),
CallSite("app/assets/scanner.py", "live_references_safely", "scanner.fast_scan_failed"),