Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

test(secure-fs): make the directory snapshot race independent of clock ticks (#653)

Christopher Kevinb60c4ab98ebe
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b8a635b7..c84219e9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -347,6 +347,8 @@ All notable changes to SkillEvaluator are documented in this file.
model (`--model`, `--agent-model` or `harbor.agents.<agent>.model`) before the
run starts. It no longer sends the evaluator provider's default model name to
that endpoint.
+- Make the directory-snapshot mutation test independent of filesystem timestamp
+ granularity while preserving the late-entry race.
- Make the MCP-staging restored-mtime race regression independent of filesystem
clock granularity by modeling an advanced ctime.
- HTML reports preserve successful execution when Tier 3 quality fails, avoiding
diff --git a/docs/developer-guide.mdx b/docs/developer-guide.mdx
index f1dc6aac..282dc922 100644
--- a/docs/developer-guide.mdx
+++ b/docs/developer-guide.mdx
@@ -85,6 +85,9 @@ The `dev` extra includes pytest-xdist, so you can parallelize any of these runs:
pytest -n auto
```
+The directory-snapshot race regression explicitly advances directory mtime when
+inserting its late entry, rather than assuming creation crosses a clock tick.
+
### Check the HTML report in a browser
`tests/reporting/browser_navigation_checks.py` opens the HTML report in a headless browser and checks navigation, focus, themes, print view and layout at three screen widths. pytest doesn't collect it, and it needs Playwright:
diff --git a/tests/utils/test_secure_fs.py b/tests/utils/test_secure_fs.py
index 98cbb6c6..3b6e4c19 100644
--- a/tests/utils/test_secure_fs.py
+++ b/tests/utils/test_secure_fs.py
@@ -460,6 +460,7 @@ def test_directory_mutation_after_scandir_snapshot_fails_closed(
root = tmp_path / "root"
root.mkdir()
(root / "safe.md").write_text("safe")
+ root_metadata = root.stat()
real_scandir = os.scandir
mutated = False
@@ -484,6 +485,7 @@ def test_directory_mutation_after_scandir_snapshot_fails_closed(
except StopIteration:
if not mutated:
(root / "late.md").write_text("late")
+ os.utime(root, ns=(root_metadata.st_atime_ns, root_metadata.st_mtime_ns + 2_000_000_000))
mutated = True
raise