Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

harden: contain benchmark output to the data dir and re-check backup purge entries (#5769)

LearningCircuit9bb18394026f
diff --git a/changelog.d/+benchmark-output-containment.security.md b/changelog.d/+benchmark-output-containment.security.md
new file mode 100644
index 0000000000..52203ffa58
--- /dev/null
+++ b/changelog.d/+benchmark-output-containment.security.md
@@ -0,0 +1 @@
+Benchmark files written by `run_benchmark`, and the reports of both `compare_configurations` helpers, now stay below the data directory. This guards CLI and Python API callers only: no code reads the `benchmark.output_dir` setting, and web benchmark runs, which keep their results in the database, never call these functions, so #7417's exposure was not reachable from the web UI. `run_benchmark`, the `run_*_benchmark` and `evaluate_*` helpers that call it, and the `compare_configurations` helpers in `local_deep_research.api.benchmark_functions` and `local_deep_research.benchmarks.benchmark_functions` resolve their output directory there: a relative path is taken relative to the data directory instead of the working directory, and a path outside it, or the data directory itself, is replaced, with a logged warning, by `benchmark_results` or `benchmark_comparisons` below it. If that default is not usable either, for example because it is a symlink to another disk, the call raises `BenchmarkOutputDirectoryError` before writing anything. A comparison resolves its root once, so the per-configuration results follow its report, unless a configuration name leads out of that root (for example an absolute name or `..` components); those results then land elsewhere below the data directory. `evaluate_benchmark_quality` keeps its temporary directory inside `benchmark_results`. The benchmark CLI defaults were already below the data directory. Other benchmark writers still use the directory their caller gives: the parameter optimizer (`OptunaOptimizer` and the `optimize_*` helpers), the benchmark evaluators (the SimpleQA evaluator's default path writes its files there, and both evaluators create their subdirectory there), `local_deep_research.benchmarks.comparison.compare_configurations`, and writers that take a file path (`grade_results`, `generate_report`, `Benchmark_results`). When such a caller passes `run_benchmark` a relative directory or one outside the data directory, as the BrowseComp evaluator, the SimpleQA evaluator with `use_direct_dataset=False` and several scripts under `examples/` do, those benchmark files land below the data directory while the caller's own files and the directory it prints stay where they were; the paths `run_benchmark` returns are the real ones. The backup purge after a password change now also re-checks each listed backup and skips any that is a symlink or resolves outside the user's backup directory. These checks are snapshots and do not protect against concurrent filesystem changes.
diff --git a/docs/cli-tools.md b/docs/cli-tools.md
index 297e01d35c..a65e6ad9e3 100644
--- a/docs/cli-tools.md
+++ b/docs/cli-tools.md
@@ -42,11 +42,13 @@ python -m local_deep_research.benchmarks.cli.benchmark_commands simpleqa [option
| `--search-provider` | (default) | LLM provider |
| `--eval-model` | (default) | Model for answer evaluation |
| `--eval-provider` | (default) | Provider for evaluation |
-| `--output-dir` | ~/.local-deep-research/benchmark_results | Results directory |
+| `--output-dir` | data directory/benchmark_results | Results directory, kept below the data directory (see **Output location**) |
| `--human-eval` | false | Use human evaluation |
| `--no-eval` | false | Skip evaluation phase |
| `--custom-dataset` | - | Path to custom dataset |
+**Output location:** benchmark result, evaluation and report files are written below the data directory (`LDR_DATA_DIR`, or the platform default). A relative `--output-dir` is taken relative to the data directory, not the current directory. A path outside the data directory, or the data directory itself, is replaced by `benchmark_results` there, with a warning in the log. If that default is not usable either, for example because it is a symlink to another disk, the command stops with an error before running. The report and results paths it prints are the real locations.
+
**Example:**
```bash
@@ -91,7 +93,7 @@ python -m local_deep_research.benchmarks.cli.benchmark_commands compare [options
|--------|---------|-------------|
| `--dataset` | simpleqa | Dataset to use (simpleqa, browsecomp) |
| `--examples` | 20 | Examples per configuration |
-| `--output-dir` | ~/.local-deep-research/benchmark_results/comparison | Results directory |
+| `--output-dir` | data directory/benchmark_results/comparison | Results directory, kept below the data directory (see **Output location** under `simpleqa`; the default replacement here is `benchmark_comparisons`) |
**Example:**
diff --git a/examples/benchmarks/README.md b/examples/benchmarks/README.md
index 61099e0c4a..8c4992dcef 100644
--- a/examples/benchmarks/README.md
+++ b/examples/benchmarks/README.md
@@ -17,7 +17,7 @@ Options:
- `--iterations`: Number of search iterations (default: 3)
- `--questions`: Questions per iteration (default: 3)
- `--search-tool`: Search tool to use (default: "searxng")
-- `--output-dir`: Directory to save results (default: "benchmark_results")
+- `--output-dir`: Directory to save results (default: "benchmark_results", relative to the LDR data directory; see [Where results are written](#where-results-are-written))
- `--no-eval`: Skip evaluation
- `--human-eval`: Use human evaluation
- `--eval-model`: Model to use for evaluation
@@ -36,7 +36,7 @@ Options:
- `--iterations`: Number of search iterations (default: 1)
- `--questions`: Questions per iteration (default: 1)
- `--search-tool`: Search tool to use (default: "searxng")
-- `--output-dir`: Directory to save results (default: "browsecomp_results")
+- `--output-dir`: Directory to save results (default: "browsecomp_results", relative to the current directory; this script writes its files itself)
See `browsecomp_benchmark_readme.md` for more information on how BrowseComp works.
@@ -52,9 +52,50 @@ python run_simpleqa.py
python run_browsecomp.py --iterations 3 --questions 3
```
+## Where results are written
+
+Scripts that hand an output directory to the library's benchmark runner
+(`evaluate_simpleqa`, `evaluate_browsecomp`, `run_simpleqa_benchmark`,
+`run_browsecomp_benchmark` or `compare_configurations`) get those benchmark
+files below the LDR data directory (`LDR_DATA_DIR`, or the platform
+default). A relative directory is placed relative to the data directory,
+not the current directory. A directory outside the data directory, such as
+the project-root `benchmark_results/<timestamp>` paths some scripts build,
+is replaced by `benchmark_results` there, with a warning in the log. If that
+default is not usable either, for example because it is a symlink to another
+disk, the run stops with an error before it starts.
+
+This applies to `run_simpleqa.py`, `run_gemini_benchmark.py` and
+`gemini/run_gemini_benchmark_fixed.py`, which call `evaluate_simpleqa` and
+`evaluate_browsecomp`, and to `../run_benchmark.py`,
+`../optimization/run_gemini_benchmark.py` and
+`../optimization/run_parallel_benchmark.py`. The report and results paths
+these functions return are the real locations. A script's own summary JSON
+and its "Results saved to" line still use the directory the script created,
+which may then hold only that summary.
+
+`claude_grading/benchmark.py`, `scripts/run_benchmark_with_claude_grading.py`
+and `scripts/run_focused_benchmark_fixed.py` call the benchmark evaluators
+directly, on their own and through `CompositeBenchmarkEvaluator`, so their
+output is split. The SimpleQA evaluator's default path writes its results,
+evaluation and report files itself, into a `simpleqa` subdirectory of the
+directory it is given, so they stay in the script's own directory tree with
+its summary JSON (for example
+`./benchmark_results/claude_grading_<timestamp>/simpleqa/simpleqa`). The
+BrowseComp evaluator creates its `browsecomp` subdirectory there too, but
+`run_browsecomp_benchmark` writes its benchmark files below the data
+directory, so that subdirectory stays empty. Parameter optimizer runs
+(`../optimization/`) split the same way and keep the optimizer's own study
+files in the directory they are given.
+
+`run_browsecomp.py`, `run_resumable_parallel_benchmark.py`,
+`browsecomp/run_browsecomp_fixed_v2.py` and `scripts/run_grader_only.py`
+call `quick_summary`, `grade_results` or `generate_report` themselves and
+write every file to the paths they are given.
+
## Evaluating Results
-Results are saved in the specified output directories and include:
+Results are saved in the output directories described above and include:
- Raw results (JSONL format)
- Evaluation results (JSONL format)
- Summary reports (Markdown format)
diff --git a/src/local_deep_research/api/benchmark_functions.py b/src/local_deep_research/api/benchmark_functions.py
index 573a3f04fc..ad9fb5509a 100644
--- a/src/local_deep_research/api/benchmark_functions.py
+++ b/src/local_deep_research/api/benchmark_functions.py
@@ -16,6 +16,7 @@ from ..benchmarks import (
run_simpleqa_benchmark,
run_xbench_deepsearch_benchmark,
)
+from ..benchmarks.runners import resolve_output_dir
def evaluate_simpleqa(
@@ -224,7 +225,11 @@ def compare_configurations(
dataset_type: Type of dataset to use
num_examples: Number of examples to evaluate
configurations: List of search configurations to compare
- output_dir: Directory to save results
+ output_dir: Directory to save results, resolved below the data
+ directory like ``run_benchmark``'s: a value outside it (or the
+ data directory itself) is replaced by ``benchmark_comparisons``
+ there, and ``BenchmarkOutputDirectoryError`` is raised if that
+ is not usable either.
Returns:
Dictionary with comparison results
@@ -252,7 +257,16 @@ def compare_configurations(
},
]
- # Create output directory
+ # Resolve once, so the per-configuration results and the comparison
+ # report share one root below the data directory, also when the
+ # requested root is replaced. The exception is a configuration name
+ # that leads out of that root (for example an absolute name or ".."
+ # components): run_benchmark then resolves that subdirectory on its
+ # own, so those results land elsewhere below the data directory (in
+ # benchmark_results when the path leaves the data directory).
+ output_dir = str(
+ resolve_output_dir(output_dir, fallback="benchmark_comparisons")
+ )
import os
os.makedirs(output_dir, exist_ok=True)
diff --git a/src/local_deep_research/benchmarks/benchmark_functions.py b/src/local_deep_research/benchmarks/benchmark_functions.py
index e879b1f72b..b2a48f895c 100644
--- a/src/local_deep_research/benchmarks/benchmark_functions.py
+++ b/src/local_deep_research/benchmarks/benchmark_functions.py
@@ -20,6 +20,7 @@ from ..benchmarks import (
run_simpleqa_benchmark,
run_xbench_deepsearch_benchmark,
)
+from .runners import resolve_output_dir
def evaluate_simpleqa(
@@ -327,7 +328,11 @@ def compare_configurations(
dataset_type: Type of dataset to use
num_examples: Number of examples to evaluate
configurations: List of search configurations to compare
- output_dir: Directory to save results
+ output_dir: Directory to save results, resolved below the data
+ directory like ``run_benchmark``'s: a value outside it (or the
+ data directory itself) is replaced by ``benchmark_comparisons``
+ there, and ``BenchmarkOutputDirectoryError`` is raised if that
+ is not usable either.
Returns:
Dictionary with comparison results
@@ -355,8 +360,16 @@ def compare_configurations(
},
]
- # Create output directory
-
+ # Resolve once, so the per-configuration results and the comparison
+ # report share one root below the data directory, also when the
+ # requested root is replaced. The exception is a configuration name
+ # that leads out of that root (for example an absolute name or ".."
+ # components): run_benchmark then resolves that subdirectory on its
+ # own, so those results land elsewhere below the data directory (in
+ # benchmark_results when the path leaves the data directory).
+ output_dir = str(
+ resolve_output_dir(output_dir, fallback="benchmark_comparisons")
+ )
# nosemgrep: semgrep.rules.path-traversal-risk, reason: CLI/Python benchmark API intentionally accepts an output directory; the web runner does not call this function
Path(output_dir).mkdir(parents=True, exist_ok=True)
diff --git a/src/local_deep_research/benchmarks/comparison/evaluator.py b/src/local_deep_research/benchmarks/comparison/evaluator.py
index b56f442119..fb82d3bf9f 100644
--- a/src/local_deep_research/benchmarks/comparison/evaluator.py
+++ b/src/local_deep_research/benchmarks/comparison/evaluator.py
@@ -75,7 +75,8 @@ def compare_configurations(
Args:
query: Research query to use for evaluation
configurations: List of parameter configurations to compare
- output_dir: Directory to save comparison results
+ output_dir: Directory to save comparison results, used as given
+ (not resolved below the data directory)
model_name: Name of the LLM model to use
provider: LLM provider
search_tool: Search engine to use
diff --git a/src/local_deep_research/benchmarks/evaluators/base.py b/src/local_deep_research/benchmarks/evaluators/base.py
index eabb20715b..338fb3ed1c 100644
--- a/src/local_deep_research/benchmarks/evaluators/base.py
+++ b/src/local_deep_research/benchmarks/evaluators/base.py
@@ -60,6 +60,15 @@ class BaseBenchmarkEvaluator(ABC):
"""
Create a benchmark-specific subdirectory for output.
+ The subdirectory is created where the caller says (a relative
+ path is relative to the working directory); it is not resolved
+ below the data directory. Files that ``run_benchmark`` writes for
+ it (the BrowseComp evaluator and the SimpleQA evaluator with
+ ``use_direct_dataset=False``) follow ``runners.resolve_output_dir``
+ instead, so unless ``output_dir`` is an absolute path below the
+ data directory they land elsewhere below the data directory and
+ this subdirectory stays empty.
+
Args:
output_dir: Parent directory for output
diff --git a/src/local_deep_research/benchmarks/evaluators/simpleqa.py b/src/local_deep_research/benchmarks/evaluators/simpleqa.py
index 1e196ff598..ccfecdde66 100644
--- a/src/local_deep_research/benchmarks/evaluators/simpleqa.py
+++ b/src/local_deep_research/benchmarks/evaluators/simpleqa.py
@@ -112,7 +112,10 @@ class SimpleQAEvaluator(BaseBenchmarkEvaluator):
Args:
system_config: Search and LLM configuration parameters
num_examples: Number of benchmark examples to run
- output_dir: Directory to save evaluation results
+ output_dir: Directory to save evaluation results. The files
+ are written there directly; unlike ``run_benchmark``, this
+ path does not resolve the directory below the data
+ directory.
Returns:
Dictionary with benchmark results
diff --git a/src/local_deep_research/benchmarks/metrics/calculation.py b/src/local_deep_research/benchmarks/metrics/calculation.py
index 83c1aeb18c..f102b359e7 100644
--- a/src/local_deep_research/benchmarks/metrics/calculation.py
+++ b/src/local_deep_research/benchmarks/metrics/calculation.py
@@ -131,17 +131,27 @@ def evaluate_benchmark_quality(
Args:
system_config: Configuration parameters to evaluate
num_examples: Number of benchmark examples to use
- output_dir: Directory to save results (temporary if None)
+ output_dir: Directory to save results. If None, a temporary
+ directory is created inside the default ``benchmark_results``
+ output directory and removed afterwards.
Returns:
Dictionary with benchmark metrics
"""
- from ..runners import run_simpleqa_benchmark
+ from ..runners import resolve_output_dir, run_simpleqa_benchmark
- # Create temporary directory if not provided
+ # Create a temporary directory if none was provided. It must be below
+ # the data directory: run_benchmark replaces an output directory outside
+ # it, so results for a system temp directory would land elsewhere and
+ # outlive the cleanup below. Putting it inside run_benchmark's own
+ # default root keeps both under the same rule.
temp_dir = None
if output_dir is None:
- temp_dir = tempfile.mkdtemp(prefix="ldr_benchmark_")
+ scratch_root = resolve_output_dir(None, fallback="benchmark_results")
+ scratch_root.mkdir(parents=True, exist_ok=True)
+ temp_dir = tempfile.mkdtemp(
+ prefix="ldr_benchmark_", dir=str(scratch_root)
+ )
output_dir = temp_dir
try:
diff --git a/src/local_deep_research/benchmarks/optimization/optuna_optimizer.py b/src/local_deep_research/benchmarks/optimization/optuna_optimizer.py
index 88f55c4cdf..867051ba4e 100644
--- a/src/local_deep_research/benchmarks/optimization/optuna_optimizer.py
+++ b/src/local_deep_research/benchmarks/optimization/optuna_optimizer.py
@@ -117,7 +117,10 @@ class OptunaOptimizer:
Args:
base_query: The research query to use for all experiments
- output_dir: Directory to save optimization results
+ output_dir: Directory to save optimization results. It is used
+ as given, not resolved below the data directory; benchmark
+ files that ``run_benchmark`` writes for a trial follow
+ ``runners.resolve_output_dir`` instead.
model_name: Name of the LLM model to use
provider: LLM provider
search_tool: Search engine to use
diff --git a/src/local_deep_research/benchmarks/runners.py b/src/local_deep_research/benchmarks/runners.py
index 5baaf2b8ae..73185151e6 100644
--- a/src/local_deep_research/benchmarks/runners.py
+++ b/src/local_deep_research/benchmarks/runners.py
@@ -38,6 +38,80 @@ def format_query(question: str, dataset_type: str = "simpleqa") -> str:
return question
+class BenchmarkOutputDirectoryError(ValueError):
+ """No benchmark output directory below the data directory is usable."""
+
+
+def _below_data_directory(value: str, data_dir: Path) -> Optional[Path]:
+ """Resolve ``value`` and return it if it is strictly below ``data_dir``.
+
+ Returns None for a value that resolves to ``data_dir`` itself, outside
+ it, or cannot be resolved.
+ """
+ from ..security.path_validator import resolve_allow_missing
+
+ try:
+ path = Path(os.path.expandvars(str(value))).expanduser()
+ if not path.is_absolute():
+ path = data_dir / path
+ resolved = resolve_allow_missing(path)
+ except (OSError, RuntimeError, ValueError):
+ return None
+ if resolved != data_dir and resolved.is_relative_to(data_dir):
+ return resolved
+ return None
+
+
+def resolve_output_dir(output_dir: Optional[str], *, fallback: str) -> Path:
+ """Return the benchmark output directory, resolved below the data dir.
+
+ ``$VAR`` and ``~`` are expanded, and a relative value is taken relative
+ to the data directory (``LDR_DATA_DIR`` or the platform default), not
+ the working directory. ``None``, ``""`` and ``"."`` select ``fallback``.
+ Any other value that resolves outside the data directory, to the data
+ directory itself, or cannot be resolved, is replaced by ``fallback``
+ (relative to the data directory) and a warning is logged.
+
+ Raises:
+ BenchmarkOutputDirectoryError: ``fallback`` is not usable either, for
+ example because ``<data dir>/<fallback>`` is a symlink to
+ another disk. Benchmark files are never written into the data
+ directory root.
+ ValueError, OSError: ``LDR_DATA_DIR`` is invalid or the data
+ directory cannot be resolved.
+
+ Symlinks are followed with ``resolve_allow_missing`` before the
+ comparison, so a link that leaves the data directory, a symlink loop and
+ a chain whose resolved form outgrows PATH_MAX are not accepted. The
+ check is a snapshot: it does not protect against a concurrent change to
+ the filesystem.
+ """
+ # Lazy imports: path_validator imports config.paths at module level.
+ from ..config.paths import get_data_directory
+ from ..security.path_validator import resolve_allow_missing
+
+ data_dir = resolve_allow_missing(get_data_directory())
+ if output_dir is not None and Path(str(output_dir)) != Path("."):
+ resolved = _below_data_directory(output_dir, data_dir)
+ if resolved is not None:
+ return resolved
+ logger.warning(
+ "Benchmark output directory is not below the data directory or "
+ f"cannot be resolved; using the default {fallback!r} there"
+ )
+ resolved = _below_data_directory(fallback, data_dir)
+ if resolved is not None:
+ return resolved
+ raise BenchmarkOutputDirectoryError(
+ "No benchmark output directory below the data directory is usable: "
+ f"neither the requested directory nor the default {fallback!r} "
+ "resolves to a directory inside it. Choose an output directory "
+ "inside the data directory, set LDR_DATA_DIR to a data directory "
+ "that contains the location you want, or remove the symlink that "
+ f"points {fallback!r} outside the data directory."
+ )
+
+
def run_benchmark(
dataset_type: str,
dataset_path: Optional[str] = None,
@@ -58,7 +132,12 @@ def run_benchmark(
dataset_type: Type of dataset ("simpleqa" or "browsecomp")
dataset_path: Optional custom dataset path
num_examples: Number of examples to use
- output_dir: Directory to save results
+ output_dir: Directory to save results. It is resolved below the
+ data directory by :func:`resolve_output_dir`: a relative value
+ is relative to the data directory, and a value outside it (or
+ the data directory itself) is replaced by ``benchmark_results``
+ there. Raises ``BenchmarkOutputDirectoryError`` before any
+ work when ``benchmark_results`` is not usable either.
run_evaluation: Whether to evaluate results
evaluation_config: Custom LLM config for evaluation
search_config: Custom search parameters
@@ -67,8 +146,13 @@ def run_benchmark(
seed: Random seed for reproducibility
Returns:
- Dictionary with benchmark results and metrics
+ Dictionary with benchmark results and metrics. The returned file
+ paths are inside the resolved output directory.
"""
+ output_dir = str(
+ resolve_output_dir(output_dir, fallback="benchmark_results")
+ )
+
# Ensure output directory exists
os.makedirs(output_dir, exist_ok=True)
diff --git a/src/local_deep_research/database/backup/backup_service.py b/src/local_deep_research/database/backup/backup_service.py
index 8814763e1e..45d3806007 100644
--- a/src/local_deep_research/database/backup/backup_service.py
+++ b/src/local_deep_research/database/backup/backup_service.py
@@ -1113,8 +1113,21 @@ class BackupService:
# purge cannot delete is checked again with the new one.
_get_backup_verifications(self.username).clear()
- # Delete all existing backup files
+ # Delete all existing backup files, including quarantined
+ # damaged copies (they are still under the old key).
+ # list_backups() reports only bare names from a filtered glob,
+ # but re-check each joined path before removing it, so an entry
+ # that is not a bare in-root name (a poisoned listing or a future
+ # caller) cannot redirect the unlink outside backup_dir.
+ # is_safe_glob_result also skips a symlink entry. The check is a
+ # snapshot, not atomic with the unlink.
for backup_path in self._managed_backup_paths():
+ if not is_safe_glob_result(backup_path, self.backup_dir):
+ logger.warning(
+ f"Refusing to delete backup {backup_path.name}: it "
+ "is a symlink or resolves outside the backup directory"
+ )
+ continue
try:
self._remove_backup_files(backup_path)
logger.debug(f"Purged old-key backup: {backup_path.name}")
diff --git a/tests/api/test_benchmark_functions.py b/tests/api/test_benchmark_functions.py
index 7e064ece6a..289defe0f9 100644
--- a/tests/api/test_benchmark_functions.py
+++ b/tests/api/test_benchmark_functions.py
@@ -2,6 +2,20 @@
from unittest.mock import patch
+import pytest
+
+
[email protected](autouse=True)
+def _pin_data_directory(tmp_path, monkeypatch):
+ """Keep comparison output out of the real user data directory.
+
+ compare_configurations resolves its output root inside LDR_DATA_DIR and
+ replaces any other location with ``benchmark_comparisons`` there.
+ Pinning the data directory to this test's tmp_path keeps that root per
+ test.
+ """
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+
class TestGetAvailableBenchmarks:
def test_returns_list(self):
diff --git a/tests/api/test_benchmark_functions_deep_coverage.py b/tests/api/test_benchmark_functions_deep_coverage.py
index d8331614c8..4a9e224371 100644
--- a/tests/api/test_benchmark_functions_deep_coverage.py
+++ b/tests/api/test_benchmark_functions_deep_coverage.py
@@ -10,10 +10,24 @@ Targets:
from unittest.mock import patch
+import pytest
+
MODULE = "local_deep_research.api.benchmark_functions"
[email protected](autouse=True)
+def _pin_data_directory(tmp_path, monkeypatch):
+ """Keep comparison output out of the real user data directory.
+
+ compare_configurations resolves its output root inside LDR_DATA_DIR and
+ replaces any other location with ``benchmark_comparisons`` there.
+ Pinning the data directory to this test's tmp_path keeps that root per
+ test.
+ """
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+
+
class TestEvaluateSimpleqa:
@patch(f"{MODULE}.run_simpleqa_benchmark")
def test_evaluation_config_with_model_only(self, mock_run):
diff --git a/tests/benchmarks/test_benchmark_functions_extended.py b/tests/benchmarks/test_benchmark_functions_extended.py
index 6738172927..b1c799bb62 100644
--- a/tests/benchmarks/test_benchmark_functions_extended.py
+++ b/tests/benchmarks/test_benchmark_functions_extended.py
@@ -15,6 +15,18 @@ from local_deep_research.benchmarks.benchmark_functions import (
)
[email protected](autouse=True)
+def _pin_data_directory(tmp_path, monkeypatch):
+ """Keep comparison output out of the real user data directory.
+
+ compare_configurations resolves its output root inside LDR_DATA_DIR and
+ replaces any other location with ``benchmark_comparisons`` there.
+ Pinning the data directory to this test's tmp_path keeps that root per
+ test.
+ """
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+
+
# ── get_available_benchmarks ───────────────────────────────────────
diff --git a/tests/benchmarks/test_benchmark_functions_extra_coverage.py b/tests/benchmarks/test_benchmark_functions_extra_coverage.py
index d878859498..801491cec3 100644
--- a/tests/benchmarks/test_benchmark_functions_extra_coverage.py
+++ b/tests/benchmarks/test_benchmark_functions_extra_coverage.py
@@ -13,9 +13,23 @@ Targets the 36 missing lines not covered by test_benchmark_functions_coverage.py
from unittest.mock import patch
+import pytest
+
MODULE = "local_deep_research.benchmarks.benchmark_functions"
[email protected](autouse=True)
+def _pin_data_directory(tmp_path, monkeypatch):
+ """Keep comparison output out of the real user data directory.
+
+ compare_configurations resolves its output root inside LDR_DATA_DIR and
+ replaces any other location with ``benchmark_comparisons`` there.
+ Pinning the data directory to this test's tmp_path keeps that root per
+ test.
+ """
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+
+
def _no_settings(key, *args, **kwargs):
return None
diff --git a/tests/benchmarks/test_calculation_coverage.py b/tests/benchmarks/test_calculation_coverage.py
index dcfbfcfdcf..f85ad9c478 100644
--- a/tests/benchmarks/test_calculation_coverage.py
+++ b/tests/benchmarks/test_calculation_coverage.py
@@ -255,11 +255,14 @@ class TestEvaluateBenchmarkQuality:
assert result["quality_score"] == 0.75
mock_run.assert_called_once()
- def test_uses_temp_dir_when_output_dir_none(self):
+ def test_uses_temp_dir_when_output_dir_none(self, tmp_path, monkeypatch):
from local_deep_research.benchmarks.metrics.calculation import (
evaluate_benchmark_quality,
)
+ # The temporary directory is created inside the default
+ # benchmark_results directory below the data directory.
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
mock_results = {"metrics": {"accuracy": 0.5}}
with patch(
"local_deep_research.benchmarks.runners.run_simpleqa_benchmark",
@@ -343,12 +346,18 @@ class TestEvaluateBenchmarkQuality:
assert search_config["model_name"] == "gpt-4"
assert search_config["provider"] == "openai"
- def test_temp_dir_cleaned_up_on_success(self):
- """When output_dir is None, a temp dir is created and cleaned up."""
+ def test_temp_dir_cleaned_up_on_success(self, tmp_path, monkeypatch):
+ """When output_dir is None, a temp dir is created and cleaned up.
+
+ It is created inside the default benchmark_results directory below
+ the data directory, so run_benchmark keeps its results there and the
+ cleanup removes them too.
+ """
from local_deep_research.benchmarks.metrics.calculation import (
evaluate_benchmark_quality,
)
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
created_dirs = []
original_mkdtemp = __import__("tempfile").mkdtemp
@@ -375,6 +384,10 @@ class TestEvaluateBenchmarkQuality:
)
assert len(created_dirs) == 1
+ assert (
+ Path(created_dirs[0]).parent
+ == (tmp_path / "benchmark_results").resolve()
+ )
# The temp dir should have been cleaned up
assert not Path(created_dirs[0]).exists()
diff --git a/tests/benchmarks/test_runners.py b/tests/benchmarks/test_runners.py
index 31611cd87b..dff96e933c 100644
--- a/tests/benchmarks/test_runners.py
+++ b/tests/benchmarks/test_runners.py
@@ -12,6 +12,19 @@ from unittest.mock import Mock, patch
from pathlib import Path
import tempfile
+import pytest
+
+
[email protected](autouse=True)
+def _pin_data_directory(tmp_path, monkeypatch):
+ """Keep benchmark output out of the real user data directory.
+
+ run_benchmark resolves its output directory inside LDR_DATA_DIR and replaces
+ any other location with a default directory there. Pinning the data
+ directory to this test's tmp_path keeps that default per test.
+ """
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+
class TestFormatQuery:
"""Tests for the format_query function."""
@@ -61,11 +74,15 @@ class TestFormatQuery:
class TestRunBenchmark:
"""Tests for the run_benchmark function."""
- def test_run_benchmark_creates_output_dir(self):
+ def test_run_benchmark_creates_output_dir(self, monkeypatch):
"""run_benchmark creates output directory."""
from local_deep_research.benchmarks.runners import run_benchmark
with tempfile.TemporaryDirectory() as tmpdir:
+ # run_benchmark output_dir is now confined to the LDR data directory,
+ # so point the data dir at tmpdir to keep this out-of-tree output
+ # location honoured rather than clamped to the default.
+ monkeypatch.setenv("LDR_DATA_DIR", tmpdir)
output_dir = Path(tmpdir) / "new_dir"
# Mock the dataset loading and search
diff --git a/tests/benchmarks/test_runners_coverage.py b/tests/benchmarks/test_runners_coverage.py
index 8816b1befc..f4e07c105a 100644
--- a/tests/benchmarks/test_runners_coverage.py
+++ b/tests/benchmarks/test_runners_coverage.py
@@ -12,10 +12,23 @@ import tempfile
from pathlib import Path
from unittest.mock import Mock, patch
+import pytest
+
MODULE = "local_deep_research.benchmarks.runners"
[email protected](autouse=True)
+def _pin_data_directory(tmp_path, monkeypatch):
+ """Keep benchmark output out of the real user data directory.
+
+ run_benchmark resolves its output directory inside LDR_DATA_DIR and replaces
+ any other location with a default directory there. Pinning the data
+ directory to this test's tmp_path keeps that default per test.
+ """
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+
+
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
@@ -923,7 +936,9 @@ class TestOutputFileCleanup:
@patch(f"{MODULE}.DatasetRegistry")
@patch(f"{MODULE}.time")
- def test_existing_files_are_removed(self, mock_time, mock_registry):
+ def test_existing_files_are_removed(
+ self, mock_time, mock_registry, monkeypatch
+ ):
from local_deep_research.benchmarks.runners import run_benchmark
mock_time.strftime.return_value = "20260101_000000"
@@ -934,19 +949,23 @@ class TestOutputFileCleanup:
mock_registry.create_dataset.return_value = mock_dataset
with tempfile.TemporaryDirectory() as tmpdir:
+ # run_benchmark output_dir is now confined below the LDR data
+ # directory (never its root); point the data dir at tmpdir so
+ # cleanup operates on a directory inside it.
+ monkeypatch.setenv("LDR_DATA_DIR", tmpdir)
+ out = Path(tmpdir) / "results"
+ out.mkdir()
for suffix in ["_results.jsonl", "_evaluation.jsonl", "_report.md"]:
- p = Path(tmpdir) / f"simpleqa_20260101_000000{suffix}"
+ p = out / f"simpleqa_20260101_000000{suffix}"
p.write_text("old content")
run_benchmark(
dataset_type="simpleqa",
- output_dir=tmpdir,
+ output_dir=str(out),
run_evaluation=False,
)
- results_file = (
- Path(tmpdir) / "simpleqa_20260101_000000_results.jsonl"
- )
+ results_file = out / "simpleqa_20260101_000000_results.jsonl"
assert not results_file.exists() or results_file.read_text() == ""
diff --git a/tests/security/test_benchmark_and_temp_lifecycle.py b/tests/security/test_benchmark_and_temp_lifecycle.py
index 0f03ce8952..436c7ffe52 100644
--- a/tests/security/test_benchmark_and_temp_lifecycle.py
+++ b/tests/security/test_benchmark_and_temp_lifecycle.py
@@ -109,6 +109,11 @@ def test_runner_cleanup_only_unlinks_its_timestamped_outputs(
):
from local_deep_research.benchmarks import runners
+ # run_benchmark keeps its output directory below the data directory and
+ # never writes into the data directory root itself.
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+ out = tmp_path / "benchmark-output"
+ out.mkdir()
timestamp = "20000101_000000"
monkeypatch.setattr(
runners,
@@ -121,19 +126,17 @@ def test_runner_cleanup_only_unlinks_its_timestamped_outputs(
runners.DatasetRegistry, "create_dataset", lambda **_kwargs: dataset
)
for suffix in ("results.jsonl", "evaluation.jsonl", "report.md"):
- (tmp_path / f"simpleqa_{timestamp}_{suffix}").write_text("stale")
- sibling = tmp_path / f"simpleqa_{timestamp}_results-backup.jsonl"
- nested = tmp_path / "nested" / f"simpleqa_{timestamp}_results.jsonl"
+ (out / f"simpleqa_{timestamp}_{suffix}").write_text("stale")
+ sibling = out / f"simpleqa_{timestamp}_results-backup.jsonl"
+ nested = out / "nested" / f"simpleqa_{timestamp}_results.jsonl"
sibling.write_text("canary")
nested.parent.mkdir()
nested.write_text("nested canary")
- runners.run_benchmark(
- "simpleqa", output_dir=str(tmp_path), run_evaluation=False
- )
+ runners.run_benchmark("simpleqa", output_dir=str(out), run_evaluation=False)
for suffix in ("results.jsonl", "evaluation.jsonl", "report.md"):
- assert not (tmp_path / f"simpleqa_{timestamp}_{suffix}").exists()
+ assert not (out / f"simpleqa_{timestamp}_{suffix}").exists()
assert sibling.read_text() == "canary"
assert nested.read_text() == "nested canary"
@@ -235,23 +238,29 @@ def test_reporting_writer_keeps_adjacent_canary(tmp_path, monkeypatch):
def test_comparison_api_writer_keeps_adjacent_canary(tmp_path, monkeypatch):
from local_deep_research.api import benchmark_functions
+ # compare_configurations keeps its output root below the data directory
+ # and never writes into the data directory root itself.
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+ out = tmp_path / "comparison-output"
+ out.mkdir()
_allow_benchmark_writes(monkeypatch)
monkeypatch.setattr(
benchmark_functions,
"run_benchmark",
lambda **_kwargs: {"metrics": {}, "total_examples": 0},
)
- canary = tmp_path / "comparison_simpleqa_backup.md"
- nested = tmp_path / "nested" / "comparison_simpleqa.md"
+ canary = out / "comparison_simpleqa_backup.md"
+ nested = out / "nested" / "comparison_simpleqa.md"
canary.write_text("canary")
nested.parent.mkdir()
nested.write_text("nested canary")
result = benchmark_functions.compare_configurations(
- configurations=[{"name": "only"}], output_dir=str(tmp_path)
+ configurations=[{"name": "only"}], output_dir=str(out)
)
assert Path(result["report_path"]).is_file()
+ assert Path(result["report_path"]).parent == out.resolve()
assert canary.read_text() == "canary"
assert nested.read_text() == "nested canary"
@@ -261,12 +270,15 @@ def test_quality_evaluation_removes_real_mkdtemp_directory(
):
from local_deep_research.benchmarks.metrics import calculation
+ # The temporary directory is created inside the default
+ # benchmark_results directory below the data directory.
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
temp_dir = tmp_path / "ldr_benchmark_created"
temp_dir.mkdir()
monkeypatch.setattr(
calculation,
"tempfile",
- SimpleNamespace(mkdtemp=lambda prefix: str(temp_dir)),
+ SimpleNamespace(mkdtemp=lambda prefix, dir: str(temp_dir)),
)
monkeypatch.setattr(
"local_deep_research.benchmarks.runners.run_simpleqa_benchmark",
@@ -282,13 +294,16 @@ def test_quality_evaluation_rmtree_targets_only_mkdtemp_dir_on_error(
):
from local_deep_research.benchmarks.metrics import calculation
+ # The temporary directory is created inside the default
+ # benchmark_results directory below the data directory.
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
temp_dir = tmp_path / "ldr_benchmark_created"
temp_dir.mkdir()
captured = []
monkeypatch.setattr(
calculation,
"tempfile",
- SimpleNamespace(mkdtemp=lambda prefix: str(temp_dir)),
+ SimpleNamespace(mkdtemp=lambda prefix, dir: str(temp_dir)),
)
monkeypatch.setattr(
"local_deep_research.benchmarks.runners.run_simpleqa_benchmark",
diff --git a/tests/security/test_benchmark_output_containment.py b/tests/security/test_benchmark_output_containment.py
new file mode 100644
index 0000000000..888508762e
--- /dev/null
+++ b/tests/security/test_benchmark_output_containment.py
@@ -0,0 +1,334 @@
+"""Benchmark output directories stay below the LDR data directory.
+
+``resolve_output_dir`` (benchmarks/runners.py) resolves the output root of
+``run_benchmark``, of both ``compare_configurations`` helpers
+(``api.benchmark_functions`` and ``benchmarks.benchmark_functions``) and of
+``evaluate_benchmark_quality``'s temporary directory. A relative value is
+taken relative to the data directory. A value outside it, the data directory
+itself, or one that cannot be resolved is replaced by a default directory
+below it; if that default is not usable either, the resolver raises rather
+than writing into the data directory root.
+``tests/security/test_settings_path_escape.py`` drives the real runner with
+an outside directory (the #7417 tripwire).
+
+Library storage paths follow a different, operator-managed policy and are
+not covered here. Every hostile destination stays beneath ``tmp_path``.
+"""
+
+from pathlib import Path
+from unittest.mock import patch
+
+import pytest
+
+from local_deep_research.benchmarks.runners import (
+ BenchmarkOutputDirectoryError,
+ resolve_output_dir,
+)
+
+from tests.security.path_max_fixtures import build_path_max_escape
+
+
+def data_dir(tmp_path, monkeypatch):
+ path = tmp_path / "ldr-data"
+ path.mkdir()
+ monkeypatch.setenv("LDR_DATA_DIR", str(path))
+ return path.resolve()
+
+
+def outside(tmp_path):
+ path = tmp_path / "outside"
+ path.mkdir()
+ (path / "CANARY").write_text("do not alter")
+ return path
+
+
+def _files_below(directory: Path) -> dict:
+ return {
+ path.relative_to(directory): path.read_bytes()
+ for path in directory.rglob("*")
+ if path.is_file()
+ }
+
+
+# ---------------------------------------------------------------------------
+# resolve_output_dir
+# ---------------------------------------------------------------------------
+
+
+def test_relative_output_dir_resolves_under_the_data_directory(
+ data_dir, tmp_path, monkeypatch
+):
+ workspace = tmp_path / "workspace"
+ workspace.mkdir()
+ monkeypatch.chdir(workspace)
+
+ resolved = resolve_output_dir("nested/results", fallback="default")
+
+ assert resolved == data_dir / "nested" / "results"
+
+
+def test_absolute_output_dir_inside_the_data_directory_is_kept(data_dir):
+ requested = data_dir / "kept" / "results"
+
+ assert resolve_output_dir(str(requested), fallback="default") == requested
+
+
[email protected]("kind", ["absolute", "relative", "expandvars"])
+def test_outside_output_dir_is_replaced_by_the_fallback(
+ data_dir, outside, monkeypatch, kind
+):
+ monkeypatch.setenv("LDR_TEST_OUTSIDE", str(outside))
+ requested = {
+ "absolute": str(outside / "results"),
+ "relative": "../outside/results",
+ "expandvars": "$LDR_TEST_OUTSIDE/results",
+ }[kind]
+
+ resolved = resolve_output_dir(requested, fallback="benchmark_results")
+
+ assert resolved == data_dir / "benchmark_results"
+
+
+def test_symlink_out_of_the_data_directory_is_replaced(data_dir, outside):
+ link = data_dir / "link"
+ try:
+ link.symlink_to(outside, target_is_directory=True)
+ except (OSError, NotImplementedError):
+ pytest.skip("Cannot create symlinks in this environment")
+
+ resolved = resolve_output_dir("link/results", fallback="benchmark_results")
+
+ assert resolved == data_dir / "benchmark_results"
+
+
+def test_path_max_symlink_chain_out_of_the_data_directory_is_replaced(
+ data_dir, outside
+):
+ """A bare non-strict resolve() reports this escape as contained."""
+ escape = build_path_max_escape(data_dir, outside)
+
+ resolved = resolve_output_dir(
+ f"{escape}/results", fallback="benchmark_results"
+ )
+
+ assert resolved == data_dir / "benchmark_results"
+
+
[email protected]("unset", [None, "", ".", "./"])
+def test_unset_output_dir_selects_the_fallback(data_dir, unset):
+ assert (
+ resolve_output_dir(unset, fallback="benchmark_results")
+ == data_dir / "benchmark_results"
+ )
+
+
[email protected]("kind", ["absolute", "dotdot"])
+def test_data_directory_root_is_replaced_by_the_fallback(data_dir, kind):
+ requested = str(data_dir) if kind == "absolute" else "sub/.."
+
+ assert (
+ resolve_output_dir(requested, fallback="benchmark_results")
+ == data_dir / "benchmark_results"
+ )
+
+
+def test_malformed_output_dir_is_replaced_by_the_fallback(data_dir):
+ assert (
+ resolve_output_dir("bad\x00path", fallback="benchmark_results")
+ == data_dir / "benchmark_results"
+ )
+
+
+def test_unusable_fallback_raises_instead_of_using_the_data_root(
+ data_dir, outside
+):
+ with pytest.raises(BenchmarkOutputDirectoryError, match="LDR_DATA_DIR"):
+ resolve_output_dir(str(outside), fallback="bad\x00fallback")
+
+
+def test_symlinked_default_fails_closed(data_dir, outside):
+ """An operator symlinked the default directory to another disk."""
+ link = data_dir / "benchmark_results"
+ try:
+ link.symlink_to(outside, target_is_directory=True)
+ except (OSError, NotImplementedError):
+ pytest.skip("Cannot create symlinks in this environment")
+
+ for requested in (str(link), "benchmark_results", None):
+ with pytest.raises(
+ BenchmarkOutputDirectoryError, match="remove the symlink"
+ ):
+ resolve_output_dir(requested, fallback="benchmark_results")
+
+
+def test_unavailable_home_expansion_uses_the_fallback(data_dir, monkeypatch):
+ original = Path.expanduser
+
+ def expanduser(path):
+ if str(path) == "~unavailable":
+ raise RuntimeError("home unavailable")
+ return original(path)
+
+ monkeypatch.setattr(Path, "expanduser", expanduser)
+
+ assert (
+ resolve_output_dir("~unavailable", fallback="benchmark_results")
+ == data_dir / "benchmark_results"
+ )
+
+
+def test_invalid_operator_data_root_is_not_hidden():
+ with patch(
+ "local_deep_research.config.paths.get_data_directory",
+ side_effect=ValueError("invalid operator root"),
+ ):
+ with pytest.raises(ValueError, match="invalid operator root"):
+ resolve_output_dir("results", fallback="benchmark_results")
+
+
+# ---------------------------------------------------------------------------
+# Writers
+# ---------------------------------------------------------------------------
+
+
+def test_run_benchmark_writes_a_relative_output_dir_under_the_data_directory(
+ data_dir, tmp_path, monkeypatch
+):
+ from local_deep_research.benchmarks import runners
+
+ workspace = tmp_path / "workspace"
+ workspace.mkdir()
+ monkeypatch.chdir(workspace)
+
+ with (
+ patch.object(
+ runners.DatasetRegistry, "create_dataset", side_effect=RuntimeError
+ ),
+ patch.object(
+ runners,
+ "load_dataset",
+ return_value=[{"id": "1", "problem": "q", "answer": "a"}],
+ ),
+ patch.object(
+ runners,
+ "quick_summary",
+ return_value={"summary": "a", "sources": []},
+ ),
+ ):
+ result = runners.run_benchmark(
+ "simpleqa", output_dir="nested/results", run_evaluation=False
+ )
+
+ results = Path(result["results_path"])
+ assert results.parent == data_dir / "nested" / "results"
+ assert results.is_file()
+ assert list(workspace.iterdir()) == []
+
+
+def test_run_benchmark_fails_closed_without_writing(
+ data_dir, outside, monkeypatch
+):
+ """No result file lands in the data root or behind the symlink."""
+ from local_deep_research.benchmarks import runners
+
+ link = data_dir / "benchmark_results"
+ try:
+ link.symlink_to(outside, target_is_directory=True)
+ except (OSError, NotImplementedError):
+ pytest.skip("Cannot create symlinks in this environment")
+ quick_summary = []
+ monkeypatch.setattr(
+ runners, "quick_summary", lambda **kwargs: quick_summary.append(1)
+ )
+ before_outside = _files_below(outside)
+
+ with pytest.raises(BenchmarkOutputDirectoryError):
+ runners.run_benchmark(
+ "simpleqa", output_dir=str(link), run_evaluation=False
+ )
+
+ assert quick_summary == []
+ assert sorted(path.name for path in data_dir.iterdir()) == [
+ "benchmark_results"
+ ]
+ assert _files_below(outside) == before_outside
+
+
+ "module_name",
+ [
+ "local_deep_research.api.benchmark_functions",
+ "local_deep_research.benchmarks.benchmark_functions",
+ ],
+)
[email protected]("output_kind", ["absolute", "relative"])
+def test_comparison_keeps_results_and_report_under_one_root(
+ data_dir, tmp_path, module_name, output_kind
+):
+ import importlib
+
+ module = importlib.import_module(module_name)
+ outside = tmp_path / "outside-comparison"
+ requested = str(outside) if output_kind == "absolute" else "comparisons"
+ expected = data_dir / (
+ "benchmark_comparisons" if output_kind == "absolute" else "comparisons"
+ )
+ output_paths = []
+
+ def run_benchmark(**kwargs):
+ output_paths.append(Path(kwargs["output_dir"]))
+ return {"metrics": {}, "total_examples": 1}
+
+ with (
+ patch.object(module, "run_benchmark", side_effect=run_benchmark),
+ patch(
+ "local_deep_research.security.file_write_verifier.write_file_verified"
+ ) as write,
+ ):
+ result = module.compare_configurations(
+ output_dir=requested,
+ configurations=[{"name": "First Setup"}, {"name": "Second Setup"}],
+ )
+
+ assert output_paths == [expected / "First_Setup", expected / "Second_Setup"]
+ report = Path(result["report_path"])
+ assert report.parent == expected
+ write.assert_called_once()
+ assert write.call_args.args[0] == str(report)
+ assert expected.is_dir()
+ assert not outside.exists()
+
+
+def test_quality_evaluation_keeps_its_temporary_results_contained(data_dir):
+ """The runner keeps the temporary directory, so the cleanup removes it.
+
+ A system temporary directory is outside the data directory, so the
+ runner would write to its fallback instead and the results would
+ outlive the cleanup.
+ """
+ from local_deep_research.benchmarks.metrics import calculation
+
+ used = []
+
+ def run_simpleqa_benchmark(**kwargs):
+ directory = resolve_output_dir(
+ kwargs["output_dir"], fallback="benchmark_results"
+ )
+ directory.mkdir(parents=True, exist_ok=True)
+ (directory / "results.jsonl").write_text("{}\n")
+ used.append(directory)
+ return {"metrics": {"accuracy": 1.0}}
+
+ with patch(
+ "local_deep_research.benchmarks.runners.run_simpleqa_benchmark",
+ side_effect=run_simpleqa_benchmark,
+ ):
+ result = calculation.evaluate_benchmark_quality({}, output_dir=None)
+
+ assert result["accuracy"] == 1.0
+ assert len(used) == 1
+ assert used[0].parent == data_dir / "benchmark_results"
+ assert not used[0].exists()
+ assert list((data_dir / "benchmark_results").iterdir()) == []
diff --git a/tests/security/test_identity_paths_and_host_effects.py b/tests/security/test_identity_paths_and_host_effects.py
index 4b208b9786..b4f44c1204 100644
--- a/tests/security/test_identity_paths_and_host_effects.py
+++ b/tests/security/test_identity_paths_and_host_effects.py
@@ -270,26 +270,18 @@ def backup_service_in_tmpdir(tmp_path, monkeypatch):
return service
- strict=True,
- reason=(
- "DEFENCE IN DEPTH: purge_and_refresh deletes every entry returned "
- "by list_backups without resolving it and checking containment in "
- "self.backup_dir (backup_service.py purge_and_refresh). The value "
- "is server-derived today -- list_backups enumerates backup_dir -- "
- "so this is NOT a reachable attack, only a missing guard that a "
- "future caller or a poisoned directory listing could turn into an "
- "out-of-root unlink. This test monkeypatches list_backups to "
- "inject the hostile path. Fix: resolve each entry against "
- "backup_dir and skip anything outside it. Flip the marker when "
- "the containment check lands. The sibling symlink test below "
- "passes already -- bare names are resolved against backup_dir."
- ),
-)
@pytest.mark.parametrize("forged_path", ["../outside.db", "/tmp/not-used"])
def test_purge_does_not_delete_backup_entries_outside_root(
tmp_path, monkeypatch, backup_service_in_tmpdir, forged_path
):
+ """Defence in depth: a listing entry outside backup_dir is not deleted.
+
+ list_backups only reports bare names from backup_dir, so this is not a
+ reachable attack today; the test monkeypatches list_backups to inject
+ the hostile path. purge_and_refresh re-checks each joined path with
+ is_safe_glob_result and skips anything outside backup_dir. This was a
+ strict xfail until that check landed.
+ """
service = backup_service_in_tmpdir
outside_canary = tmp_path / "outside.db"
outside_canary.write_bytes(b"do not delete")
diff --git a/tests/security/test_report_cleanup_only_for_deleted_rows.py b/tests/security/test_report_cleanup_only_for_deleted_rows.py
new file mode 100644
index 0000000000..77a1547a4f
--- /dev/null
+++ b/tests/security/test_report_cleanup_only_for_deleted_rows.py
@@ -0,0 +1,132 @@
+"""Report files are removed only for research rows the route really deleted.
+
+Both FastAPI cleanup routes (``delete_research`` and ``clear_history`` in
+web/routers/research.py) share ``_unlink_report_file_if_inside_root`` for
+the containment checks; ``tests/web/routers/test_report_unlink_guard.py``
+and ``tests/security/test_report_path_deletion.py`` pin those. These tests
+pin the ordering: a row whose guarded delete was refused (claimed by a
+worker, or in progress) keeps its report file, and cleanup runs only for
+the rows that were deleted.
+
+Queue-state cleanup is scoped to the requesting user: since #7606 both routes
+pass ``acting_user=username`` to ``cleanup_queued_research_state`` so another
+user's queue row (and the task metadata it still needs) is left alone. The
+assertions below pin that keyword as well as the deleted-row IDs.
+"""
+
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from local_deep_research.constants import ResearchStatus
+from local_deep_research.web.routers import research as research_router
+
+
+def _session_context(db):
+ context = MagicMock()
+ context.__enter__.return_value = db
+ # Never swallow an exception raised inside the route's session block.
+ context.__exit__.return_value = False
+ return context
+
+
[email protected]("deleted", [0, 1])
+def test_delete_route_removes_the_report_only_after_row_deletion(deleted):
+ db = MagicMock()
+ db.query.return_value.scalar.return_value = False
+ db.query.return_value.filter_by.return_value.first.return_value = (
+ SimpleNamespace(
+ status=ResearchStatus.COMPLETED, report_path="report.md"
+ )
+ )
+ with (
+ patch.object(
+ research_router,
+ "get_user_db_session",
+ return_value=_session_context(db),
+ ),
+ patch.object(
+ research_router, "_unlink_report_file_if_inside_root"
+ ) as unlink,
+ patch.object(
+ research_router,
+ "_delete_research_with_dependents",
+ return_value=deleted,
+ ),
+ patch(
+ "local_deep_research.web.queue.lifecycle_cleanup.cleanup_queued_research_state"
+ ) as cleanup,
+ ):
+ result = research_router.delete_research(
+ MagicMock(), "research-id", "alice"
+ )
+
+ if deleted:
+ assert result == {"status": "success"}
+ unlink.assert_called_once_with("report.md")
+ cleanup.assert_called_once_with(
+ db, ["research-id"], acting_user="alice"
+ )
+ db.commit.assert_called_once()
+ else:
+ # The row still exists, so a worker claimed it after the pre-check.
+ assert result.status_code == 400
+ unlink.assert_not_called()
+ cleanup.assert_not_called()
+ db.commit.assert_not_called()
+
+
+def test_clear_history_removes_reports_only_for_deleted_rows():
+ rows = [
+ SimpleNamespace(
+ id="finished",
+ status=ResearchStatus.COMPLETED,
+ report_path="finished.md",
+ ),
+ SimpleNamespace(
+ id="claimed",
+ status=ResearchStatus.COMPLETED,
+ report_path="claimed.md",
+ ),
+ SimpleNamespace(
+ id="active",
+ status=ResearchStatus.IN_PROGRESS,
+ report_path="active.md",
+ ),
+ ]
+ db = MagicMock()
+ db.query.return_value.scalar.return_value = False
+ db.query.return_value.all.return_value = rows
+ with (
+ patch.object(
+ research_router,
+ "get_user_db_session",
+ return_value=_session_context(db),
+ ),
+ patch.object(
+ research_router, "get_active_research_ids", return_value=[]
+ ),
+ patch.object(
+ research_router, "_unlink_report_file_if_inside_root"
+ ) as unlink,
+ patch.object(
+ research_router,
+ "_delete_research_with_dependents",
+ side_effect=[1, 0],
+ ) as cascade,
+ patch(
+ "local_deep_research.web.queue.lifecycle_cleanup.cleanup_queued_research_state"
+ ) as cleanup,
+ ):
+ result = research_router.clear_history(MagicMock(), "alice")
+
+ assert result == {"status": "success"}
+ # "active" is protected before the cascade; "claimed" is refused by it.
+ assert [call.args[1] for call in cascade.call_args_list] == [
+ "finished",
+ "claimed",
+ ]
+ unlink.assert_called_once_with("finished.md")
+ cleanup.assert_called_once_with(db, ["finished"], acting_user="alice")
+ db.commit.assert_called_once()
diff --git a/tests/security/test_settings_bulk_import.py b/tests/security/test_settings_bulk_import.py
index a4f92da6c4..1656d04d17 100644
--- a/tests/security/test_settings_bulk_import.py
+++ b/tests/security/test_settings_bulk_import.py
@@ -7,8 +7,11 @@ locations and environment overrides remain supported, including locations
outside LDR_DATA_DIR, but user imports cannot redirect them. These checks
exercise the real manager and download-service path with isolated roots.
-The separate benchmark-output containment gap remains a strict xfail.
-Select validation and valid user-editable imports are ordinary controls.
+Benchmark output is contained where it is written instead: run_benchmark
+resolves its output directory inside LDR_DATA_DIR (test_settings_path_escape).
+The import itself still stores an outside benchmark.output_dir value, which
+remains a strict xfail. Select validation and valid user-editable imports are
+ordinary controls.
"""
from pathlib import Path
@@ -83,9 +86,11 @@ def test_import_cannot_redirect_saved_library_root(settings_manager, value):
@pytest.mark.xfail(
strict=True,
reason=(
- "OPEN EXPOSURE: benchmark.output_dir still accepts paths outside "
- "LDR_DATA_DIR; the library's operator-only policy does not cover "
- "benchmark output. Remove this marker when that policy is fixed."
+ "OPEN: the settings import still stores a benchmark.output_dir "
+ "outside LDR_DATA_DIR; the library's operator-only policy does not "
+ "cover it. No code reads this key, and run_benchmark contains the "
+ "output directory it is given when writing. Remove this marker if "
+ "the import gains a path policy for this key."
),
)
def test_import_rejects_benchmark_path_escape(settings_manager):
@@ -123,7 +128,8 @@ def test_import_enforces_same_value_validation_as_save_routes(
``app.timezone`` is used because select is one of the classes the
validator can actually judge. Library paths have a separate operator-only
- write policy; benchmark path containment remains a separate expected failure.
+ write policy; benchmark output paths are contained by run_benchmark, and
+ the import's acceptance of them remains a separate expected failure.
Note the two validators read their constraints from different places:
the save route judges the stored ``Setting`` row, while the import
diff --git a/tests/security/test_settings_path_escape.py b/tests/security/test_settings_path_escape.py
index 1f705acc9d..39b51651f4 100644
--- a/tests/security/test_settings_path_escape.py
+++ b/tests/security/test_settings_path_escape.py
@@ -1,4 +1,4 @@
-"""Configured library-root compatibility and benchmark containment tripwires.
+"""Configured library-root compatibility and benchmark output containment.
The selected library policy (#7152/#7224, closing #6335) preserves existing
saved roots and operator overrides outside LDR_DATA_DIR. User imports cannot
@@ -6,8 +6,9 @@ redirect them; test_settings_bulk_import covers that write boundary with the
real settings manager. These resolver tests ensure configured roots still
work and document paths remain scoped to the user's library.
-Benchmark output has a separate unresolved containment policy and retains
-its strict xfails. Every configured root and canary lives under tmp_path.
+Benchmark output follows a different rule (#7417): run_benchmark resolves
+its output directory inside LDR_DATA_DIR and replaces any other location.
+Every configured root and canary lives under tmp_path.
"""
import os
@@ -164,23 +165,16 @@ def test_relative_storage_path_still_creates_a_library(tmp_path, monkeypatch):
assert library_root.is_dir()
- strict=True,
- reason=(
- "OPEN EXPOSURE: benchmark.output_dir is a user-editable setting "
- "and run_benchmark passes it straight to os.makedirs "
- "(benchmarks/runners.py:73) and then writes the results/report "
- "files under it (:113-119), with no containment check against "
- "LDR_DATA_DIR. Fix: resolve output_dir and reject/clamp anything "
- "outside the data directory. Remove this marker when the check "
- "lands."
- ),
-)
@pytest.mark.parametrize("output_dir", ["absolute", "relative"])
def test_benchmark_writer_cannot_escape_data_directory(
isolated_data_dir, tmp_path, monkeypatch, output_dir
):
- """The real runner must not create or append results outside LDR_DATA_DIR."""
+ """The real runner must not create or append results outside LDR_DATA_DIR.
+
+ This was a strict xfail until #7417's fix: run_benchmark now resolves
+ output_dir inside the data directory and replaces an outside value with
+ ``benchmark_results`` there.
+ """
from local_deep_research.benchmarks import runners
outside = tmp_path / "outside"