Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

9.6 KB
Raw Blame

Local Deep Research - Test Suite

This directory contains the test suite for the Local Deep Research project. The tests are organized into unit tests, integration tests, and search engine-specific tests.

Tests Structure

tests/
├── conftest.py              # Common test fixtures
├── run_tests.py             # Test runner with coverage reporting
├── README.md                # Test documentation
├── unit/                    # Unit tests
│   ├── test_citation.py     # Tests for citation handling
│   ├── test_config.py       # Tests for configuration management
│   ├── test_report.py       # Tests for report generation
│   └── test_search_system.py # Tests for search system
├── integration/             # Integration tests
│   └── test_db_ops.py       # Database operation tests (placeholder)
└── search_engines/          # Search engine specific tests
    └── test_wikipedia.py    # Wikipedia search engine tests

Current Coverage

Component Coverage Notes
Citation Handler High Comprehensive tests for all methods
Search System High Tests strategy selection and core functionality
Report Generator High Tests structure determination and report formatting
Configuration High Tests loading, env vars, and defaults
Wikipedia Search High Tests API interaction and result processing
Database Operations Low Database tests outlined but need implementation
Other Search Engines Low Only Wikipedia has tests so far
Web Interface None Web interface tests not yet implemented

Initial Test Files Created 2025/04/13

  1. test_citation.py: Tests the citation handler functionality, including document creation from search results, source formatting, and citation analysis.

  2. test_search_system.py: Tests the advanced search system, including strategy selection, callback handling, and search execution.

  3. test_report_generator.py: Tests report generation, including structure determination, content research, and final formatting.

  4. test_config.py: Tests configuration loading, environment variable overrides, and default settings creation.

  5. test_wikipedia.py: Tests the Wikipedia search engine, including API interaction, error handling, and result processing.

  6. conftest.py: Provides common fixtures used across different test files, including mock LLMs, search engines, and database connections.

Running the Test Suite

Prerequisites

Before running the tests, ensure you have the required test dependencies:

pip install pytest pytest-cov

Using the Test Runner

The easiest way to run tests is to use the provided run_tests.py script:

The included run_tests.py script provides an easy way to run the tests with coverage reporting:

# Run all tests with coverage reporting
python run_tests.py

# Generate HTML coverage report
python run_tests.py --html

# Run specific test file or directory
python run_tests.py --path tests/unit/test_citation.py

Using pytest Directly

You can also run the tests using pytest directly:

# Run all tests
pytest tests/

# Run with coverage
pytest --cov=local_deep_research tests/

# Run specific test file
pytest tests/unit/test_citation.py

# Run with verbose output
pytest -v tests/

Adding New Tests

When adding new tests:

  1. Follow the existing structure and naming conventions
  2. Place tests in the appropriate directory:
    • unit/ for isolated component tests
    • integration/ for tests that involve multiple components
    • search_engines/ for search engine specific tests
  3. Use fixtures from conftest.py when possible
  4. Follow the naming convention: test_*.py for test files and test_* for test functions
  5. Make sure to add appropriate mocks for external dependencies
  6. Include docstrings explaining what each test verifies

Test Coverage

To generate a coverage report:

pytest --cov=local_deep_research [--cov-report=html] tests/

After generating an HTML report, open htmlcov/index.html in your browser to view the detailed coverage information.

Continuous Integration

It is important that:

  • Tests run on each pull request
  • Coverage reports are generated
  • Test failures block merges to main branches

For full release readiness:

  1. GitHub Actions Workflow: Set up automated testing on push/PR
  2. Coverage Requirements: Enforce minimum coverage percentages (suggest 80%)
  3. Branch Protection: Require passing tests before merging
  4. Dependency Checking: Add automated vulnerability scanning

Troubleshooting

If you encounter issues with the test suite:

  1. Ensure all dependencies are installed
  2. Check that you're running from the project root directory
  3. Verify that test fixtures are properly set up
  4. For tests requiring environment variables, make sure they're properly defined

Mocking Strategy

Many tests rely on mocked dependencies to avoid external API calls or complex setups. Key points:

  1. External API calls are always mocked
  2. Database connections use in-memory SQLite or temporary files
  3. File operations use temporary directories
  4. LLM interactions are mocked with predefined responses

Adding to Test Coverage

Priority areas for additional test coverage:

  1. Integration tests for web interface
  2. Additional search engine tests
  3. Performance tests for long-running operations
  4. Edge case handling in report generation
  5. Error recovery scenarios

ADD'L:

This should be a solid foundation for ensuring the reliability of Local Deep Research. While there are still gaps to address, the core functionality is covered and this provides patterns for extending coverage to other areas.

I recommend continuing to expand the test suite, particularly for database operations, web interfaces, and additional search engines, to achieve comprehensive coverage before final release.The two files currently in the tests/ directory are more like utility scripts than software tests.

I propose LDR make use of the all-but-standard (technically unittest is in the python standard lib) pytest for a structured testing framework.

As I said in the chat, LDR has 0% "code coverage".

Since there is already much code and no test harness, I propose a strategy (this is off the cuff; discord on my phone and laptop typing this out right now so you'll all have something to read when you get online.)

Areas to test, loosely in an order

core

  • searching implementation
  • report generation
  • citations
  • LLM integrations

conf

  • loading
  • validation
  • env
  • defaults

search

  • each
  • search
  • engine
  • needs
  • tests
  • oh, and tests for filtering too

web interface

  • API endpoints
  • frontend functionality

DB

  • CRUD
  • and we'll use the migration test that exists now if that's what it does. I only glanced.

Test implementation

Would look something like this:

tests/
├── conftest.py              # Shared pytest fixtures
├── unit/                    # Unit tests
│   ├── test_citation.py     # Test citation handling
│   ├── test_config.py       # Test configuration loading
│   ├── test_llm.py          # Test LLM integration
│   ├── test_report.py       # Test report generation
│   └── test_search.py       # Test search functionality
├── integration/             # Integration tests
│   ├── test_db_ops.py       # Test database operations
...

Example

This is what a pytest test looks like:

import pytest
from unittest.mock import MagicMock, patch
import sys
from pathlib import Path

# Handle import paths for testing
sys.path.append(str(Path(__file__).parent.parent.parent))
from local_deep_research.citation_handler import CitationHandler
from langchain_core.documents import Document


@pytest.fixture
def citation_handler():
    """Create a citation handler with a mocked LLM for testing."""
    mock_llm = MagicMock()
    mock_llm.invoke.return_value = MagicMock(content="Mocked analysis with citation [1]")
    return CitationHandler(mock_llm)


@pytest.fixture
def sample_search_results():
    """Sample search results for testing."""
    return [
        {
            "title": "Test Result 1",
            "link": "https://example.com/1",
            "snippet": "This is the first test result snippet."
        },
        {
            "title": "Test Result 2",
            "link": "https://example.com/2",
            "full_content": "This is the full content of the second test result."
        }
    ]


def test_create_documents_empty(citation_handler):
    """Test document creation with empty search results."""
    documents = citation_handler._create_documents([])
    assert len(documents) == 0


def test_create_documents_string(citation_handler):
    """Test document creation with string input (error case)."""
    documents = citation_handler._create_documents("not a list")
    assert len(documents) == 0


def test_create_documents(citation_handler, sample_search_results):
    """Test document creation with valid search results."""
    documents = citation_handler._create_documents(sample_search_results)

    # Check if the correct number of documents was created
    assert len(documents) == 2

    # Check first document
    assert documents[0].metadata["title"] == "Test Result 1"
    assert documents[0].metadata["source"] == "https://example.com/1"
    assert documents[0].metadata["index"] == 1
    assert documents[0].page_content == "This is the first test result snippet."

    # Check second document - should use full_content instead of snippet
...

There would be files like this for all the areas I mentioned and others.

Else?