~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Local Deep Research - Test Suite
This directory contains the test suite for the Local Deep Research project. The tests are organized into unit tests, integration tests, and search engine-specific tests.
Tests Structure
tests/
├── conftest.py # Common test fixtures
├── run_tests.py # Test runner with coverage reporting
├── README.md # Test documentation
├── unit/ # Unit tests
│ ├── test_citation.py # Tests for citation handling
│ ├── test_config.py # Tests for configuration management
│ ├── test_report.py # Tests for report generation
│ └── test_search_system.py # Tests for search system
├── integration/ # Integration tests
│ └── test_db_ops.py # Database operation tests (placeholder)
└── search_engines/ # Search engine specific tests
└── test_wikipedia.py # Wikipedia search engine tests
Current Coverage
| Component | Coverage | Notes |
|---|---|---|
| Citation Handler | High | Comprehensive tests for all methods |
| Search System | High | Tests strategy selection and core functionality |
| Report Generator | High | Tests structure determination and report formatting |
| Configuration | High | Tests loading, env vars, and defaults |
| Wikipedia Search | High | Tests API interaction and result processing |
| Database Operations | Low | Database tests outlined but need implementation |
| Other Search Engines | Low | Only Wikipedia has tests so far |
| Web Interface | None | Web interface tests not yet implemented |
Initial Test Files Created 2025/04/13
test_citation.py: Tests the citation handler functionality, including document creation from search results, source formatting, and citation analysis.
test_search_system.py: Tests the advanced search system, including strategy selection, callback handling, and search execution.
test_report_generator.py: Tests report generation, including structure determination, content research, and final formatting.
test_config.py: Tests configuration loading, environment variable overrides, and default settings creation.
test_wikipedia.py: Tests the Wikipedia search engine, including API interaction, error handling, and result processing.
conftest.py: Provides common fixtures used across different test files, including mock LLMs, search engines, and database connections.
Running the Test Suite
Prerequisites
Before running the tests, ensure you have the required test dependencies:
pip install pytest pytest-cov
Using the Test Runner
The easiest way to run tests is to use the provided run_tests.py script:
The included run_tests.py script provides an easy way to run the tests with coverage reporting:
# Run all tests with coverage reporting
python run_tests.py
# Generate HTML coverage report
python run_tests.py --html
# Run specific test file or directory
python run_tests.py --path tests/unit/test_citation.py
Using pytest Directly
You can also run the tests using pytest directly:
# Run all tests
pytest tests/
# Run with coverage
pytest --cov=local_deep_research tests/
# Run specific test file
pytest tests/unit/test_citation.py
# Run with verbose output
pytest -v tests/
Adding New Tests
When adding new tests:
- Follow the existing structure and naming conventions
- Place tests in the appropriate directory:
unit/for isolated component testsintegration/for tests that involve multiple componentssearch_engines/for search engine specific tests
- Use fixtures from
conftest.pywhen possible - Follow the naming convention:
test_*.pyfor test files andtest_*for test functions - Make sure to add appropriate mocks for external dependencies
- Include docstrings explaining what each test verifies
Test Coverage
To generate a coverage report:
pytest --cov=local_deep_research [--cov-report=html] tests/
After generating an HTML report, open htmlcov/index.html in your browser to view the detailed coverage information.
Continuous Integration
It is important that:
- Tests run on each pull request
- Coverage reports are generated
- Test failures block merges to main branches
For full release readiness:
- GitHub Actions Workflow: Set up automated testing on push/PR
- Coverage Requirements: Enforce minimum coverage percentages (suggest 80%)
- Branch Protection: Require passing tests before merging
- Dependency Checking: Add automated vulnerability scanning
Troubleshooting
If you encounter issues with the test suite:
- Ensure all dependencies are installed
- Check that you're running from the project root directory
- Verify that test fixtures are properly set up
- For tests requiring environment variables, make sure they're properly defined
Mocking Strategy
Many tests rely on mocked dependencies to avoid external API calls or complex setups. Key points:
- External API calls are always mocked
- Database connections use in-memory SQLite or temporary files
- File operations use temporary directories
- LLM interactions are mocked with predefined responses
Adding to Test Coverage
Priority areas for additional test coverage:
- Integration tests for web interface
- Additional search engine tests
- Performance tests for long-running operations
- Edge case handling in report generation
- Error recovery scenarios
ADD'L:
This should be a solid foundation for ensuring the reliability of Local Deep Research. While there are still gaps to address, the core functionality is covered and this provides patterns for extending coverage to other areas.
I recommend continuing to expand the test suite, particularly for database operations, web interfaces, and additional search engines, to achieve comprehensive coverage before final release.The two files currently in the tests/ directory are more like utility scripts than software tests.
I propose LDR make use of the all-but-standard (technically unittest is in the python standard lib) pytest for a structured testing framework.
As I said in the chat, LDR has 0% "code coverage".
Since there is already much code and no test harness, I propose a strategy (this is off the cuff; discord on my phone and laptop typing this out right now so you'll all have something to read when you get online.)
Areas to test, loosely in an order
core
- searching implementation
- report generation
- citations
- LLM integrations
conf
- loading
- validation
- env
- defaults
search
- each
- search
- engine
- needs
- tests
- oh, and tests for filtering too
web interface
- API endpoints
- frontend functionality
DB
- CRUD
- and we'll use the migration test that exists now if that's what it does. I only glanced.
Test implementation
Would look something like this:
tests/
├── conftest.py # Shared pytest fixtures
├── unit/ # Unit tests
│ ├── test_citation.py # Test citation handling
│ ├── test_config.py # Test configuration loading
│ ├── test_llm.py # Test LLM integration
│ ├── test_report.py # Test report generation
│ └── test_search.py # Test search functionality
├── integration/ # Integration tests
│ ├── test_db_ops.py # Test database operations
...
Example
This is what a pytest test looks like:
import pytest
from unittest.mock import MagicMock, patch
import sys
from pathlib import Path
# Handle import paths for testing
sys.path.append(str(Path(__file__).parent.parent.parent))
from local_deep_research.citation_handler import CitationHandler
from langchain_core.documents import Document
@pytest.fixture
def citation_handler():
"""Create a citation handler with a mocked LLM for testing."""
mock_llm = MagicMock()
mock_llm.invoke.return_value = MagicMock(content="Mocked analysis with citation [1]")
return CitationHandler(mock_llm)
@pytest.fixture
def sample_search_results():
"""Sample search results for testing."""
return [
{
"title": "Test Result 1",
"link": "https://example.com/1",
"snippet": "This is the first test result snippet."
},
{
"title": "Test Result 2",
"link": "https://example.com/2",
"full_content": "This is the full content of the second test result."
}
]
def test_create_documents_empty(citation_handler):
"""Test document creation with empty search results."""
documents = citation_handler._create_documents([])
assert len(documents) == 0
def test_create_documents_string(citation_handler):
"""Test document creation with string input (error case)."""
documents = citation_handler._create_documents("not a list")
assert len(documents) == 0
def test_create_documents(citation_handler, sample_search_results):
"""Test document creation with valid search results."""
documents = citation_handler._create_documents(sample_search_results)
# Check if the correct number of documents was created
assert len(documents) == 2
# Check first document
assert documents[0].metadata["title"] == "Test Result 1"
assert documents[0].metadata["source"] == "https://example.com/1"
assert documents[0].metadata["index"] == 1
assert documents[0].page_content == "This is the first test result snippet."
# Check second document - should use full_content instead of snippet
...
There would be files like this for all the areas I mentioned and others.