Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

2.6 KB
Raw Blame

CI/CD Integration for Enhanced Test Framework

Overview

The enhanced test framework from PR #361 has been integrated into the CI/CD pipeline with the following components:

1. Main Test Pipeline Integration

Updated tests/run_all_tests.py

The main test runner now includes our new tests in the unit test suite:

  • tests/test_wikipedia_url_security.py - Security validation tests
  • tests/test_search_engines_enhanced.py - Enhanced search engine tests
  • tests/test_utils.py - Test utilities validation

These tests run in all profiles:

  • fast - Quick feedback (< 30s)
  • standard - Development testing (< 5min)
  • full - Comprehensive testing (< 15min)
  • unit-only - Unit tests only (< 10s)

2. CI Workflow Integration

.github/workflows/pytest-tests.yml

The main pytest workflow runs all tests including the enhanced test framework:

  • Runs on all PRs and pushes to main/dev
  • Includes URL validation security tests
  • Includes enhanced search engine tests
  • Uploads coverage reports to GitHub Pages

3. Test Organization

New Test Structure

tests/
├── fixtures/              # Reusable test fixtures
│   ├── search_engine_mocks.py
│   └── README.md
├── mock_fixtures.py       # Mock data functions
├── mock_modules.py        # Dynamic module mocking
├── test_utils.py          # Common test utilities
├── test_search_engines_enhanced.py
└── test_wikipedia_url_security.py

4. CI Environment Variables

The tests use these environment variables in CI:

  • USE_FALLBACK_LLM=true - Uses mock LLM to avoid API calls
  • CI=true - Indicates CI environment for headless testing

5. Benefits

  1. Security: URL validation tests run on every PR
  2. Fast Feedback: Enhanced tests run in parallel workflow
  3. Coverage: New tests included in coverage reports
  4. Reliability: Mock fixtures ensure consistent test results
  5. Modularity: Fixtures can be reused across test suites

6. Running Tests Locally

To run the new tests locally:

# Run all new tests
pdm run pytest tests/test_wikipedia_url_security.py tests/test_search_engines_enhanced.py -v

# Run with coverage
pdm run pytest tests/test_wikipedia_url_security.py tests/test_search_engines_enhanced.py --cov=src --cov-report=html

# Run via the test runner
cd tests && pdm run python run_all_tests.py unit-only

7. Future Improvements

  1. Add more search engine mock fixtures
  2. Create integration tests for all search engines
  3. Add performance benchmarks
  4. Expand security validation to other engines
  5. Add fuzzing tests for URL validation