Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

12.0 KB
Raw Blame

UI Tests

Browser-based testing using Puppeteer to verify that web pages load correctly and JavaScript executes without errors.

Setup

Install Puppeteer (if not already installed):

npm install puppeteer

Running Tests

Local environment notes

The auth endpoints rate-limit at 5 logins per 15 min per IP, and registrations are rate limited per IP as well (see REGISTRATION_RATE_LIMIT in web/dependencies/rate_limit.py). The full suite re-authenticates dozens of times, so a local run against a default-configured server gets blocked after the first few tests and shows a wave of Login failed failures that are not real test failures.

Two options for local runs:

  1. Disable rate limiting (simplest):

    LDR_DISABLE_RATE_LIMITING=true python -m local_deep_research.web.app
    
  2. Use the CI test-user pattern: pre-create test_admin once, then run the suite with CI=true so auth_helper.js logs in with that user instead of registering fresh users each time.

    python scripts/ci/init_test_database.py    # one-time setup
    CI=true node tests/ui_tests/run_all_tests.js --shard=<shard>
    

    Note: CI=true requires --shard=<name> (the runner fails fast otherwise to catch matrix misconfiguration). Iterate shards manually for a full local run.

Either approach is fine; option 1 is closer to how CI runs the suite (CI launches the server with rate limiting off).

Run All UI Tests (Recommended)

cd /path/to/local-deep-research
node tests/ui_tests/run_all_tests.js

Run a Single Shard

Tests are partitioned into shards so CI can run them in parallel matrix cells. Each entry in run_all_tests.js carries a shard: property; the authoritative shard list is VALID_SHARDS at the top of that file (kept in sync with strategy.matrix.shard in .github/workflows/docker-tests.yml). Local dev rarely needs this, but you can reproduce a CI cell with:

node tests/ui_tests/run_all_tests.js --shard=settings-pages
node tests/ui_tests/run_all_tests.js --shard=library
node tests/ui_tests/run_all_tests.js --shard=accessibility

Valid shards (source of truth: VALID_SHARDS in run_all_tests.js):

Shard Scope
auth-login Login/auth flow
auth-register Registration (isolated — heavy SQLCipher DB creation)
auth-pages Page browsing, navigation, comprehensive auth
research-workflow Core research lifecycle
research-form Research form interactions + results
research-metrics Metrics charts, dashboard, progress
settings-core Settings page, errors, save, interactions
settings-pages Settings tabs, star reviews, journal quality
library Collections, documents, download manager
history-news History page, news subscriptions
mobile Mobile navigation, interactions, UI
api-crud API endpoints, CRUD operations, rate limiting
error-benchmark Error handling/recovery, benchmark, context overflow
accessibility Keyboard navigation & ARIA
chat-core Chat ARIA, CSRF, suggestion chips
chat-lifecycle Chat sessions, export, edit title

Passing an unknown shard, or leaving --shard off while CI=true is set, fails fast with an error — this catches matrix misconfiguration before a cell wastes compute running the wrong slice.

Run All Page Tests

cd /path/to/local-deep-research
node tests/ui_tests/test_pages_browser.js

Run Individual Component Tests

# Test metrics charts functionality
node tests/ui_tests/test_metrics_charts.js

# Test research results page loading
node tests/ui_tests/test_research_results.js

# Test settings page functionality
node tests/ui_tests/test_settings_page.js

# Test settings error detection
node tests/ui_tests/test_settings_errors.js

# Test settings save functionality
node tests/ui_tests/test_settings_save.js

# Test star reviews analytics page
node tests/ui_tests/test_star_reviews.js

Prerequisites

  • Web server must be running on http://127.0.0.1:5000
  • Start the server with: python -m local_deep_research.web.app

What These Tests Check

All Pages

  • Page loads without timeout
  • Has title and body elements
  • No critical JavaScript errors
  • Basic DOM structure is present

Metrics Dashboard (/metrics/)

  • Loading, content, and error elements exist
  • Metrics data loads (token counts, research counts)
  • JavaScript executes without syntax errors
  • Takes screenshot for visual debugging

Research Page (/)

  • Query input field exists and is enabled
  • Submit button exists and is enabled
  • Mode selection dropdown exists

History Page (/history/)

  • History container elements exist
  • Search functionality is present
  • Content is visible

Settings Page (/settings/)

  • Forms and input elements exist
  • Per-setting auto-save controls are available
  • Configuration interface loads

Individual Component Tests

Metrics Charts Test (test_metrics_charts.js)

  • Verifies both token consumption and search activity charts render correctly
  • Checks for Canvas elements (Chart.js charts)
  • Scrolls through the page to test dynamic loading
  • Takes screenshot for visual verification
  • Validates chart data loading from API endpoints

Research Results Test (test_research_results.js)

  • Tests loading of specific research report (research ID 67)
  • Monitors network requests to identify API failures
  • Checks for error messages vs. actual content
  • Validates research report file loading functionality

Settings Page Test (test_settings_page.js)

  • Comprehensive test of settings page loading
  • Monitors all API calls (available models, search engines, settings)
  • Counts setting form elements to verify proper loading
  • Validates 300+ setting elements are present on the page
  • Tests settings synchronization and form functionality

Settings Error Detection Test (test_settings_errors.js)

  • Tests for error messages when changing setting values
  • Monitors browser console for JavaScript errors
  • Detects network errors (4xx/5xx HTTP responses)
  • Searches for error DOM elements (.error, .alert-danger, etc.)
  • Simulates user interaction with dropdowns and input fields
  • Validates error handling and user feedback mechanisms

Settings Save Test (test_settings_save.js)

  • Tests the per-setting auto-save workflow
  • Monitors network requests to /research/settings/save_all_settings
  • Confirms the bulk-submit button is absent
  • Validates checkbox-triggered auto-save
  • Checks for proper API response handling (success/error states)
  • Monitors console logging during save operations
  • Verifies success/error message display to users
  • Tests CSRF token handling and form validation

Star Reviews Test (test_star_reviews.js)

  • Tests the star reviews analytics page and navigation
  • Monitors API endpoint /metrics/api/star-reviews functionality
  • Validates Chart.js rendering for bar charts and line charts
  • Tests period selector and data filtering
  • Checks overall statistics display and rating distribution
  • Verifies recent ratings list population
  • Tests navigation between metrics dashboard and star reviews page

Testing Strategy

The UI test suite follows a layered approach:

1. Page Load Tests (test_pages_browser.js)

  • Purpose: Verify basic page functionality works across all routes
  • Coverage: Home, metrics, history, settings pages
  • Validates: Page loads, DOM structure, basic JavaScript execution

2. Component-Specific Tests

  • Purpose: Deep dive into specific UI components and workflows
  • Coverage: Charts, forms, API integrations, file loading
  • Validates: Complex interactions, data flow, error handling

3. Error Detection Tests

  • Purpose: Catch JavaScript errors and API failures that could break user experience
  • Coverage: Console errors, network failures, validation errors
  • Validates: Proper error handling and user feedback

4. Integration Tests

  • Purpose: Test complete user workflows end-to-end
  • Coverage: Settings save, research report loading, chart rendering
  • Validates: Data persistence, file operations, API coordination

What Problems These Tests Solve

  1. Regression Detection: Catch when code changes break existing functionality
  2. JavaScript Error Monitoring: Identify runtime errors that users would encounter
  3. API Integration Validation: Ensure frontend properly communicates with backend
  4. File Path Resolution: Verify reports and assets load correctly regardless of server setup
  5. Cross-Browser Compatibility: Puppeteer testing ensures consistent behavior
  6. Performance Monitoring: Detect slow loading or hanging operations

Output

  • Console logs show detailed test progress
  • Screenshots saved to tests/ui_tests/screenshots/
  • Summary report shows pass/fail status for each page

Why These Tests Are Useful

  1. Catch JavaScript Errors: Unlike unit tests, these catch runtime JS errors in the browser
  2. End-to-End Validation: Verify the complete request → response → render cycle
  3. Regression Prevention: Detect when changes break the UI
  4. Cross-Page Coverage: Test all major application pages consistently

Example Output

🚀 Starting browser test session...

📄 Testing Metrics Dashboard: http://127.0.0.1:5000/metrics/
📝 [LOG] === METRICS SCRIPT STARTED ===
📝 [LOG] === STARTING LOADMETRICS ===
📝 [LOG] Basic data success, setting metricsData
✅ Metrics Dashboard loaded successfully
🔍 Basic checks for Metrics Dashboard:
   Has title: true
   Has body: true
   Body visible: true
📊 Metrics page checks:
   Content visible: true
   Total tokens: 41,801
   Total researches: 11

==================================================
📋 TEST SUMMARY
==================================================
✅ PASS Home/Research Page
✅ PASS Metrics Dashboard
✅ PASS History Page
✅ PASS Settings Page

Total: 4 tests
Passed: 4
Failed: 0
🎉 All tests passed!

Shared helpers in test_lib/

Common patterns live in tests/ui_tests/test_lib/ and are re-exported from ./test_lib. Use these rather than inlining the equivalent logic — duplicated logic is how the same bug class hides in multiple files (#4069 / #4127).

  • findActionButton(page, { selectors, keywords, click }) — locate a button by text content with word-boundary matching. Defaults to selectors='button, a.btn, .btn' and keywords=['create', 'new', 'add']. Returns { found, text }.
    • Why this exists: text.includes('new') matches the substring "new" inside "News" (e.g. an unrelated <a>Back to News Feed</a> button). The helper wraps \b(?:create|new|add)\b so "Create Subscription" / "New Folder" / "Add Item" still match but "News Feed" doesn't.
    • Example: const { found } = await findActionButton(page, { click: true });
  • navigateTo(page, url, options) — robust navigation with retries and CI-tuned timeouts.
  • setupTest({ authenticate: true }) / teardownTest(ctx) — launches a browser, optionally authenticates via auth_helper.js (uses the CI test user when CI=true, registers a fresh user otherwise), and returns a context with page, browser, and config.
  • TestResults(suiteName) — collector with .run(group, name, fn), .skip(...), .print(), .save() (writes JSON + JUnit XML to test-results/).

Prefer asserting browser-level contracts (HTML5 :invalid, navigation target, computed style) over heuristic scraping for .error / .invalid-feedback classes — heuristic selectors mask regressions when app markup drifts.

Adding New Page Tests

To test a new page, add an entry to the testCases array in test_pages_browser.js:

{ path: '/new-page/', name: 'New Page', tests: newPageTests }

Then create a custom test function:

const newPageTests = async (page) => {
    console.log('🧪 Running new page tests...');

    const checks = await page.evaluate(() => {
        // Check for page-specific elements
        return {
            hasSpecificElement: !!document.getElementById('specific-element')
        };
    });

    console.log(`Specific element present: ${checks.hasSpecificElement}`);
};