Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

21.6 KB
Raw Blame
#!/usr/bin/env node
/**
 * Settings Round-Trip UI Tests (Flask -> FastAPI migration)
 *
 * Settings is the largest surface in the app (473+ rendered items across
 * 7 tabs on this build), and the read/write path underneath it was rewritten
 * wholesale in the migration: GET /settings/api/{key} and POST
 * /settings/save_all_settings (routers/settings.py::api_get_db_setting /
 * save_all_settings) replaced the old Flask blueprint's equivalents. Despite
 * that, no browser test in this suite proves a user can actually change a
 * setting through the real UI and have it stick — the existing CI settings
 * tests (test_settings_interactions_ci.js, test_settings_pages_ci.js, etc.)
 * check that controls render, that tabs/search exist, and that save errors
 * surface a toast, but none of them drive a real control, wait for the
 * app's own autosave, reload the page, and assert the new value survives.
 * The only file that ever did that (test_settings_persistence.js) manipulates
 * DOM values directly via page.evaluate() rather than real click/select
 * interactions, and — because it isn't named `*_ci.js` — was never wired
 * into run_all_tests.js, so it has never actually run in CI.
 *
 * This file drives the real autosave path settings.js uses for every
 * control on the page (handleInputChange -> submitSettingsData -> POST
 * /settings/save_all_settings, 800ms debounced — see scheduleSave() in
 * components/settings.js) via genuine page.click()/page.type() interaction,
 * and proves: the change survives a full page reload (Test 1), the value is
 * actually in the database and not just the DOM/JS cache (Test 2, read back
 * through the settings API from page context), a second user's account
 * never sees it (Test 3, browser-level analogue of the cross-user settings
 * isolation this project has separately investigated at the API/python
 * level), and the search box's filter contract holds (Test 4).
 *
 * Setting chosen: app.enable_notifications ("Enable Notifications" — browser
 * push alerts for research completion/errors). It is a plain checkbox
 * (editable: true, visible: true, default true — see defaults/
 * default_settings.json), so its save path has no enum/options validation
 * to worry about, and toggling it can never disable auth, CSRF, rate
 * limiting, or move the app's data directory — the exact hazards this task
 * was told to avoid. It round-trips cleanly (verified manually against the
 * running dev server before writing this file).
 *
 * NOT used: app.theme, despite being the more obviously "cosmetic" choice.
 * While surveying candidates, changing the theme select and later trying to
 * restore it to its own shipped default ("dark") turned up a genuine,
 * reproducible defect unrelated to this migration: default_settings.json
 * ships "app.theme": {"value": "dark", ...}, but manager.py dynamically
 * replaces that field's `options` with theme_registry.get_settings_options()
 * (a scan of the CSS-file-backed theme registry, which has no "dark.css" —
 * "dark" is just the app's un-themed baseline, not a registered theme).
 * validate_setting() in routers/settings.py then rejects "dark" outright:
 * POST /settings/save_all_settings {"app.theme": "dark"} -> 400 "Value must
 * be one of: ayu-mirage, catppuccin, ... " (dark/system absent from the
 * list). So the value every brand-new account actually has for app.theme
 * cannot be round-tripped through the save endpoint at all — confirmed via
 * both a real <select> interaction (Puppeteer's page.select() finds no
 * matching <option> once the value has ever moved off "dark", so the
 * browser's native fallback silently selects and saves index 0,
 * "Ayu Mirage", instead — a silent wrong-value save with a "UI Theme
 * updated" success toast) and a direct POST replaying the app's exact
 * request shape (400, see above). Root cause: DYNAMIC_SETTINGS in
 * routers/settings.py (= ["llm.provider", "llm.model", "search.tool"]) never
 * had "app.theme" added to it even though manager.py has dynamically injected
 * its options the same way since search.search_strategy and the LLM/search
 * dynamic settings were exempted. This is a real bug worth fixing, but it is
 * orthogonal to the Flask->FastAPI rewrite and — since a correct fix would
 * make "dark" round-trip successfully — asserting on today's broken behavior
 * here would make this file fail once someone fixes it. Reported in this
 * comment plus the task's final summary instead of pinned as a test.
 *
 * Run: CI=true node test_settings_persistence_ci.js
 */

const crypto = require('crypto');
const puppeteer = require('puppeteer');
const AuthHelper = require('./auth_helper');
const { getPuppeteerLaunchOptions } = require('./puppeteer_config');
const { expandSettingsSectionFor } = require('./test_lib');
const { capture, captureOnFailure } = require('./screenshot_helper');

const BASE_URL = process.env.BASE_URL || 'http://127.0.0.1:5000';
const isCI = !!process.env.CI;

const TIMEOUTS = {
    navigation: isCI ? 60000 : 30000,
    selector: isCI ? 30000 : 10000,
    save: isCI ? 20000 : 10000,
};

const SETTING_SELECTOR = '#setting-app-enable_notifications';
const SETTING_KEY = 'app.enable_notifications';
const SCREENSHOT_PREFIX = 'settings_persistence';

/** True for a real, attributable console error — not the browser's own favicon probe noise. */
function isRealConsoleError(msg) {
    return msg.type() === 'error' && !msg.text().startsWith('Failed to load resource');
}

/** Wait for the specific save_all_settings POST the app's own JS issues, and return its Response. */
function waitForSaveResponse(page) {
    return page.waitForResponse(
        (r) => r.url().includes('/settings/save_all_settings') && r.request().method() === 'POST',
        { timeout: TIMEOUTS.save }
    );
}

async function run() {
    console.log(`Running settings persistence round-trip tests (CI mode: ${isCI})`);

    const browser = await puppeteer.launch(getPuppeteerLaunchOptions());
    const page = await browser.newPage();
    await page.setViewport({ width: 1280, height: 800 });
    if (isCI) {
        page.setDefaultTimeout(60000);
        page.setDefaultNavigationTimeout(60000);
    }

    page.on('pageerror', (e) => console.log('PAGE ERROR:', e.message));
    page.on('console', (m) => {
        if (isRealConsoleError(m)) console.log('BROWSER ERROR:', m.text());
    });

    let passed = 0;
    let failed = 0;

    // Shared across tests, and read by the cleanup block in `finally` below.
    let originalValue = null;
    let changedValue = null;
    let changeWasApplied = false;
    // AuthHelper.getPage() can return a DIFFERENT page object than `page`
    // above if it recovered from a detached frame during auth (see
    // auth_helper.js). The `finally` cleanup block below must act on
    // whichever page actually did the mutating, so track it here in the
    // outer scope rather than re-reading the (possibly stale) `page`.
    let workingPage = page;

    try {
        const auth = new AuthHelper(page, BASE_URL);
        await auth.ensureAuthenticatedWithTimeout();
        workingPage = auth.getPage();

        await workingPage.goto(`${BASE_URL}/settings/`, {
            waitUntil: 'domcontentloaded',
            timeout: TIMEOUTS.navigation,
        });
        await workingPage.waitForSelector(SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
        // Settings sections start collapsed on every viewport, so the
        // checkbox renders inside a `display: none` body. Open its section
        // before any click — page.click() needs a real box to hit.
        await expandSettingsSectionFor(workingPage, SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
        await capture(workingPage, SCREENSHOT_PREFIX, 'before_toggle', { fullPage: true });

        // -----------------------------------------------------------------
        // Test 1: a setting changed via a real UI click persists across a
        // full page reload. The click fires settings.js's 'change' handler,
        // which schedules an 800ms-debounced autosave — waitForSaveResponse
        // is the explicit, non-sleep wait for that save to actually land.
        // -----------------------------------------------------------------
        console.log(`Test 1: toggling ${SETTING_KEY} via a real click, waiting for the app's own save, then reloading`);
        try {
            originalValue = await workingPage.$eval(SETTING_SELECTOR, (el) => el.checked);

            const [saveResp] = await Promise.all([
                waitForSaveResponse(workingPage),
                workingPage.click(SETTING_SELECTOR),
            ]);
            changeWasApplied = true;

            const saveBody = await saveResp.json().catch(() => null);
            if (saveResp.status() !== 200 || !saveBody || saveBody.status !== 'success') {
                throw new Error(`Save request did not succeed: status=${saveResp.status()} body=${JSON.stringify(saveBody)}`);
            }
            if (!Array.isArray(saveBody.updated) || !saveBody.updated.includes(SETTING_KEY)) {
                throw new Error(`Save response did not report ${SETTING_KEY} as updated: ${JSON.stringify(saveBody.updated)}`);
            }

            await workingPage.reload({ waitUntil: 'domcontentloaded', timeout: TIMEOUTS.navigation });
            await workingPage.waitForSelector(SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
            await expandSettingsSectionFor(workingPage, SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
            const afterReload = await workingPage.$eval(SETTING_SELECTOR, (el) => el.checked);

            if (afterReload === originalValue) {
                throw new Error(
                    `Checkbox shows the original value (${originalValue}) after reload — the change did not persist`
                );
            }
            changedValue = afterReload;
            console.log(`PASSED (original=${originalValue} -> changed=${changedValue}, survived reload)`);
            passed++;
            await capture(workingPage, SCREENSHOT_PREFIX, 'after_toggle', { fullPage: true });
        } catch (e) {
            console.log(`FAILED: ${e.message}`);
            await captureOnFailure(workingPage, SCREENSHOT_PREFIX, 'persist_across_reload', false);
            failed++;
        }

        // -----------------------------------------------------------------
        // Test 2: the change is actually in the database, not just the DOM
        // or settings.js's in-memory allSettings cache. Read it back through
        // the real settings API from page context.
        // -----------------------------------------------------------------
        console.log(`Test 2: reading ${SETTING_KEY} back through GET /settings/api/${SETTING_KEY} confirms server-side persistence`);
        try {
            if (changedValue === null) {
                throw new Error('Skipped: Test 1 did not establish a changed value to verify');
            }
            const apiResult = await workingPage.evaluate(async (key) => {
                const r = await fetch(`/settings/api/${key}`, { credentials: 'same-origin' });
                return { status: r.status, json: await r.json().catch(() => null) };
            }, SETTING_KEY);

            if (apiResult.status !== 200) {
                throw new Error(`Expected HTTP 200 from the settings API, got ${apiResult.status}`);
            }
            if (apiResult.json?.value !== changedValue) {
                throw new Error(
                    `Settings API value (${apiResult.json?.value}) does not match the UI-persisted value (${changedValue})`
                );
            }
            console.log(`PASSED (API value=${apiResult.json.value})`);
            passed++;
        } catch (e) {
            console.log(`FAILED: ${e.message}`);
            failed++;
        }

        // -----------------------------------------------------------------
        // Test 3: per-user isolation at the UI level. A second user,
        // registered in a fresh incognito browser context (no shared
        // cookies with user A), must NOT see user A's changed value.
        // This is the browser-level analogue of the cross-user settings
        // isolation this project has investigated at the API/python level —
        // worth pinning here too since it exercises the real cookie/session
        // boundary a browser enforces, not just two DB-session fixtures.
        // -----------------------------------------------------------------
        console.log("Test 3: a second user in a fresh incognito context does not see user A's changed setting");
        try {
            if (changedValue === null) {
                throw new Error("Skipped: no changed value from Test 1 to check isolation against");
            }
            const incognitoContext = await browser.createBrowserContext();
            try {
                const pageB = await incognitoContext.newPage();
                if (isCI) {
                    pageB.setDefaultTimeout(60000);
                    pageB.setDefaultNavigationTimeout(60000);
                }
                const authB = new AuthHelper(pageB, BASE_URL);
                // Name the account explicitly. Calling this with no username
                // is what broke this test: in CI the helper prefers the shared
                // CI_TEST_USER (test_admin), which is the account user A is
                // already using. A fresh incognito context gives a new cookie
                // jar, not a new user — so the test compared user A against
                // user A and reported the match as a leak. Passing a username
                // now disables that shortcut (see auth_helper.js), so the
                // account named here is the account that gets used.
                // The cleanup block at the bottom of this file already
                // documents the shared-CI_TEST_USER behaviour; this is that
                // same fact applied where it decides whether the assertion
                // means anything.
                const userBName =
                    `isolation_b_${Date.now()}_` +
                    `${crypto.randomBytes(4).toString('hex')}`;
                await authB.ensureAuthenticatedWithTimeout(
                    userBName,
                    'T3st!Secure#2024$LDR' // pragma: allowlist secret
                );
                const workingPageB = authB.getPage();

                await workingPageB.goto(`${BASE_URL}/settings/`, {
                    waitUntil: 'domcontentloaded',
                    timeout: TIMEOUTS.navigation,
                });
                await workingPageB.waitForSelector(SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
                const userBValue = await workingPageB.$eval(SETTING_SELECTOR, (el) => el.checked);

                if (userBValue === changedValue) {
                    throw new Error(
                        `GENUINE DEFECT: user B (${userBName}, a freshly registered account) reads ` +
                        `${SETTING_KEY} = ${userBValue}, matching user A's explicitly-changed value ` +
                        `(${changedValue}) instead of the account default (${originalValue}). ` +
                        `A setting is leaking across user accounts.`
                    );
                }
                console.log(`PASSED (user A=${changedValue}, user B=${userBValue} — independent per-user state)`);
                passed++;
            } finally {
                await incognitoContext.close();
            }
        } catch (e) {
            console.log(`FAILED: ${e.message}`);
            failed++;
        }

        // -----------------------------------------------------------------
        // Test 4: the settings search box narrows the visible list and the
        // narrowed list still contains a known match. Resilient by design:
        // asserts a relationship (fewer visible, expected key present, full
        // count restored on clear), never a hardcoded total.
        // -----------------------------------------------------------------
        console.log('Test 4: settings search narrows the visible list and keeps the expected match');
        try {
            await workingPage.waitForSelector('#settings-content .ldr-settings-item', { timeout: TIMEOUTS.selector });
            // Defensive: make sure no leftover query text from a prior run/step.
            await workingPage.evaluate(() => {
                const el = document.querySelector('#settings-search');
                if (el) el.value = '';
            });

            const countItems = () => workingPage.$$eval('#settings-content .ldr-settings-item', (els) => els.length);
            const initialCount = await countItems();
            if (initialCount === 0) {
                throw new Error('No settings rendered before filtering');
            }

            await workingPage.type('#settings-search', SETTING_KEY);
            await workingPage.waitForFunction(
                (n) => document.querySelectorAll('#settings-content .ldr-settings-item').length < n,
                { timeout: 5000 },
                initialCount
            );
            const filteredCount = await countItems();
            const hasExpectedMatch = await workingPage.evaluate(
                (key) => !!document.querySelector(`#settings-content .ldr-settings-item[data-key="${key}"]`),
                SETTING_KEY
            );

            if (!(filteredCount < initialCount)) {
                throw new Error(`Filtering did not narrow the list (initial=${initialCount}, filtered=${filteredCount})`);
            }
            if (!hasExpectedMatch) {
                throw new Error(`Filtered list is missing the expected match for "${SETTING_KEY}"`);
            }

            // Clearing restores the full list — proves this is a real filter
            // over the rendered set, not a one-way destructive re-render.
            await workingPage.$eval('#settings-search', (el) => {
                el.value = '';
                el.dispatchEvent(new Event('input', { bubbles: true }));
            });
            await workingPage.waitForFunction(
                (n) => document.querySelectorAll('#settings-content .ldr-settings-item').length === n,
                { timeout: 5000 },
                initialCount
            );
            const restoredCount = await countItems();
            if (restoredCount !== initialCount) {
                throw new Error(`Clearing search did not restore the full list (initial=${initialCount}, restored=${restoredCount})`);
            }

            console.log(`PASSED (${initialCount} -> ${filteredCount} filtered -> ${restoredCount} restored)`);
            passed++;
        } catch (e) {
            console.log(`FAILED: ${e.message}`);
            await captureOnFailure(workingPage, SCREENSHOT_PREFIX, 'search_filter', false);
            failed++;
        }
    } catch (e) {
        console.log(`Test suite error: ${e.message}`);
        failed++;
    } finally {
        // ---------------------------------------------------------------
        // Cleanup: restore the setting to its original value no matter what
        // happened above. In this dev environment every run registers a
        // brand-new user, so there is nothing to leak across runs here —
        // but in real CI, AuthHelper tries the shared CI_TEST_USER first
        // (see auth_helper.js), so every other settings-shard test may run
        // against the SAME account this file just modified. A failed
        // restore is reported as a suite failure, not swallowed.
        // ---------------------------------------------------------------
        try {
            if (changeWasApplied && originalValue !== null) {
                await workingPage.bringToFront().catch(() => {});
                await workingPage.waitForSelector(SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
                // The cleanup click needs the owning section open too.
                await expandSettingsSectionFor(workingPage, SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
                const currentValue = await workingPage.$eval(SETTING_SELECTOR, (el) => el.checked);

                if (currentValue !== originalValue) {
                    const [restoreResp] = await Promise.all([
                        waitForSaveResponse(workingPage),
                        workingPage.click(SETTING_SELECTOR),
                    ]);
                    const restoreBody = await restoreResp.json().catch(() => null);
                    const finalValue = await workingPage.$eval(SETTING_SELECTOR, (el) => el.checked);
                    const restoredOk = restoreResp.status() === 200 && restoreBody?.status === 'success' && finalValue === originalValue;

                    if (restoredOk) {
                        console.log(`Cleanup: restored ${SETTING_KEY} to its original value (${originalValue})`);
                    } else {
                        console.log(
                            `FAILED: Cleanup could not restore ${SETTING_KEY} to ${originalValue} ` +
                            `(status=${restoreResp.status()}, body=${JSON.stringify(restoreBody)}, final=${finalValue})`
                        );
                        failed++;
                    }
                } else {
                    console.log(`Cleanup: ${SETTING_KEY} already at its original value, nothing to restore`);
                }
            }
        } catch (cleanupError) {
            console.log(`FAILED: Cleanup error while restoring ${SETTING_KEY}: ${cleanupError.message}`);
            failed++;
        }

        await browser.close();
    }

    console.log('-'.repeat(50));
    console.log(`Settings Persistence Tests — passed: ${passed}, failed: ${failed}`);
    console.log('-'.repeat(50));
    if (failed > 0) process.exit(1);
}

run().catch((e) => {
    console.error('Test runner error:', e);
    process.exit(1);
});