~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
#!/usr/bin/env node
/**
* Settings Round-Trip UI Tests (Flask -> FastAPI migration)
*
* Settings is the largest surface in the app (473+ rendered items across
* 7 tabs on this build), and the read/write path underneath it was rewritten
* wholesale in the migration: GET /settings/api/{key} and POST
* /settings/save_all_settings (routers/settings.py::api_get_db_setting /
* save_all_settings) replaced the old Flask blueprint's equivalents. Despite
* that, no browser test in this suite proves a user can actually change a
* setting through the real UI and have it stick — the existing CI settings
* tests (test_settings_interactions_ci.js, test_settings_pages_ci.js, etc.)
* check that controls render, that tabs/search exist, and that save errors
* surface a toast, but none of them drive a real control, wait for the
* app's own autosave, reload the page, and assert the new value survives.
* The only file that ever did that (test_settings_persistence.js) manipulates
* DOM values directly via page.evaluate() rather than real click/select
* interactions, and — because it isn't named `*_ci.js` — was never wired
* into run_all_tests.js, so it has never actually run in CI.
*
* This file drives the real autosave path settings.js uses for every
* control on the page (handleInputChange -> submitSettingsData -> POST
* /settings/save_all_settings, 800ms debounced — see scheduleSave() in
* components/settings.js) via genuine page.click()/page.type() interaction,
* and proves: the change survives a full page reload (Test 1), the value is
* actually in the database and not just the DOM/JS cache (Test 2, read back
* through the settings API from page context), a second user's account
* never sees it (Test 3, browser-level analogue of the cross-user settings
* isolation this project has separately investigated at the API/python
* level), and the search box's filter contract holds (Test 4).
*
* Setting chosen: app.enable_notifications ("Enable Notifications" — browser
* push alerts for research completion/errors). It is a plain checkbox
* (editable: true, visible: true, default true — see defaults/
* default_settings.json), so its save path has no enum/options validation
* to worry about, and toggling it can never disable auth, CSRF, rate
* limiting, or move the app's data directory — the exact hazards this task
* was told to avoid. It round-trips cleanly (verified manually against the
* running dev server before writing this file).
*
* NOT used: app.theme, despite being the more obviously "cosmetic" choice.
* While surveying candidates, changing the theme select and later trying to
* restore it to its own shipped default ("dark") turned up a genuine,
* reproducible defect unrelated to this migration: default_settings.json
* ships "app.theme": {"value": "dark", ...}, but manager.py dynamically
* replaces that field's `options` with theme_registry.get_settings_options()
* (a scan of the CSS-file-backed theme registry, which has no "dark.css" —
* "dark" is just the app's un-themed baseline, not a registered theme).
* validate_setting() in routers/settings.py then rejects "dark" outright:
* POST /settings/save_all_settings {"app.theme": "dark"} -> 400 "Value must
* be one of: ayu-mirage, catppuccin, ... " (dark/system absent from the
* list). So the value every brand-new account actually has for app.theme
* cannot be round-tripped through the save endpoint at all — confirmed via
* both a real <select> interaction (Puppeteer's page.select() finds no
* matching <option> once the value has ever moved off "dark", so the
* browser's native fallback silently selects and saves index 0,
* "Ayu Mirage", instead — a silent wrong-value save with a "UI Theme
* updated" success toast) and a direct POST replaying the app's exact
* request shape (400, see above). Root cause: DYNAMIC_SETTINGS in
* routers/settings.py (= ["llm.provider", "llm.model", "search.tool"]) never
* had "app.theme" added to it even though manager.py has dynamically injected
* its options the same way since search.search_strategy and the LLM/search
* dynamic settings were exempted. This is a real bug worth fixing, but it is
* orthogonal to the Flask->FastAPI rewrite and — since a correct fix would
* make "dark" round-trip successfully — asserting on today's broken behavior
* here would make this file fail once someone fixes it. Reported in this
* comment plus the task's final summary instead of pinned as a test.
*
* Run: CI=true node test_settings_persistence_ci.js
*/
const crypto = require('crypto');
const puppeteer = require('puppeteer');
const AuthHelper = require('./auth_helper');
const { getPuppeteerLaunchOptions } = require('./puppeteer_config');
const { expandSettingsSectionFor } = require('./test_lib');
const { capture, captureOnFailure } = require('./screenshot_helper');
const BASE_URL = process.env.BASE_URL || 'http://127.0.0.1:5000';
const isCI = !!process.env.CI;
const TIMEOUTS = {
navigation: isCI ? 60000 : 30000,
selector: isCI ? 30000 : 10000,
save: isCI ? 20000 : 10000,
};
const SETTING_SELECTOR = '#setting-app-enable_notifications';
const SETTING_KEY = 'app.enable_notifications';
const SCREENSHOT_PREFIX = 'settings_persistence';
/** True for a real, attributable console error — not the browser's own favicon probe noise. */
function isRealConsoleError(msg) {
return msg.type() === 'error' && !msg.text().startsWith('Failed to load resource');
}
/** Wait for the specific save_all_settings POST the app's own JS issues, and return its Response. */
function waitForSaveResponse(page) {
return page.waitForResponse(
(r) => r.url().includes('/settings/save_all_settings') && r.request().method() === 'POST',
{ timeout: TIMEOUTS.save }
);
}
async function run() {
console.log(`Running settings persistence round-trip tests (CI mode: ${isCI})`);
const browser = await puppeteer.launch(getPuppeteerLaunchOptions());
const page = await browser.newPage();
await page.setViewport({ width: 1280, height: 800 });
if (isCI) {
page.setDefaultTimeout(60000);
page.setDefaultNavigationTimeout(60000);
}
page.on('pageerror', (e) => console.log('PAGE ERROR:', e.message));
page.on('console', (m) => {
if (isRealConsoleError(m)) console.log('BROWSER ERROR:', m.text());
});
let passed = 0;
let failed = 0;
// Shared across tests, and read by the cleanup block in `finally` below.
let originalValue = null;
let changedValue = null;
let changeWasApplied = false;
// AuthHelper.getPage() can return a DIFFERENT page object than `page`
// above if it recovered from a detached frame during auth (see
// auth_helper.js). The `finally` cleanup block below must act on
// whichever page actually did the mutating, so track it here in the
// outer scope rather than re-reading the (possibly stale) `page`.
let workingPage = page;
try {
const auth = new AuthHelper(page, BASE_URL);
await auth.ensureAuthenticatedWithTimeout();
workingPage = auth.getPage();
await workingPage.goto(`${BASE_URL}/settings/`, {
waitUntil: 'domcontentloaded',
timeout: TIMEOUTS.navigation,
});
await workingPage.waitForSelector(SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
// Settings sections start collapsed on every viewport, so the
// checkbox renders inside a `display: none` body. Open its section
// before any click — page.click() needs a real box to hit.
await expandSettingsSectionFor(workingPage, SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
await capture(workingPage, SCREENSHOT_PREFIX, 'before_toggle', { fullPage: true });
// -----------------------------------------------------------------
// Test 1: a setting changed via a real UI click persists across a
// full page reload. The click fires settings.js's 'change' handler,
// which schedules an 800ms-debounced autosave — waitForSaveResponse
// is the explicit, non-sleep wait for that save to actually land.
// -----------------------------------------------------------------
console.log(`Test 1: toggling ${SETTING_KEY} via a real click, waiting for the app's own save, then reloading`);
try {
originalValue = await workingPage.$eval(SETTING_SELECTOR, (el) => el.checked);
const [saveResp] = await Promise.all([
waitForSaveResponse(workingPage),
workingPage.click(SETTING_SELECTOR),
]);
changeWasApplied = true;
const saveBody = await saveResp.json().catch(() => null);
if (saveResp.status() !== 200 || !saveBody || saveBody.status !== 'success') {
throw new Error(`Save request did not succeed: status=${saveResp.status()} body=${JSON.stringify(saveBody)}`);
}
if (!Array.isArray(saveBody.updated) || !saveBody.updated.includes(SETTING_KEY)) {
throw new Error(`Save response did not report ${SETTING_KEY} as updated: ${JSON.stringify(saveBody.updated)}`);
}
await workingPage.reload({ waitUntil: 'domcontentloaded', timeout: TIMEOUTS.navigation });
await workingPage.waitForSelector(SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
await expandSettingsSectionFor(workingPage, SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
const afterReload = await workingPage.$eval(SETTING_SELECTOR, (el) => el.checked);
if (afterReload === originalValue) {
throw new Error(
`Checkbox shows the original value (${originalValue}) after reload — the change did not persist`
);
}
changedValue = afterReload;
console.log(`PASSED (original=${originalValue} -> changed=${changedValue}, survived reload)`);
passed++;
await capture(workingPage, SCREENSHOT_PREFIX, 'after_toggle', { fullPage: true });
} catch (e) {
console.log(`FAILED: ${e.message}`);
await captureOnFailure(workingPage, SCREENSHOT_PREFIX, 'persist_across_reload', false);
failed++;
}
// -----------------------------------------------------------------
// Test 2: the change is actually in the database, not just the DOM
// or settings.js's in-memory allSettings cache. Read it back through
// the real settings API from page context.
// -----------------------------------------------------------------
console.log(`Test 2: reading ${SETTING_KEY} back through GET /settings/api/${SETTING_KEY} confirms server-side persistence`);
try {
if (changedValue === null) {
throw new Error('Skipped: Test 1 did not establish a changed value to verify');
}
const apiResult = await workingPage.evaluate(async (key) => {
const r = await fetch(`/settings/api/${key}`, { credentials: 'same-origin' });
return { status: r.status, json: await r.json().catch(() => null) };
}, SETTING_KEY);
if (apiResult.status !== 200) {
throw new Error(`Expected HTTP 200 from the settings API, got ${apiResult.status}`);
}
if (apiResult.json?.value !== changedValue) {
throw new Error(
`Settings API value (${apiResult.json?.value}) does not match the UI-persisted value (${changedValue})`
);
}
console.log(`PASSED (API value=${apiResult.json.value})`);
passed++;
} catch (e) {
console.log(`FAILED: ${e.message}`);
failed++;
}
// -----------------------------------------------------------------
// Test 3: per-user isolation at the UI level. A second user,
// registered in a fresh incognito browser context (no shared
// cookies with user A), must NOT see user A's changed value.
// This is the browser-level analogue of the cross-user settings
// isolation this project has investigated at the API/python level —
// worth pinning here too since it exercises the real cookie/session
// boundary a browser enforces, not just two DB-session fixtures.
// -----------------------------------------------------------------
console.log("Test 3: a second user in a fresh incognito context does not see user A's changed setting");
try {
if (changedValue === null) {
throw new Error("Skipped: no changed value from Test 1 to check isolation against");
}
const incognitoContext = await browser.createBrowserContext();
try {
const pageB = await incognitoContext.newPage();
if (isCI) {
pageB.setDefaultTimeout(60000);
pageB.setDefaultNavigationTimeout(60000);
}
const authB = new AuthHelper(pageB, BASE_URL);
// Name the account explicitly. Calling this with no username
// is what broke this test: in CI the helper prefers the shared
// CI_TEST_USER (test_admin), which is the account user A is
// already using. A fresh incognito context gives a new cookie
// jar, not a new user — so the test compared user A against
// user A and reported the match as a leak. Passing a username
// now disables that shortcut (see auth_helper.js), so the
// account named here is the account that gets used.
// The cleanup block at the bottom of this file already
// documents the shared-CI_TEST_USER behaviour; this is that
// same fact applied where it decides whether the assertion
// means anything.
const userBName =
`isolation_b_${Date.now()}_` +
`${crypto.randomBytes(4).toString('hex')}`;
await authB.ensureAuthenticatedWithTimeout(
userBName,
'T3st!Secure#2024$LDR' // pragma: allowlist secret
);
const workingPageB = authB.getPage();
await workingPageB.goto(`${BASE_URL}/settings/`, {
waitUntil: 'domcontentloaded',
timeout: TIMEOUTS.navigation,
});
await workingPageB.waitForSelector(SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
const userBValue = await workingPageB.$eval(SETTING_SELECTOR, (el) => el.checked);
if (userBValue === changedValue) {
throw new Error(
`GENUINE DEFECT: user B (${userBName}, a freshly registered account) reads ` +
`${SETTING_KEY} = ${userBValue}, matching user A's explicitly-changed value ` +
`(${changedValue}) instead of the account default (${originalValue}). ` +
`A setting is leaking across user accounts.`
);
}
console.log(`PASSED (user A=${changedValue}, user B=${userBValue} — independent per-user state)`);
passed++;
} finally {
await incognitoContext.close();
}
} catch (e) {
console.log(`FAILED: ${e.message}`);
failed++;
}
// -----------------------------------------------------------------
// Test 4: the settings search box narrows the visible list and the
// narrowed list still contains a known match. Resilient by design:
// asserts a relationship (fewer visible, expected key present, full
// count restored on clear), never a hardcoded total.
// -----------------------------------------------------------------
console.log('Test 4: settings search narrows the visible list and keeps the expected match');
try {
await workingPage.waitForSelector('#settings-content .ldr-settings-item', { timeout: TIMEOUTS.selector });
// Defensive: make sure no leftover query text from a prior run/step.
await workingPage.evaluate(() => {
const el = document.querySelector('#settings-search');
if (el) el.value = '';
});
const countItems = () => workingPage.$$eval('#settings-content .ldr-settings-item', (els) => els.length);
const initialCount = await countItems();
if (initialCount === 0) {
throw new Error('No settings rendered before filtering');
}
await workingPage.type('#settings-search', SETTING_KEY);
await workingPage.waitForFunction(
(n) => document.querySelectorAll('#settings-content .ldr-settings-item').length < n,
{ timeout: 5000 },
initialCount
);
const filteredCount = await countItems();
const hasExpectedMatch = await workingPage.evaluate(
(key) => !!document.querySelector(`#settings-content .ldr-settings-item[data-key="${key}"]`),
SETTING_KEY
);
if (!(filteredCount < initialCount)) {
throw new Error(`Filtering did not narrow the list (initial=${initialCount}, filtered=${filteredCount})`);
}
if (!hasExpectedMatch) {
throw new Error(`Filtered list is missing the expected match for "${SETTING_KEY}"`);
}
// Clearing restores the full list — proves this is a real filter
// over the rendered set, not a one-way destructive re-render.
await workingPage.$eval('#settings-search', (el) => {
el.value = '';
el.dispatchEvent(new Event('input', { bubbles: true }));
});
await workingPage.waitForFunction(
(n) => document.querySelectorAll('#settings-content .ldr-settings-item').length === n,
{ timeout: 5000 },
initialCount
);
const restoredCount = await countItems();
if (restoredCount !== initialCount) {
throw new Error(`Clearing search did not restore the full list (initial=${initialCount}, restored=${restoredCount})`);
}
console.log(`PASSED (${initialCount} -> ${filteredCount} filtered -> ${restoredCount} restored)`);
passed++;
} catch (e) {
console.log(`FAILED: ${e.message}`);
await captureOnFailure(workingPage, SCREENSHOT_PREFIX, 'search_filter', false);
failed++;
}
} catch (e) {
console.log(`Test suite error: ${e.message}`);
failed++;
} finally {
// ---------------------------------------------------------------
// Cleanup: restore the setting to its original value no matter what
// happened above. In this dev environment every run registers a
// brand-new user, so there is nothing to leak across runs here —
// but in real CI, AuthHelper tries the shared CI_TEST_USER first
// (see auth_helper.js), so every other settings-shard test may run
// against the SAME account this file just modified. A failed
// restore is reported as a suite failure, not swallowed.
// ---------------------------------------------------------------
try {
if (changeWasApplied && originalValue !== null) {
await workingPage.bringToFront().catch(() => {});
await workingPage.waitForSelector(SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
// The cleanup click needs the owning section open too.
await expandSettingsSectionFor(workingPage, SETTING_SELECTOR, { timeout: TIMEOUTS.selector });
const currentValue = await workingPage.$eval(SETTING_SELECTOR, (el) => el.checked);
if (currentValue !== originalValue) {
const [restoreResp] = await Promise.all([
waitForSaveResponse(workingPage),
workingPage.click(SETTING_SELECTOR),
]);
const restoreBody = await restoreResp.json().catch(() => null);
const finalValue = await workingPage.$eval(SETTING_SELECTOR, (el) => el.checked);
const restoredOk = restoreResp.status() === 200 && restoreBody?.status === 'success' && finalValue === originalValue;
if (restoredOk) {
console.log(`Cleanup: restored ${SETTING_KEY} to its original value (${originalValue})`);
} else {
console.log(
`FAILED: Cleanup could not restore ${SETTING_KEY} to ${originalValue} ` +
`(status=${restoreResp.status()}, body=${JSON.stringify(restoreBody)}, final=${finalValue})`
);
failed++;
}
} else {
console.log(`Cleanup: ${SETTING_KEY} already at its original value, nothing to restore`);
}
}
} catch (cleanupError) {
console.log(`FAILED: Cleanup error while restoring ${SETTING_KEY}: ${cleanupError.message}`);
failed++;
}
await browser.close();
}
console.log('-'.repeat(50));
console.log(`Settings Persistence Tests — passed: ${passed}, failed: ${failed}`);
console.log('-'.repeat(50));
if (failed > 0) process.exit(1);
}
run().catch((e) => {
console.error('Test runner error:', e);
process.exit(1);
});