~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
AGENTS.md
Overview
Local Deep Research is a Python/PDM application with a FastAPI backend and a Vite frontend. Backend source is in src/local_deep_research/; web code is in src/local_deep_research/web/; tests are in tests/. src/local_deep_research/web/app.py:main() is the process entry point and launches Uvicorn; src/local_deep_research/web/fastapi_app.py constructs the FastAPI application, mounts Socket.IO, and manages startup/shutdown through lifespan().
Source-development prerequisites: Python ≥3.12, <3.15; Node.js 24.0.0+ (frontend rebuild only), PDM, and SQLCipher; see docs/developing.md and docs/SQLCIPHER_INSTALL.md.
Architecture
Iterative research pipeline: a BaseSearchStrategy generates questions, runs parallel searches across BaseSearchEngine implementations, and synthesizes a report via report_generator.py. The web layer is FastAPI + Socket.IO (ASGI); each user has their own encrypted SQLCipher database (no shared application-data DB; a central ldr_auth.db stores only usernames). The main entry point is src/local_deep_research/search_system.py (AdvancedSearchSystem). New search engines require registration in ENGINE_REGISTRY (src/local_deep_research/web_search_engines/engine_registry.py) and the class allowlist in src/local_deep_research/security/module_whitelist.py; new strategies require wiring in create_strategy() (src/local_deep_research/search_system_factory.py) — only LLM providers are auto-discovered. See docs/architecture.md for the full thread lifecycle, QueuePool/FD budget, and cleanup model.
Where to look
| Task | Paths |
|---|---|
| Backend and server lifecycle | src/local_deep_research/web/app.py, src/local_deep_research/web/fastapi_app.py, src/local_deep_research/web/routers/, src/local_deep_research/web/services/ |
| Auth and sessions | src/local_deep_research/web/auth/, src/local_deep_research/database/encrypted_db.py |
| Settings | src/local_deep_research/settings/, src/local_deep_research/settings/env_definitions/, src/local_deep_research/web/routers/settings.py, src/local_deep_research/defaults/ |
| Security and safe HTTP | src/local_deep_research/security/ (safe_requests.py, ssrf_validator.py, egress/); policy in SECURITY.md |
| LLM and embedding providers | src/local_deep_research/llm/providers/, src/local_deep_research/config/llm_config.py, src/local_deep_research/embeddings/providers/ |
| Search engines | src/local_deep_research/web_search_engines/, src/local_deep_research/web_search_engines/engines/ |
| Research engine and strategies | src/local_deep_research/search_system.py, src/local_deep_research/search_system_factory.py, src/local_deep_research/advanced_search_system/strategies/ |
| Report generation | src/local_deep_research/report_generator.py |
| Database, models, migrations | src/local_deep_research/database/models/, src/local_deep_research/database/migrations/versions/, src/local_deep_research/database/thread_local_session.py |
| Vector store and RAG | src/local_deep_research/vector_stores/, src/local_deep_research/research_library/services/library_rag_service.py |
| Programmatic API and MCP | src/local_deep_research/api/, src/local_deep_research/mcp/ |
| News, benchmarks, notes | src/local_deep_research/news/, src/local_deep_research/benchmarks/, src/local_deep_research/research_library/notes/ |
| Frontend | src/local_deep_research/web/static/, src/local_deep_research/web/templates/, src/local_deep_research/web/themes/ |
| Tests and CI | tests/, .github/workflows/, .pre-commit-config.yaml, .pre-commit-hooks/, scripts/pre_commit/ |
| Contributor and release policy | CONTRIBUTING.md, changelog.d/README.md, docs/RELEASE_GUIDE.md |
| Architecture and decisions | docs/architecture.md, docs/decisions/ (accepted ADRs are binding) |
| Development setup and CI detail | docs/developing.md, docs/CI_CD_INFRASTRUCTURE.md |
Commands
pdm install # installs the package editable; with --no-self, prefix commands with PYTHONPATH=src
npm install
pdm run pre-commit install && pdm run pre-commit install-hooks
pdm run python -m local_deep_research.web.app
pdm run pytest tests/security/test_repo_guardrails.py -q
npm test
npm run build
pdm run pre-commit run --all-files
pdm run mypy src/ --config-file=pyproject.toml --show-error-codes --pretty
Pre-commit includes auto-fix hooks; inspect the resulting diff after it runs.
Contributing
- Keep PRs small and atomic: one change per PR, with focused tests for behavior changes; see
CONTRIBUTING.md. - Before working on an issue, claim it with a comment and open an early (draft) PR; skip issues that already have a recent claim or linked PR.
- Discuss the approach in the issue first for architectural or cross-subsystem changes, and for anything that can shift research quality: LLM prompts,
src/local_deep_research/advanced_search_system/, or search-engine query/parsing/ranking behavior (seedocs/BENCHMARKING.md). - Add a changelog fragment only for user-visible behavior:
changelog.d/<PR number>.<category>.md(+<slug>instead of a number when there is no PR yet), with category one ofbreaking,security,feature,bugfix,removal,misc. The release workflow renders them; local preview ispdm run towncrier build --draft --version <X.Y.Z>.
Filing issues
Issues are the project's backlog, and most are filed by agents with gh rather than through the GitHub web UI, so the issue templates in .github/ISSUE_TEMPLATE/ are bypassed — these rules apply instead.
- Title: name the file or function and the symptom.
- Body: prose, not a template. State what is wrong, where (path and function), why it matters, and the fix options. Cross-reference the originating PR or review (
Found during the review of #N) and any sibling issues. - Labels: one primary area label from the existing set (
database,python,javascript,ci-cd,security,documentation,tests,ui-ux,performance,llm-providers,rag,api, ...; list them withgh label list) plus one ofpriority: high,priority: medium,priority: low. - Search first to avoid duplicates:
gh issue list -R LearningCircuit/local-deep-research --search "<terms>" --state all. - File with
gh issue create --label <area> --label "priority: <level>" --title "..." --body-file <file>. - Security vulnerabilities are never filed as public issues; follow
SECURITY.md.
Invariants
- Test runs default
LDR_TESTING_WITH_MOCKStotruewhen it is unset; tests markedrequires_llmorintegrationare skipped in that mode. Set it tofalseonly when exercising real LLM or network behavior. - Do not hand-edit generated files. Committed — regenerate and commit the result:
docs/CONFIGURATION.md(scripts/generate_config_docs.py),docs/ci/workflow-status.md(scripts/generate_workflow_status.py). Gitignored — never commit:src/local_deep_research/web/static/dist/(Vite build) andsrc/local_deep_research/web/static/css/themes.css(written bylifespan()fromsrc/local_deep_research/web/themes/<category>/*.css). - User data is per-user SQLCipher data. Access it through
get_user_db_session; background threads must pass their owndb_sessiontoget_settings_manager()because they lack authenticated request context — that path raises rather than silently using defaults. Workers create their own thread-local session — never receive or share aSessionacross threads, and never hold one acrossyieldin SSE/streaming generators (convention, not hook-enforced). - Database migrations: do NOT use the
alembicCLI —src/local_deep_research/database/migrations/env.pyrejects it. Userun_migrations(engine)fromsrc/local_deep_research/database/alembic_runner.py; revision IDs are zero-padded 4-digit strings insrc/local_deep_research/database/migrations/versions/NNNN_description.py, withdown_revisionset to the previous number (check the highest existing number first). - New database models inherit
src/local_deep_research/database/base.py:Base, are imported bysrc/local_deep_research/database/models/__init__.py, useUtcDateTimefor datetime columns (enforced byscripts/pre_commit/check_datetime_timezone.py), and get a numbered migration. New FastAPI routers must be registered in_mount_all()insrc/local_deep_research/web/fastapi_app.py. - DB-backed settings keys must use the namespaces in
ALLOWED_SETTING_PREFIXESinsrc/local_deep_research/web/routers/settings.py;BLOCKED_SETTING_PREFIXES(auth.,bootstrap.,db_config.,security.,server.,testing.) are not web-editable, and env-only settings are defined insrc/local_deep_research/settings/env_definitions/. Use context managers for resource-owning services and sessions; all journal-quality DB consumers exceptbuild_db()open it read-only. - Test logging: the package disables loguru at import; use the
loguru_caplogfixture (orloguru_caplog_fullfor tracebacks) — standard pytestcaplogcaptures nothing. - Pre-commit runs 60+ custom hooks from
.pre-commit-hooks/(most block commits; therecommend-*hooks are advisory and return 0, andfix-exception-logging.pyis an auto-fixer). Common gotchas: tests must import fromlocal_deep_research(or use# allow: no-sut-import — <reason>); directos.environaccess is limited to the paths allowlisted in.pre-commit-hooks/check-env-vars.py;time.sleep(>0.1s)in tests requires@pytest.mark.slow; tautological tests are rejected (# allow: weak-testto opt out); silentexcept: pass,os.path,open()withoutencoding=, and literal settings keys outside the allowed namespaces are all rejected. - Protected infrastructure packages (
settings,utilities,security,config) must not import the web layer, except the narrow allowlists in.pre-commit-hooks/check-layer-imports.py.research_idis a string UUID, not an integer. - Logging: use
get_logs_directory()underLDR_DATA_DIR; never log secrets. Insrc/local_deep_research/llm/providers/,src/local_deep_research/embeddings/providers/andsrc/local_deep_research/web_search_engines/, importloggerfromsrc/local_deep_research/security/secure_logging.py(hook-enforced: raw loguru there leaks exception tracebacks with credentials to all sinks); elsewhere plainfrom loguru import loggeris the norm — do not mass-convert. Usepathlib.Path, explicit text encodings, and bundled local assets rather than remote resources. - Persisted research logs capture only the rendered message and selected fields — diagnostic context needed in
app_logsor exported JSONL must appear in the message text, not just Loguruextra. - Vector-store writes must use the canonical reentrant per-
(username, index_path)lock (_get_faiss_write_lock()insrc/local_deep_research/research_library/services/library_rag_service.py) andBaseVectorStore.apply(); a plainthreading.Lockself-deadlocks. report_contentis answer-only; sources are structured separately. Detailed reports append the sources sentinel; raw/quick LLM output must callformat_document_split(..., trust_sentinel=False).- Treat
.pre-commit-config.yamland its executable checks as authoritative for configured behavior, while remembering that their enforcement is intentionally scoped.
Security invariants
See SECURITY.md for the full threat model; these are the rules code changes must keep.
- The repository is deny-by-default: every tracked file must match
.file-whitelist.txt, so a new file type needs a deliberate whitelist entry. Symlinks whose target is missing or is a directory, submodule gitlinks and other non-regular entries are rejected rather than skipped. Never commit secrets, keys, databases or local data. - All server-side outbound HTTP goes through
safe_get,safe_postorSafeSession(src/local_deep_research/security/safe_requests.py), which validate every redirect hop and pin DNS. Rawrequestscalls are rejected by.pre-commit-hooks/check-safe-requests.py; do not route around it with another HTTP client. - New network paths must also respect the egress policy (
src/local_deep_research/security/egress/README.md): engines are created viacreate_search_engine(), LLMs viaget_llm(), agent URL fetches callevaluate_url(). Operator-configurable LLM and embedding base URLs are checked withassert_base_url_safe(). - In
src/local_deep_research/security/ssrf_validator.py, keepRFC_FORBIDDEN_URL_CHARS_REand theurllib3.util.parse_urlhost extraction (removing either reopens GHSA-g23j-2vwm-5c25), and never remove entries fromALWAYS_BLOCKED_METADATA_IPS. Log rejected URLs only throughredact_url_for_log(). - Switches that weaken a security control (outbound notifications, private-IP access, NAT64, unprotected egress, unencrypted databases) are env-only operator decisions and must never become web-editable settings.
LDR_BOOTSTRAP_ALLOW_UNENCRYPTEDis for isolated local development only: it disables password verification. - GitHub Actions workflows pin actions to full commit SHAs, declare least-privilege
permissions:, check out withpersist-credentials: false, and follow the existingstep-security/harden-runnerpattern; zizmor audits workflows in CI. - Hidden Unicode from the enforced set is rejected in tracked text by
.pre-commit-hooks/check-zero-width-unicode.py. The enforced set is exactly: every Bidi_Control codepoint (U+061C, U+200E–U+200F, U+202A–U+202E, U+2066–U+2069), the tag block U+E0000–E007F, the variation-selector supplement U+E0100–E01EF, the hook'sZERO_WIDTH_CODEPOINTSconstant (U+061C, U+180E, U+200B–U+200D, U+200E–U+200F, U+2060, U+FEFF), and the invisible operators / deprecated format controls U+2061–U+2064 and U+206A–U+206F. Write those characters in source as\uXXXXescapes (Python/JavaScript/TypeScript); in Markdown, which has no\uXXXXescape, use character references such as‍. Input handling fails closed at file level: bytes that do not decode as UTF-8 are refused outright (enforced codepoints seen as UTF-8 byte sequences in the raw bytes are still reported), and so is any input containing a NUL byte; between them those two rules refuse every UTF-16/UTF-32 shape except byte soup that is simultaneously NUL-free and valid UTF-8 — out of scope because it is read and scanned as the UTF-8 text it already decodes to. Variation selectors U+FE00–U+FE0F (commonly part of emoji sequences) and the Mongolian free variation selectors U+180B–U+180D are not rejected: no file is refused for carrying them. - Tracked PATHS are held to a stricter rule than tracked text: every tracked path must be pure printable ASCII (U+0020–U+007E). Hidden-Unicode or lookalike filenames (zero-width characters, bidi marks, tag characters, fullwidth/Cyrillic/Greek homoglyphs, control characters) are rejected outright by the index guard shared by
.github/scripts/file-whitelist-check.shand.pre-commit-hooks/file-whitelist-check.sh(agent_guidance_index_filter.py), which also rejectsAGENT(S).md/CLAUDE.md/GEMINI.mdbasenames at any depth in any letter case (Unicode caseless matching) except the exact root paths. The rootAGENTS.md,CLAUDE.mdandGEMINI.mdmust be regular files (tracked mode 100644, no symlink), and theCLAUDE.md/GEMINI.mdbridges' blob must be exactly@AGENTS.mdplus a trailing newline.