~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
ci(release): add three cheap sanity checks (twine, tag-on-main, SBOM) (#4225)
LearningCircuit7b136b047b87
diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.ymlindex 265c9c7ea1..c9a9600d31 100644--- a/.github/workflows/release-gate.yml+++ b/.github/workflows/release-gate.yml@@ -411,6 +411,51 @@ jobs:echo "=== Verifying dependency tree is consistent ==="pip check+ echo ""+ echo "=== twine check (PyPI metadata + long_description rendering) ==="+ # Twine validates that the wheel's PKG-INFO is well-formed and the+ # README renders cleanly on PyPI. pdm build accepts metadata twine+ # rejects (e.g. malformed long_description), so this catches a class+ # of "PyPI project page silently broken" failures pip-install-check+ # cannot see. Installed transiently inside the throwaway venv.+ pip install --quiet twine+ twine check --strict "$WHEEL"++ echo ""+ echo "=== Installing [mcp] extra so ldr-mcp can resolve ==="+ # `ldr-mcp`'s target (local_deep_research.mcp.server) unconditionally+ # imports the `mcp` package, which is declared in the [mcp] extra+ # rather than core deps — intentional, per mcp/__init__.py docstring.+ # Incremental install on top of the base install above preserves the+ # base-install verification while letting the entry-point smoke+ # exercise both scripts.+ pip install --quiet --no-cache-dir "${WHEEL}[mcp]"++ echo ""+ echo "=== CLI entry-point resolution ==="+ # `import local_deep_research` above only proves the package is+ # importable — it doesn't exercise the [project.scripts] entries.+ # We can't invoke `ldr-web`/`ldr-mcp` directly: neither parses+ # --help, both start long-running servers and would hang the job.+ # Instead, replicate exactly what the console_script wrapper does+ # at runtime — import the target module and resolve the attribute+ # — via entry_points().load(), without calling the function.+ # Catches dead module refs (the original `main:main` bug) and+ # missing/renamed target attributes.+ python - <<'PY'+ import importlib.metadata as md+ targets = {"ldr-web", "ldr-mcp"}+ found = set()+ for ep in md.entry_points(group="console_scripts"):+ if ep.name in targets:+ fn = ep.load()+ assert callable(fn), f"{ep.name} -> {ep.value} is not callable"+ print(f"OK {ep.name} -> {ep.value}")+ found.add(ep.name)+ missing = targets - found+ assert not missing, f"missing console_scripts: {missing}"+ PY+echo ""echo "✅ pip install verification passed — wheel is safe to publish"diff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex f101862cca..1f19b92d70 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -240,6 +240,20 @@ jobs:persist-credentials: falsefetch-depth: 0+ - name: Verify build commit is on main+ # `push: branches: main` is trivially on main, but `push: tags: v*`+ # and `workflow_dispatch` can fire from any ref — a tag pointing at+ # a feature-branch commit, or a manual dispatch from a stale branch,+ # would otherwise sail through every gate. `fetch-depth: 0` above+ # makes origin/main reachable for the merge-base check.+ run: |+ set -euo pipefail+ if ! git merge-base --is-ancestor "$GITHUB_SHA" origin/main; then+ echo "::error::Build commit $GITHUB_SHA is not an ancestor of origin/main — refusing to release"+ exit 1+ fi+ echo "Build commit verified on main: $GITHUB_SHA"+- name: Determine versionid: versionenv:@@ -353,6 +367,88 @@ jobs:echo "Bundle files generated (contain signature + certificate):"ls -lh sbom-*.bundle+ - name: Verify SBOM bundles and sanity-check SBOM contents+ if: steps.check_release.outputs.exists == 'false'+ env:+ GH_REPO: ${{ github.repository }}+ GH_TOKEN: ${{ github.token }}+ run: |+ set -euo pipefail++ # Round-trip: prove the README cosign-verify-blob recipe still+ # works for end users. cosign signs above but never verifies, so a+ # sigstore/cert-chain regression would silently break downstream+ # verification — caught only when a user complains.+ echo "=== Verifying SBOM bundles round-trip ==="+ for bundle in sbom-spdx.json.bundle sbom-cyclonedx.json.bundle; do+ json="${bundle%.bundle}"+ cosign verify-blob \+ --bundle "$bundle" \+ --certificate-identity-regexp="^https://github.com/${GH_REPO}/" \+ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \+ "$json"+ done+ echo "Both SBOM bundles verify cleanly"++ # Trivy can emit a near-empty SBOM on partial scan failures+ # without surfacing a non-zero exit. Set a per-format floor at+ # 80% of the previous stable release's count; fall back to a hard+ # floor of 50 when no prior SBOM is fetchable (first release, GH+ # outage, releases predating SBOM attachment). SPDX and CycloneDX+ # counts differ (virtual packages, metadata-only entries) so each+ # format gets its own floor derived from its own prior count.+ # A cap of 2000 prevents runaway drift if a prior SBOM was+ # anomalously inflated by a Trivy bug.+ echo ""+ echo "=== Sanity-checking SBOM package counts ==="+ SPDX_COUNT=$(jq '.packages | length' sbom-spdx.json)+ CYCDX_COUNT=$(jq '.components | length' sbom-cyclonedx.json)+ echo "Current: SPDX=${SPDX_COUNT}, CycloneDX=${CYCDX_COUNT}"++ HARD_FLOOR=50+ FLOOR_CAP=2000+ SPDX_FLOOR="$HARD_FLOOR"+ CYCDX_FLOOR="$HARD_FLOOR"+ PREV_TAG=$(gh release list --repo "$GH_REPO" --limit 5 \+ --json tagName,isPrerelease,isDraft \+ --jq '[.[] | select(.isPrerelease==false and .isDraft==false)][0].tagName // empty')+ if [[ -n "$PREV_TAG" ]]; then+ PREV_DIR=/tmp/prev-sbom+ rm -rf "$PREV_DIR" && mkdir -p "$PREV_DIR"+ if gh release download "$PREV_TAG" --repo "$GH_REPO" \+ --pattern 'sbom-spdx.json' --pattern 'sbom-cyclonedx.json' \+ --dir "$PREV_DIR" 2>/dev/null; then+ if [[ -f "$PREV_DIR/sbom-spdx.json" ]]; then+ PREV_SPDX=$(jq '.packages | length' "$PREV_DIR/sbom-spdx.json")+ DYN=$(( PREV_SPDX * 80 / 100 ))+ DYN=$(( DYN < FLOOR_CAP ? DYN : FLOOR_CAP ))+ [[ "$DYN" -gt "$HARD_FLOOR" ]] && SPDX_FLOOR="$DYN"+ echo "Prior SPDX: $PREV_SPDX -> floor: $SPDX_FLOOR"+ fi+ if [[ -f "$PREV_DIR/sbom-cyclonedx.json" ]]; then+ PREV_CYCDX=$(jq '.components | length' "$PREV_DIR/sbom-cyclonedx.json")+ DYN=$(( PREV_CYCDX * 80 / 100 ))+ DYN=$(( DYN < FLOOR_CAP ? DYN : FLOOR_CAP ))+ [[ "$DYN" -gt "$HARD_FLOOR" ]] && CYCDX_FLOOR="$DYN"+ echo "Prior CycloneDX: $PREV_CYCDX -> floor: $CYCDX_FLOOR"+ fi+ else+ echo "No SBOMs in $PREV_TAG (older releases predate SBOM attachment) -> hard floor $HARD_FLOOR"+ fi+ else+ echo "No previous stable release found -> hard floor $HARD_FLOOR"+ fi++ if [[ "$SPDX_COUNT" -lt "$SPDX_FLOOR" ]]; then+ echo "::error::SPDX SBOM appears truncated ($SPDX_COUNT < floor $SPDX_FLOOR). Trivy may have failed silently."+ exit 1+ fi+ if [[ "$CYCDX_COUNT" -lt "$CYCDX_FLOOR" ]]; then+ echo "::error::CycloneDX SBOM appears truncated ($CYCDX_COUNT < floor $CYCDX_FLOOR). Trivy may have failed silently."+ exit 1+ fi+ echo "SBOM counts above per-format floors (SPDX≥$SPDX_FLOOR, CycloneDX≥$CYCDX_FLOOR)"+- name: Generate hashes for SLSA provenanceif: steps.check_release.outputs.exists == 'false'id: hashdiff --git a/docs/architecture/OVERVIEW.md b/docs/architecture/OVERVIEW.mdindex bdfc084e77..2a7d8049ba 100644--- a/docs/architecture/OVERVIEW.md+++ b/docs/architecture/OVERVIEW.md@@ -94,12 +94,6 @@ graph LR- `web/routes/` - Blueprint route handlers- `web/services/` - Business logic services-### CLI (`ldr`)--**Location:** `src/local_deep_research/main.py`--Command-line interface for headless research operations.-### REST API (`/api/v1`)**Location:** `src/local_deep_research/web/api.py`diff --git a/pyproject.toml b/pyproject.tomlindex ceae99f376..e6d7aafba6 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -109,7 +109,10 @@ mcp = ["Bug Tracker" = "https://github.com/LearningCircuit/local-deep-research/issues"[project.scripts]-ldr = "local_deep_research.main:main"+# `ldr` was removed in commit c1f80e7a8 ("Remove deprecated CLI", #319) but the+# entry-point line was left behind, so the installed binary has crashed on+# import for every release since. Caught by the entry-point smoke in+# release-gate.yml. Restore only when a replacement CLI exists.ldr-web = "local_deep_research.web.app:main"ldr-mcp = "local_deep_research.mcp:run_server"diff --git a/src/local_deep_research/web/database/README.md b/src/local_deep_research/web/database/README.mdindex 4b0a2d1286..a5f2980a1a 100644--- a/src/local_deep_research/web/database/README.md+++ b/src/local_deep_research/web/database/README.md@@ -14,33 +14,22 @@ The application now follows a "database-first" approach for settings:2. Settings from TOML files are used only as fallbacks if a setting doesn't exist in the database3. The web UI settings page modifies the database values directly-## Migration+## Legacy migration (historical)-If you have existing data in the legacy databases, you need to migrate it to the new unified database.+LDR has since moved to a per-user encrypted SQLCipher database model.+The original `deep_research.db` + `research_history.db` → `ldr.db`+migration tooling described in earlier versions of this README has been+removed along with the deprecated `ldr` CLI. The database split it+addressed predates the per-user model by several releases, so anyone+upgrading from a version that old should start fresh rather than+attempting to recover the legacy data.+There is no supported migration path from pre-v0.x databases.-### Automatic Migration+## Schema upgrades-When you start the application for the first time after updating, it will check if migration is needed:--1. If legacy databases exist and `ldr.db` doesn't exist, you'll see a warning message-2. You can run migration using the command: `python -m src.local_deep_research.main --migrate-db`-3. Alternatively, start the application with auto-migration: `python -m src.local_deep_research.main --auto-migrate`--### Manual Migration--If automatic migration doesn't work, you can:--1. Run the migration script directly: `python -m src.local_deep_research.web.database.migrate_to_ldr_db`-2. Check migration results in the log output--### Schema Upgrades--If you have already migrated your database but need to update its schema:--1. The application automatically runs schema upgrades on startup-2. You can manually run schema upgrades with: `python -m src.local_deep_research.main --schema-upgrade`-3. Current schema upgrades include:- - Removing the redundant `research_log` table (consolidated into `research_logs`)+Schema migrations run automatically on application startup via Alembic+(`src/local_deep_research/database/migrations/`). No manual command is+required.## Database Schema@@ -63,16 +52,9 @@ Cleanup is handled by the `@thread_cleanup` decorator (closes sessions at threadNote: The core database module is at `src/local_deep_research/database/`, separate from this `web/database/` directory. See [Architecture - Thread & Resource Lifecycle](../../../../docs/architecture.md#thread--resource-lifecycle) for the full cleanup architecture.-## Rollback--If you need to roll back to the previous database architecture:--1. Keep backup copies of your original `deep_research.db` and `research_history.db` files-2. In case of issues, restore them and modify the database paths in the code-## Troubleshooting-If you encounter issues with database migration:+If you encounter database issues:1. Check the application logs for detailed error messages2. Ensure you have write permissions to the data directory