Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

ci(release): add three cheap sanity checks (twine, tag-on-main, SBOM) (#4225)

LearningCircuit7b136b047b87
diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml
index 265c9c7ea1..c9a9600d31 100644
--- a/.github/workflows/release-gate.yml
+++ b/.github/workflows/release-gate.yml
@@ -411,6 +411,51 @@ jobs:
echo "=== Verifying dependency tree is consistent ==="
pip check
+ echo ""
+ echo "=== twine check (PyPI metadata + long_description rendering) ==="
+ # Twine validates that the wheel's PKG-INFO is well-formed and the
+ # README renders cleanly on PyPI. pdm build accepts metadata twine
+ # rejects (e.g. malformed long_description), so this catches a class
+ # of "PyPI project page silently broken" failures pip-install-check
+ # cannot see. Installed transiently inside the throwaway venv.
+ pip install --quiet twine
+ twine check --strict "$WHEEL"
+
+ echo ""
+ echo "=== Installing [mcp] extra so ldr-mcp can resolve ==="
+ # `ldr-mcp`'s target (local_deep_research.mcp.server) unconditionally
+ # imports the `mcp` package, which is declared in the [mcp] extra
+ # rather than core deps — intentional, per mcp/__init__.py docstring.
+ # Incremental install on top of the base install above preserves the
+ # base-install verification while letting the entry-point smoke
+ # exercise both scripts.
+ pip install --quiet --no-cache-dir "${WHEEL}[mcp]"
+
+ echo ""
+ echo "=== CLI entry-point resolution ==="
+ # `import local_deep_research` above only proves the package is
+ # importable — it doesn't exercise the [project.scripts] entries.
+ # We can't invoke `ldr-web`/`ldr-mcp` directly: neither parses
+ # --help, both start long-running servers and would hang the job.
+ # Instead, replicate exactly what the console_script wrapper does
+ # at runtime — import the target module and resolve the attribute
+ # — via entry_points().load(), without calling the function.
+ # Catches dead module refs (the original `main:main` bug) and
+ # missing/renamed target attributes.
+ python - <<'PY'
+ import importlib.metadata as md
+ targets = {"ldr-web", "ldr-mcp"}
+ found = set()
+ for ep in md.entry_points(group="console_scripts"):
+ if ep.name in targets:
+ fn = ep.load()
+ assert callable(fn), f"{ep.name} -> {ep.value} is not callable"
+ print(f"OK {ep.name} -> {ep.value}")
+ found.add(ep.name)
+ missing = targets - found
+ assert not missing, f"missing console_scripts: {missing}"
+ PY
+
echo ""
echo "✅ pip install verification passed — wheel is safe to publish"
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index f101862cca..1f19b92d70 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -240,6 +240,20 @@ jobs:
persist-credentials: false
fetch-depth: 0
+ - name: Verify build commit is on main
+ # `push: branches: main` is trivially on main, but `push: tags: v*`
+ # and `workflow_dispatch` can fire from any ref — a tag pointing at
+ # a feature-branch commit, or a manual dispatch from a stale branch,
+ # would otherwise sail through every gate. `fetch-depth: 0` above
+ # makes origin/main reachable for the merge-base check.
+ run: |
+ set -euo pipefail
+ if ! git merge-base --is-ancestor "$GITHUB_SHA" origin/main; then
+ echo "::error::Build commit $GITHUB_SHA is not an ancestor of origin/main — refusing to release"
+ exit 1
+ fi
+ echo "Build commit verified on main: $GITHUB_SHA"
+
- name: Determine version
id: version
env:
@@ -353,6 +367,88 @@ jobs:
echo "Bundle files generated (contain signature + certificate):"
ls -lh sbom-*.bundle
+ - name: Verify SBOM bundles and sanity-check SBOM contents
+ if: steps.check_release.outputs.exists == 'false'
+ env:
+ GH_REPO: ${{ github.repository }}
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ set -euo pipefail
+
+ # Round-trip: prove the README cosign-verify-blob recipe still
+ # works for end users. cosign signs above but never verifies, so a
+ # sigstore/cert-chain regression would silently break downstream
+ # verification — caught only when a user complains.
+ echo "=== Verifying SBOM bundles round-trip ==="
+ for bundle in sbom-spdx.json.bundle sbom-cyclonedx.json.bundle; do
+ json="${bundle%.bundle}"
+ cosign verify-blob \
+ --bundle "$bundle" \
+ --certificate-identity-regexp="^https://github.com/${GH_REPO}/" \
+ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
+ "$json"
+ done
+ echo "Both SBOM bundles verify cleanly"
+
+ # Trivy can emit a near-empty SBOM on partial scan failures
+ # without surfacing a non-zero exit. Set a per-format floor at
+ # 80% of the previous stable release's count; fall back to a hard
+ # floor of 50 when no prior SBOM is fetchable (first release, GH
+ # outage, releases predating SBOM attachment). SPDX and CycloneDX
+ # counts differ (virtual packages, metadata-only entries) so each
+ # format gets its own floor derived from its own prior count.
+ # A cap of 2000 prevents runaway drift if a prior SBOM was
+ # anomalously inflated by a Trivy bug.
+ echo ""
+ echo "=== Sanity-checking SBOM package counts ==="
+ SPDX_COUNT=$(jq '.packages | length' sbom-spdx.json)
+ CYCDX_COUNT=$(jq '.components | length' sbom-cyclonedx.json)
+ echo "Current: SPDX=${SPDX_COUNT}, CycloneDX=${CYCDX_COUNT}"
+
+ HARD_FLOOR=50
+ FLOOR_CAP=2000
+ SPDX_FLOOR="$HARD_FLOOR"
+ CYCDX_FLOOR="$HARD_FLOOR"
+ PREV_TAG=$(gh release list --repo "$GH_REPO" --limit 5 \
+ --json tagName,isPrerelease,isDraft \
+ --jq '[.[] | select(.isPrerelease==false and .isDraft==false)][0].tagName // empty')
+ if [[ -n "$PREV_TAG" ]]; then
+ PREV_DIR=/tmp/prev-sbom
+ rm -rf "$PREV_DIR" && mkdir -p "$PREV_DIR"
+ if gh release download "$PREV_TAG" --repo "$GH_REPO" \
+ --pattern 'sbom-spdx.json' --pattern 'sbom-cyclonedx.json' \
+ --dir "$PREV_DIR" 2>/dev/null; then
+ if [[ -f "$PREV_DIR/sbom-spdx.json" ]]; then
+ PREV_SPDX=$(jq '.packages | length' "$PREV_DIR/sbom-spdx.json")
+ DYN=$(( PREV_SPDX * 80 / 100 ))
+ DYN=$(( DYN < FLOOR_CAP ? DYN : FLOOR_CAP ))
+ [[ "$DYN" -gt "$HARD_FLOOR" ]] && SPDX_FLOOR="$DYN"
+ echo "Prior SPDX: $PREV_SPDX -> floor: $SPDX_FLOOR"
+ fi
+ if [[ -f "$PREV_DIR/sbom-cyclonedx.json" ]]; then
+ PREV_CYCDX=$(jq '.components | length' "$PREV_DIR/sbom-cyclonedx.json")
+ DYN=$(( PREV_CYCDX * 80 / 100 ))
+ DYN=$(( DYN < FLOOR_CAP ? DYN : FLOOR_CAP ))
+ [[ "$DYN" -gt "$HARD_FLOOR" ]] && CYCDX_FLOOR="$DYN"
+ echo "Prior CycloneDX: $PREV_CYCDX -> floor: $CYCDX_FLOOR"
+ fi
+ else
+ echo "No SBOMs in $PREV_TAG (older releases predate SBOM attachment) -> hard floor $HARD_FLOOR"
+ fi
+ else
+ echo "No previous stable release found -> hard floor $HARD_FLOOR"
+ fi
+
+ if [[ "$SPDX_COUNT" -lt "$SPDX_FLOOR" ]]; then
+ echo "::error::SPDX SBOM appears truncated ($SPDX_COUNT < floor $SPDX_FLOOR). Trivy may have failed silently."
+ exit 1
+ fi
+ if [[ "$CYCDX_COUNT" -lt "$CYCDX_FLOOR" ]]; then
+ echo "::error::CycloneDX SBOM appears truncated ($CYCDX_COUNT < floor $CYCDX_FLOOR). Trivy may have failed silently."
+ exit 1
+ fi
+ echo "SBOM counts above per-format floors (SPDX≥$SPDX_FLOOR, CycloneDX≥$CYCDX_FLOOR)"
+
- name: Generate hashes for SLSA provenance
if: steps.check_release.outputs.exists == 'false'
id: hash
diff --git a/docs/architecture/OVERVIEW.md b/docs/architecture/OVERVIEW.md
index bdfc084e77..2a7d8049ba 100644
--- a/docs/architecture/OVERVIEW.md
+++ b/docs/architecture/OVERVIEW.md
@@ -94,12 +94,6 @@ graph LR
- `web/routes/` - Blueprint route handlers
- `web/services/` - Business logic services
-### CLI (`ldr`)
-
-**Location:** `src/local_deep_research/main.py`
-
-Command-line interface for headless research operations.
-
### REST API (`/api/v1`)
**Location:** `src/local_deep_research/web/api.py`
diff --git a/pyproject.toml b/pyproject.toml
index ceae99f376..e6d7aafba6 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -109,7 +109,10 @@ mcp = [
"Bug Tracker" = "https://github.com/LearningCircuit/local-deep-research/issues"
[project.scripts]
-ldr = "local_deep_research.main:main"
+# `ldr` was removed in commit c1f80e7a8 ("Remove deprecated CLI", #319) but the
+# entry-point line was left behind, so the installed binary has crashed on
+# import for every release since. Caught by the entry-point smoke in
+# release-gate.yml. Restore only when a replacement CLI exists.
ldr-web = "local_deep_research.web.app:main"
ldr-mcp = "local_deep_research.mcp:run_server"
diff --git a/src/local_deep_research/web/database/README.md b/src/local_deep_research/web/database/README.md
index 4b0a2d1286..a5f2980a1a 100644
--- a/src/local_deep_research/web/database/README.md
+++ b/src/local_deep_research/web/database/README.md
@@ -14,33 +14,22 @@ The application now follows a "database-first" approach for settings:
2. Settings from TOML files are used only as fallbacks if a setting doesn't exist in the database
3. The web UI settings page modifies the database values directly
-## Migration
+## Legacy migration (historical)
-If you have existing data in the legacy databases, you need to migrate it to the new unified database.
+LDR has since moved to a per-user encrypted SQLCipher database model.
+The original `deep_research.db` + `research_history.db` → `ldr.db`
+migration tooling described in earlier versions of this README has been
+removed along with the deprecated `ldr` CLI. The database split it
+addressed predates the per-user model by several releases, so anyone
+upgrading from a version that old should start fresh rather than
+attempting to recover the legacy data.
+There is no supported migration path from pre-v0.x databases.
-### Automatic Migration
+## Schema upgrades
-When you start the application for the first time after updating, it will check if migration is needed:
-
-1. If legacy databases exist and `ldr.db` doesn't exist, you'll see a warning message
-2. You can run migration using the command: `python -m src.local_deep_research.main --migrate-db`
-3. Alternatively, start the application with auto-migration: `python -m src.local_deep_research.main --auto-migrate`
-
-### Manual Migration
-
-If automatic migration doesn't work, you can:
-
-1. Run the migration script directly: `python -m src.local_deep_research.web.database.migrate_to_ldr_db`
-2. Check migration results in the log output
-
-### Schema Upgrades
-
-If you have already migrated your database but need to update its schema:
-
-1. The application automatically runs schema upgrades on startup
-2. You can manually run schema upgrades with: `python -m src.local_deep_research.main --schema-upgrade`
-3. Current schema upgrades include:
- - Removing the redundant `research_log` table (consolidated into `research_logs`)
+Schema migrations run automatically on application startup via Alembic
+(`src/local_deep_research/database/migrations/`). No manual command is
+required.
## Database Schema
@@ -63,16 +52,9 @@ Cleanup is handled by the `@thread_cleanup` decorator (closes sessions at thread
Note: The core database module is at `src/local_deep_research/database/`, separate from this `web/database/` directory. See [Architecture - Thread & Resource Lifecycle](../../../../docs/architecture.md#thread--resource-lifecycle) for the full cleanup architecture.
-## Rollback
-
-If you need to roll back to the previous database architecture:
-
-1. Keep backup copies of your original `deep_research.db` and `research_history.db` files
-2. In case of issues, restore them and modify the database paths in the code
-
## Troubleshooting
-If you encounter issues with database migration:
+If you encounter database issues:
1. Check the application logs for detailed error messages
2. Ensure you have write permissions to the data directory