~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Release Guide
🚀 Automated Release Process
Releases are fully automated end-to-end whenever a PR that bumps
src/local_deep_research/__version__.py is merged to main. No
separate tag push, manual workflow trigger, or release-page click is
required — the workflow detects the new version, runs all gates, cuts
the GitHub release, and (after one approval click) publishes to PyPI
and Docker Hub.
PRs that don't touch __version__.py merge normally and do not start
the release pipeline at all (a paths filter on the push trigger).
If a release run fails, fixes merged afterwards do not retry it: once
main is ready, run a rehearsal or start the release manually (see
Manual Release Options).
📋 How Releases Work
1. Automatic Release Creation
- Trigger: Push to
mainthat changes__version__.pyto a version that does not yet exist as a GitHub release. In practice this means "merge a PR that bumps__version__.py"; other pushes tomaindo not start the workflow. Tag pushes (v*.*.*) and manualworkflow_dispatchruns also trigger the pipeline and bypass the version-exists check. - Version: Read from
src/local_deep_research/__version__.pyby theversion-checkjob; the tag isv<version>. - Release body: Composed by
.github/workflows/release.ymlfrom three sources:- AI narrative generated by OpenRouter (
vars.AI_MODEL, defaultmoonshotai/kimi-k2-thinking). The model receives the rendered hand-written notes, the auto-generated PR list, every PR's title + body (batched via one GraphQL call), and the diff between the previous release tag and this one (filtered to drop lockfiles, generated docs, SBOM, static assets, and binary patches; capped at 700k chars). - Hand-written notes from
docs/release_notes/<version>.md— rendered from contributor-suppliedchangelog.d/*.mdfragments by the workflow itself at release time (see Release-notes flow below). No manualpdm run towncrier buildis required before merging the bump. - Auto-generated PR list from GitHub's generate-notes API, label-categorized.
- AI narrative generated by OpenRouter (
- Large release notes: GitHub accepts at most 125,000 characters in a
release body. If the composed notes exceed that limit, the workflow attaches
the exact complete text as
release-notes-full.mdand links it prominently from the shortened body. Smaller release bodies are left unchanged. - No duplicates: If a release for
v<version>already exists, theversion-checkjob setsshould_release=falseand every downstream job (security gate, CI gate, build, publish) is skipped.
2. Approval and Publishing
The release pipeline uses the release GitHub environment to gate the
publish steps. DOCKER_USERNAME / DOCKER_PASSWORD are scoped to that
environment, so any job that pushes to Docker Hub must declare
environment: release and therefore goes through the approval gate.
When you merge to main (or push a tag), the pipeline runs in this
order:
- Security gates + CI gates run automatically.
buildjob runs (version pin, SBOM, Sigstore bundles), thenprovenancejob generates SLSA provenance for those artifacts.- One
releaseenv approval prompt inrelease.yml. Approving unlocks all release-env jobs in the same run, which then execute sequentially:prerelease-docker— canonical multi-arch Docker build, cosign sign, SBOM/SLSA attestations, push asprerelease-v<ver>-<sha>and re-point the floating:prereleasetag.publish-docker— retags the prerelease manifest as:1.6.9,:1.6,:latest(no rebuild, digest-preserving), then re-verifies digest + cosign + Trivy on the promoted tag.trigger-pypi— dispatchespublish.ymlwith the release run ID, immutable package artifact ID, commit SHA and manifest digest. The gate builds the wheel and sdist once, records their hashes before anything installs or imports them (and re-checks them after the smoke tests), verifies wheel installation on Python 3.12–3.14 and sdist installation on 3.12, and retains the tested artifacts for 14 days. Publishing verifies and uploads those bytes viarepository_dispatch(PyPI Trusted Publishing requires the publish step to run in a top-level workflow, so this can't be a reusable workflow_call).monitor-pypi— pollspublish.ymlfor completion. The inner polling loop times out after 105 minutes of wall-clock time (after which the job fails): about 45 minutes of automated worst case inpublish.yml(including the 30-minuteverify-publishwindow) plus an allowance for approving itsreleaseenvironment. The surrounding GH Actionstimeout-minutesis 120 to leave a safety margin around the poll budget.create-release— publishes the GitHub Release with SBOM/sig/provenance assets. Runs last, gated on all of the above succeeding, so the public Release never points at missing Docker tags or a missing PyPI version.
If any of prerelease-docker, publish-docker, or monitor-pypi fails,
create-release is skipped and no public GitHub Release is created. The
cleanup-on-rejection job then handles failure-mode cleanup:
- If
publish-dockerfailed mid-retag (e.g.,:1.6.9landed but:latestfailed), cleanup first rechecks that this version has no GitHub Release. It removes shared tags only when their current digest matches the failed build, then removes that build's prerelease tags. Main and tag-triggered release runs use one concurrency group withqueue: max: up to 100 runs can wait instead of GitHub's default single pending slot. Once 100 are pending, GitHub cancels additional runs. GitHub processes waiting runs first-in-first-out by when each began waiting, but documents that ordering is not guaranteed. Trade-off: this workflow runs on every push to main, so a burst of merges no longer collapses to one running plus one pending run; each push takes its own serial turn. In a measured 100-run history over 161 hours, 50 runs had been replaced under the old single-pending behavior, whilequeue: maxwould have run all 100 (about 66 serial group-hours). Even a version-check-only run held the group for a median of 30 minutes (p90 99), mostly waiting for a runner. After a push burst, a real release run or theworkflow_dispatchretry recommended after a failed release can wait behind stale-SHA runs, and when 100 are pending GitHub cancels the newest arrivals rather than the stale ones. To bound this, only pushes that change__version__.pystart the workflow, andversion-checkskips a main-push run whose SHA is no longer main's tip, so only the newest commit runs the gates; tag pushes andworkflow_dispatchare never skipped. The release gate's own group is unaffected, since its standalone triggers are rare. - Missing digests, failed registry inspections and unexpected HTTP responses for a tag, including unreadable platform images, keep that tag and fail cleanup only after every other tag has been handled, with or without a valid build digest: an uncertain lookup does not prove a tag is absent, safe to delete or unrelated to the failed build. If the GitHub Release lookup or Docker Hub authentication fails, cleanup stops before touching any tag. Investigate the failed check before retrying.
- A shared tag with a different digest is kept. If it still shares a
platform image with the failed build (for example,
imagetools createre-encoded the index during promotion), or if it is:<version>itself (no GitHub Release exists for that version, so another build there is an earlier unreleased attempt), cleanup reports it and fails. Other mismatches, such as:lateston an earlier release, are preserved without failing cleanup. - If this version already has a GitHub Release, cleanup deletes nothing and succeeds.
- The build-specific tags (
prerelease-v<version>-<sha>and its-amd64/-arm64tags) are removed even without a build digest. The per-arch tags are pushed before the vulnerability scan and manifest creation finish, so a failed scan or arch build leaves them with no digest. - Without a valid build digest, cleanup keeps the shared tags and fails.
When a failed scan or arch build stopped the run before
prerelease-v<version>-<sha>was created, the shared tags were never moved and retrying cannot help. If that tag existed, cleanup logs its digest before deleting it and reports each shared tag that still points at it, shares its platform images or cannot be read, and:<version>if it points at another build; verify those tags and repair them manually. If the digest could not be read, check whether:prereleasestill points at the unsigned build and repair it manually. - Signatures, attestations and SBOM artifacts are retained. Their names identify content digests, which historical release tags and other multi-platform manifests may still reference. Tag rollback cannot prove those artifacts are unused; retaining some orphan artifacts preserves verification for existing images.
- If
publish-dockersucceeded but a later step (PyPI or create-release) failed,cleanup-on-rejectiondoes NOT fire — Docker release tags exist and their cosign artifacts must stay. See "Recovery from PyPI failure" below.
Cleanup also uses the protected release environment. If its approval is
rejected, its credentials are unavailable and no automatic deletion runs.
Have a maintainer review the failed run before manual registry recovery.
Activating the shared release queue
Before allowing the first release with the workflow-wide concurrency group to
publish, let older Create Release runs finish or cancel them after checking
whether they already published artifacts. Older workflow revisions use a
per-ref group, so their runs do not share the new lock. Retrying an old run
retains that revision; it does not adopt the new queue or cleanup logic.
Start new release attempts from a revision containing this change. If an old
attempt partially published, use the recovery guidance below before starting
another publisher for that version.
When an upstream security finding blocks a release
First find which job blocks. The release exception policy is read only by the
two alert checks in the release gate's Check Code Scanning Alerts job. For a
finding that only those checks block, the preferred temporary approach while
awaiting an upstream fix is a narrowly scoped, expiring release exception
approved through a PR. Reviewers must accept the documented risk for the
exact finding; without that decision, release stays blocked. This is temporary
risk acceptance, and the upstream fix remains required.
- Add the exact alert, scanner, category, path, severity, rationale, tracking link, and expiry to the release exception policy.
- Review and merge the policy PR before starting a release with that exception. Keep scanners, alert visibility, and release approvals in place. The gate records the decision, and publication rechecks its original expiry.
- Track the upstream fix, confirm it resolves the finding, deploy it, rebuild and scan, then remove the exception. If waiting becomes unacceptable, assess another remediation; any extension needs another reviewed policy change.
For such a finding, use this policy for the waiting period instead of dismissing the valid alert, suppressing the finding in the scanner's configuration, bypassing the gate, or carrying a local dependency backport solely to unblock publication.
An exception cannot unblock a scanner job that fails on its own findings. In
the gate, Release Gate Summary requires every job to succeed, which covers
OSV-Scanner, npm audit, Retire.js, and Gitleaks. During Docker publication, the
Trivy scans fail on fixable critical or high image findings and do not read the
policy. Each of these keeps its own reviewed suppression, such as an
[[IgnoredVulns]] entry with ignoreUntil in osv-scanner.toml or an exp:
date in .trivyignore. See
Temporary release exceptions
for each scanner's suppression and expiry, the matching rules, and the current
assessment.
Recovery from PyPI failure (atomicity hole)
The one orphan state the pipeline cannot fully clean up: publish-docker
succeeded, PyPI failed. At this point Docker :1.6.9 / :1.6 /
:latest exist and are signed; PyPI has nothing; no GitHub Release.
monitor-pypi opens a tracking issue labeled ci-cd. To recover:
- Inspect the
publish.ymlworkflow run, fix the underlying cause. - Re-run failed jobs in the matching
publish.ymlrun. Its title names the source release run and immutable artifact ID. Re-running preserves the original dispatch payload and tested distributions. If no publisher run was created, re-run the failed release jobs to dispatch it. An expired or deleted package artifact requires a fresh release gate run to build and verify new artifacts before dispatching again. The publisher deliberately has no build step. A dispatch containing only a tag and SHA is insufficient: the run ID, artifact ID and manifest SHA256 are required. Both the publisher's verification job and, after thereleaseapproval, its upload job recheck the liverelease.ymlrun, so a re-run of either job also rechecks it. If a laterrelease.ymlattempt rebuilt the package (re-ranpip Install Verification), the old dispatch is rejected even when only the upload job is re-run; publish the new build from the dispatch sent by that attempt'strigger-pypijob (re-run that attempt's failed jobs if it did not get that far). The publisher also requires the package install checks, release gate summary andbuildjob to have succeeded on that build, not an older one. Reruns that leavepip Install Verificationas a carried-over success, such as "Re-run failed jobs" after a later job failed, keep the original build, so its original artifact can still be published. The original security decision and its recorded deadline are verified too. The decision must come from the same originating release run as the tested packages and its latest executedbuildjob. These bindings are checked again after publisher approval. A decision from another run or an earlier build of the same commit is rejected; a carried-over successful build keeps its original decision. A rerun that runsbuildagain, directly or because a job it depends on was re-run, rejects the old dispatch from the start of that attempt; publish from the dispatch itstrigger-pypijob sends. Rerunning publisher jobs preserves that decision; an expired or unavailable decision requires a new release run after remediation or a separately reviewed exception update. Recovery cannot override the original deadline. - Once PyPI publishes successfully, re-run
monitor-pypiand its dependent jobs in the original release run while the original deadline is valid. The monitor must succeed beforecreate-releasecan run; retrying onlycreate-releaseleaves its failed prerequisite unchanged. In the Actions UI, select the failed monitor job and use "Re-run jobs". With the CLI, usegh run rerun <release-run-id> --job <monitor-job-database-id>(obtain that database ID withgh run view <release-run-id> --json jobs). This reruns the monitor and its dependents, includingcreate-release, which checks expiry again immediately before publication. Do not bypass an expired exception with manual publication. Already published Docker or PyPI artifacts are not automatically rolled back when an exception expires.
Earlier iterations of this refactor described a single approval gate with a pre-approval testing window. That design required
DOCKER_USERNAME/DOCKER_PASSWORDto be repo-level secrets so the canonical build could run without env approval. They are env-scoped toreleaseinstead, so the gate sits in front of the build. The atomicity refactor preserves this single-approval model — one click unlocks the whole chain, and create-release runs last so the "published Release with broken artifacts" failure mode is closed.
👥 Who Can Release
Code owners (defined in .github/CODEOWNERS):
@LearningCircuit@hashedviking@djpetti
📝 Release Workflow
For Regular Releases:
- Bump version in
src/local_deep_research/__version__.py(or merge the auto-bump PR opened by.github/workflows/version_check.yml). - Merge to main → Release automatically created. The release
workflow renders fragments from
changelog.d/*.mdintodocs/release_notes/<X.Y.Z>.md, composes the body (AI narrative + rendered changelog + auto PR list), and publishes the GitHub release. - Approve publishing in GitHub Actions (PyPI/Docker).
- Merge the cleanup PR opened automatically by the
cleanup-changelogjob (titledchore: clear changelog fragments for <X.Y.Z>). It persistsdocs/release_notes/<X.Y.Z>.mdand removes the consumed fragments fromchangelog.d/. Squash-merge — the diff has no review value beyond a sanity check that the rendered notes look right.
When changelog.d/ contains a pending breaking fragment (any file towncrier
renders under Breaking Changes, such as <id>.breaking.md,
+<slug>.breaking.md or a counter-suffixed <id>.breaking.<n>.md),
Version Auto-Bump skips automatic patch PR creation and updates, and on each run closes
any open automatic bump PR whose version is not a major release. Review the
release scope, then open Actions → Version Auto-Bump → Run workflow on
main and explicitly choose major (for example, the FastAPI migration from
1.x to 2.0.0). A manual patch is rejected. A manual minor is accepted
only before 1.0 (0.N.0); on 1.x and later the bump job fails before opening a
PR.
The release workflow applies the same rule in its build job, before it
attests, signs or publishes anything: while
breaking fragments are pending it refuses to publish any version other than
N.0.0 (or 0.N.0 before 1.0), so a stale patch PR or a handwritten bump
cannot ship breaking changes as 1.x.y. scripts/ci/check_breaking_release.py
implements both checks. Fragments stay pending until the cleanup PR (step 4)
deletes them, so merge it before the next patch release. Review the selected
version, release notes and complete release-gate results before merging the
bump PR. Running Version Auto-Bump prepares a PR; merging that PR starts the
release workflow described above.
To preview the rendered notes locally before merging the bump:
pdm run towncrier build --draft --version <X.Y.Z>
(--draft writes nothing.) Skip if no fragments exist for this release
— the workflow tolerates a missing per-version file (warns and proceeds
with auto-notes plus AI summary only).
For Hotfixes:
- Create hotfix branch from main
- Make minimal fix
- Bump patch version (e.g., 0.4.3 → 0.4.4)
- Fast-track review by code owners
- Merge to main → Automatic release
🔧 Manual Release Options
Rehearsal (dry run)
- Go to Actions → "Create Release" → "Run workflow" and tick dry_run.
- Runs every release gate (security, CI, E2E, compatibility, …) once on
the selected ref, then stops:
buildand every publishing job are skipped, so nothing is tagged, built, signed or published. A green run means the gates pass on that commit. - Use it before merging a version bump, while a major release is being
prepared, or after fixing a failed release, so that one run validates
maininstead of re-running the full gates on every merge. - Any branch can be rehearsed; only a real release is limited to
mainor a version tag.
Option A: Manual Trigger
- Go to Actions → "Create Release" → "Run workflow" (leave dry_run unticked)
- No other inputs are required: the workflow reads the version from
src/local_deep_research/__version__.pyat HEAD. To release an older or different version, use Option B (push a version tag). - Dispatch from
main, or from thev<version>tag that matches__version__.pyat that tag. The build job refuses any other ref, becausepublish.ymlwould reject its source run.
Option B: Version Tags
git tag v0.4.3 && git push origin v0.4.3- Automatically creates release; the workflow uses the tag's commit
SHA (not
mainHEAD), so this is the correct path for backporting.
🛡️ Branch Protection
- Main branch is protected
- Required reviews from code owners
- No direct pushes - only via approved PRs
- Status checks must pass (CI tests)
📦 Version Numbering
Follow Semantic Versioning:
- Major (X.0.0): Breaking changes
- Minor (0.X.0): New features, backward compatible
- Patch (0.0.X): Bug fixes, backward compatible
🚨 Emergency Procedures
If automation fails, do NOT create a GitHub release through the UI as
the first recovery step — under the atomicity refactor, a manually
created GitHub release does NOT trigger publish.yml (it listens only
on repository_dispatch) and does NOT trigger docker-publish.yml
(workflow_call only). The downstream release: listeners that DO fire
(backwards-compatibility.yml, sbom.yml) are observability-only.
Recovery, in order of preference:
- Check workflow logs in GitHub Actions to identify which job
failed, and use the targeted recovery for that failure mode:
- PyPI failure with Docker already promoted: see Recovery from PyPI failure above.
- Any other failure: re-run the failed job via the Actions UI if it's still re-runnable (typically within 30 days).
- Re-trigger the full pipeline via
workflow_dispatchif re-running individual jobs isn't possible. Merging a fix does not restart the release on its own, because only version bumps trigger it; a dry-run rehearsal first confirms the gates pass. Safe for digest-keyed cosign verification — old digests remain valid because their cosign artifacts persist; the new run produces a new digest with its own signatures. - Contact code owners if recovery requires manual Docker Hub or PyPI intervention.
📝 Release-notes flow (towncrier news fragments)
Hand-written release notes are assembled from per-PR fragments using
towncrier. This replaces the older
shared docs/release_notes/<version>.md model, which broke down at
LDR's PR throughput (multiple PRs/day racing for the same file).
Contributor side
Each PR with user-visible behavior change drops one tiny markdown file:
changelog.d/<PR-number>.<category>.md
Categories: breaking, security, feature, bugfix, removal,
misc (canonical list lives in [[tool.towncrier.type]] entries in
pyproject.toml; the pre-commit hook reads it from there). Orphan
fragments (no PR/issue number) use changelog.d/+<slug>.<category>.md.
The pre-commit hook (recommend-release-notes) nudges contributors who
add ≥20 source lines without a fragment, and validates filenames so a
typo'd category doesn't silently vanish at render time. See
changelog.d/README.md for the full
convention.
Maintainer side
The release workflow handles the render. There is nothing to run manually before merging the version bump.
.github/workflows/release.yml (create-release job):
- Sparse-checks-out
changelog.d/+pyproject.toml+requirements/towncrier.txt, then installstowncrier==24.8.0from the hash-verified lock. - Runs
towncrier build --yes --version <X.Y.Z>against the runner's throwaway workspace. Towncrier writes the rendered output todocs/release_notes/<X.Y.Z>.md(per the{version}-templatedfilename;single_file = falsemakes this a per-release file rather than appending to a master CHANGELOG) and removes the consumed fragments locally. - Reads the rendered file as input to the AI summary and as the "hand-written notes" section of the published GitHub release body.
Persistence to main happens in the cleanup-changelog job, which
re-runs the same render against the release commit (github.sha) and
opens a chore/post-release-cleanup-<X.Y.Z> PR with the deletions and
the rendered file. Squash-merge it.
If changelog.d/ is empty (maintenance release with no fragments), the
render step is skipped cleanly and the release proceeds with the AI
narrative + auto PR list only — no hard failure.
Preview without committing
pdm run towncrier build --draft --version <X.Y.Z>
--draft renders to stdout without touching any files or fragments.
Useful while iterating on a fragment locally.
📊 Release Checklist
- Version updated in
__version__.py(or auto-bump PR merged) - Code owner approval received
- CI tests passing
- Blocking security findings fixed, or covered while awaiting upstream fixes by a reviewed, expiring release exception (alert-only findings) or by the failing scanner's own reviewed suppression (see the policy)
- Merge to main completed
- Release automatically created
- PyPI/Docker publishing approved
-
chore: clear changelog fragments for <X.Y.Z>PR merged
Installed dependency vulnerability checks
The release gate audits the exact installed distribution versions after each clean wheel/sdist installation on Python 3.12–3.14, and again after enabling the MCP extra. This complements repository lockfile scanning: pip consumers resolve versions from package metadata independently of the PDM lock.
The audit reuses the verified OSV-Scanner binary and the existing
osv-scanner.toml policy. Resolution is disabled for this complete installed
inventory, so an unpublished candidate is not looked up on PyPI. Vulnerabilities
and incomplete scans block the package
job. Each job retains its exact installed name/version inventory, pinned scan
input and native SARIF as installed-dependencies-* artifacts for 14 days.
Auditing reads distribution metadata without importing application modules or
installing audit tools into the environment under test.