Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

26.2 KB
Raw Blame

Release Guide

🚀 Automated Release Process

Releases are fully automated end-to-end whenever a PR that bumps src/local_deep_research/__version__.py is merged to main. No separate tag push, manual workflow trigger, or release-page click is required — the workflow detects the new version, runs all gates, cuts the GitHub release, and (after one approval click) publishes to PyPI and Docker Hub.

PRs that don't touch __version__.py merge normally and do not start the release pipeline at all (a paths filter on the push trigger). If a release run fails, fixes merged afterwards do not retry it: once main is ready, run a rehearsal or start the release manually (see Manual Release Options).

📋 How Releases Work

1. Automatic Release Creation

  • Trigger: Push to main that changes __version__.py to a version that does not yet exist as a GitHub release. In practice this means "merge a PR that bumps __version__.py"; other pushes to main do not start the workflow. Tag pushes (v*.*.*) and manual workflow_dispatch runs also trigger the pipeline and bypass the version-exists check.
  • Version: Read from src/local_deep_research/__version__.py by the version-check job; the tag is v<version>.
  • Release body: Composed by .github/workflows/release.yml from three sources:
    1. AI narrative generated by OpenRouter (vars.AI_MODEL, default moonshotai/kimi-k2-thinking). The model receives the rendered hand-written notes, the auto-generated PR list, every PR's title + body (batched via one GraphQL call), and the diff between the previous release tag and this one (filtered to drop lockfiles, generated docs, SBOM, static assets, and binary patches; capped at 700k chars).
    2. Hand-written notes from docs/release_notes/<version>.md — rendered from contributor-supplied changelog.d/*.md fragments by the workflow itself at release time (see Release-notes flow below). No manual pdm run towncrier build is required before merging the bump.
    3. Auto-generated PR list from GitHub's generate-notes API, label-categorized.
  • Large release notes: GitHub accepts at most 125,000 characters in a release body. If the composed notes exceed that limit, the workflow attaches the exact complete text as release-notes-full.md and links it prominently from the shortened body. Smaller release bodies are left unchanged.
  • No duplicates: If a release for v<version> already exists, the version-check job sets should_release=false and every downstream job (security gate, CI gate, build, publish) is skipped.

2. Approval and Publishing

The release pipeline uses the release GitHub environment to gate the publish steps. DOCKER_USERNAME / DOCKER_PASSWORD are scoped to that environment, so any job that pushes to Docker Hub must declare environment: release and therefore goes through the approval gate.

When you merge to main (or push a tag), the pipeline runs in this order:

  1. Security gates + CI gates run automatically.
  2. build job runs (version pin, SBOM, Sigstore bundles), then provenance job generates SLSA provenance for those artifacts.
  3. One release env approval prompt in release.yml. Approving unlocks all release-env jobs in the same run, which then execute sequentially:
    1. prerelease-docker — canonical multi-arch Docker build, cosign sign, SBOM/SLSA attestations, push as prerelease-v<ver>-<sha> and re-point the floating :prerelease tag.
    2. publish-docker — retags the prerelease manifest as :1.6.9, :1.6, :latest (no rebuild, digest-preserving), then re-verifies digest + cosign + Trivy on the promoted tag.
    3. trigger-pypi — dispatches publish.yml with the release run ID, immutable package artifact ID, commit SHA and manifest digest. The gate builds the wheel and sdist once, records their hashes before anything installs or imports them (and re-checks them after the smoke tests), verifies wheel installation on Python 3.12–3.14 and sdist installation on 3.12, and retains the tested artifacts for 14 days. Publishing verifies and uploads those bytes via repository_dispatch (PyPI Trusted Publishing requires the publish step to run in a top-level workflow, so this can't be a reusable workflow_call).
    4. monitor-pypi — polls publish.yml for completion. The inner polling loop times out after 105 minutes of wall-clock time (after which the job fails): about 45 minutes of automated worst case in publish.yml (including the 30-minute verify-publish window) plus an allowance for approving its release environment. The surrounding GH Actions timeout-minutes is 120 to leave a safety margin around the poll budget.
    5. create-release — publishes the GitHub Release with SBOM/sig/provenance assets. Runs last, gated on all of the above succeeding, so the public Release never points at missing Docker tags or a missing PyPI version.

If any of prerelease-docker, publish-docker, or monitor-pypi fails, create-release is skipped and no public GitHub Release is created. The cleanup-on-rejection job then handles failure-mode cleanup:

  • If publish-docker failed mid-retag (e.g., :1.6.9 landed but :latest failed), cleanup first rechecks that this version has no GitHub Release. It removes shared tags only when their current digest matches the failed build, then removes that build's prerelease tags. Main and tag-triggered release runs use one concurrency group with queue: max: up to 100 runs can wait instead of GitHub's default single pending slot. Once 100 are pending, GitHub cancels additional runs. GitHub processes waiting runs first-in-first-out by when each began waiting, but documents that ordering is not guaranteed. Trade-off: this workflow runs on every push to main, so a burst of merges no longer collapses to one running plus one pending run; each push takes its own serial turn. In a measured 100-run history over 161 hours, 50 runs had been replaced under the old single-pending behavior, while queue: max would have run all 100 (about 66 serial group-hours). Even a version-check-only run held the group for a median of 30 minutes (p90 99), mostly waiting for a runner. After a push burst, a real release run or the workflow_dispatch retry recommended after a failed release can wait behind stale-SHA runs, and when 100 are pending GitHub cancels the newest arrivals rather than the stale ones. To bound this, only pushes that change __version__.py start the workflow, and version-check skips a main-push run whose SHA is no longer main's tip, so only the newest commit runs the gates; tag pushes and workflow_dispatch are never skipped. The release gate's own group is unaffected, since its standalone triggers are rare.
  • Missing digests, failed registry inspections and unexpected HTTP responses for a tag, including unreadable platform images, keep that tag and fail cleanup only after every other tag has been handled, with or without a valid build digest: an uncertain lookup does not prove a tag is absent, safe to delete or unrelated to the failed build. If the GitHub Release lookup or Docker Hub authentication fails, cleanup stops before touching any tag. Investigate the failed check before retrying.
  • A shared tag with a different digest is kept. If it still shares a platform image with the failed build (for example, imagetools create re-encoded the index during promotion), or if it is :<version> itself (no GitHub Release exists for that version, so another build there is an earlier unreleased attempt), cleanup reports it and fails. Other mismatches, such as :latest on an earlier release, are preserved without failing cleanup.
  • If this version already has a GitHub Release, cleanup deletes nothing and succeeds.
  • The build-specific tags (prerelease-v<version>-<sha> and its -amd64/-arm64 tags) are removed even without a build digest. The per-arch tags are pushed before the vulnerability scan and manifest creation finish, so a failed scan or arch build leaves them with no digest.
  • Without a valid build digest, cleanup keeps the shared tags and fails. When a failed scan or arch build stopped the run before prerelease-v<version>-<sha> was created, the shared tags were never moved and retrying cannot help. If that tag existed, cleanup logs its digest before deleting it and reports each shared tag that still points at it, shares its platform images or cannot be read, and :<version> if it points at another build; verify those tags and repair them manually. If the digest could not be read, check whether :prerelease still points at the unsigned build and repair it manually.
  • Signatures, attestations and SBOM artifacts are retained. Their names identify content digests, which historical release tags and other multi-platform manifests may still reference. Tag rollback cannot prove those artifacts are unused; retaining some orphan artifacts preserves verification for existing images.
  • If publish-docker succeeded but a later step (PyPI or create-release) failed, cleanup-on-rejection does NOT fire — Docker release tags exist and their cosign artifacts must stay. See "Recovery from PyPI failure" below.

Cleanup also uses the protected release environment. If its approval is rejected, its credentials are unavailable and no automatic deletion runs. Have a maintainer review the failed run before manual registry recovery.

Activating the shared release queue

Before allowing the first release with the workflow-wide concurrency group to publish, let older Create Release runs finish or cancel them after checking whether they already published artifacts. Older workflow revisions use a per-ref group, so their runs do not share the new lock. Retrying an old run retains that revision; it does not adopt the new queue or cleanup logic. Start new release attempts from a revision containing this change. If an old attempt partially published, use the recovery guidance below before starting another publisher for that version.

When an upstream security finding blocks a release

First find which job blocks. The release exception policy is read only by the two alert checks in the release gate's Check Code Scanning Alerts job. For a finding that only those checks block, the preferred temporary approach while awaiting an upstream fix is a narrowly scoped, expiring release exception approved through a PR. Reviewers must accept the documented risk for the exact finding; without that decision, release stays blocked. This is temporary risk acceptance, and the upstream fix remains required.

  1. Add the exact alert, scanner, category, path, severity, rationale, tracking link, and expiry to the release exception policy.
  2. Review and merge the policy PR before starting a release with that exception. Keep scanners, alert visibility, and release approvals in place. The gate records the decision, and publication rechecks its original expiry.
  3. Track the upstream fix, confirm it resolves the finding, deploy it, rebuild and scan, then remove the exception. If waiting becomes unacceptable, assess another remediation; any extension needs another reviewed policy change.

For such a finding, use this policy for the waiting period instead of dismissing the valid alert, suppressing the finding in the scanner's configuration, bypassing the gate, or carrying a local dependency backport solely to unblock publication.

An exception cannot unblock a scanner job that fails on its own findings. In the gate, Release Gate Summary requires every job to succeed, which covers OSV-Scanner, npm audit, Retire.js, and Gitleaks. During Docker publication, the Trivy scans fail on fixable critical or high image findings and do not read the policy. Each of these keeps its own reviewed suppression, such as an [[IgnoredVulns]] entry with ignoreUntil in osv-scanner.toml or an exp: date in .trivyignore. See Temporary release exceptions for each scanner's suppression and expiry, the matching rules, and the current assessment.

Recovery from PyPI failure (atomicity hole)

The one orphan state the pipeline cannot fully clean up: publish-docker succeeded, PyPI failed. At this point Docker :1.6.9 / :1.6 / :latest exist and are signed; PyPI has nothing; no GitHub Release. monitor-pypi opens a tracking issue labeled ci-cd. To recover:

  1. Inspect the publish.yml workflow run, fix the underlying cause.
  2. Re-run failed jobs in the matching publish.yml run. Its title names the source release run and immutable artifact ID. Re-running preserves the original dispatch payload and tested distributions. If no publisher run was created, re-run the failed release jobs to dispatch it. An expired or deleted package artifact requires a fresh release gate run to build and verify new artifacts before dispatching again. The publisher deliberately has no build step. A dispatch containing only a tag and SHA is insufficient: the run ID, artifact ID and manifest SHA256 are required. Both the publisher's verification job and, after the release approval, its upload job recheck the live release.yml run, so a re-run of either job also rechecks it. If a later release.yml attempt rebuilt the package (re-ran pip Install Verification), the old dispatch is rejected even when only the upload job is re-run; publish the new build from the dispatch sent by that attempt's trigger-pypi job (re-run that attempt's failed jobs if it did not get that far). The publisher also requires the package install checks, release gate summary and build job to have succeeded on that build, not an older one. Reruns that leave pip Install Verification as a carried-over success, such as "Re-run failed jobs" after a later job failed, keep the original build, so its original artifact can still be published. The original security decision and its recorded deadline are verified too. The decision must come from the same originating release run as the tested packages and its latest executed build job. These bindings are checked again after publisher approval. A decision from another run or an earlier build of the same commit is rejected; a carried-over successful build keeps its original decision. A rerun that runs build again, directly or because a job it depends on was re-run, rejects the old dispatch from the start of that attempt; publish from the dispatch its trigger-pypi job sends. Rerunning publisher jobs preserves that decision; an expired or unavailable decision requires a new release run after remediation or a separately reviewed exception update. Recovery cannot override the original deadline.
  3. Once PyPI publishes successfully, re-run monitor-pypi and its dependent jobs in the original release run while the original deadline is valid. The monitor must succeed before create-release can run; retrying only create-release leaves its failed prerequisite unchanged. In the Actions UI, select the failed monitor job and use "Re-run jobs". With the CLI, use gh run rerun <release-run-id> --job <monitor-job-database-id> (obtain that database ID with gh run view <release-run-id> --json jobs). This reruns the monitor and its dependents, including create-release, which checks expiry again immediately before publication. Do not bypass an expired exception with manual publication. Already published Docker or PyPI artifacts are not automatically rolled back when an exception expires.

Earlier iterations of this refactor described a single approval gate with a pre-approval testing window. That design required DOCKER_USERNAME / DOCKER_PASSWORD to be repo-level secrets so the canonical build could run without env approval. They are env-scoped to release instead, so the gate sits in front of the build. The atomicity refactor preserves this single-approval model — one click unlocks the whole chain, and create-release runs last so the "published Release with broken artifacts" failure mode is closed.

👥 Who Can Release

Code owners (defined in .github/CODEOWNERS):

  • @LearningCircuit
  • @hashedviking
  • @djpetti

📝 Release Workflow

For Regular Releases:

  1. Bump version in src/local_deep_research/__version__.py (or merge the auto-bump PR opened by .github/workflows/version_check.yml).
  2. Merge to main → Release automatically created. The release workflow renders fragments from changelog.d/*.md into docs/release_notes/<X.Y.Z>.md, composes the body (AI narrative + rendered changelog + auto PR list), and publishes the GitHub release.
  3. Approve publishing in GitHub Actions (PyPI/Docker).
  4. Merge the cleanup PR opened automatically by the cleanup-changelog job (titled chore: clear changelog fragments for <X.Y.Z>). It persists docs/release_notes/<X.Y.Z>.md and removes the consumed fragments from changelog.d/. Squash-merge — the diff has no review value beyond a sanity check that the rendered notes look right.

When changelog.d/ contains a pending breaking fragment (any file towncrier renders under Breaking Changes, such as <id>.breaking.md, +<slug>.breaking.md or a counter-suffixed <id>.breaking.<n>.md), Version Auto-Bump skips automatic patch PR creation and updates, and on each run closes any open automatic bump PR whose version is not a major release. Review the release scope, then open Actions → Version Auto-Bump → Run workflow on main and explicitly choose major (for example, the FastAPI migration from 1.x to 2.0.0). A manual patch is rejected. A manual minor is accepted only before 1.0 (0.N.0); on 1.x and later the bump job fails before opening a PR.

The release workflow applies the same rule in its build job, before it attests, signs or publishes anything: while breaking fragments are pending it refuses to publish any version other than N.0.0 (or 0.N.0 before 1.0), so a stale patch PR or a handwritten bump cannot ship breaking changes as 1.x.y. scripts/ci/check_breaking_release.py implements both checks. Fragments stay pending until the cleanup PR (step 4) deletes them, so merge it before the next patch release. Review the selected version, release notes and complete release-gate results before merging the bump PR. Running Version Auto-Bump prepares a PR; merging that PR starts the release workflow described above.

To preview the rendered notes locally before merging the bump:

pdm run towncrier build --draft --version <X.Y.Z>

(--draft writes nothing.) Skip if no fragments exist for this release — the workflow tolerates a missing per-version file (warns and proceeds with auto-notes plus AI summary only).

For Hotfixes:

  1. Create hotfix branch from main
  2. Make minimal fix
  3. Bump patch version (e.g., 0.4.3 → 0.4.4)
  4. Fast-track review by code owners
  5. Merge to main → Automatic release

🔧 Manual Release Options

Rehearsal (dry run)

  • Go to Actions → "Create Release" → "Run workflow" and tick dry_run.
  • Runs every release gate (security, CI, E2E, compatibility, …) once on the selected ref, then stops: build and every publishing job are skipped, so nothing is tagged, built, signed or published. A green run means the gates pass on that commit.
  • Use it before merging a version bump, while a major release is being prepared, or after fixing a failed release, so that one run validates main instead of re-running the full gates on every merge.
  • Any branch can be rehearsed; only a real release is limited to main or a version tag.

Option A: Manual Trigger

  • Go to Actions → "Create Release" → "Run workflow" (leave dry_run unticked)
  • No other inputs are required: the workflow reads the version from src/local_deep_research/__version__.py at HEAD. To release an older or different version, use Option B (push a version tag).
  • Dispatch from main, or from the v<version> tag that matches __version__.py at that tag. The build job refuses any other ref, because publish.yml would reject its source run.

Option B: Version Tags

  • git tag v0.4.3 && git push origin v0.4.3
  • Automatically creates release; the workflow uses the tag's commit SHA (not main HEAD), so this is the correct path for backporting.

🛡️ Branch Protection

  • Main branch is protected
  • Required reviews from code owners
  • No direct pushes - only via approved PRs
  • Status checks must pass (CI tests)

📦 Version Numbering

Follow Semantic Versioning:

  • Major (X.0.0): Breaking changes
  • Minor (0.X.0): New features, backward compatible
  • Patch (0.0.X): Bug fixes, backward compatible

🚨 Emergency Procedures

If automation fails, do NOT create a GitHub release through the UI as the first recovery step — under the atomicity refactor, a manually created GitHub release does NOT trigger publish.yml (it listens only on repository_dispatch) and does NOT trigger docker-publish.yml (workflow_call only). The downstream release: listeners that DO fire (backwards-compatibility.yml, sbom.yml) are observability-only.

Recovery, in order of preference:

  1. Check workflow logs in GitHub Actions to identify which job failed, and use the targeted recovery for that failure mode:
    • PyPI failure with Docker already promoted: see Recovery from PyPI failure above.
    • Any other failure: re-run the failed job via the Actions UI if it's still re-runnable (typically within 30 days).
  2. Re-trigger the full pipeline via workflow_dispatch if re-running individual jobs isn't possible. Merging a fix does not restart the release on its own, because only version bumps trigger it; a dry-run rehearsal first confirms the gates pass. Safe for digest-keyed cosign verification — old digests remain valid because their cosign artifacts persist; the new run produces a new digest with its own signatures.
  3. Contact code owners if recovery requires manual Docker Hub or PyPI intervention.

📝 Release-notes flow (towncrier news fragments)

Hand-written release notes are assembled from per-PR fragments using towncrier. This replaces the older shared docs/release_notes/<version>.md model, which broke down at LDR's PR throughput (multiple PRs/day racing for the same file).

Contributor side

Each PR with user-visible behavior change drops one tiny markdown file:

changelog.d/<PR-number>.<category>.md

Categories: breaking, security, feature, bugfix, removal, misc (canonical list lives in [[tool.towncrier.type]] entries in pyproject.toml; the pre-commit hook reads it from there). Orphan fragments (no PR/issue number) use changelog.d/+<slug>.<category>.md. The pre-commit hook (recommend-release-notes) nudges contributors who add ≥20 source lines without a fragment, and validates filenames so a typo'd category doesn't silently vanish at render time. See changelog.d/README.md for the full convention.

Maintainer side

The release workflow handles the render. There is nothing to run manually before merging the version bump.

.github/workflows/release.yml (create-release job):

  1. Sparse-checks-out changelog.d/ + pyproject.toml + requirements/towncrier.txt, then installs towncrier==24.8.0 from the hash-verified lock.
  2. Runs towncrier build --yes --version <X.Y.Z> against the runner's throwaway workspace. Towncrier writes the rendered output to docs/release_notes/<X.Y.Z>.md (per the {version}-templated filename; single_file = false makes this a per-release file rather than appending to a master CHANGELOG) and removes the consumed fragments locally.
  3. Reads the rendered file as input to the AI summary and as the "hand-written notes" section of the published GitHub release body.

Persistence to main happens in the cleanup-changelog job, which re-runs the same render against the release commit (github.sha) and opens a chore/post-release-cleanup-<X.Y.Z> PR with the deletions and the rendered file. Squash-merge it.

If changelog.d/ is empty (maintenance release with no fragments), the render step is skipped cleanly and the release proceeds with the AI narrative + auto PR list only — no hard failure.

Preview without committing

pdm run towncrier build --draft --version <X.Y.Z>

--draft renders to stdout without touching any files or fragments. Useful while iterating on a fragment locally.

📊 Release Checklist

  • Version updated in __version__.py (or auto-bump PR merged)
  • Code owner approval received
  • CI tests passing
  • Blocking security findings fixed, or covered while awaiting upstream fixes by a reviewed, expiring release exception (alert-only findings) or by the failing scanner's own reviewed suppression (see the policy)
  • Merge to main completed
  • Release automatically created
  • PyPI/Docker publishing approved
  • chore: clear changelog fragments for <X.Y.Z> PR merged

Installed dependency vulnerability checks

The release gate audits the exact installed distribution versions after each clean wheel/sdist installation on Python 3.12–3.14, and again after enabling the MCP extra. This complements repository lockfile scanning: pip consumers resolve versions from package metadata independently of the PDM lock.

The audit reuses the verified OSV-Scanner binary and the existing osv-scanner.toml policy. Resolution is disabled for this complete installed inventory, so an unpublished candidate is not looked up on PyPI. Vulnerabilities and incomplete scans block the package job. Each job retains its exact installed name/version inventory, pinned scan input and native SARIF as installed-dependencies-* artifacts for 14 days. Auditing reads distribution metadata without importing application modules or installing audit tools into the environment under test.